Pearl H.'s father was Milton Mann and her mother was Louisa E. Williams. Her paternal grandfather was Mann; her maternal grandfather was Byrd Williams. She had two sisters, and also two siblings of unknown gender named Allie, Lumie D., Mann and Mann. She had a half-brother, and also two half-siblings of unknown gender named Fletcher, E. G. B. and I. E..