Joseph was born in 1923. Joseph's father was Watkins Harris and his mother was Alice Mae Morris. His maternal grandparents were Israel Morris and Rose Hampson. He had a brother and a sister, named Jack and Alice. He was the second oldest of the three known children. He died at the age of 58 in 1981.