Governor Joseph was born in 1647. Governor Joseph's father was Thomas Dudley and his mother was Katherine Deighton. His paternal grandparents were Roger Dudley and Suzanne Thorne; his maternal grandparents were John Deighton and Jane Basset. He had a sibling of unknown gender named Deborah. He was the younger of the two children. He died at the age of 73 in 1720.