Joel was born in 1838 in , Hawkins, Tennessee. Joel's father was Gladson and his mother was Lucy. His paternal grandfather was Gladson. He had four brothers and a sister, named Calvin, John, James, Lee and Anna. He was the youngest of the six children.