Lee was born in 1830 in , Hawkins, Tennessee. Lee's father was Gladson and his mother was Lucy. His paternal grandfather was Gladson. He had four brothers and a sister, named Calvin, John, James, Joel and Anna. He was the third youngest of the six children.