James was born in 1827 in , Of Hawkins, Tennessee. James' father was Gladson and his mother was Lucy. His paternal grandfather was Gladson. He had four brothers and a sister, named Calvin, John, Lee, Joel and Anna. He was the third oldest of the six children.