James was born in 1844.1 James' father is Henry Skaggs and his mother was Betsy Perry. His paternal grandfather was William Skaggs and his paternal grandmother is Mary Polly Pierce. He had two brothers and a sister, named Perry, Jeremiah and Nancy. He is the second youngest of the four children.