James W. was born in 1850. James W.'s father is James W. Dye and his mother is Mary Jane Allen. His maternal grandfather was Joseph Allen and his maternal grandmother is Nancy Harris. He has a brother and two sisters, named John Taylor, Nancy C. and Martha. He is the second youngest of the four children.