James' father was James Mather and his mother was Martha Mills. His paternal grandparents were James Mather and Mary Higham. He had two brothers and three sisters, named William, John, Mary, Ann and Margaret. He was the third youngest of the six known children. He died at the age of 6 in April 1843.