James was born in 1714. James' father was John Cosby and his mother was Martha Garland. His maternal grandfather was Edward Garland. He had two brothers and four sisters, named John, William, Mary, Amy, Ann and Elizabeth. He was the third oldest of the seven children. He died at the age of 75 in 1789.