Jasper was born about 1758. Jasper's father was William Terry and his mother was Rachel Manson. He had two brothers and four sisters, named John, Miles, Susannah, Marcy, Rachel and Jemima. He was the oldest of the seven children. He had a half-brother and a half-sister, named William and Mary.