Patsey was born in 1773. Patsey's father was Ison Walton and her mother was Elizabeth Wall. Her paternal grandfather was Edward Walton. She had two brothers and three sisters, named Matthew Pate, John, Jane, Ann and Mary. She was the second oldest of the six children.