Milburn was born on August 7th, 1805. Milburn's father was Archibald Whitt and his mother is Hannah Lowe. His paternal grandparents were Richard Whitt and Susannah Skaggs. He had a brother named Abijah. He was the younger of the two children. He died at the age of 57 on February 17th, 1863.