Book records show he ws born in Pelham, NH

Birth records may actually be christening records.

http://archive.org/stream/historichomesins02cran/historichomesins02cran_djvu.txt

Birth record and name of parents:
https://archive.org/stream/vitalrecordsofbr00brad/vitalrecordsofbr00brad_djvu.txt

Massachusetts, Town and Vital Records, 1620 1988
Joseph Bailey
birth 15 Sept 1766
Birth place Gloucester
Father's name ( Bradford Bailey)

Vital Records of Gloucester, Massachusetts, to the End of the Year ..., Volume 1

Record says Joseph was born in Bradford,  probably born Sept. 17, 1766 P.R 53

Family Record in the possession of Miss Mary B.S. Bailey of Lancaster

https://books.google.com/books?id=FHHRp6wRdqkC&pg=PA790&lpg=PA790&dq=Joseph+Bailey+P.R+53&source=bl&ots=guC4_nEoCQ&sig=jBYhbjvTl8W1b7fIZdhUN-e3gPc&hl=en&sa=X&ved=0CB8Q6AEwAGoVChMI8tbX8b6vxwIVSI0NCh0Inw-B#v=onepage&q=P.R%2053&f=false

http://ma-vitalrecords.org/MA/Essex/Gloucester/aBirthsB.shtml

Living in Rockport at 1850 census with son, son's wife and grandkids

https://familysearch.org/ark:/61903/1:1:MD9P-XNW

https://familysearch.org/search/record/results?count=20&query=%2Bgivenname%3ANathaniel~%20%2Bsurname%3ABailey~%20%2Bbirth_place%3A%22New%20Hampshire%22~%20%2Bbirth_year%3A1703-1703~

Massachusetts by 1860 census

Family record now in possession of Miss Mary B.S. Bailey of Lancaster