Book Historic Homes says abt. 1733. . Town records show they moved to Newbury and were discharged to the church there.

http://archive.org/stream/historichomesins02cran/historichomesins02cran_djvu.txt

Vital Records of Newbury, Massachusetts

unknown

or possibly Ware, NH

Joseph Bailey, believed to be son of Joseph
Bailey (,4), certainly nephew if not son if the family
records are correct, settled at Pelham, New Hamp-
shire. He married Deborah ■. Their children :

Joseph, born in Pelham, New Hampshire, 1766;
Ann, born August 6, 1768; Thomas Hardy, born
September 3, 1771, sailed with John Paul Jones in

"Bon Homme Richard," in the revolutionary war.
The two latter are recorded on the town books, the

iormer is a family record. Charlotte, married

Fox; Elizabeth, married Stephen Knight, February
3, 1788; Samuel, resided at Newburyport. 

Book records show he ws born in Pelham, NH

Birth records may actually be christening records.

http://archive.org/stream/historichomesins02cran/historichomesins02cran_djvu.txt

Birth record and name of parents:
https://archive.org/stream/vitalrecordsofbr00brad/vitalrecordsofbr00brad_djvu.txt

Massachusetts, Town and Vital Records, 1620 1988
Joseph Bailey
birth 15 Sept 1766
Birth place Gloucester
Father's name ( Bradford Bailey)

Vital Records of Gloucester, Massachusetts, to the End of the Year ..., Volume 1

Record says Joseph was born in Bradford,  probably born Sept. 17, 1766 P.R 53

Family Record in the possession of Miss Mary B.S. Bailey of Lancaster

https://books.google.com/books?id=FHHRp6wRdqkC&pg=PA790&lpg=PA790&dq=Joseph+Bailey+P.R+53&source=bl&ots=guC4_nEoCQ&sig=jBYhbjvTl8W1b7fIZdhUN-e3gPc&hl=en&sa=X&ved=0CB8Q6AEwAGoVChMI8tbX8b6vxwIVSI0NCh0Inw-B#v=onepage&q=P.R%2053&f=false

http://ma-vitalrecords.org/MA/Essex/Gloucester/aBirthsB.shtml

Living in Rockport at 1850 census with son, son's wife and grandkids

https://familysearch.org/ark:/61903/1:1:MD9P-XNW

https://familysearch.org/search/record/results?count=20&query=%2Bgivenname%3ANathaniel~%20%2Bsurname%3ABailey~%20%2Bbirth_place%3A%22New%20Hampshire%22~%20%2Bbirth_year%3A1703-1703~