This area, formerly part of Guilford and known as East Guilford, was first settled about 1650 on land bought from the Nehantic and Mohegan Indians in 1641. With the population increasing, settlers here sought separate parishes. The Society of East Guilford was incorporated in 1707 and the Society in North Madison, called North Bristol, in 1753.

The Town of Madison, named for President James Madison, was incorporated in 1826.

This was once a center for fishing, shipping, shipbuilding, farming, and crayon manufacturing. Famous people born here include Thomas Chittenden, first Governor of Vermont; philanthropist Daniel Hand; artist Gilbert Munger; and Cornelius Scranton Bushnell  the chief sponsor of the Civil War ironclad warship USS Monitor.