Much of the early suburban development of Hither Green in the years around 1800 was made possible by the piecemeal enclosure of wasteland bordering Hither Green Lane.

In 1788 Lord Dartmouth leased the site on which two semi-detached houses, later called the Elms and the Chesnuts, were built. The early tenants included shopkeepers from Leadenhall Street and The Strand, and a Bermondsey vinegar manufacturer.

The Elms later became the St. Mary's Mission House It was demolished around 1891 to provide part of the site for St. Swithun's Church. The other house, known as the Chestnuts, became the vicarage in 1886 and is now the oldest building in Hither Green.