The Elms was built as a farmhouse, probably in the 1780s, after the partial demolition of the Tudor mansion known as Place House, where the farmer had lived until then.

The bay windows were added c.1870.

The gradual break-up of the old Place House farm after 1850 made much of western Catford available for building.