Holly Hedge House was built c.1743 - 1744 on an encroachment on the Heath originally established around a windmill.

It was doubled in size when the section on the right in this painting was added early in the nineteenth century for Edward Legge, Vicar of Lewisham and Bishop of Oxford.

In 1888 the Volunteer Rifles took it over and in the two World Wars it became a centre for recruiting.

The house was pulled down in 1946 following bomb damage, but the site is still used by the Army.