Built in 1772 as a country house, the Manor House was owned from 1796 by the banking family of Baring.

After 1850, when the area was becoming built up, they let the house rather than live there.

The house became a public library in 1902 and the garden a public park, to make an entrance to which the service wing and stables on the right were demolished.