The Pagoda was built in 1760 as a summer house for the Earl of Cardigan and was later used by Caroline, Princess of Wales.

It was enlarged in the mid 19th century and became a London County Council children's home a century later. It is now a private house again.

Its extensive grounds were built on in the 1860s. Though architecturally unusual, its history is similar to other large houses nearby, such as the Knoll.