Gardens have been hardest hit by rain and competition from the Olympics with the gardens and leisure sector seeing a 21.3 per cent fall in visitors in London.
Members in the rest of England saw an average fall of 4.7 per cent across 2,000 tourist sites, according to the Association of Leading Visitor Attractions (ALVA).
Historic Houses Association (HHA) president Edward Harley, speaking at the group's annual event in London last week, said a "significant decline in visitor numbers" meant some of his members have extended their visiting season this year.
He told delegates that a "tough economic environment" has hit events and spends at weddings, while the weather has reduced attendances across his 300 members' houses.
After a "disastrous visiting year for many in the HHA through cancellations and bad weather", Levens Hall's Susie Bagot questioned whether there will be any ongoing benefit from the Olympics legacy.
Harley and Heritage Alliance chairman Loyd Grossman said there would, while ALVA chief executive Bernard Donoghue said the "economic benefits for tourism of hosting the Olympics would be long-term rather than short-term".
He added: "The appalling weather during much of the year has led to one of the worst trading periods since 2001 and foot and mouth. For London attractions the Olympic period was one of their worst trading periods in living memory and for visitor attractions the summer is their equivalent of retailers' Christmas - once lost, the business can't be won back."
Also speaking at the HHA event, Professional Gardeners' Guild chairman Tony Arnold said members had questioned whether the rain or the Olympics led to lower visitor numbers at gardens, though he believed rain was more to blame this year.
He said gardens need to diversify to bring in greater numbers through adding playgrounds and exhibitions. "Garden visiting is becoming quite an expensive day out with the increasing cost of transport, admissions, food and guide books."
Harley noted that 2016 marks the 300th anniversary of Capability Brown's birth and HHA members own half of Brown's existing landscapes, which will mean many celebrations.
Guild launches DVD
The Professional Gardeners' Guild launched a five-minute DVD at the annual Historic Houses Association event in London last week.
It features gardeners such as Sam Youd, friends of the guild including Ken Thompson and John Sales, landowners such as Lord Ashbrook and students talking about the guild's benefits. See www.pgg.org.uk/ about-the-pgg.
