But one person at least is more than ready to jump to the society's defence, trumpeting its achievements and defending flaws: Lady Christine Skelmersdale, who picked up the exclusive Victoria Medal of Honour (VMH) at this year's RHS Hampton Court Palace Flower Show.
With almost 40 years' experience of the country's biggest gardening charity, few are better placed to offer an insider's verdict on recent tribulations.
"I get so cross with people always criticising the RHS - it is like hitting an auntie," she says. "It is an institution and like any big old institution it has failings, but it does work very hard to try to correct them and not enough has been said about its excellent work. We gardeners benefit enormously from the RHS, from its advisory service and the plants and from the opportunity to go to the fantastic flower shows."
Lady Skelmersdale has been exhibiting at Chelsea since 1972 when, together with her husband, she bought Broadleigh Gardens. She has held various posts with the RHS, including member of the RHS Council, vice-chairman of the Daffodil & Tulip Committee, member of the Daffodil & Tulip Year Book Committee, member of the Iris Committee, and the Gardens Committee's special adviser, or "godmother" to Rosemoor, during which time she oversaw a major redevelopment.
She has only praise for departing chief executive Inga Grimsey, but is keen to see a new face at the helm soon.
"Whoever replaces Inga must have experience of running an organisation - it is a business. You don't want a gardener to run it. It has a huge budget and it is a huge business. As a trustee, I would expect the person running it to be a business person. They don't even need to know about plants, though someone with a love of them would be great.
"The criticism about the cuts is unfair. People don't seem to realise (Inga) increased staff by 10 per cent, so it is not a cut at all, really."
She is, however, less enthusiastic about the way Chelsea has evolved over the years. Not, she is at pains to point out, because of the RHS, but rather as the type of people visiting the show has changed. It has become increasingly difficult to "sell" at the RHS' flagship show and costs are constantly on the rise.
"Chelsea is getting a little disappointing now. It used to be proper gardeners, but now I would say it is probably 90 per cent 'shufflers', who come for the spectacle. If I were to change anything, it would probably be to have a smaller stand. It is very expensive to do the whole thing these days, with overtime and hotels and everything. But the problems with the show are social; there is nothing the RHS could do about it."
One thing the RHS can and does do, she explains, is continue to offer a world-class forum for advice and education, bringing together experts from right across the spectrum.
The emphasis on education is clearly an important point for Lady Skelmersdale, who grew up wanting to be a lecturer, despite gardening from as young as five. After a stint teaching geography in Zambia, however, she dedicated her life to bulbs at Broadleigh, where she still works today. Though she does find time to incorporate some education, travelling across the country lecturing on bulbs and gardening tips and delivering travelogues.
"I must have lectured to hundreds of thousands of people over the years and I think (the VMH) is partly in recognition of that. I never thought I would win it. It is one of those things for everyone in the broader range of horticulture. I enjoy the lecturing and the travelling immensely. It is so important to see plants in the wild - you can learn so much from it. It has changed the way I grow a lot of things, seeing them in the wild."
But Broadleigh is her bread and butter. The company is a household name in bulb-growing and has been synonymous with the mail-order retail of small bulbs for over 30 years. Yet change could be on the horizon. Lady Skelmersdale intends to retire within five years and a current staff shortage has presented an "opportunity to re-examine what we do".
Whether that means selling the business or changing its offering has not yet been decided but, whatever happens, there will be more Chelseas, more travel and more passion for plants. And more writing, though the publication of her new book - a gardener's guide to growing bulbs - has been delayed to spring 2011.
"I would like to do more gardening before I am too old, as well as more writing and lecturing," says Lady Skelmersdale. "I want to walk in more mountains and do more travelling, visit London more and spend more time with my husband. So Broadleigh will change. In an ideal world I would like to find someone to buy it and carry it on, but I haven't decided how it is all going to happen."
1966-70: Studies geography and education at University of Nottingham
1972: Buys Broadleigh Bulbs
1990-2001: Member, RHS Council
1991-2002 Vice-chairman, RHS Daffodil & Tulip Committee
1992-2004: Member, RHS gardening committee with special responsibility for Rosemoor
1998-2005: President, Friends of the University of Bristol Botanic Gardens.