In his new autobiography How the Boy Next Door Turned Out, the Home Front gardener writes that RHS shows representative Hayley Monckton told him last year: "You are the one that matters to us. There are other names, but most people have never heard of them."
Gavin is set for another complicated RHS Chelsea Flower Show in 2011, after high-profile dramas with designers Bunny Guinness and Andy Sturgeon as well as the judges and the BBC at previous shows.
Gavin, who cites Crug Farm Nursery, Tendercare and Architectural Plants as his favourite suppliers, says he was offered a spot on the "wrong side" of the main avenue at Chelsea in 2010, so exhibiting was "not worthwhile".
But still chasing his first Chelsea gold, he adds: "We are talking to Chelsea about a garden for next year. Our plans are out of this world, quite literally - a garden that hangs in the sky, a garden suspended over one of the world's greatest cities. If I can make this happen, I'll be delighted. It will reflect so much of what gardens can have - beauty, excitement, technology and humour. A traditional recipe for garden builders everywhere, brought up to date."
The designer says there is a "40 per cent chance" he will be at the show in May 2011. He adds: "To design this garden for Chelsea ... it needs me and I'm not around a lot. I don't know whether I need Chelsea any more, but I love lots of it. It may be indulgent of me to do it. It's more to do with ego."
Gavin is no longer gardening on UK television and has hit out in his book about the standard of programmes in the genre, arguing that "gardening TV is stuffed with crashing bores. It's banal."
He explains: "It's bland. I don't think anyone would argue with that. A lot of gardeners might like that - not every gardener wants to be cutting-edge or told about the latest new thing. They might not want ostentation and I was responsible for a lot of ostentation."
But Gavin does praise presenters Monty Don, James Alexander-Sinclair, Jane Owen, Rachel de Thames, James Wong and Matthew Wilson.
He denies any suggestion that his gardening style is now out of fashion. "I don't think so because I'm very busy," he says. "You change and mature. TV people associate me with that time but that's fine. When I started gardening there was very little choice available other than classic and twee and I had a role to play."
In 2007, Gavin agreed to advertise Westland's West+ peat-reduced compost, but lost his Chelsea Flower Show presenting job as a result because BBC rules do not allow presenters to advertise products in the area about which they are presenting.
He admits: "I created a bit of a mess. I said yes to Westland but I didn't tell my agent and it went viral." Gavin says he kept his word to advertise because "a deal is a deal" and "I was a little bit bored with coverage of Chelsea, too. It was not a programme where I could innovate. I have no regrets."
Alan Titchmarsh nearly fell into the same trap this year by fronting a B&Q campaign. Gavin says he learnt his lesson from "someone older" than Titchmarsh - Percy Thrower, who lost a BBC contract in 1975 after advertising for ICI. "You have to have rules and I don't think Alan breached them."
Gavin adds that his spat at Chelsea 2004 with designer Bunny Guinness, about his wall neighbouring her show garden being too high, made him "livid" and now "she wouldn't look favourably at what I'm doing but I don't care".
A run-in with Andy Sturgeon in 2006 about plagiarism and poaching staff has been settled, but Gavin says "what I really wanted to write, I wasn't allowed". He refuses to say whether they talk now.
The RHS suggests Gavin sees his public profile as "David versus Goliath". He is shocked: "I was full of wonder in my first and second year at Chelsea, but now if anything it's me versus myself."
Gavin is now designing worldwide, particularly at Iberian country clubs, and has bases in Dublin and Harrow. He is writing a murder mystery novel set at Chelsea featuring a character based on RHS shows development director Bob Sweet, as well as working on Irish TV programmes Dirty Old Town - about cleaning up Irish towns with volunteers - and Designing Public Spaces.
At the moment, he is perhaps best-known on television for leading the Morrisons Let's Grow voucher collection campaign, which donates gardening products to schools.
He says: "I have a little girl myself and have worked in many countries like the townships in South Africa where there was no food. To front the campaign is a real privilege."
He adds: "People ask me because they know I'll be myself. Until I was 29 or 30 no-one asked me to do anything in my life." His legacy, he believes, is that he was part of a group - including David Stevens, Paul Cooper, Martha Schwartz and Stephen Woodhams - that gave "more choice" to "suburban" people who were looking for garden designs.
1980s: Formed garden design school that became Dublin School of Garden
1980s to date: Owner, Diarmuid Gavin Designs
1991 & 1993: Gold award, Royal Dublin Society
1995: Bronze, Chelsea Flower Show
2000s: Television presenter, Gardeners' World, Home Front
2004: National Lottery garden, Chelsea; danced in Strictly Come Dancing
2005: Hanover Quay garden, Chelsea
2006: Semi-finalist, Only Fools on Horses
2007: Silver gilt, Chelsea Flower Show
2008: Bronze, Chelsea Flower Show
2008 to date: Let's Grow campaigner, Morrisons
2010: Autobiography published, How the Boy Next Door Turned Out