There are 14 show gardens scheduled for the 19-23 May show, down from 22 in 2008, after several sponsors pulled out because of the recession.
Gavin, who is appearing at Grand Designs Live at the Excel centre in London from 25 April to 4 May, said: "Five years ago everyone was excited by garden design but Chelsea has lost its way a bit and got silly and I would be a large part of that happening."
The designer's sponsor pulled out this year after the design got "too complicated and expensive".
He added: "It will be a while before I go back to Chelsea. It's a brilliant show but it's getting a bit silly with all the money being spent on it meaning the relevance for visitors and viewers is getting less."
- Meanwhile, designer Sarah Eberle is building three "credit crunch" gardens at Chelsea. See www.HorticultureWeek.co.uk.