Writing in his column for the Society of Garden Designers' Garden Design Journal, Richardson said one perception of the profession is of well-off, middle-class, middle-aged women who take up garden design but do not need to work to survive.
He added that he does not think there is anything "necessarily wrong with this model" and said the term is bandied about by "basically jealous" designers who have to charge a professional rate.
But he called on the society to "get its house in order" and sort out qualification and accreditation as well as publish professional rates like the Landscape Institute.
At the Society of Garden Designers (SGD) Awards (24 January), attended by Richardson, host James Alexander Sinclair made several barbed references to the column.
SGD chair Juliet Sargeant said the diverse routes into garden design led to a "rich mix of experience and background unique to the SGD" and resulting in "an organisation of unparalleled character, creativity and vibrancy".
She added: "The SGD welcomes people who work flexible hours, ladies and even those who eat lunch, provided that they are committed to providing excellent service through continuing development and high professional standards."
Sargeant argued that garden designers and landscapers in general "suffer from the misunderstanding that their work is simple, easy and unendingly enjoyable, so they do not require proper remuneration" - something that she said is at least beginning to change.
After the awards event, Richardson said: "Nearly everyone seems to understand that I am not attacking 'ladies who lunch' - or indeed 'women' generally - but trying to initiate discussion about the reality of the public and professional image of garden design."