Sargeant, who is believed to be Chelsea's first black designer, said: "I don't come across any other black garden designers when I'm out and about. But that doesn't mean black people aren't interested in gardening and design. If there is an issue it is engaging people of other cultures and ethnicities."
The designer, who was born in Tanzania, called on the RHS to do more to encourage people from ethnic minorities to enter the horticulture industry.
She criticised TV gardening shows for being overly "traditional white middle class" in their coverage.
Sargeant, who studied garden design at Middlesex University from 1996-99, said: "The RHS is trying to encourage young people into the industry because there is a widening skills gap. But I don’t think it has even occurred to anybody to think about diversity. I do think the RHS could do more to promote diversity.
The former chairwoman of the Society of Garden Designers, said: "There is a fabulous array of gardening programmes but they do tend to be very traditional white middle-class in their attitude towards gardening.
"What we’re offered is very much the typical English garden that is perhaps not so relevant to people of multiple cultures and ethnicities that live in Britain."
The centrepiece of Sargeant’s garden is an oak, which represents the one politician William Wilberforce sat beneath in 1787 when he resolved to push the abolition of slavery through Parliament. It also includes oak saplings that survivors of slavery, living in a safe house in Southampton, planted in their allotment. The garden is intended to celebrate Parliament pass
ing the Modern Slavery Act last year but also highlight the estimated 13,000 people working as slaves in the UK.
She said: "The garden behind the colourful front doors is black, representing captivity and the idea that modern slavery is hidden behind closed doors. There are also two open oak doors, representing the path to freedom. "I hope visitors will go away understanding that slavery is still going on. As they walk down their streets, there are people being kept working in awful conditions in nail bars and car washes against their will, and if they keep their eyes open they may be in a position to help them by informing the police."
Sargeant added: "I do wonder if perhaps more could be done. I know people always blame the media but more could be done in the media. What we’re offered is very much the typical English garden, but perhaps that is not as relevant to people of multiple cultures and ethnicities that live In Britain.
"There are lots of gardening styles and different ways, growing different food and with different ideas.
"The question in my mind is whether that is being represented and whether we encourage young people of all cultures and ethnicities to come into gardening and design. I suspect not."
Sargeant, a former junior doctor, added: "The whole issue of attracting young people into gardening is an added problem."
She said: "It’s great to see more women at Chelsea this year. I’ve been down in the dumps for the last couple of years about the number of women on main avenue but at Hampton Court it’s great to see lots of new female talent coming through and I’m looking forward to Hampton Court 2015 designers coming to Chelsea in the future. If they don’t then we should be asking questions."
Cleve West, who is Anglo-Indian and is designing for M&G at Chelsea this year, has also questioned the lack of different ethnicities represented in high profile horticulture, but said he did not know what the solution was.
Sargeant is also helping to turn Pimlico’s Orange Square into "Freedom Street", with colourful planting and performances by gospel choirs, for the Chelsea Fringe Festival.
The RHS said: "We hope that there will in future be greater diversity within the professional horticultural industry and as part of our commitment to raise the profile of careers in horticulture, we are looking at how we can address this issue."
Its community projects include the Campaign for School Gardening, which promotes gardening in 28,000 schools. It said: "We hope this will also help us to grow a more diverse new generation of gardeners that better reflects all ethnicities and cultures."