In Inclusion by Design, CABE calls on local authorities, architects and landscape architects to explore the different ways people experience buildings and places.
The report urges local authorities to use planning powers to favour areas most in need - deprived neighbourhoods with fewer local amenities. It notes that open space in such areas is more likely to be poorly managed and maintained.
CABE chief executive Richard Simmons said: "Even though accessibility has improved over the past decade, the fact remains that poor and disadvantaged people are far more likely to live in poor-quality environments."
CABE is now setting up a 20-strong professional group to advise it on inclusive design and equality.
Landscape Institute president Neil Williamson said: "Evidence shows that high-quality green space impacts positively on people's mental health. We need to be giving as much attention to the design of public spaces and green infrastructure as we do to the traditional 'grey infrastructure' of roads and buildings."
The report picks out Spa Fields in Islington - designed by Cracknell - as an example of the benefits of involving the community in public space design.
Cracknell director Phil Heaton said the report could be useful - if people had time to read it. "If you are a tired local authority officer with a big pile of paper on your desk you don't have time to read everything," he said.
"This is another wish list - we need to be offering local authorities practical advice and action plans."
Inclusion by Design notes that the built environment professions need to better mirror the diversity of the society they serve. It highlights the fact that currently, only two per cent of registered architects are black and minority ethnic, and park workers are predominantly white men aged over 40.
The cycle of deprivation in the most needy areas continues because neglected public spaces contribute to the onset of vandalism, antisocial behaviour, graffiti and littering, according to CABE.
"Better understanding of the particular needs of our diverse and differing communities will lead to more inclusive approaches to design in the public realm and will help bring communities together," Williamson said.