Gardiner, the former Wisley curator and RHS chief curator, was one of 13 trustees of the Garden Bridge Trust, which is overseeing the £185m project designed by Thomas Heatherwick with a garden by Dan Pearson.
The Garden Bridge Trust (GBT) said: "Jim Gardiner was executive vice president of the RHS and his role as a Trustee of the GBT has been to help and advise on the horticultural aspects of the project, primarily with the planting. Jim retired from the from the RHS on 28 October and has therefore stepped down from our Board. Jim has been an excellent advisor and advocate for the Bridge and we thank him for his hard work. The Trust will look for a replacement to continue providing horticultural advice for this landmark project."
The RHS said: "Jim Gardiner has been executive vice president of the RHS and his role as a trustee of the Garden Bridge Trust has been to help and advise on the horticultural aspects of this project, primarily with the planting.
"On his retirement from the RHS on 28 October he will step down from this role.
"Should the Garden Bridge need any further help and advice on sustainable planting, the RHS will continue to provide horticultural advice, thereby supporting wildlife and the environment, but will not continue in a trustee role.
"Apart from Mr Gardiner's time, the RHS has not invested funds into the Garden Bridge and has never had a role in determining whether or not the project should go ahead. However, should the project proceed, as the UK's gardening charity, we would happily provide the GBT with our horticultural expertise and advice.
"The RHS is focusing on delivering our own £160 million investment in horticulture, including a new fifth garden, RHS Bridgewater in Salford, and increasing community projects across the UK."