The garden is now looking for a replacement who can provide a local hands-on approach to the role.
Jolliffe, who has been a board member for 11 years, said: "The garden is in wonderful health, both financial and strategic. Our Director, Huw Francis is doing a fantastic job in galvanising the garden, while our recent trustee recruitment drive has resulted in the best board since I took over the chair from Robin Lewis."
He added: "It is time for both me to move on and time for the garden to move on with a new chair at the helm."
Huw Francis said: "Rob remains a loyal and ardent supporter of the garden. We shall miss his measured and benevolent approach to chairing the board and look forward to seeing him at the garden into the future."
Since opening at the turn of the century, the The National Botanic Garden of Wales has gone from strength to strength. Last month it became the first UK botanic garden to receive a new global accreditation for science and horticultural excellence from the BGCI and in April it reported visitor numbers at a 17-year high in its annual report.