Lord Taylor, who was director of Taylor's Bulbs until he became a minister in 2010, looked after R&D, Kew and domestic forestry as part of his under secretary roles.
Secretary of state Caroline Spelman and farming minister Jim Paice also left Defra as part of a reshuffle. Owen Paterson replaced Spelman, David Heath took over from Paice and Lord de Mauley replaced Lord Taylor. Richard Benyon kept his Parliamentary under secretary role.
HTA policy manager Gary Scroby said: "We're sorry to see him go and a little bit surprised. We will miss his knowledge of the industry."
He welcomed Paterson's appointment, adding: "He seems to have a reasonable amount of knowledge around agriculture and rural issues." The HTA will look to meet him at forthcoming All-Party Parliamentary Gardening & Horticulture Group events to talk about "big issues".
Paterson was the Conservative shadow agriculture, fisheries and food minister from 2003 to 2005.