"For the gardener on the street, it is going to impact on the cost of gardening," he said. "Plants and sundries will be more expensive as we are a net importer.
"Scotland doesn't punch above its weight horticulturally. We have a great heritage in gardens but lack contemporary gardens open to the public such as Eden and Alnwick. The best gardens are hidden behind the closed doors of the wealthy. The Calyx (a planned national garden for Scotland) never happened."
Fraser said the HTA had meetings at the Scottish Parliament this summer ahead of the vote and an HTA Scottish regional manager would be installed at the Royal Botanic Garden Edinburgh by Christmas.
He added that horticulture is "on the back foot lobbying the Scottish Government behind NGOs and lobby groups" and he is worried that bodies such as Scottish National Heritage would gain influence and lobby for increased stipulation of only "native obsessions" such as juniper, heather, birch, rowan and Scots pines, many of which are "inappropriate outside the Highlands".
On the positive side, Fraser said the domestic industry might expand because it would be easier to deal with.
"Yes" campaigners said the sector could have better representation through the Scottish Parliament, in the same way the Republic of Ireland has a "more credible" horticulture offer through Bord Bia and Bord na Mona. They added that Scotland got a raw deal from Westminster in the 1980s and 1990s and the situation has improved only since the Scottish Parliament first met in 1999.