The survey found 32% unsure if they had enough workers, 18% had slightly fewer than they needed and 3% having many fewer than required.
Some 78% of respondents said recruitment had been more difficult than last year, with 20% saying it had been the hardest for years.
Growers blamed the weak pound which has reduced their workers' earning power, as well as uncertainty over Brexit.
About 80,000 seasonal workers a year pick and process British fruit and veg, including many from Eastern Europe.
Seasonal workers are not seeing out a full season, and are more reluctant to return but a new "UK settled status" would grant EU migrants who had lived in the UK for five years rights to stay and access health, education and other benefits. The new Government proposals are dependent on EU states guaranteeing Britons the same rights.
NFU horticulture advisor Amy Gray said: "Not surprisingly, numbers coming here are beginning to be harder work to get access to. We've been predicting this and it has been exacerbated by Brexit."
BPOA vice chairman Kersten Catella, of the Newey Group, added Newey's labour provider Pro-Force has not let the group down but there is "obviously a problem" finding labour and some workers are not seeing their terms out because the rewards are not there. This is disappointing "because we normally rely on returnees and labour to stay out the season", she said. It is an issue across many industries and a top lobbying topic, she added.
The NFU has been running its own surveys on the issue, finding the number of seasonal workers coming to work on British farms has dropped 17%, leaving more than 1,500 unfilled vacancies on British farms in May alone.
The NFU is calling on the Government and newly appointed Defra secretary of state Michael Gove to provide reassurances to growers that there will be clarity on how farms will access a reliable and competent workforce, both today and post-Brexit.
NFU horticulture and potatoes board chairman Ali Capper said: "A lack of clarity regarding the UK’s future relationship with the EU and a weakened sterling has contributed to the reduction in workers on farm now being reported by labour providers who source seasonal workers.
"Farmers and growers need to know how the Government will deal with the need from industries that rely on seasonal workers and the NFU is calling for reassurance farmers will be able to source a reliable and competent workforce both now and in the future.
"Without that, this trend is likely to continue and at this stage in the season any further tightening in the workforce will hit hard on farms."
The survey, covering the period January to May 2017, shows the number of labour providers unable to meet the requirements of the farms they supplied rose four-fold.
The proportion of returnees, which form a sizeable and dependable source of workers, has also dropped significantly throughout the first five months of the year, falling from 65% to 33%, a drop of nearly 50%.
Capper said: "Returnees are absolutely vital. Their past experience and technical ability makes them so valuable and losing them is a big concern.
"This robust survey represents 30%¹ of the total seasonal workforce and it is crucial Government understands the importance of seasonal workers to an industry that provides the raw ingredients to feed the nation."
HTA horticulture head Raoul Curtis-Machin said within ornamental production this year there was no crisis though numbers were down and the "fear for next year is growers are going to struggle and Defra are listening to that and have some kind of draft scheme. I've not seen the detail. We need a quantity of supply."
He said many Europeans had stayed and taken senior roles but there was fear they may leave.
The Ornamentals Round Table group including the HTA is meeting Defra minister George Eustice in early July to discuss the issue.
A Government spokesperson said: "The Government places great value on the UK's food and farming industries, both as a crucial component of the UK economy and of the fabric of rural Britain.
"We are determined to get the best deal for the UK in our negotiations to leave the EU, not least for our world-leading food and farming industry, which is a key part of our nation's economic success."