Speaking at a Conservative Party conference fringe event, Paterson said:
"The most angry people I met in my time at Defra, by far, were the fruit and vegetable growers who were being deprived of a scheme called the [Seasonal] Agricultural Workers Scheme by the existing Prime Minister when she was home secretary," he said.
"And this is a skilled job. You've got to get up early in the morning and pick strawberries, you can't squash them - and brutally, our locals won't do it. "
Paterson, who argued in favour of Brexit during the referendum campaign, said that once the UK was fully in control of immigration it could introduce a permit system tailored to industries' needs, adding: "We won't be short of fruit pickers, we can take whoever we want from all around the world - we won't be short of eye surgeons, we'll take the very best, the most skilled. We'll be completely colour blind, we'll ignore all passports."
Current Defra secretary Andrea Leadsom earlier said at the conference that more seasonal farm work should be undertaken by British workers.