He said: "This fails miserably on availability of labour. We are unhappy with the response we have had to enhance SAWS and have repeated calls to extend the scheme.
"The problem has become locked up in the big policy of immigration and Defra throws its hands in the air and says 'it's not for us, it's a Home Office issue'."
But in a later debate, sustainable food and farming minister Lord Rooker admitted he had overseen the cut in annual SAWS quotas from 25,000 to 16,000 when he was a Home Office minister.
He said: "There's been a failure to understand this is not a net-immigration scheme. Over 100 countries were involved in SAWS. The people go back, many are in higher education. But we are acutely short of evidence in Government that the present policy is causing a major problem. I've read no more than two headlines of crops not being picked on time.
"The evidence needs to be put before the Home Office; we are hamstrung at Defra when we can't bring direct evidence of the effect of SAWS on food production."
He added: "It has to be evidence-based. We are sympathetic, but you need evidence. SAWS works well, but with 27 nations in the EU, the decision was made that we did not really need it."