The AHDB has 220 staff spread across four "inadequate summer house" buildings in Stoneleigh, Warwickshire, and wants to build a £10m headquarters on the site - a building that was originally to be completed this year.
But AWM, which owns Stoneleigh Park, has only paid £1.1m of a promised £4.75m grant and has now withdrawn the rest because of the economic climate.
Attenborough said the £1.1m does not have to be repaid, but added: "This is quite a significant blow because the business case for us relocating was predicated on this grant. We want all R&D and knowledge transfer staff sitting together so they can exchange best practice, which helps funding project development, but that has been delayed."
The AHDB is now looking to lease a building or get a developer to build at Stoneleigh in return for a long lease. It hopes to move by spring 2011 if Defra approves its new business case. All 220 levy-funded staff have relocated to Stoneleigh. The AHDB has 100 more home-based or regional staff.
Defra minister Jim Paice said the planned new building was "too grandiose" and he urged the AHDB to seek a more modest accommodation.