The proposed move was orchestrated by Aida Hersham, who bought the Fawley Court estate, where Toad Hall is based, for £13million in 2009 as well as the land at Shiplake.
She wanted to bring the garden centre land back into the estate as part of planned restoration of the historic mansion.
But she did not complete the unspecified cash-plus-land deal before the set deadline of 31 July last year.
The former Wyevale site could become an eyesore, say locals.
Toad Hall owners the Stubbs family had a planning application approved to develop the Shiplake site in 2010, subject to financial contributions being made to Oxfordshire County Council for road improvements but the option to move has now expired.