Under proposals, the new tax would be imposed as an up to 3% levy on the value of land.
The annual tax is calculated using the market rental value of land so would hit those with large gardens harder.
The Labour Party’s manifesto, ahead of the June 8 election, pledges "a review into reforming council tax and business rates and consider new options such as a land value tax, to ensure local government has sustainable funding for the long term."
Council Tax is determined using the value of a property on April 1 1991 in England and April 1 2003 in Wales.
A Conservative Party estimate is that this would result in a yearly tax bill of £3,837 for an average family home in England, a 224% increase on the current average council tax bill of £1,185.
The party warned that some people might sell off their gardens to lower their bills and might build over green space.
All-Party Gardening and Horticulture Group secretary Rebecca Pow MP said: "Looking at this at face value, I would urge caution with any policy which might deter people from wanting gardens. It is well documented that taking part in gardening activities and having access to green space are hugely beneficial to health and well being. I would also be concerned that this kind of a policy would target rural areas far more than urban areas, putting an additional strain on peoples pockets, accentuating the urban/rural divide.
"It is right that those who have worked hard to achieve a home with a garden are not then penalised for it and are treated fairly."
Foreign Secretary Boris Johnson said: "Corbyn’s Garden Tax will send tax bills soaring, house prices plummeting, plunge people into negative equity and force families to build over their back gardens."
The Institute for Public Policy Research says the tax might become a "tax on gardens".
The National Farmers Union has warned that if agricultural land, which is currently exempt from council tax and business rates, is also hit it would simply lead to hikes in food prices.
Hayloft Plants' Derek Jarman said it is unlikely Labour will get to implement the tax as they are unlikely to win the election.
Labour said it would not introduce a garden tax, adding: "Labour’s manifesto commits us to making council tax fairer: our plan is to see what needs to be done to ensure that working people don’t pay more. We'll look at how we can make council tax fairer but we’ll not be introducing a separate Land Value Tax on residential property."