The group, which comprise
s 18 local authorities that joined forces to fight the project, wrote to transport secretary Justine Greening asking her to backtrack on her approval in January of the £33bn line from London to Birmingham.
Speaking on behalf of 51m, Buckinghamshire County Council leader MartinTett said the group would seek a judicial review if she did not rescind her approval.
Last week HS2, the company behind the rail scheme, launched environmental survey work on its impact. Wildlife groups argue it will destroy swathes of green space.
"We would far rather the Government had listened to the people of this country who have decisively rejected this massively expensive project," said Tett.
The group is called 51m because HS2 will cost each parliamentary constituency £51m, said the organisation.
"Areas in the Midlands and the north of England risk being bypassed and left to decline by HS2. We should be investing in our existing rail and road infrastructure."
Ray Puddifoot, vice-chairman of 51m and Leader of the London Borough of Hillingdon, said the consultation process was unfair and inadequate.
"Ordinary people whose lives and livelihoods will be severely affected were not even given an adequate chance to have their say.
"The whole project represents extremely poor value for money and it is right that we challenge the Government’s decision to progress with this misguided scheme," he said.
Puddifoot added that 51m was not against the principle of high-speed rail and a better alternative was double capacity of the current West Coast Mainline – a tenth of the cost.
The 18 authorities of 51m include Aylesbury Vale District Council, Chiltern District Council, Oxfordshire County Council, Warwick District Council and Leicestershire County Council.