He said: "Like any tourist attraction we’re very worried about what we can do."
Barter added that the RHS was "taking all precautions" to protect the public from risk.
Barter said there were "a couple" of nests found in 2015 adding that because Surrey is so full of oaks the problem is difficult to control.
He said teams were checking Wisley each day and will be eradicating the pests when they see them.
OPM came to the UK in 2005 and has since spread in the south east. Barter said tree surgeon arisings could be to blame for spread and that they should be burnt immeditately if affected. The pests can cash rashes on skin.
The Forestry Commission said: "They are emerging in oak trees in the affected areas of London, Surrey and Berkshire. The first ones were reported on Friday 8 April 2016, and carefully controlled treatment of affected trees with approved insecticide to kill the caterpillars is under way. The unusually cold weather of late April slowed their development."