The European Union clearly has its problems. Big problems. Not just the obvious economic troubles that currently dominate the headlines, but more deep-seated issues to do with its “democratic deficit” — the way that union has been managed by elites and unaccountable bureaucrats, often running ahead of public opinion and alienating those whose support it will need in a crisis.

So its easy to understand the support in some circles for David Cameron’s refusal to travel further down the road of EU integration – a stance that he calls “protect[ing] Britain’s national interest”: “I am absolutely clear that it is possible to be a both a full, committed and influential member of the EU but to stay out of arrangements where they do not protect our interests.”

Many of those supporters claim to be all in favor of Europe as a single market, or at least a free trade zone, but say that the EU has gone too far and needs to be reined in. Critics of the EU in Australia and the US are especially prone to take this line. They see themselves quite genuinely as free marketeers and internationalists; their problem is with bureaucracy, over-regulation and lack of accountability.

And yet this position, which one might call “moderate euroscepticism”, is at best naive and at worst deeply disingenuous. Ironically enough, it is itself an elite position. The mass of eurosceptic opinion is quite different: nationalistic, protectionist, hostile in principle to free trade and economic integration.

On the continent, this has never really been in doubt. Although its roots are in mundane agreements about coal and steel, the core of the EU project has always been political – it’s about overcoming the national rivalries that have time and again plunged Europe into bloodshed. Its opponents are the defenders of old-fashioned nationalism; for them, cosmopolitanism is the enemy.

Jean-Pierre Chevènement, a presidential candidate and leader of the far right in France, is representative of the type – he reacted to the latest agreement by calling for a referendum, to let the French people “decide if we must accept the abandonment of budgetary sovereignty as well as of monetary sovereignty”.

Across the continent it’s the same story: the most trenchant opponents of the EU are the nativists of both right and left, the unreconstructed supporters of import controls, restricted immigration, “national” industries and ethnic pride. For them, capitalism is at best a necessary evil.

It’s perfectly coherent to argue, as the moderates do, that a certain amount of integration will serve the purpose of peace and prosperity, but that going too far is counter-productive — there may be a point at which further “deepening” of the EU becomes harmful, and that point may indeed have already been reached (although personally I am not convinced of this).

But that is quite different from arguing that the whole project was misguided in the first place. Yet the one argument has a disturbing habit of segueing into the other, and if you scratch a moderate eurosceptic you often find a deeper hostility lurking underneath.

Check out, for example, Conservative MP and internet sensation David Hannan, who has impeccable free-market credentials. He attacks the single currency as imperial overreach, but once he warms to the theme it becomes clear that he thinks the EU was flawed from the very start, that it was an anti-democratic project whose founders were “unapologetic about vesting supreme power in the hands of appointed commissioners who were to be invulnerable to public opinion”.

And there is an element of truth in that. But we can’t go back to the 1950s; the options for now are either to kill the EU or to live with it and try to reform it. The solution to the democratic deficit is more democracy, and it makes sense that the supporters of a democratic EU — like Cameron’s deputy, Liberal Democrat leader Nick Clegg, and his continental counterparts — are also supporters of integration.

And for all its problems, the EU’s achievements are remarkable. Joshua Goldstein last month said that: “In historical perspective, this is such a monumental, gigantic accomplishment that one wonders how we all have come to take it for granted.” Critics forget how much work has gone into getting where we are today, and how grim the alternatives are.

Hannan and the rest may think it’s possible to somehow junk the EU and start again with a more liberal model, but most people will recognise that as sheer fantasy. For all practical purposes, hostility to the EU plays into the hands of those whose objection is not so much to regulation and bureaucracy, but to free trade and free movement of people. They are the eurosceptics who have a real constituency, and Cameron is playing with fire when he tries to appease them.

The EU has many faults, but it is hated much more for its virtues. Its critics need to keep that in mind.