The American security establishment calls it “going dark”, and it’s a problem Australian and UK intelligence and law enforcement agencies also claim is a major threat: the steady loss of their ability to monitor communications because we are all shifting from telephones, which are easily wiretapped by governments, to online, where it’s more complicated both legally and technically, and where telecommunications companies need only keep minimal track of communications for billing purposes.

It’s the primary reason why Australia’s intelligence and law enforcement agencies are demanding the retention of internet usage data and a host of other expansions to their powers.

The Obama administration, which has overseen a systematic crackdown on the media, whistleblowers and activists, appears on the verge of endorsing the FBI’s “huge priority” of trying to address going dark by forcing ISPs and other service providers to accommodate a means for law enforcement agencies to routinely monitor internet communications, via huge fines for companies that did not provide such backdoors.

The official line is that those backdoors would only be used with a search warrant. However, there is widespread evidence that the FBI does not believe it needs warrants to surveil internet usage and instead relies on subpoenas, and sometimes not even those.

Since 9/11, the US government has a history of extensive extra-legal internet surveillance. It is now well established that the National Security Agency can search all internet traffic going in, out and through the United States for counter-terrorism purposes. But it is also most likely monitoring the internet usage of all US citizens as well as many people outside the US, via facilities such as its mammoth $2 billion data farm in Utah. Given the recent comments of a former FBI agent, it is possible all internet usage and telecommunications are also being recorded and stored.

There are two issues of relevance for us in all this.

First, the FBI has advanced the same “going dark” logic as the Attorney-General’s Office and agencies like the Australian Federal Police and Australian Security Intelligence Organisation here, that what it seeks is not an extension of existing powers, but an application of existing powers to new technology to prevent agencies from being rendered obsolete as telecommunication practices change. As the FBI’s general counsel said in 2011:

“The Going Dark problem is not about the government having inadequate legal authority—the legal authorities we have for intercepting electronic communications are adequate. Rather, the Going Dark problem is about the government’s practical difficulties in intercepting the communications and related data that courts have authorized it to collect.”

It’s important to understand that this is argument is deeply (and deliberately) misleading. Data retention, for example, is not an extension of existing surveillance powers online that will allow authorities to “keep up” with the march of technology, it’s a radical extension of those powers to enable state surveillance of far more than what we ever used phones for. Most of us now live a huge amount of our lives online; we use the internet to consume media, to work, to play, to buy and sell, to have relationships, to stay in touch with family, for civic engagement and political activity. Under the fig leaf of “just keeping up”, the going dark argument would establish state surveillance of huge parts of our lives, even if it is confined to “telecommunications data” and not the content of our internet usage.

Second, the FBI clearly doesn’t want to go down the route of the NSA in having to spend billions of dollars establishing data farms in the desert. It wants to impose the burden of monitoring usage or storing data on the private sector — thus its preferred approach of wanting to fine service providers that don’t give the Feds the access they want.

AGD here has a similar approach. In its extended and secret discussions with industry about data retention between 2009 and last year (conducted with the full knowledge and support of Labor), it repeatedly tried to raise the issue of companies bearing the cost of storing data, which one firm estimated might cost it tens of millions of dollars. As, one by one, companies pointed out the significant flaws in AGD’s data retention proposal and their implications for cost, the bureaucrats grew increasingly frustrated at their inability to prepare costings. Their attitude was that government would make a decision, “which industry may or may not agree with … The reality is they [industry] have been reminded by law enforcement of the benefits to society of these particular policies.”

Going dark is a more significant issue for Australian and British intelligence and law enforcement agencies, because neither have direct access to the sort of vast surveillance apparatus that the US government has already established to keep its own citizens, and as many of the rest of world’s citizens as convenient, under surveillance; nor do they have a tradition of acting extra-legally in the way that US agencies and the US government itself often have. But in both cases, they have chosen the lazy option of wanting to annex the internet to existing surveillance frameworks, without understanding (or appearing to wilfully misunderstand) how internet usage differs from previous uses of communications technology, or appropriately calculating the balance that is at least theoretically supposed to be maintained between liberty, privacy and security.

British Deputy PM Nick Clegg correctly understood that data retention doesn’t achieve that balance when he rejected the “snoopers’ charter”. Whether politicians here have that understanding remains to be seen.