Documents obtained under freedom of information reveal the Attorney-General’s Department was further advanced in its preparations for a data retention régime than previously disclosed, and had a wider concept of the data to be retained than eventually accepted by the government.

Data retention is one of 44 national security reform proposals currently under consideration by the Joint Committee on Intelligence and Security. The definition of the data to be retained under the proposal has been a vexed one for both the department, which came under fire from the committee for the poor quality of the paper outlining the reform proposals, and former attorney-general Nicola Roxon. She was forced to clarify that the proposal related to limited “traffic data” rather than any data that would reveal sites visited by individuals.

Brendan Molloy of the Pirate Party this week obtained several documents relating to AGD’s secret consultations with the telecommunications and internet industries in 2009 to 2012. While heavily redacted in parts, the documents afford a glimpse of the department’s proposal and the mindset of the bureaucrats who gathered to discuss it.

The documents cover three separate processes: the industry consultation process that commenced in mid-2009 and which was still going as late as May 2012 when Roxon referred the issue to JCIS; the ongoing interdepartmental discussion of data retention conducted by the Telecommunications Experts Group, an industry consultation co-chaired by AGD and the Department of Broadband and including the AFP, ACMA and (despite a redaction) ASIO; and a joint departmental-industry “privacy forum” in 2011 that dealt with aspects of data retention.

The documents make clear that AGD originally had a much wider conception of data to be retained. In the initial consultation document circulated for the first meeting in November 2009 by Catherine Smith, the SES Band 1 officer who led the consultations, AGD specifies that telecommunications data includes “the identity of the sending and receiving parties”, which necessarily includes destination addresses for each IP address. The purpose of the data is to enable agencies to “trace all communications from end to end”.

Over the subsequent two years, with industry consultations increasingly focusing on costs and industry consistently saying cost estimates would depend heavily on both the retention model and what data had to be retained, AGD appeared to shift position to focus retention on whatever data carriage service providers were already retaining for business purposes, in order to minimise costs.

However, the department was careful to retain the big stick of compulsion. After one unidentified industry figure complained that data retention for business purposes changed regularly, reflecting the changing products that customers might switch to, and that industry should be compensated for having to retain data unnecessarily, Smith told them bluntly “the reality is that government puts forward proposals which industry may or may not agree with … the reality is they have been reminded by law enforcement of the benefits to society of these particular policies (child safety, protection etc).”

At least we know it’s not only politicians who play the child p-rn card.

Hitherto, we knew that AGD had drawn up draft legislation for data retention. The documents reveal that AGD was well advanced in preparing a Regulatory Impact Statement for the proposals, and in fact had begun preparing one as far back as 2009. The preparation of a RIS, which must be signed off by what is now called the Office of Best Practice Regulation and which includes an estimate of the likely financial impact on business, is a key step before a proposal goes to cabinet, and a RIS must accompany all legislation.

Indeed, the notes from 2009 suggest AGD wanted to move quickly on the proposal, with industry only given a short time to comment on the discussion paper, and bureaucrats discussing options if industry didn’t have time to respond comprehensively; for reasons unclear but perhaps electoral, the whole process then virtually ground to a halt in 2010 before resuming again the following year. The department was still trying to put together a RIS as late as February 2012; at another industry forum in February last year, an AGD director explained how different variables would affect the RIS’s costing of the impact on business of the proposal.

The documents also indicate the government knew all along that AGD was engaged in developing a data retention proposal and workshopping it with industry; Smith told a meeting of the Telecommunications Experts Group in June 2009 that she’d already written to the Attorney-General and (presumably via the AG) to the Prime Minister seeking approval to consult with industry on data retention.

There was considerable industry pushback on the logistics of data retention. After two years of discussion, in 2011, the parties were still at loggerheads over costings, with industry saying they couldn’t be clear about costs until AGD specified exactly what they wanted retained. One industry figure explained that IP addresses were increasingly useless as an identifier given the mobile nature of internet usage and suggested MAC addresses (misunderstood as MAP addresses by the department) as preferable. Another pointed out that privacy concerns about a single centralised data storage were “insurmountable” (AGD had initially proposed both centralised and decentralised data repositories).

But the notes from the discussions between bureaucrats almost have a “banality of evil” tone to them, as public servants discuss a substantial government assault on privacy. Only one, ACMA member Chris Cheah (previously a long-time senior official from the Department of Broadband and a veteran of the telecoms policy), raised the obvious question: when was AGD planning to reveal its proposal and consult with the public, he asked at the start of the process in September 2009. AGD SES Band 2 officer Geoff McDonald replied that there’d be no public consultation until they’d finished developing a technical model and data sets.

Cheah’s question was right on the money: AGD never consulted with the public about data retention, not for another three years, until Roxon handballed the issue to JCIS for a public inquiry even as the department was still trying to get industry to commit to cost impacts so it could move the process forward within government.

And even after all that time, AGD still hadn’t finalised its “technical model and data sets”. Or so it has insisted to both the public and the committee.