Meet Dan DeLaMare-Lyon. Dan currently works for Juniper Networks in Cambridge. But he used to work for Endace Europe, a network monitoring company. In 2007, Dan was Endace’s representative spruiking its wares in the cyber security market. His presentation has now found its way online courtesy of WikiLeaks’s Spy Files collation, a massive collection of resources from within the cyber security and surveillance industry.

According to Dan’s conference presentation, Endace is able to offer a virtual one-stop shop for service providers who want to know what’s going on within their networks, for “information security”, for “service assurance” and for “lawful intercept” — both “targeting known criminals” and “broader intelligence gathering in the interests of national security”.

You’ll see a lot of references to “LI” or lawful interception through these documents, as if the industry is careful to insist that it would never be involved in anything not strictly lawful. Of course, what’s lawful depends entirely on which country you’re in. Siemens, for example, in its presentation to the ISS Dubai conference in 2007, pitched not a network monitoring system but a behavioural analytical tool that would enable Middle Eastern régimes to “track all major contact persons of the main militant group in the country using this (sic) data sources: e-mail, fax, SMS, phone calls, bank transfer data, flight booking details, credit card records”.

“Siemens Intelligence Platform”, the company boasted, “is designed for organisations related to Law Enforcement, Government, Other investigative organisation (sic)” — without explaining who the “other investigative organisation” might be. The ISS Dubai conference is an annual affair. The next one is in February, and it features presentations on mobile phone spyware, deep packet inspection, finding ways around HTTPS encryption and “web mass interception”. The public and media are banned from some sessions.

Many of the documents provided in the Spy Files, and much of the content at ISS Dubai, revolves around trying to address the basic problem that surveillance is no longer a matter of attaching alligator clips to a phone line, and the frustration of governments and their security and intelligence services about the range of ostensibly anonymous communication platforms available to their citizens. Cyber security providers approach the problem from two angles.

The first is to uncloak anonymous individuals by constructing behaviour-based profiles based on internet and other communications usage. This, incidentally, is similar to the technique Aaron Barr claims to have used in his inordinately successful effort to uncover the leaders of Anonymous. Jean-Philippe Lion of Qosmos made a presentation to ISS Prague in 2009, in which he asked the pertinent question: “How do you accurately identify targets across multiple applications, multiple physical locations, multiple terminals and multiple identities?”

Qosmos proposed a multi-step process involving tracking all user “suspected” IDs, linking them to an IP address and then “intercepting all traffic from virtual IDs and link to physical person”, then identify everyone they’re connected with. “Summary: It Is Possible To Accurately Identify Targets!” Lion declared at the end of his presentation. Californian “communications forensics” company SS8 proposed a “social network analysis” module targeted not so much at profiling an anonymous user but at identifying the key players in a social network, to facilitate disruption of the networks: “The individuals with high closeness scores are key from a targeting perspective as removing them from the network is likely to break the network into isolated groups, thus preventing the network from communication and operating as designed.”

But compared to other firms, those techniques are absurdly subtle. The second approach to the problem of use of anonymous or encrypted communications platforms is to monitor a user’s activities end-to-end. German firm Digitask pitches the case that too much intelligence is being lost by people using internet-based applications and offers “stealth software” installed on a target’s computer to log and decrypt everything the target does. Earlier this year, the company admitted to selling the malware to the Bavarian government.

Digitask also offers an off-the-shelf wifi catcher that can intercept all wireless traffic from a target — either by locating the device “undercover on public hotspots by bringing just a small receiver unit close to the target” or by using a directional antenna from further away.

But even those techniques look needlessly nuanced compared to that offered by French firm Amesys, which in a presentation to the Prague ISS dismissed “lawful interception” as insufficient and urged the advantages of its own system of “massive interception” — the interception of all internet traffic and its retention for up to one year to enable “global search and surveillance” and a “global synthesized view”.

Or for that matter there’s defence contractor Thales (which is a key player in Australian defence procurement), which promotes a military signal gathering and analysis tool for use against civilian targets. As Dan delaMare-Lyon asked, why sample when you monitor all network traffic inexpensively?

The Spy Files material caps a year that has seen the cyber security industry exposed, for the first time, to detailed scrutiny of its highly-intrusive methods and its links with governments, via the extensive revelations of HBGary Federal’s activities and material unearthed in the wake of Arab Spring uprisings that have detailed the role of western companies in providing mass surveillance equipment to Middle Eastern dictatorships. It is an industry that faces virtually no regulation by government, but instead enjoys close links with the public sector, often via a revolving door between company boards and government, intelligence or defence agencies.

The ostensible purposes of such tools is to target criminals and terrorists, even if the definition of “criminals” by many of the clients of such companies is highly inclusive. But tomorrow, we’ll see how similar tools are being used on all of us, and what we know and don’t know about how much happens of this mass surveillance happens in Australia.