They literally named it SKYNET

[Elaro]

http://arstechnica.co.uk/security/2016/ ... nt-people/

RISK ASSESSMENT / SECURITY & HACKTIVISM
The NSA’s SKYNET program may be killing thousands of innocent people
"Ridiculously optimistic" machine learning algorithm is "completely bullshit," says expert.

by Christian Grothoff & J.M. Porup - Feb 16, 2016 3:35am EST
Share Tweet Email
53

An MQ-9 Reaper sits on the tarmac.
In 2014, the former director of both the CIA and NSA proclaimed that "we kill people based on metadata." Now, a new examination of previously published Snowden documents suggests that many of those people may have been innocent.

Last year, The Intercept published documents detailing the NSA's SKYNET programme. According to the documents, SKYNET engages in mass surveillance of Pakistan's mobile phone network, and then uses a machine learning algorithm on the cellular network metadata of 55 million people to try and rate each person's likelihood of being a terrorist.

Patrick Ball—a data scientist and the director of research at the Human Rights Data Analysis Group—who has previously given expert testimony before war crimes tribunals, described the NSA's methods as "ridiculously optimistic" and "completely bullshit." A flaw in how the NSA trains SKYNET's machine learning algorithm to analyse cellular metadata, Ball told Ars, makes the results scientifically unsound.

Somewhere between 2,500 and 4,000 people have been killed by drone strikes in Pakistan since 2004, and most of them were classified by the US government as "extremists," the Bureau of Investigative Journalism reported. Based on the classification date of "20070108" on one of the SKYNET slide decks (which themselves appear to date from 2011 and 2012), the machine learning program may have been in development as early as 2007.

In the years that have followed, thousands of innocent people in Pakistan may have been mislabelled as terrorists by that "scientifically unsound" algorithm, possibly resulting in their untimely demise.

The siren song of big data

SKYNET works like a typical modern Big Data business application. The program collects metadata and stores it on NSA cloud servers, extracts relevant information, and then applies machine learning to identify leads for a targeted campaign. Except instead of trying to sell the targets something, this campaign, given the overall business focus of the US government in Pakistan, likely involves another branch of the US government—the CIA or military—that executes their "Find-Fix-Finish" strategy using Predator drones and on-the-ground death squads.


Enlarge / From GSM metadata, we can measure aspects of each selector's pattern-of-life, social network, and travel behaviour
In addition to processing logged cellular phone call data (so-called "DNR" or Dialled Number Recognition data, such as time, duration, who called whom, etc.), SKYNET also collects user location, allowing for the creation of detailed travel profiles. Turning off a mobile phone gets flagged as an attempt to evade mass surveillance. Users who swap SIM cards, naively believing this will prevent tracking, also get flagged (the ESN/MEID/IMEI burned into the handset makes the phone trackable across multiple SIM cards).


Enlarge / Travel patterns, behaviour-based analytics, and other "enrichments" are used to analyse the bulk metadata for terroristiness.
Even handset swapping gets detected and flagged, the slides boast. Such detection, we can only speculate (since the slides do not go into detail on this point), is probably based on the fact that other metadata, such as user location in the real world and social network, remain unchanged.

Given the complete set of metadata, SKYNET pieces together people's typical daily routines—who travels together, have shared contacts, stay overnight with friends, visit other countries, or move permanently. Overall, the slides indicate, the NSA machine learning algorithm uses more than 80 different properties to rate people on their terroristiness.

The program, the slides tell us, is based on the assumption that the behaviour of terrorists differs significantly from that of ordinary citizens with respect to some of these properties. However, as The Intercept's exposé last year made clear, the highest rated target according to this machine learning program was Ahmad Zaidan, Al-Jazeera's long-time bureau chief in Islamabad.


Enlarge / The highest scoring selector who travelled to Peshawar and Lahore is "PROB AHMED ZAIDAN", Al-Jazeera's long-time bureau chief in Islamabad.
As The Intercept reported, Zaidan frequently travels to regions with known terrorist activity in order to interview insurgents and report the news. But rather than questioning the machine learning that produced such a bizarre result, the NSA engineers behind the algorithm instead trumpeted Zaidan as an example of a SKYNET success in their in-house presentation, including a slide that labelled Zaidan as a "MEMBER OF AL-QA'IDA."

Feeding the machine

Training a machine learning algorithm is like training a Bayesian spam filter: you feed it known spam and known non-spam. From these "ground truths" the algorithm learns how to filter spam correctly.

In the same way, a critical part of the SKYNET program is feeding the machine learning algorithm "known terrorists" in order to teach the algorithm to spot similar profiles.

The problem is that there are relatively few "known terrorists" to feed the algorithm, and real terrorists are unlikely to answer a hypothetical NSA survey into the matter. The internal NSA documents suggest that SKYNET uses a set of "known couriers" as ground truths, and assumes by default the rest of the population is innocent.

Pakistan has a population of around 192 million people, with about 120 million cellular handsets in use at the end of 2012, when the SKYNET presentation was made. The NSA analysed 55 million of those mobile phone records. Given 80 variables on 55 million Pakistani mobile phone users, there is obviously far too much data to make sense of manually. So like any Big Data application, the NSA uses machine learning as an aid—or perhaps a substitute, the slides do not say—for human reason and judgement.

SKYNET's classification algorithm analyses the metadata and ground truths, and then produces a score for each individual based on their metadata. The objective is to assign high scores to real terrorists and low scores to the rest of the innocent population.


Enlarge / A sample travel report produced by SKYNET
To do this, the SKYNET algorithm uses the random forest algorithm, commonly used for this kind of Big Data application. Indeed, the UK's GCHQ also appears to use similar machine learning methods, as new Snowden docs published last week indicate. "It seems the technique of choice when it comes to machine learning is Random Decision Forests," George Danezis, associate professor of Security and Privacy Engineering at University College London, wrote in a blog post analysing the released documents.

The random forest method uses random subsets of the training data to create a "forest" of decision "trees," and then combines those by averaging the predictions from the individual trees. SKYNET's algorithm takes the 80 properties of each cellphone user and assigns them a numerical score—just like a spam filter.

SKYNET then selects a threshold value above which a cellphone user is classified as a "terrorist." The slides present the evaluation results when the threshold is set to a 50 percent false negative rate. At this rate, half of the people who would be classified as "terrorists" are instead classified as innocent, in order to keep the number of false positives—innocents falsely classified as "terrorists"—as low as possible.

False positives

We can't be sure, of course, that the 50 percent false negative rate chosen for this presentation is the same threshold used to generate the final kill list. Regardless, the problem of what to do with innocent false positives remains.

"The reason they're doing this," Ball explained, "is because the fewer false negatives they have, the more false positives they're certain to have. It's not symmetric: there are so many true negatives that lowering the threshold in order to reduce the false negatives by 1 will mean accepting many thousands of additional false positives. Hence this decision."


Enlarge / Statistical algorithms are able to find the couriers at very low false alarm rates, if we're allowed to miss half of them
One NSA slide brags, "Statistical algorithms are able to find the couriers at very low false alarm rates, if we're allowed to miss half of them."

But just how low is the NSA's idea of "very low"?

"Completely bullshit"

The problem, Ball told Ars, is how the NSA trains the algorithm with ground truths.

The NSA evaluates the SKYNET program using a subset of 100,000 randomly selected people (identified by their MSIDN/MSI pairs of their mobile phones), and a a known group of seven terrorists. The NSA then trained the learning algorithm by feeding it six of the terrorists and tasking SKYNET to find the seventh. This data provides the percentages for false positives in the slide above.

"First, there are very few 'known terrorists' to use to train and test the model," Ball said. "If they are using the same records to train the model as they are using to test the model, their assessment of the fit is completely bullshit. The usual practice is to hold some of the data out of the training process so that the test includes records the model has never seen before. Without this step, their classification fit assessment is ridiculously optimistic."

The reason is that the 100,000 citizens were selected at random, while the seven terrorists are from a known cluster. Under the random selection of a tiny subset of less than 0.1 percent of the total population, the density of the social graph of the citizens is massively reduced, while the "terrorist" cluster remains strongly interconnected. Scientifically-sound statistical analysis would have required the NSA to mix the terrorists into the population set before random selection of a subset—but this is not practical due to their tiny number.

This may sound like a mere academic problem, but, Ball said, is in fact highly damaging to the quality of the results, and thus ultimately to the accuracy of the classification and assassination of people as "terrorists." A quality evaluation is especially important in this case, as the random forest method is known to overfit its training sets, producing results that are overly optimistic. The NSA's analysis thus does not provide a good indicator of the quality of the method.


Enlarge / A false positive rate of 0.18 percent across 55 million people would mean 99,000 innocents mislabelled as "terrorists"
If 50 percent of the false negatives (actual "terrorists") are allowed to survive, the NSA's false positive rate of 0.18 percent would still mean thousands of innocents misclassified as "terrorists" and potentially killed. Even the NSA's most optimistic result, the 0.008 percent false positive rate, would still result in many innocent people dying.

"On the slide with the false positive rates, note the final line that says '+ Anchory Selectors,'" Danezis told Ars. "This is key, and the figures are unreported... if you apply a classifier with a false-positive rate of 0.18 percent to a population of 55 million you are indeed likely to kill thousands of innocent people. [0.18 percent of 55 million = 99,000]. If however you apply it to a population where you already expect a very high prevalence of 'terrorism'—because for example they are in the two-hop neighbourhood of a number of people of interest—then the prior goes up and you will kill fewer innocent people."

Besides the obvious objection of how many innocent people it is ever acceptable to kill, this also assumes there are a lot of terrorists to identify. "We know that the 'true terrorist' proportion of the full population is very small," Ball pointed out. "As Cory [Doctorow] says, if this were not true, we would all be dead already. Therefore a small false positive rate will lead to misidentification of lots of people as terrorists."

"The larger point," Ball added, "is that the model will totally overlook 'true terrorists' who are statistically different from the 'true terrorists' used to train the model."

In most cases, a failure rate of 0.008% would be great...

The 0.008 percent false positive rate would be remarkably low for traditional business applications. This kind of rate is acceptable where the consequences are displaying an ad to the wrong person, or charging someone a premium price by accident. However, even 0.008 percent of the Pakistani population still corresponds to 15,000 people potentially being misclassified as "terrorists" and targeted by the military—not to mention innocent bystanders or first responders who happen to get in the way.

Security guru Bruce Schneier agreed. "Government uses of big data are inherently different from corporate uses," he told Ars. "The accuracy requirements mean that the same technology doesn't work. If Google makes a mistake, people see an ad for a car they don't want to buy. If the government makes a mistake, they kill innocents."

Killing civilians is forbidden by the Geneva Convention, to which the United States is a signatory. Many facts about the SKYNET program remain unknown, however. For instance, is SKYNET a closed loop system, or do analysts review each mobile phone user's profile before condemning them to death based on metadata? Are efforts made to capture these suspected "terrorists" and put them on trial? How can the US government be sure it is not killing innocent people, given the apparent flaws in the machine learning algorithm on which that kill list is based?

"On whether the use of SKYNET is a war crime, I defer to lawyers," Ball said. "It's bad science, that's for damn sure, because classification is inherently probabilistic. If you're going to condemn someone to death, usually we have a 'beyond a reasonable doubt' standard, which is not at all the case when you're talking about people with 'probable terrorist' scores anywhere near the threshold. And that's assuming that the classifier works in the first place, which I doubt because there simply aren't enough positive cases of known terrorists for the random forest to get a good model of them."

The leaked NSA slide decks offer strong evidence that thousands of innocent people are being labelled as terrorists; what happens after that, we don't know. We don't have the full picture, nor is the NSA likely to fill in the gaps for us. (We repeatedly sought comment from the NSA for this story, but at the time of publishing it had not responded.)

Algorithms increasingly rule our lives. It's a small step from applying SKYNET logic to look for "terrorists" in Pakistan to applying the same logic domestically to look for "drug dealers" or "protesters" or just people who disagree with the state. Killing people "based on metadata," as Hayden said, is easy to ignore when it happens far away in a foreign land. But what happens when SKYNET gets turned on us—assuming it hasn't been already?

No comment

[bilateralrope]

The NSA is collecting all that data but has no idea what to do with it. Is anyone surprised ?

That simplifies the privacy vs security argument around NSA data collection. They can't increase security, so there is nothing to gain from losing privacy.

[K. A. Pital]

Skynet? Seriously? I mean... Do people at NSA watch movies at all?

[Ace Pace]

Skynet? Seriously? I mean... Do people at NSA watch movies at all?

Actually, yes. And these code names are mostly for fun, because they assume people who get to hear the code names are either "part of the gang" or higher ups who know this is how people let off steam.
Sometimes, you need a fun hangout at 2AM to think up of code names. And you're going to make up silly ones, stupid ones. With an understanding that if it's in poor taste but not stupidly so, it'll be confirmed.

[Dartzap]

Typical Yanks, always remaking the originals

[K. A. Pital]

I think Keith Alexander was a big Trek fan, or something, right? Sometimes I think it is encouraging that assasination agents are also people with their own little fan thingies.

But sometimes I think it's not very funny, too.

[Zixinus]

Sometimes I get the impression that a great deal of war on terror exists to just have a war. I have this terrible feeling that I'm not wrong. Articles like this do not help those impressions and feelings.

I think Keith Alexander was a big Trek fan, or something, right? Sometimes I think it is encouraging that assasination agents are also people with their own little fan thingies.

But sometimes I think it's not very funny, too.

There are hardly any excesses of the most crazed psychopath that cannot easily be duplicated by a normal kindly family man who just comes in to work every day and has a job to do.

[Purple]

That needs to be the next quote of the week.

[Benny the Ball]

Sometimes I get the impression that a great deal of war on terror exists to just have a war. I have this terrible feeling that I'm not wrong. Articles like this do not help those impressions and feelings.

Yep, in all likelihood you're not wrong at all

[Benny the Ball]

snip

Whoops, I didn't catch that the video poster apparently disabled playback on other sites. In that case:
http://www.youtube.com/watch?v=V3zxBhndFdo
http://www.youtube.com/watch?v=0b2gmcD79vM

[Ace Pace]

I think Keith Alexander was a big Trek fan, or something, right? Sometimes I think it is encouraging that assasination agents are also people with their own little fan thingies.

But sometimes I think it's not very funny, too.

To people outside the system, it's hard to not look at it critically, or to think it's odd or that there's some big evil design. Far more likely is, as I said, someone reccomended and no one said no. I recall atleast one instance where someone (only very late in the approval process), vetoed Pedophile as a code word. Multiple internet memes are real code words. People think up stupid "inside jokes".

It's just people, who are working very hard and trying to let off steam. Usually....

[K. A. Pital]

I know, I did not think about malice, more like "huh? People who kill other people goofing around?".

[Grumman]

Maybe it's just me, but I don't think the questions "What would a Nazi supervillain do?" and "What are the NSA and CIA collaborating on this time?" should ever have the same answer.

[Ace Pace]

I know, I did not think about malice, more like "huh? People who kill other people goofing around?".

The same person can be perfectly serious discussing actual operation, and goof around in a canteen five minutes later. Or even a minute later.

[Starglider]

To a large extent this is the typical technically incompetent but legally astute big contractor IT scam. It just happens to be completely broken terrorist detection instead of completely broken converged healthcare databases etc. The relevant machine learning algorithms only work well when there is a large set of training data, that is a decent representative sample of the actual population so that the algorithm doesn't key on irrelevant survey selection factors. The people writing the code know this, but their managers know that the customers can easily be blinded with a buzzword storm and convinced to splash cash around to try and achieve all the impossible things big data vendors claim. Yes there are lives on the line, but really if they didn't have this dubious bit of IT suggesting targets it's not like the relevant agencies would sit back and admit defeat, they'd likely be using an even more dubious methodology.

[Simon_Jester]

Thing is, ten years ago the US wasn't launching massed drone strikes directed against individual citizens over these nations, and I don't think the lack of anything like a Predator drone was the reason.

While this particular automated 'terrorist detector' isn't a necessary condition for the current policy of drone war... I'm pretty sure SOME such system is. In which case it is very reasonable to criticize the military-industrial complex for first creating such an unreliable and potentially murderous tool, and then being dumb enough to rely on it, when there is no obvious evidence that it is even necessary to be relying on it.

[Starglider]

I'm pretty sure SOME such system is. In which case it is very reasonable to criticize the military-industrial complex for first creating such an unreliable and potentially murderous tool, and then being dumb enough to rely on it, when there is no obvious evidence that it is even necessary to be relying on it.

Well they should always be criticised for wasting money and blowing up innocent people, but whatever ratio of outright con to blind optimism led to this system, I don't think the US military would be blowing up less people or wasting less money if it didn't exist. They'd just be relying on extremely dubious conventional SIGINT or HUMINT instead.

[Zixinus]

I think part of what horrifies people about this isn't just the the blowing up innocent people part. It's the part where they do it on the judgement (or hint, as the article states that they do not know precisely how reliant they are on this system) of blind, idiot machines instead of people that can be demonized or punished. The idea that the agency (and blame) needed to get an innocent person killed can be, even just partially, put on a machine is disturbing. As is the idea that blame for the crimes can be absolved by placing it on a machine.

[Simon_Jester]

I'm pretty sure SOME such system is. In which case it is very reasonable to criticize the military-industrial complex for first creating such an unreliable and potentially murderous tool, and then being dumb enough to rely on it, when there is no obvious evidence that it is even necessary to be relying on it.

Well they should always be criticised for wasting money and blowing up innocent people, but whatever ratio of outright con to blind optimism led to this system, I don't think the US military would be blowing up less people or wasting less money if it didn't exist. They'd just be relying on extremely dubious conventional SIGINT or HUMINT instead.

Normally I'd agree.

But I think in this case, honestly, they might be blowing up less people because this is an entire category of capability that just didn't exist for launching "precise" warfare in someone else's country. Without some way to 'crunch' this kind of dataset with the pretense of reliability, they'd probably have to rely on some system that would give fewer names. Innocent people might well still be dying, but at a reduced rate.

Because when you automate the decision "what constitutes a target?" it will inevitably result in the manufacture of more targets for less effort, just as with anything else that can be automated.

[Sea Skimmer]

Well they should always be criticised for wasting money and blowing up innocent people, but whatever ratio of outright con to blind optimism led to this system, I don't think the US military would be blowing up less people or wasting less money if it didn't exist. They'd just be relying on extremely dubious conventional SIGINT or HUMINT instead.

That's already how most Pakistani drone strikes originated, often with the final stage being HUMINT placing beacons on the ground (clearly foolproof!). The Pakistani Taliban and other groups execute lots of people as suspected spies doing that, probably no small number legitimately, in the context of 'was a spy' and one huge other number murdered by chance out of fear of death from above. That's how insurgencies tend to go. America did an awesome job suppressing history of its own roundup's of 'Tory spies'.

Cell phone coverage barely exists in the Pakistani federal tribal areas where almost all drone strikes take place in the country and the steep valleys mean even near a tower coverage is generally very poor. Since the Pakistani military uses the towers they also have a habit of getting blown up, while for the local militants VHF radio is often much more useful since it can go over the LOS horizon.

Most of the cell coverage that does exist was only installed in the past couple years, as part of a Pakistani program to try to improve the economic conditions on the border region. One of the few half decent thing's they've tried to do in the area since the 1980s. People tend to ignore that for every known drone strike, all of which are approved of by Pakistan, Pakistan itself has conducted multiple air raids or artillery bombardments itself. But whatever, nobody in the west cares about that. They care about a news report with the words cellphone in it because they know they have a cellphone, while something about VHF or 2.4 GHz means nothing to them.