NSA Sweep “Waste of Time,” Analyst Says

It’d be one thing if the NSA’s mas­sive sweep of our phone records was actu­ally help­ing catch ter­ror­ists. But what if it’s not work­ing at all? A lead­ing prac­ti­tioner of the kind of analy­sis the NSA is sup­pos­edly per­form­ing in this sur­veil­lance pro­gram says that “it’s a waste of time, a waste of resources. And it lets the real ter­ror­ists run free.“
Re-​​reading the USA Today piece, one para­graph jumped out:

This kind of data col­lec­tion from phone com­pa­nies is not uncom­mon; it’s been done before, though never on this large a scale, the offi­cial said. The data are used for ‘social net­work analy­sis,’ the offi­cial said, mean­ing to study how ter­ror­ist net­works con­tact each other and how they are tied together.

So I called Valdis Krebs, who’s con­sid­ered by many to be the lead­ing author­ity on social net­work analy­sis — the art and sci­ence of find­ing the impor­tant con­nec­tions in a seemingly-​​impenetrable mass of data. His analy­sis of the social net­work sur­round­ing the 9/​11 hijack­ers is a clas­sic in the field.
Here’s what Krebs had to say about the newly-​​revealed NSA pro­gram that aims to track “every call ever made”: “If you’re look­ing for a nee­dle, mak­ing the haystack big­ger is coun­ter­in­tu­itive. It just doesn’t make sense.“
“Certain peo­ple are more sus­pi­cious than oth­ers,” he adds. They make fre­quent trips back-​​and-​​forth to Afghanistan, for instance. “So you start with them. And you work two steps out. If none of those peo­ple are con­nected, you don’t have a cell. Because if one was there, you’d find some clus­ter­ing. You don’t have to col­lect all the data in the world to do that.”

The right thing to do is to look for the best haystack, not the biggest haystack. We knew exactly which haystack to look at in the year 2000 [before the 9/​11 attacks]. We just didn’t do it…
The worst part — the thing that’s most dis­ap­point­ing to me — is that this is not the right way to do this. It’s a waste of time, a waste of resources. And it lets the real ter­ror­ists run free. 

UPDATE 2:30 PM: Shane Harris broke this story, in broad strokes, back in March, Patrick reminds us. Harris also offers a pos­si­ble expla­na­tion for some of the NSA program’s mas­sive size:

To find mean­ing­ful pat­terns in trans­ac­tional data, ana­lysts need a lot of it. They must set base­lines about what con­sti­tutes “nor­mal” behav­ior ver­sus “sus­pi­cious” activ­ity. Administration offi­cials have said that the NSA doesn’t inter­cept the con­tents of a com­mu­ni­ca­tion unless offi­cials have a “rea­son­able” basis to con­clude that at least one party is linked to a ter­ror­ist orga­ni­za­tion.
To make any rea­son­able deter­mi­na­tion like that, the agency needs hun­dreds of thou­sands, or even mil­lions, of call records, prefer­ably as soon as they are cre­ated, said a senior per­son in the defense indus­try who is famil­iar with the NSA pro­gram and is an expert in the ana­lyt­i­cal tools used to find pat­terns and con­nec­tions. Asked if this means that the NSA pro­gram is much broader and less tar­geted than admin­is­tra­tion offi­cials have described, the expert replied, “I think that’s correct.” 

Harris also fin­gers a likely pro­gram set of research efforts to help the NSA bet­ter comb through all this data: “Novel Intelligence from Massive Data,” or NIMD. Its goal is to develop “tech­niques and tools that assist ana­lysts not only in deal­ing with mas­sive data, but also in inter­ac­tively mak­ing explicit — and mod­i­fy­ing and updat­ing — their cur­rent ana­lytic (cog­ni­tive) state, which includes not only their hypothe­ses, but also their knowl­edge, inter­ests, and biases.“
You’ll be shocked to hear that NIMD’s web­site has been taken offline. But you can find Goggle caches about the pro­gram here, here, here, and here.
UPDATE 5:19 PM: “To me, it’s pretty clear that the peo­ple work­ing on this pro­gram aren’t as smart as they think they are,” says for­mer Air Force counter-​​terrorist spe­cial­ist John Robb. “Some top level think­ing indi­cates that this will quickly become a rat hole for fed­eral funds (due to wasted effort) and a major source of infringe­ment of per­sonal free­dom.” John gives a bunch of rea­sons why. Here’s just one:

It will gen­er­ate oodles of false pos­i­tives. Al Qaeda is now in a phase where most domes­tic attacks will be gen­er­ated by peo­ple not cur­rently con­nected to the move­ment (like we saw in the London bomb­ings). This means that in many respects they will look like you and me until they act. The large vol­ume of false pos­i­tives gen­er­ated will not only be hugely inef­fi­cient, it will be a major infringe­ment on US lib­er­ties. For exam­ple, a false pos­i­tive will likely get you auto­mat­i­cally added to a no-​​fly list, your boss may be vis­ited (which will cause you to lose your job), etc. 

UPDATE 6:23 PM: And now, the rebut­tal. I just got off the phone with a source who has exten­sive expe­ri­ence in these mat­ters. And he dis­agrees, strongly, with Krebs and Robb.
Really, the source said, there are two approaches to whit­tling down mas­sive amounts of infor­ma­tion: lim­it­ing what you search from the begin­ning — or tak­ing absolutely every­thing in, and sift­ing through it after­wards. In his expe­ri­ence, the source said, the approach of using “brute force… not opti­mally, not smartly” on the front end, and “clean­ing [the data] up later” worked the best. Often times, other peo­ple don’t know what you’re search­ing for (or they don’t have the same super-​​slick data-​​mining algo­rithms you’ve got). Better just to get it all.
In every­thing from speech analy­sis to sen­sor fusion, he argued, when you’ve got a weak sig­nal masked by a lot of noise, “more data seems to be the answer… More data is what’s going to allow you to get to ground truth.“
Of course, there’s a price to pay with this approach: a ton of false alarms. Several stages of fil­ter­ing should fix that, he argued. Besides, “it’s not like you call the FBI every time you get a hit.“
Think of it as the Google approach. Wouldn’t you rather have every­thing avail­able on the search engine, and then do queries your­self?
UPDATE 05/​12/​06 8:52 AM: The rebut­tal gets rebutted.
“I find it almost impos­si­ble to believe that the NSA has a sys­tem good enough to beat human int[elligence], selec­tive tap­ping, and the kind of pro­gres­sive exten­sion that Krebs cites,” an MIT pro­fes­sor says, who also passes along this handy graphic.

You need to have a good under­stand­ing of the “clas­si­fiers” and func­tions appro­pri­ate for your data set — devel­op­ing the knowl­edge and tech­niques around find­ing those clas­si­fiers has taken [com­puter] vision [research] 30 years to get where it is (able to drive a car through a pre-​​set path in a desert, rec­og­nize one face out of a thou­sand with good rejec­tion but many, many false pos­i­tives)… Meaning fine, but not great… We have almost no idea how com­plex this issue is, but it’s prob­a­bly sim­i­lar.
One thing about your “exten­sive expe­ri­ence” source is that he doesn’t really spec­ify what kind of search he was doing. People doing data min­ing may be look­ing in many dif­fer­ent ways. For instance, if you have six mil­lion exam­ples of suc­cess­ful stock price changes and six mil­lion exam­ples of unsuc­cess­ful ones, you might look for other vari­ables (past per­for­mance, loca­tion, etc.) that sig­nal a dif­fer­ence — any dif­fer­ence. Large data sets are def­i­nitely help­ful for this. Getting machine learn­ing to dis­cover a spe­cific thing — like a famil­ial bond based on tele­phone calls — may or may not work at all. If all you have is fre­quency, there may be a half dozen other types of rela­tion­ships that lead to numer­ous calls. There may never be a way of dis­cern­ing rela­tion­ship based on a sin­gle modal­ity of com­mu­ni­ca­tion. That’s why most of the peo­ple I know are using mil­lions of other sen­sors, like GPS, accelerom­e­ters, record­ing the voice, read­ing heart rate, etc. Then they may be able to say with mod­er­ate cer­tainty that they can tell some­thing from phone calls. The NSA can’t do that with what USA Today says they’re collecting.

UPDATE 05/​12/​06 11:48 AM: Click here to see if you can spot the dif­fer­ence between an Al-​​Qaeda clus­ter, and on from a Fortune 500 firm.

May 11th, 2006 | You can run... | 3214245 Comments »http://defensetech.org/2006/05/11/nsa-sweep-waste-of-time-analyst-says/NSA+Sweep+%22Waste+of+Time%2C%22+Analyst+Says2006-05-11+18%3A35%3A59david_axe You can skip to the end and leave a response. Pinging is currently not allowed.