GRAMMAR FOR SPYBOYS

Maybe sixth-​​grade English was more help­ful than we thought. One of the dullest gram­mar exer­cises is being used to help find poten­tial ter­ror­ists, and save com­pa­nies a bun­dle.
Diagramming sen­tences — pick­ing out sub­ject, verb, object, adjec­tive and other parts of speech — has been a sta­ple of mid­dle and high school gram­mar lessons for decades. Now, with financ­ing from the Central Intelligence Agency, a California firm is using the tech­nique to comb through e-​​mail mes­sages and chat room talks, which can be a rich lode of cor­po­rate and gov­ern­ment infor­ma­tion, and a tough one to mine.
Figuring out the con­nec­tions among peo­ple, places and things is some­thing com­puter algo­rithms do pretty well, as long as that infor­ma­tion is struc­tured, or cat­e­go­rized and put into a data­base. Looking through a company’s cus­tomer file for a per­son named Bonds, for exam­ple, is fairly sim­ple. But if the data is unstruc­tured — if the word “bonds” hasn’t been clas­si­fied as the name of a ballplayer or as an invest­ment option — search­ing becomes much more dif­fi­cult.
For peo­ple in busi­ness or in pub­lic ser­vice, only 20 per­cent or so of their infor­ma­tion is kept in for­mal data­bases, noted Nick Patience, an ana­lyst with the 451 Group, a tech­nol­ogy research firm. The rest is unstruc­tured, tucked away in e-​​mail mes­sages, call logs, memos and instant mes­sages.
Attensity, based in Palo Alto, Calif., and financed in part by In-​​Q-​​Tel, the C.I.A.‘s invest­ment arm, has devel­oped a method to parse elec­tronic doc­u­ments almost instantly, and dia­gram all of the sen­tences inside. (“Moby-​​Dick,” for instance, took all of nine and a half sec­onds.) By label­ing sub­jects and verbs and other parts of speech, Attensity’s soft­ware gives the doc­u­ments a defin­able struc­ture, a way to fit into a data­base. And that helps turn day-​​to-​​day chat­ter into infor­ma­tion that is rel­e­vant and usable.
My arti­cle in today’s New York Times had details.

March 3rd, 2005 | Data Diving | Comments Off Both comments and pings are currently closed.