Home » Air » ‘Big data’ poses big problem for Pentagon

‘Big data’ poses big problem for Pentagon

by Mike Hoffman on February 20, 2013

FORT LAUDERDALE, Fla. — Data scientists are the most in demand job for the military, according to Reggie Brothers, the deputy assistant secretary of defense for Research.

The military has a problem with “big data” — the problem being that it collects too much of it. The infatuation with unmanned vehicles and the sensors mounted onto them has spurred a wave of data collected on the battlefield.

Using that data has caused military leaders headaches. Brothers said here at the Association of the U.S. Army’s Winter Symposium on Wednesday that the Army and the other services have placed their focus on PED, or processing, exploitation, and dissemination.

He used the ARGUS-IS as an example of the major advances being made in the world of intelligence sensors. The ARGUS-IS can stream up to a million terabytes of data and record 5,000 hours of high definition footage per day. It can do this with the 1.8 gigapixel camera and 368 different sensors all housed in the ARGUS-IS sensor that can fly on an MQ-9 Reaper.

However, the analysis of the data collected by those sensors can’t keep up. Brothers, a former program manager for the Defense Advanced Research Projects Agency, said the military needs more data scientists to mine through these mountains of data streams. The Pentagon is in competition with private industry as these scientists are sought out throughout the commercial industry.

Military service leaders have already reached out to numerous broadcasting companies such as ESPN and National Geographic to learn ways these broadcast teams deal with the large streams of live-motion footage these companies collect.

One of the major challenges the Defense Department faces with this influx of data is dealing with the personnel demands. Many assume that UAVs require fewer people to operate. Brothers said that would be the wrong assumption.

“We just don’t have the personnel right now,” Brothers said.

Defense leaders are trying to reach the appropriate mix of autonomy in order to lessen the work load on the operators. However, balancing the amount of decision making afforded to the machines has delayed those developments.

The goal is to get to a point where, on average, less than one person is needed to operate an unmanned vehicle. Right now, the military averages well over one person operating its unmanned fleet.

“We need to do better than that,” Brothers said.

Share |

{ 17 comments… read them below or add one }

riicky February 20, 2013 at 6:45 pm

So, tech being to much for us huh. Lol.

Reply

blight_ February 20, 2013 at 7:16 pm

Military's not the only group with big data headaches.

Mining the raw data is left to algorithms. Support algorithm development, high performance computing, and ways to visualize data product for human analysis.There's only so much data transformation possible to turn something into an human-understandable 2D/3D plot.

Reply

Hyundai February 20, 2013 at 7:35 pm

What you need to do is have programs with algorithms that record what is relevant and what isn't . Or you can have a supercomputer go through all the information collected looking for specific data has been recorded. What will be needed is numerous algorithims and programs looking for specific cues that make recording relevant. This doesnt happen overnight but is a trial and error process which will see what programs and algothims work and what doesn't. A simple example is an algothim to record when someone is carrying a firearm. Another algorithim might have to to with behavior or action that is not consistent with a persons location age sex etc. What is needed are programmers and social scientist and psychologist .

Reply

Nick February 20, 2013 at 7:43 pm

Good, now all the government needs to do is recruit top programming talent with their sub par salaries and all of our problems are solved. /sarcasm

But in all seriousness, these sorts of programming/data integration problems are only solved by some of the most sought after talent in the computer industry. Good luck recruiting them with middling salaries and requirements to work in places that not a lot of programmer types live/work.

Reply

stephen russell February 20, 2013 at 9:00 pm

Do a massive hiring of Internet & IT trained populace for DoD jobs for security alone.
IE End users, software, hardware alone
From all states.
& downsize for rapid use.

Reply

Jacob February 20, 2013 at 11:40 pm

I imagine the Chinese are probably stealing more data than they can process as well.

Reply

EJ257 February 21, 2013 at 10:23 am

We should just outsource the data processing to the Chinese. They certainly have no shortage of talent or manpower.

Reply

blight_ February 21, 2013 at 10:53 am

Or the machines. Bear in mind there are an increasing number of LINPACK TOP500 machines going up in the People's Republic.

Reply

Brandon February 21, 2013 at 10:31 am

Lets hope they aren't able to process quite a bit of it.

Reply

Musson February 21, 2013 at 9:06 am

The Hadoop people will be the first to tell you that putting data across 1000 servers means there will always be a few servers that are on the fritz. So, you will only retrieve data from 998 of them.

So, you never get the same number twice!

Reply

Brandon February 21, 2013 at 10:31 am

Any thing recorded that actually has an event worth keeping can be tagged and moved to long term storage. Preset amount of time before and after the even should be kept until it can later be determined how much is needed and what is not. In the case of Argus, only things that are important in areas not even remotely close to the "event" captured would be the moments immediately before and after the event unfolds to watch for reactions and target indicators. All other data can be held more very temporary amounts of time and be automatically deleted if no events are reported. Obviously this is only half of the solution. Most live feeds from predators doing over watch missions can be temporary and automatically deleted if no worth while intel was observed or reported by the operators or ground elements. I think a lot of the problem has to do SIGINT and other communications type intelligence that actually has to saved and processed which is not quick I am sure especially given the language barrier.

Reply

Rob February 21, 2013 at 12:52 pm

With US Government underminding budget all the time, its going be even more tougher for military to try get these too few of data scientist. I've heard alot of US Programmers being retired and outsourcing the work to India since they work for cheap. It will be incredibly difficult to find a good number of technically alligned people to do this work DoD and other organizations are seeking if they keep obsoleting people coming straight out of college. Not giving them chance due to the lack of experience

Reply

@analyticbridge February 23, 2013 at 10:24 pm

One of the problems is that you need clearance or at least US citizenship to qualify for these jobs. It eliminates a large number of great candidates, even worse: these candidates might end up working for enemies.

Reply

BobsYourUncle February 24, 2013 at 6:35 am

What we need to do is filter the data coming in and decentralize its analysis. The biggest problem experienced by Air Force intel during the great man hunts of '03 to '11 was its centralized nature. Intel was collected, analyzed and funneled as it went up the chain of command and then back down and through conduits to other branches. By the time it reached people in the field it was too old to be of use. And that's the intel that wasn't lost in the static. With everything being funneled through the same hole at the top its a wonder anything of use was ever noticed.

Fighter pilots in Vietnam were known to turn off gizmos in their cockpits and switch to radio frequencies not cluttered by the other flights in their strike packages. Why would they turn of radar and surface to air missile warning alarms, why stop listening to airborne radar controllers and the other flights of fighters doing battle with the SAMs and Migs that were trying to kill them too? It was more info than they could adequately digest. They prioritized and they filtered. Something people today are afraid to do. Not every tweet is worth reading and not every facebook update warrants attention.

Too much info can do more to rob you of situational awareness than having too little. It is often bandied about that Sept 11 preparations went unnoticed because the intel agencies were taking in more info than they could process. Its pointless to subscribe to every major paper in the world if you don't have time to read them all.

We've made the Generals into analysts by giving them battlefield tele-presence and by running every juicy bit of paper across their desks. By doing so we have unwittingly set them up to fail. This wiz bang wow stuff is forcing them to micro manage and forcing them to do it poorly.

Reply

WPG February 25, 2013 at 4:43 pm

This might sound silly and maybe it is, but when i play command and conquer i can run a war from my laptop just fine lol. Its a very simple war tho…..but that template must be worth something?

Reply

dipanshu mansingka March 13, 2013 at 6:45 am

A good product using big data / distributed computing is required. UK has 1.85 million cameras, which are now going to be replaced with HD camera.

Reply

Dwight_Chroot March 17, 2013 at 3:17 am

Big Data is DU to IT.

It's also a tulip bubble…at a time when we need a New Deal.

IT needs a 0 level makeover. Instead of rooting everything with spychips then adding malware and misappropriating the WWW to Madison Avenue…we need free hardware
(free as in freedom, not beer, as in OBEDIENT to the owner, transparent and secure.)

Instead of could/tablet the industry should be building "print to order" custom hardware built to user spec, and the user burns a ROM hypervisor or bootloader at home. That's how you get network security.

But Mr. Burns needs to monitor what color of shoes Justin Bieber fans whim today? You can't milk any more money out of a collapsing middle class…USEFUL COMPUTERS, and a USEFUL WEB is the way to go…Airforce can handle SIGINT for legit snooping.

Reply

Leave a Comment

Previous post:

Next post: