A recent transpac crippled six F-22s as they made their way from Hawaii to Kadena Air Base in Okinawa, Japan. The culprit: The International Date Line.
When the fighters crossed the line, all of their computer systems went Tango Uniform — fuel subsystems, navigation, and some of the comms.
We turn to CNN’s John Roberts and retired Maj. Gen. Don Shepperd for expert commentary on the matter:
ROBERTS: Twenty five years from development to deployment, the F-22 Raptor is the most advanced fighting machine in the air. But it was no match for a computer glitch that left six of them high above the Pacific Ocean, deaf, dumb and blind as they headed to their first deployment. So what happened? We turn to a man who’s at home in the cockpit, Retired Air Force Major General Don Shepperd. Don, let me set the scene. These F-22s, eight of them, were headed from Hickam Air Force base in Hawaii to an Air Force base in Japan. They were approaching the international date line, pick it up from there.
SHEPPERD: You got it right, John. You want everything to go right with your frontline fighter, $125, $135 million to copy. The F-22 Raptor is our frontline fighter, air defense, air superiority. It also can drop bombs. It is stealthy. It’s fast and you want it all to go right on your first deployment to the Pacific and it didn’t. At the international date line, whoops, all systems dumped and when I say all systems, I mean all systems, their navigation, part of their communications, their fuel systems. They were — they could have been in real trouble. They were with their tankers. The tankers — they tried to reset their systems, couldn’t get them reset. The tankers brought them back to Hawaii. This could have been real serious. It certainly could have been real serious if the weather had been bad. It turned out OK. It was fixed in 48 hours. It was a computer glitch in the millions of lines of code, somebody made an error in a couple lines of the code and everything goes.
ROBERTS: This is almost like the feared Y2K problem that happened to these aircraft. We should point out that computers control almost every aspect of this aircraft, from their weapons systems, to the flight controls and the computers absolutely went haywire, became useless.
SHEPPERD: Absolutely. When you think of airplanes from the old days, with cables and that type of thing and direct connections between the sticks and the yolks and the controls, not that way anymore. Everything is by computer. When your computers go, your airplanes go. You have multiple systems. When they all dump at the same time, you can be in real trouble. Luckily this turned out OK.
ROBERTS: What would have happened General Shepperd if these brand-new $120 million F-22s had been going into battle?
SHEPPERD: You would have been in real trouble in the middle of combat. The good thing is that we found this out. Any time — before, you know, before we get into combat with an airplane like this. Any time you introduce a new airplane, you are going to find glitches and you are going to find things that go wrong. It happens in our civilian airliners. You just don’t hear much about it but these things absolutely happen. And luckily this time we found out about it before combat. We got it fixed with tiger teams in about 48 hours and the airplanes were flying again, completed their deployment. But this could have been real serious in combat.
ROBERTS: So basically you had these advanced air — not just superiority but air supremacy fighters that were in there, up there in the air, above the Pacific Ocean, not much more sophisticated than a little Cessna 152 only with a jet engine.
SHEPPERD: You got it. They are on a 12 to 15-hour flight from Hawaii to Okinawa, but all their systems dumped. They needed help. Had they gotten separated from their tankers or had the weather been bad, they had no attitude reference. They had no communications or navigation. They would have turned around and probably could have found the Hawaiian Islands. But if the weather had been bad on approach, there could have been real trouble. Again, you get refueling from your tankers. You don’t run — you don’t get yourself where you run out of fuel. You always have enough fuel and refueling nine, 10, 11, 12 times on a flight like this where you can get somewhere to land. But again, attitude reference and navigation are essential as is communication. In this case all of that was affected. It was a serious problem.
ROBERTS: So the fact the computers run so much of the systems on these aircraft, General Shepperd, is the — is the military at risk of over engineering here so if they did have a problem like that when they were going into a hostile situation, they could be, as you said, repeatedly in real trouble?
SHEPPERD: Well, you have redundant systems but it’s just a fact of life in the modern computer age. By the way John, you are going to have the same problem coming up on your laptop computer as we conferred from — from standard time from daylight savings time to standard time. Your program — your computer is programmed for one thing and we have changed the dates and you are going to have a problem. It’s going to have to be dealt with.
ROBERTS: Do me a favor Don. Make sure I’m not on my laptop computer when I’m flying in an F-22 on that day.
SHEPPERD: Absolutely.
And make sure you don’t try to conduct any strikes across the International Date Line. One side or the other, war planners; one side or the other.
– Ward
What does it say about the defense contractor when a fighter plane that costs $135 million is crippled by the International Date Line? Was $28,000,000,000 in development costs and twenty years of development not enough?
Lockie, you’re doing a heck of a job!
Gotta agree with livio. Bugs are a fact of of life in the Info Age. Someone screwed up, and hopefully they’ll check more carefully in that area in the future. But all the planes kept flying (speaks well of their redundancy that they did that with their software on the blink) and today they’re in Japan.
The F-22 Raptor has something like 200 million lines of code, whereas an OS like Windows XP has something in the neighbourhood of 45 million…and we all know how stable XP is
.
Oh, and clearly, they never lost the “Whole Platform”, as the fly-by-wire systems clearly functioned without a hitch, not to mention the entirely electronic throttle and engine management systems, which obviously rely on computer controlled atmospheric sensors and…well, you get the idea.
Cary M brings up a good point.
I mean… as a nerd, I format my computer once a year… while spring cleaning. It’s just something you do!
Now… does that mean that the F-22 will need formatting after a year? Probably not… with more use.. the more often you need to re-install the OS.… so are they going to re-format every sorti? lol
However, as a web programmer, I create 100’s of unexpected bugs all the time.. and often some get noted and forgotten as the deadline crunch comes. “Gotta stay on budget”. I honestly thought that running a Jet on software being a horrible idea. What happens if the “Motherboard” of the jet gets shot out? What’s the piolet going to do to control his plane!
Honestly.. Stick with the Tomcat’s.. Send the F-22’s in for the anti-radar sorti’s and let the F-14’s/F-18’s do the rest…
“The F-22 Raptor has something like 200 million lines of code“
Do you have a citation for that? I just googled that and all references I’ve seen peg the number of lines for the F-22’s software at 1.7 to 2 million. Sources included Strategypage and Boeing.
Maybe it’s splitting hairs, but it’s also ten times less of a problem.
I agree with livio and Moose. The squadron (or whoever the pros from Dover were) did a good job of rewriting the code and getting the Raptors to press on. BUT … it’s relatively humorous that it happened in the first place. Remember the unit cost of these puppies is $253 million.
I just knew that this episode would bring out the “There’s no such thing as a bad weapons system” crowd, and that they’d be minimizing the significance of the thing. They might want to ask themselves — if something like the International Date Line causes such headaches, what about genuine surprises?
In the end it doesn’t really matter much, because the F-22 white elephant is truly a solution in search of a problem. If their wings fell off in mid-air, it wouldn’t affect American security a jot.
Oh, great. If we ever have to fight the Cylons, we are so screwed. I hope we have some Phantoms that are still operational.
As a software engineer in the defense industry, I can pretty say for certain that someone is getting fired right about now.
Nasty contracts letters are being written from the Air Force to Boeing.
Software testers are finding the issue, and determining how it slipped through.
Systems Engineers will be looking through their requirements and realizing that no one ever figured that time might go backwards.
This is rich. I have been up since 3:30AM Eastern patching hundreds of servers for the upcoming Daylight Saving Time change and I read this. I wonder if the on-board computer systems on the F-22’s are patched for that changeover?
It’s impossible to find all the bugs when testing any decent sized software. Even with common data processing business apps, some bugs will only appear after users start using it. This is one such an example, the programmer didn’t think of it, and when testing the Raptor no-one flew over the date line so the bug didn’t surface. So now they know about it and fixed it.
The important thing is to have some manual backup, to at least stay above water (or in the air) then the crisis hits.
Here’s a question. With an aircraft that is so dependant on computer systems to operate, what would happen if a directed energy weapon were fired at it? Let’s say a modulated surge of electricity through a couple of ionized pathways burned through the air by powerful lasers? Such a device has been in development for quite a while now. Would the Raptor be vulnerable to this? I know it’s been shown to disable motor vehicles…
yoiks, when they fixed the code to move FORWARD through time — I hope they also fixed it for moving BACKWARD through time. Otherwise the trip home FROM Japan TO Hawaii could be another doozie…
Any bets the code is based on something commercial. Maybe an MS product. Anyone remember what happened to the USS Yorktown? I’m just glad the fly by wire system wasn’t compromised or they would have neen in deep sierra.
Military systems need to be using proprietary, fault tolerant code — not some offshoot of a badly written commercial product.
The F-22 code is definitely not based on commercial code.
Most of it is written in ADA.
http://www.stsc.hill.af.mil/Crosstalk/2000/05/moody.html
It may be apocryphal, but I once heard from an electronic engineering friend of a similar bug in a guidance system, developed in the eastern part of the Southern Hemisphere, that didn’t work quite so well for the North American customers.
The problem was one of coordinate systems: When you’re used to getting positive numbers for South and East coords, and sudden you’re getting everything in North and West, things don’t work so well. To get things resolved quickly, they effectively installed it back to front — i.e. inverting all the numbers. No-one’s going to notice if a missile’s flying upside down, right?
Ummm … doesn’t all aviation work off Zulu?
It isn’t at all clear to me why the F-22 would care about the international date line, as it would be keeping time in Zulu.
The reasonably highly automated jet I fly (MD-11) crosses the date line all the time, and it couldn’t care less, because all it knows is Z.
Giggle-giggle…snort…giggle.….MUAHA-HA-HA-HA-HA-HA-HAAAAaaaa.…*cough-cough*. Oh dear, this has got to be one of the funniest stories I’ve read in a very, very long time.
I just knew this would happen when I saw the “Intel Inside” sticker on the avionics bay access hatch.
And the Royal Danish Air Force is looking into spending my tax money on the JSF. I hope they go with Linux instead (or, so help us, DOS).
Regards and all,
Thomas L. Nielsen
Denmark
Two million lines of hard real-time embedded code, this is not the stuff of trivial permutations but still, as one commenter said, they continued to fly despite the glitch. Our modern society is built on software–think about what we’re doing here commenting on a blog, this is 99% the result of software and 1% copper and silicon–your cool phone now has in excess of a million lines of code–and you still can’t get a signal. Setting up software test is likewise non-trivial and those who think this is funny ought to try accounting for all possible usage combinations for some trivial software based product…they might just find some humility.
I write ADA with my defence contractor. I find it odd that the situation was not tested in their SWIL testing.
From the article:
SHEPPERD: You got it. They are on a 12 to 15-hour flight from Hawaii to Okinawa, but all their systems dumped.
Hold On there pardner -
Hickam to Kadena comes up as 4040 nautical miles.
Max speed for a KC-10 is a hair over 600mph,
That comes out to 7 hours.
Using the public numbers, the f-22’s would have to refuel 5 times en route
If the F-22 could supercruise all the way, it would be only 3.5 hours.
By the way, it doesn’t look like there’s such a thing as drop-tanks for an F-22
Not quite as bad as when the Hawker Typhoon entered service in 1941. Six of the brand new, rushed into service planes engaged German Focke-Wulf’s over the channel. In the middle of the fighter, five of the six Typhoons’ tails fell off in a high g turn. Without ejection seats like they do today, the pilots never had a chance to get out against the tumbling effect.
Still, this Raptor biz seems weird. Anyone feel like they’re getting handed a bogus story from the higher ups?
You see what happens when you let the bright young
men in the think tanks dictate military equipment specs. Mac and his young men (who had never been
in or near the military, let alone combat).….“airplanes don’t need machine guns, that style of fighting is over. The F-4 doesn’t need anything but rockets.” Sure.
Well, we let the little men in again and you get the high techy stuff that amazes and amuses them and an aircraft the you can’t fly without a computer. When will we learn?
As far as I’m concerned, I would have stopped with the invention of the wheel, all we really need is a cart to haul our excess baggage around on
we have way to much excess baggage and a note on those Iron curtains in officials offices In Washington I think bright red velvet would be exquisite.
I used to write test equipment software and firmware to test Avionics Defense systems. This “coulda”, “shoulda” been tested thoroughly before the jet went into the air. This is a failure of their test team. They should have a simulation of those conditions for the software and embedded software to “experience.” Crossing such locations are known “events” for an avionics system.
In addition, I agree with one of the other responders. There should be a back up, such as an automatic navigational system that comes on– line (maybe something like the Chelton Flight System or something of that type that can be installed completely independent of the rest of the system.) Of course I’m not sure what system they already have for backup in this jet.
I LOVE THIS JET!!!!!!I WANT TO FLY IT WHEN I GO INTO THE MARINE CORPS!!!! I MISS HEREING THE SOUND OF IT WHEN I WATCH THE VIDEOS OF IT!!!!!!! THATS EXTREMLY CARZZZZZZZZZZZZZYYYYYYYYYYY!!!!111
There has never been delivery of a new major military (or commercial) product like an airplane, helicopter, tank, ship, etc. that did not have some items to be improved, if not corrected. Actually, I am impressed that correction to the “millions of line of code” was accomplished in 48 hours. It is a little sad to see the world’s premier fighter — truly fifth generation — berated so heavily over an issue that could have had some consequences, but didn’t. In all mock fights to date, F-22 aircraft have not lost a single fight, to any kind or quantity of competitor(s). What I see is common in our society these days — blow off the overwhelming good of an item (or event)and look/dig for something to criticize. Such is obvious from the tone of questions in the article as well as many comments.
Mike S,
“You see what happens when you let the bright young men in the think tanks dictate military equipment specs. “
Make that the few inept young men in think tanks as truly bright young men in think thanks or elsewhere would not forget any essential specs. Please think before making a rash comment like that.