Annalee Newitz over at io9 raises an interesting ethical question when she asks, “Will we hold robots accountable for war crimes?“
Here’s a bit of her post:
This isn’t idle speculation. An automated anti-aircraft cannon’s friendly fire killed nine soldiers in South Africa last year, and computer scientists speculate that as more weapons (and aircraft) are robot-controlled that we’ll need to develop new definitions of war crimes.
So how will justice be served in these cases? Presumably, we’ll punish a guilty robot by smashing it flat or refabricating it into a Kia Sportage owned by someone who commutes into DC.
But who else would we punish, especially if these robots are autonomous? The programmer who came up with the algorithm? And would the programmer, in turn, try to prove a software glitch caused by the manufacturer? Of course, this is the same question raised when 2001: A Space Odyssey hit screens almost forty years ago and more recently in the Will Smith vehicle I Robot
.
If you want to weigh into this issue on the ground floor you should make it a point to attend the Technology in Wartime conference. Your conference fee even gets you a free T-shirt (bonus).
(Gouge: CM)
(Photos: iRobot’s Bomb Disposal robot in action, courtesy iRobot; Capture from film “I Robot” starring Will Smith, courtesy 20th Century Fox)
– Ward
{ 25 comments… read them below or add one }
This is why I prefer to keep a human as close the the “front-end” as possible whether it is CIWS, UCAV, or anything with an attack/destroy capability.
Well, I guess the question is “Historically, who shoots the wrong person more often, robots or humans?”
Plus, I don’t see why faulty automated weapons systems are any different than exploding Firestone tires or unchecked brake pads. As long as the weapons system don’t screw up more often than a human, and as long as the screw up isn’t due to bad maintaince, then I guess you’d have to just write it off as an unfortunate mechanical malfunction. Kind of like you would if the helicopter just crashed on its own.
Autonomous war machines just seem silly and very dangerous, they always should have an operator to press the button.
Good photo
We’ve had autonomous weapons since the mid 1860′s. Think land mines. Do they kill innocent people? Yes. Do they work? Yes. With autonomous weapons we’ve got to decide just how much autonomy we’re willing to give to a lethal weapons system before we field the weapon.
we need EDDI was i the only person who thought stealth was a good movie…sigh
this is off subject but couldnt the just make a drone carrier like a pocket carrier size ship?
and like a a drone carrier for the air too i mean we could build airships alot better now than we used to plus that would save fuel if we used a nuke like a peeble bed reactor?
anyways my 0.001
Humans shall not always be in the loop, they are not always in the loop now, haven’t been in the loop in the past and shall not be in the loop in the future. Nor should they always be.
When they fail, it is a mechanical/computational failure. You cannot punish a machine. Nor can you or should hold a programmer/manufacturer liable anymore than you could hold a weapons or armor company for failures.
All programs that have the capability of autonomous actions capable of harming people should have gone through a large amount of design reviews — not only at the coding level, but requirements, high level design, system design, blah blah blah. You do not have “one guy in a room” coding things.
If something that wasn’t desired happened, the company and customer and process are all at fault.
I love sci-fi ethical debates. It is not possible to hold any single personal accountable for automated weapon systems which killed the incorrect person/target. Just like Private Security in Iraq, it will be the company and/or US Government that will be responsible.
There is absolutely no reason why the US isn’t more eager to push the boundaries when it comes to automated weapons. It is the only advantage we have against an endless supply of insurgents/terrorists. Imagine the strategic despair of the AIF if there were tens of thousands of unmanned armed robots patrolling the streets of Iraq, 24/7.
Maybe that friendly fire was:
1. “bugs” in system.
2. poor comm links
3. Poor Intelligence
4. programming
5. Bad encryption coding?
6. Pentagon bureaucracy BS.
7. DoD Regulations effecting Autonomous weapons?
Choose 1.
BUT The GI on the ground should oversee the Units anyway.
ID aimed in right area & account for allies.
Maybe improve sensors.
IE improve Friend or Foe ID.
Nah, all you would need is a robot with an automatic memory-wipe. Hell, it would simply be automating what seems to be standard CIA proceedure nowadays.
Evenin’ Boys,
I don’t think one could justly blame (or credit) the automated AA for the kill any more than one might blame (or credit) a land mine, a bullet, or a deep pit. In every instance there is human who turned on the machine, planted the mine, fired the bullet, or went walkies in the dark without a light.
Beyond that, the question of ethics vis-a-vis these highly automated weapons is no more complex than the ethics of a paleolithic booby trap. A thing contrived and set to injure or kill an enemy is just as capable of injuring or killing a non-enemy. All act without conscience, reason, or remorse once set.
What will prove far more intractable than ethics is the morality of these things. As we pursue AI with all the fervor and forethought of a crackhead, we’ll raise that intractability an order of magnitude (or three). When, inevitably, we achieve the ability to have a machine become self-aware and able think independent of its initial programming, we’ll going to find ourselves in a rather deep pile of shit.
And we’ll all be cursing one another (or Geo. Bush) for letting it happen.
It is, in my humble and rather ignorant opinion, that all this pursuit of AI is a form of Utopianism owing to a degree of intellectual sloth, moral cowardice, and ethical relativism. That is, the pursuit of a thing that will relieve us of our burden of critical thinking, doing the dirty work of fighting, killing, and dying, and shifting our ethics to fit the situation.
A quick review of history should inform us that no matter how lofty our ideals there will always be a man who is quite willing to wipe his ass with high-falutin ideals.
Just as al Qaeda turned passenger jets into cruise missles, so will another evil man turn AI on other men for profit, power, or just for kicks. The image from “I, Robot” makes my point. In Asimov’s work, all these machines are programmed with an infallible set of protocols to prevent a robot from ever harming a human. These protocols worked flawlessly until a human perverted them.
Frank Herbert explored the morality, ethics, and consequences of “thinking machines” in much greater detail in the Dune chronicles. In the post-apocalyptic future of Dune, after near extinction at the hands of machines, there comes the commandment: Thou shalt create a machine in the likeness of the human mind.
This is perhaps something we ought to consider.
Cheers,
Chief B.
Correction: The Herbert quote should read “Thou shalt NOT create a machine in the likeness of the human mind.
(I’m getting old…)
Stuuuuuupid….
If “robots” (automated AA guns aren’t robots) do bad things, it won’t be any different than any other situation:
(1) Was the bad thing the result of negligence on the part of the manufacturer or the CO that employed them, or was it intentionally employed to do the bad thing?
(2) punish the CO/manuafcturer accordingly.
The only different with robots is the “I was just following orders” defense is absolute. You can’t punish the robot anyway in any meaningful sense, just like you can’t punish a gun that is used as a murder weapon. So you punish the person who pulled the trigger.
This is not rocket surgery, people.
accualy it was a machine that took over control of the ns5′s or whatever but the doctor that used them to kill himself
ai is potentaly more damaging that gunpowder,nukes,or indeed all other mans inventions all together
think of it this way for all the yrs weve existed weve been the smartest meanest and most prolific preditor on earth. imagine a machine tht thinks like a man…the question is which man would that be? george washington…..ceasar?..stalin?
i dont know if we should ever realy get self aware ai but i think we definatly need a higher thinking comp for defensive systems, future space projects, etc
but never ever take a man out of a high powered weapons system
I dont see how this is any different than an airplane crash. Machines dont intentionally go out and kill people, at least not yet. They aren’t capable of anything that can be called an intent. So treat it as an accident, find out the fault, wether its a design flaw, maintanance flaw or whatever, and if there is negligence on anyones part, punish them accordingly. If not, than just accept that accidents happen.
Isn’t the Aegis system capable of operating in an autonomous mode? I don’t recall whether or not this played a role in the shooting down of the Iranian airline some years ago nor the legal fallout of that incident?
So many of the comments have revolved around the issue of intent. I think the real question here would turn out to be one of negligence.
Say the airliner was shot down while the system was operating in autonomous mode, could a claim be made that the designers of the system were negligent in creating a system that could not accurately distinguish between friend and foe? As an attorney, I would be subpoena every record related to system testing – rates of false-positives, etc.. If they knowingly deployed a system with an unacceptable amount of errors then they were negligent.
That would be manslaughter.
Re: CG
Yes, Aegis is capable of operating in a fully autonomous mode. A capability designed to cope with a Soviet saturation attack on the battlegroup (read “the carrier”) wherein the human operator would be incapable of “hooking and shooting” inbound missles quickly enough to thwart the attack.
The salient point in re this thread is that placing Aegis in full auto mode requires a human decision maker… and on a U.S. Navy ship that would mean multiple layers of decision makers and button pushers (CO, TAO, etc.) before the silicon took over. And even then it would only execute the its programmed doctrine (e.g., shoot, shoot, look, shoot).
This is much the same as with CIWS. Absolutely capable of operating and firing autonomously, but only after a human pushed the button and only able to execute its programmed doctrine.
In both cases they are not much more than very expensive parrots repeating what they’ve been told without any awareness of self or the ability to ask “Why?” It is when Aegis becomes self-aware and can ask “Why?” (or far worse “Oh, hell, why not?”) that we’ll have a real mess on our hands. Until then its all just a more efficient way of breaking things, killing people, and devouring tax revenue.
Regarding the validation of the SM2 against a non-maneuvering Airbus A300: Aegis and the Standard missle worked flawlessly. However, it was the human operators who went and shot down the plane that they shouldn’t have shot down.
As best as any of us not actually sitting at the console aboard VINCENNES will ever know, it was a tragic accident that probably could have been prevented. In my opinion, the responsibility for the shootdown lies four-square with the Iranians, not the young men in VINCENNES.
Khomeini’s boys who were running the Iranian side of things that day were well known for their restraint, truthfulness, and desire to bet everyone’s best pal. This is evidenced by all the mine laying, shooting ASMs at tankers in the straits, using 12-year old boys to clear minefields, taking hostages, and all that cheery “death to America” stuff. Nice guys all.
While VINCENNES was busy trying to shoot Iranian speedboats with her 5″/54s (a lot like trying to swat flies with a sledgehammer), the Iranians permitted Iran Air flight 655 to depart Bandar Abbas airfield. Bandar Abbas is a dual use field with both civilian and military aircraft in operation there. While there were certainly a number of blunders on the US side, it seems that permitting the flight to depart for Dubai on that track during the middle of a running gun battle was probably meant as yet another provocation by the Iranians.
Cheers,
Chief B.
Chief B.
Please don’t think that I was trying to lay any blame at the feet of the crew of the Vincennes. I was merely using the Aegis autonomous mode as an example of a weapon system that is dependent upon algorithms.
While it does take a human decision to turn the key, once the key is turned software takes over. Software makes the friend/foe determination and acts accordingly. The human is relying on the system being accurate enough to make that determination in a way that negates the possibility of friendlies being killed, or at least minimizes it to an acceptable level. What if it was later determined that the software was prone to false-positives but the human operators were never informed? How betrayed would you feel if you were the person that made the decision to turn the key?
Again, I’m using the Aegis system as the example for future systems that integrate software into the kill decision, not the Vincennes/Iran Air incident.
If the software is flawed then where does blame lie if innocents are killed? Given that human judgement is fallible, is the goal “better than human” or “as good as human”?
Just pondering, not positing a for/against position.
If the DoD takes delivery of a weapons system, does the responsibility then shift from the manufacturer to the government?
Also, want to direct people to my new blog which will respond to breaking Aerospace and Defense news… http://www.aerospacecases.com
I just posted about the new tactical hyperspectral sensor for the Shadow, which relates to autonomous target identification…
http://www.aerospacecases.com
re CG
Rest easy. I did not take it that you laid blame with her crew.
I can tell you, from my own frustrations and experiences, that Aegis is as fraught with aggravating little bugs, quirks, and demons as is anything that complex. E.g., SPY has the ability to be an excellent weather radar, getting returns off rain, clouds, etc.; all of which have to be tuned out by an operator/supervisor.
I recall a particular baseline rollout in the 90s that was the AWS equivalent of WindowsMe. Our STO, who was really the most even-tempered LDO I’ve known, was ready to kill. After an interval of dyspepsia on our part, bureaucratic twiddling by NAVSEA, and a bit of post-doctoral head-scratching by the Aegis gods, the solution was to roll AWS back to the previous baseline to await a functional baseline release.
Make no mistake, AWS is the best damned weapon system in the world today. Witness its successes with SM3 in the ABM role — the ground-based fellas have wet dreams about their systems being so successful. I have little doubt that AWS equipped ships can ensure that “if it flies, it dies.”
Considering this, it still comes back to the wanks who write code, the techs who maintain/operate it, and the decision makers’ choices, and timing. All of which are absolutely susceptible to human fallibility, as you write. Given that, there can NEVER be absolute certainty that the track you’ve hooked and shot is what everyone believes it to be. The odds that it is not may be diminishingly small, but Murphy has a way of beating those odds. Fratricide is a risk that we try manage but can never completely eliminate.
That is just the way things are and no amount of training, no great expenditure of treasure, and no amount of planning will ever eliminate it. Call it the Lawyer’s Share.
We do the best we can, with what we have, and pray God will guide us when training, brains, and whiz-bang gadgetry are exhausted. I would not feel any remorse or place blame on any Sailor who acted in good faith and gave it 100% effort. The devil makes mischief in the world and then sends the lawyers, media, congressmen, and other demonic powers to blame it all on some poor bastard who did the best he could. In other words, Shit Happens.
Cheers,
Chief B.
Some pretty good comments here. Let me toss in a little more grist for the mill.
When you ask “Will we hold robots accountable for war crimes?”, it’s partly a question about robots, and partly a question of accountability.
Accountability can be a fuzzy concept, but we can say that it comes in a few flavors. Moral culpability results from violating social or customary (including religious) precepts. Legal culpability is more a matter of violating laws, or bearing responsibility for the consequences of intentional acts, or violations of duties of care.
As some of you have pointed out, it would take a pretty high order of AI to consciously violate either moral precepts or laws. We probably don’t have to worry about that for a while. Once machines achieve that level of consciousness, determining culpability will be pretty simple — we will hold robots to the same standard as any other moral or legal actor.
[A short digression here -- to some extent, we have already invented AIs that are legally recognized as persons. They're called "corporations", and, in some cases, perhaps too few, they're legally responsible for their actions, apart from the humans who run them.
Let's leave that can of worms unopened, for another fishing trip...]
Without intent, or the ability to understand the consequences of acts or omissions, a robot might as well be a toaster — you can’t blame the toaster for burning the bread. It’s the human operator’s fault for not making sure it’s operating properly.
So that pushes things towards the culpability of human supervisors, as some of you correctly point out. That’s where things get tricky.
Absent a law or duty of care, such as a treaty regarding the employment of military robots, you’d have a hard time defining a “war crime”. I mean, sh*taree, that’s hard enough already.
Military robots present special difficulties, because there’s relatively little formal precedent or custom to use for guidance. Weapons systems without a man in the loop are relatively new — with a few notable exceptions.
As Jack D. pointed out, land mines have been used for over a 100 years. With sea mines, they comprise truly autonomous kill mechanisms. Once emplaced (or, in the case of some sea mines, set to drift), some of them can’t be controlled.
IMHO, there seems to be an emerging international effort to eschew such autonomous weapons, absent control or safety mechanisms. Autonomous sea mines were first banned by the Hague Convention of 1907, and their use by Iran in the “tanker war” justified punitive expeditions against the Islamic Republic.
More recently, you’ve got the Ottawa Convention on anti-personnel mines, agitation regarding cluster munitions and their UXO problems, etc. A variety of countries, including the US, haven’t signed Ottawa, but many of them follow the Convention on Certain Conventional Weapons, covering similar issues, that comes out of the Geneva Conventions, and similar “laws of war”.
It’d take me a few hours to research and summarize Geneva, Hague, the Nuremberg Principles, etc., etc. Let’s let JAG officers do that, assuming it’s still part of the MOS. It should suffice to say that under current “laws of war”, lighting up civilians and noncombatants is a no-no. To the extent any robot (including a land mine) can do that, we may already have a problem.
In short, current “laws of war” may already impose duties of care on the deployment of autonomous weapons systems. At the same time, they may provide little, if any, guidance on what level of care is appropriate.
So, bottom line: this is still largely on the horizon, per the lede. It’s a problem for JAG in the short term, and international treaty in the long run. But forward thinking people may want to contemplate it.
I mean, if hindsight is 20/20, foresight has got to be…what?
http://gizmodo.com/347694/israeli-military-wants-to-build-artificially-intelligent-missile-defense-system-aka-skynet
Israeli Military Wants to Build Artificially Intelligent Missile Defense System
>Israeli Military Wants to Build Artificially
>Intelligent Missile Defense System
Is that hooked up to the SkyNet?
This was not the fault of the gun. It was the operators