Is artificial intelligence vulnerable to hacking?

Question

The paper The Limitations of Deep Learning in Adversarial Settings explores how neural networks might be corrupted by an attacker who can manipulate the data set that the neural network trains with. The authors experiment with a neural network meant to read handwritten digits, undermining its reading ability by distorting the samples of handwritten digits that the neural network is trained with.

I'm concerned that malicious actors might try hacking AI. For example

• Fooling autonomous vehicles to misinterpret stop signs vs. speed limit.
• Bypassing facial recognition, such as the ones for ATM.
• Bypassing spam filters.
• Fooling sentiment analysis of movie reviews, hotels, etc.
• Bypassing anomaly detection engines.
• Faking voice commands.
• Misclassifying machine learning based-medical predictions.

What adversarial effect could disrupt the world? How we can prevent it?

Answer 1

AI is vulnerable from two security perspectives the way I see it:

1. The classic method of exploiting outright programmatic errors to achieve some sort of code execution on the machine that is running the AI or to extract data.

2. Trickery through the equivalent of AI optical illusions for the particular form of data that the system is designed to deal with.

The first has to be mitigated in the same way as any other software. I'm uncertain if AI is any more vulnerable on this front than other software, I'd be inclined to think that the complexity maybe slightly heightens the risk.

The second is probably best mitigated by both the careful refinement of the system as noted in some of the other answers, but also by making the system more context-sensitive; many adversarial techniques rely on the input being assessed in a vacuum.

Answer 2

Programmer vs Programmer

It's a "infinity war": Programmers vs Programmers. All thing can be hackable. Prevention is linked to the level of knowledge of the professional in charge of security and programmers in application security.

eg There are several ways to identify a user trying to mess up the metrics generated by Sentiment Analysis, but there are ways to circumvent those steps as well. It's a pretty boring fight.

Agent vs Agent

An interesting point that @DukeZhou raised is the evolution of this war, involving two artificial intelligence (agents). In that case, the battle is one of the most knowledgeable. Which is the best-trained model, you know?

However, to achieve perfection in the issue of vulnerability, artificial intelligence or artificial super intelligence surpass the ability to circumvent the human. It is as if the knowledge of all hacks to this day already existed in the mind of this agent and he began to develop new ways of circumventing his own system and developing protection. Complex, right?

I believe it's hard to have an AI who thinks: "Will the human going to use a photo instead of putting his face to be identified?"

How we can prevent it

Always having a human supervising the machine, and yet it will not be 100% effective. This disregarding the possibility that an agent can improve his own model alone.

Conclusion

So I think the scenario works this way: a programmer tries to circumvent the validations of an AI and the IA developer acquiring knowledge through logs and tests tries to build a smarter and safer model trying to reduce the chances of failure.

Answer 3

How we can prevent it?

There are several works about AI verification. Automatic verifiers can prove the robustness properties of neural networks. It means that if the input X of the NN is perturbed not more that on a given limit ε (in some metric, e.g. L2), then the NN gives the same answer on it.

Such verifiers are done by:

• Stanford: https://arxiv.org/pdf/1702.01135.pdf
• ETHZ: https://www.sri.inf.ethz.ch/papers/sp2018.pdf
• Google: https://arxiv.org/pdf/1803.06567.pdf, https://arxiv.org/pdf/1805.10265.pdf
• Bosch: https://arxiv.org/pdf/1805.10265.pdf

This approach may help to check robustness properties of neural networks. The next step is to construct such a neural network, that has required robustness. Some of above papers contain also methods of how to do that.

There are different techniques to improve the robustness of neural networks:

• adversarial training (see e.g. A. Kurakin et al., ICLR 2017)

• defensive distillation (see e.g. N. Papernot et al., SSP 2016)

• MMSTV defence (Maudry et al., ICLR 2018).

At least the last one can provably make NN more robust. More literature can be found here.

Answer 4

I believe it is, no system is safe, however I am not sure if I can still say this after 20-30 years of AI development/evolution. Anyways, there are articles that showed humans fooling AI (Computer Vision).

https://www.theverge.com/2018/1/3/16844842/ai-computer-vision-trick-adversarial-patches-google

https://spectrum.ieee.org/cars-that-think/transportation/sensors/slight-street-sign-modifications-can-fool-machine-learning-algorithms

Answer 5

Is Artificial Intelligence Vulnerable to Hacking?

Invert your question for a moment and think:

What would make AI at less of a risk of hacking compared to any other kind of software?

At the end of the day, software is software and there will always be bugs and security issues. AIs are at risk to all the problems non-AI software is at risk to, being AI doesn't grant it some kind of immunity.

As for AI-specific tampering, AI is at risk to being fed false information. Unlike most programs, AI's functionality is determined by the data it consumes.

For a real world example, a few years ago Microsoft created an AI chatbot called Tay. It took the people of Twitter less than 24 hours to teach it to say "We're going to build a wall, and mexico is going to pay for it":

(Image taken from the Verge article linked below, I claim no credit for it.)

And that's just the tip of the iceberg.

Some articles about Tay:

• BBC
• The Verge

Now imagine that wasn't a chat bot, imagine that was an important piece of AI from a future where AI are in charge of things like not killing the occupants of a car (i.e. a self-driving car) or not killing a patient on the operating table (i.e. some kind of medical assistance equipment).

Granted, one would hope such AIs would be better secured against such threats, but supposing someone did find a way to feed such an AI masses of false information without being noticed (after all, the best hackers leave no trace), that genuinely could mean the difference between life and death.

Using the example of a self-driving car, imagine if false data could make the car think it needed to do an emergency stop when on a motorway. One of the applications for medical AI is life-or-death decisions in the ER, imagine if a hacker could tip the scales in favour of the wrong decision.

How we can prevent it?

Ultimately the scale of the risk depends on how reliant humans become on AI. For example, if humans took the judgement of an AI and never questioned it, they'd be opening themselves up to all sorts of manipulation. However, if they use the AI's analysis as just one part of the puzzle, it would become easier to spot when an AI is wrong, be it through accidental or malicious means.

In the case of a medical decision maker, don't just believe the AI, carry out physical tests and get some human opinions too. If two doctors disagree with the AI, throw out the AI's diagnosis.

In the case of a car, one possibility is to have several redundant systems that must essentially 'vote' about what to do. If a car had multiple AIs on separate systems that must vote about which action to take, a hacker would have to take out more than just one AI to get control or cause a stalemate. Importantly, if the AIs ran on different systems, the same exploitation used on one couldn't be done on another, further increasing the hacker's workload.

Answer 6

I concur with Akio that no system is completely safe, but the take away is AI systems are less prone to attacks when comparing with the old systems because of the ability to constantly improve.

As time passes by more people will get in the field bringing new ideas and hardware will be improving so that they are "strong AI."

Answer 7

There are many ways to hack an AI. When I was kid I figured how to beat a chess computer. I always followed the same pattern, once you learn you can exploit it. The worlds best hacker is a 4 year old that wants something he will try different things until he establishes pattern in his parents. Anyway, Get an Ai to learn the patterns of a AI and given a given combination you can figure the outcome. There is also just plain flaws or back door in code either on purpose or by chance. There is also the possibility the AI will hack itself. It is called misbehaving, remember the small child again...

BTW simple way is to make AI always fails safe... something people forget.