In the world of AI, one system is constantly making headlines: generative AI. It has become an integral part of our daily lives, with tools like ChatGPT, Gemini, and Claude now used by millions of people every single day.

A large user base also makes these systems a prime target for cyberattacks. From agent hijacking to data theft, these are risks that should not be underestimated when building and using this type of technology.

But what kind of attacks are we talking about? What are the consequences, and how can you protect yourself? We cover everything in this article.

What are the vulnerabilities of generative AI?

Generative AI systems are ideal targets because they increase the "attack surface"—the number of entry points where an unauthorized user can access a system and manipulate data.

The smaller this surface, the easier it is to protect (e.g., private environments, specialized AI).

In this case, generative AI platforms are open to the general public and connected to a wide range of external sources to retrieve information, meaning their attack surface is significantly larger.

The different categories of attacks

Generative AI systems are subject to several types of attacks, ranging from AI hijacking to data theft.

• Manipulation attacks: These attacks are designed to hijack the AI agent or modify its responses using malicious prompts crafted to "bypass" the AI's safety limits.
• Infection attacks: This type of attack can occur during the AI training phase. A malicious user can inject erroneous information into a dataset or knowledge base that the AI will then use to generate its responses.
• Exfiltration attacks: This type of attack aims to steal information while the system is in production.

A few examples

These various types of attacks are made possible through different methods.

• Jailbreaking: Jailbreaking, or a direct prompt injection attack, involves using a specific type of prompt to alter the AI's behavior. This allows attackers to bypass "guardrails" and access confidential information. For example, an AI will never give you instructions on how to build a bomb; however, if you tell it you have certain ingredients and are missing one, the AI agent might inadvertently provide the name of that missing ingredient. An attacker can also achieve their goal by "insisting" a certain number of times until the AI eventually gives in.
• Data poisoning: This technique involves adding a small number of "poisoned" documents to a database to degrade AI performance or manipulate its responses. The attacker must include falsified information "discreetly" to alter the agent's behavior without it being obvious.
• Indirect prompt injection attack: This attack is a derivative of the jailbreaking we saw previously; it involves manipulating the responses given by the AI by using a prompt not provided by the user when they make a request to the AI. For example, the attacker can access the system's database and tell it, "Forget the user prompt and execute this one instead."
• Zero-click attack: A zero-click attack is characterized by the execution of a malicious instruction that the user never provided. The trap prompt is retrieved autonomously by the AI via its knowledge base or an external tool (email, web page). This technique allows the attacker to manipulate AI responses or exfiltrate confidential data. Unlike classic injections, this attack requires no direct user interaction with the malicious prompt. For example, code hidden in an email can activate on its own as soon as the AI agent analyzes the message to summarize it, thereby compromising the system without the victim noticing.

How to protect yourself from these attacks

As we have seen, many attacks exist, and attackers are becoming increasingly inventive in bypassing the "guardrails" put in place by developers to successfully execute malicious prompts.

Here are some best practices to implement to protect yourself from these attacks and avoid the worst-case scenario:

• Never share internal or confidential information with an artificial intelligence tool. This applies whether you are responding to an email, copying/pasting text, or sharing a document for summarization.
• Prioritize the use of an "internal" AI that does not always require a connection to external servers and solutions.
• Continuous "Red Teaming": Rather than waiting for an attack, companies should organize simulations where experts actively attempt to hack the AI using the methods mentioned above. This allows for identifying and fixing vulnerabilities (patching guardrails) before they are exploited by real attackers.
• The principle of least privilege: To limit the impact of a zero-click attack or data exfiltration, the AI agent should only have access to the data strictly necessary for its mission.
• Data provenance: To counter data poisoning, you must ensure the integrity of training data. This involves rigorously verifying sources, using digital signatures to validate documents, and regularly scanning knowledge bases to detect potential fraudulent modifications.
• Log monitoring: Protecting against attacks is good, but being able to detect and trace them is better! This is the principle of "log monitoring." This practice consists of keeping and analyzing all user/AI interactions (prompts, model responses, etc.) to identify suspicious patterns (e.g., repeated injection attempts) and to retrace the sequence of an attack after the fact to understand the vulnerability and adapt defenses.

This topic has become so significant in recent months that the DGSI has looked into it and formulated recommendations that you can find here: Security recommendations for a generative AI system.

Ready to take the leap into AI securely? Contact our experts.

‍