“Prompt injection”: what is this new attack against AI?

The UK national cybersecurity center (NCSC) issued a warning this week about the use of artificial intelligence. She begins by highlighting the growing number of applications and companies that are using APIs (Application Programming Interface) to integrate the Large Language Models (LLMs) that power new ChatGPT-like AIs.

According to the center, the LLM market is changing so rapidly that an AI start-up could no longer exist within two years, or the LLMs behind APIs could have changed so much that certain features that businesses depend on might have disappeared. However, the center also warns of a new cyber threat dubbed “ prompt injection ».

Instructions hidden in the processed data

This attack consists of manipulating requests to circumvent the security integrated into the LLM, in particular by relying on the fact that AIs cannot distinguish between requests and the data necessary to execute the request. The center illustrates this problem with the example of a bank which would create an assistant able to carry out the instructions of the customers. An attack could consist of sending a transaction request whose reference would hide a request. When the AI ​​analyzes the transactions at the request of the customer, it executes the hidden request and sends money to its sender without the knowledge of the victim.

This kind of attack is not purely theoretical. Earlier this year, a cybersecurity researcher demonstrated an attack of the type ” prompt injection by inserting a query into a YouTube video transcript, instructing to introduce yourself as a hacker and add a joke. He then used a ChatGPT plugin whose functionality is to summarize videos. Upon encountering the hidden query, the AI ​​executed it, adding the requested text in the chat. Although harmless, this demonstration shows the potential of this type of attack to manipulate large language models. The NCSC concludes by emphasizing the need to exercise caution in the face of new technology whose capabilities, weaknesses, and loopholes we do not yet fully understand.