AI in software - cleverly used, securely protected

Three main categories of attack on LLM-based applications

In addition to known security risks such as SQL injection or cross-site scripting, the use of LLMs gives rise to new, specific threats. They can be divided into three main categories:
1. Poisoning Attacks
In these attacks, the model is deliberately manipulated in order to trigger incorrect behaviour. Two variants are particularly relevant:

• Training data poisoning:
  Malicious content is already placed in the training data.
• Knowledge poisoning:
  The attacker manipulates the context data provided to the model via RAG.

Knowledge poisoning in particular poses a serious risk in our application. An attacker with access to the context data could insert malicious instructions there.

2. Privacy Attacks
Here, the attacker attempts to extract confidential information from the LLM application. Possible scenarios:

• Model theft:
  An attempt is made to reconstruct the model through mass requests.
• Reconstruction of training data:
  The attacker attempts to draw conclusions about the original training data.
• Leakage of system prompts or context data:
  Sensitive information such as API keys, passwords, database names or other sensitive data in context can also be affected. 

Such attacks are generally possible in our example application. It is therefore essential to consistently keep sensitive information out of system prompts and context data and ideally only provide access to non-sensitive data via RAG.

3. Evasion Attacks (Prompt Injection)
These attacks are aimed at changing the behaviour of the LLM by manipulating inputs and, for example, causing it to carry out malicious actions. A distinction is made between 

• Direct attacks: 
  The attacker enters manipulated input directly via the user interface.
• Indirect attacks: 
  Malicious content enters the context via external sources (e.g. websites, databases). 

In our application, both variants are conceivable, for example through manipulated legal texts (which the user enters) or compromised context data (which is consulted by RAG).

Protective measures to reduce the risk

Poisoning attacks can be addressed by two measures in particular: By applying classic IT security measures, access to training data and data provided as context for RAG must be secured, thus protecting it from manipulation. If external sources (e.g. websites, public databases or previous user input) are used for training or as a context, these sources should be selected with care and, if necessary, the data should be checked and cleansed before use.
The risk of privacy attacks and prompt injections can be minimised through various measures: Firstly, for example, sensitive information must be kept strictly out of context (system prompt and data for RAG). If different users require access to different context data, the access rights of the RAG must be linked to the rights of the user. Depending on the criticality of the remaining data, specialised AI models combined with blocklists of text modules can be used to try to detect extraction attempts or other injection attacks (although this cannot offer 100% protection, it does increase the effort and detectability enormously). These measures are rounded off by classic rate limits and monitoring measures to contain attacks and recognise them during operation.

Relevance for our use case

Applied to our sample application, most of the risks presented are certainly relevant for the time being. However, especially as our example is only intended for internal use within the company and works with general company data, the risk and cost of protective measures can be weighed up against each other.

In order to keep the quality of the answers high, the protection of context and training data must be guaranteed in any case. Restricting context data is also important, for example to prevent sensitive employee data from being accidentally made accessible to other employees. The risk of injection attacks, on the other hand, is probably negligible in this example. If the application is made publicly available at some point or is used in a company that is particularly vulnerable, further measures should be considered. 

Other security aspects of AI applications

In addition to the aforementioned attacks, classic risks such as supply chain attacks should also be taken into account, for example through compromised LLMs or RAG components. All libraries and models used should be carefully checked and regularly updated.

A secure development process is essential anyway. This includes, for example

• Threat modelling to identify potential risks and assess the risks in relation to the planned area of application
• Code reviews and penetration testing if necessary
• Integrity checks of all components used

Conclusion: Think about security from the outset

The integration of AI into software applications requires a new level of security awareness. Companies should not view security aspects as an afterthought, but as an integral part of the development process. This is the only way to utilise the potential of AI safely and responsibly.

Do you want to make your AI application secure? We will be happy to support you in the secure implementation of your AI projects or the integration of AI services into existing products.