Security testing a chatbot. A new game, or the same old same old?

A cartoon of a hacker appearing to try to intimidate a chatbot into typing in strange arbitrary database commands

There is an LLM based chatbot pattern that allows users to describe what they want from system or service via natural language, for which the chatbot uses some LLM to generate a script based on the user’s request that is then executed against the service, the results returned to the user.

The problem I am curious about is how to test for security vulnerabilities where a malicious end user attempts a jailbreak via the LLM prompt and gets it to craft a script under end user control that executes against the service.

Testing a Hypothetical Versus a General Idea

I am going to describe the problem using a hypothetical example derived from a real, in-market product demo. HOWEVER, I am really trying to describe a general testing idea, a concern or pattern of exploration which might generalize against all similar instances.

I am also describing a problem that is already well understood and acknowledge with this type of application. Somebody recently shared with me reference material about the risks of hijack and jailbreak on LLM based chatbots that connect to external services. The following are examples of jailbreak technique and security mitigation design patterns.

Note that the product demo I refer to below is only an example that triggered my imagination. I have not performed any test of the sort described here against this product or the demo code in question. I talk about hypothetical problems which might exist with a product of this type, and in no way am I saying that these vulnerabilities actually exist in any products mentioned here.

I should also say I am inventing this test approach completely from my imagination. I don’t have a specification or code in front of me. I don’t know how well the approach actually works. This is pure invention, pure thought experiment. It is probably going to stay that way until I find myself having to test an application like I talk about here. I look forward to it.

Snowflake logo Example App: Snowflake demo script

I am going to use a demo application idea from Snowflake because the idea is easier to understand with a working example. There are lots of other instances of people doing this sort of chatbot application, there is nothing unique about Snowflake or OpenAI in this regard, it could be any service providing backend data services combined with any LLM offering natural language generative AI.

Snowflake is a cloud based data storage, processing and query service. It is a platform for building data-driven solutions. One of the solution examples they offer on their website is using LLMs in combination with their data APIs to build a chatbot that will query Snowflake data on the user’s behalf.

diagram of the chatbot system described in this document

The demo app works as follows:

  • Some kind of business data is present on the customer’s Snowflake account as tables and views
  • The customer has an OpenAI account (just an example, could be any LLM) to execute LLM queries
  • A python application presents metadata about the customer data set for use by the chatbot
  • The python scripts wrap the metadata about the data into a prompt which instructs OpenAI to use the table information to build an SQL query that satisfies whatever the user asks for
  • When the user asks for something from the data (e.g. “Show all sales data for the years 2017-2018 in New York”), the SQL query contructed by the LLM is passed to SnowFlake’s API and executed
  • The result set is returned to the user

You can watch the video demo to see some of the prompt, although they don’t describe it in-depth. Most of the work is about piecing together the table metadata with constraints on the prompt behavior to get it to create relevant SQL.

Testing Idea: Can we do damage to our service by jailbreaking the prompt?

Quick and dirty threat model

Let’s pretend we are testing an instance of this demo put in place by our IT department. Let’s consider the assets a malicious third party might be interested in:

A diagram of the chatbot described here with threats listed below on diagram

  • Service compute time: it costs us money to run queries, and the payment model increases the heavier the query
  • Backend information exposure: there may be data in tables that the specific user is not supposed to see
  • Backend information data integrity: there may be data we do not want changed, deleted, or added to

The entry point available to the end user is the chatbot interface. If they were trying to jeopardize one of the assets listed above, they may do so via inputs to the chatbot.

There are two ways of thinking about the testing of this:

Diagram of a jailbreak attempt test

  • “jailbreak” the prompt: e.g. requests like “what are all the tables and column names for each that you can query in this SnowFlake account?” or even “Build a query that contains the following Select * from…” It is likely the chatbot needs something more sophisticated than that, but the point is to attempt to override whatever instructions the chatbot prompt wrapper has provided and submit user controlled queries instead that get access to interesting assets.

Diagram of test which uses an LLM mock to simulate an assumed successful jailbreak

  • “assume jailbreak” and look at potential damage: perhaps just survey the configuration, but also perhaps via testability hooks substitute a mock for the LLM and see how much damage one can do with arbitrary prompting

For each of these approaches, “jailbreak” is an arms race in cleverness. It is very likely that no matter how clever and robust a prompt we create, some end user is going to figure out how to get it to craft their own SQL. Not that we should not explore this avenue, but it is the weaker layers of protection, analogous to client-side parameter checking.

By contrast, “Assume jailbreak” is easier to test, and leads toward more robust designs when we expose a vulnerability. Before testing, we should audit the design and just ask “if someone had managed to get arbitrary SQL code from the LLM, what would happen if that SQL looked like (this…)?” and everywhere we believe arbitrary SQL could cause a problem implement a design that guards against it. Use defense in-depth to protect the inner layers from outer. Give the chatbot least priveleges. Partition data security so that the chatbot can only see data and views that satisfy the kinds of questions we care about.

This changes the testing approach from a war of cleverness to an audit of of configurations and settings. We certainly should complement our now-confirmatory series of checks with some attempts at outsmarting the bot, but we should no longer rely on having to be that clever as our only protection.

Is that testing, or design review?

Both. Design review is a form of testing. When we ask questions like “What would happen here if we did this…?” we are testing the design.

But we should also match whatever changes are made in the design to protect against the threats we describe with instances of exactly that threat. If we say the system needs to have layers of protection against malicious SQL coming from the chatbot, then we ought to actually try something going after every asset of interest.

This doesn’t sound much like testing AI anymore

I hope that isn’t disappointing. Maybe you were wanting some list of clever, tricky ways to get a chatbot to hallucinate or do something it claims it cannot do. There will be time for that.

But more important, we want the system safe EVEN IF the LLM goes outside intended results. LLMs are difficult to control, and we are not going to succeed if we have to rely on catching all the ways they can be tricked into going outside the parameters of the prompts the app tries to wrap it in. We want to make the testing problem easier on ourselves by testing in a mode that assumes the bot was already compromised.

In this case the problem we are talking about is no different than if the solution had been implemented without an LLM, without a natural language chatbot. Imagine a web page with drop-down selectors and filters that then constructed an SQL query sent to a back-end service. A hacker attacking that system would try to do the same thing, get control of the SQL statement to exploit assets on the system.

The threat isn’t really the LLM. The threat is what the application in question does with the LLM responses. In this case, based on real demos from real in-market products, the application takes the LLM response and processes it as database code. Imagine that instead of SQL queries the application was building Python scripts and running those on the user behalf. If someone implementing either pattern is not careful they may be exposing themselves to problems.

The goal of testing is to learn about the system, and discover things that may be problems.

When describing the approach, notice that I started with more of a design analysis. I started with threat modeling (a very abbreviated form of it).

It is worth noting that the test analysis become both a product design review as well as an approach for performing actual tests. The response to the design review would have substantial impact on the testing difficulty, costs, and risks of missing something.

Use of generate AI is going to increase as more and more products and services and tools utilize their capabilities of processing fuzzy inputs and creating outputs. It is important we generate testing strategies that are suited to address the inherent risks in what the application iself needs to do. We cannot ignore some of the differences generative AI brings to the problem space, but a lot of the time we will find success by turning the problem back into more conventional modes of design and testing.