Jailbreak
The term borrows from “jailbreaking” a phone to remove the manufacturer’s restrictions. For SI models, a jailbreak gets around the refusals that developers build in through safety training such as RLHF. The goal might be harmless curiosity, or it might be instructions for weapons, malware or fraud.
Common techniques
- Role-play and fiction: asking the model to play a character “who has no rules”, or to write a story in which a character explains something dangerous.
- Hypothetical or “research” framing: presenting a harmful request as academic, historical or for testing.
- Encoding and translation: hiding a request in code, ciphers or less common languages that safety training covered less well.
- Breaking a task into pieces: splitting a harmful request into innocent-looking steps across a long conversation.
- Automated attacks: software that searches for prompts that slip past a model’s safeguards.
Why jailbreaks work
A 2023 paper, Jailbroken: How Does LLM Safety Training Fail?, identified two root causes. Competing objectives: a model is trained both to be helpful and to be safe, and a clever prompt can pit one against the other. Mismatched generalization: safety training doesn’t reach every area where the model has skills, for example requests written in an unusual encoding.
Why it matters
Jailbreaks matter most when a model has genuinely dangerous knowledge or abilities. Labs look for them before release through red teaming and bug-bounty programmes, and test whether a jailbroken model could give serious uplift in areas like biology or cyberattacks. Those tests feed company safety frameworks such as responsible scaling policies. Labs patch known jailbreaks, but new ones keep appearing, so developers also rely on monitoring, usage policies and limits on who can access the most capable systems.
Jailbreak vs prompt injection
The two are often confused. A jailbreak is the user trying to override the model’s rules. Prompt injection is a third party hiding instructions in content the model reads, such as a web page or email, to hijack an app or SI agent.
Frequently asked questions
Why do jailbreaks work?
A 2023 study by Alexander Wei, Nika Haghtalab and Jacob Steinhardt identified two reasons: competing objectives, where a model's drive to be helpful clashes with its safety goals, and mismatched generalization, where safety training doesn't cover areas in which the model still has skills.
Is a jailbreak the same as prompt injection?
No, though they overlap. In a jailbreak, the user tries to get around the model's own safety rules. In prompt injection, text from a third party, hidden in content the model reads, overrides the instructions of the user or developer.
Sources
- Jailbroken: How Does LLM Safety Training Fail? — Wei, Haghtalab & Steinhardt, arXiv, July 5, 2023
Written by
Luka Kušec · Editor
Editor of SI.info. Writes about Super Intelligence, technology policy and the people building frontier models.