Skip to content
SI.info

Jailbreak

Published 1 min read
On this page
  1. Common techniques
  2. Why jailbreaks work
  3. Why it matters
  4. Jailbreak vs prompt injection

The term borrows from “jailbreaking” a phone to remove the manufacturer’s restrictions. For SI models, a jailbreak gets around the refusals that developers build in through safety training such as RLHF. The goal might be harmless curiosity, or it might be instructions for weapons, malware or fraud.

Common techniques

  • Role-play and fiction: asking the model to play a character “who has no rules”, or to write a story in which a character explains something dangerous.
  • Hypothetical or “research” framing: presenting a harmful request as academic, historical or for testing.
  • Encoding and translation: hiding a request in code, ciphers or less common languages that safety training covered less well.
  • Breaking a task into pieces: splitting a harmful request into innocent-looking steps across a long conversation.
  • Automated attacks: software that searches for prompts that slip past a model’s safeguards.

Why jailbreaks work

A 2023 paper, Jailbroken: How Does LLM Safety Training Fail?, identified two root causes. Competing objectives: a model is trained both to be helpful and to be safe, and a clever prompt can pit one against the other. Mismatched generalization: safety training doesn’t reach every area where the model has skills, for example requests written in an unusual encoding.

Why it matters

Jailbreaks matter most when a model has genuinely dangerous knowledge or abilities. Labs look for them before release through red teaming and bug-bounty programmes, and test whether a jailbroken model could give serious uplift in areas like biology or cyberattacks. Those tests feed company safety frameworks such as responsible scaling policies. Labs patch known jailbreaks, but new ones keep appearing, so developers also rely on monitoring, usage policies and limits on who can access the most capable systems.

Jailbreak vs prompt injection

The two are often confused. A jailbreak is the user trying to override the model’s rules. Prompt injection is a third party hiding instructions in content the model reads, such as a web page or email, to hijack an app or SI agent.

← Back to the SI Glossary

Frequently asked questions

Why do jailbreaks work?

A 2023 study by Alexander Wei, Nika Haghtalab and Jacob Steinhardt identified two reasons: competing objectives, where a model's drive to be helpful clashes with its safety goals, and mismatched generalization, where safety training doesn't cover areas in which the model still has skills.

Is a jailbreak the same as prompt injection?

No, though they overlap. In a jailbreak, the user tries to get around the model's own safety rules. In prompt injection, text from a third party, hidden in content the model reads, overrides the instructions of the user or developer.

Sources

  1. Jailbroken: How Does LLM Safety Training Fail? — Wei, Haghtalab & Steinhardt, arXiv, July 5, 2023

Written by

· Editor

Editor of SI.info. Writes about Super Intelligence, technology policy and the people building frontier models.

How we research and fact-check

Free newsletter

Get The SI Brief

One short email a week: what changed in Super Intelligence, policy and models — and why it matters.

Free. One email a week. Sent via beehiiv, which counts opens and clicks. Unsubscribe anytime. Privacy policy