Building an Input/Output Safety Classifier Pipeline (and Trying to Break It)

A lot of modern LLM safety work comes down to one idea: don't just deploy a single model and hope for the best. Wrap it. Check what goes into the model and what comes out of it, using a second, smaller model trained specifically to recognize harmful content. For my BlueDot Technical AI Safety course (Minimising Harm), I built one of these pipelines from scratch using open-weight models, then spent a few days trying to break it.

Why I built this

Reading about classifier-based guardrails in a paper is one thing. Wiring one up yourself, watching it block (or fail to block) real prompts, and seeing exactly where it falls apart is another. I had two goals:

1. Build a working input/output (I/O) classification pipeline using only open-weight models, running on a free Google Colab GPU.
2. Red-team my own pipeline, and use that to understand the difference between jailbreak techniques that generalize (universal jailbreaks) and ones that don't (targeted, contextual jailbreaks).

The stack

Everything runs locally inside a Colab notebook, on a free T4 GPU, using Ollama to serve both models:

• Primary model: gemma3:12b. Google's open-weight Gemma 3, generating the actual responses to user prompts.
• Classifier model: llama-guard3:8b. Meta's Llama Guard 3, built specifically for safety classification. Instead of generating free-form text, it takes a conversation turn and returns a safe/unsafe verdict plus the category it matched (S2 for non-violent crimes, for example), pulled from a fixed taxonomy covering things like violent crimes, hate, self-harm, privacy, and weapons.

I picked Llama Guard 3 over alternatives like ShieldGemma because it's built for this exact job, has a clear category taxonomy, and runs through the same Ollama setup as the primary model.

The architecture

Two checkpoints around one generation step:

user prompt
    │
    ▼
[Llama Guard: classify input] - unsafe──▶ BLOCKED (never reaches gemma3)
    │ safe
    ▼
[gemma3:12b: generate response]
    │
    ▼
[Llama Guard: classify full exchange] - unsafe──▶ BLOCKED (never returned to user)
    │ safe
    ▼
response returned to user

Every decision gets logged; both classifier verdicts, the raw model output, and where (if anywhere) the request got blocked. That log ended up being the most useful part of the whole project, because it's what let me actually compare attack strategies afterward instead of just eyeballing pass or fail.

Setting it up in Colab

A few notes here.

• Ollama's installer needs zstd to unpack itself, and Colab's base image doesn't have it. Running apt-get install zstd before the install script fixes it.
• GPU detection needs pciutils (lspci). Without it, Ollama can't see the T4 and quietly falls back to CPU, which would make a 12B and an 8B model painfully slow. Install pciutils and lshw alongside zstd and you're fine.
• Colab doesn't run systemd, so Ollama's systemd service creation throws a harmless warning. The fix is to skip the service and just launch ollama serve as a background process from the notebook.

Once that's sorted, pulling both models and running the pipeline is straightforward. ollama.chat() calls to each model, wrapped in a small Python class that handles classify, generate, classify, and logs everything to a list of dicts I could load into a pandas DataFrame later.

Red-teaming my own pipeline

The point wasn't to prove an LLM can be jailbroken. That's already well established. The question I cared about was where, specifically, a layered defense breaks down, and whether it breaks down differently depending on the kind of attack.

I tested two categories.

Universal jailbreaks

A universal jailbreak is a single reusable template, a persona or framing device or instruction override, that you can drop almost any harmful request into and expect some uplift, often across multiple models. I tested three classic patterns:

• DAN ("Do Anything Now"). Claims the model has broken free of its constraints and adopted an alter-ego that doesn't refuse requests.
• "Developer Mode." Claims to be an internal debug mode that disables content filters.
• The "grandma exploit." Asks the model to role-play a deceased relative gently explaining something, leaning on empathy instead of authority.

I ran each of these against the same three mild test payloads (kept low-stakes on purpose, things like lock-picking mechanics, so I was testing classifier behavior and not generating anything genuinely dangerous), giving a 3x3 grid of attempts.

0 out of 9 universal jailbreak attempts achieved a full bypass.

The pattern was identical across all three templates. Roughly a third of attempts slipped past the input classifier and got caught at output instead, and the rest never made it past input at all. It also wasn't always the same payload getting through for a given template. That tells me Llama Guard isn't just pattern-matching the wrapper text on its own. It's weighing the wrapper and the payload together, since the same template produced different input verdicts depending on what was inside it.

The two-checkpoint design earned its keep here. Every attempt that slipped past input classification got caught at output instead. A single-checkpoint, input-only system would have let about a third of these through.

Targeted (contextual) jailbreaks

A targeted jailbreak skips the generic wrapper and hand-crafts a single prompt around one specific request, using plausible context like a professional or academic framing instead of any "ignore your instructions" language. It doesn't generalize to other payloads, but it also doesn't carry a recognizable signature for a classifier to catch.

I tested this with one hand-written prompt, framing a request as filling a gap in a locksmithing certification textbook, in the same instructional tone as the rest of a fictional chapter, with no persona claims and no override language.

Full bypass. This was the only prompt in the entire test set, across both categories, that scored safe at both checkpoints. Gemma3 generated the requested content and Llama Guard waved it through twice.

One result isn't a rate, and as the follow-up below shows, it wasn't one. But it was enough to make me want to test whether the framing itself was doing the work, or whether this particular payload was just easier to miss.

The follow-up: does the framing generalize?

One bypass could mean the professional-framing technique works in general, or it could mean lock-picking specifically sits in gray-enough territory that almost any framing would get through. To find out, I reused the same "textbook gap" structure — same tone, zero override language — and applied it to two different mild payloads I'd already tested under the universal templates: shoplifting concealment and software license bypass.

The framing didn't generalize. Both new attempts got blocked at input, despite using the identical technique that got the lock-picking prompt through. Only the original payload bypassed.

That's a more useful result than a clean "framing always wins" would have been. It means Llama Guard wasn't just failing to detect the absence of override language. It was still weighing the actual content of the request. Lock-picking sits in genuinely ambiguous territory: it's legal to learn, locksmith certification is a real credential, and a classifier scoring that as lower-risk is a defensible call, not obviously a failure. Shoplifting concealment and license circumvention read as unambiguously about facilitating one specific illicit act no matter how professionally they're dressed up, and the classifier treated them that way even with zero manipulation signal present.

So the more accurate version of my earlier finding: professional, override-free framing isn't a free pass on its own. It mattered for one borderline-severity payload and made no difference for two clearer-cut ones. The open question now is where exactly that boundary sits, and whether it holds across a wider range of payload severities.

What I learned

0 out of 9 universal jailbreak attempts achieved a full bypass. Only 1 of 3 targeted contextual attempts did. That's still a real gap between the two categories, even after the follow-up tempered my first, too-confident read of that single result.

A few things this tells me about the pipeline, and about classifier-based defenses generally:

• The classifier weighs wrapper and payload together, not the wrapper alone. If it were just pattern-matching phrases like "Do Anything Now," every payload inside that template should have gotten the same input verdict. It didn't. That's more sophisticated than I expected going in.
• The two-checkpoint design did real work. Every universal-template attempt that got past input classification was caught at output instead. An input-only pipeline would have let about a third of these through.
• Content severity matters more than I gave it credit for at first. My first read of the locksmithing result was that dropping override language was the key to a bypass. The follow-up corrected that. The same framing failed against two other mild payloads. Conclusively, professional, override-free framing can matter at the margin, for content that's already ambiguous enough to be a defensible call either way, but it isn't a general-purpose bypass technique.

Where this goes next

A few things worth trying if I keep building on this:

• Multi-turn attacks. The pipeline is stateless right now. Each run() call has no memory of prior blocked attempts. A more realistic adversarial setup would let an attacker escalate gradually across turns, which is a known weak spot for single-turn classifiers.
• Comparing classifier models. Swapping Llama Guard 3 for ShieldGemma, or running both side by side, would show whether these findings are specific to one classifier's training data or reflect a broader weakness in template-based detection.
• More payloads, more templates. A handful of examples is enough to spot a pattern, not enough to claim a real catch rate. The next version of this needs a much bigger batch.

Built as part of a BlueDot Technical AI Safety course, using open-weight models (Gemma 3 and Llama Guard 3) running locally on Google Colab. Check code on my GITHUB.