Skip to content
Journal

Archive //

How to red-team an LLM feature before it ships

A practical pre-launch checklist for product teams: scope, attack categories, evidence, and the questions to answer before real users find the gaps.

This note is from Mallace’s earlier AI red-teaming prototype and is kept as a dated record. Today Mallace finds, proves and fixes security bugs in code. See what it does now.

Launch day is the first real test, unless you run one first

Most AI features ship after a demo, a handful of happy-path prompts, and a review of the system prompt. Then real users arrive with inputs nobody wrote down. Some are curious, some are careless, and a few are deliberately trying to make the system misbehave. Red teaming means being that adversary first, on purpose, while a failure still costs a bug report rather than a screenshot on social media.

Write the scope before the first payload

Name the system under test, the endpoints you may touch, the accounts you will use, and the window you will test in, and get that in writing from whoever owns the system. Scope also defines what failure means for this product: a support assistant that reveals another customer’s order is a different severity than one that tells a bad joke. Without a scope, findings turn into arguments.

Cover the categories that matter for your product

Start with the usual families: direct prompt injection, jailbreaks and role-play escapes, extraction of the system prompt or hidden context, guardrail bypass through encoding or translation, instruction-hierarchy confusion, and confident fabrication. Then weight them. A retrieval assistant over internal documents needs heavy coverage of extraction and indirect injection. A public chatbot needs more jailbreak coverage. An agent with tools needs all of it.

Test the product, not just the model

The model is one component. The failures usually live in the seams: the retrieval step that pulls in an untrusted document, the tool call that runs with more permission than the user has, the output that gets rendered as HTML, the log that stores a pasted secret. Send your cases through the same path a user’s message takes, with the configuration you plan to ship.

Keep evidence you could hand to someone else

For every finding, record the exact input, the full response, the configuration under test, the time, and whether the run was live or simulated. A finding that cannot be reproduced cannot be fixed with confidence, and a pass that came from a timeout or a fallback is not a pass. This is the standard Mallace is being built around: every result labeled, every failure visible.

Make it a gate, then make it a habit

Decide before testing what blocks launch. Then rerun the same cases on every prompt change, model upgrade, and retrieval change, because each of those can quietly reopen a gap you already closed. A one-time exercise tells you about one version of the system. A repeatable campaign tells you whether it is getting better.

See what Mallace does today.

Scanning is free and unlimited. Each fix comes as a pull request with the test that proved the bug.