Archive //
Your eval suite is not a red team
Benchmarks and evals measure expected behavior. Adversarial testing looks for the behavior you did not expect. Shipping AI safely needs both.
This note is from Mallace’s earlier AI red-teaming prototype and is kept as a dated record. Today Mallace finds, proves and fixes security bugs in code. See what it does now.
Two questions that sound alike
An eval asks whether the system does what you designed it to do: answer correctly, follow the format, stay on topic. A red team asks what the system can be made to do by someone who does not share your goals. Teams often have the first and assume it covers the second. It does not, because the inputs that matter to an attacker are exactly the ones nobody put in the eval set.
Fixed sets go stale
A static benchmark is valuable for catching regressions, but it has a short half-life as a security measure. Public jailbreak lists get patched against. Your own test cases shape the prompt you tune. Over time the system learns to pass the test, and the score improves while real-world resistance may not.
Averages hide the failure that matters
A 98 percent pass rate sounds strong until the two percent includes leaking a customer record. Adversarial testing is less about the average and more about the worst case someone can reach on purpose. Report findings individually, with severity tied to real impact in your product, rather than folding them into a single number.
Judges need judging
Many evaluation pipelines use one model to grade another model’s output. That is practical, but a judge can be wrong, can fail to parse, or can be influenced by the content it is grading. When a judge errors, the honest result is unknown, not passed. Spot-check judged results by hand, and keep the raw response alongside every verdict.
Use both, on purpose
Keep evals for correctness and regression. Add adversarial campaigns for resistance, scoped to your product’s real risks and rerun as it changes. When a red-team finding is fixed, turn the case into a regression test so it stays fixed. That loop, from finding to fix to retest, is where the two practices meet.