November 2026 · version 0.1.0a42
Benchmark.
We ran Mallace on five well-known apps that are vulnerable on purpose, from local copies pinned to exact commits. Fixes were proved and verified locally (mallace fix --no-pr), and nothing touched a live site. These are the real numbers, misses included.
Claude Haiku: proved and fixed in a42
Mallace only fixes a finding after a test it writes fails on your unfixed code. A finding the test can’t reproduce is left alone, not “fixed”.
Claude Haiku 5.5 wrote tests and patches; Claude Sonnet 5.5 reviewed independently. Each app had a $1 budget and a three-finding limit. “Attempted” counts findings that received AI work; budget-denied findings are separate.
| App | Attempted | Fixed and verified | Not reproduced | Failed | Budget stops | Spend upper bound |
|---|---|---|---|---|---|---|
| NodeGoat | 2 | 2 | 0 | 0 | 1 | $0.76 |
| DVWA | 3 | 1 | 1 | 1 | 0 | $0.51 |
| RailsGoat | 1 | 0 | 1 | 0 | 2 | $0.68 |
| Juice Shop | 3 | 3 | 0 | 0 | 0 | $0.55 |
Six fixes verified for a combined reported upper bound of $2.50, about $0.42 per verified fix including unsuccessful work. These are rounded client reservation upper bounds, not provider invoices or a guaranteed price.
NodeGoat verified NoSQL injection and one grouped eval fix; two nearby eval findings were grouped, not counted as extra verified fixes. DVWA verified MySQL, failed to build a working PGSQL test, and did not reproduce one file-read finding. RailsGoat reproduced no fix and stopped at budget on two other findings.
Juice Shop verified command injection, dynamic evaluation and login SQL injection. One nearby command-injection finding was grouped. Other findings, including search SQL injection, were not attempted.
A $1-capped RailsGoat comparison with Sonnet writing and Haiku reviewing also verified no fixes before hitting the budget gate, at a reported upper bound of $0.69. This does not establish Sonnet’s fix quality. Haiku remains the provisional default.
This is not a controlled comparison with the earlier OpenAI runs below: scanner coverage, grouping, test harnesses, budgets and attempted findings changed. We have not established that Haiku matches Terra’s coverage.
Earlier OpenAI results (a40)
| App | Attempted | Fixed and verified | Not reproduced | AI cost |
|---|---|---|---|---|
| NodeGoat | 4 | 4 | 0 | $0.74 |
| DVWA | 5 | 3 | 2 (both real false alarms) | $0.97 |
| RailsGoat | 3 | 3 | 0 | about $0.90 |
| Juice Shop | 6 | 0 | 0 | about $3.60 |
Juice Shop failed in the earlier run because its code couldn’t be loaded in the test sandbox. TypeScript imports, path aliases, CommonJS packages and native add-on support have since improved; the a42 results above include its rerun. RailsGoat’s other two findings weren’t attempted.
Earlier scan results (a40)
The scan is free and runs on your machine. In the a42 rerun it found 18 findings in Juice Shop, 5 in NodeGoat (including NoSQL), 5 in RailsGoat after copied-library filtering, and 37 in DVWA. DSVW was not rerun.
| App | Findings | Notes |
|---|---|---|
| Juice Shop | 16 | Both documented SQL injections found (login and product search). |
| NodeGoat | 4 | All three eval injections and the SSRF. Its MongoDB $where injection was missed in this run. |
| RailsGoat | 12 | 5 in the app’s own code, 7 in copied front-end libraries. |
| DVWA | 37 | Command, SQL and file-read bugs across its lessons. |
| DSVW | 3 | Most of its bugs are missed: the whole app is packed into one-liners a scanner can’t follow. |
Remaining limits
- No checks for access control, authentication or business-logic flaws, which are common serious bugs in real apps.
- Not every finding can be reproduced or fixed within a small budget. RailsGoat and DVWA remain weaknesses in this run.
- Passing these local proof, patch and review gates is limited evidence; it does not establish coverage of every bug or every application dependency.
The benchmark also caught bugs in Mallace itself, and the fixes shipped in 0.1.0a40: large repositories failed to scan, about 30 false alarms, a test sandbox that counted a test that couldn’t run as proof, and a budget that stopped runs far too early.
The apps
| App | Stack | Commit |
|---|---|---|
| OWASP Juice Shop | Node/Express, TypeScript, SQLite | 266448a |
| OWASP NodeGoat | Node/Express, MongoDB | c5cb68a |
| OWASP RailsGoat | Ruby on Rails | f5951f1 |
| DVWA | PHP, MySQL | a1978a2 |
| DSVW | Python, single-file http.server app | 9ca3c9a |