Why failed tests must never become successful evidence

The quiet failure mode that turns an unavailable target into false confidence—and how Mallace is addressing it.

A fallback can change the meaning of a result

A simulated response is useful in a demo. Substituting one after a live connection failure changes what the result means. The operator asked what a real target did; the system answered with something it invented.

Separate execution from evaluation

A timeout, HTTP error, or provider failure must remain observable. A judge should classify an actual response, not compensate for the absence of one. Mallace now stops on live HTTP failures and invalid judge output rather than quietly falling back.

Unknown is a useful state

An unevaluated result should not default to hardened. Mallace now initializes those outcomes as unknown. This avoids confusing missing evaluation with a successful defense.

The work still ahead

Failing fast is only the first step. Per-case failure accounting, durable state, provenance, redaction, and a reviewed benchmark are still needed. Local tests demonstrate these safeguards; they do not demonstrate production security.