Metrics Framework

Traditional testing metrics (line coverage, branch coverage, test count) measure effort, not effectiveness. Visdom Testing tracks metrics that answer the question that matters: would your tests catch a bug if one existed?

Testing Metrics That Matter

Why Coverage Is Insufficient

Line coverage is the most widely tracked testing metric. It is also the most misleading. Coverage measures which lines executed during testing, not whether the tests would detect a bug on those lines.

The critical insight: by every standard metric, the traditional suite looked better. Higher coverage, higher mutation score, more tests. If you relied on metrics alone, you would conclude PBT added no value. Meanwhile, both computation bugs would ship to production. 90% line coverage while missing every computation error that matters.

Per-Layer Metrics

Layer 0: Architecture Testing

Layer 1: Property-Based Testing

Layer 2: Mutation Testing

Layer 3: Contract Testing

Mutation Score Explained

Mutation testing systematically changes the code (introduces mutants) and checks whether the test suite detects each change. It answers the question: "If the code were wrong, would my tests tell me?"

How PIT Works

PIT (PITest) is the standard mutation testing tool for the JVM. It:

1. Analyzes the codebase to identify mutation points (operators, return values, conditionals)
2. Creates mutants: modified versions of the code with one change each
3. Runs the test suite against each mutant
4. Reports which mutants were killed (test failed) vs survived (all tests passed)

Mutant Classification

Mutation Score = killed / (killed + survived)
               = detected mutations / total non-equivalent mutations

TORS: Test Oracle Reliability Score

TORS measures whether your test suite is a reliable signal or a source of noise. In AI-assisted development environments, unreliable tests cause a specific failure mode: AI agents "fix" passing tests that happened to fail due to flakiness, introducing real bugs in the process.

Definition

TORS = real_failures / total_failures

Where:
  real_failures  = test failures caused by actual code defects
  total_failures = all test failures (real + flaky + environment)

Threshold:
  TORS > 0.85 → test is reliable, include in feedback signal
  TORS < 0.85 → test is unreliable, quarantine from agent feedback

TORS Benchmarks

Defect Escape Rate

Defect escape rate measures the ultimate outcome: how many bugs reach production despite the testing layers. It is the metric that tells you whether the layers are actually working.

Defect Escape Rate = production_bugs / (production_bugs + bugs_caught_in_testing)

Benchmarks

Flaky Test Management

Flaky tests are the single largest source of wasted engineering time in CI. They erode trust, cause false urgency, and in AI-assisted workflows, trigger agents to "fix" code that is not broken.

Industry Data

Detection Strategies

Flake Budget

Following Google's model, Visdom Testing recommends a 2% flake budget:

• At any time, no more than 2% of tests may have a TORS below the reliability threshold
• Tests that breach the budget are automatically quarantined from the agent feedback signal
• Quarantined tests generate a remediation ticket with flake frequency data
• Goal is not zero flakes (impossible at scale) but managed flakes that do not erode trust

Industry Benchmarks

These benchmarks are drawn from published data by teams operating testing at scale:

CI Integration Metrics

Testing effectiveness depends on CI pipeline speed. Slow feedback loops cause developers to context-switch away and merge without waiting for results. Visdom Testing tracks CI-specific metrics to ensure the testing layers provide value within the developer's attention window.

Test Selection

Not every change requires every test. Test impact analysis determines which tests are affected by a code change and runs only those:

Parallelization

Feedback Time

End-to-End: Visdom SDLC Metrics Integration