Architecture

Mental Model

Every code change passes through four testing layers. Each layer catches a different class of defect at a different cost and speed. Lower layers are deterministic and fast; upper layers are thorough and expensive. Together they form a complete testing strategy that addresses the specific failure modes of AI-generated code.

The key insight: no single testing technique catches all defect classes. Architecture tests catch structural violations. Property-based tests catch computation bugs. Mutation testing measures whether tests would actually catch a bug. Contract tests verify cross-service correctness. Each layer compensates for the blind spots of the others.

Layer Diagram

┌──────────────────────────────────────────────────────────┐
│                   Code Change (PR / Commit)               │
└──────────────┬───────────────────────────────────────────┘
               │
               ▼
┌──────────────────────────────────────────────────────────┐
│  LAYER 0: Architecture Testing               (<10s)      │
│  ArchUnit rules. Deterministic. Zero AI dependency.      │
│  Layer boundaries, banned APIs, naming conventions.      │
└──────────────┬───────────────────────────────────────────┘
               │
               ▼
┌──────────────────────────────────────────────────────────┐
│  LAYER 1: Property-Based Testing              (~2s)      │
│  jqwik properties. 1000+ generated inputs/property.      │
│  Computation correctness. Breaks circular test problem.  │
└──────────────┬───────────────────────────────────────────┘
               │
               ▼
┌──────────────────────────────────────────────────────────┐
│  LAYER 2: Mutation Testing                    (~5 min)   │
│  PIT + AI-augmented mutants. Mutation score as the       │
│  true measure of test effectiveness. Kill the mutants.   │
└──────────────┬───────────────────────────────────────────┘
               │
               ▼
┌──────────────────────────────────────────────────────────┐
│  LAYER 3: Contract Testing                    (~10 min)  │
│  Pact consumer-driven contracts. Cross-service API       │
│  compatibility. No full integration env required.        │
└──────────────┬───────────────────────────────────────────┘
               │
               ▼
┌──────────────────────────────────────────────────────────┐
│  DASHBOARD: Quality Intelligence                         │
│  Mutation score trends, TORS, defect escape rate,        │
│  flake rate, per-layer health. CI feedback loop.         │
└──────────────────────────────────────────────────────────┘

Layer Summary

Key Design Decisions

Why Layers

Each layer catches a fundamentally different class of defect. Architecture tests catch structural violations (controller calling repository directly). Property-based tests catch computation bugs (wrong rounding mode on VAT calculation). Mutation testing catches weak tests (tests that pass regardless of code correctness). Contract tests catch integration breaks (provider changed the API shape).

No single layer covers all four. Running all four layers costs less than one production incident.

Deterministic Backstop

Layer 0 (ArchUnit) is immune to AI regression. It requires no AI model, no LLM call, no network access. The rules are deterministic Java code that runs in the test suite. Even if every AI model degrades, hallucinates, or becomes unavailable, L0 continues to enforce architecture constraints.

This is the floor, the minimum guarantee. In a world where AI-generated code is the majority of new code, having a deterministic backstop that prevents the most common structural violations is non-negotiable.

• 10/10 AI generations violated layer boundaries without ArchUnit rules
• 0/10 AI generations violated layer boundaries with ArchUnit in the test loop
• Violations compile successfully, so only a test gate catches them

Mutation Score Over Coverage

Line coverage measures which lines executed. Mutation testing measures which lines are actually verified. The difference is critical: a test that runs a function but asserts nothing increases coverage but catches zero bugs.

Mutation score is the true metric for test effectiveness:

• Killed mutant: at least one test failed when the code was changed. The test is doing its job.
• Survived mutant: no test failed. Either a missing test or a weak assertion.
• Target: mutation score >80% for business-critical code, >60% for utilities.

Test Oracle Reliability Score (TORS)

TORS measures the reliability of the test suite itself. In environments where 84% of test failures are flaky (Google data), treating every failure as a real regression wastes engineering time and erodes trust in CI.

TORS = (real failures) / (total failures)
If TORS < threshold → test is flagged as unreliable

Unreliable tests are quarantined: they still run, but their results are tracked separately and do not block the pipeline. This prevents the common failure mode where developers stop trusting CI and start merging despite failures.

• TORS > 85%: healthy test suite, failures are real signals
• TORS 50-85%: flake remediation needed, prioritize top offenders
• TORS < 50%: test suite is a liability, not an asset

Testing Strategy Selection

The classic test pyramid is not wrong; it is incomplete. The right testing shape depends on the architecture and risk profile of the system. Visdom Testing supports multiple strategies:

Visdom SDLC Integration

Visdom Testing connects to the broader Visdom AI-Native SDLC metrics framework:

Full metrics framework →