Visdom
Testing

A multi-layered testing strategy for teams shipping AI-generated code. When AI writes the code and the tests, who tests the tests?

For Leaders Technical Reference Before / After

Part of Visdom · VirtusLab's AI-Native SDLC

The circular quality problem

AI generates both implementation and tests. Traditional metrics break down.

01 Circular Tests

Tests pass because they encode the same assumptions as the code.

02 Metric Inflation

90% line coverage, 0 computation bugs caught.

03 Architecture Erosion

AI takes the shortest path. Controllers call repositories directly.

04 API Regression

AI reaches for deprecated APIs from training data.

The evidence

Same agent, same CRUD task, 10 repetitions. The numbers speak for themselves.

ArchUnit experiment — 10 runs each

Without ArchUnit With ArchUnit
Layer bypass 10/10 violations 0/10 violations
Field injection 1/10 0/10
Generic exception 1/10 0/10
Compilation required? Yes (violations compile) Violations fail the build

Property-Based Testing vs Traditional — metric comparison

Metric Traditional Property-Based Combined
Line coverage 90% 80% 90%
Mutation score 73% 55% 73%
Bugs found 0/2 2/2 2/2
90%
line coverage while missing both computation bugs in a pricing module
VirtusLab Property-Based Testing Case Study
84%
of pass-to-fail test transitions are flaky, not real regressions
Micco, ICST 2017 (Google)
1.7x
more issues in AI-generated PRs than human-written ones
CodeRabbit (2025)
73%
acceptance rate for AI-generated mutation tests
Meta ACH (2025)
The critical insight: By every standard metric, the traditional suite looked better. Property-based testing found both computation bugs — early rounding of discount rate and wrong rounding mode on VAT — that 90% line coverage and 16 hand-written tests missed entirely.

Sound familiar?

Real patterns from enterprise teams adopting AI-assisted development.

Your Engineering ManagerKatja's problem

"Our test suite takes 45 minutes and 84% of the failures are flaky. Developers don't trust CI anymore."

Your QA LeadTomek's problem

"We hit 90% coverage then three pricing bugs shipped. The AI copied the implementation logic into the assertions."

Your Senior DeveloperPriya's problem

"Copilot bypassed the service layer, used RestTemplate instead of RestClient, and the tests mocked everything."

No single shape fits all

The right testing strategy depends on your architecture. Visdom Testing adapts to your stack.

▲

Pyramid

Classic. Many unit, fewer integration, minimal E2E. Best for monoliths.

 ★

Trophy

Integration-heavy. Kent C. Dodds model. Best for modern frontend.

 ⬢

Honeycomb

Spotify model. Contract-focused. Best for microservices.

 ◆

Diamond

Wide integration layer. Best for domain services.

Go deeper

SCENARIOS

Before / After Scenarios

4 scenarios showing what changes when Visdom Testing is deployed.

 REFERENCE

Technical Reference

Full architecture, layer docs, configuration, metrics framework.

 LAYER 0

Architecture Testing

ArchUnit experiment: 10/10 violations without, 0/10 with.

From the field

RESEARCH

DORA 2025: AI Adoption Impact

90% AI adoption correlates with 9% bug rate increase and 91% more code review time.

Google DORA

 RESEARCH

Meta: LLM-Powered Bug Catchers

73% acceptance rate. 49% of tests caught faults invisible to line coverage.

Meta Engineering

 RESEARCH

PBT Finds 50x More Mutations

Each property-based test finds ~50x as many mutations as the average unit test.

OOPSLA 2025

 RESEARCH

AI Code: 1.7x More Issues

AI PRs average 10.83 issues vs 6.45 for human PRs across 470 pull requests.

CodeRabbit (2025)

 INDUSTRY

Atlassian: AI Mutation Coverage

56% to 80% mutation coverage on real Jira projects.

Atlassian

 RESEARCH

Stanford: AI Makes Code Less Secure

Developers with AI assistants wrote less secure code but believed it was more secure.

Stanford / CCS 2023

 RESEARCH

Google: Flaky Tests at Scale

84% of pass-to-fail transitions are flaky. 1.5% of all executions report flakiness.

Micco, ICST 2017

 RESEARCH

Endor Labs: AI Design Flaws

15/20 AI completions had design flaws. 12/20 exhibited design pattern drift.

Endor Labs (2025)

FAQ
See all questions →
Part of Visdom

VirtusLab's AI-Native SDLC

ViDIA Context Fabric
Fast Feedback Machine-Speed CI
VCR Code Review Testing Quality Assurance

Read the
full reference

Architecture, layer docs, metrics framework, and the evidence behind each technique.

Technical Reference Before / After Scenarios