Day 16 of 60
·
Property-based & oracle-free
Mutation testing
A green test suite is comforting, until a tool flips a == to != and your suite still passes. Mutation testing is the moment you discover you were comforting yourself with theatre.
ProblemA test suite passes; but how do you know it would catch a real bug?
How it works
Tool mutates the source (flip ==, change +/-, remove a line) and reruns the suite. Mutants that survive expose weak tests. Most useful as a quality signal on the suite itself.
What it catches
Vacuous tests, missing assertions, untested branches that coverage tools count as covered. Run nightly, not per-PR; CI cost is heavy.
Tools
Stryker (JS/.NET) · OSS Mutmut / mutpy (Py) · OSS Pitest (JVM) · OSS
Verdict by project size
Small
Skip
Medium
Opt
Large
Rec
Extra-large
Must
Cost
| Project size | Setup | Maint / mo | Tool / mo | CI / run |
|---|---|---|---|---|
| Small <10k LOC | 1d | 1h | $0 | +5m |
| Medium 10–100k LOC | 3d | 4h | $0 | +30m |
| Large 100k–1M LOC | 10d | 20h | $0 | +120m |
| Extra-large >1M LOC | 30d | 80h | $0 | +300m |
Setup = engineer-days to first useful run ·
Maint = engineer-hours / month at steady state ·
Tool = out-of-pocket $ / month ·
CI = minutes added (or saved) per pipeline run
Lifecycle & ownership
When in lifecycle
Test
Per merge · Runs after merge to main; nightly heavy jobs.
Who owns it
Developer
Authoring + the inner loop
Collaborates with: QA / Test Engineer
Reference implementations
-
StrykerJS examples
Mutation-testing examples that measure whether tests catch injected faults.
-
PIT mutation testing
Mature JVM mutation-testing implementation with real-world integration patterns.
-
mutmut
Python mutation testing tool focused on practical local developer workflows.
Quick check
Mutation testing answers which question?
One question. Pick the best answer. Your streak is saved locally on this device.
Save the lesson
Download SVG ↓
Screenshot for a 1:1, drop it in Slack, or download the SVG.