Day 15 of 60
·
Property-based & oracle-free
Metamorphic testing
When you can't say "this output is correct," you can often still say "this output should equal that one." For ML, search, and code generation, it's the only oracle you have.
ProblemNo ground-truth oracle exists, common in ML, search, code generation, simulation.
How it works
You assert relationships that must hold regardless of input. Sort twice ≡ sort once. Translating en→fr→en should land semantically near the original. Useful when "is this output correct" is unanswerable.
What it catches
Whole classes of generation/inference bugs. Surprisingly cheap to write once the team adopts the pattern. Heavily used at Google for search ranking validation.
Tools
(custom assertions in any test framework) · OSS
Verdict by project size
Small
Skip
Medium
Opt
Large
Rec
Extra-large
Must
Cost
| Project size | Setup | Maint / mo | Tool / mo | CI / run |
|---|---|---|---|---|
| Small <10k LOC | 1d | 2h | $0 | +1m |
| Medium 10–100k LOC | 3d | 8h | $0 | +3m |
| Large 100k–1M LOC | 10d | 25h | $0 | +8m |
| Extra-large >1M LOC | 30d | 80h | $0 | +15m |
Setup = engineer-days to first useful run ·
Maint = engineer-hours / month at steady state ·
Tool = out-of-pocket $ / month ·
CI = minutes added (or saved) per pipeline run
Lifecycle & ownership
When in lifecycle
Test
Per merge · Runs after merge to main; nightly heavy jobs.
Who owns it
Developer
Authoring + the inner loop
Collaborates with: QA / Test Engineer
Reference implementations
-
The Fuzzing Book: Metamorphic Testing
Worked examples for testing systems without a simple oracle.
-
Metamorphic testing examples
Worked examples of metamorphic relations across domains.
-
Metamorphic relation examples
Small examples that make oracle-free assertions concrete.
Quick check
When is metamorphic testing the right tool?
One question. Pick the best answer. Your streak is saved locally on this device.
Save the lesson
Download SVG ↓
Screenshot for a 1:1, drop it in Slack, or download the SVG.