Day 17 of 60
·
Property-based & oracle-free
Differential testing
Two implementations of the same thing? Run identical inputs through both. Disagreements are bugs in at least one. This is how the LLVM and OpenSSL teams find the bugs nobody else can.
ProblemNo oracle, but multiple candidate implementations.
How it works
Run identical inputs through two implementations (different model, different seed, reference vs. optimized) and diff outputs. Disagreements are bugs in at least one. Used by compiler teams, browsers, fuzzers.
What it catches
Logic divergence, optimisation bugs, model drift. Has a long track record in high-stakes systems (LLVM, V8, OpenSSL).
Tools
Csmith (compiler) · OSS Frankencerts (TLS) · OSS (custom diffing harnesses) · OSS
Verdict by project size
Small
Skip
Medium
Opt
Large
Rec
Extra-large
Must
Cost
| Project size | Setup | Maint / mo | Tool / mo | CI / run |
|---|---|---|---|---|
| Small <10k LOC | 2d | 2h | $0 | +2m |
| Medium 10–100k LOC | 5d | 10h | $0 | +5m |
| Large 100k–1M LOC | 20d | 40h | $0 | +15m |
| Extra-large >1M LOC | 60d | 150h | $0 | +30m |
Setup = engineer-days to first useful run ·
Maint = engineer-hours / month at steady state ·
Tool = out-of-pocket $ / month ·
CI = minutes added (or saved) per pipeline run
Lifecycle & ownership
When in lifecycle
Test
Per merge · Runs after merge to main; nightly heavy jobs.
Who owns it
Developer
Authoring + the inner loop
Collaborates with: QA / Test Engineer
Reference implementations
-
Csmith
Classic differential-testing generator for finding compiler bugs.
-
Frankencerts
Differential testing corpus for certificate validation behavior.
-
SQLancer
Differential and oracle-based testing framework for database management systems.
Quick check
Differential testing requires what to be useful?
One question. Pick the best answer. Your streak is saved locally on this device.
Save the lesson
Download SVG ↓
Screenshot for a 1:1, drop it in Slack, or download the SVG.