Day 29 of 60
·
Dynamic, fuzz & dynamic security
Race & concurrency detection
A heisenbug is a bug that fires only when nobody is watching. Race detectors and deterministic schedulers are how you build the watch.
ProblemHeisenbugs that only fire under concurrent load or specific schedules.
How it works
Runtime tools (ThreadSanitizer) instrument every memory access; deterministic schedulers (Loom, Shuttle) explore all interleavings.
What it catches
Data races, lock-order bugs, missed memory barriers, distributed-consistency violations. The hardest bugs to debug post-hoc.
Tools
ThreadSanitizer · OSS Loom (Rust) · OSS Jepsen (distributed) · OSS
Verdict by project size
Small
Skip
Medium
Opt
Large
Rec
Extra-large
Must
Cost
| Project size | Setup | Maint / mo | Tool / mo | CI / run |
|---|---|---|---|---|
| Small <10k LOC | 1d | 1h | $0 | +2m |
| Medium 10–100k LOC | 5d | 5h | $0 | +10m |
| Large 100k–1M LOC | 20d | 25h | $0 | +30m |
| Extra-large >1M LOC | 80d | 120h | $0 | +60m |
Setup = engineer-days to first useful run ·
Maint = engineer-hours / month at steady state ·
Tool = out-of-pocket $ / month ·
CI = minutes added (or saved) per pipeline run
Lifecycle & ownership
When in lifecycle
Build Test
Per merge · Runs after merge to main; nightly heavy jobs.
Who owns it
Security / AppSec
SAST, DAST, threat modelling
Collaborates with: Developer
Reference implementations
-
Loom examples
Deterministic concurrency testing examples for Rust state machines.
-
ThreadSanitizer C++ manual
Race detection setup and examples for C++ code.
-
Shuttle examples
Deterministic concurrency testing examples from AWS Labs.
Quick check
Why are race & concurrency bugs notoriously hard to find post-hoc?
One question. Pick the best answer. Your streak is saved locally on this device.
Save the lesson
Download SVG ↓
Screenshot for a 1:1, drop it in Slack, or download the SVG.