Test properties, not runs.
A unit test checks one run you thought of. A property holds for every run. Generate the runs, find the one that breaks, and shrink it. Everything on this page runs in your browser.
npm install @tinyactors/wasmBrowsers guidenpm install @tinyactors/nodeNode.js and Bun guide1. Write the chart.
A payment charges a card, retries after 30 seconds of silence, and gives up after three attempts.
<scxml xmlns="http://www.w3.org/2005/07/scxml" version="1.0"
datamodel="ecmascript" name="payment" initial="charging">
<datamodel><data id="attempts" expr="0"/></datamodel>
<state id="charging">
<onentry>
<assign location="attempts" expr="attempts + 1"/>
<send type="urn:payments" event="charge" target="card"/>
<send event="timeout" delay="30s" id="timer"/>
</onentry>
<transition event="charged" target="paid"><cancel sendid="timer"/></transition>
<transition event="timeout" cond="attempts < 3" target="charging"/>
<transition event="timeout" target="failed"/>
</state>
<final id="paid"><donedata><param name="outcome" expr="'paid'"/></donedata></final>
<final id="failed"><donedata><param name="outcome" expr="'failed'"/></donedata></final>
</scxml>2. Make the world data.
The chart only sees events. So a world is an array: does the service answer charge 1, charge 2, and so on.
// A run: the chart against a world. The world is data: answers[n] says
// whether the payment service answers the chart's charge number n. Nothing
// else decides what happens, so the same answers make the same run.
export async function run(compiled, answers, { trace = false } = {}) {
const clock = new SimulatedClock();
const charges = [];
const requests = [];
let outcome;
const host = new Host(compiled, {
system: 'property',
clock,
config: { system: { ticksPerSecond: 1000 }, ...(trace ? { telemetry: { resource: { 'service.name': 'payment' } } } : {}) },
...(trace ? { telemetry: (signal, data) => signal === 'traces' && requests.push(data) } : {}),
ioProcessors: {
'urn:payments': {
receive(send) {
const n = charges.push(clock.now) - 1;
if (answers[n]) send.answer('charged');
},
},
},
onDone: (actor, donedata) => { outcome = donedata.outcome; },
});
host.spawn('payment');
await host.idle();
while (outcome === undefined) {
clock.increment(host.nextDeadline); // the next timeout, at once
await host.idle();
}
host.close(); // the last traces go to the telemetry handler here
return { answers, charges, outcome, spans: requests.flatMap((data) => decodeTraces(data)) };
}Run [silence, answer] with tracing on. Select a row to see its transitions.
3. State the properties.
Three rules that must hold whatever the service does.
// Properties: what must hold for every run, whatever the world did.
export const properties = {
'never more than three charges': (r) => r.charges.length <= 3,
'gives up only after three charges': (r) => r.outcome === 'paid' || r.charges.length === 3,
'paid only when the service answered': (r) => r.outcome === 'failed' || r.answers.slice(0, r.charges.length).some(Boolean),
};
export const broken = (r) => Object.keys(properties).filter((name) => !properties[name](r));4. Search a thousand worlds.
A seed makes a world. The search runs seeds until one breaks a property, and the seed reproduces it.
// A generator: a world from a seed, three to eight answers, each yes with
// probability 0.4. The seed reproduces it.
export function generate(seed) {
const random = mulberry32(seed);
return Array.from({ length: 3 + Math.floor(random() * 6) }, () => random() < 0.4);
}
// The search: worlds from seeds 1 to `seeds`, until one breaks a property.
export async function search(compiled, seeds) {
for (let seed = 1; seed <= seeds; seed++) {
const r = await run(compiled, generate(seed));
if (broken(r).length) return { seed, run: r };
}
return null;
}5. Plant a bug.
Retry while attempts < 4. A unit test that answers the first charge still passes.
const buggy = compile(chart.replace('attempts < 3', 'attempts < 4')); // one retry too many
const failure = await search(buggy, 1000);
console.log(`seed ${failure.seed}: ${show(failure.run.answers)} breaks ${broken(failure.run).join(', ')}`);6. Shrink the failure.
Try smaller worlds until none still fails. What is left is the bug and nothing else.
// The shrinker: a smaller world that still breaks a property, until no
// smaller one does. Smaller is one answer fewer, or a yes turned into
// silence.
export async function shrink(compiled, answers) {
let smallest = answers;
for (let changed = true; changed; ) {
changed = false;
for (const candidate of smaller(smallest)) {
if (broken(await run(compiled, candidate)).length) {
smallest = candidate;
changed = true;
break;
}
}
}
return smallest;
}
function* smaller(answers) {
for (let n = 0; n < answers.length; n++) yield answers.toSpliced(n, 1);
for (let n = 0; n < answers.length; n++) if (answers[n]) yield answers.with(n, false);
}The smallest world is empty: the service never answers. The trace shows a fourth charge at 90 seconds.
7. Run it in CI.
The same module runs in Node.js and Bun. A failure names its seed.
// The guide's runs, in Node.js. Run: node main.mjs
import { readFileSync } from 'node:fs';
import { broken, compile, run, search, shrink } from './property.mjs';
const chart = readFileSync(new URL('./payment.scxml', import.meta.url), 'utf8');
const show = (answers) => `[${answers.map((yes) => (yes ? 'answer' : 'silence')).join(', ')}]`;
const correct = compile(chart);
const found = await search(correct, 1000);
console.log(found ? `seed ${found.seed} breaks ${broken(found.run).join(', ')}` : '1000 worlds: every property held');
const buggy = compile(chart.replace('attempts < 3', 'attempts < 4')); // one retry too many
const failure = await search(buggy, 1000);
console.log(`seed ${failure.seed}: ${show(failure.run.answers)} breaks ${broken(failure.run).join(', ')}`);
const smallest = await shrink(buggy, failure.run.answers);
const replay = await run(buggy, smallest, { trace: true });
console.log(`shrunk to ${show(smallest)}: ${replay.charges.length} charges, ${replay.outcome}`);
console.log(`its trace: ${replay.spans.filter((span) => span.name.startsWith('scxml.macrostep')).map((span) => span.name.replace('scxml.macrostep ', '')).join(' → ')}`);
correct.close();
buggy.close();1000 worlds: every property held
seed 10: [silence, silence, silence, silence, silence, answer] breaks never more than three charges, gives up only after three charges
shrunk to []: 4 charges, failed
its trace: bootstrap → timeout → timeout → timeout → timeoutGo further.
This is a pattern, not a library. It grows with your system.
- Richer worldsLate, duplicate and reordered answers are more entries in the array.
- More actorsGenerate the team chat's inputs. Every online member sees every post.
- Trace propertiesSpans are data. Assert on how a run got there, not only where it ended.