flattrace — flat file, deep trace

Flat file, deep trace. flattrace records what a tool did as small, local JSONL events: nested steps, outcomes, errors and measured durations, even across child processes. Python, Rust and TypeScript writers share one v1 envelope. Check traces, find slow steps and redact a bundle for a bug report—all offline, without a collector, server or account. Code, docs and examples are MIT licensed.

What one trace line is

Every flattrace line has the same fixed envelope. That's the format, and it never changes per tool. Here is one line a Pi agent session wrote when a bash tool call finished:

{"v":1, "seq":41, "t":"2026-10-27T10:31:05.112Z", "tool":"flattrace-pi", "tool_version":"0.1.0",
 "trace":"ae77…", "run":"r-7f3…", "span":"58ae…", "parent":"12bc…",
 "event":"pi.tool", "state":"end", "took_ms":610, "outcome":"ok",
 "data":{"toolName":"bash","toolCallId":"c1","exitCode":0,"truncated":false,"outputBytes":2210}}

The envelope is the same for every tool in the world. That's what makes flattrace one format: any reader can pair starts with ends, build the span tree and compute durations without knowing anything about Pi.
event is a static name like pi.tool.
data is free-form. Its meaning belongs to whoever declared the event.

So flattrace standardises the shape of a record, and each tool owns the meaning of its events.

The vocabulary: who says what an event means

Each owner publishes a flattrace-events.json, a dictionary of the events it writes:

{ "tool": "flattrace-pi",
  "events": {
    "run":        {"kind":"span"},
    "pi.tool":    {"kind":"span", "fields":{
                     "toolName":"Registered tool name",
                     "outputBytes":"Returned bytes when exposed, otherwise null",
                     "truncated":"Whether Pi cut the output (boolean)"}},
    "pi.journal": {"kind":"point", "fields":{"entryId":"Exact persisted journal entry ID"}} }}

What the checker enforces (SPEC, "Tool event files and checking"):

every event name in the trace exists in the dictionary;
its kind matches: a span has a start and an end, a point is a single line;
every span is paired, and an unfinished one is reported as unfinished;
optional required events, and per-commands requirements (for example, "a capture run must contain capture.fetch");
optional detail levels: debug-level events can't appear in a normal-level run.

What the checker doesn't enforce: the payload. Field descriptions are documentation with units, not a schema. That's deliberate: tools keep evolving their data without a schema migration, and their own tests own the meaning.

That's why the vocabulary matters so much. It's the contract that keeps a trace honest. Without it, a typo like pi.tol or a span that never ends passes silently.

Several owners, one run

A real agent loads extensions. Each one writes its own events into the same run, and each ships its own small dictionary:

At runtime:

Pi starts and loads flattrace-pi. It opens the process's single writer and starts the run span.
Each extension that loads calls flattrace-pi's extension point, registerVocabulary(path), before the run starts. Its events then go through the same handle (withSpan, emitPoint), which refuses any name outside the registered dictionaries.

Before the run start line is written, flattrace-pi records which dictionaries this run uses, each by fingerprint:

"data":{"vocabularies":[{"tool":"flattrace-pi","version":"0.1.0","sha256":"3f…"},
                        {"tool":"todo","version":"1.2.0","sha256":"a9…"}]}

Every event goes through the one writer, in order, with seq counting up.

Checking:

flattrace check run.jsonl --events flattrace-pi.json --events todo.json

The run is validated against the union of the dictionaries.
Collisions are errors. If two dictionaries define todo.add differently, the checker names both files. Identical duplicates are fine; for example, every file declares the reserved run.
Fingerprints are checked. If the supplied todo.json isn't the one the run declared (an older or newer version), the checker says so. You always know exactly which contract a trace was written under, even years later.
No merged copy. The run points at its dictionaries by fingerprint and nobody writes a combined file, because a copy could drift.
Who is tool? The envelope tool is the program that wrote the run; each event belongs to the dictionary that declares it. Each owner's required and commands rules apply only to its own events.

The whole extension story in one sentence: install flattrace-pi, ship your own small dictionary, register it, and your events sit in the same trace as Pi's, checked together. Any other program works the same way with its own recorder; flattrace-pi is just the one for Pi.

Why not the alternatives

Option	Why not
One big dictionary for every tool	Owners don't own their events; every change edits a hidden central registry.
A separate trace file per owner	Breaks causality: a `todo.add` must sit inside the `pi.turn` span that caused it, on one timeline with one `seq`.
Write a merged dictionary into each run	A second authority that drifts, and it bloats every trace.
Make `data` a strict schema	Payloads evolve without migrations, and each tool's tests own their meaning.

How this serves reading and learning

Because every event has an owner, a version and a fingerprint:

Readers can trust names. pi.tool always means the same thing for a given flattrace-pi version, so recipes ("all truncated tool outputs the next turn relied on") are deterministic code, not guesswork.
Detectors and graders write the same way: their annotation and verdict events come from one shared vocabulary, schema/conventions-events.json, and are checked like any other event. See docs/learning.md.
Benchmarks can tell whether two runs were written under the same dictionaries; comparability itself comes from the task, seed and hash block in docs/learning.md.
Findings know where they belong: the event's owner tells you whose code a problem is in.

An honest open limit: there is no shared monotonic clock across processes, so exact cross-process self time is reported as unknown rather than guessed (see ROADMAP.md).

When not to use flattrace: it is not live monitoring. There is no collector, sampling or alerting, only small local files studied after the run.

Try it

The smallest run, three lines:

{"v": 1, "seq": 0, "tool": "app", "tool_version": "0.3.0", "run": "readme-first", "trace": "f1f1f1f1f1f1f1f1f1f1f1f1f1f1f1f1", "span": "a1a1a1a1a1a1a1a1", "parent": null, "event": "run", "state": "start", "t": "2026-10-27T00:00:00.000Z", "data": {"argv": ["serve"], "host": "example", "version": "0.3.0"}}
{"v": 1, "seq": 1, "tool": "app", "tool_version": "0.3.0", "run": "readme-first", "trace": "f1f1f1f1f1f1f1f1f1f1f1f1f1f1f1f1", "span": "a1a1a1a1a1a1a1a1", "parent": null, "event": "app.ping", "state": "point", "t": "2026-10-27T00:00:01.000Z", "data": {"latency_ms": 3}}
{"v": 1, "seq": 2, "tool": "app", "tool_version": "0.3.0", "run": "readme-first", "trace": "f1f1f1f1f1f1f1f1f1f1f1f1f1f1f1f1", "span": "a1a1a1a1a1a1a1a1", "parent": null, "event": "run", "state": "end", "t": "2026-10-27T00:00:02.000Z", "data": {}, "took_ms": 12, "outcome": "ok"}

One owner, one dictionary. An app declares its events:

{
  "tool": "app",
  "events": {
    "run": {
      "kind": "span",
      "fields": {}
    },
    "app.request": {
      "kind": "span",
      "fields": {
        "route": "request route path",
        "status": "response status code (number)"
      }
    },
    "app.tool": {
      "kind": "span",
      "fields": {
        "name": "tool name",
        "outputBytes": "bytes returned (number)"
      }
    }
  }
}

and this run nests an app.tool span inside an app.request span:

{"v": 1, "seq": 0, "tool": "app", "tool_version": "0.3.0", "run": "readme-app", "trace": "f1f1f1f1f1f1f1f1f1f1f1f1f1f1f1f1", "span": "a1a1a1a1a1a1a1a1", "parent": null, "event": "run", "state": "start", "t": "2026-10-27T00:00:00.000Z", "data": {"argv": ["serve"], "host": "example", "version": "0.3.0"}}
{"v": 1, "seq": 1, "tool": "app", "tool_version": "0.3.0", "run": "readme-app", "trace": "f1f1f1f1f1f1f1f1f1f1f1f1f1f1f1f1", "span": "b2b2b2b2b2b2b2b2", "parent": "a1a1a1a1a1a1a1a1", "event": "app.request", "state": "start", "t": "2026-10-27T00:00:01.000Z", "data": {"route": "/tiles"}}
{"v": 1, "seq": 2, "tool": "app", "tool_version": "0.3.0", "run": "readme-app", "trace": "f1f1f1f1f1f1f1f1f1f1f1f1f1f1f1f1", "span": "c3c3c3c3c3c3c3c3", "parent": "b2b2b2b2b2b2b2b2", "event": "app.tool", "state": "start", "t": "2026-10-27T00:00:01.000Z", "data": {"name": "render"}}
{"v": 1, "seq": 3, "tool": "app", "tool_version": "0.3.0", "run": "readme-app", "trace": "f1f1f1f1f1f1f1f1f1f1f1f1f1f1f1f1", "span": "c3c3c3c3c3c3c3c3", "parent": "b2b2b2b2b2b2b2b2", "event": "app.tool", "state": "end", "t": "2026-10-27T00:00:02.000Z", "data": {"outputBytes": 512}, "took_ms": 4, "outcome": "ok"}
{"v": 1, "seq": 4, "tool": "app", "tool_version": "0.3.0", "run": "readme-app", "trace": "f1f1f1f1f1f1f1f1f1f1f1f1f1f1f1f1", "span": "b2b2b2b2b2b2b2b2", "parent": "a1a1a1a1a1a1a1a1", "event": "app.request", "state": "end", "t": "2026-10-27T00:00:02.000Z", "data": {"status": 200}, "took_ms": 9, "outcome": "ok"}
{"v": 1, "seq": 5, "tool": "app", "tool_version": "0.3.0", "run": "readme-app", "trace": "f1f1f1f1f1f1f1f1f1f1f1f1f1f1f1f1", "span": "a1a1a1a1a1a1a1a1", "parent": null, "event": "run", "state": "end", "t": "2026-10-27T00:00:02.000Z", "data": {}, "took_ms": 11, "outcome": "ok"}

flattrace check app-run.jsonl --events app-vocab.json

app-run.jsonl: 6 valid events

Several owners: a render plugin ships its own dictionary,

{
  "tool": "render",
  "events": {
    "run": {
      "kind": "span",
      "fields": {}
    },
    "render.tile": {
      "kind": "span",
      "fields": {
        "tile": "tile identity",
        "pixels": "pixel count (number)"
      }
    }
  }
}

and its spans sit inside the app's request in the same run:

{"v": 1, "seq": 0, "tool": "app", "tool_version": "0.3.0", "run": "readme-mixed", "trace": "f1f1f1f1f1f1f1f1f1f1f1f1f1f1f1f1", "span": "a1a1a1a1a1a1a1a1", "parent": null, "event": "run", "state": "start", "t": "2026-10-27T00:00:00.000Z", "data": {"argv": ["serve"], "host": "example", "version": "0.3.0", "vocabularies": [{"tool": "app", "version": "0.3.0", "sha256": "700e2fbe54385d8ce0ece742accffb9a625b70f435559894d028097cefbcdf1c"}, {"tool": "render", "version": "0.3.0", "sha256": "1a71b43658da828ffa6499b33fbdf0056c9826dd05161f97fbf214cbef0e0ad3"}]}}
{"v": 1, "seq": 1, "tool": "app", "tool_version": "0.3.0", "run": "readme-mixed", "trace": "f1f1f1f1f1f1f1f1f1f1f1f1f1f1f1f1", "span": "b2b2b2b2b2b2b2b2", "parent": "a1a1a1a1a1a1a1a1", "event": "app.request", "state": "start", "t": "2026-10-27T00:00:01.000Z", "data": {"route": "/tiles"}}
{"v": 1, "seq": 2, "tool": "app", "tool_version": "0.3.0", "run": "readme-mixed", "trace": "f1f1f1f1f1f1f1f1f1f1f1f1f1f1f1f1", "span": "c3c3c3c3c3c3c3c3", "parent": "b2b2b2b2b2b2b2b2", "event": "render.tile", "state": "start", "t": "2026-10-27T00:00:01.000Z", "data": {"tile": "t-7", "pixels": 65536}}
{"v": 1, "seq": 3, "tool": "app", "tool_version": "0.3.0", "run": "readme-mixed", "trace": "f1f1f1f1f1f1f1f1f1f1f1f1f1f1f1f1", "span": "c3c3c3c3c3c3c3c3", "parent": "b2b2b2b2b2b2b2b2", "event": "render.tile", "state": "end", "t": "2026-10-27T00:00:02.000Z", "data": {}, "took_ms": 6, "outcome": "ok"}
{"v": 1, "seq": 4, "tool": "app", "tool_version": "0.3.0", "run": "readme-mixed", "trace": "f1f1f1f1f1f1f1f1f1f1f1f1f1f1f1f1", "span": "b2b2b2b2b2b2b2b2", "parent": "a1a1a1a1a1a1a1a1", "event": "app.request", "state": "end", "t": "2026-10-27T00:00:02.000Z", "data": {"status": 200}, "took_ms": 9, "outcome": "ok"}
{"v": 1, "seq": 5, "tool": "app", "tool_version": "0.3.0", "run": "readme-mixed", "trace": "f1f1f1f1f1f1f1f1f1f1f1f1f1f1f1f1", "span": "a1a1a1a1a1a1a1a1", "parent": null, "event": "run", "state": "end", "t": "2026-10-27T00:00:03.000Z", "data": {}, "took_ms": 11, "outcome": "ok"}

flattrace check mixed.jsonl --events app-vocab.json --events render-vocab.json

mixed.jsonl: 6 valid events

When owners disagree, the checker says so. A forked plugin redefines render.tile with fewer fields:

{
  "tool": "render-fork",
  "events": {
    "run": {
      "kind": "span",
      "fields": {}
    },
    "render.tile": {
      "kind": "span",
      "fields": {
        "tile": "tile identity"
      }
    }
  }
}

{"v": 1, "seq": 0, "tool": "app", "tool_version": "0.3.0", "run": "readme-collision", "trace": "f1f1f1f1f1f1f1f1f1f1f1f1f1f1f1f1", "span": "a1a1a1a1a1a1a1a1", "parent": null, "event": "run", "state": "start", "t": "2026-10-27T00:00:00.000Z", "data": {"argv": ["serve"], "host": "example", "version": "0.3.0", "vocabularies": [{"tool": "render", "version": "0.3.0", "sha256": "1a71b43658da828ffa6499b33fbdf0056c9826dd05161f97fbf214cbef0e0ad3"}, {"tool": "render-fork", "version": "0.3.0", "sha256": "6124c4372e36d6ab197543cd73693f61c9db1c453b4d0f7992644d1930da9591"}]}}
{"v": 1, "seq": 1, "tool": "app", "tool_version": "0.3.0", "run": "readme-collision", "trace": "f1f1f1f1f1f1f1f1f1f1f1f1f1f1f1f1", "span": "b2b2b2b2b2b2b2b2", "parent": "a1a1a1a1a1a1a1a1", "event": "render.tile", "state": "start", "t": "2026-10-27T00:00:01.000Z", "data": {"tile": "t-7", "pixels": 65536}}
{"v": 1, "seq": 2, "tool": "app", "tool_version": "0.3.0", "run": "readme-collision", "trace": "f1f1f1f1f1f1f1f1f1f1f1f1f1f1f1f1", "span": "b2b2b2b2b2b2b2b2", "parent": "a1a1a1a1a1a1a1a1", "event": "render.tile", "state": "end", "t": "2026-10-27T00:00:02.000Z", "data": {}, "took_ms": 6, "outcome": "ok"}
{"v": 1, "seq": 3, "tool": "app", "tool_version": "0.3.0", "run": "readme-collision", "trace": "f1f1f1f1f1f1f1f1f1f1f1f1f1f1f1f1", "span": "a1a1a1a1a1a1a1a1", "parent": null, "event": "run", "state": "end", "t": "2026-10-27T00:00:03.000Z", "data": {}, "took_ms": 11, "outcome": "ok"}

flattrace check collision.jsonl --events render-vocab.json --events render-fork.json

collision.jsonl: conflicting event definition 'render.tile' in 'render-vocab.json' and 'render-fork.json'

The same name with a different parsed definition, even a description-only change, is an error naming both files. Byte-identical files deduplicate first, so every vocabulary can copy the reserved run declaration (schema/run-declaration.json) without colliding on it. And when a supplied file is not the one the run declared, the run fails instead of silently validating against a different contract:

{"v": 1, "seq": 0, "tool": "app", "tool_version": "0.3.0", "run": "readme-hash", "trace": "f1f1f1f1f1f1f1f1f1f1f1f1f1f1f1f1", "span": "a1a1a1a1a1a1a1a1", "parent": null, "event": "run", "state": "start", "t": "2026-10-27T00:00:00.000Z", "data": {"argv": ["serve"], "host": "example", "version": "0.3.0", "vocabularies": [{"tool": "app", "version": "0.3.0", "sha256": "0000000000000000000000000000000000000000000000000000000000000000"}]}}
{"v": 1, "seq": 1, "tool": "app", "tool_version": "0.3.0", "run": "readme-hash", "trace": "f1f1f1f1f1f1f1f1f1f1f1f1f1f1f1f1", "span": "b2b2b2b2b2b2b2b2", "parent": "a1a1a1a1a1a1a1a1", "event": "app.request", "state": "start", "t": "2026-10-27T00:00:01.000Z", "data": {"route": "/tiles"}}
{"v": 1, "seq": 2, "tool": "app", "tool_version": "0.3.0", "run": "readme-hash", "trace": "f1f1f1f1f1f1f1f1f1f1f1f1f1f1f1f1", "span": "b2b2b2b2b2b2b2b2", "parent": "a1a1a1a1a1a1a1a1", "event": "app.request", "state": "end", "t": "2026-10-27T00:00:02.000Z", "data": {"status": 200}, "took_ms": 9, "outcome": "ok"}
{"v": 1, "seq": 3, "tool": "app", "tool_version": "0.3.0", "run": "readme-hash", "trace": "f1f1f1f1f1f1f1f1f1f1f1f1f1f1f1f1", "span": "a1a1a1a1a1a1a1a1", "parent": null, "event": "run", "state": "end", "t": "2026-10-27T00:00:03.000Z", "data": {}, "took_ms": 11, "outcome": "ok"}

flattrace check hash.jsonl --events app-vocab.json

hash.jsonl: extra vocabulary file 'app-vocab.json' not listed in run data.vocabularies; vocabulary 'app' in run data.vocabularies not supplied (sha256 0000...0000)

The full rules live in the composition truth table.

To implement a writer in your own software, follow SPEC.md, the frozen conformance corpus (tests/conformance/) and flattrace check.