check-events v2 — streaming test results as a ForgeGraph standardShipped

Overview

Research (2026-08-24) confirmed no widely-adopted cross-framework protocol exists for streaming test results. The streaming formats are ecosystem-bound (TAP, subunit, go test -json, Bazel BEP); the modern cross-framework standard, CTRF, is batch-only. Every live-test product invents its own NDJSON ingestion — and Bob just did too: bob-check (worktree shim → .bob/check-events.ndjson → runner tail → check WS events → cockpit phase bars) shipped in the cockpit V2 cut.

This plan promotes that proven pipeline into a shared standard — fg-check — that covers Rust, Python, Ruby, Go, JS/TS and more by adapting each framework's native live output, and bakes it into ForgeGraph best practices: conventional target names, sample package.json scripts / justfiles / Makefiles, a forge-ci.toml [check] section, and CI pages that render live per-phase bars and per-test results. Bob's cockpit and ForgeGraph's CI UI become two consumers of one contract.

Goals

• One versioned NDJSON event envelope for typecheck/lint/test/build progress, streamable from any runtime.
• Per-runtime adapters that prefer native machine-readable streams; graceful fallback ladder for everything else.
• CTRF as the stored end-of-run report — attached to ForgeGraph build rows as evidence.
• Repo conventions + samples (npm scripts, justfile, Makefile) so any onboarded repo gets live test reporting for free.
• Bob cockpit and ForgeGraph CI pages consume the identical stream.

Non-goals

• Not adopting subunit's binary wire format (dead producer ecosystem; JSON everywhere downstream). We keep its design lessons: timestamps, stream routing, attachments.
• Not replacing Forgejo Actions or the manifest CI runner — fg-check wraps commands inside whatever CI executes them.
• Not building test analytics (flake detection, history) yet — CTRF storage makes that possible later.

Decisions what the exploration settled

The envelope fg-check events v2

{"v":2,"phase":"test","event":"run_started","framework":"pytest","stream":"api-tests","command":"pytest -q","at":"…"}
{"v":2,"phase":"test","event":"test_finished","stream":"api-tests",
 "test":{"name":"test_streak_gaps","suite":"tests/lib/test_streak.py","status":"failed",
         "duration":812,"message":"expected 3, got 4","trace":"…"},
 "counts":{"passed":41,"failed":1,"total":58},"at":"…"}
{"v":2,"phase":"test","event":"run_finished","stream":"api-tests","status":"failed",
 "counts":{"passed":57,"failed":1,"skipped":0,"total":58},"durationMs":8120,"at":"…"}

• phase: typecheck | lint | test | e2e | build — each a separately-streamed configuration. Unit tests and end-to-end/integration tests are distinct configurations (mapping to forge-ci.toml's existing test.command and test.integration), and lint/typecheck pre-checks are their own configurations so Bob can identify them as streaming checks rather than test noise. Cockpit/CI phase bars render one segment per configuration.
• event: run_started | test_started | test_finished | output | run_finished | skipped. v1 events (no v field) remain parseable — consumers treat them as run_* only.
• test: CTRF test object (name, status, duration, suite, message, trace, flaky, retries).
• stream: source discriminator (package, test binary, shard). Required whenever a phase has >1 concurrent producer.
• counts: running totals per stream; consumers sum across streams.

Adapter matrix what each runtime streams natively

Repo conventions best practices + samples

The contract: a repo exposes some subset of typecheck · lint · test · build as runnable targets. fg-check detects them in priority order — package.json scripts → justfile → Makefile → forge-ci.toml [check] explicit override — and picks the adapter from lockfiles/manifests (Cargo.toml, pyproject.toml, Gemfile, go.mod…). Samples ship in docs and in the fg-onboard-repo skill.

{
  "scripts": {
    "typecheck": "tsc --noEmit",
    "lint": "oxlint .",
    "test": "vitest run",
    "build": "next build"
  }
}

typecheck:
    cargo check --all-targets

lint:
    cargo clippy -- -D warnings

test:
    cargo nextest run

build:
    cargo build --release

.PHONY: typecheck lint test build
typecheck:
	mypy src
lint:
	ruff check .
test:
	pytest -q
build:
	python -m build

version = 1

[build]
command = "pnpm build"

[test]                    # unit tests → configuration "test"
command = "pnpm vitest run --project unit"

[test.integration]        # e2e → separate configuration "e2e"
command = "pnpm playwright test"
stage   = "beta"

# lint/typecheck configurations come from conventional
# script targets — detected, streamed, never invented

Architecture

flowchart LR
  subgraph Repo["repo conventions"]
    C1["npm scripts / justfile / Makefile<br/>typecheck · lint · test · build"]
    C2["forge-ci.toml [check] overrides"]
  end
  subgraph Wrapper["fg-check wrapper (per run)"]
    D[detect runtime + targets] --> R[run command<br/>tee stdout to human log]
    R --> A{adapter}
    A -->|go test -json| N[normalize]
    A -->|nextest libtest-json| N
    A -->|pytest --report-log tail| N
    A -->|rspec formatter --out tail| N
    A -->|mocha json-stream / node:test tap| N
    A -->|teamcity msgs| N
    A -->|TAP| N
    A -->|scrape fallback| N
  end
  N --> E[".fg/check-events.ndjson<br/>NDJSON, CTRF-shaped tests"]
  E -->|tail| AG[node agent / bob runner]
  AG -->|WS events| HUB[(hub ws.forgegraf.com<br/>bob ws-gateway)]
  HUB --> UI["CI run page · cockpit tiles<br/>live phase bars + counts"]
  E -->|fold at end| CT["CTRF report"]
  CT -->|ci-report tests field| DB[(builds row<br/>evidence)]
  C1 --> D
  C2 --> D
  classDef new stroke-dasharray: 5 5,stroke:#e6a23c,color:#e6a23c;
  class D,R,A,N,E,CT new;

Two injection modes cover every adapter: parse-stdout (go, nextest, TAP, TeamCity, mocha json-stream — tee and parse, human log stays pristine for logHead/logTail evidence) and side-channel file/socket (pytest --report-log, RSpec --out, JUnit socket — tail it, exactly like bob-check's events file today).

Mockups animated · what the consumers render as events arrive

Mockups A, C, D, and E are live simulations, not stills: each plays a looped run driven by pure CSS keyframes (postplan executes no scripts), showing exactly what a viewer sees as v2 events stream in — counts ticking, bars creeping, failures popping the moment they happen. Under prefers-reduced-motion every mockup degrades to its final static frame, matching the cockpit's own motion-level philosophy.

A — ForgeGraph CI run page, mid-run

B — same page, after the run: per-test results

C — Bob cockpit wall tile: today vs v2

D — cockpit PR pipeline strip with local-vs-CI drift

E — what the agent sees in the terminal

Bob in the cockpit first consumer, already live

Bob is not a future consumer — it's the migration source. bob-check v1 shipped with cockpit V2: shim in every agent worktree, worktree-watch.ts tails .bob/check-events.ndjson, the ws-gateway fans check session events, tiles render per-phase bars. fg-check replaces the shim in place; everything downstream keeps working, then gets richer:

• Same shim path, new binary. The runner drops fg-check at ./.bob/bin/bob-check (name kept so session prompts and SKILL.md stay valid). v2 events flow through the existing check SessionEventType untouched — the gateway forwards event types it doesn't inspect.
• Tiles upgrade from counts to tests. Today a tile shows test 57/58 ✗ from scraped counts. With CTRF-shaped test_finished events the tile can show the failing test names live, and ops mode can expand a phase bar into the per-test list with messages — no new transport, just richer payloads on events the cockpit already subscribes to.
• PR pipeline "code" stage gets trustworthy pre-CI signal. The cockpit already renders the last local check result before Forgejo CI starts; v2 adds per-stream counts so parallel suites don't produce the interleaving lies the scraper can produce today.
• Agents get a better feedback loop. fg-check keeps printing the human summary for the agent (that part of the contract is untouched) — the same run feeds agent, wall, and evidence.
• Mobile rides along free. The Bob+OODA mobile plan consumes the same gateway session events (check included) for tablet session views — per the cockpit plan's rule: shared contracts, never sibling endpoints.
• Cross-repo dogfooding. Bob's agents work on many repos (JS today). The moment an agent lands in a Rust/Python/Ruby repo, cockpit phase bars work because the adapter ladder ships in the wrapper, not per-repo.

Phases

Phase 1 — envelope spec + shared wrapper Done

Phase 2 — native runtime adapters Done

Phase 3 — ForgeGraph integration Done

Phase 4 — Bob migration + best-practices rollout Bob PR open

What shipped, and what production taught us 2026-08-26

Every phase above is built and running. The parts a plan cannot predict are below — each one cost a real outage or a real near-miss, so they are recorded as findings rather than as tidy checkmarks.

Proven end to end, in production

• Live streaming: a demo repo pushed to a real node showed … test running mid-run, then ✓ test passed 3/3.
• CTRF artifact: forge ci checks <run> --ctrf fetched the folded report off node disk — 3 tests, per-test names and durations.
• Adapter honesty: the TAP output was sniffed, not scraped; confidence is reported per phase rather than assumed.

Consumer sweep

1,246 package.json files scanned across the dev tree: the only consumers of @forgegraph/check-events are ForgeGraph itself (via workspace:*, so always the fixed source) and Bob (root bob-check plus ooda-runner, both moved to 0.1.7). Bob's lockfile had pinned 0.1.4 and 0.1.6 exactly — the vulnerable versions — and now carries a single deduplicated 0.1.7. Nothing vulnerable was ever installed on a node, so this was caught before it shipped.

Risks & mitigations

Verification

• Golden-fixture corpus: recorded real output per framework/format → expected event stream; runs in both repos' CI.
• End-to-end: one PR in the ForgeGraph monorepo shows live phase bars on the CI run page; one Bob agent session shows failing test names on its cockpit tile.
• Folded CTRF reports validate against the upstream CTRF JSON Schema.
• Onboarding drill: fresh Rust and Python repos through fg-onboard-repo produce live results with only the conventional targets present.

Open questions all resolved 2026-08-24

• CTRF storage → full report uploaded as a build artifact, summary inline in ci-report. Decided
• JVM adapter → deferred; adapter slot reserved, no envelope changes required to add it later. Decided
• Manifest schema → no [check] section; fg-check wraps the existing test.command / test.integration / build.command. Unit and e2e/integration run as separate configurations, and lint/typecheck pre-checks are their own configurations so Bob identifies them as streaming checks. Decided
• Vitest/jest → build first-class in-process NDJSON reporters (phase 2); TAP/scrape stays only for frameworks without reporters. Decided