Running the conformance kit in CI

Thin recipe for implementers: how to run @ggui-ai/protocol-conformance in your own CI so a protocol regression on either side shows up as a red build. What conformance means is on the Conformance page; this page is only the wiring.

Install

The kit is a plain npm package with no monorepo assumptions — npm install resolves the kit plus @ggui-ai/protocol and ws, and it runs standalone. The receipt behind that is the reference server’s own in-process conformance suite (conformance.test.ts, exact-set assertions), which runs the kit with a host, a refusalRegistry and the server’s own refusal projector at 19 pass / 0 fail / 11 skip on 0.15.0 — the 11 skips are the three browser-host claims, the two transport-refusal rows the suite hands no endpoint projector for, and the six domain-error rows: the reference server serves resources/read and the live channel and has no tools/call plane, so it binds no driver, and that catalog is graded on the first-party server instead (domain-error.conformance.test.ts, 6/6). A hostless, flagless CLI run against the same server is 0 pass / 0 fail / 30 skip and exits 2, by design (see Invoke).

npm install -D @ggui-ai/protocol-conformance

Invoke

Programmatic runner (the real pass/fail signal). Boot your implementation, hand the runner a ConformanceHost so setup directives (session provisioning, fault injection) can run, and apply the kit’s red-build guard:

import { runConformance } from "@ggui-ai/protocol-conformance";

const result = await runConformance({
  serverUrl: "http://127.0.0.1:3100", // bare origin — the runner derives ws://…/ws
  auth: { kind: "bearer", token: process.env.TOKEN },
  host: myConformanceHost, // dispatchSetup / dispatchTeardown / readSessionField
});

// A failed fixture is a red build — and so is a run that executed
// ZERO fixtures (all skips prove nothing). A warned fixture executed,
// and a warning never fails the run.
if (result.failed.length > 0 || result.passed.length + result.warned.length === 0) process.exit(1);

CLI. The bundled CLI has no host wiring, and every wire fixture in the current catalog declares setup steps — so a hostless run with no catalog flag skips them all and exits 2 by design. Use the bare form as a connectivity/handshake smoke check, not as your conformance gate:

npx ggui-protocol-conformance --url ws://127.0.0.1:3100/ws --auth bearer:$TOKEN

Four catalogs cannot be reached by a URL alone: refusal-envelope, registry-completeness and transport-refusal grade an in-process function and a data table, and domain-error — the kit’s first tools/call driver — grades what a tools/call returns for a Plane-2 failure. The CLI hands them their inputs by flag: --registry <file.json> (the deployment’s refusal-code registry, a JSON object keyed by code; grades registry-completeness), --projector <module> (an ES module whose project(refusal) — a named or default export — returns the SPEC §7.1 tool result for a refusal, or null when the code has no envelope on the render-gate surface; grades refusal-envelope), and --transport-projector <module> (the same shape, returning the endpoint’s { httpStatus, error } or null when the code has no transport envelope; grades transport-refusal), and --tool-call-driver <module> (an ES module exporting drive(scenario) — or a default export — that performs one tools/call and returns the raw result, or null when the tool is not bound; may be async; grades domain-error). File and module paths resolve against the working directory. A flag left out leaves its catalog on the scorecard as SKIPPED with the flag named — an ungraded obligation stays visible rather than vanishing. --only <catalog>/<case> filters catalog rows the way it filters wire fixtures.

npx ggui-protocol-conformance --url ws://127.0.0.1:3100/ws --auth bearer:$TOKEN \
  --registry ./refusal-registry.json --projector ./project.mjs --transport-projector ./endpoint.mjs \
  --tool-call-driver ./tool-call-driver.mjs

Against the reference server with the kit’s sample inputs (cli-samples/), that run is 18 pass / 0 fail / 12 skip and exits 0 — every catalog row graded, every wire fixture SKIPPED for want of a host. Without --tool-call-driver it is 12 / 0 / 18: the six domain-error rows skip with the flag named. The sample driver is canned — it answers the six scenarios the way a conformant server does without calling the server — so those six passes prove the flag’s plumbing, not the reference server’s wire, which has no tools/call plane; the live receipt for the wire is the first-party server’s suite linked above.

A projector must be synchronous. One that throws, returns undefined, or returns a non-projection is graded as a FAIL on that case (exit 1) and the remaining cases are still graded — never a crash. Catalog rows count as executed fixtures: a hostless run given a catalog flag can exit 0 while every wire fixture is SKIPPED, so read the scorecard, not the exit code alone, for a wire signal.

Exit codes: 0 — at least one fixture executed and none failed; 1 — any fixture failed; 2 — invocation error (bad arguments, an unreadable registry or projector, an unreachable server) or a zero-executed (all-skip) run. A run that proved nothing never reads as success.

What a failure means

• FAIL — your implementation violates a protocol obligation the fixture freezes (wrong ack shape, missing CONTRACT_VIOLATION rejection, version-handshake drift, …). Path-A fails are server-side vendor-neutrality bugs: fix the implementation, not the fixture.
• WARN — not a failure. A SHOULD fixture ("level": "should") whose expectation your implementation does not meet: a recommendation it declined, with the same evidence a failure carries (result.warned). A warning never fails a run or changes the CLI’s exit code. The bundled CLI has no host, so it skips every fixture that needs one, the SHOULD case included; a warning appears only in a programmatic run with a host.
• SKIP — not a failure. Path-B (browser-host) claims and setup directives your host doesn’t implement skip with a precise reason. But pin your pass, skip and warned sets as exact sets in CI: a fixture silently degrading to a skip, or a warning appearing unnoticed, should fail your build, and re-pinning should be a deliberate act.
• All-skip — red (exit 2 / the programmatic guard above). Zero executed fixtures carries no signal. Catalog rows handed in by flag count as executed, so with a catalog flag the exit code alone cannot tell you the wire fixtures ran — pin the wire fixtures’ pass set as above.

A change is breaking iff a fixture that passed against version N fails against N+1 — the kit, not opinion, is the arbiter (see Version policy).

See also

• Conformance — the contract/protocol bars the kit makes observable.
• @ggui-ai/protocol-reference-server — a minimal target to sanity-check your CI wiring against before pointing the kit at your own implementation.
• Downstream SDK integrations — including the platform-composed (guuey-sdk) golden-path sample — gate their side of the seam with this same kit in their own CI; the protocol itself requires no particular platform, and the kit runs against any implementation of the wire.