Driving Gapreel from a coding agent

Gapreel records what an agent does as a video plus a labelled timeline: actions, marks for the moments that matter, and gaps for every wait. An agent controls it through MCP tools, through the gapreel command line, or by being wrapped in gapreel run. This page is the how-to; the exact tool schemas and errors are in docs/contracts/mcp-sdk.md and docs/contracts/browser-control.v0.md.

Per-agent setup: Claude Code, Codex CLI, Muse Code, Command Code, Playwright and browser-use. Which Chrome profile each browser path needs: Chrome profiles. An installable skill with the same workflow: skill/gapreel/SKILL.md. Settings > General > Command line > Install the skill (or gapreel skill install) copies it into ~/.claude/skills/gapreel, ~/.codex/skills/gapreel ($CODEX_HOME/skills when set) and ~/.agents/skills/gapreel, the folder headless muse exec reads; each copy carries a .gapreel-skill-version stamp, the daemon refreshes stamped copies when it starts on a newer version, a gapreel folder Gapreel did not write is left alone, and gapreel skill status --json reports each root as current, stale, missing or foreign.

Set up once

1. Open the Gapreel app; it runs the local daemon on 127.0.0.1:4317.
2. In Settings > General > Command line, press Install the gapreel command. gapreel then works in any new shell (it lives in /usr/local/bin, or ~/.local/bin when that is not writable).
3. Check: gapreel status prints the version, capture state, permissions and connected browsers. gapreel status --json gives the same as one JSON line.
4. Register the MCP server with your agent, using the profile that fits its job (see the per-agent pages): gapreel mcp --profile evidence for coding agents, --profile browser for agents that drive the browser, no flag for every tool.
5. Optional: Settings > General > Open Gapreel when an agent needs it lets the MCP server open the app when it is closed.

The recording workflow

1. gapreel_status: is the daemon up, may it capture the screen, is a browser connected.
2. rec_start with a target: {"kind": "screen"}, {"kind": "window", "ref": "<ref>"} (refs come from capture_targets), or {"kind": "browser_tab", "ref": "tb_<n>"} for a tab opened with browser_tab_new. profile is padded (keeps wall time) or clips_only (cuts the waits). A browser_tab recording needs the Gapreel extension in that browser; without one rec_start answers no_browser at once.
3. Bracket each step: rec_action_begin {"label": "Submit the form"} before, rec_action_end {"label": "Submit the form"} after. browser_* act tools record their own actions. rec_mark {"note": "Order confirmed"} pins a checkpoint (severity warning or critical for a failure); rec_screenshot adds a still.
4. rec_status confirms the recording runs; for a browser tab, feed.frames should grow.
5. rec_stop. On a browser tab it answers at once while the last frames drain.
6. rec_export {"profile": "clips_only", "wait_ms": 20000} renders and waits up to 20 s; call it again (or events_poll for export.ready) until status is ready. The answer's artifact gives path, mime, size_bytes, duration_ms, width, height and sha256. Add "purpose": "evidence", "package": true for a folder with the video, timeline, screenshots, diagnostics and a hashes.json, or "range": {"from_action": "<id>"} for one part. On Claude Code v2.1.304+ (main conversation, not subagents or a one-shot claude -p) the same call comes back as an MCP task that completes with the ready export; Claude Code polls it, so no wait_ms loop is needed.
7. Narration spends Saytape credits: call narrate_preflight first, show the estimate, voice and balance to the user, then rec_export with "narrate": true; it runs only when the user left the Spend Saytape credits switch (narrate:spend) on for this agent.

Recordings are also OpenTelemetry traces; see traces.md for sending them to Jaeger or Langfuse and following a span to its frames. After a restart of the agent's MCP process, rec_attach takes the recording back. rec_list, rec_get, rec_frames and rec_evidence read past recordings; shot_take and shot_text take a screenshot or read text without recording.

The same from a shell

Every verb takes --json and answers {"ok": true, "data": ...} or {"ok": false, "code", "message", "next"}; exit code 69 means the app is not running. See cli.md.

Approvals

Browser control and recording are local, and the user stays in charge of consent:

• Gapreel asks once. The first browser_* or recording tool call from a new agent (rec_start, shot_take, clip_history, resources/list and the rest) answers approval_required with approval.approval_id and approval.expires_in_s. The Gapreel window lists every capability as a switch: browse, act, observe, user tabs, run JavaScript, read and make recordings, screenshots, delete recordings, read and set the clipboard, send spans to the trace collector (rec:send), spend Saytape credits (narrate:spend) and publish share links (share:create), with what triggered the request and where data would go. Tell the user to answer it, then repeat the same call. After Allow the agent works with no more prompts for what was switched on. On Claude Code v2.1.304+ the call does not stop: it becomes an MCP task that waits for the user's answer in the Gapreel window and runs as soon as they allow it (or ends with approval_denied or approval_expired). The approval itself always stays in the Gapreel window.
• A later call that needs a capability the user switched off answers scope_denied: the window asks for more access, listing only what is missing. Repeat the same call after the user answers. approval_denied means no, and cooldown_until_ms says when asking again is allowed.
• A few single actions still ask every time: rec_delete, a file upload, typing into a password or card field, JavaScript while the run-JavaScript switch is off, a site outside the grant and taking over a tab the user opened.
• An agent run started over HTTP by a program other than the Gapreel app asks once for agent:run; later runs start with no prompt.
• Without the Gapreel app attached to the daemon there is nobody to ask: screenshots, shot_*, clip_*, rec_start on a screen, window or app, narrate_timeline, narration, rec_otlp_send, share_create, the credit-spending saytape_* tools and rec_delete answer confirmation_unavailable. Open the app and retry.
• The Stop all control button ends the recordings agents started and drops their recording, screenshot, share, send, narration, agent-run and clipboard access; the next call asks for it again. Revoking an agent in Settings stops the recordings it started.
• control_stopped means the user pressed Stop: do not retry.
• Every error carries next (what to do) and retry (safe, after_approval, after_fix or never).

Reporting a problem

Call report_issue once when Gapreel misbehaves: a wrong result, an error whose next text does not help, or a surprising slowdown. It needs no grant. Send kind (bug, wrong-result, slow, ux or other) and a summary of up to 200 characters that says what you expected and what you saw. Add details (up to 4000 characters), and when they apply tool, error_code and recording_id. Never put secrets, tokens or personal data in a report.

The answer is {"reported": true, "issue_id": "iss_<16 hex>", "duplicate": bool, "count": n, "next": "..."}. A repeat of a known problem answers duplicate: true and raises count; do not file it again. Each caller may send 30 reports per hour, after which the call answers issue_rate_limited.

The same call is POST /issues over HTTP, reportIssue in the TypeScript SDK and report_issue in the Python SDK. The owner reads reports in Gapreel Settings > Agents > Reports from agents, and from a shell with gapreel issues list, gapreel issues report and gapreel issues resolve.

When something is off

Claude Code

Measured with Claude Code 2.1.289.

Install the plugin

Install Gapreel from inside Claude Code:

/plugin marketplace add HENIART/gapreel
/plugin install gapreel@gapreel

The plugin adds the gapreel MCP server (gapreel mcp), recording hooks for PreToolUse, PostToolUse and PostToolUseFailure, and the /gapreel:gapreel agent skill. Install the stable gapreel launcher first with gapreel-core launcher install (or Settings > General > Install the gapreel command) and open the Gapreel app before recording. The plugin does not bundle the app or core binary. The MCP server and the hooks do not run gapreel from PATH: both start /bin/sh "${CLAUDE_PLUGIN_ROOT}/scripts/gapreel-launch", where ${CLAUDE_PLUGIN_ROOT} is the placeholder Claude Code replaces with the plugin's absolute install directory. The script executes only a launcher it finds at /usr/local/bin/gapreel or ~/.local/bin/gapreel whose second line is GAPREEL_LAUNCHER=1, so a binary that merely sits earlier on PATH is never run. A launcher installed elsewhere with launcher install --dir is not found by the plugin; without a match the script prints a one-line message and exits with status 127.

The manual claude mcp add registration below and gapreel-core hooks install claude-code remain the fallback. When the installed Gapreel plugin is enabled, gapreel-core connect status and connect install --client claude-code report the client state as plugin and add no second MCP entry. gapreel-core hooks install claude-code refuses to write hooks because running both would record every tool call twice; --force explicitly overrides that guard. Disable the plugin before using the fallback, or remove existing manual hooks with gapreel-core hooks uninstall claude-code (add --user for user hooks) before enabling the plugin.

The guard reads plugins/installed_plugins.json and enabledPlugins in user, project and local settings; CLAUDE_CONFIG_DIR selects the user config directory instead of ~/.claude. A user hook install also checks recorded project and local plugin installs. The guard checks installed configuration, so session-only --plugin-dir plugins and launch-time settings overrides still need manual coordination. See the Claude Code plugin manifest and plugin loading reference.

Register

claude mcp add gapreel -- gapreel mcp --profile evidence

Add -s user to use it in every project. Use --profile browser when this Claude Code session drives the browser through Gapreel, or leave the flag out for every tool. claude mcp list should show gapreel connected; the first tool call needs the Gapreel app running.

Claude Code lists Gapreel's resources and prompts at startup: gapreel://recordings/<id>/timeline and /context can be @-mentioned, and the record-test-run and export-demo prompts appear as slash commands.

Record each tool call (optional)

gapreel hooks install claude-code          # this project, .claude/settings.local.json
gapreel hooks install claude-code --user   # every project, ~/.claude/settings.json
gapreel hooks doctor claude-code

With the hooks, every tool Claude Code runs becomes an action on the recording it opened with rec_start, matched by CLAUDE_CODE_SESSION_ID. Claude Code runs hooks without a trust step.

Headless runs

gapreel run -- claude -p "<task>" --output-format stream-json --verbose

records the run from its event stream: each tool_use becomes an action, with no hook or MCP entry.

First recording

Ask Claude Code: "Call gapreel_status, start a Gapreel recording of window <id> with rec_start, mark 'checkpoint', stop it, export clips_only with wait_ms 20000 and tell me artifact.path." The answer should end with a path to an MP4 under the recordings folder.

Claude in Chrome

Claude in Chrome drives your everyday Chrome profile; Gapreel films a tab there only when the Gapreel extension is loaded and paired in that same profile (see Chrome profiles). Start with rec_start {"target": {"kind": "browser_tab"}}: with no ref the extension films the active tab of the last focused window, and rec_status should show feed.frames growing within a few seconds.

Troubleshooting

See the table in agents.md. Claude Code specific: if claude mcp list shows the server failed, run gapreel mcp --help in a shell; with the manual registration, command not found means the launcher is not installed or ~/.local/bin is not on PATH. With the plugin, a no Gapreel launcher in /usr/local/bin or ~/.local/bin message means the launcher is not installed in one of those two folders (the plugin ignores PATH).

Codex CLI

Measured with Codex CLI 0.160.0.

Register

codex mcp add gapreel -- gapreel mcp --profile evidence

-s project keeps it to this project. Codex starts MCP servers with a reduced environment (HOME, PATH, USER, LANG plus anything set under mcp_servers.gapreel.env), so Gapreel links Codex's calls to its recording by the Codex process id, not by a session variable. Codex lists only Gapreel's tools; resources and prompts are not used.

Record each tool call (optional)

gapreel hooks install codex                # this project, .codex/hooks.json
gapreel hooks install codex --user         # ~/.codex/hooks.json
gapreel hooks doctor codex

Codex trusts hooks by content hash and silently skips new or changed ones in codex exec. After installing, open an interactive codex, run /hooks and trust the Gapreel hooks (the edit can also move other hooks in the file, which then need trusting again). gapreel hooks doctor codex lists each hook as trusted, untrusted or unknown and exits 1 until every Gapreel hook is trusted. Gapreel never trusts hooks for you and never uses --dangerously-bypass-hook-trust.

Headless runs

gapreel run -- codex exec --json "<task>"

records the run from Codex's JSON events: each shell command, MCP call, file change or web search becomes an action (Shell: <command>), with no hook and no trust step.

First recording

Ask Codex: "Call gapreel_status, then rec_start with target window <id>, rec_mark 'checkpoint', rec_stop, and rec_export clips_only with wait_ms 20000; print artifact.path." Or run any task under gapreel run and read the recording id from the last stderr line.

Troubleshooting

See agents.md. Codex specific: hooks installed but nothing recorded means they are untrusted; run gapreel hooks doctor codex.

Muse Code

Measured with Muse Code 1.4.2.

Register

Muse reads MCP servers from the workspace's .mcp.json:

{ "mcpServers": { "gapreel": { "command": "gapreel", "args": ["mcp", "--profile", "evidence"] } } }

Muse passes MUSE_SESSION_ID to the server, which Gapreel uses to link the run's calls to its recording (and rec_attach after Muse restarts the server).

Tool calls

Muse has no Gapreel hook adapter: its hooks are declared in plugin manifests, which this repo has not verified on 1.4.2. Gapreel's own tools are recorded; to record Muse's other tool calls, run it under gapreel run or have it bracket steps with rec_action_begin / rec_action_end.

Headless runs

gapreel run -- muse exec --json "<task>"

turns every tool.<name> task in Muse's event stream into an action (bash for a shell command).

First recording

gapreel run --label first-muse -- muse exec --json "Run the shell command true, then reply OK." ends with recorded rec_... (1 actions from a muse stream) on stderr.

Troubleshooting

See agents.md.

Command Code

Register

command-code mcp add -s project gapreel -- gapreel mcp --profile evidence

-s project writes the workspace's .mcp.json; -s user writes the user-level setting. Command Code passes no session variable to the server, so Gapreel links its calls by the Command Code process id. It lists only Gapreel's tools.

Tool calls

There is no Gapreel hook adapter: Command Code's hooks live in its user-level settings, which gapreel hooks install does not touch. Gapreel's own tools are recorded; for the rest, ask the agent to bracket its steps with rec_action_begin / rec_action_end. gapreel run -- command-code -p "<task>" records the run start to finish, but has no adapter for Command Code's event stream yet, so it adds no per-tool actions.

First recording

Ask Command Code: "Call gapreel_status, then rec_start with target window <id>, rec_mark 'checkpoint', rec_stop, and rec_export clips_only with wait_ms 20000; print artifact.path."

Troubleshooting

See agents.md.

Playwright and browser-use

Scripts that already drive Chrome over CDP have two ways in.

SDK: record your own CDP session

TypeScript, Playwright:

import { BrowserRecorder, playwrightCdpSession } from "@gapreel/sdk";
const recorder = await BrowserRecorder.start(await playwrightCdpSession(page), { label: "checkout" });
await recorder.step("Submit the form", () => page.click("#submit"));
const stopped = await recorder.stop();   // stopped.masking counts masked and blind frames
const exported = await recorder.export("clips_only");

Python (Playwright async or browser-use):

from gapreel_sdk.cdp import AsyncBrowserRecorder
recorder = await AsyncBrowserRecorder.start_playwright(page, label="checkout")
await recorder.step("Submit the form", lambda: page.click("#submit"))
await recorder.stop()

The SDK reads the token like every client (GAPREEL_CORE_TOKEN, GAPREEL_CORE_TOKEN_PATH, else the app's token file). It measures secure fields (password, one-time code, card number, data-private) for every frame and sends the boxes with it; the daemon blacks them out before the frame is written and refuses a frame that arrives without that verdict. Closed shadow roots and cross-origin frames cannot be measured; stop() reports how many frames showed one.

CDP relay: connect Playwright to a Gapreel tab

For a script token (gpr_..., minted in Settings > Browser extension > Script tokens):

const browser = await chromium.connectOverCDP("ws://127.0.0.1:4317/cdp?tab=tb_3", {
  headers: { Authorization: "Bearer gpr_..." },
});

The tab must be one this token opened or claimed with browser_tab_new / browser_tab_claim, and it must live in the Gapreel-managed Chrome profile. Each connection asks for browser:cdp in the Gapreel window, records the run, and ends at once when the token or grant is revoked, the scope expires or the user presses Stop.

Troubleshooting

See agents.md and Chrome profiles.

Which Chrome profile each browser path needs

Commands:

• Open the managed profile: gapreel browser open-managed-profile (--browser chrome|edge|...). gapreel browser managed-profile --json reports whether its host manifest is installed and current.
• See which browsers have the host: gapreel browser list-hosts --json.
• Install or remove the host for every detected browser: gapreel browser install-host --all, gapreel browser uninstall-host --all. Installing it into your everyday profile lets agents ask to drive that profile, so do it only if you want that.
• Pair the extension: load it, then in Gapreel Settings > Browser extension press Show pairing code and paste the code on the extension's options page. gapreel status --json lists connected browsers under data.browser.browsers.

Command line for agents and scripts

gapreel-core (or the gapreel launcher that Settings > General > Command line installs) drives the running Gapreel daemon from a shell. Agents that have a shell tool but no MCP registration, sandboxed runs and CI scripts use it the same way; every verb reads the core token itself and never prints it.

Verbs

gapreel-core --help prints this list, --version the version; neither starts a daemon.

--json

Every verb takes --json and then prints exactly one line:

{"ok":true,"data":{"state":"idle"}}
{"ok":false,"code":"daemon_unreachable","message":"Gapreel is not running: nothing answers on 127.0.0.1:4317 (...). Open the Gapreel app (port 4317) and retry this call.","next":"Open the Gapreel app, then retry the same call."}

data is the daemon's answer for that verb. On failure code is the daemon's own error code (the same codes the MCP tools return, see mcp-sdk.md), message explains it, and next, when present, names the step to take. Exit codes: 0 ok, 1 the daemon or the operation refused, 2 a usage error (invalid_request), 69 the daemon is not running.

gapreel status --json | jq -r '.data.capture.state'
gapreel clip list --json --limit 5 | jq '.data.items[].text'

Without --json the verbs keep their human output: status prints a short summary, the others print the daemon's JSON answer or a one-line message.

Recording a headless agent run

gapreel-core run [--target screen|window:<id>|browser_tab] [--label <text>] [--profile padded|clips_only] [--project <name>] -- <agent command...> starts a recording (session run-<pid>, the screen by default), runs the agent with its output passed through unchanged, and turns its JSON event stream into the timeline with no hook, no MCP entry and no consent dialog, which suits CI, cron and cloud runners:

Each action carries a W3C trace context: one trace id per run and a span id derived from the stream's item id. Actions are timed when the line arrives, so their wall spans follow the agent live. The run ends with <agent> exited with code N (critical when non-zero), the recording stops, gapreel-core run exits with the agent's exit code, and stderr ends with recorded <recording_id> (<n> actions from a <agent> stream). When the daemon is not running the agent still runs and nothing is recorded. Measured 2026-10-05 with Codex CLI 0.160.0, Claude Code 2.1.289 and Muse Code 1.4.2: two commands gave two actions, one Bash call one action, one bash task one action.

Agent MCP connections

Settings > General > Agent connections registers Gapreel with Claude Code or Codex in one click, shows each connection's state, and removes only Gapreel's matching entry on Disconnect. The equivalent shell commands are:

gapreel-core connect status --json
gapreel-core connect install --client claude-code --json
gapreel-core connect install --client codex --json
gapreel-core connect uninstall --client claude-code --json
gapreel-core connect uninstall --client codex --json

The registered command is the installed stable launcher path plus mcp, otherwise the absolute running gapreel-core path plus mcp. Install the launcher before connecting to survive app moves and updates. Status returns {command, args, clients}; each client includes client, label, state, optional binary and optional detail. States are connected, stale, missing, foreign, not_installed and failed. stale means the entry runs another Gapreel binary (gapreel-core or the gapreel launcher) with mcp, for example one registered before the launcher was installed; install replaces it and uninstall removes it. A foreign entry is never overwritten or removed; inspect it in the client before connecting. --client filters status and selects a mutation; --all selects every supported client through the same client table. --json wraps results in the shared {ok, data} or {ok:false, code, message} envelope.

Claude Code uses claude mcp add --scope user gapreel -- <command> mcp, claude mcp get gapreel, and claude mcp remove --scope user gapreel. Codex uses codex mcp add gapreel -- <command> mcp, codex mcp get gapreel --json, and codex mcp remove gapreel; the installed CLI's help confirms these commands. Client discovery searches PATH plus ~/.local/bin, /opt/homebrew/bin, /usr/local/bin and ~/.claude/local. GAPREEL_CONNECT_CLAUDE_BIN and GAPREEL_CONNECT_CODEX_BIN override discovery for each client.

For compatibility, bare connect status still reports the Saytape connection; use connect status --json, connect status --all, or connect mcp status for MCP registrations. connect saytape and connect disconnect retain their existing meanings.

The card also installs tool-call hooks for both clients using hooks install claude-code --user --json and hooks install codex --user --json. The hooks CLI calls global scope --user, not --global. Installation preserves other hooks, the UI follows the agent_hooks license upgrade flow, and the Codex trust instruction returned by installation is shown in the card. Its browser extension setup link opens Settings > Browser extension.

Recordings in a trace backend

Every Gapreel recording is also an OpenTelemetry trace: one span per action, with the recording's marks as span events. Agents that already emit traces can pass their own context so the recording's spans join the agent's trace, and the span attributes point back to the recording's frames.

What the spans carry

GET /session/otlp?session=<key> (MCP sess_otlp) returns the live recording as OTLP/JSON; the bundle keeps the same document as otlp.json. For each action:

• traceId: the trace_id the agent passed to rec_action_begin or rec_mark (32 hex), else one derived from the recording id. parentSpanId is the agent's span_id when given.
• name: the action label. startTimeUnixNano / endTimeUnixNano: the action's wall-clock span.
• Attributes: gapreel.recording_id, gapreel.timeline_id, gapreel.event_id, gapreel.wall.start_ms / gapreel.wall.end_ms (milliseconds from the recording start), gapreel.frame.before / gapreel.frame.after (key-frame paths relative to the recording folder), and gapreel.export.<export id>.at_ms, .end_at_ms, .included (where the action lands in each export).
• Span events: the marks inside the action.

Labels and attributes pass through the same secret scrubber as the rest of the recording.

Sending a recording

Nothing is sent by default. The user sets a collector once in Settings > General > Trace collector (an OTLP/HTTP traces URL; http://localhost works for a local collector). Then rec_otlp_send (or POST /library/recordings/<id>/otlp/send) sends one recording, and every send is logged in otlp-audit.jsonl beside Gapreel's settings. Headers a hosted backend needs, such as an API key, come from GAPREEL_OTLP_HEADERS=key=value,key2=value2 in the environment the Gapreel daemon starts with; they never appear in a tool result.

Jaeger (OTLP-native)

Checked 2026-10-05 with jaegertracing/all-in-one:1.62.0:

docker run -d --rm --name jaeger -p 127.0.0.1:4318:4318 -p 127.0.0.1:16686:16686 jaegertracing/all-in-one:1.62.0

1. Settings > General > Trace collector: http://localhost:4318/v1/traces.
2. Record with your agent's trace context, for example rec_action_begin {"label": "Open checkout", "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736", "span_id": "00f067aa0ba902b7"}, then rec_stop and rec_otlp_send. The answer lists the trace_ids it sent.
3. Open http://localhost:16686/trace/4bf92f3577b34da6a3ce929d0e0e4736: the action is a span named Open checkout under service gapreel, with tags gapreel.recording_id and gapreel.frame.before (for example frames/evt-001.before.jpg).

Langfuse

Langfuse accepts OTLP/HTTP at https://cloud.langfuse.com/api/public/otel/v1/traces (or your own host) with basic auth built from the project's public and secret keys. This recipe follows Langfuse's documentation and has not been run against a Langfuse project from this repo:

1. Settings > General > Trace collector: https://cloud.langfuse.com/api/public/otel/v1/traces.
2. Start the Gapreel daemon with GAPREEL_OTLP_HEADERS=Authorization=Basic <base64 of public_key:secret_key>.
3. rec_otlp_send, then open the trace in Langfuse by the trace id the answer lists.

From a span to the frame

1. Read gapreel.recording_id and gapreel.event_id from the span.
2. rec_frames {"recording_id": "<id>", "action_id": "<event id>"} returns JPEG paths around that action (it also creates the frames/<event id>.before.jpg / .after.jpg key frames the attributes name, as does rec_context).
3. Or open rec_get {"recording_id": "<id>"}: exports[].track_url is the video, and gapreel.export.<export id>.at_ms is the second to seek to.