{ "project": "action", "version": "0.3.1", "generatedBy": "Dewey 0.3.6", "generatedAt": "2026-10-11T18:15:29.763716+00:00", "sections": [ { "id": "agent-layer", "title": "Agent layer", "content": "# Agent layer\n\nAction's agent layer attaches an app or Action-owned browser to a prepared,\npersistent virtual monitor. The monitor and each task have separate lifetimes.\nUse MCP for the complete browser workflow.\n\n## Prepare the monitor once\n\nProvision the monitor explicitly when the user is ready for a display topology\nchange. Preparation creates the virtual display and may change macOS modes,\narrangement, or window placement; it does not promise an unchanged desktop.\nCheck capabilities and quarantine first. Preparation never clears quarantine.\n\n```bash\nlats action layer display-status\nlats action layer prepare --width 1440 --height 900\n```\n\n`display-status` is read-only. `prepare` reuses an existing live monitor when\nthe requested size matches, or when no size is supplied. A new monitor defaults\nto 1440×900. A different size requires explicit release and preparation; an\nordinary task must not silently resize or recreate the monitor.\n\nOnce prepared, `layer open` attaches a task without connecting another display.\n`layer close` restores that task's windows and leaves the monitor connected.\nWhen the user wants to disconnect the monitor, use `lats action layer release`;\nit first closes the task, then releases the persistent display. Release can\nalso change the desktop topology. A restart, crashed monitor owner, or missing\nmonitor requires explicit preparation again, not automatic recovery during\nthe next task.\n\n## Check capabilities before opening\n\nRun `lats action capabilities`, then call `capabilities` on the active MCP\nconnection. Check its `tools/list` for `layer_open`, `layer_snapshot`,\n`layer_click`, `layer_type`, `layer_js`, and `layer_close`.\nUse `lats action layer display-status` to confirm a prepared monitor is active.\nIf it is missing, stop the task and report the preparation requirement.\n\nCompare the orchestration source digest, signed native build manifest, and\nnative executable hash with the intended deployment. A version match or an\ninstaller message saying the app is current does not prove feature parity.\nFor a standard packaged runtime, require `runtimeParity.state` to be\n`matched-source-digest`; resolve `source-digest-mismatch` or `unverified`\nbefore use. This compares bundled artifacts; it does not verify a running app.\nAn overridden native host is unverified until its own artifact is checked.\nReconnect MCP after changing its runtime or configuration, then repeat discovery.\n\nCheck `safety.privateLayer` before any display or browser is opened. If blocked,\nstop and report its reason and evidence path. Do not clear the quarantine or\nretry through the CLI/native host. Do not replace requested isolation with\nordinary background browser opening; it can still activate or reflow the desktop.\n\nAn absent quarantine means only that no host block is recorded. It does not\nprove acceptance. Inspect `acceptance.privateLayer` in the capabilities reply.\nRequire `state: \"accepted\"` for ordinary use that requires desktop preservation;\nmissing, stale, invalid, or unverified evidence means stop before opening.\nThe record lives at `~/Library/Application Support/Action/agent-layer/isolation-acceptance.json`.\nIt binds acceptance to the native executable and source, orchestration source,\nhost/OS, and physical display configuration. Capabilities checks that binding;\nit does not run display QA or certify a build automatically. Older servers\nwithout the field are unverified. See [Acceptance](#acceptance) for evidence\nrequirements. Controlled QA requires authorization for that host and a\npreserved rollback and layout; do not manufacture a passing record.\n\n**Known failed acceptance, 2026-10-11:** the retained Air trace shows the\nphysical display change from 2560×1664 to 1470×956 during\n`CGVirtualDisplay.applySettings`, before browser setup and Action's positioning\ntransaction. Air remains quarantined. The guarded build prevents display\ncreation; it does not repair the display defect. A successful close or restored\nfocus does not pass the preservation requirement.\n\n**Mini foreground check, 2026-10-11:** retained desktop traces from both the\nPiP and no-PiP browser runs show no physical-display, pointer, or unrelated\nwindow-frame drift. Both show the frontmost process changing to macOS\n`UserNotificationCenter` during `layer_open`, before the first snapshot or\npage input. That foreground check failed and is confounded by system\nnotification UI; it is not evidence of a Chrome/CDP input defect or an\nisolation pass.\n\nThose traces exercised the earlier per-task display-creation path. Keeping a\nmonitor connected removes that hotplug step from ordinary task opening; it\ndoes not retroactively pass those runs.\n\n**Bounded arts-mini task pass, 2026-10-11:** two task sessions reused the same\nprepared display and passed strict, continuous 50 ms desktop-preservation\nmonitoring across open, interaction, and close. This evidence covers that host\nand tested build, which predates the new persistent viewer controls. It does\nnot certify the current controls build, Air, cold provisioning, monitor-owner\nrestart, or release. Provisioning, task attachment, and release need distinct\nevidence. Acceptance for ordinary tasks uses a baseline taken with the\npersistent monitor already prepared; a passing run is not a current matching\nacceptance record by itself.\n\nBuild and diagnose on the development host. `native:doctor` rebuilds and signs;\nit is not a read-only preflight and must not be used to repair an installation\ntarget implicitly. Install a matched native app and orchestration runtime with\nrollback instead. For a packaged runtime, set `ACTION_ROOT=/products/action`,\nnot the extraction root. Keep the archive's sibling `bin/` and `products/`\ndirectories together so the runtime can import `bin/host-pairing.ts`.\n\n## Private browser workflow\n\nCurrent MCP servers advertise underscore names. Use the active `tools/list`;\ndotted `action.*` names are optional compatibility aliases.\n\n```text\nlayer_open { browser: true, url: \"https://example.com\", task: \"Read the page\", pip: true }\nlayer_snapshot {}\nlayer_click { label: \"More\" }\nlayer_type { text: \"Search text\" }\nlayer_js { code: \"document.title\" }\nlayer_go { url: \"https://example.com/next\" }\nlayer_close { outcome: \"done\", summary: \"Read the page\" }\n```\n\n`browser: true` selects Action-owned Chrome. Browser page interaction uses\nCDP exclusively. If CDP is missing, a label or selector cannot be resolved,\ncoordinates reach browser chrome or leave the viewport, or a shortcut is not\nsupported by the page route, the operation fails without native input.\nUse `layer_go` for navigation instead of `cmd+l`. This input contract does not\nestablish that virtual-display creation preserves the physical desktop.\nCDP-only input also does not prevent page JavaScript from requesting popups\nor focus, system permission/notification UI from appearing, or display\ncreation from changing the desktop. Acceptance must observe those effects.\n\nRetain `session` from `layer_open`: `mode`, `pid`, `displayId`, `startedAt`,\nand `browserPid` when present. Action results include the session and execution\nroute (`devtools` with `pageTargetId`, or `blink` with a native `host`). A\nreplacement layer invalidates the binding; do not reuse an earlier snapshot\nor act against the replacement. Check `layer_status` after a connection loss\nor timeout before retrying. Reobserve before repeating an operation whose\noutcome is uncertain.\n\nUse the layer's own snapshot and primitives. `layer_open` begins a drive lease\nunless `leaseId` supplies one; it returns that lease and `layer_close` releases\nthe lease it created. A supplied lease remains the caller's responsibility.\nDo not begin a second drive for the layer or use a desktop snapshot as its\ncoordinate reference. If setup fails, MCP attempts to close the layer it created\nand reports any unconfirmed cleanup. Inspect `layer_status` before retrying.\nCheck `layer_status` before the first open too: opening replaces an existing\nlayer. Do not replace another caller's layer without user authority.\n\nFor native apps, open `{ bundleId: \"com.apple.calculator\", width: 1280,\nheight: 800, pip: true }`. Select `windowId` or `windowTitle` to borrow only\nmatching windows. Native input can temporarily borrow focus or the pointer;\nthat path does not guarantee continuous focus isolation.\n\n## Transport boundaries\n\n| Surface | Layer support |\n| --- | --- |\n| Action MCP | Complete `layer_*` workflow, including `browser: true`, URL navigation, input, and JavaScript |\n| CLI | Monitor `layer prepare`, `display-status`, `release`; task `open`, `status`, `snapshot`, `pip`, `record start/stop`, `close` |\n| Native host | Native display/app primitives; no MCP browser orchestration |\n| Action WebSocket | Raw agent RPC catalog; route private-layer work to MCP |\n| Lattices daemon | Workspace window groups; route private-layer work to Action MCP |\n\nCLI `layer open` accepts `--bundle-id` or `--pid`, `--width`, `--height`,\n`--window-id` or `--window-title`, and `--pip`. It rejects `--browser`, `--url`,\nunknown flags, and unsupported interaction verbs with an MCP routing error.\nThe CLI task lifetime is detached; always close it when work ends. Width and\nheight on `open` must match the prepared monitor; they do not configure a new\ndisplay. Closing a task does not release the monitor.\n\n## Acceptance\n\nRead-only lookup is `lats action capabilities` or MCP `capabilities`, under\n`acceptance.privateLayer`. The native host supplies the current host and\nphysical-display context; inability to read that context leaves acceptance\nunverified. The record's matching status does not override `safety.privateLayer`\nquarantine, prove the running process matches the installed bundle, or replace\nreview of the retained evidence.\n\nThe operator-reviewed record has this schema:\n\n| Field | Required value |\n| --- | --- |\n| `schemaVersion`, `outcome` | `1`, `\"passed\"` |\n| `acceptedAt`, `expiresAt`, `reviewer` | Valid ISO timestamps and a nonempty reviewer; expired records are stale. |\n| `binding` | `executableSha256`, `nativeSourceDigest`, `orchestrationSourceDigest`, `hostFingerprint`, `osBuild`, `displayConfigurationDigest`, matching the current context. |\n| `checks` | `physicalDisplayMode`, `scale`, `arrangement`, `frontmostApp`, `pointer`, `unrelatedWindowFrames`, each `\"unchanged-throughout\"`. |\n| `phases` | `[\"open\", \"interaction\", \"close\"]` |\n| `evidence` | Nonempty list of `{ path, sha256 }` entries; absolute local artifact paths with verified hashes. |\n\nCapabilities discovers and validates this record. It does not write one.\nCreate or renew it only from reviewed, authorized acceptance evidence.\n\nFor ordinary task acceptance, prepare the monitor first under authorized setup\nconditions, then capture physical modes (logical and pixel sizes, scale and refresh rate),\ndisplay arrangement/mirroring, frontmost app, pointer, and unrelated window\nframes before opening. Monitor those values throughout task attachment, interaction,\nand teardown. Verify a layer snapshot and a real browser interaction, then\nconfirm closure and window restoration. Fail on the first unexpected change.\nA final sample that matches the baseline cannot erase an earlier failure.\nTest initial preparation, monitor-owner restart, and explicit release\nseparately; those operations change display topology and must not be folded\ninto a claim that ordinary tasks never connect or disconnect a display.\n\n## What it does\n\n- Attaches to the prepared monitor and moves the subject app's windows onto it. The original frame of each window is recorded. The persistent monitor owner holds the `CGVirtualDisplay` independently of the task.\n- `windowId` (the `kCGWindowNumber`) or `windowTitle` (a case-insensitive substring) moves only the matching windows. Use it to borrow one window of an app the user is also in, such as a browser.\n- Starts one ScreenCaptureKit stream on the layer display and keeps it running for the layer's life. The viewer, snapshots and recordings all come off that stream (see [Feed](#feed)).\n- Launches the subject in the background if it isn't running, and asks an app with no windows for one (the reopen a Dock click sends). Background launch requests no activation; this is not an isolation acceptance result.\n- `url` (a URL, a bare host read as `https`, or a file path) opens in the subject without activating it, once its windows are on the layer. It needs `bundleId` or `pid`, and can't be combined with `windowId` or `windowTitle`.\n- Adopts windows the subject opens while the layer is up, such as a `cmd+n` window, a pop-up, or an `open-app` with `input.url`. They move onto the layer as they appear, cascaded. A layer that borrowed one window by `windowId` or `windowTitle` adopts nothing more.\n- Shows the PiP viewer unless `pip` is `false`.\n- Writes a state file: display id, global bounds (top-left origin, points), PiP state, and the moved windows with their original frames.\n- Keeps one layer at a time. Opening a second layer closes the first, so its windows go back before the new app moves.\n\n## Primitives\n\nThe layer's own tools are the short way to work it. Look, then act on what you saw:\n\n```\nlayer_open { bundleId: \"com.apple.Safari\", url: \"news.ycombinator.com\" }\nlayer_snapshot {} → p.png, 950×560\nlayer_click { x: 475, y: 196 } pixels in that snapshot\nlayer_click { label: \"new\" } or a control by its label\nlayer_type { text: \"lobste.rs\\n\" } a trailing \\n submits\nlayer_press { key: \"cmd+l\" }\nlayer_drag { from: [95, 224], to: [570, 252] }\nlayer_scroll { x: 475, y: 280, dy: -600 }\nlayer_close {}\n```\n\nPoints are pixels in the last `layer_snapshot` image, read through its crop and scale, so an agent never adds the layer's origin or works out the display's scale. `space: \"layer\"` takes points on the layer display instead. A point outside the last snapshot is an error, and so is a point with no snapshot taken yet.\n\nBrowser primitives use the bound CDP page and return `route: \"devtools\"` with\n`pageTargetId`. Native primitives use `action.act.execute` and return\n`route: \"blink\"` with the native `host`. Both report the layer session; input\nprimitives normally return a fresh snapshot (`look: false` skips it). A lease\nis taken for the caller when none is passed. The native routing below is not a\nfallback for a browser session.\n\n## Native blink acts\n\nWhile a layer is up, `action.act.execute` routes these acts as blink acts:\n\n| Act | Host command | When |\n|---|---|---|\n| `click` with a point | `blink-click` | the point is on the layer display |\n| `type` | `blink-type` | no accessibility label path applies |\n| `press-key` | `blink-key` | always |\n| `drag` | `blink-drag` | both points are on the layer, and no `filePath` |\n| `scroll` | `blink-scroll` | the point is on the layer |\n\nAccessibility comes first. A target with a bundle id and label is pressed or set through accessibility and never blinks. `blink-click` hit-tests its point before it touches the pointer: if the element there is a button, checkbox, radio, pop-up, menu item, disclosure triangle, link or tab that takes `AXPress`, it is pressed and the result says `via=ax`. A text field, text area, combo box or search field there is focused through accessibility instead of clicked into. Nothing moves and focus does not change.\n\n`blink-type` works the same way. It inserts the text at the caret of the app's focused text field through accessibility (`via=ax`), and only borrows focus for keystrokes (`via=keys`) when the field doesn't take it, is secure, or a `delayMs` cadence is requested. Web fields usually take keystrokes. When the field's value is readable, the blink holds focus until the text shows up in it (`via=keys verified`), because a web view consumes keys in its own content process after the app's main loop has already drained. A trailing newline in `type` text (`\"lobste.rs\\n\"`) splits off: the text goes in through accessibility and the field is submitted with `AXConfirm`, a Return that moves nothing. `press-key` tries accessibility too: a chord that is a menu item's shortcut (`cmd+l`, `cmd+n`) presses that menu item (`via=ax menu=\"Open Location…\"`), and a bare Return or Escape in a text field confirms or cancels it. Anything else borrows focus for the keystroke (`via=keys`).\n\n`press-key` takes a chord in `input.key`, written however the agent reads it: `\"return\"`, `\"cmd+l\"`, `\"Cmd-Shift-T\"`, `\"⌘⇧T\"`, `[\"cmd\", \"l\"]` in `input.keys`, or `key` plus `input.modifiers`. Aliases such as `enter`, `esc`, `backspace` and `ArrowDown` map to the host's names. An unknown key is an error that points at `type`, never text typed one character at a time.\n\nThe pointer is the fallback: no pressable element at the point, a requested `holdMs`, or `--pointer` on the host command. A pointer blink saves the cursor position and the frontmost app, clicks, and restores both within tens of milliseconds. Its result says `via=pointer`.\n\n`type` and `press-key` land in the app's focused window. If that window is off the layer, the blink raises the app's layer window first, without activating the app, and refuses if focus still isn't on the layer. Keystrokes never reach a window on the user's screen.\n\n`type` and `press-key` target `input.bundleId` or `input.pid` if given, else the layer's app. Accessibility paths (`press-accessibility-element`, `set-accessibility-value`) do not change. A click outside the layer display uses `click-point`.\n\nDrag and scroll have no accessibility equivalent and always use the pointer. It leaves the user's screens for the length of the gesture (200 ms for a drag by default; an instant for a scroll) and comes back. Scroll wheel events activate nothing. A drag's mouse-down activates the app it lands in for up to a second, and focus is handed back afterwards.\n\nBlink acts report the `blink` tier. A background drive lease allows them. They do not show the pointer-focus countdown.\n\n## Feed\n\nThe layer captures its own display, not the subject app. A stream that targets an app makes macOS badge that app's windows as shared, and the badge is drawn into the window, so it would show in every frame. The framing to the subject's windows happens downstream, by crop. The stream only delivers frames when something on the layer changes, so an idle layer costs next to nothing.\n\n- **Viewer.** The PiP grows out of the screen corner on its first frame and frames the subject's windows, following them as they move and resize. Drag the header to move it; double-click the header to enlarge it, and double-click again to restore its size. Hide leaves the layer running; `layer pip` / `layer_pip` brings it back.\n- **State.** One dot in the viewer's bottom-right corner: solid coral while the agent can act, breathing while a take records, hollow while paused.\n- **Marks.** The host posts where native acts landed and the viewer rings it: a circle for a click, an outline around the focused field for typing and keys. Accessibility paths can avoid pointer movement; pointer and key fallbacks may temporarily borrow pointer or focus.\n- **Note.** The latest `action.drive.note` shows along the bottom edge for about 12 seconds.\n- **Snapshot.** `layer snapshot` / `layer_snapshot` writes a PNG from the latest frame: the subject's windows by default, one window with `windowId`, or the whole layer with `full`. Nothing is captured on request, so it answers in tens of milliseconds. `unchangedMs` is how long the layer has been still, not how stale the picture is.\n- **Recording.** `layer record start|stop` / `layer_record` adds a recording output to the running stream, so the take starts on the next frame. It records the whole layer display at its native size. One take at a time; closing the layer finishes a take in progress.\n\n## Operator controls\n\nThe viewer has a persistent hostname header and native controls outside the captured image:\n\n- **Pause / Resume.** Pauses layer input dispatch. While paused, `state.json` has `\"paused\": true`; observations remain available. Resume is an operator action. Events or page scripts already dispatched cannot be undone.\n- **Stop.** Pauses input, records a session-specific `handoff.json`, and ends the layer without activating the target app. This stops the layer session, not the agent's entire harness or unrelated work.\n- **Agent.** Looks for the opener's app on this host. Action records its process chain at `open` (`--owner-pids`), activates the nearest app in that chain, and otherwise asks Lattices (`terminals.search`, then `window.focus`) which local terminal window shows it. If nothing resolves, the viewer beeps. This does not return from an SSH host to a conversation on another machine.\n- **Take over.** Records a session-specific handoff, ends the layer, and activates the target app. Borrowed windows return to their recorded frames; windows created on the layer are closed, subject to unsaved-changes prompts. This is not an in-place remote-control mode.\n- **Hide.** Hides only the viewer; the layer keeps running.\n\nStop and Take over refuse further agent input during teardown and after cleanup. `layer close` does not clear the handoff. A new explicit `layer open` acknowledges it; callers must obtain fresh user authorization before opening again. The prepared monitor remains connected after either action.\n\nThese are host-local controls. An interactive viewer streamed to another machine or into a chat is not implemented.\n\nRequests go through `control.request.json` next to the state file and a `SIGUSR2`; the reply lands in `control.reply.json`.\n\n## Lifetime\n\n`layer_close` is the intended teardown. It creates the stop file and waits for the layer to put the windows back and exit. Windows that were opened after the layer went up (`born` in the state file) never sat on the user's screen, so they are closed instead of moved there. One that asks before closing (unsaved changes) stays open and goes back to the frame it appeared at. If the layer does not exit, Action sends `SIGTERM`.\n\nThis tears down the task's feed and window attachment, not the monitor.\n`layer release` explicitly closes any task and then stops the persistent\nmonitor owner. It must not disconnect the monitor when task closure fails.\n\nThe layer also tears down when its owner dies:\n\n- **MCP.** The layer watches the MCP server (`--parent-pid`) and restores the windows if the server exits.\n- **CLI.** The CLI exits once the layer is up, so the layer is detached. Run `layer close` to take it down when the work is done.\n\nA detached layer doesn't wait for `layer close` forever. It also comes down when:\n\n- **Its owners exit.** Every process in the opener's chain that was alive at `open` has exited.\n- **It goes idle.** 30 minutes pass with no act, no snapshot or recording request, and no use of the viewer. A recording in progress keeps the layer up. Pass `--idle-timeout ` to the host to change the lease; `0` turns it off.\n\nEither way the windows go back first, as with `layer close`.\n\n`layer status` reports the live layer. If the recorded process is gone, status removes the stale files and reports the layer as inactive.\n\nTask and monitor state live in `~/Library/Application Support/Action/agent-layer/`.\nUse `layer status` for the task and `layer display-status` for the monitor;\nan inactive task with a ready monitor is the normal state between tasks.\n\n## Requirements and limits\n\n- The feed needs Screen Recording for Action. Without it the layer still works (windows move, blink acts land), but there is no viewer, snapshot or recording; the layer's `layer.log` says `feed failed`. Recording needs macOS 15.\n- The display touches the bottom-right-most display only at its corner. A pointer pushed exactly through that corner can still cross onto it.\n- A pointer blink click on another app activates that app for a moment before focus goes back. The menu bar can flicker for that moment.\n- Blink acts refuse targets that aren't on an agent layer: a point off the layer, or an app with no window on it. `--any-display` lifts that for tests.\n- Killing a task with `SIGKILL` skips its window restoration; the persistent monitor remains. Killing the monitor owner disconnects the display, allowing macOS to relocate its windows. Neither is an ordinary task teardown or a preservation guarantee." }, { "id": "overview", "title": "Overview", "content": "# Overview\n\n`action` is a native-first macOS demo automation project.\n\nIt is aimed at a workflow where a human operator or an AI agent can:\n\n- inspect an app, browser surface, or bounded region\n- execute deterministic actions\n- record raw capture plus structured runtime traces\n- turn those traces into polished demo or promo outputs later\n\n## What Exists Today\n\nThe current repo is early, but it already has a meaningful native core:\n\n- a signed `Action.app` bundle\n- a real AppKit launcher with menus and WebKit support\n- a local `Action` agent runtime reachable over WebSocket\n- native screenshot and recording commands\n- permission and diagnostics wrappers for local development\n\nThe strongest proof point right now is native capture:\n\n- screenshot flows work\n- `ScreenCaptureKit` recording now works through a real app lifecycle path\n\n## Current Architecture Direction\n\nThe project is deliberately split into two responsibilities:\n\n- `Action.app` owns AppKit lifecycle, menus, WebKit, settings, and permission UX\n- the local agent owns transport, automation-facing methods, and runtime orchestration\n\nThis split exists because UI lifecycle and automation lifecycle are not the same\nproblem on macOS. Earlier experiments showed that trying to make a command-style\nruntime also own WebKit and recording behavior leads to brittle failures.\n\n## Repository Shape\n\n- [README.md](README.md): top-level project framing\n- [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md): deeper product and systems architecture\n- [docs/VISION.md](docs/VISION.md): product intent and precedent learnings\n- [native/engine](native/engine): Swift native engine, app host, local agent, and scripts\n- [packages](packages): JS-side tooling and operator surfaces\n\n## What Matters Most Right Now\n\nAt this stage, the most important technical goal is reliable native capture.\n\nThat means:\n\n- real AppKit lifecycle correctness\n- stable `ScreenCaptureKit` recording behavior\n- clear artifacts and finished markers\n- preserving a clean boundary between UI-owned behavior and agent-owned behavior" }, { "id": "getting-started", "title": "Getting-started", "content": "# Getting Started\n\nThis is the shortest path to a useful local dev loop for `action`.\n\n## Prerequisites\n\n- macOS on Apple Silicon\n- Bun\n- Swift/Xcode native build tooling\n- willingness to grant Accessibility and Screen Recording permissions for native automation tests\n\n## Install\n\n```bash\nbun install\n```\n\n## Build The Native App\n\n```bash\nbun run native:app:build\n```\n\nThis produces a signed app bundle at:\n\n`native/dist/Action.app`\n\n## Check Native Health\n\nOn an authorized development host, use the doctor wrapper before debugging capture:\n\n```bash\nbun run native:doctor\n```\n\nThis command changes the native build. Do not use it as read-only preflight on\nan installation target; install a matched app/runtime with rollback there.\nFor private layers, read [Agent layer](agent-layer.md) and verify capabilities,\nhost quarantine, and desktop-preservation acceptance before opening.\n\nThe development-host doctor command:\n\n- builds the app if needed\n- signs it\n- verifies signature state\n- reports current Accessibility and Screen Recording status\n\n## Launch The App\n\n```bash\nbun run native:launch\n```\n\nRebuild and relaunch after a native change:\n\n```bash\nbun run native:relaunch\n```\n\nFrom the Lattices repo root, `bun run action:launch` and `bun run action:dev` call the same scripts.\n\n## Useful Smoke Commands\n\nCheck permissions:\n\n```bash\nbun run native:permissions:status\n```\n\nRequest permissions:\n\n```bash\nbun run native:permissions:request\n```\n\nRun a screenshot smoke test:\n\n```bash\nbun run native:test:screenshot\n```\n\nRun a recording smoke test:\n\n```bash\nbun run native:test:record\n```\n\n## Important Runtime Note\n\nRecording is asynchronous.\n\nThe initial CLI response means recording startup was accepted. Completion is\nrepresented by the artifact plus a finished marker file written later by the\nrecording path.\n\nIf you are debugging recording, inspect:\n\n- the `.mov` output\n- the `.finished` marker\n- the debug log passed through `--debug-log`\n\n## Where To Read Next\n\n- [docs/native-runtime.md](docs/native-runtime.md)\n- [docs/recording.md](docs/recording.md)\n- [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md)" }, { "id": "native-runtime", "title": "Native-runtime", "content": "# Native Runtime\n\nThis page describes the runtime split that currently makes `action` behave like\na real macOS app instead of a command wrapper pretending to be one.\n\n## The Split\n\nThere are two native roles:\n\n## `Action.app`\n\nOwns:\n\n- AppKit lifecycle\n- menus and app-switcher behavior\n- WebKit windows\n- launcher UI\n- permission UX\n- recording probe execution\n\nMain file:\n\n- [ActionHostMain.swift](native/engine/Sources/ActionHostMain.swift)\n\n## Local Agent Runtime\n\nOwns:\n\n- WebSocket transport\n- automation-facing request handling\n- local runtime methods\n- bridge-friendly command orchestration\n\nMain files:\n\n- [ActionAgentRuntime.swift](native/engine/CoreSources/ActionAgentRuntime.swift)\n- [ActionAgentClient.swift](native/engine/CoreSources/ActionAgentClient.swift)\n- [ActionAgentCommandBridge.swift](native/engine/Sources/ActionAgentCommandBridge.swift)\n\n## Why This Exists\n\nTwo important failures pushed the design here:\n\n### 1. WebKit\n\nWebKit behaved unreliably when UI modes were entered through the wrong runtime\npath. The fix was to let AppKit own the UI lifecycle cleanly.\n\n### 2. ScreenCaptureKit Recording\n\nRecording was not reliable when attempted inside the wrong lifecycle context.\nThe stable path now launches a fresh `Action.app` instance in `recording-probe`\nmode for the actual capture work.\n\nThe lesson is simple:\n\n**AppKit-dependent work should run inside a real app lifecycle.**\n\n## Current Execution Flow\n\nFor normal operator usage:\n\n1. `Action.app` launches\n2. launcher UI comes up\n3. local agent is started if needed\n4. UI talks to the agent for runtime state and automation methods\n\nFor recording:\n\n1. command or UI requests recording\n2. host talks to the local agent\n3. agent launches `Action.app` again in `recording-probe` mode\n4. probe performs the actual `ScreenCaptureKit` recording\n5. artifacts and finished markers are written\n\n## Important Boundaries\n\nKeep these rules intact:\n\n- do not move WebKit lifecycle back into a fake headless command path\n- do not move recording implementation back into the plain headless agent path\n- keep the agent useful for orchestration and transport\n- keep AppKit-owned concerns in the app process\n\n## Private-layer discovery\n\nUse `lats action capabilities` and the active MCP `capabilities` / `tools/list`\nfor build provenance and routing. Check host quarantine before opening a private\ndisplay. Native support, runtime health, and version strings do not establish\nphysical desktop preservation. See [Agent layer](agent-layer.md).\n\n`native:doctor` builds and signs. Run it on an authorized development host.\nOn an installation target, preserve the installed bundle and rollback; deploy\na matched app and MCP runtime instead of rebuilding implicitly. Reconnect MCP\nafter runtime changes.\n\nTo assemble a matched runtime candidate on the development host, use:\n\n```bash\nbun scripts/package-agent-runtime.ts --app /path/Action.app --output /new/action-runtime --instructions-root /path/to/lattices-docs-checkout\n```\n\nThe command verifies the signed app's recorded MCP source fingerprint, bundles\nthe source and locked production dependencies, and includes the selected layer\ndocs and Action skill. It rejects a mismatched pair or existing output and does\nnot install anything. `runtime-provenance.json` records the artifact identities\nand marks host desktop-isolation acceptance as uncertified. Keep the previous\nruntime for rollback and preserve host quarantine when installing a candidate.\n\nThe extracted archive contains sibling `bin/` and `products/` directories.\nSet `ACTION_ROOT=/products/action`, not the extraction root, and keep\n`bin/host-pairing.ts` in place; the runtime imports that canonical dependency.\n\n## Scripts And Helpers\n\nCall native work through package.json scripts from `products/action`:\n\n```bash\nbun run native:app:build\nbun run native:doctor\nbun run native:launch\nbun run native:relaunch\nbun run native:host -- status\nbun run native:test:screenshot\nbun run native:test:record\n```\n\nThe scripts wrap helpers in [native/engine/scripts](native/engine/scripts). The most important wrappers are:\n\n- `build-app.sh`\n- `doctor.sh`\n- `run-app-host.sh`\n- `test-screenshot.sh`\n- `test-record.sh`" }, { "id": "recording", "title": "Recording", "content": "# Recording\n\nRecording is the most important runtime path in this project.\n\nThis page documents how it currently works and what assumptions should remain\ntrue while the product is still being hardened.\n\n## What Is Stable Right Now\n\nThe native recording path now succeeds for:\n\n- bounded region recording\n- app-window recording\n\nThe successful path produces:\n\n- a `.mov` file\n- a `.finished` marker file\n- an optional debug log if `--debug-log` is supplied\n\n## The Key Discovery\n\nThe main recording bug was not just a bad capture configuration.\n\nThe deeper issue was lifecycle ownership:\n\n- `ScreenCaptureKit` recording was unreliable or crash-prone when hosted from the\n wrong runtime path\n- a minimal real AppKit app lifecycle succeeded with the same capture logic\n\nThat is why the current solution uses a dedicated `recording-probe` mode inside\n`Action.app`.\n\n## Current Recording Path\n\nFor a region recording request:\n\n1. host command accepts the request\n2. host sends a recording method to the local agent\n3. agent launches `Action.app` with `recording-probe`\n4. probe runner creates a tiny AppKit window and starts recording\n5. recording continues until the stop file appears\n6. probe writes the finished marker and exits\n\nCore files:\n\n- [ActionRecordingProbeLauncher.swift](native/engine/CoreSources/ActionRecordingProbeLauncher.swift)\n- [RecordingProbeAppRunner.swift](native/engine/Sources/RecordingProbeAppRunner.swift)\n- [ActionHostMain.swift](native/engine/Sources/ActionHostMain.swift)\n\n## Testing Recording\n\nThe wrapper script is:\n\n```bash\nbun run native:test:record\n```\n\nFor lower-level testing, `run-app-host.sh` can call recording commands\ndirectly.\n\n## Important Caveat\n\nThe initial CLI reply usually means:\n\n`recording started successfully`\n\nIt does **not** mean:\n\n`recording completed successfully`\n\nCompletion is represented by the finished marker file.\n\n## What To Preserve\n\n- real AppKit lifecycle for actual recording work\n- stop-file and finished-file based orchestration\n- debug-log passthrough for probe runs\n- clean app/agent separation even if the app is doing the final recording work\n\n## What To Avoid\n\n- pushing real recording back into a plain headless lifecycle\n- assuming a WebSocket ack is equivalent to a completed recording\n- removing the probe path before a better lifecycle-safe replacement exists" }, { "id": "ARCHITECTURE", "title": "ARCHITECTURE", "content": "# Architecture\n\n`action` is a ground-up rewrite for agentic macOS demo automation.\n\nIt exists because `vif` and `stage` both produced valuable learnings, but neither is the right long-term core:\n\n- `vif` proved there is real value in combining capture, browser/app automation, overlays, audio, and export.\n- `stage` proved the architecture should be cleaner, more session-oriented, and more explicit about primitives.\n- Both also showed a failure mode: if the scene format, CLI behavior, and runtime lifecycle get tangled together, the tool becomes hard to reason about and easy to break.\n\nThe central architectural decision for `action` is:\n\n**The runtime is the product. The scene format is a compiled input, not the architecture.**\n\nAnother useful way to say the same thing:\n\n**`action` is an action runtime that can observe, act, reflect, and record.**\n\n## Goals\n\n- Provide a deterministic, session-based runtime for observing apps and browsers, resolving targets, executing actions, reflecting on what happened, and recording structured traces.\n- Support agent-authored demo and promo workflows without relying on hidden heuristics or magical state.\n- Separate live automation from final composition so polished output can improve without destabilizing the runtime.\n- Lean fully into macOS-native fidelity instead of designing for fake cross-platform flexibility.\n\n## Non-Goals\n\n- Not a generic automation framework for every operating system\n- Not an Electron-first runtime\n- Not a GUI editor in v0\n- Not a vision-only \u201cclick whatever looks right\u201d system\n- Not a scene DSL that owns business logic and runtime state\n- Not a VM/container-first automation stack for host UI, browser, or capture\n work\n\n## Core Layers\n\n`action` should be implemented as a small number of strict layers.\n\n### 1. Native Engine\n\nSwift-based macOS engine responsible for:\n\n- ScreenCaptureKit capture\n- Accessibility inspection\n- Input synthesis\n- Window enumeration and positioning\n- Overlay windows when needed during live capture\n- Audio session and telemetry primitives\n\nThe engine should be treated as a capability provider, not the place where product workflows live.\n\nThe native layer should be split into separate processes:\n\n- `Action.app`: normal AppKit lifecycle, WebKit, menus, settings, permissions UX\n- `ActionAgent`: headless local service exposing WebSocket and local IPC surfaces\n- `ActionCLI`: thin operational client that talks to the agent, not directly to UI\n\nThis separation is not optional. UI lifecycle and agent lifecycle have different\nconstraints, and combining them leads to brittle behavior.\n\n### Remote Workers\n\nRemote VMs and containers can still be useful for auxiliary work such as\nisolated command execution, rendering, browser-free tests, or heavyweight model\njobs. They should not own the core Action browser, Accessibility, AppKit, or\nScreenCaptureKit paths because those capabilities depend on the user's macOS\nhost, permissions, windows, and dedicated local Chrome profiles.\n\n### 2. Runtime\n\nThe runtime is the heart of the system. It owns:\n\n- session creation and teardown\n- lifecycle transitions\n- current surfaces and active targets\n- observations and resolution logic\n- action execution\n- reflection and inspection workflows\n- traces and artifacts\n\nAll mutable run state belongs to a concrete session. Avoid hidden globals and avoid lifecycle behavior that relies on shell cleanup or broad process-kill patterns.\n\n### 3. Compiler\n\nThe compiler converts author intent into an executable timeline.\n\nIt should:\n\n- validate scene/input structure\n- normalize high-level user intent into explicit runtime actions\n- attach fallbacks and preconditions\n- separate narrative cues from execution steps\n- produce an inspectable timeline before execution\n\nThe compiler should eventually support more than one input source:\n\n- hand-authored scenario intent\n- recorded runtime traces converted into scenario drafts\n\nThis matters because one of the most valuable workflows is likely to be:\n\n1. a user demonstrates a flow manually\n2. the runtime captures media plus trace\n3. `action` derives a reusable scenario from that run\n4. the system reruns or polishes it later\n\n### 4. Composer\n\nComposition is a separate phase from capture and execution.\n\nThe composer should:\n\n- transform raw captures and runtime traces into a render manifest\n- infer or apply zooms, cursor polish, subtitles, callouts, chapter cards, and audio timing\n- delegate actual rendering to a backend such as Remotion\n\nThis separation is essential. Styling, transitions, and branding should be able to evolve independently from action execution.\n\nThe composer should not own live app control, session lifecycle, or target\nresolution. It is post-capture orchestration, not the runtime.\n\n### 5. Frontends\n\nFrontends must stay thin.\n\n- CLI: operational control and local authoring workflows\n- MCP: agent-facing execution surface\n- TS HUD: richer operator console for logs, diagnostics, artifacts, scenario control, and reflection output\n- native HUD: minimal always-on capture companion tied closely to runtime truth and immediate controls\n\nThey should call the same runtime interfaces, not implement separate behavior.\n\nThe HUD split should be deliberate:\n\n- the native HUD should stay minimal and capture-adjacent\n- the TS HUD can carry more product richness and operator affordances\n- neither HUD should become the owner of runtime truth\n- if a HUD is on screen during a live session, it should expose explicit controls instead of acting like passive decoration\n\n## Core Primitives\n\n`action` should keep its vocabulary small and stable.\n\n### Session\n\nOne automation run with:\n\n- stable id\n- explicit lifecycle\n- owned resources\n- trace output\n- artifact list\n\nSuggested lifecycle:\n\n- `created`\n- `preflight`\n- `ready`\n- `running`\n- `paused`\n- `completing`\n- `completed`\n- `failed`\n- `cancelled`\n\n### Surface\n\nA place where the runtime can observe or act.\n\nExamples:\n\n- desktop\n- app window\n- browser tab\n- bounded region\n\n### Observation\n\nA typed fact gathered from the engine or browser adapter.\n\nExamples:\n\n- window bounds\n- DOM node snapshot\n- AX element\n- cursor position\n- recording state\n- audio level\n\n### Reflection\n\nA structured interpretation of current state or recent action.\n\nExamples:\n\n- UI critique from a model provider\n- QA finding attached to a screenshot\n- target suggestion derived from vision plus AX context\n- operator-facing summary of what changed after an action\n\nReflection is not the same thing as raw observation.\n\n- observations are facts gathered from the system\n- reflections are interpretations produced from those facts\n\nThe runtime should own reflection sessions, prompts, findings, and artifacts, but\nprovider-specific model clients should stay outside the native engine. See\n[LIVE_INSPECTION_RUNTIME.md](docs/LIVE_INSPECTION_RUNTIME.md).\n\n### Target\n\nA stable reference to something actionable.\n\nTarget resolution should support:\n\n- semantic ids\n- AX/DOM-based resolution\n- bounded text/role matching\n- calibrated anchors\n- coordinates as last resort\n\n### Action\n\nA deterministic atomic operation.\n\nExamples:\n\n- click\n- type\n- press key\n- focus window\n- open app\n- start recording\n- stop recording\n- show cue\n- wait for condition\n\n### Cue\n\nA narrative or editorial element that matters to the viewer.\n\nExamples:\n\n- label\n- chapter marker\n- subtitle\n- callout\n- caption\n\n### Effect\n\nA render-time presentation element, not necessarily a live overlay.\n\nExamples:\n\n- auto zoom\n- smart reframe\n- cursor emphasis\n- background dimming\n- click pulse\n- transition\n\n### Artifact\n\nAnything the run produces.\n\nExamples:\n\n- screenshot\n- raw capture\n- trace log\n- focus metadata\n- subtitle file\n- final rendered video\n\n## Target Resolution\n\nTarget picking is a v0 feature, not a nice-to-have.\n\nThe system must not normalize blind coordinate clicking as the default interaction model.\n\nRecommended resolution order:\n\n1. semantic target\n2. exact accessibility or DOM match\n3. bounded fuzzy text/role match\n4. calibrated anchor\n5. coordinate fallback\n\nRecommended API shape:\n\n- `observe()`\n- `target.resolve(query, context)`\n- `act(action, target)`\n\nAnd the resolver must be conservative:\n\n- return confidence\n- return ambiguity explicitly\n- avoid guessing when multiple candidates are plausible\n- allow human or agent inspection before acting\n\n## Execution Model\n\nThere are two major phases.\n\n### Live Phase\n\nThe runtime:\n\n- prepares the environment\n- resolves targets\n- executes actions\n- captures raw media\n- records metadata and trace events\n\n### Compose Phase\n\nThe composer:\n\n- consumes raw media and trace data\n- computes polish such as zooms and cursor motion\n- aligns narration, subtitles, music, and chapter cues\n- renders exports via a backend\n\nThis split is a hard requirement. It keeps the runtime reliable and the output quality flexible.\n\n## Scene Model\n\nScenes should describe intent, not be forced to contain all operational detail.\n\nGood scenes should express:\n\n- goal\n- context\n- surfaces\n- targets\n- sequence\n- narration\n- style\n- export preferences\n\nThe compiler should lower those into an executable timeline with explicit dependencies and fallbacks.\n\n## Render Backend Boundary\n\n`action` should define its own render manifest and allow multiple composition backends.\n\nNear-term plan:\n\n- `composer-core`: manifest schema and transformation logic\n- `composer-remotion`: first production backend\n\nThis allows `action` to borrow high-quality composition capabilities from tools like Remotion without turning React video rendering into the core product architecture.\n\n## Operational Design Rules\n\n- No daemon mystery states\n- No hidden global runtime state\n- No broad shell cleanup as the main lifecycle mechanism\n- No UI-only source of truth\n- No premature cross-platform abstraction\n- No scene format that bypasses runtime primitives\n\n## Proposed Package Boundaries\n\n- `packages/protocol`\n- `packages/runtime`\n- `packages/compiler`\n- `packages/composer-core`\n- `packages/composer-remotion`\n- `packages/cli`\n- `packages/mcp`\n- `native/engine`\n\n## v0 Definition of Done\n\n`action` v0 should feel solid before it feels broad.\n\nThat means:\n\n- explicit session lifecycle\n- reliable target resolution\n- deterministic action execution\n- raw capture plus trace metadata\n- polished but backend-driven composition\n- clean CLI and MCP surfaces\n\nIf v0 cannot reliably observe, resolve, act, record, and compose a basic feature demo on macOS, it is not done." }, { "id": "AX_BACKGROUND_AUTOMATION", "title": "AX_BACKGROUND_AUTOMATION", "content": "# AX Background Automation\n\nAction should treat macOS Accessibility as the default path for host-native\nautomation.\n\nThe product goal is not to create a second hardware cursor. The goal is to act\nthrough semantic app surfaces while presenting a visual cursor/caret overlay\nthat explains what is happening.\n\n## Model\n\nFor a target app, Action should prefer:\n\n1. read the AX tree\n2. resolve a specific element\n3. use semantic AX actions or settable attributes\n4. show a non-interactive overlay describing the action\n5. verify by reading AX state again\n\nThe overlay is presentation only. It must not be treated as the input channel.\n\n## Action Ladder\n\nUse the least attention-taking primitive that can complete the task:\n\n1. `observe`: AX tree, window bounds, screenshot.\n2. `semantic`: `AXPress`, `AXShowMenu`, set `AXValue`, set `AXSelectedText`.\n3. `target-focus`: set `AXFocused`, then send direct app/process events if the\n app requires a focused control.\n4. `app-api`: Chrome DevTools Protocol, AppleScript, Shortcuts, or app-specific\n APIs when they are more deterministic than generic AX.\n5. `attention`: activate/raise an app, system hotkeys, pointer warping, or HID\n events.\n\nThe runtime should record which tier was used for every action.\n\n## Warning Policy\n\n`observe` and `semantic` actions can run silently with the top-right trace.\n\n`target-focus` actions should show a subtle amber notice:\n\n> Target focus may change inside {app}.\n\n`attention` actions should require an explicit, visible warning before execution:\n\n> Action needs foreground control of {app} and may move focus or the pointer.\n\nThe user should be able to cancel or defer attention-taking actions.\n\n## Current Daily App Findings\n\nUse:\n\n```bash\nbun run native:ax:audit\n```\n\nThe audit is read-only. It snapshots AX nodes for daily apps and reports:\n\n- roles\n- available AX actions\n- settable attributes\n- pressable elements\n- text-like writable controls\n- whether the frontmost app changed during the audit\n\nEarly local findings:\n\n- Chrome exposes many `AXPress` controls and an `AXTextField` for the omnibox.\n- iTerm exposes titlebar/search controls plus a large terminal `AXTextArea`.\n- Codex currently exposes a coarse Electron shell: mostly groups and titlebar\n buttons, not a rich editor/input AX tree in this sample.\n- Cursor should be audited when running; the script intentionally does not\n launch apps because launching can itself become attention-taking behavior." }, { "id": "CAPTURE_CONTROLS_PRECEDENT", "title": "CAPTURE_CONTROLS_PRECEDENT", "content": "# Capture Controls Precedent\n\n## Keep\n\n- a countdown that is obvious before capture starts\n- controls that stay reachable while the take is live\n- runtime-owned truth for recording state and interruption\n\n## Avoid\n\n- a HUD that shows buttons without a real control path behind them\n- fake pause states when native capture is not actually paused\n- interruption paths that leave recording hanging without a stop marker\n\n## Adapt\n\n- use the native stage HUD as the immediate operator surface\n- keep the control channel file-based so the overlay and runner stay loosely coupled\n- treat interrupt as a clean stop/cancel path instead of pretending the take can be paused safely\n\n## Decision\n\n- the guided native session will surface countdown skip/cancel before capture starts\n- the live take will surface an interrupt control that stops recording cleanly\n- cancelled runs will report cancellation explicitly instead of pretending they completed\n\n## Reference Weight\n\nBlend of `vif` product feel and `stage` runtime-boundary discipline." }, { "id": "COMPOSITION_AND_SCENARIOS", "title": "COMPOSITION_AND_SCENARIOS", "content": "# Composition And Scenarios\n\n## Purpose\n\nThis document explains what \"composition\" means in `action`, and how it relates\nto agentic automation, manual capture, and reusable scenarios.\n\nThe key point is simple:\n\n**Composition happens after capture.**\n\nIt is not the runtime. It is not the session controller. It is not the source of\ntruth for what happened.\n\n## Core Idea\n\n`action` should separate three concerns:\n\n1. live interaction and capture\n2. scenario representation\n3. final presentation\n\nThose are related, but they should not be fused into one system.\n\n### Live Interaction And Capture\n\nThis is the runtime phase.\n\nIt includes:\n\n- observing windows, surfaces, and targets\n- acting through clicks, typing, focus changes, and waits\n- recording video and screenshots\n- collecting structured traces\n\nThis is where truth is created.\n\n### Scenario Representation\n\nA scenario is a reusable, inspectable description of what should happen.\n\nIt may be authored:\n\n- directly by a human\n- directly by an agent\n- derived from a recorded session\n- derived from a hybrid of user actions plus later cleanup\n\nThis is where intent is organized.\n\n### Final Presentation\n\nThis is the composition phase.\n\nIt includes:\n\n- zooms\n- reframing\n- cursor emphasis\n- chapter cards\n- subtitles\n- narration timing\n- music\n- transitions\n\nThis is where polish is applied.\n\n## What Composer-Core Should Mean\n\n`composer-core` should be a narrow package.\n\nIt should own:\n\n- render-manifest types\n- composition-friendly timing structures\n- focus-window descriptions\n- chapter and subtitle placement data\n- effect definitions that can be consumed by a renderer\n\nIt should not own:\n\n- live app control\n- target resolution\n- session lifecycle\n- capture start/stop\n- HUD control\n- input synthesis\n\nThat means `composer-core` is not \"the video editor.\" It is the contract between\nthe runtime's artifacts and a rendering backend such as Remotion.\n\n## Agentic First Still Holds\n\nThis architecture is still agentic first.\n\nThe agent-facing value comes from:\n\n- stable targets\n- deterministic runs\n- inspectable traces\n- reusable scenarios\n\nThe important shift is that the agent should not be forced to author everything\nup front.\n\nThere are at least three valid ways to create a scenario.\n\n## Three Scenario Sources\n\n### 1. Authored Scenario\n\nA human or agent writes the intended sequence before execution.\n\nExample:\n\n- open Calculator\n- type `12`\n- click `+`\n- type `30`\n- press `=`\n\nThis is the most direct \"agentic automation\" mode.\n\n### 2. Recorded Session\n\nA human drives the product manually while `action` records:\n\n- video\n- focus changes\n- target resolutions\n- clicks\n- keys\n- timing\n- window state\n\nLater, `action` can convert that into a structured scenario draft.\n\nThis is the mode you were pointing at: a user navigates a website or UI, and the\nsystem turns the observed session into something reusable.\n\n### 3. Hybrid Scenario\n\nThe user records an initial pass, and then the system or agent cleans it up:\n\n- replace weak coordinate clicks with semantic targets\n- remove dead time\n- add preconditions\n- normalize waits\n- insert cues\n\nThis is likely one of the strongest long-term workflows for `action`.\n\n## Why Recorded Sessions Matter\n\nRecorded sessions solve a real product problem.\n\nMany users know how to show the flow they want, but they do not want to author a\nscene format from scratch.\n\nIf `action` can observe a manual run and convert it into a scenario draft, then:\n\n- onboarding becomes much easier\n- the runtime captures real truth from the product\n- agents get a better starting point than a blank file\n- deterministic reruns become possible after the initial demonstration\n\nThis is one of the most compelling directions in the product.\n\n## Proposed Pipeline\n\nThe long-term pipeline should look like this:\n\n1. run a live session\n2. capture media plus trace\n3. derive or refine a scenario from the trace\n4. replay or rerun that scenario deterministically\n5. compose a polished export from the captured or rerun artifacts\n\nIn short:\n\n`capture -> trace -> scenario -> rerun -> compose`\n\nNot every workflow needs every step, but this is the most powerful full loop.\n\n## Trace-To-Scenario\n\nThe compiler should not only accept hand-authored scene input.\n\nIt should also eventually accept recorded traces as source material.\n\nThat implies two compiler inputs:\n\n- `intent -> timeline`\n- `trace -> scenario draft -> timeline`\n\nThe second path is important because it lets `action` learn from a real user-run\nsession without turning the runtime into a scene DSL.\n\n## What The Trace Must Preserve\n\nIf we want recorded sessions to become reusable scenarios later, the trace needs\nto preserve enough structure now.\n\nThat includes:\n\n- window and surface identity\n- target query and resolution result\n- confidence levels\n- click positions\n- key presses and text entry\n- timing boundaries\n- state transitions\n- artifacts created during the run\n\nThis is another reason the runtime matters more than the scene format.\n\n## What Composition Consumes\n\nThe composition layer should consume:\n\n- raw video capture\n- screenshots\n- runtime trace\n- viewport and focus metadata\n- cues or chapters\n- optional voice and music timing\n\nIt should then produce:\n\n- a render manifest\n- renderer-specific instructions\n- final output artifacts\n\n## A Practical Product Reading\n\nA useful way to think about `action` is:\n\n- the runtime is the camera operator and automation system\n- the trace is the shooting log\n- the scenario is the shot list or repeatable script\n- the composer is post-production\n\nThat framing keeps the architecture honest.\n\n## Implication For Early Milestones\n\nThe first milestone should focus on capture and trace, not polished composition.\n\nThat is why the guided capture loop is the right first implementation target.\n\nOnce that exists, the system can support:\n\n- direct deterministic demos\n- manual guided recordings\n- trace-derived scenario drafts\n- polished exports later" }, { "id": "COMPUTER_USE_AGENT", "title": "COMPUTER_USE_AGENT", "content": "# Computer Use Agent \u2014 Action Spec\n\n## Context\n\n`action` today is a guided demo workstation: it stages scenes, records sessions, and produces artifacts for post-production. The agent runtime (`ActionAgentRuntime`) handles WebSocket transport and dispatches a small set of deterministic actions (click, type, press-key, drag, recording).\n\nA computer use agent requires a different loop: **observe \u2192 think \u2192 act \u2192 repeat** with autonomous goal pursuit, ongoing perception, and a much wider action surface.\n\nThis document describes the gap and the changes needed to close it.\n\n---\n\n## 1. Current State\n\n### What Exists\n\n| Layer | What | Limitation |\n|-------|------|------------|\n| **Actions** | `click`, `type`, `press-key`, `drag`, `start/stop-recording`, `show-cue`, `wait-for-condition` | No scroll, no context menu, no hotkey combos beyond simple modifiers |\n| **Target Resolution** | `semantic`, `accessibility`, `dom`, `textual`, `anchor`, `coordinate` | Resolution is one-shot; no ongoing observation |\n| **Perception** | Screenshots on demand, AX snapshots, recording | No continuous frame stream; perception is pull-only |\n| **Agent Loop** | None | Session is scripted/guided; no autonomous iteration |\n| **Memory** | Trace + artifact persistence only | No working memory, no context accumulation across steps |\n| **Tool Surface** | `CaptureEngine` interface | Not exposed as agent tools; no tool-call protocol |\n\n### What Is Missing for Computer Use\n\n- Continuous screen observation (video stream or periodic frames)\n- Rich element interaction (scroll, hover, context menu, double-click, text selection)\n- Autonomous goal decomposition and retry\n- Working memory and context carry-across\n- Mouse cursor control with smooth motion\n- Key sequences and system hotkeys\n- Clipboard access (read/write)\n- File system operations\n- Process and window management\n- Natural language instruction parsing\n- Self-correction on failure\n\n---\n\n## 2. Perception Layer\n\n### 2.1 Continuous Frame Stream\n\nToday: screenshots are captured on demand via `captureScreenshot`.\n\nComputer use requires: a periodic frame grab (e.g., every 1\u20132 seconds) that feeds the agent's vision input.\n\n**Changes:**\n\n- Add a `FrameStream` service in the native layer\n - Captures `CGImage` frames at a configurable interval (default: 1fps)\n - Pushes frames to the agent runtime via WebSocket or a shared buffer\n - Supports pause/resume so the agent only pays for frames when active\n- Add `startVisionStream()` / `stopVisionStream()` to `CaptureEngine`\n- Frames are small JPEG thumbnails at ~720p for token efficiency\n\n### 2.2 Accessibility Tree Persistence\n\nToday: AX snapshots are taken as discrete artifacts.\n\nComputer use requires: a live, queryable AX tree that the agent can re-query on every step without re-capturing the whole screen.\n\n**Changes:**\n\n- Add an `AccessibilityMonitor` that polls AX tree every ~500ms when active\n- Cache the tree in the agent runtime\n- Provide `queryAccessibleElement(query: TargetQuery)` method that hits the cache first\n- Fall back to a fresh AX snapshot only when the cache is stale (>2s) or query misses\n\n### 2.3 Cursor and Surface Tracking\n\n**Changes:**\n\n- Add `cursorPosition()` to report current mouse coordinates in real time\n- Add `activeWindowInfo()` to report frontmost app, window title, and bounds\n- These feed into the agent's observation payload on each step\n\n---\n\n## 3. Action Surface Expansion\n\n### 3.1 Interaction Actions\n\nAdd these to `ActionKind` in the protocol:\n\n| New Action | Description |\n|------------|-------------|\n| `scroll` | Scroll a target or the active surface by a delta |\n| `double-click` | Double-click at a point or on a target |\n| `right-click` | Context menu at a point or on a target |\n| `hover` | Move mouse to a point without clicking |\n| `text-select` | Select text in a text field or document |\n| `text-copy` | Copy selected text to clipboard |\n| `text-paste` | Paste clipboard contents |\n| `screenshot` | One-off screenshot |\n\n### 3.2 System Actions\n\n| New Action | Description |\n|------------|-------------|\n| `open-url` | Open a URL in the default browser |\n| `open-file` | Open a file with its default application |\n| `launch-app` | Launch an application by bundle ID or name |\n| `quit-app` | Quit an application |\n| `focus-app` | Bring an app's window to front |\n| `get-clipboard` | Read current clipboard contents |\n| `set-clipboard` | Write text to clipboard |\n| `run-shell-command` | Execute a shell command and return output |\n\n### 3.3 Interaction Execution\n\nThe existing `executeInteractionAction` in `packages/runtime/src/interaction/index.ts` handles the current action set via `runHost` calls to CLI commands. This pattern should be extended:\n\n```\nAction \u2192 InteractionExecutor \u2192 HostCLI\n \u2198 NativeAutomationBridge (for AX-dependent actions)\n```\n\nFor `scroll`, `double-click`, `right-click`, `hover`: new host CLI commands.\n\nFor `get-clipboard` / `set-clipboard`: native macOS `NSPasteboard` calls.\n\nFor `focus-app`, `launch-app`, `quit-app`: `NSRunningApplication` and `NSWorkspace`.\n\n### 3.4 Mouse Motion Path\n\nToday `click-point` snaps the cursor. For computer use with UI agents:\n\n- Add `move-to` command with optional duration for smooth motion\n- Use `CGEvent` posting with `kCGHIDEventTap` for smooth cursor movement\n- Configurable speed (fast for demos, slow for screen recording legibility)\n\n---\n\n## 4. Agent Loop\n\n### 4.1 Cycle Architecture\n\n```\nAgentRuntime\n \u2195 vision frames + AX tree\n Agent Loop (external or embedded)\n \u2195 tool calls\n CaptureEngine\n \u2195 AX + CGEvent\n Action.app / Native Layer\n```\n\nThe agent loop itself can live:\n- **Embedded** in `ActionAgentRuntime.swift` \u2014 simple, synchronous loop\n- **External** via WebSocket RPC \u2014 the agent runs in a separate process (safer isolation)\n\nOption B (external) is preferred: it keeps the native app lifecycle clean and allows the agent to be upgraded without rebuilding Action.app.\n\n### 4.2 Tool Call Protocol\n\nAdd a `tools.list` and `tools.call` RPC surface to the WebSocket interface:\n\n```typescript\n// Request: list tools\n{ \"method\": \"tools.list\" }\n\n// Response:\n{ \"tools\": [\n { \"name\": \"screenshot\", \"description\": \"...\", \"parameters\": {...} },\n { \"name\": \"click\", \"description\": \"...\", \"parameters\": {...} },\n ...\n]}\n\n// Request: call tool\n{ \"method\": \"tools.call\", \"params\": { \"name\": \"click\", \"args\": { \"target\": {...} } } }\n\n// Response:\n{ \"success\": true, \"result\": {...} }\n// or\n{ \"success\": false, \"error\": \"...\" }\n```\n\n### 4.3 Observation Payload\n\nOn each loop iteration, the agent receives:\n\n```typescript\ninterface AgentObservation {\n frame: string; // base64 JPEG thumbnail\n axTree: AXNode[]; // cached accessibility tree\n cursorPosition: Point;\n activeWindow: SurfaceRef;\n recentActions: ActionTrace[]; // last 5 actions + outcomes\n sessionHistory: ActionTrace[]; // full history\n}\n```\n\n### 4.4 Self-Correction\n\nWhen an action fails (AX element not found, app not responding):\n\n1. Increment failure counter on the step\n2. Retry with relaxed target (e.g., fall back to coordinate click)\n3. After 2 retries, mark step as `failed` and report to agent\n4. Agent decides whether to skip, substitute, or abort\n\n---\n\n## 5. Memory and Context\n\n### 5.1 Working Memory\n\nAdd a `MemoryStore` to the protocol:\n\n```typescript\ninterface MemoryEntry {\n key: string;\n value: string;\n ttlMs?: number;\n}\n\ninterface MemoryStore {\n set(entry: MemoryEntry): void;\n get(key: string): string | undefined;\n delete(key: string): void;\n clear(): void;\n}\n```\n\nUse cases:\n- Remember the last button clicked so the agent can say \"I clicked the submit button\"\n- Store intermediate task results (e.g., \"found the settings menu\")\n- Carry context across steps (e.g., \"we are in the Safari preferences pane\")\n\n### 5.2 Session Context Carry-Over\n\nCurrently each session is stateless. For computer use:\n\n- Add a `session.context: Record` field\n- The agent can write/read this across the session lifetime\n- Context is persisted to `manifest.json` so interrupted sessions can resume\n\n---\n\n## 6. Native Layer Changes\n\n### 6.1 ActionHostMain.swift\n\n- Add a `handleToolCall(method: String, args: [String: Any]) -> Any?` entry point\n- Route `tools.call` to the appropriate native handler or forward to `ActionAgentRuntime`\n\n### 6.2 AccessibilityMonitor\n\nNew Swift class that:\n- Creates an `AXUIElement` for the systemwide accessibility observer\n- Polls at a configurable interval\n- Returns a serialized AX tree as JSON\n\n### 6.3 FrameStreamService\n\nNew Swift class that:\n- Uses `CGDisplayStream` or periodic `CGWindowListCreateImage` to capture frames\n- Compresses to JPEG and pushes via a delegate or notification\n\n### 6.4 AutomationBridge\n\nRefactor `ActionAgentCommandBridge.swift` to expose a cleaner `AutomationBridge` interface:\n\n```swift\nprotocol AutomationBridge {\n func click(at point: CGPoint) async throws\n func type(text: String, delayMs: Int?) async throws\n func scroll(delta: CGPoint, in bounds: CGRect?) async throws\n func getClipboard() async throws -> String?\n func setClipboard(text: String) async throws\n func getCursorPosition() -> CGPoint\n func getFrontmostApp() async throws -> AppInfo\n func launchApp(bundleId: String) async throws\n func quitApp(bundleId: String) async throws\n}\n```\n\nThis protocol is implemented in `Action.app` (AppKit context) and called from the agent via XPC or direct Swift invocation.\n\n---\n\n## 7. Protocol Changes\n\n### 7.1 New `ActionKind` values\n\n```typescript\nexport type ActionKind =\n | \"click\"\n | \"type\"\n | \"press-key\"\n | \"focus-window\"\n | \"open-app\"\n | \"drag\"\n | \"start-recording\"\n | \"stop-recording\"\n | \"show-cue\"\n | \"wait-for-condition\"\n | \"scroll\" // NEW\n | \"double-click\" // NEW\n | \"right-click\" // NEW\n | \"hover\" // NEW\n | \"text-select\" // NEW\n | \"text-copy\" // NEW\n | \"text-paste\" // NEW\n | \"open-url\" // NEW\n | \"open-file\" // NEW\n | \"quit-app\" // NEW\n | \"get-clipboard\" // NEW\n | \"set-clipboard\" // NEW\n | \"run-shell-command\" // NEW\n | \"screenshot\"; // NEW (rename from captureScreenshot alias)\n```\n\n### 7.2 New `RuntimeAction.input` fields\n\n```typescript\n// scroll\n{ \"deltaX\": number, \"deltaY\": number, \"targetId\"?: string }\n\n// double-click / right-click\n{ \"point\"?: Point, \"targetId\"?: string }\n\n// hover\n{ \"point\": Point }\n\n// text-select\n{ \"start\": number, \"end\": number, \"targetId\": string }\n\n// open-url\n{ \"url\": string }\n\n// open-file\n{ \"path\": string }\n\n// quit-app\n{ \"bundleId\": string }\n\n// run-shell-command\n{ \"command\": string, \"timeoutMs\"?: number }\n\n// get-clipboard / set-clipboard\n{ \"text\": string } // for set-clipboard\n// get-clipboard has no input\n```\n\n### 7.3 New `SurfaceRef` kinds\n\n```typescript\n// Extend SurfaceRef\n{ id: string; kind: \"desktop\" | \"window\" | \"browser-tab\" | \"region\" | \"menu\" | \"dialog\"; label: string; bounds?: Bounds }\n```\n\n---\n\n## 8. Agent-External vs Agent-Embedded\n\nTwo integration strategies:\n\n### Option A: Embedded Agent Loop\n\nThe agent logic runs inside `ActionAgentRuntime.swift`, called via WebSocket messages from an external orchestrator.\n\n- **Pros**: Simple IPC, shared memory\n- **Cons**: Agent code is coupled to the app; hard to update agent independently\n\n### Option B: External Agent (Recommended)\n\nA separate agent process (Node.js, Python, or custom) connects to `ActionAgentRuntime` via WebSocket as a client. The agent sends `tools.call` messages and receives observation payloads.\n\n- **Pros**: Agent can be updated independently; agent can run on a different machine; cleaner isolation\n- **Cons**: Requires a stable WebSocket protocol with backpressure handling\n\n**Recommendation**: Start with Option B using a JSON-RPC-over-WebSocket protocol. Define it in `packages/protocol/src/agent.ts`:\n\n```typescript\nexport interface AgentMessage {\n jsonrpc: \"2.0\";\n id: string;\n method: string;\n params?: Record;\n}\n\nexport interface AgentNotification {\n jsonrpc: \"2.0\";\n method: string;\n params: Record;\n}\n```\n\nMethods:\n- `agent.observation` (notification from app \u2192 agent, pushed each loop tick)\n- `tools.list` / `tools.call` (request/response)\n- `memory.get` / `memory.set` / `memory.delete`\n- `session.finish`\n\n---\n\n## 9. Security Considerations\n\nComputer use implies autonomous system interaction. Key concerns:\n\n| Concern | Mitigation |\n|---------|------------|\n| Unintended file access | Sandboxed to `~/action-sessions/` output dir; no arbitrary file read |\n| Password / credential exposure | Never store secrets; agent sees only AX tree and screen frames |\n| Unbounded loop | Agent loop must report a maximum step count; session times out |\n| Click spam | Cooldown between actions (configurable, default 300ms) |\n| Permission escalation | `requestPermissions()` must be user-gesture initiated |\n\nThe `Action.app` sandbox profile should be updated to reflect these constraints. For now, this is a local-only tool \u2014 no network-exposed agent by default.\n\n---\n\n## 10. Verification\n\n### Smoke Tests for Computer Use Readiness\n\n```bash\n# Vision stream\n$ curl -X POST ws://localhost:9234 --json '{ \"method\": \"tools.call\", \"params\": { \"name\": \"startVisionStream\", \"args\": {} } }'\n$ # Should receive frame payloads every second\n\n# AX tree query\n$ curl -X POST ws://localhost:9234 --json '{ \"method\": \"tools.call\", \"params\": { \"name\": \"queryAccessible\", \"args\": { \"query\": { \"role\": \"AXButton\" } } } }'\n$ # Should return list of buttons\n\n# Continuous observation loop (5 steps)\n$ curl -X POST ws://localhost:9234 --json '{ \"method\": \"agent.startLoop\", \"params\": { \"goal\": \"Open Safari and go to example.com\", \"maxSteps\": 5 } }'\n$ # Should stream back 5 observation payloads with decreasing uncertainty\n```\n\n---\n\n## 11. Priority Order\n\n1. **Frame stream + AX cache** \u2014 foundation for all perception\n2. **Expanded action surface** (scroll, right-click, double-click, hover)\n3. **Tool call protocol over WebSocket** \u2014 makes the external agent possible\n4. **Clipboard + shell commands** \u2014 unlocks most practical workflows\n5. **Memory store** \u2014 enables context carry-over\n6. **External agent loop** \u2014 the actual computer use brain\n7. **Self-correction / retry** \u2014 closes the loop reliably\n8. **Smooth mouse motion** \u2014 better demo output\n\n---\n\n## 12. Open Questions\n\n- Should the agent runtime support **interrupt** mid-action (user says \"stop\")?\n- Should there be a **dry-run** mode where the agent describes what it would do without executing?\n- Do we want **multiple simultaneous agent sessions**? Today sessions are single.\n- What is the **authorization model** for shell commands? Any agent-requested shell is a risk.\n- Should we support a **headless variant** of Action.app for server-side computer use?" }, { "id": "LIVE_INSPECTION_RUNTIME", "title": "LIVE_INSPECTION_RUNTIME", "content": "# Live Inspection Runtime\n\n## Why This Matters\n\n`action` should not stop at predetermined demo recording.\n\nThe stronger product direction is:\n\n- inspect a live app, browser surface, or bounded region\n- send the current state to a model or analysis tool\n- get structured findings back\n- decide whether to act, record, annotate, or keep inspecting\n\nThat is useful for:\n\n- dev workflow capture\n- UI critique and QA\n- agent-assisted debugging\n- exploratory automation before a scenario exists\n\n## Keep\n\n- native capture truth in the runtime\n- AppKit-owned UI lifecycle in `Action.app`\n- the app/agent split\n- traces and artifacts as the durable session record\n\n## Avoid\n\n- baking a specific VLM vendor into the native engine\n- turning the app into a chat client with hidden state\n- vision-only action without observable targets or trace output\n- mixing post-capture review with live runtime control\n\n## Adapt\n\n- treat vision analysis as an inspection provider, not as the runtime itself\n- let the runtime own snapshots, context packaging, artifact storage, and action boundaries\n- let provider adapters own external API calls and response normalization\n\n## Decision\n\n`action` should gain a **live inspection mode** that sits beside guided capture.\n\nThe runtime loop becomes:\n\n`observe -> snapshot -> analyze -> inspect findings -> act or record -> trace`\n\n## Fit With Current Architecture\n\nThis direction matches the repo's existing split well:\n\n- `Action.app` owns native windows, overlays, permissions UX, and any always-on inspection HUD\n- `ActionAgent` owns transport, orchestration, provider calls, and session state\n- the native engine keeps providing screenshots, window bounds, AX data, and input primitives\n- review remains a consumer of saved inspection sessions, not the owner of live inspection\n\nThe existing design already points this way:\n\n- `observe` is a first-class tool family in [packages/mcp/src/index.ts](packages/mcp/src/index.ts)\n- the architecture document already centers `observe`, `resolve`, and `act`\n- there are already native inspection-style commands such as [ActionHostMain.swift](native/engine/Sources/ActionHostMain.swift#L2115)\n\n## New Product Primitive\n\nAdd a first-class primitive:\n\n### Inspection Session\n\nAn inspection session is a live runtime session where the main artifact is not only video.\n\nIt owns:\n\n- active surface or viewport\n- latest screenshot or sampled frame\n- optional AX/DOM context\n- model prompts and responses\n- findings\n- optional follow-up actions\n- trace and artifacts\n\nSuggested lifecycle:\n\n- `created`\n- `staging`\n- `observing`\n- `analyzing`\n- `awaiting-decision`\n- `acting`\n- `recording`\n- `completed`\n- `failed`\n- `cancelled`\n\n## Proposed Capability Split\n\n### Native Engine\n\nShould provide:\n\n- screenshot current viewport/window/screen\n- window and surface metadata\n- accessibility tree slices\n- optional region selection and overlay framing\n- deterministic input execution\n\nShould not provide:\n\n- direct Minimax or OpenAI client logic\n- prompt templating\n- provider-specific parsing\n\n### Agent Runtime\n\nShould provide:\n\n- inspection session creation\n- snapshot packaging\n- provider adapter selection\n- request/response logging\n- trace event emission\n- finding normalization\n- action gating\n\n### Provider Adapters\n\nShould provide:\n\n- external API invocation\n- auth and model selection\n- response normalization into `action` finding types\n\nExamples:\n\n- Minimax M2.7 adapter\n- OpenAI vision adapter\n- local CLI adapter for custom workflows\n\n## Minimal API Shape\n\nThe next useful surface is not \u201cchat with the app.\u201d\n\nIt is a small inspection API:\n\n- `session.inspect.create`\n- `observe.snapshot`\n- `observe.axTree`\n- `inspect.analyze`\n- `inspect.findings.latest`\n- `act.execute`\n- `record.start`\n- `record.stop`\n\n`inspect.analyze` should accept:\n\n- image artifact path\n- target surface metadata\n- optional AX payload\n- prompt or preset id\n- provider id\n\nIt should return structured findings, not only raw prose.\n\nSuggested normalized finding fields:\n\n- `id`\n- `source`\n- `summary`\n- `severity`\n- `kind`\n- `bounds`\n- `targetHint`\n- `evidence`\n- `recommendedAction`\n\n## First Increment To Actually Build\n\nDo not start with open-ended streaming vision.\n\nStart with one bounded loop:\n\n1. pick a target app window or viewport\n2. capture a screenshot plus AX snapshot\n3. send both to a provider adapter\n4. store the response as session artifacts\n5. show findings in Action\n6. optionally let the operator or agent run a follow-up action\n\nThat gets the core value without forcing us to solve:\n\n- continuous video analysis\n- autonomous acting\n- multi-provider prompt orchestration\n- generalized browser/DOM fusion\n\n## Recommended v0 Scope\n\nBuild **Inspect Current Surface** first.\n\nInputs:\n\n- current focused app or selected viewport\n- prompt preset such as `ui-critique`, `qa-pass`, or `interaction-audit`\n- provider selection\n\nArtifacts:\n\n- screenshot\n- optional AX snapshot JSON\n- provider request JSON\n- provider response JSON\n- normalized findings JSON\n- trace\n\nUI outcome:\n\n- findings list in the launcher review surface\n- \u201cRun action from finding\u201d later, not in v0\n\n## Why This Is Better Than \u201cJust Ask A VLM\u201d\n\nIf the model call lives outside the runtime, we lose the product.\n\nThe value of `action` is that it can:\n\n- create a stable snapshot\n- preserve inspection context\n- bind findings to a concrete surface and time\n- keep a trace of what was observed and what happened next\n- turn inspection into a reusable workflow instead of a one-off screenshot prompt\n\n## Suggested Next Build Order\n\n1. add inspection session types and artifacts to the runtime/protocol\n2. add a provider adapter interface in the agent/runtime layer\n3. implement `inspect current surface` using screenshot + AX snapshot\n4. render findings in the launcher review UI\n5. add optional \u201cact on finding\u201d follow-up commands\n\nFor the precise implementation checklist, see\n[RUNTIME_BUILD_CHECKLIST.md](docs/RUNTIME_BUILD_CHECKLIST.md)." }, { "id": "MILESTONE_GUIDED_CAPTURE_LOOP", "title": "MILESTONE_GUIDED_CAPTURE_LOOP", "content": "# Guided Capture Loop\n\n## Purpose\n\nThis is the first meaningful milestone for `action`.\n\nThe goal is not to finish the whole product. The goal is to prove that `action`\ncan run a polished, inspectable, native-first capture session end to end.\n\nIf this milestone works, we will have validated the most important live-system\nassumptions:\n\n- a session-oriented runtime\n- a useful operator HUD\n- a visible stage with a controlled viewport\n- real recording lifecycle control\n- deterministic actions against a native app\n- artifacts and logs that make the run inspectable afterward\n\n## Mission\n\nBuild a native macOS guided capture session where a user can:\n\n1. choose a backdrop and target app\n2. see a HUD that reflects session state and logs\n3. stage a window inside a viewport\n4. trigger a recording countdown\n5. record a deterministic Calculator demo\n6. save video, screenshot, and trace artifacts\n7. replay the last run\n\nThis milestone is intentionally live-phase heavy. It validates the runtime and\nengine before investing deeply in final-video composition.\n\n## Why This Is First\n\nThis milestone gives us the highest-value proof earliest.\n\n- It makes the system feel real immediately.\n- It tests the runtime where most failure risk lives.\n- It creates the traces and artifacts the composition layer will later consume.\n- It gives us a visible product loop instead of isolated backend pieces.\n\nIf this is weak, composition will not save the product. If this is solid,\ncomposition can evolve safely later.\n\n## Precedent Review Requirement\n\nBefore implementing each major part of this milestone, review how the same area\nworked in earlier iterations such as `vif` and `stage`.\n\nFor each of these feature areas, write down:\n\n- `Keep`\n- `Avoid`\n- `Adapt`\n- `Decision`\n\nApply that review to:\n\n- HUD\n- stage and viewport\n- countdown and recording state\n- Calculator scenario flow\n- replay and artifact presentation\n\nThe purpose is to preserve the strong product instincts from prior work without\nre-importing the old architectural problems.\n\n## Scope\n\n### Included\n\n- macOS-only session flow\n- native HUD window\n- backdrop and viewport framing\n- recording countdown\n- clear recording-state visuals\n- Calculator launch and focus\n- click, key press, and type support\n- session logs\n- screenshot capture\n- raw video capture\n- replay of the most recent run\n\n### Deferred\n\n- browser flows\n- Notes scenario\n- subtitle generation\n- chapter cards\n- voiceover and music timing\n- Remotion final export\n- multi-window stage layouts\n- vision-only target selection\n\n## Success Criteria\n\nThe milestone is successful when all of the following are true:\n\n- A user can start a guided session from a single control surface.\n- The HUD remains visible and useful during the full run.\n- The system can visibly count into recording.\n- The viewport clearly indicates what is being captured.\n- The area outside the viewport dims during countdown and recording.\n- Calculator can be launched, focused, and fit into the viewport.\n- The runtime can execute a short deterministic demo using both clicks and keys.\n- Every major state transition and action appears in the HUD log.\n- The session saves at least one video, one screenshot, and one trace file.\n- The most recent run can be replayed from the HUD.\n\n## Primary Demo Scenario\n\nCalculator is the first target because it is simple, native, and good at proving\nbasic interaction primitives.\n\n### Scenario\n\n1. Start a new session.\n2. Select a backdrop preset.\n3. Open Calculator.\n4. Fit Calculator into the viewport.\n5. Trigger a `3, 2, 1` countdown.\n6. Start recording.\n7. Execute a short expression using both keyboard and click input.\n8. Capture a screenshot during or after the interaction.\n9. Stop recording automatically or through the HUD.\n10. Save artifacts and offer replay.\n\n### Suggested Interaction Sequence\n\n- type `12`\n- click `+`\n- type `30`\n- press `=`\n\nThis is enough to prove:\n\n- target window selection\n- focus control\n- key input\n- pointer movement and click\n- action logging\n- artifact capture\n\n## User Experience Requirements\n\n### HUD\n\nThe HUD is a first-class product surface, not a debug panel.\n\nIt should show:\n\n- current session state\n- elapsed time or recording timer\n- current target app\n- control buttons\n- recent log events\n- recent artifacts\n\nControls for v1 of the HUD:\n\n- `Start`\n- `Pause`\n- `Stop`\n- `Replay Last Run`\n- `Quit`\n\nThe HUD should feel deliberate and presentable:\n\n- compact\n- readable\n- elegant\n- stable in position\n- visually distinct from the captured viewport\n\nThere may be two HUD layers over time:\n\n- a minimal native HUD for always-on capture state and immediate controls\n- a richer TS-side HUD for logs, diagnostics, artifacts, and operator workflows\n\nFor this milestone, the TS HUD is an acceptable first implementation surface as\nlong as runtime truth still lives below it.\n\n### Stage, Backdrop, and Viewport\n\nThe stage is the operator's framing surface.\n\nIt should support:\n\n- a chosen backdrop treatment\n- one primary viewport\n- clear distinction between on-stage content and surrounding space\n- window fitting that does not feel accidental\n\nThe viewport is the authoritative capture region for the live demo.\n\n### Recording State\n\nRecording must be unmistakable.\n\nThe system should provide:\n\n- visible countdown\n- dimming outside the viewport\n- explicit recording indicator\n- obvious transition from staging to recording\n\n## Architectural Implications\n\nThis milestone is mostly about three layers:\n\n### 1. Native Engine\n\nOwns:\n\n- window enumeration\n- window movement and sizing\n- ScreenCaptureKit session control\n- native overlays for countdown or viewport emphasis\n- input synthesis\n- app launch/focus integration\n\n### 2. Runtime\n\nOwns:\n\n- session creation\n- lifecycle transitions\n- command dispatch\n- action execution sequencing\n- trace collection\n- artifact registration\n- replay metadata\n\n### 3. HUD Frontend\n\nOwns:\n\n- user controls\n- session-state presentation\n- logs view\n- artifact list\n- replay trigger\n\nThe composer is not the main concern in this milestone. The live system is.\n\n## Session Lifecycle For This Milestone\n\nThe generic runtime lifecycle should be specialized into an operator-facing flow:\n\n- `created`\n- `staging`\n- `countdown`\n- `recording`\n- `paused`\n- `completing`\n- `completed`\n- `failed`\n- `cancelled`\n\nNotes:\n\n- `staging` means the target is being prepared and framed.\n- `countdown` is a visible pre-recording phase.\n- `recording` means capture is live.\n- `paused` should preserve enough session state to resume or stop cleanly.\n\nThe shared protocol can continue using the broader lifecycle, but the product\nsurface should expose this narrower guided-capture vocabulary.\n\n## Runtime Events\n\nThe run should emit structured events that are useful both for the HUD and for\nlater composition.\n\nMinimum event categories:\n\n- session state changed\n- backdrop selected\n- viewport updated\n- app launched\n- window focused\n- countdown tick\n- recording started\n- recording paused\n- recording stopped\n- action planned\n- action started\n- action completed\n- action failed\n- screenshot captured\n- video artifact finalized\n- replay requested\n\nEvery event should have:\n\n- timestamp\n- session id\n- type\n- human-readable summary\n- structured payload\n\n## Artifacts\n\nMinimum artifacts for this milestone:\n\n- raw screen capture video\n- screenshot\n- trace log\n- session metadata\n\nRecommended session folder layout:\n\n```text\nartifacts/sessions//\n session.json\n trace.json\n capture.mov\n screenshot-final.png\n```\n\n## Package Responsibilities\n\n### `packages/protocol`\n\n- session states\n- action and observation types\n- HUD event payloads\n- artifact metadata types\n\n### `packages/runtime`\n\n- session manager\n- lifecycle rules\n- trace storage\n- artifact registration\n- guided-session orchestration\n\n### `packages/cli`\n\n- local operator commands for development\n- session start/stop/replay wiring\n\n### `packages/composer-core`\n\n- not primary for this milestone\n- may define the future shape of render-manifest inputs\n\n### `native/engine`\n\n- native capture control\n- window fitting\n- overlays\n- app launch/focus\n- input synthesis\n\n## Delivery Plan\n\n### Phase 1: Session Skeleton\n\n- finalize milestone-specific lifecycle states\n- add runtime event stream for HUD consumption\n- define artifact directory layout\n\n### Phase 2: HUD Shell\n\n- create HUD window\n- add controls\n- add status area\n- add scrolling event log\n- add artifact list and replay control\n\n### Phase 3: Stage and Viewport\n\n- create backdrop surface\n- define viewport region\n- fit Calculator into viewport\n- add non-captured dimmed surroundings\n\n### Phase 4: Recording Control\n\n- implement countdown\n- implement start/stop recording\n- implement screenshot capture\n- save artifacts per session\n\n### Phase 5: Deterministic Calculator Demo\n\n- open Calculator\n- focus window\n- run click and keyboard actions\n- record action traces in real time\n\n### Phase 6: Replay\n\n- make last run discoverable\n- open or preview the latest capture from the HUD\n\n## Definition Of Done\n\nThis milestone is done when:\n\n- the full Calculator scenario runs from a single operator flow\n- the HUD stays informative and responsive throughout\n- recording state is visually clear\n- artifacts are saved to a predictable session folder\n- replay of the latest run works\n- the trace is useful enough to debug failures afterward\n\n## Follow-On Milestones\n\nIf this milestone lands well, the next likely milestones are:\n\n1. manual website capture with trace recording\n2. Notes scenario with richer typing and selection behavior\n3. trace-to-scenario compilation\n4. composer-driven polish from captured sessions\n\nThat sequence preserves the architectural idea that live capture creates the raw\ntruth, and composition upgrades presentation later." }, { "id": "PRECEDENT_HUD_STAGE", "title": "PRECEDENT_HUD_STAGE", "content": "# HUD And Stage Precedent Review\n\n## Feature\n\nHUD, backdrop, viewport, and recording-state presentation for the guided capture\nmilestone.\n\n## Keep\n\n- `vif`: the product feeling that the stage is visible, deliberate, and clearly\n in a recording mode when capture starts.\n- `vif`: viewport-first framing with backdrop dimming outside the captured\n region.\n- `vif`: a control surface that feels like part of the product, not a raw debug\n console.\n- `stage`: clearer separation between operator controls and stage primitives\n such as backdrop and viewport.\n\n## Avoid\n\n- `vif`: HUD and overlay behavior becoming tightly coupled to ad hoc runner\n logic.\n- `vif`: letting the UI invent state that diverges from runtime truth.\n- `stage`: keeping the stage too abstract or too tool-like to feel like a real\n product surface.\n\n## Adapt\n\n- Keep the richer TS-side operator HUD, but make it render the runtime session\n state directly.\n- Treat backdrop and viewport as first-class stage concepts, but avoid binding\n them to a specific renderer too early.\n- Use native overlays later for always-on capture-state presence, while the TS\n HUD remains the richer inspection and control surface.\n\n## Decision\n\n- For this milestone, the TS HUD will become stage-aware: it should visibly show\n backdrop, viewport, countdown, recording state, and artifacts.\n- The native side remains responsible for capture truth and will later gain the\n thin always-on stage overlay.\n- The product contract stays viewport-first: the stage shows what is captured,\n and the rest of the scene is visually subordinate.\n\n## Reference Weight\n\nBlend leaning `vif` for product feel and `stage` for boundaries." }, { "id": "PRECEDENT_REVIEW", "title": "PRECEDENT_REVIEW", "content": "# Precedent Review\n\n## Purpose\n\n`action` is a rewrite, not a fresh invention.\n\nThat means every meaningful feature should be implemented with awareness of prior\niterations such as `vif`, `stage`, and other reference products already named in\nthe vision docs.\n\nThe point is not to cargo-cult old code or old abstractions. The point is to\navoid relearning the same lessons and to preserve the parts that were genuinely\ngood.\n\n## Primary References\n\nThere are two primary implementation precedents for this repo:\n\n### `vif`\n\n`vif` is the more mature reference.\n\nUse it for:\n\n- richer product instincts\n- features that were already proven in practice\n- examples of end-to-end flows that felt real\n- areas where the UX had already become compelling\n\nTreat it carefully because it also carries more historical complexity:\n\n- broader product surface\n- accumulated lifecycle complications\n- architectural entanglement\n- features that worked but were hard to reason about internally\n\n### `stage`\n\n`stage` is the less mature reference, but often the cleaner conceptual one.\n\nUse it for:\n\n- stronger boundaries\n- cleaner framing of capabilities\n- improved instincts around agent-facing primitives\n- ideas that look more like a v2 architecture\n\nTreat it carefully because it is also less proven:\n\n- some concepts were still underdeveloped\n- some reliability questions were unresolved\n- some abstractions were cleaner than their implementation maturity\n\n## How To Read The Two Together\n\nWhen both references are relevant:\n\n- use `vif` to understand what felt good and what actually mattered in practice\n- use `stage` to understand how that idea wanted to be cleaned up architecturally\n- do not inherit `vif` complexity just because it is more complete\n- do not inherit `stage` abstractions just because they are cleaner on paper\n\nIn short:\n\n- `vif` is often the better product reference\n- `stage` is often the better architecture prompt\n- `action` should combine the strongest parts of both without importing their\n weaknesses\n\n## Additional Reference: Peekaboo\n\nPeekaboo is a strong native implementation reference for:\n\n- permission handling\n- Swift-first macOS automation ownership\n- CLI and agent surfaces over one underlying automation stack\n- separating visual feedback concerns from the lowest-level automation core\n\nIt is not the primary product precedent for `action`, but it is highly relevant\nwhen we are deciding how to structure:\n\n- native permissions\n- capture services\n- macOS automation boundaries\n- app/CLI/MCP sharing\n\nUse Peekaboo mainly as an implementation precedent for native host architecture,\nnot as a reason to broaden `action` beyond the guided demo mission.\n\n## Rule\n\nBefore implementing a substantial feature, do a short precedent review.\n\nA substantial feature includes things like:\n\n- HUD\n- stage and viewport\n- countdown and recording controls\n- target resolution\n- timeline/scenario format\n- overlays and cues\n- replay and artifact presentation\n- composition/export behavior\n\n## What A Precedent Review Should Ask\n\nFor the specific feature being built:\n\n1. What did `vif` do that was genuinely good?\n2. What did `stage` do that was genuinely good?\n3. What failed in those implementations?\n4. What should be preserved in spirit but re-implemented differently?\n5. What should be explicitly rejected this time?\n\nAlso ask:\n\n6. Is this feature better informed by `vif` as product precedent, `stage` as\n architecture precedent, or both?\n\nThe review does not need to be long. It does need to be explicit.\n\n## Expected Output\n\nFor each substantial feature, produce a short note before or during\nimplementation with these headings:\n\n### Keep\n\n- behaviors, UX ideas, or implementation instincts worth preserving\n\n### Avoid\n\n- failure modes, overreach, or known weak abstractions\n\n### Adapt\n\n- ideas that were good but need a different architecture in `action`\n\n### Decision\n\n- what this repo will actually build now\n\n### Reference Weight\n\n- whether this feature leans more on `vif`, `stage`, or a blend of both\n\n## Example: HUD\n\n### Keep\n\n- a persistent control surface\n- visible logs and state\n- polished feeling, not a debug-only panel\n- recording controls that are always easy to reach\n\n### Avoid\n\n- HUD logic owning runtime state directly\n- UI behavior diverging from runtime truth\n- messy control flow tied to CLI lifecycle hacks\n\n### Adapt\n\n- preserve the operator-first feeling\n- move state authority into the runtime session model\n- let the HUD render events instead of inventing its own truth\n\n### Decision\n\n- HUD is a thin frontend over guided-session events and controls\n\n## Working Agreement\n\nWhen implementing features in this repo:\n\n- review the relevant precedent first\n- write down the keep/avoid/adapt/declaration summary\n- then implement\n\nThis keeps the rewrite grounded without letting old architecture silently\ncontrol the new one." }, { "id": "REVIEW_UX_V1", "title": "REVIEW_UX_V1", "content": "# Review UX V1 (Agent Notes)\n\n## Goal\nTurn session review into a fast, reliable \"agent note\" workflow:\n\n1. seek quickly\n2. anchor feedback precisely (point/range/region)\n3. write actionable instruction\n4. save and revisit notes\n\nThis is not a social commenting UI. It is a structured note system for an AI agent.\n\n## Interaction Model\n\n### 1) Single Composer\n- One explicit composer entrypoint: `+ Feedback`\n- Composer has anchor mode tabs:\n - `Point`: click timeline to stamp a single time\n - `Range`: drag timeline to set in/out\n - `Region`: drag on video frame to select area\n- Composer always shows current anchor summary and short mode instructions.\n\n### 2) Timeline As Primary Control\n- Timeline must always support click and drag seek.\n- Anchor creation overlays on top of seek behavior, not instead of it.\n- In range mode, drag previews the range before commit.\n- Timeline markers represent saved notes and jump playback on click.\n\n### 3) Frame Region Selection\n- Region mode is explicit and visible.\n- Drag on frame creates a highlighted area with clear border.\n- Region remains visible after selection and after save when focused.\n\n### 4) Notes Rail\n- Composer and saved notes live in a dedicated \"notes rail\".\n- Saved notes are scrollable and selectable.\n- Selecting a saved note focuses timeline + playback + region context.\n\n## State Rules\n\n- `isComposingFeedback` gates anchor creation.\n- `anchorMode` controls source of anchor input:\n - timeline click for point\n - timeline drag for range\n - frame drag for region\n- `Save` requires non-empty instruction.\n- `Clear Anchors` only clears draft anchors, not saved notes.\n\n## Quality Bar\n\n- No hidden interaction modes.\n- Every click/drag should produce immediate visual response.\n- No ambiguous labels like \"marking\" without visible result.\n- Primary actions should be discoverable without documentation.\n\n## Follow-up (V1.1)\n\n- Keyboard controls:\n - `N` open composer\n - `1/2/3` switch anchor mode\n - `Esc` cancel region mode\n - `Cmd+Enter` save note\n- Inline marker hover previews.\n- Optional snap-to-nearest-marker and frame-accurate stepping." }, { "id": "RUNTIME_BUILD_CHECKLIST", "title": "RUNTIME_BUILD_CHECKLIST", "content": "# Runtime Build Checklist\n\nThis document is the practical implementation list for turning `action` into a\nruntime that can observe, act, reflect, and record.\n\nIt is intentionally precise. Items here should be implementable as concrete\ntasks, not aspirational themes.\n\n## Product Contract\n\nWhen this roadmap lands, `action` should support this loop:\n\n1. select a live surface\n2. inspect the current state\n3. run reflection against that state\n4. decide whether to act, record, annotate, or stop\n5. preserve a full trace and artifacts\n\n## Workstream 1: Shared Runtime Model\n\n### 1.1 Session Vocabulary\n\n- [ ] Decide whether to keep one session type with modes, or separate `capture`\n and `inspection` session types.\n- [ ] Add a stable session mode enum such as `capture`, `inspection`, `hybrid`.\n- [ ] Define reflection-specific phases and make them canonical across TS and\n Swift:\n - [ ] `observing`\n - [ ] `analyzing`\n - [ ] `awaiting-decision`\n - [ ] `acting`\n- [ ] Define phase transition rules so invalid jumps are impossible.\n- [ ] Define which phases allow recording, acting, replay, or dismissal.\n\n### 1.2 Trace Model\n\n- [ ] Add trace events for:\n - [ ] snapshot captured\n - [ ] AX snapshot captured\n - [ ] inspection requested\n - [ ] inspection response received\n - [ ] finding recorded\n - [ ] finding accepted or dismissed\n - [ ] follow-up action started\n - [ ] follow-up action completed\n - [ ] follow-up action failed\n- [ ] Record control-origin events:\n - [ ] HUD button pressed\n - [ ] CLI command issued\n - [ ] MCP tool request issued\n- [ ] Record provider metadata in the trace without leaking secrets.\n\n### 1.3 Artifact Model\n\n- [ ] Finalize artifact kinds for:\n - [ ] screenshot\n - [ ] raw capture\n - [ ] trace\n - [ ] AX snapshot\n - [ ] inspection request\n - [ ] inspection response\n - [ ] normalized findings\n- [ ] Standardize per-session artifact paths.\n- [ ] Add a session manifest file listing all artifacts and their roles.\n- [ ] Add versioning for saved findings and inspection result formats.\n\n## Workstream 2: Native Engine Capabilities\n\n### 2.1 Surface Discovery\n\n- [ ] Add a native API to enumerate observable surfaces:\n - [ ] running apps\n - [ ] top-level windows\n - [ ] active browser window if applicable\n - [ ] bounded region selection\n- [ ] Add a stable surface identifier format.\n- [ ] Add a native command to return the currently focused surface.\n\n### 2.2 Snapshot Capture\n\n- [ ] Add a native command to capture the current focused window screenshot.\n- [ ] Add a native command to capture a selected viewport screenshot.\n- [ ] Add a native command to capture a full screen screenshot without starting\n a recording session.\n- [ ] Add timestamped, deterministic snapshot naming.\n- [ ] Add geometry metadata to every snapshot artifact.\n\n### 2.3 Accessibility Inspection\n\n- [ ] Generalize the existing Calculator-only inspection commands into\n app-agnostic AX inspection.\n- [ ] Add a native command to fetch an AX tree slice for a bundle id or surface\n id.\n- [ ] Add a native command to fetch a focused AX element snapshot.\n- [ ] Add bounds and role metadata normalization for AX nodes.\n- [ ] Add size limits and pruning rules so AX payloads stay usable for model\n calls.\n\n### 2.4 Live Overlay And Controls\n\n- [ ] Keep an always-available native control surface whenever a live session is\n on screen.\n- [ ] Make controls phase-aware for:\n - [ ] inspect\n - [ ] apply\n - [ ] start recording\n - [ ] interrupt\n - [ ] cancel\n - [ ] dismiss\n- [ ] Add a visible reflecting or analyzing state in the native HUD.\n- [ ] Add explicit control affordances for non-recording live sessions.\n- [ ] Add an operator-safe way to stop a session that is waiting on external\n analysis.\n\n### 2.5 Input And Acting\n\n- [ ] Keep deterministic input primitives as the only execution path.\n- [ ] Add generic click, type, keypress, drag, and focus commands to the\n runtime-facing API where missing.\n- [ ] Add optional screenshot-before and screenshot-after hooks for actions.\n- [ ] Add a way to bind a follow-up action to a finding target hint.\n\n## Workstream 3: Agent Runtime Core\n\n### 3.1 Session Registry\n\n- [ ] Add a session registry in the agent/runtime layer.\n- [ ] Track active session state in one place instead of distributing it across\n UI-only paths.\n- [ ] Add explicit resource cleanup per session:\n - [ ] overlay stop files\n - [ ] recording stop files\n - [ ] child processes\n - [ ] temporary inspection request files\n\n### 3.2 Snapshot Packaging\n\n- [ ] Build a snapshot packaging service that can assemble:\n - [ ] screenshot path\n - [ ] viewport metadata\n - [ ] AX snapshot path\n - [ ] surface metadata\n - [ ] runtime notes\n- [ ] Add payload size budgeting before a provider call.\n- [ ] Add redaction hooks for sensitive text or window titles later.\n\n### 3.3 Reflection Orchestration\n\n- [ ] Add an inspection service in the runtime layer.\n- [ ] It should:\n - [ ] create inspection requests\n - [ ] invoke a provider adapter\n - [ ] store raw request and response artifacts\n - [ ] normalize findings\n - [ ] emit trace events\n - [ ] transition the session between `observing`, `analyzing`, and\n `awaiting-decision`\n- [ ] Add timeout and cancellation handling for provider calls.\n- [ ] Add retry policy rules that avoid duplicate artifact corruption.\n\n### 3.4 Action Gating\n\n- [ ] Require an explicit decision point before executing a model-suggested\n action.\n- [ ] Add a runtime policy switch:\n - [ ] operator approval required\n - [ ] agent approval required\n - [ ] auto-apply allowed\n- [ ] Log the decision source in the trace.\n\n## Workstream 4: Provider Adapter Layer\n\n### 4.1 Adapter Interface\n\n- [ ] Add a provider adapter interface in the runtime layer.\n- [ ] Standardize adapter input:\n - [ ] prompt preset id\n - [ ] freeform prompt\n - [ ] image path\n - [ ] AX context path\n - [ ] surface metadata\n- [ ] Standardize adapter output:\n - [ ] summary\n - [ ] findings\n - [ ] optional recommended actions\n - [ ] raw response reference\n\n### 4.2 Minimax Adapter\n\n- [ ] Add a concrete Minimax M2.7 adapter.\n- [ ] Read credentials from environment or config, not from the native app.\n- [ ] Save the exact request payload sent to Minimax as an artifact.\n- [ ] Save the raw response as an artifact.\n- [ ] Normalize the response into the internal findings format.\n\n### 4.3 Prompt Presets\n\n- [ ] Define a preset registry for:\n - [ ] `ui-critique`\n - [ ] `qa-pass`\n - [ ] `interaction-audit`\n - [ ] `target-suggestion`\n- [ ] Keep preset definitions in a single runtime-owned location.\n- [ ] Add versioning so findings can be traced back to a prompt revision.\n\n## Workstream 5: Frontend Surfaces\n\n### 5.1 Launcher\n\n- [ ] Add a top-level inspection workflow in `Action.app`.\n- [ ] Add controls to:\n - [ ] inspect current surface\n - [ ] inspect selected app\n - [ ] inspect selected viewport\n - [ ] start recording from current session\n- [ ] Add a findings list view.\n- [ ] Add a raw context view:\n - [ ] screenshot preview\n - [ ] AX summary\n - [ ] provider summary\n- [ ] Add a decision bar for:\n - [ ] apply suggested action\n - [ ] ignore finding\n - [ ] record next step\n - [ ] rerun inspection\n\n### 5.2 Native HUD\n\n- [ ] Keep HUD controls reachable during all live phases.\n- [ ] Show when the runtime is:\n - [ ] observing\n - [ ] reflecting\n - [ ] waiting for a decision\n - [ ] acting\n - [ ] recording\n- [ ] Make sure the HUD never shows a disabled dead-end state without a visible\n escape path.\n\n### 5.3 TS HUD\n\n- [ ] Add inspection session rendering to the TS HUD.\n- [ ] Show latest findings inline with the current screenshot.\n- [ ] Show trace events for reflection and action gating.\n- [ ] Add artifact previews for inspection request, response, and findings JSON.\n\n## Workstream 6: Agent-Facing Interfaces\n\n### 6.1 CLI\n\n- [ ] Add CLI commands for:\n - [ ] `inspect current-surface`\n - [ ] `inspect app --bundle-id ...`\n - [ ] `inspect viewport --x ... --y ... --width ... --height ...`\n - [ ] `inspect findings`\n - [ ] `session act`\n - [ ] `session record-start`\n - [ ] `session record-stop`\n- [ ] Make CLI output distinguish startup acknowledgement from completed\n inspection or recording.\n\n### 6.2 MCP\n\n- [ ] Add MCP tools for:\n - [ ] create inspection session\n - [ ] capture snapshot\n - [ ] request analysis\n - [ ] list findings\n - [ ] execute action\n - [ ] start recording\n - [ ] stop recording\n- [ ] Make MCP tools return session ids and artifact references, not only prose.\n\n### 6.3 WebSocket Agent Protocol\n\n- [ ] Expand the agent protocol beyond the current narrow native command set.\n- [ ] Add methods for:\n - [ ] session creation\n - [ ] surface enumeration\n - [ ] snapshot capture\n - [ ] AX inspection\n - [ ] provider-backed analysis\n - [ ] findings retrieval\n - [ ] gated action execution\n\n## Workstream 7: Session Storage And Review\n\n### 7.1 Session Layout\n\n- [ ] Standardize a session directory layout that works for both capture and\n inspection sessions.\n- [ ] Add `session.json` with:\n - [ ] mode\n - [ ] lifecycle\n - [ ] target surface\n - [ ] provider info\n - [ ] key artifact paths\n\n### 7.2 Findings Persistence\n\n- [ ] Add `findings.json` as a first-class saved artifact.\n- [ ] Add support for finding status:\n - [ ] open\n - [ ] accepted\n - [ ] dismissed\n - [ ] acted-on\n- [ ] Add finding-to-action linkage so review can explain what happened next.\n\n### 7.3 Review UI\n\n- [ ] Extend the existing review surface to load inspection sessions, not only\n demo captures.\n- [ ] Show inspection summary cards beside recordings in the library.\n- [ ] Let a user reopen a session and see:\n - [ ] original snapshot\n - [ ] findings\n - [ ] follow-up actions\n - [ ] whether recording happened later\n\n## Workstream 8: Reliability And Safety\n\n### 8.1 Cancellation\n\n- [ ] Ensure every live phase has a tested cancel path.\n- [ ] Ensure cancellation writes a final session status and does not strand\n child processes.\n- [ ] Ensure provider timeouts do not leave the UI in `analyzing` forever.\n\n### 8.2 Error Handling\n\n- [ ] Normalize native errors, provider errors, and runtime errors into one\n session-visible format.\n- [ ] Record failures as trace events and artifacts where useful.\n- [ ] Add user-visible recovery guidance for common failure classes.\n\n### 8.3 Permissions\n\n- [ ] Make inspection flows clearly report when Accessibility or Screen\n Recording permissions are missing.\n- [ ] Distinguish which operations need which permission.\n\n## Workstream 9: Testing\n\n### 9.1 Protocol And Runtime Tests\n\n- [ ] Add TypeScript tests for:\n - [ ] session phase transitions\n - [ ] finding normalization\n - [ ] artifact manifests\n - [ ] provider adapter contracts\n\n### 9.2 Native Tests\n\n- [ ] Add smoke tests for:\n - [ ] current surface screenshot\n - [ ] AX snapshot fetch\n - [ ] live HUD controls during inspect phases\n - [ ] inspection cancellation\n - [ ] recording start from an inspection session\n\n### 9.3 End-To-End Tests\n\n- [ ] Add one end-to-end inspection path:\n - [ ] stage Calculator or another stable app\n - [ ] capture screenshot\n - [ ] capture AX snapshot\n - [ ] run mock provider analysis\n - [ ] persist findings\n - [ ] apply one follow-up action\n - [ ] optionally record the result\n\n## Recommended Build Order\n\nThis is the order that keeps the architecture practical.\n\n### Phase A: Foundation\n\n- [ ] finalize session mode and phase model\n- [ ] finalize artifact and trace schema\n- [ ] add session registry in the runtime\n\n### Phase B: Native Observe\n\n- [ ] implement surface enumeration\n- [ ] implement focused surface screenshot\n- [ ] implement generic AX snapshot\n- [ ] keep HUD controls live during non-recording sessions\n\n### Phase C: Reflection\n\n- [ ] add provider adapter interface\n- [ ] add Minimax adapter\n- [ ] add prompt preset registry\n- [ ] add findings persistence\n\n### Phase D: Decision And Action\n\n- [ ] add action gating\n- [ ] add finding-to-action execution\n- [ ] add screenshot before and after action hooks\n\n### Phase E: Record And Review\n\n- [ ] start recording from an inspection session\n- [ ] persist hybrid inspection-plus-recording sessions\n- [ ] render findings in the launcher and TS HUD\n\n## Done Criteria For The Vision\n\nWe are meaningfully on the intended path when all of the following are true:\n\n- [ ] a live app surface can be inspected without starting a demo scenario\n- [ ] a model-backed reflection step can run on the captured state\n- [ ] findings are stored as first-class session artifacts\n- [ ] a user or agent can choose whether to act on a finding\n- [ ] a recording can begin from the same live runtime session\n- [ ] the full loop is preserved as trace plus artifacts" }, { "id": "SURFACE_ADAPTER_PLAN", "title": "SURFACE_ADAPTER_PLAN", "content": "# Surface Adapter Plan\n\n> Formalized as [ACT-001: Surface Adapter Architecture](decisions/ACT-001-surface-adapter-architecture.md).\n> Keep ACT-001 as the source of truth for the decision/proposal record.\n\n## Principle\n\nEvery surface starts with the macOS accessibility tree.\n\nFirst-class surfaces get complementary semantic channels. Those channels should\nmake Action faster and more precise, but they should not replace AX as the\nbaseline source of UI truth.\n\nIn practice:\n\n- AX tells Action what app/window/control exists in the OS.\n- Native capture tells Action what is visible.\n- Adapter-specific channels tell Action what the app means internally.\n- Vision verifies visual outcomes when AX or semantic state is incomplete.\n\n## Runtime Shape\n\nAction should expose one runtime contract and many adapters.\n\n```ts\ninterface SurfaceAdapter {\n id: string;\n label: string;\n priority: number;\n\n canHandle(surface: SurfaceRef): Promise;\n observe(context: ObserveContext): Promise;\n resolve(query: TargetQuery, observation: SurfaceObservation): Promise;\n act(action: RuntimeAction, target: TargetCandidate): Promise;\n extract(query: ExtractionQuery, observation: SurfaceObservation): Promise;\n captureHints(target: TargetCandidate): Promise;\n verify(result: ActionResult, context: VerifyContext): Promise;\n}\n```\n\nThe shared observation envelope should look roughly like this:\n\n```ts\ninterface SurfaceObservation {\n surface: SurfaceRef;\n ax: AXSnapshot;\n native: NativeWindowState;\n semantic?: ChromeDomState | BrowserPageState | TerminalState | EditorState | SiteState;\n vision?: ScreenshotReflection;\n freshness: {\n axCapturedAt: string;\n semanticCapturedAt?: string;\n screenshotCapturedAt?: string;\n };\n}\n```\n\nThe target resolver should produce candidates with evidence, not just a point.\n\n```ts\ninterface TargetCandidate {\n id: string;\n label: string;\n role?: string;\n rect?: Rect;\n confidence: number;\n stabilityKey?: string;\n evidence: TargetEvidence[];\n preferredActionChannel: \"ax\" | \"dom\" | \"tmux\" | \"editor\" | \"native\" | \"hid\";\n fallbackChannels: string[];\n}\n```\n\n## Action Policy\n\nThe adapter should choose the safest available channel for each action.\n\nRecommended order:\n\n1. App-native semantic action, such as DOM value set, tmux send-keys, or editor command.\n2. AX action, such as `AXPress`, `AXValue`, `AXSelectedText`, or settable focused value.\n3. Process-directed key or text event.\n4. HID event that may move focus or the real pointer.\n\nWhen an action may disturb the user's current focus, Action should warn through\nthe overlay before doing it.\n\nPost-action verification should use a different channel when possible. For\nexample, a DOM set should be verified by DOM state plus AX or screenshot.\n\n## Adapter Families\n\n## 1. Browser App Adapters\n\nBrowsers are first-class because a large share of user work happens inside\nweb apps, and AX alone is too lossy inside complex pages.\n\n### Chrome\n\nChannels:\n\n- AX snapshot for window, omnibox, tabs, web area, and native Chrome controls.\n- Chrome extension for DOM observe/act/extract.\n- Optional Chrome DevTools Protocol for development and debugging flows.\n- Native capture for screenshots, previews, and recording.\n\nCore abilities:\n\n- Inspect current tab URL/title/loading state.\n- Return visible DOM nodes with roles, labels, text, rects, selectors, and image URLs.\n- Resolve form fields, buttons, links, menus, image cards, and content regions.\n- Set input and textarea values with browser events.\n- Click page-local elements without moving the OS pointer.\n- Return semantic crop hints for top-right preview or recording focus.\n\n### Safari\n\nChannels:\n\n- AX snapshot for window, toolbar, web area, form controls, and page content.\n- Safari extension later if the product needs Chrome-like DOM fidelity.\n- AppleScript or Safari automation only where reliable.\n- Native capture.\n\nCore abilities:\n\n- Browser-window awareness.\n- URL/title extraction through AX or scripting.\n- Basic target resolution and form interaction.\n- Conservative fallbacks when Safari does not expose stable internals.\n\n### Firefox\n\nChannels:\n\n- AX snapshot for browser shell and web area.\n- Firefox extension for DOM observe/act/extract if we want parity with Chrome.\n- Native capture.\n\nCore abilities:\n\n- Same conceptual model as Chrome, with a smaller initial scope.\n- Good enough for users who do not want Chrome.\n\n## 2. Browser Site Adapters And Recipes\n\nSite adapters live inside the browser extension world. They should not be\nseparate native apps. They specialize the generic browser adapter for high-volume\nweb products.\n\nNot every product deserves a bundled first-class adapter. The default path\nshould be:\n\n1. generic browser adapter\n2. user-authored or bundled browser recipe\n3. first-class site adapter only when usage volume and workflow depth justify it\n\n### GitHub\n\nWhy:\n\n- Pull requests, issues, reviews, checks, diffs, and file browsing are common\nagent workflows.\n\nExtra semantics:\n\n- PR title/status/checks/review state.\n- File tree and diff hunks.\n- Comment boxes and review submission controls.\n- Repo/branch/commit context.\n\n### Google Search And Docs-Like Surfaces\n\nWhy:\n\n- Search, docs, sheets, and web forms appear in many demos.\n\nExtra semantics:\n\n- Search box and result list.\n- Document title and editable region.\n- Save/sync state when exposed.\n- Selection and visible cursor context where possible.\n\n### Midjourney Recipe\n\nWhy:\n\n- It is a strong demo and dogfood target for background generation, visual\npreview, and rendered result monitoring.\n- It is probably too specific for the initial first-class adapter set.\n\nRecipe semantics:\n\n- Create prompt composer target.\n- Submit control target.\n- Job/result card targets.\n- Prompt text associated with result cards.\n- Image URL and crop hint extraction.\n\nThis should start as a bundled recipe on top of the Chrome companion extension,\nnot as a bespoke core adapter.\n\n## 3. Terminal Adapters\n\nTerminals are first-class because the user base lives in shells, and terminal\nAX output is often weaker than the shell's own state.\n\n### Terminal Base Adapter\n\nChannels:\n\n- AX snapshot for app/window/pane bounds.\n- Native capture for terminal screenshots.\n- Optional shell integration markers.\n- Process/window metadata.\n\nCore abilities:\n\n- Identify terminal app, window title, and visible text region.\n- Capture a visible pane region.\n- Warn before focus-taking input.\n- Detect when only HID/focus-based typing is available.\n\n### tmux\n\nChannels:\n\n- `tmux list-sessions`, `list-windows`, `list-panes`.\n- `tmux capture-pane`.\n- `tmux send-keys`.\n- `tmux display-message` for current command, cwd, and pane metadata.\n- AX/native capture for the terminal window that hosts the pane.\n\nCore abilities:\n\n- Resolve a pane by session/window/pane id, title, cwd, or visible text.\n- Send keys without using the OS pointer.\n- Extract command output reliably.\n- Know whether a command exited and with what status when shell integration is available.\n\n### iTerm\n\nChannels:\n\n- AX snapshot for windows, tabs, split panes, and text areas.\n- iTerm scripting API where available.\n- tmux adapter when iTerm hosts tmux.\n- Native capture.\n\nCore abilities:\n\n- Resolve window/tab/session.\n- Read visible text where possible.\n- Send text/keys through the safest available path.\n- Capture exact pane bounds.\n\n### Ghostty\n\nChannels:\n\n- AX snapshot and native capture.\n- tmux/shell integration when present.\n- App-specific hooks later if Ghostty exposes stable automation APIs.\n\nCore abilities:\n\n- Window/pane bounds.\n- Visible terminal text via AX/capture.\n- tmux-backed control when applicable.\n\n### Warp\n\nChannels:\n\n- AX snapshot and native capture.\n- Block-aware semantics if Warp exposes them through AX or local APIs.\n- Shell integration where possible.\n\nCore abilities:\n\n- Resolve command blocks.\n- Extract visible block output.\n- Warn when input requires focus.\n\n## 4. Coding Surface Adapters\n\nCoding apps are first-class because Action's users spend most of their time in\neditors and agent consoles.\n\n### Cursor\n\nChannels:\n\n- AX snapshot for editor shell, command palette, sidebars, terminal panels, and chat.\n- VS Code-compatible extension channel if available.\n- Filesystem/workspace state.\n- Native capture.\n\nCore abilities:\n\n- Identify active workspace, file, selection, editor region, terminal panel, and chat panel.\n- Run editor commands through extension APIs where possible.\n- Capture focused code regions.\n- Avoid stealing focus unless the requested action truly requires it.\n\n### VS Code\n\nChannels:\n\n- AX snapshot.\n- VS Code extension API.\n- Filesystem/workspace state.\n- Native capture.\n\nCore abilities:\n\n- Active file/selection/diagnostics.\n- Command palette actions.\n- Terminal panel state.\n- Extension-mediated text edits and navigation.\n\n### Codex\n\nChannels:\n\n- AX snapshot for the desktop app UI.\n- Local workspace and thread context when available.\n- Native capture and overlay.\n\nCore abilities:\n\n- Understand chat/input/output regions.\n- Track running commands and visible logs where exposed.\n- Capture meaningful preview regions.\n- Warn before actions that may interfere with the user's current Codex focus.\n\n### Conductor\n\nChannels:\n\n- AX snapshot.\n- Local app/project metadata if exposed.\n- Native capture.\n\nCore abilities:\n\n- Identify active project/session.\n- Resolve chat, task, and result regions.\n- Capture agent progress and output state.\n\n## 5. System And Long-Tail Adapters\n\n### Finder And File Pickers\n\nWhy:\n\n- Files, save dialogs, open dialogs, downloads, and permissions are frequent\nautomation edges.\n\nChannels:\n\n- AX snapshot.\n- Filesystem APIs.\n- Native capture.\n\nCore abilities:\n\n- Resolve file picker fields and buttons.\n- Navigate known folders.\n- Select files without fragile coordinate clicking when possible.\n\n### System Dialogs And Permissions\n\nWhy:\n\n- macOS permission dialogs and settings are unavoidable for capture and\nautomation products.\n\nChannels:\n\n- AX snapshot.\n- System settings URLs.\n- Native capture.\n\nCore abilities:\n\n- Detect permission state.\n- Open correct settings pane.\n- Explain when manual user action is required.\n\n### Generic AX Adapter\n\nThis is the long-tail fallback for every other app.\n\nChannels:\n\n- AX snapshot.\n- Native capture.\n- Optional vision reflection.\n\nCore abilities:\n\n- Inspect app/window/control tree.\n- Resolve buttons, text fields, menus, lists, and scroll areas.\n- Prefer AX actions over coordinates.\n- Use coordinate/HID fallback only with explicit risk labeling.\n\n## 6. User-Authored Adapters\n\nThe adapter system should let users teach Action their own high-volume surfaces\nwithout waiting for a bundled adapter release.\n\nThere should be three authoring levels.\n\n### Level 1: Recipes\n\nRecipes are declarative target and extraction definitions. They should be enough\nfor most site-specific and app-specific customizations.\n\nGood fits:\n\n- internal web apps\n- niche SaaS tools\n- personal dashboards\n- demo targets like Midjourney\n- simple editor or terminal workflows that mostly need better labels and crops\n\nExample shape:\n\n```json\n{\n \"id\": \"user.midjourney.create\",\n \"label\": \"Midjourney Create\",\n \"surface\": {\n \"kind\": \"browser-page\",\n \"urlMatches\": [\"https://www.midjourney.com/imagine*\"]\n },\n \"targets\": {\n \"prompt\": {\n \"dom\": { \"role\": \"textbox\", \"text\": \"What will you imagine?\" },\n \"ax\": { \"role\": \"AXTextArea\" }\n },\n \"latestResult\": {\n \"dom\": { \"nearTextTarget\": \"prompt\", \"image\": true },\n \"capture\": { \"padding\": 32, \"preferAspectRatio\": \"16:9\" }\n }\n },\n \"actions\": {\n \"submitPrompt\": [\n { \"setValue\": \"prompt\" },\n { \"key\": \"Enter\" },\n { \"waitFor\": \"latestResult\" }\n ]\n }\n}\n```\n\nRecipes should be inspectable, exportable, and safe by default. They can define\ntargets, fallback selectors, capture hints, waits, and verification checks. They\nshould not run arbitrary shell commands.\n\n### Level 2: TypeScript Adapter Modules\n\nTypeScript adapters implement a constrained subset of the adapter contract for\nusers who need logic, not just selectors.\n\nGood fits:\n\n- teams with complex internal tools\n- workflow products with stable local APIs\n- browser apps where extracting state requires page-specific logic\n\nThey should run in a sandboxed runtime with explicit capabilities:\n\n- browser DOM access\n- AX snapshot read access\n- capture hint generation\n- local filesystem read access only when granted\n- no native input or shell by default\n\n### Level 3: Native Capability Adapters\n\nNative capability adapters are for integrations that need privileged local\nbehavior, such as deep terminal or editor control.\n\nThese should usually be bundled, reviewed, or explicitly installed because they\ncan affect the user's OS session.\n\nGood fits:\n\n- tmux\n- terminal hosts\n- editor extension bridges\n- permission/settings flows\n\n### Sharing Model\n\nAdapters and recipes should be packageable as small packs:\n\n- local user recipes\n- team-shared packs\n- bundled Action packs\n- optional community packs later\n\nEach pack should declare:\n\n- surfaces matched\n- capabilities requested\n- actions it can perform\n- whether actions are background-safe\n- whether actions may steal focus\n- verification strategy\n\n## Initial Adapter Set\n\nThe first shippable set should be roughly 15 to 18 adapters:\n\n1. Generic AX\n2. Chrome\n3. Chrome extension generic web\n4. GitHub site adapter\n5. Google search/docs-like site adapter\n6. Safari\n7. Firefox\n8. Terminal base\n9. tmux\n10. iTerm\n11. Ghostty\n12. Warp\n13. Cursor\n14. VS Code\n15. Codex\n16. Conductor\n17. Finder/file pickers\n18. System dialogs/permissions\n\nBundled recipes can cover specific demo or workflow targets without promoting\nthem into core adapters. Midjourney is a good first bundled recipe because it\nexercises browser DOM resolution, background action, result monitoring, image\nextraction, and semantic preview crops.\n\nThis is enough to cover the highest-volume places while preserving a credible\nlong-tail story through AX.\n\n## Milestones\n\n### Milestone 1: Adapter Registry And Observation Envelope\n\n- Add a runtime-level adapter registry.\n- Normalize AX, native window, screenshot, and semantic observations into one envelope.\n- Add target candidates with evidence, confidence, rects, and preferred action channel.\n- Add action risk classification.\n\n### Milestone 2: Chrome Companion Extension\n\n- Build a minimal Manifest V3 extension.\n- Add content-script DOM observe/resolve/act/extract.\n- Add native messaging or local WebSocket pairing with Action.\n- Use extension rects to drive native top-right semantic previews.\n- Keep AX as the Chrome window and toolbar source of truth.\n\n### Milestone 3: tmux And Terminal Family\n\n- Add tmux observe/extract/send primitives.\n- Add terminal host detection for iTerm, Ghostty, Warp, and Terminal.app.\n- Map tmux pane geometry back to native capture regions.\n- Warn when a terminal action requires focus-taking input.\n\n### Milestone 4: Coding Surfaces\n\n- Start with Cursor and VS Code because their extension story is strongest.\n- Add active file, selection, diagnostics, terminal panel, command palette, and chat surface observations.\n- Add Codex and Conductor as AX-first app adapters with optional local state hooks.\n\n### Milestone 5: Recipe Authoring\n\n- Add a declarative recipe format for browser and AX targets.\n- Add local recipe loading, validation, and capability summaries.\n- Ship a Midjourney recipe as dogfood, not as a core adapter.\n- Add import/export for team-shared recipes.\n\n### Milestone 6: Site Adapters\n\n- Layer GitHub and Google-like flows on top of the browser adapter.\n- Promote recipes to first-class site adapters only after repeated usage proves they deserve it.\n- Keep site adapters small and evidence-driven.\n- Avoid hard-coding pixel layouts when DOM or AX can provide stable targets.\n\n### Milestone 7: Long-Tail Hardening\n\n- Improve generic AX resolution.\n- Add screenshot plus local VLM verification.\n- Add better ambiguity reporting.\n- Add \"focus will be stolen\" warnings for unsafe actions.\n\n## Definition Of Done For A First-Class Adapter\n\nAn adapter is first-class when it can:\n\n- Observe the current state from AX and at least one complementary channel.\n- Resolve targets with evidence and confidence.\n- Act through the safest available channel.\n- Verify the result after acting.\n- Provide capture hints for screenshots, zooms, and overlays.\n- Explain when it cannot act without interfering with the user.\n- Fall back cleanly to generic AX behavior.\n\n## Product Rule\n\nAction should be excellent in the places builders actually live, and still\ncompetent everywhere else." }, { "id": "V0", "title": "V0", "content": "# v0 Scope\n\n## Included\n\n- macOS only\n- Native session lifecycle\n- Observe / resolve / act primitives\n- Browser and app target resolution\n- Raw screen capture plus cursor/click metadata\n- Chapters, labels, subtitles\n- Auto-zoom and click emphasis during composition\n- Voice and background music tracks\n- CLI and MCP frontends\n- Remotion composition backend\n\n## Deferred\n\n- Full GUI editor\n- Multi-platform support\n- Vision-only autonomous clicking\n- Complex collaborative timelines\n- Multiple simultaneous app surfaces in one live session\n\n## Target Resolution Order\n\n1. semantic ids\n2. accessibility or DOM matches\n3. bounded text-based fuzzy match\n4. calibrated anchor\n5. coordinates\n\nAutomatic actions should only execute when confidence crosses an explicit threshold." }, { "id": "VISION", "title": "VISION", "content": "# Vision\n\n`action` is an agentic system for producing polished software demos and promotional videos on macOS.\n\nIt is designed for a workflow where an engineer, founder, or AI agent can say:\n\n> Show the core features of this product clearly, stylishly, and repeatably.\n\nAnd the system can turn that intent into a deterministic run, a trace of what happened, and a polished output that is ready to share.\n\n## Why This Exists\n\nExisting work in this space tends to split into a few categories:\n\n- capture tools that are visually polished but not programmable enough\n- automation tools that can click around but do not produce presentation-quality output\n- DSL-driven systems that become fragile because execution, lifecycle, and rendering are all mixed together\n\n`action` is meant to bridge those gaps.\n\nIt should feel like:\n\n- a serious automation runtime\n- a serious demo-production system\n- a tool an agent can use confidently\n\n## What We Learned From Prior Work\n\n### From `vif`\n\nKeep:\n\n- ambition around browser/app automation plus media output\n- promo/demo orientation\n- support for overlays, audio, and export\n- the insight that this is useful for AI-agent workflows\n\nDo not keep:\n\n- loosely bounded product surface\n- CLI-owned lifecycle complexity\n- runtime state spread across unrelated modules\n- architectural drift caused by feature accumulation\n\n### From `stage`\n\nKeep:\n\n- cleaner module instincts\n- stronger agent-tool framing\n- the desire for explicit capabilities and clearer boundaries\n\nDo not keep:\n\n- incomplete lifecycle/error handling\n- underdeveloped reliability story\n- early-stage abstractions that still need significant hardening\n\n### From Recordly\n\nBorrow:\n\n- the render-time polish mindset\n- auto zoom and stylish motion language\n- cursor smoothing and emphasis\n- project/artifact-oriented composition\n\nDo not borrow:\n\n- editor-first assumptions as the primary architecture\n- GUI-centric runtime control\n\n### From Stagehand\n\nBorrow:\n\n- the conceptual separation of `observe`, `act`, and `extract`\n- conservative, inspectable target resolution\n- agent-friendly interaction model\n\n### From Peekaboo\n\nBorrow:\n\n- native-first Swift ownership of macOS automation concerns\n- explicit permission handling as a first-class product concern\n- thin consumer surfaces over a shared automation core\n- the idea that visual feedback can be a distinct layer instead of being mixed into runtime truth\n\nDo not borrow:\n\n- a broad automation surface before the guided demo loop is solid\n- product sprawl caused by trying to cover every macOS interaction mode too early\n\n### From QuickRecorder and similar macOS-native tools\n\nBorrow:\n\n- native-first capture fidelity\n- macOS-specific optimizations\n- acceptance that cursor, capture, and media quality are better when treated as native concerns\n\n### From Remotion\n\nBorrow:\n\n- rendering and templating as a backend capability\n- separation of composition from capture/runtime concerns\n\nDo not borrow:\n\n- the temptation to make the rendering layer define the whole product\n\n## Product Vision\n\nThe ideal end-state for `action` is:\n\n1. an agent can inspect an app or website\n2. resolve meaningful targets safely\n3. execute a demo scenario deterministically\n4. record raw media and rich metadata\n5. automatically produce a polished output with zooms, cues, subtitles, and audio\n\nThe user should not need to hand-author every coordinate or every transition.\n\nThe user should be able to define intent, constraints, and style, while `action` handles the hard parts around:\n\n- target resolution\n- runtime stability\n- capture fidelity\n- metadata collection\n- composition orchestration\n\n## What `action` Should Be Great At\n\n### 1. Agentic Demos\n\nAn AI agent should be able to:\n\n- inspect available windows and surfaces\n- resolve likely targets\n- ask for confirmation when resolution is ambiguous\n- execute a feature demo in a repeatable way\n\n### 2. Product Storytelling\n\nThe output should feel closer to a polished product walkthrough than a raw screen recording.\n\nThat means:\n\n- chapter cues\n- focus guidance\n- subtitles\n- tasteful zooming and reframing\n- cursor clarity\n- music and narration support\n\n### 3. Deterministic Runs\n\nA run should leave behind a trace that explains:\n\n- what was observed\n- what targets were resolved\n- what actions were taken\n- what failed or was ambiguous\n- what assets were produced\n\nThis is essential both for debugging and for agent trust.\n\n### 4. Strong Native Fidelity\n\nThe system should feel designed for macOS, not merely compatible with it.\n\nThat includes:\n\n- ScreenCaptureKit quality\n- Accessibility API targeting\n- native window handling\n- native cursor and input behavior\n\n## What We Explicitly Do Not Want\n\n- a haunted runtime with hidden background state\n- click automation that guesses recklessly\n- a giant scene DSL that becomes the true implementation\n- a framework that claims to be universal before it is reliable\n- premature editor complexity\n- a product that is visually polished but operationally fragile\n\n## Design Taste\n\n`action` should feel:\n\n- deliberate\n- inspectable\n- native\n- cinematic\n- composable\n- dependable\n\nThe cinematic part matters. This is not merely automation; it is automation in service of making good-looking software demos.\n\n## v0 Aspiration\n\nThe first successful version of `action` should let a user or agent:\n\n- define a short demo scenario\n- run it against a macOS app or browser flow\n- get a raw capture plus trace\n- generate a polished edit with:\n - zooms\n - click emphasis\n - labels or chapters\n - subtitles\n - optional voice/music\n\nThat would already be meaningfully better than either a plain recorder or a plain automation tool.\n\n## Strategic Direction\n\n`action` should be built as a stable core with borrowable edges.\n\nThe system should own:\n\n- runtime behavior\n- session model\n- targeting model\n- trace model\n- render manifest\n\nAnd it should delegate where appropriate:\n\n- Remotion for polished render backends\n- FFmpeg or similar tools for specific media-processing tasks\n- browser adapters for DOM-specific observation and action\n\nThis allows the product to move quickly without surrendering architectural control.\n\n## Working Name\n\n`action` is a good working name because it fits the cinema lexicon and reinforces the runtime emphasis.\n\nIt may or may not be the final public name. The architectural direction matters more than the branding for now." }, { "id": "api", "title": "Api", "content": "# API\n\nThis page documents the current local agent protocol for `action`.\n\nThe protocol is intentionally small right now. It is enough to support native\nhealth checks, permissions, window control, screenshots, and recording.\n\nPrimary source:\n\n- [ActionAgentProtocol.swift](native/engine/CoreSources/ActionAgentProtocol.swift)\n\n## Transport\n\n- Protocol: WebSocket\n- Default host: `127.0.0.1`\n- Default port: `4319`\n\n## Request Shape\n\n```json\n{\n \"id\": \"uuid-string\",\n \"method\": \"capture.recordRegion\",\n \"params\": {\n \"output\": \"/tmp/example.mov\"\n }\n}\n```\n\nFields:\n\n- `id`: request id, defaults to a UUID in native clients\n- `method`: string name from `ActionAgentMethod`\n- `params`: flat string map\n\n## Response Shape\n\n```json\n{\n \"id\": \"uuid-string\",\n \"ok\": true,\n \"result\": {\n \"status\": \"finished\"\n },\n \"error\": null\n}\n```\n\nFields:\n\n- `id`: mirrors request id\n- `ok`: success flag\n- `result`: optional flat string map\n- `error`: optional error string\n\n## Methods\n\n| Method | Purpose | Key Params |\n|---|---|---|\n| `ping` | basic liveness check | none |\n| `status` | server metadata and supported methods | none |\n| `permissions.snapshot` | current permission state | none |\n| `permissions.request` | request or prompt for permissions | none |\n| `settings.openAccessibility` | open Accessibility settings pane | none |\n| `settings.openScreenRecording` | open Screen Recording settings pane | none |\n| `app.activate` | bring an app forward | `bundleId` |\n| `window.setFrame` | move/resize an app window | `bundleId`, `x`, `y`, `width`, `height` |\n| `window.getFrame` | inspect current app window frame | `bundleId` |\n| `capture.recordAppWindow` | record a target app window | `bundleId`, `output`, optional `stopFile`, `finishedFile`, `debugLog` |\n| `capture.recordRegion` | record a bounded region | `x`, `y`, `width`, `height`, `output`, optional `fps`, `scale`, `stopFile`, `finishedFile`, `debugLog` |\n| `capture.screenshotAppWindow` | screenshot a target app window | `bundleId`, `output` |\n| `capture.screenshotRegion` | screenshot a bounded region | `x`, `y`, `width`, `height`, `output` |\n| `capture.screenshotScreen` | screenshot the main display | `output` |\n\n## Recording Contract\n\nRecording methods have a special lifecycle:\n\n- startup success means the recording path was accepted\n- actual completion is represented by the finished marker file\n- for debugging, pass `debugLog`\n\nThis matters because recording is currently performed by launching a real\n`Action.app` probe instance rather than keeping the full recording lifecycle\ninside the headless agent process." }, { "id": "browser-profiles", "title": "Browser-profiles", "content": "# Browser Identity: Regular Chrome, Action Browsers, Seeded Identities\n\nHow Action gives an agent a **browser identity** without taking over your daily\nChrome — and which kind of control you get in each case.\n\n## Three browsers, two kinds of control\n\nWhen an agent says \"I'll open it in Chrome\", it can mean one of three things.\nThey are not interchangeable.\n\n| | What it is | What the agent can do |\n|---|---|---|\n| **1. Your regular Chrome** | The browser you use all day. Your real profiles (`Default`, `Profile 1` / \"Work\"), your tabs, history, extensions, and logins. | **Open a URL, and that's it.** `browser_open { mode: \"regular\" }` is a visible handoff with no DevTools attachment. To make the agent *act* there, use Action's native macOS control: screen capture + accessibility (`action.observe.snapshot`, `action.resolve.target`, `action.act.execute`). |\n| **2. An Action browser** | A real, non-headless Chrome that Action owns, on its own user-data-dir under `~/Library/Application Support/Action/ChromeProfiles/`. The default identity `agent-browser` is blank and signed into nothing. | **Full DOM control** over CDP: `browser_snapshot`, `browser_click`, `browser_fill`, `browser_resize`, `browser_screenshot`, `browser_tabs`. |\n| **3. An Action browser identity seeded from one of your Chrome profiles** | The same Action-owned Chrome under a name you choose (`work`, `mira`, …), carrying cookies copied from one of your real profiles for an explicit allowlist of domains. | **Full DOM control, on sites you're already signed in to.** |\n\nThe tradeoff in one line: **your regular Chrome has your session but only\nscreen-and-accessibility control; an Action browser has DOM control but starts\nas a stranger.** Seeding (option 3) is how you get both.\n\nThis is a boundary Action does not try to cross. [Chrome 136 and later ignore\nremote-debugging switches for the default personal data\ndirectory](https://developer.chrome.com/blog/remote-debugging-port), and Action\ndoes not attempt to bypass it. Pointing automation at your everyday user-data-dir\nwould also fight Chrome's directory lock and hand an agent your full history,\nextensions, and sessions by accident.\n\n### Picking one\n\n- The user wants to *look* at something in their own browser → **regular**.\n- The agent needs to click through a public site → **Action browser**.\n- The agent needs to click through a site you're logged into → **seeded identity**.\n- The agent needs to act inside your real signed-in session, and cookies aren't\n enough (passkeys, SSO device trust, an extension's state) → **regular Chrome\n plus Action's native observe/act**, not the browser tools.\n\n## The canonical example: seed \"Profile 1\" into `work`, then drive it\n\nYour Chrome profile *directory* names are `Default`, `Profile 1`, `Profile 2`, …\nand are not the display names you see in Chrome's avatar menu. A browser you\ncall \"Work\" is usually the `Profile 1` directory. Map them first, then seed.\n\nFrom an agent, over MCP:\n\n```text\nbrowser_import_cookies { listSourceProfiles: true }\n# -> [{ dir: \"Default\", name: \"Personal\" }, { dir: \"Profile 1\", name: \"Work\" }, ...]\n\nbrowser_import_cookies { into: \"work\", source: \"Profile 1\", domains: [\"github.com\"] }\n# dry run: lists exactly which cookies would be copied\n\nbrowser_import_cookies { into: \"work\", source: \"Profile 1\", domains: [\"github.com\"], confirm: true }\n# writes them into the Action identity `work`\n\nbrowser_open { url: \"https://github.com/notifications\", profile: \"work\" }\nbrowser_snapshot\nbrowser_screenshot\n```\n\nThe same thing from the terminal:\n\n```bash\n# map display names to profile directories\nbun run chrome:companion:import:cookies -- --list-profiles\n\n# dry run\nbun run chrome:companion:import:cookies -- list --into work --source \"Profile 1\" --domains github.com\n\n# write\nbun run chrome:companion:import:cookies -- import --into work --source \"Profile 1\" --domains github.com --confirm\n```\n\nAfter that, `work` is an ordinary Action identity: `browser_use_profile\n{ profile: \"work\" }` or `browser_open { url, profile: \"work\" }`, and every DOM\ntool works against it.\n\n## Named identities\n\nIdentities are created on first use — any unused name gives you a fresh blank\none. Shipped conventions:\n\n- `agent-browser` — the default blank agent identity\n- `work` — seeded from your work Chrome profile, per the example above\n- `mira` — the creative/Midjourney identity used by the companion tooling\n\nCreate and prepare one with the companion extension loaded:\n\n```bash\n# from the Action product root (`products/action`)\nbun run chrome:companion:profile -- setup work\n```\n\nThis builds the companion extension, creates the profile directory, opens\n`chrome://extensions` in that profile, and reveals `packages/chrome-companion/dist`\nso you can **Load unpacked** once.\n\nLater:\n\n```bash\nbun run chrome:companion:profile -- launch work\nbun run chrome:companion:profile -- check work\nbun run chrome:companion:profile -- path work\n```\n\nEnvironment aliases (shared between companion tooling and Action Browser MCP):\n\n| Variable | Meaning |\n|----------|---------|\n| `ACTION_BROWSER_PROFILE` / `ACTION_CHROME_COMPANION_PROFILE` | Active identity name |\n| `ACTION_BROWSER_PROFILE_DIR` / `ACTION_CHROME_COMPANION_PROFILE_DIR` | Absolute user-data-dir override |\n| `ACTION_BROWSER_PROFILE_ROOT` / `ACTION_CHROME_COMPANION_PROFILE_ROOT` | Root for named identities |\n| `ACTION_BROWSER_DEBUG_PORT` | CDP port (MCP default `9334`) |\n| `ACTION_ROOT` | Action product root (`/products/action`); cookie tools use it when the MCP process starts elsewhere |\n\n## Cookie seeding\n\nCopy **selected** cookies from a regular Chrome profile into an Action identity:\n\n```bash\n# list your Chrome profiles (directory + display name)\nbun run chrome:companion:import:cookies -- --list-profiles\n\n# list Action identities\nbun run chrome:companion:import:cookies -- --list-action-profiles\n\n# dry-run\nbun run chrome:companion:import:cookies -- list --into work --domains github.com\n\n# write\nbun run chrome:companion:import:cookies -- import --into work --domains github.com --confirm\n\n# narrow to specific cookies\nbun run chrome:companion:import:cookies -- import --into mira \\\n --domains midjourney.com \\\n --only cf_clearance \\\n --confirm\n```\n\nOr via the profile CLI:\n\n```bash\nbun run chrome:companion:profile -- import-cookies import --into work --domains github.com --confirm\n```\n\nNotes:\n\n- Requires a Keychain allow for Chrome Safe Storage only when decrypting values;\n the import path copies encrypted blobs as-is for same-machine use.\n- Prefer domain allowlists over full-jar dumps.\n- Cookies are not a complete identity (localStorage / passkeys may still need a\n one-time interactive login in the Action identity).\n\n## Chrome Companion (generic extension)\n\nPackage: `packages/chrome-companion`\n\n- Manifest V3 extension with DOM observe / resolve / act helpers\n- Localhost bridge on `http://127.0.0.1:4321` (WebSocket for the extension)\n- One-time **Load unpacked** per Action identity (`dist/` after build)\n\n```bash\nbun run chrome:companion:build\nbun run chrome:companion:bridge\nbun run chrome:companion:health\nbun run chrome:companion:profile -- setup work\n```\n\nChrome Stable often ignores `--load-extension`; the reliable path is manual\nunpacked install inside the Action-owned identity.\n\n## Action Browser MCP\n\nServer: `bin/mcp/toolsets/browser/index.ts` in the lattices repo root — the\nbrowser tools are a toolset inside the lattices MCP, served by `lattices mcp`.\nSee [the lattices MCP doc](../../../docs/mcp.md).\n\n### Tools\n\n| Tool | Purpose |\n|------|---------|\n| `browser_profiles` | List Action identities, the active one, and the three-surface policy |\n| `browser_use_profile` | Switch the active identity (`agent-browser`, `work`, `mira`, …) |\n| `browser_profile_info` | Active path, cookie readiness, companion hints |\n| `browser_import_cookies` | Dry-run or confirm a seed from a regular Chrome profile |\n| `browser_companion_status` | Extension dist + bridge health |\n| `browser_open` | Open in an Action browser, or `mode: \"regular\"` for an open-only handoff |\n| `browser_tabs` / `browser_snapshot` / `browser_click` / `browser_fill` / `browser_screenshot` / `browser_close` | DOM automation via CDP — Action browsers only |\n| `browser_resize` | Set a tab to an explicit viewport width and height for responsive checks; `target: \"window\"` resizes the real Chrome window instead |\n\n### Two MCP surfaces\n\n- **Native runtime MCP** (`action`): observe / resolve / act / record on macOS\n through Action.app. This is what controls your *regular* Chrome window, and\n every other native app, via screen capture and accessibility.\n- **Browser MCP** (the `browser` toolset of `lattices mcp`): Action-owned Chrome\n identities plus DOM-level tools over CDP.\n\nRegister it. The entry names the `lattices` binary and no path, so a repo move or\na version bump cannot break it:\n\n```bash\nclaude mcp add lattices -s user -- lattices mcp\n```\n\nTo pin a default identity, add the environment variable to that same entry:\n\n```bash\nclaude mcp add lattices -s user -e ACTION_BROWSER_PROFILE=work -- lattices mcp\n```\n\n`ACTION_ROOT` is only needed for the Chrome Companion paths below, and only when\nthe tools run from a published `lattices` install rather than a checkout.\n\n### Agent workflow\n\n```text\nbrowser_profiles # what identities exist\nbrowser_use_profile { profile: \"work\" } # pick one\nbrowser_open { url: \"https://github.com\" } # drive it\nbrowser_snapshot\nbrowser_screenshot\nbrowser_companion_status # optional richer DOM path\nbrowser_close { scope: \"browser\" } # when the task is done\n```\n\nTo open a URL in the user's own Chrome instead:\n\n```text\nbrowser_open { url: \"https://github.com\", mode: \"regular\" }\n```\n\nRegular mode is deliberately not controllable, and the result says so:\n`controlAvailable: false`, plus the native control path. Snapshot, click, fill,\nand screenshot continue to target the Action browser.\n\n### Responsive breakpoints\n\n`browser_resize` gives an already-open tab an explicit viewport, so a narrow\nlayout can be captured instead of assumed.\n\n```text\nbrowser_open { url: \"http://localhost:3000\" }\nbrowser_resize { width: 1440, height: 900 }\nbrowser_screenshot\nbrowser_resize { width: 390, height: 844, mobile: true, matchMedia: [\"(max-width: 767px)\"] }\nbrowser_screenshot\nbrowser_resize { reset: true }\n```\n\nTwo targets, deliberately not interchangeable:\n\n| | What moves | Cost |\n|---|---|---|\n| `target: \"tab\"` (default) | CSS emulation for that one tab | Exact to the pixel, other tabs unaffected, fully reversible |\n| `target: \"window\"` | The real Chrome window hosting the tab | Every tab in that window moves, the display is a hard ceiling, and the tab's emulated viewport is dropped first |\n\nA device-metrics override belongs to the CDP session that set it, and Action\nBrowser opens a fresh session per tool call. Two consequences shape the design:\n\n- The server keeps the requested viewport in memory and re-applies it on every\n session, so a resize survives the later `browser_screenshot` that is the whole\n point of it. It is per-tab, in-memory only, and dropped when the tab or the\n browser closes — nothing is written to the profile.\n- `reset` cannot simply call `clearDeviceMetricsOverride`: a foreign session's\n clear is accepted and does nothing, leaving Chrome stuck at the stale size. The\n reset re-sets the metrics to adopt ownership first, then clears them.\n\n`deviceScaleFactor` defaults to `1`, which makes screenshot pixels equal CSS\npixels — a 390-wide viewport returns a 390px-wide PNG. Read `exact` in the reply\nbefore trusting a capture: a page with a hard `min-width` can refuse to lay out\nas narrow as it was asked to.\n\n## Plugin versioning\n\nHarnesses cache the Action Browser skill and interface metadata by plugin\nversion. Bump all seven manifests together when that metadata changes:\n\n```bash\nbun run plugin:version # check that every manifest agrees\nbun run plugin:version -- 0.4.1 # set them all\n```\n\nThe check covers the root and Action product marketplace/Kimi manifests, plus\nthe browser plugin's Claude, Codex, and Kimi manifests. Reinstall or refresh the\nplugin in each harness so its cached skill is replaced.\n\nThe plugin no longer carries an MCP server or a `SERVER_VERSION`. Browser tools\nship with the lattices CLI; updating that CLI and restarting its MCP process\nupdates the server separately from refreshing this skill-only plugin.\n\n## What is intentionally not done\n\n- Automation attachment to the currently open regular Chrome window\n- Silent full cookie jar import\n- Claiming a page is authenticated without observing it after seed/login\n\n## Related\n\n- [packages/chrome-companion/README.md](../packages/chrome-companion/README.md)\n- [ACT-001 surface adapter architecture](decisions/ACT-001-surface-adapter-architecture.md)\n- [plugins/action-browser/skills/action-browser/SKILL.md](../plugins/action-browser/skills/action-browser/SKILL.md)\n\n## Browser request deadlines\n\n`browser_open.waitMs` is the total operation budget (15,000 ms by default),\nincluding profile selection, Chrome startup, HTTP/CDP connection, navigation,\nand readiness checks. Zero fails immediately without opening a tab. Expiry\nreturns an MCP tool error and cancels pending transport work; increasing\n`waitMs` does not remove the 10-second limit on individual HTTP/CDP operations.\n\nConfigure the MCP server as `command: \"lattices\", args: [\"mcp\"]` — no path, so it\ncannot be pinned to a stale checkout or plugin cache. On the next MCP process\nstart, `initialize` reports the lattices version, and `tools/list` exposes\n`browser_open.mode`, `profile`, and `newTab`. Existing MCP processes retain their\nloaded implementation until reloaded.\n\nThe `lattices` launcher resolves Bun from `LATTICES_BUN_BIN`, PATH, or the\nstandard user and Homebrew locations, so it survives the bare system PATH a\nGUI-launched harness hands it. An explicit invalid override fails with an\nactionable error." }, { "id": "native-runtime.agent", "title": "Native-runtime.agent", "content": "# Native Runtime (Agent)\n\n## Ownership Model\n\n### `Action.app`\n\nOwns:\n\n- AppKit lifecycle\n- launcher UI\n- WebKit windows\n- menus and app identity\n- recording probe mode\n\nPrimary files:\n\n- [ActionHostMain.swift](native/engine/Sources/ActionHostMain.swift)\n- [ActionLauncherController.swift](native/engine/Sources/ActionLauncherController.swift)\n- [ActionLauncherViewModel.swift](native/engine/Sources/ActionLauncherViewModel.swift)\n\n### Agent Runtime\n\nOwns:\n\n- WebSocket listener\n- request dispatch\n- automation method surface\n- launching the recording probe\n\nPrimary files:\n\n- [ActionAgentRuntime.swift](native/engine/CoreSources/ActionAgentRuntime.swift)\n- [ActionAgentClient.swift](native/engine/CoreSources/ActionAgentClient.swift)\n- [ActionAgentCommandBridge.swift](native/engine/Sources/ActionAgentCommandBridge.swift)\n\n## Recording Rule\n\n- Do not implement real recording directly in the headless agent path.\n- Use [ActionRecordingProbeLauncher.swift](native/engine/CoreSources/ActionRecordingProbeLauncher.swift).\n- The probe launches `Action.app` with `recording-probe` args.\n- The probe runner performs the actual `WindowRecorder` call.\n\n## WebKit Rule\n\n- WebKit belongs to the app lifecycle.\n- If a change touches browser windows, inspect the AppKit entry path before changing configuration.\n\n## Debug Order\n\n1. Confirm the signed app build exists.\n2. Confirm whether the failure is app lifecycle, agent transport, or probe execution.\n3. For recording, inspect `.mov`, `.finished`, and debug log together.\n4. Do not trust initial `\"recording\"` replies as proof of completion." }, { "id": "overview.agent", "title": "Overview.agent", "content": "# Overview (Agent)\n\n## Project\n\n- Name: `action`\n- Type: monorepo with native macOS runtime focus\n- Primary native surface: `Action.app`\n- Primary automation surface: local WebSocket agent runtime\n\n## Current Truth\n\n- Native app lifecycle matters more than CLI convenience for WebKit and recording.\n- `Action.app` is the owner of AppKit lifecycle, menus, WebKit, and permission UX.\n- Recording stability currently depends on running capture in a fresh app instance via `recording-probe`.\n- The agent is for transport and orchestration, not for pretending to be a full UI lifecycle host.\n\n## Entry Points\n\n- Root framing: [README.md](README.md)\n- Native host main: [ActionHostMain.swift](native/engine/Sources/ActionHostMain.swift)\n- Agent runtime: [ActionAgentRuntime.swift](native/engine/CoreSources/ActionAgentRuntime.swift)\n- Recording probe launcher: [ActionRecordingProbeLauncher.swift](native/engine/CoreSources/ActionRecordingProbeLauncher.swift)\n- Recording probe app runner: [RecordingProbeAppRunner.swift](native/engine/Sources/RecordingProbeAppRunner.swift)\n\n## Use This Page For\n\n- quick repo orientation\n- understanding app-vs-agent ownership\n- finding the files that currently define runtime truth\n\n## Do Not Assume\n\n- that command-mode execution owns the correct macOS UI lifecycle\n- that the agent should directly host all AppKit-dependent behavior\n- that older architecture docs always describe the latest capture implementation details" }, { "id": "recording.agent", "title": "Recording.agent", "content": "# Recording (Agent)\n\n## Status\n\n- Region recording: working through app-backed probe path\n- App-window recording: working through app-backed probe path\n- Success artifacts: `.mov` + `.finished`\n\n## Control Flow\n\n1. caller invokes `record-region` or `record-app-window`\n2. host sends request to agent\n3. agent calls [ActionRecordingProbeLauncher.swift](native/engine/CoreSources/ActionRecordingProbeLauncher.swift)\n4. launcher runs `open -n Action.app --args recording-probe ...`\n5. probe runner uses [RecordingProbeAppRunner.swift](native/engine/Sources/RecordingProbeAppRunner.swift)\n6. probe uses `WindowRecorder` inside a real AppKit lifecycle\n\n## Files To Read\n\n- [ActionHostMain.swift](native/engine/Sources/ActionHostMain.swift)\n- [ActionAgentRuntime.swift](native/engine/CoreSources/ActionAgentRuntime.swift)\n- [ActionRecordingProbeLauncher.swift](native/engine/CoreSources/ActionRecordingProbeLauncher.swift)\n- [RecordingProbeAppRunner.swift](native/engine/Sources/RecordingProbeAppRunner.swift)\n\n## Debugging Rules\n\n- Initial `\"recording\"` response is startup ack only.\n- Finished marker is the completion signal.\n- If present, `--debug-log` should be inspected before changing capture configuration.\n- If recording fails again, validate lifecycle assumptions before tweaking `ScreenCaptureKit` settings.\n\n## Invariants\n\n- Recording implementation should stay app-backed until a safer replacement exists.\n- Probe mode must remain available from the signed app bundle.\n- App lifecycle correctness is a recording dependency, not an optional detail." }, { "id": "skill", "title": "Skill", "content": "# Action Skills\n\nThese are project-specific skills or task patterns an AI agent should follow.\n\n## Native Runtime Review\n\nUse when:\n\n- changing AppKit lifecycle code\n- changing agent startup or bridge logic\n- changing WebKit entry paths\n\nInstructions:\n\n1. read [docs/native-runtime.md](docs/native-runtime.md)\n2. verify whether the change belongs in app code or agent code\n3. keep AppKit-owned behavior in `Action.app`\n4. verify the signed app still builds\n\n## Recording Review\n\nUse when:\n\n- changing `ScreenCaptureKit` recording\n- changing stop-file or finished-file handling\n- changing probe launch behavior\n\nInstructions:\n\n1. read [docs/recording.md](docs/recording.md)\n2. inspect the probe launcher and probe app runner\n3. preserve `.mov` + `.finished` behavior\n4. test a real recording path, not just a unit-level refactor\n\n## Agent API Review\n\nUse when:\n\n- adding WebSocket methods\n- changing request or response shape\n- updating CLI-to-agent behavior\n\nInstructions:\n\n1. read [docs/api.md](docs/api.md)\n2. update the protocol source of truth\n3. confirm method names remain documented\n4. verify callers still match the transport contract\n\n## Hermes Action\n\nUse when:\n\n- connecting Hermes to the native Action runtime\n- using Action MCP tools from a harness\n- recording or inspecting native macOS surfaces through Hermes\n\nInstructions:\n\n1. read [docs/harnesses/hermes-action.md](harnesses/hermes-action.md)\n2. use [skills/hermes-action/SKILL.md](../skills/hermes-action/SKILL.md) as the harness doctrine\n3. prefer observe and resolve before act\n4. treat recording start as asynchronous until `.finished` or `action.record.status` confirms completion" } ] }