{ "version": 1, "project": { "name": "action", "title": "Action", "description": "Observe and control apps, browsers, and recordings." }, "generatedAt": "2026-10-28T16:13:29.025Z", "artifacts": { "llms": "/action/docs/llms.txt", "agents": "/action/docs/AGENTS.md", "fullManifest": "/action/docs/agent-docs.json", "allMarkdown": "/action/docs/llms-full.txt", "markdownBase": "/action/docs/markdown/" }, "recommendedReadOrder": [ "action:overview", "action:getting-started", "action:concepts", "action:agent-setup", "action:browser", "action:browser-profiles", "action:agent-layer", "action:recording", "action:stage", "action:api", "action:native-runtime", "action:remote-engine", "action:skill" ], "docs": [ { "id": "action:overview", "slug": "overview", "product": "action", "kind": "doc", "catalogKind": "guide", "current": true, "title": "Overview", "description": "What Action is, what it installs, and how it relates to Lattices", "order": 0, "sourcePath": "products/action/docs/overview.md", "url": "/action/docs/overview", "rawUrl": "/action/docs/markdown/overview.md", "markdownUrl": "/action/docs/overview.md", "headings": [ { "depth": 2, "text": "What Action does", "anchor": "what-action-does" }, { "depth": 2, "text": "What gets installed", "anchor": "what-gets-installed" }, { "depth": 2, "text": "How Action relates to Lattices", "anchor": "how-action-relates-to-lattices" }, { "depth": 2, "text": "Platforms", "anchor": "platforms" }, { "depth": 2, "text": "Next steps", "anchor": "next-steps" } ], "tokensEstimate": 580, "frontmatter": { "title": "Overview", "description": "What Action is, what it installs, and how it relates to Lattices", "order": 0 }, "markdown": "---\ntitle: Overview\ndescription: What Action is, what it installs, and how it relates to Lattices\norder: 0\n---\n\nAction is a native macOS app that lets agents inspect and control Mac apps and\nbrowser surfaces, and record what they did. It is a separate product in the\nLattices family. It has its own signed `Action.app`, its own local agent, its\nown release tags (`action-vX.Y.Z`), and its own lifecycle.\n\n## What Action does\n\n| Job | Surfaces |\n| --- | --- |\n| Inspect | Screenshots, accessibility trees, Apple Vision OCR, optional provider vision |\n| Control | Target resolution, clicks, typing, key chords, drags, scrolls, focus, app launch |\n| Record | ScreenCaptureKit video of a window or region, plus session traces and artifacts |\n| Stage | A flat backdrop behind listed windows for a clean take (see [Stage](/action/docs/stage)) |\n| Work out of sight | A private virtual display for an app or Action-owned browser (see [Agent layer](/action/docs/agent-layer)) |\n\n## What gets installed\n\n| Part | Where | Purpose |\n| --- | --- | --- |\n| `Action.app` | `/Applications/Action.app` (bundle id `dev.lattices.Action`) | AppKit lifecycle, menus, WebKit, permission UI, recording probes |\n| Action agent | `Action.app/Contents/Helpers/ActionAgent.app`, `ws://127.0.0.1:4319` | Local WebSocket RPC for native requests and drive leases |\n| Agent runtime | A runtime bundle or a `products/action` checkout | TypeScript runtime, product CLI, and the Action MCP server |\n\n`lats action install` installs the app only. The app alone does not include MCP\norchestration. To connect an agent over MCP, install the matching runtime as\ndescribed in [Agent setup](/action/docs/agent-setup).\n\n## How Action relates to Lattices\n\nLattices 0.15.1 owns workspace organization on `lattices.sock`. Action 0.4.0\nowns native computer use on `action.sock`. Both sockets are in\n`~/.lattices/run`. Quitting one app does not stop the other. Remote access goes only through the opt-in Lattices gateway. See\n[API](/action/docs/api#private-connection).\n\nThe Lattices CLI is the family entry point. `lats action` installs, launches,\nquits, and queries Action, and forwards other verbs to the Action CLI. Two\nnearby commands do not reach Action:\n\n- `lats computer` calls the core daemon's `computer.*` methods. See the\n [Lattices API](/lattices/docs/api).\n- `lats mcp` serves the `browser_*` tools that drive Action-owned Chrome\n profiles over CDP. That server is part of the Lattices CLI, not Action.app.\n See [Browser](/action/docs/browser) and the [Lattices MCP](/lattices/docs/mcp).\n\n[Concepts](/action/docs/concepts) has the full command-to-process table.\n\n## Platforms\n\nAction.app runs on macOS. There is no Linux build of Action.app. The Action\nruntime can drive a Linux desktop that runs `lattices-host`. See\n[Remote engine](/action/docs/remote-engine).\n\n## Next steps\n\n- [Getting started](/action/docs/getting-started): install, launch, and check the app.\n- [Agent setup](/action/docs/agent-setup): connect an MCP agent to a matched runtime.\n- [Concepts](/action/docs/concepts): processes, ports, sessions, and leases.\n- [API](/action/docs/api): the agent WebSocket protocol.\n", "content": "Action is a native macOS app that lets agents inspect and control Mac apps and\nbrowser surfaces, and record what they did. It is a separate product in the\nLattices family. It has its own signed `Action.app`, its own local agent, its\nown release tags (`action-vX.Y.Z`), and its own lifecycle.\n\n## What Action does\n\n| Job | Surfaces |\n| --- | --- |\n| Inspect | Screenshots, accessibility trees, Apple Vision OCR, optional provider vision |\n| Control | Target resolution, clicks, typing, key chords, drags, scrolls, focus, app launch |\n| Record | ScreenCaptureKit video of a window or region, plus session traces and artifacts |\n| Stage | A flat backdrop behind listed windows for a clean take (see [Stage](/action/docs/stage)) |\n| Work out of sight | A private virtual display for an app or Action-owned browser (see [Agent layer](/action/docs/agent-layer)) |\n\n## What gets installed\n\n| Part | Where | Purpose |\n| --- | --- | --- |\n| `Action.app` | `/Applications/Action.app` (bundle id `dev.lattices.Action`) | AppKit lifecycle, menus, WebKit, permission UI, recording probes |\n| Action agent | `Action.app/Contents/Helpers/ActionAgent.app`, `ws://127.0.0.1:4319` | Local WebSocket RPC for native requests and drive leases |\n| Agent runtime | A runtime bundle or a `products/action` checkout | TypeScript runtime, product CLI, and the Action MCP server |\n\n`lats action install` installs the app only. The app alone does not include MCP\norchestration. To connect an agent over MCP, install the matching runtime as\ndescribed in [Agent setup](/action/docs/agent-setup).\n\n## How Action relates to Lattices\n\nLattices 0.15.1 owns workspace organization on `lattices.sock`. Action 0.4.0\nowns native computer use on `action.sock`. Both sockets are in\n`~/.lattices/run`. Quitting one app does not stop the other. Remote access goes only through the opt-in Lattices gateway. See\n[API](/action/docs/api#private-connection).\n\nThe Lattices CLI is the family entry point. `lats action` installs, launches,\nquits, and queries Action, and forwards other verbs to the Action CLI. Two\nnearby commands do not reach Action:\n\n- `lats computer` calls the core daemon's `computer.*` methods. See the\n [Lattices API](/lattices/docs/api).\n- `lats mcp` serves the `browser_*` tools that drive Action-owned Chrome\n profiles over CDP. That server is part of the Lattices CLI, not Action.app.\n See [Browser](/action/docs/browser) and the [Lattices MCP](/lattices/docs/mcp).\n\n[Concepts](/action/docs/concepts) has the full command-to-process table.\n\n## Platforms\n\nAction.app runs on macOS. There is no Linux build of Action.app. The Action\nruntime can drive a Linux desktop that runs `lattices-host`. See\n[Remote engine](/action/docs/remote-engine).\n\n## Next steps\n\n- [Getting started](/action/docs/getting-started): install, launch, and check the app.\n- [Agent setup](/action/docs/agent-setup): connect an MCP agent to a matched runtime.\n- [Concepts](/action/docs/concepts): processes, ports, sessions, and leases.\n- [API](/action/docs/api): the agent WebSocket protocol." }, { "id": "action:getting-started", "slug": "getting-started", "product": "action", "kind": "doc", "catalogKind": "guide", "current": true, "title": "Getting started", "description": "Install Action.app, grant permissions, and check the local agent", "order": 1, "sourcePath": "products/action/docs/getting-started.md", "url": "/action/docs/getting-started", "rawUrl": "/action/docs/markdown/getting-started.md", "markdownUrl": "/action/docs/getting-started.md", "headings": [ { "depth": 2, "text": "Prerequisites", "anchor": "prerequisites" }, { "depth": 2, "text": "Install and launch", "anchor": "install-and-launch" }, { "depth": 2, "text": "Query the agent", "anchor": "query-the-agent" }, { "depth": 2, "text": "Connect an agent", "anchor": "connect-an-agent" }, { "depth": 2, "text": "Quit", "anchor": "quit" }, { "depth": 2, "text": "Build from source", "anchor": "build-from-source" }, { "depth": 2, "text": "Recording is asynchronous", "anchor": "recording-is-asynchronous" }, { "depth": 2, "text": "Next steps", "anchor": "next-steps" } ], "tokensEstimate": 720, "frontmatter": { "title": "Getting started", "description": "Install Action.app, grant permissions, and check the local agent", "order": 1 }, "markdown": "---\ntitle: Getting started\ndescription: Install Action.app, grant permissions, and check the local agent\norder: 1\n---\n\nThis page installs the released `Action.app` and checks that its agent runs.\nTo build Action from source, see [Build from source](#build-from-source).\n\n## Prerequisites\n\n- macOS on Apple Silicon.\n- [Bun](https://bun.sh/docs/installation).\n- The Lattices CLI. Install it with `bun add --global @lattices/lattices`.\n\n## Install and launch\n\n1. Install the app:\n\n ```sh\n lats action install\n ```\n\n The command downloads `Action.dmg` from the latest `action-v*` GitHub\n release and installs `/Applications/Action.app`. It does not need a source\n checkout. Add `--launch` to open the app after installation.\n\n2. Launch the app:\n\n ```sh\n lats action launch\n ```\n\n3. Grant Accessibility and Screen Recording to `Action.app` when macOS asks.\n Native control needs Accessibility. Screenshots and recording need Screen\n Recording.\n\n4. Check the status:\n\n ```sh\n lats action status --json\n ```\n\n Expect `installed: true`, `running: true`, and `agentReachable: true`. The\n `agentUrl` is `ws://127.0.0.1:4319`.\n\nIf `agentReachable` is `false`, launch the app again with `lats action launch`.\n`Action.app` starts its agent helper when it launches.\n\n## Query the agent\n\n```sh\nlats action call ping\nlats action call status\nlats action call permissions.snapshot\n```\n\n`status` lists the supported methods, the protocol version, and the path of\nthe capability-token file. Parameters are a JSON object; the CLI sends each\nvalue as a string:\n\n```sh\nlats action call capture.screenshotScreen '{\"output\":\"/tmp/action-screen.png\"}'\n```\n\nSee [API](/action/docs/api) for every method.\n\n## Connect an agent\n\nThe app alone does not include the MCP server. To connect Claude Code, Codex,\nor another MCP harness, install the matching runtime bundle and configure it as\ndescribed in [Agent setup](/action/docs/agent-setup). After that, verify the\ninstall:\n\n```sh\nlats action capabilities\n```\n\nExpect `runtimeParity.state` to be `matched-source-digest`. A version number\nalone does not prove that the app and runtime match.\n\n## Quit\n\n```sh\nlats action quit\n```\n\nThis quits the Action launcher. Prepared [agent-layer](/action/docs/agent-layer)\ndisplays stay connected until you release them explicitly.\n\n## Build from source\n\nUse this path only on a development host. Requirements are macOS 14 or newer,\nBun, and a Swift 6.2 toolchain. Without a developer signing identity, the build\nuses ad hoc signing, and permission checks are not stable across rebuilds.\n\nFrom `products/action` in a Lattices checkout:\n\n1. Install dependencies:\n\n ```sh\n bun install\n ```\n\n2. Build the signed app:\n\n ```sh\n bun run native:app:build\n ```\n\n The build writes `native/dist/Action.app`, including the agent helper.\n\n3. Launch the app:\n\n ```sh\n bun run native:launch\n ```\n\n The launch command rebuilds only if the app is missing or older than its\n sources. After a native change, use `bun run native:relaunch`.\n\n4. Check the bundle and permissions:\n\n ```sh\n bun run native:verify\n bun run native:permissions:status\n ```\n\n5. With permissions granted, run the capture smoke checks:\n\n ```sh\n bun run native:test:screenshot\n bun run native:test:record\n ```\n\n The recording check waits for completion and reports the video, the\n `.finished` marker, and the debug log.\n\n`bun run native:doctor` builds, signs, verifies, and reports permissions in one\nstep. It changes the native build. Do not run it as a read-only check on an\ninstallation target. From the Lattices repository root, `bun run action:launch`\nand `bun run action:dev` call the same launch and relaunch scripts.\n\n## Recording is asynchronous\n\nA recording start reply means that startup was accepted. It does not mean that\nthe recording finished. Completion is the movie file plus a `.finished` marker.\nSee [Recording](/action/docs/recording).\n\n## Next steps\n\n- [Concepts](/action/docs/concepts)\n- [Native runtime](/action/docs/native-runtime)\n- [Browser](/action/docs/browser)\n", "content": "This page installs the released `Action.app` and checks that its agent runs.\nTo build Action from source, see [Build from source](#build-from-source).\n\n## Prerequisites\n\n- macOS on Apple Silicon.\n- [Bun](https://bun.sh/docs/installation).\n- The Lattices CLI. Install it with `bun add --global @lattices/lattices`.\n\n## Install and launch\n\n1. Install the app:\n\n ```sh\n lats action install\n ```\n\n The command downloads `Action.dmg` from the latest `action-v*` GitHub\n release and installs `/Applications/Action.app`. It does not need a source\n checkout. Add `--launch` to open the app after installation.\n\n2. Launch the app:\n\n ```sh\n lats action launch\n ```\n\n3. Grant Accessibility and Screen Recording to `Action.app` when macOS asks.\n Native control needs Accessibility. Screenshots and recording need Screen\n Recording.\n\n4. Check the status:\n\n ```sh\n lats action status --json\n ```\n\n Expect `installed: true`, `running: true`, and `agentReachable: true`. The\n `agentUrl` is `ws://127.0.0.1:4319`.\n\nIf `agentReachable` is `false`, launch the app again with `lats action launch`.\n`Action.app` starts its agent helper when it launches.\n\n## Query the agent\n\n```sh\nlats action call ping\nlats action call status\nlats action call permissions.snapshot\n```\n\n`status` lists the supported methods, the protocol version, and the path of\nthe capability-token file. Parameters are a JSON object; the CLI sends each\nvalue as a string:\n\n```sh\nlats action call capture.screenshotScreen '{\"output\":\"/tmp/action-screen.png\"}'\n```\n\nSee [API](/action/docs/api) for every method.\n\n## Connect an agent\n\nThe app alone does not include the MCP server. To connect Claude Code, Codex,\nor another MCP harness, install the matching runtime bundle and configure it as\ndescribed in [Agent setup](/action/docs/agent-setup). After that, verify the\ninstall:\n\n```sh\nlats action capabilities\n```\n\nExpect `runtimeParity.state` to be `matched-source-digest`. A version number\nalone does not prove that the app and runtime match.\n\n## Quit\n\n```sh\nlats action quit\n```\n\nThis quits the Action launcher. Prepared [agent-layer](/action/docs/agent-layer)\ndisplays stay connected until you release them explicitly.\n\n## Build from source\n\nUse this path only on a development host. Requirements are macOS 14 or newer,\nBun, and a Swift 6.2 toolchain. Without a developer signing identity, the build\nuses ad hoc signing, and permission checks are not stable across rebuilds.\n\nFrom `products/action` in a Lattices checkout:\n\n1. Install dependencies:\n\n ```sh\n bun install\n ```\n\n2. Build the signed app:\n\n ```sh\n bun run native:app:build\n ```\n\n The build writes `native/dist/Action.app`, including the agent helper.\n\n3. Launch the app:\n\n ```sh\n bun run native:launch\n ```\n\n The launch command rebuilds only if the app is missing or older than its\n sources. After a native change, use `bun run native:relaunch`.\n\n4. Check the bundle and permissions:\n\n ```sh\n bun run native:verify\n bun run native:permissions:status\n ```\n\n5. With permissions granted, run the capture smoke checks:\n\n ```sh\n bun run native:test:screenshot\n bun run native:test:record\n ```\n\n The recording check waits for completion and reports the video, the\n `.finished` marker, and the debug log.\n\n`bun run native:doctor` builds, signs, verifies, and reports permissions in one\nstep. It changes the native build. Do not run it as a read-only check on an\ninstallation target. From the Lattices repository root, `bun run action:launch`\nand `bun run action:dev` call the same launch and relaunch scripts.\n\n## Recording is asynchronous\n\nA recording start reply means that startup was accepted. It does not mean that\nthe recording finished. Completion is the movie file plus a `.finished` marker.\nSee [Recording](/action/docs/recording).\n\n## Next steps\n\n- [Concepts](/action/docs/concepts)\n- [Native runtime](/action/docs/native-runtime)\n- [Browser](/action/docs/browser)" }, { "id": "action:concepts", "slug": "concepts", "product": "action", "kind": "doc", "catalogKind": "guide", "current": true, "title": "Concepts", "description": "Action's processes, ports, command routes, sessions, leases, and capture lifecycle", "order": 2, "sourcePath": "products/action/docs/concepts.md", "url": "/action/docs/concepts", "rawUrl": "/action/docs/markdown/concepts.md", "markdownUrl": "/action/docs/concepts.md", "headings": [ { "depth": 2, "text": "Runtime versions", "anchor": "runtime-versions" }, { "depth": 2, "text": "Processes", "anchor": "processes" }, { "depth": 2, "text": "Three native paths", "anchor": "three-native-paths" }, { "depth": 2, "text": "Command destinations", "anchor": "command-destinations" }, { "depth": 3, "text": "Current mismatches", "anchor": "current-mismatches" }, { "depth": 2, "text": "Sessions and artifacts", "anchor": "sessions-and-artifacts" }, { "depth": 2, "text": "Observe, resolve, act", "anchor": "observe-resolve-act" }, { "depth": 2, "text": "Drive leases", "anchor": "drive-leases" }, { "depth": 2, "text": "Stage and agent layer", "anchor": "stage-and-agent-layer" }, { "depth": 2, "text": "Capability discovery", "anchor": "capability-discovery" }, { "depth": 2, "text": "MCP tool names", "anchor": "mcp-tool-names" } ], "tokensEstimate": 1378, "frontmatter": { "title": "Concepts", "description": "Action's processes, ports, command routes, sessions, leases, and capture lifecycle", "order": 2 }, "markdown": "---\ntitle: Concepts\ndescription: Action's processes, ports, command routes, sessions, leases, and capture lifecycle\norder: 2\n---\n\nThis page names the parts of Action and shows which process receives each\ncommand.\n\n## Runtime versions\n\nThe port numbers and connection behavior below describe published builds.\nMatching development builds use private IPC: the Action agent listens on\n`~/.lattices/run/action.sock`, and the runtime and `lats action call` supply\nthe connection token. Lattices core uses its own private socket. Remote access\ngoes through a separate, explicitly started gateway with approved method grants.\nSee [Private transport](/action/docs/api#private-connection) and\n[Remote Linux setup](/action/docs/remote-engine) for those builds.\n\n## Processes\n\n| Process | Role | Address or entry point |\n| --- | --- | --- |\n| `Action.app` | AppKit lifecycle, launcher, menus, WebKit, permission UI. Also runs one-shot native host commands. | `/Applications/Action.app` |\n| Action agent | Long-lived local RPC server for native requests and drive leases | `ws://127.0.0.1:4319` |\n| Recording probe | A fresh `Action.app` instance in `recording-probe` mode that performs one ScreenCaptureKit capture | Launched per recording |\n| Agent runtime | TypeScript sessions, observations, targets, actions, artifacts | `products/action/packages/runtime` |\n| Action CLI | Product CLI over the runtime | `products/action/packages/cli/src/main.ts` |\n| Action MCP | Stdio MCP server over the runtime | `products/action/packages/mcp/src/index.ts` |\n\n`Action.app` starts the agent as a child process from\n`Contents/Helpers/ActionAgent.app`. The helper has its own bundle id and runs\nas a UI-less app, so macOS does not count it as a second `Action.app`\ninstance. The agent exits with its parent.\n\n## Three native paths\n\nThe runtime reaches native code in three ways. Each path exists for a reason.\n\n```text\nruntime ──WebSocket──▶ Action agent (4319) drive leases, capture methods, raw RPC\nruntime ──open -n────▶ Action.app one-shot host commands with a JSON reply file\nagent ────open -n────▶ Action.app recording-probe one ScreenCaptureKit recording\n```\n\n- **Agent RPC.** Fast, persistent, connection-scoped. Drive leases live here\n because a lease must end when its connection closes.\n- **Host commands.** `native/engine/scripts/run-app-host.sh` launches\n `Action.app` with a command such as `drape`, `raise-window`, `agent-layer`,\n `click-point`, or `blink-click`, and reads a JSON reply file. These commands\n carry the app bundle's permission identity, and some need AppKit windows.\n They are not agent WebSocket methods.\n- **Recording probe.** ScreenCaptureKit recording is reliable only inside a\n real app lifecycle, so each recording gets its own app instance. See\n [Recording](/action/docs/recording).\n\n## Command destinations\n\nSeveral Lattices family commands look related to Action. Only some reach it.\n\n| Command | Receiver | Notes |\n| --- | --- | --- |\n| `lats action install`, `update` | Lattices CLI | Downloads `Action.dmg` from the latest `action-v*` release to `/Applications` |\n| `lats action launch`, `quit`, `status` | Lattices CLI and `Action.app` | Starts or stops the app. Prepared layer displays survive `quit`. |\n| `lats action call ` | Action agent, port 4319 | Raw RPC. The CLI does not add credentials; privileged methods require `protocolVersion` and `authToken` parameters. |\n| `lats action mcp` | Action MCP server (stdio) | Needs a runtime bundle, `ACTION_ROOT`, or a checkout |\n| `lats action capabilities`, `layer …`, other verbs | Action CLI | Forwarded with `bun --cwd action …`. Fails without a runtime or checkout. |\n| `lats computer …` | Lattices core daemon, port 9399 | `computer.*` methods are implemented in Lattices core, not in Action |\n| `lats mcp` (`browser_*` tools) | Lattices MCP server | Drives Action-owned Chrome profiles over CDP. It does not call Action.app. |\n| Action MCP with `ACTION_REMOTE_HOST` | `lattices-host` on the remote machine, port 9399 | The Linux path uses the Lattices host protocol, not the Action agent |\n\nThe Lattices CLI finds an Action root in this order: `ACTION_ROOT`, the\n`products/action` directory next to the CLI, the managed runtime at\n`~/Library/Application Support/Lattices/runtime/current/products/action`, and\n`products/action` in the current Git checkout.\n\n### Current mismatches\n\nThese product boundaries apply to both runtime versions. They are documented here so that a reader\ndoes not infer a different owner from the command name.\n\n- `lats computer` sounds like computer use, but it calls the Lattices core\n daemon. Action's native observe and act tools are in the Action MCP server.\n- The `browser_*` tools are documented with Action because they drive\n Action-owned Chrome identities, but the Lattices CLI serves them.\n- On Linux, the Action runtime's `RemoteEngine` uses the Lattices host protocol\n and the Lattices CLI's paired identity. Published builds connect to the host\n on port 9399; matching development builds connect through the gateway.\n No Action agent runs on the Linux host.\n\n## Sessions and artifacts\n\nThe runtime records each run as a session. A session directory holds\nobservations (screenshots, accessibility snapshots, OCR), the action trace, a\nmanifest, and any recordings. CLI inspection writes sessions under\n`artifacts/sessions/`. MCP `session_create` creates one, and\n`artifacts_list` reads it.\n\n## Observe, resolve, act\n\nThe MCP tools follow one loop:\n\n1. `observe_snapshot` captures the surface: screenshot, accessibility, and OCR.\n2. `resolve_target` turns a query (a label, text, or point) into a target.\n3. `act_execute` runs one act: click, type, press-key, drag, scroll,\n focus-window, or open-app.\n\nAccessibility comes first. A target with a bundle id and label is pressed or\nset through accessibility. The pointer is a fallback.\n\n## Drive leases\n\nA drive lease marks that an agent is acting on the desktop. The supervision\nHUD shows the lease's agent and task. `drive_begin` and `drive_release` manage\nleases from MCP. An `attention` lease, which takes the system pointer, requires\nexplicit approval and the agent's capability token. See [API](/action/docs/api).\n\n## Stage and agent layer\n\nThese features solve opposite problems.\n\n| Feature | Use it to | Changes |\n| --- | --- | --- |\n| [Stage](/action/docs/stage) | Present listed windows on a flat backdrop for a recording | Puts a sheet behind the listed windows on the user's screen |\n| [Agent layer](/action/docs/agent-layer) | Work an app or browser out of the user's sight | Moves the task to a prepared virtual display |\n\n## Capability discovery\n\nVersion strings do not prove feature parity. Before private-layer work, read\n`lats action capabilities` or the MCP `capabilities` tool. Check the source and\nnative fingerprints, `runtimeParity.state`, host quarantine, and\n`acceptance.privateLayer.state`.\n\n## MCP tool names\n\nThe Action MCP server advertises underscore names by default\n(`observe_snapshot`, `layer_open`, `drive_begin`). Dotted names such as\n`action.observe.snapshot` are compatibility aliases, enabled with\n`ACTION_MCP_TOOL_NAMES=legacy` or `both`. Use the names from the active\n`tools/list`.\n", "content": "This page names the parts of Action and shows which process receives each\ncommand.\n\n## Runtime versions\n\nThe port numbers and connection behavior below describe published builds.\nMatching development builds use private IPC: the Action agent listens on\n`~/.lattices/run/action.sock`, and the runtime and `lats action call` supply\nthe connection token. Lattices core uses its own private socket. Remote access\ngoes through a separate, explicitly started gateway with approved method grants.\nSee [Private transport](/action/docs/api#private-connection) and\n[Remote Linux setup](/action/docs/remote-engine) for those builds.\n\n## Processes\n\n| Process | Role | Address or entry point |\n| --- | --- | --- |\n| `Action.app` | AppKit lifecycle, launcher, menus, WebKit, permission UI. Also runs one-shot native host commands. | `/Applications/Action.app` |\n| Action agent | Long-lived local RPC server for native requests and drive leases | `ws://127.0.0.1:4319` |\n| Recording probe | A fresh `Action.app` instance in `recording-probe` mode that performs one ScreenCaptureKit capture | Launched per recording |\n| Agent runtime | TypeScript sessions, observations, targets, actions, artifacts | `products/action/packages/runtime` |\n| Action CLI | Product CLI over the runtime | `products/action/packages/cli/src/main.ts` |\n| Action MCP | Stdio MCP server over the runtime | `products/action/packages/mcp/src/index.ts` |\n\n`Action.app` starts the agent as a child process from\n`Contents/Helpers/ActionAgent.app`. The helper has its own bundle id and runs\nas a UI-less app, so macOS does not count it as a second `Action.app`\ninstance. The agent exits with its parent.\n\n## Three native paths\n\nThe runtime reaches native code in three ways. Each path exists for a reason.\n\n```text\nruntime ──WebSocket──▶ Action agent (4319) drive leases, capture methods, raw RPC\nruntime ──open -n────▶ Action.app one-shot host commands with a JSON reply file\nagent ────open -n────▶ Action.app recording-probe one ScreenCaptureKit recording\n```\n\n- **Agent RPC.** Fast, persistent, connection-scoped. Drive leases live here\n because a lease must end when its connection closes.\n- **Host commands.** `native/engine/scripts/run-app-host.sh` launches\n `Action.app` with a command such as `drape`, `raise-window`, `agent-layer`,\n `click-point`, or `blink-click`, and reads a JSON reply file. These commands\n carry the app bundle's permission identity, and some need AppKit windows.\n They are not agent WebSocket methods.\n- **Recording probe.** ScreenCaptureKit recording is reliable only inside a\n real app lifecycle, so each recording gets its own app instance. See\n [Recording](/action/docs/recording).\n\n## Command destinations\n\nSeveral Lattices family commands look related to Action. Only some reach it.\n\n| Command | Receiver | Notes |\n| --- | --- | --- |\n| `lats action install`, `update` | Lattices CLI | Downloads `Action.dmg` from the latest `action-v*` release to `/Applications` |\n| `lats action launch`, `quit`, `status` | Lattices CLI and `Action.app` | Starts or stops the app. Prepared layer displays survive `quit`. |\n| `lats action call ` | Action agent, port 4319 | Raw RPC. The CLI does not add credentials; privileged methods require `protocolVersion` and `authToken` parameters. |\n| `lats action mcp` | Action MCP server (stdio) | Needs a runtime bundle, `ACTION_ROOT`, or a checkout |\n| `lats action capabilities`, `layer …`, other verbs | Action CLI | Forwarded with `bun --cwd action …`. Fails without a runtime or checkout. |\n| `lats computer …` | Lattices core daemon, port 9399 | `computer.*` methods are implemented in Lattices core, not in Action |\n| `lats mcp` (`browser_*` tools) | Lattices MCP server | Drives Action-owned Chrome profiles over CDP. It does not call Action.app. |\n| Action MCP with `ACTION_REMOTE_HOST` | `lattices-host` on the remote machine, port 9399 | The Linux path uses the Lattices host protocol, not the Action agent |\n\nThe Lattices CLI finds an Action root in this order: `ACTION_ROOT`, the\n`products/action` directory next to the CLI, the managed runtime at\n`~/Library/Application Support/Lattices/runtime/current/products/action`, and\n`products/action` in the current Git checkout.\n\n### Current mismatches\n\nThese product boundaries apply to both runtime versions. They are documented here so that a reader\ndoes not infer a different owner from the command name.\n\n- `lats computer` sounds like computer use, but it calls the Lattices core\n daemon. Action's native observe and act tools are in the Action MCP server.\n- The `browser_*` tools are documented with Action because they drive\n Action-owned Chrome identities, but the Lattices CLI serves them.\n- On Linux, the Action runtime's `RemoteEngine` uses the Lattices host protocol\n and the Lattices CLI's paired identity. Published builds connect to the host\n on port 9399; matching development builds connect through the gateway.\n No Action agent runs on the Linux host.\n\n## Sessions and artifacts\n\nThe runtime records each run as a session. A session directory holds\nobservations (screenshots, accessibility snapshots, OCR), the action trace, a\nmanifest, and any recordings. CLI inspection writes sessions under\n`artifacts/sessions/`. MCP `session_create` creates one, and\n`artifacts_list` reads it.\n\n## Observe, resolve, act\n\nThe MCP tools follow one loop:\n\n1. `observe_snapshot` captures the surface: screenshot, accessibility, and OCR.\n2. `resolve_target` turns a query (a label, text, or point) into a target.\n3. `act_execute` runs one act: click, type, press-key, drag, scroll,\n focus-window, or open-app.\n\nAccessibility comes first. A target with a bundle id and label is pressed or\nset through accessibility. The pointer is a fallback.\n\n## Drive leases\n\nA drive lease marks that an agent is acting on the desktop. The supervision\nHUD shows the lease's agent and task. `drive_begin` and `drive_release` manage\nleases from MCP. An `attention` lease, which takes the system pointer, requires\nexplicit approval and the agent's capability token. See [API](/action/docs/api).\n\n## Stage and agent layer\n\nThese features solve opposite problems.\n\n| Feature | Use it to | Changes |\n| --- | --- | --- |\n| [Stage](/action/docs/stage) | Present listed windows on a flat backdrop for a recording | Puts a sheet behind the listed windows on the user's screen |\n| [Agent layer](/action/docs/agent-layer) | Work an app or browser out of the user's sight | Moves the task to a prepared virtual display |\n\n## Capability discovery\n\nVersion strings do not prove feature parity. Before private-layer work, read\n`lats action capabilities` or the MCP `capabilities` tool. Check the source and\nnative fingerprints, `runtimeParity.state`, host quarantine, and\n`acceptance.privateLayer.state`.\n\n## MCP tool names\n\nThe Action MCP server advertises underscore names by default\n(`observe_snapshot`, `layer_open`, `drive_begin`). Dotted names such as\n`action.observe.snapshot` are compatibility aliases, enabled with\n`ACTION_MCP_TOOL_NAMES=legacy` or `both`. Use the names from the active\n`tools/list`." }, { "id": "action:agent-setup", "slug": "agent-setup", "product": "action", "kind": "doc", "catalogKind": "guide", "current": true, "title": "Connect Action to your agent", "description": "Read Action's capabilities and available tools. Report the connected host, native/runtime match, and any setup requirements before taking control.", "order": null, "sourcePath": "apps/site/content/action/agent-setup.md", "url": "/action/docs/agent-setup", "rawUrl": "/action/docs/markdown/agent-setup.md", "markdownUrl": "/action/docs/agent-setup.md", "headings": [ { "depth": 1, "text": "Connect Action to your agent", "anchor": "connect-action-to-your-agent" }, { "depth": 2, "text": "Install the matching runtime", "anchor": "install-the-matching-runtime" }, { "depth": 2, "text": "Configure MCP", "anchor": "configure-mcp" }, { "depth": 2, "text": "Before a private workspace", "anchor": "before-a-private-workspace" }, { "depth": 2, "text": "Linux", "anchor": "linux" } ], "tokensEstimate": 542, "frontmatter": {}, "markdown": "# Connect Action to your agent\n\nThe macOS quick start proves that native capture works. MCP browser orchestration needs an additional runtime. Installing a skill alone does not install that runtime.\n\n## Install the matching runtime\n\n1. Install [Bun](https://bun.sh/docs/installation).\n2. Open the [Action 0.3.1 release](https://github.com/arach/lattices/releases/tag/action-v0.3.1). Download `Action-agent-runtime-0.3.1.tar.gz` and `SHA256SUMS` from that release into the same directory. This is a pinned, published runtime, not the multi-workspace development build shown in the video.\n3. In that directory, verify the archive before extracting it:\n\n```sh\nshasum -a 256 Action-agent-runtime-0.3.1.tar.gz\ncat SHA256SUMS\n```\n\nThe computed hash must equal the entry for the runtime archive. Stop if it differs.\n\n4. Extract into a new directory. Keep its `bin/` and `products/` directories together:\n\n```sh\nmkdir action-runtime-0.3.1\ntar -xzf Action-agent-runtime-0.3.1.tar.gz -C action-runtime-0.3.1\ncd action-runtime-0.3.1\nexport ACTION_ROOT=\"$PWD/products/action\"\nopen \"$ACTION_ROOT/native/dist/Action.app\"\nbun \"$ACTION_ROOT/packages/cli/src/main.ts\" capabilities\n```\n\nQuit any other Action app before opening the bundled app. Grant permissions to the bundled app when requested. Keep the previous app/runtime for rollback. Expect `runtimeParity.state` to be `matched-source-digest`; a version number alone is not a match.\n\n## Configure MCP\n\nAdd a local stdio MCP server in your agent's settings. Use the absolute path returned by `command -v bun` as the command, and the absolute path to `products/action/packages/mcp/src/index.ts` as its argument.\n\nThe following is a configuration template. Replace `/absolute/path` with your extracted runtime directory and `/absolute/path/to/bun` with your Bun executable. Your harness may use a different configuration format.\n\n```json\n{\n \"mcpServers\": {\n \"action\": {\n \"command\": \"/absolute/path/to/bun\",\n \"args\": [\"/absolute/path/products/action/packages/mcp/src/index.ts\"],\n \"env\": {\n \"ACTION_ROOT\": \"/absolute/path/products/action\"\n }\n }\n }\n}\n```\n\nRestart the MCP connection. Ask your agent:\n\n> Read Action's capabilities and available tools. Report the connected host, native/runtime match, and any setup requirements before taking control.\n\nExpect a capabilities report, not immediate browser movement. Do not treat an open connection as proof that every feature is available.\n\n## Before a private workspace\n\nRead the [agent-layer guide](https://lattices.dev/action/agent-layer.md). Check fresh MCP capabilities, `tools/list`, host safety and private-layer acceptance before opening. A matching runtime does not certify a display configuration. Preparing a virtual monitor changes display topology and requires explicit user approval. Private browser input stays CDP-only; an input failure does not authorize control of the user's desktop.\n\nThe demo's named stage and agent workspaces are a development preview. Do not assume the published release supports them. Do not rebuild on an installation target to obtain a missing feature.\n\n## Linux\n\nUse the [remote Linux setup](https://lattices.dev/action/remote-engine.md). This is a remote-host workflow, not a Linux build of Action.app.\n", "content": "# Connect Action to your agent\n\nThe macOS quick start proves that native capture works. MCP browser orchestration needs an additional runtime. Installing a skill alone does not install that runtime.\n\n## Install the matching runtime\n\n1. Install [Bun](https://bun.sh/docs/installation).\n2. Open the [Action 0.3.1 release](https://github.com/arach/lattices/releases/tag/action-v0.3.1). Download `Action-agent-runtime-0.3.1.tar.gz` and `SHA256SUMS` from that release into the same directory. This is a pinned, published runtime, not the multi-workspace development build shown in the video.\n3. In that directory, verify the archive before extracting it:\n\n```sh\nshasum -a 256 Action-agent-runtime-0.3.1.tar.gz\ncat SHA256SUMS\n```\n\nThe computed hash must equal the entry for the runtime archive. Stop if it differs.\n\n4. Extract into a new directory. Keep its `bin/` and `products/` directories together:\n\n```sh\nmkdir action-runtime-0.3.1\ntar -xzf Action-agent-runtime-0.3.1.tar.gz -C action-runtime-0.3.1\ncd action-runtime-0.3.1\nexport ACTION_ROOT=\"$PWD/products/action\"\nopen \"$ACTION_ROOT/native/dist/Action.app\"\nbun \"$ACTION_ROOT/packages/cli/src/main.ts\" capabilities\n```\n\nQuit any other Action app before opening the bundled app. Grant permissions to the bundled app when requested. Keep the previous app/runtime for rollback. Expect `runtimeParity.state` to be `matched-source-digest`; a version number alone is not a match.\n\n## Configure MCP\n\nAdd a local stdio MCP server in your agent's settings. Use the absolute path returned by `command -v bun` as the command, and the absolute path to `products/action/packages/mcp/src/index.ts` as its argument.\n\nThe following is a configuration template. Replace `/absolute/path` with your extracted runtime directory and `/absolute/path/to/bun` with your Bun executable. Your harness may use a different configuration format.\n\n```json\n{\n \"mcpServers\": {\n \"action\": {\n \"command\": \"/absolute/path/to/bun\",\n \"args\": [\"/absolute/path/products/action/packages/mcp/src/index.ts\"],\n \"env\": {\n \"ACTION_ROOT\": \"/absolute/path/products/action\"\n }\n }\n }\n}\n```\n\nRestart the MCP connection. Ask your agent:\n\n> Read Action's capabilities and available tools. Report the connected host, native/runtime match, and any setup requirements before taking control.\n\nExpect a capabilities report, not immediate browser movement. Do not treat an open connection as proof that every feature is available.\n\n## Before a private workspace\n\nRead the [agent-layer guide](https://lattices.dev/action/agent-layer.md). Check fresh MCP capabilities, `tools/list`, host safety and private-layer acceptance before opening. A matching runtime does not certify a display configuration. Preparing a virtual monitor changes display topology and requires explicit user approval. Private browser input stays CDP-only; an input failure does not authorize control of the user's desktop.\n\nThe demo's named stage and agent workspaces are a development preview. Do not assume the published release supports them. Do not rebuild on an installation target to obtain a missing feature.\n\n## Linux\n\nUse the [remote Linux setup](https://lattices.dev/action/remote-engine.md). This is a remote-host workflow, not a Linux build of Action.app." }, { "id": "action:browser", "slug": "browser", "product": "action", "kind": "doc", "catalogKind": "guide", "current": true, "title": "Browser", "description": "The three browser surfaces an agent can drive with Action, and which tools reach each one", "order": 4, "sourcePath": "products/action/docs/browser.md", "url": "/action/docs/browser", "rawUrl": "/action/docs/markdown/browser.md", "markdownUrl": "/action/docs/browser.md", "headings": [ { "depth": 2, "text": "Choose a surface", "anchor": "choose-a-surface" }, { "depth": 2, "text": "Regular Chrome", "anchor": "regular-chrome" }, { "depth": 2, "text": "Action browsers", "anchor": "action-browsers" }, { "depth": 2, "text": "Private-layer browsers", "anchor": "private-layer-browsers" } ], "tokensEstimate": 697, "frontmatter": { "title": "Browser", "description": "The three browser surfaces an agent can drive with Action, and which tools reach each one", "order": 4 }, "markdown": "---\ntitle: Browser\ndescription: The three browser surfaces an agent can drive with Action, and which tools reach each one\norder: 4\n---\n\nAn agent can use a browser with Action in three ways. They differ in which\ntools reach the page, which identity the page has, and whether the user sees it.\n\n## Choose a surface\n\n| Surface | Served by | Control | Identity |\n| --- | --- | --- | --- |\n| Your regular Chrome | Action MCP native tools | Screen capture and accessibility only (`observe_snapshot`, `resolve_target`, `act_execute`) | Your real Chrome profiles |\n| An Action browser | Lattices MCP `browser_*` tools | DOM control over the Chrome DevTools Protocol (CDP) | An Action-owned Chrome profile, blank or seeded |\n| A private-layer browser | Action MCP `layer_*` tools | CDP only, on a prepared virtual display | An Action-owned profile per workspace |\n\nUse regular Chrome when the task needs your live session and visible tabs. Use\nan Action browser for DOM-level work in a window you can watch. Use a\nprivate-layer browser when the work must stay off your screens.\n\n## Regular Chrome\n\n`browser_open {mode:\"regular\"}` opens a URL in your own Chrome as a visible\nhandoff. Nothing attaches DevTools to it. To act in that window, use the Action\nMCP native loop: `observe_snapshot`, then `resolve_target`, then\n`act_execute`.\n\n## Action browsers\n\nThe `browser_*` tools are part of the Lattices MCP server, not `Action.app`.\nRegister it once:\n\n```sh\nclaude mcp add lattices -s user -- lattices mcp\n```\n\nThe tools launch a real, non-headless Chrome on an Action-owned user data\ndirectory and drive it over CDP. The debug port defaults to `9334` and can be\nset with `ACTION_BROWSER_DEBUG_PORT`.\n\n| Tool | Purpose |\n| --- | --- |\n| `browser_open` | Open a URL in an Action browser, or hand it to regular Chrome |\n| `browser_snapshot` | Read the page as an element list with refs |\n| `browser_click`, `browser_fill` | Act on an element ref |\n| `browser_screenshot`, `browser_resize` | Capture or size the viewport |\n| `browser_tabs`, `browser_close` | List or close tabs |\n| `browser_console` | Read console output |\n| `browser_profiles`, `browser_profile_info`, `browser_use_profile` | List and select browser identities |\n| `browser_import_cookies` | Seed an identity with cookies from one of your Chrome profiles |\n| `browser_companion_status` | Check the optional Chrome companion bridge on `ws://127.0.0.1:4321` |\n\nIdentities, cookie seeding, and the companion bridge are covered in\n[Browser profiles](/action/docs/browser-profiles). The full tool reference is\nin the [Lattices MCP](/lattices/docs/mcp) page.\n\n## Private-layer browsers\n\nA private-layer browser runs on a prepared virtual display in an agent-layer\nworkspace. Prepare the display once with the CLI, then use the Action MCP:\n\n```text\nlayer_open {workspace, browser:true, url, task}\nlayer_snapshot {workspace}\nlayer_click / layer_type / layer_press / layer_js / layer_go\nlayer_close {workspace}\n```\n\n- `layer_open` begins a drive lease unless you pass an existing `leaseId`.\n `layer_close` releases it.\n- Coordinates come from the last `layer_snapshot`.\n- The path is CDP only. There is no native fallback, so do not mix in desktop\n `observe_*` or `drive_aim` calls.\n- Browser flags, navigation, input, and JavaScript need MCP. The CLI `layer`\n commands cover lifecycle, snapshot, picture-in-picture, and recording.\n\nRead `capabilities` first and confirm that the host and build accept private\nlayers. See [Agent layer](/action/docs/agent-layer).\n", "content": "An agent can use a browser with Action in three ways. They differ in which\ntools reach the page, which identity the page has, and whether the user sees it.\n\n## Choose a surface\n\n| Surface | Served by | Control | Identity |\n| --- | --- | --- | --- |\n| Your regular Chrome | Action MCP native tools | Screen capture and accessibility only (`observe_snapshot`, `resolve_target`, `act_execute`) | Your real Chrome profiles |\n| An Action browser | Lattices MCP `browser_*` tools | DOM control over the Chrome DevTools Protocol (CDP) | An Action-owned Chrome profile, blank or seeded |\n| A private-layer browser | Action MCP `layer_*` tools | CDP only, on a prepared virtual display | An Action-owned profile per workspace |\n\nUse regular Chrome when the task needs your live session and visible tabs. Use\nan Action browser for DOM-level work in a window you can watch. Use a\nprivate-layer browser when the work must stay off your screens.\n\n## Regular Chrome\n\n`browser_open {mode:\"regular\"}` opens a URL in your own Chrome as a visible\nhandoff. Nothing attaches DevTools to it. To act in that window, use the Action\nMCP native loop: `observe_snapshot`, then `resolve_target`, then\n`act_execute`.\n\n## Action browsers\n\nThe `browser_*` tools are part of the Lattices MCP server, not `Action.app`.\nRegister it once:\n\n```sh\nclaude mcp add lattices -s user -- lattices mcp\n```\n\nThe tools launch a real, non-headless Chrome on an Action-owned user data\ndirectory and drive it over CDP. The debug port defaults to `9334` and can be\nset with `ACTION_BROWSER_DEBUG_PORT`.\n\n| Tool | Purpose |\n| --- | --- |\n| `browser_open` | Open a URL in an Action browser, or hand it to regular Chrome |\n| `browser_snapshot` | Read the page as an element list with refs |\n| `browser_click`, `browser_fill` | Act on an element ref |\n| `browser_screenshot`, `browser_resize` | Capture or size the viewport |\n| `browser_tabs`, `browser_close` | List or close tabs |\n| `browser_console` | Read console output |\n| `browser_profiles`, `browser_profile_info`, `browser_use_profile` | List and select browser identities |\n| `browser_import_cookies` | Seed an identity with cookies from one of your Chrome profiles |\n| `browser_companion_status` | Check the optional Chrome companion bridge on `ws://127.0.0.1:4321` |\n\nIdentities, cookie seeding, and the companion bridge are covered in\n[Browser profiles](/action/docs/browser-profiles). The full tool reference is\nin the [Lattices MCP](/lattices/docs/mcp) page.\n\n## Private-layer browsers\n\nA private-layer browser runs on a prepared virtual display in an agent-layer\nworkspace. Prepare the display once with the CLI, then use the Action MCP:\n\n```text\nlayer_open {workspace, browser:true, url, task}\nlayer_snapshot {workspace}\nlayer_click / layer_type / layer_press / layer_js / layer_go\nlayer_close {workspace}\n```\n\n- `layer_open` begins a drive lease unless you pass an existing `leaseId`.\n `layer_close` releases it.\n- Coordinates come from the last `layer_snapshot`.\n- The path is CDP only. There is no native fallback, so do not mix in desktop\n `observe_*` or `drive_aim` calls.\n- Browser flags, navigation, input, and JavaScript need MCP. The CLI `layer`\n commands cover lifecycle, snapshot, picture-in-picture, and recording.\n\nRead `capabilities` first and confirm that the host and build accept private\nlayers. See [Agent layer](/action/docs/agent-layer)." }, { "id": "action:browser-profiles", "slug": "browser-profiles", "product": "action", "kind": "doc", "catalogKind": "guide", "current": true, "title": "Browser profiles", "description": "Regular Chrome, Action browsers, and identities seeded from your Chrome profiles", "order": 5, "sourcePath": "products/action/docs/browser-profiles.md", "url": "/action/docs/browser-profiles", "rawUrl": "/action/docs/markdown/browser-profiles.md", "markdownUrl": "/action/docs/browser-profiles.md", "headings": [ { "depth": 2, "text": "Three browsers, two kinds of control", "anchor": "three-browsers-two-kinds-of-control" }, { "depth": 3, "text": "Picking one", "anchor": "picking-one" }, { "depth": 2, "text": "The canonical example: seed \"Profile 1\" into work, then drive it", "anchor": "the-canonical-example-seed-profile-1-into-work-then-drive-it" }, { "depth": 2, "text": "Named identities", "anchor": "named-identities" }, { "depth": 2, "text": "Cookie seeding", "anchor": "cookie-seeding" }, { "depth": 2, "text": "Chrome companion (generic extension)", "anchor": "chrome-companion-generic-extension" }, { "depth": 2, "text": "Action browser MCP", "anchor": "action-browser-mcp" }, { "depth": 3, "text": "Tools", "anchor": "tools" }, { "depth": 3, "text": "Two MCP surfaces", "anchor": "two-mcp-surfaces" }, { "depth": 3, "text": "Agent workflow", "anchor": "agent-workflow" }, { "depth": 3, "text": "Responsive breakpoints", "anchor": "responsive-breakpoints" }, { "depth": 2, "text": "Plugin versioning", "anchor": "plugin-versioning" }, { "depth": 2, "text": "What is intentionally not done", "anchor": "what-is-intentionally-not-done" }, { "depth": 2, "text": "Related", "anchor": "related" }, { "depth": 2, "text": "Browser request deadlines", "anchor": "browser-request-deadlines" } ], "tokensEstimate": 2727, "frontmatter": { "title": "Browser profiles", "description": "Regular Chrome, Action browsers, and identities seeded from your Chrome profiles", "order": 5 }, "markdown": "---\ntitle: Browser profiles\ndescription: Regular Chrome, Action browsers, and identities seeded from your Chrome profiles\norder: 5\n---\n\nHow Action gives an agent a **browser identity** without taking over your daily\nChrome, and which kind of control you get in each case.\n\n## Three browsers, two kinds of control\n\nWhen an agent says \"I'll open it in Chrome\", it can mean one of three things.\nThey are not interchangeable.\n\n| | What it is | What the agent can do |\n|---|---|---|\n| **1. Your regular Chrome** | The browser you use all day. Your real profiles (`Default`, `Profile 1` / \"Work\"), your tabs, history, extensions, and logins. | **Open a URL, and that's it.** `browser_open { mode: \"regular\" }` is a visible handoff with no DevTools attachment. To make the agent *act* there, use Action's native macOS control: screen capture + accessibility (`action.observe.snapshot`, `action.resolve.target`, `action.act.execute`). |\n| **2. An Action browser** | A real, non-headless Chrome that Action owns, on its own user-data-dir under `~/Library/Application Support/Action/ChromeProfiles/`. The default identity `agent-browser` is blank and signed into nothing. | **Full DOM control** over CDP: `browser_snapshot`, `browser_click`, `browser_fill`, `browser_resize`, `browser_screenshot`, `browser_tabs`. |\n| **3. An Action browser identity seeded from one of your Chrome profiles** | The same Action-owned Chrome under a name you choose (`work`, `mira`, …), carrying cookies copied from one of your real profiles for an explicit allowlist of domains. | **Full DOM control, on sites you're already signed in to.** |\n\nThe tradeoff in one line: **your regular Chrome has your session but only\nscreen-and-accessibility control; an Action browser has DOM control but starts\nas a stranger.** Seeding (option 3) is how you get both.\n\nThis is a boundary Action does not try to cross. [Chrome 136 and later ignore\nremote-debugging switches for the default personal data\ndirectory](https://developer.chrome.com/blog/remote-debugging-port), and Action\ndoes not attempt to bypass it. Pointing automation at your everyday user-data-dir\nwould also fight Chrome's directory lock and hand an agent your full history,\nextensions, and sessions by accident.\n\n### Picking one\n\n- The user wants to *look* at something in their own browser → **regular**.\n- The agent needs to click through a public site → **Action browser**.\n- The agent needs to click through a site you're logged into → **seeded identity**.\n- The agent needs to act inside your real signed-in session, and cookies aren't\n enough (passkeys, SSO device trust, an extension's state) → **regular Chrome\n plus Action's native observe/act**, not the browser tools.\n\n## The canonical example: seed \"Profile 1\" into `work`, then drive it\n\nYour Chrome profile *directory* names are `Default`, `Profile 1`, `Profile 2`, …\nand are not the display names you see in Chrome's avatar menu. A browser you\ncall \"Work\" is usually the `Profile 1` directory. Map them first, then seed.\n\nFrom an agent, over MCP:\n\n```text\nbrowser_import_cookies { listSourceProfiles: true }\n# -> [{ dir: \"Default\", name: \"Personal\" }, { dir: \"Profile 1\", name: \"Work\" }, ...]\n\nbrowser_import_cookies { into: \"work\", source: \"Profile 1\", domains: [\"github.com\"] }\n# dry run: lists exactly which cookies would be copied\n\nbrowser_import_cookies { into: \"work\", source: \"Profile 1\", domains: [\"github.com\"], confirm: true }\n# writes them into the Action identity `work`\n\nbrowser_open { url: \"https://github.com/notifications\", profile: \"work\" }\nbrowser_snapshot\nbrowser_screenshot\n```\n\nThe same thing from the terminal:\n\n```bash\n# map display names to profile directories\nbun run chrome:companion:import:cookies -- --list-profiles\n\n# dry run\nbun run chrome:companion:import:cookies -- list --into work --source \"Profile 1\" --domains github.com\n\n# write\nbun run chrome:companion:import:cookies -- import --into work --source \"Profile 1\" --domains github.com --confirm\n```\n\nAfter that, `work` is an ordinary Action identity: `browser_use_profile\n{ profile: \"work\" }` or `browser_open { url, profile: \"work\" }`, and every DOM\ntool works against it.\n\n## Named identities\n\nIdentities are created on first use: any unused name gives you a fresh blank\none. Shipped conventions:\n\n- `agent-browser`: the default blank agent identity\n- `work`: seeded from your work Chrome profile, per the example above\n- `mira`: the creative/Midjourney identity used by the companion tooling\n\nCreate and prepare one with the companion extension loaded:\n\n```bash\n# from the Action product root (`products/action`)\nbun run chrome:companion:profile -- setup work\n```\n\nThis builds the companion extension, creates the profile directory, opens\n`chrome://extensions` in that profile, and reveals `packages/chrome-companion/dist`\nso you can **Load unpacked** once.\n\nLater:\n\n```bash\nbun run chrome:companion:profile -- launch work\nbun run chrome:companion:profile -- check work\nbun run chrome:companion:profile -- path work\n```\n\nEnvironment aliases (shared between companion tooling and Action Browser MCP):\n\n| Variable | Meaning |\n|----------|---------|\n| `ACTION_BROWSER_PROFILE` / `ACTION_CHROME_COMPANION_PROFILE` | Active identity name |\n| `ACTION_BROWSER_PROFILE_DIR` / `ACTION_CHROME_COMPANION_PROFILE_DIR` | Absolute user-data-dir override |\n| `ACTION_BROWSER_PROFILE_ROOT` / `ACTION_CHROME_COMPANION_PROFILE_ROOT` | Root for named identities |\n| `ACTION_BROWSER_DEBUG_PORT` | CDP port (MCP default `9334`) |\n| `ACTION_ROOT` | Action product root (`/products/action`); cookie tools use it when the MCP process starts elsewhere |\n\n## Cookie seeding\n\nCopy **selected** cookies from a regular Chrome profile into an Action identity:\n\n```bash\n# list your Chrome profiles (directory + display name)\nbun run chrome:companion:import:cookies -- --list-profiles\n\n# list Action identities\nbun run chrome:companion:import:cookies -- --list-action-profiles\n\n# dry-run\nbun run chrome:companion:import:cookies -- list --into work --domains github.com\n\n# write\nbun run chrome:companion:import:cookies -- import --into work --domains github.com --confirm\n\n# narrow to specific cookies\nbun run chrome:companion:import:cookies -- import --into mira \\\n --domains midjourney.com \\\n --only cf_clearance \\\n --confirm\n```\n\nOr via the profile CLI:\n\n```bash\nbun run chrome:companion:profile -- import-cookies import --into work --domains github.com --confirm\n```\n\nNotes:\n\n- Requires a Keychain allow for Chrome Safe Storage only when decrypting values;\n the import path copies encrypted blobs as-is for same-machine use.\n- Prefer domain allowlists over full-jar dumps.\n- Cookies are not a complete identity (localStorage / passkeys may still need a\n one-time interactive login in the Action identity).\n\n## Chrome companion (generic extension)\n\nPackage: `packages/chrome-companion`\n\n- Manifest V3 extension with DOM observe / resolve / act helpers\n- In Action versions before 0.4.0, a localhost bridge on `http://127.0.0.1:4321` (WebSocket for the extension)\n- One-time **Load unpacked** per Action identity (`dist/` after build)\n\nAction 0.4.0 uses Chrome native messaging instead\nof the extension's TCP connection. The registered native host connects to\n`~/.lattices/run/action-chrome.sock` with `action-chrome.token`. Use the matching\nextension, native host, and Action runtime, including for a local installation.\nRegister the host for the exact extension ID and Chrome user-data directory;\nregistration for regular Chrome does not cover a separate Action identity.\n\n```bash\nbun run chrome:companion:build\nbun run chrome:companion:bridge\nbun run chrome:companion:health\nbun run chrome:companion:profile -- setup work\n```\n\nChrome Stable often ignores `--load-extension`; the reliable path is manual\nunpacked install inside the Action-owned identity.\n\n## Action browser MCP\n\nServer: `bin/mcp/toolsets/browser/index.ts` in the lattices repo root. The\nbrowser tools are a toolset inside the lattices MCP, served by `lattices mcp`.\nSee [the Lattices MCP doc](/lattices/docs/mcp).\n\n### Tools\n\n| Tool | Purpose |\n|------|---------|\n| `browser_profiles` | List Action identities, the active one, and the three-surface policy |\n| `browser_use_profile` | Switch the active identity (`agent-browser`, `work`, `mira`, …) |\n| `browser_profile_info` | Active path, cookie readiness, companion hints |\n| `browser_import_cookies` | Dry-run or confirm a seed from a regular Chrome profile |\n| `browser_companion_status` | Extension dist + bridge health |\n| `browser_open` | Open in an Action browser, or `mode: \"regular\"` for an open-only handoff |\n| `browser_tabs` / `browser_snapshot` / `browser_click` / `browser_fill` / `browser_screenshot` / `browser_close` | DOM automation via CDP, Action browsers only |\n| `browser_console` | Read console messages from an Action browser tab |\n| `browser_resize` | Set a tab to an explicit viewport width and height for responsive checks; `target: \"window\"` resizes the real Chrome window instead |\n\n### Two MCP surfaces\n\n- **Native runtime MCP** (`action`): observe / resolve / act / record on macOS\n through Action.app. This is what controls your *regular* Chrome window, and\n every other native app, via screen capture and accessibility.\n- **Browser MCP** (the `browser` toolset of `lattices mcp`): Action-owned Chrome\n identities plus DOM-level tools over CDP.\n\nRegister it. The entry names the `lattices` binary and no path, so a repo move or\na version bump cannot break it:\n\n```bash\nclaude mcp add lattices -s user -- lattices mcp\n```\n\nTo pin a default identity, add the environment variable to that same entry:\n\n```bash\nclaude mcp add lattices -s user -e ACTION_BROWSER_PROFILE=work -- lattices mcp\n```\n\n`ACTION_ROOT` is only needed for the Chrome Companion paths below, and only when\nthe tools run from a published `lattices` install rather than a checkout.\n\n### Agent workflow\n\n```text\nbrowser_profiles # what identities exist\nbrowser_use_profile { profile: \"work\" } # pick one\nbrowser_open { url: \"https://github.com\" } # drive it\nbrowser_snapshot\nbrowser_screenshot\nbrowser_companion_status # optional richer DOM path\nbrowser_close { scope: \"browser\" } # when the task is done\n```\n\nTo open a URL in the user's own Chrome instead:\n\n```text\nbrowser_open { url: \"https://github.com\", mode: \"regular\" }\n```\n\nRegular mode is deliberately not controllable, and the result says so:\n`controlAvailable: false`, plus the native control path. Snapshot, click, fill,\nand screenshot continue to target the Action browser.\n\n### Responsive breakpoints\n\n`browser_resize` gives an already-open tab an explicit viewport, so a narrow\nlayout can be captured instead of assumed.\n\n```text\nbrowser_open { url: \"http://localhost:3000\" }\nbrowser_resize { width: 1440, height: 900 }\nbrowser_screenshot\nbrowser_resize { width: 390, height: 844, mobile: true, matchMedia: [\"(max-width: 767px)\"] }\nbrowser_screenshot\nbrowser_resize { reset: true }\n```\n\nTwo targets, deliberately not interchangeable:\n\n| | What moves | Cost |\n|---|---|---|\n| `target: \"tab\"` (default) | CSS emulation for that one tab | Exact to the pixel, other tabs unaffected, fully reversible |\n| `target: \"window\"` | The real Chrome window hosting the tab | Every tab in that window moves, the display is a hard ceiling, and the tab's emulated viewport is dropped first |\n\nA device-metrics override belongs to the CDP session that set it, and Action\nBrowser opens a fresh session per tool call. Two consequences shape the design:\n\n- The server keeps the requested viewport in memory and re-applies it on every\n session, so a resize survives the later `browser_screenshot` that is the whole\n point of it. It is per-tab, in-memory only, and dropped when the tab or the\n browser closes. Nothing is written to the profile.\n- `reset` cannot simply call `clearDeviceMetricsOverride`: a foreign session's\n clear is accepted and does nothing, leaving Chrome stuck at the stale size. The\n reset re-sets the metrics to adopt ownership first, then clears them.\n\n`deviceScaleFactor` defaults to `1`, which makes screenshot pixels equal CSS\npixels: a 390-wide viewport returns a 390px-wide PNG. Read `exact` in the reply\nbefore trusting a capture: a page with a hard `min-width` can refuse to lay out\nas narrow as it was asked to.\n\n## Plugin versioning\n\nHarnesses cache the Action Browser skill and interface metadata by plugin\nversion. Bump all seven manifests together when that metadata changes:\n\n```bash\nbun run plugin:version # check that every manifest agrees\nbun run plugin:version -- 0.4.1 # set them all\n```\n\nThe check covers the root and Action product marketplace/Kimi manifests, plus\nthe browser plugin's Claude, Codex, and Kimi manifests. Reinstall or refresh the\nplugin in each harness so its cached skill is replaced.\n\nThe plugin no longer carries an MCP server or a `SERVER_VERSION`. Browser tools\nship with the lattices CLI; updating that CLI and restarting its MCP process\nupdates the server separately from refreshing this skill-only plugin.\n\n## What is intentionally not done\n\n- Automation attachment to the currently open regular Chrome window\n- Silent full cookie jar import\n- Claiming a page is authenticated without observing it after seed/login\n\n## Related\n\n- [packages/chrome-companion/README.md](../packages/chrome-companion/README.md)\n- [ACT-001 surface adapter architecture](decisions/ACT-001-surface-adapter-architecture.md)\n- [plugins/action-browser/skills/action-browser/SKILL.md](../plugins/action-browser/skills/action-browser/SKILL.md)\n\n## Browser request deadlines\n\n`browser_open.waitMs` is the total operation budget (15,000 ms by default),\nincluding profile selection, Chrome startup, HTTP/CDP connection, navigation,\nand readiness checks. Zero fails immediately without opening a tab. Expiry\nreturns an MCP tool error and cancels pending transport work; increasing\n`waitMs` does not remove the 10-second limit on individual HTTP/CDP operations.\n\nConfigure the MCP server as `command: \"lattices\", args: [\"mcp\"]`, no path, so it\ncannot be pinned to a stale checkout or plugin cache. On the next MCP process\nstart, `initialize` reports the lattices version, and `tools/list` exposes\n`browser_open.mode`, `profile`, and `newTab`. Existing MCP processes retain their\nloaded implementation until reloaded.\n\nThe `lattices` launcher resolves Bun from `LATTICES_BUN_BIN`, PATH, or the\nstandard user and Homebrew locations, so it survives the bare system PATH a\nGUI-launched harness hands it. An explicit invalid override fails with an\nactionable error.\n", "content": "How Action gives an agent a **browser identity** without taking over your daily\nChrome, and which kind of control you get in each case.\n\n## Three browsers, two kinds of control\n\nWhen an agent says \"I'll open it in Chrome\", it can mean one of three things.\nThey are not interchangeable.\n\n| | What it is | What the agent can do |\n|---|---|---|\n| **1. Your regular Chrome** | The browser you use all day. Your real profiles (`Default`, `Profile 1` / \"Work\"), your tabs, history, extensions, and logins. | **Open a URL, and that's it.** `browser_open { mode: \"regular\" }` is a visible handoff with no DevTools attachment. To make the agent *act* there, use Action's native macOS control: screen capture + accessibility (`action.observe.snapshot`, `action.resolve.target`, `action.act.execute`). |\n| **2. An Action browser** | A real, non-headless Chrome that Action owns, on its own user-data-dir under `~/Library/Application Support/Action/ChromeProfiles/`. The default identity `agent-browser` is blank and signed into nothing. | **Full DOM control** over CDP: `browser_snapshot`, `browser_click`, `browser_fill`, `browser_resize`, `browser_screenshot`, `browser_tabs`. |\n| **3. An Action browser identity seeded from one of your Chrome profiles** | The same Action-owned Chrome under a name you choose (`work`, `mira`, …), carrying cookies copied from one of your real profiles for an explicit allowlist of domains. | **Full DOM control, on sites you're already signed in to.** |\n\nThe tradeoff in one line: **your regular Chrome has your session but only\nscreen-and-accessibility control; an Action browser has DOM control but starts\nas a stranger.** Seeding (option 3) is how you get both.\n\nThis is a boundary Action does not try to cross. [Chrome 136 and later ignore\nremote-debugging switches for the default personal data\ndirectory](https://developer.chrome.com/blog/remote-debugging-port), and Action\ndoes not attempt to bypass it. Pointing automation at your everyday user-data-dir\nwould also fight Chrome's directory lock and hand an agent your full history,\nextensions, and sessions by accident.\n\n### Picking one\n\n- The user wants to *look* at something in their own browser → **regular**.\n- The agent needs to click through a public site → **Action browser**.\n- The agent needs to click through a site you're logged into → **seeded identity**.\n- The agent needs to act inside your real signed-in session, and cookies aren't\n enough (passkeys, SSO device trust, an extension's state) → **regular Chrome\n plus Action's native observe/act**, not the browser tools.\n\n## The canonical example: seed \"Profile 1\" into `work`, then drive it\n\nYour Chrome profile *directory* names are `Default`, `Profile 1`, `Profile 2`, …\nand are not the display names you see in Chrome's avatar menu. A browser you\ncall \"Work\" is usually the `Profile 1` directory. Map them first, then seed.\n\nFrom an agent, over MCP:\n\n```text\nbrowser_import_cookies { listSourceProfiles: true }\n# -> [{ dir: \"Default\", name: \"Personal\" }, { dir: \"Profile 1\", name: \"Work\" }, ...]\n\nbrowser_import_cookies { into: \"work\", source: \"Profile 1\", domains: [\"github.com\"] }\n# dry run: lists exactly which cookies would be copied\n\nbrowser_import_cookies { into: \"work\", source: \"Profile 1\", domains: [\"github.com\"], confirm: true }\n# writes them into the Action identity `work`\n\nbrowser_open { url: \"https://github.com/notifications\", profile: \"work\" }\nbrowser_snapshot\nbrowser_screenshot\n```\n\nThe same thing from the terminal:\n\n```bash\n# map display names to profile directories\nbun run chrome:companion:import:cookies -- --list-profiles\n\n# dry run\nbun run chrome:companion:import:cookies -- list --into work --source \"Profile 1\" --domains github.com\n\n# write\nbun run chrome:companion:import:cookies -- import --into work --source \"Profile 1\" --domains github.com --confirm\n```\n\nAfter that, `work` is an ordinary Action identity: `browser_use_profile\n{ profile: \"work\" }` or `browser_open { url, profile: \"work\" }`, and every DOM\ntool works against it.\n\n## Named identities\n\nIdentities are created on first use: any unused name gives you a fresh blank\none. Shipped conventions:\n\n- `agent-browser`: the default blank agent identity\n- `work`: seeded from your work Chrome profile, per the example above\n- `mira`: the creative/Midjourney identity used by the companion tooling\n\nCreate and prepare one with the companion extension loaded:\n\n```bash\n# from the Action product root (`products/action`)\nbun run chrome:companion:profile -- setup work\n```\n\nThis builds the companion extension, creates the profile directory, opens\n`chrome://extensions` in that profile, and reveals `packages/chrome-companion/dist`\nso you can **Load unpacked** once.\n\nLater:\n\n```bash\nbun run chrome:companion:profile -- launch work\nbun run chrome:companion:profile -- check work\nbun run chrome:companion:profile -- path work\n```\n\nEnvironment aliases (shared between companion tooling and Action Browser MCP):\n\n| Variable | Meaning |\n|----------|---------|\n| `ACTION_BROWSER_PROFILE` / `ACTION_CHROME_COMPANION_PROFILE` | Active identity name |\n| `ACTION_BROWSER_PROFILE_DIR` / `ACTION_CHROME_COMPANION_PROFILE_DIR` | Absolute user-data-dir override |\n| `ACTION_BROWSER_PROFILE_ROOT` / `ACTION_CHROME_COMPANION_PROFILE_ROOT` | Root for named identities |\n| `ACTION_BROWSER_DEBUG_PORT` | CDP port (MCP default `9334`) |\n| `ACTION_ROOT` | Action product root (`/products/action`); cookie tools use it when the MCP process starts elsewhere |\n\n## Cookie seeding\n\nCopy **selected** cookies from a regular Chrome profile into an Action identity:\n\n```bash\n# list your Chrome profiles (directory + display name)\nbun run chrome:companion:import:cookies -- --list-profiles\n\n# list Action identities\nbun run chrome:companion:import:cookies -- --list-action-profiles\n\n# dry-run\nbun run chrome:companion:import:cookies -- list --into work --domains github.com\n\n# write\nbun run chrome:companion:import:cookies -- import --into work --domains github.com --confirm\n\n# narrow to specific cookies\nbun run chrome:companion:import:cookies -- import --into mira \\\n --domains midjourney.com \\\n --only cf_clearance \\\n --confirm\n```\n\nOr via the profile CLI:\n\n```bash\nbun run chrome:companion:profile -- import-cookies import --into work --domains github.com --confirm\n```\n\nNotes:\n\n- Requires a Keychain allow for Chrome Safe Storage only when decrypting values;\n the import path copies encrypted blobs as-is for same-machine use.\n- Prefer domain allowlists over full-jar dumps.\n- Cookies are not a complete identity (localStorage / passkeys may still need a\n one-time interactive login in the Action identity).\n\n## Chrome companion (generic extension)\n\nPackage: `packages/chrome-companion`\n\n- Manifest V3 extension with DOM observe / resolve / act helpers\n- In Action versions before 0.4.0, a localhost bridge on `http://127.0.0.1:4321` (WebSocket for the extension)\n- One-time **Load unpacked** per Action identity (`dist/` after build)\n\nAction 0.4.0 uses Chrome native messaging instead\nof the extension's TCP connection. The registered native host connects to\n`~/.lattices/run/action-chrome.sock` with `action-chrome.token`. Use the matching\nextension, native host, and Action runtime, including for a local installation.\nRegister the host for the exact extension ID and Chrome user-data directory;\nregistration for regular Chrome does not cover a separate Action identity.\n\n```bash\nbun run chrome:companion:build\nbun run chrome:companion:bridge\nbun run chrome:companion:health\nbun run chrome:companion:profile -- setup work\n```\n\nChrome Stable often ignores `--load-extension`; the reliable path is manual\nunpacked install inside the Action-owned identity.\n\n## Action browser MCP\n\nServer: `bin/mcp/toolsets/browser/index.ts` in the lattices repo root. The\nbrowser tools are a toolset inside the lattices MCP, served by `lattices mcp`.\nSee [the Lattices MCP doc](/lattices/docs/mcp).\n\n### Tools\n\n| Tool | Purpose |\n|------|---------|\n| `browser_profiles` | List Action identities, the active one, and the three-surface policy |\n| `browser_use_profile` | Switch the active identity (`agent-browser`, `work`, `mira`, …) |\n| `browser_profile_info` | Active path, cookie readiness, companion hints |\n| `browser_import_cookies` | Dry-run or confirm a seed from a regular Chrome profile |\n| `browser_companion_status` | Extension dist + bridge health |\n| `browser_open` | Open in an Action browser, or `mode: \"regular\"` for an open-only handoff |\n| `browser_tabs` / `browser_snapshot` / `browser_click` / `browser_fill` / `browser_screenshot` / `browser_close` | DOM automation via CDP, Action browsers only |\n| `browser_console` | Read console messages from an Action browser tab |\n| `browser_resize` | Set a tab to an explicit viewport width and height for responsive checks; `target: \"window\"` resizes the real Chrome window instead |\n\n### Two MCP surfaces\n\n- **Native runtime MCP** (`action`): observe / resolve / act / record on macOS\n through Action.app. This is what controls your *regular* Chrome window, and\n every other native app, via screen capture and accessibility.\n- **Browser MCP** (the `browser` toolset of `lattices mcp`): Action-owned Chrome\n identities plus DOM-level tools over CDP.\n\nRegister it. The entry names the `lattices` binary and no path, so a repo move or\na version bump cannot break it:\n\n```bash\nclaude mcp add lattices -s user -- lattices mcp\n```\n\nTo pin a default identity, add the environment variable to that same entry:\n\n```bash\nclaude mcp add lattices -s user -e ACTION_BROWSER_PROFILE=work -- lattices mcp\n```\n\n`ACTION_ROOT` is only needed for the Chrome Companion paths below, and only when\nthe tools run from a published `lattices` install rather than a checkout.\n\n### Agent workflow\n\n```text\nbrowser_profiles # what identities exist\nbrowser_use_profile { profile: \"work\" } # pick one\nbrowser_open { url: \"https://github.com\" } # drive it\nbrowser_snapshot\nbrowser_screenshot\nbrowser_companion_status # optional richer DOM path\nbrowser_close { scope: \"browser\" } # when the task is done\n```\n\nTo open a URL in the user's own Chrome instead:\n\n```text\nbrowser_open { url: \"https://github.com\", mode: \"regular\" }\n```\n\nRegular mode is deliberately not controllable, and the result says so:\n`controlAvailable: false`, plus the native control path. Snapshot, click, fill,\nand screenshot continue to target the Action browser.\n\n### Responsive breakpoints\n\n`browser_resize` gives an already-open tab an explicit viewport, so a narrow\nlayout can be captured instead of assumed.\n\n```text\nbrowser_open { url: \"http://localhost:3000\" }\nbrowser_resize { width: 1440, height: 900 }\nbrowser_screenshot\nbrowser_resize { width: 390, height: 844, mobile: true, matchMedia: [\"(max-width: 767px)\"] }\nbrowser_screenshot\nbrowser_resize { reset: true }\n```\n\nTwo targets, deliberately not interchangeable:\n\n| | What moves | Cost |\n|---|---|---|\n| `target: \"tab\"` (default) | CSS emulation for that one tab | Exact to the pixel, other tabs unaffected, fully reversible |\n| `target: \"window\"` | The real Chrome window hosting the tab | Every tab in that window moves, the display is a hard ceiling, and the tab's emulated viewport is dropped first |\n\nA device-metrics override belongs to the CDP session that set it, and Action\nBrowser opens a fresh session per tool call. Two consequences shape the design:\n\n- The server keeps the requested viewport in memory and re-applies it on every\n session, so a resize survives the later `browser_screenshot` that is the whole\n point of it. It is per-tab, in-memory only, and dropped when the tab or the\n browser closes. Nothing is written to the profile.\n- `reset` cannot simply call `clearDeviceMetricsOverride`: a foreign session's\n clear is accepted and does nothing, leaving Chrome stuck at the stale size. The\n reset re-sets the metrics to adopt ownership first, then clears them.\n\n`deviceScaleFactor` defaults to `1`, which makes screenshot pixels equal CSS\npixels: a 390-wide viewport returns a 390px-wide PNG. Read `exact` in the reply\nbefore trusting a capture: a page with a hard `min-width` can refuse to lay out\nas narrow as it was asked to.\n\n## Plugin versioning\n\nHarnesses cache the Action Browser skill and interface metadata by plugin\nversion. Bump all seven manifests together when that metadata changes:\n\n```bash\nbun run plugin:version # check that every manifest agrees\nbun run plugin:version -- 0.4.1 # set them all\n```\n\nThe check covers the root and Action product marketplace/Kimi manifests, plus\nthe browser plugin's Claude, Codex, and Kimi manifests. Reinstall or refresh the\nplugin in each harness so its cached skill is replaced.\n\nThe plugin no longer carries an MCP server or a `SERVER_VERSION`. Browser tools\nship with the lattices CLI; updating that CLI and restarting its MCP process\nupdates the server separately from refreshing this skill-only plugin.\n\n## What is intentionally not done\n\n- Automation attachment to the currently open regular Chrome window\n- Silent full cookie jar import\n- Claiming a page is authenticated without observing it after seed/login\n\n## Related\n\n- [packages/chrome-companion/README.md](../packages/chrome-companion/README.md)\n- [ACT-001 surface adapter architecture](decisions/ACT-001-surface-adapter-architecture.md)\n- [plugins/action-browser/skills/action-browser/SKILL.md](../plugins/action-browser/skills/action-browser/SKILL.md)\n\n## Browser request deadlines\n\n`browser_open.waitMs` is the total operation budget (15,000 ms by default),\nincluding profile selection, Chrome startup, HTTP/CDP connection, navigation,\nand readiness checks. Zero fails immediately without opening a tab. Expiry\nreturns an MCP tool error and cancels pending transport work; increasing\n`waitMs` does not remove the 10-second limit on individual HTTP/CDP operations.\n\nConfigure the MCP server as `command: \"lattices\", args: [\"mcp\"]`, no path, so it\ncannot be pinned to a stale checkout or plugin cache. On the next MCP process\nstart, `initialize` reports the lattices version, and `tools/list` exposes\n`browser_open.mode`, `profile`, and `newTab`. Existing MCP processes retain their\nloaded implementation until reloaded.\n\nThe `lattices` launcher resolves Bun from `LATTICES_BUN_BIN`, PATH, or the\nstandard user and Homebrew locations, so it survives the bare system PATH a\nGUI-launched harness hands it. An explicit invalid override fails with an\nactionable error." }, { "id": "action:agent-layer", "slug": "agent-layer", "product": "action", "kind": "doc", "catalogKind": "guide", "current": true, "title": "Agent layer", "description": "Work an app or Action-owned browser on a prepared private virtual display", "order": 6, "sourcePath": "products/action/docs/agent-layer.md", "url": "/action/docs/agent-layer", "rawUrl": "/action/docs/markdown/agent-layer.md", "markdownUrl": "/action/docs/agent-layer.md", "headings": [ { "depth": 2, "text": "Named virtual workspaces", "anchor": "named-virtual-workspaces" }, { "depth": 2, "text": "Prepare the monitor once", "anchor": "prepare-the-monitor-once" }, { "depth": 2, "text": "Check capabilities before opening", "anchor": "check-capabilities-before-opening" }, { "depth": 2, "text": "Private browser workflow", "anchor": "private-browser-workflow" }, { "depth": 2, "text": "Transport boundaries", "anchor": "transport-boundaries" }, { "depth": 2, "text": "Acceptance", "anchor": "acceptance" }, { "depth": 2, "text": "What it does", "anchor": "what-it-does" }, { "depth": 2, "text": "Primitives", "anchor": "primitives" }, { "depth": 2, "text": "Native blink acts", "anchor": "native-blink-acts" }, { "depth": 2, "text": "Feed", "anchor": "feed" }, { "depth": 2, "text": "Operator controls", "anchor": "operator-controls" }, { "depth": 2, "text": "Lifetime", "anchor": "lifetime" }, { "depth": 2, "text": "Requirements and limits", "anchor": "requirements-and-limits" } ], "tokensEstimate": 5714, "frontmatter": { "title": "Agent layer", "description": "Work an app or Action-owned browser on a prepared private virtual display", "order": 6 }, "markdown": "---\ntitle: Agent layer\ndescription: Work an app or Action-owned browser on a prepared private virtual display\norder: 6\n---\n\nAction's agent layer attaches an app or Action-owned browser to a prepared,\npersistent virtual monitor. The monitor and each task have separate lifetimes.\nUse MCP for the complete browser workflow.\n\n## Named virtual workspaces\n\nThe multi-workspace candidate adds independent virtual displays. This is not a\nhost-isolation acceptance result. Validate the matched native app and runtime\non the target host before using it for desktop-preserving work.\n\nUse `--workspace ` on each CLI layer command. Use `workspace` on each MCP\nlayer call. Omission selects `default`; it never selects the last workspace\nused. Names contain 1–32 lowercase letters, digits, or hyphens and start with a\nletter. The existing unnamed display remains the `default` workspace.\n\nEach workspace has its own display owner, task, browser profile, preview,\noperator controls, and recording. Closing a task leaves its display connected.\nOpening a task never replaces an active task. An MCP caller cannot close another\ncaller's task. Cross-process transaction locks protect each workspace's lifecycle\nand control files; concurrent mutations fail without overwriting another request.\nAn app process can belong to only one active workspace. Use separate browser\nprocesses for browser tasks on different workspaces.\n\nPreparation changes the macOS display topology. Arrange a setup interval before\nrunning these commands. Build only in the native development workspace; copy a\nmatched bundle and runtime to installation targets.\n\n```bash\nlats action layer prepare --workspace stage --width 1920 --height 1080\nlats action layer prepare --workspace agent --width 1440 --height 900\nlats action layer list\n```\n\nAfter preflight and host acceptance, attach tasks through MCP:\n\n```text\nlayer_open {workspace:\"stage\", previewFull:true}\nlayer_open {workspace:\"agent\", browser:true, url:\"https://lattices.dev/action\", previewWorkspace:\"stage\"}\nlayer_snapshot {workspace:\"agent\"}\nlayer_snapshot {workspace:\"stage\", full:true}\nlayer_record {workspace:\"stage\", action:\"start\", out:\"/absolute/path/stage.mov\"}\nlayer_record {workspace:\"agent\", action:\"start\", out:\"/absolute/path/agent.mov\"}\nlayer_go {workspace:\"agent\", url:\"https://lattices.dev/docs/agents\"}\nlayer_record {workspace:\"agent\", action:\"stop\"}\nlayer_record {workspace:\"stage\", action:\"stop\"}\n```\n\n`previewWorkspace` places the agent preview on the prepared stage display. The\ndisplays are peers, not nested virtual machines. The outer recording captures\nthe stage and embedded preview. The inner recording captures the agent display.\nVerify both recording completion results and movie files before reporting a take\nas finished. An empty stage task does not populate or arrange application windows.\n\nSet `previewFull:true` on the stage's `layer_open` or `layer_pip` call to show its\nentire display, including the embedded agent preview. Otherwise the preview follows\nthe subject windows and can crop out the embedded preview. CLI equivalent:\n`lats action layer pip --workspace stage --preview-full`. Each preview identifies\nits workspace and host. Preview framing does not change full-display recording.\n\nUse `layer_pip {workspace:\"agent\", previewWorkspace:\"stage\"}` to move an existing\npreview. CLI equivalent:\n\n```bash\nlats action layer pip --workspace agent --preview-workspace stage\n```\n\nAction rejects self-previews and recursive preview chains. A preview that loses\nits destination must not fall back to a physical screen. A display cannot be\nreleased while another workspace previews onto it. Close the dependent agent\ntask before releasing the stage display:\n\n```text\nlayer_close {workspace:\"agent\"}\nlayer_close {workspace:\"stage\"}\n```\n\n```bash\nlats action layer release --workspace agent\nlats action layer release --workspace stage\n```\n\nBrowser profiles are separate for each named workspace. A busy debugging port\nis reusable only when its process matches the expected browser executable,\nprofile path, and port. A mismatch fails without adopting or stopping that\nprocess. Port overrides for named workspaces use\n`ACTION_LAYER_BROWSER_PORT_`, with uppercase letters and underscores\nin place of hyphens. `ACTION_LAYER_BROWSER_PORT` applies to `default` only.\n\nMultiple virtual displays do not create independent macOS keyboards or focus.\nBrowser interactions use CDP. Native input uses a shared exclusive input lease\nand can still cause temporary focus changes. Validate physical display modes,\narrangement, pointer, frontmost app, and unrelated window frames throughout the\ntwo-workspace lifecycle. Also verify independent pause, stop, recording, crash\ncleanup, and preview-destination loss. Passing a one-display test does not prove\nthat the two-display configuration is accepted.\n\n## Prepare the monitor once\n\nProvision the monitor explicitly when the user is ready for a display topology\nchange. Preparation creates the virtual display and may change macOS modes,\narrangement, or window placement; it does not promise an unchanged desktop.\nCheck capabilities and quarantine first. Preparation never clears quarantine.\n\n```bash\nlats action layer display-status\nlats action layer prepare --width 1440 --height 900\n```\n\n`display-status` is read-only. `prepare` reuses an existing live monitor when\nthe requested size matches, or when no size is supplied. A new monitor defaults\nto 1440×900. A different size requires explicit release and preparation; an\nordinary task must not silently resize or recreate the monitor.\n\nOnce prepared, `layer open` attaches a task without connecting another display.\n`layer close` restores that task's windows and leaves the monitor connected.\nWhen the user wants to disconnect the monitor, use `lats action layer release`;\nit refuses while a task is active, so close the task first with `layer close`. Release can\nalso change the desktop topology. A restart, crashed monitor owner, or missing\nmonitor requires explicit preparation again, not automatic recovery during\nthe next task.\n\n## Check capabilities before opening\n\nRun `lats action capabilities`, then call `capabilities` on the active MCP\nconnection. Check its `tools/list` for `layer_open`, `layer_snapshot`,\n`layer_click`, `layer_type`, `layer_js`, and `layer_close`.\nUse `lats action layer display-status` to confirm a prepared monitor is active.\nIf it is missing, stop the task and report the preparation requirement.\n\nCompare the orchestration source digest, signed native build manifest, and\nnative executable hash with the intended deployment. A version match or an\ninstaller message saying the app is current does not prove feature parity.\nFor a standard packaged runtime, require `runtimeParity.state` to be\n`matched-source-digest`; resolve `source-digest-mismatch` or `unverified`\nbefore use. This compares bundled artifacts; it does not verify a running app.\nAn overridden native host is unverified until its own artifact is checked.\nReconnect MCP after changing its runtime or configuration, then repeat discovery.\n\nCheck `safety.privateLayer` before any display or browser is opened. If blocked,\nstop and report its reason and evidence path. Do not clear the quarantine or\nretry through the CLI/native host. Do not replace requested isolation with\nordinary background browser opening; it can still activate or reflow the desktop.\n\nAn absent quarantine means only that no host block is recorded. It does not\nprove acceptance. Inspect `acceptance.privateLayer` in the capabilities reply.\nRequire `state: \"accepted\"` for ordinary use that requires desktop preservation;\nmissing, stale, invalid, or unverified evidence means stop before opening.\nThe record lives at `~/Library/Application Support/Action/agent-layer/isolation-acceptance.json`.\nIt binds acceptance to the native executable and source, orchestration source,\nhost/OS, and physical display configuration. Capabilities checks that binding;\nit does not run display QA or certify a build automatically. Older servers\nwithout the field are unverified. See [Acceptance](#acceptance) for evidence\nrequirements. Controlled QA requires authorization for that host and a\npreserved rollback and layout; do not manufacture a passing record.\n\n**Known failed acceptance, 2026-10-11:** the retained Air trace shows the\nphysical display change from 2560×1664 to 1470×956 during\n`CGVirtualDisplay.applySettings`, before browser setup and Action's positioning\ntransaction. Air remains quarantined. The guarded build prevents display\ncreation; it does not repair the display defect. A successful close or restored\nfocus does not pass the preservation requirement.\n\n**Mini foreground check, 2026-10-11:** retained desktop traces from both the\nPiP and no-PiP browser runs show no physical-display, pointer, or unrelated\nwindow-frame drift. Both show the frontmost process changing to macOS\n`UserNotificationCenter` during `layer_open`, before the first snapshot or\npage input. That foreground check failed and is confounded by system\nnotification UI; it is not evidence of a Chrome/CDP input defect or an\nisolation pass.\n\nThose traces exercised the earlier per-task display-creation path. Keeping a\nmonitor connected removes that hotplug step from ordinary task opening; it\ndoes not retroactively pass those runs.\n\n**Bounded arts-mini task pass, 2026-10-11:** two task sessions reused the same\nprepared display and passed strict, continuous 50 ms desktop-preservation\nmonitoring across open, interaction, and close. This evidence covers that host\nand tested build, which predates the new persistent viewer controls. It does\nnot certify the current controls build, Air, cold provisioning, monitor-owner\nrestart, or release. Provisioning, task attachment, and release need distinct\nevidence. Acceptance for ordinary tasks uses a baseline taken with the\npersistent monitor already prepared; a passing run is not a current matching\nacceptance record by itself.\n\nBuild and diagnose on the development host. `native:doctor` rebuilds and signs;\nit is not a read-only preflight and must not be used to repair an installation\ntarget implicitly. Install a matched native app and orchestration runtime with\nrollback instead. For a packaged runtime, set `ACTION_ROOT=/products/action`,\nnot the extraction root. Keep the archive's sibling `bin/` and `products/`\ndirectories together so the runtime can import `bin/host-pairing.ts`.\n\n## Private browser workflow\n\nCurrent MCP servers advertise underscore names. Use the active `tools/list`;\ndotted `action.*` names are optional compatibility aliases.\n\n```text\nlayer_open { browser: true, url: \"https://example.com\", task: \"Read the page\", pip: true }\nlayer_snapshot {}\nlayer_click { label: \"More\" }\nlayer_type { text: \"Search text\" }\nlayer_js { code: \"document.title\" }\nlayer_go { url: \"https://example.com/next\" }\nlayer_close { outcome: \"done\", summary: \"Read the page\" }\n```\n\n`browser: true` selects Action-owned Chrome. Browser page interaction uses\nCDP exclusively. If CDP is missing, a label or selector cannot be resolved,\ncoordinates reach browser chrome or leave the viewport, or a shortcut is not\nsupported by the page route, the operation fails without native input.\nUse `layer_go` for navigation instead of `cmd+l`. This input contract does not\nestablish that virtual-display creation preserves the physical desktop.\nCDP-only input also does not prevent page JavaScript from requesting popups\nor focus, system permission/notification UI from appearing, or display\ncreation from changing the desktop. Acceptance must observe those effects.\n\nRetain `session` from `layer_open`: `mode`, `pid`, `displayId`, `startedAt`,\nand `browserPid` when present. Action results include the session and execution\nroute (`devtools` with `pageTargetId`, or `blink` with a native `host`). A\nreplacement layer invalidates the binding; do not reuse an earlier snapshot\nor act against the replacement. Check `layer_status` after a connection loss\nor timeout before retrying. Reobserve before repeating an operation whose\noutcome is uncertain.\n\nUse the layer's own snapshot and primitives. `layer_open` begins a drive lease\nunless `leaseId` supplies one; it returns that lease and `layer_close` releases\nthe lease it created. A supplied lease remains the caller's responsibility.\nDo not begin a second drive for the layer or use a desktop snapshot as its\ncoordinate reference. If setup fails, MCP attempts to close the layer it created\nand reports any unconfirmed cleanup. Inspect `layer_status` before retrying.\nCheck `layer_status` before the first open too: opening fails while the\nworkspace already has an active task. Close it explicitly first, and do not\nclose another caller's task without user authority.\n\nFor native apps, open `{ bundleId: \"com.apple.calculator\", width: 1280,\nheight: 800, pip: true }`. Select `windowId` or `windowTitle` to borrow only\nmatching windows. Native input can temporarily borrow focus or the pointer;\nthat path does not guarantee continuous focus isolation.\n\n## Transport boundaries\n\n| Surface | Layer support |\n| --- | --- |\n| Action MCP | Complete `layer_*` workflow, including `browser: true`, URL navigation, input, and JavaScript |\n| CLI | Monitor `layer prepare`, `display-status`, `release`; task `open`, `status`, `snapshot`, `pip`, `record start/stop`, `close` |\n| Native host | Native display/app primitives; no MCP browser orchestration |\n| Action WebSocket | Raw agent RPC catalog; route private-layer work to MCP |\n| Lattices daemon | Workspace window groups; route private-layer work to Action MCP |\n\nCLI `layer open` accepts `--bundle-id` or `--pid`, `--width`, `--height`,\n`--window-id` or `--window-title`, and `--pip`. It rejects `--browser`, `--url`,\nunknown flags, and unsupported interaction verbs with an MCP routing error.\nThe CLI task lifetime is detached; always close it when work ends. Width and\nheight on `open` must match the prepared monitor; they do not configure a new\ndisplay. Closing a task does not release the monitor.\n\n## Acceptance\n\nRead-only lookup is `lats action capabilities` or MCP `capabilities`, under\n`acceptance.privateLayer`. The native host supplies the current host and\nphysical-display context; inability to read that context leaves acceptance\nunverified. The record's matching status does not override `safety.privateLayer`\nquarantine, prove the running process matches the installed bundle, or replace\nreview of the retained evidence.\n\nThe operator-reviewed record has this schema:\n\n| Field | Required value |\n| --- | --- |\n| `schemaVersion`, `outcome` | `1`, `\"passed\"` |\n| `acceptedAt`, `expiresAt`, `reviewer` | Valid ISO timestamps and a nonempty reviewer; expired records are stale. |\n| `binding` | `executableSha256`, `nativeSourceDigest`, `orchestrationSourceDigest`, `hostFingerprint`, `osBuild`, `displayConfigurationDigest`, matching the current context. |\n| `checks` | `physicalDisplayMode`, `scale`, `arrangement`, `frontmostApp`, `pointer`, `unrelatedWindowFrames`, each `\"unchanged-throughout\"`. |\n| `phases` | `[\"open\", \"interaction\", \"close\"]` |\n| `evidence` | Nonempty list of `{ path, sha256 }` entries; absolute local artifact paths with verified hashes. |\n\nCapabilities discovers and validates this record. It does not write one.\nCreate or renew it only from reviewed, authorized acceptance evidence.\n\nFor ordinary task acceptance, prepare the monitor first under authorized setup\nconditions, then capture physical modes (logical and pixel sizes, scale and refresh rate),\ndisplay arrangement/mirroring, frontmost app, pointer, and unrelated window\nframes before opening. Monitor those values throughout task attachment, interaction,\nand teardown. Verify a layer snapshot and a real browser interaction, then\nconfirm closure and window restoration. Fail on the first unexpected change.\nA final sample that matches the baseline cannot erase an earlier failure.\nTest initial preparation, monitor-owner restart, and explicit release\nseparately; those operations change display topology and must not be folded\ninto a claim that ordinary tasks never connect or disconnect a display.\n\n## What it does\n\n- Attaches to the prepared monitor and moves the subject app's windows onto it. The original frame of each window is recorded. The persistent monitor owner holds the `CGVirtualDisplay` independently of the task.\n- `windowId` (the `kCGWindowNumber`) or `windowTitle` (a case-insensitive substring) moves only the matching windows. Use it to borrow one window of an app the user is also in, such as a browser.\n- Starts one ScreenCaptureKit stream on the layer display and keeps it running for the layer's life. The viewer, snapshots and recordings all come off that stream (see [Feed](#feed)).\n- Launches the subject in the background if it isn't running, and asks an app with no windows for one (the reopen a Dock click sends). Background launch requests no activation; this is not an isolation acceptance result.\n- `url` (a URL, a bare host read as `https`, or a file path) opens in the subject without activating it, once its windows are on the layer. It needs `bundleId` or `pid`, and can't be combined with `windowId` or `windowTitle`.\n- Adopts windows the subject opens while the layer is up, such as a `cmd+n` window, a pop-up, or an `open-app` with `input.url`. They move onto the layer as they appear, cascaded. A layer that borrowed one window by `windowId` or `windowTitle` adopts nothing more.\n- Shows the PiP viewer unless `pip` is `false`.\n- Writes a state file: display id, global bounds (top-left origin, points), PiP state, and the moved windows with their original frames.\n- Keeps one task per workspace. Opening while a task is active fails; close the task first.\n\n## Primitives\n\nThe layer's own tools are the short way to work it. Look, then act on what you saw:\n\n```\nlayer_open { bundleId: \"com.apple.Safari\", url: \"news.ycombinator.com\" }\nlayer_snapshot {} → p.png, 950×560\nlayer_click { x: 475, y: 196 } pixels in that snapshot\nlayer_click { label: \"new\" } or a control by its label\nlayer_type { text: \"lobste.rs\\n\" } a trailing \\n submits\nlayer_press { key: \"cmd+l\" }\nlayer_drag { from: [95, 224], to: [570, 252] }\nlayer_scroll { x: 475, y: 280, dy: -600 }\nlayer_close {}\n```\n\nPoints are pixels in the last `layer_snapshot` image, read through its crop and scale, so an agent never adds the layer's origin or works out the display's scale. `space: \"layer\"` takes points on the layer display instead. A point outside the last snapshot is an error, and so is a point with no snapshot taken yet.\n\nBrowser primitives use the bound CDP page and return `route: \"devtools\"` with\n`pageTargetId`. Native primitives use `action.act.execute` and return\n`route: \"blink\"` with the native `host`. Both report the layer session; input\nprimitives normally return a fresh snapshot (`look: false` skips it). A lease\nis taken for the caller when none is passed. The native routing below is not a\nfallback for a browser session.\n\n## Native blink acts\n\nWhile a layer is up, `action.act.execute` routes these acts as blink acts:\n\n| Act | Host command | When |\n|---|---|---|\n| `click` with a point | `blink-click` | the point is on the layer display |\n| `type` | `blink-type` | no accessibility label path applies |\n| `press-key` | `blink-key` | always |\n| `drag` | `blink-drag` | both points are on the layer, and no `filePath` |\n| `scroll` | `blink-scroll` | the point is on the layer |\n\nAccessibility comes first. A target with a bundle id and label is pressed or set through accessibility and never blinks. `blink-click` hit-tests its point before it touches the pointer: if the element there is a button, checkbox, radio, pop-up, menu item, disclosure triangle, link or tab that takes `AXPress`, it is pressed and the result says `via=ax`. A text field, text area, combo box or search field there is focused through accessibility instead of clicked into. Nothing moves and focus does not change.\n\n`blink-type` works the same way. It inserts the text at the caret of the app's focused text field through accessibility (`via=ax`), and only borrows focus for keystrokes (`via=keys`) when the field doesn't take it, is secure, or a `delayMs` cadence is requested. Web fields usually take keystrokes. When the field's value is readable, the blink holds focus until the text shows up in it (`via=keys verified`), because a web view consumes keys in its own content process after the app's main loop has already drained. A trailing newline in `type` text (`\"lobste.rs\\n\"`) splits off: the text goes in through accessibility and the field is submitted with `AXConfirm`, a Return that moves nothing. `press-key` tries accessibility too: a chord that is a menu item's shortcut (`cmd+l`, `cmd+n`) presses that menu item (`via=ax menu=\"Open Location…\"`), and a bare Return or Escape in a text field confirms or cancels it. Anything else borrows focus for the keystroke (`via=keys`).\n\n`press-key` takes a chord in `input.key`, written however the agent reads it: `\"return\"`, `\"cmd+l\"`, `\"Cmd-Shift-T\"`, `\"⌘⇧T\"`, `[\"cmd\", \"l\"]` in `input.keys`, or `key` plus `input.modifiers`. Aliases such as `enter`, `esc`, `backspace` and `ArrowDown` map to the host's names. An unknown key is an error that points at `type`, never text typed one character at a time.\n\nThe pointer is the fallback: no pressable element at the point, a requested `holdMs`, or `--pointer` on the host command. A pointer blink saves the cursor position and the frontmost app, clicks, and restores both within tens of milliseconds. Its result says `via=pointer`.\n\n`type` and `press-key` land in the app's focused window. If that window is off the layer, the blink raises the app's layer window first, without activating the app, and refuses if focus still isn't on the layer. Keystrokes never reach a window on the user's screen.\n\n`type` and `press-key` target `input.bundleId` or `input.pid` if given, else the layer's app. Accessibility paths (`press-accessibility-element`, `set-accessibility-value`) do not change. A click outside the layer display uses `click-point`.\n\nDrag and scroll have no accessibility equivalent and always use the pointer. It leaves the user's screens for the length of the gesture (200 ms for a drag by default; an instant for a scroll) and comes back. Scroll wheel events activate nothing. A drag's mouse-down activates the app it lands in for up to a second, and focus is handed back afterwards.\n\nBlink acts report the `blink` tier. A background drive lease allows them. They do not show the pointer-focus countdown.\n\n## Feed\n\nThe layer captures its own display, not the subject app. A stream that targets an app makes macOS badge that app's windows as shared, and the badge is drawn into the window, so it would show in every frame. The framing to the subject's windows happens downstream, by crop. The stream only delivers frames when something on the layer changes, so an idle layer costs next to nothing.\n\n- **Viewer.** The PiP grows out of the screen corner on its first frame and frames the subject's windows, following them as they move and resize. Drag the header to move it; double-click the header to enlarge it, and double-click again to restore its size. Hide leaves the layer running; `layer pip` / `layer_pip` brings it back.\n- **State.** One dot in the viewer's bottom-right corner: solid coral while the agent can act, breathing while a take records, hollow while paused.\n- **Marks.** The host posts where native acts landed and the viewer rings it: a circle for a click, an outline around the focused field for typing and keys. Accessibility paths can avoid pointer movement; pointer and key fallbacks may temporarily borrow pointer or focus.\n- **Note.** The latest `action.drive.note` shows along the bottom edge for about 12 seconds.\n- **Snapshot.** `layer snapshot` / `layer_snapshot` writes a PNG from the latest frame: the subject's windows by default, one window with `windowId`, or the whole layer with `full`. Nothing is captured on request, so it answers in tens of milliseconds. `unchangedMs` is how long the layer has been still, not how stale the picture is.\n- **Recording.** `layer record start|stop` / `layer_record` adds a recording output to the running stream, so the take starts on the next frame. It records the whole layer display at its native size. One take at a time; closing the layer finishes a take in progress.\n\n## Operator controls\n\nThe viewer has a persistent hostname header and native controls outside the captured image:\n\n- **Pause / Resume.** Pauses layer input dispatch. While paused, `state.json` has `\"paused\": true`; observations remain available. Resume is an operator action. Events or page scripts already dispatched cannot be undone.\n- **Stop.** Pauses input, records a session-specific `handoff.json`, and ends the layer without activating the target app. This stops the layer session, not the agent's entire harness or unrelated work.\n- **Agent.** Looks for the opener's app on this host. Action records its process chain at `open` (`--owner-pids`), activates the nearest app in that chain, and otherwise asks Lattices (`terminals.search`, then `window.focus`) which local terminal window shows it. If nothing resolves, the viewer beeps. This does not return from an SSH host to a conversation on another machine.\n- **Take over.** Records a session-specific handoff, ends the layer, and activates the target app. Borrowed windows return to their recorded frames; windows created on the layer are closed, subject to unsaved-changes prompts. This is not an in-place remote-control mode.\n- **Hide.** Hides only the viewer; the layer keeps running.\n\nStop and Take over refuse further agent input during teardown and after cleanup. `layer close` does not clear the handoff. A new explicit `layer open` acknowledges it; callers must obtain fresh user authorization before opening again. The prepared monitor remains connected after either action.\n\nThese are host-local controls. An interactive viewer streamed to another machine or into a chat is not implemented.\n\nRequests go through `control.request.json` next to the state file and a `SIGUSR2`; the reply lands in `control.reply.json`.\n\n## Lifetime\n\n`layer_close` is the intended teardown. It creates the stop file and waits for the layer to put the windows back and exit. Windows that were opened after the layer went up (`born` in the state file) never sat on the user's screen, so they are closed instead of moved there. One that asks before closing (unsaved changes) stays open and goes back to the frame it appeared at. If the layer does not exit, Action sends `SIGTERM`.\n\nThis tears down the task's feed and window attachment, not the monitor.\n`layer release` stops the persistent monitor owner only after its task has been\nexplicitly closed. It refuses active tasks and incoming previews without closing\nthem. Close dependent tasks first, then release their displays.\n\nThe layer also tears down when its owner dies:\n\n- **MCP.** The layer watches the MCP server (`--parent-pid`) and restores the windows if the server exits.\n- **CLI.** The CLI exits once the layer is up, so the layer is detached. Run `layer close` to take it down when the work is done.\n\nA detached layer doesn't wait for `layer close` forever. It also comes down when:\n\n- **Its owners exit.** Every process in the opener's chain that was alive at `open` has exited.\n- **It goes idle.** 30 minutes pass with no act, no snapshot or recording request, and no use of the viewer. A recording in progress keeps the layer up. Pass `--idle-timeout ` to the host to change the lease; `0` turns it off.\n\nEither way the windows go back first, as with `layer close`.\n\n`layer status` reports the live layer. If the recorded process is gone, status removes the stale files and reports the layer as inactive.\n\nTask and monitor state live in `~/Library/Application Support/Action/agent-layer/`.\nUse `layer status` for the task and `layer display-status` for the monitor;\nan inactive task with a ready monitor is the normal state between tasks.\n\n## Requirements and limits\n\n- The feed needs Screen Recording for Action. Without it the layer still works (windows move, blink acts land), but there is no viewer, snapshot or recording; the layer's `layer.log` says `feed failed`. Recording needs macOS 15.\n- The display touches the bottom-right-most display only at its corner. A pointer pushed exactly through that corner can still cross onto it.\n- A pointer blink click on another app activates that app for a moment before focus goes back. The menu bar can flicker for that moment.\n- Blink acts refuse targets that aren't on an agent layer: a point off the layer, or an app with no window on it. `--any-display` lifts that for tests.\n- Killing a task with `SIGKILL` skips its window restoration; the persistent monitor remains. Killing the monitor owner disconnects the display, allowing macOS to relocate its windows. Neither is an ordinary task teardown or a preservation guarantee.\n", "content": "Action's agent layer attaches an app or Action-owned browser to a prepared,\npersistent virtual monitor. The monitor and each task have separate lifetimes.\nUse MCP for the complete browser workflow.\n\n## Named virtual workspaces\n\nThe multi-workspace candidate adds independent virtual displays. This is not a\nhost-isolation acceptance result. Validate the matched native app and runtime\non the target host before using it for desktop-preserving work.\n\nUse `--workspace ` on each CLI layer command. Use `workspace` on each MCP\nlayer call. Omission selects `default`; it never selects the last workspace\nused. Names contain 1–32 lowercase letters, digits, or hyphens and start with a\nletter. The existing unnamed display remains the `default` workspace.\n\nEach workspace has its own display owner, task, browser profile, preview,\noperator controls, and recording. Closing a task leaves its display connected.\nOpening a task never replaces an active task. An MCP caller cannot close another\ncaller's task. Cross-process transaction locks protect each workspace's lifecycle\nand control files; concurrent mutations fail without overwriting another request.\nAn app process can belong to only one active workspace. Use separate browser\nprocesses for browser tasks on different workspaces.\n\nPreparation changes the macOS display topology. Arrange a setup interval before\nrunning these commands. Build only in the native development workspace; copy a\nmatched bundle and runtime to installation targets.\n\n```bash\nlats action layer prepare --workspace stage --width 1920 --height 1080\nlats action layer prepare --workspace agent --width 1440 --height 900\nlats action layer list\n```\n\nAfter preflight and host acceptance, attach tasks through MCP:\n\n```text\nlayer_open {workspace:\"stage\", previewFull:true}\nlayer_open {workspace:\"agent\", browser:true, url:\"https://lattices.dev/action\", previewWorkspace:\"stage\"}\nlayer_snapshot {workspace:\"agent\"}\nlayer_snapshot {workspace:\"stage\", full:true}\nlayer_record {workspace:\"stage\", action:\"start\", out:\"/absolute/path/stage.mov\"}\nlayer_record {workspace:\"agent\", action:\"start\", out:\"/absolute/path/agent.mov\"}\nlayer_go {workspace:\"agent\", url:\"https://lattices.dev/docs/agents\"}\nlayer_record {workspace:\"agent\", action:\"stop\"}\nlayer_record {workspace:\"stage\", action:\"stop\"}\n```\n\n`previewWorkspace` places the agent preview on the prepared stage display. The\ndisplays are peers, not nested virtual machines. The outer recording captures\nthe stage and embedded preview. The inner recording captures the agent display.\nVerify both recording completion results and movie files before reporting a take\nas finished. An empty stage task does not populate or arrange application windows.\n\nSet `previewFull:true` on the stage's `layer_open` or `layer_pip` call to show its\nentire display, including the embedded agent preview. Otherwise the preview follows\nthe subject windows and can crop out the embedded preview. CLI equivalent:\n`lats action layer pip --workspace stage --preview-full`. Each preview identifies\nits workspace and host. Preview framing does not change full-display recording.\n\nUse `layer_pip {workspace:\"agent\", previewWorkspace:\"stage\"}` to move an existing\npreview. CLI equivalent:\n\n```bash\nlats action layer pip --workspace agent --preview-workspace stage\n```\n\nAction rejects self-previews and recursive preview chains. A preview that loses\nits destination must not fall back to a physical screen. A display cannot be\nreleased while another workspace previews onto it. Close the dependent agent\ntask before releasing the stage display:\n\n```text\nlayer_close {workspace:\"agent\"}\nlayer_close {workspace:\"stage\"}\n```\n\n```bash\nlats action layer release --workspace agent\nlats action layer release --workspace stage\n```\n\nBrowser profiles are separate for each named workspace. A busy debugging port\nis reusable only when its process matches the expected browser executable,\nprofile path, and port. A mismatch fails without adopting or stopping that\nprocess. Port overrides for named workspaces use\n`ACTION_LAYER_BROWSER_PORT_`, with uppercase letters and underscores\nin place of hyphens. `ACTION_LAYER_BROWSER_PORT` applies to `default` only.\n\nMultiple virtual displays do not create independent macOS keyboards or focus.\nBrowser interactions use CDP. Native input uses a shared exclusive input lease\nand can still cause temporary focus changes. Validate physical display modes,\narrangement, pointer, frontmost app, and unrelated window frames throughout the\ntwo-workspace lifecycle. Also verify independent pause, stop, recording, crash\ncleanup, and preview-destination loss. Passing a one-display test does not prove\nthat the two-display configuration is accepted.\n\n## Prepare the monitor once\n\nProvision the monitor explicitly when the user is ready for a display topology\nchange. Preparation creates the virtual display and may change macOS modes,\narrangement, or window placement; it does not promise an unchanged desktop.\nCheck capabilities and quarantine first. Preparation never clears quarantine.\n\n```bash\nlats action layer display-status\nlats action layer prepare --width 1440 --height 900\n```\n\n`display-status` is read-only. `prepare` reuses an existing live monitor when\nthe requested size matches, or when no size is supplied. A new monitor defaults\nto 1440×900. A different size requires explicit release and preparation; an\nordinary task must not silently resize or recreate the monitor.\n\nOnce prepared, `layer open` attaches a task without connecting another display.\n`layer close` restores that task's windows and leaves the monitor connected.\nWhen the user wants to disconnect the monitor, use `lats action layer release`;\nit refuses while a task is active, so close the task first with `layer close`. Release can\nalso change the desktop topology. A restart, crashed monitor owner, or missing\nmonitor requires explicit preparation again, not automatic recovery during\nthe next task.\n\n## Check capabilities before opening\n\nRun `lats action capabilities`, then call `capabilities` on the active MCP\nconnection. Check its `tools/list` for `layer_open`, `layer_snapshot`,\n`layer_click`, `layer_type`, `layer_js`, and `layer_close`.\nUse `lats action layer display-status` to confirm a prepared monitor is active.\nIf it is missing, stop the task and report the preparation requirement.\n\nCompare the orchestration source digest, signed native build manifest, and\nnative executable hash with the intended deployment. A version match or an\ninstaller message saying the app is current does not prove feature parity.\nFor a standard packaged runtime, require `runtimeParity.state` to be\n`matched-source-digest`; resolve `source-digest-mismatch` or `unverified`\nbefore use. This compares bundled artifacts; it does not verify a running app.\nAn overridden native host is unverified until its own artifact is checked.\nReconnect MCP after changing its runtime or configuration, then repeat discovery.\n\nCheck `safety.privateLayer` before any display or browser is opened. If blocked,\nstop and report its reason and evidence path. Do not clear the quarantine or\nretry through the CLI/native host. Do not replace requested isolation with\nordinary background browser opening; it can still activate or reflow the desktop.\n\nAn absent quarantine means only that no host block is recorded. It does not\nprove acceptance. Inspect `acceptance.privateLayer` in the capabilities reply.\nRequire `state: \"accepted\"` for ordinary use that requires desktop preservation;\nmissing, stale, invalid, or unverified evidence means stop before opening.\nThe record lives at `~/Library/Application Support/Action/agent-layer/isolation-acceptance.json`.\nIt binds acceptance to the native executable and source, orchestration source,\nhost/OS, and physical display configuration. Capabilities checks that binding;\nit does not run display QA or certify a build automatically. Older servers\nwithout the field are unverified. See [Acceptance](#acceptance) for evidence\nrequirements. Controlled QA requires authorization for that host and a\npreserved rollback and layout; do not manufacture a passing record.\n\n**Known failed acceptance, 2026-10-11:** the retained Air trace shows the\nphysical display change from 2560×1664 to 1470×956 during\n`CGVirtualDisplay.applySettings`, before browser setup and Action's positioning\ntransaction. Air remains quarantined. The guarded build prevents display\ncreation; it does not repair the display defect. A successful close or restored\nfocus does not pass the preservation requirement.\n\n**Mini foreground check, 2026-10-11:** retained desktop traces from both the\nPiP and no-PiP browser runs show no physical-display, pointer, or unrelated\nwindow-frame drift. Both show the frontmost process changing to macOS\n`UserNotificationCenter` during `layer_open`, before the first snapshot or\npage input. That foreground check failed and is confounded by system\nnotification UI; it is not evidence of a Chrome/CDP input defect or an\nisolation pass.\n\nThose traces exercised the earlier per-task display-creation path. Keeping a\nmonitor connected removes that hotplug step from ordinary task opening; it\ndoes not retroactively pass those runs.\n\n**Bounded arts-mini task pass, 2026-10-11:** two task sessions reused the same\nprepared display and passed strict, continuous 50 ms desktop-preservation\nmonitoring across open, interaction, and close. This evidence covers that host\nand tested build, which predates the new persistent viewer controls. It does\nnot certify the current controls build, Air, cold provisioning, monitor-owner\nrestart, or release. Provisioning, task attachment, and release need distinct\nevidence. Acceptance for ordinary tasks uses a baseline taken with the\npersistent monitor already prepared; a passing run is not a current matching\nacceptance record by itself.\n\nBuild and diagnose on the development host. `native:doctor` rebuilds and signs;\nit is not a read-only preflight and must not be used to repair an installation\ntarget implicitly. Install a matched native app and orchestration runtime with\nrollback instead. For a packaged runtime, set `ACTION_ROOT=/products/action`,\nnot the extraction root. Keep the archive's sibling `bin/` and `products/`\ndirectories together so the runtime can import `bin/host-pairing.ts`.\n\n## Private browser workflow\n\nCurrent MCP servers advertise underscore names. Use the active `tools/list`;\ndotted `action.*` names are optional compatibility aliases.\n\n```text\nlayer_open { browser: true, url: \"https://example.com\", task: \"Read the page\", pip: true }\nlayer_snapshot {}\nlayer_click { label: \"More\" }\nlayer_type { text: \"Search text\" }\nlayer_js { code: \"document.title\" }\nlayer_go { url: \"https://example.com/next\" }\nlayer_close { outcome: \"done\", summary: \"Read the page\" }\n```\n\n`browser: true` selects Action-owned Chrome. Browser page interaction uses\nCDP exclusively. If CDP is missing, a label or selector cannot be resolved,\ncoordinates reach browser chrome or leave the viewport, or a shortcut is not\nsupported by the page route, the operation fails without native input.\nUse `layer_go` for navigation instead of `cmd+l`. This input contract does not\nestablish that virtual-display creation preserves the physical desktop.\nCDP-only input also does not prevent page JavaScript from requesting popups\nor focus, system permission/notification UI from appearing, or display\ncreation from changing the desktop. Acceptance must observe those effects.\n\nRetain `session` from `layer_open`: `mode`, `pid`, `displayId`, `startedAt`,\nand `browserPid` when present. Action results include the session and execution\nroute (`devtools` with `pageTargetId`, or `blink` with a native `host`). A\nreplacement layer invalidates the binding; do not reuse an earlier snapshot\nor act against the replacement. Check `layer_status` after a connection loss\nor timeout before retrying. Reobserve before repeating an operation whose\noutcome is uncertain.\n\nUse the layer's own snapshot and primitives. `layer_open` begins a drive lease\nunless `leaseId` supplies one; it returns that lease and `layer_close` releases\nthe lease it created. A supplied lease remains the caller's responsibility.\nDo not begin a second drive for the layer or use a desktop snapshot as its\ncoordinate reference. If setup fails, MCP attempts to close the layer it created\nand reports any unconfirmed cleanup. Inspect `layer_status` before retrying.\nCheck `layer_status` before the first open too: opening fails while the\nworkspace already has an active task. Close it explicitly first, and do not\nclose another caller's task without user authority.\n\nFor native apps, open `{ bundleId: \"com.apple.calculator\", width: 1280,\nheight: 800, pip: true }`. Select `windowId` or `windowTitle` to borrow only\nmatching windows. Native input can temporarily borrow focus or the pointer;\nthat path does not guarantee continuous focus isolation.\n\n## Transport boundaries\n\n| Surface | Layer support |\n| --- | --- |\n| Action MCP | Complete `layer_*` workflow, including `browser: true`, URL navigation, input, and JavaScript |\n| CLI | Monitor `layer prepare`, `display-status`, `release`; task `open`, `status`, `snapshot`, `pip`, `record start/stop`, `close` |\n| Native host | Native display/app primitives; no MCP browser orchestration |\n| Action WebSocket | Raw agent RPC catalog; route private-layer work to MCP |\n| Lattices daemon | Workspace window groups; route private-layer work to Action MCP |\n\nCLI `layer open` accepts `--bundle-id` or `--pid`, `--width`, `--height`,\n`--window-id` or `--window-title`, and `--pip`. It rejects `--browser`, `--url`,\nunknown flags, and unsupported interaction verbs with an MCP routing error.\nThe CLI task lifetime is detached; always close it when work ends. Width and\nheight on `open` must match the prepared monitor; they do not configure a new\ndisplay. Closing a task does not release the monitor.\n\n## Acceptance\n\nRead-only lookup is `lats action capabilities` or MCP `capabilities`, under\n`acceptance.privateLayer`. The native host supplies the current host and\nphysical-display context; inability to read that context leaves acceptance\nunverified. The record's matching status does not override `safety.privateLayer`\nquarantine, prove the running process matches the installed bundle, or replace\nreview of the retained evidence.\n\nThe operator-reviewed record has this schema:\n\n| Field | Required value |\n| --- | --- |\n| `schemaVersion`, `outcome` | `1`, `\"passed\"` |\n| `acceptedAt`, `expiresAt`, `reviewer` | Valid ISO timestamps and a nonempty reviewer; expired records are stale. |\n| `binding` | `executableSha256`, `nativeSourceDigest`, `orchestrationSourceDigest`, `hostFingerprint`, `osBuild`, `displayConfigurationDigest`, matching the current context. |\n| `checks` | `physicalDisplayMode`, `scale`, `arrangement`, `frontmostApp`, `pointer`, `unrelatedWindowFrames`, each `\"unchanged-throughout\"`. |\n| `phases` | `[\"open\", \"interaction\", \"close\"]` |\n| `evidence` | Nonempty list of `{ path, sha256 }` entries; absolute local artifact paths with verified hashes. |\n\nCapabilities discovers and validates this record. It does not write one.\nCreate or renew it only from reviewed, authorized acceptance evidence.\n\nFor ordinary task acceptance, prepare the monitor first under authorized setup\nconditions, then capture physical modes (logical and pixel sizes, scale and refresh rate),\ndisplay arrangement/mirroring, frontmost app, pointer, and unrelated window\nframes before opening. Monitor those values throughout task attachment, interaction,\nand teardown. Verify a layer snapshot and a real browser interaction, then\nconfirm closure and window restoration. Fail on the first unexpected change.\nA final sample that matches the baseline cannot erase an earlier failure.\nTest initial preparation, monitor-owner restart, and explicit release\nseparately; those operations change display topology and must not be folded\ninto a claim that ordinary tasks never connect or disconnect a display.\n\n## What it does\n\n- Attaches to the prepared monitor and moves the subject app's windows onto it. The original frame of each window is recorded. The persistent monitor owner holds the `CGVirtualDisplay` independently of the task.\n- `windowId` (the `kCGWindowNumber`) or `windowTitle` (a case-insensitive substring) moves only the matching windows. Use it to borrow one window of an app the user is also in, such as a browser.\n- Starts one ScreenCaptureKit stream on the layer display and keeps it running for the layer's life. The viewer, snapshots and recordings all come off that stream (see [Feed](#feed)).\n- Launches the subject in the background if it isn't running, and asks an app with no windows for one (the reopen a Dock click sends). Background launch requests no activation; this is not an isolation acceptance result.\n- `url` (a URL, a bare host read as `https`, or a file path) opens in the subject without activating it, once its windows are on the layer. It needs `bundleId` or `pid`, and can't be combined with `windowId` or `windowTitle`.\n- Adopts windows the subject opens while the layer is up, such as a `cmd+n` window, a pop-up, or an `open-app` with `input.url`. They move onto the layer as they appear, cascaded. A layer that borrowed one window by `windowId` or `windowTitle` adopts nothing more.\n- Shows the PiP viewer unless `pip` is `false`.\n- Writes a state file: display id, global bounds (top-left origin, points), PiP state, and the moved windows with their original frames.\n- Keeps one task per workspace. Opening while a task is active fails; close the task first.\n\n## Primitives\n\nThe layer's own tools are the short way to work it. Look, then act on what you saw:\n\n```\nlayer_open { bundleId: \"com.apple.Safari\", url: \"news.ycombinator.com\" }\nlayer_snapshot {} → p.png, 950×560\nlayer_click { x: 475, y: 196 } pixels in that snapshot\nlayer_click { label: \"new\" } or a control by its label\nlayer_type { text: \"lobste.rs\\n\" } a trailing \\n submits\nlayer_press { key: \"cmd+l\" }\nlayer_drag { from: [95, 224], to: [570, 252] }\nlayer_scroll { x: 475, y: 280, dy: -600 }\nlayer_close {}\n```\n\nPoints are pixels in the last `layer_snapshot` image, read through its crop and scale, so an agent never adds the layer's origin or works out the display's scale. `space: \"layer\"` takes points on the layer display instead. A point outside the last snapshot is an error, and so is a point with no snapshot taken yet.\n\nBrowser primitives use the bound CDP page and return `route: \"devtools\"` with\n`pageTargetId`. Native primitives use `action.act.execute` and return\n`route: \"blink\"` with the native `host`. Both report the layer session; input\nprimitives normally return a fresh snapshot (`look: false` skips it). A lease\nis taken for the caller when none is passed. The native routing below is not a\nfallback for a browser session.\n\n## Native blink acts\n\nWhile a layer is up, `action.act.execute` routes these acts as blink acts:\n\n| Act | Host command | When |\n|---|---|---|\n| `click` with a point | `blink-click` | the point is on the layer display |\n| `type` | `blink-type` | no accessibility label path applies |\n| `press-key` | `blink-key` | always |\n| `drag` | `blink-drag` | both points are on the layer, and no `filePath` |\n| `scroll` | `blink-scroll` | the point is on the layer |\n\nAccessibility comes first. A target with a bundle id and label is pressed or set through accessibility and never blinks. `blink-click` hit-tests its point before it touches the pointer: if the element there is a button, checkbox, radio, pop-up, menu item, disclosure triangle, link or tab that takes `AXPress`, it is pressed and the result says `via=ax`. A text field, text area, combo box or search field there is focused through accessibility instead of clicked into. Nothing moves and focus does not change.\n\n`blink-type` works the same way. It inserts the text at the caret of the app's focused text field through accessibility (`via=ax`), and only borrows focus for keystrokes (`via=keys`) when the field doesn't take it, is secure, or a `delayMs` cadence is requested. Web fields usually take keystrokes. When the field's value is readable, the blink holds focus until the text shows up in it (`via=keys verified`), because a web view consumes keys in its own content process after the app's main loop has already drained. A trailing newline in `type` text (`\"lobste.rs\\n\"`) splits off: the text goes in through accessibility and the field is submitted with `AXConfirm`, a Return that moves nothing. `press-key` tries accessibility too: a chord that is a menu item's shortcut (`cmd+l`, `cmd+n`) presses that menu item (`via=ax menu=\"Open Location…\"`), and a bare Return or Escape in a text field confirms or cancels it. Anything else borrows focus for the keystroke (`via=keys`).\n\n`press-key` takes a chord in `input.key`, written however the agent reads it: `\"return\"`, `\"cmd+l\"`, `\"Cmd-Shift-T\"`, `\"⌘⇧T\"`, `[\"cmd\", \"l\"]` in `input.keys`, or `key` plus `input.modifiers`. Aliases such as `enter`, `esc`, `backspace` and `ArrowDown` map to the host's names. An unknown key is an error that points at `type`, never text typed one character at a time.\n\nThe pointer is the fallback: no pressable element at the point, a requested `holdMs`, or `--pointer` on the host command. A pointer blink saves the cursor position and the frontmost app, clicks, and restores both within tens of milliseconds. Its result says `via=pointer`.\n\n`type` and `press-key` land in the app's focused window. If that window is off the layer, the blink raises the app's layer window first, without activating the app, and refuses if focus still isn't on the layer. Keystrokes never reach a window on the user's screen.\n\n`type` and `press-key` target `input.bundleId` or `input.pid` if given, else the layer's app. Accessibility paths (`press-accessibility-element`, `set-accessibility-value`) do not change. A click outside the layer display uses `click-point`.\n\nDrag and scroll have no accessibility equivalent and always use the pointer. It leaves the user's screens for the length of the gesture (200 ms for a drag by default; an instant for a scroll) and comes back. Scroll wheel events activate nothing. A drag's mouse-down activates the app it lands in for up to a second, and focus is handed back afterwards.\n\nBlink acts report the `blink` tier. A background drive lease allows them. They do not show the pointer-focus countdown.\n\n## Feed\n\nThe layer captures its own display, not the subject app. A stream that targets an app makes macOS badge that app's windows as shared, and the badge is drawn into the window, so it would show in every frame. The framing to the subject's windows happens downstream, by crop. The stream only delivers frames when something on the layer changes, so an idle layer costs next to nothing.\n\n- **Viewer.** The PiP grows out of the screen corner on its first frame and frames the subject's windows, following them as they move and resize. Drag the header to move it; double-click the header to enlarge it, and double-click again to restore its size. Hide leaves the layer running; `layer pip` / `layer_pip` brings it back.\n- **State.** One dot in the viewer's bottom-right corner: solid coral while the agent can act, breathing while a take records, hollow while paused.\n- **Marks.** The host posts where native acts landed and the viewer rings it: a circle for a click, an outline around the focused field for typing and keys. Accessibility paths can avoid pointer movement; pointer and key fallbacks may temporarily borrow pointer or focus.\n- **Note.** The latest `action.drive.note` shows along the bottom edge for about 12 seconds.\n- **Snapshot.** `layer snapshot` / `layer_snapshot` writes a PNG from the latest frame: the subject's windows by default, one window with `windowId`, or the whole layer with `full`. Nothing is captured on request, so it answers in tens of milliseconds. `unchangedMs` is how long the layer has been still, not how stale the picture is.\n- **Recording.** `layer record start|stop` / `layer_record` adds a recording output to the running stream, so the take starts on the next frame. It records the whole layer display at its native size. One take at a time; closing the layer finishes a take in progress.\n\n## Operator controls\n\nThe viewer has a persistent hostname header and native controls outside the captured image:\n\n- **Pause / Resume.** Pauses layer input dispatch. While paused, `state.json` has `\"paused\": true`; observations remain available. Resume is an operator action. Events or page scripts already dispatched cannot be undone.\n- **Stop.** Pauses input, records a session-specific `handoff.json`, and ends the layer without activating the target app. This stops the layer session, not the agent's entire harness or unrelated work.\n- **Agent.** Looks for the opener's app on this host. Action records its process chain at `open` (`--owner-pids`), activates the nearest app in that chain, and otherwise asks Lattices (`terminals.search`, then `window.focus`) which local terminal window shows it. If nothing resolves, the viewer beeps. This does not return from an SSH host to a conversation on another machine.\n- **Take over.** Records a session-specific handoff, ends the layer, and activates the target app. Borrowed windows return to their recorded frames; windows created on the layer are closed, subject to unsaved-changes prompts. This is not an in-place remote-control mode.\n- **Hide.** Hides only the viewer; the layer keeps running.\n\nStop and Take over refuse further agent input during teardown and after cleanup. `layer close` does not clear the handoff. A new explicit `layer open` acknowledges it; callers must obtain fresh user authorization before opening again. The prepared monitor remains connected after either action.\n\nThese are host-local controls. An interactive viewer streamed to another machine or into a chat is not implemented.\n\nRequests go through `control.request.json` next to the state file and a `SIGUSR2`; the reply lands in `control.reply.json`.\n\n## Lifetime\n\n`layer_close` is the intended teardown. It creates the stop file and waits for the layer to put the windows back and exit. Windows that were opened after the layer went up (`born` in the state file) never sat on the user's screen, so they are closed instead of moved there. One that asks before closing (unsaved changes) stays open and goes back to the frame it appeared at. If the layer does not exit, Action sends `SIGTERM`.\n\nThis tears down the task's feed and window attachment, not the monitor.\n`layer release` stops the persistent monitor owner only after its task has been\nexplicitly closed. It refuses active tasks and incoming previews without closing\nthem. Close dependent tasks first, then release their displays.\n\nThe layer also tears down when its owner dies:\n\n- **MCP.** The layer watches the MCP server (`--parent-pid`) and restores the windows if the server exits.\n- **CLI.** The CLI exits once the layer is up, so the layer is detached. Run `layer close` to take it down when the work is done.\n\nA detached layer doesn't wait for `layer close` forever. It also comes down when:\n\n- **Its owners exit.** Every process in the opener's chain that was alive at `open` has exited.\n- **It goes idle.** 30 minutes pass with no act, no snapshot or recording request, and no use of the viewer. A recording in progress keeps the layer up. Pass `--idle-timeout ` to the host to change the lease; `0` turns it off.\n\nEither way the windows go back first, as with `layer close`.\n\n`layer status` reports the live layer. If the recorded process is gone, status removes the stale files and reports the layer as inactive.\n\nTask and monitor state live in `~/Library/Application Support/Action/agent-layer/`.\nUse `layer status` for the task and `layer display-status` for the monitor;\nan inactive task with a ready monitor is the normal state between tasks.\n\n## Requirements and limits\n\n- The feed needs Screen Recording for Action. Without it the layer still works (windows move, blink acts land), but there is no viewer, snapshot or recording; the layer's `layer.log` says `feed failed`. Recording needs macOS 15.\n- The display touches the bottom-right-most display only at its corner. A pointer pushed exactly through that corner can still cross onto it.\n- A pointer blink click on another app activates that app for a moment before focus goes back. The menu bar can flicker for that moment.\n- Blink acts refuse targets that aren't on an agent layer: a point off the layer, or an app with no window on it. `--any-display` lifts that for tests.\n- Killing a task with `SIGKILL` skips its window restoration; the persistent monitor remains. Killing the monitor owner disconnects the display, allowing macOS to relocate its windows. Neither is an ordinary task teardown or a preservation guarantee." }, { "id": "action:recording", "slug": "recording", "product": "action", "kind": "doc", "catalogKind": "guide", "current": true, "title": "Recording", "description": "How Action records a window or region and signals completion", "order": 8, "sourcePath": "products/action/docs/recording.md", "url": "/action/docs/recording", "rawUrl": "/action/docs/markdown/recording.md", "markdownUrl": "/action/docs/recording.md", "headings": [ { "depth": 2, "text": "What is stable now", "anchor": "what-is-stable-now" }, { "depth": 2, "text": "The key discovery", "anchor": "the-key-discovery" }, { "depth": 2, "text": "Current recording path", "anchor": "current-recording-path" }, { "depth": 2, "text": "Testing recording", "anchor": "testing-recording" }, { "depth": 2, "text": "Important caveat", "anchor": "important-caveat" }, { "depth": 2, "text": "What to preserve", "anchor": "what-to-preserve" }, { "depth": 2, "text": "What to avoid", "anchor": "what-to-avoid" } ], "tokensEstimate": 426, "frontmatter": { "title": "Recording", "description": "How Action records a window or region and signals completion", "order": 8 }, "markdown": "---\ntitle: Recording\ndescription: How Action records a window or region and signals completion\norder: 8\n---\n\nRecording is the most important runtime path in this project.\n\nThis page documents how it currently works and what assumptions should remain\ntrue while the product is still being hardened.\n\n## What is stable now\n\nThe native recording path now succeeds for:\n\n- bounded region recording\n- app-window recording\n\nThe successful path produces:\n\n- a `.mov` file\n- a `.finished` marker file\n- an optional debug log if `--debug-log` is supplied\n\n## The key discovery\n\nThe main recording bug was not just a bad capture configuration.\n\nThe deeper issue was lifecycle ownership:\n\n- `ScreenCaptureKit` recording was unreliable or crash-prone when hosted from the\n wrong runtime path\n- a minimal real AppKit app lifecycle succeeded with the same capture logic\n\nThat is why the current solution uses a dedicated `recording-probe` mode inside\n`Action.app`.\n\n## Current recording path\n\nFor a region recording request:\n\n1. host command accepts the request\n2. host sends a recording method to the local agent\n3. agent launches `Action.app` with `recording-probe`\n4. probe runner creates a tiny AppKit window and starts recording\n5. recording continues until the stop file appears\n6. probe writes the finished marker and exits\n\nCore files:\n\n- [ActionRecordingProbeLauncher.swift](../native/engine/CoreSources/ActionRecordingProbeLauncher.swift)\n- [RecordingProbeAppRunner.swift](../native/engine/Sources/RecordingProbeAppRunner.swift)\n- [ActionHostMain.swift](../native/engine/Sources/ActionHostMain.swift)\n\n## Testing recording\n\nThe wrapper script is:\n\n```bash\nbun run native:test:record\n```\n\nFor lower-level testing, `run-app-host.sh` can call recording commands\ndirectly.\n\n## Important caveat\n\nThe initial CLI reply usually means:\n\n`recording started successfully`\n\nIt does **not** mean:\n\n`recording completed successfully`\n\nCompletion is represented by the finished marker file.\n\n## What to preserve\n\n- real AppKit lifecycle for actual recording work\n- stop-file and finished-file based orchestration\n- debug-log passthrough for probe runs\n- clean app/agent separation even if the app is doing the final recording work\n\n## What to avoid\n\n- pushing real recording back into a plain headless lifecycle\n- assuming a WebSocket ack is equivalent to a completed recording\n- removing the probe path before a better lifecycle-safe replacement exists\n", "content": "Recording is the most important runtime path in this project.\n\nThis page documents how it currently works and what assumptions should remain\ntrue while the product is still being hardened.\n\n## What is stable now\n\nThe native recording path now succeeds for:\n\n- bounded region recording\n- app-window recording\n\nThe successful path produces:\n\n- a `.mov` file\n- a `.finished` marker file\n- an optional debug log if `--debug-log` is supplied\n\n## The key discovery\n\nThe main recording bug was not just a bad capture configuration.\n\nThe deeper issue was lifecycle ownership:\n\n- `ScreenCaptureKit` recording was unreliable or crash-prone when hosted from the\n wrong runtime path\n- a minimal real AppKit app lifecycle succeeded with the same capture logic\n\nThat is why the current solution uses a dedicated `recording-probe` mode inside\n`Action.app`.\n\n## Current recording path\n\nFor a region recording request:\n\n1. host command accepts the request\n2. host sends a recording method to the local agent\n3. agent launches `Action.app` with `recording-probe`\n4. probe runner creates a tiny AppKit window and starts recording\n5. recording continues until the stop file appears\n6. probe writes the finished marker and exits\n\nCore files:\n\n- [ActionRecordingProbeLauncher.swift](../native/engine/CoreSources/ActionRecordingProbeLauncher.swift)\n- [RecordingProbeAppRunner.swift](../native/engine/Sources/RecordingProbeAppRunner.swift)\n- [ActionHostMain.swift](../native/engine/Sources/ActionHostMain.swift)\n\n## Testing recording\n\nThe wrapper script is:\n\n```bash\nbun run native:test:record\n```\n\nFor lower-level testing, `run-app-host.sh` can call recording commands\ndirectly.\n\n## Important caveat\n\nThe initial CLI reply usually means:\n\n`recording started successfully`\n\nIt does **not** mean:\n\n`recording completed successfully`\n\nCompletion is represented by the finished marker file.\n\n## What to preserve\n\n- real AppKit lifecycle for actual recording work\n- stop-file and finished-file based orchestration\n- debug-log passthrough for probe runs\n- clean app/agent separation even if the app is doing the final recording work\n\n## What to avoid\n\n- pushing real recording back into a plain headless lifecycle\n- assuming a WebSocket ack is equivalent to a completed recording\n- removing the probe path before a better lifecycle-safe replacement exists" }, { "id": "action:stage", "slug": "stage", "product": "action", "kind": "doc", "catalogKind": "guide", "current": true, "title": "Stage", "description": "Present listed windows on a flat backdrop for a clean take", "order": 9, "sourcePath": "products/action/docs/stage.md", "url": "/action/docs/stage", "rawUrl": "/action/docs/markdown/stage.md", "markdownUrl": "/action/docs/stage.md", "headings": [ { "depth": 2, "text": "What it does", "anchor": "what-it-does" }, { "depth": 2, "text": "Lifetime", "anchor": "lifetime" }, { "depth": 2, "text": "Surfaces", "anchor": "surfaces" } ], "tokensEstimate": 507, "frontmatter": { "title": "Stage", "description": "Present listed windows on a flat backdrop for a clean take", "order": 9 }, "markdown": "---\ntitle: Stage\ndescription: Present listed windows on a flat backdrop for a clean take\norder: 9\n---\n\nAction owns the look of a take. The wallpaper is never written.\n\n`action.stage.set` is the primitive: declare the world, Action makes it so.\n\n```json\n{\n \"mode\": \"drape\",\n \"color\": \"0e0d0a\",\n \"subjects\": [\n { \"bundleId\": \"to.talkie.agent.dev\", \"title\": \"Settings\" },\n { \"bundleId\": \"com.googlecode.iterm2\" }\n ]\n}\n```\n\n## What it does\n\n- Puts up a flat color sheet at ordinary window level (`NSWindow.Level.normal`).\n- `AXRaise`s only the listed windows. Same level is what lets them sit on the sheet.\n- Reads on-screen window order in the staged bounds (or the sheet's frames). The scene is the listed subjects on the sheet. Anything else above the sheet, including the driver terminal, is an intruder unless it was listed.\n- Fails `stage.set` if a listed subject is still buried or a non-subject occupies the scene. It raises again a couple of times before refusing. The sheet stays up so `stage.status` can show who is actually on top.\n- Leaves every other app alone. No hiding. No desktop-picture writes.\n- Dies with the process that asked for it. `action.stage.clear` takes it down on purpose.\n\n## Lifetime\n\n`action.stage.clear` is the intended teardown. Two backstops cover the case where it never runs:\n\n- **Owner.** A drape set through MCP watches the MCP server and dismisses itself if that server dies. A drape set through the CLI cannot do this: the CLI process exits the moment the sheet is up, so a drape watching it would take itself down immediately. CLI drapes are detached.\n- **`seconds`.** An optional lifetime after which the drape dismisses itself. The CLI defaults to 1800; pass `--seconds` to change it. MCP omits it by default because the owner check already covers a dead server.\n\n`stage status` reports `owner`, `pid`, whether the drape is still up, and `scene`: the windows actually on top of the sheet.\n\n`mode: \"space\"` keeps the sheet on the current Space only. Instantiate subjects on that Space; windows on other Spaces will not compose into the frame.\n\n`level: \"desktop\"` is the other sheet in the repertoire. It sits under all app windows and is the wrong default for a take.\n\n## Surfaces\n\n- MCP: `action.stage.set`, `action.stage.clear`, `action.stage.status`\n- CLI: `bun packages/cli/src/main.ts stage set|clear|status`\n\nTo work an app out of the user's sight instead of staging it for a take, see [Agent layer](/action/docs/agent-layer).\n", "content": "Action owns the look of a take. The wallpaper is never written.\n\n`action.stage.set` is the primitive: declare the world, Action makes it so.\n\n```json\n{\n \"mode\": \"drape\",\n \"color\": \"0e0d0a\",\n \"subjects\": [\n { \"bundleId\": \"to.talkie.agent.dev\", \"title\": \"Settings\" },\n { \"bundleId\": \"com.googlecode.iterm2\" }\n ]\n}\n```\n\n## What it does\n\n- Puts up a flat color sheet at ordinary window level (`NSWindow.Level.normal`).\n- `AXRaise`s only the listed windows. Same level is what lets them sit on the sheet.\n- Reads on-screen window order in the staged bounds (or the sheet's frames). The scene is the listed subjects on the sheet. Anything else above the sheet, including the driver terminal, is an intruder unless it was listed.\n- Fails `stage.set` if a listed subject is still buried or a non-subject occupies the scene. It raises again a couple of times before refusing. The sheet stays up so `stage.status` can show who is actually on top.\n- Leaves every other app alone. No hiding. No desktop-picture writes.\n- Dies with the process that asked for it. `action.stage.clear` takes it down on purpose.\n\n## Lifetime\n\n`action.stage.clear` is the intended teardown. Two backstops cover the case where it never runs:\n\n- **Owner.** A drape set through MCP watches the MCP server and dismisses itself if that server dies. A drape set through the CLI cannot do this: the CLI process exits the moment the sheet is up, so a drape watching it would take itself down immediately. CLI drapes are detached.\n- **`seconds`.** An optional lifetime after which the drape dismisses itself. The CLI defaults to 1800; pass `--seconds` to change it. MCP omits it by default because the owner check already covers a dead server.\n\n`stage status` reports `owner`, `pid`, whether the drape is still up, and `scene`: the windows actually on top of the sheet.\n\n`mode: \"space\"` keeps the sheet on the current Space only. Instantiate subjects on that Space; windows on other Spaces will not compose into the frame.\n\n`level: \"desktop\"` is the other sheet in the repertoire. It sits under all app windows and is the wrong default for a take.\n\n## Surfaces\n\n- MCP: `action.stage.set`, `action.stage.clear`, `action.stage.status`\n- CLI: `bun packages/cli/src/main.ts stage set|clear|status`\n\nTo work an app out of the user's sight instead of staging it for a take, see [Agent layer](/action/docs/agent-layer)." }, { "id": "action:api", "slug": "api", "product": "action", "kind": "doc", "catalogKind": "guide", "current": true, "title": "API", "description": "The Action agent WebSocket protocol over authenticated private action.sock", "order": 3, "sourcePath": "products/action/docs/api.md", "url": "/action/docs/api", "rawUrl": "/action/docs/markdown/api.md", "markdownUrl": "/action/docs/api.md", "headings": [ { "depth": 2, "text": "Transport", "anchor": "transport" }, { "depth": 3, "text": "Private connection", "anchor": "private-connection" }, { "depth": 2, "text": "Request", "anchor": "request" }, { "depth": 2, "text": "Response", "anchor": "response" }, { "depth": 2, "text": "Methods", "anchor": "methods" }, { "depth": 3, "text": "Runtime and permissions", "anchor": "runtime-and-permissions" }, { "depth": 3, "text": "Apps, windows, and input", "anchor": "apps-windows-and-input" }, { "depth": 3, "text": "Drive leases", "anchor": "drive-leases" }, { "depth": 3, "text": "Workspace drag", "anchor": "workspace-drag" }, { "depth": 3, "text": "Capture", "anchor": "capture" }, { "depth": 2, "text": "Recording contract", "anchor": "recording-contract" }, { "depth": 2, "text": "What is not in this protocol", "anchor": "what-is-not-in-this-protocol" }, { "depth": 2, "text": "From the Lattices CLI", "anchor": "from-the-lattices-cli" }, { "depth": 2, "text": "Remote Linux hosts", "anchor": "remote-linux-hosts" } ], "tokensEstimate": 1805, "frontmatter": { "title": "API", "description": "The Action agent WebSocket protocol over authenticated private action.sock", "order": 3 }, "markdown": "---\ntitle: API\ndescription: The Action agent WebSocket protocol over authenticated private action.sock\norder: 3\n---\n\nThe Action agent is a local WebSocket server inside `Action.app`. This page\ndocuments its raw RPC protocol. Most agents use the Action MCP server instead\n(see [Agent setup](/action/docs/agent-setup)). The MCP server adds sessions,\ntarget resolution, recording status, stage, and private layers on top of this\nprotocol.\n\nSource of truth:\n[`ActionAgentProtocol.swift`](https://github.com/arach/lattices/blob/main/products/action/native/engine/CoreSources/ActionAgentProtocol.swift)\nand\n[`ActionAgentRuntime.swift`](https://github.com/arach/lattices/blob/main/products/action/native/engine/CoreSources/ActionAgentRuntime.swift).\n\n## Transport\n\n| Property | Value |\n| --- | --- |\n| Protocol | WebSocket, text frames, JSON |\n| Address | `~/.lattices/run/action.sock` |\n| Protocol version | `2`, reported by `status` |\n| Authentication | Connection token for every WebSocket; additional capability token for privileged methods |\n\n`status` reports `authentication: \"capability-token\"` and `authTokenFile`, the\npath of a token file that the agent writes at each launch. The file is\nuser-only (`0600`) in a user-only directory (`0700`).\n\nPrivileged requests must include `protocolVersion` and `authToken` in their\nparams. The privileged requests are:\n\n- `drive.begin` with `mode: \"attention\"`;\n- `workspace.dragFile`;\n- `workspace.cancelOperation`.\n\nEvery method requires an authenticated connection.\n\n### Private connection\n\nAction 0.4.0 uses the private connection below. Install its matching runtime\nand Lattices CLI 0.15.1. Earlier TCP clients do not automatically fall back.\n\n| Property | Value |\n| --- | --- |\n| Address | Unix socket `~/.lattices/run/action.sock` (override with `ACTION_SOCKET_PATH`) |\n| Connection token | `~/.lattices/run/action.token`, sent as `x-lattices-token` on the WebSocket upgrade with `Host: localhost` and no `Origin` |\n| Local access | User-owned directory mode `0700`, socket and token mode `0600`, plus the connection token. The server does not perform a separate peer-UID check. |\n| TCP | `127.0.0.1:4319` opens only with `ACTION_ALLOW_TCP=1`, and still requires the connection token |\n\nEvery connection now needs the token, not only the privileged methods. The\nprivileged methods above still check `authToken` in their params as well. The\nAction runtime and `lats action call` read the token for you.\n\nRemote machines cannot reach this socket. A paired tailnet client can call a\nreviewed subset of Action methods through the Lattices gateway, one exact grant\nper method, for example `--grant action:status`. The gateway opens one Action\nconnection per call. It therefore denies `drive.begin`, `drive.touch`,\n`drive.release`, `workspace.dragFile` and `workspace.cancelOperation`, which\ncreate or use a lease owned by a persistent connection. `drive.status` remains\navailable with its own read grant. See\n[Private IPC and the gateway](/lattices/docs/concepts#private-ipc-and-the-gateway).\n\nThe receive side stays armed while a request runs, so a cancellation can arrive\nduring a gesture. Responses are serialized and correlated by request `id`.\n\n## Request\n\n```json\n{\n \"id\": \"4f2c9b1e\",\n \"method\": \"capture.screenshotScreen\",\n \"params\": { \"output\": \"/tmp/screen.png\" }\n}\n```\n\n- `id`: request id. Native clients default to a UUID.\n- `method`: a method name from the table below.\n- `params`: a flat map of string keys to string values. Send numbers and\n booleans as strings (`\"1440\"`, `\"true\"`).\n\n## Response\n\n```json\n{\n \"id\": \"4f2c9b1e\",\n \"ok\": true,\n \"result\": { \"outputPath\": \"/tmp/screen.png\", \"status\": \"screenshot\" },\n \"error\": null\n}\n```\n\n- `ok`: success flag.\n- `result`: a flat string map. Structured values, such as a drive lease, arrive\n as JSON strings inside this map.\n- `error`: an error string when `ok` is `false`.\n\n## Methods\n\n### Runtime and permissions\n\n| Method | Purpose | Params |\n| --- | --- | --- |\n| `ping` | Liveness check | none |\n| `status` | Service metadata, method list, protocol version, build provenance, token path | none |\n| `permissions.snapshot` | Accessibility and Screen Recording state | none |\n| `permissions.request` | Prompt for both permissions | none |\n| `permissions.requestAccessibility` | Prompt for Accessibility | none |\n| `settings.openAccessibility` | Open the Accessibility settings pane | none |\n| `settings.openScreenRecording` | Open the Screen Recording settings pane | none |\n\n### Apps, windows, and input\n\n| Method | Purpose | Params |\n| --- | --- | --- |\n| `app.launch` | Launch an app | `bundleId` |\n| `app.activate` | Bring an app forward | `bundleId` |\n| `window.setFrame` | Move and resize the app's window | `bundleId`, `x`, `y`, `width`, `height` |\n| `window.getFrame` | Read the app's window frame | `bundleId` |\n| `input.typeText` | Type text into the focused element | `text` |\n| `input.drag` | Pointer drag | `fromX`, `fromY`, `toX`, `toY`, optional `durationMs` (default 300), optional `filePath` |\n| `accessibility.pressElement` | Press an element by label | `bundleId`, `label`, optional `role` |\n| `accessibility.setValue` | Set an element's value | `bundleId`, `label`, `value`, optional `role` |\n\n`calculator.buttons`, `calculator.clickButton`, and `calculator.displayValue`\nexist for the Calculator demo scenario. They are not a general API.\n\n### Drive leases\n\nA drive lease tells other clients that an agent is acting on the desktop. The\nlease belongs to the connection that created it. Closing the connection\ncancels that connection's leases only.\n\n| Method | Purpose | Params |\n| --- | --- | --- |\n| `drive.begin` | Acquire a lease | `agent`, `task`, optional `mode`, `sessionId`; attention mode also needs approval, run and workspace ids, capability, resource, `protocolVersion`, `authToken` |\n| `drive.touch` | Renew a lease | `leaseId`, optional `axTier`, `actionKind` |\n| `drive.release` | Release a lease | `leaseId`, optional `outcome`, `summary` |\n| `drive.status` | Snapshot of current leases | none |\n| `sessions.prune` | Remove stale drive sessions | none |\n| `sessions.clear` | Remove all drive sessions | none |\n\n### Workspace drag\n\n| Method | Purpose | Params |\n| --- | --- | --- |\n| `workspace.dragFile` | Drag one exact run fixture from Finder into one exact Scout drop zone | lease, run, and workspace ids; fixture URL; Finder and Scout window identities; destination AX identifier; display bounds; `protocolVersion`; `authToken` |\n| `workspace.cancelOperation` | Cancel one in-flight operation | `leaseId`, `operationId`, `protocolVersion`, `authToken` |\n\n`workspace.dragFile` is not a generic coordinate drag. It rejects unknown\nparams and accepts no source point, destination point, duration, command, or\nexecutable. Action resolves exactly one Finder item by AXURL inside exactly one\ntitled Finder window. Action resolves exactly one run-scoped AX drop-zone\nidentifier inside exactly one titled `app.openscout.scout` window. All resolved\nframes and points must be inside the supplied display bounds.\n\nAn approved attention `drive.begin` acquires the global `system-pointer`\nresource and grants one `workspace.drag-file` capability. The capability is\nbound to the connection, run, and workspace, and is consumed before the\ngesture. During the drag, Action checks the lease and operation stop signals\non every step. Cleanup always attempts mouse-up and restores the saved pointer\nposition.\n\n### Capture\n\n| Method | Purpose | Params |\n| --- | --- | --- |\n| `capture.screenshotScreen` | Screenshot the main display | `output` |\n| `capture.screenshotRegion` | Screenshot a region | `x`, `y`, `width`, `height`, `output` |\n| `capture.screenshotAppWindow` | Screenshot an app window | `output`, `bundleId` or `pid` |\n| `capture.recordRegion` | Record a region | `x`, `y`, `width`, `height`, `output`, optional `fps`, `scale`, `stopFile`, `finishedFile`, `debugLog`, `includeSupervisionOverlay` |\n| `capture.recordAppWindow` | Record an app window | `bundleId`, `output`, optional `stopFile`, `finishedFile`, `debugLog` |\n\n## Recording contract\n\nThe agent does not record in its own process. A record method launches a fresh\n`Action.app` instance in `recording-probe` mode, and that probe performs the\nScreenCaptureKit capture.\n\n- A successful reply means the probe started.\n- The recording ends when the `stopFile` appears.\n- Completion is the movie at `output` plus the `finishedFile` marker. The\n marker can also contain a recording error.\n- Pass `debugLog` to collect probe logs.\n\nSee [Recording](/action/docs/recording).\n\n## What is not in this protocol\n\n- Private layers (`layer_*`), stage, observation, target resolution, and\n session artifacts are MCP and runtime features. The raw agent does not\n orchestrate them. See [Agent layer](/action/docs/agent-layer).\n- The `browser_*` tools belong to the Lattices MCP. See\n [Browser](/action/docs/browser).\n- `computer.*` methods belong to the Lattices core daemon on `lattices.sock`, not to\n this agent. See the [Lattices API](/lattices/docs/api).\n\n## From the Lattices CLI\n\n`lats action call [params-json]` sends one request and prints the\nresult. In released builds it does not attach a capability token, so it cannot\ncall privileged methods. The matching runtime reads and attaches the token automatically;\nsee [Private connection](#private-connection).\n\n```sh\nlats action call status\nlats action call window.getFrame '{\"bundleId\":\"com.apple.calculator\"}'\n```\n\n## Remote Linux hosts\n\nThe raw agent protocol is local to the Mac. For a Linux desktop, the Action\nruntime connects to `lattices-host` instead. See\n[Remote engine](/action/docs/remote-engine).\n", "content": "The Action agent is a local WebSocket server inside `Action.app`. This page\ndocuments its raw RPC protocol. Most agents use the Action MCP server instead\n(see [Agent setup](/action/docs/agent-setup)). The MCP server adds sessions,\ntarget resolution, recording status, stage, and private layers on top of this\nprotocol.\n\nSource of truth:\n[`ActionAgentProtocol.swift`](https://github.com/arach/lattices/blob/main/products/action/native/engine/CoreSources/ActionAgentProtocol.swift)\nand\n[`ActionAgentRuntime.swift`](https://github.com/arach/lattices/blob/main/products/action/native/engine/CoreSources/ActionAgentRuntime.swift).\n\n## Transport\n\n| Property | Value |\n| --- | --- |\n| Protocol | WebSocket, text frames, JSON |\n| Address | `~/.lattices/run/action.sock` |\n| Protocol version | `2`, reported by `status` |\n| Authentication | Connection token for every WebSocket; additional capability token for privileged methods |\n\n`status` reports `authentication: \"capability-token\"` and `authTokenFile`, the\npath of a token file that the agent writes at each launch. The file is\nuser-only (`0600`) in a user-only directory (`0700`).\n\nPrivileged requests must include `protocolVersion` and `authToken` in their\nparams. The privileged requests are:\n\n- `drive.begin` with `mode: \"attention\"`;\n- `workspace.dragFile`;\n- `workspace.cancelOperation`.\n\nEvery method requires an authenticated connection.\n\n### Private connection\n\nAction 0.4.0 uses the private connection below. Install its matching runtime\nand Lattices CLI 0.15.1. Earlier TCP clients do not automatically fall back.\n\n| Property | Value |\n| --- | --- |\n| Address | Unix socket `~/.lattices/run/action.sock` (override with `ACTION_SOCKET_PATH`) |\n| Connection token | `~/.lattices/run/action.token`, sent as `x-lattices-token` on the WebSocket upgrade with `Host: localhost` and no `Origin` |\n| Local access | User-owned directory mode `0700`, socket and token mode `0600`, plus the connection token. The server does not perform a separate peer-UID check. |\n| TCP | `127.0.0.1:4319` opens only with `ACTION_ALLOW_TCP=1`, and still requires the connection token |\n\nEvery connection now needs the token, not only the privileged methods. The\nprivileged methods above still check `authToken` in their params as well. The\nAction runtime and `lats action call` read the token for you.\n\nRemote machines cannot reach this socket. A paired tailnet client can call a\nreviewed subset of Action methods through the Lattices gateway, one exact grant\nper method, for example `--grant action:status`. The gateway opens one Action\nconnection per call. It therefore denies `drive.begin`, `drive.touch`,\n`drive.release`, `workspace.dragFile` and `workspace.cancelOperation`, which\ncreate or use a lease owned by a persistent connection. `drive.status` remains\navailable with its own read grant. See\n[Private IPC and the gateway](/lattices/docs/concepts#private-ipc-and-the-gateway).\n\nThe receive side stays armed while a request runs, so a cancellation can arrive\nduring a gesture. Responses are serialized and correlated by request `id`.\n\n## Request\n\n```json\n{\n \"id\": \"4f2c9b1e\",\n \"method\": \"capture.screenshotScreen\",\n \"params\": { \"output\": \"/tmp/screen.png\" }\n}\n```\n\n- `id`: request id. Native clients default to a UUID.\n- `method`: a method name from the table below.\n- `params`: a flat map of string keys to string values. Send numbers and\n booleans as strings (`\"1440\"`, `\"true\"`).\n\n## Response\n\n```json\n{\n \"id\": \"4f2c9b1e\",\n \"ok\": true,\n \"result\": { \"outputPath\": \"/tmp/screen.png\", \"status\": \"screenshot\" },\n \"error\": null\n}\n```\n\n- `ok`: success flag.\n- `result`: a flat string map. Structured values, such as a drive lease, arrive\n as JSON strings inside this map.\n- `error`: an error string when `ok` is `false`.\n\n## Methods\n\n### Runtime and permissions\n\n| Method | Purpose | Params |\n| --- | --- | --- |\n| `ping` | Liveness check | none |\n| `status` | Service metadata, method list, protocol version, build provenance, token path | none |\n| `permissions.snapshot` | Accessibility and Screen Recording state | none |\n| `permissions.request` | Prompt for both permissions | none |\n| `permissions.requestAccessibility` | Prompt for Accessibility | none |\n| `settings.openAccessibility` | Open the Accessibility settings pane | none |\n| `settings.openScreenRecording` | Open the Screen Recording settings pane | none |\n\n### Apps, windows, and input\n\n| Method | Purpose | Params |\n| --- | --- | --- |\n| `app.launch` | Launch an app | `bundleId` |\n| `app.activate` | Bring an app forward | `bundleId` |\n| `window.setFrame` | Move and resize the app's window | `bundleId`, `x`, `y`, `width`, `height` |\n| `window.getFrame` | Read the app's window frame | `bundleId` |\n| `input.typeText` | Type text into the focused element | `text` |\n| `input.drag` | Pointer drag | `fromX`, `fromY`, `toX`, `toY`, optional `durationMs` (default 300), optional `filePath` |\n| `accessibility.pressElement` | Press an element by label | `bundleId`, `label`, optional `role` |\n| `accessibility.setValue` | Set an element's value | `bundleId`, `label`, `value`, optional `role` |\n\n`calculator.buttons`, `calculator.clickButton`, and `calculator.displayValue`\nexist for the Calculator demo scenario. They are not a general API.\n\n### Drive leases\n\nA drive lease tells other clients that an agent is acting on the desktop. The\nlease belongs to the connection that created it. Closing the connection\ncancels that connection's leases only.\n\n| Method | Purpose | Params |\n| --- | --- | --- |\n| `drive.begin` | Acquire a lease | `agent`, `task`, optional `mode`, `sessionId`; attention mode also needs approval, run and workspace ids, capability, resource, `protocolVersion`, `authToken` |\n| `drive.touch` | Renew a lease | `leaseId`, optional `axTier`, `actionKind` |\n| `drive.release` | Release a lease | `leaseId`, optional `outcome`, `summary` |\n| `drive.status` | Snapshot of current leases | none |\n| `sessions.prune` | Remove stale drive sessions | none |\n| `sessions.clear` | Remove all drive sessions | none |\n\n### Workspace drag\n\n| Method | Purpose | Params |\n| --- | --- | --- |\n| `workspace.dragFile` | Drag one exact run fixture from Finder into one exact Scout drop zone | lease, run, and workspace ids; fixture URL; Finder and Scout window identities; destination AX identifier; display bounds; `protocolVersion`; `authToken` |\n| `workspace.cancelOperation` | Cancel one in-flight operation | `leaseId`, `operationId`, `protocolVersion`, `authToken` |\n\n`workspace.dragFile` is not a generic coordinate drag. It rejects unknown\nparams and accepts no source point, destination point, duration, command, or\nexecutable. Action resolves exactly one Finder item by AXURL inside exactly one\ntitled Finder window. Action resolves exactly one run-scoped AX drop-zone\nidentifier inside exactly one titled `app.openscout.scout` window. All resolved\nframes and points must be inside the supplied display bounds.\n\nAn approved attention `drive.begin` acquires the global `system-pointer`\nresource and grants one `workspace.drag-file` capability. The capability is\nbound to the connection, run, and workspace, and is consumed before the\ngesture. During the drag, Action checks the lease and operation stop signals\non every step. Cleanup always attempts mouse-up and restores the saved pointer\nposition.\n\n### Capture\n\n| Method | Purpose | Params |\n| --- | --- | --- |\n| `capture.screenshotScreen` | Screenshot the main display | `output` |\n| `capture.screenshotRegion` | Screenshot a region | `x`, `y`, `width`, `height`, `output` |\n| `capture.screenshotAppWindow` | Screenshot an app window | `output`, `bundleId` or `pid` |\n| `capture.recordRegion` | Record a region | `x`, `y`, `width`, `height`, `output`, optional `fps`, `scale`, `stopFile`, `finishedFile`, `debugLog`, `includeSupervisionOverlay` |\n| `capture.recordAppWindow` | Record an app window | `bundleId`, `output`, optional `stopFile`, `finishedFile`, `debugLog` |\n\n## Recording contract\n\nThe agent does not record in its own process. A record method launches a fresh\n`Action.app` instance in `recording-probe` mode, and that probe performs the\nScreenCaptureKit capture.\n\n- A successful reply means the probe started.\n- The recording ends when the `stopFile` appears.\n- Completion is the movie at `output` plus the `finishedFile` marker. The\n marker can also contain a recording error.\n- Pass `debugLog` to collect probe logs.\n\nSee [Recording](/action/docs/recording).\n\n## What is not in this protocol\n\n- Private layers (`layer_*`), stage, observation, target resolution, and\n session artifacts are MCP and runtime features. The raw agent does not\n orchestrate them. See [Agent layer](/action/docs/agent-layer).\n- The `browser_*` tools belong to the Lattices MCP. See\n [Browser](/action/docs/browser).\n- `computer.*` methods belong to the Lattices core daemon on `lattices.sock`, not to\n this agent. See the [Lattices API](/lattices/docs/api).\n\n## From the Lattices CLI\n\n`lats action call [params-json]` sends one request and prints the\nresult. In released builds it does not attach a capability token, so it cannot\ncall privileged methods. The matching runtime reads and attaches the token automatically;\nsee [Private connection](#private-connection).\n\n```sh\nlats action call status\nlats action call window.getFrame '{\"bundleId\":\"com.apple.calculator\"}'\n```\n\n## Remote Linux hosts\n\nThe raw agent protocol is local to the Mac. For a Linux desktop, the Action\nruntime connects to `lattices-host` instead. See\n[Remote engine](/action/docs/remote-engine)." }, { "id": "action:native-runtime", "slug": "native-runtime", "product": "action", "kind": "doc", "catalogKind": "guide", "current": true, "title": "Native runtime", "description": "How Action.app and the Action agent split native responsibilities", "order": 7, "sourcePath": "products/action/docs/native-runtime.md", "url": "/action/docs/native-runtime", "rawUrl": "/action/docs/markdown/native-runtime.md", "markdownUrl": "/action/docs/native-runtime.md", "headings": [ { "depth": 2, "text": "The split", "anchor": "the-split" }, { "depth": 2, "text": "Action.app", "anchor": "action-app" }, { "depth": 2, "text": "Local agent runtime", "anchor": "local-agent-runtime" }, { "depth": 2, "text": "Why this exists", "anchor": "why-this-exists" }, { "depth": 3, "text": "1. WebKit", "anchor": "1-webkit" }, { "depth": 3, "text": "2. ScreenCaptureKit recording", "anchor": "2-screencapturekit-recording" }, { "depth": 2, "text": "Current execution flow", "anchor": "current-execution-flow" }, { "depth": 2, "text": "Important boundaries", "anchor": "important-boundaries" }, { "depth": 2, "text": "Private-layer discovery", "anchor": "private-layer-discovery" }, { "depth": 2, "text": "Scripts and helpers", "anchor": "scripts-and-helpers" } ], "tokensEstimate": 747, "frontmatter": { "title": "Native runtime", "description": "How Action.app and the Action agent split native responsibilities", "order": 7 }, "markdown": "---\ntitle: Native runtime\ndescription: How Action.app and the Action agent split native responsibilities\norder: 7\n---\n\nThis page describes the runtime split that currently makes `action` behave like\na real macOS app instead of a command wrapper pretending to be one.\n\n## The split\n\nThere are two native roles:\n\n## `Action.app`\n\nOwns:\n\n- AppKit lifecycle\n- menus and app-switcher behavior\n- WebKit windows\n- launcher UI\n- permission UX\n- recording probe execution\n\nMain file:\n\n- [ActionHostMain.swift](../native/engine/Sources/ActionHostMain.swift)\n\n## Local agent runtime\n\n`Action.app` starts the agent as a child helper,\n`Contents/Helpers/ActionAgent.app`, on `ws://127.0.0.1:4319`. The helper exits\nwith its parent.\n\nOwns:\n\n- WebSocket transport\n- automation-facing request handling\n- local runtime methods\n- bridge-friendly command orchestration\n\nMain files:\n\n- [ActionAgentRuntime.swift](../native/engine/CoreSources/ActionAgentRuntime.swift)\n- [ActionAgentClient.swift](../native/engine/CoreSources/ActionAgentClient.swift)\n- [ActionAgentCommandBridge.swift](../native/engine/Sources/ActionAgentCommandBridge.swift)\n\n## Why this exists\n\nTwo important failures pushed the design here:\n\n### 1. WebKit\n\nWebKit behaved unreliably when UI modes were entered through the wrong runtime\npath. The fix was to let AppKit own the UI lifecycle cleanly.\n\n### 2. ScreenCaptureKit recording\n\nRecording was not reliable when attempted inside the wrong lifecycle context.\nThe stable path now launches a fresh `Action.app` instance in `recording-probe`\nmode for the actual capture work.\n\nThe lesson is simple:\n\n**AppKit-dependent work should run inside a real app lifecycle.**\n\n## Current execution flow\n\nFor normal operator usage:\n\n1. `Action.app` launches\n2. launcher UI comes up\n3. local agent is started if needed\n4. UI talks to the agent for runtime state and automation methods\n\nFor recording:\n\n1. command or UI requests recording\n2. host talks to the local agent\n3. agent launches `Action.app` again in `recording-probe` mode\n4. probe performs the actual `ScreenCaptureKit` recording\n5. artifacts and finished markers are written\n\n## Important boundaries\n\nKeep these rules intact:\n\n- do not move WebKit lifecycle back into a fake headless command path\n- do not move recording implementation back into the plain headless agent path\n- keep the agent useful for orchestration and transport\n- keep AppKit-owned concerns in the app process\n\n## Private-layer discovery\n\nUse `lats action capabilities` and the active MCP `capabilities` / `tools/list`\nfor build provenance and routing. Check host quarantine before opening a private\ndisplay. Native support, runtime health, and version strings do not establish\nphysical desktop preservation. See [Agent layer](/action/docs/agent-layer).\n\n`native:doctor` builds and signs. Run it on an authorized development host.\nOn an installation target, preserve the installed bundle and rollback; deploy\na matched app and MCP runtime instead of rebuilding implicitly. Reconnect MCP\nafter runtime changes.\n\nTo assemble a matched runtime candidate on the development host, use:\n\n```bash\nbun scripts/package-agent-runtime.ts --app /path/Action.app --output /new/action-runtime --instructions-root /path/to/lattices-docs-checkout\n```\n\nThe command verifies the signed app's recorded MCP source fingerprint, bundles\nthe source and locked production dependencies, and includes the selected layer\ndocs and Action skill. It rejects a mismatched pair or existing output and does\nnot install anything. `runtime-provenance.json` records the artifact identities\nand marks host desktop-isolation acceptance as uncertified. Keep the previous\nruntime for rollback and preserve host quarantine when installing a candidate.\n\nThe extracted archive contains sibling `bin/` and `products/` directories.\nSet `ACTION_ROOT=/products/action`, not the extraction root, and keep\n`bin/host-pairing.ts` in place; the runtime imports that canonical dependency.\n\n## Scripts and helpers\n\nCall native work through package.json scripts from `products/action`:\n\n```bash\nbun run native:app:build\nbun run native:doctor\nbun run native:launch\nbun run native:relaunch\nbun run native:host -- status\nbun run native:test:screenshot\nbun run native:test:record\n```\n\nThe scripts wrap helpers in [native/engine/scripts](../native/engine/scripts). The most important wrappers are:\n\n- `build-app.sh`\n- `doctor.sh`\n- `run-app-host.sh`\n- `test-screenshot.sh`\n- `test-record.sh`\n", "content": "This page describes the runtime split that currently makes `action` behave like\na real macOS app instead of a command wrapper pretending to be one.\n\n## The split\n\nThere are two native roles:\n\n## `Action.app`\n\nOwns:\n\n- AppKit lifecycle\n- menus and app-switcher behavior\n- WebKit windows\n- launcher UI\n- permission UX\n- recording probe execution\n\nMain file:\n\n- [ActionHostMain.swift](../native/engine/Sources/ActionHostMain.swift)\n\n## Local agent runtime\n\n`Action.app` starts the agent as a child helper,\n`Contents/Helpers/ActionAgent.app`, on `ws://127.0.0.1:4319`. The helper exits\nwith its parent.\n\nOwns:\n\n- WebSocket transport\n- automation-facing request handling\n- local runtime methods\n- bridge-friendly command orchestration\n\nMain files:\n\n- [ActionAgentRuntime.swift](../native/engine/CoreSources/ActionAgentRuntime.swift)\n- [ActionAgentClient.swift](../native/engine/CoreSources/ActionAgentClient.swift)\n- [ActionAgentCommandBridge.swift](../native/engine/Sources/ActionAgentCommandBridge.swift)\n\n## Why this exists\n\nTwo important failures pushed the design here:\n\n### 1. WebKit\n\nWebKit behaved unreliably when UI modes were entered through the wrong runtime\npath. The fix was to let AppKit own the UI lifecycle cleanly.\n\n### 2. ScreenCaptureKit recording\n\nRecording was not reliable when attempted inside the wrong lifecycle context.\nThe stable path now launches a fresh `Action.app` instance in `recording-probe`\nmode for the actual capture work.\n\nThe lesson is simple:\n\n**AppKit-dependent work should run inside a real app lifecycle.**\n\n## Current execution flow\n\nFor normal operator usage:\n\n1. `Action.app` launches\n2. launcher UI comes up\n3. local agent is started if needed\n4. UI talks to the agent for runtime state and automation methods\n\nFor recording:\n\n1. command or UI requests recording\n2. host talks to the local agent\n3. agent launches `Action.app` again in `recording-probe` mode\n4. probe performs the actual `ScreenCaptureKit` recording\n5. artifacts and finished markers are written\n\n## Important boundaries\n\nKeep these rules intact:\n\n- do not move WebKit lifecycle back into a fake headless command path\n- do not move recording implementation back into the plain headless agent path\n- keep the agent useful for orchestration and transport\n- keep AppKit-owned concerns in the app process\n\n## Private-layer discovery\n\nUse `lats action capabilities` and the active MCP `capabilities` / `tools/list`\nfor build provenance and routing. Check host quarantine before opening a private\ndisplay. Native support, runtime health, and version strings do not establish\nphysical desktop preservation. See [Agent layer](/action/docs/agent-layer).\n\n`native:doctor` builds and signs. Run it on an authorized development host.\nOn an installation target, preserve the installed bundle and rollback; deploy\na matched app and MCP runtime instead of rebuilding implicitly. Reconnect MCP\nafter runtime changes.\n\nTo assemble a matched runtime candidate on the development host, use:\n\n```bash\nbun scripts/package-agent-runtime.ts --app /path/Action.app --output /new/action-runtime --instructions-root /path/to/lattices-docs-checkout\n```\n\nThe command verifies the signed app's recorded MCP source fingerprint, bundles\nthe source and locked production dependencies, and includes the selected layer\ndocs and Action skill. It rejects a mismatched pair or existing output and does\nnot install anything. `runtime-provenance.json` records the artifact identities\nand marks host desktop-isolation acceptance as uncertified. Keep the previous\nruntime for rollback and preserve host quarantine when installing a candidate.\n\nThe extracted archive contains sibling `bin/` and `products/` directories.\nSet `ACTION_ROOT=/products/action`, not the extraction root, and keep\n`bin/host-pairing.ts` in place; the runtime imports that canonical dependency.\n\n## Scripts and helpers\n\nCall native work through package.json scripts from `products/action`:\n\n```bash\nbun run native:app:build\nbun run native:doctor\nbun run native:launch\nbun run native:relaunch\nbun run native:host -- status\nbun run native:test:screenshot\nbun run native:test:record\n```\n\nThe scripts wrap helpers in [native/engine/scripts](../native/engine/scripts). The most important wrappers are:\n\n- `build-app.sh`\n- `doctor.sh`\n- `run-app-host.sh`\n- `test-screenshot.sh`\n- `test-record.sh`" }, { "id": "action:remote-engine", "slug": "remote-engine", "product": "action", "kind": "doc", "catalogKind": "guide", "current": true, "title": "Action with a remote Linux host", "description": "Action can observe and act on a Linux desktop running lattices-host. Runs, traces and artifacts stay on the controller; the remote host performs the work.", "order": null, "sourcePath": "apps/site/content/action/remote-engine.md", "url": "/action/docs/remote-engine", "rawUrl": "/action/docs/markdown/remote-engine.md", "markdownUrl": "/action/docs/remote-engine.md", "headings": [ { "depth": 1, "text": "Action with a remote Linux host", "anchor": "action-with-a-remote-linux-host" }, { "depth": 2, "text": "Prerequisites", "anchor": "prerequisites" }, { "depth": 2, "text": "Pair through the gateway", "anchor": "pair-through-the-gateway" }, { "depth": 2, "text": "Connect the MCP runtime", "anchor": "connect-the-mcp-runtime" } ], "tokensEstimate": 651, "frontmatter": {}, "markdown": "# Action with a remote Linux host\n\nAction can observe and act on a Linux desktop running `lattices-host`. Runs, traces and artifacts stay on the controller; the remote host performs the work.\n\n## Prerequisites\n\n- A reachable Linux desktop with `lattices-host` installed and configured.\n- An Action runtime that includes `RemoteEngine` support, plus Bun on the controller.\n- The host's address and an approved pairing for this controller.\n\nThe macOS app installer does not provision a Linux host. See the [Linux host source and setup](https://github.com/arach/lattices/tree/main/packages/host-linux) before continuing. Match setup instructions to the revision you install; the homepage demo is not a Linux acceptance test.\n\n## Pair through the gateway\n\nUse Lattices 0.15.0 with Action 0.4.0 and a host at the same Lattices version. Earlier versions use direct-host transport; see [Runtime versions](/lattices/docs/concepts#private-ipc-and-the-gateway).\n\nStart the local host on the Linux desktop, then start its gateway:\n\n```sh\nlats gateway start --tailnet\n```\n\nExpect the gateway to bind the host's Tailscale address. Both machines need Tailscale. The default policy admits devices owned by the host's Tailscale user. The gateway runs until stopped; this command does not install a login service.\n\nOn the controller, request read-only access:\n\n```sh\nlats --host your-linux-host pair --read-only\n```\n\nThe command waits for approval. On the Linux host, inspect the request and compare its fingerprint with the controller's code. Replace `CLIENT_ID` with the matching pending client:\n\n```sh\nlats gateway clients\nlats gateway approve CLIENT_ID\n```\n\nAfter the controller reports that it is paired, verify the destination:\n\n```sh\nlats --host your-linux-host call host.describe\n```\n\nExpect the intended Linux hostname, platform, and capabilities. Stop if the identity differs. Existing direct-host pairings do not grant gateway access; pair again because the gateway has its own key and trust store.\n\nRead-only access is enough for this check and Action's health check below. Desktop input and app launching require a separate request with `lats --host your-linux-host pair --drive`, followed by host approval of the expanded grants. The `--drive` scope covers Lattices host operations used by `RemoteEngine`, including screen capture and the `shell` scope for typing and launching commands. It does not grant access to the separate Action agent. Review [Gateway permissions](/lattices/docs/concepts#private-ipc-and-the-gateway) before approving control.\n\n## Connect the MCP runtime\n\nFrom the Action product directory in a runtime or checkout with remote-engine support, replace `your-linux-host` with the paired gateway host name:\n\n```sh\nACTION_REMOTE_HOST=your-linux-host bun packages/mcp/src/index.ts\n```\n\nFor an MCP harness, set `ACTION_REMOTE_HOST` in the server environment rather than relying on the environment of an unrelated terminal. A custom port uses `host:port`.\n\nAsk the agent to call `action.health` first. Expect the remote host's platform and capabilities in `diagnostics`. Stop if it reports the wrong host or a connection error.\n\nThe runtime supports host snapshots and OCR, point/text target resolution, and input actions supported by that host. Native macOS Accessibility, stage/backdrop overlays, and the companion worker are not available on Linux. `action.record.*` remains native-only; guided sessions use the remote engine's recording path instead.\n\nImplementation reference: [Remote engine](https://github.com/arach/lattices/blob/main/products/action/docs/remote-engine.md).\n", "content": "# Action with a remote Linux host\n\nAction can observe and act on a Linux desktop running `lattices-host`. Runs, traces and artifacts stay on the controller; the remote host performs the work.\n\n## Prerequisites\n\n- A reachable Linux desktop with `lattices-host` installed and configured.\n- An Action runtime that includes `RemoteEngine` support, plus Bun on the controller.\n- The host's address and an approved pairing for this controller.\n\nThe macOS app installer does not provision a Linux host. See the [Linux host source and setup](https://github.com/arach/lattices/tree/main/packages/host-linux) before continuing. Match setup instructions to the revision you install; the homepage demo is not a Linux acceptance test.\n\n## Pair through the gateway\n\nUse Lattices 0.15.0 with Action 0.4.0 and a host at the same Lattices version. Earlier versions use direct-host transport; see [Runtime versions](/lattices/docs/concepts#private-ipc-and-the-gateway).\n\nStart the local host on the Linux desktop, then start its gateway:\n\n```sh\nlats gateway start --tailnet\n```\n\nExpect the gateway to bind the host's Tailscale address. Both machines need Tailscale. The default policy admits devices owned by the host's Tailscale user. The gateway runs until stopped; this command does not install a login service.\n\nOn the controller, request read-only access:\n\n```sh\nlats --host your-linux-host pair --read-only\n```\n\nThe command waits for approval. On the Linux host, inspect the request and compare its fingerprint with the controller's code. Replace `CLIENT_ID` with the matching pending client:\n\n```sh\nlats gateway clients\nlats gateway approve CLIENT_ID\n```\n\nAfter the controller reports that it is paired, verify the destination:\n\n```sh\nlats --host your-linux-host call host.describe\n```\n\nExpect the intended Linux hostname, platform, and capabilities. Stop if the identity differs. Existing direct-host pairings do not grant gateway access; pair again because the gateway has its own key and trust store.\n\nRead-only access is enough for this check and Action's health check below. Desktop input and app launching require a separate request with `lats --host your-linux-host pair --drive`, followed by host approval of the expanded grants. The `--drive` scope covers Lattices host operations used by `RemoteEngine`, including screen capture and the `shell` scope for typing and launching commands. It does not grant access to the separate Action agent. Review [Gateway permissions](/lattices/docs/concepts#private-ipc-and-the-gateway) before approving control.\n\n## Connect the MCP runtime\n\nFrom the Action product directory in a runtime or checkout with remote-engine support, replace `your-linux-host` with the paired gateway host name:\n\n```sh\nACTION_REMOTE_HOST=your-linux-host bun packages/mcp/src/index.ts\n```\n\nFor an MCP harness, set `ACTION_REMOTE_HOST` in the server environment rather than relying on the environment of an unrelated terminal. A custom port uses `host:port`.\n\nAsk the agent to call `action.health` first. Expect the remote host's platform and capabilities in `diagnostics`. Stop if it reports the wrong host or a connection error.\n\nThe runtime supports host snapshots and OCR, point/text target resolution, and input actions supported by that host. Native macOS Accessibility, stage/backdrop overlays, and the companion worker are not available on Linux. `action.record.*` remains native-only; guided sessions use the remote engine's recording path instead.\n\nImplementation reference: [Remote engine](https://github.com/arach/lattices/blob/main/products/action/docs/remote-engine.md)." }, { "id": "action:skill", "slug": "skill", "product": "action", "kind": "doc", "catalogKind": "guide", "current": true, "title": "Skills", "description": "Review patterns an AI agent follows when changing Action", "order": 11, "sourcePath": "products/action/docs/skill.md", "url": "/action/docs/skill", "rawUrl": "/action/docs/markdown/skill.md", "markdownUrl": "/action/docs/skill.md", "headings": [ { "depth": 2, "text": "Native runtime review", "anchor": "native-runtime-review" }, { "depth": 2, "text": "Recording review", "anchor": "recording-review" }, { "depth": 2, "text": "Agent API review", "anchor": "agent-api-review" }, { "depth": 2, "text": "Hermes Action", "anchor": "hermes-action" } ], "tokensEstimate": 293, "frontmatter": { "title": "Skills", "description": "Review patterns an AI agent follows when changing Action", "order": 11 }, "markdown": "---\ntitle: Skills\ndescription: Review patterns an AI agent follows when changing Action\norder: 11\n---\n\nThese are project-specific skills or task patterns an AI agent should follow.\n\n## Native runtime review\n\nUse when:\n\n- changing AppKit lifecycle code\n- changing agent startup or bridge logic\n- changing WebKit entry paths\n\nInstructions:\n\n1. read [Native runtime](/action/docs/native-runtime)\n2. verify whether the change belongs in app code or agent code\n3. keep AppKit-owned behavior in `Action.app`\n4. verify the signed app still builds\n\n## Recording review\n\nUse when:\n\n- changing `ScreenCaptureKit` recording\n- changing stop-file or finished-file handling\n- changing probe launch behavior\n\nInstructions:\n\n1. read [Recording](/action/docs/recording)\n2. inspect the probe launcher and probe app runner\n3. preserve `.mov` + `.finished` behavior\n4. test a real recording path, not just a unit-level refactor\n\n## Agent API review\n\nUse when:\n\n- adding WebSocket methods\n- changing request or response shape\n- updating CLI-to-agent behavior\n\nInstructions:\n\n1. read [Api](/action/docs/api)\n2. update the protocol source of truth\n3. confirm method names remain documented\n4. verify callers still match the transport contract\n\n## Hermes Action\n\nUse when:\n\n- connecting Hermes to the native Action runtime\n- using Action MCP tools from a harness\n- recording or inspecting native macOS surfaces through Hermes\n\nInstructions:\n\n1. read [docs/harnesses/hermes-action.md](harnesses/hermes-action.md)\n2. use [skills/hermes-action/SKILL.md](../skills/hermes-action/SKILL.md) as the harness doctrine\n3. prefer observe and resolve before act\n4. treat recording start as asynchronous until `.finished` or `action.record.status` confirms completion\n", "content": "These are project-specific skills or task patterns an AI agent should follow.\n\n## Native runtime review\n\nUse when:\n\n- changing AppKit lifecycle code\n- changing agent startup or bridge logic\n- changing WebKit entry paths\n\nInstructions:\n\n1. read [Native runtime](/action/docs/native-runtime)\n2. verify whether the change belongs in app code or agent code\n3. keep AppKit-owned behavior in `Action.app`\n4. verify the signed app still builds\n\n## Recording review\n\nUse when:\n\n- changing `ScreenCaptureKit` recording\n- changing stop-file or finished-file handling\n- changing probe launch behavior\n\nInstructions:\n\n1. read [Recording](/action/docs/recording)\n2. inspect the probe launcher and probe app runner\n3. preserve `.mov` + `.finished` behavior\n4. test a real recording path, not just a unit-level refactor\n\n## Agent API review\n\nUse when:\n\n- adding WebSocket methods\n- changing request or response shape\n- updating CLI-to-agent behavior\n\nInstructions:\n\n1. read [Api](/action/docs/api)\n2. update the protocol source of truth\n3. confirm method names remain documented\n4. verify callers still match the transport contract\n\n## Hermes Action\n\nUse when:\n\n- connecting Hermes to the native Action runtime\n- using Action MCP tools from a harness\n- recording or inspecting native macOS surfaces through Hermes\n\nInstructions:\n\n1. read [docs/harnesses/hermes-action.md](harnesses/hermes-action.md)\n2. use [skills/hermes-action/SKILL.md](../skills/hermes-action/SKILL.md) as the harness doctrine\n3. prefer observe and resolve before act\n4. treat recording start as asynchronous until `.finished` or `action.record.status` confirms completion" } ], "prompts": [] }