Agent API

Lattices 0.15.1 and later serve the agent API over authenticated WebSocket on ~/.lattices/run/lattices.sock. The same app hosts voice input on ~/.lattices/run/voice.sock (see Voice). ~120 RPC methods and real-time events. Prefer desktop.snapshot when you need to know what is in front of the user, and lats map when you want that same state drawn as ASCII.

Quick start

1. Launch the server (it starts with the menu bar app):

lats app

2. Check that it's running:

lats daemon status

3. Call a method from Node.js:

import { daemonCall } from '@lattices/lattices'

const windows = await daemonCall('windows.list')
console.log(windows) // [{ wid, app, title, frame, ... }, ...]

From another language, use a WebSocket client that supports Unix sockets and the authentication headers below.

Private connection

Install the Lattices app and CLI at version 0.15.2 together. Older clients that connect directly to TCP must migrate to the private transport.

• The daemon listens on a private Unix socket, ~/.lattices/run/lattices.sock, and writes a capability token to ~/.lattices/run/lattices.token. The directory is mode 0700; both files are mode 0600.
• Directory and token permissions restrict local access to the same user. The macOS server also checks the peer's user ID. The WebSocket upgrade must use Host: localhost, carry the token in x-lattices-token, and send no Origin header.
• lats and daemonCall() find the socket and token for you. Code above keeps working. A plain websocat ws://127.0.0.1:9399 does not: no TCP port is open by default. The macOS app can enable loopback TCP with LATTICES_ALLOW_LOOPBACK_TCP=1; that port still requires the token. The Linux host has no TCP compatibility listener.
• The shared client sends local tokens over private IPC by default. An explicit TCP target, including a loopback address that could be an SSH tunnel, receives no local token unless loopback compatibility is explicitly enabled. The CLI and Pi client require LATTICES_ALLOW_LOOPBACK_TCP=1; programmatic daemonCallTo() callers can deliberately supply tokenFile for a loopback endpoint. Non-loopback targets never receive local tokens.
• Remote clients reach a host only through the opt-in tailnet gateway, after a person on that host approves the pairing. See Remote hosts.

See Private IPC and the gateway for every product's socket.

Wire protocol

lattices uses a JSON-RPC-style protocol over authenticated WebSocket on the private socket.

Request

{
  "id": "unique-string",
  "method": "windows.list",
  "params": { "wid": 1234 }
}

Response

{
  "id": "unique-string",
  "result": [ ... ],
  "error": null
}

Errors

Method names and aliases

Methods are named <domain>.<verb>, with plural domains: windows.place, sessions.launch, layers.switch (LAT-012). The pre-LAT-012 names still work as aliases, and api.schema lists them under aliases (old name → new name):

Action receipts keep their own type values (window.place, window.move); those name the action, not the RPC method. The Node.js daemonCall() client and LatticesKit retry a new name under its old one when an older daemon answers Unknown method.

Connection lifecycle

• The server starts when the menu bar app launches and stops when it quits.
• Every connection requires the local token at the WebSocket handshake; see Private connection.
• The Node.js daemonCall() client opens a fresh connection per call and closes it when the response arrives. For event subscriptions, hold the connection open (see Reactive event pattern).
• If the server restarts (e.g. after lats app restart), existing connections are dropped. Clients should reconnect and treat it as stateless. There is no session resumption.

Remote hosts

In 0.15.1, neither the macOS app nor lattices-host opens a remote listener. Remote access uses the separate gateway, which the host user starts explicitly:

# On the host
lats gateway start --tailnet

# On the client: request access, then wait for approval on the host
lats --host HOST pair --read-only
# A viewer also needs screen contents (screenshots, streams, OCR):
lats --host HOST pair --read-only --screen
# Or request exactly these methods instead of the read-only scope:
lats --host HOST pair --grant lattices:windows.list --grant speech:speech.enqueue

# On the host: check the fingerprint and grants, then approve
lats gateway clients
lats gateway approve CLIENT_ID

The gateway binds only the host's Tailscale address, on port 9399 by default. Core method names do not change. Methods of other products go through gateway.call with { "product": "speech", "method": "speech.enqueue", "params": { "text": "Build finished" } }. Each grant names one exact product:method, and only reviewed methods can be granted. Re-pairing with the same client key adds approved grants to the existing set. To reduce access, revoke the client and pair again. --grant cannot be combined with --read-only, --drive, --screen or --shell.

Core scopes are read, screen (screenshots, streams, OCR, terminal contents), act, drive (desktop input) and shell (typing text and launching commands). --read-only requests read, plain pair requests read, screen and act, and --drive requests all five. Stored grants are exact methods, so clients paired before screen and shell were split out keep what they had. Clients paired directly with a host before this change must pair again with the gateway.

The native Machines/Hosts view reuses this CLI gateway pairing and signs its requests. Pairing a companion device alone does not authorize remote host status or screens. The Pi extension and Swift SDK do not add gateway signatures to explicit remote endpoints; a host or URL override alone is not a paired gateway client.

Remote core event subscriptions are limited to windows.changed, spaces.changed, layers.changed and host.healthChanged. A client needs lattices:events.subscribe and the corresponding read grant (windows.list, spaces.list, layers.list or daemon.status). Unsubscribing requires lattices:events.unsubscribe. Capture stream lifecycle and activity events go directly to the connection that owns the stream; see capture.stream.* below. Other event streams stay local.

Node.js client

lattices ships a zero-dependency WebSocket client that works with Node.js 18+. It handles connection, framing, and request/response matching internally.

daemonCall(method, params?, timeoutMs?)

Send an RPC call and await the response.

import { daemonCall } from '@lattices/lattices'

// Read-only
const status = await daemonCall('daemon.status')
const windows = await daemonCall('windows.list')
const win = await daemonCall('windows.get', { wid: 1234 })

// Mutations
await daemonCall('sessions.launch', { path: '/Users/you/dev/myapp' })
await daemonCall('windows.place', {
  session: 'myapp-a1b2c3',
  placement: { kind: 'tile', value: 'left' }
})

// Custom timeout (default: 3000ms)
await daemonCall('projects.scan', null, 10000)

Returns the result field from the response. Throws if the server returns an error, the connection fails, or the timeout is reached.

isDaemonRunning()

Check if the server is reachable.

import { isDaemonRunning } from '@lattices/lattices'

if (await isDaemonRunning()) {
  console.log('daemon is up')
}

Returns true if daemon.status responds within 1 second.

TypeScript SDK facade

The CLI is a human/debug surface. Product code and agents should prefer the typed SDK facade, which validates params with dependency-free schemas and calls the same daemon methods directly.

import { cua } from '@lattices/lattices/cua'

await cua.magicCursor({
  app: 'Scout',
  xRatio: 0.52,
  yRatio: 0.91,
  text: 'What are the most important docs in this project, and what would an agent say after reading them?',
  treatment: 'execute',
  trail: 'comet',
  motion: 'rush',
  trajectory: 'overshoot',
  glow: 'halo',
  idle: 'wiggle',
  edge: 'ripple',
})

await cua.click({
  app: 'Scout',
  xRatio: 0.74,
  yRatio: 0.95,
  transport: 'ax',
  axLabel: 'Send',
  noFocus: true,
  treatment: 'execute',
})

@lattices/lattices/cua exposes this typed CUA module. Existing imports from @arach/lattices/cua and @lattices/cli/cua remain supported.

Error handling

daemonCall throws on errors — always wrap calls in try/catch:

import { daemonCall } from '@lattices/lattices'

try {
  await daemonCall('sessions.launch', { path: '/nonexistent' })
} catch (err) {
  // err.message is one of:
  //   "Not found"              — resource doesn't exist
  //   "Missing parameter: ..." — required param missing
  //   "Unknown method: ..."    — bad method name
  //   "Daemon request timed out" — no response within timeout
  //   ECONNREFUSED             — daemon not running
  console.error('Daemon error:', err.message)
}

Runs And Capture

Runs are local executions that produce trace events and artifacts. Capture methods write into the Lattices run store under ~/Library/Application Support/Lattices/Runs/.

capture.screenshotWindow

Capture a window as a PNG artifact. If no target is provided, Lattices captures the frontmost non-Lattices window.

Params:

await daemonCall('capture.screenshotWindow', { source: 'agent' })

await daemonCall('capture.screenshotWindow', {
  session: 'frontend-a1b2c3',
  filename: 'before-layout.png'
})

capture.still

Return one compressed frame inline. Other machines pull these one at a time to show this Mac in their remote viewer. Nothing is written to disk or the clipboard. Pass a window or a region, or neither for a display.

The result has format, width, height, base64 data, and frame: the captured rect in the same coordinates that computer.* input takes.

capture.stream.*

Push H.264 of a display, window or region as binary WebSocket messages on the connection that asked, while a viewer is on screen. capture.still stays for thumbnails, idle views and hosts without streaming. host.describe lists capture.stream only when the host can stream now.

capture.stream.start takes the same target fields as capture.still, plus:

The result is {streamId, codec, width, height, frame:{x,y,w,h}, display}. display is {index, name} for a display target and null otherwise. Hosts that report capture activity also include activity, as described below.

• capture.stream.ack {streamId, seq} acknowledges every frame up to seq. Send one per frame received. Hosts limit the number of frames in flight. With frames outstanding and no ack for 3 s, the host ends the stream. An ack for an old seq changes nothing and still answers {ok:true}, so viewers use it to check that a quiet stream is alive. Hosts that report capture activity include the current activity in successful ACK responses.
• capture.stream.keyframe {streamId} makes the next frame an IDR.
• capture.stream.stop {streamId} stops the stream without an event.
• Streams belong to their connection. Only that connection can ack, request a keyframe or stop them, and they all stop when it closes.
• When the host ends a stream itself, it sends {event:"capture.stream.ended", data:{streamId, reason}} to the owning connection, whatever it subscribed to. Reasons are ack-timeout, window-closed, encoder-failed: … and, on a Mac, capture-stopped: ….

Host activity and variable frame delivery

Current macOS source builds suppress unchanged pictures before H.264 encoding. The host compares the visible pixels of complete capture frames and accepts ScreenCaptureKit's explicit unchanged-frame status as idle evidence. After at least one second without a picture change, with unchanged-frame evidence, the host reports idle. A pixel or capture-geometry change reports active immediately and resumes frame delivery, subject to the ACK limit.

The capture rate remains at the requested fps ceiling so the host can observe new changes promptly. The encoded video rate can fall to zero while the picture is unchanged. The viewer holds the last picture. A requested recovery keyframe can still send the same picture without changing the activity state.

Supporting hosts include this optional object in start and successful ACK responses:

{"activity":{"state":"idle","source":"host","activeFPS":30,"idleFPS":0,"revision":2}}

state is active, idle or unknown. unknown means the host does not have current evidence of picture activity, including before the first frame or when capture is blank or suspended. activeFPS is the configured frame rate ceiling; idleFPS is the intended unchanged-picture rate, currently zero. Neither field is a measurement of received frames.

revision is a nonnegative integer, starting at zero for each stream and increasing on each state change. Viewers ignore activity reports with an older revision so a delayed ACK cannot overwrite a newer activity event.

When the state changes, the host sends {event:"capture.stream.activity", data:{streamId, activity}} only to the connection that owns the stream. The gateway forwards this event for streams owned by that viewer. No event subscription is required.

Older hosts and the current Linux host omit activity. Viewers must not infer idle from missing frames, low received FPS or an omitted activity object. Those hosts do not promise the macOS unchanged-picture suppression behavior.

Binary frames

Each binary message is one access unit, big-endian:

Every keyframe carries its parameter sets. There are no B-frames. The codec lives in apps/mac/Sources/Core/Hosts/RemoteVideoFrame.swift and packages/host-linux/src/stream.ts.

capture.screenshotDisplay

Capture one complete composited display as a PNG artifact without focusing a window or changing visible overlays. The image includes the menu bar, pointer by default, and Lattices-owned surfaces already visible on that display, which makes it suitable for snapshots of transient UI such as Hyper+G. When no display is supplied, Lattices captures the display under the pointer.

await daemonCall('capture.screenshotDisplay', {
  displayIndex: 0,
  clipboard: true,
  filename: 'hyper-g-layout.png',
  source: 'agent'
})

The CLI uses the positional index as display, always writes the run artifact, and copies the PNG only when --clipboard is present. --delay is a CLI-side wait measured in seconds; it is not an RPC parameter.

lats capture display 0 --clipboard --delay 3 --filename hyper-g-layout.png

capture.screenshotRegion

Capture an explicit screen rectangle as a PNG artifact. If no rectangle is provided, the endpoint resolves wid/session/app/frontmost target and captures that window frame.

await daemonCall('capture.screenshotRegion', {
  x: 120,
  y: 160,
  width: 640,
  height: 420,
  filename: 'composer-region.png'
})

capture.zoomArtifact

Crop and zoom an existing image artifact into a new PNG artifact on the same run. Coordinates are image pixels; ratio fields are relative to the source image.

await daemonCall('capture.zoomArtifact', {
  runId: 'run_...',
  artifactId: 'art_...',
  xRatio: 0.35,
  yRatio: 0.7,
  widthRatio: 0.3,
  heightRatio: 0.2,
  scale: 3
})

vision.analyzeWindow / vision.analyzeArtifact

Run explicit local OCR-based visual analysis. These endpoints do not call a cloud vision model; they use the same on-device Vision OCR path as Lattices OCR and return fullText, text blocks, a short answer, and optional verified when contains or notContains is supplied.

await daemonCall('vision.analyzeWindow', {
  app: 'Safari',
  instruction: 'Read the visible page heading',
  contains: 'Documentation'
})

CLI:

lats capture window
lats capture display 0 --clipboard --delay 3
lats runs
lats runs run_20260617-120000_a1b2c3
lats terminals
lats terminals --refresh
lats computer prepare --text "# hello" --treatment stage
lats call computer.windowState '{"app":"Finder","maxDepth":4}'
lats call computer.pressKey '{"app":"Finder","key":"escape","treatment":"stage"}'
lats call computer.hotkey '{"app":"Xcode","shortcut":"command+b","treatment":"stage"}'
lats computer focus-window --wid 7258 --treatment present
lats computer cursor --style marker --shape chevron --angle-deg -8 --label typing
lats computer launch-app Scout
lats computer scout --treatment present
lats computer scout "Draft memo text" --execute
lats computer type-window --app Scout --text "Draft memo text" --x-ratio .5 --y-ratio .86 --execute
lats computer click --app Scout --x-ratio .5 --y-ratio .86 --execute
lats cua click --app Scout --x-ratio .74 --y-ratio .95 --transport ax --ax-label Send --execute
lats call computer.scroll '{"app":"Safari","direction":"down","amount":420,"treatment":"stage"}'
lats call computer.drag '{"app":"Finder","fromXRatio":0.2,"fromYRatio":0.2,"toXRatio":0.7,"toYRatio":0.7,"treatment":"stage"}'
lats call computer.doubleClick '{"app":"Finder","xRatio":0.5,"yRatio":0.5,"treatment":"stage"}'
lats call computer.rightClick '{"app":"Finder","xRatio":0.5,"yRatio":0.5,"treatment":"stage"}'
lats call computer.verify '{"app":"Safari","mode":"ocr","contains":"Docs"}'
lats call capture.screenshotRegion '{"app":"Safari","filename":"safari-region.png"}'
lats call vision.analyzeWindow '{"app":"Safari","instruction":"Read visible text"}'
lats call browser.getText '{"app":"Safari"}'
lats call browser.queryDom '{"app":"Safari","selector":"h1","allowAutomation":true}'
lats computer type-text --text "# hello from lattices"
lats computer demo-terminal --dry-run

Computer Action Treatments

Computer-use endpoints accept a treatment field that controls how intrusive the action may be:

The legacy dryRun: true flag maps to stage.

Background-Safety Receipts

Computer-use action responses include a backgroundSafety receipt plus top-level compatibility fields:

{
  "backgroundSafety": {
    "route": "ax.press",
    "lane": "same_session",
    "session": "parent",
    "background_safe": true,
    "backgroundSafe": true,
    "cursor_moved": false,
    "foreground_changed": false
  }
}

Agents should treat background_safe: true as the signal that the call stayed out of the user's foreground session. AX, tmux, iTerm session, overlay, staged, and observe routes can be background-safe. Pointer input, app activation, window focus, global keyboard input, and foreground typing report background_safe: false.

computer.prepare

Resolve and score terminal candidates for a future computer-use action. This is the least intrusive endpoint: by default it observes the target and captures an artifact, but it does not focus or type.

Params:

await daemonCall('computer.prepare', {
  text: '# review before typing',
  treatment: 'stage'
})

computer.windowState

Inspect a target window's Accessibility tree and return snapshot-local element ids (e1, e2, ...), a flat elements list, and a compact treeMarkdown view. mode: "ax" avoids Screen Recording. Use mode: "both" or capture: true when you also want a screenshot artifact linked to a run. The endpoint is classified as a write because those capture modes create run artifacts.

Params:

await daemonCall('computer.windowState', {
  app: 'Finder',
  maxDepth: 4,
  maxElements: 120
})

await daemonCall('computer.windowState', {
  wid: 7258,
  mode: 'both',
  source: 'agent'
})

computer.elementAction

Stage or execute an Accessibility action against an element returned by a recent computer.windowState call. Snapshot element ids are intentionally local to the daemon process and expire from the in-memory cache after a short window.

Params:

const state = await daemonCall('computer.windowState', {
  app: 'Calculator',
  maxDepth: 6
})

await daemonCall('computer.elementAction', {
  snapshotId: state.snapshotId,
  elementId: 'e7',
  action: 'press',
  treatment: 'execute',
  backgroundSafe: true
})

computer.typeElement / computer.setValue

Stage or execute text/value insertion against an element returned by a recent computer.windowState call. computer.typeElement accepts text; computer.setValue accepts value and otherwise uses the same behavior. In stage, Lattices records the intended value without changing the app.

Params:

const state = await daemonCall('computer.windowState', {
  app: 'Notes',
  maxDepth: 8
})

await daemonCall('computer.typeElement', {
  snapshotId: state.snapshotId,
  elementId: 'e12',
  text: 'Draft note',
  treatment: 'execute',
  backgroundSafe: true
})

computer.pressKey / computer.hotkey

Stage or execute keyboard input. computer.pressKey sends one key, such as escape, enter, tab, an arrow key, or a single character. computer.hotkey sends a shortcut from either shortcut: "command+shift+p" or key plus modifiers. Both default to stage; execute requires an explicit wid, app, or session target unless allowGlobal: true is passed.

Params:

await daemonCall('computer.pressKey', {
  app: 'Finder',
  key: 'escape',
  treatment: 'stage'
})

await daemonCall('computer.hotkey', {
  app: 'Xcode',
  shortcut: 'command+b',
  treatment: 'execute'
})

computer.focusWindow

Resolve a target window, optionally capture it, focus it, and verify the focused window id. Use treatment: 'observe' or stage to plan without presenting.

Params:

await daemonCall('computer.focusWindow', {
  app: 'iTerm2',
  treatment: 'present'
})

computer.showCursor

Resolve the current cursor location and show a visible cursor appearance. This is the cursor equivalent of a typing action: it records a run, cursor target, appearance parameters, and trace events. Use observe or stage to plan without showing anything.

Params:

await daemonCall('computer.showCursor', {
  style: 'marker',
  shape: 'chevron',
  angleDeg: -8,
  size: 'regular',
  color: 'white',
  treatment: 'present'
})

computer.launchApp

Launch or focus a normal macOS app and record the result as a run. Use dryRun: true or treatment: 'stage' to plan without launching.

Params:

await daemonCall('computer.launchApp', {
  app: 'Scout',
  treatment: 'present'
})

computer.typeWindowText

Focus a normal app window and insert text. If click coordinates are provided, Lattices clicks that target before typing. Coordinates can be absolute screen points (x, y) or ratios inside the target window (xRatio, yRatio). For window ratios, 0,0 is the top-left of the window and 1,1 is the bottom-right.

Params:

await daemonCall('computer.typeWindowText', {
  app: 'Scout',
  text: 'Draft memo text',
  xRatio: 0.5,
  yRatio: 0.86,
  treatment: 'execute'
})

computer.click / computer.doubleClick / computer.rightClick

Stage or execute a click target. stage records the target without clicking. In execute, transport: "auto" prefers AXPress on the resolved accessibility button/control before falling back to a pointer click. Use transport: "ax" or noFocus: true when the action must not focus the app or move the hardware pointer. computer.doubleClick forces two pointer clicks; computer.rightClick forces the right button. When a window target is provided, ratios are relative to that window.

Params:

await daemonCall('computer.click', {
  app: 'Scout',
  xRatio: 0.5,
  yRatio: 0.86,
  treatment: 'execute'
})

await daemonCall('computer.click', {
  app: 'Scout',
  xRatio: 0.74,
  yRatio: 0.95,
  transport: 'ax',
  axLabel: 'Send',
  noFocus: true,
  treatment: 'execute'
})

computer.scroll / computer.drag

Stage or execute richer pointer input while keeping the same run/capture model. computer.scroll posts scroll wheel events at an absolute or window-relative point. computer.drag interpolates a pointer drag from a start point to an end point. Both default to stage.

Scroll params:

Drag params:

await daemonCall('computer.scroll', {
  app: 'Safari',
  direction: 'down',
  amount: 420,
  treatment: 'execute'
})

await daemonCall('computer.drag', {
  app: 'Finder',
  fromXRatio: 0.2,
  fromYRatio: 0.2,
  toXRatio: 0.7,
  toYRatio: 0.7,
  treatment: 'stage'
})

computer.verify

Verify a post-action outcome using OCR, AX snapshot text/value, or artifact hash comparison. OCR verification captures/analyzes the target when needed. AX verification can use a snapshotId/elementId from computer.windowState.

await daemonCall('computer.verify', {
  app: 'Safari',
  mode: 'ocr',
  contains: 'Lattices'
})

const state = await daemonCall('computer.windowState', { app: 'Notes' })
await daemonCall('computer.verify', {
  mode: 'ax',
  snapshotId: state.snapshotId,
  elementId: 'e4',
  contains: 'Draft'
})

browser.getText / browser.queryDom / browser.executeJavascript

Browser primitives are intentionally gated. browser.getText reads visible text through Accessibility and does not enable JavaScript automation. DOM querying and arbitrary JavaScript use AppleScript browser automation and require allowAutomation: true; JavaScript mutation also requires treatment: "execute".

Supported browser app names: Safari, Google Chrome, Chrome, Brave Browser, Microsoft Edge, and Arc.

await daemonCall('browser.getText', { app: 'Safari' })

await daemonCall('browser.queryDom', {
  app: 'Safari',
  selector: 'h1, main a',
  allowAutomation: true
})

await daemonCall('browser.executeJavascript', {
  app: 'Safari',
  script: 'document.title',
  treatment: 'execute',
  allowAutomation: true
})

computer.demoScout

Warm up a Scout memo/demo recording run. In present mode it launches or focuses Scout and records a run without typing. In execute mode it can click the likely composer area, type a draft, and optionally press Enter when enter or send is true. Dry-run/stage mode does not capture by default, so it works before Screen Recording permission is granted.

Params:

await daemonCall('computer.demoScout', { dryRun: true })

await daemonCall('computer.demoScout', {
  treatment: 'present',
  capture: false
})

await daemonCall('computer.demoScout', {
  text: 'Draft memo text',
  treatment: 'execute',
  send: false
})

computer.typeText

Resolve a terminal target and insert text after safety checks. Lattices prefers the least intrusive available transport: tmux pane input when a tmux pane is known, then target-pid key events/pasteboard insertion when window focus is required. Enter is never pressed unless enter: true is provided.

Params:

await daemonCall('computer.typeText', {
  text: '# hello from lattices',
  transport: 'auto',
  enter: false
})

computer.demoTerminal

Compatibility endpoint for the original terminal demo. It follows the same treatment, safety, capture, and transport rules as computer.typeText, but provides a default text payload when text is omitted.

Run a bounded computer-use sequence against a terminal window:

1. synthesize and score terminal candidates
2. choose a safe shell-like terminal unless wid or tty is supplied
3. capture a before screenshot artifact
4. focus the terminal window
5. insert text without pressing Enter by default
6. capture an after screenshot artifact

Params:

await daemonCall('computer.demoTerminal', { dryRun: true })

await daemonCall('computer.demoTerminal', {
  app: 'iTerm2',
  text: '# hello from lattices',
  enter: false
})

Overlay UI

The macOS app exposes a shared desktop overlay canvas for lightweight agent-visible UI. Use overlay.publish for transient passive visuals, and overlay.actor.* for persistent, movable actor surfaces.

Persistent actors are useful for representing agents or processes on the desktop. Each actor has a stable id, can be moved independently through the API, dragged by the user, hidden/restored with Hyper+B, and closed with right-click. Click event callbacks and action surfaces are planned follow-on capabilities.

overlay.publish

Publish a transient layer on the screen overlay canvas.

Params:

Example:

await daemonCall('overlay.publish', {
  kind: 'highlight',
  x: 160,
  y: 120,
  w: 480,
  h: 260,
  text: 'Needs review',
  style: 'warning',
  ttlMs: 3000
})

overlay.actor.publish

Create or update a generative overlay actor. Actors default to persistent: omit ttlMs or pass 0, and dismissible defaults to false.

Params:

Bundled sprite assets:

Example:

await daemonCall('overlay.actor.publish', {
  id: 'agent-scout',
  renderer: 'sprite',
  asset: 'scout-ranger',
  state: 'waiting',
  name: 'Scout',
  message: 'Waiting for feedback',
  placement: 'point',
  x: 640,
  y: 320,
  ttlMs: 0
})

overlay.actor.moveTo

Move an actor with app-owned animation. The app interpolates position and switches directional sprite states while moving.

Params:

Example:

await daemonCall('overlay.actor.moveTo', {
  id: 'agent-scout',
  x: 820,
  y: 280,
  durationMs: 800,
  easing: 'spring'
})

overlay.actor.hud

Attach a transparent, blurred hover HUD to an actor. The HUD is backed by a native WKWebView, so apps can point it at a local static HTML dashboard or, in development, a local URL.

Params:

Example:

await daemonCall('overlay.actor.hud', {
  id: 'switch-talkie',
  hudUrl: '/Users/you/dev/talkie/.lattices/hud/index.html',
  hudReadAccess: '/Users/you/Library/Application Support/Talkie/HUD',
  hudWidth: 380,
  hudHeight: 260
})

overlay.actor.visibility

Show, hide, toggle, or inspect the persistent actor layer without destroying the actors. This is the daemon equivalent of the app's Hyper+B shortcut.

Params:

Example:

await daemonCall('overlay.actor.visibility', { action: 'toggle' })

Static HUD Manifests

Apps and projects can expose a local hover dashboard without running a web server by publishing a static bundle at:

.lattices/hud/
  manifest.json
  index.html
  assets/

Minimal manifest:

{
  "version": 1,
  "id": "talkie",
  "name": "Talkie",
  "bundleId": "com.usabletalkie.Talkie",
  "icon": "./assets/icon.png",
  "entry": "./index.html",
  "readAccess": "~/Library/Application Support/Talkie/HUD",
  "surface": { "width": 380, "height": 260 },
  "actor": {
    "labelHidden": true,
    "click": "activateApp"
  },
  "sources": [
    {
      "path": "~/Library/Application Support/Talkie/HUD/activity.jsonl",
      "format": "jsonl",
      "schema": "talkie.activity.v1",
      "presentation": "timeline"
    }
  ]
}

The CLI resolves this manifest into overlay.actor.publish with a file-backed HUD URL. The macOS app loads entry through WKWebView.loadFileURL, allowing read access to the HUD folder by default, or to the manifest's readAccess folder when one is declared.

sources is descriptive metadata for app-owned state, logs, or event streams. Lattices does not append to those logs. The app writes them in its normal runtime location, and the custom HUD renderer decides how to present them.

Useful commands:

lats hud register .lattices/hud/manifest.json --publish
lats hud publish talkie
lats hud sync
lats hud discover ~/dev --register

For packaged apps, keep the renderer files in the app bundle and point mutable sources at an app-owned folder such as ~/Library/Application Support/....

System

deck.manifest

Return the shared DeckKit manifest exposed by the macOS app. This is the contract a future Lattices companion can consume to discover pages, capabilities, and security mode.

Params: none

deck.snapshot

Return the current DeckKit runtime snapshot for the macOS host.

Params: none

Returns: a DeckRuntimeSnapshot object containing:

• voice
• desktop
• switcher
• history
• questions

deck.perform

Perform a DeckKit action against the running macOS host and return a DeckActionResult.

Params:

Example:

{
  "id": "1",
  "method": "deck.perform",
  "params": {
    "pageID": "layout",
    "actionID": "layout.optimize",
    "payload": {
      "strategy": "balanced",
      "region": "right"
    }
  }
}

solo.status, solo.enter, solo.exit, solo.toggle

Control the same reversible Focus Mode exposed through the Hyper+Z chord and command palette. solo.enter targets the currently frontmost window; solo.exit restores the exact frame captured on entry.

lats call solo.enter
lats call solo.status
lats call solo.exit

Each method returns the resulting active state. Mutating methods also return ok; solo.enter returns ok: false when there is no eligible frontmost window.

stage.status, stage.enter, stage.exit, stage.toggle, stage.cycle

Stage is on or off. The windows on the display you are using decide the grid: one window takes the stage, and more windows fill the grid around it. A window that appears while Stage is on joins the grid. The window you are in grows to 40% of the display width and comes forward. The others stay in their cells, slightly smaller. Clicking another window in the set, or stage.cycle, moves the stage. Escape and stage.exit put every window back where it was when it joined. Hyper+X toggles it. Control-Option-Tab moves through the set while it is on.

lats call stage.enter
lats call stage.cycle '{"by":-1}'
lats call stage.exit

stage.enter turns Stage on whatever the window count is. stage.cycle takes by (default 1).

daemon.status

Health check and basic stats.

Params: none

Returns:

{
  "uptime": 3600.5,
  "clientCount": 2,
  "version": "1.0.0",
  "windowCount": 12,
  "tmuxSessionCount": 3,
  "frontmostWid": 1234,
  "permissions": {
    "accessibility": true,
    "screenRecording": true
  }
}

api.schema

Return the full API schema including version, models, and method definitions. Useful for agent self-discovery.

Params: none

CLI shortcut:

lats call api.schema

diagnostics.list

Return recent diagnostic log entries from the daemon.

Params:

Mouse & Input

Mouse shortcut rules are data. Prefer shortcut.send for hotkeys an agent can define or change directly; do not add a named action unless the behavior cannot be expressed as data.

Create or replace a gesture that sends Hyper+D:

await daemonCall('mouse.shortcuts.upsert', {
  rule: {
    id: 'middle-up-voice',
    enabled: true,
    device: 'any',
    trigger: { button: 'middle', kind: 'drag', direction: 'up' },
    action: {
      type: 'shortcut.send',
      shortcut: {
        key: 'd',
        keyCode: 2,
        modifiers: ['control', 'option', 'shift', 'command']
      }
    }
  }
})

If an agent edits ~/.lattices/mouse-shortcuts.json itself, refresh the running app explicitly:

await daemonCall('mouse.shortcuts.reload')

All write methods persist the config, checkpoint the previous version in ~/.lattices/mouse-shortcuts.history/, and update the active event-tap snapshot immediately. No app restart is required.

Supported action types:

Windows & Spaces

desktop.snapshot

One call for “what is in front of the user?” Structured state — not a drawing. lats map is the ASCII render of this same current-space view (it now reads desktop.snapshot when the app is new enough).

Params:

Returns:

{
  "frontmost": { "wid": 1234, "app": "iTerm2", "title": "…", "pid": 1 },
  "activeLayer": { "id": "web", "label": "Web", "index": 0, "slot": 1, "active": true, "layout": null, "entries": [] },
  "displays": [ { "displayIndex": 0, "name": "Built-in", "frame": {}, "visibleFrame": {}, "currentSpaceId": 1, "spaces": [] } ],
  "currentSpaceId": 1,
  "windows": [
    {
      "wid": 1234,
      "app": "iTerm2",
      "title": "…",
      "pid": 5678,
      "frame": { "x": 0, "y": 25, "w": 960, "h": 1050 },
      "spaceIds": [1],
      "isOnScreen": true,
      "zIndex": 0,
      "focused": true,
      "latticesSession": "lattices-a1b2c3"
    }
  ],
  "sessions": [ { "name": "lattices-a1b2c3", "attached": true, "windowCount": 1 } ],
  "permissions": { "accessibility": true, "screenRecording": true }
}

Window frames are top-left global, same as windows.list. Does not recapture windows or run OCR.

lats call desktop.snapshot
lats map

windows.list

List all visible windows tracked by the desktop model.

Params: none

Returns: array of window objects:

[
  {
    "wid": 1234,
    "app": "Terminal",
    "pid": 5678,
    "title": "[lattices:myapp-a1b2c3] zsh",
    "frame": { "x": 0, "y": 25, "w": 960, "h": 1050 },
    "spaceIds": [1],
    "isOnScreen": true,
    "latticesSession": "myapp-a1b2c3"
  }
]

The latticesSession field is present only on windows that belong to a lattices session (matched via the [lattices:name] title tag).

windows.get

Get a single window by its CGWindowID.

Params:

Returns: a single window object (same shape as windows.list items). Errors: Not found if the window ID doesn't exist.

windows.search

Search windows by text query across title, app name, session tags, and OCR content. Returns results with matchSource indicating how the match was found, and ocrSnippet for OCR matches.

Params:

Returns: array of window objects with additional search fields:

[
  {
    "wid": 265,
    "app": "iTerm2",
    "title": "✳ Claude Code",
    "matchSource": "ocr",
    "ocrSnippet": "…~/dev/vox StatusBarIconFolder…",
    "frame": { "x": 688, "y": 3, "w": 1720, "h": 720 },
    "isOnScreen": true
  }
]

matchSource values: "title", "app", "session", "ocr".

CLI usage:

# Basic search (uses windows.search)
lats search vox

# Deep search — adds terminal tab/process inspection for ranking
lats search vox --deep

# Same as --deep (all search sources)
# A TTY matched only at app level (see terminals.list) adds 1 point to each
# window of its terminal app, with the source "terminal-app". `lats place`
# does not act on a result that has only that source.
lats search vox --all

# Pipeable output
lats search vox --wid
lats search vox --json

# Search + focus + tile in one step
lats place vox right

windows.preview

Return a cached PNG preview for a tracked window. When the cache is cold, load starts an asynchronous warmup and the caller can poll until ok is true.

The response includes cached and loading. A ready preview also includes mimeType: "image/png", base64 image data, and pixel dimensions.

spaces.list

List macOS display spaces (virtual desktops).

Display frame and visibleFrame use the same top-left global coordinate system as window frames returned by windows.list. See Workspace Map for the complete coordinate contract, terminal/JSON projection, and visible-surface lifecycle guidance.

Params: none

Returns: array of display objects:

[
  {
    "displayIndex": 0,
    "displayId": "main",
    "name": "Built-in Liquid Retina XDR Display",
    "frame": { "x": 0, "y": 0, "w": 1728, "h": 1117 },
    "visibleFrame": { "x": 0, "y": 38, "w": 1728, "h": 1079 },
    "currentSpaceId": 1,
    "spaces": [
      { "id": 1, "index": 1, "name": "Desktop 1", "display": 0, "isCurrent": true },
      { "id": 2, "index": 2, "name": "Desktop 2", "display": 0, "isCurrent": false }
    ]
  }
]

windows.place

Canonical window placement mutation. Use this when an agent needs a single, typed placement contract across voice, CLI, and daemon clients.

Params:

Target resolution priority is wid → session → app/title → frontmost window.

Placement strings: left, right, top, bottom, top-left, top-right, bottom-left, bottom-right, left-third, center-third, right-third, top-third, middle-third, bottom-third, left-quarter, right-quarter, top-quarter, bottom-quarter, maximize, center, grid:CxR:C,R, or compact CxR:C,R. The canonical grid: form is 0-indexed; the compact form is 1-indexed for command entry.

Typed placement examples:

{ "kind": "tile", "value": "top-right" }
{ "kind": "grid", "columns": 3, "rows": 2, "column": 2, "row": 0 }
{ "kind": "fractions", "x": 0.5, "y": 0, "w": 0.5, "h": 1 }

Returns: execution receipt including resolved target, placement, and trace.

windows.pick

Open Hyperspace in a read-only selection mode and wait for the user to choose one visible window. Letter hints, Tab/Return, and clicking a survey tile select; Escape cancels. Placement, layer editing, and normal motion commands are disabled for the duration of the pick.

Returns status: "selected" with a window, or cancelled, timeout, or unavailable without one.

windows.focus

Focus a window — bring it to front and switch Spaces if needed.

Params (one of):

Provide either wid or session. If wid is given, it takes priority.

windows.move

Move a specific window to a display and desktop in one verified operation. Without a placement, the window keeps its normalized position and size on the target display. A Space ID identifies one desktop on one display; Desktop 2 on a physical monitor differs from Desktop 2 on a virtual display.

Params:

Provide wid or session, plus at least one of display, spaceId, or placement. There is no frontmost-window fallback.

• spaceId selects its owning display when display is omitted. If both are supplied, the Space must belong to that display. A mismatch is an error.
• display alone selects that display's current desktop. Display indexes resolve through stable display UUIDs.
• placement can accompany either destination. Without a placement, a cross-display move maps the source frame onto the target visible frame and clamps it to fit. A same-display desktop move preserves the frame.
• Unknown displays, Spaces, and placements are errors. A window must belong to one ordinary desktop; full-screen and multi-Space windows are rejected when their relocation cannot be verified.

For an inactive source desktop, Lattices first carries the window to the source display's current desktop. It then moves the window to the target display and carries it to the requested desktop. Mission Control can appear during these steps. Keep the pointer free until the operation finishes.

Returns: an execution receipt with status (ok/planned/blocked/failed), before/target/after frames in mutations, sourceDisplay, display, spaceId, fromSpaceIds, and afterSpaceIds. verified requires verifiedFrame, verifiedDisplay, and verifiedSpace to succeed. An AX call or a desktop view switch alone cannot report success. A placement receipt retains action.type: "window.place" and compatibilityMethod: "window.move".

On partial failure, Lattices attempts to restore the original display, desktop, and frame. The receipt reports failureReason, rollback, and the observed final state. Successful moves support actions.undo, including the original desktop. Allow up to 60 seconds for verification and recovery.

// Move to the current desktop on display 0, preserving relative geometry.
await daemonCall('window.move', { wid: 4182, display: 0 }, 60000)

// Choose the exact physical display and desktop in one operation.
await daemonCall('window.move', { wid: 4182, display: 0, spaceId: 3 }, 60000)

// Preview a move and placement without changing the window or desktop view.
await daemonCall('window.move', {
  wid: 4182, display: 0, spaceId: 3, placement: 'left', dryRun: true
})

spaces.optimize

Canonical space-balancing mutation. Use this when the goal is to make the current workspace coherent rather than placing one specific window.

Params:

If windowIds is provided, scope is inferred as selection. If app is provided and scope is omitted, scope is inferred as app.

Returns: execution receipt including resolved scope, strategy, affected window IDs, and trace.

layout.distribute

Compatibility wrapper for spaces.optimize with scope=visible and strategy=balanced.

Params: none

Sessions

All session methods require tmux to be installed.

tmux.list

List tmux sessions that belong to lattices.

Params:

Returns: array of session objects:

[
  {
    "name": "myapp-a1b2c3",
    "windowCount": 1,
    "attached": true,
    "panes": [
      {
        "id": "%0",
        "windowIndex": 0,
        "windowName": "main",
        "title": "claude",
        "currentCommand": "claude",
        "pid": 9876,
        "isActive": true
      }
    ]
  }
]

With includeOrphans: true, returns every tmux session instead, including orphans (sessions not tracked by lattices):

{
  "all": [ ... ],
  "orphans": [ ... ]
}

Both arrays contain session objects (same shape as tmux.list).

sessions.launch

Launch a new tmux session for a project. If a session already exists, it will be reattached. The project must be in the scanned project list — call projects.list to check, or projects.scan to refresh.

Params:

Returns: a receipt, not a queued ok:

{
  "ok": true,
  "session": "myapp-a1b2c3",
  "alreadyRunning": false,
  "wid": 1234,
  "verified": true
}

ok / verified are false if no tagged window and no tmux session appeared within ~4s. Errors: Not found if the path isn't in the scanned project list.

sessions.kill

Kill a tmux session by name.

Params: