AX vs screenshots: what UI automation can see

Silent educational animation. All diagrams are illustrative; no live desktop was inspected.

Opening: Lattices / Field Guide. PIXELS + STRUCTURE. Two ways to understand a screen. A screenshot shows appearance. AX describes exposed controls.

[0–9 seconds] A screenshot is an image: shapes, colors, and visible text. A tool must interpret the pixels to locate a button.

[9–18 seconds] AX is structured information supplied by the app. A button may identify its role, title, and supported Press action.

[18–27 seconds] Each view has limits. Pixels do not directly encode a button action. AX can omit a custom control or provide incomplete information.

[27–36 seconds] Use the evidence the app provides, then verify what changed. Accessibility access and Screen Recording are separate macOS permissions.

On-screen diagram labels: 01 / Image; 02 / Structure; Screenshot; Accessibility · AX; What it looks like: Visible text, shapes and colors; What the app says: Roles, labels and supported actions; Role: Button; Title: Export; Action: Press; A picture does not contain a Press action; Some controls may be missing or incomplete; Screen Recording: Permission to capture screen content; Seeing the screen does not grant AX control; Accessibility: Permission for access and control through AX; AX access does not grant screen capture.

Footer: Lattices / Field Guide. Illustrative diagram · not a live desktop inspection. lattices.dev/learn.