AX and screenshots: two ways to see a Mac

See the difference between interface information and pixels, and why a successful automation may use both.

36-second silent visual explanation. The interfaces and values are illustrative. Download video · Captions (VTT) · Transcript (text)

Read the visual transcript

Opening: Lattices / Field Guide. PIXELS + STRUCTURE. Two ways to understand a screen. A screenshot shows appearance. AX describes exposed controls.
0–9 seconds
A screenshot is an image: shapes, colors, and visible text. A tool must interpret the pixels to locate a button.
9–18 seconds
AX is structured information supplied by the app. A button may identify its role, title, and supported Press action.
18–27 seconds
Each view has limits. Pixels do not directly encode a button action. AX can omit a custom control or provide incomplete information.
27–36 seconds
Use the evidence the app provides, then verify what changed. Accessibility access and Screen Recording are separate macOS permissions.
On-screen diagram labels: 01 / Image; 02 / Structure; Screenshot; Accessibility · AX; What it looks like: Visible text, shapes and colors; What the app says: Roles, labels and supported actions; Role: Button; Title: Export; Action: Press; A picture does not contain a Press action; Some controls may be missing or incomplete; Screen Recording: Permission to capture screen content; Seeing the screen does not grant AX control; Accessibility: Permission for access and control through AX; AX access does not grant screen capture.
Footer: Lattices / Field Guide. Illustrative diagram · not a live desktop inspection. lattices.dev/learn.

A screenshot shows appearance. An accessibility tree supplies the interface information the app exposes. Each answers a different question.

Documentation reviewed 2026-10-02. Examples are illustrative; this is not a hands-on product benchmark.