Computer Use

Computer use

computer reads an app's accessibility tree and acts on the numbered elements in it. The tree is the interface; a screenshot is confirmation. This design, and most of the rules below, come from Orca (MIT).

Before anything else

Use this only when there is no better route. A file is read with read, a computation is run in the shell, a web page is driven by the browser connector. Reach for computer when the capability lives in a desktop app and nowhere else.

The user sees a permission prompt on the first call of a session. Say what you intend to do before you make it.

Never push, submit a form, send a message, buy, delete, or change an account setting unless the user asked for that exact thing. If an app holds content the user did not ask about, read only what the task needs.

The loop

Copy code
computer(action: "list_apps")
computer(action: "get_state", app: "com.spotify.client")
computer(action: "click", app: "com.spotify.client", elementIndex: 42)
computer(action: "get_state", app: "com.spotify.client")

Observe, act once, observe again. Every action already returns the new state, so the reply to an action is also the next thing you read.

Element indexes go stale

The numbers in the tree are valid for that snapshot of that window. Navigation, scrolling, a focus change, a dialog, an app re-render, or simply a few seconds passing can invalidate them.

• Re-read the tree after anything that changes the UI.
• Indexes are sparse — noisy sections are omitted. Never infer a valid index from the element count, and never guess one that is not printed.
• One window at a time: once you pick windowId (or windowIndex), pass the same one to every later call until you mean to change window.

Picking the app

Prefer the bundle id from list_apps; a name is fine when unambiguous; pid:<number> only when neither is. app names a desktop application, never a website — Gmail is com.google.Chrome, not "Gmail".

When an app has several windows, run list_windows first. Use windowId when the listed id is not null, windowIndex otherwise.

Which action

• Editable field → set_value. It writes the value and can usually read it back, which is the only way to prove the text landed.
• Button, link, checkbox, menu item → click.
• An action the tree explicitly lists on an element → perform_action with that exact name.
• type_text only after a field is focused and the tree confirms a focused text receiver. Synthetic keystrokes go into the void; they are never verified.
• press_key for one key (Return, Escape, Tab, arrows). hotkey for one modifier chord plus one key (CmdOrCtrl+A). Prefer CmdOrCtrl+… so the same call works on every platform.
• Modifier-click is click with modifiers. Never send separate modifier-down/modifier-up calls: an interruption between them leaves a modifier logically held down on the user's machine.
• paste_text for a long or exact body, into a field you have confirmed has focus.

Verified vs. unverified

Every action reports whether its effect could be read back:

An unverified action is not a completed action. Read the state back before telling the user it worked, and if the action could have sent, submitted, bought or deleted something, say plainly that the effect is unproven.

Screenshots

Off by default: the tree is what you act on, and an image you did not need is pure cost. Pass screenshot: true when pixels are the answer — confirming something visually, reading a chart or canvas the tree cannot describe, or picking coordinates.

The reply carries the image and its path. If it reports a scale other than 1, convert before using pixel coordinates:

Copy code
x = screenshot_pixel_x / scale
y = screenshot_pixel_y / scale

Coordinates are window-local, and only valid for the window that screenshot came from. Prefer an element index whenever one exists.

On Linux and Windows a screenshot may come from the visible desktop region, so another window can cover the target. Pass restoreWindow: true when the pixels matter; when you cannot take focus, trust the tree over the pixels.

App notes

Browsers. Set the address field directly with set_value, then press_key: "Return" — do not assume typed text reached the address bar. Use restoreWindow: true when the browser is not frontmost. A large tab strip is deliberately compacted to the active tab plus an "inactive browser tabs omitted" marker; work on the current page unless the user asked about tabs.

Forms inside a browser page (a compose window, a web app) expose accessibility actions without necessarily moving DOM focus. After each field action, check which element actually has focus; if it did not move, Tab from a known field or fall back to window-local coordinates from a fresh screenshot.

Instrument and analysis software often renders its plots as one opaque element. Read the controls from the tree, and take a screenshot for the plot.

Shallow trees. Some apps draw their own interface and expose almost nothing. Two different things look the same from here:

• The state report says the tree is being built — that is an app whose accessibility is switched on the moment you read it. Read again in a few seconds; the real tree arrives. Do not go to coordinates.
• The tree really is empty, and stays empty on a second read. Then take a screenshot, work from coordinates, and say in your answer that you are working from pixels, because that path cannot be verified.

Errors

Each is a named recovery, not a retry.

• app_not_found — run list_apps, use the bundle id. For a web app, target the browser app, not the site name.
• app_blocked — stop. That app is deliberately out of reach.
• window_not_found / window_stale — run list_windows, take a current selector, read state again.
• window_not_focused — retry once with restoreWindow: true. If it says restore was already requested, stop retrying and ask the user to bring the app forward. For editable fields prefer set_value, which often works unfocused. On a coordinate click the message names what was in front instead: another app's window over that point, or nothing frontmost at all. Move that window out of the way rather than retrying the same click.
• element_not_found — the index is stale. Read state again.
• element_not_clickable — no actionable frame. Use a parent or child element that has one, or coordinates from a fresh screenshot.
• action_not_supported — read the element's listed actions and use one of those, or fall back to click / set_value.
• value_not_settable — the element refuses direct writes. Focus it and use keyboard input, and inspect the result.
• unsupported_capability — this platform's provider cannot do it. Use a different verb, or tell the user. On Linux this can also mean a missing desktop dependency, which the message names.
• invalid_argument — fix the arguments. Do not send the same call again.
• action_timeout — read the state before retrying, then use a simpler action.
• screenshot_failed — the tree may still be enough. If the message names Screen Recording, the user grants it in Settings → Computer Use. Note it is granted to Open Science itself, not to the helper that Accessibility is granted to — macOS attributes screen capture to the app that started the request.
• permission_denied / accessibility_error — on macOS, Accessibility has not been granted to Open Science Computer Use. Tell the user to open Settings → Computer Use and grant it; you cannot grant it for them.

An empty tree with no screenshot usually means the app has no visible window, is minimized, or the permission is missing.