agent-desktop: let AI agents operate desktop apps through the accessibility tree, not screenshots

The usual way to let an AI agent "use a computer" is to take a screenshot and ask the model to guess where the button is. Vercel Labs' open-source agent-desktop takes a different route: it reads the accessibility tree the operating system already exposes, so the agent sees the screen as structured data and then issues commands.

This article is based on the project's README. We have not fully tested it in our own environment, so check the official docs for installation and permission details.

How it works

The project describes itself as computer use for AI agents: read and operate desktop apps through the OS accessibility tree. The loop is:

1. Snapshot: get the current app's UI structure as JSON. Every actionable element gets a ref.
2. Act: the agent uses a ref to click, type and so on.
3. Snapshot again to verify: read the screen once more and check the result.

There is no pixel matching, so resolution, theme and scaling matter much less.

Highlights

• About 56 commands covering observation, click and type, keyboard and mouse, app and window management, notifications, clipboard and screenshots.
• macOS, Windows and Linux (Linux uses AT-SPI).
• Headless by default: it does not move the cursor or steal focus. Add --headed when you want that.
• A C library (FFI) for calling it from code; the project page has a Python ctypes example.
• Built-in documentation written for AI agents to read.
• Apache-2.0 license.

Install and permissions

npm install -g agent-desktop
agent-desktop version

Permissions differ by platform. macOS needs the Accessibility permission in System Settings. On Windows, apps at the same integrity level as the terminal need no extra grant. Linux needs a graphical session and at-spi2-core.

The quick example is to snapshot an app to get a ref, click it, then snapshot again to confirm.

Why agent developers should care

• Structured data is cheaper and steadier than screenshots: a JSON UI tree is usually shorter than an image and does not depend on the model's vision.
• Every step can be verified: the act-then-read loop lets an agent notice when an action did not take effect.
• Cross-platform: the same commands run on all three desktop systems.

There are limits too: it only works with apps that expose proper accessibility information. Content that is fully custom-drawn, with no accessibility nodes, needs screenshots or other methods alongside it.

The model side: pointing it at BazaarLink

agent-desktop handles seeing and doing; reasoning comes from the model behind your agent. The README does not say which model APIs the core CLI calls, so model choice depends on the agent or framework you use.

These agents usually run a loop: read the screen, ask the model for the next step, act, read again. One task means many model calls, so cost and latency add up. Being able to switch models freely, with one key and one bill, is handy.

If your agent or code supports an OpenAI-compatible API, changing three settings sends it through BazaarLink:

from openai import OpenAI

client = OpenAI(
    base_url="https://api.bazaarlink.ai/v1",
    api_key="sk-bl-YOUR_API_KEY",
)

resp = client.chat.completions.create(
    model="MODEL_ID_FROM_BAZAARLINK_MODEL_PAGE",
    messages=[{"role": "user", "content": "Given this UI tree, which ref should I click next?"}],
)
print(resp.choices[0].message.content)

• Set base_url to https://api.bazaarlink.ai/v1 (https://bazaarlink.ai/api/v1 also works).
• Set api_key to a key you created in BazaarLink.
• Set model to an ID from the models page.

One caution: agents typically rely on tool calling and long conversations. Before switching models, run your own agent workflow end to end to confirm the model supports what you need.

Summary

agent-desktop turns desktop control into a loop of reading structure, issuing commands and verifying, which suits developers who want agents to operate native apps. On the model side, any agent that supports an OpenAI-compatible API can call different models through BazaarLink with one key and one bill.

TWD billing · Taiwan invoices · leading AI models · OpenAI-compatible API