How a coding agent should see and drive the Simulator

Once a coding agent can build your app, the next thing it wants is to check its work: launch the app, look at the screen, tap something, look again. How fast and how reliably it can do that decides whether an agent can really test a change, or just guess from screenshots.

We wanted to know which way of doing it works best, so we benchmarked them: Apple’s simctl, the open-source tools AXe and idb, the Simulator’s own lower layers, and a small helper running inside the app. To judge them fairly, we wrote a test app that reports every touch, every state change and every frame it draws, so each tool’s “done” could be checked against what the app actually did. Here’s what we learned.

1. The slow part is waiting, not tapping

Sending a tap is quick with almost any tool. Knowing when the app has finished reacting is the hard part, and most setups don’t know, so they wait a fixed time after every action, often a full second. That sleep ends up being most of every step, and it’s wrong both ways: too long for a button that reacts at once, and too short for a screen that loads slowly or animates in.

The fix is to wait for the screen to settle: no input, no animation and no change on screen for a short window, then take the screenshot. In our tests, the only approach with no early screenshots combined two signals: a “nothing is happening” signal from inside the app, and a check from outside that the pixels had stopped changing. Each one alone was fooled sometimes. Watching pixels alone, for example, settled too early on alerts.

If you can’t add anything to the app, at least replace fixed sleeps with “wait until two screenshots in a row match”, and give slow screens a longer limit than fast ones.

2. Read the screen as text first, pixels second

An app’s accessibility tree lists every element on screen with its label, role and position. For an agent it’s far cheaper than a screenshot: a few hundred words instead of an image, and it says exactly what each button is called and where to tap it.

• Ask for the tree first, and a screenshot only when the tree isn’t enough: custom drawing, charts, images, or checking that something looks right.
• Send small screenshots. A scaled-down JPEG is plenty for a model to read a screen. Keep full-resolution images for when someone needs to see detail.
• Label your icon-only buttons. A button with only an icon shows up in the tree with no name, and the agent has to guess from pixels. An accessibility label fixes it for agents and for VoiceOver users at the same time.

3. Watch the device-wide accessibility switch

Tools that read the tree from outside the app depend on accessibility being turned on for the whole simulated device, and some tools turn it off when they exit. So one tool can quietly break the next one’s tree reads, and the failures look random. In our reliability runs, most failed tree reads from host-side tools came down to this.

Use one tool per simulated device, and check that accessibility is on at the start of a session rather than assuming it.

4. Keep one process running

Command-line tools that start a new process, and a new connection to the Simulator, for every tap pay that setup cost on every single action. Tools that keep a session open, or run many steps in one call, avoid it. AXe has a batch mode that runs many steps in one process, and idb’s companion stays running between commands. For an agent that takes hundreds of steps a session, this matters more than anything else on this list except the waiting.

5. Type by inserting text, not by pressing keys

Typing by sending one key press per character is slow, and fast synthetic key presses can drop characters in text fields. Inserting the text directly into the focused field is quicker and reliable. Keep real key presses for tests that are about the keyboard itself, such as autocorrect, the return key or keyboard shortcuts.

6. Speed up animations, but keep a real-speed pass

Animations are a big part of the wait after each tap. Two common shortcuts, and what they cost:

• Speeding animations up (for example, raising the speed of the app window’s layer, a long-standing trick in UI tests) keeps every animation and its completion code running, just faster. In our tests the final screen always matched the real-speed one. It can hide bugs that only happen when someone taps during an animation.
• Turning animations off is faster still, but code that waits for an animation to finish may never run, and some transitions take a different path through the app.

Turning on Reduce Motion in the Simulator also helps: the app swaps some long animations for short ones, although then you’re testing its Reduce Motion paths, not the default ones.

We’d run two profiles: an everyday one with sped-up animations for the agent’s inner loop, and a slower fidelity one at real speed, with fresh launches, before a change is merged.

7. Don’t make taps too short

Shaving every millisecond off a tap backfires. A press that’s too short can be ignored by some controls; switches were the usual victims in our tests. Keep a short, realistic hold on each tap. It costs almost nothing and removes a whole class of “the agent tapped and nothing happened” failures.

8. Reset to a known screen without relaunching

Tests need a known starting point. Booting a simulated device is the slowest thing you can do, so boot once and keep it running. Between unrelated tests, relaunch the app:

xcrun simctl launch --terminate-running-process "$SIMULATOR_ID" com.example.MyApp

Within one session, a deep link to a known screen is usually faster than a relaunch, and keeps state such as a signed-in account:

xcrun simctl openurl "$SIMULATOR_ID" "myapp://home"

What we’d recommend

• Keep the simulated device booted, and use one tool per device.
• Wait for the screen to settle instead of sleeping a fixed time.
• Read the accessibility tree first; send small screenshots only when needed.
• Type by inserting text; keep a realistic hold on taps.
• Use a fast everyday profile for the agent’s loop and a real-speed profile before merging.
• If you build your own driver, test it against an app that reports what really happened, over a long run of random steps. Ours found bugs that no single test did.

About the numbers

We measured every step of this, in milliseconds. We’ll publish the full results once we’ve run them again on a released version of Xcode. Join the list to get them when they’re out.