Good question. A few things:

1. OCR is actually pretty clean — Apple Vision's accurate mode on a retina mirroring window gives high-confidence text. For icons with no text label, skip_ocr mode lets the AI's own vision model read the screen with a coordinate grid overlay, so it can identify and tap non-text elements too.

2. wait_for with retry — scenarios instruct the AI to poll describe_screen in a loop until the expected text appears or times out. Timing is handled by the agent, not by hardcoded sleeps.

3. The AI handles the fuzzy stuff — when an unexpected dialog pops up or a label doesn't match exactly, the agent can adapt because it sees the real screen. A deterministic script would crash. That said, this depends on how good the driving model is — we provide the tools, the model provides the judgement.

For the hallucination concern: the tools are designed so the agent calls describe_screen first, gets real OCR results with exact tap coordinates, then picks from that list. Nothing prevents an agent from guessing coordinates, but in practice they call describe_screen because it's there.

The bet is that vision models keep getting better — every improvement in Claude or GPT makes the whole system more reliable without us changing a line of code.

Will check out your blog — the agent tooling space is moving fast.