The problem is not vision - it is targeting
Modern coding agents can see images. What still fails is targeting: which button, which card, which of three identical Save controls. A raw screenshot forces the model to guess. Every wrong guess costs a turn, tokens, and focus.
Visual context for agents means three things: a capture of the real UI, a mark that disambiguates the target, and intent in language the agent can act on. Miss any one and you are back to cartography by prose.
Approaches that work today
Teams use a few recurring patterns. None is universally best; pick by surface and how often you switch agents.
| Approach | Best for | Watch-outs |
|---|---|---|
| Raw screenshot + chat | Quick one-offs | Ambiguous on dense UIs |
| Human screenshot apps (CleanShot, Shottr) | Pretty captures for people | Agent still guesses the target |
| In-app DOM tools (Agentation) | Web apps you control | Not native / arbitrary screens |
| Structured capture apps (SlimSnap) | Element-level intent export | Mac-centric; format-specific |
| Numbered region overlays (Casso) | Any screen, multi-region, any agent | You still paste into the agent |
| Agent browser / skills | Automated page tours | Setup cost; not always your bug |
A reliable loop
Whatever tool you use, a tight loop beats a perfect stack:
- Reproduce the UI state you care about (right breakpoint, right theme, right data).
- Mark the target so a stranger (or model) cannot miss it.
- State intent in one sentence per mark: expected vs actual.
- Hand off image + structure + intent in one paste.
- Verify visually; re-mark only what is still wrong.
Why numbered regions help
Numbers create a shared coordinate system between the image and the text. When the prompt says "increase padding on [2] to match [1]," the model maps tokens to pixels without re-describing layout. Multi-region feedback (layout + copy + icon) stays coherent in a single turn.
Casso is built around that pattern: draw boxes, optional notes, local annotated composite, clipboard prompt with [n] references. It does not call an LLM. Your agent of choice does the reasoning.
Privacy and local capture
UI screenshots can include PII, production data, or unreleased product. Prefer tools that keep captures local until you explicitly paste them into an agent you trust. Casso writes sessions under a local temp path and never uploads the screen.
FAQ
Related
Try Casso free for 7 days
Numbered screen annotations for Claude Code, Codex, and other agents. 7-day free trial, then $29 once - not a subscription.