Computer-use agents are AI systems that can perceive graphical user interfaces — typically via screenshots — and execute actions such as mouse clicks, keyboard input, and scrolling to accomplish tasks on a computer. They close the loop between language understanding and physical computer interaction, acting as autonomous operators of software that was designed for humans and lacks a programmatic API.

Think of it like an intern who has never been given access to your internal systems but can do work by watching your screen and using your keyboard, following the same visual interface you do.

A computer-use agent assigned to extract data from a legacy web portal with no API would load the page, navigate menus, fill search fields, read results from the rendered HTML, and export findings — all by issuing click and keystroke commands based on screenshots. Anthropic's Computer Use feature, released in 2024, and OpenAI's similar capabilities provide the model layer for this. Orchestration frameworks wrap the model in a loop that captures screenshots, sends them as Multimodal inputs, parses action outputs, and executes them.

Computer-use agents depend on strong visual understanding, which in Transformer terms means processing image patches through Attention layers alongside text tokens. They are a specific realisation of the Operator Pattern, where the LLM acts as an autonomous intermediary fulfilling a user's intent. MCP Servers offer a complementary approach when structured APIs are available, making computer use the fallback for unstructured surfaces. Prompt Engineering for computer-use agents emphasises grounding (anchoring actions to visible UI elements) and error recovery strategies. Hallucination of non-existent UI elements is a key failure mode that agent designers must guard against.

You will encounter computer-use agents in robotic process automation, accessibility tooling, software testing, and enterprise workflows that involve legacy systems.