Anthropic: Claude AI can control the computer

The AI ​​company Anthropic has a new tool presented which can independently take over tasks on the user’s computer. It simulates a user, accepts mouse input and can completely take over various everyday tasks.

Automation of routine tasks

The feature, which was presented as part of the further developments of Anthropic’s Claude and Haiku models, is simply called “Computer Use” and is currently only available via the API of the middle model Claude 3.5 Sonnet. “Computer Use” allows users to give the AI ​​system multi-step instructions, which Anthropic says can consist of dozens or even hundreds of steps.

The tool can perform tasks by viewing the screen, moving the mouse pointer, clicking buttons and entering text. To make this possible, the Claude AI analyzes screenshots of the screen, counts the pixels and determines where, for example, the mouse arrow needs to be moved. Claude AI performs tasks like a real computer user

However, there are also limitations. The tool does not work with a live video stream, but with quick successive screenshots. This can cause it to miss short-lived notifications or other screen changes. Even simple actions such as drag-and-drop are not yet possible with the tool, it said. In addition, it is sometimes “complicated and prone to errors,” admits Anthropic. In one example from the test phase, it was reported that the system left a programming task unfinished for unknown reasons and instead began browsing images of Yellowstone National Park – which, however, can also be very reminiscent of a human user.

Beta is running

The tool is currently in public beta, having previously been tested in collaboration with companies such as Amazon, Canva, Asana and Notion. During development, the focus is on automating common tasks and simplifying time-consuming work. In a demo, Anthropic shows how the AI ​​is given the task of supplementing a data collection with information about a specific company.

To do this, the AI ​​switches to the browser, uses the search function and gets the required data from a website. The tool then transfers this to the appropriate form. Anthropic has released “Computer Use” for public testing to find possible problems and improve the tool together with external users.