Demonstrations of AI agents often appear impressive, yet achieving consistent and error-free performance in real-world applications remains challenging. Current AI models are capable of answering questions and engaging in conversations with near-human proficiency, forming the basis of chatbots like OpenAI’s ChatGPT and Google’s Gemini. These models also execute tasks on computers through simple commands by interacting with screens and input devices such as keyboards and trackpads, or through low-level software interfaces.
Anthropic claims that their AI agent, Claude, surpasses other agents on several critical benchmarks, including SWE-bench, which evaluates software development capabilities, and OSWorld, which assesses an agent’s ability to navigate a computer operating system. However, these claims have not yet been independently verified. According to Anthropic, Claude completes tasks in OSWorld correctly 14.9% of the time, trailing human performance, which averages around 75%, but still significantly ahead of other leading AI agents like OpenAI’s GPT-4, which achieves a success rate of approximately 7.7%.
Several companies have begun testing the agentic version of Claude, as reported by Anthropic. These include Canva, which is utilizing it to automate design and editing tasks, and Replit, which applies the model to enhance coding tasks. Additional early adopters include The Browser Company, Asana, and Notion.
Ofir Press, a postdoctoral researcher at Princeton University who contributed to the development of SWE-bench, points out that agentic AI often lacks the ability to anticipate long-term scenarios and frequently faces challenges in error recovery. He suggests that demonstrating their utility requires achieving strong performance on challenging and realistic benchmarks, such as reliably planning a wide range of trips and booking all necessary tickets.
Kaplan notes that Claude possesses the capability to troubleshoot certain errors effectively. For instance, when encountering a terminal error while attempting to start a web server, the model accurately modified its command to resolve the issue. It also managed to enable popups when encountering obstacles while browsing the web.
Numerous technology companies are currently in a race to develop AI agents in pursuit of market share and dominance. It might not be long before such agents become accessible to many users. Microsoft, which has invested over $13 billion in OpenAI, is testing agents that can operate on Windows computers. Similarly, Amazon, having made significant investments in Anthropic, is exploring the potential of agents in recommending and eventually purchasing goods for its customers.
Sonya Huang, a partner at the venture firm Sequoia with a focus on AI companies, suggests that amid the enthusiasm surrounding AI agents, many companies are merely rebranding AI-powered tools. In an interview with WIRED prior to the Anthropic announcement, she remarked that the technology is currently most effective in narrow domains, such as tasks related to coding. “You need to choose problem spaces where if the model fails, that’s okay,” she observes. “Those are the problem spaces where truly agent native companies will arise.”
A primary challenge with agentic AI is that its errors can be more problematic than merely receiving a garbled response from a chatbot. Anthropic has placed specific restrictions on what Claude can do, such as limiting its ability to use a person’s credit card for purchases.
According to Press from Princeton University, if errors can be sufficiently minimized, users may start to perceive AI—and computers—in a completely new way. “I’m super excited about this new era,” he expresses.
