What can AI agents do? Tasks listed, limits and results

What the vendor pages list as work

OpenAI’s Help Center page on Codex Cloud, read on 6 November 2026, says Codex Cloud runs coding tasks on OpenAI-managed computers. It gives three examples of what to ask for: investigate a bug, make a code change, or run the project’s tests. Each task starts from a prepared setup in its own isolated workspace, and the page says cloud tasks can continue while your computer is asleep. The page’s own next step is to review the changes and test results before using the work. OpenAI’s product page for Codex Cloud puts it in three steps, choosing repositories, starting tasks from a published environment, and then inspecting the changed files and results before you commit or open a pull request.

Anthropic’s overview of Claude Code, which carries no date, lists the tedious jobs it handles: writing tests for untested code, fixing lint errors across a project, resolving merge conflicts, updating dependencies and writing release notes. For a feature or a bug, it says Claude Code plans the approach, writes the code across several files and verifies that it works. It works with git, staging changes, writing commit messages, creating branches and opening pull requests, and its overview says that in CI you can automate code review and issue triage with GitHub Actions or GitLab CI/CD. A second Anthropic page describes the loop as three phases, gather context, take action and verify results, with dozens of actions chained together and corrected along the way. It sorts the built-in tools into five groups: file operations, search, execution, web and code intelligence.

Google’s Jules FAQ, also undated, calls Jules a software coding agent that helps you fix bugs, add documentation, update your app and implement new features. Each task runs in a fresh virtual machine where Jules clones your repository, installs dependencies and makes changes based on your prompt, and the page says you can leave the app after submitting a task. GitHub’s application card describes the Copilot cloud agent as an asynchronous agent on GitHub.com that can create branches, write code and open pull requests in response to assigned issues, in an ephemeral, firewalled environment with automated security scanning. GitHub Docs add that a session can run on GitHub Actions or in a cloud sandbox.

The one entry that is not about code is OpenAI’s Work. The Help Center page on ChatGPT Work and Codex, read on 6 November 2026, describes Work as an agent designed for longer, multi-step work and finished deliverables, such as researching a topic or creating a document, spreadsheet, presentation, report or Site, and says Codex remains dedicated to software development. On the same page, an action that needs approval on web or mobile is approved on screen: “Spoken approval is not supported.”

The word agent is not used the same way across these pages. Anthropic’s engineering post of 19 December 2024 defines agents as systems where language models dynamically direct their own processes and tool use, while OpenAI uses the word for Work, a product for longer tasks that end in a document. Our page does not choose between them; what an agent is has the definitions side by side.

The one repository where results were counted

The vendor pages above list tasks. The pages we read from OpenAI, Anthropic and Google do not give a success rate for them. The numbers come from a Microsoft .NET Blog post of 23 March 2026, which counts every pull request the Copilot coding agent opened in the dotnet/runtime repository from 19 May 2025, when the agent launched, to 22 March 2026. That is 878 pull requests: 535 merged, 253 closed and 90 still open. The post calls merged out of merged plus closed the success rate, and gives it as 67.9%. For human pull requests in the same repository it gives 87.1% for Microsoft contributors and 79.7% for the community.

The post says the comparison is not like for like. Every agent pull request was created at the explicit request of a human with maintainer rights, the agent cannot open them on its own, and the post says agent and human pull requests are different populations facing different selection pressures. It also notes that agent pull requests tend to be more bounded in scope than the average human one. So 67.9% against 87.1% is two figures from one repository, not a verdict on the agent.

The more useful table in the post splits the agent’s pull requests by kind of task. Removal and cleanup had the highest rate, 84.7% across 77 pull requests, then testing at 75.6% across 99. Bug fixes, the largest group at 317, came in at 69.4%, features at 64.5% across 131, and performance work at 54.5% across 46. A category the post calls Other had 26 pull requests and 11.5%. The post reads the pattern this way: mechanical tasks with a well-defined scope succeed, and tasks that need judgment, exploration or domain expertise struggle.

Size tells a less tidy story. Pull requests of 1 to 10 changed lines had an 80.0% rate across 160 decided ones, 11 to 50 lines 76.9%, and 101 to 500 lines 64.0%, while those of more than 1,000 lines rebounded to 71.9%. The post concludes that “task scope matters more than size”. On quality after merging, 3 of the 535 merged agent pull requests were later reverted, 0.6%, against 33 of 4,251 merged pull requests that were not the agent’s, 0.8%. The post says the samples are too small for the difference to be statistically meaningful.

All of this is one repository and one product. The post is about the Copilot coding agent, and the same table does not tell you what OpenAI’s, Anthropic’s or Google’s agents would score on the same work.

Where the vendors say it stops

GitHub Docs list limits in plain numbers. Each Copilot session that runs on GitHub Actions has a maximum execution time of 59 minutes, which the page says cannot be extended or bypassed, and a task that exceeds it stops. In Actions-based and cloud sandbox sessions Copilot can work on one branch at a time and open one pull request per task. When it runs on GitHub Actions it can make changes only in the repository named when the task starts, while sessions in a cloud sandbox can work across several repositories. The page also says Copilot works only with repositories hosted on GitHub. In the same section GitHub warns that Copilot can produce incorrect or suboptimal code, including code with security vulnerabilities, and says to review and test the output before using it in production.

Google’s Jules FAQ says long-running processes such as dev servers or watch scripts are not currently supported in setup scripts, and asks for discrete install and test commands instead. If a task fails, Jules retries automatically and then marks it failed; the FAQ names broken setup scripts and vague prompts as common causes. The code runs in a virtual machine with internet access, and the page states that you are responsible for the code you run. It lists JavaScript and TypeScript, Python, Go, Java and Rust as the languages it works best with. The FAQ still says Jules is in Public Beta, while Google’s announcement of 6 August 2025 says Jules is out of beta and available for everyone, powered by Gemini 2.5. The FAQ shows no date and the announcement is dated 6 August 2025, so we report both.

Anthropic’s page on how Claude Code works draws a line at remote systems. File edits are snapshotted before the change, and you can rewind them, but the checkpoints cover file changes only. Actions that affect remote systems, such as databases, APIs and deployments, cannot be checkpointed, and you control those with the permission mode and permission rules. Pressing Esc stops Claude immediately. OpenAI’s Codex Cloud page adds a smaller limit of the same kind: saved virtual machine state is recoverable for up to 7 days, and a new task does not recover another task’s uncommitted changes, so the page says to commit important work to source control.

Anthropic’s older post, dated 19 December 2024 and marked by Anthropic as describing a tooling landscape that has changed since, gives the general reasoning. It says agents suit open-ended problems where it is hard or impossible to predict the number of steps, and that their autonomy means higher costs and the potential for compounding errors, so it recommends extensive testing in sandboxed environments with guardrails. For code in particular, it says automated tests make solutions verifiable and let the agent iterate on the results, but that human review remains crucial for checking that solutions fit the wider system.

What went wrong inside finished tasks

A merged pull request is not the same as a correct one, and the Microsoft post is specific about why. It says the agent can produce tests that validate existing incorrect behavior, which encodes a bug as the expected result, and calls that actively harmful because such tests create false confidence. The post adds that 65.7% of the lines added in the 535 merged agent pull requests were test code, against 49.9% in a sample of 500 merged human ones; the post notes the high test share is not unique to the agent and says this risk demands careful review attention. On performance, it says the agent makes claims such as a 2x improvement without validation. The team does not trust those claims at face value, and its policy is that no performance pull request merges without empirical evidence, whoever sent it.

Cleanup work has its own failure. In one pull request that needed every occurrence of two build symbols found, the post says the agent constrained its searches to only a portion of the repository, and the post’s author had to ask explicitly for a full search, after which the agent found and fixed the rest. Platform is another. The post says the agent’s environment runs on Linux, so Windows-only code cannot be compiled by the agent. One pull request on a Windows cryptography call took more than 20 commits and was eventually closed; the post says the agent struggles with problems that need architectural judgment. The author sums up an early impression, which the post says held up over the following months: excellent at implementing well-specified changes, very good at investigating issues, and relatively poor at architecting solutions in large codebases that need broad understanding.

The post also reports what changed the numbers. Pull requests created before the team’s first setup changes, which included firewall rules for package feeds and an instruction file explaining how to build the repository, had a 38.1% success rate, and those created after had 69%. The post draws the conclusion that preparation mattered more than the model, and that what you assign matters too: from 85% for removal and cleanup to 55% for performance work, a spread it attributes to task selection.

What METR measured, and how old it is

Microsoft measured whether pull requests merged. METR measured time. Its post of 10 July 2025 reports a randomized trial with 16 experienced open-source developers working on 246 real issues in repositories they knew. Each issue was assigned at random to allow or disallow AI tools, and the headline is that with the tools allowed, the issues took 19% longer. The developers had forecast a 24% speed-up, and afterwards still believed the tools had sped them up by 20%. The tools were those of early 2025, and the page now carries the line “These results are out of date.” with a pointer to the update.

The update of 24 February 2026 describes a new experiment begun in August 2025, with data from 57 developers. METR says it gives an unreliable signal of the current effect. The main reason it gives is that a significant number of developers declined to take part because they did not want to work without AI, which it says likely biases the estimate downwards. In a survey, 30% to 50% of developers said they were choosing not to submit some tasks for that reason. METR also names a lower pay rate as a further selection effect.

It still reports its raw results. For the developers who were also in the first study it writes a speedup of -18%, with a confidence interval from -38% to +9%, and for newly recruited developers -4%, with an interval from -15% to +9%. METR says its raw results show some evidence for speedup, believes developers are likely more sped up in early 2026 than a year earlier, and says its data are only very weak evidence for the size of that increase. We report the figures as METR writes them, with no reading of our own: the text calls -18% a speedup and does not spell out the sign convention, so the two METR studies (19% longer in 2025, a negative speedup in 2026) should not be put on one line as if they were the same kind of number.

How to read this page

Three kinds of source sit side by side here, and they do not measure the same thing. The vendor pages say what an agent is built to do and where it stops, and none of the pages we read from OpenAI, Anthropic or Google puts a share of successful tasks on it. Microsoft counts merged pull requests in a single repository for a single product. METR times developers on their own projects. A merge rate and a change in minutes cannot be put on the same scale, and the pages we read do not put two vendors’ agents on one test.

Dates matter more here than on most subjects. The Microsoft figures end on 22 March 2026. The METR trial used tools from early 2025 and is marked out of date by METR itself. The Anthropic post on building agents is from December 2024. The vendor pages, read on 6 November 2026, show relative or no update dates, and the OpenAI Help Center pages said they were updated recently when we read them. The pages we read do not say which tasks are safe to leave without review, and this page does not either. For the cost side, what Claude Code costs sets out prices and token rates from Anthropic.