SWE-agent
In plain wordsAn open-source research system from Princeton that turns a language model into a software-fixing agent by giving it a specially designed set of commands for browsing, editing and testing code.
An open-source system that lets a large language modellarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code. resolve real software issues autonomously by working inside a repository through a custom agent–computer interfaceagent–computer interface, n.: The commands, tools and feedback designed for an AI agent to work with a computer — the agent's version of a user interface, shaped around how language models read and act.: commands to open and scroll files a window at a time, search the codebase, make line-range edits checked by a linter that rejects syntax errors, and run code and tests, each returning compact feedback. Run with GPT-4 Turbo, it fixed 12.5% of SWE-benchSWE-bench, prop. n.: A test that asks AI to fix real bugs and add features in real open-source Python projects, checked by running each project's own tests; the best-known yardstick for AI coding agents. issues and 87.7% of HumanEvalFix problems, well beyond what models had achieved without interacting with the code, and ablations credited much of the gain to the interface rather than the model. SWE-agent became a standard research baseline; a later minimal variant, mini-SWE-agent, showed that newer models could perform well with little more than a shell.
We benchmark new models under SWE-agent so the scaffold is held constant.
Origin
Developed at Princeton by John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan and Ofir Press; the code was released in April 2024, the paper posted on 6 May 2024 and published at NeurIPS 2024.
Related
- agent–computer interfaceagent–computer interface, n.: The commands, tools and feedback designed for an AI agent to work with a computer — the agent's version of a user interface, shaped around how language models read and act.
- SWE-benchSWE-bench, prop. n.: A test that asks AI to fix real bugs and add features in real open-source Python projects, checked by running each project's own tests; the best-known yardstick for AI coding agents.
- coding agentcoding agent, n.: An AI agent that writes and fixes software by itself: it reads a codebase, edits files, runs commands and tests, and keeps working until a feature is built or a bug is fixed.
- ReActReAct, prop. n.: A method in which an AI alternates between thinking out loud about what to do next and taking an action, such as a web search, then uses what it observed to plan its next step.
- harnessharness, n.: The software around an AI model that gives it instructions and tools, carries out its actions and decides when it stops. The model is the engine; the harness is the rest of the car.
Sources
- Yang, J., Jimenez, C. E., Wettig, A., Lieret, K., Yao, S., Narasimhan, K. & Press, O. (2024). SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering. arXiv:2405.15793. arxiv.org/abs/2405.15793
- Yang, J. et al. (2024). SWE-agent. Advances in Neural Information Processing Systems 37 (NeurIPS 2024). papers.neurips.cc/paper_files/paper/2024/hash/5a7c947568c1b1328ccc5230172e1e7c-Abstract-Conference.html
Mentioned in
coding agentcoding agent, n.: An AI agent that writes and fixes software by itself: it reads a codebase, edits files, runs commands and tests, and keeps working until a feature is built or a bug is fixed., SWE-benchSWE-bench, prop. n.: A test that asks AI to fix real bugs and add features in real open-source Python projects, checked by running each project's own tests; the best-known yardstick for AI coding agents., agent–computer interfaceagent–computer interface, n.: The commands, tools and feedback designed for an AI agent to work with a computer — the agent's version of a user interface, shaped around how language models read and act.