init

.gitignore

@@ -0,0 +1,189 @@
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+pip-wheel-metadata/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+*.py,cover
+.hypothesis/
+.pytest_cache/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+db.sqlite3
+db.sqlite3-journal
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# IPython
+profile_default/
+ipython_config.py
+
+# pyenv
+.python-version
+
+# pipenv
+#   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
+#   However, in case of collaboration, if having platform-specific dependencies or dependencies
+#   having no cross-platform support, pipenv may install dependencies that don't work, or not
+#   install all needed dependencies.
+#Pipfile.lock
+
+# PEP 582; used by e.g. github.com/David-OConnor/pyflow
+__pypackages__/
+
+# Celery stuff
+celerybeat-schedule
+celerybeat.pid
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+.env
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
+.dmypy.json
+dmypy.json
+
+# Pyre type checker
+.pyre/
+
+# Mac files
+*.DS_Store
+
+# Custom
+keys.cfg
+
+# iPython Notebooks
+*.ipynb
+
+# Evaluation folders
+results/
+testbed/
+temp/
+
+# Ignore all YAML files in data/
+data/*/ic-*
+data/*/single-issues
+
+# Fine tuning data
+fine_tune/*.ipynb
+fine_tune/subtasks/*.jsonl
+temp*.jsonl
+
+# Inspector
+inspector/*.json
+
+# Ignore all files in the private folder
+private/
+
+### Website
+
+# dependencies
+website/frontend/node_modules
+website/frontend/package-lock.json
+website/frontend/.pnp
+*.pnp.js
+
+# testing
+website/frontend/coverage
+
+# production
+website/frontend/build
+
+# misc
+*.env.local
+*.env.development.local
+*.env.test.local
+*.env.production.local
+.api_key
+*npm-debug.log*
+*yarn-debug.log*
+*yarn-error.log*
+
+
+# demo yamls (for editing)
+*.demo.yaml
+
+# trajectory files
+trajectories/*

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2024 John Yang, Carlos E. Jimenez, Alexander Wettig, Shunyu Yao, Karthik Narasimhan, Ofir Press
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -0,0 +1,98 @@
+<p align="center">
+  <a href="https://www.swe-agent.com/">
+    <img src="assets/swe-agent-banner.png" alt="swe-agent.com" />
+  </a>
+</p>
+
+
+<p align="center">
+  <a href="https://www.swe-agent.com"><strong>Website & Demo</strong></a>&nbsp; | &nbsp;
+  <a href="https://discord.gg/AVEFbBn2rH"><strong>Discord</strong></a>&nbsp; | &nbsp;
+  <strong>Paper [coming April 10th]</strong>
+</p>
+
+
+## 👋 Overview <a name="overview"></a>
+SWE-agent turns LMs (e.g. GPT-4) into software engineering agents that can fix bugs and issues in real GitHub repositories.
+
+On the full [SWE-bench](https://github.com/princeton-nlp/SWE-bench) test set, SWE-agent resolves **12.29%** of issues, achieving the state-of-the-art performance on the full test set.
+
+### ✨ Agent-Computer Interface (ACI) <a name="aci"></a>
+We accomplish these results by designing simple LM-centric commands and feedback formats to make it easier for the LM to browse the repository, view, edit and execute code files. We call this an **Agent-Computer Interface** (ACI) and build the SWE-agent repository to make it easy to iterate on ACI design for repository-level coding agents.
+
+Just like how typical language models requires good prompt engineering, good ACI design leads to much better results when using agents. As we show in our paper, a baseline agent without a well-tuned ACI does much worse than SWE-agent.
+
+SWE-agent contains features that we discovered to be immensly helpful during the agent-computer interface design process:
+1. We add a linter that runs when an edit command is issued, and do not let the edit command go through if the code isn't syntactically correct.
+2. We supply the agent with a special-built file viewer, instead of having it just ```cat``` files. We found that this file viewer works best when displaying just 100 lines in each turn. The file editor that we built has commands for scrolling up and down and for performing a search within the file.
+3. We supply the agent with a special-built full-directory string searching command. We found that it was important for this tool to succintly list the matches- we simply list each file that had at least one match. Showing the model more context about each match proved to be too confusing for the model. 
+4. When commands have an empty output we return a message saying "Your command ran successfully and did not produce any output."
+
+Read our paper for more details.
+
+```
+@misc{yang2024sweagent,
+      title={SWE-agent: Agent Computer Interfaces Enable Software Engineering Language Models}, 
+      author={John Yang and Carlos E. Jimenez and Alexander Wettig and Shunyu Yao and Karthik Narasimhan and Ofir Press},
+      year={2024},
+}
+```
+
+## 🚀 Setup <a name="setup"></a>
+1. [Install Docker](https://docs.docker.com/engine/install/), then start Docker locally.
+2. [Install Miniconda](https://docs.anaconda.com/free/miniconda/miniconda-install/), then create the `swe-agent` environment with `conda env create -f environment.yml`
+3. Activate using `conda activate swe-agent`.
+4. Run `./setup.sh` to create the `swe-agent` docker image.
+5. Create a `keys.cfg` file at the root of this repository and fill in the following:
+```
+OPENAI_API_KEY: 'OpenAI API Key Here if using OpenAI Model (optional)'
+ANTHROPIC_API_KEY: 'Anthropic API Key Here if using Anthropic Model (optional)'
+GITHUB_TOKEN: 'GitHub Token Here (required)'
+```
+See the following links for tutorials on obtaining [Anthropic](https://docs.anthropic.com/claude/reference/getting-started-with-the-api), [OpenAI](https://platform.openai.com/docs/quickstart/step-2-set-up-your-api-key), and [Github]() tokens.
+
+## 💽 Usage <a name="usage"></a>
+There are two steps to the SWE-agent pipeline. First SWE-agent takes an input GitHub issue and returns a pull request that attempts to fix it. We call that step *inference*. The second step (currently, only available for issues in the SWE-bench benchmark) is to *evaluate* the pull request to verify that it has indeed fixed the issue. 
+
+### 👩‍💻 Inference <a name="inference"></a>
+**Inference on *any* GitHub Issue**: Using this script, you can run SWE-agent on any GitHub issue!
+```
+python run.py --model_name gpt4 \
+  --data_path https://github.com/pvlib/pvlib-python/issues/1603 --config_file config/default_from_url.yaml
+```
+
+**Inference on SWE-bench**: Run SWE-agent on [SWE-bench Lite](https://www.swebench.com/lite.html) and generate patches.
+```
+python run.py --model_name gpt4 \
+  --per_instance_cost_limit 2.00 \
+  --config_file ./config/default.yaml
+```
+
+If you'd like to run on a *single* issue from SWE-bench, use the `--instance_filter` option as follows:
+```
+python run.py --model_name gpt4 \
+  --instance_filter marshmallow-code__marshmallow-1359
+```
+* See the [`scripts/`](scripts/) folder for other useful scripts and details.
+* See the [`config/`](config/) folder for details about how you can define your own configuration!
+* See the [`swe-agent/agent/`](agent/) folder for details about the logic behind configuration based workflows.
+* See the [`swe-agent/environment/`](swe-agent/environment/) folder for details about the `SWEEnv` environment (interface + implementation).
+* See the [`trajectories/`](trajectories) folder for details about the output of `run.py`.
+
+### 🧪 Evaluation <a name="evaluation"></a>
+This step is only available for issues from the SWE-bench set. To evaluate generated pull requests:
+```
+cd evaluation/
+./run_eval.sh <predictions_path>
+```
+Replace `<predictions_path>` with the path to the model's predictions, which should be generated from the *Inference* step. The `<predictions_path>` arguments should look like `../trajectories/<username>/<model>-<dataset>-<hyperparams>/all_preds.jsonl`
+* See the [`evaluation/`](evaluation/) folder for details about how evaluation works.
+
+
+## 💫 Contributions <a name="contributions"></a>
+- If you'd like to ask questions, learn about upcoming features, and participate in future development, join our [Discord community](https://discord.gg/AVEFbBn2rH)!
+- If you'd like to contribute to the codebase, we welcome [issues](https://github.com/princeton-nlp/SWE-agent/issues) and [pull requests](https://github.com/princeton-nlp/SWE-agent/pulls)!
+- If you'd like to see a post or tutorial about some topic, please let us know via an [issue](https://github.com/princeton-nlp/SWE-agent/issues).
+
+## 🪪 License <a name="license"></a>
+MIT. Check `LICENSE`.

build_deploy.sh

@@ -0,0 +1,6 @@
+# !bin/bash
+
+python3 -m build
+
+python3 -m twine upload --skip-existing --repository pypi dist/*
+# python3 -m twine upload --skip-existing --repository testpypi dist/*

config/README.md

@@ -0,0 +1,78 @@
+# Configuration
+
+This folder contains details describing how to write your own configurations to control how agents can interact with the `SWEEnv` environment.
+A configuration is represented as a single `.yaml` file, allowing you to...
+* Define the **commands** that agents may use to traverse + modify a codebase.
+* Write **prompts** that are determiniscally/conditionally shown to the agent over the course of a single trajectory.
+* Control the **input/output interface** that sits between the agent and `SWEEnv`.
+
+## Configuration File Fields
+The configuration is a `.yaml` file that consists of several fields. They are fully represented in this following outline:
+
+```yaml
+# Prompt Templates: Control how observations of environment are shown to agent
+system_template: | # .yaml syntax for multi-line string value
+  First `system` message shown to agent
+instance_template: |- # .yaml syntax for multi-line string value w/ no new line
+  Instance prompt, contains task instance-specific content
+next_step_template: |-
+  Format template of per-turn observation (Contains standard output from agent's action)
+next_step_no_output_template: |-
+  Format template of observation when there is no standard output from the agent's action
+format_error_template: |-
+  Format template of error message (Used when agent's action causes an error)
+demonstration_template: |
+  Format template for showing a demonstration to the agent
+demonstrations:
+- `trajectories/<username>/<experiment folder>/*.traj` 
+- File is a demonstration of how to solve a task. This could an agent generated trajectory.
+- You can include 1+ demonstrations
+
+# Environment States: Define features of the SWEEnv environment
+env_variables:
+# Default variables for SWEEnv at the beginning of each instance
+  CURRENT_FILE: 0
+  CURRENT_LINE:
+  OVERLAP:
+  SEARCH_FILES:
+  SEARCH_INDEX:
+  SEARCH_RESULTS:
+  WINDOW_SIZE:
+  START_INDEX:
+  END_INDEX:
+  START_CURSOR:
+  END_CUROSR:
+  START_CURSORS_MARK:
+  END_CURSOR_MARK:
+state_command: |
+# `state_command` allows you to update state variables to reflect any aspect of the environment (e.g. current working directory)
+  name: state
+  code: |
+    state() { echo '{"pwd": "'$PWD'"}';
+
+# Action Interface: Define how an agent interacts with the SWEEnv environment
+command_files:
+- path/to/bash_file.sh
+- Each file contains a list of commands implemented in bash
+- You can include 1+ command files
+parse_command: Reference to functionality for defining command documentation
+history_processor: Reference to functionality for controlling agent's message history
+parse_function: Parser run on agent output
+```
+
+We recommend looking at...
+* `configs/` for examples of properly formatted configuration files. Each configuration differs in its set of commands, input/output format, demonstrations, etc.
+* `commands/` for the bash implementations of the custom commands that SWE-agent uses to navigate + edit the codebase.
+
+## How a Configuration File is Processed
+Some notes on processing that occurs on config fields when SWE-agent is run:
+* Commands specified in `command_files` will be parsed into a single block of documentation text that can be referenced as `{command_docs}`.
+* `env_variables` are the default variables for the bash environment at the beginning of each instance.
+* `state_command` is used to extract state information from the bash environment (formatted as json) to be used in the templates given to the agent.
+
+Possible variables that can be used in templates are:
+- `{command_docs}` (an automatically compiled collection of available commands + their docstrings)
+- any variable given in `env_variables` (same spelling), e.g., `{WINDOW_SIZE}`
+- any variable extracted as json as part of the `state_command` function
+- the last observation `{observation}` 
+- ... this list will grow as we implement more features!