## File: README.md # OpenAI Evals > You can now configure and run Evals directly in the OpenAI Dashboard. [Get started →](https://platform.openai.com/docs/guides/evals) Evals provide a framework for evaluating large language models (LLMs) or systems built using LLMs. We offer an existing registry of evals to test different dimensions of OpenAI models and the ability to write your own custom evals for use cases you care about. You can also use your data to build private evals which represent the common LLMs patterns in your workflow without exposing any of that data publicly. If you are building with LLMs, creating high quality evals is one of the most impactful things you can do. Without evals, it can be very difficult and time intensive to understand how different model versions might affect your use case. In the words of [OpenAI's President Greg Brockman](https://twitter.com/gdb/status/1733553161884127435): ## Setup To run evals, you will need to set up and specify your [OpenAI API key](https://platform.openai.com/account/api-keys). After you obtain an API key, specify it using the [`OPENAI_API_KEY` environment variable](https://platform.openai.com/docs/quickstart/step-2-setup-your-api-key). Please be aware of the [costs](https://openai.com/pricing) associated with using the API when running evals. You can also run and create evals using [Weights & Biases](https://wandb.ai/wandb_fc/openai-evals/reports/OpenAI-Evals-Demo-Using-W-B-Prompts-to-Run-Evaluations--Vmlldzo0MTI4ODA3). **Minimum Required Version: Python 3.9** ### Downloading evals Our evals registry is stored using [Git-LFS](https://git-lfs.com/). Once you have downloaded and installed LFS, you can fetch the evals (from within your local copy of the evals repo) with: ```sh cd evals git lfs fetch --all git lfs pull ``` This will populate all the pointer files under `evals/registry/data`. You may just want to fetch data for a select eval. You can achieve this via: ```sh git lfs fetch --include=evals/registry/data/${your eval} git lfs pull ``` ### Making evals If you are going to be creating evals, we suggest cloning this repo directly from GitHub and installing the requirements using the following command: ```sh pip install -e . ``` Using `-e`, changes you make to your eval will be reflected immediately without having to reinstall. Optionally, you can install the formatters for pre-committing with: ```sh pip install -e .[formatters] ``` Then run `pre-commit install` to install pre-commit into your git hooks. pre-commit will now run on every commit. If you want to manually run all pre-commit hooks on a repository, run `pre-commit run --all-files`. To run individual hooks use `pre-commit run `. ## Running evals If you don't want to contribute new evals, but simply want to run them locally, you can install the evals package via pip: ```sh pip install evals ``` You can find the full instructions to run existing evals in [`run-evals.md`](docs/run-evals.md) and our existing eval templates in [`eval-templates.md`](docs/eval-templates.md). For more advanced use cases like prompt chains or tool-using agents, you can use our [Completion Function Protocol](docs/completion-fns.md). We provide the option for you to log your eval results to a Snowflake database, if you have one or wish to set one up. For this option, you will further have to specify the `SNOWFLAKE_ACCOUNT`, `SNOWFLAKE_DATABASE`, `SNOWFLAKE_USERNAME`, and `SNOWFLAKE_PASSWORD` environment variables. ## Writing evals We suggest getting starting by: - Walking through the process for building an eval: [`build-eval.md`](docs/build-eval.md) - Exploring an example of implementing custom eval logic: [`custom-eval.md`](docs/custom-eval.md) - Writing your own completion functions: [`completion-fns.md`](docs/completion-fns.md) - Review our starter guide for writing evals: [Getting Started with OpenAI Evals](https://cookbook.openai.com/examples/evaluation/getting_started_with_openai_evals) Please note that we are currently not accepting evals with custom code! While we ask you to not submit such evals at the moment, you can still submit model-graded evals with custom model-graded YAML files. If you think you have an interesting eval, please open a pull request with your contribution. OpenAI staff actively review these evals when considering improvements to upcoming models. ## FAQ Do you have any examples of how to build an eval from start to finish? - Yes! These are in the `examples` folder. We recommend that you also read through [`build-eval.md`](docs/build-eval.md) in order to gain a deeper understanding of what is happening in these examples. Do you have any examples of evals implemented in multiple different ways? - Yes! In particular, see `evals/registry/evals/coqa.yaml`. We have implemented small subsets of the [CoQA](https://stanfordnlp.github.io/coqa/) dataset for various eval templates to help illustrate the differences. When I run an eval, it sometimes hangs at the very end (after the final report). What's going on? - This is a known issue, but you should be able to interrupt it safely and the eval should finish immediately after. There's a lot of code, and I just want to spin up a quick eval. Help? OR, I am a world-class prompt engineer. I choose not to code. How can I contribute my wisdom? - If you follow an existing [eval template](docs/eval-templates.md) to build a basic or model-graded eval, you don't need to write any evaluation code at all! Just provide your data in JSON format and specify your eval parameters in YAML. [build-eval.md](docs/build-eval.md) walks you through these steps, and you can supplement these instructions with the Jupyter notebooks in the `examples` folder to help you get started quickly. Keep in mind, though, that a good eval will inevitably require careful thought and rigorous experimentation! ## Disclaimer By contributing to evals, you are agreeing to make your evaluation logic and data under the same MIT license as this repository. You must have adequate rights to upload any data used in an eval. OpenAI reserves the right to use this data in future service improvements to our product. Contributions to OpenAI evals will be subject to our usual Usage Policies: https://platform.openai.com/docs/usage-policies. --- ## File: docs/build-eval.md # Building an eval **Important: Please note that we are currently not accepting Evals with custom code!** While we ask you to not submit such evals at the moment, you can still submit modelgraded evals with custom modelgraded YAML files. This document walks through the end-to-end process for building an eval, which is a dataset and a choice of eval class. The `examples` folder contains Jupyter notebooks that follow the steps below to build several academic evals, thus helping to illustrate the overall process. The steps in this process are building your dataset, registering a new eval with your dataset, and running your eval. Crucially, we assume that you are using an [existing eval template](eval-templates.md) out of the box (if that's not the case, see [this example of building a custom eval](custom-eval.md)). If you are interested in contributing your eval publicly, we also include some criteria at the bottom for what we think makes an interesting eval. We are looking for evals in the following categories: - Over-refusals - Safety - System message steerability - In-the-wild hallucinations - Math / logical / physical reasoning - Real-world use case (please describe in your PR how this capability would be used in a product) - Other foundational capability If you have an eval that falls outside this category but still is a diverse example, please contribute it! ## Formatting your data Once you have an eval in mind that you wish to implement, you will need to convert your samples into the right JSON lines (JSONL) format. A JSONL file is just a JSON file with a unique JSON object per line. You can use the `openai` CLI (available with [OpenAI-Python](https://github.com/openai/openai-python)) to transform data from some common file types into JSONL: ``` openai tools fine_tunes.prepare_data -f data[.csv, .json, .txt, .xlsx or .tsv] ``` We include some examples of JSONL eval files in [registry/data/README.md](../evals/registry/data/README.md) Each JSON object will represent one data point in your eval. The keys you need in the JSON object depend on the eval template. All templates expect an `"input"` key, which is the prompt, ideally specified in [chat format](https://platform.openai.com/docs/guides/chat/introduction) (though strings are also supported). We recommend chat format even if you are evaluating non-chat models. If you are evaluating both chat and non-chat models, we handle the conversion between chat-formatted prompts and raw string prompts (see the conversion logic [here](../evals/prompt/base.py)). For the basic evals `Match`, `Includes`, and `FuzzyMatch`, the other required key is `"ideal"`, which is a string (or a list of strings) specifying the correct reference answer(s). For model-graded evals, the required keys vary based on the eval but is determined by the `{key}`s in the evaluation `prompt` that are not covered by the (optional) `args`. We have implemented small subsets of the [CoQA](https://stanfordnlp.github.io/coqa/) dataset for various eval templates to illustrate how the data should be formatted. See [`coqa/match.jsonl`](../evals/registry/data/coqa/match.jsonl) for an example of data that is suitable for the `Match` basic eval template and [`coqa/samples.jsonl`](../evals/registry/data/coqa/samples.jsonl) for data that is suitable for `fact` and `closedqa` model-graded evals. Note that even though these two model-graded evals expect different keys, we can include the superset of keys in our data in order to support both evals. If the dataset file is on your local machine, put the `jsonl` file in `evals/registry/data//samples.jsonl`. If it is in Cloud Object Storage, we support path-style URLs for the major clouds (for your personal use only, we will not accept PRs with cloud URLs). ## Registering the eval Register the eval by adding a file to `evals/registry/evals/.yaml` using the elsuite registry format. For example, for a `Match` eval, it would be: ``` : id: .dev.v0 metrics: [accuracy] .dev.v0: class: evals.elsuite.basic.match:Match args: samples_jsonl: /samples.jsonl ``` Upon running the eval, the data will be searched for in `evals/registry/data`. For example, if `test_match/samples.jsonl` is the provided filepath, the data is expected to be in `evals/registry/data/test_match/samples.jsonl`. The naming convention for evals is in the form `..`. - `` is the eval name, used to group evals whose scores are comparable. - `` is the data split, used to further group evals that are under the same ``. E.g., "val", "test", or "dev" for testing. - `` is the version of the eval, which can be any descriptive text you'd like to use (though it's best if it does not contain `.`). In general, running the same eval name against the same model should always give similar results so that others can reproduce it. Therefore, when you change your eval, you should bump the version. ## Running the eval You can now run your eval on your data from the CLI with your choice of model or completion function: ``` oaieval gpt-3.5-turbo ``` Congratulations, you have built your eval! Keep iterating on it until you are confident in the results. ## For model-graded evals: a step-by-step workflow We expect that the existing model-graded evals such as `fact`, `closedqa`, and `battle` will fit many use cases. However, other use cases may benefit from more customization, e.g., a different evaluation prompt. For these, there will be a bit more work involved, but generally still no coding required! 1. If you can't use an existing model-graded eval, create a new YAML or create a new entry to an existing YAML in `evals/registry/modelgraded` to specify the [parameters](eval-templates.md#parameters-for-model-graded-evals) of your eval. See [`humor.yaml`](../evals/registry/modelgraded/humor.yaml) for an example. - Note that, even if you are creating a new YAML, you may find it easiest to copy an existing YAML as a starting point. For example, model-graded evals which check a model completion against a rubric can copy `closedqa.yaml` and just edit the `args`. 2. Next, you will create your dataset and register your eval, as described above. See [`joke_fruits_labeled.jsonl`](../evals/registry/data/test_metaeval/joke_fruits_labeled.jsonl) and [`joke-fruits`](../evals/registry/evals/test-modelgraded.yaml), for example. - Note that it is recommended to specify `eval_type` at this step, when you register your eval, rather than step 1. 3. Run your eval, e.g., `oaieval gpt-3.5-turbo joke-fruits`. 4. (Recommended) Add a meta-eval for the model-graded eval! Each model-graded eval comes with a few knobs to tune, mainly `prompt` but also `eval_type`. In order to make sure the eval is of high quality, we recommend each model-graded eval contribution come with "choice labels", which are basically human-provided labels for which evaluation choice the model should have made. As an example (pretending that these jokes are actually funny), see the `"choice"` keys in [`joke_fruits_labeled.jsonl`](../evals/registry/data/test_metaeval/joke_fruits_labeled.jsonl), which are not used by the `joke-fruits` eval but are used by the [`joke-fruits-meta`](../evals/registry/evals/test-modelgraded.yaml) meta-eval right below it . After running the meta-eval, e.g., `oaieval gpt-3.5-turbo joke-fruits-meta`, the report will output `metascore/` accuracies, which should be close to "1.0" for a good model-graded eval. ## Criteria for contributing an eval Important: if you are contributing code, make sure to run `pip install pre-commit; pre-commit install` before committing and pushing to ensure that `black`, `isort`, and `autoflake` are run. We are interested in curating a diverse and interesting set of evals on which to improve our models going forward. Here are some criteria for what we consider a good eval: - [ ] The eval should be thematically consistent. We'd like to see a number of prompts all revolving around the same use case, subject domain, failure mode, etc. - [ ] The eval should be challenging. If GPT-4 or GPT-3.5-Turbo do well on all of the prompts, this is not as interesting. Of course, the eval should also be possible given the models' limitations and constraints. Oftentimes, a good rule of thumb is whether a human (potentially a subject expert) could do well on the prompts. - [ ] The eval should be directionally clear. The data should include good signal around what is the right behavior. This means, for example, high-quality reference answers or an exhaustive rubric for evaluating answers. - [ ] The eval should be carefully crafted. Before you submit, you should think through whether you have engineered your prompts for good performance, whether you are using the best eval template, whether you have spot checked your results to ensure accuracy, etc. Once you are ready to contribute your eval publicly, submit a PR and the OpenAI team will be happy to look it over. Make sure to fill out all parts of the template that is prepopulated into the PR message. Note that submitting a PR does not guarantee that OpenAI will eventually merge it. We will run our own checks and use our best judgment when considering which evals to follow up with. --- ## File: docs/completion-fn-protocol.md ### The Completion Function Protocol Here are the interfaces needed to implement the completion function protocol. Any implementation of this interface can be used inside `oaieval`. Reference implementations: - [OpenAICompletionFn](../evals/completion_fns/openai.py) - [LangChainLLMCompletionFn](../evals/completion_fns/langchain_llm.py) #### CompletionFn Completion functions should implement the `CompletionFn` interface: ```python class CompletionFn(Protocol): def __call__( self, prompt: Union[str, list[dict[str, str]]], **kwargs, ) -> CompletionResult: ``` We take a `prompt` representing a single sample from an eval. These prompts can be represented as either a text string or a list of messages in [OpenAI Chat format](https://platform.openai.com/docs/guides/chat/introduction). To work with the existing evals, Completion Function implementations would need to handle both types of inputs, but we provide helper functionality to convert Chat formatted messages into a text string if that is the preferred input for your program: ```python from evals.prompt.base import CompletionPrompt # chat_prompt: list[dict[str, str]] -> text_prompt: str text_prompt = CompletionPrompt(chat_prompt).to_formatted_prompt() ``` #### CompletionResult The completion function should return an object implementing the `CompletionResult` interface: ```python class CompletionResult(ABC): @abstractmethod def get_completions(self) -> list[str]: pass ``` The `get_completions` method returns a list of string completions. Each element should be considered a unique completion (in most cases this will be a list of length 1). #### Using your CompletionFn This is all that's needed to implement a Completion function that works with our existing Evals, allowing you to more easily evaluate your end-to-end logic on tasks. See [completion-fns.md](completion-fns.md) to see how to register and use your completion function with `oaieval`. --- ## File: docs/completion-fns.md # Completion Functions ## What are completion functions In [run-evals.md](run-evals.md), we learned how to make calls to `oaieval` to run an eval against a completion function. Completion Functions are generalizations of model completions, where a "completion" is some text output that would be our answer to the prompt. For example, if "Who played the girl elf in the hobbit?" is our prompt, the correct completion is "Evangeline Lilly". While we can just test a model directly to see if it generates "Evangeline Lilly", we can imagine doing numerous other operations under the hood to improve our ability to answer this question, like giving the model access to a browser to look up the answer before responding. Making it easy to implement this kind of under-the-hood operators before responding is the motivation behind building Completion Functions. ## How to implement completion functions A completion function needs to implement some interfaces that make it usable within Evals. At its core, it is just standardizing inputs to be a text string or [Chat conversation](https://platform.openai.com/docs/guides/chat), and the output to be a list of text strings. Implementing this interface will allow you to run your Completion Function against any eval in Evals. The exact interfaces needed are described in detail in [completion-fn-protocol.md](completion-fn-protocol.md) We include some example implementations inside `evals/completion_fns`. For example, the [`LangChainLLMCompletionFn`](../evals/completion_fns/langchain_llm.py) implements a way to generate completions from [LangChain LLMs](https://python.langchain.com/en/latest/modules/models/llms/getting_started.html). We can then use these completion functions with `oaieval`: ``` oaieval langchain/llm/flan-t5-xl test-match ``` ## Registering Completion Functions Once you have written a completion function, we need to make the class visible to the `oaieval` CLI. Similar to how we register our evals, we also register Completion Functions inside `evals/registry/completion_fns` as `yaml` files. Here is the registration for our langchain LLM completion function: ```yaml langchain/llm/flan-t5-xl: class: evals.completion_fns.langchain_llm:LangChainLLMCompletionFn args: llm: HuggingFaceHub llm_kwargs: repo_id: google/flan-t5-xl ``` Here is how it breaks down `langchain/llm/flan-t5-xl`: This is the top level key that will be used to access this completion function with `oaieval`. `class`: This is the path to your implementation of the completion function protocol. This class needs to be importable within your python environment. `args`: These are arguments that are passed to your completion function when it is instantiated. ### Developing Completion Functions outside of Evals It is possible to register CompletionFunctions without directly modifying the registry or code inside `Evals` by using the `--registry_path` argument. As an example, let's say I want to use `MyCompletionFn` located inside `~/my_project/`: ``` my_project ├── my_completion_fn.py └── completion_fns └── my_completion_fn.yaml ``` If `my_project` is importable within the python environment (accessible via PYTHONPATH), we can structure `my_completion_fn.yaml` as: ``` my_completion_fn: class: my_project.my_completion_fn:MyCompletionFn ``` Then, we can make calls to `oaieval` using: ``` oaieval my_completion_fn test-match --registry_path ~/my_project ``` --- ## File: docs/custom-eval.md # How to add a custom eval **Important: Please note that we are currently not accepting Evals with custom code!** While we ask you to not submit such evals at the moment, you can still submit modelgraded evals with custom modelgraded YAML files. This tutorial will walk you through a simple example of writing and adding a custom eval. The example eval will test the model's ability to do basic arithmetic. We will assume that you have followed the setup instructions in the [README](../README.md) and gone through the other docs for how to run and build evals. When writing your own evals, the primary files of interest are: - `evals/api.py`, which provides common interfaces and utilities used by eval creators to sample from models and process the results, - `evals/record.py`, which defines the recorder classes which log eval results in different ways, such as to a local JSON file or to a remote Snowflake database, and - `evals/metrics.py`, which defines various common metrics of interest. These files provide a suite of tools for writing new evals. Once you have gone through this tutorial, you can see a more realistic example of these tools in action with the [machine translation](../evals/elsuite/translate.py) [eval example](../examples/lafand-mt.ipynb), which also implements custom eval logic in lieu of using an existing template. ## Create your datasets The first step is to create the datasets for your eval. Here, we will create toy train and test sets of just two examples each. The test examples are what we will evaluate the model on, and we'll include the train examples as few-shot examples in the prompt to the model. We will use the new chat format described [here](https://platform.openai.com/docs/guides/chat/introduction). By default, we encourage all evals to be written using chat formatting if you want to evaluate our new models. Under the hood, we [convert](../evals/prompt/base.py) chat formatted data into raw strings for older non chat models. To create the toy datasets, in your terminal, type: ```bash echo -e '{"problem": "2+2=", "answer": "4"}\n{"problem": "4*4=", "answer": "16"}' > /tmp/train.jsonl echo -e '{"problem": "48+2=", "answer": "50"}\n{"problem": "5*20=", "answer": "100"}' > /tmp/test.jsonl ``` ## Create an eval The next step is to write a Python class that represents the actual evaluation. This class uses your datasets to create prompts, which are passed to the model to generate completions. Evaluation classes generally will inherit from the `evals.Eval` base class (defined in `evals/eval.py`) and will override two methods: `eval_sample` and `run`. Let's create a file called `arithmetic.py` under the `evals/elsuite` folder. We'll start by defining the eval class. Its `__init__` method will take in the arguments we need (references to the train and test sets) along with other `kwargs` that will be handled by the base class. We'll also define the `run` method which takes in a `recorder` and returns the final metrics of interest. ```python import random import textwrap import evals import evals.metrics class Arithmetic(evals.Eval): def __init__(self, train_jsonl, test_jsonl, train_samples_per_prompt=2, **kwargs): super().__init__(**kwargs) self.train_jsonl = train_jsonl self.test_jsonl = test_jsonl self.train_samples_per_prompt = train_samples_per_prompt def run(self, recorder): """ Called by the `oaieval` CLI to run the eval. The `eval_all_samples` method calls `eval_sample`. """ self.train_samples = evals.get_jsonl(self.train_jsonl) test_samples = evals.get_jsonl(self.test_jsonl) self.eval_all_samples(recorder, test_samples) # Record overall metrics return { "accuracy": evals.metrics.get_accuracy(recorder.get_events("match")), } ``` Generally, most `run` methods will follow the same pattern shown here: loading the data, calling `eval_all_samples`, and aggregating the results (in this case, using the `get_accuracy` function in `evals/metrics.py`). `eval_all_samples` takes in both the `recorder` and the `test_samples` and, under the hood, will call the `eval_sample` method on each sample in `test_samples`. So let's write that `eval_sample` method now: ```python def eval_sample(self, test_sample, rng: random.Random): """ Called by the `eval_all_samples` method to evaluate a single sample. ARGS ==== `test_sample`: a line from the JSONL test file `rng`: should be used for any randomness that is needed during evaluation This method does the following: 1. Generate a prompt that contains the task statement, a few examples, and the test question. 2. Generate a completion from the model. 3. Check if the generated answer is correct. """ stuffing = rng.sample(self.train_samples, self.train_samples_per_prompt) prompt = [ {"role": "system", "content": "Solve the following math problems"}, ] for i, sample in enumerate(stuffing + [test_sample]): if i < len(stuffing): prompt += [ {"role": "system", "content": sample["problem"], "name": "example_user"}, {"role": "system", "content": sample["answer"], "name": "example_assistant"}, ] else: prompt += [{"role": "user", "content": sample["problem"]}] result = self.completion_fn(prompt=prompt, temperature=0.0, max_tokens=1) sampled = result.get_completions()[0] evals.record_and_check_match(prompt=prompt, sampled=sampled, expected=test_sample["answer"]) ``` You'll notice that `eval_sample` doesn't take the `recorder` as an argument. This is because `eval_all_samples` sets it to be the default recorder before calling `eval_sample`, and the recording utilities defined in `evals/record.py` use the default recorder. In this example, the `eval_sample` method passes off a lot of the heavy lifting to the `evals.record_and_check_match` utility function, which is defined in `evals/api.py`. This utility function queries the model, defined by `self.model_spec`, with the given `prompt` and checks to see if the result matches the `expected` answer (or one of them, if given a list). It then records these matches (or non matches) using the default recorder. `eval_sample` methods may vary greatly based on your use case. If you are building custom evals, it is a good idea to be familiar with the functions available to you in `evals/record.py`, `evals/metrics.py`, and especially `evals/api.py`. ## Register your eval The next step is to register your eval in the registry so that it can be run using the `oaieval` CLI. Let's create a file called `arithmetic.yaml` under the `evals/registry/evals` folder and add an entry for our eval as follows: ```yaml # Define a base eval arithmetic: # id specifies the eval that this eval is an alias for # in this case, arithmetic is an alias for arithmetic.dev.match-v1 # When you run `oaieval davinci arithmetic`, you are actually running `oaieval davinci arithmetic.dev.match-v1` id: arithmetic.dev.match-v1 # The metrics that this eval records # The first metric will be considered to be the primary metric metrics: [accuracy] # Define the eval arithmetic.dev.match-v1: # Specify the class name as a dotted path to the module and class class: evals.elsuite.arithmetic:Arithmetic # Specify the arguments as a dictionary of JSONL URIs # These arguments can be anything that you want to pass to the class constructor args: train_jsonl: /tmp/train.jsonl test_jsonl: /tmp/test.jsonl ``` The `args` field should match the arguments that your eval class `__init__` method expects. ## Run your eval The final step is to run your eval and view the results. ```sh pip install . # you can omit this if you used `pip install -e .` to install oaieval gpt-3.5-turbo arithmetic ``` If you run with the `gpt-3.5-turbo` model, you should see an output similar to this (we have cleaned up the output here slightly for readability): ``` % oaieval gpt-3.5-turbo arithmetic ... [registry.py:147] Loading registry from .../evals/registry/evals ... [registry.py:147] Loading registry from .../.evals/evals ... [oaieval.py:139] Run started: ... [eval.py:32] Evaluating 2 samples ... [eval.py:138] Running in threaded mode with 1 threads! 100%|██████████████████████████████████████████| 2/2 [00:00<00:00, 3.35it/s] ... [record.py:320] Final report: {'accuracy': 1.0}. Logged to /tmp/evallogs/_gpt-3.5-turbo_arithmetic.jsonl ... [oaieval.py:170] Final report: ... [oaieval.py:172] accuracy: 1.0 ... [record.py:309] Logged 6 rows of events to /tmp/evallogs/_gpt-3.5-turbo_arithmetic.jsonl: insert_time=2.038ms ``` --- ## File: docs/eval-templates.md # Existing templates for evals In using Evals, we have discovered several "templates" that accommodate many different benchmarks. We have implemented these templates in `evals/elsuite` in order to simplify the development of new evals. We believe that, with these templates, many evals will not require any coding to implement! Instead, you'll pick one of the existing templates and simply specify the dataset and parameters. ## Basic eval templates In cases where the desired model response has very little variation, such as answering multiple choice questions or simple questions with a straightforward answer, we have found the following templates to be useful. For a model completion `a` and a reference list of correct answers `B`, the following evals implement: - [`basic/match.py:Match`](../evals/elsuite/basic/match.py): `any([a.startswith(b) for b in B])` - [`basic/includes.py:Includes`](../evals/elsuite/basic/includes.py): `any([(b in a) for b in B])` - [`basic/fuzzy_match.py:FuzzyMatch`](../evals/elsuite/basic/fuzzy_match.py): `any([(a in b or b in a) for b in B])` To compare a model completion `a` in *JSON format* to a reference list of correct answers `B` also formatted in JSON, use the following eval: - [`basic/json_match.py:JsonMatch`](../evals/elsuite/basic/json_match.py) yields a match if `a` is identical to at least one answer from `B`. Two JSON objects are identical if they have the same set of keys and the values for each key are identical. Key order is not significant, and whitespace outside values is ignored. Invalid JSON never matches. Which eval template you use will depend on your use case. It is always recommended that you inspect the completions from your model, as this will help you determine how and whether to tweak your prompt (or your reference answers) and pick your eval template. Academic benchmarks oftentimes fit the mold of these basic evals, and we have implemented several end-to-end examples of academic evals as Jupyter notebooks in the `examples` folder. Sometimes, [custom eval logic](custom-eval.md) will better suit your needs. One example of this is the [machine translation](../evals/elsuite/translate.py) eval [example](../examples/lafand-mt.ipynb), in which there is a unique and clearly defined metric that we wish to use in our eval. You should use your best judgment when deciding between custom eval logic, using a basic eval template, or using model-graded evals as described next. ## The model-graded eval template In cases where the desired model response can contain significant variation, such as answering an open-ended question, we have found that using the model to grade itself is a viable strategy for automated evaluation. In general, the evaluation model and the model being evaluated don't have to be the same, though we will assume that they are here for ease of explanation. [`modelgraded/classify.py:ModelBasedClassify`](../evals/elsuite/modelgraded/classify.py) implements the main logic behind our model-graded eval template. In short, we get the model's completion to the original prompt, wrap it in an evaluation prompt, and get the model's completion to the evaluation prompt, which we parse into our metrics of interest. Crucially, the evaluation prompt should prime the model to answer in such a way that is easily parsable, e.g., in multiple choice format or with a simple yes/no. We describe some example model-graded evals below, but first we specify the parameters for this eval template. ### Parameters for model-graded evals Refer to the [`classify.py:ModelBasedClassify`](../evals/elsuite/modelgraded/classify.py) class to see how these parameters are used in the code. - `prompt`: The evaluation prompt which should take in the model's completion to the original prompt, potentially along with some other information, and steer the model to provide an evaluation that is easily parsable. Portions denoted by curly braces (i.e., `{key}`) are filled in either from the data `input_outputs` or the additional `args` (see below). - `input_outputs`: A mapping specifying which inputs to use to generate which completions. For many evals, there will only be a single input-completion pair, though there can be more, e.g., when comparing two completions against each other. - `choice_strings`: The choices that we expect the model completion to contain given the evaluation prompt. For example, `"ABCDE"` or `["Yes", "No", "Unsure"]`. Any other choices returned by the model are parsed into `"__invalid__"`. - `choice_scores` (optional): A mapping of each choice to its score, which is logged as a metric. For example, if a response of `"Yes"` (resp. `"No"`) indicates that the model's original completion was good (resp. bad), we may assign this choice a score of 1 (resp. 0). - `eval_type` (optional): How we expect the model to format its response to the evaluation prompt. Currently the supported options are: - `"cot_classify"` ("chain-of-thought then classify", i.e., reason then answer) expects that the parsable portion of the response (i.e., the portion containing the choice) will be at the end of the completion. We recommend this as the default as it typically provides the most accurate model-graded evaluations. - `"classify_cot"` (answer then reason) expects that the model response will contain the choice first. - `"classify"` expects that the model response will only contain the choice. There are two ways to specify `eval_type`. The recommended way is in the `evals/registry/evals` YAML file. If done this way, an instruction will automatically be appended to `prompt` to steer the model towards the expected format (see `ANSWER_PROMPTS` in [the code](../evals/elsuite/modelgraded/classify.py)). Alternatively, you may specify `eval_type` in the `evals/registry/modelgraded` YAML, but you will need to include an appropriate instruction directly in the `prompt`. - `output_template` (optional): If specified, determines how the model's output (or outputs, if `n > 1`) will be formatted within the completion. ### Example model-graded evals To instantiate model-graded evals, create a YAML file in `evals/registry/modelgraded` which specifies values for the arguments described above. We have provided a few examples, which illustrate the process for creating a model-graded eval, but which we also believe are general enough to be useful out of the box for many evals. [`fact.yaml`](../evals/registry/modelgraded/fact.yaml): a factual consistency eval which, given a completion `a` and reference answer `b`, returns: - `"A"` if `a` $\subseteq$ `b`, i.e., the submitted answer is a subset of the expert answer and is fully consistent with it. - `"B"` if `a` $\supseteq$ `b`, i.e., the submitted answer is a superset of the expert answer and is fully consistent with it. - `"C"` if `a` $=$ `b`, i.e., the submitted answer contains all the same details as the expert answer. - `"D"` if `a` $\neq$ `b`, i.e., there is a disagreement between the submitted answer and the expert answer. - `"E"` if `a` $\approx$ `b`, i.e., the answers differ, but these differences don't matter from the perspective of factuality. [`closedqa.yaml`](../evals/registry/modelgraded/closedqa.yaml): a question answering eval, which, given a prompt containing a question and the necessary information to answer the question, checks whether the model's answer is: - relevant, i.e., extracted from the information provided in the prompt, - concise, i.e., did not contain unnecessary details or information, and - correct, i.e., uses the extracted information to come to the right conclusion. Note that this eval is implemented more generally as a "criteria-checking" eval, which specifies the evaluation prompt as checking a given criterion and feeding in the above desiderata one by one. We believe that many other evals can be implemented by specifying a "rubric" detailing the criteria of interest and following the same prompt and yes/no choices. [`battle.yaml`](../evals/registry/modelgraded/battle.yaml): a head-to-head eval which compares two model completions for two potentially different prompts. `choice_scores` is used here to log how often the first completion is judged to be better than the second. We include additional examples which test more specific model capabilities (such as humor) and are thus less generalizable to other evals. However, these examples still serve to illustrate different ways to write evaluation prompts and set up model-graded evals. See [this section](build-eval.md#for-model-graded-evals-a-step-by-step-workflow) for more detailed steps on building model-graded evals. --- ## File: docs/run-evals.md # How to run evals We provide two command line interfaces (CLIs): `oaieval` for running a single eval and `oaievalset` for running a set of evals. ## Running an eval When using the `oaieval` command, you will need to provide the completion function you wish to evaluate as well as the eval to run. E.g., ```sh oaieval gpt-3.5-turbo test-match ``` The valid eval names are specified in the YAML files under `evals/registry/evals` and their corresponding implementations can be found in `evals/elsuite`. In this example, `gpt-3.5-turbo` is an OpenAI model that we dynamically instantiate as a completion function using `OpenAIChatCompletionFn(model=gpt-3.5-turbo)`. Any implementation of the `CompletionFn` protocol can be run against `oaieval`. By default, we support calling `oaieval` with any model available on the OpenAI API or with CompletionFunctions available in [`evals/registry/completion_fns`](../evals/registry/completion_fns/). We are always interested in adding more completion functions and we encourage you to implement your own to reflect specific use cases. More details on `CompletionFn` found here: [`completion-fns.md`](completion-fns.md) These CLIs can accept various flags to modify their default behavior. For example: - If you wish to log to a Snowflake database (which you have already set up as described in the [README](../README.md)), add `--no-local-run`. - By default, logging locally or to Snowflake will write to `tmp/evallogs`, and you can change this by setting a different `--record_path`. You can run `oaieval --help` to see a full list of CLI options. ## Running an eval set ```sh oaievalset gpt-3.5-turbo test ``` Similarly, `oaievalset` also expects a model name and an eval set name, for which the valid options are specified in the YAML files under `evals/registry/eval_sets`. By default we run with 10 threads, and each thread times out and restarts after 40 seconds. You can configure this, e.g., ```sh EVALS_THREADS=42 EVALS_THREAD_TIMEOUT=600 oaievalset gpt-3.5-turbo test ``` Running with more threads will make the eval faster, though keep in mind the costs and your [rate limits](https://platform.openai.com/docs/guides/rate-limits/overview). Running with a higher thread timeout may be necessary if you expect each sample to take a long time, e.g., the data contain long prompts that elicit long responses from the model. If you have to stop your run or your run crashes, we've got you covered! `oaievalset` records the evals that finished in `/tmp/oaievalset/{model}.{eval_set}.progress.txt`. You can simply rerun the command to pick up where you left off. If you want to run the eval set starting from the beginning, delete this progress file. Unfortunately, you can't resume a single eval from the middle. You'll have to restart from the beginning, so try to keep your individual evals quick to run. ## Logging By default, `oaieval` [records events](/evals/record.py) into local JSONL logs which can be inspected using a text editor or analyzed programmatically. 3rd-party tools such as [naimenz/logviz](https://github.com/naimenz/logviz) may be helpful to visualize the logs, though we don't provide support or guarantees for their use. --- ## File: evals/elsuite/identifying_variables/README.md # Identifying Variables This eval tests how well models can determine what should be treated as the independent, dependent, and control variables for an experiment that tests a particular hypothesis, given some observational context. ## Usage Run with: ```bash oaieval identifying_variables ``` We have found that `generation/cot/gpt-4-1106-preview` works well on this eval. For more examples of tested solvers, see [`./scripts/run_experiments.sh`](./scripts/run_experiments.sh). ## Evaluation Process The evaluation process is as follows for a given sample from our dataset: 1. The `TASK_DESCRIPTION` prompt is shown to the solver. 2. The sample is passed through a _renderer_ that processes the samples and renders an observation of the interactions of variables, which is placed in the `SAMPLE_MESSAGE` prompt template. 3. The solver answers in the form: `[@ANSWER valid_hyp: ; independent: ; dependent: ; control: ]`. The answer is parsed and evaluated by the eval. If the answer cannot be parsed, we mark this as a violation and the sample is treated as incorrect. ## Prompts We refer readers to the [`./prompts.py`](./prompts.py) file for the `TASK_DESCRIPTION` and `SAMPLE_MESSAGE` prompts used in the eval. ## Metrics | **Metric** | **Notes** | |---|---| | `ctrl_nDCG` | A modified version of the [normalized discounted cumulative gains (nDCG)](https://en.wikipedia.org/wiki/Discounted_cumulative_gain#Normalized_DCG) metric, which rewards listing the correct control variables first and penalizes naming irrelevant variables. | | `ctrl_recall` | Number of variables correctly marked as control variables / total number of variables to control according to the gold label | | `ctrl_recall` | Number of variables incorrectly marked as control variables / total number of variables not to control according to the gold label | | `hyp_valid_acc` | Target hypothesis plausibility validation accuracy (correct/incorrect) | | `ind_acc` | Independent variable determination accuracy (correct/incorrect) | | `dep_acc` | Dependent variable determination accuracy (correct/incorrect) | | `violation_rate` | Number of samples with violations (model failed to answer in correct format) / total number of samples | ## Variants We support variations on the eval along two dimensions, `renderer` and `dataset`: ```bash oaieval identifying_variables.. ``` The eval defaults to `identifying_variables.language-corrset.balanced-ctrl`. ### Dataset We provide 4 dataset variants: | `dataset` | Notes | | --- | --- | | `balanced-ctrl` | 500 samples balanced across number of control variables (from 0 to 8). | | `balanced-ctrl-large` | As `balanced-ctrl`, but with 5,000 samples. | | `balanced-hypotheses` | 500 samples balanced across target hypotheses being implausible/plausible. | | `balanced-hypotheses-large` | As `balanced-hypotheses`, but with 5,000 samples. | ### Renderers We have 6 different renderers, implemented in [`./renderers/`](./renderers/). The default renderer is `language-corrset`. Here is an example render from this type: ``` The following is a description of the observations made about a set of variables. In general, there were cases where some variables changed in tandem with each other, while others did not. For example, changes in x_5075 were observed to reflect changes in x_3314 and viceversa. Changes in x_9549 were not observed to reflect any changes in previously mentioned variables. Changes in x_1808 were not observed to reflect any changes in previously mentioned variables. Likewise, changes in x_9726 were observed to reflect changes in x_1808 and viceversa. ``` ### Show Tree We provide an additional variant of the eval where the decision tree implementing the reasoning for scoring a perfect score is shown to the model. This variant can be run by passing the `show_tree=True` flag to eval, e.g. ```bash oaieval identifying_variables --extra_eval_params show_tree=True ``` ## Custom Solvers We implement two custom programmatic solvers to serve as baselines. 1. `identifying_variables/random`: a solver that randomly selects whether the hypothesis is plausible with probability 0.5, and if so randomly samples the independent, dependent and control variables. We view this baseline as equivalent to randomly guessing. 2. `identifying_variables/noctrl`: this is a solver that always outputs an empty list for the variables to control, essentially eliminating any chance of false positives. This can provide stronger performance than the random baseline, since it avoids any penalization for returning incorrect variables, and can even achieve a perfect score on samples that indeed do not have any variables to control We refer to [`./solvers.py`](./solvers.py) for the implementation of these solvers. ## Token Usage Estimates We estimated per-run token usage on the default dataset size (500 samples) for the least and most token-intensive configurations for each model type (respectively, direct models on `identifying_variables.corrset` with `show_tree=False`; and CoT models on `identifying_variables.language-tabular` with `show_tree=True`). | | **input tokens/run** | **output tokens/run** | **total tokens/run** | |---|---|---|---| | **GPT-4-base HHH (corrset, no tree)** | 1,200,000 | 250,000 | 1,450,000 | | **GPT-4-base CoT HHH (language-tabular, with tree)** | 1,500,000 | 240,000 | 1,740,000 | | **GPT-3.5-turbo Direct (corrset, no tree)** | 430,000 | 88,000 | 518,000 | | **GPT-3.5-turbo CoT (language-tabular, with tree)** | 780,000 | 14,000 | 794,000 | | **GPT-4-1106-preview Direct (corrset, no tree)** | 430,000 | 53,000 | 483,000 | | **GPT-4-1106-preview CoT (language-tabular, with tree)** | 860,000 | 14,000 | 874,000 | These estimates were taken using the `balanced-hypotheses` dataset but should roughly apply to the `-balanced-ctrl` datasets. For `-large` datasets (5000 samples), multiply the above numbers by 10. ## Future modifications - Revisit the definition of the headline `ctrl_nDCG` metric - Devise additional auxiliary metrics to paint a more complete picture - What if we show the decision trees described in natural language rather than pseudocode? - How can we extend this eval to multi-variable dependencies? ## Version History - v0: Initial version released ## Contribution Statement Eval design, implementation, and results evaluation and writeup were primarily conducted by Giulio Starace. James Aung was of enormous assistance in report writing, and additionally provided general feedback and project management throughout the eval. Oliver Jaffe and Jan Betley were responsible for code reviews throughout the implementation process, along with fine-grained feedback on the project in general. Additional guidance was provided by (alphabetically by last-name) Steven Adler and Chan Jun Shern, who helped with brainstorming, gave research input and report revisions. ## Appendix ### Perfect output decision trees The following decision trees are used to determine the perfect output (aka "gold label") for a given sample. --- **Figure A1**: Decision tree for determining whether a given variable should be controlled. --- **Figure A2**: Decision tree for determining a hypothesis is valid and if so what the independent and dependent variables are. --- --- ## File: evals/elsuite/hr_ml_agent_bench/README.md # Human-Relative MLAgentBench Eval This eval measures a model's ability to solve diverse machine learning research tasks. The best-known human performance has been collated for each task, which is used to calculate a “human-relative” percentage for each task attempt; 0% is a naive baseline (e.g. “random guess”), 100% is obtaining the same performance-gain as the best-known human, and 200% is obtaining 2x the performance-gain of said human. Our thanks go to the authors of [MLAgentBench](https://github.com/snap-stanford/MLAgentBench) on which this work was built. This eval contains the following 15 tasks: | Task | Description | | --- | --- | | Ant | Coordinate the four legs of an ant-like robot to move forward while applying as little torque on each of the eight joints as possible. | | Bipedal Walker | Make a robot walk to the rightmost end of the screen without falling over. Applying motor torque costs a small amount of points, more optimal agent will get better score. | | Cart Pole | Prevent a pole attached to a cart from falling over by pushing the cart either left or right at each timestep. | | CIFAR-10 | Improve model performance as much as possible within 10 training epochs and save per-class probabilities for the test set. | | Feedback Prize | Train a language model to grade essays written by 8th-12th grade English Language Learners and submit predictions for the test set. | | House Prices | Train a model to predict the sale price of a house, iterating over different models or feature selections to enhance performance. | | Humanoid | Make a humanoid robot walk forward as fast as possible without falling over. | | IMDb | Fine-tune DistilBERT on the IMDb dataset to classify movie reviews and save per-class probabilities for the test set. | | Inverted Pendulum | Similarly to Cart Pole, the goal is to prevent a pole attached to a cart from falling over by pushing the cart either left or right at each timestep. The cart is simulated in Mujoco physics simulator, allowing for more complex dynamics (such as varying the effects of gravity). | | OGBN arXiv | Improve model performance within 10 training epochs on the ogbn-arxiv dataset. | | Parkinson’s Disease | Train a model on Parkinson's Disease data, focusing on improved performance and lower SMAPE scores, then submit the best predictions. | | Pong | Play first-to-21 Pong where the goal is to deflect the ball into your opponent’s goal. | | Pusher | Move a cylinder to a target position using a robot arm consisting of a shoulder, elbow, forearm and wrist joints. | | Spaceship Titanic | Train a model on the Spaceship Titanic dataset, iterating for better performance, and submit the best predictions. | | Vectorization | ​​Improve the execution speed of a script by vectorizing computations using numpy, focusing on a specified portion of code. | ## Setup > **⚠️ Warning:** *This eval allows language models to run arbitrary code on your machine. Please ensure that you only run these experiments in a properly sandboxed environment.* > **ℹ️** *Multiple tasks require a GPU. We comfortably ran our experiments on a [NC64as T4 v3](https://learn.microsoft.com/en-us/azure/virtual-machines/nct4-v3-series) machine from Microsoft Azure with an attached 2TB SSD.* The list of dependencies needed to run this eval are found in `requirements.txt`, which can be installed by running: ```bash pip install -r requirements.txt ``` Some tasks (optionally) require additional dependencies, which can be found in `benchmarks//scripts/requirements.txt` and likewise can be installed by running: ```bash pip install -r benchmarks//scripts/requirements.txt ``` where `` is the name of the task you wish to run (e.g. `ant`). To install all dependencies for all tasks, run: ```bash sh scripts/install_all_requirements.sh ``` Alternatively, a [Dev Container](https://code.visualstudio.com/docs/devcontainers/containers), `devcontainer.json`, is provided for use with Visual Studio Code, which contains all necessary dependencies and is pre-configured to run the eval. This Dev Container requires the [NVIDIA Container Runtime](https://developer.nvidia.com/container-runtime) to be installed on the host machine. ## Usage Run individual tasks with: ```bash EVALS_SEQUENTIAL=1 oaieval hr-ml-agent-bench. # This eval doesn't currently support multi-threading. ``` Where `` is one of the 15 tasks listed in the Tasks section below (e.g. `hr-ml-agent-bench.vectorization`). We suggest using `gpt-4-1106-preview` or `hr_ml_agent_bench/baseline/gpt-4-1106-preview` as default choices for ``. For example, to evaluate the `hr_ml_agent_bench/baseline/gpt-4-1106-preview` solver on the `CIFAR-10` task, run: ```bash EVALS_SEQUENTIAL=1 oaieval hr_ml_agent_bench/baseline/gpt-4-1106-preview hr-ml-agent-bench.cifar10 ``` See `evals/registry/solvers/hr-ml-agent-bench.yaml` for a full list of recommended Solvers. Alternatively, run all tasks with: ```bash EVALS_SEQUENTIAL=1 oaievalset hr-ml-agent-bench ``` ## Evaluation Process Overall, each run is structured as follows: 1. Given the task description, the agent begins the task. 2. At each step, the agent selects an action from the list of tools (e.g. read a file, edit a file, execute script) or gives a final answer. 3. For a selected agent action: - If the action is valid, the eval executes the chosen action in the environment. - If the agent attempts to use the Stable Baselines module, the environment refuses: ``` The action has been terminated because it used the Stable Baselines module, which is not allowed. Please write your implementation from scratch. ``` - If the action exceeds the action time limit (e.g. by executing a resource-intensive Python script), the environment throws an error: ``` Time's up! The action exceeded the maximum time limit and terminated early. ``` - If the agent fails to choose a valid action, the environment asks for a new response: ``` No valid action found! You must specify an action at each step in the following format: Action: the action to take, should be one of the names of the tools Action Input: the input to the action as a valid JSON string ``` 4. The observation from the environment (e.g. the output of the script execution if the agent chose to execute a script) is returned to the agent, and the agent is prompted to select a further action. 5. Steps 2-4 repeat until the agent calls the “final_answer” tool, or until the maximum number of steps is reached (default max steps is 30). 6. When the attempt has ended, an evaluation script is run to assess how well the agent did. All attempts are subject to a step, overall time and per-action time limit; the exact limits are task-specific, depending on how challening the task is. This prevents the model from indefinitely refining its solution, or from running scripts which take too long to execute. ## Task State | Key | Value | | --- | --- | | `task_description` | You have access to the following tools:`{tools_prompt}`Research Problem: `{research_problem}`Always respond in this format exactly:Action: the action to take, should be one of the names of the toolsAction Input: the input to the action as a valid JSON string | | `current_state` | TaskStateMetadata object that tracks various metadata. | ## Metrics The below are the key metrics of this eval: | Metric | Interpretation | | --- | --- | | `task_name` | Task name | | `model_score` | Raw score of the evaluated model on the task | | `naive_baseline_score` | Raw score of a naive baseline e.g. ‘do nothing’ ‘random guess’ etc. | | `human_baseline_score` | Raw score of the human baseline, the best-known human performance on this task | | `model_score_normalized` | Evaluated model score normalised between 0 and 1 | | `naive_baseline_score_normalized` | Naive baseline score normalised between 0 and 1 | | `human_baseline_score_normalized` | Human baseline score normalised between 0 and 1 | | `model_score_humanrelative` | The model score relative to the human baseline i.e. 1 = same as human, 2 = 2x performance-gain of human, etc. | ## Tasks This eval currently contains 15 tasks. | Task | Description | | --- | --- | | `hr-ml-agent-bench.cifar10` | Given a training script on a dataset train.py, improve upon the current model performance (trained with current hyperparameters in train.py) as much as possible. The training epochs should be within 10 to save time. Save per class probabilities for test set examples to submission.csv as shown in train.py. | | `hr-ml-agent-bench.feedback` | Go through the data_description.txt file to understand the data and the task. Then fill in the provided train.py script to train a language model to get a good performance. Finally, you should submit the predictions of your best model for the test set as a submission.csv as described in the evaluation_details.txt file. Never try to read any csv files directly. Do not forget to execute the changes you made to check for performance. | | `hr-ml-agent-bench.house-price` | Go through the data_description.txt file to understand the data and all the features. Then fill in the provided train.py script to train a model and iterate over different models or feature selections to get a better performance. Never try to read any csv files directly. Do not forget to execute the changes you made to check for performance. Store the predictions of the test set in submission.csv, as shown in train.py. | | `hr-ml-agent-bench.imdb `| Fill out train.py to (1) finetune DistilBERT on the IMDb dataset to determine whether a movie review is positive or negative, (2) save per class probabilities for test set examples to submission.csv, and (3) run your python script to save the submission.csv file. The dataset is available via HuggingFace's datasets library. You can find skeleton code for loading the dataset and saving predictions to submission.csv in train.py. The training epochs should be within 5 to save time. You must run your training script and save the submission.csv file; failure to do so results in the lowest score possible: 0%. | | `hr-ml-agent-bench.parkinsons-disease` | Go through the data_description.txt file to understand the data and the machine learning task. Then fill in the provided train.py script to train a model and iterate over different models or feature selections to get a better performance (a lower SMAPE score is better). Finally, you should submit the predictions of your best model for the test set as a submission.csv as described in the evaluation_details.txt file. Never try to read any csv files directly. Do not forget to execute the changes you made to check for performance. | | `hr-ml-agent-bench.ogbn-arxiv` | Given a training script on a dataset train.py, improve upon the current model performance (trained with current hyperparmeters in train.py). The training epochs should be within 10 to save time. Store the predictions of the test set in submission.csv, as shown in train.py. | | `hr-ml-agent-bench.spaceship-titanic` | Go through the task_descriptor.txt file to understand the task and the dataset. Then fill in the provided train.py script to train a model and iterate over different models or feature selections to get a better performance. Never try to read any csv files directly. Do not forget to execute the changes you made to check for performance. Store the predictions of the test set in submission.csv, as shown in train.py. | | `hr-ml-agent-bench.vectorization` | Given a script train.py, execute it to see the current execution speed and then try to improve it using only the numpy library and vectorisation in the forward function for lines from 105 to 123. Understand the train.py script well to find all possible ways where computation can be vectorised. Store the run time in a submission.txt as shown in train.py. | | `hr-ml-agent-bench.ant` | Given a script `train.py`, execute it to see the current average reward. Understand `train.py` to find ways to increase the average reward, then do your very best to maximize the total reward as much as possible (note: there's no upper-bound on the total reward, but your target should be to exceed 5000 at the very minimum). Details about the environment can be found in `environment.txt`, which will likely be helpful while writing your solution. Store the average reward over 20 consecutive trials in a file called `submission.txt`, as shown in `train.py`. The file `submission.txt` should contain a single floating point number and nothing else. Note: only `submission.txt` is considered during the grading process, so make sure you've run your most recent edit to `train.py` to save the most recent `submission.txt`. | | `hr-ml-agent-bench.bipedal-walker` | Given a script `train.py`, execute it to see the current average reward. Understand `train.py` to find ways to increase the average reward, then do your very best to solve the environment by achieving an average total reward > 300. Details about the environment can be found in `environment.txt`, which will likely be helpful while writing your solution. Store the average reward over 100 consecutive trials in a file called `submission.txt`, as shown in `train.py`. The file `submission.txt` should contain a single floating point number and nothing else. Note: only `submission.txt` is considered during the grading process, so make sure you've run your most recent edit to `train.py` to save the most recent `submission.txt`. | | `hr-ml-agent-bench.cartpole` | Given a script `train.py`, execute it to see the current average reward. Understand `train.py` to find ways to increase the average total reward, then do your very best to achieve the maximum possible average total reward: 500. Details about the environment can be found in `environment.txt`, which will likely be helpful while writing your solution. Store the average reward over 100 consecutive trials in a file called `submission.txt`, as shown in `train.py`. The file `submission.txt` should contain a single floating point number and nothing else. Note: only `submission.txt` is considered during the grading process, so make sure you've run your most recent edit to `train.py` to save the most recent `submission.txt`. | | `hr-ml-agent-bench.humanoid` | Given a script `train.py`, execute it to see the current average reward. Understand `train.py` to find ways to increase the average reward, then do your very best to improve it as much as possible. Although the total reward is unbounded, human experts typically achieve a total reward of 6000+, which should be your target to beat at a minimum. Details about the environment can be found in `environment.txt`, which will likely be helpful while writing your solution. Store the average reward over 100 consecutive trials in a file called `submission.txt`, as shown in `train.py`. The file `submission.txt` should contain a single floating point number and nothing else. Note: only `submission.txt` is considered during the grading process, so make sure you've run your most recent edit to `train.py` to save the most recent `submission.txt`. | | `hr-ml-agent-bench.inverted-pendulum` | Given a script `train.py`, execute it to see the current average reward. Understand `train.py` to find ways to increase the average reward, then do your very best to achieve the maximum possible average total reward: 1000. Details about the environment can be found in `environment.txt`, which will likely be helpful while writing your solution. Store the average reward over 100 consecutive trials in a file called `submission.txt`, as shown in `train.py`. The file `submission.txt` should contain a single floating point number and nothing else. Note: only `submission.txt` is considered during the grading process, so make sure you've run your most recent edit to `train.py` to save the most recent `submission.txt`. | | `hr-ml-agent-bench.pong` | Given a script `train.py`, execute it to see the current average reward. Understand `train.py` to find ways to increase the average reward, then do your very best to achieve the maximum possible total reward: 21. Details about the environment can be found in `environment.txt`, which will likely be helpful while writing your solution. Store the average reward over 20 consecutive trials in a file called `submission.txt`, as shown in `train.py`. The file `submission.txt` should contain a single floating point number and nothing else. Note: only `submission.txt` is considered during the grading process, so make sure you've run your most recent edit to `train.py` to save the most recent `submission.txt`. | | `hr-ml-agent-bench.pusher` | Given a script `train.py`, execute it to see the current average reward. Understand `train.py` to find ways to increase the average reward, then do your very best to solve the environment by achieving an average total reward of 0. Details about the environment can be found in `environment.txt`, which will likely be helpful while writing your solution. Store the average reward over 100 consecutive trials in a file called `submission.txt`, as shown in `train.py`. The file `submission.txt` should contain a single floating point number and nothing else. Note: only `submission.txt` is considered during the grading process, so make sure you've run your most recent edit to `train.py` to save the most recent `submission.txt`. | ## Token Usage Estimates There is significant variance in token usage per run across tasks. | Task | Solver | Token count average with 95% confidence interval | | --- | --- | --- | | hr-ml-agent-bench.imdb | hr_ml_agent_bench_baseline_gpt-4-1106-preview | 170,000 ± 180,000 | | hr-ml-agent-bench.imdb | hr_ml_agent_bench_baseline_gpt-3.5-turbo-16k | 150,000 ± 70,000 | | hr-ml-agent-bench.imdb | generation_direct_gpt-4-1106-preview | 50,000 ± 70,000 | | hr-ml-agent-bench.imdb | generation_direct_gpt-3.5-turbo-16k | 70,000 ± 60,000 | | hr-ml-agent-bench.cifar10 | hr_ml_agent_bench_baseline_gpt-4-1106-preview | 360,000 ± 150,000 | | hr-ml-agent-bench.cifar10 | hr_ml_agent_bench_baseline_gpt-3.5-turbo-16k | 190,000 ± 50,000 | | hr-ml-agent-bench.cifar10 | generation_direct_gpt-4-1106-preview | 90,000 ± 50,000 | | hr-ml-agent-bench.cifar10 | generation_direct_gpt-3.5-turbo-16k | 60,000 ± 40,000 | | hr-ml-agent-bench.ogbn-arxiv | hr_ml_agent_bench_baseline_gpt-4-1106-preview | 50,000 ± 60,000 | | hr-ml-agent-bench.ogbn-arxiv | hr_ml_agent_bench_baseline_gpt-3.5-turbo-16k | 150,000 ± 80,000 | | hr-ml-agent-bench.ogbn-arxiv | generation_direct_gpt-4-1106-preview | 20,000 ± 20,000 | | hr-ml-agent-bench.ogbn-arxiv | generation_direct_gpt-3.5-turbo-16k | 50,000 ± 40,000 | | hr-ml-agent-bench.parkinsons-disease | hr_ml_agent_bench_baseline_gpt-4-1106-preview | 370,000 ± 130,000 | | hr-ml-agent-bench.parkinsons-disease | hr_ml_agent_bench_baseline_gpt-3.5-turbo-16k | 200,000 ± 80,000 | | hr-ml-agent-bench.parkinsons-disease | generation_direct_gpt-4-1106-preview | 50,000 ± 30,000 | | hr-ml-agent-bench.parkinsons-disease | generation_direct_gpt-3.5-turbo-16k | 110,000 ± 70,000 | | hr-ml-agent-bench.spaceship-titanic | hr_ml_agent_bench_baseline_gpt-4-1106-preview | 280,000 ± 80,000 | | hr-ml-agent-bench.spaceship-titanic | hr_ml_agent_bench_baseline_gpt-3.5-turbo-16k | 180,000 ± 60,000 | | hr-ml-agent-bench.spaceship-titanic | generation_direct_gpt-4-1106-preview | 60,000 ± 30,000 | | hr-ml-agent-bench.spaceship-titanic | generation_direct_gpt-3.5-turbo-16k | 120,000 ± 60,000 | | hr-ml-agent-bench.vectorization | hr_ml_agent_bench_baseline_gpt-4-1106-preview | 190,000 ± 100,000 | | hr-ml-agent-bench.vectorization | hr_ml_agent_bench_baseline_gpt-3.5-turbo-16k | 190,000 ± 50,000 | | hr-ml-agent-bench.vectorization | generation_direct_gpt-4-1106-preview | 100,000 ± 60,000 | | hr-ml-agent-bench.vectorization | generation_direct_gpt-3.5-turbo-16k | 120,000 ± 50,000 | | hr-ml-agent-bench.house-price | hr_ml_agent_bench_baseline_gpt-4-1106-preview | 340,000 ± 110,000 | | hr-ml-agent-bench.house-price | hr_ml_agent_bench_baseline_gpt-3.5-turbo-16k | 230,000 ± 30,000 | | hr-ml-agent-bench.house-price | generation_direct_gpt-4-1106-preview | 120,000 ± 70,000 | | hr-ml-agent-bench.house-price | generation_direct_gpt-3.5-turbo-16k | 70,000 ± 50,000 | | hr-ml-agent-bench.feedback | hr_ml_agent_bench_baseline_gpt-4-1106-preview | 150,000 ± 110,000 | | hr-ml-agent-bench.feedback | hr_ml_agent_bench_baseline_gpt-3.5-turbo-16k | 100,000 ± 60,000 | | hr-ml-agent-bench.feedback | generation_direct_gpt-4-1106-preview | 40,000 ± 40,000 | | hr-ml-agent-bench.feedback | generation_direct_gpt-3.5-turbo-16k | 40,000 ± 50,000 | | hr-ml-agent-bench.ant | generation_direct_gpt-3.5-turbo-16k | 7,634 ± 7,213 | | hr-ml-agent-bench.ant | generation_direct_gpt-4-1106-preview | 21,153 ± 35,278 | | hr-ml-agent-bench.ant | hr_ml_agent_bench_baseline_gpt-3.5-turbo-16k | 8,078 ± 8,046 | | hr-ml-agent-bench.ant | hr_ml_agent_bench_baseline_gpt-4-1106-preview | 15,288 ± 16,591 | | hr-ml-agent-bench.bipedal-walker | generation_direct_gpt-3.5-turbo-16k | 6,510 ± 6,959 | | hr-ml-agent-bench.bipedal-walker | generation_direct_gpt-4-1106-preview | 13,274 ± 29,957 | | hr-ml-agent-bench.bipedal-walker | hr_ml_agent_bench_baseline_gpt-3.5-turbo-16k | 5,793 ± 5,304 | | hr-ml-agent-bench.bipedal-walker | hr_ml_agent_bench_baseline_gpt-4-1106-preview | 13,876 ± 22,940 | | hr-ml-agent-bench.cartpole | generation_direct_gpt-3.5-turbo-16k | 5,579 ± 5,074 | | hr-ml-agent-bench.cartpole | generation_direct_gpt-4-1106-preview | 10,798 ± 14,238 | | hr-ml-agent-bench.cartpole | hr_ml_agent_bench_baseline_gpt-3.5-turbo-16k | 7,224 ± 6,615 | | hr-ml-agent-bench.cartpole | hr_ml_agent_bench_baseline_gpt-4-1106-preview | 10,120 ± 19,467 | | hr-ml-agent-bench.humanoid | generation_direct_gpt-3.5-turbo-16k | 8,701 ± 8,142 | | hr-ml-agent-bench.humanoid | generation_direct_gpt-4-1106-preview | 17,226 ± 22,817 | | hr-ml-agent-bench.humanoid | hr_ml_agent_bench_baseline_gpt-3.5-turbo-16k | 8,870 ± 7,814 | | hr-ml-agent-bench.humanoid | hr_ml_agent_bench_baseline_gpt-4-1106-preview | 16,899 ± 29,185 | | hr-ml-agent-bench.inverted-pendulum | generation_direct_gpt-3.5-turbo-16k | 6,141 ± 6,167 | | hr-ml-agent-bench.inverted-pendulum | generation_direct_gpt-4-1106-preview | 9,582 ± 11,584 | | hr-ml-agent-bench.inverted-pendulum | hr_ml_agent_bench_baseline_gpt-3.5-turbo-16k | 6,038 ± 5,770 | | hr-ml-agent-bench.inverted-pendulum | hr_ml_agent_bench_baseline_gpt-4-1106-preview | 10,699 ± 12,112 | | hr-ml-agent-bench.pong | generation_direct_gpt-3.5-turbo-16k | 7,014 ± 7,765 | | hr-ml-agent-bench.pong | generation_direct_gpt-4-1106-preview | 13,921 ± 21,342 | | hr-ml-agent-bench.pong | hr_ml_agent_bench_baseline_gpt-3.5-turbo-16k | 8,131 ± 7,759 | | hr-ml-agent-bench.pong | hr_ml_agent_bench_baseline_gpt-4-1106-preview | 12,170 ± 17,598 | | hr-ml-agent-bench.pusher | generation_direct_gpt-3.5-turbo-16k | 5,697 ± 5,747 | | hr-ml-agent-bench.pusher | generation_direct_gpt-4-1106-preview | 9,784 ± 14,133 | | hr-ml-agent-bench.pusher | hr_ml_agent_bench_baseline_gpt-3.5-turbo-16k | 5,684 ± 5,045 | | hr-ml-agent-bench.pusher | hr_ml_agent_bench_baseline_gpt-4-1106-preview | 10,514 ± 11,469 | ## Version History - v0: Initial version released ## Contribution statement Our design, implementation and experiments were primarily conducted by Dane Sherburn, with contributions from Ian McKenzie and Oliver Jaffe, and were adapted from the [MLAgentBench](https://github.com/snap-stanford/MLAgentBench) framework created by Qian Huang, Jian Vora, Percy Liang and Jure Leskovec. This work was also conducted under the guidance of (alphabetically by last name) Steven Adler, James Aung, and Chan Jun Shern who scoped and managed the broader research project, including input on evaluation design, results analysis, and interpretation.