evals Documentation

File: README.md Copy Chapter

OpenAI Evals

You can now configure and run Evals directly in the OpenAI Dashboard. Get started →

Evals provide a framework for evaluating large language models (LLMs) or systems built using LLMs. We offer an existing registry of evals to test different dimensions of OpenAI models and the ability to write your own custom evals for use cases you care about. You can also use your data to build private evals which represent the common LLMs patterns in your workflow without exposing any of that data publicly.

If you are building with LLMs, creating high quality evals is one of the most impactful things you can do. Without evals, it can be very difficult and time intensive to understand how different model versions might affect your use case. In the words of OpenAI's President Greg Brockman:

Setup Copy Chapter

To run evals, you will need to set up and specify your OpenAI API key. After you obtain an API key, specify it using the OPENAI_API_KEY environment variable. Please be aware of the costs associated with using the API when running evals. You can also run and create evals using Weights & Biases.

Minimum Required Version: Python 3.9

Downloading evals Copy Section

Our evals registry is stored using Git-LFS. Once you have downloaded and installed LFS, you can fetch the evals (from within your local copy of the evals repo) with:

cd evals
git lfs fetch --all
git lfs pull

This will populate all the pointer files under evals/registry/data.

You may just want to fetch data for a select eval. You can achieve this via:

git lfs fetch --include=evals/registry/data/${your eval}
git lfs pull

Making evals Copy Section

If you are going to be creating evals, we suggest cloning this repo directly from GitHub and installing the requirements using the following command:

pip install -e .

Using -e, changes you make to your eval will be reflected immediately without having to reinstall.

Optionally, you can install the formatters for pre-committing with:

pip install -e .[formatters]

Then run pre-commit install to install pre-commit into your git hooks. pre-commit will now run on every commit.

If you want to manually run all pre-commit hooks on a repository, run pre-commit run --all-files. To run individual hooks use pre-commit run <hook_id>.

Running evals Copy Chapter

If you don't want to contribute new evals, but simply want to run them locally, you can install the evals package via pip:

pip install evals

You can find the full instructions to run existing evals in run-evals.md and our existing eval templates in eval-templates.md. For more advanced use cases like prompt chains or tool-using agents, you can use our Completion Function Protocol.

We provide the option for you to log your eval results to a Snowflake database, if you have one or wish to set one up. For this option, you will further have to specify the SNOWFLAKE_ACCOUNT, SNOWFLAKE_DATABASE, SNOWFLAKE_USERNAME, and SNOWFLAKE_PASSWORD environment variables.

Writing evals Copy Chapter

We suggest getting starting by:

• Walking through the process for building an eval: build-eval.md
• Exploring an example of implementing custom eval logic: custom-eval.md
• Writing your own completion functions: completion-fns.md
• Review our starter guide for writing evals: Getting Started with OpenAI Evals

Please note that we are currently not accepting evals with custom code! While we ask you to not submit such evals at the moment, you can still submit model-graded evals with custom model-graded YAML files.

If you think you have an interesting eval, please open a pull request with your contribution. OpenAI staff actively review these evals when considering improvements to upcoming models.

FAQ Copy Chapter

Do you have any examples of how to build an eval from start to finish?

• Yes! These are in the examples folder. We recommend that you also read through build-eval.md in order to gain a deeper understanding of what is happening in these examples.

Do you have any examples of evals implemented in multiple different ways?

• Yes! In particular, see evals/registry/evals/coqa.yaml. We have implemented small subsets of the CoQA dataset for various eval templates to help illustrate the differences.

When I run an eval, it sometimes hangs at the very end (after the final report). What's going on?

• This is a known issue, but you should be able to interrupt it safely and the eval should finish immediately after.

There's a lot of code, and I just want to spin up a quick eval. Help? OR,

I am a world-class prompt engineer. I choose not to code. How can I contribute my wisdom?

• If you follow an existing eval template to build a basic or model-graded eval, you don't need to write any evaluation code at all! Just provide your data in JSON format and specify your eval parameters in YAML. build-eval.md walks you through these steps, and you can supplement these instructions with the Jupyter notebooks in the examples folder to help you get started quickly. Keep in mind, though, that a good eval will inevitably require careful thought and rigorous experimentation!

Disclaimer Copy Chapter

By contributing to evals, you are agreeing to make your evaluation logic and data under the same MIT license as this repository. You must have adequate rights to upload any data used in an eval. OpenAI reserves the right to use this data in future service improvements to our product. Contributions to OpenAI evals will be subject to our usual Usage Policies: https://platform.openai.com/docs/usage-policies.

File: docs/build-eval.md Copy Chapter

Building an eval

Important: Please note that we are currently not accepting Evals with custom code! While we ask you to not submit such evals at the moment, you can still submit modelgraded evals with custom modelgraded YAML files.

This document walks through the end-to-end process for building an eval, which is a dataset and a choice of eval class. The examples folder contains Jupyter notebooks that follow the steps below to build several academic evals, thus helping to illustrate the overall process.

The steps in this process are building your dataset, registering a new eval with your dataset, and running your eval. Crucially, we assume that you are using an existing eval template out of the box (if that's not the case, see this example of building a custom eval). If you are interested in contributing your eval publicly, we also include some criteria at the bottom for what we think makes an interesting eval.

We are looking for evals in the following categories:

• Over-refusals
• Safety
• System message steerability
• In-the-wild hallucinations
• Math / logical / physical reasoning
• Real-world use case (please describe in your PR how this capability would be used in a product)
• Other foundational capability

If you have an eval that falls outside this category but still is a diverse example, please contribute it!

Formatting your data Copy Chapter

Once you have an eval in mind that you wish to implement, you will need to convert your samples into the right JSON lines (JSONL) format. A JSONL file is just a JSON file with a unique JSON object per line.

You can use the openai CLI (available with OpenAI-Python) to transform data from some common file types into JSONL:

openai tools fine_tunes.prepare_data -f data[.csv, .json, .txt, .xlsx or .tsv]

We include some examples of JSONL eval files in registry/data/README.md

Each JSON object will represent one data point in your eval. The keys you need in the JSON object depend on the eval template. All templates expect an "input" key, which is the prompt, ideally specified in chat format (though strings are also supported). We recommend chat format even if you are evaluating non-chat models. If you are evaluating both chat and non-chat models, we handle the conversion between chat-formatted prompts and raw string prompts (see the conversion logic here).

For the basic evals Match, Includes, and FuzzyMatch, the other required key is "ideal", which is a string (or a list of strings) specifying the correct reference answer(s). For model-graded evals, the required keys vary based on the eval but is determined by the {key}s in the evaluation prompt that are not covered by the (optional) args.

We have implemented small subsets of the CoQA dataset for various eval templates to illustrate how the data should be formatted. See coqa/match.jsonl for an example of data that is suitable for the Match basic eval template and coqa/samples.jsonl for data that is suitable for fact and closedqa model-graded evals. Note that even though these two model-graded evals expect different keys, we can include the superset of keys in our data in order to support both evals.

If the dataset file is on your local machine, put the jsonl file in evals/registry/data/<eval_name>/samples.jsonl. If it is in Cloud Object Storage, we support path-style URLs for the major clouds (for your personal use only, we will not accept PRs with cloud URLs).

Registering the eval Copy Chapter

Register the eval by adding a file to evals/registry/evals/<eval_name>.yaml using the elsuite registry format. For example, for a Match eval, it would be:

<eval_name>:
  id: <eval_name>.dev.v0
  metrics: [accuracy]

<eval_name>.dev.v0:
  class: evals.elsuite.basic.match:Match
  args:
    samples_jsonl: <eval_name>/samples.jsonl

Upon running the eval, the data will be searched for in evals/registry/data. For example, if test_match/samples.jsonl is the provided filepath, the data is expected to be in evals/registry/data/test_match/samples.jsonl.

The naming convention for evals is in the form <eval_name>.<split>.<version>.

• <eval_name> is the eval name, used to group evals whose scores are comparable.
• <split> is the data split, used to further group evals that are under the same ``. E.g., "val", "test", or "dev" for testing.
• <version> is the version of the eval, which can be any descriptive text you'd like to use (though it's best if it does not contain .).

In general, running the same eval name against the same model should always give similar results so that others can reproduce it. Therefore, when you change your eval, you should bump the version.

Running the eval Copy Chapter

You can now run your eval on your data from the CLI with your choice of model or completion function:

oaieval gpt-3.5-turbo <eval_name>

Congratulations, you have built your eval! Keep iterating on it until you are confident in the results.

For model-graded evals: a step-by-step workflow Copy Chapter

We expect that the existing model-graded evals such as fact, closedqa, and battle will fit many use cases. However, other use cases may benefit from more customization, e.g., a different evaluation prompt. For these, there will be a bit more work involved, but generally still no coding required!

1. If you can't use an existing model-graded eval, create a new YAML or create a new entry to an existing YAML in evals/registry/modelgraded to specify the parameters of your eval. See humor.yaml for an example.
   • Note that, even if you are creating a new YAML, you may find it easiest to copy an existing YAML as a starting point. For example, model-graded evals which check a model completion against a rubric can copy closedqa.yaml and just edit the args.
2. Next, you will create your dataset and register your eval, as described above. See joke_fruits_labeled.jsonl and joke-fruits, for example.
   • Note that it is recommended to specify eval_type at this step, when you register your eval, rather than step 1.
3. Run your eval, e.g., oaieval gpt-3.5-turbo joke-fruits.
4. (Recommended) Add a meta-eval for the model-graded eval! Each model-graded eval comes with a few knobs to tune, mainly prompt but also eval_type. In order to make sure the eval is of high quality, we recommend each model-graded eval contribution come with "choice labels", which are basically human-provided labels for which evaluation choice the model should have made. As an example (pretending that these jokes are actually funny), see the "choice" keys in joke_fruits_labeled.jsonl, which are not used by the joke-fruits eval but are used by the joke-fruits-meta meta-eval right below it . After running the meta-eval, e.g., oaieval gpt-3.5-turbo joke-fruits-meta, the report will output metascore/ accuracies, which should be close to "1.0" for a good model-graded eval.

Criteria for contributing an eval Copy Chapter

Important: if you are contributing code, make sure to run pip install pre-commit; pre-commit install before committing and pushing to ensure that black, isort, and autoflake are run.

We are interested in curating a diverse and interesting set of evals on which to improve our models going forward. Here are some criteria for what we consider a good eval:

• The eval should be thematically consistent. We'd like to see a number of prompts all revolving around the same use case, subject domain, failure mode, etc.
• The eval should be challenging. If GPT-4 or GPT-3.5-Turbo do well on all of the prompts, this is not as interesting. Of course, the eval should also be possible given the models' limitations and constraints. Oftentimes, a good rule of thumb is whether a human (potentially a subject expert) could do well on the prompts.
• The eval should be directionally clear. The data should include good signal around what is the right behavior. This means, for example, high-quality reference answers or an exhaustive rubric for evaluating answers.
• The eval should be carefully crafted. Before you submit, you should think through whether you have engineered your prompts for good performance, whether you are using the best eval template, whether you have spot checked your results to ensure accuracy, etc.

Once you are ready to contribute your eval publicly, submit a PR and the OpenAI team will be happy to look it over. Make sure to fill out all parts of the template that is prepopulated into the PR message. Note that submitting a PR does not guarantee that OpenAI will eventually merge it. We will run our own checks and use our best judgment when considering which evals to follow up with.

File: docs/completion-fn-protocol.md Copy Chapter

The Completion Function Protocol Copy Section

Here are the interfaces needed to implement the completion function protocol. Any implementation of this interface can be used inside oaieval.

Reference implementations:

• OpenAICompletionFn
• LangChainLLMCompletionFn

CompletionFn

Completion functions should implement the CompletionFn interface:

class CompletionFn(Protocol):
    def __call__(
        self,
        prompt: Union[str, list[dict[str, str]]],
        **kwargs,
    ) -> CompletionResult:

We take a prompt representing a single sample from an eval. These prompts can be represented as either a text string or a list of messages in OpenAI Chat format. To work with the existing evals, Completion Function implementations would need to handle both types of inputs, but we provide helper functionality to convert Chat formatted messages into a text string if that is the preferred input for your program:

from evals.prompt.base import CompletionPrompt

# chat_prompt: list[dict[str, str]] -> text_prompt: str
text_prompt = CompletionPrompt(chat_prompt).to_formatted_prompt()

CompletionResult

The completion function should return an object implementing the CompletionResult interface:

class CompletionResult(ABC):
    @abstractmethod
    def get_completions(self) -> list[str]:
        pass

The get_completions method returns a list of string completions. Each element should be considered a unique completion (in most cases this will be a list of length 1).

Using your CompletionFn

This is all that's needed to implement a Completion function that works with our existing Evals, allowing you to more easily evaluate your end-to-end logic on tasks.

See completion-fns.md to see how to register and use your completion function with oaieval.

File: docs/completion-fns.md Copy Chapter

Completion Functions

What are completion functions Copy Chapter

In run-evals.md, we learned how to make calls to oaieval to run an eval against a completion function. Completion Functions are generalizations of model completions, where a "completion" is some text output that would be our answer to the prompt. For example, if "Who played the girl elf in the hobbit?" is our prompt, the correct completion is "Evangeline Lilly". While we can just test a model directly to see if it generates "Evangeline Lilly", we can imagine doing numerous other operations under the hood to improve our ability to answer this question, like giving the model access to a browser to look up the answer before responding. Making it easy to implement this kind of under-the-hood operators before responding is the motivation behind building Completion Functions.

How to implement completion functions Copy Chapter

A completion function needs to implement some interfaces that make it usable within Evals. At its core, it is just standardizing inputs to be a text string or Chat conversation, and the output to be a list of text strings. Implementing this interface will allow you to run your Completion Function against any eval in Evals.

The exact interfaces needed are described in detail in completion-fn-protocol.md

We include some example implementations inside evals/completion_fns. For example, the LangChainLLMCompletionFn implements a way to generate completions from LangChain LLMs. We can then use these completion functions with oaieval:

oaieval langchain/llm/flan-t5-xl test-match

Registering Completion Functions Copy Chapter

Once you have written a completion function, we need to make the class visible to the oaieval CLI. Similar to how we register our evals, we also register Completion Functions inside evals/registry/completion_fns as yaml files. Here is the registration for our langchain LLM completion function:

langchain/llm/flan-t5-xl:
  class: evals.completion_fns.langchain_llm:LangChainLLMCompletionFn
  args:
    llm: HuggingFaceHub
    llm_kwargs:
      repo_id: google/flan-t5-xl

Here is how it breaks down
langchain/llm/flan-t5-xl: This is the top level key that will be used to access this completion function with oaieval.
class: This is the path to your implementation of the completion function protocol. This class needs to be importable within your python environment.
args: These are arguments that are passed to your completion function when it is instantiated.

Developing Completion Functions outside of Evals Copy Section

It is possible to register CompletionFunctions without directly modifying the registry or code inside Evals by using the --registry_path argument. As an example, let's say I want to use MyCompletionFn located inside ~/my_project/:

my_project
├── my_completion_fn.py
└── completion_fns
    └── my_completion_fn.yaml

If my_project is importable within the python environment (accessible via PYTHONPATH), we can structure my_completion_fn.yaml as:

my_completion_fn:
  class: my_project.my_completion_fn:MyCompletionFn

Then, we can make calls to oaieval using:

oaieval my_completion_fn test-match --registry_path ~/my_project

File: docs/custom-eval.md Copy Chapter

How to add a custom eval

Important: Please note that we are currently not accepting Evals with custom code! While we ask you to not submit such evals at the moment, you can still submit modelgraded evals with custom modelgraded YAML files.

This tutorial will walk you through a simple example of writing and adding a custom eval. The example eval will test the model's ability to do basic arithmetic. We will assume that you have followed the setup instructions in the README and gone through the other docs for how to run and build evals.

When writing your own evals, the primary files of interest are:

• evals/api.py, which provides common interfaces and utilities used by eval creators to sample from models and process the results,
• evals/record.py, which defines the recorder classes which log eval results in different ways, such as to a local JSON file or to a remote Snowflake database, and
• evals/metrics.py, which defines various common metrics of interest.

These files provide a suite of tools for writing new evals. Once you have gone through this tutorial, you can see a more realistic example of these tools in action with the machine translation eval example, which also implements custom eval logic in lieu of using an existing template.

Create your datasets Copy Chapter

The first step is to create the datasets for your eval. Here, we will create toy train and test sets of just two examples each. The test examples are what we will evaluate the model on, and we'll include the train examples as few-shot examples in the prompt to the model.

We will use the new chat format described here. By default, we encourage all evals to be written using chat formatting if you want to evaluate our new models. Under the hood, we convert chat formatted data into raw strings for older non chat models.

To create the toy datasets, in your terminal, type:

echo -e '{"problem": "2+2=", "answer": "4"}\n{"problem": "4*4=", "answer": "16"}' > /tmp/train.jsonl
echo -e '{"problem": "48+2=", "answer": "50"}\n{"problem": "5*20=", "answer": "100"}' > /tmp/test.jsonl

Create an eval Copy Chapter

The next step is to write a Python class that represents the actual evaluation. This class uses your datasets to create prompts, which are passed to the model to generate completions. Evaluation classes generally will inherit from the evals.Eval base class (defined in evals/eval.py) and will override two methods: eval_sample and run.

Let's create a file called arithmetic.py under the evals/elsuite folder. We'll start by defining the eval class. Its __init__ method will take in the arguments we need (references to the train and test sets) along with other kwargs that will be handled by the base class. We'll also define the run method which takes in a recorder and returns the final metrics of interest.

import random
import textwrap

import evals
import evals.metrics

class Arithmetic(evals.Eval):
    def __init__(self, train_jsonl, test_jsonl, train_samples_per_prompt=2, **kwargs):
        super().__init__(**kwargs)
        self.train_jsonl = train_jsonl
        self.test_jsonl = test_jsonl
        self.train_samples_per_prompt = train_samples_per_prompt

    def run(self, recorder):
        """
        Called by the `oaieval` CLI to run the eval. The `eval_all_samples` method calls `eval_sample`.
        """
        self.train_samples = evals.get_jsonl(self.train_jsonl)
        test_samples = evals.get_jsonl(self.test_jsonl)
        self.eval_all_samples(recorder, test_samples)

        # Record overall metrics
        return {
            "accuracy": evals.metrics.get_accuracy(recorder.get_events("match")),
        }

Generally, most run methods will follow the same pattern shown here: loading the data, calling eval_all_samples, and aggregating the results (in this case, using the get_accuracy function in evals/metrics.py). eval_all_samples takes in both the recorder and the test_samples and, under the hood, will call the eval_sample method on each sample in test_samples. So let's write that eval_sample method now:

def eval_sample(self, test_sample, rng: random.Random):
        """
        Called by the `eval_all_samples` method to evaluate a single sample.

        ARGS
        ====
        `test_sample`: a line from the JSONL test file
        `rng`: should be used for any randomness that is needed during evaluation

        This method does the following:
        1. Generate a prompt that contains the task statement, a few examples, and the test question.
        2. Generate a completion from the model.
        3. Check if the generated answer is correct.
        """
        stuffing = rng.sample(self.train_samples, self.train_samples_per_prompt)

        prompt = [
            {"role": "system", "content": "Solve the following math problems"},
        ]

        for i, sample in enumerate(stuffing + [test_sample]):
            if i < len(stuffing):
                prompt += [
                    {"role": "system", "content": sample["problem"], "name": "example_user"},
                    {"role": "system", "content": sample["answer"], "name": "example_assistant"},
                ]
            else:
                prompt += [{"role": "user", "content": sample["problem"]}]


        result = self.completion_fn(prompt=prompt, temperature=0.0, max_tokens=1)
        sampled = result.get_completions()[0]

        evals.record_and_check_match(prompt=prompt, sampled=sampled, expected=test_sample["answer"])

You'll notice that eval_sample doesn't take the recorder as an argument. This is because eval_all_samples sets it to be the default recorder before calling eval_sample, and the recording utilities defined in evals/record.py use the default recorder. In this example, the eval_sample method passes off a lot of the heavy lifting to the evals.record_and_check_match utility function, which is defined in evals/api.py. This utility function queries the model, defined by self.model_spec, with the given prompt and checks to see if the result matches the expected answer (or one of them, if given a list). It then records these matches (or non matches) using the default recorder.

eval_sample methods may vary greatly based on your use case. If you are building custom evals, it is a good idea to be familiar with the functions available to you in evals/record.py, evals/metrics.py, and especially evals/api.py.

Register your eval Copy Chapter

The next step is to register your eval in the registry so that it can be run using the oaieval CLI.

Let's create a file called arithmetic.yaml under the evals/registry/evals folder and add an entry for our eval as follows:

# Define a base eval
arithmetic:
  # id specifies the eval that this eval is an alias for
  # in this case, arithmetic is an alias for arithmetic.dev.match-v1
  # When you run `oaieval davinci arithmetic`, you are actually running `oaieval davinci arithmetic.dev.match-v1`
  id: arithmetic.dev.match-v1
  # The metrics that this eval records
  # The first metric will be considered to be the primary metric
  metrics: [accuracy]
# Define the eval
arithmetic.dev.match-v1:
  # Specify the class name as a dotted path to the module and class
  class: evals.elsuite.arithmetic:Arithmetic
  # Specify the arguments as a dictionary of JSONL URIs
  # These arguments can be anything that you want to pass to the class constructor
  args:
    train_jsonl: /tmp/train.jsonl
    test_jsonl: /tmp/test.jsonl

The args field should match the arguments that your eval class __init__ method expects.

Run your eval Copy Chapter

The final step is to run your eval and view the results.

pip install .  # you can omit this if you used `pip install -e .` to install
oaieval gpt-3.5-turbo arithmetic

If you run with the gpt-3.5-turbo model, you should see an output similar to this (we have cleaned up the output here slightly for readability):

% oaieval gpt-3.5-turbo arithmetic
... [registry.py:147] Loading registry from .../evals/registry/evals
... [registry.py:147] Loading registry from .../.evals/evals
... [oaieval.py:139] Run started: <run_id>
... [eval.py:32] Evaluating 2 samples
... [eval.py:138] Running in threaded mode with 1 threads!
100%|██████████████████████████████████████████| 2/2 [00:00<00:00,  3.35it/s]
... [record.py:320] Final report: {'accuracy': 1.0}. Logged to /tmp/evallogs/<run_id>_gpt-3.5-turbo_arithmetic.jsonl
... [oaieval.py:170] Final report:
... [oaieval.py:172] accuracy: 1.0
... [record.py:309] Logged 6 rows of events to /tmp/evallogs/<run_id>_gpt-3.5-turbo_arithmetic.jsonl: insert_time=2.038ms

File: docs/eval-templates.md Copy Chapter

Existing templates for evals

In using Evals, we have discovered several "templates" that accommodate many different benchmarks. We have implemented these templates in evals/elsuite in order to simplify the development of new evals. We believe that, with these templates, many evals will not require any coding to implement! Instead, you'll pick one of the existing templates and simply specify the dataset and parameters.

Basic eval templates Copy Chapter

In cases where the desired model response has very little variation, such as answering multiple choice questions or simple questions with a straightforward answer, we have found the following templates to be useful.

For a model completion a and a reference list of correct answers B, the following evals implement:

• basic/match.py:Match: any([a.startswith(b) for b in B])
• basic/includes.py:Includes: any([(b in a) for b in B])
• basic/fuzzy_match.py:FuzzyMatch: any([(a in b or b in a) for b in B])

To compare a model completion a in JSON format to a reference list of correct answers B also formatted in JSON, use the following eval:

• basic/json_match.py:JsonMatch yields a match if a is identical to at least one answer from B. Two JSON objects are
  identical if they have the same set of keys and the values for each key are identical. Key order is not significant, and whitespace outside values is ignored. Invalid JSON never matches.

Which eval template you use will depend on your use case. It is always recommended that you inspect the completions from your model, as this will help you determine how and whether to tweak your prompt (or your reference answers) and pick your eval template. Academic benchmarks oftentimes fit the mold of these basic evals, and we have implemented several end-to-end examples of academic evals as Jupyter notebooks in the examples folder.

Sometimes, custom eval logic will better suit your needs. One example of this is the machine translation eval example, in which there is a unique and clearly defined metric that we wish to use in our eval. You should use your best judgment when deciding between custom eval logic, using a basic eval template, or using model-graded evals as described next.

The model-graded eval template Copy Chapter

In cases where the desired model response can contain significant variation, such as answering an open-ended question, we have found that using the model to grade itself is a viable strategy for automated evaluation. In general, the evaluation model and the model being evaluated don't have to be the same, though we will assume that they are here for ease of explanation.

modelgraded/classify.py:ModelBasedClassify implements the main logic behind our model-graded eval template. In short, we get the model's completion to the original prompt, wrap it in an evaluation prompt, and get the model's completion to the evaluation prompt, which we parse into our metrics of interest. Crucially, the evaluation prompt should prime the model to answer in such a way that is easily parsable, e.g., in multiple choice format or with a simple yes/no. We describe some example model-graded evals below, but first we specify the parameters for this eval template.

Parameters for model-graded evals Copy Section

Refer to the classify.py:ModelBasedClassify class to see how these parameters are used in the code.

• prompt: The evaluation prompt which should take in the model's completion to the original prompt, potentially along with some other information, and steer the model to provide an evaluation that is easily parsable. Portions denoted by curly braces (i.e., {key}) are filled in either from the data input_outputs or the additional args (see below).

• input_outputs: A mapping specifying which inputs to use to generate which completions. For many evals, there will only be a single input-completion pair, though there can be more, e.g., when comparing two completions against each other.

• choice_strings: The choices that we expect the model completion to contain given the evaluation prompt. For example, "ABCDE" or ["Yes", "No", "Unsure"]. Any other choices returned by the model are parsed into "__invalid__".

• choice_scores (optional): A mapping of each choice to its score, which is logged as a metric. For example, if a response of "Yes" (resp. "No") indicates that the model's original completion was good (resp. bad), we may assign this choice a score of 1 (resp. 0).

• eval_type (optional): How we expect the model to format its response to the evaluation prompt. Currently the supported options are:

  • "cot_classify" ("chain-of-thought then classify", i.e., reason then answer) expects that the parsable portion of the response (i.e., the portion containing the choice) will be at the end of the completion. We recommend this as the default as it typically provides the most accurate model-graded evaluations.
  • "classify_cot" (answer then reason) expects that the model response will contain the choice first.
  • "classify" expects that the model response will only contain the choice.

  There are two ways to specify eval_type. The recommended way is in the evals/registry/evals YAML file. If done this way, an instruction will automatically be appended to prompt to steer the model towards the expected format (see ANSWER_PROMPTS in the code). Alternatively, you may specify eval_type in the evals/registry/modelgraded YAML, but you will need to include an appropriate instruction directly in the prompt.

• output_template (optional): If specified, determines how the model's output (or outputs, if n > 1) will be formatted within the completion.

Example model-graded evals Copy Section

To instantiate model-graded evals, create a YAML file in evals/registry/modelgraded which specifies values for the arguments described above. We have provided a few examples, which illustrate the process for creating a model-graded eval, but which we also believe are general enough to be useful out of the box for many evals.

fact.yaml: a factual consistency eval which, given a completion a and reference answer b, returns:

• "A" if a $\subseteq$ b, i.e., the submitted answer is a subset of the expert answer and is fully consistent with it.
• "B" if a $\supseteq$ b, i.e., the submitted answer is a superset of the expert answer and is fully consistent with it.
• "C" if a $=$ b, i.e., the submitted answer contains all the same details as the expert answer.
• "D" if a $\neq$ b, i.e., there is a disagreement between the submitted answer and the expert answer.
• "E" if a $\approx$ b, i.e., the answers differ, but these differences don't matter from the perspective of factuality.

closedqa.yaml: a question answering eval, which, given a prompt containing a question and the necessary information to answer the question, checks whether the model's answer is:

• relevant, i.e., extracted from the information provided in the prompt,
• concise, i.e., did not contain unnecessary details or information, and
• correct, i.e., uses the extracted information to come to the right conclusion.

Note that this eval is implemented more generally as a "criteria-checking" eval, which specifies the evaluation prompt as checking a given criterion and feeding in the above desiderata one by one. We believe that many other evals can be implemented by specifying a "rubric" detailing the criteria of interest and following the same prompt and yes/no choices.

battle.yaml: a head-to-head eval which compares two model completions for two potentially different prompts. choice_scores is used here to log how often the first completion is judged to be better than the second.

We include additional examples which test more specific model capabilities (such as humor) and are thus less generalizable to other evals. However, these examples still serve to illustrate different ways to write evaluation prompts and set up model-graded evals. See this section for more detailed steps on building model-graded evals.

File: docs/run-evals.md Copy Chapter

How to run evals

We provide two command line interfaces (CLIs): oaieval for running a single eval and oaievalset for running a set of evals.

Running an eval Copy Chapter

When using the oaieval command, you will need to provide the completion function you wish to evaluate as well as the eval to run. E.g.,

oaieval gpt-3.5-turbo test-match

The valid eval names are specified in the YAML files under evals/registry/evals and their corresponding implementations can be found in evals/elsuite.

In this example, gpt-3.5-turbo is an OpenAI model that we dynamically instantiate as a completion function using OpenAIChatCompletionFn(model=gpt-3.5-turbo). Any implementation of the CompletionFn protocol can be run against oaieval. By default, we support calling oaieval with any model available on the OpenAI API or with CompletionFunctions available in evals/registry/completion_fns. We are always interested in adding more completion functions and we encourage you to implement your own to reflect specific use cases.

More details on CompletionFn found here: completion-fns.md

These CLIs can accept various flags to modify their default behavior. For example:

• If you wish to log to a Snowflake database (which you have already set up as described in the README), add --no-local-run.
• By default, logging locally or to Snowflake will write to tmp/evallogs, and you can change this by setting a different --record_path.

You can run oaieval --help to see a full list of CLI options.

Running an eval set Copy Chapter

oaievalset gpt-3.5-turbo test

Similarly, oaievalset also expects a model name and an eval set name, for which the valid options are specified in the YAML files under evals/registry/eval_sets.

By default we run with 10 threads, and each thread times out and restarts after 40 seconds. You can configure this, e.g.,

EVALS_THREADS=42 EVALS_THREAD_TIMEOUT=600 oaievalset gpt-3.5-turbo test

Running with more threads will make the eval faster, though keep in mind the costs and your rate limits. Running with a higher thread timeout may be necessary if you expect each sample to take a long time, e.g., the data contain long prompts that elicit long responses from the model.

If you have to stop your run or your run crashes, we've got you covered! oaievalset records the evals that finished in /tmp/oaievalset/{model}.{eval_set}.progress.txt. You can simply rerun the command to pick up where you left off. If you want to run the eval set starting from the beginning, delete this progress file.

Unfortunately, you can't resume a single eval from the middle. You'll have to restart from the beginning, so try to keep your individual evals quick to run.

Logging Copy Chapter

By default, oaieval records events into local JSONL logs which can be inspected using a text editor or analyzed programmatically. 3rd-party tools such as naimenz/logviz may be helpful to visualize the logs, though we don't provide support or guarantees for their use.

File: evals/elsuite/identifying_variables/README.md Copy Chapter

Identifying Variables

This eval tests how well models can determine what should be treated as the
independent, dependent, and control variables for an experiment that tests a
particular hypothesis, given some observational context.

Usage Copy Chapter

Run with:

oaieval <solver> identifying_variables

We have found that generation/cot/gpt-4-1106-preview works well on this eval. For more examples of tested solvers, see ./scripts/run_experiments.sh.

Evaluation Process Copy Chapter

The evaluation process is as follows for a given sample from our dataset:

1. The TASK_DESCRIPTION prompt is shown to the solver.
2. The sample is passed through a renderer that processes the samples and
   renders an observation of the interactions of variables, which is placed in
   the SAMPLE_MESSAGE prompt template.
3. The solver answers in the form: [@ANSWER valid_hyp: ; independent: <var>; dependent: <var>; control: <vars>]. The answer is parsed and evaluated by the eval. If the answer cannot be parsed, we mark this as a violation and the sample is treated as incorrect.

Prompts Copy Chapter

We refer readers to the ./prompts.py file for the
TASK_DESCRIPTION and SAMPLE_MESSAGE prompts used in the eval.

Metrics Copy Chapter

Variants Copy Chapter

We support variations on the eval along two dimensions, renderer and dataset:

oaieval <solver> identifying_variables.<renderer>.<dataset>

The eval defaults to identifying_variables.language-corrset.balanced-ctrl.

Dataset Copy Section

We provide 4 dataset variants:

Renderers Copy Section

We have 6 different renderers, implemented in ./renderers/.

The default renderer is language-corrset. Here is an example render from this type:

The following is a description of the observations made about a set of variables.

In general, there were cases where some variables changed in tandem with each other, while others did not.
For example, changes in x_5075 were observed to reflect changes in x_3314 and viceversa.
Changes in x_9549 were not observed to reflect any changes in previously mentioned variables.
Changes in x_1808 were not observed to reflect any changes in previously mentioned variables.
Likewise, changes in x_9726 were observed to reflect changes in x_1808 and viceversa.

Show Tree Copy Section

We provide an additional variant of the eval where the decision tree implementing
the reasoning for scoring a perfect score is shown to the model. This variant
can be run by passing the show_tree=True flag to eval, e.g.

oaieval <solver> identifying_variables --extra_eval_params show_tree=True

Custom Solvers Copy Chapter

We implement two custom programmatic solvers to serve as baselines.

1. identifying_variables/random: a solver that randomly selects whether the
   hypothesis is plausible with probability 0.5, and if so randomly samples the
   independent, dependent and control variables. We view this baseline as
   equivalent to randomly guessing.
2. identifying_variables/noctrl: this is a solver that always outputs an empty
   list for the variables to control, essentially eliminating any chance of
   false positives. This can provide stronger performance than the random
   baseline, since it avoids any penalization for returning incorrect variables,
   and can even achieve a perfect score on samples that indeed do not have any
   variables to control

We refer to ./solvers.py for the implementation of these
solvers.

Token Usage Estimates Copy Chapter

We estimated per-run token usage on the default dataset size (500 samples)
for the least and most token-intensive configurations for each model type
(respectively, direct models on identifying_variables.corrset with
show_tree=False; and CoT models on identifying_variables.language-tabular
with show_tree=True).

These estimates were taken using the balanced-hypotheses dataset but should
roughly apply to the -balanced-ctrl datasets. For -large datasets (5000
samples), multiply the above numbers by 10.

Future modifications Copy Chapter

• Revisit the definition of the headline ctrl_nDCG metric
• Devise additional auxiliary metrics to paint a more complete picture
• What if we show the decision trees described in natural language rather than
  pseudocode?
• How can we extend this eval to multi-variable dependencies?

Version History Copy Chapter

• v0: Initial version released

Contribution Statement Copy Chapter

Eval design, implementation, and results evaluation and writeup were primarily
conducted by Giulio Starace. James Aung was of enormous assistance in report
writing, and additionally provided general feedback and project management
throughout the eval. Oliver Jaffe and Jan Betley were responsible for code
reviews throughout the implementation process, along with fine-grained feedback
on the project in general. Additional guidance was provided by (alphabetically
by last-name) Steven Adler and Chan Jun Shern, who helped with brainstorming,
gave research input and report revisions.

Appendix Copy Chapter

Perfect output decision trees Copy Section

The following decision trees are used to determine the perfect output (aka "gold
label") for a given sample.

Figure A1: Decision tree for determining whether a given variable should be
controlled.

Figure A2: Decision tree for determining a hypothesis is valid and if so
what the independent and dependent variables are.

File: evals/elsuite/hr_ml_agent_bench/README.md Copy Chapter

Human-Relative MLAgentBench Eval

This eval measures a model's ability to solve diverse machine learning research tasks. The best-known human performance has been collated for each task, which is used to calculate a “human-relative” percentage for each task attempt; 0% is a naive baseline (e.g. “random guess”), 100% is obtaining the same performance-gain as the best-known human, and 200% is obtaining 2x the performance-gain of said human. Our thanks go to the authors of MLAgentBench on which this work was built.

This eval contains the following 15 tasks:

Setup Copy Chapter

⚠️ Warning: This eval allows language models to run arbitrary code on your machine. Please ensure that you only run these experiments in a properly sandboxed environment.

ℹ️ Multiple tasks require a GPU. We comfortably ran our experiments on a NC64as T4 v3 machine from Microsoft Azure with an attached 2TB SSD.

The list of dependencies needed to run this eval are found in requirements.txt, which can be installed by running:

pip install -r requirements.txt

Some tasks (optionally) require additional dependencies, which can be found in benchmarks/<taskid>/scripts/requirements.txt and likewise can be installed by running:

pip install -r benchmarks/<taskid>/scripts/requirements.txt

where <taskid> is the name of the task you wish to run (e.g. ant).

To install all dependencies for all tasks, run:

sh scripts/install_all_requirements.sh

Alternatively, a Dev Container, devcontainer.json, is provided for use with Visual Studio Code, which contains all necessary dependencies and is pre-configured to run the eval. This Dev Container requires the NVIDIA Container Runtime to be installed on the host machine.

Usage Copy Chapter

Run individual tasks with:

EVALS_SEQUENTIAL=1 oaieval <solver> hr-ml-agent-bench.<taskid>
# This eval doesn't currently support multi-threading.

Where <taskid> is one of the 15 tasks listed in the Tasks section below (e.g. hr-ml-agent-bench.vectorization).

We suggest using gpt-4-1106-preview or hr_ml_agent_bench/baseline/gpt-4-1106-preview as default choices for <solver>.

For example, to evaluate the hr_ml_agent_bench/baseline/gpt-4-1106-preview solver on the CIFAR-10 task, run:

EVALS_SEQUENTIAL=1 oaieval hr_ml_agent_bench/baseline/gpt-4-1106-preview hr-ml-agent-bench.cifar10

See evals/registry/solvers/hr-ml-agent-bench.yaml for a full list of recommended Solvers.

Alternatively, run all tasks with:

EVALS_SEQUENTIAL=1 oaievalset <solver> hr-ml-agent-bench

Evaluation Process Copy Chapter

Overall, each run is structured as follows:

1. Given the task description, the agent begins the task.
2. At each step, the agent selects an action from the list of tools (e.g. read a file, edit a file, execute script) or gives a final answer.
3. For a selected agent action:
   • If the action is valid, the eval executes the chosen action in the environment.
     • If the agent attempts to use the Stable Baselines module, the environment refuses:
     text Copy

     The action has been terminated because it used the Stable Baselines module, which is not allowed. Please write your implementation from scratch.

     • If the action exceeds the action time limit (e.g. by executing a resource-intensive Python script), the environment throws an error:
     text Copy

     Time's up! The action exceeded the maximum time limit and terminated early.

   • If the agent fails to choose a valid action, the environment asks for a new response:
   text Copy

   No valid action found! You must specify an action at each step in the following format:
   
   Action: the action to take, should be one of the names of the tools
   Action Input: the input to the action as a valid JSON string

4. The observation from the environment (e.g. the output of the script execution if the agent chose to execute a script) is returned to the agent, and the agent is prompted to select a further action.
5. Steps 2-4 repeat until the agent calls the “final_answer” tool, or until the maximum number of steps is reached (default max steps is 30).
6. When the attempt has ended, an evaluation script is run to assess how well the agent did.

All attempts are subject to a step, overall time and per-action time limit; the exact limits are task-specific, depending on how challening the task is. This prevents the model from indefinitely refining its solution, or from running scripts which take too long to execute.

Task State Copy Chapter

Metrics Copy Chapter

The below are the key metrics of this eval:

Tasks Copy Chapter

This eval currently contains 15 tasks.

Token Usage Estimates Copy Chapter

There is significant variance in token usage per run across tasks.

Version History Copy Chapter

• v0: Initial version released

Contribution statement Copy Chapter

Our design, implementation and experiments were primarily conducted by Dane Sherburn, with contributions from Ian McKenzie and Oliver Jaffe, and were adapted from the MLAgentBench framework created by Qian Huang, Jian Vora, Percy Liang and Jure Leskovec. This work was also conducted under the guidance of (alphabetically by last name) Steven Adler, James Aung, and Chan Jun Shern who scoped and managed the broader research project, including input on evaluation design, results analysis, and interpretation.