lm-evaluation-harness Documentation

Technical Documentation: EleutherAI/lm-evaluation-harness

ℹ️ Provenance: Multi-Tier Ingestion · CodeWiki Reference

Overview (`README.md`) Copy Section

Language Model Evaluation Harness

Latest News 📣 Copy Chapter

• [2026/09] Plugins: model backends, filters, metrics and aggregations can now be registered from your own package instead of forking — declare an lm_eval.* entry point for zero-config discovery, or point --plugins at a local module. See the Plugin Guide.
• [2025/12] CLI refactored with subcommands (run, ls, validate) and YAML config file support via --config. See the CLI Reference and Configuration Guide.
• [2025/12] Lighter install: Base package no longer includes transformers/torch. Install model backends separately: pip install lm_eval[hf], lm_eval[vllm], etc.
• [2025/07] Added think_end_token arg to hf (token/str), vllm and sglang (str) for stripping CoT reasoning traces from models that support it.
• [2025/03] Added support for steering HF models!
• [2025/02] Added SGLang support!
• [2024/09] We are prototyping allowing users of LM Evaluation Harness to create and evaluate on text+image multimodal input, text output tasks, and have just added the hf-multimodal and vllm-vlm model types and mmmu task as a prototype feature. We welcome users to try out this in-progress feature and stress-test it for themselves, and suggest they check out lmms-eval, a wonderful project originally forking off of the lm-evaluation-harness, for a broader range of multimodal tasks, models, and features.
• [2024/07] API model support has been updated and refactored, introducing support for batched and async requests, and making it significantly easier to customize and use for your own purposes. To run Llama 405B, we recommend using VLLM's OpenAI-compliant API to host the model, and use the local-completions model type to evaluate the model.
• [2024/07] New Open LLM Leaderboard tasks have been added ! You can find them under the leaderboard task group.

Announcement Copy Chapter

A new v0.4.0 release of lm-evaluation-harness is available !

New updates and features include:

• New Open LLM Leaderboard tasks have been added ! You can find them under the leaderboard task group.
• Internal refactoring
• Config-based task creation and configuration
• Easier import and sharing of externally-defined task config YAMLs
• Support for Jinja2 prompt design, easy modification of prompts + prompt imports from Promptsource
• More advanced configuration options, including output post-processing, answer extraction, and multiple LM generations per document, configurable fewshot settings, and more
• Speedups and new modeling libraries supported, including: faster data-parallel HF model usage, vLLM support, MPS support with HuggingFace, and more
• Logging and usability changes
• New tasks including CoT BIG-Bench-Hard, Belebele, user-defined task groupings, and more

Please see our updated documentation pages in docs/ for more details.

Development will be continuing on the main branch, and we encourage you to give us feedback on what features are desired and how to improve the library further, or ask questions, either in issues or PRs on GitHub, or in the EleutherAI discord!

Overview Copy Chapter

This project provides a unified framework to test generative language models on a large number of different evaluation tasks.

Features:

• Over 60 standard academic benchmarks for LLMs, with hundreds of subtasks and variants implemented.
• Support for models loaded via transformers (including quantization via GPTQModel and AutoGPTQ), GPT-NeoX, and Megatron-DeepSpeed, with a flexible tokenization-agnostic interface.
• Support for fast and memory-efficient inference with vLLM.
• Support for commercial APIs including OpenAI, and TextSynth.
• Support for evaluation on adapters (e.g. LoRA) supported in HuggingFace's PEFT library.
• Support for local models and benchmarks.
• Evaluation with publicly available prompts ensures reproducibility and comparability between papers.
• Easy support for custom prompts and evaluation metrics.

The Language Model Evaluation Harness is the backend for 🤗 Hugging Face's popular Open LLM Leaderboard, has been used in hundreds of papers, and is used internally by dozens of organizations including NVIDIA, Cohere, BigScience, BigCode, Nous Research, and Mosaic ML.

Install Copy Chapter

To install the lm-eval package from the github repository, run:

git clone --depth 1 https://github.com/EleutherAI/lm-evaluation-harness
cd lm-evaluation-harness
pip install -e .

Installing Model Backends Copy Section

The base installation provides the core evaluation framework. Model backends must be installed separately using optional extras:

For HuggingFace transformers models:

pip install "lm_eval[hf]"

For vLLM inference:

pip install "lm_eval[vllm]"

For API-based models (OpenAI, Anthropic, etc.):

pip install "lm_eval[api]"

Multiple backends can be installed together:

pip install "lm_eval[hf,vllm,api]"

A detailed table of all optional extras is available at the end of this document.

Basic Usage Copy Chapter

Documentation Copy Section

Use lm-eval -h to see available options, or lm-eval run -h for evaluation options.

List available tasks with:

lm-eval ls tasks

Hugging Face `transformers` Copy Section

[!Important]
To use the HuggingFace backend, first install: pip install "lm_eval[hf]"

To evaluate a model hosted on the HuggingFace Hub (e.g. GPT-J-6B) on hellaswag you can use the following command (this assumes you are using a CUDA-compatible GPU):

lm_eval --model hf \
    --model_args pretrained=EleutherAI/gpt-j-6B \
    --tasks hellaswag \
    --device cuda:0 \
    --batch_size 8

Additional arguments can be provided to the model constructor using the --model_args flag. Most notably, this supports the common practice of using the revisions feature on the Hub to store partially trained checkpoints, or to specify the datatype for running a model:

lm_eval --model hf \
    --model_args pretrained=EleutherAI/pythia-160m,revision=step100000,dtype="float" \
    --tasks lambada_openai,hellaswag \
    --device cuda:0 \
    --batch_size 8

Models that are loaded via both transformers.AutoModelForCausalLM (autoregressive, decoder-only GPT style models) and transformers.AutoModelForSeq2SeqLM (such as encoder-decoder models like T5) in Huggingface are supported.

Batch size selection can be automated by setting the --batch_size flag to auto. This will perform automatic detection of the largest batch size that will fit on your device. On tasks where there is a large difference between the longest and shortest example, it can be helpful to periodically recompute the largest batch size, to gain a further speedup. To do this, append :N to above flag to automatically recompute the largest batch size N times. For example, to recompute the batch size 4 times, the command would be:

lm_eval --model hf \
    --model_args pretrained=EleutherAI/pythia-160m,revision=step100000,dtype="float" \
    --tasks lambada_openai,hellaswag \
    --device cuda:0 \
    --batch_size auto:4

[!Note]
Just like you can provide a local path to transformers.AutoModel, you can also provide a local path to lm_eval via --model_args pretrained=/path/to/model

Evaluating GGUF Models

lm-eval supports evaluating models in GGUF format using the Hugging Face (hf) backend. This allows you to use quantized models compatible with transformers, AutoModel, and llama.cpp conversions.

To evaluate a GGUF model, pass the path to the directory containing the model weights, the gguf_file, and optionally a separate tokenizer path using the --model_args flag.

🚨 Important Note:
If no separate tokenizer is provided, Hugging Face will attempt to reconstruct the tokenizer from the GGUF file — this can take hours or even hang indefinitely. Passing a separate tokenizer avoids this issue and can reduce tokenizer loading time from hours to seconds.

✅ Recommended usage:

lm_eval --model hf \
    --model_args pretrained=/path/to/gguf_folder,gguf_file=model-name.gguf,tokenizer=/path/to/tokenizer \
    --tasks hellaswag \
    --device cuda:0 \
    --batch_size 8

[!Tip]
Ensure the tokenizer path points to a valid Hugging Face tokenizer directory (e.g., containing tokenizer_config.json, vocab.json, etc.).

Evaluating GGUF Models Served by llama.cpp

The gguf model type evaluates models hosted by a llama.cpp server (llama-server) through its OpenAI-compatible /v1/completions endpoint:

lm_eval --model gguf \
    --model_args base_url=http://127.0.0.1:8080 \
    --tasks hellaswag

• Requires a llama.cpp release from December 2024 or newer, which returns logprobs in the modern OpenAI format (logprobs.content, see llama.cpp#10783). The deprecated legacy format (token_logprobs) is not supported.
• If the server runs in router mode (multiple models), pass the model name or alias: --model_args base_url=http://127.0.0.1:8080,model=my-model-alias.
• Requests are issued concurrently. The default degree of parallelism is auto-detected from the server's slot count (/props → total_slots, i.e. llama-server's --parallel setting); override it with parallel=<N>. loglikelihood requests are additionally pinned to slots via llama.cpp's id_slot parameter, with consecutive requests sharing a context (e.g. the candidate continuations of one multiple-choice question) assigned to the same slot so they reuse its cached prompt prefix. generate_until requests are left unpinned, since the server's dynamic idle-slot assignment load-balances variable-length generations better.
• loglikelihood is implemented via exact teacher forcing: llama.cpp ignores echo and never returns prompt logprobs, so the continuation is tokenized with the server's /tokenize endpoint (context and continuation encoded separately, matching the HF backend), and each continuation token is scored with its natural prefix as a token-id prompt plus a logit_bias that forces the server to sample it. llama.cpp reports the forced token's pre-sampling logprob and an unbiased top_logprobs, yielding loglikelihoods that match the HF backend to within quantization noise. (A naive alternative — forcing the continuation with a GBNF grammar — does not work: grammar-constrained decoding can pick a fragmented tokenization whose logprobs are not the natural-tokenization loglikelihoods, which systematically deflates scores.)
• loglikelihood_rolling (perplexity tasks such as wikitext) is not implemented for this model type.

Multi-GPU Evaluation with Hugging Face `accelerate`

We support three main ways of using Hugging Face's accelerate 🚀 library for multi-GPU evaluation.

To perform data-parallel evaluation (where each GPU loads a separate full copy of the model), we leverage the accelerate launcher as follows:

accelerate launch -m lm_eval --model hf \
    --tasks lambada_openai,arc_easy \
    --batch_size 16

(or via accelerate launch --no-python lm_eval).

For cases where your model can fit on a single GPU, this allows you to evaluate on K GPUs K times faster than on one.

WARNING: This setup does not work with FSDP model sharding, so in accelerate config FSDP must be disabled, or the NO_SHARD FSDP option must be used.

The second way of using accelerate for multi-GPU evaluation is when your model is too large to fit on a single GPU.

In this setting, run the library outside the accelerate launcher, but passing parallelize=True to --model_args as follows:

lm_eval --model hf \
    --tasks lambada_openai,arc_easy \
    --model_args parallelize=True \
    --batch_size 16

This means that your model's weights will be split across all available GPUs.

For more advanced users or even larger models, we allow for the following arguments when parallelize=True as well:

• device_map_option: How to split model weights across available GPUs. defaults to "auto".
• max_memory_per_gpu: the max GPU memory to use per GPU in loading the model.
• max_cpu_memory: the max amount of CPU memory to use when offloading the model weights to RAM.
• offload_folder: a folder where model weights will be offloaded to disk if needed.

The third option is to use both at the same time. This will allow you to take advantage of both data parallelism and model sharding, and is especially useful for models that are too large to fit on a single GPU.

accelerate launch --multi_gpu --num_processes {nb_of_copies_of_your_model} \
    -m lm_eval --model hf \
    --tasks lambada_openai,arc_easy \
    --model_args parallelize=True \
    --batch_size 16

To learn more about model parallelism and how to use it with the accelerate library, see the accelerate documentation

Warning: We do not natively support multi-node evaluation using the hf model type! Please reference our GPT-NeoX library integration for an example of code in which a custom multi-machine evaluation script is written.

Note: we do not currently support multi-node evaluations natively, and advise using either an externally hosted server to run inference requests against, or creating a custom integration with your distributed framework as is done for the GPT-NeoX library.

Tensor Parallelism (native PyTorch)

For models that support PyTorch's native Tensor Parallelism (via DTensor), you can shard model weights across GPUs without accelerate's device-map by passing tp_plan=auto in --model_args. Launch with torchrun or accelerate launch:

torchrun --nproc-per-node=4 -m lm_eval \
    --model hf \
    --model_args pretrained=google/gemma-4-31B-it,tp_plan=auto \
    --tasks lambada_openai,arc_easy \
    --batch_size 16

Constraints:

• tp_plan and parallelize=True are mutually exclusive — use one or the other.
• The number of key-value heads in the model must be divisible by --nproc-per-node (the TP degree).
• Requires PyTorch >= 2.4 and a transformers version that exposes a TP plan for the model (v4.47+).

Steered Hugging Face `transformers` models Copy Section

To evaluate a Hugging Face transformers model with steering vectors applied, specify the model type as steered and provide the path to either a PyTorch file containing pre-defined steering vectors, or a CSV file that specifies how to derive steering vectors from pretrained sparsify or sae_lens models (you will need to install the corresponding optional dependency for this method).

Specify pre-defined steering vectors:

import torch

steer_config = {
    "layers.3": {
        "steering_vector": torch.randn(1, 768),
        "bias": torch.randn(1, 768),
        "steering_coefficient": 1,
        "action": "add"
    },
}
torch.save(steer_config, "steer_config.pt")

Specify derived steering vectors:

import pandas as pd

pd.DataFrame({
    "loader": ["sparsify"],
    "action": ["add"],
    "sparse_model": ["EleutherAI/sae-pythia-70m-32k"],
    "hookpoint": ["layers.3"],
    "feature_index": [30],
    "steering_coefficient": [10.0],
}).to_csv("steer_config.csv", index=False)

Run the evaluation harness with steering vectors applied:

lm_eval --model steered \
    --model_args pretrained=EleutherAI/pythia-160m,steer_path=steer_config.pt \
    --tasks lambada_openai,hellaswag \
    --device cuda:0 \
    --batch_size 8

NVIDIA `nemo` models Copy Section

NVIDIA NeMo Framework is a generative AI framework built for researchers and pytorch developers working on language models.

To evaluate a nemo model, start by installing NeMo following the documentation. We highly recommended to use the NVIDIA PyTorch or NeMo container, especially if having issues installing Apex or any other dependencies (see latest released containers). Please also install the lm evaluation harness library following the instructions in the Install section.

NeMo models can be obtained through NVIDIA NGC Catalog or in NVIDIA's Hugging Face page. In NVIDIA NeMo Framework there are conversion scripts to convert the hf checkpoints of popular models like llama, falcon, mixtral or mpt to nemo.

Run a nemo model on one GPU:

lm_eval --model nemo_lm \
    --model_args path=<path_to_nemo_model> \
    --tasks hellaswag \
    --batch_size 32

It is recommended to unpack the nemo model to avoid the unpacking inside the docker container - it may overflow disk space. For that you can run:

mkdir MY_MODEL
tar -xvf MY_MODEL.nemo -c MY_MODEL

Multi-GPU evaluation with NVIDIA `nemo` models

By default, only one GPU is used. But we do support either data replication or tensor/pipeline parallelism during evaluation, on one node.

1. To enable data replication, set the model_args of devices to the number of data replicas to run. For example, the command to run 8 data replicas over 8 GPUs is:

torchrun --nproc-per-node=8 --no-python lm_eval \
    --model nemo_lm \
    --model_args path=<path_to_nemo_model>,devices=8 \
    --tasks hellaswag \
    --batch_size 32

1. To enable tensor and/or pipeline parallelism, set the model_args of tensor_model_parallel_size and/or pipeline_model_parallel_size. In addition, you also have to set up devices to be equal to the product of tensor_model_parallel_size and/or pipeline_model_parallel_size. For example, the command to use one node of 4 GPUs with tensor parallelism of 2 and pipeline parallelism of 2 is:

torchrun --nproc-per-node=4 --no-python lm_eval \
    --model nemo_lm \
    --model_args path=<path_to_nemo_model>,devices=4,tensor_model_parallel_size=2,pipeline_model_parallel_size=2 \
    --tasks hellaswag \
    --batch_size 32

Note that it is recommended to substitute the python command by torchrun --nproc-per-node=<number of devices> --no-python to facilitate loading the model into the GPUs. This is especially important for large checkpoints loaded into multiple GPUs.

Not supported yet: multi-node evaluation and combinations of data replication with tensor or pipeline parallelism.

Megatron-LM models Copy Section

Megatron-LM is NVIDIA's large-scale transformer training framework. This backend allows direct evaluation of Megatron-LM checkpoints without conversion.

Requirements:

• Megatron-LM must be installed or accessible via MEGATRON_PATH environment variable
• PyTorch with CUDA support

Setup:

# Set environment variable pointing to Megatron-LM installation
export MEGATRON_PATH=/path/to/Megatron-LM

Basic usage (single GPU):

lm_eval --model megatron_lm \
    --model_args load=/path/to/checkpoint,tokenizer_type=HuggingFaceTokenizer,tokenizer_model=/path/to/tokenizer \
    --tasks hellaswag \
    --batch_size 1

Supported checkpoint formats:

• Standard Megatron checkpoints (model_optim_rng.pt)
• Distributed checkpoints (.distcp format, auto-detected)

Parallelism Modes

The Megatron-LM backend supports the following parallelism modes:

[!Note]

• Pipeline Parallelism (PP > 1) is not currently supported.
• Expert Parallelism (EP) cannot be combined with Tensor Parallelism (TP).

Data Parallelism (4 GPUs, each with full model replica):

torchrun --nproc-per-node=4 -m lm_eval --model megatron_lm \
    --model_args load=/path/to/checkpoint,tokenizer_model=/path/to/tokenizer,devices=4 \
    --tasks hellaswag

Tensor Parallelism (TP=2):

torchrun --nproc-per-node=2 -m lm_eval --model megatron_lm \
    --model_args load=/path/to/checkpoint,tokenizer_model=/path/to/tokenizer,devices=2,tensor_model_parallel_size=2 \
    --tasks hellaswag

Expert Parallelism for MoE models (EP=4):

torchrun --nproc-per-node=4 -m lm_eval --model megatron_lm \
    --model_args load=/path/to/moe_checkpoint,tokenizer_model=/path/to/tokenizer,devices=4,expert_model_parallel_size=4 \
    --tasks hellaswag

Using extra_args for additional Megatron options:

lm_eval --model megatron_lm \
    --model_args load=/path/to/checkpoint,tokenizer_model=/path/to/tokenizer,extra_args="--no-rope-fusion --trust-remote-code" \
    --tasks hellaswag

[!Note]
The --use-checkpoint-args flag is enabled by default, which loads model architecture parameters from the checkpoint. For checkpoints converted via Megatron-Bridge, this typically includes all necessary model configuration.

Multi-GPU evaluation with OpenVINO models

Pipeline parallelism during evaluation is supported with OpenVINO models

To enable pipeline parallelism, set the model_args of pipeline_parallel. In addition, you also have to set up device to value HETERO:<GPU index1>,<GPU index2> for example HETERO:GPU.1,GPU.0 For example, the command to use pipeline parallelism of 2 is:

lm_eval --model openvino \
    --tasks wikitext \
    --model_args pretrained=<path_to_ov_model>,pipeline_parallel=True \
    --device HETERO:GPU.1,GPU.0

Tensor + Data Parallel and Optimized Inference with `vLLM` Copy Section

We also support vLLM for faster inference on supported model types, especially faster when splitting a model across multiple GPUs. For single-GPU or multi-GPU — tensor parallel, data parallel, or a combination of both — inference, for example:

lm_eval --model vllm \
    --model_args pretrained={model_name},tensor_parallel_size={GPUs_per_model},dtype=auto,gpu_memory_utilization=0.8,data_parallel_size={model_replicas} \
    --tasks lambada_openai \
    --batch_size auto

To use vllm, do pip install "lm_eval[vllm]". For a full list of supported vLLM configurations, please reference our vLLM integration and the vLLM documentation.

[!Note]
data_parallel_size>1 dispatches each replica as a separate ray actor and requires pip install ray. Each actor reserves tensor_parallel_size GPUs (default 1).

vLLM occasionally differs in output from Huggingface. We treat Huggingface as the reference implementation and provide a script for checking the validity of vllm results against HF.

[!Tip]
For fastest performance, we recommend using --batch_size auto for vLLM whenever possible, to leverage its continuous batching functionality!

[!Tip]
Passing max_model_len=4096 or some other reasonable default to vLLM through model args may cause speedups or prevent out-of-memory errors when trying to use auto batch size, such as for Mistral-7B-v0.1 which defaults to a maximum length of 32k.

Tensor + Data Parallel and Fast Offline Batching Inference with `SGLang` Copy Section

We support SGLang for efficient offline batch inference. Its Fast Backend Runtime delivers high performance through optimized memory management and parallel processing techniques. Key features include tensor parallelism, continuous batching, and support for various quantization methods (FP8/INT4/AWQ/GPTQ).

To use SGLang as the evaluation backend, please install it in advance via SGLang documents here.

[!Tip]
Due to the installing method of Flashinfer-- a fast attention kernel library, we don't include the dependencies of SGLang within pyproject.toml. Note that the Flashinfer also has some requirements on torch version.

SGLang's server arguments are slightly different from other backends, see here for more information. We provide an example of the usage here:

lm_eval --model sglang \
    --model_args pretrained={model_name},dp_size={data_parallel_size},tp_size={tensor_parallel_size},dtype=auto \
    --tasks gsm8k_cot \
    --batch_size auto

[!Tip]
When encountering out-of-memory (OOM) errors (especially for multiple-choice tasks), try these solutions:

1. Use a manual batch_size, rather than auto.
2. Lower KV cache pool memory usage by adjusting mem_fraction_static - Add to your model arguments for example --model_args pretrained=...,mem_fraction_static=0.7.
3. Increase tensor parallel size tp_size (if using multiple GPUs).

ONNX Runtime GenAI Copy Section

We support ONNX Runtime GenAI for cross-platform evaluation of ONNX LLMs produced by the ONNX Runtime GenAI Model Builder. Unlike the Windows ML backend, this backend runs on Linux, macOS, and Windows and selects execution providers with the cross-platform og.Config API (CPU, CUDA, DirectML, WebGPU, and AMD NPU via VitisAI/RyzenAI).

Install the backend along with the execution-provider wheel that matches your hardware:

# CPU
pip install "lm_eval[onnxruntime-genai]"
# CUDA / DirectML instead of the CPU wheel (mutually exclusive):
#   pip install onnxruntime-genai-cuda
#   pip install onnxruntime-genai-directml

Evaluate a Model Builder ONNX model on a chosen execution provider:

lm_eval --model onnxruntime-genai \
    --model_args pretrained=/path/to/model_builder_output,execution_provider=cuda \
    --tasks hellaswag \
    --batch_size 1

The pretrained path is a Model Builder output directory (containing genai_config.json, the ONNX graph(s), and an HF tokenizer) or a .onnx file inside one. execution_provider defaults to cpu (which runs with no provider appended); pass cuda, dml, VitisAI, etc. for accelerators. Extra provider settings can be passed via provider_options.

[!Note]
Supported architectures are whatever the Model Builder supports (Llama, Phi, Qwen, Gemma, Mistral, Granite, ChatGLM, …). Inference is batch-size 1 and runs one execution provider per run.

ONNX Runtime Copy Section

We also support running the same Model Builder export through a raw onnxruntime.InferenceSession instead of the GenAI loop. Use this backend when you want scores to come from the exact runtime a deployment uses, or when you need an execution provider that onnxruntime-genai does not build — notably ROCm and MIGraphX for AMD GPUs.

# CPU
pip install "lm_eval[onnxruntime]"
# CUDA / ROCm instead of the CPU wheel (mutually exclusive):
#   pip install onnxruntime-gpu
#   pip install onnxruntime-rocm

lm_eval --model onnxruntime \
    --model_args pretrained=/path/to/model_builder_output,execution_provider=rocm \
    --tasks hellaswag,arc_easy,wikitext \
    --batch_size 1

execution_provider accepts short aliases (cpu, cuda, rocm, migraphx, dml, openvino, tensorrt, vitisai, webgpu, qnn) as well as full ONNX Runtime provider names. As with the onnxruntime-genai backend, leaving it unset honors the providers declared in the export's genai_config.json, so an export built for a specific device runs there without extra flags. Non-CPU providers keep CPUExecutionProvider as a per-node fallback, since Model Builder graphs use contrib ops that not every provider implements.

Because both ONNX backends share one scoring implementation and end up in the same ORT kernels, they agree on the same model and provider; tests/models/test_onnxruntime_parity.py enforces that as a standing check.

[!Note]
This backend currently covers loglikelihood, multiple_choice, and loglikelihood_rolling (hellaswag, arc, mmlu, wikitext, …). For generative tasks such as gsm8k, use --model onnxruntime-genai with the same model directory.

The installed onnxruntime must be new enough to load the contrib-op schemas your Model Builder version emitted. An older runtime can reject the graph outright — for example a 12-input GroupQueryAttention fails to load on ONNX Runtime 1.22.

Windows ML Copy Section

We support Windows ML for hardware-accelerated inference on Windows platforms. This enables evaluation on CPU, GPU, and NPU (Neural Processing Unit) devices.

Windows ML?
https://learn.microsoft.com/en-us/windows/ai/new-windows-ml/overview

To use Windows ML, install the required dependencies:

pip install wasdk-Microsoft.Windows.AI.MachineLearning[all] wasdk-Microsoft.Windows.ApplicationModel.DynamicDependency.Bootstrap onnxruntime-windowsml onnxruntime-genai-winml

Evaluate an ONNX Runtime GenAI LLM on NPU/GPU/CPU on Windows:

lm_eval --model winml \
    --model_args pretrained=/path/to/onnx/model \
    --tasks mmlu \
    --batch_size 1

[!Note]
The Windows ML backend is ONLY for ONNX Runtime GenAI model format. Models targeting transformers.js won't work. You can verify this by finding the genai_config.json file in the model folder.

[!Note]
To run an ONNX Runtime GenAI model on the target device, you MUST convert the original model to that vendor and device type. Converted models won't work / work well on other vendor or device types. To learn more on model conversion, please visit Microsoft AI Tool Kit

Model APIs and Inference Servers Copy Section

[!Important]
To use API-based models, first install: pip install "lm_eval[api]"

Our library also supports the evaluation of models served via several commercial APIs, and we hope to implement support for the most commonly used performant local/self-hosted inference servers.

To call a hosted model, use:

export OPENAI_API_KEY=YOUR_KEY_HERE
lm_eval --model openai-completions \
    --model_args model=davinci-002 \
    --tasks lambada_openai,hellaswag

We also support using your own local inference server with servers that mirror the OpenAI Completions and ChatCompletions APIs.

lm_eval --model local-completions --tasks gsm8k --model_args model=facebook/opt-125m,base_url=http://{yourip}:8000/v1/completions,num_concurrent=1,max_retries=3,tokenized_requests=False,batch_size=16

Note that for externally hosted models, configs such as --device which relate to where to place a local model should not be used and do not function. Just like you can use --model_args to pass arbitrary arguments to the model constructor for local models, you can use it to pass arbitrary arguments to the model API for hosted models. See the documentation of the hosting service for information on what arguments they support.

Models which do not supply logits or logprobs can be used with tasks of type generate_until only, while local models, or APIs that supply logprobs/logits of their prompts, can be run on all task types: generate_until, loglikelihood, loglikelihood_rolling, and multiple_choice.

For more information on the different task output_types and model request types, see our documentation.

[!Note]
For best performance with closed chat model APIs such as Anthropic Claude 3 and GPT-4, we recommend carefully looking at a few sample outputs using --limit 10 first to confirm answer extraction and scoring on generative tasks is performing as expected. providing system="<some system prompt here>" within --model_args for anthropic-chat-completions, to instruct the model what format to respond in, may be useful.

Other Frameworks Copy Section

A number of other libraries contain scripts for calling the eval harness through their library. These include GPT-NeoX, Megatron-DeepSpeed, and mesh-transformer-jax.

To create your own custom integration you can follow instructions from this tutorial.

Additional Features Copy Section

[!Note]
For tasks unsuitable for direct evaluation — either due risks associated with executing untrusted code or complexities in the evaluation process — the --predict_only flag is available to obtain decoded generations for post-hoc evaluation.

If you have a Metal compatible Mac, you can run the eval harness using the MPS back-end by replacing --device cuda:0 with --device mps (requires PyTorch version 2.1 or higher). Note that the PyTorch MPS backend is still in early stages of development, so correctness issues or unsupported operations may exist. If you observe oddities in model performance on the MPS back-end, we recommend first checking that a forward pass of your model on --device cpu and --device mps match.

[!Note]
You can inspect what the LM inputs look like by running the following command:

bash Copy

python write_out.py \
    --tasks <task1,task2,...> \
    --num_fewshot 5 \
    --num_examples 10 \
    --output_base_path /path/to/output/folder

This will write out one text file for each task.

To verify the data integrity of the tasks you're performing in addition to running the tasks themselves, you can use the --check_integrity flag:

lm_eval --model openai \
    --model_args engine=davinci-002 \
    --tasks lambada_openai,hellaswag \
    --check_integrity

Advanced Usage Tips Copy Chapter

For models loaded with the HuggingFace transformers library, any arguments provided via --model_args get passed to the relevant constructor directly. This means that anything you can do with AutoModel can be done with our library. For example, you can pass a local path via pretrained= or use models finetuned with PEFT by taking the call you would run to evaluate the base model and add ,peft=PATH to the model_args argument:

lm_eval --model hf \
    --model_args pretrained=EleutherAI/gpt-j-6b,parallelize=True,load_in_4bit=True,peft=nomic-ai/gpt4all-j-lora \
    --tasks openbookqa,arc_easy,winogrande,hellaswag,arc_challenge,piqa,boolq \
    --device cuda:0

Models provided as delta weights can be easily loaded using the Hugging Face transformers library. Within --model_args, set the delta argument to specify the delta weights, and use the pretrained argument to designate the relative base model to which they will be applied:

lm_eval --model hf \
    --model_args pretrained=Ejafa/llama_7B,delta=lmsys/vicuna-7b-delta-v1.1 \
    --tasks hellaswag

GPTQ quantized models can be loaded using GPTQModel (faster) or AutoGPTQ

GPTQModel: add ,gptqmodel=True to model_args

lm_eval --model hf \
    --model_args pretrained=model-name-or-path,gptqmodel=True \
    --tasks hellaswag

AutoGPTQ: add ,autogptq=True to model_args:

lm_eval --model hf \
    --model_args pretrained=model-name-or-path,autogptq=model.safetensors,gptq_use_triton=True \
    --tasks hellaswag

We support wildcards in task names, for example you can run all of the machine-translated lambada tasks via --task lambada_openai_mt_*.

Saving & Caching Results Copy Chapter

To save evaluation results provide an --output_path. We also support logging model responses with the --log_samples flag for post-hoc analysis.

[!TIP]
Use --use_cache <DIR> to cache evaluation results and skip previously evaluated samples when resuming runs of the same (model, task) pairs. Note that caching is rank-dependent, so restart with the same GPU count if interrupted. You can also use --cache_requests to save dataset preprocessing steps for faster evaluation resumption.

To push results and samples to the Hugging Face Hub, first ensure an access token with write access is set in the HF_TOKEN environment variable. Then, use the --hf_hub_log_args flag to specify the organization, repository name, repository visibility, and whether to push results and samples to the Hub - example dataset on the HF Hub. For instance:

lm_eval --model hf \
    --model_args pretrained=model-name-or-path,autogptq=model.safetensors,gptq_use_triton=True \
    --tasks hellaswag \
    --log_samples \
    --output_path results \
    --hf_hub_log_args hub_results_org=EleutherAI,hub_repo_name=lm-eval-results,push_results_to_hub=True,push_samples_to_hub=True,public_repo=False \

This allows you to easily download the results and samples from the Hub, using:

from datasets import load_dataset

load_dataset("EleutherAI/lm-eval-results-private", "hellaswag", "latest")

For a full list of supported arguments, check out the interface guide in our documentation!

Visualizing Results Copy Chapter

You can seamlessly visualize and analyze the results of your evaluation harness runs using both Weights & Biases (W&B) and Zeno.

Zeno Copy Section

You can use Zeno to visualize the results of your eval harness runs.

First, head to hub.zenoml.com to create an account and get an API key on your account page.
Add this key as an environment variable:

export ZENO_API_KEY=[your api key]

You'll also need to install the lm_eval[zeno] package extra.

To visualize the results, run the eval harness with the log_samples and output_path flags.
We expect output_path to contain multiple folders that represent individual model names.
You can thus run your evaluation on any number of tasks and models and upload all of the results as projects on Zeno.

lm_eval \
    --model hf \
    --model_args pretrained=EleutherAI/gpt-j-6B \
    --tasks hellaswag \
    --device cuda:0 \
    --batch_size 8 \
    --log_samples \
    --output_path output/gpt-j-6B

Then, you can upload the resulting data using the zeno_visualize script:

python scripts/zeno_visualize.py \
    --data_path output \
    --project_name "Eleuther Project"

This will use all subfolders in data_path as different models and upload all tasks within these model folders to Zeno.
If you run the eval harness on multiple tasks, the project_name will be used as a prefix and one project will be created per task.

You can find an example of this workflow in examples/visualize-zeno.ipynb.

Weights and Biases Copy Section

With the Weights and Biases integration, you can now spend more time extracting deeper insights into your evaluation results. The integration is designed to streamline the process of logging and visualizing experiment results using the Weights & Biases (W&B) platform.

The integration provide functionalities

• to automatically log the evaluation results,
• log the samples as W&B Tables for easy visualization,
• log the results.json file as an artifact for version control,
• log the <task_name>_eval_samples.json file if the samples are logged,
• generate a comprehensive report for analysis and visualization with all the important metric,
• log task and cli specific configs,
• and more out of the box like the command used to run the evaluation, GPU/CPU counts, timestamp, etc.

First you'll need to install the lm_eval[wandb] package extra. Do pip install lm_eval[wandb].

Authenticate your machine with an your unique W&B token. Visit https://wandb.ai/authorize to get one. Do wandb login in your command line terminal.

Run eval harness as usual with a wandb_args flag. Use this flag to provide arguments for initializing a wandb run (wandb.init) as comma separated string arguments.

lm_eval \
    --model hf \
    --model_args pretrained=microsoft/phi-2,trust_remote_code=True \
    --tasks hellaswag,mmlu_abstract_algebra \
    --device cuda:0 \
    --batch_size 8 \
    --output_path output/phi-2 \
    --limit 10 \
    --wandb_args project=lm-eval-harness-integration \
    --log_samples

In the stdout, you will find the link to the W&B run page as well as link to the generated report. You can find an example of this workflow in examples/visualize-wandb.ipynb, and an example of how to integrate it beyond the CLI.

Contributing Copy Chapter

Check out our open issues and feel free to submit pull requests!

For more information on the library and how everything fits together, see our documentation pages.

To get started with development, first clone the repository and install the dev dependencies:

git clone https://github.com/EleutherAI/lm-evaluation-harness
cd lm-evaluation-harness
pip install -e ".[dev,hf]"

Implementing new tasks Copy Section

To implement a new task in the eval harness, see this guide.

In general, we follow this priority list for addressing concerns about prompting and other eval details:

1. If there is widespread agreement among people who train LLMs, use the agreed upon procedure.
2. If there is a clear and unambiguous official implementation, use that procedure.
3. If there is widespread agreement among people who evaluate LLMs, use the agreed upon procedure.
4. If there are multiple common implementations but not universal or widespread agreement, use our preferred option among the common implementations. As before, prioritize choosing from among the implementations found in LLM training papers.

These are guidelines and not rules, and can be overruled in special circumstances.

We try to prioritize agreement with the procedures used by other groups to decrease the harm when people inevitably compare runs across different papers despite our discouragement of the practice. Historically, we also prioritized the implementation from Language Models are Few Shot Learners as our original goal was specifically to compare results with that paper.

Support Copy Section

The best way to get support is to open an issue on this repo or join the EleutherAI Discord server. The #lm-thunderdome channel is dedicated to developing this project and the #release-discussion channel is for receiving support for our releases. If you've used the library and have had a positive (or negative) experience, we'd love to hear from you!

Optional Extras Copy Chapter

Extras dependencies can be installed via pip install -e ".[NAME]"

Model Backends Copy Section

These extras install dependencies required to run specific model backends:

Task Dependencies Copy Section

These extras install dependencies required for specific evaluation tasks:

Development & Utilities Copy Section

Cite as Copy Chapter

@misc{eval-harness,
  author       = {Gao, Leo and Tow, Jonathan and Abbasi, Baber and Biderman, Stella and Black, Sid and DiPofi, Anthony and Foster, Charles and Golding, Laurence and Hsu, Jeffrey and Le Noac'h, Alain and Li, Haonan and McDonell, Kyle and Muennighoff, Niklas and Ociepa, Chris and Phang, Jason and Reynolds, Laria and Schoelkopf, Hailey and Skowron, Aviya and Sutawika, Lintang and Tang, Eric and Thite, Anish and Wang, Ben and Wang, Kevin and Zou, Andy},
  title        = {The Language Model Evaluation Harness},
  month        = 07,
  year         = 2024,
  publisher    = {Zenodo},
  version      = {v0.4.3},
  doi          = {10.5281/zenodo.12608602},
  url          = {https://zenodo.org/records/12608602}
}

Overview (`docs/README.md`) Copy Section

Eval Harness Documentation

Welcome to the docs for the LM Evaluation Harness!

Table of Contents Copy Chapter

• To learn about the public interface of the library, as well as how to evaluate via the command line or as integrated into an external library, see the Interface.
• To learn how to add a new library, API, or model type to the library, as well as a quick explainer on the types of ways to evaluate an LM, see the Model Guide.
  • For an extended description of how to extend the library to new model classes served over an API, see the API Guide.
  • To ship a model, filter, metric or aggregation from your own package instead of the lm-eval source tree, see the Plugin Guide.
• For a crash course on adding new tasks to the library, see our New Task Guide.
• To learn more about pushing the limits of task configuration that the Eval Harness supports, see the Task Configuration Guide.

Overview (`docs/chat-template-readme.md`) Copy Section

Chat Template Delimiter Handling Update

Overview Copy Chapter

This change modifies how delimiters are handled when applying chat templates in the request construction process for likelihood and multiple-choice based tasks. When apply_chat_template is set to True, the target delimiter is now set to an empty string instead of using the configured delimiter.

Background Copy Chapter

By default, the system uses a target delimiter (typically a whitespace " ") between the context and target text when constructing prompts. The full string is constructed as:

doc_to_text(doc) + target_delimiter + doc_to_target(doc)

While this worked well for base models where we wanted the model to predict a single whitespace followed by the answer, chat models have their own formatting conventions that handle spacing differently.

The Change Copy Chapter

• When apply_chat_template=True, the target delimiter is now empty ("") instead of the default whitespace
• This prevents interference between chat template formatting and the default delimiter system
• Particularly important for multiple choice tasks where the template itself handles spacing

Example Copy Chapter

# Before (with default delimiter " ")
<user>Question: What color is the sky?\nAnswer:<assistant> blue

# After
<user>Question: What color is the sky?\nAnswer:<assistant>blue

Overview (`lm_eval/tasks/README.md`) Copy Section

Tasks

A list of supported tasks and task groupings can be viewed with lm-eval ls tasks.

For more information, including a full list of task names and their precise meanings or sources, follow the links
provided to the individual README.md files for each subfolder.

Multimodal Tasks Copy Chapter

Multimodal Tasks Copy Chapter

Overview (`scripts/clean_training_data/README.md`) Copy Section

Clean Training Data

janitor.py contains a script to remove benchmark data contamination from training data sets.
It uses the approach described in the GPT-3 paper.

Algorithm Copy Chapter

1. Collects all contamination text files that are to be removed from training data
2. Filters training data by finding Ngram matches between the training data
   and any contamination
   1. Ngrams ignore case and punctuation and are split on whitespace.
   2. Matching Ngram substrings are removed, as is a window_to_remove character window around
      the match, splitting the training data into chunks
   3. Any chunks less than minimum_slice_length are removed
   4. Training data sets split into more than too_dirty_cutoff are considered
      completely contaminated and removed

OpenAI used:

ngram_n = 13
window_to_remove = 200
minimum_slice_length = 200
too_dirty_cutoff = 10

Compiling Copy Chapter

Janitor can be used as a pure python program, but it is much faster if the ngram
code is run in C++. To compile the C++ code, run

pip install pybind11
c++ -O3 -Wall -shared -std=c++11 -fPIC $(python3 -m pybind11 --includes) janitor_util.cpp -o janitor_util$(python3-config --extension-suffix)

MacOS users: If your compiler isn't linked to Python, you may need to add to the above -undefined dynamic_lookup.
Linux users: If your compiler isn't linked to Python, you may need to follow these steps:

1. Rename the compiled code file to janitor_util.so.
2. Before running import Janitor in your code, add sys.path.append("your/relative/path/to/janitor_util.so") so that Python knows the location of janitor_util.so.

Overview (`templates/new_yaml_task/README.md`) Copy Section

Task-name

Paper Copy Chapter

Title: paper titles goes here

Abstract: link to paper PDF or arXiv abstract goes here

Short description of paper / benchmark goes here:

Homepage: homepage to the benchmark's website goes here, if applicable

Citation Copy Section

BibTeX-formatted citation goes here

Groups, Tags, and Tasks Copy Section

Groups

• group_name: Short description

Tags

• tag_name: Short description

Tasks

• task_name: 1-sentence description of what this particular task does
• task_name2: ...

Checklist Copy Section

For adding novel benchmarks/datasets to the library:

• Is the task an existing benchmark in the literature?
  • Have you referenced the original paper that introduced the task?
  • If yes, does the original paper provide a reference implementation? If so, have you checked against the reference implementation and documented how to run such a test?

If other tasks on this dataset are already supported:

• Is the "Main" variant of this task clearly denoted?
• Have you provided a short sentence in a README on what each new variant adds / evaluates?
• Have you noted which, if any, published evaluation setups are matched by this variant?

Changelog Copy Section

Overview (`lm_eval/tasks/aclue/README.md`) Copy Section

ACLUE

Paper Copy Section

Can Large Language Model Comprehend Ancient Chinese? A Preliminary Test on ACLUE
https://arxiv.org/abs/2310.09550

The Ancient Chinese Language Understanding Evaluation (ACLUE) is an evaluation benchmark focused on ancient Chinese language comprehension. It aims to assess the performance of large-scale language models on understanding ancient Chinese. The benchmark comprises 15 tasks spanning various domains, including lexical, syntactic, semantic, inference, and knowledge. ACLUE's tasks are derived from a combination of manually curated questions from publicly available resources, and automatically
generated questions from classical Chinese language corpora. The range of questions span from the Xia dynasty (2070 BCE) to the Ming dynasty (1368 CE). ACLUE adopts a multiple-choice question format for all tasks.

Homepage: https://github.com/isen-zhang/ACLUE

Citation Copy Section

@inproceedings{zhang-li-2023-large,
    title = "Can Large Language Model Comprehend {A}ncient {C}hinese? A Preliminary Test on {ACLUE}",
    author = "Zhang, Yixuan  and Li, Haonan",
    booktitle = "Proceedings of the Ancient Language Processing Workshop",
    month = sep,
    year = "2023",
    address = "Varna, Bulgaria",
    publisher = "INCOMA Ltd., Shoumen, Bulgaria",
    url = "https://aclanthology.org/2023.alp-1.9",
    pages = "80--87"
}

Groups, Tags, and Tasks Copy Section

Groups

• aclue: All 15 subjects of the ACLUE dataset, evaluated following the methodology in CMMLU's original implementation.

Tasks

The following tasks evaluate subjects in the ACLUE dataset using loglikelihood-based multiple-choice scoring:

• aclue_{subject_english}

Checklist Copy Section

• Is the task an existing benchmark in the literature?
  • Have you referenced the original paper that introduced the task?
  • If yes, does the original paper provide a reference implementation?
    • Yes, original implementation contributed by author of the benchmark

If other tasks on this dataset are already supported:

• Is the "Main" variant of this task clearly denoted?
• Have you provided a short sentence in a README on what each new variant adds / evaluates?
• Have you noted which, if any, published evaluation setups are matched by this variant?

Overview (`lm_eval/tasks/acpbench/README.md`) Copy Section

ACPBench

Homepage: https://ibm.github.io/ACPBench/

Papers Copy Section

Title: ACPBench: Reasoning About Action, Change, and Planning
Pdf: https://arxiv.org/pdf/2410.05669
Task: acp_bench
Abstract:

There is an increasing body of work using Large Language Models (LLMs) as agents for orchestrating workflows and making decisions in domains that require planning and multi-step reasoning. As a result, it is imperative to evaluate LMs on core skills required for planning. ACPBench is a benchmark for evaluating the reasoning tasks in the field of planning. The benchmark consists of 7 reasoning tasks over 13 planning domains. The collection is constructed from planning domains described in a formal language. This allows the synthesized problems to have provably correct solutions across many tasks and domains. Further, it allows the luxury to scale without additional human effort, i.e., many additional problems can be created automatically.

Title: ACPBench Hard: Unrestrained Reasoning about Action, Change, and Planning
Pdf: https://arxiv.org/abs/2503.24378
Task: acp_bench_hard
Abstract:

We introduce ACPBench Hard, a dataset of generative, open-ended questions which LLM models needs to answer in order to plan. Models that perform well on these tasks could in principle be integrated into a planner or be used directly as a policy. We discuss the complexity of these tasks as well as the complexity of validating the correctness of their answers and present validation algorithms for each task. Equipped with these validators, we test the performance of a variety of models on our tasks and find that for most of these tasks, the performance of even the largest models is still subpar. Our experiments show that no model outperforms any other in these tasks, and with a few exceptions, all tested language models score below 65%, indicating that even the current frontier language models as well as so-called reasoning models have a long way to go before they can reliably reason about planning.

The dataset is available on HuggingFace.

Citation Copy Section

@inproceedings{kokel2025acp
  author       = {Harsha Kokel and
                  Michael Katz and
                  Kavitha Srinivas and
                  Shirin Sohrabi},
  title        = {ACPBench: Reasoning about Action, Change, and Planning},
  booktitle    = {{AAAI}},
  publisher    = {{AAAI} Press},
  year         = {2025}
}

@misc{KokelKSS25ACPHard,
  title       = {ACPBench Hard: Unrestrained Reasoning about Action, Change, and Planning},
  author      = {Harsha Kokel and
                 Michael Katz and
                 Kavitha Srinivas and
                 Shirin Sohrabi},
  year        = {2025},
  eprint      = {2503.24378},
  archivePrefix = {arXiv},
  primaryClass  = {cs.AI},
  url         = {https://arxiv.org/abs/2503.24378},
}

Groups, Tags, and Tasks Copy Section

Groups

• None

Tags

• acp_bench : Evaluates acp_bool_cot_2shot and acp_mcq_cot_2shot (Main variant for ACPBench paper)
• acp_bool_cot_2shot : Evaluates acp_areach_bool, acp_app_bool, acp_just_bool, acp_land_bool, acp_prog_bool, acp_reach_bool, acp_val_bool with chain-of-thought and 2 shots
• acp_mcq_cot_2shot : Evaluates acp_areach_mcq, acp_app_mcq, acp_just_mcq, acp_land_mcq, acp_prog_mcq, acp_reach_mcq, acp_val_mcq with chain-of-thought and 2 shots
• acp_bench_hard : Evaluates acp_gen_2shot (Main variant for ACPBench Hard paper)
• acp_gen_2shot : Evaluates acp_areach_gen, acp_app_gen, acp_just_gen, acp_land_gen, acp_nexta_gen, acp_prog_gen, acp_reach_gen, acp_val_gen with 2 shots
• acp_bench_hard_with_pddl : Evaluates acp_gen_2shot_with_pddl
• acp_gen_2shot_with_pddl : Evaluates acp_areach_gen_with_pddl, acp_app_gen_with_pddl, acp_just_gen_with_pddl, acp_land_gen_with_pddl, acp_nexta_gen_with_pddl, acp_prog_gen_with_pddl, acp_reach_gen_with_pddl, acp_val_gen_with_pddl with 2 shots

Tasks

7 Boolean tasks

• acp_areach_bool
• acp_app_bool
• acp_just_bool
• acp_land_bool
• acp_prog_bool
• acp_reach_bool
• acp_val_bool

7 MCQ tasks

• acp_areach_mcq
• acp_app_mcq
• acp_just_mcq
• acp_land_mcq
• acp_prog_mcq
• acp_reach_mcq
• acp_val_mcq

8 Generative tasks (with just natural language description in context)

• acp_areach_gen
• acp_app_gen
• acp_just_gen
• acp_land_gen
• acp_nexta_gen
• acp_prog_gen
• acp_reach_gen
• acp_val_gen

and the same 8 generative tasks with natural language as well as the PDDL description of the domain and problem in context.

• acp_areach_gen_with_pddl
• acp_app_gen_with_pddl
• acp_just_gen_with_pddl
• acp_land_gen_with_pddl
• acp_nexta_gen_with_pddl
• acp_prog_gen_with_pddl
• acp_reach_gen_with_pddl
• acp_val_gen_with_pddl

! The evaluation scripts are taken from original github https://github.com/IBM/ACPBench

Checklist Copy Section

For adding novel benchmarks/datasets to the library:

• Is the task an existing benchmark in the literature?
  • Have you referenced the original paper that introduced the task?
  • If yes, does the original paper provide a reference implementation? If so, have you checked against the reference implementation and documented how to run such a test?

If other tasks on this dataset are already supported:

• Is the "Main" variant of this task clearly denoted?
• Have you provided a short sentence in a README on what each new variant adds / evaluates?
• Have you noted which, if any, published evaluation setups are matched by this variant?

Change Log Copy Section

• 03/17/2025 Initial Commit
• 05/13/2025 Adding ACPBench Hard tasks (with and without PDDL)

Overview (`lm_eval/tasks/aexams/README.md`) Copy Section

Arabic EXAMS

Paper Copy Section

EXAMS: a resource specialized in multilingual high school exam questions.
The original paper EXAMS

The Arabic EXAMS dataset includes five subjects

• Islamic studies
• Biology
• Physics
• Science
• Social

The original dataset EXAMS-QA

EXAMS is a benchmark dataset for cross-lingual and multilingual question answering for high school examinations.
With 24,000 high-quality high school exam questions in 16 languages, covering 8 language families and 24 school subjects from Natural Sciences and Social Sciences, among others.
EXAMS offers unique fine-grained evaluation framework across multiple languages and subjects

Homepage for Arabic EXAMS: EXAMS Arabic Homepage

Citation Copy Section

Groups, Tags, and Tasks Copy Section

Groups

• aexams: Arabic EXAMS dataset, including IslamicStudies, Biology, Science, Physics, Social subjects.

Tasks

The following tasks evaluate subjects in Arabic EXAMS dataset using loglikelihood-based multiple-choice scoring:

• aexams_IslamicStudies
• aexams_Biology
• aexams_Science
• aexams_Physics
• aexams_Social

Checklist Copy Section

• Is the task an existing benchmark in the literature?
  • Have you referenced the original paper that introduced the task?
  • If yes, does the original paper provide a reference implementation?
    • Yes, original implementation contributed by author of the benchmark

If other tasks on this dataset are already supported:

• Is the "Main" variant of this task clearly denoted?
• Have you provided a short sentence in a README on what each new variant adds / evaluates?
• Have you noted which, if any, published evaluation setups are matched by this variant?

Overview (`lm_eval/tasks/afrimgsm/README.md`) Copy Section

AfriMGSM

Paper Copy Section

IrokoBench: A New Benchmark for African Languages in the Age of Large Language Models
https://arxiv.org/pdf/2406.03368

IrokoBench is a human-translated benchmark dataset for 16 typologically diverse
low-resource African languages covering three tasks: natural language inference (AfriXNLI),
mathematical reasoning (AfriMGSM), and multi-choice knowledge-based QA (AfriMMLU).

Citation Copy Section

@misc{adelani2024irokobenchnewbenchmarkafrican,
      title={IrokoBench: A New Benchmark for African Languages in the Age of Large Language Models},
      author={David Ifeoluwa Adelani and Jessica Ojo and Israel Abebe Azime and Jian Yun Zhuang and Jesujoba O. Alabi and Xuanli He and Millicent Ochieng and Sara Hooker and Andiswa Bukula and En-Shiun Annie Lee and Chiamaka Chukwuneke and Happy Buzaaba and Blessing Sibanda and Godson Kalipe and Jonathan Mukiibi and Salomon Kabongo and Foutse Yuehgoh and Mmasibidi Setaka and Lolwethu Ndolela and Nkiruka Odu and Rooweither Mabuya and Shamsuddeen Hassan Muhammad and Salomey Osei and Sokhar Samb and Tadesse Kebede Guge and Pontus Stenetorp},
      year={2024},
      eprint={2406.03368},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2406.03368},
}

Groups and Tasks Copy Section

Every AfriMGSM task is evaluated with five different prompt templates (the
updated IrokoBench/AfroBench prompts), so each per-language task name carries a
_prompt_{i} suffix where i is the prompt id (1-5). The original
afrimgsm_direct_{language_code}, afrimgsm_en_cot_{language_code} and
afrimgsm_translate_{language_code} names without the suffix are no longer
registered.

Groups

• afrimgsm-irokobench: evaluates all direct tasks over all five prompts and reports a size-weighted mean accuracy
• afrimgsm_cot-irokobench: same as above with chain-of-thought prompting
• afrimgsm_tt-irokobench: same as above in the translate-test setting
• afrimgsm_tt_cot-irokobench: chain-of-thought in the translate-test setting

Tags

• afrimgsm_tasks: runs all direct tasks (all five prompts)
• afrimgsm_tasks_prompt_{i}: runs all direct tasks for prompt i (1-5)
• afrimgsm_cot_tasks: runs all chain-of-thought tasks (all five prompts)
• afrimgsm_cot_tasks_prompt_{i}: runs all chain-of-thought tasks for prompt i (1-5)
• afrimgsm_tt_tasks: runs all translate-test tasks
• afrimgsm_tt_cot_tasks: runs all chain-of-thought translate-test tasks

Tasks

• afrimgsm_{language_code}_prompt_{i}: evaluates one language with prompt i (1-5) on the curated dataset, e.g. afrimgsm_amh_prompt_1
• afrimgsm_cot_{language_code}_prompt_{i}: same as above with chain-of-thought prompting
• afrimgsm_translate_{language_code}_prompt_{i}: same as above in the translate-test setting
• afrimgsm_cot_translate_{language_code}_prompt_{i}: chain-of-thought in the translate-test setting

with language_code one of amh, eng, ewe, fra, hau, ibo, kin,
lin, lug, orm, sna, sot, swa, twi, vai, wol, xho, yor,
zul (the translate-test variants have no eng, and afrimgsm_translate has
no vai).

Checklist Copy Section

For adding novel benchmarks/datasets to the library:

• Is the task an existing benchmark in the literature?
  • Have you referenced the original paper that introduced the task?
  • If yes, does the original paper provide a reference implementation? If so, have you checked against the reference implementation and documented how to run such a test?

If other tasks on this dataset are already supported:

• Is the "Main" variant of this task clearly denoted?
• Have you provided a short sentence in a README on what each new variant adds / evaluates?
• Have you noted which, if any, published evaluation setups are matched by this variant?
  • Checked for equivalence with v0.3.0 LM Evaluation Harness

Overview (`lm_eval/tasks/afrimmlu/README.md`) Copy Section

AfriMMLU

Paper Copy Section

IrokoBench: A New Benchmark for African Languages in the Age of Large Language Models
https://arxiv.org/pdf/2406.03368

IrokoBench is a human-translated benchmark dataset for 16 typologically diverse
low-resource African languages covering three tasks: natural language inference (AfriXNLI),
mathematical reasoning (AfriMGSM), and multi-choice knowledge-based QA (AfriMMLU).

Citation Copy Section

@misc{adelani2024irokobenchnewbenchmarkafrican,
      title={IrokoBench: A New Benchmark for African Languages in the Age of Large Language Models},
      author={David Ifeoluwa Adelani and Jessica Ojo and Israel Abebe Azime and Jian Yun Zhuang and Jesujoba O. Alabi and Xuanli He and Millicent Ochieng and Sara Hooker and Andiswa Bukula and En-Shiun Annie Lee and Chiamaka Chukwuneke and Happy Buzaaba and Blessing Sibanda and Godson Kalipe and Jonathan Mukiibi and Salomon Kabongo and Foutse Yuehgoh and Mmasibidi Setaka and Lolwethu Ndolela and Nkiruka Odu and Rooweither Mabuya and Shamsuddeen Hassan Muhammad and Salomey Osei and Sokhar Samb and Tadesse Kebede Guge and Pontus Stenetorp},
      year={2024},
      eprint={2406.03368},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2406.03368},
}

Groups and Tasks Copy Section

Every AfriMMLU task is evaluated with five different prompt templates (the
updated IrokoBench/AfroBench prompts), so each per-language task name carries a
_prompt_{i} suffix where i is the prompt id (1-5). The original
afrimmlu_direct_{language_code} and afrimmlu_translate_{language_code}
names without the suffix are no longer registered.

Groups

• afrimmlu-irokobench: evaluates all direct tasks over all five prompts and reports a size-weighted mean accuracy
• afrimmlu_tt-irokobench: same as above in the translate-test setting

Tags

• afrimmlu_tasks: runs all direct tasks (all five prompts)
• afrimmlu_tasks_prompt_{i}: runs all direct tasks for prompt i (1-5)
• afrimmlu_tt_tasks: runs all translate-test tasks
• afrobench_mmlu_tasks: the AfriMMLU subset of the AfroBench benchmark (all direct tasks)

Tasks

• afrimmlu_direct_{language_code}_prompt_{i}: evaluates one language with prompt i (1-5) on the curated dataset, e.g. afrimmlu_direct_amh_prompt_1
• afrimmlu_translate_{language_code}_prompt_{i}: same as above in the translate-test setting

with language_code one of amh, eng, ewe, fra, hau, ibo, kin,
lin, lug, orm, sna, sot, swa, twi, wol, xho, yor, zul
(eng has no translate-test variant).

Checklist Copy Section

For adding novel benchmarks/datasets to the library:

• Is the task an existing benchmark in the literature?
  • Have you referenced the original paper that introduced the task?
  • If yes, does the original paper provide a reference implementation? If so, have you checked against the reference implementation and documented how to run such a test?

If other tasks on this dataset are already supported:

• Is the "Main" variant of this task clearly denoted?
• Have you provided a short sentence in a README on what each new variant adds / evaluates?
• Have you noted which, if any, published evaluation setups are matched by this variant?
  • Checked for equivalence with v0.3.0 LM Evaluation Harness

Overview (`lm_eval/tasks/afrixnli/README.md`) Copy Section

AfriXNLI

Paper Copy Section

IrokoBench: A New Benchmark for African Languages in the Age of Large Language Models
https://arxiv.org/pdf/2406.03368

IrokoBench is a human-translated benchmark dataset for 16 typologically diverse
low-resource African languages covering three tasks: natural language inference (AfriXNLI),
mathematical reasoning (AfriMGSM), and multi-choice knowledge-based QA (AfriMMLU).

Citation Copy Section

@misc{adelani2024irokobenchnewbenchmarkafrican,
      title={IrokoBench: A New Benchmark for African Languages in the Age of Large Language Models},
      author={David Ifeoluwa Adelani and Jessica Ojo and Israel Abebe Azime and Jian Yun Zhuang and Jesujoba O. Alabi and Xuanli He and Millicent Ochieng and Sara Hooker and Andiswa Bukula and En-Shiun Annie Lee and Chiamaka Chukwuneke and Happy Buzaaba and Blessing Sibanda and Godson Kalipe and Jonathan Mukiibi and Salomon Kabongo and Foutse Yuehgoh and Mmasibidi Setaka and Lolwethu Ndolela and Nkiruka Odu and Rooweither Mabuya and Shamsuddeen Hassan Muhammad and Salomey Osei and Sokhar Samb and Tadesse Kebede Guge and Pontus Stenetorp},
      year={2024},
      eprint={2406.03368},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2406.03368},
}

Groups and Tasks Copy Section

AfriXNLI ships two generations of prompts. The newer per-prompt variants (the
updated IrokoBench/AfroBench prompts) evaluate every language with five
different prompt templates, so their task names carry a _prompt_{i} suffix
where i is the prompt id (1-5). The original anli- and Lai-prompt tasks are
still registered under their per-language names without the suffix.

Groups

• afrixnli-irokobench: evaluates all direct tasks over all five prompts and reports a size-weighted mean accuracy
• afrixnli_tt-irokobench: same as above in the translate-test setting

Tags

• afrixnli_tasks: runs all direct per-prompt tasks (all five prompts)
• afrixnli_tasks_prompt_{i}: runs all direct per-prompt tasks for prompt i (1-5)
• afrixnli_tt_tasks: runs all per-prompt translate-test tasks
• afrixnli: runs all anli- and Lai-prompt tasks
• afrixnli_en_direct: anli prompt on the curated dataset
• afrixnli_native_direct: anli prompt translated to the respective languages on the curated dataset
• afrixnli_translate: anli prompt in the translate-test setting
• afrixnli_manual_direct: Lai's prompt on the curated dataset
• afrixnli_manual_translate: Lai's prompt in the translate-test setting

Tasks

Per-prompt variants:

• afrixnli_{language_code}_prompt_{i}: evaluates one language with prompt i (1-5) on the curated dataset, e.g. afrixnli_amh_prompt_1
• afrixnli_translate_{language_code}_prompt_{i}: same as above in the translate-test setting

Original anli- and Lai-prompt variants:

• afrixnli_en_direct_{language_code}: each task evaluates for one language
• afrixnli_native_direct_{language_code}: each task evaluates for one language
• afrixnli_translate_{language_code}: each task evaluates for one language
• afrixnli_manual_direct_{language_code}: each task evaluates for one language
• afrixnli_manual_translate_{language_code}: each task evaluates for one language

with language_code one of amh, eng, ewe, fra, hau, ibo, kin,
lin, lug, orm, sna, sot, swa, twi, wol, xho, yor, zul
(eng has no translate-test variant).

Checklist Copy Section

For adding novel benchmarks/datasets to the library:

• Is the task an existing benchmark in the literature?
  • Have you referenced the original paper that introduced the task?
  • If yes, does the original paper provide a reference implementation? If so, have you checked against the reference implementation and documented how to run such a test?

If other tasks on this dataset are already supported:

• Is the "Main" variant of this task clearly denoted?
• Have you provided a short sentence in a README on what each new variant adds / evaluates?
• Have you noted which, if any, published evaluation setups are matched by this variant?
  • Checked for equivalence with v0.3.0 LM Evaluation Harness

Overview (`lm_eval/tasks/afrobench/README.md`) Copy Section

AfroBench

Paper Copy Section

Title: AfroBench: How Good are Large Language Models on African Languages?

Paper Link: https://arxiv.org/abs/2311.07978

Abstract Copy Chapter

Large-scale multilingual evaluations, such as MEGA, often include only a handful of African languages due to the scarcity of high-quality evaluation data and the limited discoverability of existing African datasets. This lack of representation hinders comprehensive LLM evaluation across a diverse range of languages and tasks. To address these challenges, we introduce AfroBench -- a multi-task benchmark for evaluating the performance of LLMs across 64 African languages, 15 tasks and 22 datasets. AfroBench consists of nine natural language understanding datasets, six text generation datasets, six knowledge and question answering tasks, and one mathematical reasoning task. We present results comparing the performance of prompting LLMs to fine-tuned baselines based on BERT and T5-style models. Our results suggest large gaps in performance between high-resource languages, such as English, and African languages across most tasks; but performance also varies based on the availability of monolingual data resources. Our findings confirm that performance on African languages continues to remain a hurdle for current LLMs, underscoring the need for additional efforts to close this gap.

HomePage: https://mcgill-nlp.github.io/AfroBench/

Groups, and Tasks Copy Section

Groups

• afrobench : Runs all that tasks, datasets and prompts in this benchmark
• afrobench_lite: Runs the lite version of the benchmark which includes; afrimgsm, afrimmlu, afrixnli, sib, intent, adr and flores

Dataset specific grouping that listing all prompts, allowing users to review or edit them.

• adr afrihate afrisenti belebele african_flores injongointent mafand masakhaner masakhapos naijarc nollysenti african_ntrex openai_mmlu salt sib uhura xlsum

Task Tags

• adr_tasks: all datasets in this benchmark relating to Automatic Diacritics Restoration task
• afrihate_tasks: all datasets in this benchmark relating to Hate Speech detection task
• afrimgsm_tasks: all datasets in this benchmark relating to Mathematical reasoning task
• afrixnli_tasks: all datasets in this benchmark relating to Natural Language Inference task
• afrobench_xqa_tasks: all datasets in this benchmark relating to Crosslingual QA (XQA) task
• afrobench_sentiment_tasks: all datasets in this benchmark relating to Sentiment Classification task
• afrobench_MT_tasks: all datasets in this benchmark relating to Machine Translation task
• afrobench_TC_tasks: all datasets in this benchmark relating to Topic Classification task
• afrobench_mmlu_tasks: all datasets in this benchmark relating to MMLU task
• injongointent_tasks: all datasets in this benchmark relating to Intent Detection task
• masakhaner_tasks: all datasets in this benchmark relating to Named Entity Recognition (NER) task
• masakhapos_tasks: all datasets in this benchmark relating to Part of Speech Tagging (POS) task
• RC_tasks: all datasets in this benchmark relating to Reading Comprehension task
• uhura_arc_easy_tasks: all datasets in this benchmark relating to Arc-Easy (XQA) task
• xlsum_tasks: all datasets in this benchmark relating to Summarization task

We've included sample run scripts for easier integration with the benchmark: sample run scripts

For better understanding of the run interface see interface.md

All dataset used in this benchmark are available at huggingface

Citation Copy Section

@misc{ojo2025afrobenchgoodlargelanguage,
      title={AfroBench: How Good are Large Language Models on African Languages?},
      author={Jessica Ojo and Odunayo Ogundepo and Akintunde Oladipo and Kelechi Ogueji and Jimmy Lin and Pontus Stenetorp and David Ifeoluwa Adelani},
      year={2025},
      eprint={2311.07978},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2311.07978},
}

Please cite datasets used. Citations for individual datasets are included in their respective repository readme files within this benchmark.

Checklist Copy Section

For adding novel benchmarks/datasets to the library:

• Is the task an existing benchmark in the literature?
  • Have you referenced the original paper that introduced the task?
  • If yes, does the original paper provide a reference implementation? If so, have you checked against the reference implementation and documented how to run such a test? The original paper doesn't have an associated implementation, but there is an official entry in BigBench. I use the same prompting format as BigBench.

If other tasks on this dataset are already supported:

• Is the "Main" variant of this task clearly denoted?
• Have you provided a short sentence in a README on what each new variant adds / evaluates?
• Have you noted which, if any, published evaluation setups are matched by this variant?

Overview (`lm_eval/tasks/agieval/README.md`) Copy Section

AGIEval

Paper Copy Section

Title: AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models

Abstract: https://arxiv.org/abs/2304.06364.pdf

AGIEval is a human-centric benchmark specifically designed to evaluate the general abilities of foundation models in tasks pertinent to human cognition and problem-solving.
This benchmark is derived from 20 official, public, and high-standard admission and qualification exams intended for general human test-takers, such as general college admission tests (e.g., Chinese College Entrance Exam (Gaokao) and American SAT), law school admission tests, math competitions, lawyer qualification tests, and national civil service exams.

Homepage: https://github.com/ruixiangcui/AGIEval

Citation Copy Section

@misc{zhong2023agieval,
      title={AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models},
      author={Wanjun Zhong and Ruixiang Cui and Yiduo Guo and Yaobo Liang and Shuai Lu and Yanlin Wang and Amin Saied and Weizhu Chen and Nan Duan},
      year={2023},
      eprint={2304.06364},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

Please make sure to cite all the individual datasets in your paper when you use them. We provide the relevant citation information below:

/* Detailed source-code truncated for AI context efficiency. */

Groups, Tags, and Tasks Copy Section

Groups

• agieval: Evaluates all tasks listed below.

• agieval_en: Evaluates all English subtasks: agieval_aqua_rat, agieval_gaokao_english, agieval_logiqa_en, agieval_lsat_*, agieval_sat_*, agieval_math

• agieval_cn: Evaluates all Chinese subtasks:
  agieval_gaokao_biology, agieval_gaokao_chemistry, agieval_gaokao_chinese, agieval_gaokao_geography,
  agieval_gaokao_history, agieval_gaokao_mathqa, agieval_gaokao_mathcloze, agieval_gaokao_physics, agieval_jec_qa_ca, agieval_jec_qa_kd, agieval_logiqa_zh

• agieval_nous: Evaluates a specific subset of AGIEval tasks (multiple-choice and english-only), namely those in https://github.com/teknium1/LLM-Benchmark-Logs/blob/main/benchmark-logs/Mistral-7B-Base.md

Tags

None.

Tasks

• agieval_aqua_rat
• agieval_gaokao_biology
• agieval_gaokao_chemistry
• agieval_gaokao_chinese
• agieval_gaokao_english
• agieval_gaokao_geography
• agieval_gaokao_history
• agieval_gaokao_mathqa
• agieval_gaokao_mathcloze
• agieval_gaokao_physics
• agieval_jec_qa_ca
• agieval_jec_qa_kd
• agieval_logiqa_en
• agieval_logiqa_zh
• agieval_lsat_ar
• agieval_lsat_lr
• agieval_lsat_rc
• agieval_sat_en
• agieval_sat_en_without_passage
• agieval_sat_math
• agieval_math

Overview (`lm_eval/tasks/aime/README.md`) Copy Section

AIME

Citation Copy Section

@dataset{aime_1983_2024,
  author = {Hemish Veeraboina},
  title = {AIME Problem Set 1983-2024},
  year = {2024},
  publisher = {Kaggle},
  url = {https://www.kaggle.com/datasets/hemishveeraboina/aime-problem-set-1983-2024}
}

@dataset{aime_2024,
  author = {Maxwell Jia},
  title = {AIME Problem Set 2024},
  year = {2024},
  publisher = {Huggingface},
  url = {https://huggingface.co/datasets/Maxwell-Jia/AIME_2024}
}

@dataset{aime_2025,
  author = {math-ai},
  title = {AIME Problem Set 2025},
  year = {2025},
  publisher = {Huggingface},
  url = {https://huggingface.co/datasets/math-ai/aime25}
}

Groups, Tags, and Tasks Copy Section

Groups

• math_word_problems

Tasks

• aime: AIME 1983-2024 problems
• aime24: AIME 2024 problems
• aime25: AIME 2025 problems

Checklist Copy Section

For adding novel benchmarks/datasets to the library:

• Is the task an existing benchmark in the literature?
  • Have you referenced the original paper that introduced the task?
  • If yes, does the original paper provide a reference implementation? If so, have you checked against the reference implementation and documented how to run such a test?

If other tasks on this dataset are already supported:

• Is the "Main" variant of this task clearly denoted?
• Have you provided a short sentence in a README on what each new variant adds / evaluates?
• Have you noted which, if any, published evaluation setups are matched by this variant?

Overview (`lm_eval/tasks/anli/README.md`) Copy Section

ANLI

Paper Copy Section

Title: Adversarial NLI: A New Benchmark for Natural Language Understanding

Paper Link: https://arxiv.org/abs/1910.14599

Adversarial NLI (ANLI) is a dataset collected via an iterative, adversarial
human-and-model-in-the-loop procedure. It consists of three rounds that progressively
increase in difficulty and complexity, and each question-answer includes annotator-
provided explanations.

Homepage: https://github.com/facebookresearch/anli

Citation Copy Section

@inproceedings{nie-etal-2020-adversarial,
    title = "Adversarial {NLI}: A New Benchmark for Natural Language Understanding",
    author = "Nie, Yixin  and
      Williams, Adina  and
      Dinan, Emily  and
      Bansal, Mohit  and
      Weston, Jason  and
      Kiela, Douwe",
    booktitle = "Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics",
    year = "2020",
    publisher = "Association for Computational Linguistics",
}

Groups and Tasks Copy Section

Groups

• anli: Evaluates anli_r1, anli_r2, and anli_r3

Tasks

• anli_r1: The data collected adversarially in the first round.
• anli_r2: The data collected adversarially in the second round, after training on the previous round's data.
• anli_r3: The data collected adversarially in the third round, after training on the previous multiple rounds of data.

Checklist Copy Section

For adding novel benchmarks/datasets to the library:

• Is the task an existing benchmark in the literature?
• Have you referenced the original paper that introduced the task?
• If yes, does the original paper provide a reference implementation? If so, have you checked against the reference implementation and documented how to run such a test?

If other tasks on this dataset are already supported:

• Is the "Main" variant of this task clearly denoted?
• Have you provided a short sentence in a README on what each new variant adds / evaluates?
• Have you noted which, if any, published evaluation setups are matched by this variant?

Overview (`lm_eval/tasks/arab_culture/README.md`) Copy Section

Arab Culture

Paper Copy Section

Title: Commonsense Reasoning in Arab Culture

Abstract: https://arxiv.org/abs/2502.12788

Despite progress in Arabic large language models, such as Jais and AceGPT, their evaluation on commonsense reasoning has largely relied on machine-translated datasets, which lack cultural depth and may introduce Anglocentric biases. Commonsense reasoning is shaped by geographical and cultural contexts, and existing English datasets fail to capture the diversity of the Arab world. To address this, we introduce \datasetname, a commonsense reasoning dataset in Modern Standard Arabic (MSA), covering cultures of 13 countries across the Gulf, Levant, North Africa, and the Nile Valley. The dataset was built from scratch by engaging native speakers to write and validate culturally relevant questions for their respective countries. \datasetname spans 12 daily life domains with 54 fine-grained subtopics, reflecting various aspects of social norms, traditions, and everyday experiences. Zero-shot evaluations show that open-weight language models with up to 32B parameters struggle to comprehend diverse Arab cultures, with performance varying across regions. These findings highlight the need for more culturally aware models and datasets tailored to the Arabic-speaking world.

Homepage: https://github.com/fajri91/ArabicCulture

Citation Copy Section

@misc{sadallah2025commonsensereasoningarabculture,
      title={Commonsense Reasoning in Arab Culture},
      author={Abdelrahman Sadallah and Junior Cedric Tonga and Khalid Almubarak and Saeed Almheiri and Farah Atif and Chatrine Qwaider and Karima Kadaoui and Sara Shatnawi and Yaser Alesh and Fajri Koto},
      year={2025},
      eprint={2502.12788},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2502.12788},
}

There are two variant of this task: `arab_culture`, and `arab_culture_completion` Copy Section

• The arab_culture is the normal MCQ evaluation type, which appends the answers to the question, and then measure the likelihood of the different choices markers (A,B,C or "أ","ب","ج"). For more info, follow the MMLU style tempelate
• The arab_culture_completion do the evaluation in a sentence completion manner, by appending each asnwer to the question separetley and chooses the answer with the higher likelihood. See this for more information

Groups and Tasks Copy Section

Groups

• arabculture: evaluates all ArabCulture tasks.

• arab_culture_gulf: evaluates Gulf countires ArabCulture tasks.

• arab_culture_levant: evaluates Levant countires ArabCulture tasks.

• arab_culture_nile_valley: evaluates Nile Valley countires ArabCulture tasks.

• arab_culture_north_africa: evaluates North Africa ArabCulture tasks.

Evaluation modes Copy Section

This bechmark allows for different evaluation settings by allowing to adding more extra context for the model:

We have three settings:

• without any information

COUNTRY=False
REGION=False

• with only region information

COUNTRY=False
REGION=True

• with region and country information

COUNTRY=True
REGION=True

Please add these flags add environment variables.

• We also allow for prompting in English, which we found to achieve higher results on most of the evaluated models (please refer to our paper).

• To change the language of the prompt, Define the ARABIC environment variable.

Overview (`lm_eval/tasks/arab_culture_completion/README.md`) Copy Section

Arab Culture

Paper Copy Section

Title: Commonsense Reasoning in Arab Culture

Abstract: https://arxiv.org/abs/2502.12788

Despite progress in Arabic large language models, such as Jais and AceGPT, their evaluation on commonsense reasoning has largely relied on machine-translated datasets, which lack cultural depth and may introduce Anglocentric biases. Commonsense reasoning is shaped by geographical and cultural contexts, and existing English datasets fail to capture the diversity of the Arab world. To address this, we introduce \datasetname, a commonsense reasoning dataset in Modern Standard Arabic (MSA), covering cultures of 13 countries across the Gulf, Levant, North Africa, and the Nile Valley. The dataset was built from scratch by engaging native speakers to write and validate culturally relevant questions for their respective countries. \datasetname spans 12 daily life domains with 54 fine-grained subtopics, reflecting various aspects of social norms, traditions, and everyday experiences. Zero-shot evaluations show that open-weight language models with up to 32B parameters struggle to comprehend diverse Arab cultures, with performance varying across regions. These findings highlight the need for more culturally aware models and datasets tailored to the Arabic-speaking world.

Homepage: https://github.com/fajri91/ArabicCulture

Citation Copy Section

@misc{sadallah2025commonsensereasoningarabculture,
      title={Commonsense Reasoning in Arab Culture},
      author={Abdelrahman Sadallah and Junior Cedric Tonga and Khalid Almubarak and Saeed Almheiri and Farah Atif and Chatrine Qwaider and Karima Kadaoui and Sara Shatnawi and Yaser Alesh and Fajri Koto},
      year={2025},
      eprint={2502.12788},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2502.12788},
}

There are two variant of this task: `arab_culture`, and `arab_culture_completion` Copy Section

• The arab_culture is the normal MCQ evaluation type, which appends the answers to the question, and then measure the likelihood of the different choices markers (A,B,C or "أ","ب","ج"). For more info, follow the MMLU style tempelate
• The arab_culture_completion do the evaluation in a sentence completion manner, by appending each asnwer to the question separetley and chooses the answer with the higher likelihood. See this for more information

Groups and Tasks Copy Section

Groups

• arabculture: evaluates all ArabCulture tasks.

• arab_culture_gulf: evaluates Gulf countires ArabCulture tasks.

• arab_culture_levant: evaluates Levant countires ArabCulture tasks.

• arab_culture_nile_valley: evaluates Nile Valley countires ArabCulture tasks.

• arab_culture_north_africa: evaluates North Africa ArabCulture tasks.

Evaluation modes Copy Section

This bechmark allows for different evaluation settings by allowing to adding more extra context for the model:

We have three settings:

• without any information

COUNTRY=False
REGION=False

• with only region information

COUNTRY=False
REGION=True

• with region and country information

COUNTRY=True
REGION=True

Please add these flags add environment variables.

• We also allow for prompting in English, which we found to acheive higher results on most of the evaluated models (please refer to our paper).

• To change the language of the prompt, Define the ARABIC environment variable.

Overview (`lm_eval/tasks/arabic_leaderboard_complete/README.md`) Copy Section

Arabic Leaderboard

Title: Open Arabic LLM Leaderboard

The Open Arabic LLM Leaderboard evaluates language models on a large number of different evaluation tasks that reflect the characteristics of the Arabic language and culture.
The benchmark uses several datasets, most of them translated to Arabic, and validated by native Arabic speakers. They also used benchmarks from other papers or prepared benchmarks from scratch natively for Arabic.

Homepage: https://huggingface.co/spaces/OALL/Open-Arabic-LLM-Leaderboard

Citation Copy Section

/* Detailed source-code truncated for AI context efficiency. */

Groups and Tasks Copy Section

• arabic_leaderboard_alghafa: A multiple-choice evaluation benchmark for zero- and few-shot evaluation of Arabic LLMs prepared from scratch natively for Arabic.
  • Paper: https://aclanthology.org/2023.arabicnlp-1.21.pdf
  • You can find the list of the tasks as follows:
    • arabic_leaderboard_alghafa_mcq_exams_test_ar
    • arabic_leaderboard_alghafa_meta_ar_dialects
    • arabic_leaderboard_alghafa_meta_ar_msa
    • arabic_leaderboard_alghafa_multiple_choice_facts_truefalse_balanced_task
    • arabic_leaderboard_alghafa_multiple_choice_grounded_statement_soqal_task
    • arabic_leaderboard_alghafa_multiple_choice_grounded_statement_xglue_mlqa_task
    • arabic_leaderboard_alghafa_multiple_choice_rating_sentiment_no_neutral_task
    • arabic_leaderboard_alghafa_multiple_choice_rating_sentiment_task
    • arabic_leaderboard_alghafa_multiple_choice_sentiment_task
• arabic_leaderboard_arabic_exams: A question answering benchmark for high school examinations in different school subjects that requires knowledge and reasoning in different languages in multiple domains.
  • Paper: https://aclanthology.org/2020.emnlp-main.438.pdf
• arabic_leaderboard_arabic_mmlu: A multi-task language understanding benchmark for the Arabic language, sourced from school exams across diverse educational levels in different countries with native speakers in the region.
  The data comprises multiple choice questions in 40 tasks.
  • Paper: https://arxiv.org/pdf/2402.12840
  • You can find the list of the tasks as follows:
    • arabic_leaderboard_arabic_mmlu_abstract_algebra
    • arabic_leaderboard_arabic_mmlu_anatomy
    • arabic_leaderboard_arabic_mmlu_astronomy
    • arabic_leaderboard_arabic_mmlu_business_ethics
    • arabic_leaderboard_arabic_mmlu_clinical_knowledge
    • arabic_leaderboard_arabic_mmlu_college_biology
    • arabic_leaderboard_arabic_mmlu_college_chemistry
    • arabic_leaderboard_arabic_mmlu_college_computer_science
    • arabic_leaderboard_arabic_mmlu_college_mathematics
    • arabic_leaderboard_arabic_mmlu_college_medicine
    • arabic_leaderboard_arabic_mmlu_college_physics
    • arabic_leaderboard_arabic_mmlu_computer_security
    • arabic_leaderboard_arabic_mmlu_conceptual_physics
    • arabic_leaderboard_arabic_mmlu_econometrics
    • arabic_leaderboard_arabic_mmlu_electrical_engineering
    • arabic_leaderboard_arabic_mmlu_elementary_mathematics
    • arabic_leaderboard_arabic_mmlu_formal_logic
    • arabic_leaderboard_arabic_mmlu_global_facts
    • arabic_leaderboard_arabic_mmlu_high_school_biology
    • arabic_leaderboard_arabic_mmlu_high_school_chemistry
    • arabic_leaderboard_arabic_mmlu_high_school_computer_science
    • arabic_leaderboard_arabic_mmlu_high_school_european_history
    • arabic_leaderboard_arabic_mmlu_high_school_geography
    • arabic_leaderboard_arabic_mmlu_high_school_government_and_politics
    • arabic_leaderboard_arabic_mmlu_high_school_macroeconomics
    • arabic_leaderboard_arabic_mmlu_high_school_mathematics
    • arabic_leaderboard_arabic_mmlu_high_school_microeconomics
    • arabic_leaderboard_arabic_mmlu_high_school_physics
    • arabic_leaderboard_arabic_mmlu_high_school_psychology
    • arabic_leaderboard_arabic_mmlu_high_school_statistics
    • arabic_leaderboard_arabic_mmlu_high_school_us_history
    • arabic_leaderboard_arabic_mmlu_high_school_us_history
    • arabic_leaderboard_arabic_mmlu_human_aging
    • arabic_leaderboard_arabic_mmlu_human_sexuality
    • arabic_leaderboard_arabic_mmlu_international_law
    • arabic_leaderboard_arabic_mmlu_jurisprudence
    • arabic_leaderboard_arabic_mmlu_logical_fallacies
    • arabic_leaderboard_arabic_mmlu_machine_learning
    • arabic_leaderboard_arabic_mmlu_management
    • arabic_leaderboard_arabic_mmlu_marketing
    • arabic_leaderboard_arabic_mmlu_medical_genetics
    • arabic_leaderboard_arabic_mmlu_miscellaneous
    • arabic_leaderboard_arabic_mmlu_moral_disputes
    • arabic_leaderboard_arabic_mmlu_moral_scenarios
    • arabic_leaderboard_arabic_mmlu_nutrition
    • arabic_leaderboard_arabic_mmlu_philosophy
    • arabic_leaderboard_arabic_mmlu_prehistory
    • arabic_leaderboard_arabic_mmlu_professional_accounting
    • arabic_leaderboard_arabic_mmlu_professional_law
    • arabic_leaderboard_arabic_mmlu_professional_medicine
    • arabic_leaderboard_arabic_mmlu_professional_psychology
    • arabic_leaderboard_arabic_mmlu_public_relations
    • arabic_leaderboard_arabic_mmlu_security_studies
    • arabic_leaderboard_arabic_mmlu_sociology
    • arabic_leaderboard_arabic_mmlu_us_foreign_policy
    • arabic_leaderboard_arabic_mmlu_virology
    • arabic_leaderboard_arabic_mmlu_world_religions
• arabic_leaderboard_arabic_mt_arc_challenge: AI2 Reasoning Challenge (ARC) is a multiple-choice question task. The dataset contains only natural, grade-school science questions,
  written for human tests. The challenge set contains only questions answered incorrectly by both a retrieval-based algorithm and a word co-occurence algorithm. (machine translated benchmark - part of the Alghafa Arabic translated LLM benchmark)
  • Paper: https://aclanthology.org/2023.arabicnlp-1.21.pdf
• arabic_leaderboard_arabic_mt_arc_easy: This dataset is the same as arabic_arc_challenge, except it is not from the challenge set.
  • Paper: https://aclanthology.org/2023.arabicnlp-1.21.pdf
• arabic_leaderboard_arabic_mt_boolq: A true/false questions dataset that contains the columns passage, question, and the answer (i.e., true/false). (machine translated benchmark - part of the Alghafa Arabic translated LLM benchmark)
  • Paper: https://aclanthology.org/2023.arabicnlp-1.21.pdf
• arabic_leaderboard_arabic_mt_copa: Choice Of Plausible Alternatives (COPA) is a multiple-choice question dataset, which involves open-domain commonsense causal reasoning. (machine translated benchmark - part of the Alghafa Arabic translated LLM benchmark)
  • Paper: https://aclanthology.org/2023.arabicnlp-1.21.pdf
• arabic_leaderboard_arabic_mt_hellaswag: The tesk is to choose the next set of sentences, based on the given candidates. The tasks involve reading comprehension and information retrieval challenges
  by testing the abilities of the models on basic knowledge (i.e., from 3rd grade to 9th) and commonsense inference. (machine translated benchmark - part of the Alghafa Arabic translated LLM benchmark)
  • Paper: https://aclanthology.org/2023.arabicnlp-1.21.pdf
• arabic_leaderboard_arabic_mt_mmlu: A multiple-choice question answering dataset from various branches of knowledge including humanities, social sciences, hard sciences, and other areas. The examples in the English dataset are translated into Arabic using ChatGPT with a translation prompt.
  • Paper: https://aclanthology.org/2023.arabicnlp-1.21.pdf
• arabic_leaderboard_arabic_mt_openbook_qa: A multiple-choice openbook question answering dataset that requires external knowledge and reasoning. The open book that comes with these questions is
  based on elementary level science facts. (machine translated benchmark - part of the Alghafa Arabic translated LLM benchmark)
  • Paper: https://aclanthology.org/2023.arabicnlp-1.21.pdf
• arabic_leaderboard_arabic_mt_piqa: Physical Interaction Question Answering (PIQA) is a multiple-choice question answering based on physical commonsense reasoning. (machine translated benchmark - part of the Alghafa Arabic translated LLM benchmark)
  • Paper: https://aclanthology.org/2023.arabicnlp-1.21.pdf
• arabic_leaderboard_arabic_mt_race: A multiple-choice questions dataset to assess reading comprehension tasks based on English exams in China - designed for middle school and high school students
  (machine translated benchmark - part of the Alghafa Arabic translated LLM benchmark)
  • Paper: https://aclanthology.org/2023.arabicnlp-1.21.pdf
• arabic_leaderboard_arabic_mt_sciq: A multiple-choice Science Question Answering task to assess understanding of scientific concepts about physics, chemistry, and biology. (machine translated benchmark - part of the Alghafa Arabic translated LLM benchmark)
  • Paper: https://aclanthology.org/2023.arabicnlp-1.21.pdf
• arabic_leaderboard_arabic_mt_toxigen: This benchmark consists of tasks designed to evaluate language models and classify input text as hateful or not hateful. (machine translated benchmark - part of the Alghafa Arabic translated LLM benchmark)
  • Paper: https://aclanthology.org/2023.arabicnlp-1.21.pdf
• arabic_leaderboard_acva: Arabic-Culture-Value-Alignment (ACVA) is a yes/no question dataset, generated by GPT3.5 Turbo from Arabic topics to assess model alignment with Arabic values and cultures.
  • Paper: https://arxiv.org/pdf/2309.12053
  • You can find the list of the tasks as follows:
    • arabic_leaderboard_acva_Algeria
    • arabic_leaderboard_acva_Ancient_Egypt
    • arabic_leaderboard_acva_Arab_Empire
    • arabic_leaderboard_acva_Arabic_Architecture
    • arabic_leaderboard_acva_Arabic_Art
    • arabic_leaderboard_acva_Arabic_Astronomy
    • arabic_leaderboard_acva_Arabic_Calligraphy
    • arabic_leaderboard_acva_Arabic_Ceremony
    • arabic_leaderboard_acva_Arabic_Clothing
    • arabic_leaderboard_acva_Arabic_Culture
    • arabic_leaderboard_acva_Arabic_Food
    • arabic_leaderboard_acva_Arabic_Funeral
    • arabic_leaderboard_acva_Arabic_Geography
    • arabic_leaderboard_acva_Arabic_History
    • arabic_leaderboard_acva_Arabic_Language_Origin
    • arabic_leaderboard_acva_Arabic_Literature
    • arabic_leaderboard_acva_Arabic_Math
    • arabic_leaderboard_acva_Arabic_Medicine
    • arabic_leaderboard_acva_Arabic_Music
    • arabic_leaderboard_acva_Arabic_Ornament
    • arabic_leaderboard_acva_Arabic_Philosophy
    • arabic_leaderboard_acva_Arabic_Physics_and_Chemistry
    • arabic_leaderboard_acva_Arabic_Wedding
    • arabic_leaderboard_acva_Bahrain
    • arabic_leaderboard_acva_Comoros
    • arabic_leaderboard_acva_Egypt_modern
    • arabic_leaderboard_acva_InfluenceFromAncientEgypt
    • arabic_leaderboard_acva_InfluenceFromByzantium
    • arabic_leaderboard_acva_InfluenceFromChina
    • arabic_leaderboard_acva_InfluenceFromGreece
    • arabic_leaderboard_acva_InfluenceFromIslam
    • arabic_leaderboard_acva_InfluenceFromPersia
    • arabic_leaderboard_acva_InfluenceFromRome
    • arabic_leaderboard_acva_Iraq
    • arabic_leaderboard_acva_Islam_Education
    • arabic_leaderboard_acva_Islam_branches_and_schools
    • arabic_leaderboard_acva_Islamic_law_system
    • arabic_leaderboard_acva_Jordan
    • arabic_leaderboard_acva_Kuwait
    • arabic_leaderboard_acva_Lebanon
    • arabic_leaderboard_acva_Libya
    • arabic_leaderboard_acva_Mauritania
    • arabic_acva_Mesopotamia_civilization
    • arabic_leaderboard_acva_Morocco
    • arabic_leaderboard_acva_Oman
    • arabic_leaderboard_acva_Palestine
    • arabic_leaderboard_acva_Qatar
    • arabic_leaderboard_acva_Saudi_Arabia
    • arabic_leaderboard_acva_Somalia
    • arabic_leaderboard_acva_Sudan
    • arabic_leaderboard_acva_Syria
    • arabic_leaderboard_acva_Tunisia
    • arabic_leaderboard_acva_United_Arab_Emirates
    • arabic_leaderboard_acva_Yemen
    • arabic_leaderboard_acva_communication
    • arabic_leaderboard_acva_computer_and_phone
    • arabic_leaderboard_acva_daily_life
    • arabic_leaderboard_acva_entertainment

Checklist Copy Section

For adding novel benchmarks/datasets to the library:

• Is the task an existing benchmark in the literature?
  • Have you referenced the original paper that introduced the task?
  • If yes, does the original paper provide a reference implementation? If so, have you checked against the reference implementation and documented how to run such a test?

If other tasks on this dataset are already supported:

• Is the "Main" variant of this task clearly denoted?
• Have you provided a short sentence in a README on what each new variant adds / evaluates?
• Have you noted which, if any, published evaluation setups are matched by this variant?

Overview (`lm_eval/tasks/arabic_leaderboard_light/README.md`) Copy Section

Arabic Leaderboard Light

Title: Open Arabic LLM Leaderboard Light

This leaderboard follows all the details as in arabic_leaderboard_complete, except that a light version - 10% random sample of the test set of each benchmark - is used to test the language models.

NOTE: In ACVA benchmark, there is Yemen subset, and it is a small dataset - it has only 10 samples in the test split. So, for this specific subset dataset, to have more reliable results, we consider the original dataset, instead of 10% of its test samples.

Checklist Copy Section

For adding novel benchmarks/datasets to the library:

• Is the task an existing benchmark in the literature?
  • Have you referenced the original paper that introduced the task?
  • If yes, does the original paper provide a reference implementation? If so, have you checked against the reference implementation and documented how to run such a test?

If other tasks on this dataset are already supported:

• Is the "Main" variant of this task clearly denoted?
• Have you provided a short sentence in a README on what each new variant adds / evaluates?
• Have you noted which, if any, published evaluation setups are matched by this variant?

Overview (`lm_eval/tasks/arabicmmlu/README.md`) Copy Section

ArabicMMLU

Paper Copy Section

Title: ArabicMMLU: Assessing Massive Multitask Language Understanding in Arabic

Abstract: https://arxiv.org/abs/2402.12840

The focus of language model evaluation has
transitioned towards reasoning and knowledge intensive tasks, driven by advancements in pretraining large models. While state-of-the-art models are partially trained on large Arabic texts, evaluating their performance in Arabic remains challenging due to the limited availability of relevant datasets. To bridge this gap, we present ArabicMMLU, the first multi-task language understanding benchmark for Arabic language, sourced from school exams across diverse educational levels in different countries spanning North Africa, the Levant, and the Gulf regions. Our data comprises 40 tasks and 14,575 multiple-choice questions in Modern Standard Arabic (MSA), and is carefully constructed by collaborating with native speakers in the region. Our comprehensive evaluations of 35 models reveal substantial room for improvement, particularly among the best open-source models. Notably, BLOOMZ, mT0, LLama2, and Falcon struggle to achieve a score of 50%, while even the top-performing Arabic centric model only achieves a score of 62.3%.

The authors of the paper conducted studies by varying the language of the initial prompt and answer keys between English and Arabic. However, they set English initial prompts and answer keys as the standard, which is the version implemented in this task.

Homepage: https://github.com/mbzuai-nlp/ArabicMMLU

Citation Copy Section

@misc{koto2024arabicmmlu,
      title={ArabicMMLU: Assessing Massive Multitask Language Understanding in Arabic},
      author={Fajri Koto and Haonan Li and Sara Shatnawi and Jad Doughman and Abdelrahman Boda Sadallah and Aisha Alraeesi and Khalid Almubarak and Zaid Alyafeai and Neha Sengupta and Shady Shehata and Nizar Habash and Preslav Nakov and Timothy Baldwin},
      year={2024},
      eprint={2402.12840},
      archivePrefix={arXiv},
      primaryClass={id='cs.CL' full_name='Computation and Language' is_active=True alt_name='cmp-lg' in_archive='cs' is_general=False description='Covers natural language processing. Roughly includes material in ACM Subject Class I.2.7. Note that work on artificial languages (programming languages, logics, formal systems) that does not explicitly address natural-language issues broadly construed (natural-language processing, computational linguistics, speech, text retrieval, etc.) is not appropriate for this area.'}
}

Groups and Tasks Copy Section

Groups

• arabicmmlu: evaluates all ArabicMMLU tasks.

• arabicmmlu_stem: evaluates STEM ArabicMMLU tasks.

• arabicmmlu_stem_social_science: evaluates social science ArabicMMLU tasks.

• arabicmmlu_stem_humanities: evaluates humanities ArabicMMLU tasks.

• arabicmmlu_stem_language: evaluates Arabic language ArabicMMLU tasks.

• arabicmmlu_stem_other: evaluates other ArabicMMLU tasks.

Overview (`lm_eval/tasks/aradice/README.md`) Copy Section

AraDiCE

Paper Copy Section

Title: AraDiCE: Benchmarks for Dialectal and Cultural Capabilities in LLMs

Abstract: Arabic, with its rich diversity of dialects, remains significantly underrepresented in Large Language Models, particularly in dialectal variations. We address this gap by introducing seven synthetic datasets in dialects alongside Modern Standard Arabic (MSA), created using Machine Translation (MT) combined with human post-editing. We present AraDiCE, a benchmark for Arabic Dialect and Cultural Evaluation. We evaluate LLMs on dialect comprehension and generation, focusing specifically on low-resource Arabic dialects. Additionally, we introduce the first-ever fine-grained benchmark designed to evaluate cultural awareness across the Gulf, Egypt, and Levant regions, providing a novel dimension to LLM evaluation. Our findings demonstrate that while Arabic-specific models like Jais and AceGPT outperform multilingual models on dialectal tasks, significant challenges persist in dialect identification, generation, and translation. This work contributes ~45K post-edited samples, a cultural benchmark, and highlights the importance of tailored training to improve LLM performance in capturing the nuances of diverse Arabic dialects and cultural contexts. We will release the dialectal translation models and benchmarks curated in this study.

Homepage:
https://huggingface.co/datasets/QCRI/AraDiCE

Citation Copy Section

@article{mousi2024aradicebenchmarksdialectalcultural,
      title={{AraDiCE}: Benchmarks for Dialectal and Cultural Capabilities in LLMs},
      author={Basel Mousi and Nadir Durrani and Fatema Ahmad and Md. Arid Hasan and Maram Hasanain and Tameem Kabbani and Fahim Dalvi and Shammur Absar Chowdhury and Firoj Alam},
      year={2024},
      publisher={arXiv:2409.11404},
      url={https://arxiv.org/abs/2409.11404},
}

Groups, Tags, and Tasks Copy Section

Groups

• AraDiCE: Overall results for all tasks associated with different datasets.

Tasks

• aradice: Overall results for all tasks associated with different datasets.
• arabicmmlu: TODO

Checklist Copy Section

• Is the task an existing benchmark in the literature?
  • Have you referenced the original paper that introduced the task?
  • If yes, does the original paper provide a reference implementation? If so, have you checked against the reference implementation and documented how to run such a test?

If other tasks on this dataset are already supported:

• Is the "Main" variant of this task clearly denoted?
• Have you provided a short sentence in a README on what each new variant adds / evaluates?
• Have you noted which, if any, published evaluation setups are matched by this variant?

Overview (`lm_eval/tasks/arc/README.md`) Copy Section

ARC

Paper Copy Section

Title: Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge

Abstract: https://arxiv.org/abs/1803.05457

The ARC dataset consists of 7,787 science exam questions drawn from a variety
of sources, including science questions provided under license by a research
partner affiliated with AI2. These are text-only, English language exam questions
that span several grade levels as indicated in the files. Each question has a
multiple choice structure (typically 4 answer options). The questions are sorted
into a Challenge Set of 2,590 “hard” questions (those that both a retrieval and
a co-occurrence method fail to answer correctly) and an Easy Set of 5,197 questions.

Homepage: https://allenai.org/data/arc

Citation Copy Section

@article{Clark2018ThinkYH,
  title={Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge},
  author={Peter Clark and Isaac Cowhey and Oren Etzioni and Tushar Khot and Ashish Sabharwal and Carissa Schoenick and Oyvind Tafjord},
  journal={ArXiv},
  year={2018},
  volume={abs/1803.05457}
}

Groups, Tags, and Tasks Copy Section

Groups

None.

Tags

• ai2_arc: Evaluates arc_easy and arc_challenge

Tasks

• arc_easy
• arc_challenge

Checklist Copy Section

For adding novel benchmarks/datasets to the library:

• Is the task an existing benchmark in the literature?
  • Have you referenced the original paper that introduced the task?
  • If yes, does the original paper provide a reference implementation? If so, have you checked against the reference implementation and documented how to run such a test?

If other tasks on this dataset are already supported:

• Is the "Main" variant of this task clearly denoted?
• Have you provided a short sentence in a README on what each new variant adds / evaluates?
• Have you noted which, if any, published evaluation setups are matched by this variant?

Overview (`lm_eval/tasks/arc_mt/README.md`) Copy Section

arc mt

arc mt is an implementation of tasks to support machine translated arc
challenge evals, to improve eval support across a number of additional
languages.

The main page for the effort is
here and we will
include more data and analysis there.

Initial datasets include a number of European languages, and we plan to expand
more in the future.

Overview (`lm_eval/tasks/arithmetic/README.md`) Copy Section

Arithmetic

Paper Copy Section

Title: Language Models are Few-Shot Learners
Abstract: https://arxiv.org/abs/2005.14165

A small battery of 10 tests that involve asking language models a simple arithmetic
problem in natural language.

Homepage: https://github.com/openai/gpt-3/tree/master/data

Citation Copy Section

@inproceedings{NEURIPS2020_1457c0d6,
    author = {Brown, Tom and Mann, Benjamin and Ryder, Nick and Subbiah, Melanie and Kaplan, Jared D and Dhariwal, Prafulla and Neelakantan, Arvind and Shyam, Pranav and Sastry, Girish and Askell, Amanda and Agarwal, Sandhini and Herbert-Voss, Ariel and Krueger, Gretchen and Henighan, Tom and Child, Rewon and Ramesh, Aditya and Ziegler, Daniel and Wu, Jeffrey and Winter, Clemens and Hesse, Chris and Chen, Mark and Sigler, Eric and Litwin, Mateusz and Gray, Scott and Chess, Benjamin and Clark, Jack and Berner, Christopher and McCandlish, Sam and Radford, Alec and Sutskever, Ilya and Amodei, Dario},
    booktitle = {Advances in Neural Information Processing Systems},
    editor = {H. Larochelle and M. Ranzato and R. Hadsell and M. F. Balcan and H. Lin},
    pages = {1877--1901},
    publisher = {Curran Associates, Inc.},
    title = {Language Models are Few-Shot Learners},
    url = {https://proceedings.neurips.cc/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf},
    volume = {33},
    year = {2020}
}

Groups, Tags, and Tasks Copy Section

Tags

• arithmetic: Evaluates 1dc to 5ds

Tasks

• arithmetic_1dc
• arithmetic_2da
• arithmetic_2dm
• arithmetic_2ds
• arithmetic_3da
• arithmetic_3ds
• arithmetic_4da
• arithmetic_4ds
• arithmetic_5da
• arithmetic_5ds

Checklist Copy Section

For adding novel benchmarks/datasets to the library:

• Is the task an existing benchmark in the literature?
  • Have you referenced the original paper that introduced the task?
  • If yes, does the original paper provide a reference implementation? If so, have you checked against the reference implementation and documented how to run such a test?

If other tasks on this dataset are already supported:

• Is the "Main" variant of this task clearly denoted?
• Have you provided a short sentence in a README on what each new variant adds / evaluates?
• Have you noted which, if any, published evaluation setups are matched by this variant?

Changelog Copy Section

version 2.0: (2025-Feb-14) set target delimiter to "" as the targets already start with a space.

Overview (`lm_eval/tasks/asdiv/README.md`) Copy Section

ASDiv

Paper Copy Section

Title: ASDiv: A Diverse Corpus for Evaluating and Developing English Math Word Problem Solvers

Abstract: https://arxiv.org/abs/2106.15772

ASDiv (Academia Sinica Diverse MWP Dataset) is a diverse (in terms of both language
patterns and problem types) English math word problem (MWP) corpus for evaluating
the capability of various MWP solvers. Existing MWP corpora for studying AI progress
remain limited either in language usage patterns or in problem types. We thus present
a new English MWP corpus with 2,305 MWPs that cover more text patterns and most problem
types taught in elementary school. Each MWP is annotated with its problem type and grade
level (for indicating the level of difficulty).

NOTE: We currently ignore formulas for answer generation.

Homepage: https://github.com/chaochun/nlu-asdiv-dataset

Citation Copy Section

@misc{miao2021diverse,
    title={A Diverse Corpus for Evaluating and Developing English Math Word Problem Solvers},
    author={Shen-Yun Miao and Chao-Chun Liang and Keh-Yih Su},
    year={2021},
    eprint={2106.15772},
    archivePrefix={arXiv},
    primaryClass={cs.AI}
}

Groups, Tags, and Tasks Copy Section

Groups

• Not part of a group yet.

Tasks

• asdiv
• asdiv_cot_llama: ASDIV with prompt formatting modified to conform to the evaluation settings described by Meta here: https://huggingface.co/datasets/meta-llama/Meta-Llama-3.1-8B-Instruct-evals/viewer/Meta-Llama-3.1-8B-Instruct-evals__gsm8k__details?row=0
  • Note that the CoT prompt from (https://arxiv.org/pdf/2201.11903) is used exactly as in GSM8k-CoT
  • This file is setup to run identically to the task gsm8k_cot_llama but for asdiv.
  • Use this task with --fewshot_as_multiturn and --apply_chat_template to run correctly with Llama Instruct models.

Checklist Copy Section

For adding novel benchmarks/datasets to the library:

• Is the task an existing benchmark in the literature?
  • Have you referenced the original paper that introduced the task?
  • If yes, does the original paper provide a reference implementation? If so, have you checked against the reference implementation and documented how to run such a test?

If other tasks on this dataset are already supported:

• Is the "Main" variant of this task clearly denoted?
• Have you provided a short sentence in a README on what each new variant adds / evaluates?
• Have you noted which, if any, published evaluation setups are matched by this variant?

Overview (`lm_eval/tasks/babi/README.md`) Copy Section

bAbI

Paper Copy Section

Title: Towards ai-complete question answering: A set of prerequisite toy tasks
Abstract: https://arxiv.org/abs/1502.05698

One long-term goal of machine learning research is to produce methods that are applicable to reasoning and natural language, in particular building an intelligent dialogue agent. To measure progress towards that goal, we argue for the usefulness of a set of proxy tasks that evaluate reading comprehension via question answering. Our tasks measure understanding in several ways: whether a system is able to answer questions via chaining facts, simple induction, deduction and many more. The tasks are designed to be prerequisites for any system that aims to be capable of conversing with a human. We believe many existing learning systems can currently not solve them, and hence our aim is to classify these tasks into skill sets, so that researchers can identify (and then rectify) the failings of their systems. We also extend and improve the recently introduced Memory Networks model, and show it is able to solve some, but not all, of the tasks.

Homepage: https://github.com/facebookarchive/bAbI-tasks

Citation Copy Section

@article{weston2015towards,
  title={Towards ai-complete question answering: A set of prerequisite toy tasks},
  author={Weston, Jason and Bordes, Antoine and Chopra, Sumit and Rush, Alexander M and Van Merri{\"e}nboer, Bart and Joulin, Armand and Mikolov, Tomas},
  journal={arXiv preprint arXiv:1502.05698},
  year={2015}
}

Groups, Tags, and Tasks Copy Section

Groups

• Not part of a group yet

Tags

• No tags applied.

Tasks

• babi

Checklist Copy Section

For adding novel benchmarks/datasets to the library:

• Is the task an existing benchmark in the literature?
  • Have you referenced the original paper that introduced the task?
  • If yes, does the original paper provide a reference implementation? If so, have you checked against the reference implementation and documented how to run such a test?

If other tasks on this dataset are already supported:

• Is the "Main" variant of this task clearly denoted?
• Have you provided a short sentence in a README on what each new variant adds / evaluates?
• Have you noted which, if any, published evaluation setups are matched by this variant?

Overview (`lm_eval/tasks/babilong/README.md`) Copy Section

Babilong

Paper Copy Section

Title: Babilong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack
Abstract: https://arxiv.org/abs/2406.10149

In recent years, the input context sizes of large language models (LLMs) have increased dramatically. However, existing evaluation methods have not kept pace, failing to comprehensively assess the efficiency of models in handling long contexts. To bridge this gap, we introduce the BABILong benchmark, designed to test language models' ability to reason across facts distributed in extremely long documents. BABILong includes a diverse set of 20 reasoning tasks, including fact chaining, simple induction, deduction, counting, and handling lists/sets. These tasks are challenging on their own, and even more demanding when the required facts are scattered across long natural text. Our evaluations show that popular LLMs effectively utilize only 10-20% of the context and their performance declines sharply with increased reasoning complexity. Among alternatives to in-context reasoning, Retrieval-Augmented Generation methods achieve a modest 60% accuracy on single-fact question answering, independent of context length. Among context extension methods, the highest performance is demonstrated by recurrent memory transformers after fine-tuning, enabling the processing of lengths up to 50 million tokens. The BABILong benchmark is extendable to any length to support the evaluation of new upcoming models with increased capabilities, and we provide splits up to 10 million token lengths.

Homepage: https://github.com/booydar/babilong

Citation Copy Section

@article{kuratov2024babilong,
    title={Babilong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack},
    author={Kuratov, Yuri and Bulatov, Aydar and Anokhin, Petr and Rodkin, Ivan and Sorokin, Dmitry and Burtsev, Mikhail},
    journal={arXiv preprint arXiv:2406.10149},
    year={2024}
}

Groups and Tasks Copy Section

Groups

• babilong: All Babilong tasks at 0k context length
• babilong_longctx: Babilong tasks between qa1-qa5 at context lengths up to 128k

Tasks

The benchmark includes 1000 samples of 20 reasoning tasks at various context lengths:

QA Tasks (qa1-qa20):

• babilong_qa1: Single supporting fact QA
• babilong_qa2: Two supporting facts QA
• babilong_qa3: Three supporting facts QA
• babilong_qa4: Two argument relations
• babilong_qa5: Three argument relations
• babilong_qa6: Yes/No questions
• babilong_qa7: Counting
• babilong_qa8: Lists and sets
• babilong_qa9: Simple negation
• babilong_qa10: Indefinite knowledge
• babilong_qa11: Track person through temporal references
• babilong_qa12: Conjunction
• babilong_qa13: Compound coreference
• babilong_qa14: Time reasoning
• babilong_qa15: Basic deduction
• babilong_qa16: Basic induction
• babilong_qa17: Positional reasoning
• babilong_qa18: Size reasoning
• babilong_qa19: Path finding
• babilong_qa20: Motivation deduction

[!NOTE]
When using babilong tasks, please note:

1. This is the implementation with 1000 samples per length. You can change the dataset path to RMT-team/babilong in common_utils.py for the dataset with 100 samples per length, which supports context lengths up to 10M tokens.
2. Supported lengths are 0k, 1, 2, 4, 8, 16, 32, 64, 128k tokens for tasks qa1-5. Tasks qa6-20 only have a length of 0k.
3. The default maximum sequence length is 0k. For calculating metrics of different max seq lengths, specify additional lengths using the metadata parameter:
   --metadata '{"max_seq_lengths":"0k,1k,2k,4k,8k,16k,32k,128k"}'. The config currently only takes one context length at a time. The metadata parameter can also be passed to the TaskManager (metadata: dict).

Checklist Copy Section

For adding novel benchmarks/datasets to the library:

• Is the task an existing benchmark in the literature?
  • Have you referenced the original paper that introduced the task?
  • If yes, does the original paper provide a reference implementation? If so, have you checked against the reference implementation and documented how to run such a test?

If other tasks on this dataset are already supported:

• Is the "Main" variant of this task clearly denoted?
• Have you provided a short sentence in a README on what each new variant adds / evaluates?
• Have you noted which, if any, published evaluation setups are matched by this variant?

Overview (`lm_eval/tasks/bangla/README.md`) Copy Section

Bangla Benchmark Datasets

This repository contains resources related to Bangla MMLU, Bangla OpenBookQA, Bangla BoolQA, Bangla PIQA, Bangla CommonsenseQA, benchmark datasets designed for evaluating Bangla language models. The dataset is used for training, development, and comparative evaluation of language models in the Bangla language.

Overview Copy Chapter

TituLLMs is a family of Bangla large language models (LLMs) with comprehensive benchmarking designed to advance natural language processing for the Bangla language. The benchmarking datasets covers questions across a diverse range of topics in Bangla.

These datasets were primarily used to train, validate, and evaluate Bangla language models(TituLLM) and compare their performance with other existing models.

For more details, please refer to the original research paper:
https://arxiv.org/abs/2502.11187

Dataset Copy Chapter

The datasets can be found on Hugging Face:
https://huggingface.co/datasets/hishab/titulm-bangla-mmlu

https://huggingface.co/datasets/hishab/boolq_bn

https://huggingface.co/datasets/hishab/openbookqa-bn

https://huggingface.co/datasets/hishab/piqa-bn

https://huggingface.co/datasets/hishab/commonsenseqa-bn

These datasets were used as a benchmark in the development and evaluation of TituLLMs.

Usage Copy Chapter

The dataset is intended for use within the [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) repository to evaluate and compare the performance of Bangla language models. Copy Chapter

Citation Copy Chapter

If you use this dataset or model, please cite the original paper:

@misc{nahin2025titullmsfamilybanglallms,
      title={TituLLMs: A Family of Bangla LLMs with Comprehensive Benchmarking},
      author={Shahriar Kabir Nahin and Rabindra Nath Nandi and Sagor Sarker and Quazi Sarwar Muhtaseem and Md Kowsher and Apu Chandraw Shill and Md Ibrahim and Mehadi Hasan Menon and Tareq Al Muntasir and Firoj Alam},
      year={2025},
      eprint={2502.11187},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2502.11187},
}

---

### Overview (`lm_eval/tasks/basque_bench/README.md`)

# BasqueBench

### Paper

BasqueBench is a benchmark for evaluating language models in Basque tasks. This is, it evaluates the ability of a language model to understand and generate Basque text. BasqueBench offers a combination of pre-existing, open datasets and datasets developed exclusivelly for this benchmark. All the details of BasqueBench will be published in a paper soon.

The new evaluation datasets included in BasqueBench are:
| Task     | Category                   | Homepage                                      |
|:--------:|:--------------------------:|:---------------------------------------------:|
| ARC_eu   | Question Answering         | https://huggingface.co/datasets/HiTZ/ARC-eu   |
| MGSM_eu  | Math                       | https://huggingface.co/datasets/HiTZ/MGSM-eu  |
| PAWS_eu  | Paraphrasing               | https://huggingface.co/datasets/HiTZ/PAWS-eu  |
| PIQA_eu  | Question Answering         | https://huggingface.co/datasets/HiTZ/PIQA-eu  |
| WNLI_eu  | Natural Language Inference | https://huggingface.co/datasets/HiTZ/WNLI-eu  |
| XCOPA_eu | Commonsense Reasoning      | https://huggingface.co/datasets/HiTZ/XCOPA-eu |

The datasets included in BasqueBench that have been made public in previous publications are:

| Task          | Category       | Paper title          | Homepage  |
|:-------------:|:-----:|:-------------:|:-----:|
| Belebele_eu | Reading Comprehension | [The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants](https://arxiv.org/abs/2308.16884) | https://huggingface.co/datasets/facebook/belebele |
| EusExams | Question Answering | [Latxa: An Open Language Model and Evaluation Suite for Basque](https://arxiv.org/abs/2403.20266) | https://huggingface.co/datasets/HiTZ/EusExams |
| EusProficiency | Question Answering | [Latxa: An Open Language Model and Evaluation Suite for Basque](https://arxiv.org/abs/2403.20266) | https://huggingface.co/datasets/HiTZ/EusProficiency |
| EusReading | Reading Comprehension | [Latxa: An Open Language Model and Evaluation Suite for Basque](https://arxiv.org/abs/2403.20266) | https://huggingface.co/datasets/HiTZ/EusReading |
| EusTrivia | Question Answering | [Latxa: An Open Language Model and Evaluation Suite for Basque](https://arxiv.org/abs/2403.20266) | https://huggingface.co/datasets/HiTZ/EusTrivia |
| FLORES_eu | Translation | [No Language Left Behind: Scaling Human-Centered Machine Translation](https://arxiv.org/abs/2207.04672) | https://huggingface.co/datasets/facebook/flores |
| QNLIeu | Natural Language Inference | [BasqueGLUE: A Natural Language Understanding Benchmark for Basque](https://aclanthology.org/2022.lrec-1.172/) | https://huggingface.co/datasets/orai-nlp/basqueGLUE |
| XNLIeu | Natural Language Inference | [XNLIeu: a dataset for cross-lingual NLI in Basque](https://arxiv.org/abs/2404.06996) | https://huggingface.co/datasets/HiTZ/xnli-eu |
| XStoryCloze_eu | Commonsense Reasoning | [Few-shot Learning with Multilingual Generative Language Models](https://aclanthology.org/2022.emnlp-main.616/) | https://huggingface.co/datasets/juletxara/xstory_cloze |


### Citation

@inproceedings{baucells-etal-2025-iberobench,
title = "{I}bero{B}ench: A Benchmark for {LLM} Evaluation in {I}berian Languages",
author = "Baucells, Irene and
Aula-Blasco, Javier and
de-Dios-Flores, Iria and
Paniagua Su{'a}rez, Silvia and
Perez, Naiara and
Salles, Anna and
Sotelo Docio, Susana and
Falc{~a}o, J{'u}lia and
Saiz, Jose Javier and
Sepulveda Torres, Robiert and
Barnes, Jeremy and
Gamallo, Pablo and
Gonzalez-Agirre, Aitor and
Rigau, German and
Villegas, Marta",
editor = "Rambow, Owen and
Wanner, Leo and
Apidianaki, Marianna and
Al-Khalifa, Hend and
Eugenio, Barbara Di and
Schockaert, Steven",
booktitle = "Proceedings of the 31st International Conference on Computational Linguistics",
month = jan,
year = "2025",
address = "Abu Dhabi, UAE",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2025.coling-main.699/",
pages = "10491--10519",
}

### Groups and Tasks

#### Groups

- `basque_bench`: All tasks included in BasqueBench.
- `flores_eu`: All FLORES translation tasks from or to Basque.

#### Tasks

The following tasks evaluate tasks on BasqueBench dataset using various scoring methods.
  - `arc_eu_challenge`
  - `arc_eu_easy`
  - `belebele_eus_Latn`
  - `eus_exams_eu`
  - `eus_proficiency`
  - `eus_reading`
  - `eus_trivia`
  - `flores_eu`
  - `flores_eu-ca`
  - `flores_eu-de`
  - `flores_eu-en`
  - `flores_eu-es`
  - `flores_eu-fr`
  - `flores_eu-gl`
  - `flores_eu-it`
  - `flores_eu-pt`
  - `flores_ca-eu`
  - `flores_de-eu`
  - `flores_en-eu`
  - `flores_es-eu`
  - `flores_fr-eu`
  - `flores_gl-eu`
  - `flores_it-eu`
  - `flores_pt-eu`
  - `mgsm_direct_eu`
  - `mgsm_native_cot_eu`
  - `paws_eu`
  - `piqa_eu`
  - `qnlieu`
  - `wnli_eu`
  - `xcopa_eu`
  - `xnli_eu`
  - `xnli_eu_native`
  - `xstorycloze_eu`

Some of these tasks are taken from benchmarks already available in LM Evaluation Harness. These are:
- `belebele_eus_Latn`: Belebele Basque
- `qnlieu`: From BasqueGLUE


### Checklist

* [x] Is the task an existing benchmark in the literature?
  * [ ] Have you referenced the original paper that introduced the task?
  * [ ] If yes, does the original paper provide a reference implementation?
    * [ ] Yes, original implementation contributed by author of the benchmark

If other tasks on this dataset are already supported:
* [ ] Is the "Main" variant of this task clearly denoted?
* [ ] Have you provided a short sentence in a README on what each new variant adds / evaluates?
* [ ] Have you noted which, if any, published evaluation setups are matched by this variant?

### Changelog

* v1.1 (2026-01-16, PR #3465): excluded line breaks from stop criteria in `mgsm_direct_eu` and `msgm_native_cot_eu`

---

### Overview (`lm_eval/tasks/basqueglue/README.md`)

# BasqueGLUE

### Paper

Title: `BasqueGLUE: A Natural Language Understanding Benchmark for Basque`

Abstract: `https://aclanthology.org/2022.lrec-1.172/`

Natural Language Understanding (NLU) technology has improved significantly over the last few years and multitask benchmarks such as GLUE are key to evaluate this improvement in a robust and general way. These benchmarks take into account a wide and diverse set of NLU tasks that require some form of language understanding, beyond the detection of superficial, textual clues. However, they are costly to develop and language-dependent, and therefore they are only available for a small number of languages. In this paper, we present BasqueGLUE, the first NLU benchmark for Basque, a less-resourced language, which has been elaborated from previously existing datasets and following similar criteria to those used for the construction of GLUE and SuperGLUE. We also report the evaluation of two state-of-the-art language models for Basque on BasqueGLUE, thus providing a strong baseline to compare upon. BasqueGLUE is freely available under an open license.

Homepage: `https://github.com/orai-nlp/BasqueGLUE`

Title: `Latxa: An Open Language Model and Evaluation Suite for Basque`

Abstract: `https://arxiv.org/abs/2403.20266`

The use of BasqueGLUE for evaluating the performance of decoder models in Basque is presented in this paper.

Homepage: `https://github.com/hitz-zentroa/latxa`

### Citation

@InProceedings{urbizu2022basqueglue,
author = {Urbizu, Gorka and San Vicente, Iñaki and Saralegi, Xabier and Agerri, Rodrigo and Soroa, Aitor},
title = {BasqueGLUE: A Natural Language Understanding Benchmark for Basque},
booktitle = {Proceedings of the Language Resources and Evaluation Conference},
month = {June},
year = {2022},
address = {Marseille, France},
publisher = {European Language Resources Association},
pages = {1603--1612},
url = {https://aclanthology.org/2022.lrec-1.172}
}

@misc{etxaniz2024latxa,
title={Latxa: An Open Language Model and Evaluation Suite for Basque},
author={Julen Etxaniz and Oscar Sainz and Naiara Perez and Itziar Aldabe and German Rigau and Eneko Agirre and Aitor Ormazabal and Mikel Artetxe and Aitor Soroa},
year={2024},
eprint={2403.20266},
archivePrefix={arXiv},
primaryClass={cs.CL}
}

### Groups, Tags, and Tasks

#### Groups

None.

#### Tags

* `basque-glue`: First version of the implementation. Calls all subtasks, but does not average.

#### Tasks

* `bhtc_v2`: Topic classification of news extracts with 12 categories.
* `bec2016eu`: Sentiment analysis on tweets about the campaign for the 2016 Basque elections.
* `vaxx_stance`: Stance detection on tweets around the anti-vaccine movement.
* `qnlieu`: Q&A NLI as in [glue/qnli](../glue/qnli).
* `wiceu`: Word-in-Context as in [super_glue/wic](../super_glue/wic).
* `epec_koref_bin`: Correference detection as in [super_glue/wsc](../super_glue/wsc).

### Checklist

For adding novel benchmarks/datasets to the library:
* [ ] Is the task an existing benchmark in the literature?
  * [ ] Have you referenced the original paper that introduced the task?
  * [ ] If yes, does the original paper provide a reference implementation? If so, have you checked against the reference implementation and documented how to run such a test?


If other tasks on this dataset are already supported:
* [ ] Is the "Main" variant of this task clearly denoted?
* [ ] Have you provided a short sentence in a README on what each new variant adds / evaluates?
* [ ] Have you noted which, if any, published evaluation setups are matched by this variant?

---

### Overview (`lm_eval/tasks/bbh/README.md`)

# BigBenchHard

## Paper
Title: `Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them`
Abstract: https://arxiv.org/abs/2210.09261

A suite of 23 challenging BIG-Bench tasks which we call BIG-Bench Hard (BBH).
These are the task for which prior language model evaluations did not outperform
the average human-rater.

Homepage: https://github.com/suzgunmirac/BIG-Bench-Hard


## Citation

@article{suzgun2022challenging,
title={Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them},
author={Suzgun, Mirac and Scales, Nathan and Sch{"a}rli, Nathanael and Gehrmann, Sebastian and Tay, Yi and Chung, Hyung Won and Chowdhery, Aakanksha and Le, Quoc V and Chi, Ed H and Zhou, Denny and and Wei, Jason},
journal={arXiv preprint arXiv:2210.09261},
year={2022}
}

### Groups, Tags, and Tasks

#### Groups

- `bbh`: is the same as `bbh_cot_fewshot`.
- `bbh_zeroshot`
- `bbh_fewshot`
- `bbh_cot_fewshot`
- `bbh_cot_zeroshot`

#### Tags

None.

#### Tasks

- ...

### Checklist

- [x] Is in Eval-harness v1.0 ?
- [ ] Has been checked for regression from v1.0?
- [ ] Has been checked for equivalence with original paper methodology?
- [ ] "Main" checked variant clearly denoted?

### Variant Wishlist

- [ ] Variant with Calculator (see https://github.com/openai/grade-school-math/blob/master/grade_school_math/calculator.py for example implementation)
- [ ] Using Verifiers
- [ ] Majority voting "without CoT"

### Changelog
- no version change: changed dataset to `SaylorTwift/bbh`. Do not expect any change in the results.
- `bbh_cot_fewshot` v.4.0; 2025-07-14:
  - PR #3140. Removed duplicate "Let's think step by step" from the fewshots.
  - set target_delimiter to "" as the fewshot samples end with a newline character.

---

### Overview (`lm_eval/tasks/bbq/README.md`)

# BBQ

### Paper

Title: BBQ: A Hand-Built Bias Benchmark for Question Answering

Abstract: https://aclanthology.org/2022.findings-acl.165/

BBQ measures the bias in the output for the question answering task.
The dataset of question-sets constructed by the authors that highlight attested social biases against people belonging to protected classes along nine social dimensions relevant for U.S. English-speaking contexts.
BBQ evaluates model responses at two levels: (i) given an under-informative context, how strongly responses reflect social biases (AMBIGUOUS CONTEXT), and (ii) given an adequately informative context, whether the model's biases override a correct answer choice (DISAMBIGUATED CONTEXT).

Homepage: https://github.com/nyu-mll/BBQ


### Citation

@inproceedings{parrish-etal-2022-bbq,
title = "{BBQ}: A hand-built bias benchmark for question answering",
author = "Parrish, Alicia and
Chen, Angelica and
Nangia, Nikita and
Padmakumar, Vishakh and
Phang, Jason and
Thompson, Jana and
Htut, Phu Mon and
Bowman, Samuel",
editor = "Muresan, Smaranda and
Nakov, Preslav and
Villavicencio, Aline",
booktitle = "Findings of the Association for Computational Linguistics: ACL 2022",
month = may,
year = "2022",
address = "Dublin, Ireland",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2022.findings-acl.165",
doi = "10.18653/v1/2022.findings-acl.165",
pages = "2086--2105"
}

### Groups and Tasks

#### Groups

* `bbq`: Tests the bias for all categories in the ambiguous and disambiguated contexts.

#### Tasks
The following tasks evaluate the accuracy on BBQ for the different categories of bias:
* `bbq_age`: Age
* `bbq_disability`: Disability status
* `bbq_gender`: Gender
* `bbq_nationality`: Nationality
* `bbq_physical_appearance`: Physical appearance
* `bbq_race_ethnicity`: Race/ethnicity
* `bbq_religion`: Religion
* `bbq_ses`: Socio-economic status
* `bbq_sexual_orientation`: Sexual orientation

Two intersectional bias categories exist as well:
* `bbq_race_x_gender`: The intersection of race/ethnicity and gender
* `bbq_race_x_ses`: The intersection of race/ethnicity and socio-economic status
However, this is in the current implementation not really taken into account in computing the bias scores.

### Checklist

For adding novel benchmarks/datasets to the library:
* [ ] Is the task an existing benchmark in the literature?
  * [ ] Have you referenced the original paper that introduced the task?
  * [ ] If yes, does the original paper provide a reference implementation? If so, have you checked against the reference implementation and documented how to run such a test?


If other tasks on this dataset are already supported:
* [ ] Is the "Main" variant of this task clearly denoted?
* [ ] Have you provided a short sentence in a README on what each new variant adds / evaluates?
* [ ] Have you noted which, if any, published evaluation setups are matched by this variant?

---

### Overview (`lm_eval/tasks/bear/README.md`)

# BEAR

Dataset to evaluate common factual knowledge in language models. This dataset was created as part of the [paper "BEAR: A Unified Framework for Evaluating Relational Knowledge in Causal and Masked Language Models"](https://aclanthology.org/2024.findings-naacl.155/).

For more information visit the [LM Pub Quiz website](https://lm-pub-quiz.github.io/).

## Paper

Title: BEAR: A Unified Framework for Evaluating Relational Knowledge in Causal and Masked Language Models

Abstract: https://aclanthology.org/2024.findings-naacl.155/

Knowledge probing assesses to which degree a language model (LM) has successfully learned relational knowledge during pre-training. Probing is an inexpensive way to compare LMs of different sizes and training configurations. However, previous approaches rely on the objective function used in pre-training LMs and are thus applicable only to masked or causal LMs. As a result, comparing different types of LMs becomes impossible. To address this, we propose an approach that uses an LM’s inherent ability to estimate the log-likelihood of any given textual statement. We carefully design an evaluation dataset of 7,731 instances (40,916 in a larger variant) from which we produce alternative statements for each relational fact, one of which is correct. We then evaluate whether an LM correctly assigns the highest log-likelihood to the correct statement. Our experimental evaluation of 22 common LMs shows that our proposed framework, BEAR, can effectively probe for knowledge across different LM types. We release the BEAR datasets and an open-source framework that implements the probing approach to the research community to facilitate the evaluation and development of LMs.

Homepage: https://lm-pub-quiz.github.io/

### Citation

/* Detailed source-code truncated for AI context efficiency. */

### Groups, Tags, and Tasks

#### Groups

* Not part of a group yet.

#### Tags

* No specific tags yet

#### Tasks

* `bear`: BEAR knowledge probe (factual multiple choice task)
* `bear_big`: Big variant of BEAR (more questions, larger answer spaces)



### Changelog

---

### Overview (`lm_eval/tasks/belebele/README.md`)

# Belebele

### Paper

The Belebele Benchmark for Massively Multilingual NLU Evaluation
https://arxiv.org/abs/2308.16884

Belebele is a multiple-choice machine reading comprehension (MRC) dataset spanning 122 language variants. This dataset enables the evaluation of mono- and multi-lingual models in high-, medium-, and low-resource languages. Each question has four multiple-choice answers and is linked to a short passage from the FLORES-200 dataset. The human annotation procedure was carefully curated to create questions that discriminate between different levels of generalizable language comprehension and is reinforced by extensive quality checks. While all questions directly relate to the passage, the English dataset on its own proves difficult enough to challenge state-of-the-art language models. Being fully parallel, this dataset enables direct comparison of model performance across all languages. Belebele opens up new avenues for evaluating and analyzing the multilingual abilities of language models and NLP systems.

Homepage: https://github.com/facebookresearch/belebele

### Citation

```bibtex
@misc{bandarkar2023belebele,
      title={The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants},
      author={Lucas Bandarkar and Davis Liang and Benjamin Muller and Mikel Artetxe and Satya Narayan Shukla and Donald Husa and Naman Goyal and Abhinandan Krishnan and Luke Zettlemoyer and Madian Khabsa},
      year={2023},
      eprint={2308.16884},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

Groups and Tasks Copy Section

Groups

• belebele: All 122 languages of the Belebele dataset, evaluated following the methodology in MMLU's original implementation.

Tasks

The following tasks evaluate languages in the Belebele dataset using loglikelihood-based multiple-choice scoring:

• belebele_{language}

The variant evaluated here is the 0-shot or few-shot evaluation with English Instructions.

Checklist Copy Section

• Is the task an existing benchmark in the literature?
  • Have you referenced the original paper that introduced the task?
  • If yes, does the original paper provide a reference implementation?
    • Yes, original implementation contributed by author of the benchmark

If other tasks on this dataset are already supported:

• Is the "Main" variant of this task clearly denoted?
• Have you provided a short sentence in a README on what each new variant adds / evaluates?
• Have you noted which, if any, published evaluation setups are matched by this variant?

Overview (`lm_eval/tasks/benchmarks/README.md`) Copy Section

Changelog Copy Section

• 2025-Mar-17 OpenLLM v2: Fixed few-shot split to correctly use train set for arc_challenge.
• 2026-Sep-14 group rte_flan v1.1: Replace duplicate rte_prompt-1 entry in group config with rte_prompt-2

Overview (`lm_eval/tasks/bertaqa/README.md`) Copy Section

BertaQA

Paper Copy Section

Title: BertaQA: How Much Do Language Models Know About Local Culture?

Abstract: https://arxiv.org/abs/2406.07302

Large Language Models (LLMs) exhibit extensive knowledge about the world, but most evaluations have been limited to global or anglocentric subjects. This raises the question of how well these models perform on topics relevant to other cultures, whose presence on the web is not that prominent. To address this gap, we introduce BertaQA, a multiple-choice trivia dataset that is parallel in English and Basque. The dataset consists of a local subset with questions pertinent to the Basque culture, and a global subset with questions of broader interest. We find that state-of-the-art LLMs struggle with local cultural knowledge, even as they excel on global topics. However, we show that continued pre-training in Basque significantly improves the models' performance on Basque culture, even when queried in English. To our knowledge, this is the first solid evidence of knowledge transfer from a low-resource to a high-resource language. Our analysis sheds light on the complex interplay between language and knowledge, and reveals that some prior findings do not fully hold when reassessed on local topics. Our dataset and evaluation code are available under open licenses at https://github.com/juletx/BertaQA.

Homepage: https://github.com/juletx/BertaQA

Citation Copy Section

@misc{etxaniz2024bertaqa,
      title={BertaQA: How Much Do Language Models Know About Local Culture?},
      author={Julen Etxaniz and Gorka Azkune and Aitor Soroa and Oier Lopez de Lacalle and Mikel Artetxe},
      year={2024},
      eprint={2406.07302},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

Groups and Tasks Copy Section

Groups

• bertaqa: Group of BertaQA tasks.

Tasks

• bertaqa_eu: Trivia questions in Basque.
• bertaqa_en: Trivia questions in English, human-translated from Basque.
• bertaqa_en_mt_*: Trivia questions in English, machine-translated from Basque with different models.

Checklist Copy Section

For adding novel benchmarks/datasets to the library:

• Is the task an existing benchmark in the literature?
  • Have you referenced the original paper that introduced the task?
  • If yes, does the original paper provide a reference implementation? If so, have you checked against the reference implementation and documented how to run such a test?

If other tasks on this dataset are already supported:

• Is the "Main" variant of this task clearly denoted?
• Have you provided a short sentence in a README on what each new variant adds / evaluates?
• Have you noted which, if any, published evaluation setups are matched by this variant?

Overview (`lm_eval/tasks/bhs/README.md`) Copy Section

BHS: Controlled Evaluation of Syntactic Knowledge in Basque, Hindi, and Swahili

Paper Copy Chapter

Title: Controlled Evaluation of Syntactic Knowledge in Multilingual Language Models

Abstract:

Language models (LMs) are capable of acquiring elements of human-like syntactic knowledge. Targeted syntactic evaluation tests have been employed to measure how well they form generalizations about syntactic phenomena in high-resource languages such as English. However, we still lack a thorough understanding of LMs' capacity for syntactic generalizations in low-resource languages, which are responsible for much of the diversity of syntactic patterns worldwide. In this study, we develop targeted syntactic evaluation tests for three low-resource languages (Basque, Hindi, and Swahili) and use them to evaluate five families of open-access multilingual Transformer LMs. We find that some syntactic tasks prove relatively easy for LMs while others (agreement in sentences containing indirect objects in Basque, agreement across a prepositional phrase in Swahili) are challenging. We additionally uncover issues with publicly available Transformers, including a bias toward the habitual aspect in Hindi in multilingual BERT and underperformance compared to similar-sized models in XGLM-4.5B. (Kryvosheieva & Levy, 2025)

Homepage: https://github.com/dariakryvosheieva/syntactic_generalization_multilingual

Citation Copy Section

@inproceedings{kryvosheieva-levy-2025-controlled,
    title = "Controlled Evaluation of Syntactic Knowledge in Multilingual Language Models",
    author = "Kryvosheieva, Daria and Levy, Roger",
    editor = "Hettiarachchi, Hansi and Ranasinghe, Tharindu and Rayson, Paul and Mitkov, Ruslan and Gaber, Mohamed and Premasiri, Damith and Tan, Fiona Anting and Uyangodage, Lasitha",
    booktitle = "Proceedings of the First Workshop on Language Models for Low-Resource Languages",
    month = jan,
    year = "2025",
    address = "Abu Dhabi, United Arab Emirates",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2025.loreslm-1.30/",
    pages = "402--413"
}

Groups, Tags, and Tasks Copy Section

• bhs_basque: Run all Basque tasks (listed below) and calculate mean performance. In all tasks, the goal is for the model to predict the auxiliary verb (AUX) that correctly agrees with the subject (S), direct object (DO), and indirect object (IO). Each task manipulates a different one of these, e.g., for bhs__basque__DO__S_IO_DO_V_AUX, the two presented sentences (with S_IO_DO_V_AUX structure) have auxiliary verbs that agree with the subject and indirect object, and the task is to correctly assign the one that also agrees with the direct object (DO) a higher probability than the one that does not. For specific examples, see Kryvosheieva & Levy (2025).

  • bhs__basque__DO__S_DO_V_AUX
  • bhs__basque__DO__S_IO_DO_V_AUX
  • bhs__basque__IO__IO_S_V_AUX
  • bhs__basque__IO__S_IO_DO_V_AUX
  • bhs__basque__S__IO_S_V_AUX
  • bhs__basque__S__S_DO_V_AUX
  • bhs__basque__S__S_IO_DO_V_AUX
  • bhs__basque__S__S_V_AUX

• bhs_hindi: Run all Hindi tasks (listed below) and calculate mean performance. In all tasks, the goal is for the model to predict that in a sentence with the 'ne' clitic, the final verb should be in a perfective form, and in sentences without, it should be in a non-perfective form (in this case, habitual or progressive) by assigning a higher probability to the correct verb. For specific examples, see Kryvosheieva & Levy (2025).

  • bhs__hindi__S_O_V
  • bhs__hindi__S_PossPRN_O_V
  • bhs__hindi__S_PossPRN_PossN_O_V
  • bhs__hindi__S_ne_O_V
  • bhs__hindi__S_ne_PossPRN_O_V
  • bhs__hindi__S_ne_PossPRN_PossN_O_V

• bhs_swahili: Run all Swahili tasks (listed below) and calculate mean performance. In all tasks, the goal is for the model to assign the final word - a verb (V) or adjective (A/AN) a higher probability if it correctly agrees with the initial noun (in terms of noun class) than if it does not. For specific examples, see Kryvosheieva & Levy (2025).

  • bhs__swahili__N_of_Poss_D_AP_V_ni_AN
  • bhs__swahili__N_of_Poss_D_AP_ni_AN
  • bhs__swahili__N_of_Poss_D_A_V
  • bhs__swahili__N_of_Poss_D_A_V1_V2
  • bhs__swahili__N_of_Poss_D_V
  • bhs__swahili__N_of_Poss_D_ni_A
  • bhs__swahili__N_of_Poss_V
  • bhs__swahili__N_of_Poss_ni_A

Implementation Note: The original implementation normalizes the log-probability of the final word by its length in number of tokens, which is not supported by the Language Model Evaluation Harness (see [1], [2], [3]). For this reason, the implementation provided here includes both the acc (accuracy based on comparing the unnormalized log-probability of the correct and incorrect versions of each sentence) and acc_norm (the same as acc but with sentence log-probability normalized by number of bytes) metrics.

Checklist Copy Section

For adding novel benchmarks/datasets to the library:

• Is the task an existing benchmark in the literature?
  • Have you referenced the original paper that introduced the task?
  • If yes, does the original paper provide a reference implementation? If so, have you checked against the reference implementation and documented how to run such a test?

Changelog Copy Section

Overview (`lm_eval/tasks/bigbench/README.md`) Copy Section

BigBench

Paper Copy Section

Title: Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models

Abstract: https://arxiv.org/abs/2206.04615

The Beyond the Imitation Game Benchmark (BIG-bench) is a collaborative benchmark intended to probe large language models and extrapolate their future capabilities.

Homepage: https://github.com/google/BIG-bench

Citation Copy Section

/* Detailed source-code truncated for AI context efficiency. */

Groups and Tasks Copy Section

Groups

• group_name: Short description

Tags

• bigbench_generate_until
• bigbench_multiple_choice_a
• bigbench_multiple_choice_b

Tasks

• task_name: 1-sentence description of what this particular task does
• task_name2: ...

Checklist Copy Section

For adding novel benchmarks/datasets to the library:

• Is the task an existing benchmark in the literature?
  • Have you referenced the original paper that introduced the task?
  • If yes, does the original paper provide a reference implementation? If so, have you checked against the reference implementation and documented how to run such a test?

If other tasks on this dataset are already supported:

• Is the "Main" variant of this task clearly denoted?
• Have you provided a short sentence in a README on what each new variant adds / evaluates?
• Have you noted which, if any, published evaluation setups are matched by this variant?

Overview (`lm_eval/tasks/blimp/README.md`) Copy Section

Task-name

Paper Copy Section

Title: BLiMP: A Benchmark of Linguistic Minimal Pairs for English
Abstract: https://arxiv.org/abs/1912.00582

BLiMP is a challenge set for evaluating what language models (LMs) know about
major grammatical phenomena in English. BLiMP consists of 67 sub-datasets, each
containing 1000 minimal pairs isolating specific contrasts in syntax, morphology,
or semantics. The data is automatically generated according to expert-crafted
grammars.

Homepage: https://github.com/alexwarstadt/blimp

Citation Copy Section

/* Detailed source-code truncated for AI context efficiency. */

Subtasks Copy Section

List or describe tasks defined in this folder, and their names here:

• task_name: 1-sentence description of what this particular task does
• task_name2: .....

Checklist Copy Section

For adding novel benchmarks/datasets to the library:

• Is the task an existing benchmark in the literature?
  • Have you referenced the original paper that introduced the task?
  • If yes, does the original paper provide a reference implementation? If so, have you checked against the reference implementation and documented how to run such a test?

If other tasks on this dataset are already supported:

• Is the "Main" variant of this task clearly denoted?
• Have you provided a short sentence in a README on what each new variant adds / evaluates?
• Have you noted which, if any, published evaluation setups are matched by this variant?

Overview (`lm_eval/tasks/blimp_nl/README.md`) Copy Section

BLiMP-NL: A Corpus of Dutch Minimal Pairs and Acceptability Judgments for Language Model Evaluation

Paper Copy Chapter

Title: BLiMP-NL: A Corpus of Dutch Minimal Pairs and Acceptability Judgments for Language Model Evaluation

Abstract:

[A] corpus of 8400 Dutch sentence pairs, intended primarily for the grammatical evaluation of language models. Each pair consists of a grammatical sentence and a minimally different ungrammatical sentence. The corpus covers 84 paradigms, classified into 22 syntactic phenomena. Ten sentence pairs of each paradigm were created by hand, while the remaining 90 were generated semi-automatically and manually validated afterwards.
(Suijkerbuijk et al., 2025)

Homepage: https://data.ru.nl/collections/ru/cls/blimp-nl_dsc_550

Citation Copy Section

@article{10.1162/coli_a_00559,
    author = {Suijkerbuijk, Michelle and Prins, Zo{\"e} and de Heer Kloots, Marianne and Zuidema, Willem and Frank, Stefan L.},
    title = {BLiMP-NL: A Corpus of Dutch Minimal Pairs and Acceptability Judgments for Language Model Evaluation},
    journal = {Computational Linguistics},
    pages = {1-35},
    year = {2025},
    month = {05},
    issn = {0891-2017},
    doi = {10.1162/coli_a_00559},
    url = {https://doi.org/10.1162/coli\_a\_00559},
}

Groups, Tags, and Tasks Copy Section

Groups

• blimp_nl: Runs all tasks of the large BLiMP-NL benchmark

Phenomena (runs all paradigms within each phenomenon and calculates the mean across all of them):

• blimp_nl__adpositional_phrases: "This covers the characteristics of different types of adpositional phrases, such as the PP-complement of a noun phrase or containing an R-word." (Suijkerbuijk et al., 2025).
• blimp_nl__adverbial_modification: "This covers the position of adverbs in the sentence." (Suijkerbuijk et al., 2025).
• blimp_nl__anaphor_agreement: "This covers the requirement that reflexive pronouns such as mezelf ('myself') agree with their antecedents in person and number." (Suijkerbuijk et al., 2025).
• blimp_nl__argument_structure: This covers the different verb types and their characteristics, such as the number of arguments (in-/di-)transitive verbs take and the specific auxiliary (a)telic unaccusative and NOM-DAT verbs select." (Suijkerbuijk et al., 2025).
• blimp_nl__auxiliaries: "This covers the different types of auxiliary verbs and their behavior." (Suijkerbuijk et al., 2025).
• blimp_nl__binding_principle_a: " This covers the structural relationship between the reflexive pronoun and its antecedent." (Suijkerbuijk et al., 2025).
• blimp_nl__complementive: "This covers the possibility of having secondary predication on (in-/di)transitive verbs and the position of that predication." (Suijkerbuijk et al., 2025).
• blimp_nl__crossing_dependencies: "This covers the specific feature that verbs and arguments are ordered cross-serially." (Suijkerbuijk et al., 2025).
• blimp_nl__determiners: "This covers the special determiner geen ('no') and its characteristics." (Suijkerbuijk et al., 2025).
• blimp_nl__extraposition: " This covers the possibility of extraposing nouns and adverbs" (Suijkerbuijk et al., 2025).
• blimp_nl__finite_argument_clause: "This covers the argument clause that is finite, and specifically the obligatory complementizer, the position of the clause, and the verbs that select this clause." (Suijkerbuijk et al., 2025).
• blimp_nl__infinitival_argument_clause: " This covers the argument clause that is infinitival, and specifically the verbs that select this clause and the differences between the infinitival markers te and om te." (Suijkerbuijk et al., 2025).
• blimp_nl__nominalization: "This covers the ways in which words from different categories can be turned into nouns." (Suijkerbuijk et al., 2025).
• blimp_nl__parasitic_gaps: "This covers the characteristics of parasitic gap formation." (Suijkerbuijk et al., 2025).
• blimp_nl__passive: "This covers the formation of the impersonal and regular passive construction." (Suijkerbuijk et al., 2025).
• blimp_nl__quantifiers: " This covers the behavior of quantifiers, specifically their agreement with nouns and verbs." (Suijkerbuijk et al., 2025).
• blimp_nl__r_words: "This covers the formation and extraction of R-words (e.g., daar and er)." (Suijkerbuijk et al., 2025).
• blimp_nl__relativization: "This covers the characteristics of relativization and the restrictions thereon." (Suijkerbuijk et al., 2025).
• blimp_nl__topicalization: "This covers the characteristics of topicalization and the restrictions thereon." (Suijkerbuijk et al., 2025).
• blimp_nl__verb_second: "This covers the different word order restrictions in main and embedded clauses." (Suijkerbuijk et al., 2025).
• blimp_nl__wh_movement: "This covers the requirements for wh-movement and the related phenomenon stranding." (Suijkerbuijk et al., 2025).
• blimp_nl__wh_movement_restrictions: "This covers the restrictions that exist on wh-movement, such as island and superiority constraints." (Suijkerbuijk et al., 2025).

Each of these is further divided into specific experimental paradigms (which here are represented as individual tasks; 100 items each), which are described in the Suijkerbuijk et al., (2025).

Implementation note: The original implementation as discussed in the paper uses masked language models and compares syntactic log-odds ratios (SLOG; Pauls & Klein, 2012) between sentences, which normalizes for word frequency. Neither masked langauge models nor SLOG are currently supported by the Harness, and so the implementation provided here includes both un-normalized accuracy (acc) and byte-length-normalized accuracy (acc_norm).

Checklist Copy Section

For adding novel benchmarks/datasets to the library:

• Is the task an existing benchmark in the literature?
  • Have you referenced the original paper that introduced the task?
  • If yes, does the original paper provide a reference implementation? If so, have you checked against the reference implementation and documented how to run such a test?

Changelog Copy Section

Overview (`lm_eval/tasks/c4/README.md`) Copy Section

Colossal Clean Crawled Corpus(C4)

Paper Copy Section

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

A colossal, cleaned version of Common Crawl's web crawl corpus. Based on Common Crawl dataset.

This is the processed version of Google's C4 dataset.

Homepage

Citation Copy Section

@misc{raffel2023exploringlimitstransferlearning,
      title={Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer},
      author={Colin Raffel and Noam Shazeer and Adam Roberts and Katherine Lee and Sharan Narang and Michael Matena and Yanqi Zhou and Wei Li and Peter J. Liu},
      year={2023},
      eprint={1910.10683},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/1910.10683},
}

Groups, Tags, and Tasks Copy Section

Groups

• Not part of a group yet.

Tasks

• c4: measure perplexity on the C4 dataset, via rolling loglikelihoods.

Checklist Copy Section

For adding novel benchmarks/datasets to the library:

• Is the task an existing benchmark in the literature?
  • Have you referenced the original paper that introduced the task?
  • If yes, does the original paper provide a reference implementation? If so, have you checked against the reference implementation and documented how to run such a test?

If other tasks on this dataset are already supported:

• Is the "Main" variant of this task clearly denoted?
• Have you provided a short sentence in a README on what each new variant adds / evaluates?
• Have you noted which, if any, published evaluation setups are matched by this variant?

Changelog Copy Section

Overview (`lm_eval/tasks/cabbq/README.md`) Copy Section

Catalan Bias Benchmark for Question Answering (CaBBQ)

Paper Copy Section

Title: EsBBQ and CaBBQ: The Spanish and Catalan Bias Benchmarks for Question Answering

Abstract: https://arxiv.org/abs/2507.11216

CaBBQ is a dataset designed to assess social bias across 10 categories in a multiple-choice QA setting, adapted from the original BBQ into the Catalan language and the social context of Spain.

It is fully parallel with the esbbq task group, the version in Spanish.

Citation Copy Section

@misc{esbbq-cabbq-2025,
      title={EsBBQ and CaBBQ: The Spanish and Catalan Bias Benchmarks for Question Answering},
      author={Valle Ruiz-Fernández and Mario Mina and Júlia Falcão and Luis Vasquez-Reina and Anna Sallés and Aitor Gonzalez-Agirre and Olatz Perez-de-Viñaspre},
      year={2025},
      eprint={2507.11216},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2507.11216},
}

Groups and Tasks Copy Section

Groups

• cabbq: Contains the subtasks that covers all demographic categories.

Tasks Copy Section

for category in ["age", "disability_status", "gender", "lgbtqia", "nationality", "physical_appearance", "race_ethnicity", "religion", "ses", "spanish_region"]:

• cabbq_{category}: Subtask that evaluates on the given category's subset.

Metrics Copy Section

CaBBQ is evaluated with the following 4 metrics, at the level of each subtask and with aggregated values for the entire group:

• acc_ambig: Accuracy over ambiguous instances.
• acc_disambig: Accuracy over disambiguated instances.
• bias_score_ambig: Bias score over ambiguous instances.
• bias_score_disambig: Bias score over disambiguated instances.

See the paper for a thorough explanation and the formulas of these metrics.

Checklist Copy Section

For adding novel benchmarks/datasets to the library:

• Is the task an existing benchmark in the literature?
  • Have you referenced the original paper that introduced the task?
  • If yes, does the original paper provide a reference implementation? If so, have you checked against the reference implementation and documented how to run such a test?

If other tasks on this dataset are already supported:

• Is the "Main" variant of this task clearly denoted?
• Have you provided a short sentence in a README on what each new variant adds / evaluates?
• Have you noted which, if any, published evaluation setups are matched by this variant?

Overview (`lm_eval/tasks/careqa/README.md`) Copy Section

CareQA

Paper Copy Section

Title: Automatic Evaluation of Healthcare LLMs Beyond Question-Answering

Abstract: https://arxiv.org/abs/2502.06666

CareQA originates from the Spanish Specialised Healthcare Training (MIR) exams by the
Spanish Ministry of Health. The close-ended version is a multiple-choice question
answering (MCQA) including 5,621 QA pairs across six categories: medicine, nursing,
biology, chemistry, psychology, and pharmacology, sourced from the 2020 to 2024 exam
editions. CareQA is available in both English and Spanish. The open-ended version
(English only) contains 3,730 QA pairs.

Homepage:
https://huggingface.co/datasets/HPAI-BSC/CareQA

Tasks

• careqa_en: MCQA in english.
• careqa_es: MCQA in spanish.
• careqa_open: Open-Ended QA in english.
• careqa_open_perplexity: Open-Ended QA in english, evaluated with perplexity.

Citation Copy Section

@misc{ariasduart2025automaticevaluationhealthcarellms,
      title={Automatic Evaluation of Healthcare LLMs Beyond Question-Answering},
      author={Anna Arias-Duart and Pablo Agustin Martin-Torres and Daniel Hinjos and Pablo Bernabeu-Perez and Lucia Urcelay Ganzabal and Marta Gonzalez Mallo and Ashwin Kumar Gururajan and Enrique Lopez-Cuena and Sergio Alvarez-Napagao and Dario Garcia-Gasulla},
      year={2025},
      eprint={2502.06666},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2502.06666},
}

Overview (`lm_eval/tasks/catalan_bench/README.md`) Copy Section

CatalanBench

Paper Copy Section

CatalanBench is a benchmark for evaluating language models in Catalan tasks. This is, it evaluates the ability of a language model to understand and generate Catalan text. CatalanBench offers a combination of pre-existing, open datasets and datasets developed exclusivelly for this benchmark. All the details of CatalanBench will be published in a paper soon.

The new evaluation datasets included in CatalanBench are:

The datasets included in CatalanBench that have been made public in previous pubications are:

Citation Copy Section

Paper for CatalanBench coming soon.

@inproceedings{baucells-etal-2025-iberobench,
    title = "{I}bero{B}ench: A Benchmark for {LLM} Evaluation in {I}berian Languages",
    author = "Baucells, Irene  and
      Aula-Blasco, Javier  and
      de-Dios-Flores, Iria  and
      Paniagua Su{\'a}rez, Silvia  and
      Perez, Naiara  and
      Salles, Anna  and
      Sotelo Docio, Susana  and
      Falc{\~a}o, J{\'u}lia  and
      Saiz, Jose Javier  and
      Sepulveda Torres, Robiert  and
      Barnes, Jeremy  and
      Gamallo, Pablo  and
      Gonzalez-Agirre, Aitor  and
      Rigau, German  and
      Villegas, Marta",
    editor = "Rambow, Owen  and
      Wanner, Leo  and
      Apidianaki, Marianna  and
      Al-Khalifa, Hend  and
      Eugenio, Barbara Di  and
      Schockaert, Steven",
    booktitle = "Proceedings of the 31st International Conference on Computational Linguistics",
    month = jan,
    year = "2025",
    address = "Abu Dhabi, UAE",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2025.coling-main.699/",
    pages = "10491--10519",
}

Groups and Tasks Copy Section

Groups

• catalan_bench: All tasks included in CatalanBench.
• flores_ca: All FLORES translation tasks from or to Catalan.

Tags

• cabreu: Three CaBREU tasks for each type of summary (extractive, abstractive and extreme).
• phrases_va: Two Phrases_va tasks for language adaptation between Catalan and Valencian.

Tasks

The following tasks evaluate tasks on CatalanBench dataset using various scoring methods.

• arc_ca_challenge
• arc_ca_easy
• belebele_cat_Latn
• cabreu
• catalanqa
• catcola
• cieacova
• cocoteros_va
• copa_ca
• coqcat
• flores_ca
• flores_ca-de
• flores_ca-en
• flores_ca-es
• flores_ca-eu
• flores_ca-fr
• flores_ca-gl
• flores_ca-it
• flores_ca-pt
• flores_de-ca
• flores_en-ca
• flores_es-ca
• flores_eu-ca
• flores_fr-ca
• flores_gl-ca
• flores_it-ca
• flores_pt-ca
• mgsm_direct_ca
• openbookqa_ca
• parafraseja
• paws_ca
• phrases_ca
• piqa_ca
• siqa_ca
• teca
• terretaqa
• truthfulqa_va
• veritasqa_gen_ca
• veritasqa_mc1_ca
• veritasqa_mc2_ca
• wnli_ca
• xnli_ca
• xnli_va
• xquad_ca
• xstorycloze_ca

Some of these tasks are taken from benchmarks already available in LM Evaluation Harness. These are:

• belebele_cat_Latn: Belebele Catalan

Checklist Copy Section

• Is the task an existing benchmark in the literature?
  • Have you referenced the original paper that introduced the task?
  • If yes, does the original paper provide a reference implementation?
    • Yes, original implementation contributed by author of the benchmark

If other tasks on this dataset are already supported:

• Is the "Main" variant of this task clearly denoted?
• Have you provided a short sentence in a README on what each new variant adds / evaluates?
• Have you noted which, if any, published evaluation setups are matched by this variant?

Changelog Copy Section

version 2.0: (2025-Mar-18) add cococteros_va task.
version 2.1: (2025-Jul-30) add xnli_va task.
version 2.2: (2025-Jul-30) add truthfulqa_va task.
version 2.3: (2026-Jan-16) exclude line breaks from stop criteria in mgsm_direct_ca
version 2.4: (2026-Aug-24) add terretaqa task.
version 2.5: (2026-Aug-24) add CieaCOVA

Overview (`lm_eval/tasks/ceval/README.md`) Copy Section

C-Eval (Validation)

Paper Copy Section

C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models
https://arxiv.org/pdf/2305.08322.pdf

C-Eval is a comprehensive Chinese evaluation suite for foundation models.
It consists of 13948 multi-choice questions spanning 52 diverse disciplines
and four difficulty levels.

Homepage: https://cevalbenchmark.com/

Citation Copy Section

@article{huang2023ceval,
    title={C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models},
    author={Huang, Yuzhen and Bai, Yuzhuo and Zhu, Zhihao and Zhang, Junlei and Zhang, Jinghan and Su, Tangjun and Liu, Junteng and Lv, Chuancheng and Zhang, Yikai and Lei, Jiayi and Fu, Yao and Sun, Maosong and He, Junxian},
    journal={arXiv preprint arXiv:2305.08322},
    year={2023}
}

SUBJECTS = {
"computer_network":"计算机网络",
"operating_system":"操作系统",
"computer_architecture":"计算机组成",
"college_programming":"大学编程",
"college_physics":"大学物理",
"college_chemistry":"大学化学",
"advanced_mathematics":"高等数学",
"probability_and_statistics":"概率统计",
"discrete_mathematics":"离散数学",
"electrical_engineer":"注册电气工程师",
"metrology_engineer":"注册计量师",
"high_school_mathematics":"高中数学",
"high_school_physics":"高中物理",
"high_school_chemistry":"高中化学",
"high_school_biology":"高中生物",
"middle_school_mathematics":"初中数学",
"middle_school_biology":"初中生物",
"middle_school_physics":"初中物理",
"middle_school_chemistry":"初中化学",
"veterinary_medicine":"兽医学",
"college_economics":"大学经济学",
"business_administration":"工商管理",
"marxism":"马克思主义基本原理",
"mao_zedong_thought":"毛泽东思想和中国特色社会主义理论体系概论",
"education_science":"教育学",
"teacher_qualification":"教师资格",
"high_school_politics":"高中政治",
"high_school_geography":"高中地理",
"middle_school_politics":"初中政治",
"middle_school_geography":"初中地理",
"modern_chinese_history":"近代史纲要",
"ideological_and_moral_cultivation":"思想道德修养与法律基础",
"logic":"逻辑学",
"law":"法学",
"chinese_language_and_literature":"中国语言文学",
"art_studies":"艺术学",
"professional_tour_guide":"导游资格",
"legal_professional":"法律职业资格",
"high_school_chinese":"高中语文",
"high_school_history":"高中历史",
"middle_school_history":"初中历史",
"civil_servant":"公务员",
"sports_science":"体育学",
"plant_protection":"植物保护",
"basic_medicine":"基础医学",
"clinical_medicine":"临床医学",
"urban_and_rural_planner":"注册城乡规划师",
"accountant":"注册会计师",
"fire_engineer":"注册消防工程师",
"environmental_impact_assessment_engineer":"环境影响评价工程师",
"tax_accountant":"税务师",
"physician":"医师资格"
}

CMMLU

Paper Copy Section

CMMLU: Measuring massive multitask language understanding in Chinese
https://arxiv.org/abs/2306.09212

CMMLU is a comprehensive evaluation benchmark specifically designed to evaluate the knowledge and reasoning abilities of LLMs within the context of Chinese language and culture.
CMMLU covers a wide range of subjects, comprising 67 topics that span from elementary to advanced professional levels.

Homepage: https://github.com/haonan-li/CMMLU

Citation Copy Section

@misc{li2023cmmlu,
      title={CMMLU: Measuring massive multitask language understanding in Chinese},
      author={Haonan Li and Yixuan Zhang and Fajri Koto and Yifei Yang and Hai Zhao and Yeyun Gong and Nan Duan and Timothy Baldwin},
      year={2023},
      eprint={2306.09212},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

Groups and Tasks Copy Section

Groups

• ceval-valid: All 52 subjects of the C-Eval dataset, evaluated following the methodology in MMLU's original implementation. This implementation consists solely of the validation set of C-Eval, as the test set requires submission of model predictions to an external site.

Tasks

The following tasks evaluate subjects in the C-Eval dataset using loglikelihood-based multiple-choice scoring:

• ceval-valid_{subject_english}

Checklist Copy Section

• Is the task an existing benchmark in the literature?
  • Have you referenced the original paper that introduced the task?
  • If yes, does the original paper provide a reference implementation?

If other tasks on this dataset are already supported:

• Is the "Main" variant of this task clearly denoted?
• Have you provided a short sentence in a README on what each new variant adds / evaluates?
• Have you noted which, if any, published evaluation setups are matched by this variant?

Overview (`lm_eval/tasks/chartqa/README.md`) Copy Section

Task-name

Paper Copy Section

Title: ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning

Abstract: In this work, we present a large-scale benchmark covering 9.6K human-written questions as well as 23.1K questions generated from human-written chart summaries.

Short description of paper / benchmark goes here:

Homepage: https://github.com/vis-nlp/ChartQA

Citation Copy Section

@misc{masry2022chartqabenchmarkquestionanswering,
      title={ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning},
      author={Ahmed Masry and Do Xuan Long and Jia Qing Tan and Shafiq Joty and Enamul Hoque},
      year={2022},
      eprint={2203.10244},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2203.10244},
}

Groups, Tags, and Tasks Copy Section

Tasks

• chartqa: Prompt taken from on mistral-evals: https://github.com/mistralai/mistral-evals/blob/main/eval/tasks/chartqa.py
• chartqa_llama: variant as implemented in https://github.com/meta-llama/llama-models/blob/main/models/llama3_3/eval_details.md
• chartqa_llama_90: similar to chartqa_llama but specific to the 90B models of llama 3.2

Checklist Copy Section

For adding novel benchmarks/datasets to the library:

• Is the task an existing benchmark in the literature?
  • Have you referenced the original paper that introduced the task?
  • If yes, does the original paper provide a reference implementation? If so, have you checked against the reference implementation and documented how to run such a test?

If other tasks on this dataset are already supported:

• Is the "Main" variant of this task clearly denoted?
• Have you provided a short sentence in a README on what each new variant adds / evaluates?
• Have you noted which, if any, published evaluation setups are matched by this variant?

Changelog Copy Section

Overview (`lm_eval/tasks/click/README.md`) Copy Section

click

Paper Copy Section

Title: CLIcK: A Benchmark Dataset of Cultural and Linguistic Intelligence in Korean

Abstract: Despite the rapid development of large language models (LLMs) for the Korean language, there remains an obvious lack of benchmark datasets that test the requisite Korean cultural and linguistic knowledge. Because many existing Korean benchmark datasets are derived from the English counterparts through translation, they often overlook the different cultural contexts. For the few benchmark datasets that are sourced from Korean data capturing cultural knowledge, only narrow tasks such as bias and hate speech detection are offered. To address this gap, we introduce a benchmark of Cultural and Linguistic Intelligence in Korean (CLIcK), a dataset comprising 1,995 QA pairs. CLIcK sources its data from official Korean exams and textbooks, partitioning the questions into eleven categories under the two main categories of language and culture. For each instance in CLIcK, we provide fine-grained annotation of which cultural and linguistic knowledge is required to answer the question correctly. Using CLIcK, we test 13 language models to assess their performance. Our evaluation uncovers insights into their performances across the categories, as well as the diverse factors affecting their comprehension. CLIcK offers the first large-scale comprehensive Korean-centric analysis of LLMs' proficiency in Korean culture and language.

Homepage: https://huggingface.co/datasets/EunsuKim/CLIcK

Citation Copy Section

@misc{kim2024click,
      title={CLIcK: A Benchmark Dataset of Cultural and Linguistic Intelligence in Korean},
      author={Eunsu Kim and Juyoung Suk and Philhoon Oh and Haneul Yoo and James Thorne and Alice Oh},
      year={2024},
      eprint={2403.06412},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

Groups, Tags, and Tasks Copy Section

Groups

• click: All 11 categories of the CLIcK dataset
• click_lang: "Language" category of the CLIcK dataset, consisting of 3 subcategories
• click_cul: "Culture" category of the CLIcK dataset, consisting of 8 subcategories

Tasks

• Three tasks under click_lang:

  • click_lang_text
  • click_lang_grammar
  • click_lang_function

• Eight tasks under click_cul:

  • click_cul_society
  • click_cul_tradition
  • click_cul_politics
  • click_cul_economy
  • click_cul_law
  • click_cul_history
  • click_cul_geography
  • click_cul_kpop

Checklist Copy Section

For adding novel benchmarks/datasets to the library:

• Is the task an existing benchmark in the literature?
  • Have you referenced the original paper that introduced the task?
  • If yes, does the original paper provide a reference implementation? If so, have you checked against the reference implementation and documented how to run such a test?

If other tasks on this dataset are already supported:

• Is the "Main" variant of this task clearly denoted?
• Have you provided a short sentence in a README on what each new variant adds / evaluates?
• Have you noted which, if any, published evaluation setups are matched by this variant?

Overview (`lm_eval/tasks/cmmlu/README.md`) Copy Section

CMMLU

Paper Copy Section

CMMLU: Measuring massive multitask language understanding in Chinese
https://arxiv.org/abs/2306.09212

CMMLU is a comprehensive evaluation benchmark specifically designed to evaluate the knowledge and reasoning abilities of LLMs within the context of Chinese language and culture.
CMMLU covers a wide range of subjects, comprising 67 topics that span from elementary to advanced professional levels.

Homepage: https://github.com/haonan-li/CMMLU

Citation Copy Section

@misc{li2023cmmlu,
      title={CMMLU: Measuring massive multitask language understanding in Chinese},
      author={Haonan Li and Yixuan Zhang and Fajri Koto and Yifei Yang and Hai Zhao and Yeyun Gong and Nan Duan and Timothy Baldwin},
      year={2023},
      eprint={2306.09212},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

Groups and Tasks Copy Section

Groups

• cmmlu: All 67 subjects of the CMMLU dataset, evaluated following the methodology in MMLU's original implementation.

Tasks

The following tasks evaluate subjects in the CMMLU dataset using loglikelihood-based multiple-choice scoring:

• cmmlu_{subject_english}

Checklist Copy Section

• Is the task an existing benchmark in the literature?
  • Have you referenced the original paper that introduced the task?
  • If yes, does the original paper provide a reference implementation?
    • Yes, original implementation contributed by author of the benchmark

If other tasks on this dataset are already supported:

• Is the "Main" variant of this task clearly denoted?
• Have you provided a short sentence in a README on what each new variant adds / evaluates?
• Have you noted which, if any, published evaluation setups are matched by this variant?

Overview (`lm_eval/tasks/cnn_dailymail/README.md`) Copy Section

CNN-DailyMail

Paper Copy Chapter

Teaching Machines to Read and Comprehend https://arxiv.org/abs/1506.03340

The CNN/DailyMail dataset is an English-language dataset containing just over 300k unique news articles as written by journalists at CNN and the Daily Mail. It is widely used for abstractive text summarization tasks. This task uses the non-anonymized Version 3.0.0.

Homepage: https://huggingface.co/datasets/cnn_dailymail

Citation Copy Chapter

@inproceedings{hermann2015teaching,
  title={Teaching Machines to Read and Comprehend},
  author={Hermann, Karl Moritz and Kocisky, Tomas and Grefenstette, Edward and Espeholt, Lasse and Kay, Will and Suleyman, Mustafa and Blunsom, Phil},
  booktitle={Advances in Neural Information Processing Systems},
  year={2015}
}

Groups and Tasks Copy Chapter

Groups Copy Section

• summarization

Tasks Copy Section

• cnn_dailymail: The Version 3.0.0 (non-anonymized) dataset. It evaluates models on their ability to generate multi-sentence abstractive summaries. It uses a zero-shot prompt format (Summarize the following article...) and evaluates using ROUGE (1/2/L) and BERTScore (P/R/F1).

Checklist Copy Chapter

For adding novel benchmarks/datasets to the library:

• Is the task an existing benchmark in the literature?

  • Have you referenced the original paper that introduced the task?

  • If yes, does the original paper provide a reference implementation? If so, have you checked against the reference implementation and documented how to run such a test?

    • The task uses the standard 3.0.0 (non-anonymized) split from HuggingFace. The implementation uses a standard zero-shot prompt: "Summarize the following article:\n\n{{article}}\n\nSummary:".

    • Evaluation metrics include ROUGE (1, 2, L) and BERTScore (Precision, Recall, F1), matching standard reporting practices for abstractive summarization.

If other tasks on this dataset are already supported:

• Is the "Main" variant of this task clearly denoted?

• Have you provided a short sentence in a README on what each new variant adds / evaluates?

• Have you noted which, if any, published evaluation setups are matched by this variant?

Overview (`lm_eval/tasks/commonsense_qa/README.md`) Copy Section

Task-name

Paper Copy Section

Title: COMMONSENSEQA: A Question Answering Challenge Targeting Commonsense Knowledge

Abstract: https://arxiv.org/pdf/1811.00937.pdf

CommonsenseQA is a multiple-choice question answering dataset that requires different types of commonsense knowledge to predict the correct answers.
It contains 12,102 questions with one correct answer and four distractor answers.

Homepage: https://www.tau-nlp.org/commonsenseqa

Citation Copy Section

@inproceedings{talmor-etal-2019-commonsenseqa,
    title = "{C}ommonsense{QA}: A Question Answering Challenge Targeting Commonsense Knowledge",
    author = "Talmor, Alon  and
      Herzig, Jonathan  and
      Lourie, Nicholas  and
      Berant, Jonathan",
    booktitle = "Proceedings of the 2019 Conference of the North {A}merican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)",
    month = jun,
    year = "2019",
    address = "Minneapolis, Minnesota",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/N19-1421",
    doi = "10.18653/v1/N19-1421",
    pages = "4149--4158",
    archivePrefix = "arXiv",
    eprint        = "1811.00937",
    primaryClass  = "cs",
}

Groups and Tasks Copy Section

Groups

• Not part of a group yet.

Tasks

• commonsense_qa: Represents the "random" split from the paper. Uses an MMLU-style prompt, as (presumably) used by Llama evaluations.

Checklist Copy Section

For adding novel benchmarks/datasets to the library:

• Is the task an existing benchmark in the literature?
  • Have you referenced the original paper that introduced the task?
  • If yes, does the original paper provide a reference implementation? If so, have you checked against the reference implementation and documented how to run such a test?

If other tasks on this dataset are already supported:

• Is the "Main" variant of this task clearly denoted?
• Have you provided a short sentence in a README on what each new variant adds / evaluates?
• Have you noted which, if any, published evaluation setups are matched by this variant?

Overview (`lm_eval/tasks/copal_id/README.md`) Copy Section

COPAL

Paper Copy Section

Title: COPAL-ID: Indonesian Language Reasoning with Local Culture and Nuances

Abstract: https://arxiv.org/abs/2311.01012

COPAL-ID is an Indonesian causal commonsense reasoning dataset that captures local nuances. It provides a more natural portrayal of day-to-day causal reasoning within the Indonesian (especially Jakartan) cultural sphere. Professionally written and validatid from scratch by natives, COPAL-ID is more fluent and free from awkward phrases, unlike the translated XCOPA-ID.

Homepage: https://github.com/haryoa/copal-id

Citation Copy Section

@article{wibowo2023copal,
  title={COPAL-ID: Indonesian Language Reasoning with Local Culture and Nuances},
  author={Wibowo, Haryo Akbarianto and Fuadi, Erland Hilman and Nityasya, Made Nindyatama and Prasojo, Radityo Eko and Aji, Alham Fikri},
  journal={arXiv preprint arXiv:2311.01012},
  year={2023}
}

Groups and Tasks Copy Section

Groups

• copal_id

Tasks

• copal_id_standard: Standard version of COPAL dataset, use formal language and less local nuances
• copal_id_colloquial: Colloquial version of COPAL dataset, use informal language and more local nuances

Checklist Copy Section

For adding novel benchmarks/datasets to the library:

• Is the task an existing benchmark in the literature?
  • Have you referenced the original paper that introduced the task?
  • If yes, does the original paper provide a reference implementation? If so, have you checked against the reference implementation and documented how to run such a test?

If other tasks on this dataset are already supported:

• Is the "Main" variant of this task clearly denoted?
• Have you provided a short sentence in a README on what each new variant adds / evaluates?
• Have you noted which, if any, published evaluation setups are matched by this variant?

Overview (`lm_eval/tasks/coqa/README.md`) Copy Section

CoQA

Paper Copy Section

Title: CoQA: A Conversational Question Answering Challenge

Abstract: https://arxiv.org/pdf/1808.07042.pdf

CoQA is a large-scale dataset for building Conversational Question Answering
systems. The goal of the CoQA challenge is to measure the ability of machines to
understand a text passage and answer a series of interconnected questions that
appear in a conversation.

Homepage: https://stanfordnlp.github.io/coqa/

Citation Copy Section

BibTeX-formatted citation goes here

Groups and Tasks Copy Section

Groups

• Not part of a group yet

Tasks

• coqa

Checklist Copy Section

For adding novel benchmarks/datasets to the library:

• Is the task an existing benchmark in the literature?
  • Have you referenced the original paper that introduced the task?
  • If yes, does the original paper provide a reference implementation? If so, have you checked against the reference implementation and documented how to run such a test?

If other tasks on this dataset are already supported:

• Is the "Main" variant of this task clearly denoted?
• Have you provided a short sentence in a README on what each new variant adds / evaluates?
• Have you noted which, if any, published evaluation setups are matched by this variant?

Overview (`lm_eval/tasks/crows_pairs/README.md`) Copy Section

CrowS-Pairs

Paper Copy Section

CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models
https://aclanthology.org/2020.emnlp-main.154/
French CrowS-Pairs: Extending a challenge dataset for measuring social bias in masked
language models to a language other than English
https://aclanthology.org/2022.acl-long.583/

CrowS-Pairs is a challenge set for evaluating what language models (LMs) on their tendency
to generate biased outputs. CrowS-Pairs comes in 2 languages and the English subset has
a newer version which fixes some of the issues with the original version.

Homepage: https://github.com/nyu-mll/crows-pairs, https://gitlab.inria.fr/french-crows-pairs

Citation Copy Section

/* Detailed source-code truncated for AI context efficiency. */

Groups and Tasks Copy Section

Groups

• crows_pairs_english: The entire English subset of the CrowS-Pairs dataset.
• crows_pairs_french: The entire French subset of the CrowS-Pairs dataset.

Tasks

The following tasks evaluate sub-areas of bias in the English CrowS-Pairs dataset:

• crows_pairs_english_age
• crows_pairs_english_autre
• crows_pairs_english_disability
• crows_pairs_english_gender
• crows_pairs_english_nationality
• crows_pairs_english_physical_appearance
• crows_pairs_english_race_color
• crows_pairs_english_religion
• crows_pairs_english_sexual_orientation
• crows_pairs_english_socioeconomic

The following tasks evaluate sub-areas of bias in the French CrowS-Pairs dataset:

• crows_pairs_french_age
• crows_pairs_french_autre
• crows_pairs_french_disability
• crows_pairs_french_gender
• crows_pairs_french_nationality
• crows_pairs_french_physical_appearance
• crows_pairs_french_race_color
• crows_pairs_french_religion
• crows_pairs_french_sexual_orientation
• crows_pairs_french_socioeconomic

All tasks evaluate the percentage of more-stereotypical sentences that are rated as more likely by a model than the non-stereotypical sentences (pct_stereotype), as well as the average absolute difference of loglikelihoods between the sentences in the pairs.

Checklist Copy Section

• Is the task an existing benchmark in the literature?
  • Have you referenced the original paper that introduced the task?
  • If yes, does the original paper provide a reference implementation?
    • The original paper does not for causal language models, so this is a novel formulation of the task for autoregressive LMs.

If other tasks on this dataset are already supported:

• Is the "Main" variant of this task clearly denoted?
• Have you provided a short sentence in a README on what each new variant adds / evaluates?
• Have you noted which, if any, published evaluation setups are matched by this variant?

Overview (`lm_eval/tasks/cruxeval/README.md`) Copy Section

CRUXEval

Paper Copy Chapter

CRUXEval: A Benchmark for Code Reasoning, Understanding and Execution

https://arxiv.org/abs/2401.03065

CRUXEval is a benchmark of 800 Python functions paired with input-output pairs. It evaluates a model's ability to reason about code execution in two directions: given a function and input, predict the output (cruxeval_output); and given a function and output, find a valid input (cruxeval_input). The benchmark includes chain-of-thought variants that prompt the model to reason step by step before answering.

Homepage: https://crux-eval.github.io

Citation Copy Chapter

@article{gu2024cruxeval,
  title={CRUXEval: A Benchmark for Code Reasoning, Understanding and Execution},
  author={Alex Gu and Baptiste Rozière and Hugh Leather and Armando Solar-Lezama and Gabriel Synnaeve and Sida I. Wang},
  year={2024},
  eprint={2401.03065},
  archivePrefix={arXiv},
  primaryClass={cs.SE}
}

Tasks

• cruxeval_input: Input prediction — given a function and its output, predict a valid input. pass@1 at temperature 0.2. Matches the published evaluation setup from the paper.
• cruxeval_input_08: Input prediction at temperature 0.8 for pass@5 evaluation. Matches the published pass@5 setup from the paper.
• cruxeval_input_cot: Input prediction with chain-of-thought reasoning prompting.
• cruxeval_output: Output prediction — given a function and its input, predict the output. pass@1 at temperature 0.2. Matches the published evaluation setup from the paper.
• cruxeval_output_08: Output prediction at temperature 0.8 for pass@5 evaluation. Matches the published pass@5 setup from the paper.
• cruxeval_output_cot: Output prediction with chain-of-thought reasoning prompting.

Checklist Copy Section

For adding novel benchmarks/datasets to the library:

• Is the task an existing benchmark in the literature?
  • Have you referenced the original paper that introduced the task?
  • If yes, does the original paper provide a reference implementation? If so, have you checked against the reference implementation and documented how to run such a test?

If other tasks on this dataset are already supported:

• Is the "Main" variant of this task clearly denoted?
• Have you provided a short sentence in a README on what each new variant adds / evaluates?
• Have you noted which, if any, published evaluation setups are matched by this variant?

Changelog Copy Section

v1.0: Initial implementation with input/output prediction tasks and chain-of-thought variants.

Overview (`lm_eval/tasks/darija_bench/README.md`) Copy Section

DarijaBench

Paper Copy Section

Title: Atlas-Chat: Adapting Large Language Models for Low-Resource Moroccan Arabic Dialect

Abstract: https://arxiv.org/abs/2409.17912

DarijaBench, a comprehensive evaluation dataset tailored for Moroccan Darija. DarijaBench includes different datasets for core NLP tasks such as translation (based on four datasets, DODa-10K, FLORES+, NLLB-Seed and MADAR), summarization (based on MArSum) and, sentiment analysis (based on five datasets, MAC, MYC, MSAC, MSDA and, ElectroMorocco2016), in addition to a new transliteration task to convert between Darija (written in Arabic letters) and Arabizi (written in Latin letters) it is based on DODa-10K dataset.

Homepage: https://huggingface.co/datasets/MBZUAI-Paris/DarijaBench

Citation Copy Section

@article{shang2024atlaschatadaptinglargelanguage,
      title={Atlas-Chat: Adapting Large Language Models for Low-Resource Moroccan Arabic Dialect},
      author={Guokan Shang and Hadi Abdine and Yousef Khoubrane and Amr Mohamed and Yassine Abbahaddou and Sofiane Ennadir and Imane Momayiz and Xuguang Ren and Eric Moulines and Preslav Nakov and Michalis Vazirgiannis and Eric Xing},
      year={2024},
      eprint={2409.17912},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2409.17912},
}

Groups and Tasks Copy Section

Groups

• darija_sentiment: evaluates all Darija sentiment analysis tasks.
• darija_summarization: evaluates Darija summarization task.
• darija_translation: evaluates all Darija Translation tasks.
• darija_transliteration: evaluates Darija transliteration task.

Tasks

• darija_sentiment_mac: evaluates Darija translation task from MAC dataset.
• darija_sentiment_myc: evaluates Darija translation task from MYC dataset.
• darija_sentiment_msac: evaluates Darija translation task from MSAC dataset.
• darija_sentiment_msda: evaluates Darija translation task from MSDA dataset.
• darija_sentiment_electrom: evaluates Darija translation task from ElectroMorocco2016 dataset.
• darija_summarization_task: evaluates Darija summarization task from MArSum corpus.
• darija_translation_doda: evaluates Darija translation task from DODa-10k corpus.
• darija_translation_flores: evaluates Darija translation task from FLORES+ dataset.
• darija_translation_madar: evaluates Darija translation task from MADAR dataset.
• darija_translation_seed: evaluates Darija translation task from NLLB-Seed datasets.
• darija_transliteration_task: evaluates Darija transliteration task from DODa-10K corpus.

Note: depending on the model, padding and padding side could affect the results. The default padding side in this library is forced to left. Use batch size equal to 1 to avoid problems.

Checklist Copy Section

For adding novel benchmarks/datasets to the library:

• Is the task an existing benchmark in the literature?
  • Have you referenced the original paper that introduced the task?
  • If yes, does the original paper provide a reference implementation? If so, have you checked against the reference implementation and documented how to run such a test?

If other tasks on this dataset are already supported:

• Is the "Main" variant of this task clearly denoted?
• Have you provided a short sentence in a README on what each new variant adds / evaluates?
• Have you noted which, if any, published evaluation setups are matched by this variant?

Overview (`lm_eval/tasks/darijahellaswag/README.md`) Copy Section

DarijaHellaSwag

Paper Copy Section

Title: Atlas-Chat: Adapting Large Language Models for Low-Resource Moroccan Arabic Dialect

Abstract: https://arxiv.org/abs/2409.17912

DarijaHellaSwag is a challenging multiple-choice benchmark designed to evaluate machine reading comprehension and commonsense reasoning in Moroccan Darija. It is a translated version of the HellaSwag validation set, which presents scenarios where models must choose the most plausible continuation of a passage from four options.

Homepage: https://huggingface.co/datasets/MBZUAI-Paris/DarijaHellaSwag

Citation Copy Section

@article{shang2024atlaschatadaptinglargelanguage,
      title={Atlas-Chat: Adapting Large Language Models for Low-Resource Moroccan Arabic Dialect},
      author={Guokan Shang and Hadi Abdine and Yousef Khoubrane and Amr Mohamed and Yassine Abbahaddou and Sofiane Ennadir and Imane Momayiz and Xuguang Ren and Eric Moulines and Preslav Nakov and Michalis Vazirgiannis and Eric Xing},
      year={2024},
      eprint={2409.17912},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2409.17912},
}

Groups and Tasks Copy Section

Groups

• Not part of a group yet

Tasks

• darijahellaswag

Checklist Copy Section

For adding novel benchmarks/datasets to the library:

• Is the task an existing benchmark in the literature?
  • Have you referenced the original paper that introduced the task?
  • If yes, does the original paper provide a reference implementation? If so, have you checked against the reference implementation and documented how to run such a test?

If other tasks on this dataset are already supported:

• Is the "Main" variant of this task clearly denoted?
• Have you provided a short sentence in a README on what each new variant adds / evaluates?
• Have you noted which, if any, published evaluation setups are matched by this variant?

Overview (`lm_eval/tasks/darijammlu/README.md`) Copy Section

DarijaMMLU

Paper Copy Section

Title: Atlas-Chat: Adapting Large Language Models for Low-Resource Moroccan Arabic Dialect

Abstract: https://arxiv.org/abs/2409.17912

DarijaMMLU is an evaluation benchmark designed to assess large language models' (LLM) performance in Moroccan Darija, a variety of Arabic. It consists of 22,027 multiple-choice questions, translated from selected subsets of the Massive Multitask Language Understanding (MMLU) and ArabicMMLU benchmarks to measure model performance on 44 subjects in Darija. DarijaMMLU is constructed by translating selected subsets from two major benchmarks into Darija from English and MSA: Massive Multitask Language Understanding (MMLU) and ArabicMMLU.

Homepage: https://huggingface.co/datasets/MBZUAI-Paris/DarijaMMLU

Citation Copy Section

@article{shang2024atlaschatadaptinglargelanguage,
      title={Atlas-Chat: Adapting Large Language Models for Low-Resource Moroccan Arabic Dialect},
      author={Guokan Shang and Hadi Abdine and Yousef Khoubrane and Amr Mohamed and Yassine Abbahaddou and Sofiane Ennadir and Imane Momayiz and Xuguang Ren and Eric Moulines and Preslav Nakov and Michalis Vazirgiannis and Eric Xing},
      year={2024},
      eprint={2409.17912},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2409.17912},
}

Groups and Tasks Copy Section

Groups

• darijammlu: evaluates all DarijaMMLU tasks.

Tags

Source-based tags:

• darijammlu_mmlu: evaluates DarijaMMLU tasks that were translated from MMLU.
• darijammlu_ar_mmlu: evaluates DarijaMMLU tasks that were translated from ArabicMMLU.

Category-based tags:

• darijammlu_stem: evaluates DarijaMMLU STEM tasks.
• darijammlu_social_sciences: evaluates DarijaMMLU social sciences tasks.
• darijammlu_humanities: evaluates DarijaMMLU humanities tasks.
• darijammlu_language: evaluates DarijaMMLU language tasks.
• darijammlu_other: evaluates other DarijaMMLU tasks.

Checklist Copy Section

For adding novel benchmarks/datasets to the library:

• Is the task an existing benchmark in the literature?
  • Have you referenced the original paper that introduced the task?
  • If yes, does the original paper provide a reference implementation? If so, have you checked against the reference implementation and documented how to run such a test?

If other tasks on this dataset are already supported:

• Is the "Main" variant of this task clearly denoted?
• Have you provided a short sentence in a README on what each new variant adds / evaluates?
• Have you noted which, if any, published evaluation setups are matched by this variant?

Overview (`lm_eval/tasks/discrim_eval/README.md`) Copy Section

Discrim-Eval

Paper Copy Section

Title: Evaluating and Mitigating Discrimination in Language Model Decisions

Abstract: https://arxiv.org/abs/2312.03689

This benchmark consists of prompts for 70 decision-making scenarios. Each prompt asks whether a person should receive a favorable outcome ("Yes" or "No"). Persons vary by age, gender, and race, yielding 135 prompts per scenario. The dataset includes both explicit and implicit mentions of demographics.

For each prompt, we compare the probabilities assigned to the answers "Yes" and "No".
Let $p(\text{yes})$ and $p(\text{no})$ denote the model probabilities. We compute
the normalized probability of "Yes"
$$\text{pnorm}(\text{yes}) = \frac{p(\text{yes})}{p(\text{yes}) + p(\text{no})}$$
and then take its log-odds
$$\logit\bigl(\text{pnorm}(\text{yes})\bigr) = \log\frac{\text{pnorm}(\text{yes})}{1 - \text{pnorm}(\text{yes})}.$$
This quantity equals $\log p(\text{yes}) - \log p(\text{no})$.
For each demographic (race, gender, and age) we report the largest difference
between groups in the mean logit scores.

Homepage: https://huggingface.co/datasets/Anthropic/discrim-eval

Citation Copy Section

@misc{tamkin2023discrim,
      title={Evaluating and Mitigating Discrimination in Language Model Decisions},
      author={Alex Tamkin and Amanda Askell and Liane Lovitt and Esin Durmus and Nicholas Joseph and Shauna Kravec and Karina Nguyen and Jared Kaplan and Deep Ganguli},
      year={2023},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

Overview (`lm_eval/tasks/drop/README.md`) Copy Section

DROP

Paper Copy Section

Title: DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs

Abstract: https://aclanthology.org/attachments/N19-1246.Supplementary.pdf

DROP is a QA dataset which tests comprehensive understanding of paragraphs. In
this crowdsourced, adversarially-created, 96k question-answering benchmark, a
system must resolve multiple references in a question, map them onto a paragraph,
and perform discrete operations over them (such as addition, counting, or sorting).

Homepage: https://allenai.org/data/drop

Acknowledgement: This implementation is based on the official evaluation for DROP:
https://github.com/allenai/allennlp-reading-comprehension/blob/master/allennlp_rc/eval/drop_eval.py

Citation Copy Section

@misc{dua2019drop,
    title={DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs},
    author={Dheeru Dua and Yizhong Wang and Pradeep Dasigi and Gabriel Stanovsky and Sameer Singh and Matt Gardner},
    year={2019},
    eprint={1903.00161},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}

Groups and Tasks Copy Section

Groups

• Not part of a group yet.

Tasks

• drop

Checklist Copy Section

For adding novel benchmarks/datasets to the library:

• Is the task an existing benchmark in the literature?
  • Have you referenced the original paper that introduced the task?
  • If yes, does the original paper provide a reference implementation? If so, have you checked against the reference implementation and documented how to run such a test?

If other tasks on this dataset are already supported:

• Is the "Main" variant of this task clearly denoted?
• Have you provided a short sentence in a README on what each new variant adds / evaluates?
• Have you noted which, if any, published evaluation setups are matched by this variant?

Overview (`lm_eval/tasks/egyhellaswag/README.md`) Copy Section

EgyHellaSwag

Paper Copy Section

Title: NileChat: Towards Linguistically Diverse and Culturally Aware LLMs for Local Communities

Abstract: https://arxiv.org/abs/2505.18383

EgyHellaSwag is a benchmark for evaluating commonsense reasoning in Egyptian Arabic. It is a translated version of the HellaSwag benchmark, consisting of multiple-choice sentence completion tasks. The dataset was translated using the google/gemma-3-27b-it model.

Homepage: https://huggingface.co/datasets/UBC-NLP/EgyHellaSwag

Citation Copy Section

@article{mekki2025nilechatlinguisticallydiverseculturally,
  title={NileChat: Towards Linguistically Diverse and Culturally Aware LLMs for Local Communities},
  author={Abdellah El Mekki and Houdaifa Atou and Omer Nacar and Shady Shehata and Muhammad Abdul-Mageed},
  year={2025},
  eprint={2505.18383},
  archivePrefix={arXiv},
  primaryClass={cs.CL},
  url={https://arxiv.org/abs/2505.18383},
}

Groups and Tasks Copy Section

Groups

• Not part of a group.

Tags

• egyhellaswag

Checklist Copy Section

For adding novel benchmarks/datasets to the library:

• Is the task an existing benchmark in the literature?
  • Have you referenced the original paper that introduced the task?
  • If yes, does the original paper provide a reference implementation? If so, have you checked against the reference implementation and documented how to run such a test?

If other tasks on this dataset are already supported:

• Is the "Main" variant of this task clearly denoted?
• Have you provided a short sentence in a README on what each new variant adds / evaluates?
• Have you noted which, if any, published evaluation setups are matched by this variant?

Overview (`lm_eval/tasks/egymmlu/README.md`) Copy Section

EgyMMLU

Paper Copy Section

Title: NileChat: Towards Linguistically Diverse and Culturally Aware LLMs for Local Communities

Abstract: https://arxiv.org/abs/2505.18383

EgyMMLU is a benchmark designed to evaluate the performance of large language models in Egyptian Arabic. It contains 22,027 multiple-choice questions covering 44 subjects, translated from parts of the Massive Multitask Language Understanding (MMLU) and ArabicMMLU benchmarks. The dataset was translated using google/gemma-3-27b-it.

Homepage: https://huggingface.co/datasets/UBC-NLP/EgyMMLU

Citation Copy Section

@article{mekki2025nilechatlinguisticallydiverseculturally,
  title={NileChat: Towards Linguistically Diverse and Culturally Aware LLMs for Local Communities},
  author={Abdellah El Mekki and Houdaifa Atou and Omer Nacar and Shady Shehata and Muhammad Abdul-Mageed},
  year={2025},
  eprint={2505.18383},
  archivePrefix={arXiv},
  primaryClass={cs.CL},
  url={https://arxiv.org/abs/2505.18383},
}

Groups and Tasks Copy Section

Groups

• egymmlu: evaluates all EgyMMLU tasks.

Tags

Source-based tags:

• egymmlu_mmlu: evaluates EgyMMLU tasks that were translated from MMLU.
• egymmlu_ar_mmlu: evaluates EgyMMLU tasks that were translated from ArabicMMLU.

Category-based tags:

• egymmlu_stem: evaluates EgyMMLU STEM tasks.
• egymmlu_social_sciences: evaluates EgyMMLU social sciences tasks.
• egymmlu_humanities: evaluates EgyMMLU humanities tasks.
• egymmlu_language: evaluates EgyMMLU language tasks.
• egymmlu_other: evaluates other EgyMMLU tasks.

Checklist Copy Section

For adding novel benchmarks/datasets to the library:

• Is the task an existing benchmark in the literature?
  • Have you referenced the original paper that introduced the task?
  • If yes, does the original paper provide a reference implementation? If so, have you checked against the reference implementation and documented how to run such a test?

If other tasks on this dataset are already supported:

• Is the "Main" variant of this task clearly denoted?
• Have you provided a short sentence in a README on what each new variant adds / evaluates?
• Have you noted which, if any, published evaluation setups are matched by this variant?

Overview (`lm_eval/tasks/eq_bench/README.md`) Copy Section

EQ-Bench

Title: EQ-Bench: An Emotional Intelligence Benchmark for Large Language Models

Abstract: https://arxiv.org/abs/2312.06281

EQ-Bench is a benchmark for language models designed to assess emotional intelligence.

Why emotional intelligence? One reason is that it represents a subset of abilities that are important for the user experience, and which isn't explicitly tested by other benchmarks. Another reason is that it's not trivial to improve scores by fine tuning for the benchmark, which makes it harder to "game" the leaderboard.

EQ-Bench is a little different from traditional psychometric tests. It uses a specific question format, in which the subject has to read a dialogue then rate the intensity of possible emotional responses of one of the characters. Every question is interpretative and assesses the ability to predict the magnitude of the 4 presented emotions. The test is graded without the need for a judge (so there is no length bias). It's cheap to run (only 171 questions), and produces results that correlate strongly with human preference (Arena ELO) and multi-domain benchmarks like MMLU.

Homepage: https://eqbench.com/

NOTE: There are some key differences between the lm-evaluation-harness version and the implementation described in the EQ-Bench paper (These have been OK'd by the author):

• The lm-eval version uses the EQ-Bench v2 test set (171 questions) and score calculation. It does not incorporate the revision part of the prompt, as per v2.1 (https://github.com/EQ-bench/EQ-Bench)
• No retries in lm-eval version (EQ-Bench pipeline retries with successively higher temps if it encounters unparsable answers)
• In the original implementation, unparsable answers are excluded from the final score, and 83% of answers have to be parseable or a fail is returned. The lm-eval version instead assigns 0 to unparsable answers and has no fail criteria. So for lower performing models, there may be differences with the EQ-Bench leaderboard.

Citation Copy Section

@misc{paech2023eqbench,
	title={EQ-Bench: An Emotional Intelligence Benchmark for Large Language Models},
	author={Samuel J. Paech},
	year={2023},
	eprint={2312.06281},
	archivePrefix={arXiv},
	primaryClass={cs.CL}
}

Groups and Tasks Copy Section

Groups

• Not part of a group yet

Tasks

• eq_bench

Checklist Copy Section

• Is the task an existing benchmark in the literature?
  • Have you referenced the original paper that introduced the task?
  • If yes, does the original paper provide a reference implementation? If so, have you checked against the reference implementation and documented how to run such a test?

If other tasks on this dataset are already supported:

• Is the "Main" variant of this task clearly denoted?
• Have you provided a short sentence in a README on what each new variant adds / evaluates?
• Have you noted which, if any, published evaluation setups are matched by this variant?

Overview (`lm_eval/tasks/esbbq/README.md`) Copy Section

Spanish Bias Benchmark for Question Answering (EsBBQ)

Paper Copy Section

Title: EsBBQ and CaBBQ: The Spanish and Catalan Bias Benchmarks for Question Answering

Abstract: https://arxiv.org/abs/2507.11216

EsBBQ is a dataset designed to assess social bias across 10 categories in a multiple-choice QA setting, adapted from the original BBQ into the Spanish language and the social context of Spain.

It is fully parallel with the cabbq task group, the version in Catalan.

Citation Copy Section

@misc{esbbq-cabbq-2025,
      title={EsBBQ and CaBBQ: The Spanish and Catalan Bias Benchmarks for Question Answering},
      author={Valle Ruiz-Fernández and Mario Mina and Júlia Falcão and Luis Vasquez-Reina and Anna Sallés and Aitor Gonzalez-Agirre and Olatz Perez-de-Viñaspre},
      year={2025},
      eprint={2507.11216},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2507.11216},
}

Groups and Tasks Copy Section

Groups

• esbbq: Contains the subtasks that covers all demographic categories.

Tasks Copy Section

for category in ["age", "disability_status", "gender", "lgbtqia", "nationality", "physical_appearance", "race_ethnicity", "religion", "ses", "spanish_region"]:

• esbbq_{category}: Subtask that evaluates on the given category's subset.

Metrics Copy Section

EsBBQ is evaluated with the following 4 metrics, at the level of each subtask and with aggregated values for the entire group:

• acc_ambig: Accuracy over ambiguous instances.
• acc_disambig: Accuracy over disambiguated instances.
• bias_score_ambig: Bias score over ambiguous instances.
• bias_score_disambig: Bias score over disambiguated instances.

See the paper for a thorough explanation and the formulas of these metrics.

Checklist Copy Section

For adding novel benchmarks/datasets to the library:

• Is the task an existing benchmark in the literature?
  • Have you referenced the original paper that introduced the task?
  • If yes, does the original paper provide a reference implementation? If so, have you checked against the reference implementation and documented how to run such a test?

If other tasks on this dataset are already supported:

• Is the "Main" variant of this task clearly denoted?
• Have you provided a short sentence in a README on what each new variant adds / evaluates?
• Have you noted which, if any, published evaluation setups are matched by this variant?

Overview (`lm_eval/tasks/eus_exams/README.md`) Copy Section

EusExams

Paper Copy Section

Title: Latxa: An Open Language Model and Evaluation Suite for Basque

Abstract: https://arxiv.org/abs/2403.20266

EusExams is a collection of tests designed to prepare individuals for Public Service examinations conducted by several Basque institutions, including the public health system Osakidetza, the Basque Government, the City Councils of Bilbao and Gasteiz, and the University of the Basque Country (UPV/EHU). Within each of these groups, there are different exams for public positions, such as administrative and assistant roles. Each multiple-choice question contains 2 to 4 choices (3.90 on average) and one correct answer. The dataset is mostly parallel with 16k questions in Basque and 18k in Spanish.

Homepage: https://github.com/hitz-zentroa/latxa

Citation Copy Section

@misc{etxaniz2024latxa,
      title={Latxa: An Open Language Model and Evaluation Suite for Basque},
      author={Julen Etxaniz and Oscar Sainz and Naiara Perez and Itziar Aldabe and German Rigau and Eneko Agirre and Aitor Ormazabal and Mikel Artetxe and Aitor Soroa},
      year={2024},
      eprint={2403.20266},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

Groups and Tasks Copy Section

Tags

• eus_exams_eu: The Basque version of the exams.
• eus_exams_es: The Spanish version of the exams.

Tasks

Basque and Spanish versions of the exams are available as separate tasks starting with eus_exams_eu and eus_exams_es respectively.

Checklist Copy Section

For adding novel benchmarks/datasets to the library:

• Is the task an existing benchmark in the literature?
  • Have you referenced the original paper that introduced the task?
  • If yes, does the original paper provide a reference implementation? If so, have you checked against the reference implementation and documented how to run such a test?

If other tasks on this dataset are already supported:

• Is the "Main" variant of this task clearly denoted?
• Have you provided a short sentence in a README on what each new variant adds / evaluates?
• Have you noted which, if any, published evaluation setups are matched by this variant?

Overview (`lm_eval/tasks/eus_proficiency/README.md`) Copy Section

EusProficiency

Paper Copy Section

Title: Latxa: An Open Language Model and Evaluation Suite for Basque

Abstract: https://arxiv.org/abs/2403.20266

EusProficiency comprises 5,169 exercises on different topics from past EGA exams, the official C1-level certificate of proficiency in Basque. We collected the atarikoa exercises from EGA exams through the years 1998 to 2008. Atarikoa is the first qualifying test of EGA, which measures different aspects of language competency, such as reading comprehension, grammar, vocabulary, spelling, and writing. Each test generally has 85 multiple-choice questions, with 4 choices and a single correct answer.

Homepage: https://github.com/hitz-zentroa/latxa

Citation Copy Section

@misc{etxaniz2024latxa,
      title={Latxa: An Open Language Model and Evaluation Suite for Basque},
      author={Julen Etxaniz and Oscar Sainz and Naiara Perez and Itziar Aldabe and German Rigau and Eneko Agirre and Aitor Ormazabal and Mikel Artetxe and Aitor Soroa},
      year={2024},
      eprint={2403.20266},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

Groups and Tasks Copy Section

Groups

There are no groups.

Tasks

• eus_proficiency: EusProficiency comprises 5,169 exercises on different topics from past EGA exams, the official C1-level certificate of proficiency in Basque.

Checklist Copy Section

For adding novel benchmarks/datasets to the library:

• Is the task an existing benchmark in the literature?
  • Have you referenced the original paper that introduced the task?
  • If yes, does the original paper provide a reference implementation? If so, have you checked against the reference implementation and documented how to run such a test?

If other tasks on this dataset are already supported:

• Is the "Main" variant of this task clearly denoted?
• Have you provided a short sentence in a README on what each new variant adds / evaluates?
• Have you noted which, if any, published evaluation setups are matched by this variant?

Overview (`lm_eval/tasks/eus_reading/README.md`) Copy Section

EusReading

Paper Copy Section

Title: Latxa: An Open Language Model and Evaluation Suite for Basque

Abstract: https://arxiv.org/abs/2403.20266

EusReading consists of 352 reading comprehension exercises (irakurmena) sourced from the set of past EGA exams from 1998 to 2008. Each test generally has 10 multiple-choice questions, with 4 choices and a single correct answer. These exercises are more challenging than Belebele due to the complexity and length of the input texts. As a result, EusReading is useful to measure long context understanding of models.

Homepage: https://github.com/hitz-zentroa/latxa

Citation Copy Section

@misc{etxaniz2024latxa,
      title={Latxa: An Open Language Model and Evaluation Suite for Basque},
      author={Julen Etxaniz and Oscar Sainz and Naiara Perez and Itziar Aldabe and German Rigau and Eneko Agirre and Aitor Ormazabal and Mikel Artetxe and Aitor Soroa},
      year={2024},
      eprint={2403.20266},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

Groups and Tasks Copy Section

Groups

There are no groups.

Tasks

• eus_reading: EusReading consists of 352 reading comprehension exercises (irakurmena) sourced from the set of past EGA exams from 1998 to 2008.

Checklist Copy Section

For adding novel benchmarks/datasets to the library:

• Is the task an existing benchmark in the literature?
  • Have you referenced the original paper that introduced the task?
  • If yes, does the original paper provide a reference implementation? If so, have you checked against the reference implementation and documented how to run such a test?

If other tasks on this dataset are already supported:

• Is the "Main" variant of this task clearly denoted?
• Have you provided a short sentence in a README on what each new variant adds / evaluates?
• Have you noted which, if any, published evaluation setups are matched by this variant?

Overview (`lm_eval/tasks/eus_trivia/README.md`) Copy Section

EusTrivia

Paper Copy Section

Title: Latxa: An Open Language Model and Evaluation Suite for Basque

Abstract: https://arxiv.org/abs/2403.20266

EusTrivia consists of 1,715 trivia questions from multiple online sources. 56.3% of the questions are elementary level (grades 3-6), while the rest are considered challenging. A significant portion of the questions focus specifically on the Basque Country, its language and culture. Each multiple-choice question contains two, three or four choices (3.84 on average) and a single correct answer. Five areas of knowledge are covered:

• Humanities and Natural Sciences (27.8%): This category encompasses questions about history, geography, biology, ecology and other social and natural sciences.
• Leisure and Art (24.5%): This category includes questions on sports and athletes, performative and plastic arts and artists, architecture, cultural events, and related topics.
• Music (16.0%): Here are grouped all the questions about music and musicians, both classical and contemporary.
• Language and Literature (17.1%): This category is concerned with all kinds of literature productions and writers, as well as metalinguistic questions (e.g., definitions, synonyms, and word usage).
• Mathematics and ICT (14.5%): This category covers mathematical problems and questions about ICT, as well as questions about people known for their contributions to these fields of knowledge.

Homepage: https://github.com/hitz-zentroa/latxa

Citation Copy Section

@misc{etxaniz2024latxa,
      title={Latxa: An Open Language Model and Evaluation Suite for Basque},
      author={Julen Etxaniz and Oscar Sainz and Naiara Perez and Itziar Aldabe and German Rigau and Eneko Agirre and Aitor Ormazabal and Mikel Artetxe and Aitor Soroa},
      year={2024},
      eprint={2403.20266},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

Groups and Tasks Copy Section

Groups