[](https://aka.ms/fara1.5)
[](https://arxiv.org/abs/2606.20785)
[](https://aka.ms/fara1.5-4B-foundry)
[](https://aka.ms/fara1.5-9B-foundry)
[](https://aka.ms/fara1.5-27B-foundry)
[](https://aka.ms/fara1.5-hf)
[](https://huggingface.co/datasets/microsoft/WebTailBench)
[](https://huggingface.co/datasets/microsoft/CUAVerifierBench)
` element at `/finish` (synthetic envs only). |
| `env_state_raw` | `str` | Raw text from that `` element (debugging / fallback if `env_state_json` parse failed). |
| `screenshots` | `List[str]` | Screenshot filenames captured during the run, **relative** to the trajectory dir if `is_rel_paths` is true (the default), else absolute. |
| `is_aborted` | `bool` | True if the trajectory was killed mid-execution by an environment error. Aborted trajectories are typically retried up to `--max_error_task_retries` (see `webeval/core.py`). |
| `is_rel_paths` | `bool` | Whether `screenshots` paths are relative to `trajectory.path`. New runs use `True`. |
| `token_usage` | `Dict[str, RequestUsage]` | Per-component LLM token usage. The example uses `{"FaraAgent": {"prompt_tokens": 23383, "completion_tokens": 1920}}` — the key is whichever component name `WebSurferSystem.get_answer()` attributes the call to. Aggregated via `RequestUsage.__add__`. |
Persisted via `FinalAnswer.save(path)`; loaded via `FinalAnswer.load(path)`.
`Trajectory.__init__` calls `FinalAnswer.load(...)` on the single
`*_final_answer.json` it finds in the dir (raises `ValueError` if 0
or >1 are present), reattaches the loaded `FinalAnswer` as
`trajectory.answer`, and exposes the screenshot list directly as
`trajectory.screenshots`. `trajectory.is_aborted` is just a
pass-through to `trajectory.answer.is_aborted`.
#### `final_answer == ""` is the load-bearing flag
`FinalAnswer` is constructed with `final_answer = ""` as the
default, and `WebSurferSystem.get_answer()` only **overwrites it with
the agent's actual answer string when the agent calls `terminate(...)`
within `--max_rounds` steps**. Three trajectory outcomes therefore look
different on disk:
| Outcome | `is_aborted` | `final_answer` | Notes |
|--------------------------------------------|--------------|-----------------------|---|
| Agent called `terminate(...)` within budget| `False` | the model's response | Only this case has a real string to score. |
| Agent never called `terminate`, hit `--max_rounds` | `False` | `""` | Trajectory ran to completion-by-cutoff; downstream is free to score the final state but the model itself produced no answer. |
| Run errored mid-execution (browser crash, timeout, …) | `True` | `""` | Will be retried up to `--max_error_task_retries` (`webeval/core.py:run_eval_single_example`); only the final attempt's `_final_answer.json` is kept. |
**Filtering "actually completed" runs**: anywhere you want trajectories
that finished within step budget AND produced a real model answer, the
canonical check is
```python
candidate.answer.final_answer != "" and not candidate.is_aborted
```
This is what `webeval/post_eval_analysis.py` uses when computing
"average score across non-aborted, non-step-budget-exceeded
trajectories", and it's also why `WebTailBenchBenchmark.evaluator`
short-circuits with `verifier_reasoning='The last action must be
"terminate"'` for any trajectory whose last logged action is not
`terminate` — those are guaranteed to have `final_answer ==
""` and therefore nothing the rubric judge can grade.
> **Auto-0 vs excluded**: two superficially similar cases are scored
> very differently.
>
> - **Over-budget** (``, `is_aborted=False`): WebTailBench
> short-circuits to `score=0` without calling the rubric judge,
> writes it to `scores/.json`, and **counts it as 0 in
> the mean**.
> - **Aborted / errored mid-run** (`is_aborted=True`, loader fails,
> or evaluator raises): returns `score=None`, no score file is
> written, and every aggregator filters on `score is not None` —
> so aborted trajectories are **excluded from all final metrics**
> rather than counted as 0.
`screenshot_{i}.png` files keep accumulating regardless of how the
trajectory ended, so screenshot count alone is **not** a proxy for
"in budget" — always combine it with the `final_answer` check above.
Note: `trajectory.latest_screenshot` is a static convenience path
(`trajectory.path / "screenshot_scaled.png"`, see
`webeval/trajectory.py:131`) that may not exist on disk for every
run — the example directory, for instance, has only
`screenshot_1..4.png`.
### `times.json`
`{"start_time": , "end_time": , "duration":
}`, written around the solver call in
`webeval/core.py:run_single_task` (`webeval/core.py:86-93`). Both
stamps are `time.time()` values (not ISO-8601); e.g. the example is
`{"start_time": 1762809276.0520103, "end_time": 1762809454.79363,
"duration": 178.74161958694458}`. `evaluate_single_example` reads
`duration` to populate `EvalResult.duration`.
### `scores/.json`
Written by `webeval/core.py:evaluate_single_example` after a successful
`Benchmark.evaluate_example()` call. Single writer, single shape across
all benchmarks (top-level `{score, gpt_response_text}`); the
benchmark-specific verdict lives inside `gpt_response_text`.
| Field | Type | Notes |
|---------------------|------------------|---|
| `score` | `float` or `int` | Top-line success metric (0/1 for outcome-style judges; fraction for rubric-style judges). |
| `gpt_response_text` | `str` (often a JSON-encoded dict) | Free-form judge output. See per-benchmark table below. |
#### Where each benchmark's verdict lives
The score file is the **single source of truth for an evaluation
result**. `final_answer.json` is never overwritten by the verifier.
`task_data.json` is a separate dump the WebTailBench rubric verifier
writes alongside the score file (`benchmarks/webtailbench.py:346` →
`[Rubric] Saved rubric results to …/task_data.json`), so a fresh
solver-only run won't have one, but any trajectory that's been
re-scored — including the checked-in example — will. Re-running the
verifier (`--redo_eval` or `scripts/verify_trajectories.py`) rewrites
the score file in place and, for webtailbench, refreshes
`task_data.json` with the latest rubric payload.
| Benchmark | Score file path under `traj//` | `gpt_response_text` payload |
|---|---|---|
| `webvoyager` | `scores/gpt_eval.json` | gpt-4o judge text — verdict is `"SUCCESS"` / `"NOT SUCCESS"` parsed back to `score ∈ {0.0, 1.0}` (`benchmarks/webvoyager/webvoyager.py`). |
| `om2w` | `scores/-.json` (e.g. `WebJudge_Online_Mind2Web_eval-3.json`) | o4-mini judge text — verdict via `extract_predication(...)` from the OnlineMind2Web judge methods (`benchmarks/om2w/om2w.py`). |
| `webtailbench` | `scores/mmrubric_--.json` | JSON-encoded dict assembled in `benchmarks/webtailbench/webtailbench.py:355-442`: `is_success` (int 0/1), `success_criterion` (`outcome` / `process` / `both`), `final_answer`, `outcome_success` (bool), `outcome_reasoning`, `outcome_primary_intent`, `rubric_is_success` (int), the `rubric_*` family forwarded from MMRubricAgent (`rubric_items`, `rubric_total_earned_points`, `rubric_total_max_points`, `rubric_all_scores_list`, `rubric_all_rubric_dicts`, `rubric_intermediate_mm_rubric_steps`, `rubric_outcome_verification`, `rubric_majority_vote_metadata`), and the error-taxonomy fields nested inside the rubric payload (`first_point_of_failure`, `is_ambiguous` / `ambiguity_codes`, `is_invalid` / `invalid_task_codes` — `mm_rubric_agent.py:392-406`). Top-level `score` equals `is_success`, which is driven by `--success`: `outcome` → `int(outcome_success)`, `process` → `rubric_is_success`, `both` → both. |
| `verify_trajectories.py` (stand-alone) | same path + shape as `webtailbench` | same Universal Verifier output. |
### `core.log`
Per-task log lines from the `webeval.core.` logger. Useful
when a task hangs or errors and the rest of the pipeline keeps
moving. A run's `core.log` typically mixes:
- `[Execution ] Start / Trajectory is not complete. Reexecuting... / Completed / Error`
- BrowserBase plumbing: `Initializing BrowserBase session...`,
`Connected to BrowserBase session: `, `Closing browser...`
- Every `WebSurferEvent` event re-emitted as its dataclass
`repr(...)` (so the same step shows up both here and in
`web_surfer.log`, just unparsed)
- `[Evaluation ] Start / Loading from: / Completed: score=, duration= / Error`
- `[Rubric] Saved rubric results to /task_data.json` when the
WebTailBench rubric verifier runs
The example's `core.log` shows the full sequence for one solve on
2025-11-10 plus three separate rubric re-eval passes on 2026-03-26,
all against the same trajectory directory.
### `task_data.json` (optional, rubric-verifier output)
Only written by the WebTailBench rubric verifier
(`benchmarks/webtailbench/webtailbench.py:346`), so trajectories that
have never been re-scored won't have one. When present, it's a large
JSON dump of the task's rubric/outcome state — everything MMRubricAgent
produced plus the task metadata it loaded. In the example,
`task_data.json` has 76 keys, including `task_summary`, `task_type`,
`rubric_score`, `verifier_agent_response`, `verifier_rubric`,
`start_timestamp`, `end_timestamp`, `token_usage`, `plan_list`,
`facts_list`, `observations_list`, `actions`, `actions_summary`, and
the boolean error-taxonomy flags (`did_web_surfing_fail`,
`web_surfing_stalled`, `encountered_captcha`, …). Treat this file as
diagnostic detail; the authoritative score still lives in
`scores/.json`.
## `Trajectory` consumers
```python
from webeval.trajectory import Trajectory
traj = Trajectory.from_folder("/path/to//")
# Returns None if the dir is missing the answer file or the log can't be parsed.
traj.events # list[dict] of parsed action/observation events
traj.actions # list[str] of JSON-encoded action-arg dicts
traj.thoughts # list[str] parallel to .actions
traj.screenshots # list[Path] absolute paths to screenshot files
traj.answer # FinalAnswer
traj.is_aborted # bool (delegates to .answer.is_aborted)
traj.latest_screenshot # convenience: trajectory.path / "screenshot_scaled.png"
```
Three places consume it today:
1. **Benchmark verifiers** (`webeval/benchmarks/*/.py`): each
benchmark's `evaluator(task_data, candidate)` receives a
`Trajectory` and returns `(score, gpt_response_text)`. WebTailBench
wraps `MMRubricAgent` (the Universal Verifier) for this; webvoyager
/ om2w call their own GPT judges.
2. **`webeval/scripts/verify_trajectories.py`**: stand-alone parallel
runner that points `MMRubricAgent` at any directory of
`Trajectory`-shaped folders and writes per-task
`scores/mmrubric_<…>.json`. Useful for re-scoring an existing
eval run with new judge parameters.
3. **`webeval/post_eval_analysis.py`**: aggregates
`_final_answer.json` + `scores/<…>.json` + counts of
`WebSurferEvent` lines in `web_surfer.log` for run-level reporting.
## Where each file is written
| File | Writer |
|-----------------------------------|-------------------------------------------------------------------------------------------|
| `web_surfer.log` | `LogHandler` attached in `WebSurferSystem.get_answer()` (capturing `FaraAgent` log calls) |
| `_final_answer.json` | `WebSurferSystem.get_answer()` → `FinalAnswer.save()` |
| `screenshot{i}.png` | `FaraAgent.execute_action` (browser screenshots; one per action by default) |
| `times.json` | `webeval/core.py:run_single_task` |
| `core.log` | `_logger = logger.getChild(task_id)` in `webeval/core.py:run_eval_single_example` |
| `scores/.json` | `webeval/core.py:evaluate_single_example` |
| `scores/mmrubric_<…>.json` | `webeval/scripts/verify_trajectories.py` |
---
### Overview (`webeval/src/webeval/benchmarks/webvoyager/README.md`)
https://github.com/MinorJerry/WebVoyager/blob/main/evaluation/auto_eval.py to add
---
### Overview (`webeval/src/webeval/benchmarks/om2w/impl/README.md`)
Online-Mind2Web Benchmark
Tianci Xue,1,
Weijian Qi*,1,
Tianneng Shi*2,
Chan Hee Song1,
Boyu Gou1,
Dawn Song,2,
Huan Sun†,1
Yu Su†,1
1The Ohio State University, 2University of California, Berkeley
*Equal contribution, †Equal advising
📃 Paper
•
📃 Blog
•
🏆 Leaderboard
•
🤗 Data
# Online-Mind2Web benchmark
## News
* [05/11/2025] Check out our updates in the [paper](https://arxiv.org/abs/2504.01382).
* The performance of Claude Computer Use 3.7.
* WebJudge(o4-mini) achieves high agreement (86%) with a low success rate gap (3.8%) compared with humans.
* Release [WebJudge-7B](https://huggingface.co/osunlp/WebJudge-7B), a robust and reliable reward model for Reinforcement learning.
## Tasks
Online-Mind2Web includes 300 diverse tasks from 136 popular websites across various domains. It covers a diverse set of real-world user tasks, such as clothing, food, housing, and transportation, to evaluate web agents' performance in a real-world online environment.
## Update Tasks
We will regularly update Online-Mind2Web by replacing outdated or invalid tasks (e.g., due to website changes) to maintain its value as a rigorous benchmark for web agents. If you find any tasks are outdated, please reach out to us, and we will update them.
To ensure fair comparisons, we will aim to keep the updated tasks on the same websites as before and with a similar reference length. Additionally, once agent performance saturates on Online-Mind2Web, we will also revise simple tasks to preserve its long-term value.
# Automatic Evaluator via LLM-as-a-Judge (WebJudge)
To enhance the reliability and scalability of the evaluation process in online environments, We propose a more reliable automatic evaluation method called **WebJudge**, which consists of three parts. (1) Key Point Identification: The model is prompted to identify several key points necessary for completing the task, based on the given instruction and task description. (2) Key Screenshot Identification: Important screenshots are selected from the agent’s trajectory to retain relevant visual evidence while discarding uninformative frames. (3) Outcome Judgment: Output the judgement result based on the task description, key points, key screenshots, and the action history. Our method preserves critical intermediate screenshots while mitigating the token overload issue.
# Results
## Comparison against Existing Evaluation Methods on Online-Mind2Web
Model
Auto-Eval
SeeAct
Agent-E
Browser Use
Claude 3.5
Claude 3.7
Operator
Avg AR
GPT-4o
Autonomous Eval
84.7
85.0
76.0
83.7
75.5
71.7
79.4
AgentTrek Eval
73.0
64.3
63.3
--
--
--
66.9
WebVoyager
--
75.3
71.3
74.0
72.0
76.7
73.9
WebJudge
86.7
86.0
81.4
86.3
79.1
81.8
83.6
o4-mini
Autonomous Eval
79.7
85.7
86.0
84.3
68.0
73.3
79.5
WebVoyager
--
80.3
79.0
81.7
74.3
78.3
78.7
WebJudge
85.3
86.3
89.3
87.0
82.3
83.7
85.7
WebJudge-7B
86.0
87.3
88.3
89.7
84.3
86.3
87.0
WebJudge powered by GPT-4o and o4-mini consistently achieves the highest agreement, with averages of 83.6% and 85.7%, respectively. Meanwhile, WebJudge-7B even outperforms o4-mini, reaching a high agreement with human judgment of 87%.
## Excellent generalization capabilities on [AgentRewardBench](https://agent-reward-bench.github.io/) (5 OOD benchmarks)
| **Methods** | **AB** | **VWA** | **WA** | **Work** | **Wk++** | **Overall** |
|--------------|--------|--------|--------|----------|----------|--------------|
| *Rule-based** | 25.0 | **85.2** | 79.0 | 100.0 | 83.3 | 83.8 |
| Autonomous Eval* | 83.3 | 61.2 | 67.6 | 96.4 | 59.3 | 67.6 |
| GPT-4o (A11y Tree)* | 77.8 | 63.0 | 70.2 | 94.6 | 63.0 | 69.8 |
| WebJudge (GPT-4o) | 66.7 | 69.8 | 72.6 | 92.3 | 75.0 | 73.7 |
| WebJudge-7B | 80.0 | 66.7 | 77.5 | 100.0 | 70.0 | 75.7 |
| WebJudge (o4-mini) | **100.0** | 74.5 | **81.2** | **100.0** | **90.0** | **82.0** |
WebJudge significantly outperforms existing methods, achieving impressive overall precision of 73.7% 75.7% and 82.0% on WebArena (WA), VisualWebArena (VWA), AssistantBench (AB), WorkArena (Work) and WorkArena++ (Wk++) across 1302 trajectories.
The high precision suggests that WebJudge holds potential as a robust and scalable reward model for downstream applications such as Rejection Sampling Fine-Tuning, Reflection, and Reinforcement Learning.
# Model Release
We have released the fine-tuned [WebJudge-7B](https://huggingface.co/osunlp/WebJudge-7B) weights, which are now available on Hugging Face.
# Setup Environment
Create a conda environment and install dependencies:
```
conda create -n Online_Mind2Web python=3.11
conda activate Online_Mind2Web
pip install -r requirements.txt
```
# Evaluation
You can run the provided example evaluation script directly to perform the evaluation. Adjust the "mode" parameter to choose among various auto-eval methods.
```bash
bash ./script/eval.sh
```
## Important Notes for Reliable Evaluation on Online-Mind2Web:
> [!IMPORTANT]
> - **Start from the specified websites, not Google Search**:To enable fair comparisons, please ensure that each task starts from the specified website in our benchmark. Starting from Google Search or alternative websites can lead agents to use different websites to solve the task, resulting in varying difficulty levels and potentially skewed evaluation results.
> - **Include only factual actions, not agent outputs**: The action history should contain only the factual actions taken by the agent to complete the task (e.g., clicking elements and Typing text). Do not include the final response or any other agent's outputs, as they may contain hallucinated content and result in a high rate of false positives.
> - **Use o4-mini for WebJudge**: WebJudge powered by o4-mini demonstrates a higher alignment with human judgment, achieving an average agreement rate of 85.7% and maintaining a narrow success rate gap of just 3.8%. Therefore, please use o4-mini as the backbone for automatic evaluation.
# Evaluation Results
In certain scenarios, testing on the full Online-Mind2Web dataset may not be feasible due to cost, privacy, or legal constraints. To facilitate fair and apple-to-apple comparisons, we release both our human evaluation labels and auto-eval details.
- **Human Evaluation**: Task-level human evaluation labels are provided in the [file](https://github.com/OSU-NLP-Group/Online-Mind2Web/tree/main/data/evaluation_results/online_mind2web_evaluation_results/human_label.json).
- **Auto-Evaluation**: The results of WebJudge are available in the [folder](https://github.com/OSU-NLP-Group/Online-Mind2Web/tree/main/data/evaluation_results/online_mind2web_evaluation_results).
## 📚 Citation
Note: Online-Mind2Web is derived from the original Mind2Web dataset. We kindly ask that you cite both the original and this work when using or referencing the data.
```
@article{xue2025illusionprogressassessingcurrent,
title={An Illusion of Progress? Assessing the Current State of Web Agents},
author={Tianci Xue and Weijian Qi and Tianneng Shi and Chan Hee Song and Boyu Gou and Dawn Song and Huan Sun and Yu Su},
year={2025},
eprint={2504.01382},
archivePrefix={arXiv},
primaryClass={cs.AI},
url={https://arxiv.org/abs/2504.01382},
}
@inproceedings{deng2023mind2web,
author = {Deng, Xiang and Gu, Yu and Zheng, Boyuan and Chen, Shijie and Stevens, Sam and Wang, Boshi and Sun, Huan and Su, Yu},
booktitle = {Advances in Neural Information Processing Systems},
editor = {A. Oh and T. Naumann and A. Globerson and K. Saenko and M. Hardt and S. Levine},
pages = {28091--28114},
publisher = {Curran Associates, Inc.},
title = {Mind2Web: Towards a Generalist Agent for the Web},
url = {https://proceedings.neurips.cc/paper_files/paper/2023/file/5950bf290a1570ea401bf98882128160-Paper-Datasets_and_Benchmarks.pdf},
volume = {36},
year = {2023}
}
```
---
### Overview (`webeval/src/webeval/benchmarks/om2w/impl/data/README.md`)
## Example
You can refer to the [example](https://github.com/OSU-NLP-Group/Online-Mind2Web/tree/main/data/example/fb7b4f784cfde003e2548fdf4e8d6b4f) format when conducting auto-eval. For each task, the result should be stored in a folder named as its `task_id`, containing :
- `trajectory/`: Stores screenshots of each step.
- `result.json`: Task metadata and action history.
## 🧠 Human Labels
The file `human_label.json` provides human evaluation labels for each task across different agents. The meaning of labels represent as following:
- `0`: ❌ **Failure** – The agent did **not** successfully complete the task.
- `1`: ✅ **Success** – The agent **successfully** completed the task.
- `2`: 🚫 **Not Executable** – The agent was **unable to execute** the task due to external or system-related limitations.
## ℹ️ Details on Label 2
The label 2 indicates that the agent could not execute the task at all. The specific reasons vary by agent:
- **SeeAct**: Some tasks raise internal bugs when the agent extracts elements from the web page.
- **Operator**: Certain websites (e.g., Reddit) are inaccessible, preventing task execution.
- **Claude Computer Use 3.7**: Due to updates on the websites, a few tasks became outdated during the testing phase..
- **Agent-E** and **Browser Use**: Label 2 typically corresponds to tasks blocked by **CAPTCHAs**. There are only **two** tasks for Agent-E and **one** task for Browser Use.
## Note
The results on the leaderboard are averaged over three runs, so there may be slight differences compared to the auto-eval results shown here.
---
### Eval Reproducibility (`docs/eval_reproducibility.md`)
# Eval Reproducibility (webeval)
> **Note:** This document covers the `webeval/` pipeline that produced the
> published Fara-7B benchmark numbers (WebVoyager, Online-Mind2Web,
> WebTailBench). It predates the Fara-1.5 release and will be updated as the
> repo is refactored to the Fara-1.5 evaluation stack. For the trajectory
> on-disk format, see [`webeval/README.md`](../webeval/README.md).
Agentic evaluations on live websites present unique challenges due to day-to-day changes. We implement several measures to ensure reliable and comparable evaluations:
**BrowserBase Integration**
We employ BrowserBase to manage browser session hosting, enabling reliable browser instance management.
**Time-sensitive Task Updates**
Tasks in benchmarks like WebVoyager can become stale or impossible. We:
- Removed ~48 impossible tasks from the original WebVoyager benchmark
- Updated ~50 tasks with future dates to keep them achievable
- Example: *"Search for a hotel in Bali from Jan 1 to Jan 4, 2024"* → *"Search for a hotel in Bali from Jan 1 to Jan 4, 2026"*
- Our updated WebVoyager benchmark is available at `webeval/data/webvoyager/WebVoyager_data_08312025.jsonl`
**Environment Error Handling**
Browser errors (connection drops, page timeouts) are handled robustly:
- Trajectories are retried up to 5 times when environment errors occur
- Complete yet incorrect trajectories are never retried
- Each retry starts with a fresh browser session, with no retained state
**Step Budget**
Each trajectory is capped at a maximum of 100 actions across all online benchmarks. Trajectories exceeding this budget without choosing to stop are considered incorrect.
## WebEval Package Installation
```bash
conda create --name fara_webeval python=3.12
conda activate fara_webeval
# Install fara package (with vllm extras for GPU hosting)
pip install -e .[vllm]
# Install webeval
cd webeval
pip install -e .
# Install playwright
playwright install
```
The webeval package no longer depends on `autogen-core` / `autogen-ext` —
all chat completion clients are vendored under `webeval/src/webeval/oai_clients/`
(see `GracefulRetryClient`, `OpenAIClientWrapper`, `AzureOpenAIClientWrapper`,
etc.). You no longer need to clone or install the autogen submodule.
> **Always activate the `fara_webeval` env before running any of the eval
> scripts below.** It pins `vllm==0.10.0` + `torch==2.7.1`; running with
> a newer `vllm` (≥ 0.19) under `torch` ≥ 2.10 trips a known
> `torch._dynamo.symbolic_convert` crash during CUDA-graph capture, which
> can be worked around with `--enforce_eager` but at a meaningful
> throughput cost. Stick to the pinned env.
## Running Evaluations
Navigate to the scripts directory:
```bash
cd webeval/scripts
```
Make sure you set a valid OpenAI GPT-4o endpoint in `endpoint_configs_gpt4o/dev` in order to run the WebVoyager LLM-as-a-judge!
**Option 1: Self-hosted vLLM**
```
/* Detailed source-code truncated for AI context efficiency. */
```
**Option 2: Azure Foundry Deployment**
Deploy [Fara1.5-9B on Foundry endpoint(s)](https://ai.azure.com/catalog/models/Fara1.5-9B), then place endpoint URLs and keys in JSONs under `endpoint_configs/`:
```bash
python webvoyager.py --model_endpoint ../../endpoint_configs/ --eval_oai_config ../endpoint_configs_gpt4o/dev/ --out_url /path/to/save/eval/files --processes 1 --run_id 1_endpoint --max_rounds 100
python om2w.py --model_endpoint ../../endpoint_configs/ --eval_oai_config ../endpoint_configs_o4/dev/ --eval_model o4-mini --out_url /path/to/save/eval/files --processes 1 --run_id 1_endpoint --max_rounds 100
python webtailbench.py --model_endpoint ../../endpoint_configs/ --eval_oai_config ../../endpoint_configs/judge_active/prod --judge_eval_model gpt-5.2 --judge_o4_eval_model o4-mini --out_url /path/to/Fara/eval --processes 1 --run_id 1_endpoint --max_rounds 100
```
### Notes
- We use the same LLM-as-a-judge prompts and model (GPT-4o) as WebVoyager, hence the `--eval_oai_config` argument
- Set `--browserbase` for browser session management (requires exported API key and project ID environment variables)
- Avoid overloading a single vLLM deployment with more than ~10 concurrent processes due to known issues
- See debugging output in `fara/webeval/scripts/stdout.txt`
---
## Analyzing Evaluation Results
### Evaluation Output Structure
Evaluation results are stored under `--out_url` in folders organized by:
- Model name
- Dataset
- Username
- Run ID
Example path:
```
/runs/WebSurfer-fara-100-max_n_images-3/fara1.5-9b//WebVoyager_WebVoyager_data_08312025.jsonl/
```
Each evaluation folder contains:
- `gpt_eval/` - LLM-as-a-judge evaluation results
- `traj/` - Per-task trajectory subdirectories containing:
- `*-final_answer.json` (e.g., `Amazon--1_final_answer.json`) - `` indicates abortion or step budget exceeded
- `scores/*_eval.json` - LLM judge scores (`gpt_eval.json` for WebVoyager, `WebJudge_Online_Mind2Web_eval-3.json` for Online-Mind2Web)
- `web_surfer.log` - Action history and errors
- `screenshot_X.png` - Screenshots captured before each action X
- `times.json` - contains start and end times of task
- `core.log` - contains high level logs like if the trajectory needs to be started or is already cached/done, the evaluation score, duration, and errors encountered
### Running Analysis
Use the analysis notebook to compute metrics:
```bash
cd webeval/scripts/analyze_eval_results/
jupyter notebook analyze.ipynb
```
The script:
- Identifies trajectories aborted mid-execution and diagnostic reasons
- Computes average scores across non-aborted trajectories
- Distinguishes between aborted trajectories (errors during sampling) and completed trajectories (with terminate() call or step budget exceeded)
To re-run failed tasks, execute the evaluation script again with the same `run_id` and `username` - it will skip non-aborted tasks.
Example WebVoyager GPT Eval Result
```json
{
"score": 1.0,
"gpt_response_text": "To evaluate the task, we need to verify if the criteria have been met:\n\n1. **Recipe Requirement**: A vegetarian lasagna recipe with zucchini and at least a four-star rating.\n\n2. **Search and Results**:\n - The screenshots show that the search term used was \"vegetarian lasagna zucchini.\"\n - Among the search results, \"Debbie's Vegetable Lasagna\" is prominently featured.\n \n3. **Evaluation of the Recipe**:\n - Rating: \"Debbie's Vegetable Lasagna\" has a rating of 4.7, which satisfies the requirement of being at least four stars.\n - The presence of zucchini in the recipe is implied through the search conducted, though the screenshots do not explicitly show the ingredients list. However, the result response confirms the match to the criteria.\n\nGiven the information provided, the task seems to have fulfilled the requirement of finding a vegetarian lasagna recipe with zucchini and a four-star rating or higher. \n\n**Verdict: SUCCESS**"
}
```
Example of web_surfer.log (action history)
```
/* Detailed source-code truncated for AI context efficiency. */
```
---
--- METRICS ---
- Files Extracted: 6
- Estimated Token Budget: ~13826 tokens
- Canonical Reference: https://codewiki.google/github.com/microsoft/fara