Technical Documentation: microsoft/fara

ℹ️ Provenance: Multi-Tier Ingestion · CodeWiki Reference

Overview (`README.md`) Copy Section

Updates Copy Chapter

• 2026-07-22 — Fara1.5 released! A family of native computer use
  agents at three scales (Fara1.5-4B, Fara1.5-9B, Fara1.5-27B) built on
  Qwen3.5 and trained on data from the FaraGen1.5 pipeline. All three models have weights available on Hugging Face and can be hosted through Microsoft Foundry (Fara1.5-4B, Fara1.5-9B, Fara1.5-27B).
  Fara1.5-9B
  reaches 63.4% on Online-Mind2Web and 86.6% on WebVoyager, a new state of
  the art for its size class; Fara1.5-27B achieves 72.3% on Online-Mind2Web,
  outperforming much larger proprietary systems. Read the
  paper and the
  blog post.
• 2026-05-12 — WebTailBench is now a first-class benchmark in this
  repo: the loader auto-downloads tasks and rubrics from
  microsoft/WebTailBench
  (use the refreshed test_v2 split; a V1↔V2 diff is hosted
  here),
  the Universal Verifier (MMRubricAgent) is the official judge, and the
  reproducibility CLI lives in webeval/scripts/webtailbench.py (stand-alone
  re-scoring via webeval/scripts/verify_trajectories.py).
• 2026-04-19 — Released CUAVerifierBench,
  a human-annotated benchmark for evaluating CUA verifiers (i.e. judges that
  score agent trajectories). Two splits — fara7b_om2w_browserbase (106
  Fara-7B Online-Mind2Web/Browserbase trajectories, ~2 reviewers each) and
  internal (154 trajectories from a heldout aurora-v2 task suite) —
  with per-judge UV-blind / UV-informed labels, Universal Verifier
  outputs, and legacy verifier outputs side-by-side. The build script
  that produced the dataset lives alongside the data on Hugging Face.

Overview Copy Chapter

Fara1.5 is a family of native Computer Use Agents (CUAs) at three scales — Fara1.5-4B, Fara1.5-9B, and Fara1.5-27B — built on Qwen3.5 and trained with supervised finetuning on data from FaraGen1.5, our scalable data pipeline of environments, solvers, and verifiers. Each model sets a new state of the art for its size class on browser-use benchmarks, and Fara1.5-27B outperforms much larger proprietary systems such as OpenAI Operator and Gemini 2.5 Computer Use on Online-Mind2Web.

Fara1.5 models operate through an observe-think-act loop: given a screenshot of the browser and the conversation history, the model reasons about the state of the task and outputs an action — mouse and keyboard inputs on directly predicted coordinates, web searches, or context management operations — with no accessibility trees or separate parsing models.

All three models are available on Microsoft Foundry: Fara1.5-4B, Fara1.5-9B, and Fara1.5-27B.

Try Fara1.5-9B as follows (see Installation for detailed instructions):

# 1. Clone repository
git clone https://github.com/microsoft/fara.git
cd fara

# 2. Setup environment
python3 -m venv .venv 
source .venv/bin/activate
pip install -e .
playwright install

Deploy Fara1.5-9B from the Microsoft Foundry catalog and put your endpoint in a config JSON (e.g. azure_foundry_config.json):

{
    "model": "Fara1.5-9B",
    "base_url": "https://your-endpoint.inference.ml.azure.com/",
    "api_key": "YOUR_API_KEY_HERE"
}

Then you can iteratively query it with:

fara-cli --task "whats the weather in new york now" --endpoint_config azure_foundry_config.json

To try Fara inside Magentic-UI — a sandboxed browser environment with auditable action logging and user prompts at critical points — follow the instructions in the Magentic-UI repo. You will need a model endpoint as before, but instead of fara-cli you can use Magentic-UI which has a nice UI (see video demos below).

Note: If you're using Windows, we highly recommend using WSL2 (Windows Subsystem for Linux). Please see the Windows instructions in the Installation section.

What Makes Fara1.5 Unique Copy Section

Unlike traditional chat models that generate text-based responses, Fara1.5 leverages computer interfaces—mouse and keyboard—to perform multi-step tasks on behalf of users. The models:

• Operate visually by perceiving webpages and taking actions like scrolling, typing, and clicking on directly predicted coordinates without accessibility trees or separate parsing models
• Enable efficient deployment thanks to their compact sizes (4B/9B/27B), resulting in reduced latency and improved privacy
• Interact with users: trained with a user simulator on multi-turn rollouts, Fara1.5 can ask for missing information, flag ambiguous tasks, and pause for approval before irreversible actions
• Remain a research preview: safety measures include refusing harmful tasks based on responsible AI policies, prompting users at critical interaction points, and auditable action logging through Magentic-UI's sandboxed browser environment

FaraGen1.5: Scalable Learning Environments Copy Section

Collecting computer use data from human demonstrations is expensive and slow. FaraGen1.5 is a scalable data pipeline composed of three modular components:

• Environments — open-internet tasks on live websites plus six synthetic FaraEnvs (Mail, Calendar, Stream, ML, Stay, Scheduler): functional UI clones that faithfully simulate domains gated by authentication or requiring irreversible actions, and unlock execution-based, ground-truth verification
• Solvers — a solver harness that can be powered by multiple models, including strong frontier models such as GPT-5.4, paired with a user simulator to enable multi-turn rollouts
• Verifiers — three complementary filters covering task correctness (the Universal Verifier LLM-judge), efficiency (scoring redundant actions), and critical-point adherence / user interaction (missing information, ambiguous tasks, unapproved irreversible actions)

The resulting training mix contains roughly 2M samples: ~60% web trajectories, 12.8% synthetic environments, 12.5% form filling, 8.8% grounding, 4.9% VQA, and 0.8% GUI drag tasks.

Key Capabilities Copy Section

Fara1.5 can automate everyday web tasks including:

• Searching for information and summarizing results
• Filling out forms and managing accounts
• Booking travel, movie tickets, and restaurant reservations
• Shopping and comparing prices across retailers
• Finding job postings and real estate listings

Performance Highlights Copy Section

Each Fara1.5 model sets a new state of the art for its size class. Fara1.5-9B improves over Fara-7B by +29.3 points on Online-Mind2Web, +13.1 points on WebVoyager, and +8.2 points on WebTailBench outcome success:

Table: Task success rate (%) on WebVoyager, Online-Mind2Web, and WebTailBench v1.5. For WebTailBench v1.5 we report both Process Success (correct intermediate steps) and Outcome Success (final task state correct). All Fara1.5 and Fara-7B numbers are averaged over three independent runs. The GPT-5.4-based FaraGen1.5 solver is an upper-bound reference for the SFT-based distillation. † denotes numbers sourced from the model's official release or leaderboard rather than re-run by us.

WebTailBench: Evaluating Real-World Web Tasks Copy Section

WebTailBench is our evaluation benchmark focusing on 11 real-world task types that are underrepresented or missing in existing benchmarks. The benchmark includes 609 tasks across diverse categories, with the first 8 segments testing single skills or objectives (usually on a single website), and the remaining 3 evaluating more difficult multi-step or cross-site tasks (shopping lists, comparison shopping, and compositional tasks). Tasks and precomputed rubrics are on Hugging Face (use the refreshed test_v2 split), the Universal Verifier (MMRubricAgent) is the official judge, and the reproducibility CLI lives in webeval/scripts/webtailbench.py.

CUAVerifierBench: Evaluating the Verifiers Themselves Copy Section

While WebTailBench measures agents, CUAVerifierBench measures the judges that score those agents. Each row pairs a Fara agent trajectory (instruction, screenshots, web_surfer log, final answer) with one human reviewer's verdict, plus the verdicts produced by the Universal Verifier (MMRubricAgent) and several legacy verifiers — so researchers can compute verifier–human agreement (Cohen's κ, accuracy, F1) on a fixed corpus and iterate on new judge prompts / architectures against a frozen ground-truth set.

The dataset is exposed as two Hugging Face configs joinable on task_id:

Two splits ship today:

Reviewer identities are anonymized as Judge1 … JudgeN using a single shared map across both splits. The build script that produced the dataset (with full schema + provenance) ships alongside the data on Hugging Face at microsoft/CUAVerifierBench; see the dataset README for the full column list.

from datasets import load_dataset

trajs = load_dataset("microsoft/CUAVerifierBench", "trajectories",
                     split="fara7b_om2w_browserbase")
anns  = load_dataset("microsoft/CUAVerifierBench", "annotations",
                     split="fara7b_om2w_browserbase")

Evaluation Infrastructure Copy Section

Our evaluation setup leverages:

1. Playwright - A cross-browser automation framework that replicates browser environments
2. Abstract Web Agent Interface - Allows integration of any model from any source into the evaluation environment
3. Fara-Agent Class - Reference implementation for running the Fara models

Note: Fara1.5 is a research preview designed to invite hands-on exploration and feedback from the community. We recommend running it in a sandboxed environment, monitoring its execution, and avoiding sensitive data or high-risk domains.

Installation Copy Chapter

Linux Copy Section

The following instructions are for Linux systems, see the Windows section below for Windows instructions.

Install the package using pip and set up the environment with Playwright:

# 1. Clone repository
git clone https://github.com/microsoft/fara.git
cd fara

# 2. Setup environment
python3 -m venv .venv 
source .venv/bin/activate
pip install -e .[vllm]
playwright install

Note: If you plan on hosting with Microsoft Foundry only, you can skip the [vllm] and just do pip install -e .

Windows Copy Section

For Windows, we highly recommend using WSL2 (Windows Subsystem for Linux) to provide a Linux-like environment. However, if you prefer to run natively on Windows, follow these steps:

# 1. Clone repository
git clone https://github.com/microsoft/fara.git
cd fara

# 2. Setup environment
python3 -m venv .venv
.venv\Scripts\activate
pip install -e .
python3 -m playwright install

Hosting the Model Copy Section

Recommended: The easiest way to get started is using Microsoft Foundry hosting, which requires no GPU hardware or model downloads. All three Fara1.5 models are available on Foundry.

Microsoft Foundry Hosting

Deploy Fara1.5 models from the Microsoft Foundry catalog (Fara1.5-4B, Fara1.5-9B, Fara1.5-27B) without needing to download weights or manage GPU infrastructure.

Setup:

1. Deploy the model on Microsoft Foundry and obtain your endpoint URL and API key

Then create a endpoint configuration JSON file (e.g., azure_foundry_config.json):

{
    "model": "Fara1.5-9B",
    "base_url": "https://your-endpoint.inference.ml.azure.com/",
    "api_key": "YOUR_API_KEY_HERE"
}

2. Run the Fara agent:

fara-cli --task "how many pages does wikipedia have" --endpoint_config azure_foundry_config.json [--headful]

Note: you can also specify the endpoint config with the args --base_url [your_base_url] --api_key [your_api_key] --model [your_model_name] instead of using a config JSON file.

Note: If you see an error that the fara-cli command is not found, then try:

python -m fara.run_fara --task "what is the weather in new york now"

That's it! No GPU or model downloads required.

Self-hosting with vLLM

If you have access to GPU resources, you can download the model weights from Hugging Face, self-host Fara models with vLLM, and point fara-cli at the resulting OpenAI-compatible endpoint:

vllm serve <model> --port 5000 --dtype auto

The previous-generation Fara-7B weights remain available on Hugging Face (with GGUF variants for LM Studio / Ollama). Pass --fara-7b to run the Fara-7B agent instead of Fara1.5:

fara-cli --fara-7b --task "how many pages does wikipedia have" --endpoint_config fara7b_config.json

If you didn't use vLLM to host, specify --base_url [your_base_url] --api_key [your_api_key] --model [your_model_name] when running fara-cli. Please ensure that context length is set to at least 15000 tokens and temperature to 0 for best results.

Runs save a full trajectory — per-step screenshots and a data_point.json (task, actions, observations, and outcome) — to the folder passed via --output_folder.

Reproducibility Copy Chapter

Instructions to reproduce our benchmark results (WebVoyager, Online-Mind2Web, WebTailBench) with the webeval/ framework — including installation, the per-benchmark CLIs, BrowserBase setup, and result analysis — live in docs/eval_reproducibility.md.

Note: the webeval pipeline documented there produced the previous-generation (Fara-7B) numbers and is being updated for the Fara1.5 evaluation stack.

Citation Copy Chapter

If you use Fara1.5 in your research, please use the following BibTeX entry.

@article{fara152026,
  title={Fara1.5: Scalable Learning Environments for Computer Use Agents},
  author={Awadallah, Ahmed and Gupta, Sahil and Lara, Yash and Lu, Yadong and Mozannar, Hussein and Nambi, Akshay and Nussbaum, Zach and Pandya, Yash and Rajeswaran, Aravind and Rosset, Corby and Taymanov, Alexey and do Valle, Luiz and Vineet, Vibhav and Whitehead, Spencer and Zhao, Andrew},
  journal={arXiv:2606.20785},
  year={2026}
}

For the previous generation, Fara-7B:

@article{fara7b2025,
  title={Fara-7B: An Efficient Agentic Model for Computer Use},
  author={Awadallah, Ahmed and Lara, Yash and Magazine, Raghav and Mozannar, Hussein and Nambi, Akshay and Pandya, Yash and Rajeswaran, Aravind and Rosset, Corby and Taymanov, Alexey and Vineet, Vibhav and Whitehead, Spencer and Zhao, Andrew},
  journal={arXiv:2511.19663},
  year={2025}
}

Overview (`webeval/README.md`) Copy Section

webeval

Evaluation framework for Fara-7B
on agentic web benchmarks (WebVoyager, OnlineMind2Web, WebTailBench).

For installation + the full reproducibility CLI, see the repo-root
README.md. Activate the fara_webeval conda env
before running anything below.

This document focuses on a single internal contract that every
benchmark, system, and verifier reads or writes: the trajectory —
i.e. the on-disk record of one agent run on one task.

Trajectory format Copy Chapter

Every executed task lives in its own directory under
<out_url>/runs/<system.hash>/<model_prefix>///<run_id>/traj/<task_id>/,
e.g.:

/path/to/Fara/eval/runs/WebSurfer-fara-100-max_n_images-3/
  fara-7b/<user>/WebTailBench_hf/<run_id>/traj/<task_id>/
    web_surfer.log                 # newline-delimited JSON action+observation log
    <task_id>_final_answer.json    # FinalAnswer dump (final_answer, screenshots, token_usage)
    screenshot_1.png ... screenshot_N.png
    times.json                     # {start_time, end_time, duration} of the solver run
    core.log                       # webeval.core per-task log (Execution / Evaluation lifecycle)
    task_data.json                 # only present after the rubric verifier runs (see below)
    scores/
      <benchmark.eval_hash>.json   # {score: float, gpt_response_text: str|json}

A concrete example is checked in at
webeval/data/example_trajectory/ — a
single WebTailBench AllTrails task (alltrails_find_23) with its
web_surfer.log, alltrails_find_23_final_answer.json,
screenshot_{1..4}.png frames, times.json, core.log,
task_data.json (written by the rubric verifier on re-eval), and the
scores/mmrubric_0.8-5-3.json rubric verdict (score=1). Point
Trajectory.from_folder(...) at that dir to see every field below
populated end-to-end.

`web_surfer.log` (one JSON object per line) Copy Section

Written by the LogHandler in webeval/utils.py, attached to the
WebSurferLogger inside WebSurferSystem.get_answer(). The handler
runs at DEBUG level so the FaraAgent.{generate_model_call, execute_action} methods (src/fara/fara_agent.py) — which emit at
logger.debug(...) — actually flow through. Every line carries a
top-level type field (LogHandler.emit in webeval/utils.py:124).
In the current FaraAgent pipeline only two of them are actually
produced:

Three more event dataclasses (OrchestrationEvent, AgentEvent,
LLMCallEvent) are declared in webeval/systems/messages.py /
webeval/utils.py and handled by LogHandler.emit, but have no
construction callsites in the current repo — they're reserved for
other solver systems / legacy trajectories and won't appear in a
fresh FaraAgent run.

Trajectory.__init__ reads every line and either:

1. Uses each event's top-level action field directly (the
   WebSurferEvent shape), OR
2. If no event has an action field (the all-OtherEvent case), runs
   parse_text_based_event to extract action + arguments (with
   thoughts) from the Thought #N: ... Action #N: executing tool …
   text. This is how trajectories from older WebSurferSystem runs
   are recovered.

After parsing, Trajectory.actions is a list of JSON-encoded action
arg dicts (with thoughts stripped) and Trajectory.thoughts is the
parallel list of thought strings.

Gotcha: if web_surfer.log is empty, Trajectory.events == []
and Trajectory.actions == []. Downstream evaluators (e.g.
WebTailBenchBenchmark.evaluator) treat that as
verifier_reasoning="No actions found in the trajectory" and return
score 0 without invoking MMRubricAgent. The fix is in
WebSurferSystem.get_answer(), which now bumps the logger to DEBUG
before the FaraAgent runs.

`_final_answer.json` (`FinalAnswer` dataclass) Copy Section

Written by WebSurferSystem.get_answer() once the agent terminates or
hits max_rounds. Field reference (webeval/trajectory.py:FinalAnswer):

Persisted via FinalAnswer.save(path); loaded via FinalAnswer.load(path).
Trajectory.__init__ calls FinalAnswer.load(...) on the single
*_final_answer.json it finds in the dir (raises ValueError if 0
or >1 are present), reattaches the loaded FinalAnswer as
trajectory.answer, and exposes the screenshot list directly as
trajectory.screenshots. trajectory.is_aborted is just a
pass-through to trajectory.answer.is_aborted.

`final_answer == ""` is the load-bearing flag

FinalAnswer is constructed with final_answer = "<no_answer>" as the
default, and WebSurferSystem.get_answer() only overwrites it with
the agent's actual answer string when the agent calls terminate(...)
within --max_rounds steps. Three trajectory outcomes therefore look
different on disk:

Filtering "actually completed" runs: anywhere you want trajectories
that finished within step budget AND produced a real model answer, the
canonical check is

candidate.answer.final_answer != "<no_answer>" and not candidate.is_aborted

This is what webeval/post_eval_analysis.py uses when computing
"average score across non-aborted, non-step-budget-exceeded
trajectories", and it's also why WebTailBenchBenchmark.evaluator
short-circuits with verifier_reasoning='The last action must be "terminate"' for any trajectory whose last logged action is not
terminate — those are guaranteed to have final_answer == "<no_answer>" and therefore nothing the rubric judge can grade.

Auto-0 vs excluded: two superficially similar cases are scored
very differently.

• Over-budget (<no_answer>, is_aborted=False): WebTailBench
  short-circuits to score=0 without calling the rubric judge,
  writes it to scores/<eval_hash>.json, and counts it as 0 in
  the mean.
• Aborted / errored mid-run (is_aborted=True, loader fails,
  or evaluator raises): returns score=None, no score file is
  written, and every aggregator filters on score is not None —
  so aborted trajectories are excluded from all final metrics
  rather than counted as 0.

screenshot_{i}.png files keep accumulating regardless of how the
trajectory ended, so screenshot count alone is not a proxy for
"in budget" — always combine it with the final_answer check above.
Note: trajectory.latest_screenshot is a static convenience path
(trajectory.path / "screenshot_scaled.png", see
webeval/trajectory.py:131) that may not exist on disk for every
run — the example directory, for instance, has only
screenshot_1..4.png.

`times.json` Copy Section

{"start_time": , "end_time": , "duration": <float-seconds>}, written around the solver call in
webeval/core.py:run_single_task (webeval/core.py:86-93). Both
stamps are time.time() values (not ISO-8601); e.g. the example is
{"start_time": 1762809276.0520103, "end_time": 1762809454.79363, "duration": 178.74161958694458}. evaluate_single_example reads
duration to populate EvalResult.duration.

`scores/.json` Copy Section

Written by webeval/core.py:evaluate_single_example after a successful
Benchmark.evaluate_example() call. Single writer, single shape across
all benchmarks (top-level {score, gpt_response_text}); the
benchmark-specific verdict lives inside gpt_response_text.

Where each benchmark's verdict lives

The score file is the single source of truth for an evaluation
result. final_answer.json is never overwritten by the verifier.
task_data.json is a separate dump the WebTailBench rubric verifier
writes alongside the score file (benchmarks/webtailbench.py:346 →
[Rubric] Saved rubric results to …/task_data.json), so a fresh
solver-only run won't have one, but any trajectory that's been
re-scored — including the checked-in example — will. Re-running the
verifier (--redo_eval or scripts/verify_trajectories.py) rewrites
the score file in place and, for webtailbench, refreshes
task_data.json with the latest rubric payload.

`core.log` Copy Section

Per-task log lines from the webeval.core.<task_id> logger. Useful
when a task hangs or errors and the rest of the pipeline keeps
moving. A run's core.log typically mixes:

• [Execution <task_id>] Start / Trajectory is not complete. Reexecuting... / Completed / Error
• BrowserBase plumbing: Initializing BrowserBase session...,
  Connected to BrowserBase session: , Closing browser...
• Every WebSurferEvent event re-emitted as its dataclass
  repr(...) (so the same step shows up both here and in
  web_surfer.log, just unparsed)
• [Evaluation <task_id>] Start / Loading from: / Completed: score=<n>, duration=<seconds> / Error
• [Rubric] Saved rubric results to /task_data.json when the
  WebTailBench rubric verifier runs

The example's core.log shows the full sequence for one solve on
2025-11-10 plus three separate rubric re-eval passes on 2026-03-26,
all against the same trajectory directory.

`task_data.json` (optional, rubric-verifier output) Copy Section

Only written by the WebTailBench rubric verifier
(benchmarks/webtailbench/webtailbench.py:346), so trajectories that
have never been re-scored won't have one. When present, it's a large
JSON dump of the task's rubric/outcome state — everything MMRubricAgent
produced plus the task metadata it loaded. In the example,
task_data.json has 76 keys, including task_summary, task_type,
rubric_score, verifier_agent_response, verifier_rubric,
start_timestamp, end_timestamp, token_usage, plan_list,
facts_list, observations_list, actions, actions_summary, and
the boolean error-taxonomy flags (did_web_surfing_fail,
web_surfing_stalled, encountered_captcha, …). Treat this file as
diagnostic detail; the authoritative score still lives in
scores/<eval_hash>.json.

`Trajectory` consumers Copy Chapter

from webeval.trajectory import Trajectory
traj = Trajectory.from_folder("/path/to/<task_id>/")
# Returns None if the dir is missing the answer file or the log can't be parsed.
traj.events                # list[dict] of parsed action/observation events
traj.actions               # list[str]  of JSON-encoded action-arg dicts
traj.thoughts              # list[str]  parallel to .actions
traj.screenshots           # list[Path] absolute paths to screenshot files
traj.answer                # FinalAnswer
traj.is_aborted            # bool (delegates to .answer.is_aborted)
traj.latest_screenshot     # convenience: trajectory.path / "screenshot_scaled.png"

Three places consume it today:

1. Benchmark verifiers (webeval/benchmarks/*/.py): each
   benchmark's evaluator(task_data, candidate) receives a
   Trajectory and returns (score, gpt_response_text). WebTailBench
   wraps MMRubricAgent (the Universal Verifier) for this; webvoyager
   / om2w call their own GPT judges.
2. webeval/scripts/verify_trajectories.py: stand-alone parallel
   runner that points MMRubricAgent at any directory of
   Trajectory-shaped folders and writes per-task
   scores/mmrubric_<…>.json. Useful for re-scoring an existing
   eval run with new judge parameters.
3. webeval/post_eval_analysis.py: aggregates
   _final_answer.json + scores/<…>.json + counts of
   WebSurferEvent lines in web_surfer.log for run-level reporting.

Where each file is written Copy Chapter

Overview (`webeval/src/webeval/benchmarks/webvoyager/README.md`) Copy Section

https://github.com/MinorJerry/WebVoyager/blob/main/evaluation/auto_eval.py to add

Overview (`webeval/src/webeval/benchmarks/om2w/impl/README.md`) Copy Section

Online-Mind2Web Benchmark

Online-Mind2Web benchmark

News Copy Chapter

• [05/11/2025] Check out our updates in the paper.
  • The performance of Claude Computer Use 3.7.
  • WebJudge(o4-mini) achieves high agreement (86%) with a low success rate gap (3.8%) compared with humans.
  • Release WebJudge-7B, a robust and reliable reward model for Reinforcement learning.

Tasks Copy Chapter

Online-Mind2Web includes 300 diverse tasks from 136 popular websites across various domains. It covers a diverse set of real-world user tasks, such as clothing, food, housing, and transportation, to evaluate web agents' performance in a real-world online environment.

Update Tasks Copy Chapter

We will regularly update Online-Mind2Web by replacing outdated or invalid tasks (e.g., due to website changes) to maintain its value as a rigorous benchmark for web agents. If you find any tasks are outdated, please reach out to us, and we will update them.

To ensure fair comparisons, we will aim to keep the updated tasks on the same websites as before and with a similar reference length. Additionally, once agent performance saturates on Online-Mind2Web, we will also revise simple tasks to preserve its long-term value.

Automatic Evaluator via LLM-as-a-Judge (WebJudge)

To enhance the reliability and scalability of the evaluation process in online environments, We propose a more reliable automatic evaluation method called WebJudge, which consists of three parts. (1) Key Point Identification: The model is prompted to identify several key points necessary for completing the task, based on the given instruction and task description. (2) Key Screenshot Identification: Important screenshots are selected from the agent’s trajectory to retain relevant visual evidence while discarding uninformative frames. (3) Outcome Judgment: Output the judgement result based on the task description, key points, key screenshots, and the action history. Our method preserves critical intermediate screenshots while mitigating the token overload issue.

Results

Comparison against Existing Evaluation Methods on Online-Mind2Web Copy Chapter

WebJudge powered by GPT-4o and o4-mini consistently achieves the highest agreement, with averages of 83.6% and 85.7%, respectively. Meanwhile, WebJudge-7B even outperforms o4-mini, reaching a high agreement with human judgment of 87%.

Excellent generalization capabilities on [AgentRewardBench](https://agent-reward-bench.github.io/) (5 OOD benchmarks) Copy Chapter

WebJudge significantly outperforms existing methods, achieving impressive overall precision of 73.7% 75.7% and 82.0% on WebArena (WA), VisualWebArena (VWA), AssistantBench (AB), WorkArena (Work) and WorkArena++ (Wk++) across 1302 trajectories.

The high precision suggests that WebJudge holds potential as a robust and scalable reward model for downstream applications such as Rejection Sampling Fine-Tuning, Reflection, and Reinforcement Learning.

Model Release

We have released the fine-tuned WebJudge-7B weights, which are now available on Hugging Face.

Setup Environment

Create a conda environment and install dependencies:

conda create -n Online_Mind2Web python=3.11
conda activate Online_Mind2Web
pip install -r requirements.txt

Evaluation

You can run the provided example evaluation script directly to perform the evaluation. Adjust the "mode" parameter to choose among various auto-eval methods.

bash ./script/eval.sh

Important Notes for Reliable Evaluation on Online-Mind2Web: Copy Chapter

[!IMPORTANT]

• Start from the specified websites, not Google Search:To enable fair comparisons, please ensure that each task starts from the specified website in our benchmark. Starting from Google Search or alternative websites can lead agents to use different websites to solve the task, resulting in varying difficulty levels and potentially skewed evaluation results.
• Include only factual actions, not agent outputs: The action history should contain only the factual actions taken by the agent to complete the task (e.g., clicking elements and Typing text). Do not include the final response or any other agent's outputs, as they may contain hallucinated content and result in a high rate of false positives.
• Use o4-mini for WebJudge: WebJudge powered by o4-mini demonstrates a higher alignment with human judgment, achieving an average agreement rate of 85.7% and maintaining a narrow success rate gap of just 3.8%. Therefore, please use o4-mini as the backbone for automatic evaluation.

Evaluation Results

In certain scenarios, testing on the full Online-Mind2Web dataset may not be feasible due to cost, privacy, or legal constraints. To facilitate fair and apple-to-apple comparisons, we release both our human evaluation labels and auto-eval details.

• Human Evaluation: Task-level human evaluation labels are provided in the file.
• Auto-Evaluation: The results of WebJudge are available in the folder.

📚 Citation Copy Chapter

Note: Online-Mind2Web is derived from the original Mind2Web dataset. We kindly ask that you cite both the original and this work when using or referencing the data.

@article{xue2025illusionprogressassessingcurrent,
      title={An Illusion of Progress? Assessing the Current State of Web Agents}, 
      author={Tianci Xue and Weijian Qi and Tianneng Shi and Chan Hee Song and Boyu Gou and Dawn Song and Huan Sun and Yu Su},
      year={2025},
      eprint={2504.01382},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2504.01382}, 
}

@inproceedings{deng2023mind2web,
 author = {Deng, Xiang and Gu, Yu and Zheng, Boyuan and Chen, Shijie and Stevens, Sam and Wang, Boshi and Sun, Huan and Su, Yu},
 booktitle = {Advances in Neural Information Processing Systems},
 editor = {A. Oh and T. Naumann and A. Globerson and K. Saenko and M. Hardt and S. Levine},
 pages = {28091--28114},
 publisher = {Curran Associates, Inc.},
 title = {Mind2Web: Towards a Generalist Agent for the Web},
 url = {https://proceedings.neurips.cc/paper_files/paper/2023/file/5950bf290a1570ea401bf98882128160-Paper-Datasets_and_Benchmarks.pdf},
 volume = {36},
 year = {2023}
}

Overview (`webeval/src/webeval/benchmarks/om2w/impl/data/README.md`) Copy Section

Example Copy Chapter

You can refer to the example format when conducting auto-eval. For each task, the result should be stored in a folder named as its task_id, containing :

• trajectory/: Stores screenshots of each step.
• result.json: Task metadata and action history.

🧠 Human Labels Copy Chapter

The file human_label.json provides human evaluation labels for each task across different agents. The meaning of labels represent as following:

• 0: ❌ Failure – The agent did not successfully complete the task.
• 1: ✅ Success – The agent successfully completed the task.
• 2: 🚫 Not Executable – The agent was unable to execute the task due to external or system-related limitations.

ℹ️ Details on Label 2 Copy Chapter

The label 2 indicates that the agent could not execute the task at all. The specific reasons vary by agent:

• SeeAct: Some tasks raise internal bugs when the agent extracts elements from the web page.
• Operator: Certain websites (e.g., Reddit) are inaccessible, preventing task execution.
• Claude Computer Use 3.7: Due to updates on the websites, a few tasks became outdated during the testing phase..
• Agent-E and Browser Use: Label 2 typically corresponds to tasks blocked by CAPTCHAs. There are only two tasks for Agent-E and one task for Browser Use.

Note Copy Chapter

The results on the leaderboard are averaged over three runs, so there may be slight differences compared to the auto-eval results shown here.

Eval Reproducibility (`docs/eval_reproducibility.md`) Copy Section

Eval Reproducibility (webeval)

Note: This document covers the webeval/ pipeline that produced the
published Fara-7B benchmark numbers (WebVoyager, Online-Mind2Web,
WebTailBench). It predates the Fara-1.5 release and will be updated as the
repo is refactored to the Fara-1.5 evaluation stack. For the trajectory
on-disk format, see webeval/README.md.

Agentic evaluations on live websites present unique challenges due to day-to-day changes. We implement several measures to ensure reliable and comparable evaluations:

BrowserBase Integration
We employ BrowserBase to manage browser session hosting, enabling reliable browser instance management.

Time-sensitive Task Updates
Tasks in benchmarks like WebVoyager can become stale or impossible. We:

• Removed ~48 impossible tasks from the original WebVoyager benchmark
• Updated ~50 tasks with future dates to keep them achievable
• Example: "Search for a hotel in Bali from Jan 1 to Jan 4, 2024" → "Search for a hotel in Bali from Jan 1 to Jan 4, 2026"
• Our updated WebVoyager benchmark is available at webeval/data/webvoyager/WebVoyager_data_08312025.jsonl

Environment Error Handling
Browser errors (connection drops, page timeouts) are handled robustly:

• Trajectories are retried up to 5 times when environment errors occur
• Complete yet incorrect trajectories are never retried
• Each retry starts with a fresh browser session, with no retained state

Step Budget
Each trajectory is capped at a maximum of 100 actions across all online benchmarks. Trajectories exceeding this budget without choosing to stop are considered incorrect.

WebEval Package Installation Copy Chapter

conda create --name fara_webeval python=3.12
conda activate fara_webeval

# Install fara package (with vllm extras for GPU hosting)
pip install -e .[vllm]

# Install webeval
cd webeval
pip install -e .

# Install playwright
playwright install

The webeval package no longer depends on autogen-core / autogen-ext —
all chat completion clients are vendored under webeval/src/webeval/oai_clients/
(see GracefulRetryClient, OpenAIClientWrapper, AzureOpenAIClientWrapper,
etc.). You no longer need to clone or install the autogen submodule.

Always activate the fara_webeval env before running any of the eval
scripts below. It pins vllm==0.10.0 + torch==2.7.1; running with
a newer vllm (≥ 0.19) under torch ≥ 2.10 trips a known
torch._dynamo.symbolic_convert crash during CUDA-graph capture, which
can be worked around with --enforce_eager but at a meaningful
throughput cost. Stick to the pinned env.

Running Evaluations Copy Chapter

Navigate to the scripts directory:

cd webeval/scripts

Make sure you set a valid OpenAI GPT-4o endpoint in endpoint_configs_gpt4o/dev in order to run the WebVoyager LLM-as-a-judge!

Option 1: Self-hosted vLLM

/* Detailed source-code truncated for AI context efficiency. */

Option 2: Azure Foundry Deployment

Deploy Fara1.5-9B on Foundry endpoint(s), then place endpoint URLs and keys in JSONs under endpoint_configs/:

python webvoyager.py --model_endpoint ../../endpoint_configs/ --eval_oai_config ../endpoint_configs_gpt4o/dev/ --out_url /path/to/save/eval/files --processes 1 --run_id 1_endpoint --max_rounds 100
python om2w.py --model_endpoint ../../endpoint_configs/ --eval_oai_config ../endpoint_configs_o4/dev/ --eval_model o4-mini --out_url /path/to/save/eval/files --processes 1 --run_id 1_endpoint --max_rounds 100
python webtailbench.py --model_endpoint ../../endpoint_configs/ --eval_oai_config ../../endpoint_configs/judge_active/prod --judge_eval_model gpt-5.2 --judge_o4_eval_model o4-mini --out_url /path/to/Fara/eval --processes 1 --run_id 1_endpoint --max_rounds 100

Notes Copy Section

• We use the same LLM-as-a-judge prompts and model (GPT-4o) as WebVoyager, hence the --eval_oai_config argument
• Set --browserbase for browser session management (requires exported API key and project ID environment variables)
• Avoid overloading a single vLLM deployment with more than ~10 concurrent processes due to known issues
• See debugging output in fara/webeval/scripts/stdout.txt

Analyzing Evaluation Results Copy Chapter

Evaluation Output Structure Copy Section

Evaluation results are stored under --out_url in folders organized by:

• Model name
• Dataset
• Username
• Run ID

Example path:

/runs/WebSurfer-fara-100-max_n_images-3/fara1.5-9b/<username>/WebVoyager_WebVoyager_data_08312025.jsonl/<run_id>

Each evaluation folder contains:

• gpt_eval/ - LLM-as-a-judge evaluation results
• traj/ - Per-task trajectory subdirectories containing:
  • *-final_answer.json (e.g., Amazon--1_final_answer.json) - <no_answer> indicates abortion or step budget exceeded
  • scores/*_eval.json - LLM judge scores (gpt_eval.json for WebVoyager, WebJudge_Online_Mind2Web_eval-3.json for Online-Mind2Web)
  • web_surfer.log - Action history and errors
  • screenshot_X.png - Screenshots captured before each action X
  • times.json - contains start and end times of task
  • core.log - contains high level logs like if the trajectory needs to be started or is already cached/done, the evaluation score, duration, and errors encountered

Running Analysis Copy Section

Use the analysis notebook to compute metrics:

cd webeval/scripts/analyze_eval_results/
jupyter notebook analyze.ipynb

The script:

• Identifies trajectories aborted mid-execution and diagnostic reasons
• Computes average scores across non-aborted trajectories
• Distinguishes between aborted trajectories (errors during sampling) and completed trajectories (with terminate() call or step budget exceeded)

To re-run failed tasks, execute the evaluation script again with the same run_id and username - it will skip non-aborted tasks.

Example WebVoyager GPT Eval Result

{
  "score": 1.0,
  "gpt_response_text": "To evaluate the task, we need to verify if the criteria have been met:\n\n1. **Recipe Requirement**: A vegetarian lasagna recipe with zucchini and at least a four-star rating.\n\n2. **Search and Results**:\n   - The screenshots show that the search term used was \"vegetarian lasagna zucchini.\"\n   - Among the search results, \"Debbie's Vegetable Lasagna\" is prominently featured.\n   \n3. **Evaluation of the Recipe**:\n   - Rating: \"Debbie's Vegetable Lasagna\" has a rating of 4.7, which satisfies the requirement of being at least four stars.\n   - The presence of zucchini in the recipe is implied through the search conducted, though the screenshots do not explicitly show the ingredients list. However, the result response confirms the match to the criteria.\n\nGiven the information provided, the task seems to have fulfilled the requirement of finding a vegetarian lasagna recipe with zucchini and a four-star rating or higher. \n\n**Verdict: SUCCESS**"
}

Example of web_surfer.log (action history)

/* Detailed source-code truncated for AI context efficiency. */

--- METRICS ---

• Files Extracted: 6
• Estimated Token Budget: ~13826 tokens
• Canonical Reference: https://codewiki.google/github.com/microsoft/fara