OmniParser Documentation

File: README.md Copy Chapter

OmniParser: Screen Parsing tool for Pure Vision Based GUI Agent

📢 [Project Page] [V2 Blog Post] [Models V2] [Models V1.5] [HuggingFace Space Demo]

OmniParser is a comprehensive method for parsing user interface screenshots into structured and easy-to-understand elements, which significantly enhances the ability of GPT-4V to generate actions that can be accurately grounded in the corresponding regions of the interface.

News Copy Chapter

• [2026/7] We add a YOLOv9-E interactive region detector. Its inference-only weight is available in Hugging Face PR #37.
• [2025/3] We support local logging of trajecotry so that you can use OmniParser+OmniTool to build training data pipeline for your favorate agent in your domain. [Documentation WIP]
• [2025/3] We are gradually adding multi agents orchstration and improving user interface in OmniTool for better experience.
• [2025/2] We release OmniParser V2 checkpoints. Watch Video
• [2025/2] We introduce OmniTool: Control a Windows 11 VM with OmniParser + your vision model of choice. OmniTool supports out of the box the following large language models - OpenAI (4o/o1/o3-mini), DeepSeek (R1), Qwen (2.5VL) or Anthropic Computer Use. Watch Video
• [2025/1] V2 is coming. We achieve new state of the art results 39.5% on the new grounding benchmark Screen Spot Pro with OmniParser v2 (will be released soon)! Read more details here.
• [2024/11] We release an updated version, OmniParser V1.5 which features 1) more fine grained/small icon detection, 2) prediction of whether each screen element is interactable or not. Examples in the demo.ipynb.
• [2024/10] OmniParser was the #1 trending model on huggingface model hub (starting 10/29/2024).
• [2024/10] Feel free to checkout our demo on huggingface space! (stay tuned for OmniParser + Claude Computer Use)
• [2024/10] Both Interactive Region Detection Model and Icon functional description model are released! Hugginface models
• [2024/09] OmniParser achieves the best performance on Windows Agent Arena!

Install Copy Chapter

First clone the repo, and then install environment:

cd OmniParser
conda create -n "omni" python==3.12
conda activate omni
pip install -r requirements.txt

Until Hugging Face PR #37 is merged, download the latest YOLOv9-E detector from the PR:

huggingface-cli download microsoft/OmniParser-v2.0 icon_detect_v3/model.pt \
  --revision refs/pr/37 --local-dir weights

OmniParser prefers this local weight. After the PR is merged, it will download the same weight automatically on first use. Download the caption weights into the weights folder:

for f in icon_caption/{config.json,generation_config.json,model.safetensors}; do huggingface-cli download microsoft/OmniParser-v2.0 "$f" --local-dir weights; done
   mv weights/icon_caption weights/icon_caption_florence

Examples: Copy Chapter

We put together a few simple examples in the demo.ipynb.

Gradio Demo Copy Chapter

To run gradio demo, simply run:

python gradio_demo.py

Model Weights License Copy Chapter

icon_detect_v3 is based on the MIT-licensed YOLOv9 implementation. Earlier Ultralytics-based icon detectors retain their original AGPL license. The caption models are under the MIT license.

📚 Citation Copy Chapter

Our technical report can be found here.
If you find our work useful, please consider citing our work:

@misc{lu2024omniparserpurevisionbased,
      title={OmniParser for Pure Vision Based GUI Agent}, 
      author={Yadong Lu and Jianwei Yang and Yelong Shen and Ahmed Awadallah},
      year={2024},
      eprint={2408.00203},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2408.00203}, 
}

File: requirements.txt Copy Chapter

torch
easyocr
torchvision
supervision==0.18.0
openai==1.3.5
transformers
huggingface_hub
ultralytics==8.3.70
azure-identity
numpy==1.26.4
opencv-python
opencv-python-headless
gradio
dill
accelerate
timm
einops==0.8.0
paddlepaddle
paddleocr
ruff==0.6.7
pre-commit==3.8.0
pytest==8.3.3
pytest-asyncio==0.23.6
pyautogui==0.9.54
streamlit>=1.38.0
anthropic[bedrock,vertex]>=0.37.1
jsonschema==4.22.0
boto3>=1.28.57
google-auth<3,>=2
screeninfo
uiautomation
dashscope
groq

File: SECURITY.md Copy Chapter

Security Copy Chapter

Microsoft takes the security of our software products and services seriously, which includes all source code repositories managed through our GitHub organizations, which include Microsoft, Azure, DotNet, AspNet and Xamarin.

If you believe you have found a security vulnerability in any Microsoft-owned repository that meets Microsoft's definition of a security vulnerability, please report it to us as described below.

Reporting Security Issues Copy Chapter

Please do not report security vulnerabilities through public GitHub issues.

Instead, please report them to the Microsoft Security Response Center (MSRC) at https://msrc.microsoft.com/create-report.

If you prefer to submit without logging in, send email to [email protected]. If possible, encrypt your message with our PGP key; please download it from the Microsoft Security Response Center PGP Key page.

You should receive a response within 24 hours. If for some reason you do not, please follow up via email to ensure we received your original message. Additional information can be found at microsoft.com/msrc.

Please include the requested information listed below (as much as you can provide) to help us better understand the nature and scope of the possible issue:

• Type of issue (e.g. buffer overflow, SQL injection, cross-site scripting, etc.)
• Full paths of source file(s) related to the manifestation of the issue
• The location of the affected source code (tag/branch/commit or direct URL)
• Any special configuration required to reproduce the issue
• Step-by-step instructions to reproduce the issue
• Proof-of-concept or exploit code (if possible)
• Impact of the issue, including how an attacker might exploit the issue

This information will help us triage your report more quickly.

If you are reporting for a bug bounty, more complete reports can contribute to a higher bounty award. Please visit our Microsoft Bug Bounty Program page for more details about our active programs.

Preferred Languages Copy Chapter

We prefer all communications to be in English.

Policy Copy Chapter

Microsoft follows the principle of Coordinated Vulnerability Disclosure.

File: docs/Evaluation.md Copy Chapter

Eval setup for ScreenSpot Pro

We adapt the eval code from ScreenSpot Pro (ss pro) official repo. This folder contains the inference script/results on this benchmark. We going through legal review proces to release omniparser v2. Once it is done, we will update the file so that it can load the v2 model.

1. eval/ss_pro_gpt4o_omniv2.py: contains the prompt we use, it can be dropped in replacement for this file in the original ss pro repo.
2. eval/logs_sspro_omniv2.json: contains the inferenced results for ss pro using GPT4o+OmniParserv2.