openvino Documentation

OpenVINO™ is an open source toolkit for optimizing and deploying AI inference

11,042 stars C++ C++ 1 file · ~157,228 tokens #ai#computer-vision#deep-learning#deploy-ai#diffusion-models#generative-ai#good-first-issue#inference
RAW Doc

Technical Documentation: openvinotoolkit/openvino

ℹ️ Provenance: Multi-Tier Ingestion · CodeWiki Reference

Overview (README.md)

  • Inference Optimization: Boost deep learning performance in computer vision, automatic speech recognition, generative AI, natural language processing with large and small language models, and many other common tasks.
  • Flexible Model Support: Use models trained with popular frameworks such as PyTorch, TensorFlow, ONNX, Keras, PaddlePaddle, and JAX/Flax. Directly integrate models built with transformers and diffusers from the Hugging Face Hub using Optimum Intel. Convert and deploy models without original frameworks.
  • Broad Platform Compatibility: Reduce resource demands and efficiently deploy on a range of platforms from edge to cloud. OpenVINO™ supports inference on CPU (x86, ARM), GPU (Intel integrated & discrete GPU) and AI accelerators (Intel NPU).
  • Community and Ecosystem: Join an active community contributing to the enhancement of deep learning performance across various domains.

Check out the OpenVINO Cheat Sheet and Key Features for a quick reference.

Installation

Get your preferred distribution of OpenVINO or use this command for quick installation:

sh
pip install -U openvino

Verify Installation

After installation, you can verify OpenVINO is installed correctly by running:

python
import openvino as ov
print(ov.__version__)

Check system requirements and supported devices for detailed information.

Tutorials and Examples

OpenVINO Quickstart example will walk you through the basics of deploying your first model.

Learn how to optimize and deploy popular models with the OpenVINO Notebooks 📚:

Discover more examples in the OpenVINO Samples (Python & C++) and Notebooks (Python).

Here are easy-to-follow code examples demonstrating how to run PyTorch and TensorFlow model inference using OpenVINO:

PyTorch Model

python
import openvino as ov
import torch
import torchvision

# load PyTorch model into memory
model = torch.hub.load("pytorch/vision", "shufflenet_v2_x1_0", weights="DEFAULT")

# convert the model into OpenVINO model
example = torch.randn(1, 3, 224, 224)
ov_model = ov.convert_model(model, example_input=(example,))

# compile the model for CPU device
core = ov.Core()
compiled_model = core.compile_model(ov_model, 'CPU')

# infer the model on random data
output = compiled_model({0: example.numpy()})

TensorFlow Model

python
import numpy as np
import openvino as ov
import tensorflow as tf

# load TensorFlow model into memory
model = tf.keras.applications.MobileNetV2(weights='imagenet')

# convert the model into OpenVINO model
ov_model = ov.convert_model(model)

# compile the model for CPU device
core = ov.Core()
compiled_model = core.compile_model(ov_model, 'CPU')

# infer the model on random data
data = np.random.rand(1, 224, 224, 3)
output = compiled_model({0: data})

OpenVINO supports the CPU, GPU, and NPU devices and works with models from PyTorch, TensorFlow, ONNX, TensorFlow Lite, PaddlePaddle, and JAX/Flax frameworks. It includes APIs in C++, Python, C, NodeJS, and offers the GenAI API for optimized model pipelines and performance.

Generative AI with OpenVINO

Get started with the OpenVINO GenAI installation and refer to the detailed guide to explore the capabilities of Generative AI using OpenVINO.

Learn how to run LLMs and GenAI with Samples in the OpenVINO™ GenAI repo. See GenAI in action with Jupyter notebooks: LLM-powered Chatbot and LLM Instruction-following pipeline.

Documentation

User documentation contains detailed information about OpenVINO and guides you from installation through optimizing and deploying models for your AI applications.

Developer documentation focuses on the OpenVINO architecture and describes building and contributing processes.

OpenVINO Ecosystem

OpenVINO Tools

Integrations

  • 🤗Optimum Intel - grab and use models leveraging OpenVINO within the Hugging Face API.
  • Torch.compile - use OpenVINO for Python-native applications by JIT-compiling code into optimized kernels.
  • ExecuTorch - use ExecuTorch with OpenVINO to optimize and run AI models efficiently.
  • OpenVINO LLMs inference and serving with vLLM​ - enhance vLLM's fast and easy model serving with the OpenVINO backend.
  • OpenVINO Execution Provider for ONNX Runtime - use OpenVINO as a backend with your existing ONNX Runtime code.
  • LLMWare - seamlessly build enterprise AI apps, agentic workflows, RAG pipelines, multimodal use cases, and more, using 100+ inference-ready OpenVINO-optimized SLMs.
  • LlamaIndex - build context-augmented GenAI applications with the LlamaIndex framework and enhance runtime performance with OpenVINO.
  • LangChain - integrate OpenVINO with the LangChain framework to enhance runtime performance for GenAI applications.
  • Keras 3 - Keras 3 is a multi-backend deep learning framework. Users can switch model inference to the OpenVINO backend using the Keras API.

Check out the Awesome OpenVINO repository to discover a collection of community-made AI projects based on OpenVINO!

Performance

Explore OpenVINO Performance Benchmarks to discover the optimal hardware configurations and plan your AI deployment based on verified data.

Contribution and Support

Check out Contribution Guidelines for more details.
Read the Good First Issues section, if you're looking for a place to start contributing. We welcome contributions of all kinds!

You can ask questions and get support on:

Resources

Telemetry

OpenVINO™ collects software performance and usage data for the purpose of improving OpenVINO™ tools.
This data is collected directly by OpenVINO™ or through the use of Google Analytics 4.
You can opt-out at any time by running the command:

bash
opt_in_out --opt_out

More Information is available at OpenVINO™ Telemetry.

License

OpenVINO™ Toolkit is licensed under Apache License Version 2.0.
By contributing to the project, you agree to the license and copyright terms therein and release your contribution under these terms.

* Other names and brands may be claimed as the property of others.

Dev Index (`docs/dev/index.md`)

OpenVINO™ Developer Documentation

Welcome to the OpenVINO™ Developer Documentation. This documentation helps deeper understand the OpenVINO architecture and gives detailed information on the concepts and ideas used inside.

The OpenVINO™ provides a functionality to load models from different frameworks and run them on different accelerators.

text
/* Detailed source-code truncated for AI context efficiency. */

[Get Started](./get_started.md)

OpenVINO Repository Structure

The repository is organized in such a way that the components contain all dependencies (for example, third-party, tests, documentation, and others).

The OpenVINO Repository includes the following components. Click on the component name to get more information:

[openvino/](../../README.md) // OpenVINO Repository .github/ // Github actions and PR templates cmake/ // Global CMake scripts docs/ // OpenVINO documentation licensing/ // Licenses samples/ // OpenVINO samples scripts/ // Helper scripts [src/](../../src/README.md) // Folder with core OpenVINO components tests/ // Infrastructure tests which validate full pipelines thirdparty/ // Common third-party dependencies tools/ // OpenVINO tools

OpenVINO Components

OpenVINO Components include:

  • OpenVINO™ Runtime - is a set of C++ libraries with C and Python bindings providing a common API to deliver inference solutions on the platform of your choice.
    • core - provides the base API for model representation and modification.
    • inference - provides an API to infer models on the device.
    • transformations - contains the set of common transformations which are used in OpenVINO plugins.
    • low precision transformations - contains the set of transformations that are used in low precision models
    • bindings - contains all available OpenVINO bindings which are maintained by the OpenVINO team.
      • c - C API for OpenVINO™ Runtime
      • python - Python API for OpenVINO™ Runtime
  • Plugins - contains OpenVINO plugins which are maintained in open-source by the OpenVINO team. For more information, take a look at the list of supported devices.
  • Frontends - contains available OpenVINO frontends that allow reading models from the native framework format.
  • OpenVINO Model Converter (OVC) - is a cross-platform command-line tool that facilitates the transition between training and deployment environments, and adjusts deep learning models for optimal execution on end-point target devices.
  • Samples - applications in C, C++ and Python languages that show basic OpenVINO use cases.

OpenVINO Component Structure

The OpenVINO component contains all dependencies (for example, third-party, tests, documentation, and others). An example component structure with comments and marks for optional folders is presented below.

text
ov_component/           // Component folder
    cmake/              // (optional) CMake scripts that are related only to this component
    dev_api/            // (optional) Developer API is used when the component provides API for internal developers
    docs/               // (optional) Contains detailed component documentation
    include/            // (optional) Public component API
    src/                // Sources of the component
    tests/              // Tests for the component
    thirdparty/         // (optional) Third-party dependencies
    CMakeLists.txt      // Main CMake script
    README.md           // (optional) Entry point for the developer documentation

Features

See Also

Openvino Sphinx Theme README (`docs/openvino_sphinx_theme/README.md`)

openvino_sphinx_theme

Installation and usage

  1. Install the openvino_sphinx_theme using python:
text
cd openvino/docs/openvino_sphinx_theme
python -m pip install --user .
  1. Update the html_theme variable in your conf.py:
text
html_theme = 'openvino_sphinx_theme'

Configuration

To add a logo at the left of your navigation bar, use html_logo variable to set the path to the logo file.

text
html_logo = <path to the logo file>

Version and language selectors

To enable a version and language selectors, add the following configuration to your conf.py in html_context:

text
html_context = {
    'current_version': 'latest',
    'current_language': 'en',
    'languages': (('English', '/en/latest/'), ('Chinese', '/cn/latest/')),
    'versions': (('latest', '/en/latest/'), ('2022.1', '/en/2022.1'))
}

You can add selectors only for versions or languages.
If you want to add version selector you must define both current_version and versions properties.
If you want to add version selector you must define both current_language and languages properties.

Maintainers

  • OpenVINO Documentation team

Articles En Physical Ai Explanation README (`docs/articles_en/physical-ai/explanation/README.md`)

Explanation

{toctree}
:hidden:
:maxdepth: 1

architecture
inference
runtime
configuration
manifests
cli
robots
cameras

This section contains short concept pages for the runtime package.

Read these pages when you need the model behind the APIs rather than a step-by-step guide.

Pages

Articles En Physical Ai Getting Started README (`docs/articles_en/physical-ai/getting-started/README.md`)

Getting Started

{toctree}
:hidden:
:maxdepth: 1

installation
quickstart
run-a-policy

This section contains short tutorials for the first successful Physical AI workflows.

  1. Installation
  2. Quickstart
  3. Run a Policy

Minimal Path

bash
pip install physicalai
python
from physicalai.inference import InferenceModel

model = InferenceModel("./exports/act_policy")
model.reset()
action = model.select_action(observation)

Use Python when you need direct control over objects. Use YAML and the CLI when you need a reproducible run that can be shared or repeated.

Articles En Physical Ai Getting Started Quickstart (`docs/articles_en/physical-ai/getting-started/quickstart.md`)

Quickstart

This tutorial shows the smallest useful inference flow: load an exported policy package and compute one action.

python
from physicalai.inference import InferenceModel

model = InferenceModel("./exports/act_policy")
model.reset()

action = model.select_action(observation)

The observation input is a dictionary of NumPy arrays. The keys must match the names expected by the exported policy.

For example, an observation may look like this.

python
observation = {
    "state": joint_positions,
    "image.wrist": wrist_image,
    "image.front": front_image,
}

Chunk Policies

Some policies produce action chunks internally. Even in that case, select_action() still returns a single action on each call.

python
for _ in range(100):
    action = model.select_action(observation)
    observation = env.step(action)

If you are building a robot control loop, use chunk prediction through PolicyRuntime instead of managing timing and buffering yourself.

python
chunk = model.predict_action_chunk(observation)

Manifest Path

Exported packages include a manifest.

text
exports/act_policy/
├── manifest.json
├── model.xml
└── stats.safetensors

The manifest records the exported artifacts, the runner configuration, the preprocessing and postprocessing pipeline, and the hardware metadata.

Articles En Physical Ai How To README (`docs/articles_en/physical-ai/how-to/README.md`)

How-To Guides

{toctree}
:hidden:
:maxdepth: 2

runtime/run-policy-on-robot
runtime/use-execution-modes
runtime/add-runtime-callbacks
inference/load-exported-policy
inference/use-manifest
inference/configure-pre-post-processing
config/write-runtime-config
config/write-inference-config
config/instantiate-components
cli/run

This section contains task-oriented guides for common runtime workflows.

Runtime

Inference

Config

CLI

Articles En Physical Ai Reference README (`docs/articles_en/physical-ai/reference/README.md`)

Reference

{toctree}
:hidden:
:maxdepth: 1

cli
config-schema
manifest-schema
inference-api
runtime-api
robot-api
camera-api
benchmark-api

This section provides exact lookup material for commands, schemas, and public contracts.

Pages

Dev Ci Github Actions Overview (`docs/dev/ci/github_actions/overview.md`)

Overview of the OpenVINO GitHub Actions CI

Welcome to the OpenVINO Developer guide on the GitHub Actions infrastructure.
This document will give you an overview of the setup used in OpenVINO and point you at more
detailed instructions where necessary.

Table of Contents

Workflows

GitHub Actions workflows are configurable and automated processes that run one or multiple
consecutive jobs (for more details, refer to the
official GitHub Actions documentation).
They include:

  • a series of commands that you would usually execute in a terminal, one by one
  • information about the environment in which the commands should be executed

You can find all workflows for this repository in the workflows folder.
The main workflows providing broad coverage for different operating systems are:

Additionally, several supporting workflows build and test OpenVINO for other operating systems and processor architectures:

Reusing GitHub Actions

The OpenVINO workflows use both official and community-built actions, such as actions/checkout
and actions/upload-artifact. Additionally, jobs featured in several workflows are extracted
into reusable workflows. You can learn more about using and writing them,
check how to reuse workflows,
and see what and how to obtain additional actions.

Workflows' Triggers and Schedule

Workflows run whenever they are triggered by predefined events.
These triggers are not mutually exclusive and multiple can be used by one workflow.
The Linux workflow uses the following triggers:

  • on: schedule - schedule trigger
    • This trigger runs the workflow on a specified interval (e.g., nightly).
    • In the example below: '0 0 * * 3,6' - learn more on cron syntax
  • on: workflow_dispatch - manual trigger
  • on: pull_request - pre-commit trigger
    • This trigger runs the workflow when a pull request (PR) is created targeting the master or release
      branch and every time the PR is updated with new commits.
  • on: merge_group - merge queue validation trigger
  • on: push - post-commit trigger.
    • This trigger runs the workflow when a commit is pushed to the master or release branch
      (e.g., when a PR is merged).

Expensive validation is selected by Smart CI instead of workflow-level path filters.

The triggers for each workflow can be found at the beginning of a workflow file, in the on
key. You should also learn how to use
paths.

yaml
on:
  schedule:
    # Wednesday and Saturday with the latest oneDNN
    - cron: '0 0 * * 3,6'
    # Tuesday and Thursday without the latest oneDNN
    - cron: '0 0 * * 2,4'
  workflow_dispatch:
  pull_request:
    types: [opened, synchronize, reopened, ready_for_review]
  merge_group:
  push:
    branches:
      - master
      - 'releases/**'

NOTE

Required checks are configured with branch protection and may change independently of this
document. Check the required status checks shown on the pull request. Post-commit and scheduled
workflows, such as the macOS ARM64 workflow, do not validate pull requests.

Workflow Structure

The workflow structures for Linux, Windows, and macOS are mostly the same:

  1. Clone the OpenVINO repository and required resources
  2. Install build dependencies
  3. Build OpenVINO from source
  4. Pack and upload the artifacts (the built OpenVINO and tests)
  5. Download and use the artifacts in the parallel jobs with different tests
  6. Collect the test results and upload them as artifacts

NOTE: some workflows may use the same structure, while others may lack the last 3 steps,
with tests coming right after the Build step.

Overview of the Linux workflow.
There are several jobs present:

yaml
jobs:
  Smart_CI: ...
  Docker: ...
  Build: ...
  Debian_Packages: ...
  Samples: ...
  Conformance: ...
  CXX_Unit_Tests: ...
  Python_Unit_Tests: ...
  Python_API_Tests: ...
  CPU_Functional_Tests: ...
  TensorFlow_Models_Tests_Precommit: ...
  PyTorch_Models_Tests: ...
  JAX_Models_Tests_Precommit: ...
  GGUF_Models_Tests_Precommit: ...
  NVIDIA_Plugin: ...
  Openvino_tokenizers: ...
  iGPU: ...
  Overall_Status: ...

The Build job executes the first 4 steps:

  • clones OpenVINO
  • installs dependencies
  • builds from source with cmake
  • packs and uploads the artifacts using actions/upload-artifact

The other jobs are responsible for running different tests using the built artifacts. They:

  • download and unpack the artifacts using actions/download-artifact
  • install the needed dependencies
  • run tests
  • collect test results
  • upload test results as pipeline artifacts

Single Job Overview

Each job has several keys that describe its environment. Consider checking a comprehensive
syntax overview.

This section describes the specifics of the OpenVINO CI environment.

Overview of the Linux workflow's Python_Unit_Tests job:

yaml
Python_Unit_Tests:
    name: Python unit tests
    needs: [ Docker, Build, Smart_CI ]
    uses: ./.github/workflows/job_python_unit_tests.yml
    with:
      runner: 'aks-linux-4-cores-16gb'
      image: ${{ fromJSON(needs.docker.outputs.images).ov_test.ubuntu_22_04_x64 }}
      affected-components: ${{ needs.smart_ci.outputs.affected_components }}
      python-version: '3.11'
  • Test jobs that consume build artifacts have Build in needs. Jobs using custom images also
    depend on Docker, and component-aware jobs depend on Smart_CI.
  • The machine that is used for a job is specified using the runs-on key.
    • In this case aks-linux-4-cores-16gb is used. Read more on what machines are
      available and how to choose one for a job.
  • Some jobs run inside a Docker container. Reusable workflows receive the image through an
    image input and configure the container internally.
    • Read more on what images are available and when to use one.
  • Some jobs may benefit from caching, for example, Python dependencies or cmake build artifacts.
    • Read more on how to utilize cache for a job.
  • A job must define steps - a series of commands to execute in the predefined environment.
    • All the steps are executed in the shell specified by the shell key under defaults: run:
      unless a shell is specified directly in a step.

Security considerations

Please consult workflow security guidelines before submitting a PR with GitHub Actions workflows changes.

Finding Results, Artifacts, and Logs

Results

To understand which jobs have successfully passed, which are running and which have failed,
check the following:

For Pull Requests:

  • Open a Pull Request and scroll to the bottom of the page. You will see a list of jobs,
    both finished and still running for the most recent commit:

For scheduled runs:

  • Navigate to the OpenVINO Repository Actions
  • Select the required workflow from the list on the left
  • Filter the runs by clicking on Event and selecting schedule
    • You can additionally filter the results per branch, actor and result

Artifacts

Artifacts, that is files produced by the workflow, are available only for the completed pipelines,
both successful or failed. To find artifacts for a pipeline, follow these steps:

  1. Open a Pull Request and scroll to the list of jobs, as described above.

  2. Click Details, to the right of the selected job, to see more information about it.

  3. Click Summary, above the list of jobs on the left side of the window:

  4. Scroll to the bottom of the page

  5. You will find the artifacts produced by all the jobs in this pipeline:

  6. Click on the artifact name to download it.

Initial GOLD BOM baseline

Missing GOLD BOMs can be proposed by the manual
Initialize GOLD BOM Baseline workflow.
It accepts successful build workflow run IDs, collects every bom_* artifact from those runs,
and creates a PR against the branch selected when starting the workflow, including master or
releases/*. Manifests are stored under boms/<configuration>/<component>.json. The
configuration directory comes from the artifact name (for example,
bom_ubuntu_22_04_x86_64 becomes boms/ubuntu_22_04_x86_64/), so Linux distributions,
architectures, and build variants can each have their own baseline. The platform subdirectory
inside the artifact is only an artifact layout detail; it is not repeated in the GOLD path.

Run it from Actions → Initialize GOLD BOM Baseline → Run workflow, selecting the target
branch first. Enter a JSON array of successful build run IDs from the same repository,
covering the configurations to initialize. Both PR runs and branch runs are accepted,
regardless of their source or target branch.
The branch selected for initialization only determines where the GOLD BOM PR is created.
For example:

json
["123456789", "123456790", "123456791"]

The workflow adds only BOM files that are missing on the selected branch. Existing GOLD files
are never overwritten. If all selected manifests already exist, the workflow fails without
creating a PR. For an existing configuration, developers must update changed BOMs manually in
the PR that changes the packaged file list.

When the regular BOM comparison is added, it should be configuration-specific. If the expected
GOLD BOM is absent, there is no baseline to compare: the check should emit a warning with a link
to this documentation and skip comparison for that BOM. This bootstrap workflow can then
initialize the missing BOM through a reviewed PR without changing existing baselines.

Logs

To find logs for a pipeline:

  1. Open a Pull Request and scroll to the list of jobs, as described above.
  2. Click Details, to the right of the selected job, to see more information about it.
  3. Click on a step to see its logs

Custom Actions

Several actions are written specifically for the needs of the OpenVINO workflows. Read more
about the available custom actions and what they do in the custom actions document.

Check the Reusing GitHub Actions section for more information.

Machines

The machines that execute the commands from the workflows are called runners in GitHub Actions.
Two types of runners are available for the OpenVINO organization:

Workflows utilize appropriate runners based on their jobs' needs. Learn more about the
available runners and how to choose one in the OpenVINO Runner Overview.

Docker Images

You can run jobs in Docker containers.
Refer to the documentation for syntax overview.

Workflows utilize appropriate Docker images based on their jobs' needs. Learn more about the
available images and how to choose one in the OpenVINO Docker Image Overview.

Caches

Three types of caches are available:

workflows utilize appropriate caches based on their jobs' needs. Learn more about the
available caches and how to use one in the OpenVINO Cache Overview.

Adding New Tests

If you would like to add new tests, refer to the How to add Tests document.

Optimizing workflows based on PR changes

To optimize pre-commit workflow by running only the jobs that are actually required to validate
changes in a pull request, you can use the Smart CI feature - learn more about it.

Merge Queue

The OpenVINO GitHub organisation utilises Merge Queue to automate the Pull Requests' merging process. Learn more about MQ and available settings in the official documentation.

Merge Queue has several settings that are configured on a per-repository basis. The OpenVINO repository uses the following settings:

  • Merge method: Squash and Merge
  • Build concurrency:
    • Maximum pull requests to build: 5
  • Only merge non-failing Pull Requests: No
  • Status check timeout: 360 minutes
  • Merge limits:
    • Minimum pull requests to merge: 1
    • Maximum pull requests to merge: 2

If you would like to enable Merge Queue for another repository in the OpenVINO GitHub organisation, use these settings as the default ones and adjust if needed.

Agentic Workflows

OpenVINO CI also includes agentic workflows — GitHub Actions workflows that delegate an
investigative or maintenance task to an AI agent instead of running a fixed script. Currently these are
the two CI failure doctors: an on-demand pull request investigator (/ci-doctor) and an automatic
merge-queue failure investigator. Learn more in the Agentic Workflows document.

See also

Src Bindings C Docs Api Overview (`src/bindings/c/docs/api_overview.md`)

Overview of OpenVINO C* API

This API provides a simplified interface for OpenVINO functionality that allows to:

  • handle the models
  • load and configure OpenVINO plugins based on device names
  • perform inference in synchronous and asynchronous modes with arbitrary number of infer requests (the number of infer requests may be limited by target device capabilities)

Supported OSes

The C API is built as part of the OpenVINO Runtime distributions. Refer to the
OpenVINO system requirements
for the maintained operating-system and architecture matrix.

Setting Up the Environment

To configure the environment for the OpenVINO C* API, run:

  • On Linux: source /setupvars.sh

  • On Windows 10/11:

    • . /setupvars.ps1 in PowerShell
    • \setupvars.bat in Command Prompt

The script automatically detects latest installed C* version and configures required environment if the version is supported.

Struct

text
typedef struct ov_version {

    const char* buildNumber;

    const char* description;

} ov_version_t;
text
typedef struct {

    const char* device_name;

    ov_version_t version;

} ov_core_version_t;
text
typedef struct {

    ov_core_version_t* versions;

    size_t size;

} ov_core_version_list_t;
text
typedef struct {

    char** devices;

    size_t size;

} ov_available_devices_t;
text
typedef struct ov_dimension {

    int64_t min;

    int64_t max;

} ov_dimension_t;
text
typedef struct {

    int64_t rank;

    int64_t* dims;

} ov_shape_t;
text
typedef struct ov_partial_shape {

    ov_rank_t rank;

    ov_dimension_t* dims;

} ov_partial_shape_t;
text
typedef struct {

    enum Status {

        NOT_RUN,

        OPTIMIZED_OUT,

        EXECUTED

    } status;

    int64_t real_time;

    int64_t cpu_time;

    const char* node_name;

    const char* exec_type;

    const char* node_type;

} ov_profiling_info_t;
text
typedef struct {

    ov_profiling_info_t* profiling_infos;

    size_t size;

} ov_profiling_info_list_t;
text
typedef struct {

    void(OPENVINO_C_API_CALLBACK* callback_func)(void* args);

    void* args;

} ov_callback_t;
text
typedef enum {

    DYNAMIC = 0U,  //!< Dynamic element type

    BOOLEAN,         //!< boolean element type

    BF16,            //!< bf16 element type

    F16,             //!< f16 element type

    F32,             //!< f32 element type

    F64,             //!< f64 element type

    I4,              //!< i4 element type

    I8,              //!< i8 element type

    I16,             //!< i16 element type

    I32,             //!< i32 element type

    I64,             //!< i64 element type

    U1,              //!< binary element type

    U2,              //!< u2 element type

    U3,              //!< u3 element type

    U4,              //!< u4 element type

    U6,              //!< u6 element type

    U8,              //!< u8 element type

    U16,             //!< u16 element type

    U32,             //!< u32 element type

    U64,             //!< u64 element type

    NF4,             //!< nf4 element type

    F8E4M3,          //!< f8e4m3 element type

    F8E5M3,          //!< f8e5m2 element type

    STRING,          //!< string element type

    F4E2M1,          //!< f4e2m1 element type

    F8E8M0,          //!< f8e8m0 element type

} ov_element_type_e;
text
typedef enum {

    OK = 0,  //!< SUCCESS

    GENERAL_ERROR = -1,       //!< GENERAL_ERROR

    NOT_IMPLEMENTED = -2,     //!< NOT_IMPLEMENTED

    NETWORK_NOT_LOADED = -3,  //!< NETWORK_NOT_LOADED

    PARAMETER_MISMATCH = -4,  //!< PARAMETER_MISMATCH

    NOT_FOUND = -5,           //!< NOT_FOUND

    OUT_OF_BOUNDS = -6,       //!< OUT_OF_BOUNDS

    UNEXPECTED = -7,          //!< UNEXPECTED

    REQUEST_BUSY = -8,        //!< REQUEST_BUSY

    RESULT_NOT_READY = -9,    //!< RESULT_NOT_READY

    NOT_ALLOCATED = -10,      //!< NOT_ALLOCATED

    INFER_NOT_STARTED = -11,  //!< INFER_NOT_STARTED

    NETWORK_NOT_READ = -12,   //!< NETWORK_NOT_READ

    INFER_CANCELLED = -13,    //!< INFER_CANCELLED

    INVALID_C_PARAM = -14,         //!< INVALID_C_PARAM

    UNKNOWN_C_ERROR = -15,         //!< UNKNOWN_C_ERROR

    NOT_IMPLEMENT_C_METHOD = -16,  //!< NOT_IMPLEMENT_C_METHOD

    UNKNOW_EXCEPTION = -17,        //!< UNKNOW_EXCEPTION

} ov_status_e;
text
typedef enum {

    UNDEFINE = 0U,      //!< Undefine color format

    NV12_SINGLE_PLANE,  //!< Image in NV12 format as single tensor

    NV12_TWO_PLANES,    //!< Image in NV12 format represented as separate tensors for Y and UV planes.

    I420_SINGLE_PLANE,  //!< Image in I420 (YUV) format as single tensor

    I420_THREE_PLANES,  //!< Image in I420 format represented as separate tensors for Y, U and V planes.

    RGB,                //!< Image in RGB interleaved format (3 channels)

    BGR,                //!< Image in BGR interleaved format (3 channels)

    GRAY,               //!< Image in GRAY format (1 channel)

    RGBX,               //!< Image in RGBX interleaved format (4 channels)

    BGRX                //!< Image in BGRX interleaved format (4 channels)

} ov_color_format_e;
text
typedef enum {

    RESIZE_LINEAR,  //!< linear algorithm

    RESIZE_CUBIC,   //!< cubic algorithm

    RESIZE_NEAREST  //!< nearest algorithm

} ov_preprocess_resize_algorithm_e;

Properties

common properties

text
/* Detailed source-code truncated for AI context efficiency. */

AUTO plugin specified properties

text
OPENVINO_C_VAR(const char*) ov_property_key_intel_auto_device_bind_buffer;

OPENVINO_C_VAR(const char*) ov_property_key_intel_auto_enable_startup_fallback;

OPENVINO_C_VAR(const char*) ov_property_key_intel_auto_enable_runtime_fallback;

GPU plugin specified properties

text
OPENVINO_C_VAR(const char*) ov_property_key_intel_gpu_context_type;

OPENVINO_C_VAR(const char*) ov_property_key_intel_gpu_ocl_context;

OPENVINO_C_VAR(const char*) ov_property_key_intel_gpu_ocl_context_device_id;

OPENVINO_C_VAR(const char*) ov_property_key_intel_gpu_tile_id;

OPENVINO_C_VAR(const char*) ov_property_key_intel_gpu_ocl_queue;

OPENVINO_C_VAR(const char*) ov_property_key_intel_gpu_va_device;

OPENVINO_C_VAR(const char*) ov_property_key_intel_gpu_shared_mem_type;

OPENVINO_C_VAR(const char*) ov_property_key_intel_gpu_mem_handle;

OPENVINO_C_VAR(const char*) ov_property_key_intel_gpu_dev_object_handle;

OPENVINO_C_VAR(const char*) ov_property_key_intel_gpu_va_plane;

OV Core

This struct represents OpenVINO entity and allows you to manipulate with plugins using unified interfaces.

Create

  • ov_status_e ov_core_create(ov_core_t** core)

    Note: Constructs OpenVINO Core instance by default.

    • Parameters:

      • core - A pointer to the newly created ov_core_t.

    • Return value: Status code of the operation: OK(0) for success.

  • ov_status_e ov_core_create_with_config(const char* xml_config_file, ov_core_t** core)

    Note: Constructs OpenVINO Core instance using XML configuration file with devices description.

    • Parameters:

      • xml_config_file- A full path to.xml file containing plugins configuration. If the parameter is not specified, the default configuration is handled automatically.
      • core - A pointer to the newly created ov_core_t.

    • Return value: Status code of the operation: OK(0) for success.

    • Usage examples:

      Create an ov_core_t t instance with a custom configuration location specified:

      text
      char *xml_config_file="/localdisk/plugins/my_custom_cfg.xml";
ov_core_t* core;
ov_status_e status = ov_core_create_with_config(xml_config_file, &core);

      .xml file has the following structure:

      text
      <ie>
    <plugins>
        <plugin name="" location="" optional="yes/no">
            <extensions>
                <extension location=""/>
            </extensions>
            <properties>
                <property key="" value=""/>
            </properties>
         </plugin>
    </plugins>
</ie>

Methods

  • ov_status_e ov_get_openvino_version(ov_version_t* version)

    • Description: Get version of OpenVINO.

    • Parameters:

      • ov_version_t - a pointer to the version.

    • Return value: Status of the operation: OK(0) for success.

    • Usage example:

      text
      ov_version_t version = {.description = NULL, .buildNumber = NULL};
  ov_get_openvino_version(&version);
  printf("description : %s \n", version.description);
  printf("build number: %s \n", version.buildNumber);
  ov_version_free(&version);

  • ov_status_e ov_core_read_model(const ov_core_t* core, const char* model_path, const char* bin_path, ov_model_t** model)

    • Description: Reads models from IR / ONNX / PDPD / TF / TFLite formats to create ov_model_t.
      You can create as many ov_model_t as you need and use them simultaneously (up to the limitation of the hardware resources).

    • Parameters:

      • core - A pointer to ov_core_t instance.
      • model_path - Path to a model.
      • bin_path - Path to a data file.
      • model - A pointer to the newly created model.

    • Return value: Status code of the operation: OK(0) for success.

    • Usage example:

      text
      ov_core_t* core = NULL;
  ov_core_create(&core);
  ov_model_t* model = NULL;
  ov_core_read_model(core, "model.xml", "model.bin", &model);

  • ov_status_e ov_core_compile_model(const ov_core_t* core, const ov_model_t* model, const char* device_name, const size_t property_args_size, ov_compiled_model_t** compiled_model, ...);

    • Description: Creates a compiled model from a source model object.

    • Parameters:

      • core- A pointer to ov_core_t instance.
      • model Model - An object acquired from Core::read_model.
      • device_name - Name of a device to load a model to.
      • property_args_size - How many properties args will be passed, each property contains 2 args: key and value.
      • compiled_model - A pointer to the newly created compiled_model.
      • ... - property parameter, optional pack of pairs: <char* property_key, char* property_value> relevant only for this load operation operation.

    • Return value: Status code of the operation: OK(0) for success.

    • Usage example:

      text
      ov_core_t* core = nullptr;
  ov_core_create(&core);
  ov_model_t* model = nullptr;
  ov_core_read_model(core, xml_file_name.c_str(), bin_file_name.c_str(), &model);
  const char* key = ov_property_key_hint_performance_mode;
  const char* num = "LATENCY";
  ov_compiled_model_t* compiled_model = nullptr;
  ov_core_compile_model(core, model, "CPU", 2, &compiled_model, key, num);
  ...
  ov_compiled_model_free(compiled_model);
  ov_model_free(model);
  ov_core_free(core);

  • ov_status_e ov_core_set_property(const ov_core_t* core, const char* device_name, ...)

    • Description: Sets properties for a device, acceptable keys can be found in ov_property_key_xxx.

    • Parameters:

      • core - A pointer to ov_core_t instance.
      • device_name - Name of a device.
      • ... - property paramaters, optional pack of pairs: <char* property_key, char* property_value>.

    • Return value: Status code of the operation: OK(0) for success.

    • Usage example:

      text
      ov_core_t* core = nullptr;
  ov_core_create(&core);
  const char* key_1 = ov_property_key_inference_num_threads;
  const char* value_1 = "12";
  const char* key_2 = ov_property_key_num_streams;
  const char* value_2 = "7";
  ov_core_set_property(core, "CPU", key_1, value_1, key_2, value_2);
  ...
  ov_core_free(core);

  • ov_status_e ov_core_get_property(const ov_core_t* core, const char* device_name, const char* property_key, char** property_value)

    • Description: Gets properties related to device behaviour.

    • Parameters:

      • core - A pointer to ov_core_t instance.
      • device_name - Name of a device.
      • property_key - Property key.
      • property_value - A pointer to property value with string format.

    • Return value: Status code of the operation: 0 for success.

    • Usage example:

      text
      ov_core_t* core = nullptr;
  ov_core_create(&core);
  const char* key = ov_property_key_hint_performance_mode;
  const char* mode = "LATENCY";
  ov_core_set_property(core, "CPU", key, mode);
  char* ret = nullptr;
  ov_core_get_property(core, "CPU", key, &ret);
  ov_free(ret);
  ...
  ov_core_free(core);

  • ov_status_e ov_core_import_model(const ov_core_t* core, const char* content, const size_t content_size, const char* device_name, ov_compiled_model_t** compiled_model);

    • Description: Imports a compiled model from the previously exported one.

    • Parameters:

      • core - A pointer ov_core_t instance.
      • content - A pointer to content of the exported model.
      • content_size - Number of bytes in the exported network.
      • device_name - Name of a device to import a compiled model for.
      • compiled_model - A pointer to the newly created compiled_model.

    • Return value: Status code of the operation: OK(0) for success.

  • ov_status_e ov_core_get_versions_by_device_name(const ov_core_t* core, const char* device_name, ov_core_version_list_t* versions)

    • Description: Returns device plugins version information.
    • Parameters:
      • core - A pointer ov_core_t instance.
      • device_name - A device name to identify a plugin.
      • versions - A pointer to versions corresponding to device_name.
    • Return value: Status code of the operation: OK(0) for success.

  • ov_status_e ov_core_create_context(const ov_core_t* core, const char* device_name, const size_t context_args_size, ov_remote_context_t** context, ...);

    • Description: Creates a new remote shared context object on the specified accelerator device using specified plugin-specific low-level device API parameters (device handle, pointer, context, etc.).
    • Parameters:
      • core - A pointer ov_core_t instance.
      • device_name - Device name to identify a plugin.
      • context_args_size - How many property args will be for this remote context creation.
      • context - A pointer to the newly created remote context.
      • ... - variadic parmameters Actual property parameter for remote context
    • Return value: Status code of the operation: OK(0) for success.

  • ov_status_e ov_core_compile_model_with_context(const ov_core_t* core, const ov_model_t* model, const ov_remote_context_t* context, const size_t property_args_size, ov_compiled_model_t** compiled_model, ...);

    • Description: Creates a compiled model from a source model within a specified remote context.

    • Parameters:

      • core - A pointer ov_core_t instance.
      • model - Model object acquired from ov_core_read_model.
      • context - A pointer to the newly created remote context.
      • property_args_size - How many args will be for this compiled model.
      • compiled_model - A pointer to the newly created compiled_model.
      • ... - variadic parmameters Actual property parameter for remote context

    • Return value: Status code of the operation: OK(0) for success.

  • ov_status_e ov_core_add_extension(const ov_core_t* core, const char* library_path)

    • Description: Adds an extension to the core.
    • Parameters:
      • core - A pointer ov_core_t instance.
      • library_path - Path to an extension.
    • Return value: Status code of the operation: OK(0) for success.

OV Model

This struct contains the information about the model read from IR and allows you to manipulate with some model parameters such as layers affinity and output layers.

Methods

  • ov_status_e ov_model_free(ov_model_t* model)

    • Description: Release the memory allocated by ov_model_t.
    • Parameters:
      • model - A pointer to the ov_model_t to free memory..
    • Return value: Status code of the operation: OK(0) for success.

  • ov_status_e ov_model_const_input(const ov_model_t* model, ov_output_const_port_t** input_port);

    • Description: Get a const input port of ov_model_t,which only support single input model.
    • Parameters:
      • model - A pointer to the ov_model_t.
      • input_port - A pointer to the ov_output_const_port_t.
    • Return value: Status code of the operation: OK(0) for success.

  • ov_status_e ov_model_input(const ov_model_t* model, ov_output_port_t** input_port);

  • Description: Get single input port of ov_model_t, which only support single input model.

  • Parameters:

    • model - A pointer to the ov_model_t.
    • input_port - A pointer to the ov_output_port_t.

  • Return value: Status code of the operation: OK(0) for success.

  • ov_status_e ov_model_input_by_name(const ov_model_t* model, const char* tensor_name, ov_output_port_t** input_port)

    • Description: Get an input port of ov_model_t by name.
    • Parameters:
      • model - A pointer to the ov_model_t.
      • tensor_name - Input tensor name (char *).
      • input_port - A pointer to the ov_output_port_t.
    • Return value: Status code of the operation: OK(0) for success.

  • ov_status_e ov_model_input_by_index(const ov_model_t* model, const size_t index, ov_output_port_t** input_port)

    • Description: Get an input port of ov_model_t by name.
    • Parameters:
      • model - A pointer to the ov_model_t.
      • index - Input tensor index.
      • input_port - A pointer to the ov_output_port_t.
    • Return value: Status code of the operation: OK(0) for success.

  • ov_status_e ov_model_const_output(const ov_model_t* model, ov_output_const_port_t** output_port);

    • Description: Get a single const output port of ov_model_t, which only support single output model.
    • Parameters:
      • model - A pointer to the ov_model_t.
      • output_port - A pointer to the ov_output_const_port_t.
    • Return value: Status code of the operation: OK(0) for success.

  • ov_status_e ov_model_output(const ov_model_t* model, ov_output_port_t** output_port);

    • Description: Get a single output port of ov_model_t, which only support single output model.
    • Parameters:
      • model - A pointer to the ov_model_t.
      • output_port - A pointer to the ov_output_port_t.
    • Return value: Status code of the operation: OK(0) for success.

  • ov_status_e ov_model_inputs_size(const ov_model_t* model, size_t* input_size);

    • Description: Get the input size of ov_model_t.
    • Parameters:
      • model - A pointer to the ov_model_t.
      • input_size - The model's input size.
    • Return value: Status code of the operation: OK(0) for success.

  • ov_status_e ov_model_outputs_size(const ov_model_t* model, size_t* output_size);

    • Description: Get the input size of ov_model_t.
    • Parameters:
      • model - A pointer to the ov_model_t.
      • output_size - The model's output size.
    • Return value: Status code of the operation: OK(0) for success.

  • bool ov_model_is_dynamic(const ov_model_t* model)

    • Description: Returns true if any of the ops defined in the model is dynamic shape.
    • Parameters:
      • model - A pointer to the ov_model_t.
    • Return value: true if model contains dynamic shapes.

  • ov_status_e ov_model_reshape(const ov_model_t* model, const char** tensor_names, const ov_partial_shape_t* partial_shapes, size_t size)

    • Description: Do reshape in model with a list of <name, partial shape>.
    • Parameters:
      • model - A pointer to the ov_model_t.
      • tensor_names - The list of input tensor names.
      • partialShape - A PartialShape list.
      • size - The item count in the list.
    • Return value: Status code of the operation: OK(0) for success.

  • ov_status_e ov_model_get_friendly_name(const ov_model_t* model, char** friendly_name)

    • Description: Gets the friendly name for a model.
    • Parameters:
      • model - A pointer to the ov_model_t.
      • friendly_name - The model's friendly name.
    • Return value: Status code of the operation: OK(0) for success.

Node

This struct contains the information about the model's port.

Methods

  • ov_status_e ov_const_port_get_shape(const ov_output_const_port_t* port, ov_shape_t* tensor_shape)

    • Description: Get the shape of port object.
    • Parameters:
      • port - A pointer to ov_output_const_port_t.
      • tensor_shape - Returned tensor shape.
    • Return value: Status code of the operation: OK(0) for success.

  • ov_status_e ov_port_get_shape(const ov_output_port_t* port, ov_shape_t* tensor_shape)

    • Description: Get the shape of port object.
    • Parameters:
      • port - A pointer to ov_output_port_t.
      • tensor_shape - Returned tensor shape.
    • Return value: Status code of the operation: OK(0) for success.

  • ov_status_e ov_port_get_any_name(const ov_output_const_port_t* port, char** tensor_name)

    • Description: Get the tensor name of port.
    • Parameters:
      • port - A pointer to ov_output_port_t.
      • tensor_name - Returned tensor name.
    • Return value: Status code of the operation: OK(0) for success.

  • ov_status_e ov_port_get_partial_shape(const ov_output_const_port_t* port, ov_partial_shape_t* partial_shape)

    • Description: Get the partial shape of port.
    • Parameters:
      • port - A pointer to ov_output_const_port_t.
      • partial_shape - Partial shape.
    • Return value: Status code of the operation: OK(0) for success.

  • ov_status_e ov_port_get_element_type(const ov_output_const_port_t* port, ov_element_type_e* tensor_type)

    • Description: Get the tensor type of port.
    • Parameters:
      • port - A pointer to ov_output_const_port_t.
      • tensor_type - Returned tensor type.
    • Return value: Status code of the operation: OK(0) for success.

  • void ov_output_port_free(ov_output_port_t* port)

    • Description: free port object.
    • Parameters:
      • port - A pointer to ov_output_port_t.
    • Return value: no return.

  • void ov_output_const_port_free(ov_output_const_port_t* port)

    • Description: free const port object.
    • Parameters:
      • port - A pointer to ov_output_const_port_t.
    • Return value: no return.

CompiledModel

This struct represents a compiled model instance loaded to plugin and ready for inference.

Methods

  • ov_status_e ov_compiled_model_inputs_size(const ov_compiled_model_t* compiled_model, size_t* size)

    • Description: Get the input size of ov_compiled_model_t.
    • Parameters:
      • compiled_model - A pointer to the ov_compiled_model_t instance.
      • input_size - The compiled_model's input size.
    • Return value: Status code of the operation: OK(0) for success.

  • ov_status_e ov_compiled_model_input(const ov_compiled_model_t* compiled_model, ov_output_const_port_t** input_port)

    • Description: - Get the single const input port of ov_compiled_model_t, which only support single input model.
    • Parameters:
      • compiled_model - A pointer to the ov_compiled_model_t instance.
      • input_port - A pointer to the ov_output_const_port_t instance.
    • Return value: Status code of the operation: OK(0) for success.

  • ov_status_e ov_compiled_model_input_by_index(const ov_compiled_model_t* compiled_model, const size_t index, ov_output_const_port_t** input_port)

    • Description: Get a const input port of ov_compiled_model_t by port index.
    • Parameters:
      • compiled_model - A pointer to the ov_compiled_model_t instance.
      • index: Input index.
      • input_port - A pointer to the ov_output_const_port_t instance.
    • Return value: Status code of the operation: OK(0) for success.

  • ov_status_e ov_compiled_model_input_by_name(const ov_compiled_model_t* compiled_model, const char* name, ov_output_const_port_t** input_port)

    • Description: - Get a const input port of ov_compiled_model_t by name.
    • Parameters:
      • compiled_model - A pointer to the ov_compiled_model_t instance.
      • name - input tensor name.
      • input_port - A pointer to the ov_output_const_port_t instance.
    • Return value: Status code of the operation: OK(0) for success.

  • ov_compiled_model_outputs_size(const ov_compiled_model_t* compiled_model, size_t* size)

    • Description: - Get the output size of ov_compiled_model_t.
    • Parameters:
      • compiled_model - A pointer to the ov_compiled_model_t instance.
      • size - The compiled_model's output size.
    • Return value: Status code of the operation: OK(0) for success.

  • ov_status_e ov_compiled_model_output(const ov_compiled_model_t* compiled_model, ov_output_const_port_t** output_port)

    • Description: - Get the single const output port of ov_compiled_model_t, which only support single output model.
    • Parameters:
      • compiled_model - A pointer to the ov_compiled_model_t instance.
      • output_port - A pointer to the ov_output_const_port_t instance.
    • Return value: Status code of the operation: OK(0) for success.

  • ov_status_e ov_compiled_model_output_by_index(const ov_compiled_model_t* compiled_model, const size_t index, ov_output_const_port_t** output_port)

    • Description: Get a const input port of ov_compiled_model_t by port index.
    • Parameters:
      • compiled_model - A pointer to the ov_compiled_model_t instance.
      • index: Output index.
      • output_port - A pointer to the ov_output_const_port_t instance.
    • Return value: Status code of the operation: OK(0) for success.

  • ov_status_e ov_compiled_model_output_by_name(const ov_compiled_model_t* compiled_model, const char* name, ov_output_const_port_t** output_port)

    • Description: - Get a const output port of ov_compiled_model_t by name.
    • Parameters:
      • compiled_model - A pointer to the ov_compiled_model_t instance.
      • name - input tensor name.
      • output_port - A pointer to the ov_output_const_port_t instance.
    • Return value: Status code of the operation: OK(0) for success.

  • ov_status_e ov_compiled_model_get_runtime_model(const ov_compiled_model_t* compiled_model, ov_model_t** model)

    • Description: - Gets runtime model information from a device.
    • Parameters:
      • compiled_model - A pointer to the ov_compiled_model_t instance.
      • model - A pointer to the ov_model_t instance.
    • Return value: Status code of the operation: OK(0) for success.

  • ov_status_e ov_compiled_model_create_infer_request(const ov_compiled_model_t* compiled_model, ov_infer_request_t** infer_request)

    • Description: - Creates an inference request object used to infer the compiled model.
    • Parameters:
      • compiled_model - A pointer to ov_compiled_model_t instance.
      • infer_request - A pointer to ov_infer_request_t instance.
    • Return value: Status code of the operation: OK(0) for success.

  • ov_status_e ov_compiled_model_set_property(const ov_compiled_model_t* compiled_model, ...)

    • Description: - Sets properties for a device, acceptable keys can be found in ov_property_key_xxx.
    • Parameters:
      • compiled_model - A pointer to ov_compiled_model_t instance.
      • ... variadic paramaters, the format is <char *property_key, char* property_value>.
    • Return value: Status code of the operation: OK(0) for success.

  • ov_status_e ov_compiled_model_get_property(const ov_compiled_model_t* compiled_model, const char* property_key, char** property_value)

    • Description: - Gets properties for current compiled model.
    • Parameters:
      • compiled_model - A pointer to ov_compiled_model_t instance.
      • property_key - Property key.
      • property_value - A pointer to property value.
    • Return value: Status code of the operation: OK(0) for success.

  • ov_status_e ov_compiled_model_export_model(const ov_compiled_model_t* compiled_model, const char* export_model_path)

    • Description: - Exports the current compiled model to an output stream std::ostream.
    • Parameters:
      • compiled_model - A pointer to ov_compiled_model_t instance.
      • export_model_path - Path to the file.
    • Return value: Status code of the operation: OK(0) for success.

  • void ov_compiled_model_free(ov_compiled_model_t* compiled_model)

    • Description: - Release the memory allocated by ov_compiled_model_t`.
    • Parameters:
      • compiled_model - A pointer to ov_compiled_model_t instance.
    • Return value: None

InferRequest

This struct provides an interface to infer requests of ov_compiled_model_t and serves to handle infer requests execution and to set and get output data.

Methods

  • ov_status_e ov_infer_request_set_tensor(ov_infer_request_t* infer_request, const char* tensor_name, const ov_tensor_t* tensor)

    • Description: Set an input/output tensor to infer on by the name of tensor.
    • Parameters:
      • infer_request - A pointer to ov_infer_request_t instance.
      • tensor_name - Name of the input or output tensor.
      • tensor - Reference to the tensor.
    • Return value: Status code of the operation: OK(0) for success.

  • ov_status_e ov_infer_request_set_tensor_by_port(ov_infer_request_t* infer_request, const ov_output_port_t* port, const ov_tensor_t* tensor)

    • Description: Set an input/output tensor to infer request for the port.
    • Parameters:
      • infer_request - A pointer to ov_infer_request_t instance.
      • port - Port of the input or output tensor, which can be got by calling ov_model_t/ov_compiled_model_t interface.
      • tensor - Reference to the tensor.
    • Return value: Status code of the operation: OK(0) for success.

  • ov_status_e ov_infer_request_set_input_tensor_by_index(ov_infer_request_t* infer_request, const size_t idx, const ov_tensor_t* tensor)

    • Description: Set an input tensor to infer on by the index of tensor.
    • Parameters:
      • infer_request - A pointer to ov_infer_request_t instance.
      • idx - Index of the input port.
      • tensor - Reference to the tensor.
    • Return value: Status code of the operation: OK(0) for success.

  • ov_status_e ov_infer_request_set_input_tensor(ov_infer_request_t* infer_request, const ov_tensor_t* tensor)

    • Description: Set an input tensor for the model with single input to infer on.
    • Parameters:
      • infer_request - A pointer to ov_infer_request_t instance.
      • tensor - Reference to the tensor.
    • Return value: Status code of the operation: OK(0) for success.

  • ov_status_e ov_infer_request_set_output_tensor_by_index(ov_infer_request_t* infer_request, const size_t idx, const ov_tensor_t* tensor)

    • Description: Set an output tensor to infer by the index of output tensor.
    • Parameters:
      • infer_request - A pointer to ov_infer_request_t instance.
      • idx - Index of the input port.
      • tensor - Reference to the tensor.
    • Return value: Status code of the operation: OK(0) for success.

  • ov_status_e ov_infer_request_set_output_tensor(ov_infer_request_t* infer_request, const ov_tensor_t* tensor)

    • Description: Set an output tensor to infer models with single output.
    • Parameters:
      • infer_request - A pointer to ov_infer_request_t instance.
      • tensor - Reference to the tensor.
    • Return value: Status code of the operation: OK(0) for success.

  • ov_status_e ov_infer_request_get_tensor(const ov_infer_request_t* infer_request, const char* tensor_name, ov_tensor_t** tensor)

    • Description: Get an input/output tensor by the name of tensor.
    • Parameters:
      • infer_request - A pointer to ov_infer_request_t instance.
      • tensor_name - Name of the input or output tensor.
      • tensor - Reference to the tensor.
    • Return value: Status code of the operation: OK(0) for success.

  • ov_status_e ov_infer_request_get_tensor_by_port(const ov_infer_request_t* infer_request, const ov_output_port_t* port, ov_tensor_t** tensor)

    • Description: Get an input/output tensor by port.
    • Parameters:
      • infer_request - A pointer to ov_infer_request_t instance.
      • port - Port of the tensor to get.
      • tensor - Reference to the tensor.
    • Return value: Status code of the operation: OK(0) for success.

  • ov_status_e ov_infer_request_get_input_tensor_by_index(const ov_infer_request_t* infer_request, const size_t idx, ov_tensor_t** tensor)

    • Description: Get an input tensor by the index of input tensor.
    • Parameters:
      • infer_request - A pointer to ov_infer_request_t instance.
      • idx - Index of the tensor to get.
      • tensor - Reference to the tensor.
    • Return value: Status code of the operation: OK(0) for success.

  • ov_status_e ov_infer_request_get_output_tensor_by_index(const ov_infer_request_t* infer_request, const size_t idx, ov_tensor_t** tensor)

    • Description: Get an output tensor by the index of output tensor.
    • Parameters:
      • infer_request - A pointer to ov_infer_request_t instance.
      • idx - Index of the tensor to get.
      • tensor - Reference to the tensor.
    • Return value: Status code of the operation: OK(0) for success.

  • ov_status_e ov_infer_request_infer(ov_infer_request_t* infer_request)

    • Description: Infer specified input(s) in synchronous mode.
    • Parameters:
      • infer_request - A pointer to ov_infer_request_t instance.
    • Return value: Status code of the operation: OK(0) for success.

  • ov_status_e ov_infer_request_start_async(ov_infer_request_t* infer_request)

    • Description: Start inference of specified input(s) in asynchronous mode.
    • Parameters:
      • infer_request - A pointer to ov_infer_request_t instance.
    • Return value: Status code of the operation: OK(0) for success.

  • ov_status_e ov_infer_request_cancel(ov_infer_request_t* infer_request)

    • Description: Cancel inference request.
    • Parameters:
      • infer_request - A pointer to ov_infer_request_t instance.
    • Return value: Status code of the operation: OK(0) for success.

  • ov_status_e ov_infer_request_wait_for(ov_infer_request_t* infer_request, const int64_t timeout);

    • Description: Waits for the result to become available. Blocks until the specified timeout has elapsed or the result becomes available, whichever comes first.
    • Parameters:
      • infer_request - A pointer to ov_infer_request_t instance.
      • timeout - Maximum duration, in milliseconds, to block for.
    • Return value: Status code of the operation: OK(0) for success.

  • ov_status_e ov_infer_request_set_callback(ov_infer_request_t* infer_request, const ov_callback_t* callback)

    • Description: Set callback function, which will be called when inference is done.
    • Parameters:
      • infer_request - A pointer to ov_infer_request_t instance.
      • callback A function to be called.
    • Return value: Status code of the operation: OK(0) for success.

  • void ov_infer_request_free(ov_infer_request_t* infer_request)

    • Description: Release the memory allocated by ov_infer_request_t.
    • Parameters:
      • infer_request - A pointer to ov_infer_request_t instance.
    • Return value: None.

  • void ov_infer_request_get_profiling_info(const ov_infer_request_t* infer_request, ov_profiling_info_list_t* profiling_infos)

    • Description: Query performance measures per layer to identify the most time consuming operation.
    • Parameters:
      • infer_request - A pointer to ov_infer_request_t instance.
      • profiling_infos - Vector of profiling information for operations in a model.
    • Return value: None.

Tensor

Methods

  • ov_status_e ov_tensor_create_from_host_ptr(const ov_element_type_e type, const ov_shape_t shape, void* host_ptr, ov_tensor_t** tensor)

    • Description: Constructs Tensor using element type, shape and external host ptr.
    • Parameters:
      • type - Tensor element type
      • shape - Tensor shape
      • host_ptr - Pointer to pre-allocated host memory
      • tensor - A point to ov_tensor_t
    • Return value: Status code of the operation: OK(0) for success.

  • ov_status_e ov_tensor_create_from_string_array(const char** string_array, const size_t array_size, const ov_shape_t shape, ov_tensor_t** tensor)

    • Description: Constructs Tensor using string array and size.
      • string_array - String array
      • array_size - Number of elements in string array
      • shape - Tensor shape
      • tensor - A point to ov_tensor_t
    • Return value: Status code of the operation: OK(0) for success.

  • ov_status_e ov_tensor_create(const ov_element_type_e type, const ov_shape_t shape, ov_tensor_t** tensor)

    • Description: Constructs Tensor using element type and shape. Allocate internal host storage using default allocator.
    • Parameters:
      • type - Tensor element type
      • shape - Tensor shape
      • tensor - A point to ov_tensor_t
    • Return value: Status code of the operation: OK(0) for success.

  • ov_status_e ov_tensor_get_shape(const ov_tensor_t* tensor, ov_shape_t* shape)

    • Description: Get shape for tensor.
    • Parameters:
      • tensor - A point to ov_tensor_t
      • shape - Tensor shape
    • Return value: Status code of the operation: OK(0) for success.

  • ov_status_e ov_tensor_get_element_type(const ov_tensor_t* tensor, ov_element_type_e* type)

    • Description: Get type for tensor.
    • Parameters:
      • tensor - A point to ov_tensor_t
      • type - Tensor element type.
    • Return value: Status code of the operation: OK(0) for success.

  • ov_status_e ov_tensor_get_byte_size(const ov_tensor_t* tensor, size_t* byte_size)

    • Description: Get byte size for tensor.
    • Parameters:
      • tensor - A point to ov_tensor_t
      • byte_size - The size of the current Tensor in bytes.
    • Return value: Status code of the operation: OK(0) for success.

  • ov_status_e ov_tensor_data(const ov_tensor_t* tensor, void** data)

    • Description: Provides an access to the underlaying host memory.
    • Parameters:
      • tensor - A point to ov_tensor_t
      • data - A point to host memory.
    • Return value: Status code of the operation: OK(0) for success.

  • ov_status_e ov_tensor_set_string_data(ov_tensor_t* tensor, const char** string_array, size_t array_size)

    • Description: Set string data for tensor
    • Parameters:
      • tensor - A point to ov_tensor_t
      • string_array - String array
      • array_size - Number of elements in string array
    • Return value: Status code of the operation: OK(0) for success.

  • void ov_tensor_free(ov_tensor_t* tensor)

    • Description: Free ov_tensor_t.
    • Parameters:
      • tensor - A point to ov_tensor_t
    • Return value: None.

Src Bindings Js Docs README (`src/bindings/js/docs/README.md`)

OpenVINO™ JavaScript Bindings

Folders

`openvino-node` Package Developer Documentation

Components

  • include - header files for current API.
  • lib - TypeScript sources for current API.
  • src - C++ sources for current API.
  • tests - tests directory for current API.

Build

  • Make sure that all submodules are updated:
    bash
    git submodule update --init --recursive
  • Create the build directory:
    bash
    mkdir build && cd build
  • Configure building of the binaries:
    bash
    cmake \
  -DCMAKE_BUILD_TYPE=Release \
  -DCPACK_GENERATOR=NPM \
  -DENABLE_SYSTEM_TBB=OFF -UTBB* \
  -DENABLE_TESTS=OFF \
  -DENABLE_SAMPLES=OFF \
  -DENABLE_WHEEL=OFF \
  -DENABLE_PYTHON=OFF \
  -DENABLE_INTEL_GPU=OFF \
  -DCMAKE_INSTALL_PREFIX="../src/bindings/js/node/bin" \
  ..
  • Build the bindings:
    bash
    cmake --build . --config Release --verbose -j4
  • Install binaries for the openvino-node package:
    bash
    cmake --install .
  • Navigate to the npm package folder:
    bash
    cd ../src/bindings/js/node
  • Now, you can install dependencies packages and transpile ts to js code:
    bash
    npm install
  • Run tests to make sure that openvino-node has been built successfully:
    bash
    npm run test

Usage

  • Add the openvino-node package to your project by specifying it in package.json:
    json
    "openvino-node": "file:*path-to-current-directory*"
  • Make sure to require it:
    js
    const { addon: ov } = require('openvino-node');

Samples

OpenVINO™ Node.js Bindings Examples of Usage

Contributing

Your contributions are welcome! Make sure to read the Contribution Guide to learn how you can get involved.

See Also

Src Common Transformations Docs README (`src/common/transformations/docs/README.md`)

Transformations documentation

Writing transformations

Writing tests

Debug capabilities

Key Contacts

For assistance regarding snippets, contact a member of openvino-ie-transformations-maintainers group.

See also

Src Common Snippets Docs Debug Capabilities README (`src/common/snippets/docs/debug_capabilities/README.md`)

Debug capabilities

Debug capabilities are the set of useful debug features, most of them are controlled by environment variables.

They can be activated at runtime and might be used to analyze issues, locate faulty code, get more execution details, benchmark execution time, etc.

Use the following cmake option to enable snippets debug capabilities:

-DENABLE_DEBUG_CAPS=ON

Src Common Transformations Docs Debug Capabilities README (`src/common/transformations/docs/debug_capabilities/README.md`)

Debug capabilities

Debug capabilities are the set of useful debug features for OpenVINO transformations, controlled by environment variables.

They can be activated at runtime and are useful for analyzing transformation behavior, profiling pass execution, and inspecting model state between passes.

  • Matcher logging
    When to use: a MatcherPass transformation is not firing or matching unexpectedly — logs the pattern matching process to show why matches succeed or fail.
    Requires: -DENABLE_DEBUG_CAPS=ON
    Example: OV_MATCHER_LOGGING=true OV_MATCHERS_TO_LOG=EliminateSplitConcat ./your_program

  • Transformation profiling
    When to use: slow model compilation or need to identify which transformation passes take the most time.
    Example: OV_ENABLE_PROFILE_PASS=true

  • Model visualization
    When to use: need to see the model graph structure after specific passes — generates .svg files.
    Example: OV_ENABLE_VISUALIZE_TRACING=true or OV_ENABLE_VISUALIZE_TRACING="Pass1,Pass2"

  • Model serialization
    When to use: need to inspect model IR (.xml/.bin) after specific passes — useful for diffing model state before and after a transformation.
    Example: OV_ENABLE_SERIALIZE_TRACING=true or OV_ENABLE_SERIALIZE_TRACING="Pass1,Pass2"

  • Subgraph extraction
    When to use: working with a large model where full compilation or inference is slow, or where the serialized IR is too large to read or render in graph visualization tools — extract only the relevant subgraph as a standalone ov::Model, serialize to IR, and continue investigation on the smaller model independently.
    Example: extract_subgraph(model, {{"MatMul_0", 0}}, {{"Softmax_0", 0}})

See also

  • debug-matcher-pass skill — automated diagnosis workflow for MatcherPass transformations that are not firing. Collects matcher logs, identifies root cause, and generates a reproducer test.

Src Plugins Intel Cpu Docs Debug Capabilities README (`src/plugins/intel_cpu/docs/debug_capabilities/README.md`)

Debug capabilities

Debug capabilities are the set of useful debug features, controlled by environment variables.

They can be activated at runtime and might be used for analyzing issues, getting more context, comparing execution results, etc.

Use the following cmake option to enable debug capabilities:

-DENABLE_DEBUG_CAPS=ON

  • Verbose mode
    When to use: need to understand execution flow — traces node execution with types, shapes, algorithms, and kernels.
    Example: OV_CPU_VERBOSE=1

  • Blob dumping
    When to use: wrong results / accuracy issues — dump layer I/O tensors and compare against a reference.
    Warning: raw blobs can have very big sizes
    Example: OV_CPU_BLOB_DUMP_NODE_NAME="*" OV_CPU_BLOB_DUMP_DIR=blob_dump

  • Execution graph serialization
    When to use: need to have full execution flow picture, i.e. complete graph structure, how nodes are related, who are the producers / consumers of a node.
    Example: OV_CPU_EXEC_GRAPH_PATH=graph.xml

  • IR serialization
    When to use: unexpected graph structure — serialize IR at each transformation stage to see what changed.
    Example: OV_CPU_DUMP_IR="transformations dir=path/dumpdir formats=svg,xml,dot"

  • Graph transformation disabling
    When to use: crash or wrong results suspected in a transformation — disable groups to isolate the fault.
    Example: OV_CPU_DISABLE="transformations=common,preLpt,lpt,postLpt,snippets,specific"

  • Logging
    When to use: need to debug a specific code path — enables [ DEBUG ] logs with source location, supports breakpoint traps. Logs can be filtered and used as breakpoints with OV_CPU_DEBUG_LOG_BRK.
    Example: OV_CPU_DEBUG_LOG=-

  • Inference Precision
    When to use: precision-related accuracy issues — override inference precision for specific nodes.
    Example: OV_CPU_INFER_PRC_POS_PATTERN="^FullyConnected@"

  • Average counters
    When to use:

    • performance analysis across multiple inferences — collects averaged execution counters.
    • list of executed nodes, their types and primitive types
      Example: OV_CPU_AVERAGE_COUNTERS=filename

  • Performance summary
    When to use: slow inference — displays per-node timing summary when the model is destructed.
    Example: OV_CPU_SUMMARY_PERF=1

  • Max ISA cap
    When to use: reproduce lower-ISA behavior on higher-ISA hardware — caps runtime ISA dispatch. OV_CPU_MAX_ISA covers OV kernels, ONEDNN_MAX_CPU_ISA covers oneDNN primitives; independent, set both to cap everything.
    Example: OV_CPU_MAX_ISA=AVX2 ONEDNN_MAX_CPU_ISA=AVX2

  • Memory statistics
    When to use:

    • high memory usage or just memory profiling — dumps memory usage statistics per compiled model.
      Example: OV_CPU_MEMORY_STATISTICS_PATH=<file_path>.csv

CMakeLists (`docs/CMakeLists.txt`)

SPDX-License-Identifier: Apache-2.0

add_subdirectory(snippets)

set(ENABLE_CPP_API OFF CACHE BOOL "Build with C/C++ API.")
set(ENABLE_PYTHON_API OFF CACHE BOOL "Build with Python API.")
set(ENABLE_GENAI_API OFF CACHE BOOL "Build with GenAI API.")
set(ENABLE_NOTEBOOKS OFF CACHE BOOL "Build with openvino notebooks.")
set(ENABLE_OVMS OFF CACHE BOOL "Build with ovms.")
set(OVMS_DOCS_DIR "" CACHE PATH "Path to model server documentation dir.")

if(${ENABLE_OVMS} AND (OVMS_DOCS_DIR STREQUAL ""))
message( FATAL_ERROR "You want to build OVMS, but OVMS_DOCS_DIR variable is empty." )
endif()

function(build_docs)
find_package(Doxygen REQUIRED dot)
find_package(LATEX REQUIRED)

text
set(DOCS_BUILD_DIR "${CMAKE_CURRENT_BINARY_DIR}")
set(DOCS_SOURCE_DIR "${OpenVINO_SOURCE_DIR}/docs")
set(ARTICLES_EN_DIR "${OpenVINO_SOURCE_DIR}/docs/articles_en")
set(SCRIPTS_DIR "${DOCS_SOURCE_DIR}/scripts")

# Preprocessing scripts
set(REMOVE_XML_SCRIPT "${SCRIPTS_DIR}/remove_xml.py")
set(FILE_HELPER_SCRIPT "${SCRIPTS_DIR}/filehelper.py")
set(ARTICLES_HELPER_SCRIPT "${SCRIPTS_DIR}/articles_helper.py")
set(COPY_IMAGES_SCRIPT "${SCRIPTS_DIR}/copy_images.py")
set(DOXYGEN_MAPPING_SCRIPT "${SCRIPTS_DIR}/create_mapping.py")
set(DOCS_MAPPING_SCRIPT "${SCRIPTS_DIR}/create_doc_mapping.py")
set(BREATHE_APIDOC_SCRIPT "${SCRIPTS_DIR}/apidoc.py")
set(OV_INSTALLATION_SCRIPT "${SCRIPTS_DIR}/install_appropriate_openvino_version.py")

# Doxygen/Sphinx setup
set(DOXYGEN_XML_OUTPUT "${DOCS_BUILD_DIR}/xml")
set(SPHINX_SETUP_DIR "${DOCS_SOURCE_DIR}/sphinx_setup")
set(SPHINX_SOURCE_DIR "${DOCS_BUILD_DIR}/sphinx_source")
set(SPHINX_OUTPUT "${DOCS_BUILD_DIR}/_build")

list(APPEND commands COMMAND ${CMAKE_COMMAND} -E cmake_echo_color --green "STARTED preprocessing OpenVINO articles")
list(APPEND commands COMMAND ${Python3_EXECUTABLE} ${ARTICLES_HELPER_SCRIPT}
    --filetype=rst
    --input_dir=${ARTICLES_EN_DIR}
    --output_dir=${SPHINX_SOURCE_DIR}
    --exclude_dir=${SPHINX_SOURCE_DIR})
list(APPEND commands COMMAND ${CMAKE_COMMAND} -E cmake_echo_color --green "FINISHED preprocessing OpenVINO articles")

if(${ENABLE_CPP_API})
    # Doxygen config
    set(DOXYFILE_SOURCE "${DOCS_SOURCE_DIR}/Doxyfile.config")
    set(DOXYFILE_BUILD "${DOCS_BUILD_DIR}/Doxyfile.config")
    configure_file(${DOXYFILE_SOURCE} ${DOXYFILE_BUILD} @ONLY)
    list(APPEND commands COMMAND ${CMAKE_COMMAND} -E cmake_echo_color --green "STARTED preprocessing OpenVINO C/C++ API reference")
    list(APPEND commands COMMAND ${Python3_EXECUTABLE} ${REMOVE_XML_SCRIPT} ${DOXYGEN_XML_OUTPUT})
    list(APPEND commands COMMAND ${DOXYGEN_EXECUTABLE} ${DOXYFILE_BUILD})
    list(APPEND post_commands COMMAND ${Python3_EXECUTABLE} ${BREATHE_APIDOC_SCRIPT} ${DOXYGEN_XML_OUTPUT} -o "${SPHINX_SOURCE_DIR}/api/c_cpp_api" -m -T -p openvino -g class,group,struct,union,enum)
    list(APPEND commands COMMAND ${CMAKE_COMMAND} -E cmake_echo_color --green "FINISHED preprocessing OpenVINO C/C++ API reference")
endif()

if(${ENABLE_PYTHON_API} OR ${ENABLE_GENAI_API})
    list(APPEND commands COMMAND ${CMAKE_COMMAND} -E cmake_echo_color --green "STARTED preprocessing OpenVINO Python API")
    list(APPEND commands COMMAND ${Python3_EXECUTABLE} ${OV_INSTALLATION_SCRIPT}
    --ov_dir=${SPHINX_SETUP_DIR}
    --python=${Python3_EXECUTABLE}
    --enable_genai=${ENABLE_GENAI_API})
    list(APPEND commands COMMAND ${CMAKE_COMMAND} -E cmake_echo_color --green "FINISHED preprocessing OpenVINO Python API")
endif()

if(${ENABLE_OVMS})
    list(APPEND commands COMMAND ${CMAKE_COMMAND} -E cmake_echo_color --green "STARTED preprocessing OVMS")
    list(APPEND commands COMMAND ${Python3_EXECUTABLE} ${FILE_HELPER_SCRIPT}
    --filetype=md
    --input_dir=${OVMS_DOCS_DIR}
    --output_dir=${SPHINX_SOURCE_DIR}/model-server
    --exclude_dir=${SPHINX_SOURCE_DIR})
    list(APPEND commands COMMAND ${CMAKE_COMMAND} -E cmake_echo_color --green "FINISHED preprocessing OVMS")
endif()


# Preprocess docs
add_custom_target(preprocess_docs
                  COMMAND ${CMAKE_COMMAND} -E remove_directory ${SPHINX_SOURCE_DIR}
                  ${commands}
                  WORKING_DIRECTORY ${DOCS_BUILD_DIR}
                  VERBATIM)

add_custom_command(TARGET preprocess_docs
                   POST_BUILD
                   COMMAND ${Python3_EXECUTABLE} ${COPY_IMAGES_SCRIPT} ${DOXYGEN_XML_OUTPUT} ${SPHINX_SOURCE_DIR}
                   COMMAND ${Python3_EXECUTABLE} ${DOXYGEN_MAPPING_SCRIPT} ${DOXYGEN_XML_OUTPUT} ${DOCS_BUILD_DIR} ${OpenVINO_SOURCE_DIR}/../
                   COMMAND ${Python3_EXECUTABLE} ${DOCS_MAPPING_SCRIPT} ${DOCS_BUILD_DIR} ${ARTICLES_EN_DIR}
                   COMMAND ${CMAKE_COMMAND} -E copy_directory ${SPHINX_SETUP_DIR} ${SPHINX_SOURCE_DIR}
                   ${post_commands}
                   VERBATIM)

# Build docs
add_custom_target(sphinx_docs
                  DEPENDS preprocess_docs
                  COMMAND ${CMAKE_COMMAND} -E cmake_echo_color --green "STARTED sphinx documentation build"
                  COMMAND sphinx-build -j auto -w ${DOCS_BUILD_DIR}/sphinx.log -b html ${SPHINX_SOURCE_DIR} ${SPHINX_OUTPUT}
                  COMMAND ${CMAKE_COMMAND} -E cmake_echo_color --green "FINISHED sphinx documentation build"
                  WORKING_DIRECTORY ${SPHINX_SOURCE_DIR}
                  VERBATIM)

set_target_properties(preprocess_docs sphinx_docs PROPERTIES FOLDER docs)

endfunction()

if(ENABLE_DOCS)
build_docs()
endif()

RELEASE (`docs/RELEASE.MD`)

OpenVINO Release Management

The process described below reflects the approach to managing OpenVINO releases.

Release Milestones

  • Planning
  • Execution (development of new features)
  • Stabilization (Feature Freeze, Code Freeze milestones)
  • Validation
  • Distribution

Planning

This phase takes 2-4 weeks and involves scoping the backlog, prioritizing it, analyzing, and making commitments by developers for timelines specified by the release manager.

Execution (development of new features)

Stabilization (Feature Freeze, Code Freeze milestones)

  • Feature Freeze: This milestone ensures that no new features are added to the software after a certain point. This allows the development team to focus on stabilizing and refining the existing features, fixing bugs, and improving performance without the risk of introducing new issues.
  • Code Freeze: This milestone marks the point where no new code changes are allowed except for critical bug fixes. This helps in ensuring that the final product is stable and reliable, as it minimizes the risk of last-minute changes that could introduce new bugs or instability.

Release Validation

  • This is a continuous process executed on a regular basis with cadence based on testing type: nightly, bi-weekly, weekly.
  • After Code Freeze, the testing team can perform final regression testing to ensure that recent changes have not introduced new bugs and that the software meets the required quality standards.

Distribution

  • OpenVINO has different types of build distribution: Regular releases, Long-Term Support, Pre-release releases, Nightly builds. Read more here: OpenVINO Release Policy
  • Different distribution channels are supported. Explore different options here: OpenVINO Download

Documentation Build Instructions (`docs/documentation_build_instructions.md`)

Building documentation with CMake

NOTE: Instructions were tested inside container based on Ubuntu 22.04 docker image.

  1. Clone the OpenVINO repository and setup its submodules
text
$ git clone <openvino_repository_url> <repository_path>
$ cd <repository_path>
$ git submodule update --init --recursive
  1. Install build dependencies using the install_build_dependencies.sh script located in OpenVINO root directory
text
$ chmod +x install_build_dependencies.sh
$ ./install_build_dependencies.sh
  1. Install additional packages needed to build documentation
text
$ apt install -y doxygen graphviz texlive
  1. Create python virtualenv and install needed libraries
text
$ python3 -m venv env
$ source env/bin/activate
(env) $ pip install --upgrade setuptools pip
(env) $ pip install -r docs/requirements.txt
  1. Install the sphinx theme
text
(env) $ python -m pip install docs/openvino_sphinx_theme
  1. Install the custom sphinx sitemap
text
(env) $ pip install docs/openvino_custom_sphinx_sitemap
  1. Create build folder:
text
(env) $ mkdir build && cd build
  1. Build documentation using these commands:
text
(env) $ cmake .. -DENABLE_DOCS=ON
(env) $ cmake --build . --target sphinx_docs

Depending on the needs, following variables can be added to first cmake call:

  • building C/C++ API: -DENABLE_CPP_API=ON
  • building Python API: -DENABLE_PYTHON_API=ON
  • building GenAI API: -DENABLE_GENAI_API=ON
  • building Notebooks: -DENABLE_NOTEBOOKS=ON
  • building OVMS: -DENABLE_OVMS=ON -DOVMS_DOCS_DIR=

Note: When building the Python API documentation, the OpenVINO package version specified in conf.py is automatically downloaded using the installation script.
If you need a specific version or want to use an already installed OpenVINO package, modify these files accordingly.

Requirements (`docs/requirements.txt`)

alabaster==1.0.0
atomicwrites==1.4.1
attrs==26.1.0
Babel==2.18.0
beautifulsoup4==4.15.0
breathe==4.36.0
certifi==2026.7.22
cmake>=3.26
colorama==0.4.6
Cython==3.3.0
docutils==0.21.2
idna==3.20
imagesize==2.0.1
importlib-metadata>=4.8.0,<10.0.0
iniconfig==2.1.0
ipython==9.9.0
Jinja2==3.1.6
jsonschema==4.26.0
lxml>=4.9.2
MarkupSafe==3.0.3
mistune==3.3.4
myst-parser==4.0.1
packaging==26.3
pluggy==1.6.0
pydata-sphinx-theme==0.14.4
Pygments==2.21.0
pyparsing==3.3.2
pytest==9.1.1
pytest-html==4.2.0
pytest-metadata==3.1.1
py>=1.11.0
pytz==2026.3.post1
pyyaml==6.0.3
requests==2.34.2
six==1.17.0
snowballstemmer==3.1.1
soupsieve==2.9.2
sphinx==8.1.3
sphinx-copybutton==0.5.2
sphinx-design==0.6.1
sphinx-inline-tabs==2025.12.21.14
sphinx-sitemap==2.6.0
sphinxcontrib-applehelp==2.0.0
sphinxcontrib-devhelp==2.0.0
sphinxcontrib-htmlhelp==2.1.0
sphinxcontrib-jsmath==1.0.1
sphinxcontrib-qthelp==2.0.0
sphinxcontrib-serializinghtml==2.0.0
toml==0.10.2
urllib3==2.8.0
zipp==4.1.0

Articles En About Openvino (`docs/articles_en/about-openvino.rst`)

About OpenVINO

.. toctree::
:maxdepth: 1
:hidden:

about-openvino/key-features
about-openvino/performance-benchmarks
OpenVINO Ecosystem <about-openvino/openvino-ecosystem>
about-openvino/contributing
Release Notes <about-openvino/release-notes-openvino>

OpenVINO is a toolkit for simple and efficient deployment of various deep learning models.
In this section you will find information on the product itself, as well as the software
and hardware solutions it supports.

OpenVINO (Open Visual Inference and Neural network Optimization) is an open-source software
toolkit designed to optimize, accelerate, and deploy deep learning models for user applications.
OpenVINO is actively developed by Intel® to work efficiently on a wide range of Intel® hardware platforms,
including CPUs (x86 and Arm), GPUs, and NPUs.

Features
##############################################################

One of the main purposes of the OpenVINO toolkit is to streamline integration and deployment of
deep learning models. Yet its feature set is much wider, offering various optimization and implementation options.

To learn about the main properties of OpenVINO, see the :doc:Key Features <about-openvino/key-features>.

Architecture
##############################################################

To learn more about how OpenVINO works, read the Developer documentation on its
architecture <https://github.com/openvinotoolkit/openvino/blob/master/src/docs/architecture.md>__
and
core components <https://github.com/openvinotoolkit/openvino/blob/master/src/README.md>__.

Community
##############################################################

OpenVINO community plays a vital role in the growth and development of the open-sourced toolkit.
Users can contribute to OpenVINO and get support using the following channels:

  • OpenVINO GitHub issues, discussions and pull requests <https://github.com/openvinotoolkit/openvino>__
  • OpenVINO Blog <https://blog.openvino.ai/>__
  • Community Forum <https://community.intel.com/t5/Intel-Distribution-of-OpenVINO/bd-p/distribution-openvino-toolkit>__
  • OpenVINO video <https://www.youtube.com/watch?v=_Jnjt21ZDS8&list=PLg-UKERBljNxdIQir1wrirZJ50yTp4eHv>__
  • Support Information <https://www.intel.com/content/www/us/en/support/products/96066/software/development-software/openvino-toolkit.html>__

Case Studies
##############################################################

OpenVINO has been employed in various case studies across a wide range of industries and
applications, including healthcare, retail, safety and security, transportation, and more.
Read about how OpenVINO enhances efficiency, accuracy, and safety in different sectors on the
success stories page <https://www.intel.com/content/www/us/en/internet-of-things/ai-in-production/success-stories.html>__.

Articles En Documentation (`docs/articles_en/documentation.rst`)

Documentation

.. meta::
:description: Explore the OpenVINO toolkit workflow that entails preparing,
optimizing and deploying models in deep learning applications.

.. toctree::
:maxdepth: 1
:hidden:

API Reference <api/api_reference>
OpenVINO IR format and Operation Sets <documentation/openvino-ir-format>
Compatibility and Support <documentation/compatibility-and-support>
Legacy Features <documentation/legacy-features>
OpenVINO Extensibility <documentation/openvino-extensibility>
OpenVINO Security <documentation/openvino-security>

This section provides reference documents for the OpenVINO toolkit, such as API and Operation
listing.

| :doc:API Reference doc path <api/api_reference>
| A collection of reference articles for OpenVINO C++, C, Node.js, and Python APIs, as well as
the Python API for OpenVINO GenAI.

| :doc:OpenVINO IR format <documentation/openvino-ir-format>
| A section describing the OpenVINO IR model format and its opsets.

| :doc:Legacy Features <documentation/legacy-features>
| The information on all OpenVINO components that have recently been deprecated or discontinued.

| :doc:OpenVINO Extensibility Mechanism <documentation/openvino-extensibility>
| The Intel® Distribution of OpenVINO™ toolkit supports neural network models trained with
various frameworks, including TensorFlow, PyTorch, ONNX, TensorFlow Lite, and PaddlePaddle.
Learn how to extend OpenVINO functionality with custom settings.

| :doc:OpenVINO™ Security <documentation/openvino-security>
| Learn how to use OpenVINO securely and protect your data to meet specific security and privacy
requirements.

Articles En Get Started (`docs/articles_en/get-started.rst`)

GET STARTED

.. meta::
:description: Learn how to install Intel® Distribution of OpenVINO™ toolkit
on Windows, macOS, and Linux operating systems, using various
installation methods.

.. toctree::
:maxdepth: 1
:hidden:

Install OpenVINO <get-started/install-openvino>
Learn OpenVINO <get-started/learn-openvino>
System Requirements <./about-openvino/release-notes-openvino/system-requirements>

.. raw:: html

Welcome to OpenVINO! This guide introduces installation and learning materials for Intel® Distribution of OpenVINO™ toolkit. The guide walks through the following steps:
Quick Start Example
Install OpenVINO
Learn OpenVINO

For a quick reference, check out
the Quick Start Guide [pdf] <https://docs.openvino.ai/2026/_static/download/OpenVINO_Quick_Start_Guide.pdf>__

.. _quick-start-example:

  1. Quick Start Example (No Installation Required)

#################################################

.. image:: https://user-images.githubusercontent.com/15709723/127752390-f6aa371f-31b5-4846-84b9-18dd4f662406.gif
:width: 400

Try out OpenVINO's capabilities with this quick start example <https://github.com/openvinotoolkit/openvino_notebooks/tree/latest/notebooks/vision-monodepth>__
that estimates depth in a scene using an OpenVINO monodepth model to quickly see how to load a model, prepare an image, inference the image, and display the result.

.. _install-openvino-gsg:

  1. Install OpenVINO

###################

See the :doc:installation overview page <get-started/install-openvino> for options to install OpenVINO and set up a development environment on your device.

.. _get-started-learn-openvino/interactive-tutorials-python:

  1. Learn OpenVINO

#################

OpenVINO provides a wide array of examples and documentation showing how to work with models, run inference, and deploy applications. Step through the sections below to learn the basics of OpenVINO and explore its advanced optimization features. For further details, visit :doc:OpenVINO documentation <documentation>.

.. _openvino-basics:

OpenVINO Basics
+++++++++++++++

Learn the basics of working with models and inference in OpenVINO. Begin with “Hello World” Interactive Tutorials that show how to prepare models, run inference, and retrieve results using the OpenVINO API. Then, explore OpenVINO Code Samples that can be adapted for your own application.

.. _interactive-learn-openvino/interactive-tutorials-python:

Interactive Tutorials - Jupyter Notebooks

Start with :doc:interactive Python <get-started/learn-openvino/interactive-tutorials-python> that show the basics of model inference, the OpenVINO API, how to convert models to OpenVINO format, and more.

  • Hello Image Classification <https://github.com/openvinotoolkit/openvino_notebooks/tree/latest/notebooks/hello-world>__
    • Load an image classification model in OpenVINO and use it to apply a label to an image
  • OpenVINO Runtime API Tutorial <https://github.com/openvinotoolkit/openvino_notebooks/tree/latest/notebooks/openvino-api>__
    • Learn the basic Python API for working with models in OpenVINO
  • Convert TensorFlow Models to OpenVINO <https://github.com/openvinotoolkit/openvino_notebooks/tree/latest/notebooks/tensorflow-classification-to-openvino>__
  • Convert PyTorch Models to OpenVINO <https://github.com/openvinotoolkit/openvino_notebooks/blob/latest/notebooks/pytorch-to-openvino/pytorch-onnx-to-openvino.ipynb>__

.. _code-samples:

OpenVINO Code Samples

View :doc:sample code <get-started/learn-openvino/openvino-samples> for various C++ and Python applications that can be used as a starting point for your own application. For C++ developers, step through the :doc:Get Started with C++ Samples <get-started/learn-openvino/openvino-samples/get-started-demos> to learn how to build and run an image classification program that uses OpenVINO’s C++ API.

.. _integrate-openvino:

Integrate OpenVINO With Your Application

Learn how to :doc:use the OpenVINO API to implement an inference pipeline <openvino-workflow/running-inference> in your application.

.. _openvino-advanced-features:

OpenVINO Advanced Features
++++++++++++++++++++++++++

OpenVINO provides features to improve your model’s performance, optimize your runtime, maximize your application’s throughput on target hardware, and much more. Visit the links below to learn more about these features and how to use them.

Model Compression and Quantization

Use OpenVINO’s model compression tools to reduce your model’s latency and memory footprint while maintaining good accuracy.

  • Tutorial - Quantization-Aware Training in PyTorch with NNCF <https://github.com/openvinotoolkit/openvino_notebooks/tree/latest/notebooks/pytorch-quantization-aware-training>__
  • :doc:Model Optimization Guide <openvino-workflow/model-optimization>

Automated Device Configuration

OpenVINO’s hardware device configuration options enable you to write an application once and deploy it anywhere with optimal performance.

  • Increase application portability and perform parallel inference across processors with :doc:Automatic Device Selection (AUTO) <openvino-workflow/running-inference/inference-devices-and-modes/auto-device-selection>
  • Efficiently split inference between hardware cores with :doc:Heterogeneous Execution (HETERO) <openvino-workflow/running-inference/inference-devices-and-modes/hetero-execution>

Flexible Model and Pipeline Configuration

Pipeline and model configuration features in OpenVINO Runtime allow you to easily optimize your application’s performance on any target hardware.

  • :doc:Automatic Batching <openvino-workflow/running-inference/inference-devices-and-modes/automatic-batching> performs on-the-fly grouping of inference requests to maximize utilization of the target hardware’s memory and processing cores.
  • :doc:Performance Hints <openvino-workflow/running-inference/optimize-inference/high-level-performance-hints> automatically adjust runtime parameters to prioritize for low latency or high throughput
  • :doc:Dynamic Shapes <openvino-workflow/running-inference/model-input-output/dynamic-shapes> reshapes models to accept arbitrarily-sized inputs, increasing flexibility for applications that encounter different data shapes
  • :doc:Benchmark Tool <get-started/learn-openvino/openvino-samples/benchmark-tool> characterizes model performance in various hardware and pipeline configurations

.. _additional-about-openvino/additional-resources:

Additional Resources

  • OpenVINO Success Stories <https://www.intel.com/content/www/us/en/internet-of-things/ai-in-production/success-stories.html>__ - See how Intel partners have successfully used OpenVINO in production applications to solve real-world problems.
  • :doc:Performance Benchmarks <about-openvino/performance-benchmarks> - View results from benchmarking models with OpenVINO on Intel hardware.

Articles En Openvino Workflow Generative (`docs/articles_en/openvino-workflow-generative.rst`)

Generative AI workflow

.. meta::
:description: learn how to use OpenVINO to run generative AI models.

.. toctree::
:maxdepth: 1
:hidden:

Generative Model Preparation <openvino-workflow-generative/genai-model-preparation>
Inference with OpenVINO GenAI <openvino-workflow-generative/inference-with-genai>
Inference with Optimum Intel <openvino-workflow-generative/inference-with-optimum-intel>
OpenVINO Tokenizers <openvino-workflow-generative/ov-tokenizers>

Generative AI is a specific area of Deep Learning models used for producing new and “original”
data, based on input in the form of image, sound, or natural language text. Due to their
complexity and size, generative AI pipelines are more difficult to deploy and run efficiently.
OpenVINO™ simplifies the process and ensures high-performance integrations, with the following
options:

.. tab-set::

.. tab-item:: OpenVINO™ GenAI

text
| - Suggested for production deployment for the supported use cases.
  | - Smaller footprint and fewer dependencies.
  | - More optimization and customization options.
  | - Available in both Python and C++.
  | - A limited set of supported use cases.

  :doc:`Install the OpenVINO GenAI package <../get-started/install-openvino/install-openvino-genai>`
  and run generative models out of the box. With custom
  API and tokenizers, among other components, it manages the essential tasks such as the
  text generation loop, tokenization, and scheduling, offering ease of use and high
  performance.

  `Check out the OpenVINO GenAI Quick-start Guide [PDF] <https://docs.openvino.ai/2026/_static/download/GenAI_Quick_Start_Guide.pdf>`__

.. tab-item:: Optimum Intel (Hugging Face integration)

text
| - Suggested for prototyping and, if the use case is not covered by OpenVINO GenAI, production.
  | - Bigger footprint and more dependencies.
  | - Limited customization due to Hugging Face dependency.
  | - Not usable for C++ applications.
  | - A very wide range of supported models.

  Using Optimum Intel is a great way to experiment with different models and scenarios,
  thanks to a simple interface for the popular API and infrastructure offered by Hugging Face.
  It also enables weight compression with
  `Neural Network Compression Framework (NNCF) <https://github.com/openvinotoolkit/nncf>`__,
  as well as conversion on the fly. For integration with the final product it may offer
  lower performance, though.

.. tab-item:: OpenVINO™ Model Server

text
| - Easy and quick deployment of models to edge or cloud.
  | - Includes endpoints for serving generative AI models.
  | - Available in both Python and C++.
  | - Allows client applications in any programming language that supports REST or gRPC.

  :doc:`OpenVINO™ Model Server <model-server/ovms_what_is_openvino_model_server>`
  provides a set of REST API endpoints dedicated to generative use cases. The endpoints
  simplify writing AI applications, ensure scalability, and provide state-of-the-art
  performance optimizations. They include OpenAI API for:
  `text generation <https://openvino-doc.iotg.sclab.intel.com/seba-test-8/model-server/ovms_docs_rest_api_chat.html>`__,
  `embeddings <https://openvino-doc.iotg.sclab.intel.com/seba-test-8/model-server/ovms_docs_rest_api_embeddings.html>`__,
  and `reranking <https://openvino-doc.iotg.sclab.intel.com/seba-test-8/model-server/ovms_docs_rest_api_rerank.html>`__.
  The model server supports deployments as containers or binary applications on Linux and Windows with CPU or GPU acceleration.
  See the :doc:`demos <model-server/ovms_docs_demos>`.

The advantages of using OpenVINO for generative model deployment:

| Fewer dependencies and smaller footprint
| More compact than frameworks such as Hugging Face and PyTorch, with a smaller binary size
and reduced memory footprint, makes deployments easier and updates more manageable.

| Compression and precision management
| Techniques such as 8-bit and 4-bit weight compression, including embedding layers, and storage
format reduction. This includes fp16 precision for non-compressed models and int8/int4 for
compressed models, like GPTQ models from Hugging Face <https://huggingface.co/models>__.

| Enhanced inference capabilities
| Advanced features like in-place KV-cache, dynamic quantization, KV-cache quantization and
encapsulation, dynamic beam size configuration, and speculative sampling, and more are
available.

| Stateful model optimization
| Models from the Hugging Face Transformers are converted into a stateful form, optimizing
inference performance and memory usage in long-running text generation tasks by managing past
KV-cache tensors more efficiently internally. This feature is automatically activated for
many supported models, while unsupported ones remain stateless. Learn more about the
:doc:Stateful models and State API <../openvino-workflow/running-inference/inference-request/stateful-models>.

| Optimized LLM inference
| Includes a Python API for rapid development and C++ for further optimization, offering
better performance than Python-based runtimes.

You can run Generative AI models, using native OpenVINO API, although it is not recommended.
If you want to learn how to do it, refer to
the 24.6 documentation <https://docs.openvino.ai/2024/learn-openvino/llm_inference_guide/llm-inference-native-ov.html>__.

Proceed to guides on:

  • :doc:OpenVINO GenAI <./openvino-workflow-generative/inference-with-genai>
  • :doc:Hugging Face and Optimum Intel <./openvino-workflow-generative/inference-with-optimum-intel>
  • Generative AI with OpenVINO Inference Engine <https://docs.openvino.ai/2024/learn-openvino/llm_inference_guide/llm-inference-native-ov.html>__

Articles En Openvino Workflow (`docs/articles_en/openvino-workflow.rst`)

Conventional AI Workflow

.. meta::
:description: OpenVINO toolkit workflow usually involves preparation,
optimization, and compression of models, running inference and
deploying deep learning applications.

.. toctree::
:maxdepth: 1
:hidden:

Conventional Model Preparation <openvino-workflow/model-preparation>
openvino-workflow/model-optimization
Running Inference <openvino-workflow/running-inference>
Deployment on a Local System <openvino-workflow/deployment-locally>
openvino-workflow/torch-compile

OpenVINO offers multiple workflows, depending on the use case and personal or project preferences.
This section will give you a detailed view of how you can go from preparing your model,
through optimizing it, to executing inference, and deploying your solution.

Once you obtain a model in one of the :doc:supported model formats <openvino-workflow/model-preparation>,
you can decide how to proceed:

.. tab-set::

.. tab-item:: Workflow for convenience

text
This approach assumes you run your model directly.

  .. image:: ./assets/images/ov_workflow_diagram_convenience.svg
     :align: center
     :alt: OpenVINO workflow diagram for convenience

.. tab-item:: Workflow for performance (recommended for production)

text
This approach assumes you convert your model to OpenVINO IR explicitly, which means the
  conversion stage is not part of the final application.

  .. image:: ./assets/images/ov_workflow_diagram_performance.svg
     :align: center
     :alt: OpenVINO workflow diagram for performance

OpenVINO uses the following functions for reading, converting, and saving models:

.. tab-set::

.. tab-item:: read_model

text
* Creates an ov.Model from a file.
  * Supported file formats: OpenVINO IR, ONNX, PaddlePaddle, TensorFlow and TensorFlow Lite. PyTorch files are not directly supported.
  * OpenVINO files are read directly while other formats are converted automatically.

.. tab-item:: compile_model

text
* Creates an ov.CompiledModel from a file or ov.Model object.
  * Supported file formats: OpenVINO IR, ONNX, PaddlePaddle, TensorFlow and TensorFlow Lite. PyTorch files are not directly supported.
  * OpenVINO files are read directly while other formats are converted automatically.

.. tab-item:: convert_model

text
* Creates an ov.Model from a file or Python memory object.
  * Supported file formats: ONNX, PaddlePaddle, TensorFlow and TensorFlow Lite.
  * Supported framework objects: PaddlePaddle, TensorFlow and PyTorch.
  * This method is only available in the Python API.

.. tab-item:: save_model

text
* Saves an ov.Model to OpenVINO IR format.
  * Compresses weights to FP16 by default.
  * This method is only available in the Python API.

| :doc:Model Preparation <openvino-workflow/model-preparation>
| Learn how to convert pre-trained models to OpenVINO IR.

| :doc:Model Optimization and Compression <openvino-workflow/model-optimization>
| Find out how to optimize a model to achieve better inference performance, utilizing
multiple optimization methods for both in-training compression and post-training quantization.

| :doc:Running Inference <openvino-workflow/running-inference>
| See how to run inference with OpenVINO, which is the most basic form of deployment,
and the quickest way of running a deep learning model.

| :doc:Deployment Option 1. Using OpenVINO Runtime <openvino-workflow/deployment-locally>
| Deploy a model locally, reading the file directly from your application and utilizing resources available to the system.
| Deployment on a local system uses the steps described in the section on running inference.

| :doc:Deployment Option 2. Using Model Server <../model-server/ovms_what_is_openvino_model_server>
| Deploy a model remotely, connecting your application to an inference server and utilizing external resources, with no impact on the app's performance.
| Deployment on OpenVINO Model Server is quick and does not require any additional steps described in the section on running inference.

| :doc:Deployment Option 3. Using torch.compile for PyTorch 2.0 <openvino-workflow/torch-compile>
| Deploy a PyTorch model using OpenVINO in a PyTorch-native application.

Articles En Physical Ai (`docs/articles_en/physical-ai.md`)

explore_github_url: https://github.com/openvinotoolkit/physicalai

OpenVINO™ Physical AI

{toctree}
:hidden:
:maxdepth: 2

physical-ai/getting-started/README
physical-ai/how-to/README
physical-ai/explanation/README
physical-ai/reference/README

OpenVINO™ Physical AI provides runtime components for working with exported robot policies. The documentation is organized around the main deployment tasks and the runtime concepts behind them.

I want to Go to
Install the package Installation
Run first inference Quickstart
Run a policy on a robot Run a Policy
Write runtime YAML Runtime Config
Use the runtime CLI CLI Run
Understand architecture Architecture
Look up schemas Config Schema

Documentation Structure

text
docs/
├── getting-started/  # tutorials
├── how-to/           # task guides
├── explanation/      # concepts and boundaries
└── reference/        # exact commands, schemas, APIs

Workflow

Most deployment workflows follow the same path from an exported package to a running robot loop.

text
exported policy package
    -> InferenceModel
    -> PolicyRuntime
    -> Robot

Python example:

python
from physicalai.inference import InferenceModel
from physicalai.runtime import PolicyRuntime, SyncExecution
from physicalai.robot import SO101
from physicalai.capture import UVCCamera

model = InferenceModel("./exports/act_policy")
robot = SO101(port="/dev/ttyACM0")
cameras = {"wrist": UVCCamera(device="/dev/video0", width=640, height=480)}

runtime = PolicyRuntime(
    fps=30,
    robot=robot,
    model=model,
    cameras=cameras,
    execution=SyncExecution(),
)

with runtime:
    runtime.run(duration_s=60)

Preview: The CLI remains a planned API. See #121 for status.

CLI example:

bash
physicalai run --config runtime.yaml --run.duration_s=60

Dev Build (`docs/dev/build.md`)

How to build OpenVINO

mermaid
gantt 
    %% Use a hack for centry as a persantage
    dateFormat YYYY
    axisFormat %y
    todayMarker off
    title       OpenVINO getting started pipeline
    Setup environment :env, 2000, 1716w
    Build openvino :crit, build, after env, 1716w
    Run tests :active, run, after build, 1716w

The articles below provide the basic informations about the process of building OpenVINO.

NOTE: For the details on how to build static OpenVINO, refer to Building static OpenVINO libraries

See also

Dev Build Android (`docs/dev/build_android.md`)

Build OpenVINO™ Runtime for Android systems

This article describes how to build OpenVINO for Android operating systems.

Software requirements

Use the prebuilt x86-64 archive

For Android x86-64 targets, you can use the prebuilt OpenVINO 2026.3.1 Runtime package instead of building OpenVINO from source:

sh
mkdir openvino-android
export OPV_HOME_DIR=${PWD}/openvino-android
curl -L https://storage.openvinotoolkit.org/repositories/openvino/packages/2026.3.1/linux/openvino_toolkit_android_2026.3.1.22476.56d9685302d_x86_64.tgz \
  --output $OPV_HOME_DIR/openvino_toolkit_android_2026.3.1.tgz
tar -xf $OPV_HOME_DIR/openvino_toolkit_android_2026.3.1.tgz -C $OPV_HOME_DIR
export OpenVINO_DIR=$OPV_HOME_DIR/openvino_toolkit_android_2026.3.1.22476.56d9685302d_x86_64/runtime/cmake

The archive provides the OpenVINO headers, x86-64 Runtime libraries, CMake configuration, and required oneTBB libraries. For other Android architectures or a custom configuration, build OpenVINO from source as described below. The ADB example later in this guide demonstrates an aarch64 source build and does not apply directly to this x86-64 archive.

How to build

Create a work directory

sh
mkdir openvino-android
export OPV_HOME_DIR=${PWD}/openvino-android
export HOST_OS_TYPE=$(uname -s | tr '[:upper:]' '[:lower:]') # linux/darwin

Download and unpack Android packages

  • Download and unpack Android NDK
    sh
    wget https://dl.google.com/android/repository/android-ndk-r28c-${HOST_OS_TYPE}.zip --directory-prefix $OPV_HOME_DIR

unzip $OPV_HOME_DIR/android-ndk-r28c-${HOST_OS_TYPE}.zip -d $OPV_HOME_DIR
mv $OPV_HOME_DIR/android-ndk-r28c $OPV_HOME_DIR/android-ndk
export ANDROID_NDK_PATH=$OPV_HOME_DIR/android-ndk
  • Download and unpack Android SDK Platform Tools
    sh
    wget https://dl.google.com/android/repository/platform-tools-latest-${HOST_OS_TYPE}.zip --directory-prefix $OPV_HOME_DIR

unzip $OPV_HOME_DIR/platform-tools-latest-${HOST_OS_TYPE}.zip -d $OPV_HOME_DIR
export ANDROID_TOOLS_PATH=$OPV_HOME_DIR/platform-tools

For Windows and Mac operating systems, the downloading and unpacking steps are similar.

Set the environment variables for building

sh
# If you have no android devices please set CURRENT_ANDROID_ABI according to your preferences e.g. export CURRENT_ANDROID_ABI=arm64-v8a
export CURRENT_ANDROID_ABI=`$ANDROID_TOOLS_PATH/adb shell getprop ro.product.cpu.abi`
export CURRENT_ANDROID_PLATFORM=35
export CURRENT_ANDROID_STL=c++_shared
export CURRENT_CMAKE_TOOLCHAIN_FILE=$ANDROID_NDK_PATH/build/cmake/android.toolchain.cmake
  • ANDROID_ABI specifies the target architecture:
    • x86_64 for x64 build
    • armeabi-v7a with NEON for ARM with NEON support
    • arm64-v8a for ARM 64 bits
    • riscv64 for RISC-V 64 bits (experimental)
  • ANDROID_PLATFORM specifies the Android API version.
  • ANDROID_STL indicates that a shared C++ runtime is used.

Build and install OneTBB™ (Not for RISC-V 64 architecture)

To improve the parallelism performance of the OpenVINO™ library using OneTBB, it is required to separately build OneTBB for a specific version of the Android NDK:
Android OpenVINO build with THREADING=TBB requires this separate OneTBB build.
OpenVINO CMake configure must be called with -DTBB_DIR=$OPV_HOME_DIR/one-tbb-install/lib/cmake/TBB.
If TBB_DIR is not provided, Android configure fails by design.

sh
# Clone OneTBB™ repository 
git clone --recursive https://github.com/oneapi-src/oneTBB $OPV_HOME_DIR/one-tbb
# Create build and install directory 
mkdir $OPV_HOME_DIR/one-tbb-build $OPV_HOME_DIR/one-tbb-install
# Configure OneTBB™ CMake project 
cmake -S $OPV_HOME_DIR/one-tbb \
      -B $OPV_HOME_DIR/one-tbb-build \
      -DCMAKE_BUILD_TYPE=Release \
      -DCMAKE_INSTALL_PREFIX=$OPV_HOME_DIR/one-tbb-install \
      -DCMAKE_TOOLCHAIN_FILE=$CURRENT_CMAKE_TOOLCHAIN_FILE \
      -DANDROID_ABI=$CURRENT_ANDROID_ABI \
      -DANDROID_PLATFORM=$CURRENT_ANDROID_PLATFORM \
      -DANDROID_STL=$CURRENT_ANDROID_STL \
      -DTBB_TEST=OFF \
      -DCMAKE_SHARED_LINKER_FLAGS="-Wl,--undefined-version"
# Build OneTBB™ project 
cmake --build $OPV_HOME_DIR/one-tbb-build --parallel
# Install OneTBB™ project 
cmake --install $OPV_HOME_DIR/one-tbb-build

Android OpenVINO configuration expects the separately built OneTBB package.
TBB_DIR must be provided when configuring OpenVINO.

Clone OpenVINO™ GenAI (Optional)

sh
git clone --recursive https://github.com/openvinotoolkit/openvino.genai $OPV_HOME_DIR/openvino.genai

Build and install OpenVINO™

sh
# Clone OpenVINO™ repository 
git clone --recursive https://github.com/openvinotoolkit/openvino $OPV_HOME_DIR/openvino
# Create build and install directory 
mkdir $OPV_HOME_DIR/openvino-build $OPV_HOME_DIR/openvino-install
# Configure OpenVINO™ CMake project 
cmake -S $OPV_HOME_DIR/openvino \
      -B $OPV_HOME_DIR/openvino-build \
      -DCMAKE_BUILD_TYPE=Release \
      -DCMAKE_INSTALL_PREFIX=$OPV_HOME_DIR/openvino-install \
      -DCMAKE_TOOLCHAIN_FILE=$CURRENT_CMAKE_TOOLCHAIN_FILE \
      -DANDROID_ABI=$CURRENT_ANDROID_ABI \
      -DANDROID_PLATFORM=$CURRENT_ANDROID_PLATFORM \
      -DANDROID_STL=$CURRENT_ANDROID_STL \
      -DTBB_DIR=$OPV_HOME_DIR/one-tbb-install/lib/cmake/TBB \
      -DOPENVINO_EXTRA_MODULES=$OPV_HOME_DIR/openvino.genai
# Build OpenVINO™ project 
cmake --build $OPV_HOME_DIR/openvino-build --parallel
# Install OpenVINO™ project 
cmake --install $OPV_HOME_DIR/openvino-build

Download the example model

We will use mobelinet-v3-tf example model

sh
mkdir $OPV_HOME_DIR/mobelinet-v3-tf
wget https://storage.openvinotoolkit.org/repositories/openvino_notebooks/models/mobelinet-v3-tf/FP32/v3-small_224_1.0_float.xml -P $OPV_HOME_DIR/mobelinet-v3-tf/
wget https://storage.openvinotoolkit.org/repositories/openvino_notebooks/models/mobelinet-v3-tf/FP32/v3-small_224_1.0_float.bin -P $OPV_HOME_DIR/mobelinet-v3-tf/

Use the ADB tool to run the example model and check the library

This example is demonstrated for aarch64 architecture

sh
# Copy OpenVINO™ libraries to android device
$ANDROID_TOOLS_PATH/adb push --sync $OPV_HOME_DIR/openvino-install/runtime/lib/aarch64/* /data/local/tmp/
# Copy OneTBB libraries to android device
$ANDROID_TOOLS_PATH/adb push --sync $OPV_HOME_DIR/one-tbb-install/lib/* /data/local/tmp/
# Copy shared STL library to android device
$ANDROID_TOOLS_PATH/adb push --sync $ANDROID_NDK_PATH/toolchains/llvm/prebuilt/$HOST_OS_TYPE-x86_64/sysroot/usr/lib/aarch64-linux-android/libc++_shared.so /data/local/tmp/
# Copy example model files
$ANDROID_TOOLS_PATH/adb push --sync $OPV_HOME_DIR/mobelinet-v3-tf /data/local/tmp/
# Copy OpenVINO™ benchmark_app tool to android device
$ANDROID_TOOLS_PATH/adb push --sync $OPV_HOME_DIR/openvino/bin/aarch64/Release/benchmark_app /data/local/tmp/
# Run OpenVINO™ benchmark_app tool
$ANDROID_TOOLS_PATH/adb shell "LD_LIBRARY_PATH=/data/local/tmp ./data/local/tmp/benchmark_app -m /data/local/tmp/mobelinet-v3-tf/v3-small_224_1.0_float.xml -hint latency"

See also

Dev Build Linux (`docs/dev/build_linux.md`)

Build OpenVINO™ Runtime for Linux systems

Current validation environments include:

  • Ubuntu 22.04, 24.04, and 26.04 (64-bit) in GitHub Actions

The external infrastructure Docker repository also contains RHEL 8.10 and 9.6 build images, but
the current product CI configurations do not reference them. Other compatible distributions may
build OpenVINO, but are not covered by the current CI matrix.

Software requirements

  • CMake 3.26 or higher
  • GCC 7.5 or higher to build OpenVINO Runtime
  • Python 3.11 - 3.14 for OpenVINO Runtime Python API
  • (Optional) Install Intel® Graphics Compute Runtime for OpenCL™ Driver package to enable inference on Intel integrated GPUs.

How to build

  1. Clone OpenVINO repository and init submodules:

    sh
    git clone https://github.com/openvinotoolkit/openvino.git
cd openvino
git submodule update --init --recursive

    (Optional) For users in China, clone submodules via gitee mirrors

    sh
    chmod +x scripts/submodule_update_with_gitee.sh
./scripts/submodule_update_with_gitee.sh

  2. Install build dependencies using the install_build_dependencies.sh script in the
    project root folder.

    sh
    sudo ./install_build_dependencies.sh

  3. Create a build folder:

    sh
    mkdir build && cd build

NOTE: It is recommended to disable the oneAPI environment before compiling OpenVINO from source on Linux, as it may cause build failures.

  1. OpenVINO Runtime uses a CMake-based build system. In the created build directory, run cmake to fetch project dependencies and create Unix makefiles, then run make to build the project:
    sh
    cmake -DCMAKE_BUILD_TYPE=Release ..
  cmake --build . --parallel
    The process may take some time to finish. If you are using a system with limited resources, it is recommended to specify a lower number of parallel jobs to avoid overloading your system. This can help maintain system responsiveness and stability during the build process. Use nproc to find the number of available processing units. For example, to use 8 parallel jobs, run the following command:
    sh
    cmake --build . --parallel 8

Additional Build Options

You can use the following additional build options:

  • For IA32 operation systems, use ia32.linux.toolchain.cmake CMake toolchain file:

    sh
    cmake -DCMAKE_TOOLCHAIN_FILE=<openvino_repo>/cmake/toolchains/ia32.linux.toolchain.cmake ..

  • OpenVINO offers several CMake options that can be used to build a custom OpenVINO runtime, which can speed up compilation. These options allow you to skip compilation of other plugins and frontends by disabling/enabling them. You can find a list of available options here

  • To build the OpenVINO Runtime Python API:

    1. Enable the -DENABLE_PYTHON=ON option in the CMake step above (Step 4). To specify an exact Python version, use the following options (requires cmake 3.16 and higher):
      text
      -DPython3_EXECUTABLE=/usr/bin/python3.11
    2. To build a wheel package (.whl), enable the -DENABLE_WHEEL=ON option in the CMake step above (Step 4), and install requirements:
      sh
      pip install -r <openvino source tree>/src/bindings/python/wheel/requirements-dev.txt
    3. After the build process finishes, export the newly built Python libraries to the user environment variables:
      text
      export PYTHONPATH=<openvino_repo>/bin/intel64/Release/python:<openvino_repo>/tools/ovc:$PYTHONPATH
export LD_LIBRARY_PATH=<openvino_repo>/bin/intel64/Release:$LD_LIBRARY_PATH
export PATH=<openvino_repo>/tools/ovc/openvino/tools/ovc:$PATH
      or install the wheel with pip:
      text
      pip install <openvino_repo>/build/wheel/openvino-2022.2.0-000-cp37-cp37-manylinux_2_35_x86_64.whl

See also

Dev Build Mac Arm (`docs/dev/build_mac_arm.md`)

Build OpenVINO™ Runtime for macOS systems (Apple Silicon)

This guide shows how to build OpenVINO Runtime for later inference on Apple Silicon on macOS.

There are two options how to use OpenVINO on Apple Silicon:

  • (Native) Compile OpenVINO for arm64 architecture. Note, build steps will cover this as a default scenario.
  • (Rosetta) Compile Intel CPU plugin x86_64 architecture and run under Rosetta.

The software was:

  • validated on macOS 15 (native arm64)
  • produced for macOS 11.0 (deployment target)

Software requirements

  • brew package manager to install additional dependencies. Use install brew guide to achieve this.

  • Installation step for python and python libraries varies depending on the host architecture:

    • arm64 Python 3.11 - 3.14 for the OpenVINO Runtime Python API:
    sh
    % # let's have a look what python versions are available in brew
% brew search python
% # select preferred version of python based on available ones, e.g. 3.11
% brew install [email protected]
    • x86_64 Select universal2 installer from Python releases download page and install python-3.X.Y-macos11.pkg image. This allows you to have universal python libraries of OpenVINO Python API (build x86_64).

  • Clang compiler and other command line tools from Xcode 10.1 or higher:

    sh
    % xcode-select --install

  • CMake 3.26 or higher and other build dependencies:

    sh
    % brew install cmake scons fdupes git-lfs ninja

  • (arm64 only) Product and samples dependencies:

    sh
    % brew install tbb pugixml flatbuffers snappy protobuf

  • Additional pip dependencies to build OpenVINO Runtime Python API:

    sh
    % # update pip and setuptools to newer versions
% python3 -m pip install -U pip
% python3 -m pip install -r <openvino source tree>/src/bindings/python/requirements.txt

    Additional install requirements (after OpenVINO repo clone) in order to build OpenVINO Python API as wheel packages:

    sh
    % python3 -m pip install -r <openvino source tree>/src/bindings/python/wheel/requirements-dev.txt

How to build

  1. (Get sources) Clone the repository and submodules:
sh
git clone https://github.com/openvinotoolkit/openvino.git
cd openvino
git submodule update --init
  1. Create a build folder:
sh
mkdir build && cd build
  1. (CMake configure) OpenVINO project uses a CMake-based build system. In the created build directory, run cmake to fetch project dependencies and create build rules:
sh
cmake -G "Ninja Multi-Config" -DENABLE_SYSTEM_PUGIXML=ON -DENABLE_SYSTEM_SNAPPY=ON -DENABLE_SYSTEM_PROTOBUF=ON ..

NOTE: By default OpenVINO CMake scripts try to introspect the system and enable all possible functionality based on that. You can look at the CMake output and see warnings, which show that some functionality is turned off and the corresponding reason, guiding what to do to install additionally to enable unavailable functionality. Additionally, you can change CMake options to enable / disable some functionality, add / remove compilation flags, provide custom version of dependencies like TBB, PugiXML, Protobuf. For more information, see CMake Options for Custom Compilation.

  1. (CMake build) Build OpenVINO project:
sh
cmake --build . --config Release --parallel $(sysctl -n hw.ncpu)

All built binaries are located in <openvino_source_dir>/bin/<arm64 | intel64>/Release/ and wheel packages are located in <openvino_build_dir>/wheels.

  1. (Optional install) Once you have built OpenVINO, you can install artifacts to a preferred location:
sh
cmake -DCMAKE_INSTALL_PREFIX=<installation location> -P cmake_install.cmake

Building x86_64 binaries

Since OSX version 11.x and Xcode version 12.2, the Apple development tools allow to compile arm64 code on x86 hosts and vice-versa. Based on this, OpenVINO can be compiled as x86_64 binary, then run on Apple Silicon hosts using Rosetta. To do this, you must first install Rosetta:

sh
softwareupdate --install-rosetta

Then try to compile OpenVINO using the steps above, but adding -DCMAKE_OSX_ARCHITECTURES=x86_64 on cmake configure stage. But, don't enable any system library usage explicitly via CMake options (remove all -DENABLE_SYSTEM_* options), because they have arm64 architecture, e.g.:

sh
file /opt/homebrew/Cellar/tbb/2021.5.0_2/lib/libtbb.12.5.dylib
/opt/homebrew/Cellar/tbb/2021.5.0_2/lib/libtbb.12.5.dylib: Mach-O 64-bit dynamically linked shared library arm64

The same goes for other external dependencies like pugixml.

NOTE: When building with universal2 python libraries, wheel package is created with the openvino-2022.3.0-000-cp311-cp311-macosx_12_0_universal2.whl name and have a proper universal2 tags, so they can potentially be used on both Apple Silicon and Intel CPU.

See also

Dev Build Mac Intel Cpu (`docs/dev/build_mac_intel_cpu.md`)

Build OpenVINO™ Runtime for macOS systems (Intel CPU)

This guide shows how to build OpenVINO Runtime for later inference on Intel CPU on macOS with Intel CPU underneath. This can be done using two ways:

  • Compile on Intel CPU host using native compilation. Note, that Build steps show this scenario.
  • Cross-compile on OSX Apple Silicon.

The software was:

  • validated on macOS 15 (native arm64)

The x86_64 native and cross-compilation paths described in this guide are not exercised by the current CI matrix.

Software Requirements

  • brew package manager to install additional dependencies. Use install brew guide to achieve this.
  • Installation step for python and python libraries varies depending on the host architecture:
    • x86_64 Python 3.11 - 3.14 for the OpenVINO Runtime Python API:
    sh
    % # let's have a look what python versions are available in brew
% brew search python
% # select preferred version of python based on available ones, e.g. 3.11
% brew install [email protected]
    • arm64 Select universal2 installer from Python releases download page and install python-3.X.Y-macos11.pkg image. This allows to have universal python libraries of OpenVINO Python API (build x86_64) .
  • CMake 3.26 or higher and other development tools:
    sh
    % brew install cmake scons fdupes git-lfs ninja
  • Clang compiler and other command line tools from Xcode 10.1 or higher:
    sh
    % xcode-select --install
  • (Optional; native compilation only, x86_64) Product and samples dependencies:
    sh
    % brew install tbb pugixml flatbuffers snappy protobuf
  • Additional pip dependencies to build OpenVINO Runtime Python API:
    sh
    % # update pip and setuptools to newer versions
% python3 -m pip install -U pip
% python3 -m pip install -r <openvino source tree>/src/bindings/python/requirements.txt
    Additional install requirements (after OpenVINO repo clone) in order to build OpenVINO Python API:
    sh
    % python3 -m pip install -r <openvino source tree>/src/bindings/python/wheel/requirements-dev.txt

How to build

  1. Create a build folder:
sh
mkdir build && cd build
  1. (CMake configure) OpenVINO project uses a CMake-based build system. In the created build directory, run cmake to fetch project dependencies and create build rules:
sh
cmake -G "Ninja Multi-Config" -DENABLE_SYSTEM_PUGIXML=ON -DENABLE_SYSTEM_SNAPPY=ON -DENABLE_SYSTEM_PROTOBUF=ON ..

NOTE: By default OpenVINO CMake scripts try to introspect the system and enable all possible functionality based on that. You can look at the CMake output and see warnings, which show that some functionality is turned off and the corresponding reason, guiding what to do to install additionally to enable unavailable functionality. Additionally, you can change CMake options to enable / disable some functionality, add / remove compilation flags, provide custom version of dependencies like TBB, PugiXML, OpenCV, Protobuf. For more information, see CMake Options for Custom Compilation.

  1. (CMake build) Build OpenVINO project:
sh
cmake --build . --config Release --parallel $(sysctl -n hw.ncpu)

All built binaries are located in <openvino_source_dir>/bin/intel64/Release/ and wheel packages are located in <openvino_build_dir>/wheels.

  1. (Optional install) Once you have built OpenVINO, you can install artifacts to a preferred location:
sh
cmake -DCMAKE_INSTALL_PREFIX=<installation location> -P cmake_install.cmake

Cross-compilation

Since OSX version 11.x and Xcode version 12.2, the Apple development tools allows to compile arm64 code on x86 hosts and vice-versa. Based on this, OpenVINO can be compiled even on Apple Silicon machines, then such artifacts can be run on both Intel CPU hosts and Apple Silicon hosts (using Rosetta). To do this, add -DCMAKE_OSX_ARCHITECTURES=x86_64 in the cmake configuration step when compiling OpenVINO following the steps above. Don't enable any system library usage explicitly via CMake options (remove all -DENABLE_SYSTEM_* options), because they have arm64 architecture, e.g.:

sh
file /opt/homebrew/Cellar/tbb/2021.5.0_2/lib/libtbb.12.5.dylib
/opt/homebrew/Cellar/tbb/2021.5.0_2/lib/libtbb.12.5.dylib: Mach-O 64-bit dynamically linked shared library arm64

If you will see the errors like the one below:

sh
ld: warning: ignoring file /opt/homebrew/lib/libopencv_imgproc.4.6.0.dylib, building for macOS-x86_64 but attempting to link with file built for macOS-arm64
Undefined symbols for architecture x86_64:
  "cv::Mat::Mat(cv::Size_<int>, int, void*, unsigned long)", referenced from:
      _image_resize in opencv_c_wrapper.cpp.o
      _image_save in opencv_c_wrapper.cpp.o
....
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)

Disable its usage in cmake or completely remove such library from the system (e.g. brew uninstall opencv), because it's pure arm64 and cannot be used to compile x86_64 binaries.

Or you have to explicitly find / compile x86_64 (or even universal2) dependencies by yourself and pass it to OpenVINO cmake scripts. E.g. compile oneTBB using additional option -DCMAKE_OSX_ARCHITECTURES="x86_64;arm64", install and then set export TBBROOT= which will be used by OpenVINO.

See also

Dev Build Raspbian (`docs/dev/build_raspbian.md`)

Build OpenVINO™ Runtime for Raspbian Stretch OS

NOTE: Since 2023.0 release, you can compile OpenVINO Intel CPU plugin on ARM platforms.

Hardware Requirements

  • Raspberry Pi with Raspbian Stretch OS or Raspberry Pi OS (32 or 64-bit).

    NOTE: Despite the Raspberry Pi CPU is ARMv8, 32-bit OS detects ARMv7 CPU instruction set. The default gcc compiler applies ARMv6 architecture flag for compatibility with lower versions of boards. For more information, run the gcc -Q --help=target command and refer to the description of the -march= option.

Compilation

You can perform native compilation of the OpenVINO Runtime for Raspberry Pi, which is the most straightforward solution. However, it might take at least one hour to complete on Raspberry Pi 3.

  1. Install dependencies:
bash
sudo apt-get update
sudo apt-get install -y git cmake scons build-essential
  1. Clone the repository:
text
git clone --recurse-submodules --single-branch --branch=master https://github.com/openvinotoolkit/openvino.git
  1. Go to the cloned openvino repository:
bash
cd openvino/
  1. Create a build folder:
bash
mkdir build && cd build/
  1. Build the OpenVINO Runtime:
bash
cmake -DCMAKE_BUILD_TYPE=Release \
      -DARM_COMPUTE_SCONS_JOBS=$(nproc --all) \
.. && cmake --build . --parallel

NOTE: The build command may fail due to insufficient RAM. To fix this issue, you can increase the swap size:

  1. Deactivate the current swap:
bash
sudo dphys-swapfile swapoff
  1. Modify the swap size by setting CONF_SWAPSIZE=8192 in /etc/dphys-swapfile.
  2. Recreate the swap file:
bash
sudo dphys-swapfile setup
  1. Start swap:
bash
sudo dphys-swapfile swapon

Cross-compilation for Raspberry Pi ARM64

Cross-compilation uses one host-specific CMake toolchain file:

  • cmake/arm64.toolchain.cmake for Linux-hosted cross-compilation to Raspberry Pi AArch64 Linux;
  • cmake/rpi-aarch64-linux-from-macos.toolchain.cmake for macOS-hosted cross-compilation to Raspberry Pi AArch64 Linux;
  • cmake/rpi-aarch64-linux-from-windows.toolchain.cmake for Windows-hosted cross-compilation to Raspberry Pi AArch64 Linux.

The macOS and Windows toolchains generate compiler, binutils, and Arm Compute Library SCons wrappers in the build directory. Wrapper templates and helper sources are stored in cmake/rpi_cross/.

Create a working directory for downloaded host tools and build outputs:

bash
mkdir openvino-rpi-cross
export OV_RPI_HOME=${PWD}/openvino-rpi-cross

On Windows PowerShell:

powershell
$env:OV_RPI_HOME = "$PWD\openvino-rpi-cross"
New-Item -ItemType Directory -Force -Path $env:OV_RPI_HOME

Prepare a target sysroot

The macOS and Windows toolchains require a Raspberry Pi ARM64 Linux sysroot. The sysroot must match the Raspberry Pi userspace you will run on, especially glibc and C++ runtime versions. Use a reproducible package repository, OS image, or the sysroot bundled with the installed cross toolchain; do not copy files from a live Raspberry Pi.

The examples below use the sysroot bundled with the selected cross toolchain. If you use a distro-matched sysroot from another source, pass it with -DOV_RPI_SYSROOT=.

Linux host

Install host tools:

bash
sudo apt-get update
sudo apt-get install -y git cmake ninja-build scons python3 python3-pip \
    gcc-aarch64-linux-gnu g++-aarch64-linux-gnu pkg-config xz-utils zstd

Configure and build:

bash
cmake -S . -B build-rpi-aarch64 -G Ninja \
    -DCMAKE_BUILD_TYPE=Release \
    -DCMAKE_TOOLCHAIN_FILE=$PWD/cmake/arm64.toolchain.cmake \
    -DCMAKE_INSTALL_PREFIX=$PWD/install-rpi-aarch64

cmake --build build-rpi-aarch64 --parallel
cmake --install build-rpi-aarch64

The Linux toolchain file uses the standard aarch64-linux-gnu-* tool names and is intentionally kept minimal. If your Linux cross toolchain needs an explicit target sysroot, pass standard CMake sysroot settings that match your distribution toolchain.

macOS host

Install host tools and download the Linux AArch64 GNU cross toolchain:

bash
brew install cmake ninja scons python xz zstd
brew install messense/macos-cross-toolchains/aarch64-unknown-linux-gnu

Set the toolchain sysroot for CMake:

bash
export OV_RPI_SYSROOT="$(aarch64-linux-gnu-gcc -print-sysroot)"
export OV_RPI_TOOLCHAIN_PREFIX=aarch64-linux-gnu

Configure and build with the macOS-hosted Raspberry Pi ARM64 Linux toolchain:

bash
cmake -S . -B build-rpi-aarch64-macos -G Ninja \
    -DCMAKE_BUILD_TYPE=Release \
    -DCMAKE_TOOLCHAIN_FILE=$PWD/cmake/rpi-aarch64-linux-from-macos.toolchain.cmake \
    -DOV_RPI_TOOLCHAIN_PREFIX=$OV_RPI_TOOLCHAIN_PREFIX \
    -DOV_RPI_SYSROOT=$OV_RPI_SYSROOT \
    -DCMAKE_INSTALL_PREFIX=$PWD/install-rpi-aarch64-macos

cmake --build build-rpi-aarch64-macos --parallel
cmake --install build-rpi-aarch64-macos

The macOS toolchain file generates host wrapper scripts in the build directory and points CMake, binutils, and Arm Compute Library at the target sysroot.

Windows host

Download and unpack CMake and Ninja from PowerShell:

powershell
$env:OV_RPI_TOOLS = "$env:OV_RPI_HOME\tools"
New-Item -ItemType Directory -Force -Path $env:OV_RPI_TOOLS

$CMakeVersion = "3.30.5"
$CMakeUrl = "https://github.com/Kitware/CMake/releases/download/v$CMakeVersion/cmake-$CMakeVersion-windows-x86_64.zip"
$CMakeZip = "$env:OV_RPI_HOME\cmake-$CMakeVersion-windows-x86_64.zip"
Invoke-WebRequest -Uri $CMakeUrl -OutFile $CMakeZip
Expand-Archive -Path $CMakeZip -DestinationPath $env:OV_RPI_TOOLS -Force
Move-Item -Force "$env:OV_RPI_TOOLS\cmake-$CMakeVersion-windows-x86_64" "$env:OV_RPI_TOOLS\cmake"

$NinjaVersion = "1.12.1"
$NinjaUrl = "https://github.com/ninja-build/ninja/releases/download/v$NinjaVersion/ninja-win.zip"
$NinjaZip = "$env:OV_RPI_HOME\ninja-win.zip"
New-Item -ItemType Directory -Force -Path "$env:OV_RPI_TOOLS\ninja"
Invoke-WebRequest -Uri $NinjaUrl -OutFile $NinjaZip
Expand-Archive -Path $NinjaZip -DestinationPath "$env:OV_RPI_TOOLS\ninja" -Force

Install Python and SCons:

powershell
$PythonVersion = "3.12"
winget install --id "Python.Python.$PythonVersion" -e
py "-$PythonVersion" -m pip install --user scons
$PythonScripts = py "-$PythonVersion" -c "import site; print(site.getuserbase() + r'\Scripts')"

Download and unpack the Arm GNU Toolchain for Windows targeting AArch64 GNU/Linux:

powershell
$ToolchainUrl = "https://developer.arm.com/-/media/Files/downloads/gnu/14.3.rel1/binrel/arm-gnu-toolchain-14.3.rel1-mingw-w64-i686-aarch64-none-linux-gnu.zip"
$ToolchainZip = "$env:OV_RPI_HOME\arm-gnu-toolchain-aarch64-none-linux-gnu.zip"
Invoke-WebRequest -Uri $ToolchainUrl -OutFile $ToolchainZip
Expand-Archive -Path $ToolchainZip -DestinationPath $env:OV_RPI_TOOLS -Force
$env:OV_RPI_TOOLCHAIN_ROOT = "$env:OV_RPI_TOOLS\arm-gnu-toolchain-14.3.rel1-mingw-w64-i686-aarch64-none-linux-gnu"

The Windows-hosted Arm package uses the aarch64-none-linux-gnu prefix and provides the target sysroot under $env:OV_RPI_TOOLCHAIN_ROOT\aarch64-none-linux-gnu\libc.

Configure and build with the Windows-hosted Raspberry Pi ARM64 Linux toolchain:

powershell
$env:OV_RPI_REPO = (Get-Location).Path
$env:Path = "$env:OV_RPI_TOOLCHAIN_ROOT\bin;$env:OV_RPI_TOOLS\ninja;$env:OV_RPI_TOOLS\cmake\bin;$PythonScripts;$env:Path"
$env:OV_RPI_SYSROOT = "$env:OV_RPI_TOOLCHAIN_ROOT\aarch64-none-linux-gnu\libc"
$env:OV_RPI_TOOLCHAIN_PREFIX = "aarch64-none-linux-gnu"

& "$env:OV_RPI_TOOLS\cmake\bin\cmake.exe" -S $env:OV_RPI_REPO -B "$env:OV_RPI_REPO\build-rpi-aarch64-windows" -G Ninja `
    "-DCMAKE_MAKE_PROGRAM=$env:OV_RPI_TOOLS\ninja\ninja.exe" `
    -DCMAKE_BUILD_TYPE=Release `
    "-DCMAKE_TOOLCHAIN_FILE=$env:OV_RPI_REPO\cmake\rpi-aarch64-linux-from-windows.toolchain.cmake" `
    "-DOV_RPI_TOOLCHAIN_PREFIX=$env:OV_RPI_TOOLCHAIN_PREFIX" `
    "-DOV_RPI_SYSROOT=$env:OV_RPI_SYSROOT" `
    "-DCMAKE_INSTALL_PREFIX=$env:OV_RPI_REPO\install-rpi-aarch64-windows"

& "$env:OV_RPI_TOOLS\cmake\bin\cmake.exe" --build "$env:OV_RPI_REPO\build-rpi-aarch64-windows" --parallel
& "$env:OV_RPI_TOOLS\cmake\bin\cmake.exe" --install "$env:OV_RPI_REPO\build-rpi-aarch64-windows"

The Windows toolchain file generates host wrapper scripts, a local grep.exe required by OpenVINO's CMake probes, and Arm Compute Library SCons wrappers in the build directory. Do not mix the Arm GNU Toolchain C++ runtime headers with a different distro sysroot unless the libc and runtime versions are known to be compatible.

Build CPU benchmark runtime targets

To build only the runtime pieces needed to run benchmark_app on the Raspberry Pi CPU, build the CPU plugin, the IR frontend, and benchmark_app explicitly:

bash
cmake --build build-rpi-aarch64 --target \
    openvino_intel_cpu_plugin openvino_ir_frontend benchmark_app \
    --parallel

For macOS and Windows builds, replace build-rpi-aarch64 with build-rpi-aarch64-macos or build-rpi-aarch64-windows.

Verify and run on Raspberry Pi

After any cross-build, verify that the produced binaries are Linux AArch64 ELF files:

bash
${OV_RPI_TOOLCHAIN_PREFIX:-aarch64-linux-gnu}-readelf -h \
    bin/aarch64/Release/benchmark_app

On a Windows host, use the Arm GNU Toolchain readelf.exe:

powershell
& "$env:OV_RPI_TOOLCHAIN_ROOT\bin\aarch64-none-linux-gnu-readelf.exe" -h `
    .\bin\aarch64\Release\benchmark_app

The output must include Class: ELF64 and Machine: AArch64.

For a quick CPU smoke test, copy the runtime output directory to the Raspberry Pi and run benchmark_app with the copied libraries on LD_LIBRARY_PATH:

bash
rsync -a bin/aarch64/Release/ rpi@raspberrypi:~/openvino-rpi-cross/bin/

ssh rpi@raspberrypi \
    'cd ~/openvino-rpi-cross/bin && \
     LD_LIBRARY_PATH=$PWD ./benchmark_app \
       -m ~/models/model.xml \
       -d CPU \
       -api sync \
       -niter 10 \
       -nireq 1 \
       -nstreams 1 \
       -hint none'

Installing the build with cmake --install is the preferred way to preserve the expected runtime layout.

Additional Build Options

  • To build Python API, install libpython3-dev:armhf and python3-pip
    packages using apt-get; add the following options:
    sh
    -DENABLE_PYTHON=ON \
-DPython3_EXECUTABLE=/usr/bin/python3.8

See also

Dev Build Riscv64 (`docs/dev/build_riscv64.md`)

Cross compile OpenVINO™ Runtime for RISCV64 systems

This guide shows how to build OpenVINO Runtime for 64-bit RISC-V devices. Due to limited resources, cross compilation is used now for building OpenVINO targeting RISC-V development boards.

Cross compilation was tested on the following hosts:

  • Ubuntu 22.04 (64-bit), x64

The software was validated on the following devices:

Software requirements

  • CMake 3.26 or higher
  • GCC 7.5 or higher (for non-RVV) / riscv-gnu-toolchain (for RVV)
  • Python 3.10 for the optional OpenVINO Runtime Python API cross-build described below

How to build

Currently, there are two ways to build OpenVINO Runtime for 64-bit RISC-V platforms:

  1. The build with RVV intrinsics using riscv-gnu-toolchain. This GNU Compiler Toolchain supports RVV 0.7.1 and ratified RVV 1.0. The vector intrinsics use the common prefix __riscv_.
  2. The build without optimized primitives using installed Linux packages. The compilers in these packages don't support RVV intrinsics.

NOTE: Currently CPU Plugin in OpenVINO supports Just-In-Time (JIT) code generation for limited scope of operations on devices with RVV 1.0.
All three described above ways to build OpenVINO Runtime for 64-bit RISC-V supports JIT code generation.

Steps

  1. Prerequisite:
  • For target with RVV intrinsics using riscv-gnu-toolchain:
    sh
    git clone https://github.com/riscv-collab/riscv-gnu-toolchain.git
cd riscv-gnu-toolchain
./configure --prefix=/opt/riscv
make linux build-qemu -j$(nproc)

    NOTE: The build-qemu target is optional, as it is used to build the qemu simulator. However, it is recommended to build the qemu simulator, since it is much more convenient to validate the software on your host than on your devices. More information can be seen here.

  • For target without optimized primitives using installed Linux packages:
    sh
    apt-get update
apt-get install -y  gcc-riscv64-linux-gnu g++-riscv64-linux-gnu binutils-riscv64-linux-gnu

  1. Clone OpenVINO repository and init submodules:

    sh
    git clone --recursive https://github.com/openvinotoolkit/openvino.git
cd openvino

  2. Install build dependencies using the install_build_dependencies.sh script in the
    project root folder.

    sh
    sudo ./install_build_dependencies.sh

  3. Create a build folder:

    sh
    mkdir build && cd build

  4. To cross compile OpenVINO Runtime for RISC-V devices, run cmake with specified CMAKE_TOOLCHAIN_FILE and RISCV_TOOLCHAIN_ROOT (the last one is needed only for build using GNU toolchain).

  • For target with RVV intrinsics using riscv-gnu-toolchain:

    sh
    cmake .. \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_INSTALL_PREFIX=<openvino_install_path> \
  -DCMAKE_TOOLCHAIN_FILE=../cmake/toolchains/riscv64-gnu.toolchain.cmake \
  -DRISCV_TOOLCHAIN_ROOT=/opt/riscv

    NOTE: The riscv-gnu-toolchain is build as there are essential files used for cross compilation under /opt/riscv/sysroot. The latest stable versions of Clang or GCC both support compiling source code into RISC-V instructions, so it is acceptable to choose your preferable compilers by specifying -DCMAKE_C_COMPILER and CMAKE_CXX_COMPILER. But remember to add the key -DCMAKE_SYSROOT=/opt/riscv/sysroot, otherwise many fundamental headers and libs could not be found during cross compilation.

  • For target without optimized primitives using installed Linux packages:

    sh
    cmake .. \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_INSTALL_PREFIX=<openvino_install_path> \
  -DCMAKE_TOOLCHAIN_FILE=../cmake/toolchains/riscv64.linux.toolchain.cmake

    Then run make to build the project:

    sh
    make install -j$(nproc)

(Optional) Build the OpenVINO Runtime Python API

To enable cross-compilation with python, the library libpython3-dev:riscv64 should be on the host machine.

When installing packages using the utilities apt or apt-get the packages are downloaded from apt software repositories. On Ubuntu the apt software repositories are defined in the /etc/apt/sources.list file or in separate files under the /etc/apt/sources.list.d/ directory. Host machine contains host-specific repositories (for example, x86-x64) in these files.

  1. Add riscv64 repositories to download riscv64-specific packages:

    sh
    echo deb [arch=riscv64] http://ports.ubuntu.com/ubuntu-ports/ jammy main >> riscv64-sources.list
echo deb [arch=riscv64] http://ports.ubuntu.com/ubuntu-ports/ jammy universe >> riscv64-sources.list
echo deb [arch=riscv64] http://ports.ubuntu.com/ubuntu-ports/ jammy-updates main >> riscv64-sources.list
echo deb [arch=riscv64] http://ports.ubuntu.com/ubuntu-ports/ jammy-security main >> riscv64-sources.list
mv riscv64-sources.list /etc/apt/sources.list.d/
dpkg --add-architecture riscv64
apt-get update -o Dir::Etc::sourcelist=/etc/apt/sources.list.d/riscv64-sources.list

  2. Install libpython3-dev:riscv64 using apt-get:

    sh
    apt-get install -y --no-install-recommends libpython3-dev:riscv64

    Create symbolink to allow python to find riscv64-linux-gnu/python3.10/pyconfig.h in /usr/include/python3.10/ (this header is initially stored in /usr/include/riscv64-linux-gnu/)

    sh
    ln -s /usr/include/riscv64-linux-gnu/ /usr/include/python3.10/

  3. Add the keys -DENABLE_PYTHON=ON -DENABLE_WHEEL=ON to cmake command during OpenVINO build.

Note: This optional Python 3.10 cross-build is not exercised by the current GitHub Actions
RISC-V workflow, which builds the C++ Runtime with ENABLE_PYTHON=OFF. The target device must
provide Python 3.10 when using these manually verified instructions.

RISC-V Emulation software

In order to test applications without hardware one can use emulation software. The command line example to launch executable file with riscv64 emulation:

sh
<riscv_toolchain_root>/bin/qemu-riscv64 -cpu=<target_cpu> <executable_file_path>

<riscv_toolchain_root> matches the RISCV_TOOLCHAIN_ROOT used during the build (for example, /opt/riscv), or use qemu-riscv64 from your system PATH.

For example, to emulate RVV 0.7.1:

sh
<riscv_toolchain_root>/bin/qemu-riscv64 -cpu rv64,v=true,vext_spec=v0.7.1 <executable_file_path>

Or to emulate RVV 1.0:

sh
<riscv_toolchain_root>/bin/qemu-riscv64 -cpu rv64,v=true,vext_spec=v1.0 <executable_file_path>

See also

Dev Build Webassembly (`docs/dev/build_webassembly.md`)

Build OpenVINO™ Runtime for WebAssembly

This guide shows how to build OpenVINO for WebAssembly using Emscripten SDK. Emscripten SDK can be directly downloaded and used, but it is more easier to use the emscripten/emsdk docker image.

The approach is validated on Linux, Windows and macOS operation systems.

Software Requirements

How to build

  1. Clone OpenVINO repository and init submodules:
sh
git clone https://github.com/openvinotoolkit/openvino.git
cd openvino
git submodule update --init --recursive
  1. Run docker image and mount a volume with OpenVINO source code:
sh
$ docker pull emscripten/emsdk
$ docker run -it --rm -v `pwd`:/openvino emscripten/emsdk bash
  1. (CMake configure) Run cmake configure step using helper emscripten command:
sh
$ mkdir build && cd build
$ emcmake cmake -DCMAKE_BUILD_TYPE=Release /openvino
  1. (CMake build) Build OpenVINO project:
sh
$ emmake make -j$(nproc)

openvino.wasm and openvino.js files are located in:

  • <openvino_source_dir>/bin/ia32/Release/ on host machine file system.
  • /openvino/bin/ia32/Release in docker environment.
    These files can be used in browser applications.

See also

Dev Build Windows (`docs/dev/build_windows.md`)

Build OpenVINO™ Runtime for Windows systems

OpenVINO can be compiled for different architectures on Windows: X64 or ARM64. In order to build for ARM64 architecture, the machine with Windows on ARM is required as only native compilation is supported. Refer to the page for details.

Supported configurations:

  • Windows 10 x86 64-bit or higher with Visual Studio 2019 or higher build for X64 architecture.
  • Windows on ARM (shortly WoA) to build for ARM64 architecture. OpenVINO was validated on Windows DevKit 2023

Current validation includes x86_64 builds with Visual Studio 2022 and Visual Studio 2026.
Visual Studio 2019 remains the documented minimum but is not exercised by the current CI matrix.

Software requirements

  • CMake 3.26 or higher
  • Microsoft Visual Studio 2019 or higher, version 16.3 or later

    NOTE: Native Microsoft Visual Studio for WoA has been available since version 3.11.

  • Python 3.11 - 3.14 for OpenVINO Runtime Python API
  • [Git for Windows*]
  • (Windows on ARM only) LLVM for Windows on ARM (WoA)

    NOTE: After installation, make sure clang-cl compiler is available from PATH.

How to build

NOTE: By default, the build enables the OpenVINO Runtime GPU plugin to infer models on your Intel® Processor Graphics. This requires you to download and install the Intel® Graphics Driver for Windows before running the build. If you don't want to use the GPU plugin, use the -DENABLE_INTEL_GPU=OFF CMake build option and skip the installation of the Intel® Graphics Driver.

  1. Clone submodules:

    sh
    git clone https://github.com/openvinotoolkit/openvino.git
cd openvino
git submodule update --init --recursive

  2. Create build directory:

    sh
    mkdir build && cd build

  3. In the build directory, run cmake to fetch project dependencies and generate a Visual Studio solution:

    sh
    cmake -G "Visual Studio 17 2022" <path/to/openvino>

  4. Build generated solution in Visual Studio or run cmake --build . --config Release --verbose -j<number_of_jobs> to build from the command line. View the number of available processing units with WMIC cpu get numberofLogicalProcessors. Be aware that this process may take some time.

  5. Before running the samples, add paths to the Threading Building Blocks (TBB) binaries used for the build to the %PATH% environment variable. By default, TBB binaries are downloaded by the CMake-based script to the /temp//tbb/bin folder.

Building with PDB files support

If you intend to generate PDB files and debug your build or run some tools, which require PDB files, it is essential to set the CMake build type appropriately.
You should utilize one of the following CMake build type options:

  • -DCMAKE_BUILD_TYPE=RelWithDebInfo: This option generates PDB files with release information, making it suitable for debugging optimized builds.
  • -DCMAKE_BUILD_TYPE=Debug: This option generates PDB files optimized for debugging, providing comprehensive debugging information.
  • -DCMAKE_BUILD_TYPE=Release -DENABLE_PDB_IN_RELEASE=ON to build in Release mode, but with debug symbols generated.

Note, that PDB symbols files are supported for both dynamic and static builds.

To pack PDB files, it's essential to run cmake command to create a dedicated archive:

sh
cmake --install <build folder> --config <Release | Debug | RelWithDebInfo> --prefix <installation path> --component pdb

Building with Python API support

  1. Enable the -DENABLE_PYTHON=ON in the CMake (Step #3) option above. To specify an exact Python version, use the following options (requires cmake 3.16 and higher):
    sh
    -DPython3_EXECUTABLE="C:\Program Files\Python11\python.exe"
  2. To build a wheel package (.whl), enable the -DENABLE_WHEEL=ON option in the CMake step above (Step 3), and install requirements:
    sh
    pip install -r <openvino source tree>\src\bindings\python\wheel\requirements-dev.txt
  3. After the build process finishes, export the newly built Python libraries to the user environment variables:
    text
    set PYTHONPATH=<openvino_repo>/bin/<arch>/Release/python;<openvino_repo>/tools/ovc;%PYTHONPATH%
set OPENVINO_LIB_PATHS=<openvino_repo>/bin/<arch>/Release;<openvino_repo>/temp/<platform>/tbb/bin
set PATH=<openvino_repo>/tools/ovc/openvino/tools/ovc:%PATH%

or install the wheel with pip:

text
pip install build/wheels/openvino-2026.2.1-9612-cp313-cp313-win_arm64.whl

NOTE: Wheels are generated under the /wheels/ directory. Substitute the filename with the artifact that matches your Python version and target architecture.

Building OpenVINO with Ninja* Build System

sh
call "C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Auxiliary\Build\vcvars64.bat"
cmake -G Ninja -Wno-dev -DCMAKE_BUILD_TYPE=Release ..
cmake --build . --parallel

See also

Dev Building Documentation (`docs/dev/building_documentation.md`)

How to build documentation

The following procedure was validated on Windows and Ubuntu operation systems.

Table of content:

Installing dependencies

The doxygen and latex must be installed in addition to usual build dependencies:

Windows

Ubuntu systems

  • Latex and graphviz:
sh
apt-get install texlive-full graphviz
  • goxygen version >= 1.8.20:
text
$ git clone https://github.com/doxygen/doxygen.git
$ cd doxygen
$ git checkout Release_1_8_20
$ mkdir build
$ cd build
$ cmake ..
$ cmake --build . -j8
$ sudo make install

Building documentation

You should run cmake with -DENABLE_DOCS=ON and it will find all dependencies automatically on Ubuntu systems, while on Windows we still need to specify paths to the installed dependencies:

sh
cmake -DLATEX_COMPILER="C:/Program Files/MiKTeX/miktex/bin/x64/latex.exe" \
      -DDOXYGEN_DOT_EXECUTABLE="C:/Program Files (x86)/Graphviz2.38/bin/dot.exe" \
      -DDOXYGEN_EXECUTABLE="C:/Program Files/doxygen/bin/doxygen.exe" \
      -DENABLE_DOCS=ON \

Once the dependencies are found, the project must generated using CMake. The target openvino_docs must be built to generate doxygen documentation, the generated files can be found at /docs/html/index.html

See also

Dev Cmake Options For Custom Compilation (`docs/dev/cmake_options_for_custom_compilation.md`)

CMake Options for Custom Compilation

This document provides description and default values for CMake options that can be used to build a custom OpenVINO runtime using the open source version. To understand all the dependencies when creating a custom runtime from the open source repository, refer to the OpenVINO Runtime Introduction.

Table of contents:

Disable / enable plugins build and other components

  • Inference plugins:
    • ENABLE_INTEL_CPU enables CPU plugin compilation:
      • ON is default for x86 platforms; OFF, otherwise.
    • ENABLE_INTEL_GPU enables Intel GPU plugin compilation:
      • ON is default for x86 platforms; not available, otherwise.
    • ENABLE_INTEL_NPU enables Intel NPU plugin compilation:
      • ON is default for Windows and Linux x86 platforms; not available, otherwise.
    • ENABLE_HETERO enables HETERO plugin build:
      • ON is default.
    • ENABLE_MULTI enables MULTI plugin build:
      • ON is default.
    • ENABLE_AUTO enables AUTO plugin build:
      • ON is default.
    • ENABLE_TEMPLATE enables TEMPLATE plugin build:
      • ON is default.
    • ENABLE_AUTO_BATCH enables Auto Batch plugin build:
      • ON is default.
    • ENABLE_PROXY enables Proxy plugin compilation:
      • ON is default.
  • Frontends to work with models from frameworks:
    • ENABLE_OV_ONNX_FRONTEND enables ONNX frontend plugin for OpenVINO Runtime:
      • ON is default.
    • ENABLE_OV_PADDLE_FRONTEND enables PDPD frontend plugin for OpenVINO Runtime:
      • ON is default.
    • ENABLE_OV_TF_FRONTEND enables TensorFlow frontend plugin for OpenVINO Runtime:
      • ON is default.
    • ENABLE_OV_TF_LITE_FRONTEND enables TensorFlow Lite frontend plugin for OpenVINO Runtime:
      • ON is default.
    • ENABLE_OV_PYTORCH_FRONTEND enables PyTorch frontend plugin for OpenVINO Runtime:
      • ON is default.
    • ENABLE_OV_JAX_FRONTEND enables JAX frontend plugin for OpenVINO Runtime:
      • ON is default.
    • ENABLE_OV_IR_FRONTEND enables OpenVINO Intermediate Representation frontend plugin for OpenVINO Runtime:
      • ON is default.
  • OPENVINO_EXTRA_MODULES specifies path to add extra OpenVINO modules to the build.
  • ENABLE_SAMPLES enables OpenVINO Runtime samples build:
    • ON is default.
  • ENABLE_PYTHON enables Python API build:
    • ON if python requirements are satisfied (auto-discovered by CMake).
  • ENABLE_WHEEL enables Python OpenVINO Runtime and Development wheels build:
    • ON if requirements are satisfied (auto-discovered by CMake).
  • ENABLE_TESTS enables tests compilation:
    • OFF is default.
  • ENABLE_DOCS enables building the OpenVINO documentation:
    • OFF is on Debian (Ubuntu) OSes
    • OFF is in other cases.
  • ENABLE_SYSTEM_PUGIXML builds with system version of pugixml if it is available on the system.
  • ENABLE_SYSTEM_PROTOBUF use protobuf installed on the system (used by ONNX, PaddlePaddle and TensorFlow frontends):
    • OFF is default.
  • ENABLE_SYSTEM_FLATBUFFERS use FlatBuffers installed on the system (used by Tensorflow Lite frontend):
    • ON is default.
  • ENABLE_SYSTEM_TBB use TBB installed on the system:
    • ON is on Debian (Ubuntu) OSes.
    • OFF is in other cases.
  • ENABLE_SYSTEM_OPENCL use OpenCL installed on the system:
    • ON is default.

Options affecting binary size

  • ENABLE_LTO boolean option to enable Link Time Optimizations:
    • OFF is default, because it takes longer time to link binaries.
    • ON is enabled for OpenVINO release binaries.
    • Available on Unix* compilers only like GCC or CLANG.
    • Gives 30% decrease in binary size together with other optimization options used to build OpenVINO.
  • THREADING points to the OpenVINO threading interface:
    • TBB is the default option, which enables build with Intel TBB and tbb::static_partitioner.
    • TBB_AUTO enables building with Intel TBB.
    • OMP enables building with Intel OpenMP.
    • SEQ disables threading optimizations. Can be used in cases when TBB binaries are absent.
    • Note: because TBB is a template library, it increases binary size because of multiple instantiations of tbb::parallel_for
  • ENABLE_TBBBIND_2_5 enables prebuilt static TBBBind 2.5 usage:
    • ON is default, because OpenVINO Runtime should be generic out of box.

Note: TBBBind 2.5 is needed when OpenVINO inference targets CPUs with:

  • NUMA support (Non-Unified Memory Architecture), e.g. to detect a number of NUMA nodes
  • Hybrid architecture to separate Performance / Efficiency cores and schedule tasks in the optimal way.

Note: if you build OpenVINO runtime with oneTBB support where TBBBind 2.5 is automatically loaded by TBB in runtime, then set ENABLE_TBBBIND_2_5 to OFF

  • make sure libtbbbind.so is in the same folder as libtbb.so. For example, oneTBB package on Ubuntu 22.04 has libtbbbind missed. https://bugs.launchpad.net/ubuntu/+source/onetbb/+bug/2006898
  • oneTBB relies on higher version hwloc to recognize hybrid CPU core types correctly, on some machines, they require higher hwloc version to work correctly. Check if hwloc-info --version returns hwloc version >= 2.7.0, Ubuntu 20.04 with hwloc 2.1.0
  • ENABLE_SSE42 enables SSE4.2 optimizations:
    • ON is default for x86 platforms; not available for other platforms.
    • Affects only OpenVINO Runtime common part and preprocessing plugin, does not affect the oneDNN library
  • ENABLE_AVX2 enables AVX2 optimizations:
    • ON is default for x86 platforms, not available for other platforms.
    • Affects only OpenVINO Runtime common part and preprocessing plugin, does not affect the oneDNN library
  • ENABLE_AVX512F enables AVX512 optimizations:
    • ON is default for x86 platforms, not available for other platforms.
    • Affects only OpenVINO Runtime common part and preprocessing plugin, does not affect the oneDNN library
  • ENABLE_PROFILING_ITT enables profiling with Intel ITT and VTune.
    • OFF is default, because it increases binary size.
  • SELECTIVE_BUILD enables [[Conditional compilation|ConditionalCompilation]] feature.
    • OFF is default.
  • ENABLE_MLAS_FOR_CPU enables MLAS library for CPU plugin
    • ON is default for x86_64 and AARCH64 platforms
    • Affects only OpenVINO CPU plugin

Building with custom OpenCV

Some OpenVINO samples can benefit from OpenCV usage, e.g. can read more image formats as inputs. If you have OpenCV on your machine, you can pass it via CMake option:

sh
cmake -DOpenCV_DIR=<path to OpenCVConfig.cmake> ...

Building with custom TBB

When OpenVINO CMake scripts are run with TBB enabled (-DTHREADING=TBB which is default, or -DTHREADING=TBB_AUTO), CMake OpenVINO scripts automatically download prebuilt version of TBB which is ABI-compatible with the default compiler of your system. If you have a non-default compiler or want to use custom TBB, you can use:

sh
export TBBROOT=<path to TBB install root>
cmake ...

In this case OpenVINO CMake scripts take TBBROOT environment variable into account and provided TBB will be used.

Note: if you are building TBB from source files, please install TBB after and use TBBROOT to point to installation root.
Note: reference to oneTBB Note in Options affecting binary size

Test capabilities

  • ENABLE_SANITIZER builds with clang address sanitizer support:
    • OFF is default.
  • ENABLE_THREAD_SANITIZER builds with clang thread-sanitizer support:
    • OFF is default.
  • ENABLE_COVERAGE adds option to enable coverage. See dedicated guide [[how to measure test coverage|InferenceEngineTestsCoverage]]:
    • OFF is default.
  • ENABLE_FUZZING enables instrumentation of code for fuzzing:
    • OFF is default.

Other options

  • ENABLE_CLANG_FORMAT enables Clang format code style check:
    • ON is default.
  • ENABLE_FASTER_BUILD enables precompiled headers and unity build using CMake:
    • OFF is default.
  • ENABLE_INTEGRITYCHECK builds DLLs with /INTEGRITYCHECK flag:
    • OFF is default.
    • Available on MSVC compiler only.
  • ENABLE_QSPECTRE builds with /Qspectre flag:
    • OFF is default.
    • Available on MSVC compiler only.

Additional Resources

Dev Coding Style (`docs/dev/coding_style.md`)

OpenVINO Coding Guidelines

Fix order

When bringing a set of changes into compliance, address the checks in the following order:

  1. Copyright headers — fast, no build required
  2. clang-tidy — requires a clean build; fix compilation errors first, then apply tidy auto-fixes
  3. Naming style (ncc) — requires clang, but no full rebuild; fix after tidy to avoid redundant churn
  4. clang-format — purely cosmetic; run last so tidy auto-fixes don't reintroduce formatting issues

All source files must carry the standard OpenVINO copyright header.

C++ (.cpp, .hpp, .h):

cpp
// Copyright (C) 2018-<current_year> Intel Corporation
// SPDX-License-Identifier: Apache-2.0
//

Python (.py):

python
# Copyright (C) 2018-<current_year> Intel Corporation
# SPDX-License-Identifier: Apache-2.0

To check and fix headers automatically, the script at .github/scripts/check_copyright.py can be used. If it exits non-zero it writes copyright_fixes.diff, which can be applied with patch -p1 < copyright_fixes.diff.

Note: automatically generated patch can occasionally produce incorrect changes — in some cases, the manual fix is required.

Static analysis (clang-tidy)

clang-tidy-18 is used for static analysis and enforcing modern C++ patterns.

Prerequisites: clang-tidy-18 must be installed, and the project must be compiled with clang as the compiler. Before enabling it, verify that an existing build directory already uses clang:

bash
grep "CMAKE_CXX_COMPILER" <build_dir>/CMakeCache.txt | grep -i clang

If it does, re-run cmake in it to append the style flags (preserves the existing configuration):

bash
cmake -DENABLE_CLANG_FORMAT=ON -DENABLE_CLANG_TIDY=ON <build_dir>

If it doesn't (or no build directory exists), set =build-clang and configure it:

bash
cmake -B <build_dir> -DCMAKE_CXX_COMPILER=clang++-18 -DCMAKE_C_COMPILER=clang-18 \
      -DENABLE_CLANG_FORMAT=ON -DENABLE_CLANG_TIDY=ON \
      -DCMAKE_BUILD_TYPE=Release

Build a target to run checks (warnings appear inline during compilation):

bash
cmake --build  <build_dir> --target <your_target> -- -j$(nproc)

To auto-fix diagnostics, enable the fix mode and rebuild:

bash
cmake -DENABLE_CLANG_TIDY_FIX=ON <build_dir>
cmake --build  <build_dir> --target <your_target> -- -j$(nproc)

Notes:

  • Compilation errors must be resolved before clang-tidy diagnostics can be applied.
  • Auto-fix can occasionally produce incorrect changes — in some cases, the manual fix is required.
  • It is better to use a dedicated build directory for clang-tidy (e.g. build-clang/) — clang-tidy rewrites compilation databases and can interfere with incremental builds in a shared directory.
  • Do not use // NOLINT suppressions to silence diagnostics. A suppression hides the issue without fixing it and can mask real problems in future code. It is strongly recommended to suppress if if the check is a confirmed false positive with no correct alternative, and document the reason inline.

Naming style (ncc)

OpenVINO has strict rules for naming style in public API. All classes must start with a capital letter, and methods and functions should be named in snake_case style.
To check the naming style, ncc tool is integrated in the OpenVINO. Read the detailed information about the naming style can be found in the configuration file.
To activate this tool, you need to have clang on the local machine and enable the CMake option ENABLE_NCC_STYLE.
After that, you can use the ncc_all target to check the naming style.

Coding style (clang-format)

The majority of OpenVINO components use clang-format-18 for code style checks.

Prerequisites: clang-format-18 must be installed. No specific compiler is required — clang-format runs as a standalone tool independent of how the project was compiled.

The code style is based on the Google Code style with some differences. All the differences are described in the configuration file:
https://github.com/openvinotoolkit/openvino/blob/69f709028a5f8da596d1d0df9a0101e517c35708/src/.clang-format#L1-L28

To enable clang-format checks in the build, set the corresponding CMake flag:

bash
cmake -DENABLE_CLANG_FORMAT=ON <other flags> ..

Check and fix targets:

bash
cmake --build  <build_dir> --target clang_format_check_all   # report violations
cmake --build  <build_dir> --target clang_format_fix_all     # auto-fix violations

See also

Dev Conditional Compilation (`docs/dev/conditional_compilation.md`)

OpenVINO Conditional Compilation

Contents

Introduction

Conditional compilation can significantly reduce the size of OpenVINO™ binaries by excluding unnecessary components for inference of particular models. The components are:

  • layers and graph transformations in OpenVINO Runtime and plugins,
  • OpenVINO Runtime operations,
  • jit kernels in a CPU plugin,
  • arbitrary code that is not used for particular model inference.

However, conditional compilation has a significant drawback - the resulting OpenVINO runtime will properly infer only using the models and devices for which it was compiled. If just one model is used to collect statistics for compilation, only this particular model is guaranteed to work with the resulting OpenVINO runtime.

There are two conditional compilation build stages: SELECTIVE_BUILD=COLLECT and SELECTIVE_BUILD=ON.

SELECTIVE_BUILD=COLLECT enables analysis mode for annotated code regions. It can be used to collect statistics data using itt. Once this stage is completed, run an OpenVINO sample, such as benchmark_app, to generate actual statistic data. Then run itt_collector tool to dump the data into csv files. This stage can be enabled during the OpenVINO build with the options -DSELECTIVE_BUILD=COLLECT -DENABLE_PROFILING_ITT=FULL.

SELECTIVE_BUILD=ON excludes all inactive annotated code regions from the compiled binaries using csv files generated by the itt_collector tool. It is done by a header file which generated from the csv files. This stage need build OpenVINO with option -DSELECTIVE_BUILD=ON -DENABLE_PROFILING_ITT=OFF -DSELECTIVE_BUILD_STAT=<cc_data.csv>. Notice: -DENABLE_PROFILING_ITT=OFF is not must, if user wants to have ITT counters in final application, he can set it FULL: -DENABLE_PROFILING_ITT=FULL.

NOTE: If the above options are not enabled, conditional compilation will be OFF and the default behavior is kept, all OpenVINO features are enabled. You can ignore SELECTIVE_BUILD or set option -DSELECTIVE_BUILD=OFF.

Setup environment

To take advantage of conditional compilation, install the following tools:

Building for different models

Conditional compilation has two stages:

  • Collecting information about code usage
  • Building the result binaries without unused components or parts
mermaid
flowchart LR
    subgraph collect[Collect statistic]
    cmake_collect[mkdir build\ncd build\ncmake -DENABLE_PROFILING_ITT=FULL -DSELECTIVE_BUILD=COLLECT ..]
    build_collect[cmake --build .\ncmake --build . --target sea_itt_lib]
    run_collect["python ../thirdparty/itt_collector/runtool/sea_runtool.py --bindir ${OPENVINO_LIBRARY_DIR} \ \n-o ${MY_MODEL_RESULT} ! ./benchmark_app -niter 1 -nireq 1 -m ${MY_MODEL}.xml\n..."]
    cmake_collect-->build_collect
    build_collect-->run_collect
    run_collect-->run_collect
    end
    subgraph result[Build cutted OpenVINO]
    cmake_res["cmake -DSELECTIVE_BUILD=ON \ \n-DSELECTIVE_BUILD_STAT=${ABSOLUTE_PATH_TO_STATISTICS_FILES}/*.csv \ \n -DENABLE_PROFILING_ITT=OFF .."]
    build_res[cmake --build .]
    cmake_res-->build_res
    end
    collect --> result

To apply conditional compilation, follow the steps below:

  1. Perform code usage analysis:
    1. Run the CMake tool with the following options: -DENABLE_PROFILING_ITT=FULL -DSELECTIVE_BUILD=COLLECT.
    2. Select several models to be used in a specific application or target device.
    3. Use the sea_itt_lib target to build the ITT collector.
    4. Run the target application under the ITT collector for code usage analysis for each model. Statistics are generated in the .csv format.
      python thirdparty/itt_collector/runtool/sea_runtool.py --bindir ${OPENVINO_LIBRARY_DIR} -o ${MY_MODEL_RESULT} ! ./benchmark_app -niter 1 -nireq 1 -m ${MY_MODEL}.xml
  2. Build the resulting binaries:
    1. Run the CMake tool with the following options: -DSELECTIVE_BUILD=ON -DSELECTIVE_BUILD_STAT=${ABSOLUTE_PATH_TO_STATISTICS_FILES}/*.csv -DENABLE_PROFILING_ITT=OFF
    2. cmake --build <cmake_build_directory>

The -niter 1 -nireq 1 flags are highly recommended for benchmark_app. Otherwise, the trace files will be very large.
If you are using an application other than benchmark_app, remember to limit the number of inference requests and iterations.

Building for devices with different ISA

Building for devices with different ISA is quite similar to building for different models (see the previous chapter).
The differences are only in the code usage analysis step. The analysis step should be performed on target devices and all CSV files with statistics should be copied to the build machine. These files will be used for the final build.

Limitations

  • Currently, Ninja build system is not supported for the conditional compilation build.
  • Currently, conditional compilation supports OpenVINO IR models only. To enable conditional compilation, the models from alternative frameworks (PyTorch, TensorFlow, ONNX, etc.) need to be converted to OpenVINO IR models first.

See also

Dev Debug Capabilities (`docs/dev/debug_capabilities.md`)

OpenVINO Debug Capabilities

OpenVINO components provides different debug capabilities, to get more information please read:

See also

Dev Dev Doc Guide (`docs/dev/dev_doc_guide.md`)

Creating developer documentation

Developer documentation provides an overview of the component development and helps engineers understand the underlying technology.
This guide contains the requirements for writing developer documentation.

The component structure

The OpenVINO component structure is fully defined in the dedicated section of developer documentation.
This guide highlights the parts which are used for developer documentation. The structure is presented below:

text
openvino_component/
    docs/               // Contains detailed component documentation
    README.md           // Endtry point for the developer documentation

***README.md*** content

  • Include only the necessary information. Avoid overloading the document with excessive details.
  • Make sure your document has a clear structure, is easy to navigate and quickly look through.
  • Provide links to additional documentation and resources.

You can use the following template for a README.md file:

md
# Component name

Short information about the component.
 * Responsibility
 * Supported features
 * Code style

## Key contacts

The section provides information about groups of developers who can help in case of questions, and also review and merge PRs.

## Components

The section contains basic information about the included components: API, sources, tests, etc.
Detailed information can be located in the `docs/` folder. This section can contain links to these pages.

## Architecture

Is an optional section which provides main ideas about the component architecture.
It can contain references to the pages with additional information.

## Tutorials

The section contains a list of component tutorials.
Example:
 * [How to support new operation](./docs/operation_support.md)
 * [How to implement new feature](./docs/new_feature.md)
 * [How to debug the component](./docs/debug_capabilities.md)
 * ...

## See also

The section contains a list of related pages.
Example:
 * [OpenVINO™ README](../../README.md)
 * [Developer documentation](../../docs/dev/index.md)
 * ...

***docs/*** folder content

Docs folder should contain pages with a detailed description of the component features and media content for the developer documentation. Make sure that the pages in this folder provide information directly related to the topic. It is better to provide information about additional topics in a separate file and make a reference to it.

Media content

The developer documentation may include pictures, diagrams, and examples that illustrate the described features to make it easier to understand.
It is recommended to use the mermaid framework for diagrams. Read the Github Documentation to get more information. The framework allows creating diagrams and pasting them to your documents. For example, the code:

md
```mermaid
graph LR;
    A-->B;
    A-->C;
    B-->D;
    C-->D;
```

Generates the next diagram:

mermaid
graph LR;
    A-->B;
    A-->C;
    B-->D;
    C-->D;

See also

Dev Get Started (`docs/dev/get_started.md`)

Get started

Welcome to the OpenVINO Developer guides. These documents describe how to build OpenVINO, provide general ideas about its architecture and help you start contributing to OpenVINO.

If you're brand new to OpenVINO and want to jump right into code, start with the Build OpenVINO guide.

Explore other resources to learn more about OpenVINO:

See also

Dev Installing (`docs/dev/installing.md`)

Installing

Once the project is built you can install OpenVINO™ Runtime into custom location:

text
cmake --install <BUILDDIR> --prefix <INSTALLDIR>

Build and Run Samples

  1. Build samples.

    To build C++ sample applications, run the following commands:

    Linux and macOS:

    sh
    cd <INSTALLDIR>/samples/cpp
./build_samples.sh

    Windows Command Prompt:

    sh
    cd <INSTALLDIR>\samples\cpp
build_samples_msvc.bat

    Windows PowerShell:

    sh
    & <path-to-build-samples-folder>/build_samples.ps1

  2. Download a model.

    You can download an image classification model from
    Hugging Face
    to run the sample

  3. Convert the model.

    Linux and macOS:

    sh
    ovc <path-to-your-model> --compress_to_fp16=True

    Windows:

    bat
    ovc <path-to-your-model> --compress_to_fp16=True

  4. Run inference on the sample.

    Set up the OpenVINO environment variables:

    Linux and macOS:

    sh
    source <INSTALLDIR>/setupvars.sh

    Windows Command Prompt:

    bat
    <INSTALLDIR>\setupvars.bat

    Windows PowerShell:

    bat
    . <path-to-setupvars-folder>/setupvars.ps1

    The following commands run the Image Classification Code Sample using the [dog.bmp](https://storage.openvinotoolkit.org/data/test_data/images/ 224x224/dog.bmp) file as an input image, the model in IR format, and on different hardware devices:

    Linux and macOS:

    sh
    cd ~/openvino_cpp_samples_build/<architecture>/Release
./classification_sample_async -i <path-to-input-image>/dog.bmp -m <path-to-your-model>/model.xml -d CPU

    where the is the output of uname -m, for example, intel64, armhf, or aarch64.

    Windows:

    bat
    cd  %USERPROFILE%\Documents\Intel\OpenVINO\openvino_cpp_samples_build\<architecture>\Release
.\classification_sample_async.exe -i <path-to-input-image>\dog.bmp -m <path-to-your-model>\model.xml -d CPU

    where the is either intel64 or aarch64 depending on the platform architecture.

When the sample application is complete, you see the label and confidence data for the top 10 categories on the display:

Below are results of using the googlenet-v1 model.

text
Top 10 results:

Image dog.bmp

classid probability
------- -----------
156     0.6875963
215     0.0868125
218     0.0784114
212     0.0597296
217     0.0212105
219     0.0194193
247     0.0086272
157     0.0058511
216     0.0057589
154     0.0052615

Adding OpenVINO Runtime to Your Project

For CMake projects, set the OpenVINO_DIR and when you run CMake tool:

sh
cmake -DOpenVINO_DIR=<INSTALLDIR>/runtime/cmake .

Then you can find OpenVINO Runtime by [find_package]:

cmake
find_package(OpenVINO REQUIRED)
add_executable(ov_app main.cpp)
target_link_libraries(ov_app PRIVATE openvino::runtime)

add_executable(ov_c_app main.c)
target_link_libraries(ov_c_app PRIVATE openvino::runtime::c)

See also

Dev Ov Dependencies (`docs/dev/ov_dependencies.txt`)

SPDX-License-Identifier: Apache-2.0

#This file provides a comprehensive list of all dependencies of OpenVINO 2025.1
#The file is part of the automation pipeline for posting OpenVINO IR models on the HuggingFace Hub, including OneBOM dependency checks.

absl-py
accuracy-checker
addict
alabaster
appnope
argparse
astor
astpretty
astroid
astunparse
atomicwrites
attrs
babel
backcall
backports-functools-lru-cache
bandit
beautifulsoup4
black
Boost
Bootstrap
BrainMaGe
cached-property
cachetools
certifi
chardet
charset-normalizer
chosen-js
clang
click
CNPY
colorama
ComputeLibrary
coverage
cycler
cython
dataclasses
decorator
Deformable-ConvNets
defusedxml
Detectron2
distlib
distro
docopt
docutils
Doxygen
Vovkos doxyrest
enum34
eradicate
execnet
fastjsonschema
Ffmpeg
filelock
flake8
flake8_builtins
flake8_coding
flake8_commas
flake8_pep3101
flake8_quotes
flake8-annotations-complexity
flake8-broken-line
flake8-bugbear
flake8-class-attributes-order
flake8-comprehensions
flake8-debugger
flake8-docstrings
flake8-eradicate
flake8-executable
flake8-expression-complexity
flake8-plugin-utils
flake8-print
flake8-pytest-style
flake8-rst-docstrings
flake8-string-format
flake8-variables-names
flatbuffers
fonttools
freetype
funcsigs
functools32
future
futures
gast
generator
gflags
gitdb
gitpython
google-auth
google-auth-oauthlib
google fonts
google-java-format
google-pasta
Google gtest-parallel
googletest
graphviz
grpcio
Gstreamer
h5py
hamcrest-core
Hyperopt
idna
idna
imagesize
importlib-metadata
importlib-resources
import-order
iniconfig
INTEL/CAFFE
Intel Integrated Performance Primitives - Intel IPP
intel/IntelSEAPI
intel/libva
ipython
isort
itt API
jedi
jinja2
json
json-c
jsonschema
pboettch json-schema-validator
linjackson78 jstyleson
John Hauser Berkeley SoftFloat
jquery - jquery/jquery
Junit
keras-nightly
keras-preprocessing
kiwisolver
lazy-object-proxy
libjpeg-turbo
Libpng
Llohse libnpy
llvm
lxml
mando
facebookresearch/maskrcnn-benchmark
markdown
markupsafe
mathjax mathjax
matplotlib
mccabe
mistune
mongo-python-driver
more-itertools
mypy
mypy-extensions
networkx
nibabel
Nithinn ncc
numpy
oauthlib
OneDNN
OneTBB
onnx
onnxruntime
Open Model Zoo
omz-tools
OpenCL
OpenCL headers
OpenCL ICD Loader
OpenCL-hpp headers
OpenCV
opencv-python
OpenJDK
OpenVINO OSS
openvino-telemetry
opt-einsum
packaging
paddlepaddle
PapaParse
pandas
parso
pathlib2
pathspec
pbr
pep8-naming
pexpect
pickleshare
pillow
platformdirs
pluggy
pluggy
progress
prompt-toolkit
protobuf
ptyprocess
pugixml
py
pyasn1
pyasn1-modules
pycodestyle
py-cpuinfo
py-leveldb
Pybind11
pydata-sphinx-theme
pydocstyle
pyenchant
pyflakes
pygments
pylint
pymongo
pyparsing
pyrsistent
pytest
pytest-dependency
pytest-forked
pytest-html
pytest-metadata
pytest-metadata
pytest-timeout
pytest-xdist
Pytorch
Python
python-attrs
python-dateutil
python-fastjsonschema
python pandas
Python-pillow
pytz
pyyaml
radon
Rapidjson
requests
requests-oauthlib
restructuredtext-lint
retrying
Roboto Fonts
rsa
ruGPT3Medium
safestringlib
scikit-build
scipy
setuptools
six
smmap
snowballstemmer
soupsieve
sphinx
sphinxcontrib-applehelp
sphinxcontrib-devhelp
sphinxcontrib-htmlhelp
sphinxcontrib-jsmath
sphinxcontrib-qthelp
sphinxcontrib-serializinghtml
sphinx-copybutton
sphinx-inline-tabs
sphinx-panels
sphinx-sitemap
stevedore
tensorboard
tensorboard-data-server
tensorboard-plugin-wit
tensorflow
tensorflow
tensorflow-estimator
termcolor
test-generator
threading_building_blocks
foutaise/texttable/
toml
tomli
Torch7
tox
traitlets
Transformers
typed-ast
types-pkg_resources
typing
typing-extensions
urllib3
Viewer
virtualenv
Torchvision
wcwidth
werkzeug
wheel
wrapt
xbyak
zipp
Zlib
backports
contextlib2
dill
keras
singledispatch
Fluid
libusb (for MYRIAD)
libGNA (for GNA)
Benchmark
GitPython
huggingface/transformers
Hamcrest JavaHamcrest
Openvinotoolkit model-server
Openvinotoolkit nncf
pytorch/vision
Sphinx documentation builder
FlatBuffers
POCO C++ Libraries
Google Mock
Google Snappy
Microsoft onnxruntime
ML commons/inference
take-action
bitsandbytes
diffusers
einops
optimum-intel
tiktoken
timm
tokenizers
torch
optimum
psutil
vscode-extension-samples
fast_tokenizer
re2
icu4c
sentencepiece
auto-gptq
transformers-stream-generator
eigen3
gtk3-devel
gstreamer1-devel
gstreamer1-plugins-base-devel
ffmpeg-devel
libmfx-devel
cmake
git
python38-devel
python38-pip
gcc-c++
gcc
python3-dev
python3-pip
ninja-build
libgtk-3-dev
libpng-dev
libjpeg-dev
libwebp-dev
libtiff5-dev
libopenexr-dev
libopenblas-dev
libx11-dev
libavutil-dev
libavcodec-dev
libavformat-dev
libswscale-dev
libavresample-dev
libtbb2
libssl-dev
libva-dev
libmfx-dev
libgstreamer1.0-dev
libgstreamer-plugins-base1.0-dev
epel-release-latest-8.noarch.rpm
rpmfusion-free-release-8.noarch.rpm
pytest-repeat 0.9.3
pip
make
httpx
ultralytics
llama.cpp
lixsmm

Dev Public Ci (`docs/dev/public_ci.md`)

Enabling tests in OpenVINO CI

Each component contains own documentation for writing and extending tests.
If you want to get more information about it please read the documentation for interested component.

GitHub Actions is used for building and testing OpenVINO on different platforms and software combinations. Find more information in the OpenVINO GitHub Actions overview document.

See Also

Dev Static Libraries (`docs/dev/static_libraries.md`)

Building OpenVINO static libraries

Contents

Introduction

Building static OpenVINO Runtime libraries allows to additionally reduce the size of a binary when it is used together with conditional compilation.
It is possible because not all interface symbols of OpenVINO Runtime libraries are exported to end users during a static build and can be removed by linker. See Static OpenVINO libraries + Conditional compilation for particular models

System requirements

  • CMake version 3.26 or higher must be used to build static OpenVINO libraries.
  • Supported OSes:
    • Windows x64
    • Linux x64
    • All other OSes may work, but have not been explicitly tested

Configure OpenVINO Runtime in the CMake stage

The default architecture of OpenVINO Runtime assumes that the following components are subject to dynamic loading during execution:

  • (Device) Inference backends (CPU, GPU, NPU, MULTI, HETERO, etc.)
  • (Model) Frontends (IR, ONNX, PDPD, TF, JAX, etc.)

With the static OpenVINO Runtime, all these modules should be linked into a final user application and the list of modules/configuration must be known for the CMake configuration stage. To minimize the total binary size, you can explicitly turn OFF unnecessary components. Use CMake Options for Custom Compilation as a reference for OpenVINO CMake configuration.

For example, to enable only IR v11 reading and CPU inference capabilities, use:

sh
cmake -DENABLE_INTEL_GPU=OFF \
      -DENABLE_INTEL_NPU=OFF \
      -DENABLE_TEMPLATE=OFF \
      -DENABLE_HETERO=OFF \
      -DENABLE_MULTI=OFF \
      -DENABLE_AUTO=OFF \
      -DENABLE_AUTO_BATCH=OFF \
      -DENABLE_OV_ONNX_FRONTEND=OFF \
      -DENABLE_OV_PADDLE_FRONTEND=OFF \
      -DENABLE_OV_TF_FRONTEND=OFF \
      -DENABLE_OV_TF_LITE_FRONTEND=OFF \
      -DENABLE_OV_JAX_FRONTEND=OFF \
      -DENABLE_OV_PYTORCH_FRONTEND=OFF \
      -DENABLE_OV_JAX_FRONTEND=OFF \
      -DENABLE_INTEL_CPU=ON \
      -DENABLE_OV_IR_FRONTEND=ON

NOTE: Inference backends located in external repositories can also be used in a static build. Use -DOPENVINO_EXTRA_MODULES= to enable them. OpenVINODeveloperPackage.cmake must not be used to build external plugins, only OPENVINO_EXTRA_MODULES is a working solution.

NOTE: The ENABLE_LTO CMake option can also be passed to enable link time optimizations to reduce the binary size. But such property should also be enabled on the target which links with static OpenVINO libraries via set_target_properties(<target_name> PROPERTIES INTERPROCEDURAL_OPTIMIZATION_RELEASE ON)

Build static OpenVINO libraries

To build OpenVINO Runtime in a static mode, you need to specify the additional CMake option:

sh
cmake -DBUILD_SHARED_LIBS=OFF <all other CMake options> <openvino_sources root>

Then, use the usual CMake 'build' command:

sh
cmake --build . --config Release -j12

NOTE: Do not restrict the build to --target openvino — the install step below expects other components too (such as openvino_c), and fails if they were not built:

text
file INSTALL cannot find ".../bin/intel64/Release/openvino_c.lib": No error.

To build only a subset of targets, disable the components you don't need at configure time (e.g. -DENABLE_JS=OFF), or use a component-scoped install: cmake --install . --component core.

Then, the installation step:

sh
cmake -DCMAKE_INSTALL_PREFIX=<install_root> -P cmake_install.cmake

The OpenVINO runtime is located in /runtime/lib

NOTE: Build artifacts default to the source tree, not the build directory. If you build more than one static configuration from the same checkout (e.g. per-device, or /MD vs /MT), pass a separate -DOUTPUT_ROOT= per configuration, or they will clobber each other's generated files.

Once you build static OpenVINO Runtime libraries and install them, you can use one of the two ways to add them to your project:

CMake interface

Just use CMake's find_package as usual and link openvino::runtime:

cmake
find_package(OpenVINO REQUIRED)
target_link_libraries(<application> PRIVATE openvino::runtime)

openvino::runtime transitively adds all other static OpenVINO libraries to a linker command.

Pass libraries to linker directly

If you want to configure your project directly, you need to pass all libraries from /runtime/lib to linker command.

NOTE: Since the proper order of static libraries must be used (dependent library should come before dependency in a linker command), consider using the following compiler specific flags to link static OpenVINO libraries:

Microsoft Visual Studio compiler:

sh
/WHOLEARCHIVE:<ov_library 0> /WHOLEARCHIVE:<ov_library 1> ...

GCC like compiler:

sh
gcc main.cpp -Wl,--whole-archive <all libraries from <root>/runtime/lib> > -Wl,--no-whole-archive -o a.out

Static OpenVINO libraries + Conditional compilation for particular models

OpenVINO Runtime can be compiled for particular models, as shown in the Conditional compilation for particular models guide.
The conditional compilation feature can be paired with static OpenVINO libraries to build even smaller end-user applications in terms of binary size. The following procedure can be used, (based on the detailed Conditional compilation for particular models guide):

  • Build OpenVINO Runtime as usual with the CMake option of -DSELECTIVE_BUILD=COLLECT.
  • Run target applications on target models and target platforms to collect traces.
  • Build the final OpenVINO static Runtime with -DSELECTIVE_BUILD=ON -DSELECTIVE_BUILD_STAT=/path/*.csv -DBUILD_SHARED_LIBS=OFF

Building with static MSVC Runtime

In order to build with static MSVC runtime, use the special OpenVINO toolchain file:

sh
cmake -DCMAKE_TOOLCHAIN_FILE=<openvino source dir>/cmake/toolchains/mt.runtime.win32.toolchain.cmake <other options>

NOTE: all other dependent application and libraries must be built with the same mt.runtime.win32.toolchain.cmake toolchain to have conformed values of the MSVC_RUNTIME_LIBRARY target property.

Building static OpenVINO for a specific device

The general instructions above apply to every device. The notes below cover additional, device-specific behavior of -DBUILD_SHARED_LIBS=OFF that is not obvious from CMake errors alone.

CPU

No additional options are required; CPU builds statically the same way as the common runtime described above.

GPU

GPU builds statically with the default -DGPU_RT_TYPE=OCL runtime, no extra flags needed.

NOTE: OV_GPU_WITH_SYCL (SYCL-enabled GPU runtime) is not compatible with -DENABLE_LTO=ON and fails CMake configuration with FATAL_ERROR: Intel GPU plugin with SYCL is not supported with ENABLE_LTO=ON. If SYCL is enabled for GPU, do not enable ENABLE_LTO.

NPU

No additional options are required. ENABLE_INTEL_NPU_INTERNAL, which controls internal NPU tooling such as compile_tool, follows BUILD_SHARED_LIBS and is disabled by default in a static build, so the install manifest does not expect those tools (cmake/features.cmake). If you explicitly re-enable it with -DENABLE_INTEL_NPU_INTERNAL=ON, build the corresponding targets as well (or build the default target as described above), otherwise the install step will fail looking for binaries that were never built.

The NPU offline compiler is a prebuilt shared library and is not available in static builds: ENABLE_INTEL_NPU_COMPILER defaults to ${BUILD_SHARED_LIBS} and cannot be forced ON for a static build. When it's unavailable, CompilerAdapterFactory falls back to the driver compiler (NPU_COMPILER_TYPE=DRIVER) instead of the plugin compiler (NPU_COMPILER_TYPE=PLUGIN) used by default in shared builds — see compiler_adapter_factory.cpp.

Building multiple device configurations from one checkout

This only applies if you deliberately build separate static packages per device (for example, to validate each device independently, or to ship a smaller single-device binary) — not to a single build with several devices enabled together. Give each configuration its own OUTPUT_ROOT and install prefix (see the note above):

sh
cmake -S . -B build_cpu -DCMAKE_BUILD_TYPE=Release -DBUILD_SHARED_LIBS=OFF -DOUTPUT_ROOT=$PWD/build_cpu
cmake --build build_cpu --config Release -j12
cmake --install build_cpu --config Release --prefix install_cpu

cmake -S . -B build_npu -DCMAKE_BUILD_TYPE=Release -DBUILD_SHARED_LIBS=OFF -DOUTPUT_ROOT=$PWD/build_npu
cmake --build build_npu --config Release -j12
cmake --install build_npu --config Release --prefix install_npu

Validating a static build

The existing OpenVINO samples can be used to validate a static build:

  • hello_query_device reports the devices and plugins compiled into the binary; use it to confirm that the plugins you enabled (for example CPU, GPU, NPU) are present and loadable.
  • hello_classification (or benchmark_app) loads a model and runs inference on a selected device, verifying that the statically-linked plugin and frontend also work correctly at run time, not just that they link successfully.
sh
cmake -S samples/cpp -B samples_build -DOpenVINO_DIR=<install_root>/runtime/cmake
cmake --build samples_build --config Release
samples_build/intel64/Release/hello_query_device
samples_build/intel64/Release/hello_classification <model.xml> <image> <DEVICE>

These samples link against the static package the same way any consuming application would, following the steps described in Link static OpenVINO Runtime.

Limitations

  • Device support in a static build:

    Device Builds statically Notes
    CPU Yes
    GPU Yes See GPU notes above
    NPU Yes See NPU notes above
    MULTI, HETERO, AUTO, BATCH Yes
    IR, ONNX, PDPD, TF, TF Lite frontends Yes

  • Static build support means building static libraries only for OpenVINO Runtime libraries. Third-party prebuilt dependencies keep their original format — in particular TBB stays a shared library: it is not rebuilt as part of a static OpenVINO build, and the prebuilt package is copied to /runtime/3rdparty/tbb/bin, so it must be available to your application at run time (on PATH on Windows, or LD_LIBRARY_PATH on Linux).

    NOTE: The prebuilt TBB package can also be downloaded directly:

    • Linux x64: https://storage.openvinotoolkit.org/dependencies/thirdparty/linux/oneapi-tbb-2021.13.1-lin-release.tgz
    • Windows x64: https://storage.openvinotoolkit.org/dependencies/thirdparty/windows/oneapi-tbb-2021.13.3-vs2022-win.zip

    The TBB version differs per OS and changes between OpenVINO releases; check cmake/dependencies.cmake for the version matching your checkout before downloading.

Using a custom TBB build

To build against your own TBB instead of the prebuilt package, follow the oneTBB installation instructions and set TBBROOT to the installation directory before configuring OpenVINO.

If you build that oneTBB as a static library, read Static Linking of oneTBB first. It is not a recommended configuration: it is safe only if exactly one copy of oneTBB ends up in the process, and it disables features that need run-time dynamic loading (tbbbind topology constraints, tbbmalloc_proxy, TCM).

NOTE: You may notice the ENABLE_TBBBIND_2_5 option is OFF on Windows x64 in a static build. This is harmless — the option only matters for old TBB packages that lack a dynamic tbbbind_2_5 library. The prebuilt oneTBB ships one, so NUMA and hybrid-core detection work regardless.

See also

Dev Test Coverage (`docs/dev/test_coverage.md`)

OpenVINO coverage report build

The coverage report is generated using Lcov tool and based on profile data generated by GCC.
The generated reports are in HTML form and located in <openvino_build>/coverage. The reports are generated for the following components:

  1. inference - Inference component is a part of OpenVINO Runtime library.
  2. core - Core component is a part of OpenVINO Runtime library.
  3. low_precision_transformations - Low Precision transformations is a part of OpenVINO Runtime library.
  4. openvino_transformations - Common transformations is a part of OpenVINO Runtime library.
  5. snippets - OpenVINO snippets.
  6. OpenVINO open-source plugins:
    • hetero_plugin - Heterogeneous plugin.
    • auto_plugin - Auto plugin.
    • proxy_plugin - Proxy plugin.
    • auto_batch_plugin - Auto-batching plugin.
    • template_plugin - Template plugin.
    • intel_gpu_plugin - GPU plugin.
    • intel_cpu_plugin - CPU plugin.
    • intel_npu_plugin - Intel® NPU plugin.

Build with profiling data support

To build coverage report, compile OpenVINO with an additional CMake option -DENABLE_COVERAGE=ON:

bash
$ cmake -DENABLE_COVERAGE=ON .

And build OpenVINO as usual.

Generate coverage report

In order to generate coverage reports, first of all, the tests must be run. Depending on how many tests are run, the better coverage percentage can be achieved. E.g. for openvino component, InferenceEngineUnitTests, ieUnitTests, ieFuncTests must be run as well as plugin tests.

bash
$ ctest -V

OpenVINO components define several common groups which allow to run tests for separate component (with ctest argiment -L GROUP_NAME):

  • OV - core OpenVINO tests
  • IR_FE - IR frontend tests
  • ONNX_FE - ONNX frontend tests
  • PADDLE_FE - Paddle frontend tests
  • TF_FE - TensorFlow frontend tests
  • TFL_FE - TensorFlow Lite frontend tests
  • CPU - CPU plugin tests
  • GPU - GPU plugin tests
  • AUTO - Auto plugin tests
  • MULTI - Multi-device tests covered by the Auto plugin
  • AUTO_BATCH - Auto-batching plugin tests
  • PROXY - Proxy plugin tests
  • NPUW - Intel® NPU unit tests

After sufficient number of tests are executed, the coverage numbers can be calculated. In order to do this, run:

bash
$ make ov_coverage

The following tree of reports are generated:

bash
$ find coverage -maxdepth 2 -name index.html
coverage/auto_batch_plugin/index.html
coverage/auto_plugin/index.html
coverage/core/index.html
coverage/transformations/index.html
coverage/paddle_frontend/index.html
coverage/tf_frontend/index.html
coverage/openvino/index.html
coverage/onnx_frontend/index.html
coverage/ir_frontend/index.html
coverage/low_precision_transformations/index.html
coverage/proxy_plugin/index.html
coverage/template_plugin/index.html
coverage/inference/index.html
coverage/frontend_common/index.html
coverage/hetero_plugin/index.html
coverage/intel_cpu_plugin/index.html
coverage/intel_gpu_plugin/index.html
coverage/intel_npu_plugin/index.html

See also

Snippets CMakeLists (`docs/snippets/CMakeLists.txt`)

SPDX-License-Identifier: Apache-2.0

set(TARGET_NAME openvino_docs_snippets)

if(CMAKE_COMPILER_IS_GNUCXX OR OV_COMPILER_IS_CLANG OR OV_COMPILER_IS_INTEL_LLVM)
ov_add_compiler_flags(-Wall)
ov_add_compiler_flags(-Wno-unused-variable)
endif()
if(UNUSED_BUT_SET_VARIABLE_SUPPORTED)
ov_add_compiler_flags(-Wno-unused-but-set-variable)
endif()

file(GLOB SOURCES "${CMAKE_CURRENT_SOURCE_DIR}/.cpp"
"${CMAKE_CURRENT_SOURCE_DIR}/src/.cpp"
"${CMAKE_CURRENT_SOURCE_DIR}/src/.c"
"${CMAKE_CURRENT_SOURCE_DIR}/../articles_en/assets/snippets/.cpp"
"${CMAKE_CURRENT_SOURCE_DIR}/../articles_en/assets/snippets/*.c")

file(GLOB GPU_SOURCES "${CMAKE_CURRENT_SOURCE_DIR}/../articles_en/assets/snippets/gpu/*.cpp")

add GPU snippets if OpenCL has been found

if(TARGET OpenCL::OpenCL)
list(APPEND SOURCES ${GPU_SOURCES})
endif()

try to find VA libraries

find_package(PkgConfig QUIET)
if(PkgConfig_FOUND)
pkg_search_module(libva QUIET IMPORTED_TARGET libva)
if(libva_FOUND)
message(STATUS "${PKG_CONFIG_EXECUTABLE}: libva (${libva_VERSION}) is found at ${libva_PREFIX}")
endif()
endif()

find_package(OpenCV QUIET COMPONENTS core imgcodecs)
if(NOT OpenCV_FOUND OR NOT OpenCV_VERSION VERSION_GREATER_EQUAL 3)
list(REMOVE_ITEM SOURCES "${CMAKE_CURRENT_SOURCE_DIR}/../articles_en/assets/snippets/ShapeInference.cpp")
endif()

requires mfxFrameSurface1 and MSS API

list(REMOVE_ITEM SOURCES "${CMAKE_CURRENT_SOURCE_DIR}/dldt_optimization_guide2.cpp"
"${CMAKE_CURRENT_SOURCE_DIR}/dldt_optimization_guide3.cpp"
"${CMAKE_CURRENT_SOURCE_DIR}/dldt_optimization_guide4.cpp")

create a static library

add_library(${TARGET_NAME} STATIC ${SOURCES})
target_include_directories(${TARGET_NAME} PRIVATE "${OpenVINO_SOURCE_DIR}/src/inference/include"
"${OpenVINO_SOURCE_DIR}/src/inference/dev_api"
"${OpenVINO_SOURCE_DIR}/src/core/dev_api"
"${OpenVINO_SOURCE_DIR}/src/core/include"
"${OpenVINO_SOURCE_DIR}/src/bindings/c/include"
"${OpenVINO_SOURCE_DIR}/src/common/transformations/include"
"${OpenVINO_SOURCE_DIR}/src/common/preprocessing/src"
"${OpenVINO_SOURCE_DIR}/src/common/util/include"
"${OpenVINO_SOURCE_DIR}/src/common/low_precision_transformations/include"
"${OpenVINO_SOURCE_DIR}/src/frontends/common/include"
"${OpenVINO_SOURCE_DIR}/src/core/template_extension"
"${OpenVINO_SOURCE_DIR}/src/frontends/onnx/frontend/include"
"${OpenVINO_SOURCE_DIR}/src/frontends/tensorflow/include"
"${OpenVINO_SOURCE_DIR}/src/frontends/paddle/include")

ov_mark_target_as_cc(${TARGET_NAME})

if(TARGET OpenCL::OpenCL)
target_link_libraries(${TARGET_NAME} PRIVATE OpenCL::OpenCL)

text
if(libva_FOUND)
    target_compile_definitions(${TARGET_NAME} PRIVATE ENABLE_LIBVA)
    target_link_libraries(${TARGET_NAME} PRIVATE PkgConfig::libva)
endif()

if(WIN32)
    target_compile_definitions(${TARGET_NAME} PRIVATE ENABLE_DX11)
    target_link_libraries(${TARGET_NAME} PRIVATE d3d11 dxgi)
endif()

endif()

if(OpenCV_FOUND)
target_include_directories(${TARGET_NAME} SYSTEM PRIVATE ${OpenCV_INCLUDE_DIRS})
target_link_libraries(${TARGET_NAME} PRIVATE opencv_core)
endif()

ov_ncc_naming_style(FOR_TARGET "${TARGET_NAME}"

SOURCE_DIRECTORIES "${CMAKE_CURRENT_SOURCE_DIR}"

ADDITIONAL_INCLUDE_DIRECTORIES

$)

Detect OpenVINO

find_package(OpenVINO QUIET
PATHS "${CMAKE_BINARY_DIR}"
NO_DEFAULT_PATH)
if(NOT OpenVINO_FOUND)
set(OpenVINO_DIR ${CMAKE_BINARY_DIR})
endif()

Example

set(TARGET_NAME "ov_integration_snippet")

[cmake:integration_example_cpp]

cmake_minimum_required(VERSION 3.10)
set(CMAKE_CXX_STANDARD 17)

find_package(OpenVINO REQUIRED)

add_executable(${TARGET_NAME} src/main.cpp)

target_link_libraries(${TARGET_NAME} PRIVATE openvino::runtime)

[cmake:integration_example_cpp]

set(TARGET_NAME_PY "ov_integration_snippet_py")

[cmake:integration_example_cpp_py]

cmake_minimum_required(VERSION 3.10)
set(CMAKE_CXX_STANDARD 17)

if(NOT CMAKE_CROSSCOMPILING)
find_package(Python3 QUIET COMPONENTS Interpreter)
if(Python3_Interpreter_FOUND)
execute_process(
COMMAND ${Python3_EXECUTABLE} -c "from openvino.utils import get_cmake_path; print(get_cmake_path(), end='')"
OUTPUT_VARIABLE OpenVINO_DIR_PY
ERROR_QUIET)
endif()
endif()

find_package(OpenVINO REQUIRED PATHS "${OpenVINO_DIR_PY}")

add_executable(${TARGET_NAME_PY} src/main.cpp)

target_link_libraries(${TARGET_NAME_PY} PRIVATE openvino::runtime)

[cmake:integration_example_cpp_py]

set(TARGET_NAME_C "ov_integration_snippet_c")

[cmake:integration_example_c]

cmake_minimum_required(VERSION 3.10)
set(CMAKE_CXX_STANDARD 17)

find_package(OpenVINO REQUIRED)

add_executable(${TARGET_NAME_C} src/main.c)

target_link_libraries(${TARGET_NAME_C} PRIVATE openvino::runtime::c)

[cmake:integration_example_c]

Sphinx Setup Index (`docs/sphinx_setup/index.rst`)

============================ OpenVINO 2026.3

.. container::
:name: ov-homepage-banner

.. raw:: html

text
<link rel="stylesheet" type="text/css" href="_static/css/homepage_style.css">
  
     <section class="splide" aria-label="Splide Banner Carousel">
       
     		
           <li id="ov-homepage-slide2" class="splide__slide">
              OpenVINO GenAI
              Simplify GenAI model deployment!
              <a class="ov-homepage-banner-btn" href="https://docs.openvino.ai/2026/openvino-workflow-generative/inference-with-genai.html">Check out our guide</a>
              </li>
              <li id="ov-homepage-slide1" class="splide__slide">
              OpenVINO models on Hugging Face!
              Get pre-optimized OpenVINO models, no need to convert!
              <a class="ov-homepage-banner-btn" href="https://huggingface.co/OpenVINO">Visit Hugging Face</a>
              </li>
              <li id="ov-homepage-slide3" class="splide__slide">
              OpenVINO Model Hub
              See performance benchmarks for top AI models!
              <a class="ov-homepage-banner-btn" href="https://www.intel.com/content/www/us/en/developer/tools/openvino-toolkit/model-hub.html">Explore now</a>
              </li>
              <li id="ov-homepage-slide4" class="splide__slide">
              OpenVINO via PyTorch 2.0 torch.compile()
              Use OpenVINO directly in PyTorch-native applications!
              <a class="ov-homepage-banner-btn" href="https://docs.openvino.ai/2026/openvino-workflow/torch-compile.html">Learn more</a>
              </li>
        
       
     </section>

|

OpenVINO is an open-source toolkit for deploying high-performance AI solutions across
cloud, AI PCs, edge devices, and Physical AI alike. Develop your applications
with both generative and conventional AI models, coming from the most popular model frameworks.
Convert, optimize, and run inference utilizing the full potential of Intel® hardware.
There are four main tools in OpenVINO to meet all your deployment needs:

.. grid:: 1 2 2 4

.. grid-item-card:: OpenVINO Base Package
:link: ./openvino-workflow.html

text
Run and deploy conventional AI models

.. grid-item-card:: OpenVINO GenAI
:link: ./openvino-workflow-generative.html

text
Run and deploy generative AI models

.. grid-item-card:: OpenVINO Physical AI
:link: ./physical-ai.html

text
Deploy VLA models on robots

.. grid-item-card:: OpenVINO Model Server
:link: ./model-server/ovms_what_is_openvino_model_server.html

text
Deploy both generative and conventional AI inference on a server

|
| For a quick ramp-up, check out the
OpenVINO Toolkit Cheat Sheet [PDF] <https://docs.openvino.ai/2026/_static/download/OpenVINO_Quick_Start_Guide.pdf>__
and the
OpenVINO GenAI Quick-start Guide [PDF] <https://docs.openvino.ai/2026/_static/download/GenAI_Quick_Start_Guide.pdf>__
|

.. image:: ./assets/images/openvino-overview-diagram.jpg
:align: center
:alt: openvino diagram
:width: 90%

|

Where to Begin
++++++++++++++++++++++++++++

.. grid:: 2 2 3 3
:class-container: ov-homepage-higlight-grid

.. grid-item-card:: Installation
:img-top: ./assets/images/home_begin_tile_01.png
:class-card: homepage_begin_tile
:shadow: none

text
This guide introduces installation and learning materials for Intel® Distribution of OpenVINO™ toolkit.

  .. button-link:: get-started/install-openvino.html
     :color: primary
     :outline:

     Get Started

.. grid-item-card:: Performance Benchmarks
:img-top: ./assets/images/home_begin_tile_02.png
:class-card: homepage_begin_tile
:shadow: none

text
See latest benchmark numbers for OpenVINO and OpenVINO Model Server.

  .. button-link:: about-openvino/performance-benchmarks.html
     :color: primary
     :outline:

     View data

.. grid-item-card:: Framework Compatibility
:img-top: ./assets/images/home_begin_tile_03.png
:class-card: homepage_begin_tile
:shadow: none

text
Load models directly (for TensorFlow, ONNX, PaddlePaddle) or convert to OpenVINO format.

  .. button-link:: openvino-workflow/model-preparation.html
     :color: primary
     :outline:

     Load your model

.. grid-item-card:: Easy Deployment
:img-top: ./assets/images/home_begin_tile_04.png
:class-card: homepage_begin_tile
:shadow: none

text
Get started in just a few lines of code.

  .. button-link:: openvino-workflow/running-inference.html
     :color: primary
     :outline:

     Run Inference

.. grid-item-card:: Serving at scale
:img-top: ./assets/images/home_begin_tile_05.png
:class-card: homepage_begin_tile
:shadow: none

text
Cloud-ready deployments for microservice applications.

  .. button-link:: model-server/ovms_what_is_openvino_model_server.html
     :color: primary
     :outline:

     Check out Model Server

.. grid-item-card:: Model Compression
:img-top: ./assets/images/home_begin_tile_06.png
:class-card: homepage_begin_tile
:shadow: none

text
Reach for performance with post-training and training-time compression with NNCF.

  .. button-link:: openvino-workflow/model-optimization.html
     :color: primary
     :outline:

     Optimize now

|

Key Features
++++++++++++++++++++++++++++

.. button-link:: about-openvino/key-features.html
:color: primary
:outline:
:align: right
:class: key-feat-btn

See all features

.. grid:: 2 2 2 2
:class-container: homepage_begin_container

.. grid-item-card:: Model Compression
:img-top: ./assets/images/home_key_feature_01.png
:class-card: homepage_begin_key
:shadow: none

text
You can either link directly with OpenVINO Runtime to run inference locally or use OpenVINO Model Server to serve model inference from a separate server or within Kubernetes environment.

.. grid-item-card:: Fast & Scalable Deployment
:img-top: ./assets/images/home_key_feature_02.png
:class-card: homepage_begin_key
:shadow: none

text
Write an application once, deploy it anywhere, achieving maximum performance from hardware. Automatic device discovery allows for superior deployment flexibility. OpenVINO Runtime supports Linux, Windows and MacOS and provides Python, C++ and C API. Use your preferred language and OS.

.. grid-item-card:: Lighter Deployment
:img-top: ./assets/images/home_key_feature_03.png
:class-card: homepage_begin_key
:shadow: none

text
Designed with minimal external dependencies reduces the application footprint, simplifying installation and dependency management. Popular package managers enable application dependencies to be easily installed and upgraded. Custom compilation for your specific model(s) further reduces final binary size.

.. grid-item-card:: Enhanced App Start-Up Time
:img-top: ./assets/images/home_key_feature_04.png
:class-card: homepage_begin_key
:shadow: none

text
In applications where fast start-up is required, OpenVINO significantly reduces first-inference latency by using the CPU for initial inference and then switching to another device once the model has been compiled and loaded to memory. Compiled models are cached, improving start-up time even more.

.. toctree::
:maxdepth: 2
:hidden:

GET STARTED
HOW TO USE - GENERATIVE AI WORKFLOW
HOW TO USE - CONVENTIONAL AI WORKFLOW
HOW TO USE - MODEL SERVING <model-server/ovms_what_is_openvino_model_server>
HOW TO USE - PHYSICAL AI
REFERENCE DOCUMENTATION
ABOUT OPENVINO

Src Docs Architecture (`src/docs/architecture.md`)

OpenVINO Architecture

This guide encompasses existed architectural ideas and guidelines for architecture development.

Main architectual design concepts

OpenVINO development is based on next principals:

  • Performance and Scalability - OpenVINO should provide the optimal performance in different customer cases
  • Availability and Resilience - OpenVINO is high quality product and we should demonstrate this high level in different cases
  • Security - OpenVINO is applied in different applications, it is why we have a strict validation pipelines which allow to find vulnerabilities on the development stage.

Each OpenVINO component is projected with using DOTADIW (Do One Thing And Do It Well) approach, it means that OpenVINO is highly granularity product and we are avoiding mixing of responsibilities for different components.

OpenVINO Component responsibilities

mermaid
flowchart TB
    subgraph tools [Tools]
        ovc{{OpenVINO model converter}}
    
        style ovc fill:#6c9f7f
    end
    subgraph tutorials [Tutorials]
        samples[Samples]
        demos[Demos]
        notebooks[Notebooks]
    end
    subgraph api [API]
        cpp[C++]
        subgraph bindings [Bindings]
            c_api[C]
            python_api[Python]
        end
    end
    
    subgraph plugins [OV Plugins]
        auto(["AUTO"])
        cpu(["Intel_CPU"])
        gpu(["Intel_GPU"])
    end
    subgraph frontends [OV Frontends]
        direction TB
        ir_fe["IR Frontend"]
        onnx_fe["ONNX Frontend"]
        paddle_fe["Paddle Frontend"]
    end
    openvino(openvino library)
    
    frontends--Read model--->openvino
    openvino--API--->api
    openvino--infer--->plugins

All OpenVINO components can be logicaly divided to several groups:

  • OpenVINO library is a core library which provides the OpenVINO API.
  • Bindings are libraries which provide OpenVINO API for different languages.
  • Frontends are libraries which converts model from framework representation to OpenVINO format.
  • Plugins are components which allow to run models on different accelerators.
  • Tools a set of different components which provide additional opportunities for OpenVINO.
  • Tutorials different applications which show how user can work with OpenVINO.

OpenVINO Inference pipeline

OpenVINO inference pipeline has several objects which are responsible for different stages of model inference.

  • Core object is a global context for Model inference, allows to create the unique environment for inference pipeline.
    • read_model() finds the right frontend and converts Framework model to OpenVINO representation.
    • compile_model() loads hardware plugin and compiles model on the device.
  • CompiledModel represents the hardware specified model, this model can contain additional plugin optimization and on this stage model already was compiled for the specific device.
  • InferRequest provides an interface for running inference.
mermaid
flowchart TB
    subgraph core [ov::Core core]
        read_model["core.read_model()"]
        compile_model["core.compile_model()"]
    end
    fw_model[(Framework model)]
    ov_model[ov::Model]
    
    subgraph compiled_models [Compiled models]
        cpu_model["CPU"]
        gpu_model["GPU"]
        auto_model["AUTO"]
    end
    subgraph infer_reguests [InferRequests]
        cpu_req["CPU"]
        gpu_req["GPU"]
        auto_req["AUTO"]
    end
   
    result[Results]
    
    fw_model--->read_model
    read_model--->ov_model
    ov_model-->compile_model
    
    fw_model--->compile_model
    
    compile_model--"compile_model(CPU)"--->cpu_model
    compile_model--"compile_model(GPU)"--->gpu_model
    compile_model--"compile_model(AUTO)"--->auto_model
    
    cpu_model--"create_infer_request()"--->cpu_req
    gpu_model--"create_infer_request()"--->gpu_req
    auto_model--"create_infer_request()"--->auto_req
    
    cpu_req--"infer()"-->result
    gpu_req--"infer()"-->result
    auto_req--"infer()"-->result

OpenVINO Components Relationships

The diagram below shows internal components and their relationships inside the OpenVINO dynamic library.

mermaid
flowchart LR
    subgraph legend [Legend]
    interface_lib[Interface library]
    style interface_lib fill:#af8401

    object_lib[Object library]
    style object_lib fill:#6e9f01

    static_lib[(Static library)]
    style static_lib fill:#01429f
    dynamic_lib([Dynamic library])
    style dynamic_lib fill:#cb6969

    some_api{{Some library API}}
    some_dev_api{{Some developer API}}

    object_lib ===>|Link with| interface_lib

    some_api --->|Depends on headers only| static_lib
    some_api -.->|Include headers| some_dev_api
    end
text
/* Detailed source-code truncated for AI context efficiency. */

See also

Src Docs Performance Analysis ITT Counters (`src/docs/performance_analysis_ITT_counters.md`)

Performance Analysis Using ITT Counters

Contents

Introduction

OpenVINO has a powerful capabilities for performance analysis of the key stages, such as read time and load time. Most of the modules and features have been tagged with Intel ITT counters, which allows us to measure the performance of these components.

Performance analysis

For performance analysis, follow the steps below:

  1. Run the CMake tool with the following option: -DENABLE_PROFILING_ITT=FULL and build OpenVINO.

  2. Choose the tool for statistics collection using ITT counters.

    1. Intel Vtune Profiler

  3. Run an OpenVINO project with the performance analysis tool.

Intel Vtune Profiler

Example of running the tool:

sh
vtune -collect hotspots -k sampling-mode=hw -k enable-stack-collection=true -k stack-size=0 -k sampling-interval=0.5 -- ./benchmark_app -nthreads=1 -api sync -niter 1 -nireq 1 -m ./resnet-50-pytorch/resnet-50-pytorch.xml

Mandatory parameters:

  • -collect hotspots

Generated artifacts:

r000hs
Generated file can be opened with Vtune client.

Adding new ITT counters

Use API defined in openvino/itt module.

See also

Articles En About Openvino Contributing (`docs/articles_en/about-openvino/contributing.rst`)

Contribute to OpenVINO

.. toctree::
:maxdepth: 1
:hidden:

contributing/code-contribution-guide

OpenVINO™ is always looking for opportunities to improve and your contributions
play a big role in this process. Here are four ways you can make OpenVINO better:

  • Provide feedback <#provide-feedback>__
  • Contribute code changes <#contribute-code-changes>__
  • Improve documentation <#improve-documentation>__
  • Promote and support OpenVINO <#promote-and-support-openvino>__

:fas:comments Provide feedback
################################

.. rubric:: Report bugs / issues
:name: report-bugs-issues

If you notice unexpected behavior in OpenVINO or its components, you can
create a new issue <https://github.com/openvinotoolkit/openvino/issues>__
in the GitHub issue tracker.

.. rubric:: Propose improvements
:name: propose-improvements

If you want to share your ideas for improving OpenVINO:

  • Open a new GitHub Discussion <https://github.com/openvinotoolkit/openvino/discussions>__.
  • Create a Feature Request Issue <https://github.com/openvinotoolkit/openvino/issues/new?assignees=octocat&labels=enhancement%2Cfeature&projects=&template=feature_request.yml&title=%5BFeature+Request%5D%3A+>__
    if your idea is already well defined.

In both cases, provide a detailed description and list potential use cases,
benefits, and challenges. Keep in mind that even if your input is not immediately
prioritized, it may be used at a later or undertaken by the community.

:fas:code-branch Contribute code changes
##########################################

Always check if the change is still needed! Verify if
the issue <https://github.com/openvinotoolkit/openvino/issues>__ or
request <https://github.com/openvinotoolkit/openvino/pulls>__ is still open
and nobody has started working on it. If the ticket is already work in progress,
you can always ask if you can help.

Address only the issues that affect the master or
:doc:LTS release branches <./release-notes-openvino/release-policy>.

Do not start work on contributions, if a proper issue/ request has not been created.

.. tip::

If you want to start with something simple, check out
first-time contributions <https://github.com/orgs/openvinotoolkit/projects/3>__.

.. rubric:: Fix bugs
:name: fix-bugs

Choose one of the issues reported in
GitHub Issue Tracker <https://github.com/openvinotoolkit/openvino/issues>__ and
create a Pull Request <https://github.com/openvinotoolkit/openvino/blob/master/CONTRIBUTING_PR.md>__
(PR) addressing it.

If you find a new bug and want to fix it, you should still
create a new issue before working on the PR. This way, it will be easier for other
developers to track changes.

.. rubric:: Develop new features
:name: develop-new-features

If you find a Feature Request <https://github.com/openvinotoolkit/openvino/issues/new?assignees=octocat&labels=enhancement%2Cfeature&projects=&template=feature_request.yml&title=%5BFeature+Request%5D%3A+>__
you want to work on, make sure it is clearly defined. If you have any doubts,
or the change is complex, discuss it <https://github.com/openvinotoolkit/openvino/discussions>__
with OpenVINO developers first.

If you have an idea for a new feature and want
to develop it, you should still create a Feature Request before working on the
PR. This way, it will be easier for other developers to track changes.

.. rubric:: Develop a new device plugin
:name: develop-new-device-plugin

If you want to run inference on a device that is currently not supported, you
can see how to develop a new plugin for it in the
Plugin Developer Guide <https://docs.openvino.ai/2026/documentation/openvino-extensibility/openvino-plugin-library.html>__.

:fas:file-alt Improve documentation
#####################################

OpenVINO user documentation is built from several sources, mainly the files in
the docs/articles_en <https://github.com/openvinotoolkit/openvino/tree/master/docs/articles_en>__
folder, using Sphinx <https://www.sphinx-doc.org/>__ and the
reStructuredText <https://www.sphinx-doc.org/en/master/usage/restructuredtext/index.html>__
markup language.

OpenVINO developer documentation <https://github.com/openvinotoolkit/openvino/tree/master/docs/dev>__
is available only in markdown in the docs/dev <https://github.com/openvinotoolkit/openvino/tree/master/docs/dev>__
folder.

To edit docs, consider using the Editor’s
guide <https://github.com/openvinotoolkit/openvino/blob/master/CONTRIBUTING_DOCS.md>__
and contacting documentation maintainers <https://github.com/orgs/openvinotoolkit/teams/openvino-docs-maintainers>__,
who will help you with information architecture and formatting, as well as
review, adjust, and merge the PR.

.. rubric:: Review user documentation
:name: review-user-documentation

In most cases, creating a PR is enough to correct a documentation mistake, improve
the language, and update or extend the information. For your convenience, the
top-right panel of most pages includes the “Edit on GitHub” button that will
take you to the source file of the given article.

.. rubric:: Write new content
:name: write-new-content

For more extensive changes in docs, reach out to any of the
documentation maintainers <https://github.com/orgs/openvinotoolkit/teams/openvino-docs-maintainers>__
to discuss the new content.

:fas:bullhorn Promote and support OpenVINO
############################################

.. rubric:: Popularize OpenVINO
:name: popularize-openvino

Articles, tutorials, blog posts, demos, videos, and any other involvement in the
OpenVINO community is more than welcome. If you discuss or present OpenVINO on
various social platforms, you are raising awareness of the product among AI
enthusiasts and enabling other people to discover the toolkit.

Feel free to reach out to OpenVINO developers if you need help with making a
contribution. You can also contact
documentation maintainers <https://github.com/orgs/openvinotoolkit/teams/openvino-docs-maintainers>__
, if you need help with visuals, brand materials, or content creation in general.

.. rubric:: Help other community members
:name: help-community

If you are an experienced OpenVINO user and want to help, you can share your
expertise with the community at any time. Check GitHub
Discussions <https://github.com/openvinotoolkit/openvino/discussions>__
and Issues <https://github.com/openvinotoolkit/openvino/issues>__ to see if
you can help someone.

.. note::

By contributing to the OpenVINO project, you agree that your contributions
will be licensed under the terms of the OpenVINO repository <https://github.com/openvinotoolkit/openvino/blob/master/LICENSE>__.

Additional Resources
#####################

  • :doc:Code Contribution Guide <./contributing/code-contribution-guide>
  • Choose a "Good First Issue" <https://github.com/orgs/openvinotoolkit/projects/3>__.
  • Learn more about OpenVINO architecture <https://github.com/openvinotoolkit/openvino/blob/master/src/docs/architecture.md>__.
  • Check out a blog post on contributing to OpenVINO <https://medium.com/openvino-toolkit/how-to-contribute-to-an-ai-open-source-project-c741f48e009e>__.
  • Visit Intel DevHub Discord server <https://discord.gg/7pVRxUwdWG>__ to join
    discussions and talk to OpenVINO developers.

Articles En About Openvino Key Features (`docs/articles_en/about-openvino/key-features.rst`)

Key Features

Easy Integration
#########################

| :doc:Support for multiple frameworks <../openvino-workflow/model-preparation/convert-model-to-ir>
| Use deep learning models from PyTorch, TensorFlow, TensorFlow Lite, PaddlePaddle, and ONNX
directly or convert them to the optimized OpenVINO IR format for improved performance.

| :doc:Close integration with PyTorch <../openvino-workflow/torch-compile>
| For PyTorch-based applications, specify OpenVINO as a backend using
:doc:torch.compile <../openvino-workflow/torch-compile> to improve model inference. Apply
OpenVINO optimizations to your PyTorch models directly with a single line of code.

| :doc:GenAI Out Of The Box <../openvino-workflow-generative/inference-with-genai>
| With the OpenVINO GenAI, you can run generative models with just a few lines of code.
Check out the GenAI guide for instructions on how to do it.

| Python / C++ / C / NodeJS APIs <https://docs.openvino.ai/2026/api/api_reference.html>__
| OpenVINO offers the C++ API as a complete set of available methods. For less resource-critical
solutions, the Python API provides almost full coverage, while C and NodeJS ones are limited
to the methods most basic for their typical environments. The NodeJS API, is still in its
early and active development.

| :doc:Open source and easy to extend <../about-openvino/contributing>
| If you need a particular feature or inference accelerator to be supported, you are free to file
a feature request or develop new components specific to your projects yourself. As open source,
OpenVINO may be used and modified freely. See the extensibility guide for more information on
how to adapt it to your needs.

Deployment
#########################

| :doc:Local or remote <../openvino-workflow>
| Integrate the OpenVINO runtime directly with your application to run inference locally or use
OpenVINO Model Server <https://github.com/openvinotoolkit/model_server>__ to shift the inference
workload to a remote system, a separate server or a Kubernetes environment. For serving,
OpenVINO is also integrated with vLLM <https://docs.vllm.ai/en/stable/getting_started/openvino-installation.html>__
and Triton <https://github.com/triton-inference-server/openvino_backend>__ services.

| :doc:Scalable and portable <release-notes-openvino/system-requirements>
| Write an application once, deploy it anywhere, always making the most out of your hardware setup.
The automatic device selection mode gives you the ultimate deployment flexibility on all major
operating systems. Check out system requirements.

| Light-weight
| Designed with minimal external dependencies, OpenVINO does not bloat your application
and simplifies installation and dependency management. The custom compilation for your specific
model(s) may further reduce the final binary size.

Performance
#########################

| :doc:Model Optimization <../openvino-workflow/model-optimization>
| Optimize your deep learning models with NNCF, using various training-time and post-training
compression methods, such as pruning, sparsity, quantization, and weight compression. Make
your models take less space, run faster, and use less resources.

| :doc:Top performance <../about-openvino/performance-benchmarks>
| OpenVINO is optimized to work with Intel hardware, delivering confirmed high performance for
hundreds of models. Explore OpenVINO Performance Benchmarks to discover the optimal hardware
configurations and plan your AI deployment based on verified data.

| :doc:Enhanced App Start-Up Time <../openvino-workflow/running-inference/optimize-inference>
| If you need your application to launch immediately, OpenVINO will reduce first-inference latency,
running inference on CPU until a more suited device is ready to take over. Once a model
is compiled for inference, it is also cached, improving the start-up time even more.

Articles En About Openvino Openvino Ecosystem (`docs/articles_en/about-openvino/openvino-ecosystem.rst`)

OpenVINO Ecosystem

.. meta::
:description: Explore the OpenVINO™ ecosystem of tools and resources for developing deep
learning solutions.

.. toctree::
:maxdepth: 1
:hidden:

OpenVINO Integrations <openvino-ecosystem/openvino-integrations>
OpenVINO Tools <openvino-ecosystem/openvino-project>

OpenVINO™, as a toolkit should, involves multiple components and integrations that may be used
in various areas of your Deep Learning pipelines. This section will give you an overview of a
whole ecosystem of resources either developed under the OpenVINO umbrella, integrating it with
external solutions, or utilizing its potential.

| :doc:OpenVINO Integrations <./openvino-ecosystem/openvino-integrations>
| See what other tools OpenVINO is easily integrated with and how you can benefit from its
performance, without rewriting your software.

| :doc:OpenVINO Tools <./openvino-ecosystem/openvino-project>
| Check out the most noteworthy OpenVINO components.

Articles En About Openvino Performance Benchmarks (`docs/articles_en/about-openvino/performance-benchmarks.rst`)

Performance Benchmarks

.. meta::
:description: Use the benchmark results for Intel® Distribution of OpenVINO™
toolkit, that may help you decide what hardware to use or how
to plan the workload.

.. toctree::
:maxdepth: 1
:hidden:

Efficient LLMs for AI PC
Performance Information F.A.Q.
OpenVINO Accuracy
Getting Performance Numbers

This page presents benchmark results for the
Intel® Distribution of OpenVINO™ toolkit <https://software.intel.com/content/www/us/en/develop/tools/openvino-toolkit.html>__
and :doc:OpenVINO Model Server <../../model-server/ovms_what_is_openvino_model_server>, for
a representative selection of public neural networks and Intel® devices. The results may help
you decide which hardware to use in your applications or plan AI workload for the hardware
you have already implemented in your solutions. Click the buttons below to see the chosen
benchmark data.

.. note::

Benchmark data for the OpenVINO inference engine is available only on the
Model Hub page <https://www.intel.com/content/www/us/en/developer/tools/openvino-toolkit/model-hub.html>__.

.. grid:: 1 1 2 2
:gutter: 4

.. grid-item::

text
.. button-link:: https://www.intel.com/content/www/us/en/developer/tools/openvino-toolkit/model-hub.html
     :class: ov-toolkit-benchmark-results_old
     :color: primary
     :outline:
     :expand:

     :material-regular:`bar_chart;1.4em` OpenVINO Benchmark Graphs (general)

.. grid-item::

text
.. button-link:: #
     :class: ovms-toolkit-benchmark-results
     :color: primary
     :outline:
     :expand:

     :material-regular:`bar_chart;1.4em` OVMS Benchmark Graphs (general)

.. grid-item::

text
.. button-link:: ./performance-benchmarks/generative-ai-performance.html
     :class: ov-toolkit-benchmark-genai
     :color: primary
     :outline:
     :expand:

     :material-regular:`table_view;1.4em` LLM performance for AI PC

.. grid-item::

text
.. button-link:: #
     :class: ovms-toolkit-benchmark-llm-result
     :color: primary
     :outline:
     :expand:

     :material-regular:`bar_chart;1.4em` OVMS for GenAI

Key performance indicators and workload parameters

.. tab-set::

.. tab-item:: Throughput
:sync: throughput

text
For Vision and NLP Models this measures the number of inferences delivered within a latency threshold
  (for example, number of Frames Per Second - FPS).
  For GenAI (or Large Language Models) this measures the token rate after the first token aka. 2nd token
  throughput rate which is presented as tokens/sec. Please click on the "Workload Parameters" tab to
  learn more about input/output token lengths, etc.

.. tab-item:: Latency
:sync: latency

text
For Vision and NLP models this measures the synchronous execution of inference requests and
  is reported in milliseconds. Each inference request (for example: preprocess, infer,
  postprocess) is allowed to complete before the next one starts. This performance metric is
  relevant in usage scenarios where a single image input needs to be acted upon as soon as
  possible. An example would be the healthcare sector where medical personnel only request
  analysis of a single ultra sound scanning image or in real-time or near real-time applications
  such as an industrial robot's response to actions in its environment or obstacle avoidance
  for autonomous vehicles.
  For Transformer models like Stable-Diffusion this measures the time it takes to convert the prompt
  or input text into a finished image. It is presented in seconds.

.. tab-item:: Workload Parameters
:sync: workloadparameters

text
The workload parameters affect the performance results of the different models we use for
  benchmarking. Image processing models have different image size definitions and the
  Natural Language Processing models have different max token list lengths. All these can
  be found in detail in the :doc:`FAQ section `.
  All models are executed using a batch size of 1. Below are the parameters for the GenAI
  models we display.

  * Input tokens: 1024,
  * Output tokens: 128,
  * number of beams: 1

  For text to image:

  * iteration steps: 20,
  * image size (HxW): 256 x 256,
  * input token length: 1024 (the tokens for GenAI models are in English).

Platforms, Configurations, Methodology

To see the methodology used to obtain the numbers and learn how to test performance yourself,
see the guide on :doc:getting performance numbers .

For a listing of all platforms and configurations used for testing, refer to the following:

.. grid:: 1 1 2 2
:gutter: 4

.. grid-item::

text
.. button-link:: ../_static/download/benchmarking_OV_platform_list.pdf
     :color: primary
     :outline:
     :expand:

     :material-regular:`download;1.5em` Click for Hardware Platforms [PDF]

  .. button-link:: ../_static/download/benchmarking_OV_system_info_detailed.xlsx
     :color: primary
     :outline:
     :expand:

     :material-regular:`download;1.5em` Click for Configuration Details [XLSX]

Disclaimers

  • System configurations used for Intel® Distribution of OpenVINO™ toolkit performance results
    are based on release 2026.3, as of Aug 4, 2026.

  • OpenVINO Model Server performance results are based on release 2025.3, as of September 3rd, 2025.

The results may not reflect all publicly available updates. Intel technologies' features and
benefits depend on system configuration and may require enabled hardware, software, or service
activation. Learn more at intel.com, the OEM, or retailer.

See configuration disclosure for details. No product can be absolutely secure.
Performance varies by use, configuration and other factors. Learn more at
www.intel.com/PerformanceIndex <https://www.intel.com/PerformanceIndex>__.
Intel optimizations, for Intel compilers or other products, may not optimize to the same degree
for non-Intel products.

.. raw:: html

.. container:: benchmark-banner

Results may vary. For more information, see
:doc:F.A.Q. <./performance-benchmarks/performance-benchmarks-faq>
See :doc:Legal Information <./additional-resources/terms-of-use>.

Articles En About Openvino Release Notes Openvino (`docs/articles_en/about-openvino/release-notes-openvino.rst`)

============================= OpenVINO Release Notes

.. meta::
:description: See what has changed in OpenVINO with the latest release, as well as all
previous releases in this year's cycle.

.. toctree::
:maxdepth: 1
:hidden:

text
release-notes-openvino/system-requirements
release-notes-openvino/release-policy

2026.3 - 4 August 2026
#############################################################################################

:doc:System Requirements <./release-notes-openvino/system-requirements> | :doc:Release policy <./release-notes-openvino/release-policy> | :doc:Installation Guides <./../get-started/install-openvino>

We are considering transitioning to C++20 as the default standard starting with the 27.0 release. We welcome feedback from our customers and community members on this proposed change. Please share your thoughts and comments on the OpenVINO community forum <https://community.intel.com/t5/Intel-Distribution-of-OpenVINO/bd-p/distribution-openvino-toolkit>__.

What's New
++++++++++

More Gen AI coverage and frameworks integrations to minimize code changes

  • New models supported:

    • On CPU, GPU, and NPU: SmolLM3-3B, LFM2-1.2B, LFM2.5-1.2B

    • On CPU and GPU: Harrier OSS-v1-0.6B, Qwen3-8B with EAGLE-3, MiniCPM5-1B, FLUX.2-klein-4B

    • Extended to GPU and NPU: YOLO26

    • Additional models available as early releases on CPU and GPU: Qwen3-ASR, Qwen3-Omni, Gemma-3n, Qwen3-VL-Embedding-8B, Kokoro-82M

  • Support for Hugging Face Transformers v5.5, ensuring compatibility with the latest model architectures on Hugging Face

Broader LLM model support and more model compression techniques

  • OpenVINO™ GenAI extends the EAGLE-3 speculative decoding pipeline to LLMs and VLMs, enhancing existing continuous batching and adding Top-K sampling to deliver additional token-generation speedups on CPUs, GPUs, and NPUs.

  • Lazy weight loading is enabled for IR and ONNX models to automatically select an optimal loading and compilation path, minimizing peak memory usage during model initialization.

  • Neural Network Compression Framework (NNCF) now supports FP8 quantization for ONNX models, helping developers realize FP8 performance, accuracy, and memory gains while expanding low-precision inference options for production deployments.

  • OpenVINO™ GenAI now supports three additional pipelines: Omni for multimodal workloads, ASR for speech recognition, and Embedding for multimodal embedding generation.

More portability and performance to run AI at the edge, in the cloud or locally

  • Introducing support for Intel® Xeon® 6+ processors (formerly codenamed Clearwater Forest)

  • MoE offloading to disk enabled, allowing 30B MoE models like Qwen3-30B-A3B to run even on devices with 16 GB of memory while maintaining acceptable tokens-per-second (TPS) generation rates.

  • OpenVINO™ Model Server simplifies model deployment and unifies REST API endpoints, reducing command complexity while providing standard v1/chat/completions support for easier integration with other serving frameworks.

  • OpenVINO™ Model Server further improves stability, performance, and accuracy for LLMs such as Qwen3.5/3.6 with linear attention and extends tool-parser support to MiniCPM5-1B and LFM2.5, improving accuracy and reliability.

OpenVINO™ Runtime
+++++++++++++++++

Common

  • Introduced optimized memory management for constant weights when loading and compiling models from IR and ONNX formats on Windows, reducing peak memory usage and RAM pressure to enable inference on memory-constrained systems. Linux support was added in 2026.2.

  • Added official support for Ubuntu 26.04 LTS, including support for building OpenVINO from source with GCC 15.

  • Fixed constant folding for the Loop node, enabling a broader range of ONNX models to compile successfully during model optimization.

  • Resolved an XML parsing issue to safely detect and report malformed XML plugin configuration files, improving robustness and providing clearer diagnostics during plugin initialization and registration.

CPU Device Plugin

  • Enabled tree-mask support in PagedAttention for EAGLE-3 tree sampling, broadening speculative decoding capabilities for LLM inference.

  • Enabled Adaptive R-KV support, improving key-value cache efficiency during LLM inference.

  • Optimized INT8 and INT4 weight-compressed LLM inference with BF16 inference precision, time to first token (TTFT) and time per output token (TPOT) performance.

  • Added BF16 and FP16 support for GatedDeltaNet, enabling JIT kernels with native FP16 and BF16 instructions to improve TTFT.

  • Enabled Convert-Gather fusion for hybrid-precision workloads to reduce memory consumption.

GPU Device Plugin

  • Extended remote tensor support to include NT handles and CPU pointers, broadening interoperability options for GPU memory management.

  • Optimized memory usage during model loading and inference phases, reducing overall memory footprint to enable deployment on more constrained systems.

  • Introduced compiled blob compatibility checks based on OpenVINO version, GPU device properties, and GPU driver, improving reliability and safety of model caching and reuse workflows.

  • Improved inference performance for Qwen3-ASR models on GPU.

  • Introduced Mixture of Experts (MoE) weight offloading to disk, reducing memory requirements for INT4 models through partial weight offloading.

  • Optimized MoE decoding throughput in batched inference scenarios.

NPU Device Plugin

  • Added support for LFM2 in both FP16 and INT4 quantized modes on NPU. To convert an LFM2 model to an NPU-friendly INT4 quantized form, pass the additional --backup-precision int8_sym parameter to optimum-intel to ensure symmetric quantization of the linear part of the attention mechanism.

OpenVINO Python API

  • Updated core dependencies with security and stability fixes.

OpenVINO C API

  • Updated C API documentation to better define ownership and lifetime requirements for error information returned by ov_get_error_info.

OpenVINO Node.js API

  • Added GenAI API support for Image2ImagePipeline and InpaintingPipeline.

  • Added image decoding support for the image generation pipeline.

  • Refactored InitWorker to throw catchable errors for easier problem diagnosis.

  • Improved Tensor memory safety in the Node.js API by keeping the underlying TypedArray alive for the Tensor lifetime, preventing premature JavaScript garbage collection and improving application stability.

PyTorch Framework Support

  • Improved aten::layer_norm translation with type alignment, enabling mixed-precision (FP16/BF16) models.

  • Added conversion support for AWQ and INT4 pack-quantized models produced by compressed-tensors.

  • Added support for the torch_fused GPTQ quantization type (validated with gptqmodel 6.0.3).

ONNX Framework Support

  • Extended GroupQueryAttention with quantized INT8/INT4 and FP8 (float8e4m3fn) KV cache support and QK-norm.

  • Improved ScatterND to honor the reduction attribute, Reshape to honor allowzero, and LayerNormalization to produce its optional Mean and InvStdDev outputs.

  • Enabled conversion of models that accumulate tensors into a sequence across loop iterations (a common pattern in KV-cache and RNN-style graphs), including support for the SequenceAt, SequenceErase, and SequenceLength operations.

  • Fixed a numerical-stability issue in ReduceLogSumExp, improving accuracy for inputs with large values.

TensorFlow Lite Framework Support

  • Added odml.rms_norm STABLEHLO composite translation.

OpenVINO Model Server
+++++++++++++++++++++

  • Improved performance, accuracy, memory consumption, and stability, with particular focus on new models featuring linear attention architectures.

  • Added new tool parsers for LFM2.5 and MiniCPM-V5 models.

  • Added support for the Jinja chat template engine as an alternative to Minja for Vision Language Models (VLMs).

  • Introduced automatic detection of key runtime parameters for generative models, including task category, target device, and tool parsers, reducing required configuration and minimizing deployment errors.

  • Simplified deployment of classic models by enabling direct deployment without versioning by pointing --model_path to a model file, in addition to a folder with model versions.

  • Extended OVMS CLI support for configuring and deploying local models, including the ability to add classic and generative models to the same multi-model configuration file.

  • Preview: Added support for the Kokoro model in the OpenAI API /audio/speech endpoint, including multilingual support through the optional eSpeak component.

  • Extended support for Automatic Speech Recognition (ASR) models beyond the Whisper model family, enabling more generic audio endpoint support for models such as Qwen-ASR.

  • Added /v1 endpoint as an alias for the /v3 used for OpenAI API, allowing both versions to be used interchangeably. The /v1 endpoint is the recommended long-term path for closer OpenAI API compatibility; /v3 will be preserved until 2027.

  • Introduced MediaPipe graph pooling for improved resource management and pipeline reuse.

  • Preview: Added support for Omni models.

Neural Network Compression Framework
++++++++++++++++++++++++++++++++++++

  • Added FP8 support for the ONNX backend, enabling both full and weight-only quantized FP8 ONNX models through NNCF.

  • Added preliminary support for INT2 and INT3 data types for weight compression through nncf.compress_weights() for the OpenVINO backend.

  • Added support for the GroupedMatMul operation in data-free weight compression.

  • Updated ONNX dependency to version 1.22.0.

OpenVINO Tokenizers
+++++++++++++++++++

  • Optimized BPE node performance and internal caching, improving end-to-end performance for tokenizer workloads.

  • Improved compatibility with Hugging Face Transformers v5.7 and later.

  • Added ONNX translators for SentencepieceTokenizer, SentencepieceDecoder, VectorToString, StringJoin, and StringSplit operations.

OpenVINO GenAI
++++++++++++++

  • Introduced new pipelines:

    • Preview: OmniPipeline with support for Qwen3-Omni, enabling multimodal understanding with combined text and speech output.

    • EmbeddingPipeline for multimodal embedding generation with Qwen3-VL-Embedding.

    • ASRPipeline as a common interface for automatic speech recognition models.

  • Added Kokoro text-to-speech model support to Text2SpeechPipeline.

  • Added support for speech recognition with Qwen3-ASR.

  • Extended VLM support to include Gemma 3n.

  • Extended image generation support to Flux.2 Klein.

  • Enabled CDPruner visual token pruning for Qwen3-VL, reducing KV cache usage and improving TTFT and throughput.

  • Extended the EAGLE-3 speculative decoding pipeline to LLMs and VLMs, enhancing continuous batching and adding Top-K sampling to improve token generation speed on CPU, GPU, and NPU.

  • Improved VideoChat-Flash vision encoder performance.

  • Enabled independent configuration of VLM components through MODEL_PROPERTIES, including plugin settings for the language model, vision encoder, and embeddings model.

  • Extended performance metrics to report LLM sampling duration and per-stage speech recognition latency.

  • Improved apply_chat_template performance by caching compiled Minja templates, reducing TTFT during generation.

Other Changes and Known Issues
++++++++++++++++++++++++++++++

  • Released optimum-intel v2.1.0, compatible with OpenVINO 2026.3, available on PyPI <https://pypi.org/project/optimum-intel/>__.

Jupyter Notebooks

  • New models and use cases:

    • MiniCPM5-1B <https://openvinotoolkit.github.io/openvino_notebooks/?search=LLM-powered+Chatbot>__

    • OmniVoice Text-to-Speech <https://openvinotoolkit.github.io/openvino_notebooks/?search=OmniVoice+Text-to-Speech>__ (experimental)

    • Unified image & video generation with Bernini-R-1.3B <https://openvinotoolkit.github.io/openvino_notebooks/?search=Unified+image+%26+video+generation+with+Bernini-R-1.3B>__ (experimental)

    • Qwen3-VL-Embedding-8B <https://openvinotoolkit.github.io/openvino_notebooks/?search=Multimodal+Embedding+with+Qwen3-VL>__

    • Document Parsing using Unlimited-OCR <https://openvinotoolkit.github.io/openvino_notebooks/?search=Document+Parsing+using+Unlimited-OCR>__ (experimental)

    • Gemma-4-12b <https://openvinotoolkit.github.io/openvino_notebooks/?search=gemma+4>__

  • Archived notebooks (still available in 2026.2 branch and on the Archived tab):

    • Explainable AI Toolkit <https://openvinotoolkit.github.io/openvino_notebooks/?search=explainable>__

    • Visual-language assistant with GLM4-V and OpenVINO <https://openvinotoolkit.github.io/openvino_notebooks/?search=glm4-v>__

    • Multimodal understanding and generation with Janus-Pro and OpenVINO <https://openvinotoolkit.github.io/openvino_notebooks/?search=janus>__

    • Image generation with HunyuanDIT and OpenVINO <https://openvinotoolkit.github.io/openvino_notebooks/?search=Image+generation+with+HunyuanDIT>__

    • Style Transfer with OpenVINO <https://openvinotoolkit.github.io/openvino_notebooks/?search=Style+Transfer>__

    • Visual-language assistant with Pixtral and OpenVINO <https://openvinotoolkit.github.io/openvino_notebooks/?search=pixtral>__

    • Infinite Zoom Stable Diffusion v2 <https://openvinotoolkit.github.io/openvino_notebooks/?search=Infinite+Zoom+Stable+Diffusion+v2>__

    • Object segmentations with EfficientSAM and OpenVINO <https://openvinotoolkit.github.io/openvino_notebooks/?search=Object+segmentations+with+EfficientSAM>__

    • Convert and Optimize YOLOv9 with OpenVINO™ <https://openvinotoolkit.github.io/openvino_notebooks/?search=Convert+and+Optimize+YOLOv9>__

    • Convert and Optimize YOLOv10 with OpenVINO <https://openvinotoolkit.github.io/openvino_notebooks/?search=Convert+and+Optimize+YOLOv10>__

    • Convert and Optimize YOLOv12 real-time object detection with OpenVINO™ <https://openvinotoolkit.github.io/openvino_notebooks/?search=Convert+and+Optimize+YOLOv12>__

    • Convert and Optimize YOLOv8 <https://openvinotoolkit.github.io/openvino_notebooks/?search=yolov8>__

Known Issues

| Component: CPU Plugin
| ID: CVS-191433
| Description:
| The ENABLE_CPU_PINNING property is not currently supported on dual-socket CWF Windows Servers. Its default value on these systems is No. If a user changes the value to Yes in the application, the application will crash on this system.

| Component: GenAI
| ID: 191107
| Description:
| The updated chat template for Gemma 4 models cannot be parsed by the Minja templating engine because Minja does not support adjacent literal-string concatenation syntax. As a workaround, the previous chat-template version can be set manually for the GenAI pipeline with pipe.get_tokenizer().set_chat_template(chat_template). A specific revision of the chat template can be downloaded from the Hugging Face model card. For example, see the Gemma-4-12B-it chat template <https://huggingface.co/google/gemma-4-12B-it/blob/657684fef0b5ac5d6bff39284ceb6ec3710b700e/chat_template.jinja>__.

| Component: GPU Plugin
| ID: 187785, 187019
| Description:
| The Gemma 4 26B A4B model has an accuracy issue when the input prompt is long, and the agentic BFCL test fails on GPU. The Gemma 4 31B model has an accuracy issue when an image is included in the input.

| Component: GenAI NPU
| ID: 191130
| Description:
| Qwen3 LLM fails to load on NPU. Qwen3 compilation fails for context sizes greater than 8K. As a workaround, switching compilation to CID mode through the compiler in the driver should resolve the problem.

Previous 2026 releases
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

.. dropdown:: 2026.2 - 28 May 2026

What's New

  • More Gen AI coverage and frameworks integrations to minimize code changes

    • New models supported: Gemma 4 E2B and Gemma 4 E4B

      • Only on CPUs & GPUs: Qwen3-Coder-Next, Qwen3.5, Qwen3.6, Trinity-mini, LFM2-24B-A2B, LFM2-8B-A1B, LFM2.5-350M​
      • Only on CPUs: YOLO26
      • Only on GPUs: Gemma 4 31B and Gemma 4 26B-A4B
      • Extended to GPUs: GPT-OSS-120B

    • Scaled Dot-Product Attention (SDPA) path support added for LFM2 models

    • Support for Hugging Face Transformers v5.0, ensuring compatibility with the latest model architecture for enhanced interoperability.

  • Broader LLM model support and more model compression techniques

    • OpenVINO™ GenAI introduces extension support for loading custom extension libraries and registering unsupported operations via the extensions property. This gives developers the flexibility to run models with custom ops that OpenVINO doesn't support out of the box.
    • INT4 KV-cache compression is enabled for GPUs, with substantial memory reduction when KV cache size is significant, such as with large input prompts exceeding 32K tokens.
    • OpenVINO GenAI significantly reduces model loading times on GPU when using cache blobs — preventing bottlenecks for multi-stage AI pipelines, including agentic use cases that rely on multiple models.
    • Optimized IR read mode with independently managed constant buffers to reduce peak memory usage by avoiding unnecessary duplication of weight data unless required for correctness (Linux support added in this release).
    • Preview: Enhanced XAttention accuracy on CPUs and GPUs through by-channel INT8 KV-cache quantization (compared to by-token INT8 KV-cache), matching the default by-channel INT8 KV cache quantization when XAttention is not enabled.

  • More portability and performance to run AI at the edge, in the cloud or locally

    • OpenVINO™ GenAI extends its JavaScript API to include a Text-to-Speech pipeline and VLM samples for browser and Node.js developers.
    • OpenVINO™ Model Server extends tool-calling support to Qwen 3.5 and 3.6 models to enable agentic AI use cases.
    • OpenVINO™ Model Server adds streaming transcription support for speech-to-text, reducing latency for real-time voice applications.
    • Preview: Introducing OpenVINO Physical AI, a hardware-accelerated, production‑ready inferencing and deployment framework that standardizes how developers connect cameras, robots, models, and safety controls, reducing brittle custom harnesses and making complex systems easier to build, debug, and evolve on Intel platforms.

OpenVINO™ Runtime

Common Plugin

  • Filesystem path handling in the C++ API has been improved through internal frontend enhancements, eliminating platform-specific path conversion issues and reducing the risk of path-related errors during model loading and deployment workflows.
  • Introduced properties RUNTIME_REQUIREMENTS and COMPATIBILITY_CHECK which allow users to check if compiled models can be imported by the device before sending them to the OpenVINO runtime.
  • Reduced peak and average memory consumption on model compilation when using mmap on Linux.
  • Improved model serialization error handling and report errors when there is no space on disk to store model.
  • The ov::save_model function now adds runtime attributes containing the OpenVINO version used for saving, enabling better model provenance tracking and version compatibility management.
  • Constant folding failures have been resolved when unsupported floating-point precision was encountered; operations now automatically fall back to FP32 precision to ensure successful model optimization.

CPU Device Plugin

  • Enabled support for the new-generation Qwen3 series models, including Qwen3-Coder-Next, Qwen3.5 and Qwen3.6, with performance optimizations.
  • Added support and optimization for CausalConv1D and GatedDeltaNet operations with kernel implementation, enabling models such as Qwen3 and LFM2.

GPU Device Plugin

  • Enabled support for the new-generation Qwen3 series models, including Qwen3-Coder-Next, Qwen3.5 and Qwen3.6, with performance optimizations.
  • Enabled parallel loading for model cache blobs, significantly reducing model load time.
  • Enabled by-channel INT8 KV cache quantization by default for GPU XAttention (sparse attention), delivering improved accuracy and aligning the configuration with the standard path.
  • INT4 KV cache quantization has been enabled, reducing memory consumption for key-value cache storage during LLM inference.
  • Improved Qwen3-vl-4b performance for TTFT, TPOT, and model loading.
  • The GPT-OSS-20B model now supports INT8 weight precision and runs on Intel® Core™ Ultra Series 2 processors (H-SKUs) and Intel® Arc™ GPUs.
  • Improved ResNet-34 performance on Intel® Xe2 architecture.

NPU Device Plugin

  • Integrated NPU compiler version 8.1, splitting the compiler library into openvino_intel_npu_compiler_loader and openvino_intel_npu_compiler components. The loader library can return the list of supported properties and respond to queries about specific properties. The core compiler library is loaded in memory only when a model is compiled.
  • Added blob encryption and decryption support through ov::cache_encryption_callbacks, enabling secure model caching and export/import workflows. A security warning will be issued when using encryption callbacks with 'Compiler-in-Driver' on driver versions up to 32.0.100.4724 due to temporary unencrypted file usage.
  • Added support for ov::runtime_requirements to expose a human-readable compatibility description of the compiled model. Current support covers 'Compiler-in-Plugin' models, with 'Compiler-in-Driver' support planned for a future release.
  • Added support for ov::compatibility_check to check the compatibility of a compiled model based only on description (previously obtained through ov::runtime_requirements). Models are marked as supported in the current OpenVINO release if compiled for the current platform with sufficient device tiles available for execution. The compatibility check does not currently consider driver capabilities and cannot guarantee model acceptance during import. This logic will be adjusted in the upcoming release to rely on Level Zero (UMD) for more extensive compatibility checks when available in future drivers.
  • Added support for CompiledModel::release_memory(), enabling memory consumption reduction between inferences through NPU driver graph eviction mechanisms.
  • Implemented Level Zero command queue pooling to enable queue sharing across multiple compiled models, reducing doorbell utilization and recycling overhead.
  • Added model priority to be changed dynamically for compiled models through the ov::model_priority property after model compilation.
  • Exposed compiler version information through compiled model properties, providing better traceability for which compiler was automatically selected by the plugin during compilation and preserving this information in cached or exported models.
  • Added device recovery mechanisms to handle device lost errors, allowing applications to create new ov::Core instances after device resets to resume execution, aligning with future driver recovery capabilities.
  • Introduced general attention handling optimization, reducing time-to-first-token (TTFT) on most public LLMs for longer prompts (starting at 4K tokens). Use NPUW_PREFILL_ATTENTION_HINT:STATIC to revert to previous behavior if issues occur.
  • Improved compile time for large models, including LLMs.
  • Enhanced partitioning stability that previously caused compilation failures for several VLM models.
  • Enhanced support for GenAI models produced with Transformers 5.x.
  • Improved second token performance stability for LLMs on Intel® Core™ Ultra Series 2 processors.
  • Introduced new backward-compatible blob format, currently limited to classic (non-LLM) models. Blobs exported with NPUW_ENSURE_COMPATIBILITY:YES maintain compatibility with future OpenVINO versions for the given NPU architecture.

OpenVINO Node.js API

  • Error handling in the Node.js API has been enhanced for Core and AsyncInferQueue classes, providing more robust and predictable exception management during model loading and inference operations.

PyTorch Framework Support

  • GPTQ quantized model support has been added via torch.export, enabling conversion of 4-bit GPTQ models (AutoGPTQ) through the torch.export path.
  • Quantized model conversion with torch.export has been extended to support AWQ and BitNet quantized models in addition to existing TorchScript compatibility.
  • CELU operation accuracy has been fixed to match PyTorch behavior, resolving numerical precision issues.

ONNX Framework Support

  • Scan, Loop, and If operations have been improved to correctly handle models using graph initializers as direct outputs of control flow subgraphs, resolving conversion failures and incorrect results. Scan operation now validates num_scan_inputs and properly handles models where the loop body has fewer outputs than initial state values.
  • Tokenizer operation support has been added via openvino-tokenizers integration, enabling conversion of ONNX models using StringNormalizer, LabelEncoder, Tokenizer, and TfIdfVectorizer operations when openvino-tokenizers is installed. Users receive guidance to install the package if missing.

OpenVINO™ Model Server

Performance

  • Improved performance on Intel Data Center GPU Flex 60 and Flex 70 for Qwen3-30B MoE model family.
  • Improved multinomial algorithm performance, reducing latency for generation with temperature > 0.
  • Improved model loading and pipeline initialization performance for new inference requests.

New models and hardware support

  • Restored support for generative models on hosts with CPUs without Intel® AVX2 instruction set when using supported discrete GPUs.
  • Added support for Intel® Xe GPUs for MoE models, including Intel® Arc™ A770.
  • Enabled execution of GPT-OSS-20B with INT8 precision and GPT-OSS-120B with INT4 precision on GPU.
  • Enabled models and support for MoE for Qwen3.5, Qwen3.6, Qwen3-Coder-Next, and Gemma 4 (without continuous batching).
  • Fixed chat template rendering for DeepSeek and Granite models when processing non-ASCII characters.
  • Added tool parsers for Gemma 4 and LFM2 models.

Deployment ease

  • Improved default performance tuning to use resource constraints in Docker containers, with default number of REST workers, OpenVINO inference streams, threads, and CPU pinning configurations avoiding quota and ulimit settings on Linux to prevent overallocation and performance degradation in Docker and Kubernetes environments.
  • Enhanced deployment capabilities with local generative model startup options and runtime parameter configuration through CLI, enabling generative model deployment from read-only filesystems with configurable runtime parameters such as target device and cache size for seamless KServe and OpenShift integration.
  • Improved model pulling recovery mechanisms to resume interrupted Hugging Face model downloads from the previous checkpoint in case of failures or interruptions.

New or improved endpoints capabilities

  • Added initial support for /responses endpoint.
  • Fixed server readiness endpoint behavior - /v2/health/ready now correctly reports success when all models are fully initialized and returns appropriate errors when models are not loaded.
  • Added min_p sampling parameter for enhanced generation control.
  • Added skip_special_tokens sampling parameter - when set to False, returns raw model responses including special tokens to users.
  • Fixed default seed parameter to use random values, ensuring non-deterministic responses from LLM models.
  • Added LoRA adapter support for both image generation models and LLM models.
  • Added support of streaming for audio/transcriptions endpoint
  • Introduced OVMS_AUDIO_MAX_FILE_SIZE_BYTES environment variable that controls the upper bound on memory that a single audio request can allocate for decoded data.

Limitations:

  • Gemma 4 and LFM2 MoE models supported without Continuous Batching
  • /responses endpoint doesn't include built-in tools, audio input and multinomial output. There are also no session management capabilities.

Neural Network Compression Framework

  • Added support for transpose_a attribute in Scale Estimation compression algorithm.
  • Added INT4/INT8 weight compression support for Vision-Language-Action (VLA) with Pi0.5 model.

OpenVINO Tokenizers

  • Added ONNX Frontend extension with new translators for tokenization-related operations: Label Encoder, StringNormalizer, Tokenizer, TFID Vectorizer.
  • Updated TensorFlow Frontend extension with AsString operation translator.
  • Extended Python CLI with new check and diagnose tools.
  • Reduced binary size on Linux and Mac platforms.

OpenVINO™ GenAI

  • Support for hybrid attention models with linear state (CausalConv1D, GatedDeltaNet) for SDPA and PA backend.
  • VLM models enabled: Qwen3.5, Qwen3.6, Gemma 4 (SDPA backend), VideoChat-Flash.
  • Hybrid attention text models enabled: LFM2, Qwen3-Coder-Next.
  • VLMPipeline now supports in-pipeline video sampling with video metadata API, enabling raw video frames input directly without manual frame sampling.
  • Continuous Batching API now supports images_batches, videos_batches, videos_metadata_batches properties.
  • Multinomial sampling performance has been improved, and a new min_p sampling parameter has been added for enhanced generation control.
  • Whisper pipeline results now include a language field indicating the detected or specified language for audio transcription and translation tasks.
  • LoRA adapters can now be applied to Text2VideoPipeline.
  • TaylorSeer caching mechanism is now enabled by default for Flux, Stable Diffusion 3, and LTX-Video.
  • Added new TOOL_CALL : finish_reason for better integration with agentic tools.
  • Performance metrics now include apply_chat_template() latency measurements for comprehensive chat pipeline profiling.
  • A new extensions API has been added, enabling direct loading of OpenVINO extensions within GenAI pipelines for custom operation support.
  • The Node.js API has been updated to include support for Text2SpeechPipeline and Text2ImagePipeline, enabling speech and image generation.
  • Whisper pipeline on NPU now supports word-level timestamps by default.

OpenVINO™ Physical AI

  • Released OpenVINO™ Physical AI, a runtime package for deploying robot policies in real-world environments. This release packages the core deployment stack including camera capture, robot interfaces, exported-policy inference, and runtime loop integration.
  • Introduced unified camera API supporting UVC, RealSense, Basler, and shared-camera transport workflows.
  • Implemented robot interfaces for SO-101 and Trossen WidowX AI integrations.
  • Enabled inference runtime for exported policies with built-in OpenVINO and ONNX backends.
  • Provided runtime control loop with PolicyRuntime, SyncExecution, and AsyncExecution capabilities.
  • Included hardware-specific extras for camera and robot integrations.

Other Changes and Known Issues

Jupyter Notebooks

New models and use cases:

  • Qwen3.5 (0.8B, 2B, 4B, 9B, 27B, 35B-A3B), Qwen3.6 (27B, 35B-A3B) <https://openvinotoolkit.github.io/openvino_notebooks/?search=llm+code+assistant>__
  • Hunyuan Machine Translation (experimental) <https://openvinotoolkit.github.io/openvino_notebooks/?search=Hunyuan+Machine+Translation>__
  • VoxCPM2 Text-to-Speech (experimental) <https://openvinotoolkit.github.io/openvino_notebooks/?search=VoxCPM2+Text-to-Speech>__
  • Text-to-Image Generation with ERNIE-Image-Turbo (experimental) <https://openvinotoolkit.github.io/openvino_notebooks/?search=Text-to-Image+Generation+with+ERNIE-Image-Turbo>__
  • MedASR Medical Speech Recognition <https://openvinotoolkit.github.io/openvino_notebooks/?search=MedASR+Medical+Speech+Recognition>__
  • FLUX.2 Klein Image Generation (experimental) <https://openvinotoolkit.github.io/openvino_notebooks/?search=FLUX.2+Klein+Image+Generation>__
  • Document parsing with GLM-OCR (experimental) <https://openvinotoolkit.github.io/openvino_notebooks/?search=Document+parsing+with+GLM-OCR>__
  • Document parsing with MinerU 2.5 (experimental) <https://openvinotoolkit.github.io/openvino_notebooks/?search=MinerU>__
  • LFM2.5 350M, LFM2-24B-A2B <https://github.com/openvinotoolkit/openvino_notebooks/blob/latest/notebooks/llm-chatbot/llm-chatbot.ipynb>__
  • Visual-language assistant with Gemma 4 <https://openvinotoolkit.github.io/openvino_notebooks/?search=gemma+4>__

Known Issues

| Component: CPU/GPU Plugin
| ID: 186412
| Description:
| Gemma 4 INT8/INT4 weight compressed models require explicitly setting KV cache group size to
64 to preserve accuracy. This modification has been included in the latest version of Optimum
Intel for model export.

| Component: OpenVINO Runtime
| ID: 186160
| Description:
| Gemma 3, internvl2-4b, minicpm4-0.5b and minicpm4-8b models' accuracy is reduced when converted
with optimum-intel and Transformers library version 5.0.0+. As a workaround, convert the models
using following Transformers library versions: Gemma 3- Transformers 4.57.6, internvl2-4b-
Transformers 4.51.3, minicpm4-0.5b and minicpm4-8b- Transformers 4.53.3.

| Component: NPU Plugin
| ID: 214108
| Description:
| Phi 3.5/4 models accuracy is reduced when converted with optimum-intel and Transformers library
version 5.0.0+. As a workaround, convert the model using Transformers library version 4.57.6.

| Component: GPU Plugin
| ID: 187445
| Description:
| Trinity-mini model can demonstrate low accuracy on GPU platforms with INT8 precision.

| Component: OpenVINO Runtime
| ID: 187019
| Description:
| Gemma 4 models can demonstrate low accuracy on some platforms.

| Component: GPU Plugin
| ID: 187077
| Description:
| YOLO26 fails to compile on GPU.

| Component: GPU Plugin
| ID: 180852
| Description:
| GPT-OSS-120B can demonstrate low accuracy on GPU platforms and requires
ACTIVATIONS_SCALE_FACTOR 8.

.. dropdown:: 2026.1 - 7 April 2026

What's New

  • More Gen AI coverage and frameworks integrations to minimize code changes

    • New models supported on CPUs & GPUs: Qwen3 VL
    • New models supported on CPUs: GPT-OSS-120B
    • Preview: Introducing the OpenVINO backend for llama.cpp, which enables optimized inference on Intel CPUs, GPUs, and NPUs. Validated on GGUF models such as Llama-3.2-1B-Instruct-GGUF, Phi-3-mini-4k-instruct-gguf, Qwen2.5-1.5B-Instruct-GGUF, and Mistral-7B-Instruct-v0.3.
    • New notebook: Unified VLM chatbot with video file support and interactive model switching across Qwen3-VL, Qwen2.5-VL, and LLaVa-NeXT-Video.

  • Broader LLM model support and more model compression techniques

    • OpenVINO™ GenAI adds TaylorSeer Lite caching for image and video generation, accelerating diffusion-transformer inference across Flux, SD3, and LTX-Video pipelines, aligned with Hugging Face Diffusers.
    • LTX-Video generation on GPU achieves end-to-end acceleration through fusion of RMSNorm and RoPE operators, significantly improving video generation performance.
    • OpenVINO™ GenAI adds dynamic LoRA support for Qwen3-VL and VL models with LLM, allowing developers to swap adapters at runtime for efficient serving of multiple model variants in production without reloading the base model.
    • Preview: The release-weights API for ov::Model enables memory reclamation during model compilation on NPUs, delivering dramatically lower peak memory consumption for edge and client deployments. Users must set this property in ov::Model, and it will be applied during compilation.

  • More portability and performance to run AI at the edge, in the cloud or locally

    • Introducing support for Intel® Core™ Series 3 processors (formerly codenamed Wildcat Lake) and Intel® Arc™ Pro B70 Graphics with 32GB memory for single-GPU inference on 20-30B parameter LLMs.
    • Prompt Lookup Decoding extended to vision-language pipelines, delivering significantly faster token generation for multimodal workloads on Intel CPUs and GPUs.
    • OpenVINO™ GenAI now has a smaller runtime footprint after eliminating ICU DLL dependencies from tokenization, leading to reduced memory usage, faster startup, and easier deployment.
    • OpenVINO GenAI introduces WhisperPipeline for Node.js via its NPM package, delivering production-ready speech recognition with word-level audio-to-text transcription.
    • OpenVINO™ Model Server enhances support for Qwen3-MOE and GPT-OSS-20B models, delivering improved performance, accuracy, and robust concurrent request handling with continuous batching. These pre-optimized models are available on Hugging Face for easy deployment. Additionally, the Model Server introduces image inpainting and outpainting capabilities via the /image endpoint for AI image editing.

OpenVINO™ Runtime

Common

  • Introduced new properties:

    • CACHE_PATH is fully compatible to CACHE_DIR but natively supports std::filesystem::path.
    • CACHE_BLOB_ID (preview) - Allows users to specify a custom ID for the compiled model in cache. This can accelerate model import times, but users must ensure ID uniqueness to prevent collisions.

  • Improve error messages in IStreamsExecutor::Config::set_property

  • Improve ov::util::ConstantWriter class functionality to reduce chance of introducing bug in hash calculations

  • Fix static resource cleanup by allowing custom cleanup functions to be registered for OpenVINO™ components during library unloading.

  • Fix duplicate hash generation for different ov::Model, eliminating unnecessary model recompilation when OpenVINO caching is enabled.

  • Fix memory leaks and application crashes related to using CRT library (Windows).

CPU Device Plugin

  • Model inference performance has been optimized on Intel® Core™ Ultra Series 3 with 2 P-cores + 4 LPE-cores.
  • XAttention now maintains proper accuracy when enabled, resolving previous accuracy issues.
  • The accuracy issue with long prompt input has been fixed.
  • Upgraded oneDNN version to v3.10.
  • Improve Gemma3 image comprehension by implementing a custom attention mask pattern.

GPU Device Plugin

  • Performance has been improved for LTX-Video model.
  • Preview: Experimental L0 backend support for Xe2+ GPUs. Documentation is provided here <https://github.com/openvinotoolkit/openvino/pull/34727/changes#diff-c0c2a7f47dd4d4c967eb34bf9691372c31f58f9dad77d294dd237d02aa002a1a>__
  • Memory optimization for SD3.5 Flash.
  • XAttention (Block Sparse Attention with Antidiagonal Scoring) is now initially supported on Intel's Xe1 architecture to improve time-to-first token. (Xe2/3 is already supported). Performance has been improved for low threshold scenarios.

NPU Device Plugin

  • Batching changes in NPU plugin:

    • Eliminated the unconditional model clone in the Plugin batching path to reduce memory usage. The model is no longer cloned until after the initial Plugin batch-related checks have been performed.
    • Input and output layouts must be specified for Plugin batching to be applied. If layouts are not provided, the model will be compiled as-is, without any preliminary batch processing in the NPU Plugin.

  • Support for IO strides has been added. During model compilation, users can specify which input/output ports should accept tensors with strides using the new property: ov::intel_npu::enable_strides_for. All desired IO ports must be selected at compilation time. Support for all IO ports is not enabled by default since it can reduce model performance. This feature is supported only with the NPU driver, starting from 32.0.100.4621 (Windows driver) or 1.30 (Linux driver). NPU Plugin will report the property as supported only when dependencies are met. Applications should first check if this property is supported.

  • Fixed accuracy issues with INT8-ASYM Vocabulary on Gemma-2, also improved performance for this path.

  • Introduced Flash Attention support on for Intel® Core™ Ultra Series 3, allows faster LLM compilation for NPU with longer contexts.

  • Improved Long RoPE support for Phi models.

OpenVINO Python API

  • Fixed ov::Tensor creation from NumPy scalar data.
  • Added support for pathlib.Path objects in Core.add_extension and FrontEndManager.register_front_end to unify handling of file path arguments.

OpenVINO Node.js API

  • The openvino-genai-node NPM package has been updated to include the following improvements:

    • Implemented WhisperPipeline: audio-to-text pipeline with word-level transcription, allowing users to generate precise and detailed speech recognition results.
    • New method getGenerationConfig is now available for LLMPipeline, VLMPipeline, and WhisperPipeline, allowing users to quickly retrieve default configuration values and streamline their setup process.
    • ChatHistory is now supported in VLMPipeline, enabling users to manage conversation context more effectively during generation.
    • Async error handling in LLMPipeline has been refactored, preserving existing behavior while improving internal error processing during asynchronous calls, resulting in a more stable and reliable user experience.

PyTorch Framework Support

  • The torch.export path has been significantly improved, aligning TorchFX operation coverage with the TorchScript path.
  • Support for float16 and bfloat16 data types has been added for constant input value extraction.

ONNX Framework Support

  • The Attention operation (Opset 23) is now supported, including multi-head, group query, and multi-query attention modes with KV caching, causal masking, softcap, and boolean/float attention masks. This enables direct conversion of transformer-based ONNX models without manual decomposition.
  • Sequence data type support has been extended with SequenceConstruct, SequenceEmpty, SequenceInsert, and ConcatFromSequence operations, enabling loop-based sequence accumulation patterns commonly found in control-flow models.
  • Support for FP8 quantization types (f8e4m3, f8e5m2) and block-wise quantization has been added to the QuantizeLinear and DequantizeLinear operations.

TensorFlow Framework Support

  • An issue in NonMaxSuppressionV2 where the iou_threshold parameter was ignored has been fixed.

TensorFlow Lite Framework Support

  • The TransposeConv operation has been fixed to correctly apply bias inputs.

OpenVINO™ Model Server

  • Enhanced support for Qwen3-MOE models and GPT-OSS-20B delivers improved performance, accuracy, and robust concurrent request handling with continuous batching capabilities. These models are now available in pre-optimized OpenVINO™ format directly on the Hugging Face hub, making it very easy to deploy them.

  • Added support for Qwen3-VL models with function calling capabilities, enabling this vision language model in agentic scenarios.

  • Extended /image endpoint to support inpainting and outpainting capabilities. It is now possible to pass the input image along with a mask to edit parts of the image or to extend the input image.

  • Other improvements and fixes:

    • Server logs now report current KV cache allocation alongside current usage metrics. With dynamic cache size (default setting), allocation automatically scales during runtime based on the request's concurrency and processed context length.
    • Generation request cancellation is now supported for NPU devices, where requests from disconnected clients will be cancelled.
    • The finish reason now returns tool_calls when the model generates a function call, in line with OpenAI API standards.
    • Corrected tokens usage reporting in the text generation last streaming event with NPU execution.
    • Added extra streaming event right after the first token is generated, in line with OpenAI API. This will correct TTFT metric benchmarking using tools relying on streaming events.
    • Enhanced error handling for Hugging Face Hub model pulling/downloads includes retry and resume capabilities to address network connectivity issues with large model files. Download operations can now recover from previous errors or be reported in logs when recovery is not possible.

Neural Network Compression Framework

  • Added experimental support for NVFP4 data type.
  • Introduced an additional RoPe ignored pattern without the transpose node to support 4-bit compression for models like Phi-3.5-MoE-instruct.
  • Migrated TorchFX backend support from torch.ao to torchao <https://github.com/pytorch/ao>__.
  • Upgraded PyTorch version to 2.10.0.
  • Upgraded ONNX version to 1.20.1 and ONNX Runtime to 1.24.3.

OpenVINO Tokenizers

  • Added precomputed Unicode normalization maps to reduce first inference time and memory consumption.
  • Remove ICU library from dependencies, reducing binary size, build time, and complexity.

OpenVINO™ GenAI

  • Extended prompt lookup decoding support to Vision-Language Models (VLMs) to improve tokens per second (TPS) performance.
  • New AggregationMode.ADAPTIVE_RKV eviction strategy that keeps the highest attention-mass blocks and fills remaining slots with the most semantically diverse ones.
  • VLMPipeline now supports Qwen3-VL.
  • LoRA adapters can now be applied to VLMPipeline (only applied to the language-model (LLM) part), enabling task-specific fine-tuning without reloading the base model.
  • Improved VLM image resizing accuracy.
  • TaylorSeer Lite caching is now available for Flux, Stable Diffusion 3, and LTX-Video (disabled by default).
  • LoRA adapters in GGUF format can now be loaded directly into LLMPipeline and VLMPipeline.
  • TextEmbeddingPipeline now supports dynamic input shapes via the NPUW plugin, enabling NPU inference for a wider range of embedding models.
  • Improved pipeline loading time through asynchronous tokenizer warmup.

Other Changes and Known Issues

Jupyter Notebooks

New models and use cases:

  • MiniCPM-o 4.5 Multimodal Chatbot with OpenVINO (experimental) <https://openvinotoolkit.github.io/openvino_notebooks/?search=minicpm-o+4.5>__
  • LLM Code Assistant with OpenVINO <https://openvinotoolkit.github.io/openvino_notebooks/?search=llm+code+assistant>__
  • Visual-Language Models (VLM) Chatbot with Generate API <https://openvinotoolkit.github.io/openvino_notebooks/?search=vlm-powered>__
  • PaddleOCR-VL for Document Parsing (experimental) <https://openvinotoolkit.github.io/openvino_notebooks/?search=paddleocr-vl>__

Archived Notebooks Tab: Added a dedicated "Archived" tab to the OpenVINO Notebooks portal <https://openvinotoolkit.github.io/openvino_notebooks/>__. Users can now easily search and browse older or deprecated notebooks, keeping the main catalog focused on the latest updates while retaining access to historical content.

Known Issues

| Component: NPU compiler
| ID: 180626
| Description:
| gemma-3-12B accuracy on NPU is low.

| Component: NPU compiler
| ID: 183423, 181329
| Description:
| Performance regression on large number of computer vision models compared to OpenVINO 2026.0
release. For models showing regression, it's recommended to quantize and convert the model
using OpenVINO 2026.0.

.. dropdown:: 2026.0 - 23 February 2026

text
**What's New**

* More Gen AI coverage and frameworks integrations to minimize code changes

  * New models supported on CPUs & GPUs: GPT-OSS-20B, Qwen3-30B-A3B, MiniCPM-V-4_5-8B,  and MiniCPM-o-2.6.
  * New models supported on NPUs: MiniCPM-o-2.6. In addition, NPU support is now available on Qwen2.5-1B-Instruct, Qwen3-Embedding-0.6B, Qwen-2.5-coder-0.5B.
  * Preview: OpenVINO™ GenAI adds support for video generation pipeline based on LTX-Video model on CPU and GPUs.
  * OpenVINO™ GenAI now adds word-level timestamp functionality to the Whisper Pipeline on CPUs, GPUs, and NPUs, enabling more accurate transcriptions and subtitling in line with OpenAI and FasterWhisper implementations.
  * Phi-3-mini FastDraft model is now available on Hugging Face to accelerate LLM inference on NPUs. FastDraft optimizes speculative decoding for LLMs.

* Broader LLM model support and more model compression techniques

  * OpenVINO™ GenAI and OpenVINO™ Model Server introduces EAGLE-3 speculative decoding to accelerate LLM inference using smarter token prediction on Intel CPUs and GPUs. Validated on Qwen3-8B model
  * With the new int4 data-aware weight compression for 3D MatMuls, the Neural Network Compression Framework enables MoE LLMs to run with reduced memory bandwidth, and improved accuracy compared to data-free schemes-delivering faster, more efficient deployment on resource-constrained devices.
  * Preview: The Neural Network Compression Framework now supports per-layer and per-group Look-Up Tables (LUT) for FP8-4BLUT quantization. This enables fine-grained, codebook-based compression that reduces model size and bandwidth while improving inference speed and accuracy for LLMs and transformer workloads.

* More portability and performance to run AI at the edge, in the cloud or locally

  * OpenVINO™ GenAI adds VLM pipeline support to enhance Agentic AI framework integration.
  * OpenVINO GenAI now supports speculative decoding for NPUs, delivering improved performance and efficient text generation through a small draft model that is periodically validated by the full-size model.
  * Preview: NPU compiler integration with the NPU plugin enables ahead-of-time and on-device compilation without relying on OEM driver updates. Developers can enable this feature for a single, ready-to-ship package that reduces integration friction and accelerates time-to-value.
  * OpenVINO™ Model Server adds enhanced support for audio endpoint plus agentic continuous batching and concurrent runs for improved LLM performance in agentic workflows on Intel CPUs and GPUs.

**OpenVINO™ Runtime**

*Common*

* API methods that accept filesystem paths as input are now standardized to accept ``std::filesystem::path``. This makes path handling more consistent across OpenVINO™ and simplifies integration in modern C++ codebases that already rely on ``std::filesystem``. Existing ``std::string`` and ``std::wstring`` overloads are still available.

*CPU Device Plugin*

* GPT-OSS-20B model is now supported, with improved performance for Mixture-of-Experts subgraphs as well as Paged Attention with sink input.
* Rotary Position Embedding fusion and kernel optimization have been expanded to cover more LLMs, including GLM4, to enhance overall performance.
* The accuracy issue with Boolean causal masks in ScaledDotProduct Attention when using BF16/FP16 precision has been resolved, addressing accuracy problems in LFM2.
* XAttention (Block Sparse Attention with Antidiagonal Scoring) is now available as a preview feature to improve Time-To-First-Token (TTFT) performance when processing long context inputs.
* OneTBB library in OpenVINO™ Windows release has been upgraded from 2021.2.1 to 2021.13.1
* Linux docker support for offline cores on platforms with multiple NUMA nodes.

*GPU Device Plugin*

* Improved TTFT for Qwen3-30B-A3B INT4 model, support INT8 model.
* Preview support for XAttention on Intel's Xe2/Xe3 architecture to improve TTFT performance.
* 2nd token latency has been improved for GPT-OSS-20B INT4 model on Intel® Core™ Ultra Series 2, Intel® Core™ Ultra Series 3, and Intel® Arc™ B-Series Graphics.
* TTFT has been improved for vision language models including Phi-3.5-vision, Phi-4-multimodal, and LLaVa-NeXT-Video.

*NPU Device Plugin*

* NPU compiler is now included in the OpenVINO™ distribution package as a separate library. This is a preview feature and can be enabled by setting ``ov::intel_npu::compiler_type`` property to ``PREFER_PLUGIN`` to utilize compiler-in-plugin with fallback to compiler-in-driver in case of compatibility or support issues. By default, the NPU will continue using compiler-in-driver.
* A new model marshaling and serialization mechanism has been implemented to avoid weight copying during compilation, reducing peak memory consumption by up to 1x the original weights size. This mechanism is currently available only when compiler-in-plugin option is enabled.
* Added support for importing CPU virtual addresses into level zero memory through Remote Tensor APIs.
* Fixed various issues related to sliding window context handling in models like Gemma and Phi, improved compatibility with the recent transformers packages.
* Introduced new methods to handle attention, ``NPUW_LLM_PREFILL_ATTENTION_HINT`` can be set to ``PYRAMID`` to significantly improve TTFT. The default value is ``STATIC`` (no change to the existing behavior).
* Reduced KV-cache memory consumption, reaching up to 2.5 GB saving for select models on longer contexts (8..12K).

*OpenVINO Python API*

* OpenVINO™ now supports u2, u3, and u6 unsigned integer data types, enabling more efficient memory usage for quantized models. The u3 and u6 types include optimized packing that writes values into three INT8 containers using a concurrency-friendly pattern, ensuring safe concurrent read/write operations without data spanning across byte boundaries.
* Introduced ``release_gil_before_calling_cpp_dtor`` feature in Python bindings, which optimizes Global Interpreter Lock (GIL) handling during C++ destructor calls. This improves both stability and performance in multi-threaded Python applications.
* Improved PyThreadState management in the Python API for increased stability and crash prevention in complex threading scenarios.
* OpenVINO Python package now requires only NumPy as a runtime dependency. The other packaging dependencies have been removed, resulting in a lighter installation footprint and fewer potential dependency conflicts.
* Added instructions for debugging the Python API on Linux, helping developers troubleshoot and diagnose issues more effectively.

*OpenVINO Node.js API*

* The Node.js API has been improved with GenAI features:

* New parsers have been added to the LLMPipeline to extract structured outputs, reasoning steps, and tool calls from model responses. The parsing layer is fully extensible, enabling developers to plug in their own parsers to tailor how model outputs are interpreted and consumed in downstream applications.
* Added support for running Visual-Language Models, enabling richer multimodal applications that combine image, video, and text understanding in a single VLMPipeline.
* Introduced a dedicated TextRerankPipeline for re-ranking documents, providing a straightforward way to improve retrieval quality and increase relevance in search and RAG scenarios.
* Removed the legacy behaviour whereby ``LLMPipeline.generate()`` could return a string. It now always returns ``DecodedResults``, which provides consistent access to comprehensive information about the generation result, including the output text, scores, performance metrics, and parsed values.

*PyTorch Framework Support*

* The ``axis=None`` parameter is now supported for mean reduction operations, allowing for more flexible tensor averaging.
* Enhanced support for complex data types has been implemented to improve compatibility with vision-language models, such as Qwen.

*ONNX Framework Support*

* Major internal refactoring of the graph iteration mechanism has been implemented for improved performance and maintainability. The legacy path can be enabled by setting the ``ONNX_ITERATOR=0`` environment variable. This legacy path is deprecated and will be removed in future releases.

**OpenVINO™ Model Server**

* Improvements in performance and accuracy for GPT-OSS and Qwen3-MOE models.

	* Improvements in execution performance especially on Intel® Core™ Ultra Series 3 built-in GPUs
	* Improved chat template examples to fix handling agentic use cases
	* Improvements in tool parsers to be less restrictive for the generated content and improve response reliability
	* Better accuracy with INT4 precisions especially with long prompts
* Improvements in text2speech endpoint

	* Added voice parameter to choose speaker based on provided embeddings vector
	* Corrected handling of compilation cache to speed up model loading
* Improvements in speech2text endpoint:

	* Added handling for temperature sampling parameter
	* Support for timestamps in the output

* New parameters have been added to VLM pipelines to control domain name restrictions for image URLs in requests, with optional URL redirection support. By default, all URLs are blocked.
* NPU execution for text embeddings endpoint (experimental)
* Exposed tokenizer endpoint for reranker and LLM pipelines
* Added configurable preprocessing for classic models. Deployed models can include extra preprocessing layers added in at runtime. This can simplify client implementations and enable sending encoded images to models, which are accepted as an array of input. Possible options include:

	* Color format change
	* Layout change
	* Scale changes
	* Mean changes
* Added support for tool parser compatible with devstral model - take advantage of unsloth/Devstral-Small-2507 model or similar for coding tasks.
* Updated numerous demos

	* Audio endpoints
	* VLM endpoints usage
	* Agentic demo
	* Visual Studio Code integration for code assistant
	* Image classification
* Optimized file handle usage to reduce the number of open files during high-load operations on Linux deployments.

**Neural Network Compression Framework**

* Extended 4-bit compression data-aware methods (AWQ, Scale Estimation, GPTQ) to support 3D matmuls for more accurate compression of such models as GPT-OSS-20B and Qwen3-30B-A3B.
* Preview support for per-layer and per-block codebooks has been introduced for 4-bit weight compression (ADAPTIVE_CODEBOOK data type), which helps to reduce the quantization error in the case of per-channel weight compression. See the `example <https://github.com/openvinotoolkit/nncf/tree/develop/examples/llm_compression/openvino/smollm2_360m_adaptive_codebook>`__ for more details.
* Added NNCF Profiler for layer-by-layer profiling of OpenVINO™ model activations. This is useful for debugging quantization and compression issues, comparing model variants, and understanding activation distributions. See more details in `Readme <https://github.com/openvinotoolkit/nncf/blob/develop/tools/activation_profiler/README.md>`__ and `Jupyter notebook <https://github.com/openvinotoolkit/nncf/blob/develop/tools/activation_profiler/nncf_profiler_example.ipynb>`__.
* Added new API method, ``nncf.prune()``, for unstructured pruning of PyTorch models previously supported with the deprecated and removed ``nncf.create_compressed_model()`` method.
* NNCF optimization methods for TensorFlow models and TensorFlow backend in NNCF are deprecated and removed in 2026. It is recommended to use PyTorch analogous models for training-aware optimization methods and OpenVINO IR, PyTorch, and ONNX models for post-training optimization methods from NNCF.
* The following experimental NNCF methods are deprecated and removed: NAS, Structural Pruning, AutoML, Knowledge Distillation, Mixed-Precision Quantization, Movement Sparsity.

**OpenVINO Tokenizers**

* Added support for Qwen3 Reranker and LFM2 models.
* The ``UTF8Validate`` operation has been made available for use in the GGUF GenAI converter.
* Improved tokenization accuracy through improved metaspace handling when processing special tokens.

**OpenVINO™ GenAI**

* Added preview support for video generation via Text2Video pipeline with LTX-Video model.
* Support for EAGLE3 speculative decoding pipeline to boost TPS with an additional EAGLE3 draft model. Support is also enabled on Intel NPU.
* `Conditional Diversity Visual Token Pruning <https://arxiv.org/pdf/2506.10967t>`__ to minimize TTFT of Qwen2/2.5 VL models, this feature is disabled by default and must be turned on.
* Added word-level timestamp generation for detailed transcriptions with WhisperPipeline.
* Added ChatHistory API support for VLMPipeline with images and video.
* Added VLLMParser wrapper.
* Added universal video tags ``<ov_genai_video_i>`` for VLM models with video support (Qwen2-VL, Qwen2.5-VL, LLaVa-NeXT-Video)
* Introduced NPU support for text embedding pipelines (for Qwen3-Embeddings-0.6B and similar models).

**Other Changes and Known Issues**

*Jupyter Notebooks*

New models and use cases:

* `LFM2  <https://github.com/openvinotoolkit/openvino_notebooks/blob/latest/notebooks/llm-chatbot/llm-chatbot.ipynb>`__
* `Qwen3-30B-A3B <https://openvinotoolkit.github.io/openvino_notebooks/?search=chatbot>`__
* `Visual-language assistant with Qwen3-VL and OpenVINO <https://openvinotoolkit.github.io/openvino_notebooks/?search=Visual-language+assistant+with+Qwen3-VL>`__
* `Text-to-image generation with Qwen-Image and OpenVINO <https://openvinotoolkit.github.io/openvino_notebooks/?search=qwen-image>`__ (experimental)
* `Multi-speaker dialogue generation with FireRedTTS-2 and OpenVINO <https://openvinotoolkit.github.io/openvino_notebooks/?search=Fireredtts>`__ (experimental)
* `Document Parsing using DeepSeek-OCR and OpenVINO <https://openvinotoolkit.github.io/openvino_notebooks/?search=DeepSeek-OCR>`__ (experimental)
* `Text-to-image generation with Z-Image-Turbo and OpenVINO <https://openvinotoolkit.github.io/openvino_notebooks/?search=Z-Image-Turbo>`__ (experimental)
* `Text-Image to Video generation with Wan2.2 and OpenVINO <https://openvinotoolkit.github.io/openvino_notebooks/?search=wan2.2>`__ (experimental)
* `End-to-End Speech Recognition with Fun-ASR-Nano and OpenVINO <https://openvinotoolkit.github.io/openvino_notebooks/?search=Fun-ASR-Nano>`__ (experimental)
* `Text-to-Speech (TTS) system with Fun-CosyVoice 3.0 and OpenVINO <https://openvinotoolkit.github.io/openvino_notebooks/?search=CosyVoice>`__ (experimental)
* `PointPillar for 3D object detection <https://openvinotoolkit.github.io/openvino_notebooks/?search=PointPillar>`__

.. dropdown:: Deleted notebooks (still available in 2025.4 branch)

	* `Visual-language assistant with GLM-Edge-V and OpenVINO  <https://github.com/openvinotoolkit/openvino_notebooks/tree/2025.4/notebooks/glm-edge-v>`__
	* `Run inference in Keras 3 with the OpenVINO™ IR backend  <https://github.com/openvinotoolkit/openvino_notebooks/tree/2025.4/notebooks/keras-with-openvino-backend>`__
	* `Unified image generation using OmniGen and OpenVINO  <https://github.com/openvinotoolkit/openvino_notebooks/tree/2025.4/notebooks/omnigen>`__
	* `Convert a JAX Model to OpenVINO™ IR  <https://github.com/openvinotoolkit/openvino_notebooks/tree/2025.4/notebooks/jax-to-openvino>`__
	* `Text-to-Speech synthesis using Llasa and OpenVINO  <https://github.com/openvinotoolkit/openvino_notebooks/tree/2025.4/notebooks/llasa-speech-synthesis>`__
	* `Visual-language assistant with LLaVA Next and OpenVINO  <https://github.com/openvinotoolkit/openvino_notebooks/tree/2025.4/notebooks/nano-llava-multimodal-chatbot>`__
	* `Sound Generation with AudioLDM2 and OpenVINO™  <https://github.com/openvinotoolkit/openvino_notebooks/tree/2025.4/notebooks/sound-generation-audioldm2>`__
	* `Text Generation via Prompt Lookup Decoding using OpenVINO™  <https://github.com/openvinotoolkit/openvino_notebooks/tree/2025.4/notebooks/prompt-lookup-decoding>`__
	* `Stable Diffusion with KerasCV and OpenVINO  <https://github.com/openvinotoolkit/openvino_notebooks/tree/2025.4/notebooks/stable-diffusion-keras-cv>`__
	* `Text-to-image generation using PhotoMaker and OpenVINO  <https://github.com/openvinotoolkit/openvino_notebooks/tree/2025.4/notebooks/photo-maker>`__
	* `Image generation with Sana and OpenVINO  <https://github.com/openvinotoolkit/openvino_notebooks/tree/2025.4/notebooks/sana-image-generation>`__
	* `Document Visual Question Answering Using Pix2Struct and OpenVINO  <https://github.com/openvinotoolkit/openvino_notebooks/tree/2025.4/notebooks/pix2struct-docvqa>`__
	* `Running OpenCLIP models using OpenVINO™  <https://github.com/openvinotoolkit/openvino_notebooks/tree/2025.4/notebooks/open-clip>`__
	* `Magika: AI powered fast and efficient file type identification using OpenVINO  <https://github.com/openvinotoolkit/openvino_notebooks/tree/2025.4/notebooks/magika-content-type-recognition>`__
	* `Text-to-Video retrieval with S3D MIL-NCE and OpenVINO  <https://github.com/openvinotoolkit/openvino_notebooks/tree/2025.4/notebooks/s3d-mil-nce-text-to-video-retrieval>`__
	* `Named entity recognition with OpenVINO™  <https://github.com/openvinotoolkit/openvino_notebooks/tree/2025.4/notebooks/named-entity-recognition>`__
	* `Big Transfer Image Classification Model Quantization with NNCF in OpenVINO™  <https://github.com/openvinotoolkit/openvino_notebooks/tree/2025.4/notebooks/big-transfer-quantization>`__
	* `Image generation with StableCascade and OpenVINO  <https://github.com/openvinotoolkit/openvino_notebooks/tree/2025.4/notebooks/stable-cascade-image-generation>`__
	* `Optical Character Recognition with GOT-OCR 2.0 and OpenVINO  <https://github.com/openvinotoolkit/openvino_notebooks/tree/2025.4/notebooks/got-ocr2>`__
	* `Optimizing PyTorch models with Neural Network Compression Framework of OpenVINO™ by 8-bit quantization.  <https://github.com/openvinotoolkit/openvino_notebooks/tree/2025.4/notebooks/pytorch-quantization-aware-training>`__
	* `Language-Visual Saliency with CLIP and OpenVINO™  <https://github.com/openvinotoolkit/openvino_notebooks/tree/2025.4/notebooks/clip-language-saliency-map>`__
	* `LocalAI and OpenVINO  <https://github.com/openvinotoolkit/openvino_notebooks/tree/2025.4/notebooks/localai>`__
	* `Grammatical Error Correction with OpenVINO  <https://github.com/openvinotoolkit/openvino_notebooks/tree/2025.4/notebooks/grammar-correction>`__
	* `Stable Fast 3D Mesh Reconstruction and OpenVINO  <https://github.com/openvinotoolkit/openvino_notebooks/tree/2025.4/notebooks/stable-fast-3d>`__
	* `Colorize grayscale images using DDColor and OpenVINO  <https://github.com/openvinotoolkit/openvino_notebooks/tree/2025.4/notebooks/ddcolor-image-colorization>`__
	* `Image Generation with Tiny-SD  <https://github.com/openvinotoolkit/openvino_notebooks/tree/2025.4/notebooks/tiny-sd-image-generation>`__
	* `Text-to-Speech synthesis using OuteTTS and OpenVINO  <https://github.com/openvinotoolkit/openvino_notebooks/tree/2025.4/notebooks/outetts-text-to-speech>`__
	* `Text-to-Music generation using Riffusion and OpenVINO  <https://github.com/openvinotoolkit/openvino_notebooks/tree/2025.4/notebooks/riffusion-text-to-music>`__
	* `Video Classification with InternVideo2 and OpenVINO  <https://github.com/openvinotoolkit/openvino_notebooks/tree/2025.4/notebooks/intern-video2-classiciation>`__
	* `Single-step image generation using SDXL-turbo and OpenVINO  <https://github.com/openvinotoolkit/openvino_notebooks/tree/2025.4/notebooks/sdxl-turbo>`__
	* `Optimizing TensorFlow models with Neural Network Compression Framework of OpenVINO™ by 8-bit quantization  <https://github.com/openvinotoolkit/openvino_notebooks/tree/2025.4/notebooks/tensorflow-quantization-aware-training>`__
	* `Animating Open-domain Images with DynamiCrafter and OpenVINO  <https://github.com/openvinotoolkit/openvino_notebooks/tree/2025.4/notebooks/dynamicrafter-animating-images>`__
	* `Create a native Agent with OpenVINO  <https://github.com/openvinotoolkit/openvino_notebooks/tree/2025.4/notebooks/llm-native-agent-react>`__
	* `Visual-language assistant with GLM4-V and OpenVINO  <https://github.com/openvinotoolkit/openvino_notebooks/tree/2025.4/notebooks/glm4-v>`__

*Known Issues*

| **Component: Optimum**
| ID: 179936
| Description:
|   phi-4-multimodal instruct model isn't functional when converted using optimum-cli as channel-wise one (with -group-size -1) with OpenVINO 2026.0. It's recommended to use for the conversion OV 2025.4/OV 2025.4.1

| **Component: OpenVINO Runtime**
| ID: 179009
| Description:
|   Memory leak for static builds with HybridCRT enabled; impacts Windows only

| **Component: OpenVINO Runtime**
| ID: 181161
| Description:
|   GPT-OSS-20B int4 on Intel® Core™ Ultra Series 3 iGPUs is validated for production use in
	single-stream and low-concurrency text generation scenarios with short-to-medium context
	lengths; support for high-concurrency serving with long contexts is being optimized and
	will be available in an upcoming release. For production deployments requiring sustained
	concurrent workloads, CPU-based serving is the recommended configuration today.

Deprecation And Support
+++++++++++++++++++++++

Using deprecated features and components is not advised. They are available to enable a smooth transition to new solutions and will be discontinued in the future. To keep using discontinued features, you will have to revert to the last LTS OpenVINO version supporting them. For more details, refer to the OpenVINO Legacy Features and Components <https://docs.openvino.ai/2026/documentation/legacy-features.html>__ page.

Discontinued in 2026

  • The deprecated openvino.runtime namespace has been removed. Please use the openvino namespace directly.

  • The deprecated openvino.Type.undefined has been removed. Please use openvino.Type.dynamic instead.

  • Support for Debian 10 has been discontinued due to the end of its standard support.

  • The PostponedConstant constructor signature has been updated for improved usability:

    • Old (removed): Callable[[Tensor], None]

    • New: Callable[[], Tensor]

  • The deprecated OpenVINO™ GenAI predefined generation configs were removed.

  • The deprecated OpenVINO GenAI support for whisper stateless decoder model has been removed. Please use a stateful model.

  • The deprecated OpenVINO GenAI StreamerBase put method, bool return type for callbacks, and ChunkStreamer class has been removed.

  • NNCF create_compressed_model() method is now deprecated and removed in 2026. Please use nncf.prune() method for unstructured pruning and nncf.quantize() for INT8 quantization.

  • NNCF optimization methods for TensorFlow models and TensorFlow backend in NNCF are deprecated and removed in 2026. It is recommended to use PyTorch analogous models for training-aware optimization methods and OpenVINO™ IR, PyTorch, and ONNX models for post-training optimization methods from NNCF.

  • The following experimental NNCF methods are deprecated and removed: NAS, Structural Pruning, AutoML, Knowledge Distillation, Mixed-Precision Quantization, Movement Sparsity.

  • CPU plugin now requires support for the AVX2 instruction set as a minimum system requirement. The SSE instruction set will no longer be supported.

  • OpenVINO™ migrated builds based on RHEL 8 to RHEL 9.

  • Dropped support for the TensorFlow Serving (TFS) API in OpenVINO Model Server 2026.3. KServe API is recommended for classic model deployments.

Deprecated and to be removed in the future

  • auto shape and auto batch size (reshaping a model in runtime) will be removed in the future. OpenVINO™’s dynamic shape models are recommended instead.

  • Starting with 2026.0 release major internal refactoring of the graph iteration mechanism has been implemented for improved performance and maintainability. The legacy path can be enabled by setting the ONNX_ITERATOR=0 environment variable. This legacy path is deprecated and will be removed in future releases.

  • OpenVINO Model Server:

    • The dedicated OpenVINO operator for Kubernetes and OpenShift is now deprecated in favor of the recommended KServe operator. The OpenVINO operator will remain functional in upcoming OpenVINO™ Model Server releases but will no longer be actively developed. Since KServe provides broader capabilities, no loss of functionality is expected. On the contrary, more functionalities will be accessible and migration between other serving solutions and OpenVINO Model Server will be much easier.

    • Support for Stateful models <https://docs.openvino.ai/2025/model-server/ovms_docs_stateful_models.html>__ will be deprecated. These capabilities were originally introduced for Kaldi audio models which are no longer applicable. Current audio models support relies on the OpenAI API, and pipelines implemented via OpenVINO™ GenAI library.

    • Directed Acyclic Graph Scheduler <https://docs.openvino.ai/2025/model-server/ovms_docs_dag.html>__ will be deprecated in favor of pipelines managed by MediaPipe scheduler and will be removed in 2026.3. That approach gives more flexibility, includes wider range of calculators and has support for using processing accelerators.

  • OpenVINO™ GenAI:

    • start_chat() / finish_chat() APIs are deprecated and will be removed in a future major release. Pass a ChatHistory object directly to generate() instead.

Legal Information
+++++++++++++++++

You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. All information provided here is subject to change without notice. Contact your Intel representative to obtain the latest Intel product specifications and roadmaps.

No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document.

The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request.

Intel technologies' features and benefits depend on system configuration and may require enabled hardware, software or service activation. Learn more at www.intel.com or from the OEM or retailer.

No computer system can be absolutely secure.

Intel, Atom, Core, Xeon, OpenVINO, and the Intel logo are trademarks of Intel Corporation in the U.S. and/or other countries. Other names and brands may be claimed as the property of others.

Copyright © 2026, Intel Corporation. All rights reserved.

For more complete information about compiler optimizations, see our Optimization Notice.

Performance varies by use, configuration and other factors.

Articles En Documentation Compatibility And Support (`docs/articles_en/documentation/compatibility-and-support.rst`)

Compatibility and Support

.. toctree::
:maxdepth: 1
:hidden:

compatibility-and-support/supported-devices
compatibility-and-support/supported-operations
compatibility-and-support/supported-models

| :doc:Supported Devices <compatibility-and-support/supported-devices>:
| compatibility information for supported hardware accelerators.

| :doc:Supported Operations <compatibility-and-support/supported-operations>:
| a listing of operations supported by OpenVINO, based on device and frontend conformance.

| :doc:AI Models verified for OpenVINO™ <compatibility-and-support/supported-models>:
| a list of selected models confirmed to work with Intel® Core Ultra™ Processors with the
OpenVINO™ toolkit.

Articles En Documentation Legacy Features (`docs/articles_en/documentation/legacy-features.rst`)

Legacy Features and Components

.. meta::
:description: A list of deprecated OpenVINO™ components.

Since OpenVINO has grown very rapidly in recent years, a number of its features
and components have been replaced by other solutions. Some of them are still
supported to assure OpenVINO users are given enough time to adjust their projects,
before the features are fully discontinued.

This section will give you an overview of these major changes and tell you how
you can proceed to get the best experience and results with the current OpenVINO
offering.

Discontinued:
#############

.. dropdown:: OpenVINO Development Tools Package

| New solution: OpenVINO Runtime includes all supported components
| Old solution: See how to install Development Tools <https://docs.openvino.ai/2024/documentation/legacy-features/install-dev-tools.html>__
|
| OpenVINO Development Tools used to be the OpenVINO package with tools for
advanced operations on models, such as Model conversion API, Benchmark Tool,
Accuracy Checker, Annotation Converter, Post-Training Optimization Tool,
and Open Model Zoo tools. Most of these tools have been either removed,
replaced by other solutions, or moved to the OpenVINO Runtime package.

.. dropdown:: Model Optimizer / Conversion API

| New solution: :doc:Direct model support and OpenVINO Converter (OVC) <../openvino-workflow/model-preparation>
| Old solution: Legacy Conversion API <https://docs.openvino.ai/2024/documentation/legacy-features/transition-legacy-conversion-api.html>__
|
| The role of Model Optimizer and later the Conversion API was largely reduced
when all major model frameworks became supported directly. For converting model
files explicitly, it has been replaced with a more light-weight and efficient
solution, the OpenVINO Converter (launched with OpenVINO 2023.1).

.. dropdown:: Open Model ZOO

| New solution: users are encouraged to use public model repositories such as Hugging Face <https://huggingface.co/OpenVINO>__
| Old solution: Open Model ZOO <https://docs.openvino.ai/2024/documentation/legacy-features/model-zoo.html>__
|
| Open Model ZOO provided a collection of models prepared for use with OpenVINO,
and a small set of tools enabling a level of automation for the process.
Since the tools have been mostly replaced by other solutions and several
other model repositories have recently grown in size and popularity,
Open Model ZOO will no longer be maintained. You may still use its resources
until they are fully removed. Check the OMZ GitHub project <https://github.com/openvinotoolkit/open_model_zoo>__

.. dropdown:: Multi-Device Execution

| New solution: :doc:Automatic Device Selection <../openvino-workflow/running-inference/inference-devices-and-modes/auto-device-selection>
| Old solution: Check the legacy solution <https://docs.openvino.ai/2024/documentation/legacy-features/multi-device.html>__
|
| The behavior and results of the Multi-Device Execution mode are covered by the CUMULATIVE_THROUGHPUT
option of the Automatic Device Selection. The only difference is that CUMULATIVE_THROUGHPUT uses
the devices specified by AUTO, which means that adding devices manually is not mandatory,
while with MULTI, the devices had to be specified before the inference.

.. dropdown:: Caffe, and Kaldi model formats

| New solution: conversion to ONNX via external tools
| Old solution: model support discontinued with OpenVINO 2024.0
| The last version supporting Apache MXNet, Caffe, and Kaldi model formats <https://docs.openvino.ai/2023.3/mxnet_caffe_kaldi.html>__
| :doc:See the currently supported frameworks <../openvino-workflow/model-preparation>

.. dropdown:: Post-training Optimization Tool (POT)

| New solution: Neural Network Compression Framework (NNCF) now offers the same functionality
| Old solution: POT discontinued with OpenVINO 2024.0
| :doc:See how to use NNCF for model optimization <../openvino-workflow/model-optimization>
| Check the NNCF GitHub project, including documentation <https://github.com/openvinotoolkit/nncf>__

.. dropdown:: Inference API 1.0

| New solution: API 2.0 launched in OpenVINO 2022.1
| Old solution: discontinued with OpenVINO 2024.0
| 2023.2 is the last version supporting API 1.0 <https://docs.openvino.ai/archives/index.html#:~:text=2023.2,Release%20Notes>__

.. dropdown:: Compile tool

| New solution: the tool is no longer needed
| Old solution: discontinued with OpenVINO 2023.0
| If you need to compile a model for inference on a specific device, use the following script:

.. rst-class:: m-4

text
.. tab-set::

     .. tab-item:: Python
        :sync: py

        .. doxygensnippet:: docs/articles_en/assets/snippets/export_compiled_model.py
           :language: python
           :fragment: [export_compiled_model]

     .. tab-item:: C++
        :sync: cpp

        .. doxygensnippet:: docs/articles_en/assets/snippets/export_compiled_model.cpp
           :language: cpp
           :fragment: [export_compiled_model]

.. dropdown:: TensorFlow integration (OVTF)

| New solution: Direct model support and OpenVINO Converter (OVC)
| Old solution: discontinued in OpenVINO 2023.0
|
| OpenVINO now features a native TensorFlow support, with no need for explicit model
conversion.

Articles En Documentation Openvino Extensibility (`docs/articles_en/documentation/openvino-extensibility.rst`)

OpenVINO Extensibility Mechanism

.. meta::
:description: Explore OpenVINO™ Extensibility API, which allows adding
support for models with custom operations and their further implementation
in applications.

.. toctree::
:maxdepth: 1
:hidden:

openvino-extensibility/custom-openvino-operations
openvino-extensibility/frontend-extensions
openvino-extensibility/custom-gpu-operations

.. toctree::
:maxdepth: 1
:hidden:

openvino-extensibility/transformation-api
OpenVINO Plugin Developer Guide <openvino-extensibility/openvino-plugin-library>

The Intel® Distribution of OpenVINO™ toolkit supports neural-network models trained with various frameworks, including
TensorFlow, PyTorch, ONNX, TensorFlow Lite, and PaddlePaddle. The list of supported operations is different for each of the supported frameworks.
To see the operations supported by your framework, refer to :doc:Supported Framework Operations <../documentation/compatibility-and-support/supported-operations>.

Custom operations, which are not included in the list, are not recognized by OpenVINO out-of-the-box. The need for custom operation may appear in two cases:

  1. A new or rarely used regular framework operation is not supported in OpenVINO yet.
  2. A new user operation that was created for some specific model topology by the author of the model using framework extension capabilities.

Importing models with such operations requires additional steps. This guide illustrates the workflow for running inference on models featuring custom operations. This allows plugging in your own implementation for them. OpenVINO Extensibility API enables adding support for those custom operations and using one implementation for model conversion API and OpenVINO Runtime.

Defining a new custom operation basically consists of two parts:

  1. Definition of operation semantics in OpenVINO, the code that describes how this operation should be inferred consuming input tensor(s) and producing output tensor(s). The implementation of execution kernels for :doc:GPU <openvino-extensibility/custom-gpu-operations> is described in separate guides.

  2. Mapping rule that facilitates conversion of framework operation representation to OpenVINO defined operation semantics.

The first part is required for inference. The second part is required for successful import of a model containing such operations from the original framework model format. There are several options to implement each part. The following sections will describe them in detail.

Definition of Operation Semantics
#################################

If the custom operation can be mathematically represented as a combination of existing OpenVINO operations and such decomposition gives desired performance, then low-level operation implementation is not required. Refer to the latest OpenVINO operation set, when deciding feasibility of such decomposition. You can use any valid combination of existing operations. The next section of this document describes the way to map a custom operation.

If such decomposition is not possible or appears too bulky with a large number of constituent operations that do not perform well, then a new class for the custom operation should be implemented, as described in the :doc:Custom Operation Guide <openvino-extensibility/custom-openvino-operations>.

You might prefer implementing a custom operation class if you already have a generic C++ implementation of operation kernel. Otherwise, try to decompose the operation first, as described above. Then, after verifying correctness of inference and resulting performance, you may move on to optional implementation of Bare Metal C++.

Additionally, it is also possible to implement custom operations using Python. OpenVINO provides a Python API that allows you to define and register custom operations. This can be particularly useful for rapid prototyping and testing of new operations.

Mapping from Framework Operation
################################

Mapping of custom operation is implemented differently, depending on model format used for import.
If a model is represented in the ONNX (including models exported from PyTorch in ONNX), TensorFlow Lite, PaddlePaddle or
TensorFlow formats, then you should use one of the classes from :doc:Frontend Extension API <openvino-extensibility/frontend-extensions>,
the application of which is described below.

Registering Extensions
######################

A custom operation class and a new mapping frontend extension class object should be registered to be usable in OpenVINO runtime.

.. note::
This documentation is derived from the Template extension <https://github.com/openvinotoolkit/openvino/tree/master/src/core/template_extension>__, which demonstrates the details of extension development. It is based on minimalistic Identity operation that is a placeholder for your real custom operation. Review the complete, fully compilable code to see how it works.

Use the ov::Core::add_extension method to load the extensions to the ov::Core object. This method allows loading library with extensions or extensions from the code.

Load Extensions to Core
+++++++++++++++++++++++

Extensions can be loaded from a code with the ov::Core::add_extension method:

.. tab-set::

.. tab-item:: Python
:sync: py

text
.. doxygensnippet:: docs/articles_en/assets/snippets/ov_extensions.py
     :language: python
     :fragment: [add_extension]

.. tab-item:: C++
:sync: cpp

text
.. doxygensnippet:: docs/articles_en/assets/snippets/ov_extensions.cpp
     :language: cpp
     :fragment: [add_extension]

The Identity is a custom operation class defined in :doc:Custom Operation Guide <openvino-extensibility/custom-openvino-operations>. This is sufficient to enable reading OpenVINO IR which uses the Identity extension operation. In order to load original model directly to the runtime, add a mapping extension:

.. tab-set::

.. tab-item:: Python
:sync: py

text
.. doxygensnippet:: docs/articles_en/assets/snippets/ov_extensions.py
     :language: python
     :fragment: [add_frontend_extension]

.. tab-item:: C++
:sync: cpp

text
.. doxygensnippet:: docs/articles_en/assets/snippets/ov_extensions.cpp
     :language: cpp
     :fragment: [add_frontend_extension]

If custom OpenVINO operation is implemented in C++ and loaded into the runtime through a shared library, there is no way to add a frontend mapping extension that refers to this custom operation. In this case, use C++ shared library approach to implement both operations semantics and framework mapping.

.. _create_a_library_with_extensions:

Create a Library with Extensions
++++++++++++++++++++++++++++++++

An extension library should be created in the following cases:

  • Conversion of a model with custom operations in model conversion API
  • Loading a model with custom operations in a Python application. This applies to both framework model and OpenVINO IR.
  • Loading models with custom operations in tools that support loading extensions from a library, for example the benchmark_app.

To create an extension library, perform the following:

  1. Create an entry point for extension library. OpenVINO provides the OPENVINO_CREATE_EXTENSIONS() macro, which allows to define an entry point to a library with OpenVINO Extensions.
    This macro should have a vector of all OpenVINO Extensions as an argument.

Based on that, the declaration of an extension class might look like the following:

.. doxygensnippet:: src/core/template_extension/ov_extension.cpp
:language: cpp
:fragment: [ov_extension:entry_point]

  1. Configure the build of your extension library, using the following CMake script:

.. doxygensnippet:: src/core/template_extension/CMakeLists.txt
:language: cpp
:fragment: [cmake:extension]

This CMake script finds OpenVINO, using the find_package CMake command.

  1. Build the extension library, running the commands below:

.. code-block:: sh

$ cd src/core/template_extension/new
$ mkdir build
$ cd build
$ cmake -DOpenVINO_DIR= ../
$ cmake --build .

The OpenVINO python distribution could be also used. The following code snippet demonstrates how to get the OpenVINO_DIR:

.. code-block:: sh

$ cd src/core/template_extension/new
$ mkdir build
$ cd build
$ cmake -DOpenVINO_DIR=$(python3 -c "from openvino.utils import get_cmake_path; print(get_cmake_path(), end='')") ../
$ cmake --build .

  1. After the build, you may use the path to your extension library to load your extensions to OpenVINO Runtime:

.. tab-set::

.. tab-item:: Python
:sync: py

text
.. doxygensnippet:: docs/articles_en/assets/snippets/ov_extensions.py
     :language: python
     :fragment: [add_extension_lib]

.. tab-item:: C++
:sync: cpp

text
.. doxygensnippet:: docs/articles_en/assets/snippets/ov_extensions.cpp
     :language: cpp
     :fragment: [add_extension_lib]

See Also
########

  • :doc:OpenVINO Transformations <openvino-extensibility/transformation-api>
  • :doc:Using OpenVINO Runtime Samples <../get-started/learn-openvino/openvino-samples>
  • :doc:Hello Shape Infer SSD sample <../get-started/learn-openvino/openvino-samples/hello-reshape-ssd>

Articles En Documentation Openvino Ir Format (`docs/articles_en/documentation/openvino-ir-format.rst`)

OpenVINO IR format

.. meta::
:description: OpenVINO IR, known as Intermediate Representation, is the result
of model conversion in OpenVINO and is represented by two files:
an XML and a binary file.

.. toctree::
:maxdepth: 1
:hidden:

openvino-ir-format/operation-sets
openvino-ir-format/operation-sets/available-opsets
openvino-ir-format/operation-sets/operation-specs
openvino-ir-format/operation-sets/broadcast-rules
openvino-ir-format/intermediate-representation-int8-inference

OpenVINO IR is the proprietary model format of OpenVINO, benefiting from the full extent
of its features. It is obtained by converting a model from one of the
:doc:supported formats <../openvino-workflow/model-preparation>
using the model conversion API or OpenVINO Converter. The process translates common
deep learning operations of the original network to their counterpart representations in
OpenVINO and tunes them with the associated weights and biases.
The resulting OpenVINO IR format contains two files:

  • .xml - Describes the model topology.
  • .bin - Contains the weights and binary data.

:doc:See why converting to OpenVINO IR is recommended <../openvino-workflow/model-preparation/convert-model-to-ir>

IR Structure
############

OpenVINO toolkit introduces its own format of graph representation and its own operation set. A graph is represented with two files: an XML file and a binary file. This representation is commonly referred to as the Intermediate Representation or IR.

The XML file describes a model topology using a <layer> tag for an operation node and an <edge> tag for a data-flow connection.
Each operation has a fixed number of attributes that define operation flavor used for a node.
For example, the Convolution operation has such attributes as dilation, stride, pads_begin, and pads_end.

The XML file does not have big constant values like convolution weights.
Instead, it refers to a part of the accompanying binary file that stores such values in a binary format.

Here is an example of a small IR XML file that corresponds to a graph from the previous section:

.. scrollbox::   

.. code-block:: cpp

text
<?xml version="1.0" ?>
  <net name="model_file_name" version="10">
     <layers>
        <layer id="0" name="input" type="Parameter" version="opset1">
              <data element_type="f32" shape="1,3,32,100"/> 
              <output>
                 
                 
                    <dim>1</dim>
                    <dim>3</dim>
                    <dim>32</dim>
                    <dim>100</dim>
                 
              </output>
        </layer>
        <layer id="1" name="conv1/weights" type="Const" version="opset1">
              
              <data element_type="f32" offset="0" shape="64,3,3,3" size="6912"/>
              <output>
                 
                    <dim>64</dim>
                    <dim>3</dim>
                    <dim>3</dim>
                    <dim>3</dim>
                 
              </output>
        </layer>
        <layer id="2" name="conv1" type="Convolution" version="opset1">
              <data auto_pad="same_upper" dilations="1,1" output_padding="0,0" pads_begin="1,1" pads_end="1,1" strides="1,1"/>
              
                 
                    <dim>1</dim>
                    <dim>3</dim>
                    <dim>32</dim>
                    <dim>100</dim>
                 
                 
                    <dim>64</dim>
                    <dim>3</dim>
                    <dim>3</dim>
                    <dim>3</dim>
                 
              
              <output>
                 
                    <dim>1</dim>
                    <dim>64</dim>
                    <dim>32</dim>
                    <dim>100</dim>
                 
              </output>
        </layer>
        <layer id="3" name="conv1/activation" type="ReLU" version="opset1">
              
                 
                    <dim>1</dim>
                    <dim>64</dim>
                    <dim>32</dim>
                    <dim>100</dim>
                 
              
              <output>
                 
                    <dim>1</dim>
                    <dim>64</dim>
                    <dim>32</dim>
                    <dim>100</dim>
                 
              </output>
        </layer>
        <layer id="4" name="output" type="Result" version="opset1">
              
                 
                    <dim>1</dim>
                    <dim>64</dim>
                    <dim>32</dim>
                    <dim>100</dim>
                 
              
        </layer>
     </layers>
     <edges>
        
        <edge from-layer="0" from-port="0" to-layer="2" to-port="0"/>
        <edge from-layer="1" from-port="1" to-layer="2" to-port="1"/>
        <edge from-layer="2" from-port="2" to-layer="3" to-port="0"/>
        <edge from-layer="3" from-port="1" to-layer="4" to-port="0"/>
     </edges>
     <meta_data>
        
        <MO_version value="2022.3"/>
        <cli_parameters>
              
              <caffe_parser_path value="DIR"/>
              <data_type value="float"/>

              ...

              

        </cli_parameters>
     </meta_data>
  </net>

The IR does not use explicit data nodes described in the previous section. In contrast, properties of data such as tensor dimensions and their data types are described as properties of input and output ports of operations.

Additional Resources
####################

  • :doc:IR and Operation Sets <openvino-ir-format/operation-sets>

Articles En Documentation Openvino Security (`docs/articles_en/documentation/openvino-security.rst`)

OpenVINO Security

Deploying deep learning models for OpenVINO may raise security and privacy issues.
Trained models are often valuable intellectual property and you may choose to protect them
with encryption or other security tools.

Actual security and privacy requirements depend on your unique deployment scenario.
This section provides general guidance on using OpenVINO tools and libraries securely.
The main security measure for OpenVINO is its
:doc:Security Add-on <../about-openvino/openvino-ecosystem/openvino-project/openvino-security-add-on>. You can find its description
in the Ecosystem section.

.. _encrypted-models:

Using Encrypted Models with OpenVINO
##############################################

Deploying deep-learning capabilities to edge devices can present security challenges like ensuring
inference integrity, or providing copyright protection of your deep-learning models.

One possible solution is to use cryptography to protect models as they are deployed and stored
on edge devices. Model encryption, decryption and authentication are not provided by OpenVINO
but can be implemented with third-party tools (i.e., OpenSSL). While implementing encryption,
ensure that the latest versions of tools are used and follow cryptography best practices.

This guide presents how to use OpenVINO securely with protected models.

Secure Model Deployment
+++++++++++++++++++++++++++++++++++

After a model is optimized by model conversion API, it's deployed to target devices in the
OpenVINO Intermediate Representation (OpenVINO IR) format. An optimized model is stored on edge
device and is executed by the OpenVINO Runtime. TensorFlow, TensorFlow Lite, ONNX and PaddlePaddle
models can be read natively by OpenVINO Runtime as well.

Encrypting and optimizing model before deploying it to the edge device can be used to protect
deep-learning models. The edge device should keep the stored model protected all the time
and have the model decrypted in runtime only for use by the OpenVINO Runtime.

.. image:: ../assets/images/deploy_encrypted_model.svg

Loading Encrypted Models
+++++++++++++++++++++++++++++++++++

The OpenVINO Runtime requires model decryption before loading. Allocate a temporary memory block
for model decryption and use the ov::Core::read_model method to load the model from a memory
buffer. For more information, see the ov::Core Class Reference Documentation.

.. doxygensnippet:: docs/articles_en/assets/snippets/protecting_model_guide.cpp
:language: cpp
:fragment: part0

Hardware-based protection such as Intel Software Guard Extensions (Intel SGX) can be used to protect
decryption operation secrets and bind them to a device. For more information, see
the Intel Software Guard Extensions <https://software.intel.com/en-us/sgx>__.

Use the ov::Core::read_model <../api/c_cpp_api/group__ov__dev__exec__model.html#classov_1_1_core_1ae0576a95f841c3a6f5e46e4802716981>__
to set model representations and weights respectively.

Currently there is no way to read external weights from memory for ONNX models.
The ov::Core::read_model(const std::string& model, const Tensor& weights) method
should be called with weights passed as an empty ov::Tensor.

.. doxygensnippet:: docs/articles_en/assets/snippets/protecting_model_guide.cpp
:language: cpp
:fragment: part1

Encrypted models that have already been compiled, in the form of blob files,
can be loaded using the
ov::Core::import_model <../api/c_cpp_api/group__ov__runtime__cpp__api.html#_CPPv4N2ov4Core12import_modelERNSt7istreamERKNSt6stringERK6AnyMap>__
method, as shown in the code sample below:

.. code-block:: cpp

ov::Core core;
// Import a model from a blob.
std::ifstream compiled_blob(blob, std::ios_base::in | std::ios_base::binary);
auto compiled_model = core.import_model(compiled_blob, "CPU");

Additional Resources
####################

  • Intel® Distribution of OpenVINO™ toolkit home page <https://software.intel.com/en-us/openvino-toolkit>__.
  • :doc:Convert a Model <../openvino-workflow/model-preparation/convert-model-to-ir>.
  • :doc:OpenVINO™ Runtime User Guide <../openvino-workflow/running-inference>.
  • For more information on Sample Applications, see the :doc:OpenVINO Samples Overview <../get-started/learn-openvino/openvino-samples>

Articles En Get Started Install Openvino (`docs/articles_en/get-started/install-openvino.rst`)

Install OpenVINO™ 2026.3

.. meta::
:description: install OpenVINO Runtime package, using the distribution channel
of your choice.

.. toctree::
:maxdepth: 3
:hidden:

OpenVINO GenAI
OpenVINO Runtime on Linux
OpenVINO Runtime on Windows
OpenVINO Runtime on macOS
Create an OpenVINO Yocto Image
Additional Hardware Setup and Troubleshooting

.. raw:: html

OpenVINO 2026.3, described here, is not a Long-Term-Support version!
All currently supported versions are:

  • 2026.3.1 (development)
  • 2025.4.1 (maintenance)

.. dropdown:: Effortless GenAI integration with OpenVINO GenAI

OpenVINO GenAI streamlines application development by providing LLM-specific interfaces for
easy integration of language models, handling tokenization and text generation.
For installation and usage instructions, check
:doc:OpenVINO GenAI installation <../openvino-workflow-generative> and
:doc:inference with OpenVINO GenAI <../openvino-workflow-generative/inference-with-genai>.

.. dropdown:: Building OpenVINO from Source

OpenVINO Toolkit source files are available on GitHub as open source. If you want to build your own version of OpenVINO for your platform,
follow the OpenVINO Build Instructions <https://github.com/openvinotoolkit/openvino/blob/master/docs/dev/build.md>__.

Articles En Get Started Learn Openvino (`docs/articles_en/get-started/learn-openvino.rst`)

Learn OpenVINO

.. meta::
:description: Explore learning materials, including interactive
Python tutorials and sample console applications that explain
how to use OpenVINO features.

.. toctree::
:maxdepth: 1
:hidden:

Interactive Tutorials (Python) <learn-openvino/interactive-tutorials-python>
Sample Applications (Python & C++) <learn-openvino/openvino-samples>

This section will help you get a hands-on experience with OpenVINO even if you are just starting
to learn what OpenVINO is and how it works. It includes various types of learning materials
accommodating different learning needs, which means you should find it useful if you are a beginning,
as well as an experienced user.

| :doc:Tutorials <learn-openvino/interactive-tutorials-python>
| A collection of interactive Python tutorials. It introduces you to the OpenVINO™ toolkit explaining how to use the Python API and tools for optimized deep learning inference. The tutorials are available in Jupyter notebooks and can be run in your browser. No installation required.

| :doc:OpenVINO Samples <learn-openvino/openvino-samples>
| The OpenVINO samples (Python and C++) are simple console applications that show how to use specific OpenVINO API features. They can assist you in executing tasks such as loading a model, running inference, querying particular device capabilities, etc.

Articles En Openvino Workflow Generative Genai Model Preparation (`docs/articles_en/openvino-workflow-generative/genai-model-preparation.rst`)

Generative Model Preparation

.. meta::
:description: Learn how to use Hugging Face Hub and Optimum Intel APIs to
prepare generative models for inference.

Since generative AI models tend to be big and resource-heavy, it is advisable to
optimize them for efficient inference. This article will show how to prepare
LLM models for inference with OpenVINO by:

  • Downloading Models from Hugging Face <#download-generative-models-from-hugging-face-hub>__
  • Downloading Models from Model Scope <#download-generative-models-from-model-scope>__
  • Converting and Optimizing Generative Models <#convert-and-optimize-generative-models>__

Download Generative Models From Hugging Face Hub
###############################################################################

Pre-converted and pre-optimized models are available in the OpenVINO Toolkit <https://huggingface.co/OpenVINO>__
organization, under the model section <https://huggingface.co/OpenVINO#models>__, or under
different model collections:

  • LLM: <https://huggingface.co/collections/OpenVINO/llm-6687aaa2abca3bbcec71a9bd>__
  • Speech-to-Text <https://huggingface.co/collections/OpenVINO/speech-to-text-672321d5c070537a178a8aeb>__
  • Speculative Decoding Draft Models <https://huggingface.co/collections/OpenVINO/speculative-decoding-draft-models-673f5d944d58b29ba6e94161>__

You can also use the huggingface_hub package to download models:

.. code-block:: console

pip install huggingface_hub
huggingface-cli download "OpenVINO/phi-2-fp16-ov" --local-dir model_path

The models can be used in OpenVINO immediately after download. No dependencies
are required except huggingface_hub.

Download Generative Models From Model Scope
###############################################################################

To download models from Model Scope <https://www.modelscope.cn/home>__,
use the modelscope package:

.. code-block:: console

pip install modelscope
modelscope download --model "Qwen/Qwen2-7b" --local_dir model_path

Models downloaded via Model Scope are available in Pytorch format only and they must
be :doc:converted to OpenVINO IR <../../openvino-workflow/model-preparation/convert-model-to-ir>
before inference.

Convert and Optimize Generative Models
###############################################################################

OpenVINO works best with models in the OpenVINO IR format, both in full precision and quantized.
If your selected model has not been pre-optimized, you can easily do it yourself, using a single
optimum-cli command. For that, make sure optimum-intel is installed on your system:

.. code-block:: console

pip install optimum-intel[openvino]

While optimizing models, you can decide to keep the original precision or select one that is lower.

.. tab-set::

.. tab-item:: Keeping full model precision
:sync: full-precision

text
.. code-block:: console

     optimum-cli export openvino --model <model_id> --weight-format fp16 <exported_model_name>

  Examples:

  .. tab-set::

     .. tab-item:: LLM (text generation)
        :sync: llm-text-gen

        .. code-block:: console

           optimum-cli export openvino --model meta-llama/Llama-2-7b-chat-hf --weight-format fp16 ov_llama_2

     .. tab-item:: Diffusion models (text2image)
        :sync: diff-text-img

        .. code-block:: console

           optimum-cli export openvino --model stabilityai/stable-diffusion-xl-base-1.0 --weight-format fp16 ov_SDXL

     .. tab-item:: VLM (Image processing):
        :sync: vlm-img-proc

        .. code-block:: console

           optimum-cli export openvino --model openbmb/MiniCPM-V-2_6 --trust-remote-code –weight-format fp16 ov_MiniCPM-V-2_6

     .. tab-item:: Whisper models (speech2text):
        :sync: whisp-speech-txt

        .. code-block:: console

           optimum-cli export openvino --trust-remote-code --model openai/whisper-base ov_whisper

     .. tab-item:: SpeechT5 TTS models (text2speech):
        :sync: whisp-speech-txt

        .. code-block:: console

           optimum-cli export openvino --model microsoft/speecht5_tts --model-kwargs "{\"vocoder\": \"microsoft/speecht5_hifigan\"}" ov_speecht5_tts

.. tab-item:: Exporting to selected precision
:sync: low-precision

text
.. code-block:: console

     optimum-cli export openvino --model <model_id> --weight-format int4 <exported_model_name>

  Examples:

  .. tab-set::

     .. tab-item:: LLM (text generation)
        :sync: llm-text-gen

        .. code-block:: console

           optimum-cli export openvino --model meta-llama/Llama-2-7b-chat-hf --weight-format int4 ov_llama_2

     .. tab-item:: Diffusion models (text2image)
        :sync: diff-text-img

        .. code-block:: console

           optimum-cli export openvino --model stabilityai/stable-diffusion-xl-base-1.0 --weight-format int4 ov_SDXL

     .. tab-item:: VLM (Image processing)
        :sync: vlm-img-proc

        .. code-block:: console

           optimum-cli export openvino -m model_path --task text-generation-with-past --weight-format int4 ov_MiniCPM-V-2_6

.. note::

Any other model_id, for example openbmb/MiniCPM-V-2_6, or the path
to a local model file can be used.

Also, you can specify different data type like int8.

Additional Resources
###############################################################################

  • Full set of optimum-cli parameters <https://huggingface.co/docs/optimum/en/intel/openvino/export>__
  • :doc:Model conversion in OpenVINO <../../openvino-workflow/model-preparation/convert-model-to-ir>
  • :doc:Model optimization in OpenVINO <../../openvino-workflow/model-optimization>

Articles En Openvino Workflow Generative Inference With Genai (`docs/articles_en/openvino-workflow-generative/inference-with-genai.rst`)

Inference with OpenVINO GenAI

.. meta::
:description: Learn how to use OpenVINO GenAI to execute LLM models.

.. toctree::
:maxdepth: 1
:hidden:

NPU with OpenVINO GenAI

OpenVINO™ GenAI is a library of pipelines and methods, extending the OpenVINO runtime to work
with generative AI models more efficiently. This article provides reference code and guidance
on its usage. Note that the base OpenVINO version will not work with these instructions,
make sure to :doc:install OpenVINO with GenAI <../../get-started/install-openvino/install-openvino-genai>.

.. image:: ../assets/images/genai_main_diagram.svg
:align: center
:alt: OpenVINO GenAI workflow diagram

| Here is sample code for several Generative AI use case scenarios. Note that these are very basic
examples and may need adjustments for your specific needs, like changing the inference device.
| For a more extensive instruction and additional options, see the
step-by-step chat-bot guide <#chat-bot-use-case-step-by-step>__ below.

.. dropdown:: Text-to-Image Generation

OpenVINO GenAI introduces openvino_genai.Text2ImagePipeline for inference of text-to-image
models such as: as Stable Diffusion 1.5, 2.1, XL, LCM, Flex, and more.
See the following usage example for reference.

.. tab-set::

text
.. tab-item:: Python
     :sync: python

     .. tab-set::

        .. tab-item:: text2image.py
           :name: text2image

           .. code-block:: python

              import argparse

              import openvino_genai
              from PIL import Image


              def main():
                  parser = argparse.ArgumentParser()
                  parser.add_argument('model_dir')
                  parser.add_argument('prompt')
                  args = parser.parse_args()

                  device = 'CPU'  # GPU can be used as well
                  pipe = openvino_genai.Text2ImagePipeline(args.model_dir, device)

                  image_tensor = pipe.generate(
                      args.prompt,
                      width=512,
                      height=512,
                      num_inference_steps=20,
                      num_images_per_prompt=1)

                  image = Image.fromarray(image_tensor.data[0])
                  image.save("image.bmp")

        .. tab-item:: lora_text2image.py
           :name: loratext2imagepy

           .. code-block:: python

              import openvino as ov
              import openvino_genai

              def image_write(path: str, image_tensor: ov.Tensor):
                  from PIL import Image
                  image = Image.fromarray(image_tensor.data[0])
                  image.save(path)


              def main():
                  parser = argparse.ArgumentParser()
                  parser.add_argument('models_path')
                  parser.add_argument('prompt')
                  args, adapters = parser.parse_known_args()

                  prompt = args.prompt

                  device = "CPU"  # GPU, NPU can be used as well
                  adapter_config = openvino_genai.AdapterConfig()

                  # Multiple LoRA adapters applied simultaneously are supported, parse them all and corresponding alphas from cmd parameters:
                  for i in range(int(len(adapters) / 2)):
                      adapter = openvino_genai.Adapter(adapters[2 * i])
                      alpha = float(adapters[2 * i + 1])
                      adapter_config.add(adapter, alpha)

                  # LoRA adapters passed to the constructor will be activated by default in next generates
                  pipe = openvino_genai.Text2ImagePipeline(args.models_path, device, adapters=adapter_config)

                  print("Generating image with LoRA adapters applied, resulting image will be in lora.bmp")
                  image = pipe.generate(prompt,
                                        width=512,
                                        height=896,
                                        num_inference_steps=20,
                                        rng_seed=42)

                  image_write("lora.bmp", image)
                  print("Generating image without LoRA adapters applied, resulting image will be in baseline.bmp")
                  image = pipe.generate(prompt,
                                        # passing adapters in generate overrides adapters set in the constructor; openvino_genai.AdapterConfig() means no adapters
                                        adapters=openvino_genai.AdapterConfig(),
                                        width=512,
                                        height=896,
                                        num_inference_steps=20,
                                        rng_seed=42)
                  image_write("baseline.bmp", image)


     For more information, refer to the
     `Python sample <https://github.com/openvinotoolkit/openvino.genai/tree/master/samples/python/image_generation>`__

  .. tab-item:: C++
     :sync: cpp

     .. tab-set::

        .. tab-item:: text2image.cpp
           :name: text2imagecpp

           .. code-block:: cpp

              #include "openvino/genai/image_generation/text2image_pipeline.hpp"

              #include "imwrite.hpp"

              int32_t main(int32_t argc, char* argv[]) try {
                  OPENVINO_ASSERT(argc == 3, "Usage: ", argv[0], " <MODEL_DIR> ''");

                  const std::string models_path = argv[1], prompt = argv[2];
                  const std::string device = "CPU";  // GPU can be used as well

                  ov::genai::Text2ImagePipeline pipe(models_path, device);
                  ov::Tensor image = pipe.generate(prompt,
                      ov::genai::width(512),
                      ov::genai::height(512),
                      ov::genai::num_inference_steps(20),
                      ov::genai::num_images_per_prompt(1));

                  // writes `num_images_per_prompt` images by pattern name
                  imwrite("image_%d.bmp", image, true);

                  return EXIT_SUCCESS;
              } catch (const std::exception& error) {
                  try {
                      std::cerr << error.what() << '\n';
                  } catch (const std::ios_base::failure&) {}
                  return EXIT_FAILURE;
              } catch (...) {
                  try {
                      std::cerr << "Non-exception object thrown\n";
                  } catch (const std::ios_base::failure&) {}
                  return EXIT_FAILURE;
              }

        .. tab-item:: lora_text2image.cpp
           :name: loratext2imagecpp

           .. code-block:: cpp

              #include "openvino/genai/image_generation/text2image_pipeline.hpp"

              #include "imwrite.hpp"

              int32_t main(int32_t argc, char* argv[]) try {
                  OPENVINO_ASSERT(argc >= 3 && (argc - 3) % 2 == 0, "Usage: ", argv[0], " <MODEL_DIR> '' [<LORA_SAFETENSORS> <ALPHA> ...]]");

                  const std::string models_path = argv[1], prompt = argv[2];
                  const std::string device = "CPU";  // GPU, NPU can be used as well

                  ov::genai::AdapterConfig adapter_config;
                  // Multiple LoRA adapters applied simultaneously are supported, parse them all and corresponding alphas from cmd parameters:
                  for(size_t i = 0; i < (argc - 3)/2; ++i) {
                      ov::genai::Adapter adapter(argv[3 + 2*i]);
                      float alpha = std::atof(argv[3 + 2*i + 1]);
                      adapter_config.add(adapter, alpha);
                  }

                  // LoRA adapters passed to the constructor will be activated by default in next generates
                  ov::genai::Text2ImagePipeline pipe(models_path, device, ov::genai::adapters(adapter_config));

                  std::cout << "Generating image with LoRA adapters applied, resulting image will be in lora.bmp\n";
                  ov::Tensor image = pipe.generate(prompt,
                      ov::genai::width(512),
                      ov::genai::height(896),
                      ov::genai::num_inference_steps(20),
                      ov::genai::rng_seed(42));
                  imwrite("lora.bmp", image, true);

                  std::cout << "Generating image without LoRA adapters applied, resulting image will be in baseline.bmp\n";
                  image = pipe.generate(prompt,
                      ov::genai::adapters(),  // passing adapters in generate overrides adapters set in the constructor; adapters() means no adapters
                      ov::genai::width(512),
                      ov::genai::height(896),
                      ov::genai::num_inference_steps(20),
                      ov::genai::rng_seed(42));
                  imwrite("baseline.bmp", image, true);

                  return EXIT_SUCCESS;
              } catch (const std::exception& error) {
                  try {
                      std::cerr << error.what() << '\n';
                  } catch (const std::ios_base::failure&) {}
                  return EXIT_FAILURE;
              } catch (...) {
                  try {
                      std::cerr << "Non-exception object thrown\n";
                  } catch (const std::ios_base::failure&) {}
                  return EXIT_FAILURE;
              }

     For more information, refer to the
     `C++ sample <https://github.com/openvinotoolkit/openvino.genai/tree/master/samples/cpp/image_generation/>`__

.. dropdown:: Speech Recognition

The application performs inference on speech recognition Whisper Models. The samples include
the WhisperPipeline class and use audio files in WAV format at a sampling rate of 16 kHz
as input.

.. tab-set::

text
.. tab-item:: Python
     :sync: cpp

     .. code-block:: python

        import openvino_genai
        import librosa


        def read_wav(filepath):
            raw_speech, samplerate = librosa.load(filepath, sr=16000)
            return raw_speech.tolist()


        def infer(model_dir: str, wav_file_path: str):
            device = "CPU"  # GPU or NPU can be used as well.
            pipe = openvino_genai.WhisperPipeline(model_dir, device)

            # The pipeline expects normalized audio with a sampling rate of 16kHz.
            raw_speech = read_wav(wav_file_path)
            result = pipe.generate(
                raw_speech,
                max_new_tokens=100,
                language="<|en|>",
                task="transcribe",
                return_timestamps=True,
            )

            print(result)

            for chunk in result.chunks:
                print(f"timestamps: [{chunk.start_ts}, {chunk.end_ts}] text: {chunk.text}")


     For more information, refer to the
     `Python sample <https://github.com/openvinotoolkit/openvino.genai/tree/master/samples/python/whisper_speech_recognition/>`__.

  .. tab-item:: C++
     :sync: cpp

     .. code-block:: cpp

        #include "audio_utils.hpp"
        #include "openvino/genai/whisper_pipeline.hpp"

        int main(int argc, char* argv[]) try {
            if (3 > argc) {
                throw std::runtime_error(std::string{"Usage: "} + argv[0] + " <MODEL_DIR> \"<WAV_FILE_PATH>\"");
            }

            std::filesystem::path models_path = argv[1];
            std::string wav_file_path = argv[2];
            std::string device = "CPU";  // GPU or NPU can be used as well.

            ov::genai::WhisperPipeline pipeline(models_path, device);

            ov::genai::WhisperGenerationConfig config(models_path / "generation_config.json");
            config.max_new_tokens = 100;
            config.language = "<|en|>";
            config.task = "transcribe";
            config.return_timestamps = true;

            // The pipeline expects normalized audio with a sampling rate of 16kHz.
            ov::genai::RawSpeechInput raw_speech = utils::audio::read_wav(wav_file_path);
            auto result = pipeline.generate(raw_speech, config);

            std::cout << result << "\n";

            for (auto& chunk : *result.chunks) {
                std::cout << "timestamps: [" << chunk.start_ts << ", " << chunk.end_ts << "] text: " << chunk.text << "\n";
            }

        } catch (const std::exception& error) {
            try {
                std::cerr << error.what() << '\n';
            } catch (const std::ios_base::failure&) {
            }
            return EXIT_FAILURE;
        } catch (...) {
            try {
                std::cerr << "Non-exception object thrown\n";
            } catch (const std::ios_base::failure&) {
            }
            return EXIT_FAILURE;
        }

     For more information, refer to the
     `C++ sample <https://github.com/openvinotoolkit/openvino.genai/tree/master/samples/cpp/whisper_speech_recognition/>`__.

.. dropdown:: Text-to-Speech Generation

OpenVINO GenAI provides the openvino_genai.Text2SpeechPipeline API for performing inference with text-to-speech models,
such as the SpeechT5 TTS model. A speaker embedding vector can be specified to control the characteristics of the synthesized voice.
If no embedding is provided, the model defaults to a built-in speaker. Speaker embeddings can be generated using the following script:
create_speaker_embedding.py <https://github.com/openvinotoolkit/openvino.genai/blob/master/samples/python/speech_generation/create_speaker_embedding.py>__.
The example below demonstrates how to use the Text2SpeechPipeline API.

.. tab-set::

text
.. tab-item:: Python
     :sync: cpp

     .. code-block:: python

        import numpy as np
        import openvino as ov
        import openvino_genai
        import soundfile as sf

        device = "CPU"
        speaker_embedding = np.fromfile("speaker_embedding.bin", dtype=np.float32).reshape(1, 512)
        speaker_embedding = ov.Tensor(speaker_embedding)

        pipe = openvino_genai.Text2SpeechPipeline(model_dir, device)

        result = pipe.generate(args.text, speaker_embedding)

        speech = result.speeches[0]
        sf.write("output_audio.wav", speech.data[0], samplerate=16000)


     For more information, refer to the
     `Python sample <https://github.com/openvinotoolkit/openvino.genai/tree/master/samples/python/speech_generation/>`__.

  .. tab-item:: C++
     :sync: cpp

     .. code-block:: cpp

        #include "audio_utils.hpp"
        #include "openvino/genai/whisper_pipeline.hpp"

        int main(int argc, char* argv[]) try {
            OPENVINO_ASSERT(argc == 3 || argc == 4,
                            "Usage: ",
                            argv[0],
                            " <MODEL_DIR> \"\" [<SPEAKER_EMBEDDING_BIN_FILE>]");

            const std::string models_path = argv[1], prompt = argv[2];
            const std::string device = "CPU";

            ov::genai::Text2SpeechPipeline pipe(models_path, device);

            ov::genai::Text2SpeechDecodedResults gen_speech;
            if (argc == 4) {
                const std::string speaker_embedding_path = argv[3];
                auto speaker_embedding = utils::audio::read_speaker_embedding(speaker_embedding_path);
                gen_speech = pipe.generate(prompt, speaker_embedding);
            } else {
                gen_speech = pipe.generate(prompt);
            }

            std::string output_file_name = "output_audio.wav";
            auto waveform_size = gen_speech.speeches[0].get_size();
            auto waveform_ptr = gen_speech.speeches[0].data<const float>();
            auto bits_per_sample = gen_speech.speeches[0].get_element_type().bitwidth();
            utils::audio::save_to_wav(waveform_ptr, waveform_size, output_file_name, bits_per_sample);

            return EXIT_SUCCESS;
        }


     For more information, refer to the
     `C++ sample <https://github.com/openvinotoolkit/openvino.genai/tree/master/samples/cpp/speech_generation/>`__.

.. dropdown:: Using GenAI in Chat Scenario

For chat scenarios where inputs and outputs represent a conversation, maintaining KVCache
across inputs may prove beneficial. The start_chat and finish_chat chat-specific
methods are used to mark a conversation session, as shown in the samples below:

.. tab-set::

text
.. tab-item:: Python
     :sync: py

     .. code-block:: python

        import openvino_genai


        def streamer(subword):
            print(subword, end='', flush=True)
            return False


        def infer(model_dir: str):
            device = 'CPU'  # GPU can be used as well.
            pipe = openvino_genai.LLMPipeline(model_dir, device)

            config = openvino_genai.GenerationConfig()
            config.max_new_tokens = 100

            pipe.start_chat()
            while True:
                try:
                    prompt = input('question:\n')
                except EOFError:
                    break
                pipe.generate(prompt, config, streamer)
                print('\n----------')
            pipe.finish_chat()



     For more information, refer to the
     `Python sample <https://github.com/openvinotoolkit/openvino.genai/tree/master/samples/python/text_generation/chat_sample/>`__.

  .. tab-item:: C++
     :sync: cpp

     .. code-block:: cpp

        #include "openvino/genai/llm_pipeline.hpp"

        int main(int argc, char* argv[]) try {
            if (2 != argc) {
                throw std::runtime_error(std::string{"Usage: "} + argv[0] + " <MODEL_DIR>");
            }
            std::string prompt;
            std::string models_path = argv[1];

            std::string device = "CPU";  // GPU, NPU can be used as well
            ov::genai::LLMPipeline pipe(models_path, device);

            ov::genai::GenerationConfig config;
            config.max_new_tokens = 100;
            std::function streamer = [](std::string word) {
                std::cout << word << std::flush;
                return false;
            };

            pipe.start_chat();
            std::cout << "question:\n";
            while (std::getline(std::cin, prompt)) {
                pipe.generate(prompt, config, streamer);
                std::cout << "\n----------\n"
                    "question:\n";
            }
            pipe.finish_chat();
        } catch (const std::exception& error) {
            try {
                std::cerr << error.what() << '\n';
            } catch (const std::ios_base::failure&) {}
            return EXIT_FAILURE;
        } catch (...) {
            try {
                std::cerr << "Non-exception object thrown\n";
            } catch (const std::ios_base::failure&) {}
            return EXIT_FAILURE;
        }


     For more information, refer to the
     `C++ sample <https://github.com/openvinotoolkit/openvino.genai/tree/master/samples/cpp/text_generation/chat_sample/>`__

  .. tab-item:: JavaScript/TypeScript
     :sync: js

     .. code-block:: typescript

        import { LLMPipeline } from "openvino-genai-node";
        import readline from 'readline';

        const pipe = await LLMPipeline(model_path, 'CPU');

        const config = { 
            max_new_tokens: 100, 
            num_beam_groups: 3, 
            num_beams: 15, 
            diversity_penalty: 1.5 
        };

        await pipe.startChat();
        
        const rl = readline.createInterface({
            input: process.stdin,
            output: process.stdout,
        });

        console.log('question:');
        rl.on('line', async (prompt) => {
            console.log('answer:');
            const answer = await pipe.generate(prompt, config);
            console.log(answer);
            console.log('\n----------\nquestion:');
        });

        rl.on('close', async () => {
            // highlight-next-line
            await pipe.finishChat();
            process.exit(0);
        });

     For more information, refer to the
     `JavaScript sample <https://github.com/openvinotoolkit/openvino.genai/tree/master/samples/js/text_generation/chat_sample/>`__.

.. dropdown:: Using GenAI with Vision Language Models

OpenVINO GenAI introduces the openvino_genai.VLMPipeline pipeline for
inference of multimodal text-generation Vision Language Models (VLMs).
With a text prompt and an image as input, VLMPipeline can generate text using
models such as LLava or MiniCPM-V. See the chat scenario presented
in the samples below:

.. tab-set::

text
.. tab-item:: Python
     :sync: py

     .. code-block:: python

        import numpy as np
        import openvino_genai
        from PIL import Image
        from openvino import Tensor
        from pathlib import Path


        def streamer(subword: str) -> bool:
            print(subword, end='', flush=True)


        def read_image(path: str) -> Tensor:
            pic = Image.open(path).convert("RGB")
            image_data = np.array(pic.getdata()).reshape(1, pic.size[1], pic.size[0], 3).astype(np.uint8)
            return Tensor(image_data)


        def read_images(path: str) -> list[Tensor]:
            entry = Path(path)
            if entry.is_dir():
                return [read_image(str(file)) for file in sorted(entry.iterdir())]
            return [read_image(path)]


        def infer(model_dir: str, image_dir: str):
            rgbs = read_images(image_dir)
            device = 'CPU'  # GPU can be used as well.
            enable_compile_cache = dict()
            if "GPU" == device:
                enable_compile_cache["CACHE_DIR"] = "vlm_cache"
            pipe = openvino_genai.VLMPipeline(model_dir, device, **enable_compile_cache)

            config = openvino_genai.GenerationConfig()
            config.max_new_tokens = 100

            pipe.start_chat()
            prompt = input('question:\n')
            pipe.generate(prompt, images=rgbs, generation_config=config, streamer=streamer)

            while True:
                try:
                    prompt = input("\n----------\n"
                        "question:\n")
                except EOFError:
                    break
                pipe.generate(prompt, generation_config=config, streamer=streamer)
            pipe.finish_chat()


     For more information, refer to the
     `Python sample <https://github.com/openvinotoolkit/openvino.genai/tree/master/samples/python/visual_language_chat>`__.

  .. tab-item:: C++
     :sync: cpp

     .. code-block:: cpp

        #include "load_image.hpp"
        #include <openvino/genai/visual_language/pipeline.hpp>
        #include <filesystem>

        bool print_subword(std::string&& subword) {
            return !(std::cout << subword << std::flush);
        }

        int main(int argc, char* argv[]) try {
            if (3 != argc) {
                throw std::runtime_error(std::string{"Usage "} + argv[0] + " <MODEL_DIR> ");
            }

            std::vector<ov::Tensor> rgbs = utils::load_images(argv[2]);

            std::string device = "CPU";  // GPU can be used as well.
            ov::AnyMap enable_compile_cache;
            if ("GPU" == device) {
                enable_compile_cache.insert({ov::cache_dir("vlm_cache")});
            }
            ov::genai::VLMPipeline pipe(argv[1], device, enable_compile_cache);

            ov::genai::GenerationConfig generation_config;
            generation_config.max_new_tokens = 100;

            std::string prompt;

            pipe.start_chat();
            std::cout << "question:\n";

            std::getline(std::cin, prompt);
            pipe.generate(prompt,
                          ov::genai::images(rgbs),
                          ov::genai::generation_config(generation_config),
                          ov::genai::streamer(print_subword));
            std::cout << "\n----------\n"
                "question:\n";
            while (std::getline(std::cin, prompt)) {
                pipe.generate(prompt,
                              ov::genai::generation_config(generation_config),
                              ov::genai::streamer(print_subword));
                std::cout << "\n----------\n"
                    "question:\n";
            }
            pipe.finish_chat();
        } catch (const std::exception& error) {
            try {
                std::cerr << error.what() << '\n';
            } catch (const std::ios_base::failure&) {}
            return EXIT_FAILURE;
        } catch (...) {
            try {
                std::cerr << "Non-exception object thrown\n";
            } catch (const std::ios_base::failure&) {}
            return EXIT_FAILURE;
        }


     For more information, refer to the
     `C++ sample <https://github.com/openvinotoolkit/openvino.genai/tree/master/samples/cpp/visual_language_chat/>`__

|

Chat-bot use case - step by step
###############################################################################################

This example will show you how to create a chat-bot functionality, using the ov_genai.LLMPipeline
and a chat-tuned TinyLlama model. Apart from the basic implementation, it provides additional
optimization methods.

Although CPU is used as inference device in the samples below, you may choose GPU instead.
Note that tasks such as token selection, tokenization, and detokenization are always handled
by CPU only. Tokenizers, represented as a separate model, are also run on CPU.

Running the model
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

You start with exporting an LLM model via Hugging Face Optimum-Intel. Note that the precision
of int4 is used, instead of the original fp16, for better performance. The weight
compression is done by NNCF at the model export stage. The exported model contains all the
information necessary for execution, including the tokenizer/detokenizer and the generation
config, ensuring that its results match those generated by Hugging Face.

.. note::

To use meta-llama/Llama-2-7b-chat-hf model, you will need to accept license agreement.
You must be a registered user in 🤗 Hugging Face Hub. Please visit
HuggingFace model card <https://huggingface.co/meta-llama/Llama-2-7b-chat-hf>,
carefully read terms of usage and click accept button. You will need to use an access token
for the code below to run. For more information on access tokens, refer to
this section of the documentation <https://huggingface.co/docs/hub/security-tokens>.
Refer to this
document <https://huggingface.co/docs/huggingface_hub/en/guides/cli>__
to learn how to login to Hugging Face Hub.

The LLMPipeline is the main object to setup the model for text generation. You can provide the
converted model to this object, specify the device for inference, and provide additional
parameters.

.. tab-set::

.. tab-item:: Python
:sync: py

text
.. code-block:: console

     optimum-cli export openvino --model "TinyLlama/TinyLlama-1.1B-Chat-v1.0" --weight-format int4 --trust-remote-code "TinyLlama-1.1B-Chat-v1.0"

  .. code-block:: python

        import openvino_genai as ov_genai
        pipe = ov_genai.LLMPipeline(model_path, "CPU")
        print(pipe.generate("The Sun is yellow because", max_new_tokens=100))

.. tab-item:: C++
:sync: cpp

text
.. code-block:: console

     optimum-cli export openvino --model "TinyLlama/TinyLlama-1.1B-Chat-v1.0" --weight-format int4 --trust-remote-code "TinyLlama-1.1B-Chat-v1.0"

  .. code-block:: cpp

     #include "openvino/genai/llm_pipeline.hpp"
     #include 

     int main(int argc, char* argv[]) {
        std::string model_path = argv[1];
        ov::genai::LLMPipeline pipe(model_path, "CPU");
        std::cout << pipe.generate("The Sun is yellow because", ov::genai::max_new_tokens(100));
     }

.. tab-item:: JavaScript/TypeScript
:sync: js

text
.. code-block:: console

     optimum-cli export openvino --model "TinyLlama/TinyLlama-1.1B-Chat-v1.0" --weight-format int4 --trust-remote-code "TinyLlama-1.1B-Chat-v1.0"

  .. code-block:: typescript

     import { LLMPipeline, GenerationConfig } from 'openvino-genai-node';

     async function main() {
       const modelPath = process.argv[2];
       const pipe = await LLMPipeline.LLMPipeline(modelPath, 'CPU');

       const config: GenerationConfig = { maxNewTokens: 100 };
       const result = await pipe.generate('The Sun is yellow because', config);
       
       console.log(result.texts[0]);
     }

     main();

Streaming the Output
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

For more interactive UIs during generation, you can stream output tokens. In this example, a
lambda function outputs words to the console immediately upon generation:

.. tab-set::

.. tab-item:: Python
:sync: py

text
.. code-block:: python

     import openvino_genai as ov_genai
     pipe = ov_genai.LLMPipeline(model_path, "CPU")

     streamer = lambda x: print(x, end='', flush=True)
     pipe.generate("The Sun is yellow because", streamer=streamer, max_new_tokens=100)

.. tab-item:: C++

text
.. code-block:: cpp

     #include "openvino/genai/llm_pipeline.hpp"
     #include 

     int main(int argc, char* argv[]) {
        std::string model_path = argv[1];
        ov::genai::LLMPipeline pipe(model_path, "CPU");

        auto streamer = [](std::string word) {
           std::cout << word << std::flush;
           // Return flag indicating whether generation should be stopped.
           // false means continue generation.
           return false;
        };
        pipe.generate("The Sun is yellow because", ov::genai::streamer(streamer), ov::genai::max_new_tokens(100));
     }

.. tab-item:: JavaScript/TypeScript
:sync: js

text
.. code-block:: typescript

     import { LLMPipeline, GenerationConfig } from 'openvino-genai-node';

     async function main() {
       const modelPath = process.argv[2];
       const pipe = await LLMPipeline.LLMPipeline(modelPath, 'CPU');

       const config: GenerationConfig = { maxNewTokens: 100 };
       for await (const chunk of pipe.stream('The Sun is yellow because', config)) {
         process.stdout.write(chunk);
       }
     }

     main();

You can also create your custom streamer for more sophisticated processing:

.. tab-set::

.. tab-item:: Python
:sync: py

text
.. code-block:: python

     import openvino_genai as ov_genai

     class CustomStreamer(ov_genai.StreamerBase):
        def __init__(self, tokenizer):
           ov_genai.StreamerBase.__init__(self)
           self.tokenizer = tokenizer
        def put(self, token_id) -> bool:
           # Decode tokens and process them.
           # Streamer returns a flag indicating whether generation should be stopped.
           # In Python, `return` can be omitted. In that case, the function will return None
           # which will be converted to False, meaning that generation should continue.
           # return stop_flag
        def end(self):
           # Decode tokens and process them.

     pipe = ov_genai.LLMPipeline(model_path, "CPU")
     pipe.generate("The Sun is yellow because", streamer=CustomStreamer(), max_new_tokens=100)

.. tab-item:: C++

text
.. code-block:: cpp

     #include <streamer_base.hpp>

     class CustomStreamer: public StreamerBase {
     public:
        bool put(int64_t token) {
           bool stop_flag = false;
           /*
           custom decoding/tokens processing code
           tokens_cache.push_back(token);
           std::string text = m_tokenizer.decode(tokens_cache);
           ...
           */
           return stop_flag;  // Flag indicating whether generation should be stopped. If True, generation stops.
        };

        void end() {
           /* custom finalization */
        };
     };

     int main(int argc, char* argv[]) {
        auto custom_streamer = std::make_shared<CustomStreamer>();

        std::string model_path = argv[1];
        ov::genai::LLMPipeline pipe(model_path, "CPU");
        pipe.generate("The Sun is yellow because", ov::genai::streamer(custom_streamer), ov::genai::max_new_tokens(100));
     }

Optimizing Generation with Grouped Beam Search
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

For better text generation quality and more efficient batch processing, specify
generation_config to leverage grouped beam search decoding.

.. tab-set::

.. tab-item:: Python
:sync: py

text
.. code-block:: python

     import openvino_genai as ov_genai
     pipe = ov_genai.LLMPipeline(model_path, "CPU")
     config = pipe.get_generation_config()
     config.max_new_tokens = 256
     config.num_beam_groups = 3
     config.num_beams = 15
     config.diversity_penalty = 1.0
     pipe.generate("The Sun is yellow because", config)

.. tab-item:: C++
:sync: cpp

text
.. code-block:: cpp

     int main(int argc, char* argv[]) {
        std::string model_path = argv[1];
        ov::genai::LLMPipeline pipe(model_path, "CPU");
        ov::genai::GenerationConfig config = pipe.get_generation_config();
        config.max_new_tokens = 256;
        config.num_beam_groups = 3;
        config.num_beams = 15;
        config.diversity_penalty = 1.0f;

        cout << pipe.generate("The Sun is yellow because", config);
     }

.. tab-item:: JavaScript/TypeScript
:sync: js

text
.. code-block:: typescript

     import { LLMPipeline, GenerationConfig } from 'openvino-genai-node';

     async function main() {
       const modelPath = process.argv[2];
       const pipe = await LLMPipeline(modelPath, 'CPU');

       const config : GenerationConfig = {
           maxNewTokens: 256,
           numBeamGroups: 3,
           numBeams: 15,
           diversityPenalty: 1.0
       };

       const result = await pipe.generate('The Sun is yellow because', config);
       console.log(result.toString());
     }

     main();

Efficient Text Generation via Speculative Decoding
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Speculative decoding (or assisted-generation) enables faster token generation
when an additional smaller draft model is used alongside the main model. This reduces the
number of infer requests to the main model, increasing performance.

The draft model predicts the next K tokens one by one in an autoregressive manner. The main
model validates these predictions and corrects them if necessary - in case of
a discrepancy, the main model prediction is used. Then, the draft model acquires this token and
runs prediction of the next K tokens, thus repeating the cycle.

.. tab-set::

.. tab-item:: Python
:sync: py

text
.. code-block:: python

     import openvino_genai
     import queue
     import threading

     def streamer(subword):
             print(subword, end='', flush=True)
             return False

     def infer(model_dir: str, draft_model_dir: str, prompt: str):
         main_device = 'CPU'  # GPU can be used as well.
         draft_device = 'CPU'

         scheduler_config = openvino_genai.SchedulerConfig()
         scheduler_config.cache_size = 2

         draft_model = openvino_genai.draft_model(draft_model_dir, draft_device)

         pipe = openvino_genai.LLMPipeline(model_dir, main_device, scheduler_config=scheduler_config, draft_model=draft_model)

         config = openvino_genai.GenerationConfig()
         config.max_new_tokens = 100
         config.num_assistant_tokens = 5

         pipe.generate("The Sun is yellow because", config, streamer)


  For more information, refer to the
  `Python sample <https://github.com/openvinotoolkit/openvino.genai/tree/master/samples/python/text_generation/speculative_decoding_lm/>`__.

.. tab-item:: C++
:sync: cpp

text
.. code-block:: cpp

     #include <openvino/openvino.hpp>

     #include "openvino/genai/llm_pipeline.hpp"

     int main(int argc, char* argv[]) try {
         if (4 != argc) {
             throw std::runtime_error(std::string{"Usage: "} + argv[0] + " <MODEL_DIR> <DRAFT_MODEL_DIR> ''");
         }

         ov::genai::GenerationConfig config;
         config.max_new_tokens = 100;
         config.num_assistant_tokens = 5;

         std::string main_model_path = argv[1];
         std::string draft_model_path = argv[2];
         std::string prompt = argv[3];

         std::string main_device = "CPU", draft_device = "CPU";

         ov::genai::SchedulerConfig scheduler_config;
         scheduler_config.cache_size = 5;

         ov::genai::LLMPipeline pipe(
             main_model_path,
             main_device,
             ov::genai::draft_model(draft_model_path, draft_device),
             ov::genai::scheduler_config(scheduler_config));

         auto streamer = [](std::string subword) {
             std::cout << subword << std::flush;
             return false;
         };

         pipe.generate("The Sun is yellow because", config, streamer);
     } catch (const std::exception& error) {
         try {
             std::cerr << error.what() << '\n';
         } catch (const std::ios_base::failure&) {}
         return EXIT_FAILURE;
     } catch (...) {
         try {
             std::cerr << "Non-exception object thrown\n";
         } catch (const std::ios_base::failure&) {}
         return EXIT_FAILURE;
     }


  For more information, refer to the
  `C++ sample <https://github.com/openvinotoolkit/openvino.genai/tree/master/samples/cpp/text_generation/speculative_decoding_lm/>`__

Inference of GGUF (GGML Unified Format) models
###############################################################################################

Some language models on Hugging Face are distributed in the GGUF (GGML Unified Format) and can
be downloaded. You can browse all available
GGUF models on Hugging Face <https://huggingface.co/models?library=gguf>__.
A GGUF model is encapsulated in a single binary file that contains all necessary components, including metadata and model weights, to represent the entire LLM pipeline.
Once downloaded, these GGUF models can be used directly with OpenVINO GenAI (for supported architectures) without additional conversion steps.

Unlike standard Hugging Face models, GGUF models do not require conversion to OpenVINO Intermediate Representation (IR) using the optimum-intel tool.
The LLMPipeline object can be instantiated directly from a GGUF file, enabling seamless inference without intermediate steps.

This capability is currently available in preview mode and supports a limited set of topologies, including SmolLM, Qwen2.5.
For other models and architectures, we still recommend converting the model to the IR format,
using the optimum-intel tool.
See :doc:Generative Model Preparation Using Optimum-intel <./genai-model-preparation> for more details.

To perform inference with a GGUF model using OpenVINO GenAI, simply provide the path to the .gguf file when constructing the LLMPipeline object, as shown below:

.. tab-set::

.. tab-item:: Python
:sync: py

text
.. code-block:: python

     import openvino_genai

     pipe = openvino_genai.LLMPipeline("SmolLM2-135M.F16.gguf", "CPU")

     config = openvino_genai.GenerationConfig()
     config.max_new_tokens = 100

     pipe.generate("The Sun is yellow because", config)

.. tab-item:: C++
:sync: cpp

text
.. code-block:: cpp

     #include <openvino/openvino.hpp>
     #include "openvino/genai/llm_pipeline.hpp"

     int main(int argc, char* argv[]) try {
         ov::genai::GenerationConfig config;
         config.max_new_tokens = 100;

         std::string model_path = "SmolLM2-135M.F16.gguf";
         std::string prompt = "The Sun is yellow because";

         ov::genai::LLMPipeline pipe(model_path, "CPU");

         auto result = pipe.generate("The Sun is yellow because", config);
         std::cout << "result = " << result << std::endl;
     } catch (const std::exception& error) {
         try {
             std::cerr << error.what() << '\n';
         } catch (const std::ios_base::failure&) {}
         return EXIT_FAILURE;
     } catch (...) {
         try {
             std::cerr << "Non-exception object thrown\n";
         } catch (const std::ios_base::failure&) {}
         return EXIT_FAILURE;
     }

.. tab-item:: JavaScript/TypeScript
:sync: js

text
.. code-block:: typescript

     import { LLMPipeline, GenerationConfig } from 'openvino-genai-node';

     async function main() {
       const config: GenerationConfig = {
         maxNewTokens: 100
       };

       const modelPath = 'SmolLM2-135M.F16.gguf';
       const prompt = 'The Sun is yellow because';

       const pipe = await LLMPipeline(modelPath, 'CPU');

       const result = await pipe.generate(prompt, config);
       console.log('result =', result.toString());
     }

     main();

Comparing with Hugging Face Results
#######################################

You can compare the results of the above example with those generated by Hugging Face models by
running the following code:

.. tab-set::

.. tab-item:: Python

text
.. code-block:: python

     from transformers import AutoTokenizer, AutoModelForCausalLM
     import openvino_genai as ov_genai

     tokenizer = AutoTokenizer.from_pretrained("TinyLlama/TinyLlama-1.1B-Chat-v1.0")
     model = AutoModelForCausalLM.from_pretrained("TinyLlama/TinyLlama-1.1B-Chat-v1.0")

     max_new_tokens = 32
     prompt = 'table is made of'

     encoded_prompt = tokenizer.encode(prompt, return_tensors='pt', add_special_tokens=False)
     hf_encoded_output = model.generate(encoded_prompt, max_new_tokens=max_new_tokens, do_sample=False)
     hf_output = tokenizer.decode(hf_encoded_output[0, encoded_prompt.shape[1]:])
     print(f'hf_output: {hf_output}')

     pipe = ov_genai.LLMPipeline('TinyLlama-1.1B-Chat-v1.0')
     ov_output = pipe.generate(prompt, max_new_tokens=max_new_tokens)
     print(f'ov_output: {ov_output}')

     assert hf_output == ov_output

GenAI API
#######################################

The use case described here regards the following OpenVINO GenAI API classes:

  • generation_config - defines a configuration class for text generation,
    enabling customization of the generation process such as the maximum length of
    the generated text, whether to ignore end-of-sentence tokens, and the specifics
    of the decoding strategy (greedy, beam search, or multinomial sampling).
  • llm_pipeline - provides classes and utilities for processing inputs,
    text generation, and managing outputs with configurable options.
  • streamer_base - an abstract base class for creating streamers.
  • tokenizer - the tokenizer class for text encoding and decoding.

Learn more from the GenAI API reference <https://docs.openvino.ai/2026/api/genai_api/api.html>__.

Additional Resources
####################

  • OpenVINO GenAI Repo <https://github.com/openvinotoolkit/openvino.genai>__
  • OpenVINO GenAI Documentation <https://openvinotoolkit.github.io/openvino.genai/>__
  • OpenVINO GenAI Samples <https://github.com/openvinotoolkit/openvino.genai/tree/master/samples>__
  • A Jupyter notebook demonstrating
    Visual-language assistant with MiniCPM-V2 and OpenVINO <https://github.com/openvinotoolkit/openvino_notebooks/tree/latest/notebooks/minicpm-v-multimodal-chatbot>__
  • OpenVINO Tokenizers <https://github.com/openvinotoolkit/openvino_tokenizers>__
  • Neural Network Compression Framework <https://github.com/openvinotoolkit/nncf>__

Articles En Openvino Workflow Generative Inference With Optimum Intel (`docs/articles_en/openvino-workflow-generative/inference-with-optimum-intel.rst`)

Inference with Optimum Intel

.. meta::
:description: Learn how to use the native OpenVINO package to execute LLM models.

The steps below show how to load and infer LLMs from
Hugging Face <https://huggingface.co/models>__ using Optimum Intel. They also show how to
convert models into OpenVINO IR format so they can be optimized by NNCF and used with
other OpenVINO tools.

Prerequisites
############################################################

  • Create a Python environment by following the instructions on the
    :doc:Install OpenVINO PIP <../../get-started/install-openvino> page.
  • Install the necessary dependencies for Optimum Intel:

.. code-block:: console

pip install optimum[openvino,nncf]

Loading a Hugging Face Model to Optimum Intel
############################################################

To start using OpenVINO as a backend for Hugging Face, change the original Hugging Face
code in two places:

.. code-block:: diff

-from transformers import AutoModelForCausalLM
+from optimum.intel import OVModelForCausalLM
model_id = "meta-llama/Llama-2-7b-chat-hf"
-model = AutoModelForCausalLM.from_pretrained(model_id)
+model = OVModelForCausalLM.from_pretrained(model_id, export=True)

Instead of using AutoModelForCasualLM from the Hugging Face transformers library,
switch to OVModelForCasualLM from the optimum.intel library. This change enables
you to use OpenVINO's optimization features. You may also use other AutoModel types,
such as OVModelForSeq2SeqLM, though this guide will focus on CausalLM.

By setting the parameter export=True, the model is converted to OpenVINO IR format on the fly.

It is recommended to save model to disk after conversion using save_pretrained() and
loading it from disk at deployment time via from_pretrained() for better efficiency.

.. code-block:: python

model.save_pretrained("ov_model")

This will create a new folder called ov_model with the LLM in OpenVINO IR format inside.
You can change the folder and provide another model directory instead of ov_model.

Once the model is saved, you can load it with the following command:

.. code-block:: python

model = OVModelForCausalLM.from_pretrained("ov_model")

Converting a Hugging Face Model to OpenVINO IR
############################################################

The optimum-cli tool allows you to convert models from Hugging Face to
the OpenVINO IR format:

.. code-block:: python

optimum-cli export openvino --model

If you want to convert the Llama 2 model from Hugging Face to an OpenVINO IR
model and name it ov_llama_2, the command would look like this:

.. code-block:: python

optimum-cli export openvino --model meta-llama/Llama-2-7b-chat-hf ov_llama_2

In this case, you can load the converted model in OpenVINO representation directly from the disk:

.. code-block:: python

model_id = "llama_openvino"
model = OVModelForCausalLM.from_pretrained(model_id)

Optimum-Intel API also provides out-of-the-box model optimization through weight compression
using NNCF which substantially reduces the model footprint and inference latency:

.. tab-set::

.. tab-item:: CLI
:sync: CLI

text
.. code-block:: sh

     optimum-cli export openvino --model meta-llama/Llama-2-7b-chat-hf --weight-format int8 ov_llama_2

.. tab-item:: API
:sync: API

text
.. code-block:: python

     model = OVModelForCausalLM.from_pretrained(model_id, export=True, load_in_8bit=True)

     # or if the model has been already converted
     model = OVModelForCausalLM.from_pretrained(model_path, load_in_8bit=True)

     # save the model after optimization
     model.save_pretrained(optimized_model_path)

Weight compression is applied by default to models larger than one billion parameters and is
also available for CLI interface as the --int8 option.

.. note::

8-bit weight compression is enabled by default for models larger than 1 billion parameters.

Optimum Intel <https://huggingface.co/docs/optimum/intel/inference>__ also provides 4-bit
weight compression with OVWeightQuantizationConfig class to control weight quantization
parameters.

.. tab-set::

.. tab-item:: CLI
:sync: CLI

text
.. code-block:: python

     optimum-cli export openvino --model meta-llama/Llama-2-7b-chat-hf --weight-format int4 ov_llama_2

.. tab-item:: API
:sync: API

text
.. code-block:: python

     from optimum.intel import OVModelForCausalLM, OVWeightQuantizationConfig
     import nncf

     model = OVModelForCausalLM.from_pretrained(
         model_id,
         export=True,
         quantization_config=OVWeightQuantizationConfig(bits=4),
     )

     # or if the model has been already converted
     model = OVModelForCausalLM.from_pretrained(
         model_path,
         quantization_config=OVWeightQuantizationConfig(bits=4),
     )

     # use custom parameters for weight quantization
     model = OVModelForCausalLM.from_pretrained(
         model_path,
         quantization_config=OVWeightQuantizationConfig(bits=4, asym=True, ratio=0.8, dataset="ptb"),
     )

     # save the model after optimization
     model.save_pretrained(optimized_model_path)

.. note::

Optimum-Intel has a predefined set of weight quantization parameters for popular models,
such as meta-llama/Llama-2-7b or Qwen/Qwen-7B-Chat. These parameters are used by
default only when bits=4 is specified in the config.

For more details on compression options, refer to the corresponding Optimum documentation <https://huggingface.co/docs/optimum/en/intel/openvino/optimization#4-bit>__.
For native NNCF weight quantization options, refer to the :doc:weight compression guide <../../openvino-workflow/model-optimization-guide/weight-compression>.

OpenVINO also supports 4-bit models from Hugging Face Transformers <https://github.com/huggingface/transformers>__
library optimized with GPTQ <https://github.com/PanQiWei/AutoGPTQ>__. In this case, there
is no need for an additional model optimization step because model conversion will
automatically preserve the INT4 optimization results, allowing model inference to benefit
from it.

Below are some examples of using Optimum-Intel for model conversion and inference:

  • Instruction following using Databricks Dolly 2.0 and OpenVINO <https://github.com/openvinotoolkit/openvino_notebooks/blob/latest/notebooks/dolly-2-instruction-following/dolly-2-instruction-following.ipynb>__
  • Create an LLM-powered Chatbot using OpenVINO <https://github.com/openvinotoolkit/openvino_notebooks/blob/latest/notebooks/llm-chatbot/llm-chatbot.ipynb>__

.. note::

Optimum-Intel can be used for other generative AI models. See
Stable Diffusion v2.1 using Optimum-Intel OpenVINO <https://github.com/openvinotoolkit/openvino_notebooks/blob/latest/notebooks/stable-diffusion-v2/stable-diffusion-v2-optimum-demo.ipynb>__
and
Image generation with Stable Diffusion XL and OpenVINO <https://github.com/openvinotoolkit/openvino_notebooks/blob/latest/notebooks/stable-diffusion-xl/stable-diffusion-xl.ipynb>__
for more examples.

Inference Example
############################################################

For Hugging Face models, the AutoTokenizer and the pipeline function are used to create
an inference pipeline. This setup allows for easy text processing and model interaction:

.. code-block:: python

from optimum.intel import OVModelForCausalLM

new imports for inference

from transformers import AutoTokenizer

load the model

model_id = "meta-llama/Llama-2-7b-chat-hf"
model = OVModelForCausalLM.from_pretrained(model_id, export=True)

inference

prompt = "The weather is:"
tokenizer = AutoTokenizer.from_pretrained(model_id)
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=50)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

.. note::

Converting LLMs on the fly every time to OpenVINO IR is a resource intensive task.
It is a good practice to convert the model once, save it in a folder and load it for
inference.

By default, inference will run on CPU. To select a different inference device, for example, GPU,
add device="GPU" to the from_pretrained() call. To switch to a different device after
the model has been loaded, use the .to() method. The device naming convention is the same
as in OpenVINO native API:

.. code-block:: python

model.to("GPU")

.. _enabling-runtime-optimizations:

Enabling OpenVINO Runtime Optimizations
############################################################

OpenVINO runtime provides a set of optimizations for more efficient LLM inference. This
includes Dynamic quantization of activations of 4/8-bit quantized MatMuls and
KV-cache quantization.

  • Dynamic quantization enables quantization of activations of MatMul operations
    that have 4 or 8-bit quantized weights (see
    :doc:LLM Weight Compression <../../openvino-workflow/model-optimization-guide/weight-compression>).
    It improves inference latency and throughput of LLMs, though it may cause
    insignificant deviation in generation accuracy. Quantization is performed in a group-wise
    manner, with configurable group size. It means that values in a group share quantization
    parameters. Larger group sizes lead to faster inference but lower accuracy. Recommended
    group size values are 0, 32, 64, 128 or -1(per-token).

    .. note::

    The dynamic quantization group size is treated as a guideline rather than a strict requirement.
    The actual policy may vary depending on functional capabilities or performance and accuracy considerations.
    The plugin may choose to disable dynamic quantization entirely or use a smaller group size than the one
    specified by the user.

    On Intel CPU and Intel GPU, dynamic quantization is enabled by default.

    • For Intel CPU and Intel GPU without XMX support, the default group size is 32.
    • For Intel GPU with XMX support, the only supported group size is innermost axis and
      it is turned on by default.

    To disable dynamic quantization you can either:

    • (Primary Option) Set DYNAMIC_QUANTIZATION_GROUP_SIZE to the 0 value.
    • Switch execution mode from the PERFORMANCE mode to the ACCURACY mode. However, this
      option affects inference precision as well. You can learn more about both: PERFORMANCE
      and ACCURACY modes by following the :ref:Precision Control Guide <execution-mode>.

    To change a group size value (e.g. to 64), you need to execute the following code:

    .. code-block:: python

    model = OVModelForCausalLM.from_pretrained(
    model_path,
    ov_config={"DYNAMIC_QUANTIZATION_GROUP_SIZE": "64"}
    )

    .. note::

    As of release 2024.3, dynamic quantization is not enabled for BF16 inference.

  • KV-cache quantization allows lowering the precision of Key and Value cache in LLMs.
    This helps reduce memory consumption during inference, improving latency and throughput.
    KV-cache can be quantized into the following precisions: u8, u4, bf16, f16.
    For u8 or u4, group-wise quantization is used; the group sizes are controlled by
    KEY_CACHE_GROUP_SIZE and VALUE_CACHE_GROUP_SIZE. Keys and Values precisions
    can be configured independently using KEY_CACHE_PRECISION and VALUE_CACHE_PRECISION.
    Key quantization mode is set with KEY_CACHE_QUANT_MODE to either BY_CHANNEL (more accurate)
    or BY_TOKEN.

    Default KV-cache quantization settings for CPU and GPU:

    • U8 Keys: channel-wise quantization, group_size = 32 (CPU) / 16 (GPU);
    • U8 Values: token-wise quantization, group_size = hidden_dim.

    Use the following settings to enable KV-cache quantization:

    .. code-block:: python

    model = OVModelForCausalLM.from_pretrained(
    model_path,
    ov_config={"KV_CACHE_PRECISION": "u8", "DYNAMIC_QUANTIZATION_GROUP_SIZE": "32", "PERFORMANCE_HINT": "LATENCY"}
    )

    .. note::
    Currently, for KV-cache quantization, GPU ignores the DYNAMIC_QUANTIZATION_GROUP_SIZE property, using group_size = head_size. Additionally, it does not support the get_state() and set_state() APIs when KV-cache quantization is enabled.

    For GPU, KV-cache quantization is enabled by default on platforms without XMX support, and can be disabled by setting KV_CACHE_PRECISION to undefined.

Working with Models Tuned with LoRA
#########################################

Low-rank Adaptation (LoRA) is a popular method to tune Generative AI models to a downstream
task or custom data. However, it requires some extra steps to be done for efficient deployment
using the Hugging Face API. Namely, the trained adapters should be fused into the baseline
model to avoid extra computation. This is how it can be done for LLMs:

.. code-block:: python

model_id = "meta-llama/Llama-2-7b-chat-hf"
lora_adaptor = "./lora_adaptor"
model = AutoModelForCausalLM.from_pretrained(model_id, use_cache=True)
model = PeftModelForCausalLM.from_pretrained(model, lora_adaptor)
model.merge_and_unload()
model.get_base_model().save_pretrained("fused_lora_model")

Now the model can be converted to OpenVINO using Optimum Intel Python API or CLI interfaces
mentioned above.

Execution on CPU device
##########################

As mentioned in the :ref:Composability of different threading runtimes <Composability_of_different_threading_runtimes> section, OpenVINO's default threading runtime,
oneTBB, keeps CPU cores active for a while after inference is done. When using Optimum Intel Python API,
it calls Torch (via HF transformers) for postprocessing, such as beam search or gready search.
Torch uses OpenMP for threading, OpenMP needs to wait for CPU cores that are kept active by
oneTBB. By default, OpenMP uses the busy-wait <https://gcc.gnu.org/onlinedocs/libgomp/GOMP_005fSPINCOUNT.html>__ which can delay the next OpenVINO inference as well.

It is recommended to:

  • Limit the number of CPU threads used by Torch with torch.set_num_threads <https://pytorch.org/docs/stable/generated/torch.set_num_threads.html>__.
  • Set the environment variable OMP_WAIT_POLICY <https://gcc.gnu.org/onlinedocs/libgomp/OMP_005fWAIT_005fPOLICY.html>__ to PASSIVE, which disables OpenMP busy-wait <https://gcc.gnu.org/onlinedocs/libgomp/GOMP_005fSPINCOUNT.html>__.

Additional Resources
#####################

  • Optimum Intel documentation <https://huggingface.co/docs/optimum/intel/inference>__
  • :doc:LLM Weight Compression <../../openvino-workflow/model-optimization-guide/weight-compression>
  • Neural Network Compression Framework <https://github.com/openvinotoolkit/nncf>__
  • Hugging Face Transformers <https://huggingface.co/docs/transformers/index>__
  • Generation with LLMs <https://huggingface.co/docs/transformers/llm_tutorial>__
  • Pipeline class <https://huggingface.co/docs/transformers/main_classes/pipelines>__
  • GenAI Pipeline Repository <https://github.com/openvinotoolkit/openvino.genai>__
  • OpenVINO Tokenizers <https://github.com/openvinotoolkit/openvino_tokenizers>__

Articles En Openvino Workflow Generative Ov Tokenizers (`docs/articles_en/openvino-workflow-generative/ov-tokenizers.rst`)

OpenVINO Tokenizers

Tokenization is a necessary step in text processing using various models, including text
generation with LLMs. Tokenizers convert the input text into a sequence of tokens with
corresponding IDs, so that the model can understand and process it during inference. The
transformation of a sequence of numbers into a string is called detokenization.

.. image:: ../assets/images/tokenization.svg
:align: center

There are two important points in the tokenizer-model relation:

  • Every model with text input is paired with a tokenizer and cannot be used without it.
  • To reproduce the model accuracy on a specific task, it is essential to use the same tokenizer employed during the model training.

OpenVINO Tokenizers is an OpenVINO extension and a Python library designed to streamline
tokenizer conversion for seamless integration into your project. With OpenVINO Tokenizers you can:

  • Add text processing operations to OpenVINO. Both tokenizer and detokenizer are OpenVINO models, meaning that you can work with them as with any model: read, compile, save, etc.

  • Perform tokenization and detokenization without third-party dependencies.

  • Convert Hugging Face tokenizers into OpenVINO tokenizer and detokenizer for efficient deployment across different environments. See the conversion example <https://github.com/openvinotoolkit/openvino_tokenizers?tab=readme-ov-file#convert-huggingface-tokenizer>__ for more details.

  • Combine OpenVINO models into a single model. Recommended for specific models, like classifiers or RAG Embedders, where both tokenizer and a model are used once in each pipeline inference. For more information, see the OpenVINO Tokenizers Notebook <https://github.com/openvinotoolkit/openvino_notebooks/tree/latest/notebooks/openvino-tokenizers>__.

  • Add greedy decoding pipeline to text generation models.

  • Use TensorFlow models, such as TensorFlow Text MUSE model. See the MUSE model inference example <https://github.com/openvinotoolkit/openvino_tokenizers?tab=readme-ov-file#tensorflow-text-integration>__ for detailed instructions. Note that TensorFlow integration requires additional conversion extensions to work with string tensor operations like StringSplit, StaticRexexpReplace, StringLower, and others.

.. note::

OpenVINO Tokenizers can be inferred only on a CPU device.

Supported Tokenizers
#####################

.. list-table::
:widths: 30 25 20 20
:header-rows: 1

    • Hugging Face Tokenizer Type
    • Tokenizer Model Type
    • Tokenizer
    • Detokenizer
    • Fast
    • WordPiece
    • Yes
    • No
    • BPE
    • Yes
    • Yes
    • Unigram
    • No
    • No
    • Legacy
    • SentencePiece .model
    • Yes
    • Yes
    • Custom
    • tiktoken
    • Yes
    • Yes
    • RWKV
    • Trie
    • Yes
    • Yes

.. note::

The outputs of the converted and the original tokenizer may differ, either decreasing or increasing
model accuracy on a specific task. You can modify the prompt to mitigate these changes.
In the OpenVINO Tokenizers repository <https://github.com/openvinotoolkit/openvino_tokenizers>__
you can find the percentage of tests where the outputs of the original and converted tokenizer/detokenizer match.

Python Installation
###################

  1. Create and activate a virtual environment.

    .. code-block:: python

    python3 -m venv venv

    source venv/bin/activate

  2. Install OpenVINO Tokenizers.

    Installation options include using a converted OpenVINO tokenizer, converting a Hugging Face tokenizer
    into an OpenVINO tokenizer, installing a pre-release version to experiment with latest changes,
    or building and installing from source. You can also install OpenVINO Tokenizers with Conda distribution.
    Check the OpenVINO Tokenizers repository <https://github.com/openvinotoolkit/openvino_tokenizers.git>__ for more information.

    .. tab-set::

    .. tab-item:: Converted OpenVINO tokenizer

    text
    .. code-block:: python

     pip install openvino-tokenizers

    .. tab-item:: Hugging Face tokenizer

    text
    .. code-block:: python

     pip install openvino-tokenizers[transformers]

    .. tab-item:: Pre-release version

    text
    .. code-block:: python

     pip install --pre -U openvino openvino-tokenizers --extra-index-url https://storage.openvinotoolkit.org/simple/wheels/nightly

    .. tab-item:: Build from source

    text
    .. code-block:: python

     source path/to/installed/openvino/setupvars.sh

           git clone https://github.com/openvinotoolkit/openvino_tokenizers.git

     cd openvino_tokenizers

     pip install --no-deps .

C++ Installation
################

You can use converted tokenizers in C++ pipelines with prebuild binaries.

  1. Download :doc:OpenVINO archive distribution <../../get-started/install-openvino> for your OS and extract the archive.

  2. Download OpenVINO Tokenizers prebuild libraries <https://storage.openvinotoolkit.org/repositories/openvino_tokenizers/packages/2026.3.1.0/>__. To ensure compatibility, the first three numbers of the OpenVINO Tokenizers version should match the OpenVINO version and OS.

  3. Extract OpenVINO Tokenizers archive into the OpenVINO installation directory:

    .. tab-set::

    .. tab-item:: Linux_x86

    text
    .. code-block:: sh

     <openvino_dir>/runtime/lib/intel64/

    .. tab-item:: Linux_arm64

    text
    .. code-block:: sh

     <openvino_dir>/runtime/lib/aarch64/

    .. tab-item:: Windows

    text
    .. code-block:: sh

     <openvino_dir>\runtime\bin\intel64\Release\

    .. tab-item:: MacOS_arm64

    text
    .. code-block:: sh

     <openvino_dir>/runtime/lib/arm64/Release/

    After that, you can add the binary extension to the code:

    .. tab-set::

    .. tab-item:: Linux

    text
    .. code-block:: sh

     core.add_extension("libopenvino_tokenizers.so")

    .. tab-item:: Windows

    text
    .. code-block:: sh

     core.add_extension("openvino_tokenizers.dll")

    .. tab-item:: MacOS

    text
    .. code-block:: sh

     core.add_extension("libopenvino_tokenizers.dylib")

    If you use the 2023.3.0.0 version, the binary extension file is called (lib)user_ov_extension.(dll/dylib/so).

You can learn how to read and compile converted models in the
:doc:Model Preparation <../../openvino-workflow/model-preparation> guide.

Tokenizers Usage
################

  1. Convert a Tokenizer to OpenVINO Intermediate Representation (IR)
    +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

You can convert Hugging Face tokenizers to IR using either a CLI tool bundled with Tokenizers or
Python API. Skip this step if you have a converted OpenVINO tokenizer.

Install dependencies:

.. code-block:: python

pip install openvino-tokenizers[transformers]

Convert Tokenizers:

.. tab-set::

.. tab-item:: CLI

text
.. code-block:: sh

     !convert_tokenizer $model_id --with-detokenizer -o tokenizer

  Compile the converted model to use the tokenizer:

  .. code-block:: sh

     from pathlib import Path
     import openvino_tokenizers
     from openvino import Core


     tokenizer_dir = Path("tokenizer/")
     core = Core()
     ov_tokenizer = core.read_model(tokenizer_dir / "openvino_tokenizer.xml")
     ov_detokenizer = core.read_model(tokenizer_dir / "openvino_detokenizer.xml")

     tokenizer, detokenizer = core.compile_model(ov_tokenizer), core.compile_model(ov_detokenizer)

.. tab-item:: Python API

text
.. code-block:: python

     from transformers import AutoTokenizer
     from openvino_tokenizers import convert_tokenizer

     hf_tokenizer = AutoTokenizer.from_pretrained(model_id)
     ov_tokenizer, ov_detokenizer = convert_tokenizer(hf_tokenizer, with_detokenizer=True)

  Use ``save_model`` to reuse converted tokenizers later:

  .. code-block:: python

     from pathlib import Path
     from openvino import save_model

     tokenizer_dir = Path("tokenizer/")
     save_model(ov_tokenizer, tokenizer_dir / "openvino_tokenizer.xml")
     save_model(ov_detokenizer, tokenizer_dir / "openvino_detokenizer.xml")

  Compile the converted model to use the tokenizer:

  .. code-block:: python

     from openvino import compile_model

     tokenizer, detokenizer = compile_model(ov_tokenizer), compile_model(ov_detokenizer)

The result is two OpenVINO models: ov_tokenizer and ov_detokenizer.
You can find more information and code snippets in the OpenVINO Tokenizers Notebook <https://github.com/openvinotoolkit/openvino_notebooks/tree/latest/notebooks/openvino-tokenizers>__.

  1. Tokenize and Prepare Inputs
    +++++++++++++++++++++++++++++++

.. code-block:: python

import numpy as np

text_input = ["Quick brown fox jumped"]

model_input = {name.any_name: output for name, output in tokenizer(text_input).items()}

if "position_ids" in (input.any_name for input in infer_request.model_inputs):
model_input["position_ids"] = np.arange(model_input["input_ids"].shape[1], dtype=np.int64)[np.newaxis, :]

no beam search, set idx to 0

model_input["beam_idx"] = np.array([0], dtype=np.int32)

end of sentence token is where the model signifies the end of text generation

read EOS token ID from rt_info of tokenizer/detokenizer ov.Model object

eos_token = ov_tokenizer.get_rt_info(EOS_TOKEN_ID_NAME).value

  1. Generate Text
    +++++++++++++++++++++++++++

.. code-block:: python

tokens_result = np.array([[]], dtype=np.int64)

reset KV cache inside the model before inference

infer_request.reset_state()
max_infer = 10

for _ in range(max_infer):
infer_request.start_async(model_input)
infer_request.wait()

text
# get a prediction for the last token on the first inference
  output_token = infer_request.get_output_tensor().data[:, -1:]
  tokens_result = np.hstack((tokens_result, output_token))
  if output_token[0, 0] == eos_token:
     break

  # prepare input for new inference
  model_input["input_ids"] = output_token
  model_input["attention_mask"] = np.hstack((model_input["attention_mask"].data, [[1]]))
  model_input["position_ids"] = np.hstack(
     (model_input["position_ids"].data, [[model_input["position_ids"].data.shape[-1]]])
  )
  1. Detokenize Output
    +++++++++++++++++++++++++++++

.. code-block:: python

text_result = detokenizer(tokens_result)["string_output"]
print(f"Prompt:\n{text_input[0]}")
print(f"Generated:\n{text_result[0]}")

Additional Resources
####################

  • OpenVINO Tokenizers repo <https://github.com/openvinotoolkit/openvino_tokenizers>__
  • OpenVINO Tokenizers Notebook <https://github.com/openvinotoolkit/openvino_notebooks/tree/latest/notebooks/openvino-tokenizers>__
  • Text generation C++ samples that support most popular models like LLaMA 3 <https://github.com/openvinotoolkit/openvino.genai/tree/master/samples/cpp/text_generation>__
  • OpenVINO GenAI Repo <https://github.com/openvinotoolkit/openvino.genai>__

Articles En Openvino Workflow Deployment Locally (`docs/articles_en/openvino-workflow/deployment-locally.rst`)

Deploy Locally

.. toctree::
:maxdepth: 1
:hidden:

Local Distribution Libraries <./deployment-locally/local-distribution-libraries>
Optimize Binaries Size <./deployment-locally/optimial-binary-size-conditional-compilation>
Integrate OpenVINO with Ubuntu Snap <./deployment-locally/integrate-openvino-with-ubuntu-snap>

.. meta::
:description: There are several ways of deploying OpenVINO™ application once
its development has been finished.

.. note::

Note that :doc:running inference in OpenVINO Runtime <running-inference> is the most basic form of deployment. Before moving forward, make sure you know how to create a proper Inference configuration and :doc:develop your application properly <running-inference>.

Local Deployment Options
########################

  • Set a dependency on the existing prebuilt packages, also called "centralized distribution":

    • using Debian / RPM packages - a recommended way for Linux operating systems;

    • using PIP package manager on PyPI - the default approach for Python-based applications;

    • using Docker images - if the application should be deployed as a Docker image, use a pre-built OpenVINO™ Runtime Docker image as a base image in the Dockerfile for the application container image. For more information about OpenVINO Docker images, refer to :doc:Installing OpenVINO from Docker <../get-started/install-openvino/install-openvino-docker-linux>

      • Furthermore, to customize your OpenVINO Docker image, use the Docker CI Framework <https://github.com/openvinotoolkit/docker_ci>__ to generate a Dockerfile and build the image.

  • Grab a necessary functionality of OpenVINO together with your application, also called "local distribution":

    • using the :doc:local distribution <deployment-locally/local-distribution-libraries> approach;
    • using a static version of OpenVINO Runtime linked to the final app <https://github.com/openvinotoolkit/openvino/blob/master/docs/dev/static_libraries.md>__.

The table below shows which distribution type can be used for what target operating system:

.. list-table::
:header-rows: 1

    • Distribution type
    • Operating systems
    • Debian packages
    • 20.04, 22.04, 24.04 (64-bit)
    • RPM packages
    • Red Hat Enterprise Linux 8, 64-bit
    • Docker images
    • Ubuntu 20.04, 22.04, 24.04 (64-bit); Red Hat Enterprise Linux 8, 64-bit
    • :doc:Libraries for Local Distribution <deployment-locally/local-distribution-libraries>
    • All operating systems
    • Build OpenVINO statically and link to the final app <https://github.com/openvinotoolkit/openvino/blob/master/docs/dev/static_libraries.md>__
    • All operating systems

Granularity of Major Distribution Types
#######################################

The granularity of OpenVINO packages may vary for different distribution types. For example, the PyPI distribution of OpenVINO has a single 'openvino' package <https://pypi.org/project/openvino/>__ that contains all the runtime libraries and plugins, while a :doc:local distribution <deployment-locally/local-distribution-libraries> is a more configurable type providing higher granularity. Below are important details of the set of libraries included in the OpenVINO Runtime package:

.. image:: ../assets/images/deployment_simplified.svg

  • The main library openvino is used by users' C++ applications to link against with. For C language applications, openvino_c is additionally required for distribution. The library includes OpenVINO API 2.0.
  • The "optional" plugin libraries like openvino_intel_cpu_plugin (matching the openvino_.+_plugin pattern) are used to provide inference capabilities on specific devices or additional capabilities like :doc:Hetero Execution <running-inference/inference-devices-and-modes/hetero-execution>.
  • The "optional" plugin libraries like openvino_ir_frontend (matching openvino_.+_frontend) are used to provide capabilities to read models of different file formats such as OpenVINO IR, TensorFlow, ONNX, and PaddlePaddle.

Here the term "optional" means that if the application does not use the capability enabled by the plugin, the plugin library or a package with the plugin is not needed in the final distribution.

Building a local distribution will require more detailed information, and you will find it in the dedicated :doc:Libraries for Local Distribution <deployment-locally/local-distribution-libraries> article.

.. note::

Depending on your target OpenVINO devices, the following configuration might be needed for deployed machines: :doc:Configurations for GPU <../get-started/install-openvino/configurations/configurations-intel-gpu>.

Articles En Openvino Workflow Model Optimization (`docs/articles_en/openvino-workflow/model-optimization.rst`)

Model Optimization - NNCF

.. meta::
:description: Learn about the optimization methods offered by OpenVINO's NNCF

.. toctree::
:maxdepth: 1
:hidden:

model-optimization-guide/weight-compression
model-optimization-guide/quantizing-models-post-training
model-optimization-guide/compressing-models-during-training

Model optimization means altering the model itself to improve its performance and reduce
its size. It is an optional step, typically used only at the development stage, so that
a pre-optimized model is used in the final AI application.

In OpenVINO, the default optimization tool is NNCF (Neural Network Compression Framework).
It is a set of compression algorithms <https://github.com/openvinotoolkit/nncf/blob/develop/README.md>__,
organized as a Python package, that make your models smaller and faster. Note that NNCF
is not part of the OpenVINO package, so it needs to be installed separately. It supports
models in OpenVINO IR, PyTorch and ONNX formats, offering
the following main optimizations:

.. image:: ../assets/images/WHAT_TO_USE.svg

| :doc:Weight Compression <model-optimization-guide/weight-compression>:
| An easy-to-use method for Large Language Model footprint reduction and inference
acceleration.

| :doc:Post-training Quantization <model-optimization-guide/quantizing-models-post-training>:
| Designed to optimize deep learning models by applying 8-bit integer quantization. Being
the easiest way to optimize a model it does not require its retraining or fine-tuning
but may result in a drop in accuracy. If the accuracy-performance tradeoff is not
acceptable, Training-time Optimization may be a better option.

| :doc:Training-time Optimization <model-optimization-guide/compressing-models-during-training>:
| Involves a suite of advanced methods such as Sparsity, as well
as Quantization-aware Training. This kind of optimization requires the use of the PyTorch framework.

Recommended workflows
##########################

  • A common approach for most cases is to:

    1. Perform post-training quantization first, as it is the easiest option.
    2. If the accuracy drop is unacceptable, use quantization-aware training instead. It will give
      you the same level of performance boost, with a smaller impact on accuracy.

  • Weight compression works with LLMs, VLMs and other Transformer-based models.

.. image:: ../assets/images/DEVELOPMENT_FLOW_V3_crunch.svg

Installation and usage
###########################

To learn about the full scope of the framework, its installation, and technical details, visit
both the NNCF repository <https://github.com/openvinotoolkit/nncf?tab=readme-ov-file>__ and
NNCF API documentation <https://openvinotoolkit.github.io/nncf/autoapi/nncf/>__.

.. tab-set::

.. tab-item:: Installation
:sync: install

text
.. tab-set::

     .. tab-item:: PyPI
        :sync: pip

        .. code-block::

           pip install nncf

     .. tab-item:: Conda
        :sync: conda

        .. code-block::

           conda install -c conda-forge nncf

  For more installation details, see the page on
  `NNCF Installation <https://github.com/openvinotoolkit/nncf/blob/develop/docs/Installation.md>`__.

.. tab-item:: System Requirements
:sync: sys-req

text
Full requirement listing is available in the
  `NNCF GitHub Repository <https://github.com/openvinotoolkit/nncf?tab=readme-ov-file#system-requirements>`__

  Note that to optimize a model, you will need to install this model's framework as well.
  Install NNCF in the same Python environment as the framework. For a list of recommended
  framework versions, see the
  `framework compatibility table <https://github.com/openvinotoolkit/nncf/blob/develop/docs/Installation.md#corresponding-versions>`__.

.. note::

Once optimized, models may be executed with the typical OpenVINO inference workflow,
no additional changes to the inference code are required.

This is true for models optimized using NNCF, as well as those pre-optimized in their source
frameworks, such as PyTorch, and ONNX (in Q/DQ; Quantize/DeQuantize format).
The latter may be easily converted to the
:doc:OpenVINO Intermediate Representation format (IR) <../../documentation/openvino-ir-format>
right away.

Hugging Face Optimum Intel <https://huggingface.co/docs/optimum/intel/optimization_ov>__
offers OpenVINO integration with Hugging Face models and pipelines. NNCF serves as the compression
backend within the Hugging Face Optimum Intel, integrating with the widely used transformers
library to enhance model performance.

Additional Resources
#######################

  • NNCF GitHub repository <https://github.com/openvinotoolkit/nncf>__
  • NNCF Architecture <https://github.com/openvinotoolkit/nncf/blob/develop/docs/NNCFArchitecture.md>__
  • NNCF Tutorials <https://github.com/openvinotoolkit/nncf?tab=readme-ov-file#demos-tutorials-and-samples>__
  • NNCF Compressed Model Zoo <https://github.com/openvinotoolkit/nncf/blob/develop/docs/ModelZoo.md>__
  • :doc:Deployment optimization <running-inference/optimize-inference>

Articles En Openvino Workflow Model Preparation (`docs/articles_en/openvino-workflow/model-preparation.rst`)

Conventional Model Preparation

.. meta::
:description: Preparing models for OpenVINO Runtime. Learn about the methods
used to read, convert and compile models from different frameworks.

.. toctree::
:maxdepth: 1
:hidden:

Convert to OpenVINO Model <model-preparation/convert-model-to-ir>
Conversion Parameters <model-preparation/conversion-parameters>
Setting Input Shapes <model-preparation/setting-input-shapes>

OpenVINO supports the following model formats:

  • PyTorch,
  • TensorFlow,
  • TensorFlow Lite,
  • ONNX,
  • PaddlePaddle,
  • JAX/Flax (experimental feature)
  • OpenVINO IR.

The easiest way to obtain a model is to download it from an online database, such as
Kaggle <https://www.kaggle.com/models>, Hugging Face <https://huggingface.co/>, and
Torchvision models <https://pytorch.org/hub/>__. Now you have two options:

  • Skip model conversion and :doc:run inference <running-inference>
    directly from the TensorFlow, TensorFlow Lite, ONNX, or PaddlePaddle source format.
    Conversion will still be performed but it will happen automatically and "under the hood".
    This option, while convenient, offers lower performance and stability, as well as fewer
    optimization options.

    For PyTorch models, you can use the dedicated :doc:torch.compile <./torch-compile>
    implementation.

  • Explicitly :doc:convert the model to OpenVINO IR <model-preparation/convert-model-to-ir>.
    This approach offers the best possible results and is the recommended one, especially for
    production-ready solutions. Consider storing your model in this format to minimize
    first-inference latency, perform model optimizations, and save space on your drive, in some
    cases. Explicit conversion can be done in two ways:

    • the Python API functions <#convert-a-model-with-python-convert-model>__
      (openvino.convert_model and openvino.save_model)
    • the ovc <#convert-a-model-in-cli-ovc>__ command line tool.

    Once saved as :doc:OpenVINO IR <model-preparation/convert-model-to-ir>
    (a set of .xml and .bin files), the model may be deployed with maximum performance.
    Because it is already optimized for
    :doc:OpenVINO inference <running-inference>,
    it can be read, compiled, and inferred with no additional delay.

.. note::

For PyTorch and JAX/Flax models, Python API <#convert-a-model-with-python-convert-model>__ is the only
conversion option.

Different model representations
##############################################

A model in OpenVINO can be represented in three ways: saved on disk, loaded but not compiled
(ov.Model), and loaded and compiled (ov.CompiledModel).

| Saved on disk
| One or more files saved on a drive, fully representing the neural network.
Different model formats are stored in different ways, for example:
| OpenVINO IR: pair of .xml and .bin files
| ONNX: .onnx file
| TensorFlow: directory with a .pb file and two subfolders or just a .pb file
| TensorFlow Lite: .tflite file
| PaddlePaddle: .pdmodel file

| Loaded but not compiled
| A model object (ov.Model) is created in memory either by parsing a file or converting
an existing framework object. Inference cannot be done with this object yet as it is not
attached to any specific device, but it allows customization such as reshaping its input,
applying quantization or even adding preprocessing steps before compiling the model.

| Loaded and compiled
| This representation is achieved when one or more devices are specified for a model object to
run on (ov.CompiledModel), allowing device optimizations to be made and enabling
inference.

For more information on each function, see the :doc:OpenVINO workflow <../openvino-workflow>
page.

Convert a Model with Python: convert_model
##############################################

Model Conversion API in Python uses the openvino.convert_model function, to turn a model
into the openvino.Model object and load it to memory. Now it can be: saved to a drive with openvino.save_model
or further :doc:optimized with NNCF <model-optimization> before saving.

See how to use openvino.convert_model with models from some of the most popular
public repositories:

.. tab-set::

.. tab-item:: Torchvision

text
.. code-block:: py
     :force:

        import openvino as ov
        import torch
        from torchvision.models import resnet50

        model = resnet50(weights='DEFAULT')

        # prepare input_data
        input_data = torch.rand(1, 3, 224, 224)

        ov_model = ov.convert_model(model, example_input=input_data)

        ###### Option 1: Save to OpenVINO IR:

        # save model to OpenVINO IR for later use
        ov.save_model(ov_model, 'model.xml')

        ###### Option 2: Compile and infer with OpenVINO:

        # compile model
        compiled_model = ov.compile_model(ov_model)

        # run inference
        result = compiled_model(input_data)

.. tab-item:: Hugging Face Transformers

text
.. code-block:: py

     from transformers import BertTokenizer, BertModel

     tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
     model = BertModel.from_pretrained("bert-base-uncased")
     text = "Replace me by any text you'd like."
     encoded_input = tokenizer(text, return_tensors='pt')

     import openvino as ov
     ov_model = ov.convert_model(model, example_input={**encoded_input})

     ###### Option 1: Save to OpenVINO IR:

     # save model to OpenVINO IR for later use
     ov.save_model(ov_model, 'model.xml')

     ###### Option 2: Compile and infer with OpenVINO:

     # compile model
     compiled_model = ov.compile_model(ov_model)

     # prepare input_data using HF tokenizer or your own tokenizer
     # encoded_input is reused here for simplicity

     # run inference
     result = compiled_model({**encoded_input})

.. tab-item:: Keras Applications

text
.. code-block:: py

     import tensorflow as tf
     import openvino as ov

     tf_model = tf.keras.applications.ResNet50(weights="imagenet")
     ov_model = ov.convert_model(tf_model)

     ###### Option 1: Save to OpenVINO IR:

     # save model to OpenVINO IR for later use
     ov.save_model(ov_model, 'model.xml')

     ###### Option 2: Compile and infer with OpenVINO:

     # compile model
     compiled_model = ov.compile_model(ov_model)

     # prepare input_data
     import numpy as np
     input_data = np.random.rand(1, 224, 224, 3)

     # run inference
     result = compiled_model(input_data)

.. tab-item:: TensorFlow Hub

text
.. code-block:: py

     import tensorflow as tf
     import tensorflow_hub as hub
     import openvino as ov

     model = tf.keras.Sequential([
           hub.KerasLayer("https://tfhub.dev/google/imagenet/mobilenet_v1_100_224/classification/5")
     ])

     # Check model page for information about input shape: https://tfhub.dev/google/imagenet/mobilenet_v1_100_224/classification/5
     model.build([None, 224, 224, 3])

     ov_model = ov.convert_model(model)

     ###### Option 1: Save to OpenVINO IR:

     ov.save_model(ov_model, 'model.xml')

     ###### Option 2: Compile and infer with OpenVINO:

     compiled_model = ov.compile_model(ov_model)

     # prepare input_data
     import numpy as np
     input_data = np.random.rand(1, 224, 224, 3)

     # run inference
     result = compiled_model(input_data)

.. tab-item:: ONNX Model Hub

text
.. code-block:: py

     import onnx

     model = onnx.hub.load("resnet50")
     onnx.save(model, 'resnet50.onnx')  # use a temporary file for model

     import openvino as ov
     ov_model = ov.convert_model('resnet50.onnx')

     ###### Option 1: Save to OpenVINO IR:

     # save model to OpenVINO IR for later use
     ov.save_model(ov_model, 'model.xml')

     ###### Option 2: Compile and infer with OpenVINO:

     # compile model
     compiled_model = ov.compile_model(ov_model)

     # prepare input_data
     import numpy as np
     input_data = np.random.rand(1, 3, 224, 224)

     # run inference
     result = compiled_model(input_data)

  • Saving the model, Option 1, is used as a separate step, outside of deployment.
    The file it provides is then used in the final software solution, resulting in
    maximum performance due to fewer dependencies and faster model loading.

  • Compiling the model, Option 2, provides a convenient way to quickly switch from
    framework-based code to OpenVINO-based code in your existing Python inference application.
    The converted model is not saved to IR but compiled and used for inference within the same
    application.

Before saving the model to OpenVINO IR, consider
:doc:Post-training Optimization <model-optimization-guide/quantizing-models-post-training> to achieve more efficient inference and
a smaller model.

.. _convert_model_cli_ovc:

Convert a Model in CLI: ovc
###############################

ovc is a command-line model converter, combining the openvino.convert_model
and openvino.save_model functionalities, providing the exact same results, if the same set
of parameters is used for saving into OpenVINO IR. It converts files from one of the supported
formats to
:doc:OpenVINO IR <model-preparation/convert-model-to-ir>,
which can then be read, compiled, and run by the final inference application.

.. note::
PyTorch and JAX/Flax models cannot be converted with ovc, use openvino.convert_model instead.

Additional Resources
####################

The following articles describe in detail how to obtain and prepare your model depending on
the source model type:

  • :doc:Convert different model formats to the ov.Model format <model-preparation/convert-model-to-ir>.
  • :doc:Review all available conversion parameters <model-preparation/conversion-parameters>.

To achieve the best model inference performance and more compact OpenVINO IR representation
follow:

  • :doc:Post-training optimization <model-optimization-guide/quantizing-models-post-training>
  • :doc:Model inference in OpenVINO Runtime <running-inference>

Articles En Openvino Workflow Running Inference (`docs/articles_en/openvino-workflow/running-inference.rst`)

Running and Integrating Inference Pipeline

.. meta::
:description: Learn how to implement a typical inference pipeline of OpenVINO™
Runtime in an application.⠀

.. toctree::
:maxdepth: 1
:hidden:

running-inference/model-representation
running-inference/model-input-output
running-inference/inference-request
running-inference/stateful-models
running-inference/python-api-advanced-inference
running-inference/python-api-exclusives
running-inference/inference-devices-and-modes
Optimize Inference <running-inference/optimize-inference>

OpenVINO Runtime is a set of C++ libraries with C and Python bindings, providing a common
API to run inference on various devices. Each device (integrated with OpenVINO’s plugin
architecture) offers the common, as well as hardware-specific APIs for more configuration
options. Note that OpenVINO Runtime may also be integrated with other frameworks and work
as their backend, for example, using torch.compile.
The scheme below illustrates the typical workflow for deploying a trained deep learning
model in an application:

.. image:: ../assets/images/IMPLEMENT_PIPELINE_with_API_C.svg

This guide will show you how to implement a typical OpenVINO™ Runtime inference pipeline
in your application. Before proceeding, check how
:doc:model conversion <model-preparation/convert-model-to-ir>
works in OpenVINO and how it may affect your applications’ performance. Make sure you have
installed OpenVINO Runtime and set environment variables (otherwise, the find_package
calls will not find OpenVINO_DIR):

.. tab-set::

.. tab-item:: Linux
:sync: lin

text
.. code-block:: console

     /setupvars.sh

.. tab-item:: Windows
:sync: win

text
PowerShell:

  .. code-block:: console

     /setupvars.sh

  Command Prompt

  .. code-block:: console

     cd  
     setupvars.bat

Step 1. Create OpenVINO Runtime Core
###############################################################################################

Initiate working with OpenVINO in your application by including the OpenVINO™ Runtime
components:

.. tab-set::

text
.. tab-item:: Python
   :sync: py

   .. doxygensnippet:: docs/snippets/src/main.py
      :language: python
      :fragment: [import]

   .. doxygensnippet:: docs/snippets/src/main.py
      :language: python
      :fragment: [part1]

.. tab-item:: C++
   :sync: cpp

   .. doxygensnippet:: docs/snippets/src/main.cpp
      :language: cpp
      :fragment: [include]

   .. doxygensnippet:: docs/snippets/src/main.cpp
      :language: cpp
      :fragment: [part1]

.. tab-item:: C
   :sync: c

   .. doxygensnippet:: docs/snippets/src/main.c
      :language: cpp
      :fragment: [include]

   .. doxygensnippet:: docs/snippets/src/main.c
      :language: cpp
      :fragment: [part1]

Step 2. Compile the Model
###############################################################################################

Compile the model with ov::Core::compile_model(), defining the device or mode to use
for inference. The following example uses the
:doc:AUTO mode <running-inference/inference-devices-and-modes/auto-device-selection>,
which selects the device for you. To learn more about supported devices and inference modes,
see the :doc:Inference Devices and Modes <running-inference/inference-devices-and-modes>
section.

.. tab-set::

text
.. tab-item:: Python
   :sync: py

   .. tab-set::

      .. tab-item:: IR
         :sync: ir

         .. doxygensnippet:: docs/snippets/src/main.py
            :language: python
            :fragment: [part2_1]

      .. tab-item:: ONNX
         :sync: onnx

         .. doxygensnippet:: docs/snippets/src/main.py
            :language: python
            :fragment: [part2_2]

      .. tab-item:: PaddlePaddle
         :sync: paddlepaddle

         .. doxygensnippet:: docs/snippets/src/main.py
            :language: python
            :fragment: [part2_3]

      .. tab-item:: TensorFlow
         :sync: tensorflow

         .. doxygensnippet:: docs/snippets/src/main.py
            :language: python
            :fragment: [part2_4]

      .. tab-item:: TensorFlow Lite
         :sync: tensorflow_lite

         .. doxygensnippet:: docs/snippets/src/main.py
            :language: python
            :fragment: [part2_5]

      .. tab-item:: ov::Model
         :sync: openvinomodel

         .. doxygensnippet:: docs/snippets/src/main.py
            :language: python
            :fragment: [part2_6]

.. tab-item:: C++
   :sync: cpp

   .. tab-set::

      .. tab-item:: IR
         :sync: ir

         .. doxygensnippet:: docs/snippets/src/main.cpp
            :language: cpp
            :fragment: [part2_1]

      .. tab-item:: ONNX
         :sync: onnx

         .. doxygensnippet:: docs/snippets/src/main.cpp
            :language: cpp
            :fragment: [part2_2]

      .. tab-item:: PaddlePaddle
         :sync: paddlepaddle

         .. doxygensnippet:: docs/snippets/src/main.cpp
            :language: cpp
            :fragment: [part2_3]

      .. tab-item:: TensorFlow
         :sync: tensorflow

         .. doxygensnippet:: docs/snippets/src/main.cpp
            :language: cpp
            :fragment: [part2_4]

      .. tab-item:: TensorFlow Lite
         :sync: tensorflow_lite

         .. doxygensnippet:: docs/snippets/src/main.cpp
            :language: cpp
            :fragment: [part2_5]

      .. tab-item:: ov::Model
         :sync: openvinomodel

         .. doxygensnippet:: docs/snippets/src/main.cpp
            :language: cpp
            :fragment: [part2_6]

.. tab-item:: C
   :sync: c

   .. tab-set::

      .. tab-item:: IR
         :sync: ir

         .. doxygensnippet:: docs/snippets/src/main.c
            :language: cpp
            :fragment: [part2_1]

      .. tab-item:: ONNX
         :sync: onnx

         .. doxygensnippet:: docs/snippets/src/main.c
            :language: cpp
            :fragment: [part2_2]

      .. tab-item:: PaddlePaddle
         :sync: paddlepaddle

         .. doxygensnippet:: docs/snippets/src/main.c
            :language: cpp
            :fragment: [part2_3]

      .. tab-item:: TensorFlow
         :sync: tensorflow

         .. doxygensnippet:: docs/snippets/src/main.c
            :language: cpp
            :fragment: [part2_4]

      .. tab-item:: TensorFlow Lite
         :sync: tensorflow_lite

         .. doxygensnippet:: docs/snippets/src/main.c
            :language: cpp
            :fragment: [part2_5]

      .. tab-item:: ov::Model
         :sync: openvinomodel

         .. doxygensnippet:: docs/snippets/src/main.c
            :language: cpp
            :fragment: [part2_6]

The ov::CompiledModel class represents a compiled model and enables you to get
information inputs or output ports by a tensor name or index. This approach is aligned with
most frameworks. The ov::Model object represents any models inside the OpenVINO™ Runtime.
For more details, refer to
:doc:OpenVINO™ Model representation <running-inference/model-representation>.

Step 3. Create an Inference Request
###############################################################################################

Use the ov::InferRequest class methods to create an infer request. For more details,
see the article on
:doc:InferRequest <running-inference/inference-request>.

.. tab-set::

text
.. tab-item:: Python
   :sync: py

   .. doxygensnippet:: docs/snippets/src/main.py
      :language: python
      :fragment: [part3]

.. tab-item:: C++
   :sync: cpp

   .. doxygensnippet:: docs/snippets/src/main.cpp
      :language: cpp
      :fragment: [part3]

.. tab-item:: C
   :sync: c

   .. doxygensnippet:: docs/snippets/src/main.c
      :language: cpp
      :fragment: [part3]

Step 4. Set Inputs
###############################################################################################

Create ov::Tensor, you can use external memory for that , and use the
ov::InferRequest::set_input_tensor method to send this tensor to the device.
For more info on textual data as input, see the
:doc:String Tensors <running-inference/model-input-output/string-tensors> article.

.. tab-set::

text
.. tab-item:: Python
   :sync: py

   .. doxygensnippet:: docs/snippets/src/main.py
      :language: python
      :fragment: [part4]

.. tab-item:: C++
   :sync: cpp

   .. doxygensnippet:: docs/snippets/src/main.cpp
      :language: cpp
      :fragment: [part4]

.. tab-item:: C
   :sync: c

   .. doxygensnippet:: docs/snippets/src/main.c
      :language: cpp
      :fragment: [part4]

Step 5. Start Inference
###############################################################################################

Use either ov::InferRequest::start_async or ov::infer_request.infer() to start model
inference. To learn how they work, see the
:doc:OpenVINO Inference Request <running-inference/inference-request>
article. The following example uses the asynchronous option and calls
ov::InferRequest::wait to wait for the inference results.

.. tab-set::

text
.. tab-item:: Python
   :sync: py

   .. doxygensnippet:: docs/snippets/src/main.py
      :language: python
      :fragment: [part5]

.. tab-item:: C++
   :sync: cpp

   .. doxygensnippet:: docs/snippets/src/main.cpp
      :language: cpp
      :fragment: [part5]

.. tab-item:: C
   :sync: c

   .. doxygensnippet:: docs/snippets/src/main.c
      :language: cpp
      :fragment: [part5]

Step 6. Process the Inference Results
#################################################################################################

Get output tensors and process the inference results.
For more info on textual data as input, see the
:doc:String Tensors <running-inference/model-input-output/string-tensors> article.

.. tab-set::

text
.. tab-item:: Python
   :sync: py

   .. doxygensnippet:: docs/snippets/src/main.py
      :language: python
      :fragment: [part6]

.. tab-item:: C++
   :sync: cpp

   .. doxygensnippet:: docs/snippets/src/main.cpp
      :language: cpp
      :fragment: [part6]

.. tab-item:: C
   :sync: c

   .. doxygensnippet:: docs/snippets/src/main.c
      :language: cpp
      :fragment: [part6]

Step 7. [only for C] Release the allocated objects
###############################################################################################

To avoid memory leak, applications developed with the C API need to release the allocated
objects in the following order.

.. tab-set::

text
.. tab-item:: C
   :sync: c

   .. doxygensnippet:: docs/snippets/src/main.c
      :language: cpp
      :fragment: [part8]

Build Your Application
###############################################################################################

If you have integrated OpenVINO with your application, you will need to adjust your
application build process as well. Of course, there are multiple ways this stage may be
done, so you will need to choose the one best for your project. To learn about the basics of
OpenVINO build process, refer to the
documentation on GitHub <https://github.com/openvinotoolkit/openvino/blob/master/docs/dev/build.md>__.

The following example uses a C++ & C application together with CMake,
for project configuration.

  1. Create Structure for project:

    .. tab-set::

    text
    .. tab-item:: C++
   :sync: cpp

   .. doxygensnippet:: docs/snippets/src/main.cpp
      :language: cpp
      :fragment: [part7]
      :force:

.. tab-item:: C
   :sync: c

   .. doxygensnippet:: docs/snippets/src/main.c
      :language: cpp
      :fragment: [part7]
      :force:

  2. Configure the CMake build

    For details on additional CMake build options, refer to the
    CMake page <https://cmake.org/cmake/help/latest/manual/cmake.1.html#manual:cmake(1)>__.

    .. tab-set::

    text
    .. tab-item:: C++
   :sync: cpp

   .. doxygensnippet:: docs/snippets/CMakeLists.txt
      :language: cpp
      :fragment: [cmake:integration_example_cpp]

.. tab-item:: C
   :sync: c

   .. doxygensnippet:: docs/snippets/CMakeLists.txt
      :language: cpp
      :fragment: [cmake:integration_example_c]

.. tab-item:: C++ (PyPI)
  :sync: cpp

  .. doxygensnippet:: docs/snippets/CMakeLists.txt
     :language: cpp
     :fragment: [cmake:integration_example_cpp_py]

  3. Build Project

    Use CMake to build the project on your system:

    .. code-block:: sh

    cd build/
    cmake ../project
    cmake --build .

Additional Resources
####################

  • To see working implementation of the steps, check out the
    :doc:Learn OpenVINO <../get-started/learn-openvino> section, including
    OpenVINO™ Runtime API Tutorial <./../../notebooks/openvino-api-with-output.html>__.
  • Models in the OpenVINO IR format on Hugging Face <https://huggingface.co/models>__.
  • :ref:Using Encrypted Models with OpenVINO <encrypted-models>

Articles En Openvino Workflow Torch Compile (`docs/articles_en/openvino-workflow/torch-compile.rst`)

PyTorch Deployment via "torch.compile"

The torch.compile feature enables you to use OpenVINO for PyTorch-native applications.
It speeds up PyTorch code by JIT-compiling it into optimized kernels.
By default, Torch code runs in eager-mode, but with the use of torch.compile it goes
through the following steps:

  1. Graph acquisition - the model is rewritten as blocks of subgraphs that are either:

    • compiled by TorchDynamo and "flattened",
    • falling back to the eager-mode, due to unsupported Python constructs (like control-flow code).

  2. Graph lowering - all PyTorch operations are decomposed into their constituent kernels specific to the chosen backend.

  3. Graph compilation - the kernels call their corresponding low-level device-specific operations.

How to Use
####################

.. tab-set::

.. tab-item:: Image Generation

text
.. tab-set::

     .. tab-item:: Stable-Diffusion-2

        .. code-block:: py
           :force:

           import torch
           from diffusers import StableDiffusionPipeline, DPMSolverMultistepScheduler

           model_id = "stabilityai/stable-diffusion-2-1"

           # Use the DPMSolverMultistepScheduler (DPM-Solver++) scheduler here instead
           pipe = StableDiffusionPipeline.from_pretrained(model_id, torch_dtype=torch.float16)
           pipe.scheduler = DPMSolverMultistepScheduler.from_config(pipe.scheduler.config)

           + pipe.text_encoder = torch.compile(pipe.text_encoder, backend="openvino") #Optional
           + pipe.unet = torch.compile(pipe.unet, backend=“openvino”)
           + pipe.vae.decode = torch.compile(pipe.vae.decode, backend=“openvino”) #Optional

           prompt = "a photo of an astronaut riding a horse on mars"
           image = pipe(prompt).images[0]

           image.save("astronaut_rides_horse.png")


     .. tab-item:: Stable-Diffusion-3

        .. code-block:: py

           import torch
           from diffusers import StableDiffusion3Pipeline

           pipe = StableDiffusion3Pipeline.from_pretrained("stabilityai/stable-diffusion-3-medium-diffusers", torch_dtype=torch.float32)

           + pipe.transformer = torch.compile(pipe.transformer, backend="openvino")

           image = pipe(
               "A cat holding a sign that says hello world",
               negative_prompt="",
               num_inference_steps=28,
               guidance_scale=7.0,
           ).images[0]

           image.save('out.png')

     .. tab-item:: Stable-Diffusion-XL

        .. code-block:: py

           import torch
           from diffusers import UNet2DConditionModel, DiffusionPipeline, LCMScheduler

           unet = UNet2DConditionModel.from_pretrained("latent-consistency/lcm-sdxl", torch_dtype=torch.float16, variant="fp16")
           pipe = DiffusionPipeline.from_pretrained("stabilityai/stable-diffusion-xl-base-1.0", unet=unet, torch_dtype=torch.float16, variant="fp16")
           pipe.scheduler = LCMScheduler.from_config(pipe.scheduler.config)

           + pipe.text_encoder = torch.compile(pipe.text_encoder, backend="openvino") #Optional
           + pipe.unet = torch.compile(pipe.unet, backend="openvino")
           + pipe.vae.decode = torch.compile(pipe.vae.decode, backend="openvino") #Optional

           prompt = "a close-up picture of an old man standing in the rain"
           image = pipe(prompt, num_inference_steps=5, guidance_scale=8.0).images[0]
           image.save("result.png")

.. tab-item:: Text Generation

text
.. tab-set::

     .. tab-item:: Llama-3.2-1B

        .. code-block:: py

           import torch
           from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline

           model_name_or_path = "meta-llama/Llama-3.2-1B-Instruct"
           tokenizer = AutoTokenizer.from_pretrained(model_name_or_path, trust_remote_code=True, torch_dtype=torch.float32)
           model = AutoModelForCausalLM.from_pretrained(
               model_name_or_path,
               trust_remote_code=True,
               device_map='cpu',
               torch_dtype=torch.float32
           )

           prompt = "Tell me about AI"

           + model.forward = torch.compile(model.forward, backend="openvino", options={'aot_autograd': True})

           pipe = pipeline(
               "text-generation",
               model=model,
               tokenizer=tokenizer,
               max_new_tokens=64
           )
           result = pipe(prompt)
           print(result[0]['generated_text'])


     .. tab-item:: Llama-2-7B-GPTQ

        .. code-block:: py

           import torch
           from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline

           model_name_or_path = "TheBloke/Llama-2-7B-GPTQ"
           tokenizer = AutoTokenizer.from_pretrained(model_name_or_path, trust_remote_code=True, torch_dtype=torch.float32)
           model = AutoModelForCausalLM.from_pretrained(
               model_name_or_path,
               trust_remote_code=True,
               device_map='cpu',
               torch_dtype=torch.float32
           )

           prompt = "Tell me about AI"

           + model.forward = torch.compile(model.forward, backend="openvino", options={'aot_autograd': True})

           pipe = pipeline(
               "text-generation",
               model=model,
               tokenizer=tokenizer,
               max_new_tokens=64
           )
           result = pipe(prompt)
           print(result[0]['generated_text'])


     .. tab-item:: Chatglm-4-GPTQ

        .. code-block:: py

           import torch
           from transformers import AutoModelForCausalLM, AutoTokenizer

           query = "tell me about AI“

           tokenizer = AutoTokenizer.from_pretrained("mcavus/glm-4v-9b-gptq-4bit-dynamo", trust_remote_code=True)
           inputs = tokenizer.apply_chat_template([{"role": "user", "content": query}],
                                                  add_generation_prompt=True,
                                                  tokenize=True,
                                                  return_tensors="pt",
                                                  return_dict=True
                                                  )
           model = AutoModelForCausalLM.from_pretrained(
               "mcavus/glm-4v-9b-gptq-4bit-dynamo",
               torch_dtype=torch.float32,
               low_cpu_mem_usage=True,
               trust_remote_code=True
           )

           + model.transformer.encoder.forward = torch.compile(model.transformer.encoder.forward, backend="openvino", options={"aot_autograd":True})

           gen_kwargs = {"max_length": 2500, "do_sample": True, "top_k": 1}
           with torch.no_grad():
               outputs = model.generate(**inputs, **gen_kwargs)
               outputs = outputs[:, inputs['input_ids'].shape[1]:]
               print(tokenizer.decode(outputs[0], skip_special_tokens=True))

To use torch.compile, you need to define the openvino backend in your PyTorch application.
This way Torch FX subgraphs will be directly converted to OpenVINO representation without
any additional PyTorch-based tracing/scripting.
This approach works only for the package distributed via pip, as it is now configured with
torch_dynamo_backends entrypoint <https://pytorch.org/docs/stable/torch.compiler_custom_backends.html#registering-custom-backends>__.

.. code-block:: python

...
model = torch.compile(model, backend='openvino')
...

For OpenVINO installed via channels other than pip, such as conda, and versions older than
2024.1, an additional import statement is needed:

.. code-block:: python

import openvino.torch

...
model = torch.compile(model, backend='openvino')
...

.. image:: ../assets/images/torch_compile_backend_openvino.svg
:alt: torch.compile execution diagram
:width: 992px
:height: 720px
:scale: 60%
:align: center

Options
++++++++++++++++++++

It is possible to use additional arguments for torch.compile to set the backend device,
enable model caching, set the cache directory etc. You can use a dictionary of the available options:

  • device - enables selecting a specific hardware device to run the application.
    By default, the OpenVINO backend for torch.compile runs PyTorch applications
    on CPU. If you set this variable to GPU.0, for example, the application will
    use the integrated graphics processor instead.
  • aot_autograd - enables aot_autograd graph capture. The aot_autograd graph capture
    is needed to enable dynamic shapes or to finetune a model. For models with dynamic
    shapes, it is recommended to set this option to True. By default, aot_autograd
    is set to False.
  • model_caching - enables saving the optimized model files to a hard drive,
    after the first application run. This makes them available for the following
    application executions, reducing the first-inference latency. By default, this
    variable is set to False. Set it to True to enable caching.
  • cache_dir - enables defining a custom directory for the model files (if
    model_caching is set to True). By default, the OpenVINO IR is saved
    in the cache sub-directory, created in the application's root directory.
  • decompositions - enables defining additional operator decompositions. By
    default, this is an empty list. For example, to add a decomposition for
    an operator my_op, add 'decompositions': [torch.ops.aten.my_op.default]
    to the options.
  • disabled_ops - enables specifying operators that can be disabled from
    openvino execution and make it fall back to native PyTorch runtime. For
    example, to disable an operator my_op from OpenVINO execution, add
    'disabled_ops': [torch.ops.aten.my_op.default] to the options. By
    default, this is an empty list.
  • config - enables passing any OpenVINO configuration option as a dictionary
    to this variable. For details on the various options, refer to the
    :ref:OpenVINO Advanced Features <openvino-advanced-features>.

See the example below for details:

.. code-block:: python

model = torch.compile(model, backend="openvino", options = {"device" : "CPU", "model_caching" : True, "cache_dir": "./model_cache"})

You can also set OpenVINO specific configuration options by

Explore Related Technologies & Topics

View All 9,111+ Repositories →

Discover similar high-velocity repositories, agent skills, and OpenAPI specifications across the ecosystem.

#ai Browse ai catalog → #computer-vision Browse computer-vision catalog → #deep-learning Browse deep-learning catalog → #deploy-ai Browse deploy-ai catalog → #diffusion-models Browse diffusion-models catalog → #generative-ai Browse generative-ai catalog → ⚡ MCP Servers Model Context Protocol → 🧠 Agent Skills Prompt & rule files → 🔌 OpenAPI Specs REST & Swagger schemas →