label-studio Documentation

Technical Documentation: HumanSignal/label-studio

ℹ️ Provenance: Multi-Tier Ingestion · CodeWiki Reference

Overview (README.md) Copy Section

Website • Docs • Join Slack Community

What is Label Studio? Copy Chapter

Label Studio is an open source data labeling tool. It lets you label data types like audio, text, images, videos, and time series with a simple and straightforward UI and export to various model formats. It can be used to prepare raw data or improve existing training data to get more accurate ML models.

• Try out Label Studio
• What you get from Label Studio
• Included templates for labeling data in Label Studio
• Set up machine learning models with Label Studio
• Integrate Label Studio with your existing tools

Have a custom dataset? You can customize Label Studio to fit your needs. Read an introductory blog post to learn more.

Try out Label Studio Copy Chapter

Install Label Studio locally or deploy it in a cloud instance. Or sign up for a free trial of our Starter Cloud edition! You can learn more about what each edition offers here.

• Install locally with Docker
• Run with Docker Compose (Label Studio + Nginx + PostgreSQL)
• Install locally with pip
• Install locally with poetry
• Install locally with Anaconda
• Install for local development
• Deploy in a cloud instance

Install locally with Docker Copy Section

Official Label Studio docker image is here and it can be downloaded with docker pull.
Run Label Studio in a Docker container and access it at http://localhost:8080.

docker pull heartexlabs/label-studio:latest
docker run -it -p 8080:8080 -v $(pwd)/mydata:/label-studio/data heartexlabs/label-studio:latest

You can find all the generated assets, including SQLite3 database storage label_studio.sqlite3 and uploaded files, in the ./mydata directory.

Override default Docker install

You can override the default launch command by appending the new arguments:

docker run -it -p 8080:8080 -v $(pwd)/mydata:/label-studio/data heartexlabs/label-studio:latest label-studio --log-level DEBUG

Build a local image with Docker

If you want to build a local image, run:

docker build -t heartexlabs/label-studio:latest .

Run with Docker Compose Copy Section

Docker Compose script provides production-ready stack consisting of the following components:

• Label Studio
• Nginx - proxy web server used to load various static data, including uploaded audio, images, etc.
• PostgreSQL - production-ready database that replaces less performant SQLite3.

To start using the app from http://localhost run this command:

docker-compose up

Run with Docker Compose + MinIO Copy Section

You can also run it with an additional MinIO server for local S3 storage. This is particularly useful when you want to
test the behavior with S3 storage on your local system. To start Label Studio in this way, you need to run the following command:

# Add sudo on Linux if you are not a member of the docker group
docker compose -f docker-compose.yml -f docker-compose.minio.yml up -d

If you do not have a static IP address, you must create an entry in your hosts file so that both Label Studio and your
browser can access the MinIO server. For more detailed instructions, please refer to our guide on storing data.

Install locally with pip Copy Section

# Requires Python >=3.10
pip install label-studio

# Start the server at http://localhost:8080
label-studio

Install locally with poetry Copy Section

### install poetry
pip install poetry

### set poetry environment
poetry new my-label-studio
cd my-label-studio
poetry add label-studio

### activate poetry environment
poetry shell

### Start the server at http://localhost:8080
label-studio

Install locally with Anaconda Copy Section

conda create --name label-studio
conda activate label-studio
conda install psycopg2
pip install label-studio

Install for local development Copy Section

You can run the latest Label Studio version locally without installing the package from pypi.

# Install all package dependencies
pip install poetry
poetry install
# Run database migrations
python label_studio/manage.py migrate
python label_studio/manage.py collectstatic
# Start the server in development mode at http://localhost:8080
python label_studio/manage.py runserver

Deploy in a cloud instance Copy Section

You can deploy Label Studio with one click in Heroku, Microsoft Azure, or Google Cloud Platform:

Apply frontend changes

For information about updating the frontend, see label-studio/web/README.md.

Install dependencies on Windows

To run Label Studio on Windows, download and install the following wheel packages from Gohlke builds to ensure you're using the correct version of Python:

• lxml

# Upgrade pip 
pip install -U pip

# If you're running Win64 with Python 3.8, install the packages downloaded from Gohlke:
pip install lxml‑4.5.0‑cp38‑cp38‑win_amd64.whl

# Install label studio
pip install label-studio

Run test suite Copy Section

To add the tests' dependencies to your local install:

poetry install --with test

Alternatively, it is possible to run the unit tests from a Docker container in which the test dependencies are installed:

make build-testing-image
make docker-testing-shell

In either case, to run the unit tests:

cd label_studio

# sqlite3
DJANGO_DB=sqlite DJANGO_SETTINGS_MODULE=core.settings.label_studio pytest -vv

# postgres (assumes default postgres user,db,pass. Will not work in Docker
# testing container without additional configuration)
DJANGO_DB=default DJANGO_SETTINGS_MODULE=core.settings.label_studio pytest -vv

What you get from Label Studio Copy Chapter

https://github.com/user-attachments/assets/525ad5ff-6904-4398-b507-7e8954268d69

• Multi-user labeling sign up and login, when you create an annotation it's tied to your account.
• Multiple projects to work on all your datasets in one instance.
• Streamlined design helps you focus on your task, not how to use the software.
• Configurable label formats let you customize the visual interface to meet your specific labeling needs.
• Support for multiple data types including images, audio, text, HTML, time-series, and video.
• Import from files or from cloud storage in Amazon AWS S3, Google Cloud Storage, or JSON, CSV, TSV, RAR, and ZIP archives.
• Integration with machine learning models so that you can visualize and compare predictions from different models and perform pre-labeling.
• Embed it in your data pipeline REST API makes it easy to make it a part of your pipeline

Included templates for labeling data in Label Studio Copy Chapter

Label Studio includes a variety of templates to help you label your data, or you can create your own using specifically designed configuration language. The most common templates and use cases for labeling include the following cases:

Set up machine learning models with Label Studio Copy Chapter

Connect your favorite machine learning model using the Label Studio Machine Learning SDK. Follow these steps:

1. Start your own machine learning backend server. See more detailed instructions.
2. Connect Label Studio to the server on the model page found in project settings.

This lets you:

• Pre-label your data using model predictions.
• Do online learning and retrain your model while new annotations are being created.
• Do active learning by labeling only the most complex examples in your data.

Integrate Label Studio with your existing tools Copy Chapter

You can use Label Studio as an independent part of your machine learning workflow or integrate the frontend or backend into your existing tools.

Ecosystem Copy Chapter

Citation Copy Chapter

Include a citation for Label Studio in the References section of your articles:

@misc{Label Studio,
  title={{Label Studio}: Data labeling software},
  url={https://github.com/HumanSignal/label-studio},
  note={Open source software available from https://github.com/HumanSignal/label-studio},
  author={
    Maxim Tkachenko and
    Mikhail Malyuk and
    Andrey Holmanyuk and
    Nikolai Liubimov},
  year={2020-2025},
}

License Copy Chapter

This software is licensed under the Apache 2.0 LICENSE © Heartex. 2020-2025

README (`docs/README.md`) Copy Section

Documentation of Label Studio

Use and deploy Hexo Copy Chapter

Installing Dependencies Copy Section

npm install

Starting Development server Copy Section

npm run server

Starts a local server. By default, this is at http://localhost:4000/.

Deploying Documentation Copy Section

npm run publish

Deploy the docs locally using Hexo Copy Chapter

To deploy the docs locally on your machine using Hexo, use the following steps.

Prerequisites Copy Section

• Install Hexo
• Clone the Label Studio Github repository

Deploy the docs locally Copy Section

In the label-studio/docs directory of the cloned repo, do the following:

1. (First time) Install required dependencies:

npm install

2. Start the Hexo server:

hexo serve

Hexo Official Documentation Copy Chapter

https://hexo.io/docs/

Source Guide Dashboard Overview (`docs/source/guide/dashboard_overview.md`) Copy Section

title: Projects overview dashboard short: Projects overview tier: enterprise type: guide order: 0 order_enterprise: 172 meta_title: Projects Overview Dashboard meta_description: Use the analytics overview page to explore progress across multiple projects section: "Analytics" date: 2026-01-10 11:50:41 Copy Chapter

The Analytics > Overview page is a centralized starting point to explore performance insights across multiple projects.

Access by user role

Filtering Copy Chapter

Filter by project and workspace Copy Section

Select one or more workspaces or individual projects. To see all projects, select All Workspaces at the top of the workspace filter.

This does not include Personal Sandbox projects, and is limited to 50 projects at a time.

Filter by date range Copy Section

You can select a date range to apply to the project metrics. If you want to be precise, you can also include time.

Note that annotation and acceptance metrics are calculated on when they are completed, not when they begin.

For example, say you select a date range of 2 days: Monday and Tuesday. If an annotator opens an annotation on Tuesday, begins working on it, and then submits it on Wednesday, then that annotation and the time they spent on it will not be included in the metrics.

Save views Copy Section

You can save your date range and project filters as custom views. Saved views makes it easier for you to group dates and projects you want to readily access.

Note the following:

• You can save, update, and delete views.
• Views are only visible to you (meaning they are not shared with other users within your organization).

!!! note
If you click Apply after saving your date and project/workspace filters, your selections will propogate to other Analytics pages. You can reapply your view or incorporate those filters into a new or updated view.

Tasks and project states Copy Chapter

For more information about the states referenced in the Overview dashboard, see Project and task state management.

!!! note

Project and task state history tracking did not start until state management was implemented for your organization. For most Label Studio Cloud organizations, state management was implemented in February 2026. 

This means that the **Overview** dashboard will not reflect when a project or task changed states if that change happened before state management was implemented.

Project progress summaries Copy Chapter

At the top of the Overview dashboard, summary cards show aggregate metrics for the selected date range and workspaces or projects.

!!! note
A dash (--) appears when there is no data. A zero (0) appears when there is data available and the value is zero.

Project progress charts Copy Chapter

The Overview dashboard includes charts that break down metrics over the selected time period.

Dates for annotations and reviews are based on when they were completed, not when they were started.

Project progress table Copy Chapter

The Project Progress table display detailed metrics for each project.

!!! note
A dash (--) appears when there is no data. A zero (0) appears when there is data available and the value is zero.

Source Guide Prompts Overview (`docs/source/guide/prompts_overview.md`) Copy Section

title: Prompts overview and use cases short: Overview and use cases tier: enterprise type: guide order: 0 order_enterprise: 225 meta_title: Prompts overview meta_description: An overview of the Prompts feature on the HumanSignal platform section: Prompts date: 2024-05-15 14:30:14 Copy Chapter

Use Prompts to evaluate and refine your LLM prompts and then generate predictions to automate your labeling process.

All you need to get started is an LLM deployment API key and a project.

With Prompts, you can:

• Drastically improve the speed and efficiency of annotations, transforming subject matter experts (SMEs) into highly productive data scientists while reducing the dependency on non-SME manual annotators.
• Increase annotation throughput, accuracy, and consistency, making the process faster and more scalable.
• Empower users to harness the full potential of AI-driven text labeling, setting a new standard for efficiency and innovation in data labeling.
• Leverage subject matter expertise to rapidly bootstrap projects with labels, allowing you to decrease time to ML development.
• Allow your subject matter experts time focus on higher-level tasks rather than being bogged down by repetitive manual work.

Features, requirements, and constraints Copy Chapter

!!! note
For information on installing Prompts for on-prem environments, see Install Prompts.

Use cases Copy Chapter

Auto-labeling with Prompts Copy Section

Prompts allows you to leverage LLMs to swiftly generate accurate predictions, enabling instant labeling of thousands of tasks.

By utilizing AI to handle the bulk of the annotation work, you can significantly enhance the efficiency and speed of your data labeling workflows. This is particularly valuable when dealing with large datasets that require consistent and accurate labeling. Automating this process reduces the reliance on manual annotators, which not only cuts down on labor costs but also minimizes human errors and biases. With AI's ability to learn from the provided ground truth annotations, you can maintain a high level of accuracy and consistency across the dataset, ensuring high-quality labeled data for training machine learning models.

Workflow

1. If you don't already have one, create a project and import a text-based dataset.

   • Create a project
   • Sync data from external storage

2. Annotate a subset of tasks, marking as many as possible as ground truth annotations. The more data you have for the prompt evaluation, the more confident you can be with the results.

   If you want to skip this step, see the bootstrapping use case outlined below.

   • Labeling guide
   • Define ground truth annotations for a project
   • Blog - What's a ground truth dataset?

3. Go to the Prompts page and create a new Prompt. If you haven't already, you will also need to add an API key to connect to your model.

   • Create a Prompt
   • Model provider keys

4. Write a prompt and evaluate it against your ground truth dataset.

   • Draft a prompt

5. When your prompt is returning an overall accuracy that is acceptable, you can choose to apply it to the rest of the tasks in your project.

   • Generate predictions from a prompt

Bootstrapping projects with Prompts Copy Section

In this use case, you do not need a ground truth annotation set. You can use Prompts to generate predictions for tasks without returning accuracy scores for the predictions it generates.

This use case is ideal for organizations looking to kickstart new initiatives without the initial burden of creating extensive ground truth annotations, allowing you to start analyzing and utilizing your data immediately. This is particularly beneficial for projects with tight timelines or limited resources.

By generating predictions and converting them into annotations, you can also quickly build a labeled dataset, which can then be refined and improved over time with the help of subject matter experts. This approach accelerates the project initiation phase, enabling faster experimentation and iteration.

Additionally, this workflow provides a scalable solution for continuously expanding datasets, ensuring that new data can be integrated and labeled efficiently as the project evolves.

!!! note
You can still follow this use case even if you already have ground truth annotations. You will have the option to select a task sample set without taking ground truth data into consideration.

Workflow

1. If you don't already have one, create a project and import a text-based dataset.

   • Create a project
   • Sync data from external storage

2. Go to the Prompts page and create a new Prompt. If you haven't already, you will also need to add an API key to connect to your model.

   • Create a Prompt
   • Model provider keys

3. Write a prompt and run it against your task samples.

   • Draft a prompt

When you run your prompt, you create predictions for the selected sample (this can be a portion of the project tasks or all tasks). From here you have several options:

• Continue to work on your prompt and generate new predictions each time you run it against your sample.
• Return to the project and begin reviewing your predictions. If you convert your predictions into annotations, you can use subject matter experts and annotators to begin interacting with those the annotations.
• As you review the annotations, you can identify ground truths. With a ground truth dataset, you can further refine your prompt using its accuracy score.

Prompt evaluation and fine-tuning Copy Section

As you evaluate your prompt against the ground truth annotations, you will be given an accuracy score for each version of your prompt. You can use this to iterate your prompt versions for clarity, specificity, and context.

This accuracy score provides a measurable way to evaluate and refine the performance of your prompt. By tracking accuracy, you can ensure that the automated labels generated by the LLM are consistent with ground truth data.

This feedback loop allows you to iteratively fine-tune your prompts, optimizing the accuracy of predictions and enhancing the overall reliability of your data annotation processes. In industries where data accuracy directly impacts decision-making and operational efficiency, this capability is invaluable.

Workflow

1. If you don't already have one, create a project and import a text-based dataset.

   • Create a project
   • Sync data from external storage

2. Annotate a subset of tasks, marking as many as possible as ground truth annotations. The more data you have for the prompt evaluation, the more confident you can be with the results.

   • Labeling guide
   • Define ground truth annotations for a project
   • Blog - What's a ground truth dataset?

3. Go to the Prompts page and create a new Prompt. If you haven't already, you will also need to add an API key to connect to your model.

   • Create a Prompt
   • Model provider keys

4. Write a prompt and evaluate it against your ground truth dataset.

   • Draft a prompt

5. Continue iterating and refining your prompt until you reach an acceptable accuracy score.

Source Skills Index (`docs/source/skills/index.md`) Copy Section

title: Skills overview short: Overview tier: all type: skills order: 1 meta_title: Label Studio agent skills meta_description: Use HumanSignal agent skills to set up labeling projects and custom interfaces from inside your AI coding agent. Copy Chapter

HumanSignal publishes a small set of agent skills that let you set up and iterate on Label Studio projects from inside an AI coding agent (Claude Code, Codex, Cursor, and other agents supported by the skills CLI). Each skill is a self-contained package: a workflow prompt, baked-in reference material, and small Python helper scripts that talk to your Label Studio instance over its public API.

Skills with the lock emoji 🔒 next to their name are only available for Label Studio Enterprise users.

Two skills are currently published:

When to use which Copy Chapter

• You want a standard labeling project — text classification, NER/span labeling, image bounding boxes, audio transcription, taxonomy review, ranking, pairwise comparison, time-series segmentation. Use create-xml-labeling-config-skill. It writes an XML config using Label Studio's built-in tags, which is the right runtime for the vast majority of projects.

• You need a custom React-based UI — the built-in tags don't cover your case, you're working with data types that don't have a standard object tag (3D, GEOTiff, DICOM), or you want to vibe-code your labeling screen. Use 🔒 create-interface-skill. The output is a single JSX file you can paste into the Label Studio Enterprise Interfaces editor or sync with the label-studio-sdk CLI.

If you're not sure which to start with, start with the XML config skill. It's the lighter-weight path and works on every Label Studio install.

How a skill run is structured Copy Chapter

Both skills follow the same general shape:

1. You describe the task in plain English ("NER for legal contracts with labels Party / Date / Amount / Clause type"). The skill asks one or two clarifying questions if it really needs them; otherwise it picks sensible defaults and surfaces its assumptions at the approval gate.
2. The agent drafts the XML config or JSX interface using the skill's baked-in reference material. No external knowledge base or runtime lookup is required.
3. Local validation runs automatically. Both skills ship Python scripts that catch structural issues before anything reaches your Label Studio instance.
4. The agent shows you the config / interface, a sample task, and any assumptions it made, then waits for an explicit yes.
5. On approval, the skill pushes to Label Studio. Either as a new project, or — for the config skill — as an update to an existing --project-id.

The approval gate in step 4 is load-bearing: neither skill writes to your Label Studio instance without you explicitly saying yes. If the agent gets an assumption wrong, you redirect at the gate and the skill iterates.

Install Copy Chapter

Both skills install through the skills CLI, which targets the agent of your choice.

`create-xml-labeling-config-skill` Copy Section

# Claude Code
npx skills add humansignal/create-xml-labeling-config-skill --skill create-xml-labeling-config-skill -g -a claude-code

# Codex
npx skills add humansignal/create-xml-labeling-config-skill --skill create-xml-labeling-config-skill -g -a codex

# Cursor
npx skills add humansignal/create-xml-labeling-config-skill --skill create-xml-labeling-config-skill -g -a cursor

`create-interface-skill` Copy Section

# Claude Code
npx skills add humansignal/create-interface-skill --skill create-interface-skill -g -a claude-code

# Codex
npx skills add humansignal/create-interface-skill --skill create-interface-skill -g -a codex

# Cursor
npx skills add humansignal/create-interface-skill --skill create-interface-skill -g -a cursor

Restart your agent after installing.

Credentials Copy Chapter

Both skills read from a .env at the skill root. The variables they look for:

Get your API key from Account & Settings → Access Token in your Label Studio instance.

Source Copy Chapter

• github.com/HumanSignal/create-xml-labeling-config-skill
• github.com/HumanSignal/create-interface-skill

Source Tags Index (`docs/source/tags/index.md`) Copy Section

title: Introduction type: tags order: 201 meta_title: Customize the Label Studio User Interface meta_description: Customize the Label Studio labeling interface using XML-like tags for data annotation tasks in machine learning and data science projects. Copy Chapter

You can use XML-like tags in Label Studio to configure the labeling interface. Label Studio uses three different types of tags to manage labeling:

• Object tags for data types, used to display elements in a task that can be labeled such as audio, HTML, images, paragraphs, text, and more.
• Control tags that you can use to annotate the objects. For example, use labels for semantic and named entity tasks, choices for classification tasks, textarea for transcription tasks, and more.
• Visual tags to modify the visual aspects of the labeling interface, such as by controlling the view of specific labeling options, or adding a header.

Create a custom labeling configuration Copy Chapter

When you set up your labeling interface, you can choose to use tags to create a custom labeling interface. When you create a custom labeling interface, you must pay attention to the structure of your task data and set up the configuration accordingly.

!!! error enterprise
Label Studio Enterprise has the option include plugins as part of your labeling interface. Plugins allow you to use Javascript to extend your interface and customize annotator workflows. For more information, see Plugins.

!!! info Tip
You can also implement plugins as part of your labeling interface. Plugins allow you to use Javascript to extend your interface and customize annotator workflows. For more information, see Plugins.

Connecting elements Copy Section

All control and object tags require a name parameter. Every control tag also requires a toName parameter that matches the name parameter of the object tag in the configuration.

For example, if you want to apply labels to text for a named entity recognition task, you might use the following labeling configuration:

<View>
  <Labels name="lbl" toName="txt">
    <Label value="Label 1"></Label>
    <Label value="Label 2"></Label>
  </Labels>
  <Text name="txt" value="$value"></Text>
</View>

In this example, the Label tags are used with the Text tag to label text. You can combine multiple control and object tags in the same configuration and use names to connect them.

Variables Copy Section

All object tags and some control and visual tags support variables inside their parameters. You can use a variable to set up a labeling configuration once and manage what displays on the labeling interface with the data in a specific task.

To use a variable, use it with the value parameter of a tag and specify it using the $ sign and the name of the field that you want to reference. For example, for a sample task containing the following partial JSON:

{
  "header": "This is a different header for each task",
  "textlabel": "This is the text that needs to be labeled"
}

Use the following labeling configuration to reference the data fields:

<View>
  <Header value="$header"></Header>
  <Text name="text" value="$textlabel"></Text>
  <Choices name="sentiment" toName="text" choice="single" showInLine="true">
   <Choice value="Positive"/>
   <Choice value="Negative"/>
   <Choice value="Neutral"/>
  </Choices>
</View>

Then your labeling interface looks like the following:

Styles Copy Section

You can style the labeling interface using the Style tag, but you can also add styles to the View tag.

For example, you can style the View tag to create a two-column labeling interface with multiple control and object tags such as in the following labeling configuration:

<View style="display: flex;">
  <View style="flex: 50%">
    <Header value="Choose:"></Header>
    <Text name="txt-1" value="$value"></Text>
    <Choices name="chc" toName="txt-1">
      <Choice value="Choice 1"></Choice>
      <Choice value="Choice 2"></Choice>
    </Choices>
  </View>
  <View style="flex: 50%; margin-left: 1em">
    <Header value="Enter your question and rate text:"></Header>
    <Text name="txt-2" value="$value"></Text>
    <Rating name="rating" toName="txt-2"></Rating>
    <TextArea name="question"></TextArea>
  </View>
</View>

Source Tutorials Index (`docs/source/tutorials/index.md`) Copy Section

title: Tutorials meta_title: Tutorials meta_description: A curated list of tutorials to help you get started or learn how to integrate Label Studio into your workflow. layout: tutorials hide_sidebar: true Copy Chapter

CONTRIBUTING (`docs/CONTRIBUTING.md`) Copy Section

Contribute to the Label Studio documentation

Thank you for investing your time in contributing to our project! Any contribution you make will be reflected on the Label Studio documentation site at https://labelstud.io/guide/, or https://labelstud.io/templates/ if you contribute a template.

Before you start Copy Chapter

We value input from each member of the community, and we ask that you follow our code of conduct. We are a small team, but we try to respond to issues and pull requests within 2 business days.

For changes that you contribute to any of the Label Studio repositories, please do the following:

• Review the general contributor guidelines.
• Create issues for any major changes and enhancements that you want to make.
• Keep pull requests specific to one issue. Shorter pull requests are preferred and are easier to review.

Contribute updates to the documentation Copy Chapter

Some types of documentation that you might want to contribute to Label Studio:

• Small fix: Fix a typo or a broken link.
• Add a section: Add a new section to an existing topic. For example, improve the troubleshooting guidance for ML backends based on an issue that you encountered.
• Refactor: Rewrite a topic to make it clearer. For example, improve the guidance for creating custom webhook events.
• Add new: Add a new topic, such as a new template or a new ML tutorial. If you want to add something else, please open an issue and discuss it with the team.

Contribute a template Copy Chapter

All Label Studio templates are specific to a machine learning use case, such as Natural Language Processing, Computer Vision, Conversational AI, and others.

We welcome contributions of new templates and enhancements to existing templates. Before you make a contribution, you might want to file a feature request issue about your proposed template to discuss it before you put in the work of a pull request.

If you want to add a new template, ensure that it fits one of the ML use cases. If your template is a customization of an existing template that primarily affects the layout of the labeling interface, consider contributing to the existing template by adding a new section to the Enhance this template section.

Contribute a new template Copy Section

If you want to contribute a new template, do the following.

Add the template to the documentation:

1. Create a new file in the label-studio/docs/source/templates directory with the name of the template, separated by underscores, such as new_template.md.
2. Add front matter to the template, including the title, category, type, order, and meta information. The type must be templates, and the order must be 1 higher than the last relevant template in the category to which your template applies. For example:

---
title: Document Retrieval
type: templates
category: Ranking and Scoring
cat: ranking-and-scoring
order: 502
meta_title: Document Retrieval Data Labeling Template
meta_description: Template for annotating documents for document retrieval tasks with Label Studio for your machine learning and data science projects.
---

3. Add information about your template. Include 1-2 sentences introducing the machine learning use case for the template and how the template actually works. For example:

If you want to start training document retrieval or recommender models, you might want to develop a dataset with that identifies similar documents. Use this template to identify and choose documents that are related to a specific query or an existing document.

4. Add boilerplate information to allow a template preview to render:

## Interactive Template Preview

<div id="main-preview"></div>

5. Add a header ## Labeling Configuration and place the full template XML in html tags.
6. Add a section ## About the labeling configuration and describe the tags in the template.
7. (Optional) Add a section with enhancements to the template with alternate configurations, a section for related templates, or a section for related tags.

After adding the template file, add it to the relevant template gallery by doing the following:

1. Locate the template gallery file for your template use case. All template galleries are named gallery_*.html with the last portion of the file name an abbreviation of the use case. For example, gallery_cv.html is the computer vision template gallery.
2. Create a new entry for the template at the end of the file. For example, copy this and update the a href element to point to your new template file, add categories, and update the title:

<div class="column">
      <a href="/templates/image_ellipse.html">
        <div class="card">
          <div class="image-wrap">
            <div class="image" style="background-image: url(/images/templates-misc/object-detection-ellipses.png)"></div>
          </div>
          <div class="category">object detection, semantic segmentation</div>
          <div class="title">Object Detection with Ellipses</div>
        </div>
      </a>
    </div>

If you want to add an image, place it in the label-studio/docs/themes/htx/source/images/templates-misc/ directory and update the link in this section.

That's it! Feel free to put out a pull request with only part of the template documentation to get feedback too.

Contribute an enhancement to an existing template Copy Section

If you modified an existing template and want to share your changes with the community, contribute an enhancement to an existing template.

1. Locate the template document that you want to update. For example, label-studio/docs/source/templates/text_summarization.md.
2. If it doesn't exist, add a header ## Enhance this template at the bottom of the template, before the ## Related Tags section.
3. Add a new header describing your enhancement. For example, ### Display text box next to the text to summarize.
4. Describe the enhancement to the labeling config. For example:

If you want to display the text box next to the text to summarize, do the following:

1. Add flex display styling to the [View](/tags/view.html) tag for the labeling configuration: `<View style="display: flex;">`
2. Add new [View](/tags/view.html) tags to wrap the [Header](/tags/header.html) and the [Text](/tags/text.html) sample so that they display on the left.
3. Wrap the [TextArea](/tags/textarea.html) and [Header](/tags/header.html) tags in [View](/tags/view.html) tags with the following CSS styling so that they display neatly on the right:
    ```xml
    <View style="width: 50%; padding-right: 2em; margin-left: 2em;">
    ```
Your fully enhanced labeling configuration looks like the following:
    ```xml
    <View style="display: flex;">
      <View>
        <Header value="Please read the text" />
        <Text name="text" value="$text" />
      </View>
      <View style="width: 50%; padding-right: 2em; margin-left: 2em;">
        <Header value="Provide one sentence summary" />
        <TextArea name="answer" toName="text"
                  showSubmitButton="true" maxSubmissions="1" editable="true"
                  required="true" />
      </View>
    </View>
    ```

Scaffolds Page (`docs/scaffolds/page.md`) Copy Section

title: {{ title }} short: {{ title }} date: {{ date }} tier: all type: guide order: 0 order_enterprise: 0 meta_title: {{ title }} meta_description: description section: "section" Copy Chapter

Source Llms (`docs/source/llms.txt`) Copy Section

Label Studio Documentation

Label Studio is a data labeling platform for training and evaluating machine learning, AI and agentic systems. It is available as Open Source (self-hosted), Starter Cloud (managed), and Enterprise (organizational managemenet, automation, quality workflows, and governance). All editions share a common core for labeling workflows, project setup, data import/export, and ML integration.
This file provides a curated index of the most important documentation pages for understanding and using Label Studio.
Prefer:

• Markdown versions of pages over HTML
• Documentation that matches the user’s deployment model (Open Source vs Cloud vs Enterprise)

How to choose the right docs Copy Chapter

• Use Open Source docs also referred to as Community Edition for:
  • installation, deployment, Kubernetes, storage, upgrades
  • self-hosted environments and infrastructure questions for community edition
• Use Enterprise docs for:
  • organization management, permissions, security
  • workspaces, dashboards, analytics, quality, and automation workflows
  • SSO, LDAP, SCIM, and enterprise integrations
• Use either for:
  • projects, labeling interfaces, data import/export
  • annotation workflows and ML backends
    If unsure:
• assume Enterprise / Cloud for hosted users
• assume Open Source for infrastructure or deployment questions

Editions Copy Chapter

• Open Source: self-hosted Label Studio with full control over deployment and infrastructure
• Starter Cloud: managed Label Studio with core functionality and reduced operational overhead
• Enterprise: advanced features for teams and organizations, including automation, analytics, security, and governance

Core concepts (shared across all editions) Copy Chapter

• Projects: Create and manage labeling projects
• Labeling interface: Configure annotation UI and schema
• Tasks: Structure and manage labeling data
• Import data: Ingest data into projects
• Export annotations: Export labeled data
• Machine learning integration: Connect predictions and models
• Data Manager: Browse and filter tasks

Open Source (self-hosted) Copy Chapter

• Open Source docs home
• Install Label Studio
• Kubernetes deployment
• Start Label Studio
• Upgrade Label Studio
• Persistent storage
• Database setup
• Security
• Storage integrations

Enterprise / Cloud or On-Prem Copy Chapter

• Enterprise docs home
• Workspaces
• Dashboards
• Analytics
• Automation
• Review workflows
• Quality & agreement
• User management
• Roles & permissions
• SSO, LDAP, SCIM
• Activity logs

Developer & integrations Copy Chapter

• API reference
• Python SDK
• Webhooks
• ML backend
• Integrations
• Open source repository

Optional Copy Chapter

• Glossary
• Templates
• Enterprise On-Prem Release notes
• Community

Source Guide FAQ (`docs/source/guide/FAQ.md`) Copy Section

title: Troubleshoot labeling issues short: Troubleshooting hide_sidebar: true meta_title: Troubleshoot labeling issues Copy Chapter

For troubleshooting information, see the HumanSignal support center.

Source Guide Access Tokens (`docs/source/guide/access_tokens.md`) Copy Section

title: Access tokens short: Access tokens tier: all type: guide order: 381 order_enterprise: 359 meta_title: Access tokens meta_description: Access tokens to interact with the Label Studio API and SDK. section: "Manage Your Organization" date: 2025-02-18 12:03:59 parent_enterprise: "admin_settings" Copy Chapter

Label Studio has personal access tokens and legacy tokens. These tokens are also referred to as your "API keys."

Are only visible to users once
    JWT refresh tokens
    Can be manually revoked
    Require [extra steps](#HTTP-API) when used with HTTP API
     `-H 'Authorization: Bearer <token>'` with HTTP API requests
    Only need to be set once when used with the SDK | Does not expire
    Remains listed and available in your account settings
    Can be manually revoked
    Do not need to be refreshed with used with HTTP API
     `-H 'Authorization: Token  <token>'` with HTTP API requests
    Only need to be set once when used with the SDK |

Find your access tokens Copy Chapter

You can access your API keys/access tokens by clicking your user icon in the upper right and selecting Account & Settings.

If you do not see either the Personal Access Tokens or Legacy Tokens page, that means you first need to enable them for your organization.

Enable access tokens for an organization Copy Chapter

The options that users see on their Account & Settings page depend on your settings at the organization level.

From the Organization page, select Settings > Access Token Settings.

!!! note
The Organization page is only available to users in the Admin and Owner role.

From here you can enable and disable token types.

• When a certain token type is disabled, existing tokens will not be able to authenticate to the Label Studio platform.

• Use the Personal Access Token Time-to-Live to set an expiration date for personal access tokens. (Enterprise only)

"API keys" vs. "Access tokens" Copy Chapter

In Label Studio, "access tokens" and "API keys" mean the same thing and are used interchangeably.

Personal access tokens Copy Chapter

SDK Copy Section

Personal access tokens can be set directly in the script or set as the LABEL_STUDIO_API_KEY environment variable.

# Define the URL where Label Studio is accessible and the API key for your user account
LABEL_STUDIO_URL = 'http://localhost:8080'

# API key can be either your PAT or legacy access token
LABEL_STUDIO_API_KEY = 'your-token'

# Import the SDK and the client module
from label_studio_sdk import LabelStudio

# Connect to the Label Studio API 
client = LabelStudio(base_url=LABEL_STUDIO_URL, api_key=LABEL_STUDIO_API_KEY)

HTTP API Copy Section

Because this is a JWT refresh token, you must use your PAT to generate a short-lived access token. This access token is then used for API authentication.

To generate this access token, make a POST request with your personal access token in the JSON body. For example:

curl -X POST <your-label-studio-url>/api/token/refresh \
-H "Content-Type: application/json" \
-d '{"refresh": "your-personal-access-token"}'

In response, you will receive a JSON payload similar to:

{
    "access": "your-new-access-token"
}

Use this access token by including it in your API requests via the Authorization: Bearer header. For example:

curl -X <method> <Label Studio URL>/api/<endpoint> -H 'Authorization: Bearer your-new-access-token'

When that access token expires (after around 5 minutes) you’ll get a 401 response, and will need to use your personal access token again to acquire a new one. This adds an extra layer of security.

You can also check when the token is set to expire using the following script:

# pip install pyjwt
from datetime import datetime, timezone
import jwt

decoded = jwt.decode(token)
exp = decoded.get("exp")
token_is_expired = (exp <= datetime.now(timezone.utc).timestamp())

Legacy tokens Copy Chapter

Generally speaking, the legacy tokens are not as secure as personal access tokens because they must be manually revoked.

SDK Copy Section

There is no difference in how you use legacy tokens and personal access tokens with the Python SDK. See the example above.

HTTP API Copy Section

Use this access token by including it in your API requests via the Authorization: Token header (this is different than the Authorization: Bearer header used with personal access tokens).

For example:

curl -X <method> <Label Studio URL>/api/<endpoint> -H 'Authorization: Token <token>

Source Guide Active Learning (`docs/source/guide/active_learning.md`) Copy Section

title: Set up active learning with Label Studio
short: Active learning loop
tier: enterprise
type: guide
order: 0
order_enterprise: 257
meta_title: Set up an active learning loop with Label Studio
meta_description: Set up an end-to-end active learning loop with Label Studio using the ML backend SDK and webhooks to perform model training and predictions and labeling.
section: "Machine Learning"

Follow this tutorial to set up an active learning loop with Label Studio.

Use Label Studio Enterprise Edition to build an automated active learning loop with a machine learning model backend. If you use the open source Community Edition of Label Studio, you can manually sort tasks and retrieve predictions to mimic an active learning process. If you're using Label Studio Community Edition, see how to manually manage your active learning loop.

About Active Learning Copy Chapter

Creating annotated training data for supervised machine learning models can be expensive and time-consuming. Active Learning is a branch of machine learning that seeks to minimize the total amount of data required for labeling by strategically sampling observations that provide new insight into the problem.

In particular, Active Learning algorithms aim to select diverse and informative data for annotation, rather than random observations, from a pool of unlabeled data using prediction scores. For more about the practice of active learning, read this article written by our HumanSignal CTO on Towards Data Science.

Set up an automated active learning loop Copy Chapter

Continuously train and review predictions from a connected machine learning model using Label Studio.

After a user creates an annotation in Label Studio, the configured webhook sends a message to the machine learning backend with the information about the created annotation. The fit() method of the ML backend runs to train the model. When the user moves on to the next labeling task, Label Studio retrieves the latest prediction for the task from the ML backend, which runs the predict() method on the task.

To set up this active learning, do the following:

1. Set up an ML model as an ML backend for active learning.
2. Connect the ML backend for getting predictions to Label Studio.
3. Configure webhooks to send a training event to the ML backend (optional).
4. Set up task sampling with prediction scores.
5. Label the tasks.

As you label tasks, Label Studio sends webhook events to your machine learning backend and prompts it to retrain. As the model retrains, the predictions from the latest model version appear in Label Studio.

Set up an ML model as an ML backend for active learning Copy Chapter

Set up an example machine learning model as an ML backend, or create a custom machine learning model.

Connect the ML backend to Label Studio for active learning Copy Chapter

Follow the steps to connect a model to a Label Studio project and ensure the setting Start model training on annotation submission is enabled. This sends a training request to the backend after each annotation submission or update.

Configure webhooks to send a training event to the ML backend (optional) Copy Chapter

By default, Label Studio notifies your ML backend every time an annotation is created or updated so that it can start training in response.

If you want, you can set up your project to send a webhook event and use that event and payload to drive event-specific training logic in your ML backend. If you want to customize the events and payloads sent to your ML backend, do the following:

1. In the Label Studio UI, open the project that you want to use for active learning.
2. Click Settings > Webhooks.
3. Click Add Webhook.
4. Add the following URL as your Payload URL: http://localhost:9090/webhook
5. (Optional) Leave the option to Send payload enabled. The ML backend does not require a payload, but you can use it in your code to retrieve project-related details, such as the project ID that you can use to retrieve data, define training hyperparameters based on project settings, retrieve the project state, or other details.
6. Disable the option to Send for all actions and enable Annotation created and Annotation updated.
7. Click Add Webhook.

For more details on the webhook event payloads, see the full payload details for the annotation webhook.

Set up task sampling with prediction scores Copy Chapter

In order to maximize the training efficiency and effectiveness of your machine learning model, you want your annotators to focus on labeling the tasks with the least confident, or most uncertain, prediction scores from your model. To do make sure of that, set up uncertainty task sampling.

Label the tasks Copy Chapter

On the project data manager, select Label All Tasks to start labeling.

As your model retrains and a new version is updated in Label Studio, the tasks shown next to annotators are always those with the lowest prediction scores, reflecting those with the lowest model certainty. The predictions for the tasks correspond to the latest model version.

Customize your active learning loop Copy Chapter

If you want to change the behavior of the active learning loop, you can make manual changes.

• Customize the prediction score produced by the model by modifying the inference call. See Make predictions with your ML backend for details and example code.
• To change the version of the model used to show predictions to annotators, update it in the machine learning settings. See Choose which predictions to show to annotators.
• If you want to delete all predictions after your model is retrained, see how to delete predictions.
• If you need to retrieve and save predictions for all tasks, see the recommendations for retrieving predictions from a model.

Set up manual active learning Copy Section

If you're using Label Studio Community Edition, data annotators can't experience a live active learning loop. You can mimic an active learning experience by doing the following:

1. Manually retrieve predictions from a model.
2. Sort the tasks in the data manager by prediction score.
3. Select Label Tasks As Displayed when labeling tasks.

This manual active learning loop does not automatically update the order of tasks presented to annotators as the ML backend trains with each new annotation and produces new predictions. Therefore, instead of on-the-fly automated active learning, you can perform a form of batched active learning, where you perform annotation for a period, stop to train the model, then retrieve new predictions and start annotating tasks again.

Source Guide Admin Interfaces (`docs/source/guide/admin-interfaces.md`) Copy Section

title: Interface admin settings short: Interfaces tier: enterprise type: guide hide_menu: false order: 0 order_enterprise: 363 meta_title: Interface admin settings section: "Manage Your Organization" parent_enterprise: "admin_settings" date: 2025-02-18 12:03:59 Copy Chapter

The Organization > Settings > Interfaces page is where Owners and Admins turn the Interfaces feature on for the organization and configure the security allowlist that every Interface runs against.

These settings apply uniformly to every Interface in your organization — embedded iframes, the main labeling canvas, the in-product preview, and the Develop Locally playground.

Access these settings from Organization > Settings > Interfaces.

!!! info Which section do I need?
- To let an Interface call an external API/service (fetch/XHR/WebSocket) → API origins.
- To let an Interface load a third-party script/stylesheet from a CDN → Advanced: external scripts.
- To use a library with no external network at all, bundle it into the Interface. Nothing to configure here. See Use external libraries and services.

Enable Interfaces Copy Chapter

Toggle Allow projects in this organization to use Interfaces to make the Interfaces feature available across the organization.

API origins Copy Chapter

Hosts that Interfaces are allowed to reach via fetch, XHR, or WebSocket (the CSP connect-src directive).

!!! info Tip
Your Label Studio server's own origin is always allowed automatically — you don't need to add it. Only list third-party hosts.

Iframe capabilities Copy Chapter

Capabilities that Interfaces are allowed to request through the iframe allow= attribute. Click any chip to toggle it on or off. The list is intentionally narrow:

!!! warning
Stay minimal. Every enabled capability is something an Interface (including a third-party or experimental one) could call into. Only enable a capability when an Interface in your organization needs it.

Advanced: external scripts Copy Chapter

Use this section to allow Interfaces to load third-party <script> and <style> tags from a list of trusted hosts.

Enable Allow external scripts / stylesheets to unlock the origin list and then add one or more script origins, for example https://cdn.jsdelivr.net.

!!! warning
Scripts loaded from these origins run with the Interface's full privileges and can read any task data the Interface is rendering. Only add hosts you trust completely. Most organizations should leave this section disabled.

Source Guide Admin Auth (`docs/source/guide/admin_auth.md`) Copy Section

title: SSO, LDAP & SCIM short: SSO, LDAP & SCIM tier: enterprise type: guide order: 0 order_enterprise: 383 meta_title: Identity and access options for Label Studio Enterprise meta_description: Overview of identity and access options available in Label Studio Enterprise section: "Manage Your Organization" date: 2025-09-21 09:08:58 Copy Chapter

Label Studio Enterprise provides enterprise-grade identity and access management to centralize authentication, automate user lifecycle, and control access across organizations, workspaces, and projects.

!!! info Tip
SSO handles authentication; SCIM handles automated user and group provisioning.

Most enterprises enable SSO first, then SCIM for ongoing user/group sync. LDAP is typically used for on‑prem deployments.

LDAP Copy Chapter

On-prem only.

Authenticate against your directory (e.g., AD/LDAP) in self-hosted environments while keeping roles managed in Label Studio.

• Login with enterprise directory credentials
• Keep RBAC and project/workspace permissions in Label Studio

See Set up LDAP authentication for Label Studio.

SSO (SAML 2.0) Copy Chapter

Authenticate users via your IdP (Okta, Google SAML, Azure AD/Microsoft Entra, Ping Identity, OneLogin, etc.) for seamless, secure login flows.

• Centralized login with your IdP
• Single sign-on across your tools
• Enterprise security policies enforced at IdP

See Set up SSO authentication for Label Studio and our SAML API docs.

SCIM 2.0 Copy Chapter

Automate provisioning/deprovisioning and group-to-role mappings. Supports Create/Update/Deactivate Users and Push Groups, including workspace/project membership management.

• Provision/deprovision users automatically
• Sync user profile changes
• Push Groups for workspace and project membership
• Map groups to organization roles and project-level roles

See and Set up SCIM2 for Label Studio our SCIM API docs.

Source Guide Admin Logs (`docs/source/guide/admin_logs.md`) Copy Section

title: Activity logs short: Activity logs tier: enterprise type: guide order: 0 order_enterprise: 354 meta_title: User activity logs in Label Studio meta_description: How to view user activity logs in Label Studio section: "Manage Your Organization" date: 2024-02-05 16:40:16 Copy Chapter

You can view user logs from the Organization > Activity Log page. Only users with the Administrator or Owner role can access this page.

!!! note
In Label Studio Enterprise Cloud (app.humansignal.com), activity logs are retained for 180 days.

In Label Studio Enterprise On-Prem, there is no limit on the retention period of activity logs.

Logs include user ID, IP address, a timestamp, and the type of action performed by logging the API request. You can find out more about each action being performed by clicking the API request.

!!! note
In some cases, clicking the API request will not return additional information. This is typically because the request includes sensitive information, such as a password.

You can filter the logs by the following:

• User
• Project
• Request type (GET, POST, PATCH, DELETE)
• Date range
• Request keyword

Source Guide Admin Manage (`docs/source/guide/admin_manage.md`) Copy Section

title: Manage user accounts short: Manage user accounts tier: opensource type: guide order: 372 order_enterprise: 0 meta_title: Manage user accounts meta_description: Manage existing user accounts in Label Studio Community edition section: "Manage Your Organization" date: 2024-02-06 16:36:44 Copy Chapter

!!! error Enterprise
In Label Studio Community Edition, there are relatively limited options available for user management.

In Label Studio Enterprise, you can set roles for users, restrict their permissions, delete and deactivate users, and view detailed logs of user actions. For more information about the user management capabilities of Label Studio Enterprise, see [User management overview](https://docs.humansignal.com/guide/manage_users) in the Enterprise documentation. To start a free trial, [click here](https://app.heartex.com/user/trial).

Retrieve user info from the command line Copy Chapter

You can retrieve information about a user, including the API user token for a user, from the command line after starting Label Studio.

From the command line, run the following:

label-studio user --username <username>

You can see user info, including their API token, as the last line of the response. For example:

=> User info:
{'id': 1, 'first_name': 'User', 'last_name': 'Somebody', 'username': 'label-studio', 'email': '[email protected]', 'last_activity': '2021-06-15T19:37:29.594618Z', 'avatar': '/data/avatars/img.jpg', 'initials': 'el', 'phone': '', 'active_organization': 1, 'token': '<api_token>', 'status': 'ok'}

Users can also retrieve their own tokens from the Label Studio app. For more information, see Access Tokes.

Review existing accounts in Label Studio Copy Chapter

You can view the organization user list from Label Studio:

1. Open Label Studio and click the menu in the upper left.
2. Select Organization.

Click the user's row to see additional details about their activity, such as when they were last active, which projects they created, and which projects they contributed to.

Reset passwords Copy Chapter

If you forget your password or change passwords regularly for security reasons, you can change it from the command line.

On the server running Label Studio, run the following command. And then when prompted, type the username and the new password:

label-studio reset_password

You can also use optional command line arguments to reset the password for a username.

• Specify the username and type the password when prompted:

  bash Copy

  label-studio reset_password --username <username>
  New password:

- Specify both the username and the password:

    ```bash
label-studio reset_password --username <username> --password <password>

Source Guide Admin Manage Lse (`docs/source/guide/admin_manage_lse.md`) Copy Section

title: Manage user accounts short: Manage user accounts tier: enterprise type: guide order: 0 order_enterprise: 372 meta_title: Manage user accounts meta_description: How to assign users to roles, delete users, tag users, and deactivate users section: "Manage Your Organization" parent: "manage_users" parent_enterprise: "manage_users" date: 2024-02-06 12:00:25 Copy Chapter

!!! note
If you are using SSO/SAML or SCIM, you can map groups to roles on the organization level and the project level. For more information, see Set up SSO authentication for Label Studio and Set up SCIM2 for Label Studio.

If you are using an IdP with SCIM or SAML/SSO, you should also handle user deactivations and deletions from the IdP.

Assign roles to users Copy Chapter

If you invite users using the invite link option, their account is created but must be activated by assigning them to a role.

!!! info Tip
You can use the role filter above the table to see all Pending users. Select the Not Activated role.

1. Open Label Studio and click the menu in the upper left.

2. Select Organization.

   If you do not see the Organization option, you do not have access to the Organization page. Only users in the Owner or Administrator role can access this page.

3. Locate the user account to which you are assigning a role.

4. Use the drop-down menu next to the user to select a role.

Programmatically assign roles Copy Chapter

To programmatically activate and assign roles to users, you can use the API:

https://api.labelstud.io/api-reference/api-reference/organizations/members/update

You can find user ID and organization ID from the Organization page:

You can also use the API:

• List organizations
• List members

Tag members Copy Chapter

You can tag organization members with metadata.

Apply member tags via CSV Copy Section

To bulk apply member tags, go to the Organization page and click Tag Members.

From here you can upload a CSV with the following format:

email,tags
[email protected],"Data Science, Seattle"
[email protected],"Finance, New York City"

• Ensure you include email,tags as the header.
• Tags must be encapsulated in quotation marks.
• If you have multiple tags, separate them with a comma.

!!! info Tip
If you are exporting the CSV from a spreadsheet editor, make sure you check the format in a text editor before uploading. Spreadsheet editors can add unnecessary formatting.

Add and delete tags Copy Section

To delete or manually add new tags, go to Organization > Settings > Member Tags.

Manually assign tags Copy Section

If you want to manually assign tags to individual users, you can use the Tags column in the org members table.

Deactivate users Copy Chapter

You can deactivate a user account by assigning them to the Deactivated role. Deactivating a user revokes their access and opens up their seat in your license.

You can reactivate a user by assigning them to an active user role.

Delete users Copy Chapter

1. From the Organization page, select the user you want to delete.

2. Click Delete.

3. A confirmation message appears. Click Delete Member.

Once deleted, any completed work or changes that the user made will appear as belonging to a "Deleted User."

If you want to add the user again at a later date, you will need to re-invite them to the organization.

Source Guide Admin Permissions (`docs/source/guide/admin_permissions.md`) Copy Section

title: Customize organization permissions short: Permissions tier: enterprise type: guide hide_menu: false order: 0 order_enterprise: 362 meta_title: Customize organization permissions section: "Manage Your Organization" parent_enterprise: "admin_settings" date: 2025-02-18 12:03:59 Copy Chapter

!!! note
This page is only visible to users in the Owner role.

You can use this page to customize certain permissions for roles. For information about Label Studio Enterprise roles and default permissions, see User roles and permissions.

Note the following:

• Permissions for the Owner role are not configurable.
• You can use the options on this page to have a more fined-grained control on which roles can execute certain actions.
• Any restrictions you configure will also apply to the API. For example, if you restrict Managers' ability to configure cloud storage, they will not be able to complete those actions through the UI or the API.

Permissions that can be configured include the following:

!!! note
Permissions for Annotators, Reviewers, and Managers are already restricted to projects that they created (if a Manager) or have explicitly been granted access to.

!!! info tip
You can click Reset to Defaults to return all permissions to their default value.

Source Guide Admin Roles (`docs/source/guide/admin_roles.md`) Copy Section

title: User roles and permissions short: Roles and permissions tier: enterprise type: guide order: 0 order_enterprise: 375 meta_title: User roles and permissions meta_description: A description of the user roles and permissions available in Label Studio Enterprise. section: "Manage Your Organization" parent: "manage_users" parent_enterprise: "manage_users" date: 2024-02-05 17:19:21 Copy Chapter

Your user role determines your level of access to actions and information in Label Studio.

• Access for Annotators, Reviewers, and Managers is restricted based on which projects and workspaces they have access to. For more information, see Project setup.
• Owners can further restrict permissions within their organization using the Permissions page.

Roles in Label Studio Enterprise Copy Chapter

For information on how to assign users to roles, see Manage user accounts.

Permissions in Label Studio Enterprise Copy Chapter

[^1]: For workspaces or projects they created or were invited to
[^2]: Owners can restrict this in the org settings
[^3]: The project must be using manual distribution and the annotator must be granted Data Manager access
[^4]: For their own work/history

Source Guide Admin Settings (`docs/source/guide/admin_settings.md`) Copy Section

title: Organization settings short: Settings type: guide tier: enterprise order: 0 order_enterprise: 357 meta_title: Organization settings meta_description: Configure org-wide settings in Label Studio Enterprise. section: "Manage Your Organization" Copy Chapter

There are several places where you can configure org-wide settings:

Source Guide Admin Usage (`docs/source/guide/admin_usage.md`) Copy Section

title: Usage & License short: Usage & License type: guide tier: enterprise order: 0 order_enterprise: 358 meta_title: Usage & License meta_description: Find information about your Label Studio account and configure organization settings. section: "Manage Your Organization" parent_enterprise: "admin_settings" Copy Chapter

You can find information about your plan and seat usage on this page.

Label Studio Enterprise Copy Chapter

Seats in use Copy Section

Here you can see the number of seats compared with the number of active users.

Each organization has a set amount of Label Studio seats in their license. To increase your seats, contact your HumanSignal account representative.

!!! note On-premise
On-premise deployments can use fewer seats than their license allows by setting the LICENSE_MAX_USERS_OVERRIDE environment variable. It can only lower the licensed seat count, and the deployment does not start if you set it higher. Seats are counted across all organizations in the deployment. See Available Helm values.

License info Copy Section

This section includes general information about your Label Studio license, such as when it was issued and when it expires.

It also includes the following usage information:

Projects

Cloud/SaaS: Every project that belongs to your current organization, regardless of state (draft/published) or workspace (meaning that it includes sandbox projects).

On-prem: All projects on your server, regardless of state workspace.

Results

The total number of labeled regions (results) that your organization has created across all projects. A result is one item in an annotation’s result array — a bounding box, text span, relation between regions, etc.

Security Copy Section

Only the org Owner can update these settings.

Session Timeout Policy

Field	Description
Max session age (minutes)	How many minutes a user can be idle. If they exceed this limit, they are automatically logged out the next time they trigger a request. Any interaction that hits the backend (clicks that trigger API calls, saves, polling, etc.) resets this timer.
Max time between activity (minutes)	Absolute session lifetime. Regardless of activity, a session cannot last longer than this many minutes from login; once reached, the user must re-authenticate.

Single Sign-On (SSO) enabled

This indicates with your organization can use SSO.

SSO is not available if you are in a trial organization or if it is not enabled for your license.

Embedding Copy Section

You can use these fields to set up Label Studio Embeds.

!!! note
Embedding is not available to all customers. Contact your HumanSignal account manager to enable.

Features Copy Section

Only the org Owner can update these settings and request access.

The features can be requested and enabled for your license.

Email notification settings Copy Section

Determines what email preferences are available from the Account and Settings page.

Note the following:

• New users will have their email preferences enabled by default.
• If you disable a notification here, that preference is hidden from their Account and Settings page and is blocked for all users in your organization.
• If you re-enable a notification, the preference will appear again in their Account and Settings page and will revert back to whatever state the user had it in before it was disabled for the organization.

Frequently asked Enterprise billing questions Copy Section

Find answers to common questions related to billing.

What happens if my plan is inactive?

If your subscription to Label Studio Enterprise expires or is cancelled, you can no longer perform labeling, review annotations, or add new users to your organization. You can log in and export your completed annotations.

What happens if I have too many active users?

You need to purchase additional seats for your Label Studio Teams subscription if you have more active users than your subscription allows. A user counts as an active user when they are assigned a role. Invited users that have not yet accepted an invitation appear as "Not activated" and do not count toward the seat limit.

Label Studio Starter Cloud Copy Chapter

Label Studio Starter Cloud has the Usage & Billing page that shows information about your subscription and how many seats you have in use.

!!! note
Only the Organization owner can access the Usage & Billing page.

There are two components to the subscription:

• Base subscription - This is your access to Label Studio Starter Cloud and includes 1 seat.
• Additional seats - You can purchase up to 11 additional seats (for a total of 12) on an as-needed basis.

Manage seats Copy Section

Add seats

As an Owner, go to Organization > Usage & Billing. Click Manage Seats and increase your seat count.

When you add seats, the cost is prorated and is reflected on your next invoice. An organization can have up to 12 seats.

Remove seats

Before you can remove seats, you must deactivate their associated users. You cannot have more active users than seats.

To deactivate a user, go to the Organization > Members page and set the user's role to Deactivated.

Then click Manage Seats and decrease your seat count.

When you remove seats, the cost is prorated and is reflected on your next invoice.

Frequently asked Starter Cloud billing questions Copy Section

How do I cancel?

As an Owner, go to Organization > Usage & Billing and click Cancel.

After cancellation, your account remains active until the end of your paid billing period. During this time, you have full access to the Label Studio Starter Cloud features.

However, after cancellation, you cannot add user seats. If you need more users than your current seat count allows, you must reactivate your subscription.

When the billing period ends, you will retain access to Label Studio and can download your data. However, you will be unable to import new data, annotate existing data, or create new projects.

How do I renew?

If you do not cancel your subscription, it is automatically renewed at the end of the billing period.

Can I upgrade to Label Studio Enterprise?

Yes, you can upgrade to Label Studio Enterprise, please contact our sales: [email protected].

Source Guide Admin User (`docs/source/guide/admin_user.md`) Copy Section

title: Add members in Label Studio Enterprise short: Add members tier: enterprise type: guide order: 0 order_enterprise: 369 meta_title: Add and invite members to Label Studio meta_description: Overview of how you can add members to Label Studio Enterprise section: "Manage Your Organization" parent: "manage_users" parent_enterprise: "manage_users" date: 2024-02-05 17:15:19 Copy Chapter

You can view, add, and manage organization members from the Organization page in Label Studio.

Navigate to the Organization page by clicking the menu in the upper left and selecting Organization.

Only users in the Owner or Administrator role can access this page.

Invite members Copy Chapter

Once the initial organization account is created, you can begin inviting users.

From the Organization page, click Invite Members. From here you have two options:

Invite link Copy Section

You can share this link with users, who can then complete the registration process.

However, they will be unable to access Label Studio until an Owner or Administrator manually assigns them a role. See User roles and permissions.

Click Reset to reset the link. When reset, anyone who has the old link will be unable to register a new user account.

Invite via email Copy Section

Enter a list of email addresses separated by commas and then select a user role for the new accounts.

Users will receive an email with a link to create their Label Studio account, and will be able to access Label Studio as soon as their registration is complete.

Require invites for new users (on-prem deployments) Copy Section

While you can invite users to join your organization with the invite link, this does not prevent users from registering a new account through the signup page.

You can remove the option to create accounts through the /user/signup page by setting the following environment variable:

LABEL_STUDIO_DISABLE_SIGNUP_WITHOUT_LINK=true

Users will only be able to sign up through an invitation link or email.

Activate new users Copy Chapter

Users who sign up through the link will not be able to access Label Studio until they are assigned a user role.

To filter for users who are pending, select Pending from the roles filter at the top of the Organization > Members page:

Then use the drop-down menu to assign a role:

Source Guide Agreement Metrics (`docs/source/guide/agreement_metrics.md`) Copy Section

title: Built-in agreement metrics reference short: Built-in metrics tier: enterprise type: guide order: 0 order_enterprise: 309 meta_title: Built-in agreement metrics in Label Studio Enterprise meta_description: Built-in agreement metrics in Label Studio Enterprise. section: "Review & Measure Quality" parent: "stats" parent_enterprise: "stats" Copy Chapter

The following metrics are available for control tags out-of-the-box in Label Studio Enterprise. You can use them as is, or you can create your own custom metrics.

All control tags Copy Chapter

Choices and taxonomy Copy Chapter

Categorical metrics are used for categorical control tags, such as Choices and Taxonomy.

Numeric Copy Chapter

Rectangles Copy Chapter

* Nested Choices or TextArea tags inside RectangleLabels/Rectangle tags

Polygons Copy Chapter

* Nested Choices or TextArea tags inside PolygonLabels/Polygon tags

Brush Copy Chapter

Span and segment Copy Chapter

* Nested Choices or TextArea tags inside Labels tags

HTML spans Copy Chapter

* Nested Choices or TextArea tags inside HyperTextLabels tags

Text Copy Chapter

Video Copy Chapter

Keypoints Copy Chapter

Examples Copy Chapter

Exact Match Copy Section

Exact Match is the simplest agreement metric. It checks whether two annotators gave the exact same answer, and returns either a perfect score (1.0) or zero (0.0). There is no partial credit.

This is the default metric for Choices, Taxonomy, Pairwise, and DateTime tags. It is also available as an alternative metric for many other tag types.

What affects the score

Unlike Span Overlap or IoU, there is no partial credit with Exact Match. The annotators either agree completely or they don't.

Single-select classification

When a labeling configuration uses a single-select Choices tag (e.g., sentiment analysis), each annotator picks one option. Exact Match compares the two selections directly.

Example: A project classifies customer reviews as Positive, Negative, or Neutral.

Multi-select classification

When a Choices tag allows multiple selections, each annotator's response is a list. Exact Match compares the two lists and requires them to contain the same items in the same order.

Example: A project tags articles with topics: Sports, Politics, Technology, Health.

!!! note
If you want partial credit for multi-select classifications (e.g., matching 2 out of 3 selected items), use Jaccard Similarity instead of Exact Match.

Taxonomy

For Taxonomy tags, the annotator's selection is a specific path through the taxonomy tree. Exact Match requires the full path to be identical.

Example: A project uses a taxonomy to classify animals.

!!! note
If you want partial credit for taxonomy paths that share a common prefix, use Common Labels Matches or Common Subtree Matches instead.

DateTime

For DateTime tags, Exact Match compares the two date/time values. They must be identical to score 1.0.

Span Overlap Copy Section

Span Overlap measures how much two annotators agree on the position of labeled spans in text (or other one-dimensional data like audio segments or time series). It is the default metric for Labels, ParagraphLabels, TimeSeriesLabels, and TimelineLabels tags.

What affects the score

What counts as a "span"

When annotators highlight a region of text and assign it a label, Label Studio records the character positions where the highlight starts and ends, along with the label. For example, in the sentence:

Dr. Maria Chen presented her findings at the Berlin conference.

If an annotator highlights Dr. Maria Chen and labels the span as "Person", the span is recorded as characters 0 through 14.

Step 1: Check that labels match

Before measuring any positional overlap, Span Overlap first checks whether two spans share the same label. If the labels are different, the score for that pair is 0.0 regardless of how much they overlap positionally.

For example, if Annotator A labels characters 0–14 as Person and Annotator B labels characters 0–14 as Organization, the score is 0.0 even though the character ranges are identical.

Step 2: Calculate IoU (Intersection over Union)

For two spans with matching labels, the metric calculates how much they overlap using IoU:

IoU = Intersection length / Union length

• Intersection is the region where both spans overlap.
• Union is the total region covered by either span.

Example: Consider two annotators labeling the same sentence:

Dr. Maria Chen presented her findings at the Berlin conference.

The labels match (both Person), so we calculate IoU:

• Intersection: characters 4–14 (the overlap) = 10 characters
• Union: characters 0–14 (the combined extent) = 14 characters
• IoU = 10 / 14 = 0.71

Step 3: Match spans across annotations using greedy matching

When annotators create multiple spans, the metric needs to figure out which spans from Annotator A correspond to which spans from Annotator B. It does this using greedy matching:

1. For every span in Annotator A's work, find the span in Annotator B's work with the highest IoU score.
2. For every span in Annotator B's work, find the span in Annotator A's work with the highest IoU score.
3. Average all of these best-match scores together.

This two-way matching means that unmatched spans (spans one annotator created but the other didn't) naturally pull the overall score down, because their best-match score will be 0.0.

Full example: Two annotators label named entities in this sentence:

Dr. Maria Chen presented her findings at the Berlin conference.

First, compute the IoU for each possible pair:

Then, find the best match for each span:

Final score = average of all best-match scores = (0.71 + 0.35 + 0.71 + 0.35) / 4 = 0.53

How the threshold variant works

The base Span Overlap metric returns a continuous score (0.53 in the example above). The Span Overlap (Threshold) variant converts this into a simple yes-or-no match by comparing the score against a threshold.

For example, with a threshold of 0.5:

• Score 0.53 >= 0.5 → 1.0 (match)

With a threshold of 0.75:

• Score 0.53 < 0.75 → 0.0 (no match)

The threshold variant is required when using the consensus methodology, which needs binary match/no-match decisions.

Intersection over Union (for bounding boxes) Copy Section

Intersection over Union (IoU) measures how much two annotators agree on the position and size of bounding boxes drawn on an image. It is the default metric for RectangleLabels and Rectangle tags.

What affects the score

What counts as a bounding box

When an annotator draws a rectangle on an image and assigns it a label, Label Studio records the box's position (x, y coordinates of the top-left corner) and its width and height. For example, if an annotator draws a box around a dog in a photo and labels it Dog, Label Studio stores the box's coordinates along with that label.

Step 1: Check that labels match

Just like Span Overlap, IoU first checks whether two bounding boxes share the same label. If the labels are different, the score for that pair is 0.0 regardless of how much the boxes overlap.

For example, if Annotator A draws a box and labels it Dog and Annotator B draws a box in the exact same position but labels it Cat, the score is 0.0.

Step 2: Calculate IoU

For two boxes with matching labels, the metric calculates how much area they share:

IoU = Intersection area / Union area

• Intersection is the area where both boxes overlap (the region covered by both).
• Union is the total area covered by either box (both boxes combined, not double-counting the overlap).

Another way to express this:

IoU = Intersection area / (Area of box A + Area of box B − Intersection area)

Example: Two annotators are labeling dogs in a photo.

Annotator A draws a box around the dog:

• Position: x=10, y=20
• Size: 100 wide × 80 tall
• Label: Dog

Annotator B draws a slightly different box around the same dog:

• Position: x=30, y=20
• Size: 100 wide × 80 tall
• Label: Dog

The labels match (both Dog), so we calculate IoU:

• Box A covers x=10 to 110, y=20 to 100
• Box B covers x=30 to 130, y=20 to 100
• Intersection: x=30 to 110, y=20 to 100 = 80 wide × 80 tall = 6,400 sq. units
• Area of Box A = 100 × 80 = 8,000 sq. units
• Area of Box B = 100 × 80 = 8,000 sq. units
• Union = 8,000 + 8,000 − 6,400 = 9,600 sq. units
• IoU = 6,400 / 9,600 = 0.67

Step 3: Match boxes across annotations using greedy matching

When annotators draw multiple bounding boxes, the metric uses the same greedy matching algorithm described in Span Overlap:

1. For every box in Annotator A's work, find the box in Annotator B's work with the highest IoU score.
2. For every box in Annotator B's work, find the box in Annotator A's work with the highest IoU score.
3. Average all of these best-match scores together.

Full example: Two annotators label objects in a photo containing a dog and a cat.

Suppose the IoU for matching pairs is:

Best match for each box:

Final score = (0.67 + 0.85 + 0.67 + 0.85) / 4 = 0.76

How the threshold variant works

The base IoU metric returns a continuous score (0.76 in the example above). The Intersection over Union (Threshold) variant converts this into a binary match/no-match result.

For example, with a threshold of 0.5:

• Score 0.76 >= 0.5 → 1.0 (match)

With a threshold of 0.8:

• Score 0.76 < 0.8 → 0.0 (no match)

IoU for other region types Copy Section

The IoU concept applies to more than just rectangular bounding boxes. Label Studio adapts the same core idea for different annotation types:

• Intersection over Union for Polygons works the same way, but calculates the overlapping area between polygon shapes instead of rectangles. This is useful when annotators draw freeform outlines around irregular objects.
• Brush Intersection over Union compares pixel-level brush masks by counting how many pixels overlap versus the total pixels painted by either annotator. This is useful for segmentation tasks where annotators paint regions rather than drawing shapes.

In all cases, the overall process is the same: check that labels match, calculate the overlap ratio for the specific shape type, then use greedy matching to pair up regions and average the scores.

Jaccard Similarity Copy Section

Jaccard Similarity measures how much two annotators' selections overlap when they can each choose multiple items from a list. Unlike Exact Match, it gives partial credit when annotators agree on some selections but not all of them.

This metric is dynamically available for Choices tags with multi-select enabled.

What affects the score

The formula

Jaccard Similarity treats each annotator's selections as a set and computes:

Jaccard Similarity = number of shared selections / total number of distinct selections

Or more formally:

J(A, B) = |A ∩ B| / |A ∪ B|

• Intersection (A ∩ B) is the set of items both annotators selected.
• Union (A ∪ B) is the set of all items selected by either annotator.

A simple example

A project asks annotators to tag articles with all relevant topics from a list: Sports, Politics, Technology, Health, Science.

Annotator A selects: Sports, Politics, Technology
Annotator B selects: Politics, Technology, Health

• Shared selections (intersection): Politics, Technology = 2 items
• All distinct selections (union): Sports, Politics, Technology, Health = 4 items
• Jaccard Similarity = 2 / 4 = 0.50

How it compares to Exact Match

Using the same example above:

This is why Jaccard Similarity is useful for multi-select tasks — it recognizes that agreeing on 2 out of 4 topics is meaningfully different from agreeing on 0.

More examples

Note that order does not matter. [Sports, Politics] and [Politics, Sports] produce the same score because both are treated as sets.

How the threshold variant works

The base Jaccard Similarity metric returns a continuous score (like 0.50). The Jaccard Similarity (Threshold) variant converts this into a binary match/no-match result.

For example, with a threshold of 0.5:

• Score 0.50 >= 0.5 → 1.0 (match)

With a threshold of 0.75:

• Score 0.50 < 0.75 → 0.0 (no match)

The threshold variant is required when using the consensus methodology.

Text Similarity Copy Section

Text Similarity measures how closely two free-text annotations match at the surface level — that is, how similar the actual characters or words are. It is the default metric for TextArea tags.

The core idea

Text Similarity uses edit distance algorithms to calculate how many changes (insertions, deletions, substitutions) would be needed to transform one text into the other. Fewer changes means higher similarity.

By default, it uses the Levenshtein algorithm, which counts the minimum number of single-character edits needed. The raw edit distance is then normalized to a score between 0.0 and 1.0, where 1.0 means the texts are identical.

Examples

Configurable parameters

Text Similarity lets you adjust two settings:

Algorithm — the method used to compare strings:

Granularity — the level at which the text is split before comparison:

Character-level comparison is the most fine-grained and catches small typos. Word-level comparison is more forgiving of minor spelling differences but stricter about missing or extra words.

Multiple text fields

When a TextArea tag allows multiple lines of text, each annotator's response is a list of strings. Text Similarity compares each line in order and averages the scores.

Example: Annotators are asked to transcribe three lines of text from an image.

• Line 1: "John Smith" vs "John Smith" → 1.0
• Line 2: "123 Main St" vs "123 Main Street" → ~0.81
• Line 3: "New York" vs "New York" → 1.0
• Final score = (1.0 + 0.81 + 1.0) / 3 = 0.94

If one annotator writes more lines than the other, the extra lines are compared against nothing and score 0.0, pulling the average down.

How the threshold variant works

The base Text Similarity metric returns a continuous score. The Text Similarity (Threshold) variant converts this into a binary match/no-match result. The default threshold for text similarity is 0.85 (compared to 0.5 for most other metrics), reflecting the expectation that free-text annotations should be fairly close to count as agreement.

Semantic Similarity Copy Section

Semantic Similarity measures whether two text annotations convey the same meaning, regardless of how they are worded. Instead of comparing characters or words directly, it compares the underlying meaning using AI embeddings.

The core idea

Semantic Similarity works in three steps:

1. Convert each text to an embedding — A numerical representation (a vector) that captures the meaning of the text.
2. Compute cosine similarity between the two vectors — This measures how close the two meanings are on a scale from 0.0 to 1.0.
3. Compare against a threshold — If the similarity meets the threshold (default: 0.85), the texts are considered a match.

When meaning matters more than wording

The key difference from Text Similarity is that Semantic Similarity understands that different words can express the same idea.

Text Similarity would score the second and third rows low because the actual words are quite different. Semantic Similarity recognizes that the underlying meaning is essentially the same.

Configurable parameters

Semantic Similarity has one parameter:

• Threshold (default: 0.85) — the minimum cosine similarity for two texts to be considered a match. Lower values are more lenient; higher values require closer meaning.

Choosing between Text Similarity and Semantic Similarity Copy Section

Use Text Similarity when:

• Annotators are transcribing text (OCR, audio transcription)
• Annotators are entering structured data (names, addresses, codes)
• Exact wording is important to the task
• You want to detect and measure minor typos or formatting differences

Use Semantic Similarity when:

• Annotators are writing descriptions, summaries, or captions
• Multiple valid phrasings exist for the same answer
• You care about whether annotators understood the content the same way, not whether they used the same words
• Annotators may be working in slightly different styles or vocabularies

Numeric Difference Copy Section

Numeric Difference measures how close two numeric values are. Unlike Exact Match, which only cares whether two numbers are identical, Numeric Difference gives partial credit when values are close but not equal. It is the default metric for Number and Rating tags.

The formula

Numeric Difference converts the absolute difference between two values into a similarity score using this formula:

score = 1 / (1 + difference)

This produces a score between 0.0 and 1.0. Identical values score 1.0, and the score decreases smoothly as the difference increases.

Examples with a 1–5 rating scale

A project asks annotators to rate the quality of customer support responses on a scale from 1 to 5.

Examples with continuous numeric values

A project asks annotators to estimate the age of a person in a photo.

Notice that the score drops quickly for larger differences. A difference of 1 already halves the score, and a difference of 5 brings it down to 0.17. This makes the base metric most useful for detecting whether annotators are in close agreement.

How the threshold variant works

The threshold variant, Numeric Difference (Threshold), works differently from other threshold variants. Instead of binarizing the similarity score, it uses a maximum difference parameter (default: 1.0). If the absolute difference between two values is within this tolerance, the score is 1.0; otherwise it's 0.0.

Example: With a max difference of 1.0 on a 1–5 rating scale:

This is often more intuitive for rating scales: "Annotators agree if they're within 1 point of each other."

Taxonomy metrics Copy Section

Label Studio offers two alternative metrics for Taxonomy tags that provide partial credit, unlike Exact Match which requires the full taxonomy selection to be identical. Both are useful when you want to recognize that two annotators who selected nearby items in a taxonomy are in closer agreement than two who selected completely unrelated items.

Understanding taxonomy paths

A taxonomy is a tree structure. When an annotator makes a selection, it is recorded as a path from the root to the selected node. For example, in this taxonomy:

Animals
├── Dogs
│   ├── Labrador
│   └── Poodle
├── Cats
│   ├── Siamese
│   └── Persian
└── Birds
    ├── Eagle
    └── Sparrow

Selecting "Labrador" produces the path: Animals > Dogs > Labrador
Selecting "Siamese" produces the path: Animals > Cats > Siamese

Annotators can also select multiple paths (e.g., both "Labrador" and "Eagle").

Common Labels Matches

Common Labels Matches compares taxonomy selections by treating each complete path as an item, then computing Jaccard Similarity over the set of paths. A path either matches exactly or it doesn't — there is no partial credit within a single path.

Example: An annotator can select multiple species from the taxonomy above.

Note that in the last row, Labrador and Poodle are both dogs, but Common Labels Matches treats them as completely different paths. If you want credit for shared ancestry, use Common Subtree Matches instead.

Common Subtree Matches

Common Subtree Matches gives partial credit for shared ancestry in the taxonomy tree. Before comparing, it expands each selected path into all of its ancestor prefixes. Then it computes Jaccard Similarity over these expanded sets.

How expansion works: The path "Animals > Dogs > Labrador" expands into:

• Animals
• Animals > Dogs
• Animals > Dogs > Labrador

This means that even if two annotators selected different leaf nodes, they'll still get credit for agreeing on the parent categories.

Example: Comparing "Labrador" vs. "Poodle":

• Shared nodes (intersection): {Animals}, {Animals > Dogs} = 2 nodes
• All distinct nodes (union): {Animals}, {Animals > Dogs}, {Animals > Dogs > Labrador}, {Animals > Dogs > Poodle} = 4 nodes
• Score = 2 / 4 = 0.50

Compare this to "Labrador" vs. "Siamese":

• Shared nodes: {Animals} = 1 node
• All distinct nodes: {Animals}, {Animals > Dogs}, {Animals > Dogs > Labrador}, {Animals > Cats}, {Animals > Cats > Siamese} = 5 nodes
• Score = 1 / 5 = 0.20

And "Labrador" vs. "Eagle":

• Shared nodes: {Animals} = 1 node
• All distinct nodes: 5 nodes
• Score = 1 / 5 = 0.20

The scores reflect how closely related the selections are in the taxonomy tree. Two dog breeds (0.50) score higher than a dog and a cat (0.20), which makes intuitive sense.

Comparing all three taxonomy metrics

Using the taxonomy above, here's how the three available metrics score the same pairs:

• Exact Match is strictest — any difference at all scores 0.0.
• Common Labels Matches gives credit when some of the selected paths match exactly, but not for shared ancestry.
• Common Subtree Matches is most lenient — it recognizes that selecting two items in the same branch of the taxonomy represents partial agreement.

Source Guide Api (`docs/source/guide/api.md`) Copy Section

title: API Reference for Label Studio
short: API
type: guide
tier: all
order: 401
order_enterprise: 401
meta_title: API Endpoints
meta_description: API documentation for authenticating, listing data science projects, importing predictions and raw data and exporting annotated data, and user management.
section: "Integrate & Extend"

You can use the Label Studio API to import data for labeling, export annotations, set up machine learning with Label Studio, and sync tasks with cloud storage.

See the API reference documentation for further guidance and interactive examples. If you want to write Python scripts using the API, use the Label Studio Python SDK.

!!! info Tip
For additional guidance on using our API, see 5 Tips and Tricks for Label Studio’s API and SDK.

Authenticate to the API Copy Section

You must retrieve your access token so that you can authenticate to the API.

Whether you can create a token, and which types of tokens you can create, depends on your org settings.

!!! note "API keys vs. Access tokens"
In Label Studio, "access tokens" and "API keys" mean the same thing and are used interchangeably.

There are two types of access tokens/API keys: Personal Access Tokens (PATs) and Legacy Tokens.

You can read more about the differences between them here: Access Tokens.

Find your access token

1. Open Label Studio and click your user icon in the upper right. Select Account & Settings.
2. Select Personal Access Token or Legacy Token on the left.

Depending on which one you select, you will need to either generate a new token or copy the one that is displayed.

Authenticate HTTP API requests with a personal access token

PATs use 'Authorization: Bearer <token>' when used with HTTP API requests, for example:

curl -X <method> <Label Studio URL>/api/<endpoint> -H 'Authorization: Bearer <token>'

Authenticate HTTP API requests with a legacy token

Legacy tokens use 'Authorization: Token <token>' when used with HTTP API requests, for example:

curl -X <method> <Label Studio URL>/api/<endpoint> -H 'Authorization: Token <token>

Authenticate Python SDK requests

When used with the SDK, you do not need to distinguish between legacy tokens or personal access tokens.

# Define the URL where Label Studio is accessible
LABEL_STUDIO_URL = 'YOUR_BASE_URL'

# API key can be either your PAT or legacy access token
LABEL_STUDIO_API_KEY = 'YOUR_API_KEY'

# Import the SDK and the client module
from label_studio_sdk import LabelStudio
client = LabelStudio(base_url=LABEL_STUDIO_URL, api_key=LABEL_STUDIO_API_KEY)

See API documentation for authentication.

List all projects Copy Section

To perform most tasks with the Label Studio API, you must specify the project ID, sometimes referred to as the pk, or primary key. If you don't know what your project ID is, you might want to get a list of all projects in Label Studio that you can access. See the List your projects API endpoint documentation.

Create and set up a project Copy Section

Create a project and set up the labeling interface in Label Studio using the API. See the Create new project API endpoint documentation.

If you want to make sure the configuration for your labeling interface is valid before submitting it using the API, you can use the validate label config API endpoint.

Import tasks using the API Copy Section

To import tasks using the API, make sure you know the project ID that you want to add tasks to. See additional examples and parameter descriptions in the import data endpoint documentation

Retrieve tasks Copy Section

Retrieve a paginated list of tasks for a specific project. If you want, you can also retrieve tasks and annotations using this API endpoint, as an alternative to exporting annotations. See details and parameters in the list project tasks endpoint documentation.

Export annotations Copy Section

To export annotations, first see which formats are available to export for your project.

Choose your selected format from the response and then call the export endpoint. See the export annotations endpoint documentation for more details.

Source Guide Ask Ai (`docs/source/guide/ask_ai.md`) Copy Section

title: AI Assistant short: AI Assistant tier: enterprise type: guide order: 0 order_enterprise: 356 meta_title: About AI Assistant meta_description: Information about using the AI features in Label Studio section: "Manage Your Organization" date: 2025-01-28 16:40:16 Copy Chapter

The Label Studio AI Assistant is an OpenAI LLM that has been trained on the Label Studio documentation, codebase, and several other Label Studio resources.

Use AI Assistant for project setup Copy Chapter

Instead of manually building labeling interfaces or project instructions from scratch, you can prompt the AI with a description of what your labeling project needs and what you want annotators to be able to select.

You can interact with this as you would a chat, meaning that you can build off your previous instructions and request changes.

Use AI Assistant for general help Copy Chapter

You can also use AI Assistant to ask for troubleshooting and general help with Label Studio.

AI Assistant FAQ Copy Chapter

How do I enable or disable AI Assistant? Copy Section

Your organization can enable AI Assistant from the Organization > Usage & License page.

!!! note
Only users in the Owner role can modify these settings. For Starter Cloud users, open a support ticket to request disablement.

• Enable AI Features--Enable AI Assistant for your labeling interface configuration.
• Enable Ask AI--Enable the AI Assistant for general Label Studio help.

What models do you use? Copy Section

We use OpenAI Tier 5 models, primarily gpt-4o and o3-mini.

Do all your AI features use the same backend? Copy Section

Yes.

What data is used to generate answers? Copy Section

Only the data you provide in the chat window is used to generate responses, this includes:

• All chat messages
• Task samples you have uploaded
• The current labeling config

How is my data used to train models? Copy Section

The model is trained on public resources like our documentation, codebase, blog, and website. We also train it on additional examples we've written specifically for the model.

We are on a Tier 5 OpenAI account, and opt out of all requests for training data.

We track requests to our AI to use as quality control, but they are only used to test the outputs of the model. For example: If Company X asks "Make a project to label opossums," we may notice that our AI did not provide a satisfactory answer and we may create configurations with opossums in them and train the model on them. The model would not be trained on any of Company X's questions or data.

All tracked data is covered by our industry-leading security and privacy policies.

Is it HIPAA compliant? Copy Section

Your Business Associate Agreement (BAA) does not cover these features. If you are required to comply with HIPAA, we recommend that you disable Ask AI.

Source Guide Auth Ldap (`docs/source/guide/auth_ldap.md`) Copy Section

title: Set up LDAP authentication for Label Studio short: LDAP tier: enterprise type: guide order: 0 order_enterprise: 385 meta_title: LDAP authentication for Label Studio Enterprise meta_description: Label Studio Enterprise documentation for setting up LDAP authentication for your data labeling, machine learning, and data science projects. section: "Manage Your Organization" parent: "admin_auth" parent_enterprise: "admin_auth" date: 2024-05-24 09:08:58 Copy Chapter

Set up LDAP authentication to manage access to Label Studio.

!!! error Enterprise
LDAP authentication is only available in Label Studio Enterprise Edition and is only available for on-prem installations. If you're using Label Studio Community Edition, see Label Studio Features to learn more.

To more easily manage access to Label Studio Enterprise, you can map LDAP groups to both roles or workspaces.

Set up LDAP authentication Copy Chapter

After you set up LDAP authentication, you can no longer use native authentication to log in to the Label Studio UI. Set up LDAP authentication and assign LDAP users to your Label Studio Enterprise organization using environment variables in Docker.

You can also map specific LDAP groups to specific organization roles and workspaces in Label Studio Enterprise, making it easier to set up and manage role-based access control (RBAC) and project access in Label Studio Enterprise.

You can refer to this example environment variable file for your own LDAP setup:

AUTH_LDAP_ENABLED=1

# Use ldaps to secure the LDAP connection
AUTH_LDAP_SERVER_URI=ldaps://ldap.example.com
# LDAP admin credentials    
AUTH_LDAP_BIND_DN=uid=user,ou=sysadmins,o=123abc,dc=zexample,dc=com
AUTH_LDAP_BIND_PASSWORD=password123

# Allow users to use usernames (not only emails) to log into Label Studio
USE_USERNAME_FOR_LOGIN=1

# Simple user search in LDAP groups
AUTH_LDAP_USER_DN_TEMPLATE=uid=%(user)s,ou=Users,o=123abc,dc=example,dc=com

# Specify organization to assign it to all users on the platform
# Warning: the user with this email must be registered before any LDAP users log in 
[email protected]

# Populate the user from the LDAP directory:
# firstName, lastName, mail, sAMAccountName are taken from your LDAP record 
AUTH_LDAP_USER_ATTR_MAP_FIRST_NAME=firstName
AUTH_LDAP_USER_ATTR_MAP_LAST_NAME=lastName
# Specify the field to use for AUTH_LDAP_USER_QUERY_FIELD as 'email'
AUTH_LDAP_USER_ATTR_MAP_EMAIL=mail
# Specify the field to use for AUTH_LDAP_USER_QUERY_FIELD as 'username' 
AUTH_LDAP_USER_ATTR_MAP_USERNAME=sAMAccountName

# Query the authenticating user in Label Studio, it can be [email|username]
AUTH_LDAP_USER_QUERY_FIELD=email

# Group parameters
AUTH_LDAP_GROUP_SEARCH_BASE_DN=ou=Users,o=group-id,dc=example,dc=com
AUTH_LDAP_GROUP_SEARCH_FILTER_STR=(objectClass=groupOfNames)
AUTH_LDAP_GROUP_TYPE=ou

If you want to use a recursive scan to search in several LDAP groups to grant access to Label Studio, instead of relying on the simple search used by AUTH_LDAP_USER_ON_TEMPLATE, update your environment variables file like the following:

# Leave this parameter empty
AUTH_LDAP_USER_DN_TEMPLATE=""
# Specify the groups that you want to be able to log in, separated by ';' 
AUTH_LDAP_USER_SEARCH_BASES="ou=guests,dc=domain,dc=com;ou=owners,dc=domain,dc=com"

User Role Mappings Copy Chapter

Map LDAP groups to specific Label Studio Enterprise roles using ; to specify several groups.

AUTH_LDAP_ORGANIZATION_ROLE_ADMINISTRATOR=cn=admins,ou=users,o=123abc,dc=example,dc=com 
AUTH_LDAP_ORGANIZATION_ROLE_MANAGER=cn=managers,ou=users,o=123abc,dc=example,dc=com 
AUTH_LDAP_ORGANIZATION_ROLE_REVIEWER=cn=reviewers,ou=users,o=123abc,dc=example,dc=com
AUTH_LDAP_ORGANIZATION_ROLE_ANNOTATOR=cn=annotators,ou=users,o=123abc,dc=example,dc=com;cn=guests,ou=users,o=123abc,dc=example,dc=com
AUTH_LDAP_ORGANIZATION_ROLE_NOT_ACTIVATED=cn=not,ou=users,o=123abc,dc=example,dc=com 
AUTH_LDAP_ORGANIZATION_ROLE_DEACTIVATED=cn=deactivated,ou=users,o=123abc,dc=example,dc=com

Workspace Mappings Copy Chapter

Map LDAP groups to specific Label Studio workspaces. Use a JSON format where keys are workspace titles and values are LDAP groups. Split groups with ; to specify several groups.

AUTH_LDAP_ORGANIZATION_WORKSPACES='{"Workspace 1":"cn=team1,ou=users,o=60cbc901ec2e8e387a3b2d3e,dc=jumpcloud,dc=com","Workspace 2":"cn=team2,ou=users,o=60cbc901ec2e8e387a3b2d3e,dc=jumpcloud,dc=com"}'

Manage user access only with LDAP Copy Chapter

If you want to manage Label Studio roles and workspaces entirely with LDAP, add the following to your environment variable file:

MANUAL_PROJECT_MEMBER_MANAGEMENT=0
MANUAL_WORKSPACE_MANAGEMENT=0
MANUAL_ROLE_MANAGEMENT=0

Setting these options disables the Label Studio API and UI options to assign roles and workspaces for specific users within Label Studio and relies entirely on the settings in the environment variable file.

Enabling TLS on LDAP/Active Directory Copy Chapter

To enable a secure Transport Layer Security (TLS) connection on LDAP/Active Directory, add the following env variables:

AUTH_LDAP_CONNECTION_OPTIONS="OPT_X_TLS_CACERTFILE=/path/to/cert.crt;OPT_X_TLS_NEWCTX=0"

Where OPT_X_TLS_CACERTFILE points to a file with certificates. If you’re using self-signed certificates, you’ll need to add OPT_X_TLS_NEWCTX=0 as the last entry of the AUTH_LDAP_CONNECTION_OPTIONS env variable.

Source Guide Auth Setup (`docs/source/guide/auth_setup.md`) Copy Section

title: Set up SSO authentication for Label Studio short: SSO and SAML tier: enterprise type: guide order: 0 order_enterprise: 387 meta_title: SSO authentication for Label Studio Enterprise meta_description: Label Studio Enterprise documentation for setting up SSO authentication for your data labeling, machine learning, and data science projects. section: "Manage Your Organization" parent: "admin_auth" parent_enterprise: "admin_auth" Copy Chapter

Set up single sign-on using SAML to manage access to Label Studio using your existing Identity Provider (IdP).

!!! error Enterprise
SSO authentication is only available in Label Studio Enterprise Edition. If you're using Label Studio Community Edition, see Label Studio Features to learn more.

To more easily manage access to Label Studio Enterprise, you can map SAML groups to both roles or workspaces.

Set up SAML SSO Copy Chapter

The organization Owner or Administrator for Label Studio Enterprise can set up SSO & SAML for the instance. Label Studio Enterprise supports the following IdPs:

• Okta
• Google SAML
• Ping Federate and Ping Identity SAML SSO Setup Example
• OneLogin
• Microsoft Entra ID (formerly Azure Active Directory, Azure AD)
• Auth0
• Others that use SAML assertions

After setting up the SSO, you can use native authentication to access the Label Studio UI, however it's not a recommended option especially for the user with the Owner role.

• You can use SSO along with normal login. This is not a recommended option.

• You can prevent a user from creating his own organization by using DISABLE_SIGNUP_WITHOUT_LINK option.

Connect your Identity Provider to Label Studio Enterprise Copy Section

Set up Label Studio Enterprise as a Service Provider (SP) with your Identity Provider (IdP) to use SAML authentication.

The details will vary depending on your IdP, but in general you will complete the following steps:

From Label Studio:

1. Go to the Organization page.

   If you do not see the option to select Organization, you are not logged in with the appropriate role.

2. Select SSO & SAML in the upper right.

3. In the Organization field, ensure the domain matches the domain used for your organization in your IdP.

4. Copy the following URLs:

   • Assertion Consumer Service (ACS) URL with Audience (EntityID), and Recipient (Reply) details---The IdP uses this URL to redirect users to after a successful authentication.
   • Login URL---This is the URL that users will use to log in to Label Studio.
   • Logout URL---This is the URL used to redirect users after successfully logging out of Label Studio.

From your IdP:

1. Paste the URLs copied from Label Studio in the appropriate location.
2. Generate a metadata XML file, or a URL that specifies the metadata for the IdP.
3. Set up or confirm setup of the following SAML attributes. Label Studio Enterprise expects specific attribute mappings for user identities.

The default attribute names are:

!!! note Note
Different Identity Providers use different attribute names. Label Studio provides presets in the SSO & SAML settings page to quickly configure the correct attribute mappings for popular IdPs. You can also manually configure custom attribute names if your IdP uses different values.

Attribute presets by IdP:

Microsoft Entra ID with full URI format:

If your Entra ID is configured with default claim URIs, use:

From Label Studio:

1. Return to the SSO & SAML page.

2. Upload the metadata XML file or specify the metadata URL.

3. Set up group mappings. These can also be added or edited later.

   Ensure the group name you enter is the same as the group name sent as an attribute in a SAML authentication response by your IdP.

   • Organization Roles to Groups Mapping---Map groups to roles at the organization level. The role set at the organization level is the default role of the user and is automatically assigned to workspaces and projects. For more information on roles, see Roles in Label Studio Enterprise.

     You can map multiple groups to the same role. Note that users who are Not Activated or Deactivated do not count towards the seat limit for your account.

   • Workspaces to Groups Mapping---Add groups as members to workspaces. Users with Manager, Reviewer, or Annotator roles can only see workspaces after they've been added as a member to that workspace.

     Select an existing workspace or create a new one. You can map multiple groups to the same workspace.

   • Projects to Groups Mapping---Map groups to roles at the project level. Project-level roles can be Annotator, Reviewer, or Inherit.

     You can map a group to different roles across multiple projects. You can also map multiple groups to the same roles and the same projects. For more information on roles, see Roles in Label Studio Enterprise.

     If you select Inherit, the group will inherit the role set above under Organization Roles to Groups Mapping. If the group is inheriting the Not Activated role, the users are mapped to the project, but they are not actually assigned to the project until the group is synced (meaning that the user authenticates with SSO).

4. Click Save.

5. Test the configuration by logging in to Label Studio Enterprise with your SSO account.

Setup SAML SSO with Okta video tutorial Copy Section

Manage user access only with SSO Copy Chapter

If you want to manage Label Studio roles and workspaces entirely with single sign-on (SSO), add the following to your environment variable file:

MANUAL_PROJECT_MEMBER_MANAGEMENT=0
MANUAL_WORKSPACE_MANAGEMENT=0
MANUAL_ROLE_MANAGEMENT=0

Setting these options disables the Label Studio API and UI options to assign roles and workspaces for specific users within Label Studio and relies entirely on the settings in the environment variable file.

!!! info Tip
If you are using the SaaS version of Label Studio (Label Studio Enterprise Cloud) and would like to enable these restrictions for your organization, open a ticket to submit your request.

If requested, we can also disable the common login option for your organization. When disabled, users can only use the SSO login fields and the common login  option is disabled completely.

Login page URL Copy Section

You can also set the LOGIN_PAGE_URL environment variable to redirect the login page to the specified URL.

Source Guide Backup Enterprise (`docs/source/guide/backup_enterprise.md`) Copy Section

title: Create backups for on-prem Label Studio Enterprise short: Back up Label Studio tier: enterprise type: guide order: 0 order_enterprise: 78 meta_title: Back up Label Studio Enterprise on-prem meta_description: How to back up your on-prem Label Studio Enterprise environment section: "Install" parent_enterprise: "install_enterprise" date: 2024-01-22 15:54:30 Copy Chapter

Backing up your Label Studio Enterprise on-prem installation involves more than just a database backup, although that is a critical component. Here's an overview of the steps you should take to back up your Label Studio Enterprise on-premises installation:

1. Back up your databases (development and production). This is essential as it contains all your project data and configurations. If you're using PostgreSQL, you can use tools like pg_dump to create backups of your databases.

2. Back up your file storage. If you're using local file storage or have files stored in a cloud storage that's synced with Label Studio, ensure you back up these files. This includes any media files (images, audio, video) that you've uploaded for labeling.

3. Back up your configuration files that have environment variables. This includes your label-studio configuration file (which might contain your database connection settings and other configurations) and any custom scripts or machine learning models that you've integrated with Label Studio.

4. Back up your environment variables. If you're using environment variables to configure Label Studio (which might include database credentials, secret keys, etc.), make sure to back these up as well.

5. Back up your Docker volumes. If you're running Label Studio in Docker containers and you have data volumes, back up these volumes. Docker volumes may contain your database data, media uploads, and other persistent data.

6. Document your current setup. Make sure to document the versions of Label Studio and any other dependencies you're using, as well as any specific configurations or customizations you've made. This will be helpful if you need to restore from a backup.

7. Regularly test your backups. Ensure that your backup process is working by regularly testing that you can restore your Label Studio instance from the backups.

Remember to store your backups in a secure location, ideally in a different physical location than your production environment to protect against site-wide issues such as natural disasters or power outages. It's also a good practice to encrypt sensitive data in your backups to prevent unauthorized access.

Source Guide Command Palette (`docs/source/guide/command_palette.md`) Copy Section

title: Command palette short: Command Palette tier: enterprise type: guide order: 0 order_enterprise: 356 meta_title: Enhanced search with the command palette meta_description: How to use the Label Studio command palette section: "Manage Your Organization" date: 2025-01-28 16:40:16 Copy Chapter

The command palette is an enhanced search and navigation tool.

To open it, click the search icon in the upper right or press Ctrl+K (Cmd+K on Mac)

What you can do Copy Chapter

From here you can:

• Search across your Label Studio org. For example, you can search for projects and workspaces.
• Navigate to pages within the app.
• Find quick links to Label Studio documentation and resources, such as the API docs and the changelog.
• When you are in a project, you can search for settings pages or individual settings to quickly jump to them.

Role-aware results Copy Chapter

Results respect your role and access. For example, Annotators and Reviewers won’t see workspaces or projects that they can’t access.

AI assistant Copy Chapter

If AI assistant is enabled, you can enter a query and then press Tab. The assistant opens automatically with your query.

Keyboard shortcuts Copy Chapter

• Ctrl+K or Cmd+K to open the command palette
• Use the Up and Down arrow keys to navigate through the results
• Enter to select an option
• Esc to close
• Tab to open the AI assistant

Source Guide Comments Notifications (`docs/source/guide/comments_notifications.md`) Copy Section

title: Comments and notifications short: Comments and notifications type: guide tier: enterprise section: "Review & Measure Quality" order: 0 order_enterprise: 306 meta_title: Comment in Label Studio Enterprise meta_description: The comments and notifications features define how annotators, reviewers and administrators communicate and receive updates on projects and tasks. Copy Chapter

Comments are a Label Studio Enterprise feature that allows communication between reviewers and annotators:

• Ask for clarification on a task

• Provide reasoning or context for their annotation | Reviewer to annotator communication

• Answer questions

• Give feedback to help the annotator improve |

This annotator-reviewer feedback loop is an important tool in your review workflow, allowing you to:

• Speed up the labeling process
• Increase the annotation quality
• Build more solid labeling and review processes

Permissions Copy Chapter

All users have the following commenting permissions for any annotations that they would otherwise have access to view:

• View the Comments panel and read previous comments.
• Add new comments.

Additional permissions are as follows:

Add a comment Copy Chapter

You can add comments from the Comments panel when viewing an annotation. All users have access to the Comments panel.

Link comments to regions and fields Copy Section

You can link your comment to a specific area within an annotation.

This can be a region (for example, a bounding box or a span of text) or it can be a field (for example, a classification).

Note the following:

• You can link comments to fields within the labeling configuration. However, if the field is a classification, choice, or taxonomy, you can only attach a comment if there is already a selection.
• You can only link comments to an entire text area. You cannot link comments to regions within text areas.
• Linking is not supported for Ranker or Pairwise fields.

Comment actions Copy Chapter

The actions you can perform depend on your user role permissions and whether it's your comment. You can access comment actions from the overflow menu:

Filter for comments Copy Chapter

The Data Manager has several columns to display comment information:

!!! note
Users in the Reviewer or Annotator role must be granted access to view the Data Manager. You can grant access in the project settings.

Require comments Copy Chapter

You can configure the project so that annotators and reviewers are required to leave comments after taking specific actions:

• Annotation > Annotating Options > Annotators must leave a comment on skip
• Review > Reject Options > Reviewers must leave a comment on reject

Comment notifications Copy Chapter

To see notifications, click your user profile icon in the upper right.

Use the overflow menu to mark notifications as read or unread.

!!! note
Notifications that have already been read are denoted with a gray background.

Annotators Copy Section

Annotators can write comments on their own annotations.

Sent notifications

When an annotator writes a comment, the users who are notified depend on whether it is the first comment on the annotation:

• If this is the first comment on an annotation, then all users within a project that have the role of Reviewer or higher see the comment in their notification list.
• If this comment is in response to an ongoing discussion within the comments panel, then only users who are have added comments to the annotation will see a notification.

Received notifications

An annotator receives a notification any time another user leaves a comment on one of their annotations.

Reviewers and other users Copy Section

Reviewers can write comments on annotations that they are reviewing. This applies to users with the Reviewer, Manager, Admin, and Owner role.

Sent notifications

When a reviewer writes a comment, then the users who are notified depend on their role and whether they have also left comments:

• Annotators get a notification for all comments on the annotations they created.
• If other users have also left comments on the annotation (for example, if multiple users are reviewing the annotation), then they will also be notified when a new comment appears.

Received notifications

A reviewer receives a notification when:

• If the comment is the first comment on an annotation, then all users within a project that have the role of Reviewer or higher see the comment in their notification list.
• If you have already left a comment on an annotation, you receive a notification when any user, regardless of role, adds a new comment.

Source Guide Custom Metric (`docs/source/guide/custom_metric.md`) Copy Section

title: Add a custom agreement metric to Label Studio short: Custom metrics tier: enterprise type: guide order: 0 order_enterprise: 310 meta_title: Add a Custom Agreement Metric for Labeling meta_description: Label Studio Enterprise documentation about how to add a custom agreement metric to use for assessing annotator agreement or the quality of your annotation and prediction results for data labeling and machine learning projects. section: "Review & Measure Quality" parent: "stats" parent_enterprise: "stats" Copy Chapter

Write a custom agreement metric to assess the quality of the predictions and annotations in your Label Studio Enterprise project.

Label Studio Enterprise contains a variety of agreement metrics for your project, but if you want to evaluate annotations using a custom metric or a standard metric not available in Label Studio, you can write your own.

!!! note
This functionality is available out-of-the-box for Label Studio Enterprise Cloud users.

For Label Studio Enterprise on-prem environments, you must configure your cloud provider to allow Label Studio to deploy and invoke serverless functions. Label Studio supports [AWS Lambda](#AWS-Lambda) and [Google Cloud Functions](#Google-Cloud-Functions). For more information, see [On-prem deployments](#On-prem-deployments).

Label Studio Enterprise includes various annotation and labeling statistics and the ability to add your own. The open source Community Edition of Label Studio does not contain these calculations. If you're using Label Studio Community Edition, see Label Studio Features to learn more.

Add your custom agreement metric to Label Studio Copy Chapter

Custom agreement metrics are added at the project level under Settings > Quality > Agreement > Custom Agreement Metric Code.

!!! note
You must configure the labeling interface before you can add your custom agreement metric.

On-prem deployment using AWS Lambda

On-prem deployment using Google Cloud Functions

Unlike with AWS Lambda, you cannot set a prefix or tags for the Google Cloud Functions function. See How Label Studio manages your functions below.

How to write your custom agreement metric Copy Chapter

For more information about agreement metrics, see Task agreement.

You can use the agreement metric to compare two annotations, or one annotation with one prediction. Use the input parameters annotation_1 and annotation_2 to specify the annotations to compare, or annotation and prediction to compare.

Add your code to the following function defined in Label Studio:

def agreement(annotation_1, annotation_2, per_label=False) -> float:

This function takes the following arguments:

For example, given the following labeling config:

<View>
  <Image name="image" value="$image"/>
  <Choices name="choice" toName="image" showInLine="true">
    <Choice value="Positive" />
    <Choice value="Negative" />
	<Choice value="Neutral" />
  </Choices>
</View>

The following agreement metric compares two annotations for a classification task with choice options of "Positive" and "Negative":

def agreement(annotation_1, annotation_2, per_label=False) -> float:

    # Retrieve two annotations in the Label Studio JSON format
    r1 = annotation_1["result"][0]["value"]["choices"][0]
    r2 = annotation_2["result"][0]["value"]["choices"][0]
    
    # Determine annotation agreement based on specific choice values
    if r1 == r2:
        # If annotations match and include the choice "Positive", return an agreement score of 0.99
        if r1 == "Positive":
            return 0.99
        # If annotations match and include the choice "Negative", return an agreement score of 0.7
        if r1 == "Negative":
            return 0.7
    # If annotations do not match, return an agreement score of 0
    else:
        return 0

If you set per_label=True, you can define a separate method or agreement score for each label. If you do this, you must return a separate score for each label. For example, for a classification task, you could use the following function to assign a weight and return a specific agreement score for each label used in an annotation:

def agreement(annotation_1, annotation_2, per_label=False) -> float:

    label_1 = annotation_1["result"][0]["value"]["choices"][0]
    label_2 = annotation_2["result"][0]["value"]["choices"][0]
    weight = {"Positive": 0.99, "Negative": 0.01}
    
    if label_1 == label_2:
        if per_label:
            return {label_1: weight[label_1]}
        else:
            return weight[label_1]
    else:
        if per_label:
            return {label_1: 0, label_2: 0}
        else:
            return 0

Show logs Copy Section

You can view logs for your custom agreement metric for troubleshooting purposes.

For logs to generate, you must first have at least one task annotated by at least two annotators.

AWS Lambda (on-prem deployments) Copy Chapter

If you have Label Studio Enterprise deployed in a private cloud (self-managed) Amazon Web Services (AWS) Elastic Compute Cluster (EC2) instance or Amazon Elastic Kubernetes Service (EKS), you must grant additional permissions so that Label Studio Enterprise can run custom agreement metrics in AWS Lambda.

To set up the permissions, do the following:

1. Create an AWS IAM role to be used by the custom metric Lambda functions to store logs in Cloudwatch
2. Set up permissions that grant access to AWS Lambda. How you do this depends on your deployment scenario:
   • Deployed with Docker Compose running in EC2.
   • Deployed in EKS with an OIDC provider.
   • Deployed in EKS without an OIDC provider.

You must know the AWS account ID for the AWS account that you use to manage Label Studio Enterprise to perform these steps.

Step 1: Create an AWS IAM role for logging Copy Section

Using your preferred method, create an AWS IAM role.

1. Create an AWS IAM role named LSE_CustomMetricsExecuteRole. Follow the steps to create a role to delegate permissions to an AWS service in the AWS Identity and Access Management documentation for Creating a role for an AWS service (console).
2. Whether you create the role using the console or the AWS CLI, create or attach the following IAM policy to allow the role to store logs in Cloudwatch. Replace YOUR_AWS_ACCOUNT with your AWS account ID that has access to Label Studio Enterprise.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "logs:CreateLogGroup",
            "Resource": "arn:aws:logs:*:YOUR_AWS_ACCOUNT:*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ],
            "Resource": [
                "arn:aws:logs:*:YOUR_AWS_ACCOUNT:log-group:/aws/lambda/custom-metric-*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogGroup",
                "logs:PutRetentionPolicy"
            ],
           "Resource": [
               "arn:aws:logs:*:YOUR_AWS_ACCOUNT:log-group:/aws/lambda/custom-metric-*"
           ]
        }
    ]
}

Create an IAM policy to grant AWS Lambda permissions

To grant permissions to a specific user, role, or EKS node group used to manage Label Studio Enterprise access to interact with AWS Lambda, use the following IAM policy. Create an IAM policy called LSE_AllowInteractLambda and replace YOUR_AWS_ACCOUNT with your AWS account ID:

{
   "Version": "2012-10-17",
   "Statement": [
      {
         "Sid": "VisualEditor0",
         "Effect": "Allow",
         "Action": "iam:PassRole",
         "Resource": "arn:aws:iam::YOUR_AWS_ACCOUNT:role/LSE_CustomMetricsExecuteRole"
      },
      {
         "Sid": "VisualEditor1",
         "Effect": "Allow",
         "Action": [
            "lambda:CreateFunction",
            "lambda:UpdateFunctionCode",
            "lambda:InvokeFunction",
            "lambda:GetFunction",
            "lambda:DeleteFunction",
            "lambda:TagResource",
            "lambda:ListTags"
         ],
         "Resource": [
            "arn:aws:lambda:*:YOUR_AWS_ACCOUNT:function:custom-metric-*"
         ]
      },
      {
         "Sid": "VisualEditor2",
         "Effect": "Allow",
         "Action": "lambda:ListFunctions",
         "Resource": "*"
      },
      {
         "Action": [
            "logs:CreateLogGroup",
            "logs:PutRetentionPolicy",
            "logs:TagResource",
            "logs:StartQuery",
            "logs:GetQueryResults"
         ],
         "Effect": "Allow",
         "Resource": [
            "arn:aws:logs:*:YOUR_AWS_ACCOUNT:log-group:/aws/lambda/custom-metric-*"
         ]
      }
   ]
}

Step 2: Configure permissions between Label Studio Enterprise and AWS Lambda Copy Section

After creating an IAM role to manage logs for the custom agreement metric, set up permissions to allow Label Studio Enterprise to interact with AWS Lambda.

How you set up permissions depends on how you deployed Label Studio Enterprise in your self-managed cloud infrastructure:

• Deployed with Docker Compose running in EC2
• Deployed in EKS with an OIDC provider
• Deployed in EKS without an OIDC provider

Docker Compose running in EC2

If you deployed Label Studio Enterprise using Docker Compose in an AWS EC2 instance, do the following to finish setting up permissions for the custom agreement metric functionality:

1. Follow the AWS documentation steps for Creating an IAM user in your AWS account to create an IAM user with programmatic access. This type of user is granted an access key to access AWS services.
2. While creating the IAM user, for the Set permissions option, choose to Attach existing policies directly.
3. Select Create policy and attach the LSE_AllowInteractLambda policy.
4. When you finish creating the user, save the username and access key somewhere secure.
5. In the docker-compose.yaml file that you use to deploy Label Studio Enterprise, add the following environment variables in the app and rqworkers sections:

!!! attention "important"
Update:
- YOUR_AWS_ACCESS_KEY_ID, YOUR_AWS_SECRET_ACCESS_KEY and YOUR_AWS_ACCOUNT with the credentials for the account created in step 1.
- YOUR_AWS_REGION with the AWS region that your EC2 instance exists in the following:

AWS_ACCESS_KEY_ID=YOUR_AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY=YOUR_AWS_SECRET_ACCESS_KEY
LS_LAMBDA_REGION_CUSTOM_METRICS=YOUR_AWS_REGION
LS_LAMBDA_ROLE_CUSTOM_METRICS=arn:aws:iam::YOUR_AWS_ACCOUNT:role/LSE_CustomMetricsExecuteRole

After you set up these permissions in your environment, you're ready to write your custom agreement metric and add it to Label Studio Enterprise:

1. Write your custom agreement metric.
2. Add your custom agreement metric to Label Studio Enterprise.

Deployed in EKS with an OIDC provider

If you deployed Label Studio Enterprise in Amazon Elastic Kubernetes Service (EKS) with OpenID Connect (OIDC) for identity and access management (IAM), do the following to finish setting up permissions for the custom agreement metric functionality:

1. Create an AWS IAM role named LSE_ServiceAccountApp following the steps to create a role to delegate permissions to an AWS service in the AWS Identity and Access Management documentation for Creating a role for an AWS service (console).
2. When relevant, attach the LSE_AllowInteractLambda policy to the LSE_ServiceAccountApp role.
3. Update your helm values.yaml file to include the following map. Replace YOUR_AWS_ACCOUNT with your AWS account ID:

app:
  serviceAccount:
    annotations: 
      eks.amazonaws.com/role-arn: arn:aws:iam::YOUR_AWS_ACCOUNT:role/LSE_ServiceAccountApp

4. Restart your Helm release.

After you set up these permissions in your environment, you're ready to write your custom agreement metric and add it to Label Studio Enterprise:

1. Write your custom agreement metric.
2. Add your custom agreement metric to Label Studio Enterprise.

Deployed in EKS without an OIDC provider

If you deployed Label Studio Enterprise in Amazon Elastic Kubernetes Service (EKS) and are not using OpenID Connect (OIDC) for identity and access management (IAM), do the following to finish setting up permissions for the custom agreement metric functionality:

1. In the AWS console UI, go to EKS > Clusters > YOUR_CLUSTER_NAME > Node Group.
2. Select the name of YOUR_NODE_GROUP with Label Studio Enterprise deployed.
3. On the Details page, locate and select the option for Node IAM Role ARN.
4. Create the AWS IAM policy LSE_AllowInteractLambda.
5. Restart your Helm release.

After you set up these permissions in your environment, you're ready to write your custom agreement metric and add it to Label Studio Enterprise:

1. Write your custom agreement metric.
2. Add your custom agreement metric to Label Studio Enterprise.

Google Cloud Functions (on-prem deployments) Copy Chapter

If you deploy Label Studio Enterprise on Google Cloud Platform (GCP), you can run custom agreement metrics on Google Cloud Functions (2nd gen) instead of AWS Lambda.

Before you start, make sure you have your GCP project ID and know how Label Studio Enterprise is deployed (Compute Engine VM, GKE with Workload Identity, or GKE without Workload Identity).

Step 1: Enable the required GCP APIs Copy Section

Label Studio Enterprise requires the following APIs to be enabled in your GCP project:

• Cloud Functions API (cloudfunctions.googleapis.com)
• Cloud Build API (cloudbuild.googleapis.com)
• Cloud Storage API (storage.googleapis.com)
• Cloud Logging API (logging.googleapis.com)
• Artifact Registry API (artifactregistry.googleapis.com)
• Cloud Run API (run.googleapis.com)

You can enable them using the gcloud CLI:

gcloud services enable \
  cloudfunctions.googleapis.com \
  cloudbuild.googleapis.com \
  storage.googleapis.com \
  logging.googleapis.com \
  artifactregistry.googleapis.com \
  run.googleapis.com \
  --project=YOUR_GCP_PROJECT

Step 2: Create a runtime service account for Cloud Functions Copy Section

Create a dedicated service account for your Cloud Functions to run as. It only needs the Logs Writer role.

1. Create the service account:

gcloud iam service-accounts create lse-custom-metrics-runtime \
  --display-name="LSE Custom Metrics Runtime" \
  --project=YOUR_GCP_PROJECT

2. Grant the Logs Writer role:

gcloud projects add-iam-policy-binding YOUR_GCP_PROJECT \
  --member="serviceAccount:lse-custom-metrics-runtime@YOUR_GCP_PROJECT.iam.gserviceaccount.com" \
  --role="roles/logging.logWriter"

You'll need this service account email (lse-custom-metrics-runtime@YOUR_GCP_PROJECT.iam.gserviceaccount.com) later for the GCP_SERVICE_ACCOUNT_CUSTOM_METRICS environment variable.

Step 3: Grant permissions to the Label Studio Enterprise identity Copy Section

The identity Label Studio Enterprise runs as needs to manage Cloud Functions and upload function source. This identity varies by deployment — your Compute Engine VM service account, a dedicated GCP service account bound via Workload Identity, or your GKE node pool service account. The deployment-specific sections below tell you which one to bind this role to.

Create a custom IAM role named LSE_CustomMetricsDeployer with the following permissions:

title: LSE Custom Metrics Deployer
description: Allows Label Studio Enterprise to deploy and invoke custom metric Cloud Functions
stage: GA
includedPermissions:
  # Cloud Functions lifecycle
  - cloudfunctions.functions.create
  - cloudfunctions.functions.update
  - cloudfunctions.functions.delete
  - cloudfunctions.functions.get
  - cloudfunctions.functions.list
  # Source upload via generateUploadUrl
  - cloudfunctions.functions.generateUploadUrl
  # Source code download from GCS staging bucket
  - storage.objects.get
  - storage.objects.create
  # Cloud Functions invocation (2nd gen uses Cloud Run)
  - run.routes.invoke
  # View function execution logs
  - logging.logEntries.list
  - logging.logs.list
  # Act as the runtime service account
  - iam.serviceAccounts.actAs

You can create this role using the gcloud CLI:

gcloud iam roles create LSE_CustomMetricsDeployer \
  --project=YOUR_GCP_PROJECT \
  --title="LSE Custom Metrics Deployer" \
  --description="Allows Label Studio Enterprise to deploy and invoke custom metric Cloud Functions" \
  --permissions=cloudfunctions.functions.create,cloudfunctions.functions.update,cloudfunctions.functions.delete,cloudfunctions.functions.get,cloudfunctions.functions.list,cloudfunctions.functions.generateUploadUrl,storage.objects.get,storage.objects.create,run.routes.invoke,logging.logEntries.list,logging.logs.list,iam.serviceAccounts.actAs

Then bind it to the Label Studio Enterprise identity. Replace LSE_IDENTITY with the appropriate member (see the deployment-specific sections below):

gcloud projects add-iam-policy-binding YOUR_GCP_PROJECT \
  --member="LSE_IDENTITY" \
  --role="projects/YOUR_GCP_PROJECT/roles/LSE_CustomMetricsDeployer"

!!! note
Instead of a single custom role, you can also assign the following predefined roles: roles/cloudfunctions.developer, roles/run.invoker, roles/storage.objectAdmin, roles/logging.viewer, and roles/iam.serviceAccountUser. The custom role above follows the principle of least privilege.

Deployed with Docker Compose on a Compute Engine VM

If you deployed Label Studio Enterprise using Docker Compose on a Compute Engine VM:

1. Make sure the VM runs as a service account that has the LSE_CustomMetricsDeployer custom role (or the equivalent predefined roles) bound at the project level. If you need to create one, run:

gcloud iam service-accounts create lse-app \
  --display-name="LSE Application" \
  --project=YOUR_GCP_PROJECT

gcloud projects add-iam-policy-binding YOUR_GCP_PROJECT \
  --member="serviceAccount:lse-app@YOUR_GCP_PROJECT.iam.gserviceaccount.com" \
  --role="projects/YOUR_GCP_PROJECT/roles/LSE_CustomMetricsDeployer"

2. Assign that service account to your Compute Engine instance.

3. In the docker-compose.yaml file you use to deploy Label Studio Enterprise, add the following environment variables to the app and rqworkers sections:

!!! attention "important"
Replace YOUR_GCP_PROJECT with your GCP project ID and YOUR_GCP_REGION with your preferred region (for example, us-central1).

CUSTOM_METRIC_PROVIDER=gcp
GCP_PROJECT_CUSTOM_METRICS=YOUR_GCP_PROJECT
GCP_REGION_CUSTOM_METRICS=YOUR_GCP_REGION
GCP_SERVICE_ACCOUNT_CUSTOM_METRICS=lse-custom-metrics-runtime@YOUR_GCP_PROJECT.iam.gserviceaccount.com

!!! note
On Compute Engine with a properly scoped service account, Label Studio uses Application Default Credentials (ADC) automatically — no key file required.

Deployed in GKE with Workload Identity

If you're using GKE with Workload Identity:

1. Create a GCP service account for Label Studio Enterprise (if you haven't already):

gcloud iam service-accounts create lse-app \
  --display-name="LSE Application" \
  --project=YOUR_GCP_PROJECT

2. Bind the LSE_CustomMetricsDeployer custom role to the service account:

gcloud projects add-iam-policy-binding YOUR_GCP_PROJECT \
  --member="serviceAccount:lse-app@YOUR_GCP_PROJECT.iam.gserviceaccount.com" \
  --role="projects/YOUR_GCP_PROJECT/roles/LSE_CustomMetricsDeployer"

3. Allow the Kubernetes service account to impersonate the GCP service account:

gcloud iam service-accounts add-iam-policy-binding \
  lse-app@YOUR_GCP_PROJECT.iam.gserviceaccount.com \
  --role="roles/iam.workloadIdentityUser" \
  --member="serviceAccount:YOUR_GCP_PROJECT.svc.id.goog[YOUR_NAMESPACE/YOUR_K8S_SERVICE_ACCOUNT]"

4. Update your Helm values.yaml to annotate the Kubernetes service account and set the required environment variables:

app:
  serviceAccount:
    annotations:
      iam.gke.io/gcp-service-account: lse-app@YOUR_GCP_PROJECT.iam.gserviceaccount.com
  extraEnvironmentVars:
    CUSTOM_METRIC_PROVIDER: gcp
    GCP_PROJECT_CUSTOM_METRICS: YOUR_GCP_PROJECT
    GCP_REGION_CUSTOM_METRICS: YOUR_GCP_REGION
    GCP_SERVICE_ACCOUNT_CUSTOM_METRICS: lse-custom-metrics-runtime@YOUR_GCP_PROJECT.iam.gserviceaccount.com

rqworker:
  extraEnvironmentVars:
    CUSTOM_METRIC_PROVIDER: gcp
    GCP_PROJECT_CUSTOM_METRICS: YOUR_GCP_PROJECT
    GCP_REGION_CUSTOM_METRICS: YOUR_GCP_REGION
    GCP_SERVICE_ACCOUNT_CUSTOM_METRICS: lse-custom-metrics-runtime@YOUR_GCP_PROJECT.iam.gserviceaccount.com

5. Restart your Helm release.

Deployed in GKE without Workload Identity

If you're using GKE without Workload Identity, your pods authenticate as the GKE node's service account, so that's the identity you'll grant permissions to.

1. Identify the service account used by your node pool. In the GCP console, go to Kubernetes Engine > Clusters > YOUR_CLUSTER > Node Pools > YOUR_NODE_POOL > Security and note the service account.

2. Bind the LSE_CustomMetricsDeployer custom role to the node pool's service account:

gcloud projects add-iam-policy-binding YOUR_GCP_PROJECT \
  --member="serviceAccount:YOUR_NODE_SERVICE_ACCOUNT@YOUR_GCP_PROJECT.iam.gserviceaccount.com" \
  --role="projects/YOUR_GCP_PROJECT/roles/LSE_CustomMetricsDeployer"

3. Update your Helm values.yaml to set the required environment variables:

app:
  extraEnvironmentVars:
    CUSTOM_METRIC_PROVIDER: gcp
    GCP_PROJECT_CUSTOM_METRICS: YOUR_GCP_PROJECT
    GCP_REGION_CUSTOM_METRICS: YOUR_GCP_REGION
    GCP_SERVICE_ACCOUNT_CUSTOM_METRICS: lse-custom-metrics-runtime@YOUR_GCP_PROJECT.iam.gserviceaccount.com

rqworker:
  extraEnvironmentVars:
    CUSTOM_METRIC_PROVIDER: gcp
    GCP_PROJECT_CUSTOM_METRICS: YOUR_GCP_PROJECT
    GCP_REGION_CUSTOM_METRICS: YOUR_GCP_REGION
    GCP_SERVICE_ACCOUNT_CUSTOM_METRICS: lse-custom-metrics-runtime@YOUR_GCP_PROJECT.iam.gserviceaccount.com

4. Restart your Helm release.

Environment variable reference for GCP Copy Section

Cloud Function runtime configuration Copy Section

Label Studio Enterprise deploys each custom metric Cloud Function with a fixed runtime configuration. These values are not user-configurable from the UI or environment variables and apply to every deployed function:

What this means for your metric code:

• Use Python 3.13–compatible code and imports.
• Keep the agreement function lightweight and stateless. Loading large models or running expensive computation can exceed the 256 MB / 60 s limits and fail scoring.
• Scoring requests are queued one-at-a-time per function. This is fine for background workers but means metrics aren't suited to interactive use.

Network connectivity

Custom metric functions are deployed with internal-only ingress, which affects how you reach them:

• Same GCP project as Label Studio Enterprise: Works out of the box. Internal traffic from your Compute Engine or GKE workloads is allowed by default.
• Different VPC or project: You'll need to set up internal connectivity first — Shared VPC, VPC peering, or a Serverless VPC Access connector — or scoring will fail.
• Testing the function directly: You can't invoke it from outside Google's network. curl from your laptop or the Cloud Console Test button will return 403 Forbidden. To verify deployment, trigger scoring inside Label Studio Enterprise (for example, recalculate agreement on a project that uses the metric).

How Label Studio manages your functions

Label Studio creates one Cloud Function per project that uses a custom metric, reuses it across scoring runs, and resolves the function URL for you — there's no URL or function name to configure manually.

To find a function in the Google Cloud console:

1. Open Cloud Functions in your configured GCP project and region.
2. Look for functions named custom-metric--<timestamp>.
3. Open a function to view logs, deployment state, and the trigger URL.

Function lifecycle behavior

!!! attention "important"
Prefix and tags/labels are not currently exposed as user-facing fields in the UI for GCP.

Advanced users can only set such values through API-level metric parameters (if your organization exposes that workflow), not via standard settings screens.

Source Guide Dashboard Annotator (`docs/source/guide/dashboard_annotator.md`) Copy Section

title: Member performance dashboard short: Member performance tier: enterprise type: guide order: 0 order_enterprise: 175 meta_title: Member performance dashboard meta_description: Use the performance dashboard to track user work and progress. section: "Analytics" date: 2024-04-04 11:50:41 Copy Chapter

The performance dashboard provides metrics about a user's annotation and review activities over a period of time, including how many tasks a user has completed and how quickly.

This can be useful in a number of ways:

• Resource management optimization: If you have a project that is on a rigid timeline, you can determine which users have the quickest turnaround and assign them as necessary.
• Annotator payment management: If you are using outside contractors or otherwise paying users based on their output, you can use this dashboard to help calculate or verify contractor payments.
• Reduce costs associated with internal analytic reports: With this report, you no longer need to develop complex tools or workflows to track user performance.

Access the dashboard Copy Chapter

Depending on your user role, you can access this dashboard in several ways:

• From the Organization page, select a user on the Members page. Then select Member Performance on the right.

• From the Home page, click Member Performance. This will pre-select your own performance dashboard.

• From the Members dashboard on a project, click the overflow menu next to a username in the Annotation Summary table. This will pre-select that user's annotator performance dashboard for that specific project.

• From anywhere within the app, open the navigation menu and select Analytics. This will pre-select your own annotator performance dashboard.

Access by user role Copy Section

Export data Copy Chapter

You can use the Export drop-down to export the following:

• Report - Download the information in the dashboard as CSV or JSON.

• Timeline - Download a detailed timeline of all the user's annotation actions within the time frame, including when the began and submitted each annotation.

• Comments Added - Download a CSV file with all of the comments that the user has left.

Filters Copy Chapter

Filter by project and workspace Copy Section

Select one or more workspaces or individual projects. To see all projects, select All Workspaces at the top of the workspace filter.

This does not include Personal Sandbox projects.

Filter by date range Copy Section

You can select a date range to apply to the performance metrics. If you want to be precise, you can also include time.

Note that annotation and review metrics are calculated on when they are completed, not when they begin.

For example, say you select a date range of 2 days: Monday and Tuesday. If an annotator opens an annotation on Tuesday, begins working on it, and then submits it on Wednesday, then that annotation and the time they spent on it will not be included in your performance report.

Similarly, if a reviewer opens an annotation on Tuesday but then doesn't accept it until Wednesday, that annotation is not counted in their total reviews or accepted annotation metrics.

Save views Copy Section

You can save your member, date range, and project filters as custom views. Saved views makes it easier for you to group member and project combos you want to readily access.

Note the following:

• You can save, update, and delete views.
• Views are only visible to you (meaning they are not shared with other users within your organization).

!!! note
If you click Apply after saving your date and project/workspace filters, your selections will propogate to other Analytics pages. You can reapply your view or incorporate those filters into a new or updated view.

Annotations Copy Chapter

The Annotations tab shows information related to a user's annotation performance and history.

The metrics are calculated from the following data:

• last_action -- The last action taken on an annotation. This can be submitted, updated, fixed and accepted, accepted, rejected.
• submitted_or_reviewed -- Annotations that have a last action matching those listed above.
• lead_times -- The time spent with annotations that have a last action matching those listed above. Lead time includes idle time and is cumulative across multiple sessions.

!!! note
A dash (--) appears when there is no data. A zero (0) appears when there is data available and the value is zero.

Annotation performance summaries Copy Section

Annotation charts Copy Section

!!! note
The date and time are calculated based on when they completed the annotation (see last_action above), and not when they began their annotation. See Filter by data range

Annotator performance table Copy Section

This is a detailed table with performance metrics for each user.

Keep in mind that all metrics in the table, including task assignments, are filtered by the time frame selected.

Reviews Copy Chapter

The Reviews tab shows information related to a user's activity reviewing other users' annotations.

!!! note
A dash (--) appears when there is no data. A zero (0) appears when there is data available and the value is zero.

Review performance summaries Copy Section

!!! note
Data collection for review time began on September 25, 2025 for cloud environments, or for 2.30 for on-prem deployments. If you filter for earlier dates, review time will not be calculated.

Review charts Copy Section

!!! note
The date and time are calculated based on when they either completed the review (such as by accepting it) or exited the annotation. See Filter by data range

Reviewer performance table Copy Section

This is a detailed table with performance metrics for each user.

Keep in mind that all metrics in the table, including task assignments, are filtered by the time frame selected.

Source Guide Dashboard Data Quality (`docs/source/guide/dashboard_data_quality.md`) Copy Section

title: Data quality dashboard short: Data quality dashboard tier: enterprise type: guide order: 0 order_enterprise: 71 meta_title: Data quality dashboard meta_description: Use the Data quality dashboard to analyze agreement, confusion, and dimension-level quality for a project. section: "Project Management" parent: "dashboards" parent_enterprise: "dashboards" date: 2026-06-29 00:00:00 Copy Chapter

Investigate project data quality with agreement metrics, dimension-level comparisons, and confusion analysis.

The Data quality dashboard helps you understand how consistently tasks are labeled and where annotations or labels disagree. Use it to find ambiguous tasks, confusing labels, and dimensions that need clearer instructions or review.

For velocity and time spent, see the Throughput dashboard. For member-level performance, see the Members dashboard.

Access the dashboard Copy Chapter

From the project, open the Dashboard tab and select Data quality from the dashboard navigation.

The Data quality dashboard has two tabs:

• Agreement analysis: Review agreement metrics and confusion across dimensions.
• Label distribution: Compare annotation and prediction label counts by dimension.

Agreement analysis Copy Chapter

Use the Agreement analysis tab to understand where annotators, models, and ground truth annotations align or disagree.

Average Task Agreement Copy Section

The Average Task Agreement KPI card shows the average agreement score for tasks in the project.

Agreement is most useful when tasks have multiple annotations or a comparison source such as ground truth. Tasks with only one annotation do not provide an annotator-to-annotator agreement signal.

Overall Agreement Charts Copy Section

Ground Truth Agreement Copy Section

Label distribution Copy Chapter

Use the Label distribution tab to compare values from annotations and predictions for each dimension.

Each supported dimension has a card with:

• Average Agreement: Agreement for the dimension when there is enough data to compare annotations or predictions.
• Label Distribution chart: A grouped horizontal bar chart comparing values from annotations and predictions.
• Label Distribution table: Counts from annotations and predictions for each value, plus a Total row. Values in the From Annotations and From Predictions columns link to the Data Manager with filters applied for the selected dimension and value.

For more detail about supported dimensions and Label Distribution behavior, see the Label Distribution dashboard.

Interpreting data quality issues Copy Chapter

Low agreement or high confusion does not always mean poor annotator or model performance. It can also indicate:

• Ambiguous task data.
• Labels that overlap conceptually.
• Missing or unclear labeling instructions.
• A project configuration that requires higher annotation overlap or additional review strategy.

Use the Data quality dashboard together with the Members dashboard and Data Manager to decide whether to update instructions, add examples, reassign tasks, or coach specific members.

Source Guide Dashboard Distribution (`docs/source/guide/dashboard_distribution.md`) Copy Section

title: Label distribution dashboard short: Label distribution tier: enterprise type: guide order: 0 order_enterprise: 175 meta_title: Label distribution dashboard meta_description: Use the Label Distribution dashboard to compare annotation and prediction label distributions by project dimension. section: "Analytics" date: 2026-06-29 00:00:00 Copy Chapter

The Analytics > Label Distribution dashboard shows how annotation and prediction values are distributed across a project. Use it to identify imbalanced labels, compare annotator and model outputs, and drill into the tasks behind a label value.

Access by user role Copy Chapter

Label distribution by dimension Copy Chapter

The dashboard groups results by labeling dimension. A dimension can come from a traditional labeling config control tag or from an Interface output, such as ReactCode or CustomInterface fields.

Each supported dimension has a card with:

• Average Agreement: Agreement for the dimension when there is enough data to compare annotations, predictions, or ground truth.
• Label Distribution chart: A grouped horizontal bar chart comparing values from annotations and predictions.
• Label breakdown table: Counts from annotations and predictions for each value.

Label breakdown table Copy Chapter

The label breakdown table shows each value for the selected dimension, plus a Total row.

Values in the From Annotations and From Predictions columns can link to the Data Manager with filters applied for the selected project, dimension, value, and source. Use these links to inspect the underlying tasks, annotations, or predictions.

Supported dimensions Copy Chapter

Label distribution supports dimensions with a finite set of values, including:

• Label controls such as <Labels>, <RectangleLabels>, and other labels-based controls.
• Choice controls such as <Choices>.
• Taxonomy controls with finite, nested choices.
• Enum-backed ReactCode and CustomInterface dimensions.
• Other finite scalar dimensions that can be represented as a known set of values.

Dimensions that do not have a finite set of values are not shown as label distributions. This includes free text, arbitrary numeric or date values, objects, arrays, and values that are not declared in the dimension schema.

!!! note
If a dimension is unsupported, the dashboard should show that it is unsupported rather than rendering an empty chart. This helps distinguish "no data" from data that cannot be safely summarized as a label distribution.

Source Guide Dashboard Members (`docs/source/guide/dashboard_members.md`) Copy Section

title: Members dashboard short: Members dashboard tier: enterprise type: guide order: 0 order_enterprise: 72 meta_title: Members dashboard meta_description: Use the Members dashboard to monitor annotator and reviewer performance, agreement, and quality metrics within a project. section: "Project Management" parent: "dashboards" parent_enterprise: "dashboards" date: 2026-06-17 00:00:00 Copy Chapter

Get insight into the performance, agreement, and quality of members working on a project.

The Members dashboard gives you two views of your project's workforce: a Performance tab with separate tables for annotators and reviewers, and an Agreement Matrix tab for inter-annotator reliability.

This can be useful in a number of ways:

• Track annotator and reviewer performance: View key metrics like evaluation status, agreement, acceptance score, performance score, GT agreement, and time spent to identify who is aligned or needs support.
• Monitor annotation and review progress: See how many annotations have been submitted, accepted, rejected, or are still pending review for both annotators and reviewers.
• Identify quality and consistency issues: Use the Agreement Matrix tab to spot disagreements between members, then click into cells to view the specific tasks where they diverged.

For member metrics across projects and over time, see the Member Performance dashboard.

Access the dashboard Copy Chapter

From the project, open the Dashboard tab and select Members from the dashboard navigation.

The Data quality dashboard has two tabs:

• Performance: Review key metrics for individual annotators and reviewers in the project.
• Agreement Matrix: Compare pairwise agreement scores between members, models, and ground truth.

Filter by participants Copy Chapter

A participant filter is available in the top right of the dashboard. By default it shows All participants.

• Annotators are listed under All Annotators; if the project has model predictions, model versions appear under All Model Versions.
• Selecting specific participants narrows both the Performance tables and the Agreement Matrix to only those members.

Performance Copy Chapter

The Performance tab has a toggle to switch between:

• Annotations: metrics for each member's submission activity
• Reviews: metrics for each member's review activity

Use the Export button to download the currently visible table as a CSV.

Annotation Summary Copy Section

The Annotation Summary table shows annotation metrics for each project member. The table is sortable and includes a totals row at the bottom. Use the Columns picker to show or hide columns.

!!! note
The following columns are hidden by default and can be shown using the Columns picker: ID, Assigned, Pending, Median Lead Time, Prediction Agreement.

Review Summary Copy Section

The Review Summary table shows review activity for each member who has performed reviews on the project.

!!! note
The following columns are hidden by default and can be shown using the Columns picker: ID, Performance, Submitted tasks.

!!! note
Review time data collection began on September 25, 2025 for cloud environments, or version 2.30.0 for on-prem deployments. Filtering for earlier dates will not include review time.

Performance by Dimension Copy Section

The Performance by Dimension table appears below the Annotation Summary on the Annotations sub-tab. It breaks down each member's performance score per labeling dimension configured in the project.

Use the Compare with dropdown to choose how scores are calculated:

• Compare with GT — scores reflect each member's agreement with ground truth annotations on overlapping tasks.
• Compare with All Annotators — scores reflect each member's agreement with other annotators on the same tasks.

An Avg column shows each member's overall average across all dimensions. A footer row shows the average per dimension across all members. Use the Columns picker to show or hide individual dimensions.

Agreement Matrix Copy Chapter

The Agreement Matrix tab shows the Member / Model Agreement Matrix, a heatmap of pairwise agreement scores between all members (and models, if included via the participant filter) in the project.

Click on any cell to open the Data Manager filtered to tasks where both members annotated, so you can inspect disagreements directly.

!!! note
Agreement in the Members dashboard reflects an agreement score between annotators, regardless of the agreement methodology configured for the project.

Filters Copy Section

• Dimension: Filter agreement by a specific labeling dimension (control tag). Defaults to All dimensions.
• Label: Filter to tasks where at least one annotation contains the specified label. Only available when a specific dimension is selected.

Reading the matrix Copy Section

Cells are color-coded by agreement band:

• Green — High (≥ 66%)
• Yellow — Moderate (33–66%)
• Red — Low (< 33%)

Hover over any cell to see the agreement percentage and the number of tasks where both members annotated. If a Ground Truth column is present, it shows each member's agreement against the project's ground truth annotations.

An Avg row at the bottom shows the average agreement per column across all members.

The participant filter (top right) applies to this tab — select specific annotators or model versions to narrow the matrix.

Source Guide Dashboard Throughput (`docs/source/guide/dashboard_throughput.md`) Copy Section

title: Throughput dashboard short: Throughput dashboard tier: enterprise type: guide order: 0 order_enterprise: 70 meta_title: Throughput dashboard meta_description: Use the Throughput dashboard to monitor project velocity, remaining work, and time spent. section: "Project Management" parent: "dashboards" parent_enterprise: "dashboards" date: 2026-06-29 00:00:00 Copy Chapter

Track how quickly work is moving through a project and where bottlenecks are forming.

The Throughput dashboard shows project velocity over time. Use it to monitor completed work, remaining work, and the time spent annotating and reviewing tasks.

For data quality-focused metrics, see the Data quality dashboard. For member-level productivity and agreement, see the Members dashboard.

Access the dashboard Copy Chapter

From the project, open the Dashboard tab and select Throughput from the dashboard navigation.

Filter by date range Copy Chapter

Use the date range filter to update the KPI cards and charts for the selected time period.

If you include a precise time, the dashboard uses that timestamp when determining which task and work events fall within the range.

KPI cards Copy Chapter

The top of the dashboard shows summary cards for the selected date range.

!!! note
Task state metrics are based on project and task state management. If state tracking was enabled after some project work had already happened, earlier state transitions might not appear in the dashboard.

Throughput charts Copy Chapter

Source Guide Dashboards (`docs/source/guide/dashboards.md`) Copy Section

title: Dashboards short: Dashboards tier: enterprise type: guide order: 0 order_enterprise: 135 meta_title: Dashboard in Label Studio meta_description: Use the project Dashboard tab to monitor project throughput, quality, and member performance. section: "Create & Manage Projects" Copy Chapter

Use project dashboards to monitor throughput, data quality, and member performance for a single project.

The project Dashboard tab gives managers, administrators, and owners a project-level view of how work is moving, where quality issues are appearing, and how members are performing.

Open a project and select Dashboard to access the project dashboards:

• Throughput: Track velocity, remaining work, and time spent.
• Data quality: Analyze agreement, dimension-level quality, and label confusion.
• Members: Review annotator, reviewer, and model performance within the project.

Choose a project dashboard Copy Chapter

Access by user role Copy Chapter

Project dashboards are available to users who can manage or monitor the project, such as Owners, Administrators, and Managers.

Annotators and Reviewers generally do not use the project Dashboard tab. For individual work history across projects, see the Member Performance dashboard.

Filtering and drilldowns Copy Chapter

Each project dashboard has filters that apply to the metrics on that page. Depending on the dashboard, you can filter by date range, task data, annotator, model, review outcome, ground truth status, labeling dimension, or label.

Many cards, charts, and tables link to the Data Manager with relevant filters applied. Use these drilldowns to move from a summary metric to the exact tasks, annotations, predictions, or labels behind it.

Analytics dashboards Copy Chapter

Analytics dashboards are available from the main Analytics navigation and can include data across projects, workspaces, or individual members depending on your role.

Source Guide Decommission Minio (`docs/source/guide/decommission_minio.md`) Copy Section

title: Decommission minio short: Decommission minio meta_title: Decommission minio Copy Chapter

!!! warning
Decommissioning Minio is a hazardous action because you may lose all your data uploaded using direct file upload. For more information or support, contact the Heartex team.

To decommission Minio, do the following:

Configure Persistent Storage Copy Chapter

To configure Persistent Storage, follow the instructions in Set up persistent storage.

Upgrade to the latest available Release Copy Chapter

1. Add MINIO_MIGRATION: true into global.extraEnvironmentVars section of your ls-values.yaml file:

global:
  extraEnvironmentVars:
    MINIO_MIGRATION: true

2. Redeploy a release by following the steps from Upgrade Label Studio Enterprise.

Run migration script Copy Chapter

1. Run shell in lse-app pod:

kubectl exec -ti deploy/<YOUR_RELEASE_NAME>-lse-app -c lse-app -- bash

2. Copy data from Minio to a persistent storage:

JSON_LOG=0 python3 $LSE_DIR/label_studio_enterprise/manage.py minio-migrate

Ensure a successful data migration Copy Chapter

!!! warning
Minio service will be deleted in the next step.

• Try to run Export in your project.
• Ensure that this data appears in your bucket/volume by project_id/export.

!!! note
For more information or support, contact the Heartex team.

Completely remove Minio service Copy Chapter

1. Add the following map to your ls-values.yaml file:

minio:
  enabled: false

2. Remove MINIO_MIGRATION: true from global.extraEnvironmentVars section of your ls-values.yaml file.

3. Redeploy a release by following steps from Upgrade Label Studio Enterprise.

Source Guide Email Setup (`docs/source/guide/email_setup.md`) Copy Section

title: Set up email backend for Label Studio Enterprise
short: Set up email backend
tier: enterprise
type: guide
order: 0
order_enterprise: 90
meta_title: Email backends in Label Studio
section: "Install & Setup"

In Label Studio Enterprise, you can configure email backends to enable password reset via email and receive notifications. There are three available options for setting up email backends:

• Dummy console email backend - all emails will be printed in the app console.
• SMTP backend
• Sendgrid backend

Dummy Console Backend Copy Section

The Dummy Console Email Backend is a simple option for testing purposes. When this backend is configured, all emails generated by the application will be printed in the application console, rather than being sent to recipients.

FROM_EMAIL = Label Studio <[email protected]>

# option 1: app console
EMAIL_BACKEND = 'django.core.mail.backends.console.EmailBackend'

SMTP Backend Copy Section

The SMTP (Simple Mail Transfer Protocol) Backend allows you to use a custom SMTP server to send emails. This provides flexibility in configuring email delivery options and can integrate with various email service providers.

More details can be found here: https://docs.djangoproject.com/en/4.2/topics/email/#smtp-backend. If your SMTP configuration is not compatible with certificate or hostname validation, we provide the
label_studio.core.utils.mail.NoVerificationEmailBackend which can be used instead of Django's default smtp.EmailBackend, but note that this option is less secure. See https://docs.djangoproject.com/en/5.0/releases/4.2/#miscellaneous for more information.

# SMTP server,
FROM_EMAIL=Label Studio <[email protected]>
EMAIL_BACKEND=django.core.mail.backends.smtp.EmailBackend
EMAIL_HOST=smtp.gmail.com
EMAIL_PORT=587
[email protected]
EMAIL_HOST_PASSWORD=PASSWORD # It's your account password, or separate application password, if you have 2fa on your google account
EMAIL_USE_TLS=1
EMAIL_USE_SSL=0
# EMAIL_SSL_KEYFILE=/path/inside/of/ls/container/key  # optional
# EMAIL_SSL_CERTFILE=/path/inside/of/ls/container/cert  # optional
EMAIL_TIMEOUT=60

Sendgrid Backend Copy Section

The Sendgrid Backend utilizes the Sendgrid API to send emails. This option requires an active Sendgrid account and API key for authentication. The Sendgrid backend offers an easy-to-use and reliable email delivery service.

# option 2: Sendgrid
EMAIL_BACKEND=sendgrid_backend.SendgridBackend
SENDGRID_API_KEY=<SENDGRID-API-KEY>

Source Guide Embed (`docs/source/guide/embed.md`) Copy Section

title: Embed Label Studio - Beta 🧪 short: Embed Label Studio 🧪 type: guide tier: enterprise order: 409 order_enterprise: 409 meta_title: Embed Label Studio meta_description: Overview information for embedding Label Studio Enterprise in an external application. section: "Integrate & Extend" Copy Chapter

Label Studio Embeds allow you to seamlessly integrate annotation and review functionality into your own applications.

!!! note
This feature is not available to all customers. Contact your HumanSignal account manager to enable.

Before you begin, ensure you have:

• Label Studio:

  • You must have the Owner role in Label Studio.
  • From Organization > Settings > Access Token Settings, ensure that at least one option is enabled.

• Your external application:

  • Access to your server backend (to generate signed tokens).
  • The URL where your embed will be hosted.

Authorization Copy Chapter

At a high level, the embed process is a secure handshake between your app and Label Studio:

1. A user logs into your external app.

2. Your backend signs a JWT with their user_email, organization_id, and project_id using your private key.

3. Your frontend initializes the Label Studio embed with id, url, and token.

4. Label Studio validates the JWT and issues its own internal token.

sequenceDiagram
    participant User
    participant YourApp as Your App (Frontend & Backend)
    participant LSE as Label Studio Enterprise

    User->>YourApp: Logs in
    YourApp->>YourApp: Backend signs JWT (user_email + organization_id + project_id)
    YourApp->>User: Sends JWT to frontend
    User->>LSE: Initializes Embed with (id, url, token)
    LSE->>LSE: Verifies JWT with Public Key
    LSE->>User: Issues Internal Token
    User->>LSE: Annotates, Reviews, Uses Embed

Generate a JWT on your server Copy Section

You must begin by generating a JSON web token (JWT) that authenticates Label Studio.

You will also need to generate a base64-encoded public URL key, which you will add to Label Studio.

JWT claims required:

{
  "user_email": "[email protected]",
  "organization_id": 12345,
  "project_id": 67890,
  "iat": 1692445200,
  "exp": 1692452400
}

!!! note
exp, organization_id, and project_id are required. Each accepts a positive integer or a string of digits. exp is a Unix timestamp in seconds and must fall within your organization's token lifetime (60 minutes by default). project_id scopes the embed to that project and must match the project in the embed URL. iat is optional.

Notes:

• You can find your organization ID from the Organization page in Label Studio.
• The example below uses RS256 as a verification algorithm, but you can find a full list of supported algorithms from Organization > Usage & License > Embedding.
• The public key must be stored in Label Studio in base64-url encoded format.

Example: Node.js (jsonwebtoken)

const jwt = require("jsonwebtoken");
const fs = require("fs");

const privateKey = fs.readFileSync("./private.pem");

const token = jwt.sign(
  {
    user_email: "[email protected]",
    organization_id: 12345,
    project_id: 67890,
  },
  privateKey,
  { algorithm: "RS256", expiresIn: "1h" }
);

console.log(token);

Configure embedding in Label Studio Copy Chapter

From Label Studio, go to Organization > Usage & License > Embedding and complete the following fields:

Configure an embedding page in your external application Copy Chapter

At a minimum, you need to add the Embed SDK script to an HTML page in your application. For example:

<script src="https://app.humansignal.com/react-app/embed-sdk.js"></script>

But to maximize the usability of your embedding, you can also include additional Label Studio elements. See our Frontend Reference.

Here is an example:

/* Detailed source-code truncated for AI context efficiency. */

Troubleshooting Copy Chapter

Blank/white screen

• Domain not listed in Supported Domains within Label Studio.
• Misconfigured/missing public verification key
• Misconfigured/missing public verification algorithms
• Verification of the external JWT failed to verify one or more of the claims provided (invalid JWT).
• Active organization of the user was different than specified in the JWT claims.
• Your org has not enabled legacy API tokens or personal access tokens (need at least one).
• User with the provided user_email does not exist on LabelStudioEnterprise

HTTP errors in SDK

• 403 (permission errors)

  • User is not a member of the Project
  • Task was not assigned to a user, but attempted to load
  • User did not have access to an annotation or prediction, but it attempted to load anyways

• 404

  • Task was not found

• Other non-HTTP errors that caused a logical problem in the system.

Source Guide Enterprise Features (`docs/source/guide/enterprise_features.md`) Copy Section

title: "Enterprise features: Label Studio at scale" short: Enterprise features type: guide tier: opensource order: 35 order_enterprise: 0 section: "Discover & Learn" meta_title: Label Studio Enterprise Features meta_description: Overview of the features in Label Studio Enterprise. Copy Chapter

This page provides an overview of the advanced capabilities available exclusively in Label Studio Enterprise. These features extend the core functionality of Label Studio with enterprise-scale user management, security, quality assurance, and automation.

Organizations and Workspaces Copy Chapter

Label Studio Enterprise introduces a hierarchical structure for organizing projects and users through organizations and workspaces, which is not available in the community edition.

Organizations Copy Section

Organizations are the top-level entity in Label Studio Enterprise that contain workspaces, projects, and users. Each organization is isolated from others, providing complete data separation.

Key capabilities of organizations:

• Organization-level user management
• Isolated data and resources
• Centralized billing and usage tracking
• Custom branding options

Workspaces Copy Section

Workspaces provide a way to group related projects within an organization. They help structure projects by department, team, or use case.

Key capabilities of workspaces:

• Group related projects together
• Delegate management responsibilities
• Control access at the workspace level
• Share resources between projects in the same workspace

Multi-tenant SaaS architecture Copy Chapter

Organizations provide complete isolation between different Label Studio Enterprise SaaS customers, ensuring data segregation and independent user management. Workspaces within organizations group related projects by team, department, or product line. Label Studio Enterprise may also be deployed on-prem (see Enterprise Deployment Options).

User management and role-based access control Copy Chapter

Role hierarchy and permissions Copy Section

Label Studio Enterprise implements a comprehensive role-based access control (RBAC) system with five distinct roles, each providing specific capabilities and access levels. For information on how to assign users to roles, see Manage user accounts.

graph TD
    subgraph Project Level
        Project1[Project 1]
        Project2[Project 2]
        Project3[Project 3]
        ReviewerRole[Reviewer Role]
        AnnotatorRole[Annotator Role]
        Project1 --> AnnotatorRole
        Project2 --> AnnotatorRole
        Project3 --> ReviewerRole
    end

    subgraph Workspace Level
        WorkspaceA[Workspace A]
        WorkspaceB[Workspace B]
        ManagerRole[Manager Role]
        WorkspaceA --> ManagerRole
        WorkspaceB --> ManagerRole
    end

    subgraph Organization Level
        Organization
        OwnerRole[Owner Role]
        AdminRole[Admin Role]
        Organization --> OwnerRole
        Organization --> AdminRole
    end

    Organization --> WorkspaceA
    Organization --> WorkspaceB
    WorkspaceA --> Project1
    WorkspaceA --> Project2
    WorkspaceB --> Project3

Project-level roles Copy Section

Project roles are assigned independently of organization roles. For example, a user can be an Annotator in one project and a Reviewer in another project, allowing for flexible permission management:

• Annotator: Can view projects and assigned tasks and can create annotations
• Reviewer: Can review annotations, accept or reject them, and provide feedback

User lifecycle management Copy Section

Enterprise features include comprehensive user account management capabilities:

• Invitation-based signup: Control user onboarding through invitation links
• Bulk user operations: Assign roles and update workspace memberships in bulk
• User activity monitoring: Track annotation activity and system usage
• Account deactivation: Preserve annotation history while removing access

Authentication and identity management Copy Chapter

Label Studio Enterprise provides comprehensive identity management capabilities for large-scale deployments, supporting integration with existing enterprise identity providers.

SSO and SAML integration Copy Section

The enterprise edition supports Single Sign-On through SAML 2.0 assertions, allowing integration with identity providers like Okta, Google SAML, Ping Identity, Microsoft Active Directory, and other SAML-compatible systems. For setup instructions, see the Authentication and SSO Configuration Guide.

SCIM 2.0 provisioning Copy Section

System for Cross-domain Identity Management (SCIM) 2.0 enables automated user provisioning and deprovisioning from identity providers.

The SCIM integration supports:

• Create Users: Automatic user provisioning with role assignments
• Update User Attributes: Synchronize profile changes and role modifications
• Deactivate Users: Remove access while preserving annotation history
• Push Groups: Workspace and project membership management

For setup instructions, see the SCIM 2.0 Integration Guide.

LDAP integration Copy Section

Enterprise customers can integrate with LDAP directories for user authentication while maintaining local role management.

Advanced security features Copy Chapter

Enterprise security architecture Copy Section

Label Studio Enterprise includes comprehensive security features designed for enterprise compliance requirements. Enterprise security features include:

• Redis over TLS with client certificate authentication
• IP allowlisting and VPN access control for cloud storage
  (OSS supports limited IP filtering only for GCP)
• Advanced audit logging
• SCIM 2.0 user provisioning and deprovisioning
• SAML-based Single Sign-On (SSO) integration
• Role-based access controls across organizations and workspaces
• Granular task and annotator guardrails (e.g. auto-pausing users, setting trust thresholds)
• Custom plugins and front-end customization with organization-level isolation
• Deployment support for fully air-gapped, on-prem environments
• Uptime SLA and security certifications (e.g. SOC 2, HIPAA readiness)

Data security Copy Section

• End-to-end TLS/SSL: Secure communication between all components
• Audit Logging: Comprehensive logging of all user activities

Advanced cloud storage options Copy Section

Label Studio Enterprise supports a wider range of persistent storage options for scalable, production-grade deployments. These include:

• AWS S3 IAM role access: Configure secure access using IAM roles or static credentials
• Google Cloud Storage with Workload Identity Federation (WIF): Use short-lived credentials to avoid hardcoded secrets

Advanced labeling & evaluation workflows Copy Chapter

AI features and model integration Copy Section

Label Studio Enterprise provides enhanced machine learning capabilities:

• AI Features Toggle: Enable AI helper tools
• Prediction Score Display: View prediction/confidence scores in the labeling interface
• Bulk Labeling: Quickly label multiple tasks at once

Automated active learning loops Copy Section

Label Studio Enterprise supports active learning workflows that automatically update model predictions as labeling progresses.

After each annotation, Label Studio can send a webhook to a connected machine learning backend. The backend uses the new data to retrain the model (fit()), and Label Studio fetches updated predictions (predict()) for the next task, creating a continuous feedback loop.

Key components:

• ML backend setup: Register a model to receive training data and return predictions
• Prediction integration: Display updated predictions during labeling
• Webhooks: Trigger model retraining after annotations
• Task sampling: Use prediction scores to prioritize which tasks to label
• Continuous loop: As labeling continues, predictions improve with each iteration

Prompts Copy Section

For an additional fee, Label Studio Enterprise includes a built-in Prompts interface for working with language models as part of the labeling workflow.

Prompt templates allow users to define inputs, models, and outputs to send to connected LLMs. Prompts can be used to pre-label data, generate new tasks, or support review workflows.

The Prompts interface supports:

• Prompt Templates: Define inputs, outputs, and model parameters
• Inline Generation: Run prompts directly from the labeling interface
• Model Connections: Connect to OpenAI, Anthropic, or custom endpoints via API
• Pre-labeling: Populate fields using model-generated responses
• Task Creation: Generate new tasks from prompt outputs

For more information, see the Prompts Overview.

Customization with plugins Copy Section

Label Studio Enterprise supports custom JavaScript plugins to extend and tailor the labeling interface for project-specific needs.

Plugins run automatically when users interact with annotation tasks. They can be used to validate annotations, add visual elements, call external services, or apply custom logic based on events in the labeling UI.

The plugin framework supports:

• Project-specific scripts: Define custom plugins on a per-project basis
• Event-driven behavior: Trigger logic on task load, annotation switch, and other actions
• UI extensions: Add validations, overlays, or context-aware functionality
• External integration: Connect to APIs or fetch reference data in real time
• Testing and preview: Use the built-in panel to test plugins before deployment

To learn how organizations are using plugins in real-world workflows, see the blog post Build Powerful Labeling Interfaces with Plugins.

Quality workflows and analytics Copy Chapter

Label Studio Enterprise enhances the annotation workflow with features designed for quality control and team collaboration, as well as analytics to optimize annotation and dataset quality.

Quality control features Copy Section

• Auto-validation
• Review Workflows: Configure review processes for annotations
• Inter-annotator agreement: Measure agreement between annotators to ensure quality
• Annotator Performance Dashboard: Track and analyze annotator quality to identity low performers, implement training and corrective actions
• Project Performance Dashboard: View label distribution to ensure completeness
• Guardrails for low-trust annotators: Set limits or pause annotators to prevent

Inter-annotator agreement metrics and matrices Copy Section

View real-time agreement metrics to take proactive action on quality. The Enterprise edition includes 30+ built-in agreement metrics for different annotation types, including:

• Basic matching: Default evaluation for each control tag type
• Intersection over Union (IoU): For bounding boxes and polygons
• Edit distance algorithms: For text transcription tasks
• F1, Precision, Recall: For classification and detection tasks

To learn how different team structures use agreement metrics to scale high-quality labeling, see the blog post 3 Annotation Team Playbooks for Scalable Labeling.

Low-trust annotator guardrails Copy Section

• Annotator pausing: Manual and automatic pausing based on performance criteria
• Annotation limits: Configure maximum annotations per user to prevent spam

Project performance dashboards Copy Section

The project dashboard provides a high-level view of project status over time, including metrics like tasks completed, annotations submitted, reviews performed, label distribution, and lead times. Users can filter by date ranges and reorder visualizations to focus on key indicators.

Annotator performance dashboards Copy Section

Organization owners and administrators can view individual annotator dashboards to analyze task counts, average annotation times, review turnaround, and performance scoring. These dashboards help in managing resources, identifying top performers, and tracking quality without building custom analytics tools.

Project Management Copy Chapter

Label Studio Enterprise includes advanced project configuration options to support complex workflows across teams and workspaces.

Enterprise-Only Project Settings Copy Section