# Technical Documentation: HumanSignal/label-studio

> ℹ️ **Provenance:** Multi-Tier Ingestion · [CodeWiki Reference](https://codewiki.google/github.com/HumanSignal/label-studio)

### Overview (README.md)

<img src="https://user-images.githubusercontent.com/12534576/192582340-4c9e4401-1fe6-4dbb-95bb-fdbba5493f61.png"/>

  

[Website](https://labelstud.io/) • [Docs](https://labelstud.io/guide/) • [Join Slack Community <img src="https://app.heartex.ai/docs/images/slack-mini.png" width="18px"/>](https://slack.labelstud.io/?source=github-1)


## What is Label Studio?



Label Studio is an open source data labeling tool. It lets you label data types like audio, text, images, videos, and time series with a simple and straightforward UI and export to various model formats. It can be used to prepare raw data or improve existing training data to get more accurate ML models.

- [Try out Label Studio](#try-out-label-studio)
- [What you get from Label Studio](#what-you-get-from-label-studio)
- [Included templates for labeling data in Label Studio](#included-templates-for-labeling-data-in-label-studio)
- [Set up machine learning models with Label Studio](#set-up-machine-learning-models-with-Label-Studio)
- [Integrate Label Studio with your existing tools](#integrate-label-studio-with-your-existing-tools)



Have a custom dataset? You can customize Label Studio to fit your needs. Read an [introductory blog post](https://towardsdatascience.com/introducing-label-studio-a-swiss-army-knife-of-data-labeling-140c1be92881) to learn more. 

## Try out Label Studio

Install Label Studio locally or deploy it in a cloud instance. [Or sign up for a free trial of our Starter Cloud edition!](https://humansignal.com/platform/starter-cloud/) You can learn more about what each edition offers [here](https://labelstud.io/guide/label_studio_compare). 

- [Install locally with Docker](#install-locally-with-docker)
- [Run with Docker Compose (Label Studio + Nginx + PostgreSQL)](#run-with-docker-compose)
- [Install locally with pip](#install-locally-with-pip)
- [Install locally with poetry](#install-locally-with-poetry)
- [Install locally with Anaconda](#install-locally-with-anaconda)
- [Install for local development](#install-for-local-development)
- [Deploy in a cloud instance](#deploy-in-a-cloud-instance)

### Install locally with Docker
Official Label Studio docker image is [here](https://hub.docker.com/r/heartexlabs/label-studio) and it can be downloaded with `docker pull`. 
Run Label Studio in a Docker container and access it at `http://localhost:8080`.


```bash
docker pull heartexlabs/label-studio:latest
docker run -it -p 8080:8080 -v $(pwd)/mydata:/label-studio/data heartexlabs/label-studio:latest
```
You can find all the generated assets, including SQLite3 database storage `label_studio.sqlite3` and uploaded files, in the `./mydata` directory.

#### Override default Docker install
You can override the default launch command by appending the new arguments:
```bash
docker run -it -p 8080:8080 -v $(pwd)/mydata:/label-studio/data heartexlabs/label-studio:latest label-studio --log-level DEBUG
```

#### Build a local image with Docker
If you want to build a local image, run:
```bash
docker build -t heartexlabs/label-studio:latest .
```

### Run with Docker Compose
Docker Compose script provides production-ready stack consisting of the following components:

- Label Studio
- [Nginx](https://www.nginx.com/) - proxy web server used to load various static data, including uploaded audio, images, etc.
- [PostgreSQL](https://www.postgresql.org/) - production-ready database that replaces less performant SQLite3.

To start using the app from `http://localhost` run this command:
```bash
docker-compose up
```

### Run with Docker Compose + MinIO
You can also run it with an additional MinIO server for local S3 storage. This is particularly useful when you want to 
test the behavior with S3 storage on your local system. To start Label Studio in this way, you need to run the following command:
````bash
# Add sudo on Linux if you are not a member of the docker group
docker compose -f docker-compose.yml -f docker-compose.minio.yml up -d
````
If you do not have a static IP address, you must create an entry in your hosts file so that both Label Studio and your 
browser can access the MinIO server. For more detailed instructions, please refer to [our guide on storing data](docs/source/guide/storedata.md).


### Install locally with pip

```bash
# Requires Python >=3.10
pip install label-studio

# Start the server at http://localhost:8080
label-studio
```

### Install locally with poetry

```bash
### install poetry
pip install poetry

### set poetry environment
poetry new my-label-studio
cd my-label-studio
poetry add label-studio

### activate poetry environment
poetry shell

### Start the server at http://localhost:8080
label-studio
```

### Install locally with Anaconda

```bash
conda create --name label-studio
conda activate label-studio
conda install psycopg2
pip install label-studio
```

### Install for local development

You can run the latest Label Studio version locally without installing the package from pypi. 

```bash
# Install all package dependencies
pip install poetry
poetry install
# Run database migrations
python label_studio/manage.py migrate
python label_studio/manage.py collectstatic
# Start the server in development mode at http://localhost:8080
python label_studio/manage.py runserver
```

### Deploy in a cloud instance

You can deploy Label Studio with one click in Heroku, Microsoft Azure, or Google Cloud Platform: 

<a href="https://www.heroku.com/deploy?template=https://github.com/HumanSignal/label-studio/tree/heroku-persistent-pg"><img src="https://www.herokucdn.com/deploy/button.svg" alt="Deploy" height="30px"></a>
[<img src="https://aka.ms/deploytoazurebutton" height="30px">](https://portal.azure.com/#create/Microsoft.Template/uri/https%3A%2F%2Fraw.githubusercontent.com%2Fhumansignal%2Flabel-studio%2Fdevelop%2Fazuredeploy.json)
[<img src="https://deploy.cloud.run/button.svg" height="30px">](https://deploy.cloud.run)


#### Apply frontend changes

For information about updating the frontend, see [label-studio/web/README.md](https://github.com/HumanSignal/label-studio/blob/develop/web/README.md#installation-instructions).


#### Install dependencies on Windows 
To run Label Studio on Windows, download and install the following wheel packages from [Gohlke builds](https://www.lfd.uci.edu/~gohlke/pythonlibs) to ensure you're using the correct version of Python:
- [lxml](https://www.lfd.uci.edu/~gohlke/pythonlibs/#lxml)

```bash
# Upgrade pip 
pip install -U pip

# If you're running Win64 with Python 3.8, install the packages downloaded from Gohlke:
pip install lxml‑4.5.0‑cp38‑cp38‑win_amd64.whl

# Install label studio
pip install label-studio
```

### Run test suite
To add the tests' dependencies to your local install:

```bash
poetry install --with test
```

Alternatively, it is possible to run the unit tests from a Docker container in which the test dependencies are installed:


```bash
make build-testing-image
make docker-testing-shell
```

In either case, to run the unit tests:

```bash
cd label_studio

# sqlite3
DJANGO_DB=sqlite DJANGO_SETTINGS_MODULE=core.settings.label_studio pytest -vv

# postgres (assumes default postgres user,db,pass. Will not work in Docker
# testing container without additional configuration)
DJANGO_DB=default DJANGO_SETTINGS_MODULE=core.settings.label_studio pytest -vv
```
 
## What you get from Label Studio

https://github.com/user-attachments/assets/525ad5ff-6904-4398-b507-7e8954268d69

- **Multi-user labeling** sign up and login, when you create an annotation it's tied to your account.
- **Multiple projects** to work on all your datasets in one instance.
- **Streamlined design** helps you focus on your task, not how to use the software.
- **Configurable label formats** let you customize the visual interface to meet your specific labeling needs.
- **Support for multiple data types** including images, audio, text, HTML, time-series, and video. 
- **Import from files or from cloud storage** in Amazon AWS S3, Google Cloud Storage, or JSON, CSV, TSV, RAR, and ZIP archives. 
- **Integration with machine learning models** so that you can visualize and compare predictions from different models and perform pre-labeling.
- **Embed it in your data pipeline** REST API makes it easy to make it a part of your pipeline

## Included templates for labeling data in Label Studio 

Label Studio includes a variety of templates to help you label your data, or you can create your own using specifically designed configuration language. The most common templates and use cases for labeling include the following cases:

<img src="/images/template-types.png" />

## Set up machine learning models with Label Studio

Connect your favorite machine learning model using the Label Studio Machine Learning SDK. Follow these steps:

1. Start your own machine learning backend server. See [more detailed instructions](https://github.com/HumanSignal/label-studio-ml-backend).
2. Connect Label Studio to the server on the model page found in project settings.

This lets you:

- **Pre-label** your data using model predictions. 
- Do **online learning** and retrain your model while new annotations are being created. 
- Do **active learning** by labeling only the most complex examples in your data.

## Integrate Label Studio with your existing tools

You can use Label Studio as an independent part of your machine learning workflow or integrate the frontend or backend into your existing tools.  

## Ecosystem

| Project | Description |
|-|-|
| label-studio | Server, distributed as a pip package |
| [Frontend library](web/libs/editor/) | The Label Studio frontend library. This uses React to build the UI and mobx-state-tree for state management. |  
| [Data Manager library](web/libs/datamanager/) | A library for the Data Manager, our data exploration tool. | 
| [label-studio-converter](https://github.com/HumanSignal/label-studio-sdk/tree/master/src/label_studio_sdk/converter) | Encode labels in the format of your favorite machine learning library |
| [label-studio-transformers](https://github.com/HumanSignal/label-studio-transformers) | Transformers library connected and configured for use with Label Studio |

## Citation

Include a citation for Label Studio in the **References** section of your articles:

```tex
@misc{Label Studio,
  title={{Label Studio}: Data labeling software},
  url={https://github.com/HumanSignal/label-studio},
  note={Open source software available from https://github.com/HumanSignal/label-studio},
  author={
    Maxim Tkachenko and
    Mikhail Malyuk and
    Andrey Holmanyuk and
    Nikolai Liubimov},
  year={2020-2025},
}
```

## License

This software is licensed under the [Apache 2.0 LICENSE](/LICENSE) © [Heartex](https://www.heartex.com/). 2020-2025

<img src="https://user-images.githubusercontent.com/12534576/192582529-cf628f58-abc5-479b-a0d4-8a3542a4b35e.png" title="Hey everyone!" width="180" />

---

### README (`docs/README.md`)

# Documentation of Label Studio

## Use and deploy Hexo

### Installing Dependencies

```shell
npm install
```

### Starting Development server

```shell
npm run server
```

Starts a local server. By default, this is at http://localhost:4000/.

### Deploying Documentation

```shell
npm run publish
```

## Deploy the docs locally using Hexo
To deploy the docs locally on your machine using Hexo, use the following steps. 

### Prerequisites
- Install Hexo
- Clone the Label Studio Github repository 

### Deploy the docs locally
In the label-studio/docs directory of the cloned repo, do the following:
1. (First time) Install required dependencies:
```shell
npm install
```
2. Start the Hexo server:

```shell
hexo serve
```


## Hexo Official Documentation 
[https://hexo.io/docs/](https://hexo.io/docs/)

---

### Source Guide Dashboard Overview (`docs/source/guide/dashboard_overview.md`)

---
title: Projects overview dashboard
short: Projects overview
tier: enterprise
type: guide
order: 0
order_enterprise: 172
meta_title: Projects Overview Dashboard
meta_description: Use the analytics overview page to explore progress across multiple projects 
section: "Analytics"
date: 2026-01-10 11:50:41
---

The **Analytics > Overview** page is a centralized starting point to explore performance insights across multiple projects. 



#### Access by user role

| User role | Access restrictions |
|-----------|---------------------|
| **Owner** and **Admin** | Can view the **Overview** dashboard and filter for all workspaces and projects. |
| **Manager** | Can view the **Overview** dashboard, but can only filter for and view workspaces or projects in which they are a member.  |
| **Reviewer** and **Annotator** | Cannot access the **Overview** dashboard. <br/><br/>When they open **Analytics**, they are redirected to the [Member Performance dashboard](dashboard_annotator), where they can only see their own annotation and review history. |

## Filtering

### Filter by project and workspace

Select one or more workspaces or individual projects. To see all projects, select **All Workspaces** at the top of the workspace filter. 

This does not include Personal Sandbox projects, and is limited to 50 projects at a time. 


### Filter by date range

You can select a date range to apply to the project metrics. If you want to be precise, you can also include time. 

Note that annotation and acceptance metrics are calculated on when they are **completed**, not when they **begin**. 

For example, say you select a date range of 2 days: Monday and Tuesday. If an annotator opens an annotation on Tuesday, begins working on it, and then submits it on Wednesday, then that annotation and the time they spent on it will **not** be included in the metrics.   

### Save views

You can save your date range and project filters as custom views. Saved views makes it easier for you to group dates and projects you want to readily access. 

Note the following:

* You can save, update, and delete views. 
* Views are only visible to you (meaning they are not shared with other users within your organization).

!!! note
    If you click **Apply** after saving your date and project/workspace filters, your selections will propogate to other Analytics pages. You can reapply your view or incorporate those filters into a new or updated view. 

## Tasks and project states

For more information about the states referenced in the Overview dashboard, see [Project and task state management](project_states). 

!!! note

    Project and task state history tracking did not start until state management was implemented for your organization. For most Label Studio Cloud organizations, state management was implemented in February 2026. 

    This means that the **Overview** dashboard will not reflect when a project or task changed states if that change happened before state management was implemented. 


## Project progress summaries

At the top of the Overview dashboard, summary cards show aggregate metrics for the selected date range and workspaces or projects.

!!! note
    A dash (`--`) appears when there is no data. A zero (`0`) appears when there is data available and the value is zero.


| Metric | Description |
|--------|--------------|
| **Total Done Tasks** | Total number of tasks in the [**Done** state](project_states#Task-states) across the selected projects. |
| **Total Tasks Annotated** | Total number of tasks that have the required annotations completed across all selected projects. This includes tasks in the [**Done** state, **Needs Review** state, and the **In Review** state](project_states#Task-states). |
| **Total Annotations Submitted** | Total number of annotations submitted across all selected projects. This includes annotations that have been submitted or updated, whether or not they have been reviewed. |
| **Total Time Spent** | Total time spent annotating (lead time) and reviewing across all selected projects. <br/><br/> The lead time is the time spent on annotations in which the last action was one of the following: **Submit**, **Update**, **Fix + Accept**, **Accept**, **Reject**. <br/><br/>Lead time includes idle time and is cumulative across multiple sessions. |


## Project progress charts

The Overview dashboard includes charts that break down metrics over the selected time period. 

Dates for annotations and reviews are based on when they were completed, not when they were started.




| Chart | Description |
|-------|--------------|
| **Task Progress** | Shows how tasks move through [states](project_states#Task-states) over the selected period of time.|
| **Annotations Submitted vs Accepted** | Shows submitted annotations and accepted annotations over time, with an acceptance score line (the acceptance score is the number of accepted annotations divided by the number of annotations reviewed). Note that accepted annotations includes annotations that were fixed and then accepted. |
| **Time Spent** | Shows time spent annotating and reviewing over time. Includes total time in the period and average time per submitted annotation. |


## Project progress table

The **Project Progress** table display detailed metrics for each project. 



!!! note
    A dash (`--`) appears when there is no data. A zero (`0`) appears when there is data available and the value is zero.

| Column | Description |
|--------|--------------|
| **Workspace** | The workspace that contains the project. |
| **Project** | The project name. Click to open the project dashboard. |
| **Task Count** | Number of tasks in the project. This includes every task and is not date filtered. |
| **Tasks (Initial)** | Number of tasks in the [**Initial** state](project_states#Task-states). |
| **Tasks (Annotating)** | Number of tasks in the [**Annotating** state](project_states#Task-states). |
| **Tasks (Needs Review)** | Number of tasks in the [**Needs Review** state](project_states#Task-states). |
| **Tasks (In Review)** | Number of tasks in the [**In Review** state](project_states#Task-states). |
| **Tasks (Done)** | Number of tasks in the [**Done** state](project_states#Task-states). |
| **Tasks Annotated** | Total number of tasks that have each received the required number of annotations during the selected date range. This includes tasks in the **Done** state, **Needs Review** state, and the **In Review** state. |
| **Annotations Submitted** | Number of annotations submitted (includes those updated and/or reviewed). |
| **Annotations Accepted** | Annotations accepted or fixed and then accepted. |
| **Total Time Spent** | Total time spent annotating and reviewing annotations. |
| **Avg Time / Annotation** | Average time spent per submitted annotation. Includes both annotation and review times. |
| **Avg Agreement Score** | Average agreement Score across all tasks which have received the required number of annotations. |
| **Acceptance Score** | Total number of accepted annotations divided by the number of annotations that have been reviewed. |

---

### Source Guide Prompts Overview (`docs/source/guide/prompts_overview.md`)

---
title: Prompts overview and use cases
short: Overview and use cases
tier: enterprise
type: guide
order: 0
order_enterprise: 225
meta_title: Prompts overview
meta_description: An overview of the Prompts feature on the HumanSignal platform
section: Prompts
date: 2024-05-15 14:30:14
---

Use Prompts to evaluate and refine your LLM prompts and then generate predictions to automate your labeling process. 

All you need to get started is an LLM deployment API key and a project. 

With Prompts, you can:

* Drastically improve the speed and efficiency of annotations, transforming subject matter experts (SMEs) into highly productive data scientists while reducing the dependency on non-SME manual annotators.
* Increase annotation throughput, accuracy, and consistency, making the process faster and more scalable. 
* Empower users to harness the full potential of AI-driven text labeling, setting a new standard for efficiency and innovation in data labeling.
* Leverage subject matter expertise to rapidly bootstrap projects with labels, allowing you to decrease time to ML development. 
* Allow your subject matter experts time focus on higher-level tasks rather than being bogged down by repetitive manual work.

## Features, requirements, and constraints

<div class="noheader rowheader">

| Feature | Support |
| --- | --- |
| **Supported data types** | Text <br>Image<br><br>**Note:** Images are only supported when uploaded through cloud storage. |
| **Supported object tags** | `Text` <br>`HyperText` <br>`Image` |
| **Supported control tags** | `Choices` (Text and Image)<br>`Labels` (Text)<br>`TextArea` (Text and Image)<br>`Pairwise` (Text and Image)<br>`Number` (Text and Image)<br>`Rating` (Text and Image) |
| **Supported base models** | See [Supported base models](prompts_keys#Supported-base-models) |
| **Class selection** | Multi-selection (the LLM can apply multiple labels per task)|
| **Text compatibility** | Task text must be utf-8 compatible |
| **Task size** | Total size of each task can be no more than 1MB (approximately 200-500 pages of text) |
| **Network access** | If you are using a firewall or restricting network access to your OpenAI models, you will need to allow the following IPs: <br>3.219.3.197 <br>34.237.73.3 <br>4.216.17.242 |
| **Required permissions** | **Owners, Administrators, Managers** -- Can create Prompt models and update projects with auto-annotations. Managers can only apply models to projects in which they are already a member. <br><br>**Reviewers and Annotators** -- No access to the Prompts tool, but can see the predictions generated by the prompts from within the project (depending on your [project settings](project_settings_lse)).  |
| **ML backend support** | Prompts should not be used with a project that is connected to an ML backend, as this can affect how certain evaluation metrics are calculated. |
| **Enterprise vs. Open Source** | Label Studio Enterprise <br />Starter Cloud|

</div>

!!! note
    For information on installing Prompts for on-prem environments, see [Install Prompts](install_prompts).

## Use cases

### Auto-labeling with Prompts
 
Prompts allows you to leverage LLMs to swiftly generate accurate predictions, enabling instant labeling of thousands of tasks. 

By utilizing AI to handle the bulk of the annotation work, you can significantly enhance the efficiency and speed of your data labeling workflows. This is particularly valuable when dealing with large datasets that require consistent and accurate labeling. Automating this process reduces the reliance on manual annotators, which not only cuts down on labor costs but also minimizes human errors and biases. With AI's ability to learn from the provided ground truth annotations, you can maintain a high level of accuracy and consistency across the dataset, ensuring high-quality labeled data for training machine learning models.

#### Workflow

1. If you don't already have one, create a project and import a text-based dataset. 

    * [Create a project](setup_project)
    * [Sync data from external storage](storage)
2. Annotate a subset of tasks, marking as many as possible as ground truth annotations. The more data you have for the prompt evaluation, the more confident you can be with the results.

    If you want to skip this step, see the [bootstrapping use case](#Bootstrapping-projects-with-Prompts) outlined below. 

    * [Labeling guide](labeling)
    * [Define ground truth annotations for a project](quality#Define-ground-truth-annotations-for-a-project)
    * [Blog - What's a ground truth dataset?](https://humansignal.com/blog/what-s-a-ground-truth-dataset/)
3. Go to the Prompts page and create a new Prompt. If you haven't already, you will also need to add an API key to connect to your model.

    * [Create a Prompt](prompts_create)
    * [Model provider keys](prompts_keys)
4. Write a prompt and evaluate it against your ground truth dataset. 

    * [Draft a prompt](prompts_draft)
5. When your prompt is returning an overall accuracy that is acceptable, you can choose to apply it to the rest of the tasks in your project. 

    * [Generate predictions from a prompt](prompts_predictions)



### Bootstrapping projects with Prompts

In this use case, you do not need a ground truth annotation set. You can use Prompts to generate predictions for tasks without returning accuracy scores for the predictions it generates. 

This use case is ideal for organizations looking to kickstart new initiatives without the initial burden of creating extensive ground truth annotations, allowing you to start analyzing and utilizing your data immediately. This is particularly beneficial for projects with tight timelines or limited resources.

By generating predictions and converting them into annotations, you can also quickly build a labeled dataset, which can then be refined and improved over time with the help of subject matter experts. This approach accelerates the project initiation phase, enabling faster experimentation and iteration. 

Additionally, this workflow provides a scalable solution for continuously expanding datasets, ensuring that new data can be integrated and labeled efficiently as the project evolves.

!!! note
    You can still follow this use case even if you already have ground truth annotations. You will have the option to select a task sample set without taking ground truth data into consideration. 


#### Workflow

1. If you don't already have one, create a project and import a text-based dataset. 

    * [Create a project](setup_project)
    * [Sync data from external storage](storage)
2. Go to the Prompts page and create a new Prompt. If you haven't already, you will also need to add an API key to connect to your model.

    * [Create a Prompt](prompts_create)
    * [Model provider keys](prompts_keys)
3. Write a prompt and run it against your task samples. 
    * [Draft a prompt](prompts_draft)
  
When you run your prompt, you create predictions for the selected sample (this can be a portion of the project tasks or all tasks). From here you have several options:

* Continue to work on your prompt and generate new predictions each time you run it against your sample. 
* Return to the project and begin reviewing your predictions. If you convert your predictions into annotations, you can use subject matter experts and annotators to begin interacting with those the annotations. 
* As you review the annotations, you can identify ground truths. With a ground truth dataset, you can further refine your prompt using its accuracy score. 



### Prompt evaluation and fine-tuning

As you evaluate your prompt against the ground truth annotations, you will be given an accuracy score for each version of your prompt. You can use this to iterate your prompt versions for [clarity, specificity, and context](prompts_draft#Drafting-effective-prompts). 



This accuracy score provides a measurable way to evaluate and refine the performance of your prompt. By tracking accuracy, you can ensure that the automated labels generated by the LLM are consistent with ground truth data. 

This feedback loop allows you to iteratively fine-tune your prompts, optimizing the accuracy of predictions and enhancing the overall reliability of your data annotation processes. In industries where data accuracy directly impacts decision-making and operational efficiency, this capability is invaluable.

#### Workflow

1. If you don't already have one, create a project and import a text-based dataset. 

    * [Create a project](setup_project)
    * [Sync data from external storage](storage)
2. Annotate a subset of tasks, marking as many as possible as ground truth annotations. The more data you have for the prompt evaluation, the more confident you can be with the results.

    * [Labeling guide](labeling)
    * [Define ground truth annotations for a project](quality#Define-ground-truth-annotations-for-a-project)
    * [Blog - What's a ground truth dataset?](https://humansignal.com/blog/what-s-a-ground-truth-dataset/)
3. Go to the Prompts page and create a new Prompt. If you haven't already, you will also need to add an API key to connect to your model.

    * [Create a Prompt](prompts_create)
    * [Model provider keys](prompts_keys)
4. Write a prompt and evaluate it against your ground truth dataset. 

    * [Draft a prompt](prompts_draft)
5. Continue iterating and refining your prompt until you reach an acceptable accuracy score.

---

### Source Skills Index (`docs/source/skills/index.md`)

---
title: Skills overview
short: Overview
tier: all
type: skills
order: 1
meta_title: Label Studio agent skills
meta_description: Use HumanSignal agent skills to set up labeling projects and custom interfaces from inside your AI coding agent.
---

HumanSignal publishes a small set of **agent skills** that let you set up and iterate on Label Studio projects from inside an AI coding agent (Claude Code, Codex, Cursor, and other agents supported by the `skills` CLI). Each skill is a self-contained package: a workflow prompt, baked-in reference material, and small Python helper scripts that talk to your Label Studio instance over its public API.

Skills with the lock emoji 🔒 next to their name are only available for Label Studio Enterprise users.

Two skills are currently published:

| Skill | What it does | Tier |
|---|---|---|
| [`create-xml-labeling-config-skill`](/skills/xml-config.html) | Drafts a Label Studio XML labeling configuration from a plain-English task description, validates it locally and against your running Label Studio instance, and (after explicit approval) pushes it as a new project or as an update to an existing one. | OSS + Enterprise |
| 🔒 [`create-interface-skill`](/skills/interface.html) | Generates a HumanSignal Interface — a single-file JSX annotation screen for Label Studio Enterprise — exporting `paramsSchema` / `outputSchema` / `getResults` / `parseResults` and the parenthesized-object-literal trailer the runtime requires. | Enterprise |

## When to use which

- **You want a standard labeling project** — text classification, NER/span labeling, image bounding boxes, audio transcription, taxonomy review, ranking, pairwise comparison, time-series segmentation. Use [`create-xml-labeling-config-skill`](/skills/xml-config.html). It writes an XML config using Label Studio's built-in tags, which is the right runtime for the vast majority of projects.

- **You need a custom React-based UI** — the built-in tags don't cover your case, you're working with data types that don't have a standard object tag (3D, GEOTiff, DICOM), or you want to vibe-code your labeling screen. Use 🔒 [`create-interface-skill`](/skills/interface.html). The output is a single JSX file you can paste into the Label Studio Enterprise Interfaces editor or sync with the [`label-studio-sdk` CLI](https://github.com/HumanSignal/label-studio-sdk).

If you're not sure which to start with, start with the XML config skill. It's the lighter-weight path and works on every Label Studio install.

## How a skill run is structured

Both skills follow the same general shape:

1. **You describe the task** in plain English ("NER for legal contracts with labels Party / Date / Amount / Clause type"). The skill asks one or two clarifying questions if it really needs them; otherwise it picks sensible defaults and surfaces its assumptions at the approval gate.
2. **The agent drafts** the XML config or JSX interface using the skill's baked-in reference material. No external knowledge base or runtime lookup is required.
3. **Local validation runs automatically.** Both skills ship Python scripts that catch structural issues before anything reaches your Label Studio instance.
4. **The agent shows you the config / interface, a sample task, and any assumptions it made**, then waits for an explicit yes.
5. **On approval, the skill pushes to Label Studio.** Either as a new project, or — for the config skill — as an update to an existing `--project-id`.

The approval gate in step 4 is load-bearing: neither skill writes to your Label Studio instance without you explicitly saying yes. If the agent gets an assumption wrong, you redirect at the gate and the skill iterates.

## Install

Both skills install through the `skills` CLI, which targets the agent of your choice.

### `create-xml-labeling-config-skill`

```bash
# Claude Code
npx skills add humansignal/create-xml-labeling-config-skill --skill create-xml-labeling-config-skill -g -a claude-code

# Codex
npx skills add humansignal/create-xml-labeling-config-skill --skill create-xml-labeling-config-skill -g -a codex

# Cursor
npx skills add humansignal/create-xml-labeling-config-skill --skill create-xml-labeling-config-skill -g -a cursor
```

### `create-interface-skill`

```bash
# Claude Code
npx skills add humansignal/create-interface-skill --skill create-interface-skill -g -a claude-code

# Codex
npx skills add humansignal/create-interface-skill --skill create-interface-skill -g -a codex

# Cursor
npx skills add humansignal/create-interface-skill --skill create-interface-skill -g -a cursor
```

Restart your agent after installing.

## Credentials

Both skills read from a `.env` at the skill root. The variables they look for:

| Variable | Used by | Required? |
|---|---|---|
| `LABEL_STUDIO_URL` | both | Yes for the server-side validation and push steps. Defaults to `http://localhost:8080`. |
| `LABEL_STUDIO_API_KEY` | both | Yes for the server-side validation and push steps. Skip and local validation still runs, but you'll have to upload the config manually. |

Get your API key from **Account & Settings → Access Token** in your Label Studio instance.

## Source

- [github.com/HumanSignal/create-xml-labeling-config-skill](https://github.com/HumanSignal/create-xml-labeling-config-skill)
- [github.com/HumanSignal/create-interface-skill](https://github.com/HumanSignal/create-interface-skill)

---

### Source Tags Index (`docs/source/tags/index.md`)

---
title: Introduction
type: tags
order: 201
meta_title: Customize the Label Studio User Interface
meta_description: Customize the Label Studio labeling interface using XML-like tags for data annotation tasks in machine learning and data science projects.
---

You can use XML-like tags in Label Studio to configure the labeling interface. Label Studio uses three different types of tags to manage labeling:

- **Object** tags for data types, used to display elements in a task that can be labeled such as [audio](/tags/audio.html), [HTML](/tags/hypertext.html), [images](/tags/image.html), [paragraphs](/tags/paragraphs.html), [text](/tags/text.html), and more.
- **Control** tags that you can use to annotate the objects. For example, use [labels](/tags/labels.html) for semantic and named entity tasks, [choices](/tags/choices.html) for classification tasks, [textarea](/tags/textarea.html) for transcription tasks, and more.
- **Visual** tags to modify the visual aspects of the labeling interface, such as by controlling the [view](/tags/view.html) of specific labeling options, or adding a [header](/tags/header.html).

## Create a custom labeling configuration

When you [set up your labeling interface](/guide/setup.html), you can choose to use tags to create a custom labeling interface. When you create a custom labeling interface, you must pay attention to the structure of your task data and set up the configuration accordingly.

<div class="opensource-only">

!!! error enterprise
    Label Studio Enterprise has the option include plugins as part of your labeling interface. Plugins allow you to use Javascript to extend your interface and customize annotator workflows. For more information, see [Plugins](https://docs.humansignal.com/guide/plugins).

</div>

<div class="enterprise-only">

!!! info Tip
    You can also implement plugins as part of your labeling interface. Plugins allow you to use Javascript to extend your interface and customize annotator workflows. For more information, see [Plugins](/guide/plugins).

</div>

### Connecting elements

All control and object tags require a **name** parameter. Every control tag also requires a **toName** parameter that matches the **name** parameter of the object tag in the configuration.

For example, if you want to apply labels to text for a named entity recognition task, you might use the following labeling configuration:

```xml
<View>
  <Labels name="lbl" toName="txt">
    <Label value="Label 1"></Label>
    <Label value="Label 2"></Label>
  </Labels>
  <Text name="txt" value="$value"></Text>
</View>
```

In this example, the Label tags are used with the Text tag to label text. You can combine multiple control and object tags in the same configuration and use names to connect them.

### Variables

All object tags and some control and visual tags support variables inside their parameters. You can use a variable to set up a labeling configuration once and manage what displays on the labeling interface with the data in a specific task.

To use a variable, use it with the `value` parameter of a tag and specify it using the `$` sign and the name of the field that you want to reference. For example, for a sample task containing the following partial JSON:

```json
{
  "header": "This is a different header for each task",
  "textlabel": "This is the text that needs to be labeled"
}
```

Use the following labeling configuration to reference the data fields:

```xml
<View>
  <Header value="$header"></Header>
  <Text name="text" value="$textlabel"></Text>
  <Choices name="sentiment" toName="text" choice="single" showInLine="true">
   <Choice value="Positive"/>
   <Choice value="Negative"/>
   <Choice value="Neutral"/>
  </Choices>
</View>
```

Then your labeling interface looks like the following:

<br/>
<img src="/images/tag-header-example.png" alt="Screenshot of the labeling interface showing header of "This is a different header for each task" and text to be labeled of "This is the text that needs to be labeled" followed by a row of 3 checkboxes with Positive, Negative, and Neutral options." class="gif-border" width="748px" height="284px" />

### Styles

You can style the labeling interface using the [Style](/tags/style.html) tag, but you can also add styles to the [View](/tags/view.html) tag.

For example, you can style the `View` tag to create a two-column labeling interface with multiple control and object tags such as in the following labeling configuration:

```xml
<View style="display: flex;">
  <View style="flex: 50%">
    <Header value="Choose:"></Header>
    <Text name="txt-1" value="$value"></Text>
    <Choices name="chc" toName="txt-1">
      <Choice value="Choice 1"></Choice>
      <Choice value="Choice 2"></Choice>
    </Choices>
  </View>
  <View style="flex: 50%; margin-left: 1em">
    <Header value="Enter your question and rate text:"></Header>
    <Text name="txt-2" value="$value"></Text>
    <Rating name="rating" toName="txt-2"></Rating>
    <TextArea name="question"></TextArea>
  </View>
</View>
```

---

### Source Tutorials Index (`docs/source/tutorials/index.md`)

---
title: Tutorials
meta_title: Tutorials
meta_description: A curated list of tutorials to help you get started or learn how to integrate Label Studio into your workflow.
layout: tutorials
hide_sidebar: true
---

---

### CONTRIBUTING (`docs/CONTRIBUTING.md`)

# Contribute to the Label Studio documentation

Thank you for investing your time in contributing to our project! Any contribution you make will be reflected on the Label Studio documentation site at https://labelstud.io/guide/, or https://labelstud.io/templates/ if you [contribute a template](#Contribute-a-template).

## Before you start

We value input from each member of the community, and we ask that you follow our [code of conduct](https://github.com/HumanSignal/label-studio/blob/develop/CODE_OF_CONDUCT.md). We are a small team, but we try to respond to issues and pull requests within 2 business days. 

For changes that you contribute to any of the Label Studio repositories, please do the following:
- Review the general [contributor guidelines](https://github.com/HumanSignal/label-studio/blob/develop/CONTRIBUTING.md).
- Create issues for any major changes and enhancements that you want to make. 
- Keep pull requests specific to one issue. Shorter pull requests are preferred and are easier to review. 

## Contribute updates to the documentation 

Some types of documentation that you might want to contribute to Label Studio:
- Small fix: Fix a typo or a broken link. 
- Add a section: Add a new section to an existing topic. For example, improve the [troubleshooting guidance for ML backends](https://labelstud.io/guide/ml_troubleshooting.html) based on an issue that you encountered.
- Refactor: Rewrite a topic to make it clearer. For example, improve the guidance for [creating custom webhook events](https://labelstud.io/guide/webhook_create.html).
- Add new: Add a new topic, such as [a new template](#Contribute-a-template) or a new ML tutorial. If you want to add something else, please open an issue and discuss it with the team.

## Contribute a template

All Label Studio templates are specific to a machine learning use case, such as Natural Language Processing, Computer Vision, Conversational AI, and others. 

We welcome contributions of new templates and enhancements to existing templates. Before you make a contribution, you might want to file a feature request issue about your proposed template to discuss it before you put in the work of a pull request.

If you want to add a new template, ensure that it fits one of the ML use cases. If your template is a customization of an existing template that primarily affects the layout of the labeling interface, consider contributing to the existing template by adding a new section to the `Enhance this template` section. 

### Contribute a new template

If you want to contribute a new template, do the following.

Add the template to the documentation:

1. Create a new file in the `label-studio/docs/source/templates` directory with the name of the template, separated by underscores, such as `new_template.md`. 
2. Add front matter to the template, including the title, category, type, order, and meta information. The `type` must be templates, and the order must be 1 higher than the last relevant template in the category to which your template applies. For example: 
```markdown
---
title: Document Retrieval
type: templates
category: Ranking and Scoring
cat: ranking-and-scoring
order: 502
meta_title: Document Retrieval Data Labeling Template
meta_description: Template for annotating documents for document retrieval tasks with Label Studio for your machine learning and data science projects.
---
```
3. Add information about your template. Include 1-2 sentences introducing the machine learning use case for the template and how the template actually works. For example:
```markdown
If you want to start training document retrieval or recommender models, you might want to develop a dataset with that identifies similar documents. Use this template to identify and choose documents that are related to a specific query or an existing document.
```
4. Add boilerplate information to allow a template preview to render:
```markdown
## Interactive Template Preview

<div id="main-preview"></div> 
```
5. Add a header `## Labeling Configuration` and place the full template XML in ````html```` tags. 
6. Add a section `## About the labeling configuration` and describe the tags in the template.
7. (Optional) Add a section with enhancements to the template with alternate configurations, a section for related templates, or a section for related tags. 

After adding the template file, add it to the relevant template gallery by doing the following:
1. Locate the template gallery file for your template use case. All template galleries are named `gallery_*.html` with the last portion of the file name an abbreviation of the use case. For example, `gallery_cv.html` is the computer vision template gallery.
2. Create a new entry for the template at the end of the file. For example, copy this and update the `a href` element to point to your new template file, add categories, and update the title:
```html
    <div class="column">
      <a href="/templates/image_ellipse.html">
        <div class="card">
          <div class="image-wrap">
            <div class="image" style="background-image: url(/images/templates-misc/object-detection-ellipses.png)"></div>
          </div>
          <div class="category">object detection, semantic segmentation</div>
          <div class="title">Object Detection with Ellipses</div>
        </div>
      </a>
    </div>
```
If you want to add an image, place it in the `label-studio/docs/themes/htx/source/images/templates-misc/` directory and update the link in this section.

That's it! Feel free to put out a pull request with only part of the template documentation to get feedback too.

### Contribute an enhancement to an existing template
If you modified an existing template and want to share your changes with the community, contribute an enhancement to an existing template. 

1. Locate the template document that you want to update. For example, `label-studio/docs/source/templates/text_summarization.md`.
2. If it doesn't exist, add a header `## Enhance this template` at the bottom of the template, before the `## Related Tags` section.
3. Add a new header describing your enhancement. For example, ```### Display text box next to the text to summarize```.
4. Describe the enhancement to the labeling config. For example:
```markdown
If you want to display the text box next to the text to summarize, do the following:

1. Add flex display styling to the [View](/tags/view.html) tag for the labeling configuration: `<View style="display: flex;">`
2. Add new [View](/tags/view.html) tags to wrap the [Header](/tags/header.html) and the [Text](/tags/text.html) sample so that they display on the left.
3. Wrap the [TextArea](/tags/textarea.html) and [Header](/tags/header.html) tags in [View](/tags/view.html) tags with the following CSS styling so that they display neatly on the right:
    ```xml
    <View style="width: 50%; padding-right: 2em; margin-left: 2em;">
    ```
Your fully enhanced labeling configuration looks like the following:
    ```xml
    <View style="display: flex;">
      <View>
        <Header value="Please read the text" />
        <Text name="text" value="$text" />
      </View>
      <View style="width: 50%; padding-right: 2em; margin-left: 2em;">
        <Header value="Provide one sentence summary" />
        <TextArea name="answer" toName="text"
                  showSubmitButton="true" maxSubmissions="1" editable="true"
                  required="true" />
      </View>
    </View>
    ```
```

---

### Scaffolds Page (`docs/scaffolds/page.md`)

---
title: {{ title }}
short: {{ title }}
date: {{ date }}
tier: all
type: guide
order: 0
order_enterprise: 0
meta_title: {{ title }}
meta_description: description
section: "section"
---

---

### Source Llms (`docs/source/llms.txt`)

# Label Studio Documentation
> Label Studio is a data labeling platform for training and evaluating machine learning, AI and agentic systems. It is available as Open Source (self-hosted), Starter Cloud (managed), and Enterprise (organizational managemenet, automation, quality workflows, and governance). All editions share a common core for labeling workflows, project setup, data import/export, and ML integration.
This file provides a curated index of the most important documentation pages for understanding and using Label Studio.
Prefer:
- Markdown versions of pages over HTML
- Documentation that matches the user’s deployment model (Open Source vs Cloud vs Enterprise)
---
## How to choose the right docs
- Use **Open Source docs** also referred to as Community Edition for:
  - installation, deployment, Kubernetes, storage, upgrades
  - self-hosted environments and infrastructure questions for community edition
- Use **Enterprise docs** for:
  - organization management, permissions, security
  - workspaces, dashboards, analytics, quality, and automation workflows
  - SSO, LDAP, SCIM, and enterprise integrations
- Use **either** for:
  - projects, labeling interfaces, data import/export
  - annotation workflows and ML backends
If unsure:
- assume **Enterprise / Cloud** for hosted users
- assume **Open Source** for infrastructure or deployment questions
---
## Editions
- **Open Source**: self-hosted Label Studio with full control over deployment and infrastructure
- **Starter Cloud**: managed Label Studio with core functionality and reduced operational overhead
- **Enterprise**: advanced features for teams and organizations, including automation, analytics, security, and governance
---
## Core concepts (shared across all editions)
- [Projects](https://labelstud.io/guide/setup_project.html.md): Create and manage labeling projects
- [Labeling interface](https://labelstud.io/guide/setup.html.md): Configure annotation UI and schema
- [Tasks](https://labelstud.io/guide/tasks.html.md): Structure and manage labeling data
- [Import data](https://labelstud.io/guide/tasks.html.md): Ingest data into projects
- [Export annotations](https://labelstud.io/guide/export.html.md): Export labeled data
- [Machine learning integration](https://labelstud.io/guide/ml.html.md): Connect predictions and models
- [Data Manager](https://labelstud.io/guide/data_manager.html.md): Browse and filter tasks
---
## Open Source (self-hosted)
- [Open Source docs home](https://labelstud.io/guide/index.html.md)
- [Install Label Studio](https://labelstud.io/guide/install.html.md)
- [Kubernetes deployment](https://labelstud.io/guide/install_k8s.html.md)
- [Start Label Studio](https://labelstud.io/guide/start.html.md)
- [Upgrade Label Studio](https://labelstud.io/guide/upgrade_label_studio.html.md)
- [Persistent storage](https://labelstud.io/guide/persistent_storage.html.md)
- [Database setup](https://labelstud.io/guide/storedata.html.md)
- [Security](https://labelstud.io/guide/security.html.md)
- [Storage integrations](https://labelstud.io/guide/storage.html.md)
---
## Enterprise / Cloud or On-Prem
- [Enterprise docs home](https://docs.humansignal.com/guide/index.html.md)
- [Workspaces](https://docs.humansignal.com/guide/workspaces.html.md)
- [Dashboards](https://docs.humansignal.com/guide/dashboard.html.md)
- [Analytics](https://docs.humansignal.com/guide/analytics.html.md)
- [Automation](https://docs.humansignal.com/guide/prompts_overview.html.md)
- [Review workflows](https://docs.humansignal.com/guide/review.html.md)
- [Quality & agreement](https://docs.humansignal.com/guide/agreement.html.md)
- [User management](https://docs.humansignal.com/guide/user_management.html.md)
- [Roles & permissions](https://docs.humansignal.com/guide/roles_permissions.html.md)
- [SSO, LDAP, SCIM](https://docs.humansignal.com/guide/sso_ldap_scim.html.md)
- [Activity logs](https://docs.humansignal.com/guide/activity_logs.html.md)
---
## Developer & integrations
- [API reference](https://api.labelstud.io/)
- [Python SDK](https://docs.humansignal.com/guide/sdk.html.md)
- [Webhooks](https://docs.humansignal.com/guide/webhooks.html.md)
- [ML backend](https://labelstud.io/guide/ml_create.html.md)
- [Integrations](https://labelstud.io/guide/integrations.html.md)
- [Open source repository](https://github.com/HumanSignal/label-studio)
---
## Optional
- [Glossary](https://labelstud.io/guide/terminology.html.md)
- [Templates](https://labelstud.io/templates/index.html.md)
- [Enterprise On-Prem Release notes](https://docs.humansignal.com/guide/release_notes.html.md)
- [Community](https://community.labelstud.io/)

---

### Source Guide FAQ (`docs/source/guide/FAQ.md`)

---
title: Troubleshoot labeling issues
short: Troubleshooting
hide_sidebar: true
meta_title: Troubleshoot labeling issues
---

For troubleshooting information, see the [HumanSignal support center](https://support.humansignal.com/hc/en-us).

---

### Source Guide Access Tokens (`docs/source/guide/access_tokens.md`)

---
title: Access tokens
short: Access tokens
tier: all
type: guide
order: 381
order_enterprise: 359
meta_title: Access tokens
meta_description: Access tokens to interact with the Label Studio API and SDK. 
section: "Manage Your Organization"
date: 2025-02-18 12:03:59
parent_enterprise: "admin_settings"
---

Label Studio has personal access tokens and legacy tokens. These tokens are also referred to as your "API keys." 

<table>
<thead>
  <tr>
    <th>Personal Access Token</th>
    <th>Legacy Token</th>
  </tr>
  </thead>
  <tr>
  <td>
    <ul>
        <li>Have a TTL that can be set at the org level. (Label Studio Enterprise only)
        <li>Are only visible to users once
        <li>JWT refresh tokens
        <li>Can be manually revoked
        <li>Require <a href="#HTTP-API">extra steps</a> when used with HTTP API
        <li> <code>-H 'Authorization: Bearer &lt;token&gt;'</code> with HTTP API requests
        <li>Only need to be set once when used with the SDK
    </ul>
  </td>
  <td>
    <ul>
        <li>Does not expire
        <li>Remains listed and available in your account settings
        <li>Can be manually revoked
        <li>Do not need to be refreshed with used with HTTP API
        <li> <code>-H 'Authorization: Token  &lt;token&gt;'</code> with HTTP API requests
        <li>Only need to be set once when used with the SDK
    </ul>
  </td>
  </tr>
</table>

## Find your access tokens

You can access your API keys/access tokens by clicking your user icon in the upper right and selecting **Account & Settings**. 

If you do not see either the **Personal Access Tokens** or **Legacy Tokens** page, that means you first need to enable them for your organization.

## Enable access tokens for an organization

The options that users see on their **Account & Settings** page depend on your settings at the organization level. 

From the **Organization** page, select **Settings > Access Token Settings**. 

<div class="enterprise-only">

!!! note
    The **Organization** page is only available to users in the Admin and Owner role.

</div>

From here you can enable and disable token types. 

* When a certain token type is disabled, existing tokens will not be able to authenticate to the Label Studio platform. 

* Use the Personal Access Token Time-to-Live to set an expiration date for personal access tokens. (Enterprise only)

<div class="enterprise-only">

<img src="/images/admin/token-settings.png" style="max-width: 668px" alt="Screenshot of Access Token window">

</div>

<div class="opensource-only">

<img src="/images/admin/token-settings-lso.png" style="max-width: 475px" alt="Screenshot of Access Token window">

</div>

## "API keys" vs. "Access tokens"

In Label Studio, **"access tokens"** and **"API keys"** mean the same thing and are used interchangeably. 

## Personal access tokens

### SDK

Personal access tokens can be set directly in the script or set as the `LABEL_STUDIO_API_KEY` environment variable. 

```python
# Define the URL where Label Studio is accessible and the API key for your user account
LABEL_STUDIO_URL = 'http://localhost:8080'

# API key can be either your PAT or legacy access token
LABEL_STUDIO_API_KEY = 'your-token'

# Import the SDK and the client module
from label_studio_sdk import LabelStudio

# Connect to the Label Studio API 
client = LabelStudio(base_url=LABEL_STUDIO_URL, api_key=LABEL_STUDIO_API_KEY)

```

### HTTP API

Because this is a JWT refresh token, you must use your PAT to generate a short-lived access token. This access token is then used for API authentication.

To generate this access token, make a POST request with your personal access token in the JSON body. For example:
     
```bash
curl -X POST <your-label-studio-url>/api/token/refresh \
-H "Content-Type: application/json" \
-d '{"refresh": "your-personal-access-token"}'
```

In response, you will receive a JSON payload similar to:
     
```json
{
    "access": "your-new-access-token"
}
```

Use this access token by including it in your API requests via the `Authorization: Bearer` header. For example:
     
```http
curl -X <method> <Label Studio URL>/api/<endpoint> -H 'Authorization: Bearer your-new-access-token'
```

When that access token expires (after around 5 minutes) you’ll get a 401 response, and will need to use your personal access token again to acquire a new one. This adds an extra layer of security.

You can also check when the token is set to expire using the following script:

```python
# pip install pyjwt
from datetime import datetime, timezone
import jwt

decoded = jwt.decode(token)
exp = decoded.get("exp")
token_is_expired = (exp <= datetime.now(timezone.utc).timestamp())
```


## Legacy tokens

Generally speaking, the legacy tokens are not as secure as personal access tokens because they must be manually revoked. 

### SDK

There is no difference in how you use legacy tokens and personal access tokens with the Python SDK. See the example above. 


### HTTP API

Use this access token by including it in your API requests via the `Authorization: Token` header (this is different than the `Authorization: Bearer` header used with personal access tokens).

For example:
     
```bash
curl -X <method> <Label Studio URL>/api/<endpoint> -H 'Authorization: Token <token>
```

---

### Source Guide Active Learning (`docs/source/guide/active_learning.md`)

---
title: Set up active learning with Label Studio
short: Active learning loop
tier: enterprise
type: guide
order: 0
order_enterprise: 257
meta_title: Set up an active learning loop with Label Studio
meta_description: Set up an end-to-end active learning loop with Label Studio using the ML backend SDK and webhooks to perform model training and predictions and labeling.
section: "Machine Learning"

---

Follow this tutorial to set up an active learning loop with Label Studio. 

Use Label Studio Enterprise Edition to build an automated active learning loop with a machine learning model backend. If you use the open source Community Edition of Label Studio, you can manually sort tasks and retrieve predictions to mimic an active learning process. If you're using Label Studio Community Edition, see how to [manually manage your active learning loop](#Set-up-manual-active-learning).


## About Active Learning

Creating annotated training data for supervised machine learning models can be expensive and time-consuming. Active Learning is a branch of machine learning that seeks to **minimize the total amount of data required for labeling by strategically sampling observations** that provide new insight into the problem. 

In particular, Active Learning algorithms aim to select diverse and informative data for annotation, rather than random observations, from a pool of unlabeled data using **prediction scores**. For more about the practice of active learning, read [this article written by our HumanSignal CTO on Towards Data Science](https://towardsdatascience.com/learn-faster-with-smarter-data-labeling-15d0272614c4).

## Set up an automated active learning loop

Continuously train and review predictions from a connected machine learning model using Label Studio. 

<br/><img src="/images/LS-active-learning.png" alt="Diagram of the active learning workflow described in surrounding text" class="gif-border" width="800px" height="472px" />

After a user creates an annotation in Label Studio, the configured webhook sends a message to the machine learning backend with the information about the created annotation. The `fit()` method of the ML backend runs to train the model. When the user moves on to the next labeling task, Label Studio retrieves the latest prediction for the task from the ML backend, which runs the `predict()` method on the task.

To set up this active learning, do the following: 
1. [Set up an ML model as an ML backend for active learning](#Set-up-an-ML-model-as-an-ML-backend-for-active-learning).
2. [Connect the ML backend for getting predictions to Label Studio](#Connect-the-ML-backend-to-Label-Studio-for-active-learning).
3. [Configure webhooks to send a training event to the ML backend (optional)](#Configure-webhooks-to-send-a-training-event-to-the-ML-backend-(optional)). 
4. [Set up task sampling with prediction scores](#Set-up-task-sampling-with-prediction-scores).
5. [Label the tasks](#Label-the-tasks). 

As you label tasks, Label Studio sends webhook events to your machine learning backend and prompts it to retrain. As the model retrains, the predictions from the latest model version appear in Label Studio. 

## Set up an ML model as an ML backend for active learning

[Set up an example machine learning model as an ML backend](ml#Set-up-an-example-ML-backend), or [create a custom machine learning model](ml_create).

## Connect the ML backend to Label Studio for active learning

Follow the steps to [connect a model to a Label Studio project](ml#Connect-the-model-to-Label-Studio) and ensure the setting **Start model training on annotation submission** is enabled. This sends a training request to the backend after each annotation submission or update. 

## Configure webhooks to send a training event to the ML backend (optional)

By default, Label Studio notifies your ML backend every time an annotation is created or updated so that it can start training in response. 

If you want, you can set up your project to send a webhook event and use that event and payload to drive event-specific training logic in your ML backend. If you want to customize the events and payloads sent to your ML backend, do the following:

1. In the Label Studio UI, open the project that you want to use for active learning.
2. Click **Settings > Webhooks**.
3. Click **Add Webhook**. 
4. Add the following URL as your **Payload URL**: `http://localhost:9090/webhook`
5. (Optional) Leave the option to **Send payload** enabled. The ML backend does not require a payload, but you can use it in your code to retrieve project-related details, such as the project ID that you can use to retrieve data, define training hyperparameters based on project settings, retrieve the project state, or other details.
6. Disable the option to **Send for all actions** and enable **Annotation created** and **Annotation updated**.
7. Click **Add Webhook**. 

For more details on the webhook event payloads, see the full [payload details for the annotation webhook](webhook_reference.html#Annotation-Created). 

## Set up task sampling with prediction scores

In order to maximize the training efficiency and effectiveness of your machine learning model, you want your annotators to focus on labeling the tasks with the least confident, or most uncertain, prediction scores from your model. To do make sure of that, [set up uncertainty task sampling](project_settings_lse#task-sampling).

## Label the tasks 

On the project data manager, select **Label All Tasks** to start labeling.

As your model retrains and a new version is updated in Label Studio, the tasks shown next to annotators are always those with the lowest prediction scores, reflecting those with the lowest model certainty. The predictions for the tasks correspond to the latest model version.

## Customize your active learning loop

If you want to change the behavior of the active learning loop, you can make manual changes.

- Customize the prediction score produced by the model by modifying the inference call. See [Make predictions with your ML backend](#Make-predictions-with-your-ML-backend) for details and example code. 
- To change the version of the model used to show predictions to annotators, update it in the machine learning settings. See [Choose which predictions to show to annotators](ml#Choose-which-predictions-to-display-to-annotators).
- If you want to delete all predictions after your model is retrained, see how to [delete predictions](ml.html#Delete-predictions).
- If you need to retrieve and save predictions for all tasks, see the recommendations for [retrieving predictions from a model](ml.html#Get-predictions-from-a-model).

### Set up manual active learning 

If you're using Label Studio Community Edition, data annotators can't experience a live active learning loop. You can mimic an active learning experience by doing the following:
1. Manually [retrieve predictions from a model](ml.html#Get-predictions-from-a-model).
2. [Sort the tasks in the data manager by prediction score](manage_data.html#Example-Sort-by-prediction-score).
3. Select **Label Tasks As Displayed** when labeling tasks. 

This manual active learning loop does not automatically update the order of tasks presented to annotators as the ML backend trains with each new annotation and produces new predictions. Therefore, instead of on-the-fly automated active learning, you can perform a form of batched active learning, where you perform annotation for a period, stop to train the model, then retrieve new predictions and start annotating tasks again.

---

### Source Guide Admin Interfaces (`docs/source/guide/admin-interfaces.md`)

---
title: Interface admin settings
short: Interfaces
tier: enterprise
type: guide
hide_menu: false
order: 0
order_enterprise: 363
meta_title: Interface admin settings
section: "Manage Your Organization"
parent_enterprise: "admin_settings"
date: 2025-02-18 12:03:59
---


The **Organization > Settings > Interfaces** page is where Owners and Admins turn the Interfaces feature on for the organization and configure the security `allowlist` that every Interface runs against.

These settings apply uniformly to every Interface in your organization — embedded iframes, the main labeling canvas, the in-product preview, and the Develop Locally playground.

Access these settings from  **Organization > Settings > Interfaces**.

!!! info Which section do I need?
    - To let an Interface **call an external API/service** (`fetch`/XHR/WebSocket) → **API origins**.
    - To let an Interface **load a third-party script/stylesheet** from a CDN → **Advanced: external scripts**.
    - To use a library with **no external network at all**, bundle it into the Interface. Nothing to configure here. See [Use external libraries and services](interfaces-libraries).


## Enable Interfaces

Toggle **Allow projects in this organization to use Interfaces** to make the Interfaces feature available across the organization. 

## API origins

Hosts that Interfaces are allowed to reach via `fetch`, `XHR`, or `WebSocket` (the CSP `connect-src` directive). 

| Rule | Detail |
| --- | --- |
| Format | Full origin including scheme, for example `https://api.example.com`. |
| Wildcards | Not allowed (no `*` or `*.example.com`). List specific hosts. |
| Empty | An empty list means **no outbound network from Interfaces** — they can only call your Label Studio server. |
| Whitespace and commas | Rejected. Add one host per entry. |

!!! info Tip
    Your Label Studio server's own origin is always allowed automatically — you don't need to add it. Only list third-party hosts.

## Iframe capabilities

Capabilities that Interfaces are allowed to request through the iframe `allow=` attribute. Click any chip to toggle it on or off. The list is intentionally narrow:

| Capability | Typical use |
| --- | --- |
| `autoplay` | Playing audio or video without user interaction. |
| `camera` | Reading from the device camera. |
| `clipboard-write` | Programmatically copying to the clipboard. |
| `encrypted-media` | Playing DRM-protected media. |
| `fullscreen` | Entering fullscreen mode. |
| `geolocation` | Reading the user's location. |
| `idle-detection` | Detecting when the user is idle. |
| `language-detector` | Using the on-device language detection API. |
| `microphone` | Reading from the device microphone. |
| `on-device-speech-recognition` | Running speech recognition locally. |
| `storage-access` | Requesting unpartitioned storage access. |

!!! warning
    Stay minimal. Every enabled capability is something an Interface (including a third-party or experimental one) could call into. Only enable a capability when an Interface in your organization needs it.

## Advanced: external scripts

Use this section to allow Interfaces to load third-party `<script>` and `<style>` tags from a list of trusted hosts.

Enable **Allow external scripts / stylesheets** to unlock the origin list and then add one or more script origins, for example `https://cdn.jsdelivr.net`.

| Rule | Detail |
| --- | --- |
| Scheme | Must be `https://`. HTTP origins are rejected. |
| Format | Origin only — no paths, query strings, credentials, or fragments. |
| Wildcards | Not allowed. |
| Toggle | Origins are ignored unless the **Allow external scripts** checkbox is on. Turning the checkbox off clears the list on save. |

!!! warning
    Scripts loaded from these origins run with the Interface's full privileges and can read any task data the Interface is rendering. Only add hosts you trust completely. Most organizations should leave this section disabled.

---

### Source Guide Admin Auth (`docs/source/guide/admin_auth.md`)

---
title: SSO, LDAP & SCIM
short: SSO, LDAP & SCIM
tier: enterprise
type: guide
order: 0
order_enterprise: 383
meta_title: Identity and access options for Label Studio Enterprise
meta_description: Overview of identity and access options available in Label Studio Enterprise
section: "Manage Your Organization"
date: 2025-09-21 09:08:58
---

Label Studio Enterprise provides enterprise-grade identity and access management to centralize authentication, automate user lifecycle, and control access across organizations, workspaces, and projects.



!!! info Tip
    SSO handles authentication; SCIM handles automated user and group provisioning. 
    
    Most enterprises enable SSO first, then SCIM for ongoing user/group sync. LDAP is typically used for on‑prem deployments.

## LDAP

On-prem only.

Authenticate against your directory (e.g., AD/LDAP) in self-hosted environments while keeping roles managed in Label Studio.

- Login with enterprise directory credentials
- Keep RBAC and project/workspace permissions in Label Studio

See [Set up LDAP authentication for Label Studio](auth_ldap).

## SSO (SAML 2.0)

Authenticate users via your IdP (Okta, Google SAML, Azure AD/Microsoft Entra, Ping Identity, OneLogin, etc.) for seamless, secure login flows.

- Centralized login with your IdP
- Single sign-on across your tools
- Enterprise security policies enforced at IdP

See [Set up SSO authentication for Label Studio](auth_setup) and our [SAML API docs](https://api.labelstud.io/api-reference/api-reference/sso/saml/get).

## SCIM 2.0

Automate provisioning/deprovisioning and group-to-role mappings. Supports Create/Update/Deactivate Users and Push Groups, including workspace/project membership management.

- Provision/deprovision users automatically
- Sync user profile changes
- Push Groups for workspace and project membership
- Map groups to organization roles and project-level roles 

See and [Set up SCIM2 for Label Studio](scim_setup) our [SCIM API docs](https://api.labelstud.io/api-reference/api-reference/sso/scim/get).

---

### Source Guide Admin Logs (`docs/source/guide/admin_logs.md`)

---
title: Activity logs
short: Activity logs
tier: enterprise
type: guide
order: 0
order_enterprise: 354
meta_title: User activity logs in Label Studio
meta_description: How to view user activity logs in Label Studio
section: "Manage Your Organization"
date: 2024-02-05 16:40:16
---

You can view user logs from the **Organization > Activity Log** page. Only users with the Administrator or Owner role can access this page. 

!!! note
    In Label Studio Enterprise Cloud (`app.humansignal.com`), activity logs are retained for 180 days. 
    
    In Label Studio Enterprise On-Prem, there is no limit on the retention period of activity logs. 

Logs include user ID, IP address, a timestamp, and the type of action performed by logging the API request. You can find out more about each action being performed by clicking the API request. 

!!! note
    In some cases, clicking the API request will not return additional information. This is typically because the request includes sensitive information, such as a password. 

You can filter the logs by the following:

* User
* Project
* Request type (GET, POST, PATCH, DELETE)
* Date range
* Request keyword

---

### Source Guide Admin Manage (`docs/source/guide/admin_manage.md`)

---
title: Manage user accounts
short: Manage user accounts
tier: opensource
type: guide
order: 372
order_enterprise: 0
meta_title: Manage user accounts
meta_description: Manage existing user accounts in Label Studio Community edition
section: "Manage Your Organization"
date: 2024-02-06 16:36:44
---

!!! error Enterprise
    In Label Studio Community Edition, there are relatively limited options available for user management. 
    
    In Label Studio Enterprise, you can set roles for users, restrict their permissions, delete and deactivate users, and view detailed logs of user actions. For more information about the user management capabilities of Label Studio Enterprise, see [User management overview](https://docs.humansignal.com/guide/manage_users) in the Enterprise documentation. To start a free trial, [click here](https://app.heartex.com/user/trial).  

## Retrieve user info from the command line

You can retrieve information about a user, including the API user token for a user, from the command line after starting Label Studio.

From the command line, run the following:

```bash
label-studio user --username <username>
```

You can see user info, including their API token, as the last line of the response. For example:

```
=> User info:
{'id': 1, 'first_name': 'User', 'last_name': 'Somebody', 'username': 'label-studio', 'email': 'example@labelstud.io', 'last_activity': '2021-06-15T19:37:29.594618Z', 'avatar': '/data/avatars/img.jpg', 'initials': 'el', 'phone': '', 'active_organization': 1, 'token': '<api_token>', 'status': 'ok'}
```

Users can also retrieve their own tokens from the Label Studio app. For more information, see [Access Tokes](access_tokens). 


## Review existing accounts in Label Studio

You can view the organization user list from Label Studio:

1. Open Label Studio and click the menu in the upper left. 
2. Select **Organization**. 

Click the user's row to see additional details about their activity, such as when they were last active, which projects they created, and which projects they contributed to. 

## Reset passwords

If you forget your password or change passwords regularly for security reasons, you can change it from the command line.

On the server running Label Studio, run the following command. And then when prompted, type the username and the new password:

```bash
label-studio reset_password
```

You can also use optional command line arguments to reset the password for a username.

- Specify the username and type the password when prompted:

    ```bash
label-studio reset_password --username <username>
New password:
```

- Specify both the username and the password:

    ```bash
label-studio reset_password --username <username> --password <password>
```

---

### Source Guide Admin Manage Lse (`docs/source/guide/admin_manage_lse.md`)

---
title: Manage user accounts
short: Manage user accounts
tier: enterprise
type: guide
order: 0
order_enterprise: 372
meta_title: Manage user accounts
meta_description: How to assign users to roles, delete users, tag users, and deactivate users
section: "Manage Your Organization"
parent: "manage_users"
parent_enterprise: "manage_users"
date: 2024-02-06 12:00:25
---


!!! note
    If you are using SSO/SAML or SCIM, you can map groups to roles on the organization level and the project level. For more information, see [Set up SSO authentication for Label Studio](auth_setup) and [Set up SCIM2 for Label Studio](scim_setup). 

    If you are using an IdP with SCIM or SAML/SSO, you should also handle user deactivations and deletions from the IdP. 


## Assign roles to users

If you invite users using the [invite link option](admin_user#Invite-users-to-Label-Studio-Enterprise), their account is created but must be activated by assigning them to a role. 

!!! info Tip
    You can use the role filter above the table to see all Pending users. Select the **Not Activated** role.

1. Open Label Studio and click the menu in the upper left. 
2. Select **Organization**. 

    If you do not see the **Organization** option, you do not have access to the Organization page. Only users in the Owner or Administrator role can access this page. 

3. Locate the user account to which you are assigning a role. 

4. Use the drop-down menu next to the user to select a role. 




## Programmatically assign roles

To programmatically activate and assign roles to users, you can use the API:

https://api.labelstud.io/api-reference/api-reference/organizations/members/update


You can find user ID and organization ID from the **Organization** page:



You can also use the API:

* [List organizations](https://api.labelstud.io/api-reference/api-reference/organizations/list)
* [List members](https://api.labelstud.io/api-reference/api-reference/organizations/members/list)

## Tag members

You can tag organization members with metadata. 

### Apply member tags via CSV

To bulk apply member tags, go to the Organization page and click **Tag Members**. 

From here you can upload a CSV with the following format:

```csv
email,tags
heidi@humansignal.com,"Data Science, Seattle"
sally@humansignal.com,"Finance, New York City"
```

* Ensure you include `email,tags` as the header. 
* Tags must be encapsulated in quotation marks. 
* If you have multiple tags, separate them with a comma. 



!!! info Tip
    If you are exporting the CSV from a spreadsheet editor, make sure you check the format in a text editor before uploading. Spreadsheet editors can add unnecessary formatting. 


### Add and delete tags

To delete or manually add new tags, go to **Organization > Settings > Member Tags**. 

### Manually assign tags

If you want to manually assign tags to individual users, you can use the **Tags** column in the org members table. 



## Deactivate users

You can deactivate a user account by assigning them to the **Deactivated** role. Deactivating a user revokes their access and opens up their seat in your license. 

You can reactivate a user by assigning them to an active user role.


## Delete users

1. From the Organization page, select the user you want to delete. 

2. Click **Delete**. 

3. A confirmation message appears. Click **Delete Member**. 

Once deleted, any completed work or changes that the user made will appear as belonging to a "Deleted User."

If you want to add the user again at a later date, you will need to re-invite them to the organization.

---

### Source Guide Admin Permissions (`docs/source/guide/admin_permissions.md`)

---
title: Customize organization permissions
short: Permissions
tier: enterprise
type: guide
hide_menu: false
order: 0
order_enterprise: 362
meta_title: Customize organization permissions
section: "Manage Your Organization"
parent_enterprise: "admin_settings"
date: 2025-02-18 12:03:59
---

!!! note
    This page is only visible to users in the Owner role. 

You can use this page to customize certain permissions for roles. For information about Label Studio Enterprise roles and default permissions, see [User roles and permissions](admin_roles).

Note the following:

* Permissions for the Owner role are not configurable. 
* You can use the options on this page to have a more fined-grained control on which roles can execute certain actions. 
* Any restrictions you configure will also apply to the API. For example, if you restrict Managers' ability to configure cloud storage, they will not be able to complete those actions through the UI or the API. 


Permissions that can be configured include the following:

| Permission | Default Roles |
|------------|-------------|
| **Invite members to organization** | Manager+ |
| **Create API Tokens** | All roles |
| **Edit Plugins** | Manager+ |
| **Manage Cloud Storage** | Manager+ |
| **Manage Webhooks** | Manager+ |
| **Access Project Dashboard** | Manager+ |
| **Access Member Performance Dashboard** | All roles |
| **Access Project Members Dashboard** | Manager+ |
| **Use AI Assistant** | Manager+ |
| **Delete Tasks** | Manager+ |
| **Reset Project Cache** | Manager+ |
| **Drop All Tabs** | Manager+ |
| **Delete Project** | Manager+ |

!!! note
    Permissions for Annotators, Reviewers, and Managers are already restricted to projects that they created (if a Manager) or have explicitly been granted access to. 

!!! info tip
    You can click **Reset to Defaults** to return all permissions to their default value.

---

### Source Guide Admin Roles (`docs/source/guide/admin_roles.md`)

---
title: User roles and permissions
short: Roles and permissions
tier: enterprise
type: guide
order: 0
order_enterprise: 375
meta_title: User roles and permissions
meta_description: A description of the user roles and permissions available in Label Studio Enterprise. 
section: "Manage Your Organization"
parent: "manage_users"
parent_enterprise: "manage_users"
date: 2024-02-05 17:19:21
---

Your user role determines your level of access to actions and information in Label Studio. 

* Access for Annotators, Reviewers, and Managers is restricted based on which projects and workspaces they have access to. For more information, see [Project setup](setup_project). 
* Owners can further restrict permissions within their organization using the [Permissions page](admin_permissions). 

## Roles in Label Studio Enterprise

For information on how to assign users to roles, see [Manage user accounts](admin_manage_lse).

| Role          | Description                                                                                                                                                                                  |
| ------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Owner**         | Manages the organization and has full permissions at all levels.<br><br>This is not an assignable role, and there is only one Owner per organization. By default, the Owner is tied to the account that created the Label Studio organization. To change the Owner, you will need to [open a support ticket](https://support.humansignal.com/hc/en-us/requests/new).                                                           |
| **Administrator** | Has full permissions at most levels.<br><br>Can access and update all workspaces and projects, invite members to the organization, and set most ([but not all](admin_usage)) organization settings.                            |
| **Manager**       | Has full administrative access over projects and workspaces that they created or to which they have been added as a member. <br><br>Managers cannot access the Organization page. |
| **Reviewer**      | Reviews annotated tasks.<br><br>Can only view projects that include tasks assigned to them. Can review and update task annotations.                                                                              |
| **Annotator**     | Labels tasks.<br><br>Can only view projects with tasks assigned to them and label tasks in those projects.                                                                                               |


## Permissions in Label Studio Enterprise

<table>
  <tr>
    <th>Action</th>
    <th>Annotator</th>
    <th>Reviewer</th>
    <th>Manager</th>
    <th>Administrator</th>
    <th>Owner</th>
  </tr>
  <tr>
    <td colspan="6" style="text-align:center"><b>Organization Management</b></td>
  </tr>
  <tr>
    <td>View the Organization page</td>
    <td></td>
    <td></td>
    <td style="text-align:center"></td>
    <td style="text-align:center">✅</td>
    <td style="text-align:center">✅</td>
  </tr>
    <tr>
    <td>Assign member roles</td>
    <td></td>
    <td></td>
    <td></td>
    <td style="text-align:center">✅</td>
    <td style="text-align:center">✅</td>
  </tr>
  <tr>
    <td>Invite users</td>
    <td></td>
    <td></td>
    <td></td>
    <td style="text-align:center">✅</td>
    <td style="text-align:center">✅</td>
  </tr>
  <tr>
    <td>View system-wide activity log</td>
    <td></td>
    <td></td>
    <td></td>
    <td style="text-align:center">✅</td>
    <td style="text-align:center">✅</td>
  </tr>
    <tr>
    <td>Set up model providers for the org</td>
    <td></td>
    <td></td>
    <td></td>
    <td style="text-align:center">✅</td>
    <td style="text-align:center">✅</td>
  </tr>
    </tr>
    <tr>
    <td>Configure organization settings</td>
    <td></td>
    <td></td>
    <td></td>
    <td style="text-align:center">Partial</td>
    <td style="text-align:center">✅</td>
  </tr>
  <tr>
    <td colspan="6" style="text-align:center"><b>Project Management</b></td>
  </tr>
  <tr>
    <td>Workspace access</td>
    <td style="text-align:center">R</td>
    <td style="text-align:center">R</td>
    <td style="text-align:center">CRUD[^1]</td>
    <td style="text-align:center">CRUD</td>
    <td style="text-align:center">CRUD</td>
  </tr>
  <tr>
    <td>Project access</td>
    <td style="text-align:center">R</td>
    <td style="text-align:center">R</td>
    <td style="text-align:center">CRUD[^1]</td>
    <td style="text-align:center">CRUD</td>
    <td style="text-align:center">CRUD</td>
  </tr>
    <tr>
    <td>View the Data Manager</td>
    <td>Depends on project settings</td>
    <td>Depends on project settings</td>
    <td style="text-align:center">✅</td>
    <td style="text-align:center">✅</td>
    <td style="text-align:center">✅</td>
  </tr>
  <tr>
    <td>Configure project settings</td>
    <td></td>
    <td></td>
    <td style="text-align:center">✅</td>
    <td style="text-align:center">✅</td>
    <td style="text-align:center">✅</td>
  </tr>
  <tr>
    <td>View and configure project plugins</td>
    <td></td>
    <td></td>
    <td style="text-align:center">✅[^2]</td>
    <td style="text-align:center">✅</td>
    <td style="text-align:center">✅</td>
  </tr>
  <tr>
    <td>Pause annotators and reviewers</td>
    <td></td>
    <td></td>
    <td style="text-align:center">✅</td>
    <td style="text-align:center">✅</td>
    <td style="text-align:center">✅</td>
  </tr>
  <tr>
    <td colspan="6" style="text-align:center"><b>Data Access</b></td>
  </tr>
  <tr>
    <td>Import data</td>
    <td></td>
    <td></td>
    <td style="text-align:center">✅</td>
    <td style="text-align:center">✅</td>
    <td style="text-align:center">✅</td>
  </tr>
  <tr>
    <td>Export data</td>
    <td></td>
    <td></td>
    <td style="text-align:center">✅</td>
    <td style="text-align:center">✅</td>
    <td style="text-align:center">✅</td>
  </tr>
  <tr>
    <td colspan="6" style="text-align:center"><b>Data Labeling Workflows</b></td>
  </tr>
  <tr>
    <td>Assign annotators to tasks</td>
    <td></td>
    <td style="text-align:center"></td>
    <td style="text-align:center">✅</td>
    <td style="text-align:center">✅</td>
    <td style="text-align:center">✅</td>
  </tr>
  <tr>
    <td>Access labeling workflow</td>
    <td style="text-align:center">✅</td>
    <td style="text-align:center">✅</td>
    <td style="text-align:center">✅</td>
    <td style="text-align:center">✅</td>
    <td style="text-align:center">✅</td>
  </tr>
  <tr>
    <td>Access review workflow</td>
    <td></td>
    <td style="text-align:center">✅</td>
    <td style="text-align:center">✅</td>
    <td style="text-align:center">✅</td>
    <td style="text-align:center">✅</td>
  </tr>
  <tr>
    <td>Monitor annotator agreement</td>
    <td></td>
    <td style="text-align:center">✅</td>
    <td style="text-align:center">✅</td>
    <td style="text-align:center">✅</td>
    <td style="text-align:center">✅</td>
  </tr>

  <tr>
    <td>Verify annotation results</td>
    <td></td>
    <td style="text-align:center">✅</td>
    <td style="text-align:center">✅</td>
    <td style="text-align:center">✅</td>
    <td style="text-align:center">✅</td>
  </tr>
  <tr>
    <td>Assign reviewers to tasks</td>
    <td></td>
    <td></td>
    <td style="text-align:center">✅</td>
    <td style="text-align:center">✅</td>
    <td style="text-align:center">✅</td>
  </tr>
  <tr>
    <td>Perform bulk labeling</td>
    <td style="text-align:center">✅[^3]</td>
    <td style="text-align:center">✅</td>
    <td style="text-align:center">✅</td>
    <td style="text-align:center">✅</td>
    <td style="text-align:center">✅</td>
  </tr>
  <tr>
    <td colspan="6" style="text-align:center"><b>Advanced</b></td>
  </tr>
  <tr>
    <td>API access to equivalent Label Studio functionality</td>
    <td style="text-align:center">✅</td>
    <td style="text-align:center">✅</td>
    <td style="text-align:center">✅[^1]</td>
    <td style="text-align:center">✅</td>
    <td style="text-align:center">✅</td>
  </tr>
  <tr>
    <td colspan="6" style="text-align:center"><b>Analytics</b></td>
  </tr>
  <tr>
    <td>View project dashboard and members dashboard</td>
    <td></td>
    <td></td>
    <td style="text-align:center">Workspace and invited projects</td>
    <td style="text-align:center">Organization</td>
    <td style="text-align:center">Organization</td>
  </tr>
  <tr>
    <td>View annotator dashboard</td>
    <td style="text-align:center">✅[^4]</td>
    <td style="text-align:center">✅[^4]</td>
    <td style="text-align:center">✅</td>
    <td style="text-align:center">✅</td>
    <td style="text-align:center">✅</td>
  </tr>
    <tr>
    <td colspan="6" style="text-align:center"><b>Prompts</b></td>
  </tr>
  <tr>
      <td>Use Prompts (create/evaluate/run/apply to projects)</td>
      <td></td>
      <td></td>
      <td>✅</td>
      <td>✅</td>
      <td>✅</td>
    </tr>
    <tr>
      <td>See predictions generated by Prompts in projects</td>
      <td>✅</td>
      <td>✅</td>
      <td>✅</td>
      <td>✅</td>
      <td>✅</td>
    </tr>
</table>


[^1]: For workspaces or projects they created or were invited to
[^2]: Owners can restrict this in the org settings
[^3]: The project must be using manual distribution and the annotator must be granted Data Manager access
[^4]: For their own work/history

---

### Source Guide Admin Settings (`docs/source/guide/admin_settings.md`)

---
title: Organization settings
short: Settings
type: guide
tier: enterprise
order: 0
order_enterprise: 357
meta_title: Organization settings
meta_description: Configure org-wide settings in Label Studio Enterprise.  
section: "Manage Your Organization"
---

There are several places where you can configure org-wide settings:

| Field          | Access    | Description  |
| ------------- | ------------ | ------------ |
| [**Usage & License**](admin_usage) | Owner<br />Admin | This page has a mix of settings that can be set by Administrators (enabling email notifications) and Owners (enabling AI, enabling storage proxy, enabling early adopter features).   | 
| [**Access Token**](access_tokens) | Owner<br />Admin | Control which types of access tokens are available in the organization.  | 
| [**Model Providers**](model_providers)         | Owner<br />Admin | Set up model providers that can be used with [Prompts](prompts_overview) and [Chat](/tags/chat.html). | 
| [**Permissions**](admin_permissions)        |  Owner | Customize certain permissions for roles.  | 
| [**Support reports**](support_reports)      | Owner<br />Admin | Generate anonymized operational reports that help HumanSignal support understand your deployment, diagnose issues, and recommend workflow and performance improvements. |

---

### Source Guide Admin Usage (`docs/source/guide/admin_usage.md`)

---
title: Usage & License
short: Usage & License
type: guide
tier: enterprise
order: 0
order_enterprise: 358
meta_title: Usage & License
meta_description: Find information about your Label Studio account and configure organization settings.  
section: "Manage Your Organization"
parent_enterprise: "admin_settings"
---

You can find information about your plan and seat usage on this page. 

## Label Studio Enterprise

### Seats in use

Here you can see the number of seats compared with the number of active users. 

Each organization has a set amount of Label Studio seats in their license. To increase your seats, contact your HumanSignal account representative. 

!!! note On-premise
    On-premise deployments can use fewer seats than their license allows by setting the `LICENSE_MAX_USERS_OVERRIDE` environment variable. It can only lower the licensed seat count, and the deployment does not start if you set it higher. Seats are counted across all organizations in the deployment. See [Available Helm values](helm_values#The-global-extraEnvironmentVars-usage).

### License info

This section includes general information about your Label Studio license, such as when it was issued and when it expires. 

It also includes the following usage information:

<dl>
<dt>Projects</dt>
<dd>

**Cloud/SaaS:** Every project that belongs to your current organization, regardless of state (draft/published) or workspace (meaning that it includes sandbox projects). 

**On-prem:** All projects on your server, regardless of state workspace. 
</dd>

<dt>Results</dt>
<dd>

The total number of labeled regions (results) that your organization has created across all projects. A result is one item in an annotation’s result array — a bounding box, text span, relation between regions, etc. 
</dd>
</dl>

### Security 

Only the org Owner can update these settings.

<dl>
<dt>Session Timeout Policy</dt>
<dd>

| Field          | Description    |
| ------------- | ------------ |
| **Max session age (minutes)**         | How many minutes a user can be idle. If they exceed this limit, they are automatically logged out the next time they trigger a request. <br> Any interaction that hits the backend (clicks that trigger API calls, saves, polling, etc.) resets this timer. |
| **Max time between activity (minutes)** | Absolute session lifetime. Regardless of activity, a session cannot last longer than this many minutes from login; once reached, the user must re-authenticate. |

</dd>

<dt>Single Sign-On (SSO) enabled</dt>
<dd>
This indicates with your organization can use SSO. 

SSO is not available if you are in a trial organization or if it is not enabled for your license. 

</dd>
</dl>

### Embedding

 You can use these fields to set up [Label Studio Embeds](embed#Configure-embedding-in-Label-Studio). 

!!! note
    Embedding is not available to all customers. Contact your [HumanSignal account manager](mailto:sales@humansignal.com) to enable.

### Features

Only the org Owner can update these settings and request access.

The features can be requested and enabled for your license. 

| Field          | Description    |
| ------------- | ------------ |
| **Invite external annotators to projects**  | When enabled, your organization can send email invites to contractors/partners to give them project-scoped access (Annotator role). <br> When not enabled, only existing org members can be added to projects; you’d provision any new users through your normal onboarding (e.g., SSO/SCIM) first. |
| **Access to Activity Log** | Whether your organization has access to [activity logs](admin_logs). |
| **Prompts** | Whether your organization has access to [Prompts](prompts_overview). |
| **Whitelabel** | Whether your organization has whitelabeling enabled. |
| **Plugins** | Toggle whether your organization has access to plugins. And, if enabled, whether users in the Manager role can create and update them. |
| **Early Adopter** | Opt-in to new features before they're released to the public. |
| **Enable Storage Proxy** | Allow Label Studio to [proxy data from cloud storage](storage#Pre-signed-URLs-vs-Storage-proxies). |
| **Enable AI Features** | Enable [AI Assistant](ask_ai) for your labeling interface configuration. |
| **Enable Ask AI** | Enable the [AI Assistant](ask_ai) for general Label Studio help. |

### Email notification settings

Determines what email preferences are available from the [**Account and Settings** page](user_account). 

Note the following:

* New users will have their email preferences enabled by default. 
* If you disable a notification here, that preference is hidden from their **Account and Settings** page and is blocked for all users in your organization. 
* If you re-enable a notification, the preference will appear again in their **Account and Settings** page and will revert back to whatever state the user had it in before it was disabled for the organization. 

### Frequently asked Enterprise billing questions

Find answers to common questions related to billing. 

#### What happens if my plan is inactive?

If your subscription to Label Studio Enterprise expires or is cancelled, you can no longer perform labeling, review annotations, or add new users to your organization. You can log in and export your completed annotations. 

#### What happens if I have too many active users?

You need to purchase additional seats for your Label Studio Teams subscription if you have more active users than your subscription allows. A user counts as an active user when they are assigned a role. Invited users that have not yet accepted an invitation appear as "Not activated" and do not count toward the seat limit.

## Label Studio Starter Cloud

Label Studio Starter Cloud has the **Usage & Billing** page that shows information about your subscription and how many seats you have in use. 

!!! note
    Only the Organization owner can access the Usage & Billing page. 

There are two components to the subscription:

* **Base subscription** - This is your access to Label Studio Starter Cloud and includes 1 seat.
* **Additional seats** - You can purchase up to 11 additional seats (for a total of 12) on an as-needed basis.

### Manage seats

#### Add seats

As an Owner, go to **Organization > Usage & Billing**. Click **Manage Seats** and increase your seat count. 

When you add seats, the cost is prorated and is reflected on your next invoice. An organization can have up to 12 seats. 

#### Remove seats

Before you can remove seats, you must deactivate their associated users. You cannot have more active users than seats.

To deactivate a user, go to the **Organization > Members** page and set the user's role to **Deactivated**. 

Then click **Manage Seats** and decrease your seat count. 

When you remove seats, the cost is prorated and is reflected on your next invoice.

### Frequently asked Starter Cloud billing questions

#### How do I cancel?

As an Owner, go to **Organization > Usage & Billing** and click **Cancel**. 

After cancellation, your account remains active until the end of your paid billing period. During this time, you have full access to the Label Studio Starter Cloud features.

However, after cancellation, you cannot add user seats. If you need more users than your current seat count allows, you must reactivate your subscription.

When the billing period ends, you will retain access to Label Studio and can download your data. However, you will be unable to import new data, annotate existing data, or create new projects.

#### How do I renew?

If you do not cancel your subscription, it is automatically renewed at the end of the billing period.

#### Can I upgrade to Label Studio Enterprise?

Yes, you can upgrade to Label Studio Enterprise, please contact our sales: sales@humansignal.com.

---

### Source Guide Admin User (`docs/source/guide/admin_user.md`)

---
title: Add members in Label Studio Enterprise
short: Add members
tier: enterprise
type: guide
order: 0
order_enterprise: 369
meta_title: Add and invite members to Label Studio
meta_description: Overview of how you can add members to Label Studio Enterprise
section: "Manage Your Organization"
parent: "manage_users"
parent_enterprise: "manage_users"
date: 2024-02-05 17:15:19
---

You can view, add, and manage organization members from the **Organization** page in Label Studio. 

Navigate to the **Organization** page by clicking the menu in the upper left and selecting **Organization**. 

Only users in the Owner or Administrator role can access this page. 


## Invite members

Once the initial organization account is created, you can begin inviting users. 

From the Organization page, click **Invite Members**. From here you have two options:

### Invite link

You can share this link with users, who can then complete the registration process. 

However, they will be unable to access Label Studio until an Owner or Administrator manually assigns them a role. See [User roles and permissions](admin_roles). 

Click **Reset** to reset the link. When reset, anyone who has the old link will be unable to register a new user account. 

### Invite via email

Enter a list of email addresses separated by commas and then select a user role for the new accounts. 
    
Users will receive an email with a link to create their Label Studio account, and will be able to access Label Studio as soon as their registration is complete. 

### Require invites for new users (on-prem deployments)

While you can invite users to join your organization with the invite link, this does not prevent users from registering a new account through the signup page.

You can remove the option to create accounts through the `/user/signup` page by setting the following environment variable:

```bash
LABEL_STUDIO_DISABLE_SIGNUP_WITHOUT_LINK=true
```

Users will only be able to sign up through an invitation link or email. 

## Activate new users

Users who sign up through the link will not be able to access Label Studio until they are assigned a user role. 

To filter for users who are pending, select **Pending** from the roles filter at the top of the **Organization > Members** page:



Then use the drop-down menu to assign a role:

---

### Source Guide Agreement Metrics (`docs/source/guide/agreement_metrics.md`)

---
title: Built-in agreement metrics reference
short: Built-in metrics
tier: enterprise
type: guide
order: 0
order_enterprise: 309
meta_title: Built-in agreement metrics in Label Studio Enterprise
meta_description: Built-in agreement metrics in Label Studio Enterprise.
section: "Review & Measure Quality"
parent: "stats"
parent_enterprise: "stats"
---

The following metrics are available for control tags out-of-the-box in Label Studio Enterprise. You can use them as is, or you can create your own [custom metrics](custom_metric).

## All control tags

| Metric | Description | Tags | Methodology |
|--------|-------------|------|-------------|
| **Exact Match** | Evaluates whether annotation results exactly match, with optional label weights | All tags | Pairwise, Consensus |


## Choices and taxonomy

Categorical metrics are used for categorical control tags, such as `Choices` and `Taxonomy`.

| Metric | Description | Tags | Methodology |
|--------|-------------|------|-------------|
| **Common Labels Matches** | Evaluates common label matches for a taxonomy of labels assigned to regions. Computes partial credit along taxonomy paths. | Taxonomy | Pairwise |
| **Common Labels Matches (Threshold)** | Evaluates common label matches for a taxonomy of labels, returns binarized match based on threshold | Taxonomy | Pairwise, Consensus |
| **Common Subtree Matches** | Evaluates common subtree matches for a taxonomy of choices. Computes IoU over the subtree of selected taxonomy nodes. | Taxonomy | Pairwise |
| **Common Subtree Matches (Threshold)** | Evaluates common subtree matches for a taxonomy of choices, returns binarized match based on threshold | Taxonomy | Pairwise, Consensus |
| **Jaccard Similarity** | Evaluates common label matches using set intersection divided by set union | Choices (multi-select) | Pairwise |
| **Jaccard Similarity (Threshold)** | Evaluates common label matches, returns binary match based on threshold | Choices (multi-select) | Pairwise, Consensus |

## Numeric

| Metric | Description | Tags | Methodology |
|--------|-------------|------|-------------|
| **Numeric Difference** | Evaluates how similar two numeric values are based on their absolute difference | Number<br/><br/>Rating | Pairwise |
| **Numeric Difference (Threshold)** | Evaluates whether two numeric values match within a specified tolerance | Number<br/><br/>Rating | Pairwise, Consensus |

## Rectangles

| Metric | Description | Tags | Methodology |
|--------|-------------|------|-------------|
| **Intersection over Union** | Evaluates overlap between bounding box regions, returns raw IoU score | RectangleLabels<br/><br/>Rectangle | Pairwise |
| **Intersection over Union (Threshold)** | Evaluates overlap between bounding box regions, returns binarized match based on threshold | RectangleLabels<br/><br/>Rectangle | Pairwise, Consensus |
| **Bounding Box Labels Similarity** | Evaluates bbox overlap with Jaccard similarity for label matching, returns raw Jaccard score | Choices*  | Pairwise |
| **Bounding Box Labels Similarity (Threshold)** | Evaluates bbox overlap with Jaccard similarity for label matching, returns binary match based on threshold | Choices* | Pairwise, Consensus |
| **Bounding Box Text Similarity** | Evaluates bounding box overlap with text similarity for text matching, returns raw similarity score | TextArea* | Pairwise |
| **Bounding Box Text Similarity (Threshold)** | Evaluates bounding box overlap with text similarity for text matching, returns binary match based on threshold | TextArea* | Pairwise, Consensus |

\* Nested Choices or TextArea tags inside RectangleLabels/Rectangle tags

## Polygons

| Metric | Description | Tags | Methodology |
|--------|-------------|------|-------------|
| **Intersection over Union for Polygons** | Evaluates overlap between polygon regions, returns raw IoU score | PolygonLabels<br/><br/>Polygon | Pairwise |
| **Intersection over Union for Polygons (Threshold)** | Evaluates overlap between polygon regions, returns binarized match based on threshold | PolygonLabels<br/><br/>Polygon | Pairwise, Consensus |
| **Polygon Labels Similarity** | Evaluates polygon overlap with Jaccard similarity for label matching, returns raw Jaccard score | Choices* | Pairwise |
| **Polygon Labels Similarity (Threshold)** | Evaluates polygon overlap with Jaccard similarity for label matching, returns binary match based on threshold | Choices* | Pairwise, Consensus |
| **Polygon Text Similarity** | Evaluates polygon overlap with text similarity for text matching, returns raw similarity score | TextArea* | Pairwise |
| **Polygon Text Similarity (Threshold)** | Evaluates polygon overlap with text similarity for text matching, returns binary match based on threshold | TextArea* | Pairwise, Consensus |

\* Nested Choices or TextArea tags inside PolygonLabels/Polygon tags

## Brush

| Metric | Description | Tags | Methodology |
|--------|-------------|------|-------------|
| **Brush Intersection over Union** | Evaluates pixel overlap between brush mask regions, returns raw IoU score | BrushLabels<br/><br/>Brush | Pairwise |
| **Brush Intersection over Union (Threshold)** | Evaluates pixel overlap between brush mask regions, returns binarized match based on threshold | BrushLabels<br/><br/>Brush | Pairwise, Consensus |

## Span and segment

| Metric | Description | Tags | Methodology |
|--------|-------------|------|-------------|
| **Span Overlap** | Evaluates overlap between one-dimensional labeled regions, returns raw IoU score | Labels<br/><br/>ParagraphLabels<br/><br/>TimeSeriesLabels<br/><br/>TimelineLabels | Pairwise |
| **Span Overlap (Threshold)** | Evaluates overlap between labeled regions, returns binarized match based on threshold | Labels<br/><br/>ParagraphLabels<br/><br/>TimeSeriesLabels<br/><br/>TimelineLabels | Pairwise, Consensus |
| **Span Labels Similarity** | Evaluates span overlap with Jaccard similarity for label matching, returns raw Jaccard score | Choices* | Pairwise |
| **Span Labels Similarity (Threshold)** | Evaluates span overlap with Jaccard similarity, returns binary match based on threshold | Choices* | Pairwise, Consensus |
| **Span Text Similarity** | Evaluates span overlap with text edit distance, returns raw similarity score | TextArea* | Pairwise |
| **Span Text Similarity (Threshold)** | Evaluates span overlap with text similarity, returns binary match based on threshold | TextArea* | Pairwise, Consensus |
| **Unordered Naive Comparison for Timeline Labels** | Compares timeline label annotations without regard to label order | TimelineLabels | Pairwise, Consensus |

\* Nested Choices or TextArea tags inside Labels tags

## HTML spans

| Metric | Description | Tags | Methodology |
|--------|-------------|------|-------------|
| **Overlap over HTML Spans** | Evaluates whether two given hypertext spans have points in common | HyperTextLabels | Pairwise |
| **Overlap over HTML Spans (Threshold)** | Evaluates HTML span overlap, returns binarized match based on threshold | HyperTextLabels | Pairwise, Consensus |
| **HTML Span Labels Similarity** | Evaluates HTML span overlap with Jaccard similarity for label matching, returns raw Jaccard score | Choices* | Pairwise |
| **HTML Span Labels Similarity (Threshold)** | Evaluates HTML span overlap with Jaccard similarity, returns binary match based on threshold | Choices* | Pairwise, Consensus |
| **HTML Span Text Similarity** | Evaluates HTML span overlap with text edit distance, returns raw similarity score | TextArea* | Pairwise |
| **HTML Span Text Similarity (Threshold)** | Evaluates HTML span overlap with text similarity, returns binary match based on threshold | TextArea* | Pairwise, Consensus |

\* Nested Choices or TextArea tags inside HyperTextLabels tags

## Text

| Metric | Description | Tags | Methodology |
|--------|-------------|------|-------------|
| **Text Similarity** | Uses the edit distance algorithm to calculate how similar two text annotations are to one another | TextArea | Pairwise |
| **Text Similarity (Threshold)** | Uses the edit distance algorithm to determine if two text annotations match based on a similarity threshold | TextArea | Pairwise, Consensus |
| **Semantic Similarity** | Evaluates text similarity by comparing semantic meaning using embeddings | User-defined | Pairwise, Consensus |

## Video 

| Metric | Description | Tags | Methodology |
|--------|-------------|------|-------------|
| **Exact Frames Matching for Video** | Evaluates video annotations by comparing exact frame matches | VideoRectangle | Pairwise |
| **Exact Frames Matching for Video (Threshold)** | Evaluates video annotations by comparing exact frame matches, returns binarized match based on threshold | VideoRectangle | Pairwise, Consensus |
| **Video Tracking** | Evaluates video tracking by comparing bounding boxes using IoU score across frames | User-defined | Pairwise, Consensus |

## Keypoints

| Metric | Description | Tags | Methodology |
|--------|-------------|------|-------------|
| **Keypoint Distance** | Evaluates keypoint annotations by checking if corresponding labeled keypoints are within a coordinate distance threshold | KeypointLabels<br/><br/>Keypoint | Pairwise |

## Examples

### Exact Match

Exact Match is the simplest agreement metric. It checks whether two annotators gave the exact same answer, and returns either a perfect score (1.0) or zero (0.0). There is no partial credit.

This is the default metric for `Choices`, `Taxonomy`, `Pairwise`, and `DateTime` tags. It is also available as an alternative metric for many other tag types.

#### What affects the score

| Scenario | Score |
|----------|-------|
| Annotators give the exact same answer | 1.0 |
| Annotators give different answers | 0.0 |
| Neither annotator provides an answer | 1.0 |
| One annotator provides an answer, the other doesn't | 0.0 |

Unlike Span Overlap or IoU, there is no partial credit with Exact Match. The annotators either agree completely or they don't.

#### Single-select classification

When a labeling configuration uses a single-select `Choices` tag (e.g., sentiment analysis), each annotator picks one option. Exact Match compares the two selections directly.

**Example**: A project classifies customer reviews as Positive, Negative, or Neutral.

| Annotator A | Annotator B | Score |
|-------------|-------------|-------|
| Positive | Positive | **1.0** |
| Positive | Negative | **0.0** |
| Neutral | Neutral | **1.0** |

#### Multi-select classification

When a `Choices` tag allows multiple selections, each annotator's response is a list. Exact Match compares the two lists and requires them to contain the same items in the same order.

**Example**: A project tags articles with topics: Sports, Politics, Technology, Health.

| Annotator A | Annotator B | Score |
|-------------|-------------|-------|
| [Sports, Politics] | [Sports, Politics] | **1.0** |
| [Sports, Politics] | [Politics, Sports] | **0.0** (different order) |
| [Sports, Politics] | [Sports] | **0.0** (different selections) |

!!! note
    If you want partial credit for multi-select classifications (e.g., matching 2 out of 3 selected items), use [Jaccard Similarity](#Jaccard-similarity) instead of Exact Match.

#### Taxonomy

For `Taxonomy` tags, the annotator's selection is a specific path through the taxonomy tree. Exact Match requires the full path to be identical.

**Example**: A project uses a taxonomy to classify animals.

| Annotator A | Annotator B | Score |
|-------------|-------------|-------|
| Animals > Dogs > Labrador | Animals > Dogs > Labrador | **1.0** |
| Animals > Dogs > Labrador | Animals > Dogs > Poodle | **0.0** |
| Animals > Dogs > Labrador | Animals > Dogs | **0.0** |

!!! note
    If you want partial credit for taxonomy paths that share a common prefix, use [Common Labels Matches](#Common-labels-matches) or [Common Subtree Matches](#Common-subtree-matches) instead.

#### DateTime

For `DateTime` tags, Exact Match compares the two date/time values. They must be identical to score 1.0.

| Annotator A | Annotator B | Score |
|-------------|-------------|-------|
| 2025-03-19 | 2025-03-19 | **1.0** |
| 2025-03-19 | 2025-03-20 | **0.0** |

### Span Overlap

Span Overlap measures how much two annotators agree on the position of labeled spans in text (or other one-dimensional data like audio segments or time series). It is the default metric for `Labels`, `ParagraphLabels`, `TimeSeriesLabels`, and `TimelineLabels` tags.

#### What affects the score

| Scenario | Effect on score |
|----------|----------------|
| Annotators highlight the exact same characters with the same label | `1.0` (perfect agreement) |
| Spans overlap partially with the same label | Between `0.0` and `1.0`, proportional to the overlap |
| Spans have different labels, even if positions are identical | `0.0` |
| One annotator creates a span that the other doesn't | Pulls the average down (unmatched span scores `0.0`) |
| Neither annotator creates any spans | `1.0` (agreement that there is nothing to label) |

#### What counts as a "span"

When annotators highlight a region of text and assign it a label, Label Studio records the character positions where the highlight starts and ends, along with the label. For example, in the sentence:

`Dr. Maria Chen presented her findings at the Berlin conference.`

If an annotator highlights `Dr. Maria Chen` and labels the span as "Person", the span is recorded as characters 0 through 14.

#### Step 1: Check that labels match

Before measuring any positional overlap, Span Overlap first checks whether two spans share the same label. If the labels are different, the score for that pair is **0.0** regardless of how much they overlap positionally.

For example, if Annotator A labels characters 0–14 as **Person** and Annotator B labels characters 0–14 as **Organization**, the score is 0.0 even though the character ranges are identical.

#### Step 2: Calculate IoU (Intersection over Union)

For two spans with matching labels, the metric calculates how much they overlap using IoU:

```
IoU = Intersection length / Union length
```

- **Intersection** is the region where both spans overlap.
- **Union** is the total region covered by either span.

**Example**: Consider two annotators labeling the same sentence:

`Dr. Maria Chen presented her findings at the Berlin conference.`

| | Span | Characters | Label |
|---|------|------------|-------|
| Annotator A | "Dr. Maria Chen" | `0–14` | Person |
| Annotator B | "Maria Chen" | `4–14` | Person |

The labels match (both **Person**), so we calculate IoU:

- Intersection: characters 4–14 (the overlap) = **10 characters**
- Union: characters 0–14 (the combined extent) = **14 characters**
- IoU = 10 / 14 = **0.71**

#### Step 3: Match spans across annotations using greedy matching

When annotators create multiple spans, the metric needs to figure out which spans from Annotator A correspond to which spans from Annotator B. It does this using **greedy matching**:

1. For every span in Annotator A's work, find the span in Annotator B's work with the highest IoU score.
2. For every span in Annotator B's work, find the span in Annotator A's work with the highest IoU score.
3. Average all of these best-match scores together.

This two-way matching means that unmatched spans (spans one annotator created but the other didn't) naturally pull the overall score down, because their best-match score will be 0.0.

**Full example**: Two annotators label named entities in this sentence:

`Dr. Maria Chen presented her findings at the Berlin conference.`

| | Span | Characters | Label |
|---|------|------------|-------|
| Annotator A | "Dr. Maria Chen" | 0–14 | Person |
| Annotator A | "Berlin" | 41–47 | Location |
| Annotator B | "Maria Chen" | 4–14 | Person |
| Annotator B | "Berlin conference" | 41–58 | Location |

First, compute the IoU for each possible pair:

| | B: "Maria Chen" (Person) | B: "Berlin conference" (Location) |
|---|---|---|
| **A: "Dr. Maria Chen" (Person)** | Labels match → IoU = 10/14 = **0.71** | Labels differ → **0.0** |
| **A: "Berlin" (Location)** | Labels differ → **0.0** | Labels match → IoU = 6/17 = **0.35** |

Then, find the best match for each span:

| Span | Best match | Score |
|------|------------|-------|
| A: "Dr. Maria Chen" | B: "Maria Chen" | 0.71 |
| A: "Berlin" | B: "Berlin conference" | 0.35 |
| B: "Maria Chen" | A: "Dr. Maria Chen" | 0.71 |
| B: "Berlin conference" | A: "Berlin" | 0.35 |

**Final score** = average of all best-match scores = `(0.71 + 0.35 + 0.71 + 0.35) / 4` = **0.53**

#### How the threshold variant works

The base Span Overlap metric returns a continuous score (0.53 in the example above). The **Span Overlap (Threshold)** variant converts this into a simple yes-or-no match by comparing the score against a threshold.

For example, with a threshold of 0.5:
- Score 0.53 >= 0.5 → **1.0** (match)

With a threshold of 0.75:
- Score 0.53 < 0.75 → **0.0** (no match)

The threshold variant is required when using the consensus methodology, which needs binary match/no-match decisions.

### Intersection over Union (for bounding boxes)

Intersection over Union (IoU) measures how much two annotators agree on the position and size of bounding boxes drawn on an image. It is the default metric for `RectangleLabels` and `Rectangle` tags.

#### What affects the score

| Scenario | Effect on score |
|----------|----------------|
| Annotators draw boxes in the exact same position and size with the same label | 1.0 (perfect agreement) |
| Boxes overlap partially with the same label | Between 0.0 and 1.0, proportional to the overlapping area |
| Boxes have different labels, even if positions are identical | 0.0 |
| One annotator draws a box that the other doesn't | Pulls the average down (unmatched box scores 0.0) |
| Neither annotator draws any boxes | 1.0 (agreement that there is nothing to label) |

#### What counts as a bounding box

When an annotator draws a rectangle on an image and assigns it a label, Label Studio records the box's position (x, y coordinates of the top-left corner) and its width and height. For example, if an annotator draws a box around a dog in a photo and labels it **Dog**, Label Studio stores the box's coordinates along with that label.

#### Step 1: Check that labels match

Just like Span Overlap, IoU first checks whether two bounding boxes share the same label. If the labels are different, the score for that pair is **0.0** regardless of how much the boxes overlap.

For example, if Annotator A draws a box and labels it **Dog** and Annotator B draws a box in the exact same position but labels it **Cat**, the score is 0.0.

#### Step 2: Calculate IoU

For two boxes with matching labels, the metric calculates how much area they share:

```
IoU = Intersection area / Union area
```

- **Intersection** is the area where both boxes overlap (the region covered by both).
- **Union** is the total area covered by either box (both boxes combined, not double-counting the overlap).

Another way to express this:

```
IoU = Intersection area / (Area of box A + Area of box B − Intersection area)
```

**Example**: Two annotators are labeling dogs in a photo.

Annotator A draws a box around the dog:
- Position: `x=10`, `y=20`
- Size: `100` wide × `80` tall
- Label: **Dog**

Annotator B draws a slightly different box around the same dog:
- Position: `x=30`, `y=20`
- Size: `100` wide × `80` tall
- Label: **Dog**

The labels match (both **Dog**), so we calculate IoU:

- Box A covers `x=10` to `110`, `y=20` to `100`
- Box B covers `x=30` to `130`, `y=20` to `100`
- Intersection: `x=30` to `110`, `y=20` to `100` = `80` wide × `80` tall = **6,400 sq. units**
- Area of Box A = `100` × `80` = **8,000 sq. units**
- Area of Box B = `100` × `80` = **8,000 sq. units**
- Union = `8,000` + `8,000` − `6,400` = **9,600 sq. units**
- IoU = `6,400` / `9,600`  = **0.67**

#### Step 3: Match boxes across annotations using greedy matching

When annotators draw multiple bounding boxes, the metric uses the same **greedy matching** algorithm described in Span Overlap:

1. For every box in Annotator A's work, find the box in Annotator B's work with the highest IoU score.
2. For every box in Annotator B's work, find the box in Annotator A's work with the highest IoU score.
3. Average all of these best-match scores together.

**Full example**: Two annotators label objects in a photo containing a dog and a cat.

| | Box | Label |
|---|-----|-------|
| Annotator A | Box around the dog | Dog |
| Annotator A | Box around the cat | Cat |
| Annotator B | Box around the dog (shifted slightly) | Dog |
| Annotator B | Box around the cat (slightly larger) | Cat |

Suppose the IoU for matching pairs is:

| | B: Dog box | B: Cat box |
|---|---|---|
| **A: Dog box** | Labels match → IoU = **0.67** | Labels differ → **0.0** |
| **A: Cat box** | Labels differ → **0.0** | Labels match → IoU = **0.85** |

Best match for each box:

| Box | Best match | Score |
|-----|------------|-------|
| A: Dog box | B: Dog box | 0.67 |
| A: Cat box | B: Cat box | 0.85 |
| B: Dog box | A: Dog box | 0.67 |
| B: Cat box | A: Cat box | 0.85 |

**Final score** = (0.67 + 0.85 + 0.67 + 0.85) / 4 = **0.76**

#### How the threshold variant works

The base IoU metric returns a continuous score (0.76 in the example above). The **Intersection over Union (Threshold)** variant converts this into a binary match/no-match result.

For example, with a threshold of 0.5:
- Score 0.76 >= 0.5 → **1.0** (match)

With a threshold of 0.8:
- Score 0.76 < 0.8 → **0.0** (no match)


### IoU for other region types

The IoU concept applies to more than just rectangular bounding boxes. Label Studio adapts the same core idea for different annotation types:

- **Intersection over Union for Polygons** works the same way, but calculates the overlapping area between polygon shapes instead of rectangles. This is useful when annotators draw freeform outlines around irregular objects.
- **Brush Intersection over Union** compares pixel-level brush masks by counting how many pixels overlap versus the total pixels painted by either annotator. This is useful for segmentation tasks where annotators paint regions rather than drawing shapes.

In all cases, the overall process is the same: check that labels match, calculate the overlap ratio for the specific shape type, then use greedy matching to pair up regions and average the scores.

### Jaccard Similarity

Jaccard Similarity measures how much two annotators' selections overlap when they can each choose multiple items from a list. Unlike Exact Match, it gives partial credit when annotators agree on some selections but not all of them.

This metric is dynamically available for Choices tags with multi-select enabled.

#### What affects the score

| Scenario | Score |
|----------|-------|
| Annotators select the exact same items | 1.0 |
| Annotators share some but not all selections | Between 0.0 and 1.0, based on the overlap |
| Annotators select completely different items | 0.0 |
| Neither annotator selects anything | 1.0 (agreement that nothing applies) |
| One annotator selects items, the other selects nothing | 0.0 |

#### The formula

Jaccard Similarity treats each annotator's selections as a set and computes:

```
Jaccard Similarity = number of shared selections / total number of distinct selections
```

Or more formally:

```
J(A, B) = |A ∩ B| / |A ∪ B|
```

- **Intersection** (A ∩ B) is the set of items both annotators selected.
- **Union** (A ∪ B) is the set of all items selected by either annotator.

#### A simple example

A project asks annotators to tag articles with all relevant topics from a list: Sports, Politics, Technology, Health, Science.

Annotator A selects: **Sports, Politics, Technology**
Annotator B selects: **Politics, Technology, Health**

- Shared selections (intersection): Politics, Technology = **2 items**
- All distinct selections (union): Sports, Politics, Technology, Health = **4 items**
- Jaccard Similarity = 2 / 4 = **0.50**

#### How it compares to Exact Match

Using the same example above:

| Metric | Annotator A | Annotator B | Score |
|--------|-------------|-------------|-------|
| **Exact Match** | [Sports, Politics, Technology] | [Politics, Technology, Health] | **0.0** (not identical) |
| **Jaccard Similarity** | [Sports, Politics, Technology] | [Politics, Technology, Health] | **0.50** (2 out of 4 distinct items match) |

This is why Jaccard Similarity is useful for multi-select tasks — it recognizes that agreeing on 2 out of 4 topics is meaningfully different from agreeing on 0.

#### More examples

| Annotator A | Annotator B | Intersection | Union | Score |
|-------------|-------------|--------------|-------|-------|
| [Sports] | [Sports] | `1` | `1` | **1.0** |
| [Sports, Politics] | [Sports, Politics] | `2` | `2` | **1.0** |
| [Sports] | [Politics] | 0 | 2 | **0.0** |
| [Sports, Politics, Technology] | [Sports] | `1` | `3` | **0.33** |
| [Sports, Politics] | [Sports, Politics, Technology, Health] | `2` | `4` | **0.50** |

Note that **order does not matter**. `[Sports, Politics]` and `[Politics, Sports]` produce the same score because both are treated as sets.

#### How the threshold variant works

The base Jaccard Similarity metric returns a continuous score (like 0.50). The **Jaccard Similarity (Threshold)** variant converts this into a binary match/no-match result.

For example, with a threshold of 0.5:
- Score `0.50` >= `0.5` → **1.0** (match)

With a threshold of 0.75:
- Score `0.50` < `0.75` → **0.0** (no match)

The threshold variant is required when using the consensus methodology.



### Text Similarity

Text Similarity measures how closely two free-text annotations match at the surface level — that is, how similar the actual characters or words are. It is the default metric for TextArea tags.

#### The core idea

Text Similarity uses **edit distance** algorithms to calculate how many changes (insertions, deletions, substitutions) would be needed to transform one text into the other. Fewer changes means higher similarity.

By default, it uses the **Levenshtein** algorithm, which counts the minimum number of single-character edits needed. The raw edit distance is then normalized to a score between 0.0 and 1.0, where 1.0 means the texts are identical.

#### Examples

| Annotator A | Annotator B | Score | Why |
|-------------|-------------|-------|-----|
| "The cat sat on the mat" | "The cat sat on the mat" | **1.0** | Identical text |
| "The cat sat on the mat" | "The cat sat on the mat." | **~0.96** | One extra character (period) |
| "The cat sat on the mat" | "The cat sit on the mat" | **~0.96** | One character substitution (a→i) |
| "The cat sat on the mat" | "A dog lay on the rug" | **~0.43** | Many differences |
| "hello" | "world" | **~0.20** | Almost entirely different |

#### Configurable parameters

Text Similarity lets you adjust two settings:

**Algorithm** — the method used to compare strings:

| Algorithm | How it works | Good for |
|-----------|-------------|----------|
| Levenshtein (default) | Counts insertions, deletions, and substitutions | General-purpose text comparison |
| Damerau-Levenshtein | Like Levenshtein, but also counts transpositions (swapped adjacent characters) as a single edit | Text where typos often involve swapped letters |
| Jaro-Winkler | Weighs matching characters by position, with a bonus for shared prefixes | Short strings like names or codes |
| Jaro | Similar to Jaro-Winkler but without the prefix bonus | Short strings |
| Hamming | Counts positions where characters differ (strings must be the same length) | Fixed-length codes or identifiers |
| Ratcliff-Obershelp | Finds the longest common subsequences | Texts with rearranged sections |

**Granularity** — the level at which the text is split before comparison:

| Granularity | What it compares | Example |
|-------------|------------------|---------|
| Character (default) | Individual characters | "cat" → [c, a, t] |
| Bigram (2-gram) | Pairs of characters | "cat" → [ca, at] |
| Trigram (3-gram) | Triples of characters | "cats" → [cat, ats] |
| Word | Whole words | "the cat sat" → [the, cat, sat] |

Character-level comparison is the most fine-grained and catches small typos. Word-level comparison is more forgiving of minor spelling differences but stricter about missing or extra words.

#### Multiple text fields

When a TextArea tag allows multiple lines of text, each annotator's response is a list of strings. Text Similarity compares each line in order and averages the scores.

**Example**: Annotators are asked to transcribe three lines of text from an image.

| | Line 1 | Line 2 | Line 3 |
|---|--------|--------|--------|
| Annotator A | "John Smith" | "123 Main St" | "New York" |
| Annotator B | "John Smith" | "123 Main Street" | "New York" |

- Line 1: "John Smith" vs "John Smith" → **1.0**
- Line 2: "123 Main St" vs "123 Main Street" → **~0.81**
- Line 3: "New York" vs "New York" → **1.0**
- **Final score** = (1.0 + 0.81 + 1.0) / 3 = **0.94**

If one annotator writes more lines than the other, the extra lines are compared against nothing and score 0.0, pulling the average down.

#### How the threshold variant works

The base Text Similarity metric returns a continuous score. The **Text Similarity (Threshold)** variant converts this into a binary match/no-match result. The default threshold for text similarity is **0.85** (compared to 0.5 for most other metrics), reflecting the expectation that free-text annotations should be fairly close to count as agreement.

### Semantic Similarity

Semantic Similarity measures whether two text annotations convey the **same meaning**, regardless of how they are worded. Instead of comparing characters or words directly, it compares the underlying meaning using AI embeddings.

#### The core idea

Semantic Similarity works in three steps:

1. **Convert each text to an embedding** — A numerical representation (a vector) that captures the meaning of the text.
2. **Compute cosine similarity** between the two vectors — This measures how close the two meanings are on a scale from `0.0` to `1.0`.
3. **Compare against a threshold** — If the similarity meets the threshold (default: 0.85), the texts are considered a match.

#### When meaning matters more than wording

The key difference from Text Similarity is that Semantic Similarity understands that different words can express the same idea.

| Annotator A | Annotator B | Text Similarity | Semantic Similarity |
|-------------|-------------|-----------------|---------------------|
| "The cat is on the mat" | "The cat is on the mat" | **1.0** (identical) | **~1.0** (identical meaning) |
| "The cat is on the mat" | "A feline is sitting on the rug" | **~0.35** (very different words) | **~0.90** (very similar meaning) |
| "The patient has a fever" | "The patient's temperature is elevated" | **~0.42** (different words) | **~0.88** (same clinical meaning) |
| "The cat is on the mat" | "The stock market crashed today" | **~0.25** (different words) | **~0.10** (completely different meaning) |

Text Similarity would score the second and third rows low because the actual words are quite different. Semantic Similarity recognizes that the underlying meaning is essentially the same.

#### Configurable parameters

Semantic Similarity has one parameter:

- **Threshold** (default: 0.85) — the minimum cosine similarity for two texts to be considered a match. Lower values are more lenient; higher values require closer meaning.

### Choosing between Text Similarity and Semantic Similarity

| Consider | Text Similarity | Semantic Similarity |
|----------|-----------------|---------------------|
| **How it compares** | Character-by-character or word-by-word | Meaning and intent |
| **Best for** | Tasks where exact wording matters | Tasks where meaning matters more than wording |
| **Handles synonyms** | No — "couch" vs "sofa" scores low | Yes — "couch" vs "sofa" scores high |
| **Handles paraphrasing** | No — rephrased text scores low | Yes — same meaning scores high |
| **Catches typos** | Yes — small edits produce high scores | Yes — typos rarely change meaning |

**Use Text Similarity when:**
- Annotators are transcribing text (OCR, audio transcription)
- Annotators are entering structured data (names, addresses, codes)
- Exact wording is important to the task
- You want to detect and measure minor typos or formatting differences

**Use Semantic Similarity when:**
- Annotators are writing descriptions, summaries, or captions
- Multiple valid phrasings exist for the same answer
- You care about whether annotators understood the content the same way, not whether they used the same words
- Annotators may be working in slightly different styles or vocabularies

---

### Numeric Difference

Numeric Difference measures how close two numeric values are. Unlike Exact Match, which only cares whether two numbers are identical, Numeric Difference gives partial credit when values are close but not equal. It is the default metric for Number and Rating tags.

#### The formula

Numeric Difference converts the absolute difference between two values into a similarity score using this formula:

```
score = 1 / (1 + difference)
```

This produces a score between 0.0 and 1.0. Identical values score 1.0, and the score decreases smoothly as the difference increases.

#### Examples with a 1–5 rating scale

A project asks annotators to rate the quality of customer support responses on a scale from 1 to 5.

| Annotator A | Annotator B | Difference | Score | Interpretation |
|-------------|-------------|------------|-------|----------------|
| 4 | 4 | 0 | **1.0** | Perfect agreement |
| 4 | 5 | 1 | **0.50** | Close, one step apart |
| 4 | 3 | 1 | **0.50** | Close, one step apart |
| 5 | 3 | 2 | **0.33** | Moderate disagreement |
| 5 | 1 | 4 | **0.20** | Strong disagreement |

#### Examples with continuous numeric values

A project asks annotators to estimate the age of a person in a photo.

| Annotator A | Annotator B | Difference | Score |
|-------------|-------------|------------|-------|
| 30 | 30 | 0 | **1.0** |
| 30 | 32 | 2 | **0.33** |
| 30 | 35 | 5 | **0.17** |
| 30 | 50 | 20 | **0.05** |

Notice that the score drops quickly for larger differences. A difference of 1 already halves the score, and a difference of 5 brings it down to 0.17. This makes the base metric most useful for detecting whether annotators are in close agreement.

#### How the threshold variant works

The threshold variant, **Numeric Difference (Threshold)**, works differently from other threshold variants. Instead of binarizing the similarity score, it uses a **maximum difference** parameter (default: 1.0). If the absolute difference between two values is within this tolerance, the score is 1.0; otherwise it's 0.0.

**Example**: With a max difference of `1.0` on a 1–5 rating scale:

| Annotator A | Annotator B | Difference | Within tolerance? | Score |
|-------------|-------------|------------|-------------------|-------|
| 4 | 4 | 0 | Yes (`0 <= 1.0`) | **1.0** |
| 4 | 5 | 1 | Yes (`1 <= 1.0`) | **1.0** |
| 4 | 2 | 2 | No (`2 > 1.0`) | **0.0** |

This is often more intuitive for rating scales: "Annotators agree if they're within 1 point of each other."

### Taxonomy metrics

Label Studio offers two alternative metrics for `Taxonomy` tags that provide partial credit, unlike Exact Match which requires the full taxonomy selection to be identical. Both are useful when you want to recognize that two annotators who selected nearby items in a taxonomy are in closer agreement than two who selected completely unrelated items.

#### Understanding taxonomy paths

A taxonomy is a tree structure. When an annotator makes a selection, it is recorded as a **path** from the root to the selected node. For example, in this taxonomy:

```
Animals
├── Dogs
│   ├── Labrador
│   └── Poodle
├── Cats
│   ├── Siamese
│   └── Persian
└── Birds
    ├── Eagle
    └── Sparrow
```

Selecting "Labrador" produces the path: **Animals > Dogs > Labrador**
Selecting "Siamese" produces the path: **Animals > Cats > Siamese**

Annotators can also select multiple paths (e.g., both "Labrador" and "Eagle").

#### Common Labels Matches

Common Labels Matches compares taxonomy selections by treating each complete path as an item, then computing **Jaccard Similarity over the set of paths**. A path either matches exactly or it doesn't — there is no partial credit within a single path.

**Example**: An annotator can select multiple species from the taxonomy above.

| Annotator A | Annotator B | Shared paths | All distinct paths | Score |
|-------------|-------------|--------------|--------------------|-------|
| [Labrador] | [Labrador] | 1 | 1 | **1.0** |
| [Labrador, Eagle] | [Labrador, Eagle] | 2 | 2 | **1.0** |
| [Labrador, Eagle] | [Labrador, Sparrow] | 1 (Labrador) | 3 (Labrador, Eagle, Sparrow) | **0.33** |
| [Labrador] | [Poodle] | 0 | 2 | **0.0** |

Note that in the last row, Labrador and Poodle are both dogs, but Common Labels Matches treats them as completely different paths. If you want credit for shared ancestry, use Common Subtree Matches instead.

#### Common Subtree Matches

Common Subtree Matches gives partial credit for shared ancestry in the taxonomy tree. Before comparing, it **expands each selected path into all of its ancestor prefixes**. Then it computes Jaccard Similarity over these expanded sets.

**How expansion works**: The path "Animals > Dogs > Labrador" expands into:
- Animals
- Animals > Dogs
- Animals > Dogs > Labrador

This means that even if two annotators selected different leaf nodes, they'll still get credit for agreeing on the parent categories.

**Example**: Comparing "Labrador" vs. "Poodle":

| | Expanded nodes |
|---|---------------|
| Annotator A: **Labrador** | {Animals}, {Animals > Dogs}, {Animals > Dogs > Labrador} |
| Annotator B: **Poodle** | {Animals}, {Animals > Dogs}, {Animals > Dogs > Poodle} |

- Shared nodes (intersection): {Animals}, {Animals > Dogs} = **2 nodes**
- All distinct nodes (union): {Animals}, {Animals > Dogs}, {Animals > Dogs > Labrador}, {Animals > Dogs > Poodle} = **4 nodes**
- Score = 2 / 4 = **0.50**

Compare this to "Labrador" vs. "Siamese":

| | Expanded nodes |
|---|---------------|
| Annotator A: **Labrador** | {Animals}, {Animals > Dogs}, {Animals > Dogs > Labrador} |
| Annotator B: **Siamese** | {Animals}, {Animals > Cats}, {Animals > Cats > Siamese} |

- Shared nodes: {Animals} = **1 node**
- All distinct nodes: {Animals}, {Animals > Dogs}, {Animals > Dogs > Labrador}, {Animals > Cats}, {Animals > Cats > Siamese} = **5 nodes**
- Score = 1 / 5 = **0.20**

And "Labrador" vs. "Eagle":

| | Expanded nodes |
|---|---------------|
| Annotator A: **Labrador** | {Animals}, {Animals > Dogs}, {Animals > Dogs > Labrador} |
| Annotator B: **Eagle** | {Animals}, {Animals > Birds}, {Animals > Birds > Eagle} |

- Shared nodes: {Animals} = **1 node**
- All distinct nodes: 5 nodes
- Score = 1 / 5 = **0.20**

The scores reflect how closely related the selections are in the taxonomy tree. Two dog breeds (0.50) score higher than a dog and a cat (0.20), which makes intuitive sense.

#### Comparing all three taxonomy metrics

Using the taxonomy above, here's how the three available metrics score the same pairs:

| Annotator A | Annotator B | Exact Match | Common Labels Matches | Common Subtree Matches |
|-------------|-------------|-------------|----------------------|----------------------|
| Labrador | Labrador | **1.0** | **1.0** | **1.0** |
| Labrador | Poodle | **0.0** | **0.0** | **0.50** |
| Labrador | Siamese | **0.0** | **0.0** | **0.20** |
| Labrador | Eagle | **0.0** | **0.0** | **0.20** |
| [Labrador, Eagle] | [Labrador, Sparrow] | **0.0** | **0.33** | **0.67** |

- **Exact Match** is strictest — any difference at all scores 0.0.
- **Common Labels Matches** gives credit when some of the selected paths match exactly, but not for shared ancestry.
- **Common Subtree Matches** is most lenient — it recognizes that selecting two items in the same branch of the taxonomy represents partial agreement.

---

### Source Guide Api (`docs/source/guide/api.md`)

---
title: API Reference for Label Studio
short: API
type: guide
tier: all
order: 401
order_enterprise: 401
meta_title: API Endpoints
meta_description: API documentation for authenticating, listing data science projects, importing predictions and raw data and exporting annotated data, and user management.
section: "Integrate & Extend"

---

You can use the Label Studio API to import data for labeling, export annotations, set up machine learning with Label Studio, and sync tasks with cloud storage. 

See the [API reference documentation](https://api.labelstud.io/api-reference/introduction/getting-started) for further guidance and interactive examples. If you want to write Python scripts using the API, use the [Label Studio Python SDK](sdk.html). 

!!! info Tip
    For additional guidance on using our API, see [5 Tips and Tricks for Label Studio’s API and SDK](https://labelstud.io/blog/5-tips-and-tricks-for-label-studio-s-api-and-sdk/).


### Authenticate to the API

You must retrieve your access token so that you can authenticate to the API. 

Whether you can create a token, and which types of tokens you can create, depends on your org settings.

!!! note "API keys vs. Access tokens"
    In Label Studio, **"access tokens"** and **"API keys"** mean the same thing and are used interchangeably.  

There are two types of access tokens/API keys: **Personal Access Tokens (PATs)** and **Legacy Tokens**. 

You can read more about the differences between them here: [Access Tokens](access_tokens). 

#### Find your access token

1. Open Label Studio and click your user icon in the upper right. Select **Account & Settings**. 
2. Select **Personal Access Token** or **Legacy Token** on the left. 

Depending on which one you select, you will need to either generate a new token or copy the one that is displayed. 


#### Authenticate HTTP API requests with a personal access token

PATs use `'Authorization: Bearer <token>'` when used with HTTP API requests, for example:

```bash
curl -X <method> <Label Studio URL>/api/<endpoint> -H 'Authorization: Bearer <token>'
```

#### Authenticate HTTP API requests with a legacy token

Legacy tokens use `'Authorization: Token <token>'` when used with HTTP API requests, for example:

```bash
curl -X <method> <Label Studio URL>/api/<endpoint> -H 'Authorization: Token <token>
```

#### Authenticate Python SDK requests

When used with the SDK, you do not need to distinguish between legacy tokens or personal access tokens. 

```python
# Define the URL where Label Studio is accessible
LABEL_STUDIO_URL = 'YOUR_BASE_URL'

# API key can be either your PAT or legacy access token
LABEL_STUDIO_API_KEY = 'YOUR_API_KEY'

# Import the SDK and the client module
from label_studio_sdk import LabelStudio
client = LabelStudio(base_url=LABEL_STUDIO_URL, api_key=LABEL_STUDIO_API_KEY)
```

See [API documentation for authentication](https://api.labelstud.io/api-reference/introduction/getting-started#authentication).

### List all projects

To perform most tasks with the Label Studio API, you must specify the project ID, sometimes referred to as the `pk`, or primary key. If you don't know what your project ID is, you might want to get a list of all projects in Label Studio that you can access. See the [List your projects API endpoint documentation](https://api.labelstud.io/api-reference/api-reference/projects/list).

### Create and set up a project

Create a project and set up the labeling interface in Label Studio using the API. See the [Create new project API endpoint documentation](https://api.labelstud.io/api-reference/api-reference/projects/create).

If you want to make sure the configuration for your labeling interface is valid before submitting it using the API, you can use the [validate label config](https://api.labelstud.io/api-reference/api-reference/projects/validate-config) API endpoint.

### Import tasks using the API

To import tasks using the API, make sure you know the project ID that you want to add tasks to. See additional examples and parameter descriptions in the [import data endpoint documentation](https://api.labelstud.io/api-reference/api-reference/tasks/create)

### Retrieve tasks
Retrieve a paginated list of tasks for a specific project. If you want, you can also retrieve tasks and annotations using this API endpoint, as an alternative to exporting annotations. See details and parameters in the [list project tasks endpoint documentation](https://api.labelstud.io/api-reference/api-reference/tasks/list).

### Export annotations

To export annotations, first see [which formats are available to export for your project](https://api.labelstud.io/api-reference/api-reference/projects/exports/list-formats). 

Choose your selected format from the response and then call the export endpoint. See the [export annotations](https://api.labelstud.io/api-reference/api-reference/projects/exports/download-sync) endpoint documentation for more details.

---

### Source Guide Ask Ai (`docs/source/guide/ask_ai.md`)

---
title: AI Assistant
short: AI Assistant
tier: enterprise
type: guide
order: 0
order_enterprise: 356
meta_title: About AI Assistant 
meta_description: Information about using the AI features in Label Studio
section: "Manage Your Organization"
date: 2025-01-28 16:40:16
---

The Label Studio AI Assistant is an OpenAI LLM that has been trained on the Label Studio documentation, codebase, and several other Label Studio resources.

## Use AI Assistant for project setup 

Instead of manually building labeling interfaces or project instructions from scratch, you can prompt the AI with a description of what your labeling project needs and what you want annotators to be able to select. 

You can interact with this as you would a chat, meaning that you can build off your previous instructions and request changes. 




## Use AI Assistant for general help

You can also use AI Assistant to ask for troubleshooting and general help with Label Studio.  

  


## AI Assistant FAQ

### How do I enable or disable AI Assistant?

Your organization can enable AI Assistant from the **Organization > Usage & License** page. 

!!! note
    Only users in the Owner role can modify these settings. For Starter Cloud users, [open a support ticket](mailto:support@humansignal.com) to request disablement. 

 

* **Enable AI Features**--Enable AI Assistant for your labeling interface configuration. 
* **Enable Ask AI**--Enable the AI Assistant for general Label Studio help. 

### What models do you use?

We use OpenAI Tier 5 models, primarily gpt-4o and o3-mini. 

### Do all your AI features use the same backend?

Yes. 

### What data is used to generate answers?

Only the data you provide in the chat window is used to generate responses, this includes:

* All chat messages
* Task samples you have uploaded
* The current labeling config

### How is my data used to train models?

The model is trained on public resources like our documentation, codebase, blog, and website. We also train it on additional examples we've written specifically for the model. 

We are on a Tier 5 OpenAI account, and opt out of all requests for training data.

We track requests to our AI to use as quality control, but they are only used to test the outputs of the model.  For example: If Company X asks "Make a project to label opossums," we may notice that our AI did not provide a satisfactory answer and we may create configurations with opossums in them and train the model on them. The model would not be trained on any of Company X's questions or data.

All tracked data is covered by our industry-leading [security and privacy policies](https://humansignal.com/security/).

### Is it HIPAA compliant?

Your Business Associate Agreement (BAA) does not cover these features. If you are required to comply with HIPAA, we recommend that you disable Ask AI.

---

### Source Guide Auth Ldap (`docs/source/guide/auth_ldap.md`)

---
title: Set up LDAP authentication for Label Studio
short: LDAP
tier: enterprise
type: guide
order: 0
order_enterprise: 385
meta_title: LDAP authentication for Label Studio Enterprise
meta_description: Label Studio Enterprise documentation for setting up LDAP authentication for your data labeling, machine learning, and data science projects.
section: "Manage Your Organization"
parent: "admin_auth"
parent_enterprise: "admin_auth"
date: 2024-05-24 09:08:58
---

Set up LDAP authentication to manage access to Label Studio.

!!! error Enterprise
    LDAP authentication is only available in Label Studio Enterprise Edition and is only available for on-prem installations. If you're using Label Studio Community Edition, see <a href="https://labelstud.io/guide/label_studio_compare.html">Label Studio Features</a> to learn more.


To more easily [manage access to Label Studio Enterprise](manage_users.html), you can map LDAP groups to both `roles` or `workspaces`. 

## Set up LDAP authentication 

After you set up LDAP authentication, you can no longer use native authentication to log in to the Label Studio UI. Set up LDAP authentication and assign LDAP users to your Label Studio Enterprise organization using environment variables in Docker. 

You can also map specific LDAP groups to specific organization roles and workspaces in Label Studio Enterprise, making it easier to set up and manage role-based access control (RBAC) and project access in Label Studio Enterprise. 

You can refer to this example environment variable file for your own LDAP setup:

```bash 
AUTH_LDAP_ENABLED=1

# Use ldaps to secure the LDAP connection
AUTH_LDAP_SERVER_URI=ldaps://ldap.example.com
# LDAP admin credentials    
AUTH_LDAP_BIND_DN=uid=user,ou=sysadmins,o=123abc,dc=zexample,dc=com
AUTH_LDAP_BIND_PASSWORD=password123

# Allow users to use usernames (not only emails) to log into Label Studio
USE_USERNAME_FOR_LOGIN=1

# Simple user search in LDAP groups
AUTH_LDAP_USER_DN_TEMPLATE=uid=%(user)s,ou=Users,o=123abc,dc=example,dc=com

# Specify organization to assign it to all users on the platform
# Warning: the user with this email must be registered before any LDAP users log in 
AUTH_LDAP_ORGANIZATION_OWNER_EMAIL=heartex@heartex.net

# Populate the user from the LDAP directory:
# firstName, lastName, mail, sAMAccountName are taken from your LDAP record 
AUTH_LDAP_USER_ATTR_MAP_FIRST_NAME=firstName
AUTH_LDAP_USER_ATTR_MAP_LAST_NAME=lastName
# Specify the field to use for AUTH_LDAP_USER_QUERY_FIELD as 'email'
AUTH_LDAP_USER_ATTR_MAP_EMAIL=mail
# Specify the field to use for AUTH_LDAP_USER_QUERY_FIELD as 'username' 
AUTH_LDAP_USER_ATTR_MAP_USERNAME=sAMAccountName

# Query the authenticating user in Label Studio, it can be [email|username]
AUTH_LDAP_USER_QUERY_FIELD=email

# Group parameters
AUTH_LDAP_GROUP_SEARCH_BASE_DN=ou=Users,o=group-id,dc=example,dc=com
AUTH_LDAP_GROUP_SEARCH_FILTER_STR=(objectClass=groupOfNames)
AUTH_LDAP_GROUP_TYPE=ou
```

If you want to use a recursive scan to search in several LDAP groups to grant access to Label Studio, instead of relying on the simple search used by `AUTH_LDAP_USER_ON_TEMPLATE`, update your environment variables file like the following:
```bash
# Leave this parameter empty
AUTH_LDAP_USER_DN_TEMPLATE=""
# Specify the groups that you want to be able to log in, separated by ';' 
AUTH_LDAP_USER_SEARCH_BASES="ou=guests,dc=domain,dc=com;ou=owners,dc=domain,dc=com"
```


## User Role Mappings

Map LDAP groups to specific Label Studio Enterprise roles using `;` to specify several groups.

```bash
AUTH_LDAP_ORGANIZATION_ROLE_ADMINISTRATOR=cn=admins,ou=users,o=123abc,dc=example,dc=com 
AUTH_LDAP_ORGANIZATION_ROLE_MANAGER=cn=managers,ou=users,o=123abc,dc=example,dc=com 
AUTH_LDAP_ORGANIZATION_ROLE_REVIEWER=cn=reviewers,ou=users,o=123abc,dc=example,dc=com
AUTH_LDAP_ORGANIZATION_ROLE_ANNOTATOR=cn=annotators,ou=users,o=123abc,dc=example,dc=com;cn=guests,ou=users,o=123abc,dc=example,dc=com
AUTH_LDAP_ORGANIZATION_ROLE_NOT_ACTIVATED=cn=not,ou=users,o=123abc,dc=example,dc=com 
AUTH_LDAP_ORGANIZATION_ROLE_DEACTIVATED=cn=deactivated,ou=users,o=123abc,dc=example,dc=com
```


## Workspace Mappings

Map LDAP groups to specific Label Studio workspaces. Use a JSON format where keys are workspace titles and values are LDAP groups. Split groups with `;` to specify several groups. 

```bash
AUTH_LDAP_ORGANIZATION_WORKSPACES='{"Workspace 1":"cn=team1,ou=users,o=60cbc901ec2e8e387a3b2d3e,dc=jumpcloud,dc=com","Workspace 2":"cn=team2,ou=users,o=60cbc901ec2e8e387a3b2d3e,dc=jumpcloud,dc=com"}'
```

## Manage user access only with LDAP 

If you want to manage Label Studio roles and workspaces entirely with LDAP, add the following to your environment variable file:

```
MANUAL_PROJECT_MEMBER_MANAGEMENT=0
MANUAL_WORKSPACE_MANAGEMENT=0
MANUAL_ROLE_MANAGEMENT=0
```

Setting these options disables the Label Studio API and UI options to assign roles and workspaces for specific users within Label Studio and relies entirely on the settings in the environment variable file.


## Enabling TLS on LDAP/Active Directory

To enable a secure Transport Layer Security (TLS) connection on LDAP/Active Directory, add the following env variables:

```bash
AUTH_LDAP_CONNECTION_OPTIONS="OPT_X_TLS_CACERTFILE=/path/to/cert.crt;OPT_X_TLS_NEWCTX=0"
```

Where `OPT_X_TLS_CACERTFILE` points to a file with certificates. If you’re using self-signed certificates, you’ll need to add `OPT_X_TLS_NEWCTX=0` as the last entry of the `AUTH_LDAP_CONNECTION_OPTIONS` env variable.

---

### Source Guide Auth Setup (`docs/source/guide/auth_setup.md`)

---
title: Set up SSO authentication for Label Studio
short: SSO and SAML
tier: enterprise
type: guide
order: 0
order_enterprise: 387
meta_title: SSO authentication for Label Studio Enterprise
meta_description: Label Studio Enterprise documentation for setting up SSO authentication for your data labeling, machine learning, and data science projects.
section: "Manage Your Organization"
parent: "admin_auth"
parent_enterprise: "admin_auth"
---

Set up single sign-on using SAML to manage access to Label Studio using your existing Identity Provider (IdP).

!!! error Enterprise
    SSO authentication is only available in Label Studio Enterprise Edition. If you're using Label Studio Community Edition, see <a href="https://labelstud.io/guide/label_studio_compare.html">Label Studio Features</a> to learn more.


To more easily [manage access to Label Studio Enterprise](manage_users.html), you can map SAML groups to both `roles` or `workspaces`. 

## Set up SAML SSO

The organization Owner or Administrator for Label Studio Enterprise can set up SSO & SAML for the instance. Label Studio Enterprise supports the following IdPs:
- [Okta](https://www.youtube.com/watch?v=Dr-_hyWIw4M)
- [Google SAML](google_saml.html)
- [Ping Federate and Ping Identity SAML SSO Setup Example](pingone.html)
- OneLogin
- Microsoft Entra ID (formerly Azure Active Directory, Azure AD)
- Auth0
- Others that use SAML assertions

After setting up the SSO, you can use native authentication to access the Label Studio UI, however it's not a recommended option especially for the user with the Owner role.

- You can use SSO along with normal login. This is not a recommended option.

- You can prevent a user from creating his own organization by using [DISABLE_SIGNUP_WITHOUT_LINK](admin_user#Require-invites-for-new-users) option.

### Connect your Identity Provider to Label Studio Enterprise

Set up Label Studio Enterprise as a Service Provider (SP) with your Identity Provider (IdP) to use SAML authentication. 

The details will vary depending on your IdP, but in general you will complete the following steps:

#### From Label Studio:

1. Go to the **Organization** page. 
    
    If you do not see the option to select **Organization**, you are not logged in with the appropriate role. 
2. Select **SSO & SAML** in the upper right. 
3. In the **Organization** field, ensure the domain matches the domain used for your organization in your IdP.
4. Copy the following URLs:
    
    * **Assertion Consumer Service (ACS) URL with Audience (EntityID), and Recipient (Reply) details**---The IdP uses this URL to redirect users to after a successful authentication.
    * **Login URL**---This is the URL that users will use to log in to Label Studio. 
    * **Logout URL**---This is the URL used to redirect users after successfully logging out of Label Studio.

#### From your IdP:

1. Paste the URLs copied from Label Studio in the appropriate location. 
2. Generate a metadata XML file, or a URL that specifies the metadata for the IdP.
3. Set up or confirm setup of the following SAML attributes. Label Studio Enterprise expects specific attribute mappings for user identities.

**The default attribute names are:**

| Data | Default Attribute |
| --- | --- |
| Email address | Email |
| First or given name | FirstName |
| Last or family name | LastName |
| Group name | Groups | 

!!! note Note
    Different Identity Providers use different attribute names. Label Studio provides **presets** in the SSO & SAML settings page to quickly configure the correct attribute mappings for popular IdPs. You can also manually configure custom attribute names if your IdP uses different values.

**Attribute presets by IdP:**

| IdP | Email | FirstName | LastName | Groups |
| --- | --- | --- | --- | --- |
| Default | `Email` | `FirstName` | `LastName` | `Groups` |
| Auth0 | `email` | `given_name` | `family_name` | `groups` |
| Entra ID (short) | `emailAddress` | `givenName` | `surname` | `groups` |
| Google | `Email` | `FirstName` | `LastName` | `Groups` |
| PingOne | `emailAddress` | `givenName` | `surname` | `Groups` |
| Okta | `email` | `firstName` | `lastName` | `groups` |

**Microsoft Entra ID with full URI format:**

If your Entra ID is configured with default claim URIs, use:

| Attribute | URI |
| --- | --- |
| Email | `http://schemas.xmlsoap.org/ws/2005/05/identity/claims/emailaddress` |
| FirstName | `http://schemas.xmlsoap.org/ws/2005/05/identity/claims/givenname` |
| LastName | `http://schemas.xmlsoap.org/ws/2005/05/identity/claims/surname` |
| Groups | `http://schemas.microsoft.com/ws/2008/06/identity/claims/groups` |



#### From Label Studio:

1. Return to the SSO & SAML page. 
2. Upload the metadata XML file or specify the metadata URL.  
3. Set up group mappings. These can also be added or edited later.

    Ensure the group name you enter is the same as the group name sent as an attribute in a SAML authentication response by your IdP.

    * **Organization Roles to Groups Mapping**---Map groups to roles at the organization level. The role set at the organization level is the default role of the user and is automatically assigned to workspaces and projects. For more information on roles, see [Roles in Label Studio Enterprise](manage_users#Roles-in-Label-Studio-Enterprise).
    
        You can map multiple groups to the same role. Note that users who are **Not Activated** or **Deactivated** do not count towards the seat limit for your account. 
    * **Workspaces to Groups Mapping**---Add groups as members to workspaces. Users with Manager, Reviewer, or Annotator roles can only see workspaces after they've been added as a member to that workspace.
    
        Select an existing workspace or create a new one. You can map multiple groups to the same workspace. 
    * **Projects to Groups Mapping**---Map groups to roles at the project level. Project-level roles can be **Annotator**, **Reviewer**, or **Inherit**. 
    
        You can map a group to different roles across multiple projects. You can also map multiple groups to the same roles and the same projects. For more information on roles, see [Roles in Label Studio Enterprise](manage_users#Roles-in-Label-Studio-Enterprise). 
    
        If you select **Inherit**, the group will inherit the role set above under **Organization Roles to Groups Mapping.** If the group is inheriting the Not Activated role, the users are mapped to the project, but they are not actually assigned to the project until the group is synced (meaning that the user authenticates with SSO). 
4. Click **Save**.

5. Test the configuration by logging in to Label Studio Enterprise with your SSO account.


### Setup SAML SSO with Okta video tutorial

<iframe class="video-border" width="560" height="315" src="https://www.youtube.com/embed/Dr-_hyWIw4M" width="100%" height="400vh" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>


## Manage user access only with SSO 

If you want to manage Label Studio roles and workspaces entirely with single sign-on (SSO), add the following to your environment variable file:

```
MANUAL_PROJECT_MEMBER_MANAGEMENT=0
MANUAL_WORKSPACE_MANAGEMENT=0
MANUAL_ROLE_MANAGEMENT=0
```

Setting these options disables the Label Studio API and UI options to assign roles and workspaces for specific users within Label Studio and relies entirely on the settings in the environment variable file.

!!! info Tip
    If you are using the SaaS version of Label Studio (Label Studio Enterprise Cloud) and would like to enable these restrictions for your organization, [open a ticket](https://support.humansignal.com/hc/en-us/requests/new) to submit your request.  
    
    If requested, we can also disable the common login option for your organization. When disabled, users can only use the SSO login fields and the common login  option is disabled completely. 

### Login page URL

You can also set the `LOGIN_PAGE_URL` environment variable to redirect the login page to the specified URL.

---

### Source Guide Backup Enterprise (`docs/source/guide/backup_enterprise.md`)

---
title: Create backups for on-prem Label Studio Enterprise
short: Back up Label Studio
tier: enterprise
type: guide
order: 0
order_enterprise: 78
meta_title: Back up Label Studio Enterprise on-prem
meta_description: How to back up your on-prem Label Studio Enterprise environment
section: "Install"
parent_enterprise: "install_enterprise"
date: 2024-01-22 15:54:30
---

Backing up your Label Studio Enterprise on-prem installation involves more than just a database backup, although that is a critical component. Here's an overview of the steps you should take to back up your Label Studio Enterprise on-premises installation:

1. **Back up your databases (development and production).** This is essential as it contains all your project data and configurations. If you're using PostgreSQL, you can use tools like **pg_dump** to create backups of your databases.
   
2. **Back up your file storage.** If you're using local file storage or have files stored in a cloud storage that's synced with Label Studio, ensure you back up these files. This includes any media files (images, audio, video) that you've uploaded for labeling.

3. **Back up your configuration files that have environment variables.** This includes your `label-studio` configuration file (which might contain your database connection settings and other configurations) and any custom scripts or machine learning models that you've integrated with Label Studio.

4. **Back up your environment variables.** If you're using environment variables to configure Label Studio (which might include database credentials, secret keys, etc.), make sure to back these up as well.

5. **Back up your Docker volumes.** If you're running Label Studio in Docker containers and you have data volumes, back up these volumes. Docker volumes may contain your database data, media uploads, and other persistent data.

6. **Document your current setup.** Make sure to document the versions of Label Studio and any other dependencies you're using, as well as any specific configurations or customizations you've made. This will be helpful if you need to restore from a backup.

7. **Regularly test your backups.** Ensure that your backup process is working by regularly testing that you can restore your Label Studio instance from the backups.


Remember to store your backups in a secure location, ideally in a different physical location than your production environment to protect against site-wide issues such as natural disasters or power outages. It's also a good practice to encrypt sensitive data in your backups to prevent unauthorized access.

---

### Source Guide Command Palette (`docs/source/guide/command_palette.md`)

---
title: Command palette
short: Command Palette
tier: enterprise
type: guide
order: 0
order_enterprise: 356
meta_title: Enhanced search with the command palette
meta_description: How to use the Label Studio  command palette
section: "Manage Your Organization"
date: 2025-01-28 16:40:16
---

The command palette is an enhanced search and navigation tool. 

To open it, click the search icon in the upper right or press `Ctrl+K` (`Cmd+K` on Mac)



## What you can do



From here you can:

- Search across your Label Studio org. For example, you can search for projects and workspaces.
- Navigate to pages within the app. 
- Find quick links to Label Studio documentation and resources, such as the API docs and the changelog. 
- When you are in a project, you can search for settings pages or individual settings to quickly jump to them. 

## Role-aware results 

Results respect your role and access. For example, Annotators and Reviewers won’t see workspaces or projects that they can’t access.

## AI assistant

If AI assistant is enabled, you can enter a query and then press Tab. The assistant opens automatically with your query.

## Keyboard shortcuts

- `Ctrl+K` or `Cmd+K` to open the command palette 
- Use the `Up` and `Down` arrow keys to navigate through the results
- `Enter` to select an option
- `Esc` to close
- Tab to open the AI assistant

---

### Source Guide Comments Notifications (`docs/source/guide/comments_notifications.md`)

---
title: Comments and notifications
short: Comments and notifications
type: guide
tier: enterprise
section: "Review & Measure Quality"
order: 0
order_enterprise: 306
meta_title: Comment in Label Studio Enterprise
meta_description: The comments and notifications features define how annotators, reviewers and administrators communicate and receive updates on projects and tasks.
---


Comments are a Label Studio Enterprise feature that allows communication between reviewers and annotators:

<div class="noheader rowheader">

<table>
<thead>
    <tr>
      <th>Annotators</th>
      <th>Reviewers</th>
    </tr>
</thead>
<tr>
<td>

**Annotator to reviewer communication** 

- Ask for clarification on a task
- Provide reasoning or context for their annotation
  
</td>
<td>

**Reviewer to annotator communication** 

- Answer questions
- Give feedback to help the annotator improve 

</td>
</tr>
</table>

</div>

 This annotator-reviewer feedback loop is an important tool in your review workflow, allowing you to:

- Speed up the labeling process
- Increase the annotation quality
- Build more solid labeling and review processes




## Permissions

All users have the following commenting permissions for any annotations that they would otherwise have access to view:

- View the **Comments** panel and read previous comments. 
- Add new comments. 

Additional permissions are as follows:


| Action          | Annotator | Reviewer and higher roles|
|-----------------|:---------:|:--------:|
| Add comment     | true      | true     |
| View comment    | true      | true     |
| Resolve/unresolve own comment | true     | true     |
| Resolve/unresolve other comments | false     | true     |
| Delete own comment | true     | true     |
| Delete other comments | false     | false     |
| Link/unlink own comment | true     | true     |
| Link/unlink other comments | false     | false     |


## Add a comment

You can add comments from the **Comments** panel when viewing an annotation. All users have access to the **Comments** panel. 



### Link comments to regions and fields

You can link your comment to a specific area within an annotation. 

This can be a region (for example, a bounding box or a span of text) or it can be a field (for example, a classification).



Note the following:

* You can link comments to fields within the labeling configuration. However, if the field is a classification, choice, or taxonomy, you can only attach a comment if there is already a selection. 
* You can only link comments to an entire text area. You cannot link comments to regions within text areas. 
* Linking is not supported for [Ranker](/tags/ranker) or [Pairwise](/tags/pairwise) fields. 

## Comment actions

The actions you can perform depend on [your user role permissions](#Permissions) and whether it's your comment. You can access comment actions from the overflow menu:




## Filter for comments

The Data Manager has several columns to display comment information:

| Column          | Description | View permissions |
|-----------------|:---------:|:--------:|
| **Comment texts**    | The text in the comment.      | All user roles   |
| **Comments**    | Total number of comments for a particular task.      | All user roles      |
| **Unresolved comments** | Total number of comments that are not yet resolved.      | All user roles      |
| **Commented at** | The date and time at which the comment was made.      | All user roles except Annotator     |
| **Commented authors** | User who wrote comments.     | All user roles except Annotator     |

!!! note
    Users in the Reviewer or Annotator role must be granted access to view the Data Manager. You can grant access in the [project settings](project_settings_lse). 


## Require comments

You can configure the project so that annotators and reviewers are required to leave comments after taking specific actions:

* [**Annotation > Annotating Options > Annotators must leave a comment on skip**](project_settings_lse#Annotation)
* [**Review > Reject Options > Reviewers must leave a comment on reject**](project_settings_lse#reject-options)

## Comment notifications

To see notifications, click your user profile icon in the upper right. 

Use the overflow menu to mark notifications as read or unread.





!!! note
    Notifications that have already been read are denoted with a gray background. 

### Annotators

Annotators can write comments on their own annotations. 

**Sent notifications**

When an annotator writes a comment, the users who are notified depend on whether it is the first comment on the annotation:

* If this is the first comment on an annotation, then all users within a project that have the role of Reviewer or higher see the comment in their [notification list](#Notifications). 
* If this comment is in response to an ongoing discussion within the comments panel, then only users who are have added comments to the annotation will see a notification. 

**Received notifications** 

An annotator receives a notification any time another user leaves a comment on one of their annotations. 

### Reviewers and other users

Reviewers can write comments on annotations that they are reviewing. This applies to users with the Reviewer, Manager, Admin, and Owner role. 

**Sent notifications**

When a reviewer writes a comment, then the users who are notified depend on their role and whether they have also left comments: 

* Annotators get a notification for all comments on the annotations they created. 
* If other users have also left comments on the annotation (for example, if multiple users are reviewing the annotation), then they will also be notified when a new comment appears. 

**Received notifications** 

A reviewer receives a notification when:

* If the comment is the first comment on an annotation, then all users within a project that have the role of Reviewer or higher see the comment in their [notification list](#Comment-notifications). 
* If you have already left a comment on an annotation, you receive a notification when any user, regardless of role, adds a new comment.

---

### Source Guide Custom Metric (`docs/source/guide/custom_metric.md`)

---
title: Add a custom agreement metric to Label Studio
short: Custom metrics
tier: enterprise
type: guide
order: 0
order_enterprise: 310
meta_title: Add a Custom Agreement Metric for Labeling
meta_description: Label Studio Enterprise documentation about how to add a custom agreement metric to use for assessing annotator agreement or the quality of your annotation and prediction results for data labeling and machine learning projects.
section: "Review & Measure Quality"
parent: "stats"
parent_enterprise: "stats"
---

Write a custom agreement metric to assess the quality of the predictions and annotations in your Label Studio Enterprise project. 

Label Studio Enterprise contains a variety of [agreement metrics for your project](agreement_metrics), but if you want to evaluate annotations using a custom metric or a standard metric not available in Label Studio, you can write your own. 

!!! note
    This functionality is available out-of-the-box for Label Studio Enterprise Cloud users. 
    
    For Label Studio Enterprise on-prem environments, you must configure your cloud provider to allow Label Studio to deploy and invoke serverless functions. Label Studio supports [AWS Lambda](#AWS-Lambda) and [Google Cloud Functions](#Google-Cloud-Functions). For more information, see [On-prem deployments](#On-prem-deployments).   


Label Studio Enterprise includes various annotation and labeling statistics and the ability to add your own. The open source Community Edition of Label Studio does not contain these calculations. If you're using Label Studio Community Edition, see <a href="https://labelstud.io/guide/label_studio_compare.html">Label Studio Features</a> to learn more.

## Add your custom agreement metric to Label Studio

Custom agreement metrics are added at the project level under **Settings > Quality > Agreement > Custom Agreement Metric Code**.

!!! note 
    You must configure the labeling interface before you can add your custom agreement metric. 

**On-prem deployment using AWS Lambda**

| Field | Description |
| --- | --- |
| **Lambda Tags** | Add tags to the AWS Lambda function using the syntax `tag_name tag_value`. |
| **Lambda Prefix** | Select a prefix for the AWS Lambda function. |

**On-prem deployment using Google Cloud Functions**

Unlike with AWS Lambda, you cannot set a prefix or tags for the Google Cloud Functions function. See [How Label Studio manages your functions](#How-Label-Studio-manages-your-functions) below. 


## How to write your custom agreement metric

For more information about agreement metrics, see [Task agreement](stats).

You can use the agreement metric to compare two annotations, or one annotation with one prediction. Use the input parameters `annotation_1` and `annotation_2` to specify the annotations to compare, or annotation and prediction to compare. 

Add your code to the following function defined in Label Studio:
```python
def agreement(annotation_1, annotation_2, per_label=False) -> float:
```

This function takes the following arguments:

| argument | format | description |
| --- | --- | --- |
| `annotation_1` | JSON object | The first annotation or prediction to compare when calculating agreement. Retrieved in [Label Studio JSON format](export.html#Label-Studio-JSON-format-of-annotated-tasks). |
| `annotation_2` | JSON object | The second annotation or prediction to compare when calculating agreement. Retrieved in [Label Studio JSON format](export.html#Label-Studio-JSON-format-of-annotated-tasks).
| `per_label` | boolean | Whether to perform an agreement calculation for each label in the annotation, or across the entire annotation result.  |
| `return` | float | The agreement score to assign, as a float point number between 0 and 1. |

For example, given the following labeling config:

```xml
<View>
  <Image name="image" value="$image"/>
  <Choices name="choice" toName="image" showInLine="true">
    <Choice value="Positive" />
    <Choice value="Negative" />
	<Choice value="Neutral" />
  </Choices>
</View>
```

The following agreement metric compares two annotations for a classification task with choice options of "Positive" and "Negative":

```python
def agreement(annotation_1, annotation_2, per_label=False) -> float:

    # Retrieve two annotations in the Label Studio JSON format
    r1 = annotation_1["result"][0]["value"]["choices"][0]
    r2 = annotation_2["result"][0]["value"]["choices"][0]
    
    # Determine annotation agreement based on specific choice values
    if r1 == r2:
        # If annotations match and include the choice "Positive", return an agreement score of 0.99
        if r1 == "Positive":
            return 0.99
        # If annotations match and include the choice "Negative", return an agreement score of 0.7
        if r1 == "Negative":
            return 0.7
    # If annotations do not match, return an agreement score of 0
    else:
        return 0
```

If you set `per_label=True`, you can define a separate method or agreement score for each label. If you do this, you must return a separate score for each label. For example, for a classification task, you could use the following function to assign a weight and return a specific agreement score for each label used in an annotation:

```python
def agreement(annotation_1, annotation_2, per_label=False) -> float:

    label_1 = annotation_1["result"][0]["value"]["choices"][0]
    label_2 = annotation_2["result"][0]["value"]["choices"][0]
    weight = {"Positive": 0.99, "Negative": 0.01}
    
    if label_1 == label_2:
        if per_label:
            return {label_1: weight[label_1]}
        else:
            return weight[label_1]
    else:
        if per_label:
            return {label_1: 0, label_2: 0}
        else:
            return 0
```

### Show logs

You can view logs for your custom agreement metric for troubleshooting purposes. 

For logs to generate, you must first have at least one task annotated by at least two annotators. 


## AWS Lambda (on-prem deployments)

If you have Label Studio Enterprise deployed in a private cloud (self-managed) Amazon Web Services (AWS) Elastic Compute Cluster (EC2) instance or Amazon Elastic Kubernetes Service (EKS), you must grant additional permissions so that Label Studio Enterprise can run custom agreement metrics in AWS Lambda. 

To set up the permissions, do the following: 
1. [Create an AWS IAM role](#Create-an-AWS-IAM-role-for-logging) to be used by the custom metric Lambda functions to store logs in Cloudwatch 
2. Set up permissions that grant access to AWS Lambda. How you do this depends on your deployment scenario:
   - [Deployed with Docker Compose running in EC2](#Deployed-with-Docker-Compose-running-in-EC2).
   - [Deployed in EKS with an OIDC provider](#Deployed-in-EKS-with-an-OIDC-provider).
   - [Deployed in EKS without an OIDC provider](#Deployed-in-EKS-without-an-OIDC-provider).

You must know the AWS account ID for the AWS account that you use to manage Label Studio Enterprise to perform these steps. 

### Step 1: Create an AWS IAM role for logging

Using your preferred method, create an AWS IAM role. 

1. Create an AWS IAM role named `LSE_CustomMetricsExecuteRole`. Follow the steps to create a role to delegate permissions to an AWS service in the AWS Identity and Access Management documentation for [Creating a role for an AWS service (console)](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-service.html#roles-creatingrole-service-console).
2. Whether you create the role using the console or the AWS CLI, create or attach the following IAM policy to allow the role to store logs in Cloudwatch. Replace `YOUR_AWS_ACCOUNT` with your AWS account ID that has access to Label Studio Enterprise.
```json
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "logs:CreateLogGroup",
            "Resource": "arn:aws:logs:*:YOUR_AWS_ACCOUNT:*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ],
            "Resource": [
                "arn:aws:logs:*:YOUR_AWS_ACCOUNT:log-group:/aws/lambda/custom-metric-*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogGroup",
                "logs:PutRetentionPolicy"
            ],
           "Resource": [
               "arn:aws:logs:*:YOUR_AWS_ACCOUNT:log-group:/aws/lambda/custom-metric-*"
           ]
        }
    ]
}
```

#### Create an IAM policy to grant AWS Lambda permissions

To grant permissions to a specific user, role, or EKS node group used to manage Label Studio Enterprise access to interact with AWS Lambda, use the following IAM policy. Create an IAM policy called `LSE_AllowInteractLambda` and replace `YOUR_AWS_ACCOUNT` with your AWS account ID:
```json
{
   "Version": "2012-10-17",
   "Statement": [
      {
         "Sid": "VisualEditor0",
         "Effect": "Allow",
         "Action": "iam:PassRole",
         "Resource": "arn:aws:iam::YOUR_AWS_ACCOUNT:role/LSE_CustomMetricsExecuteRole"
      },
      {
         "Sid": "VisualEditor1",
         "Effect": "Allow",
         "Action": [
            "lambda:CreateFunction",
            "lambda:UpdateFunctionCode",
            "lambda:InvokeFunction",
            "lambda:GetFunction",
            "lambda:DeleteFunction",
            "lambda:TagResource",
            "lambda:ListTags"
         ],
         "Resource": [
            "arn:aws:lambda:*:YOUR_AWS_ACCOUNT:function:custom-metric-*"
         ]
      },
      {
         "Sid": "VisualEditor2",
         "Effect": "Allow",
         "Action": "lambda:ListFunctions",
         "Resource": "*"
      },
      {
         "Action": [
            "logs:CreateLogGroup",
            "logs:PutRetentionPolicy",
            "logs:TagResource",
            "logs:StartQuery",
            "logs:GetQueryResults"
         ],
         "Effect": "Allow",
         "Resource": [
            "arn:aws:logs:*:YOUR_AWS_ACCOUNT:log-group:/aws/lambda/custom-metric-*"
         ]
      }
   ]
}
```

### Step 2: Configure permissions between Label Studio Enterprise and AWS Lambda

After creating an IAM role to manage logs for the custom agreement metric, set up permissions to allow Label Studio Enterprise to interact with AWS Lambda. 

How you set up permissions depends on how you deployed Label Studio Enterprise in your self-managed cloud infrastructure:
- [Deployed with Docker Compose running in EC2](#Deployed-with-Docker-Compose-running-in-EC2)
- [Deployed in EKS with an OIDC provider](#Deployed-in-EKS-with-an-OIDC-provider)
- [Deployed in EKS without an OIDC provider](#Deployed-in-EKS-without-an-OIDC-provider)

#### Docker Compose running in EC2

If you deployed Label Studio Enterprise using Docker Compose in an AWS EC2 instance, do the following to finish setting up permissions for the custom agreement metric functionality:
1. Follow the AWS documentation steps for [Creating an IAM user in your AWS account](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_create.html) to create an IAM user with programmatic access. This type of user is granted an access key to access AWS services.
2. While creating the IAM user, for the **Set permissions** option, choose to **Attach existing policies directly**.
3. Select **Create policy** and attach the [`LSE_AllowInteractLambda` policy](#Create-an-IAM-policy-to-grant-AWS-Lambda-permissions).
4. When you finish creating the user, save the username and access key somewhere secure.
5. In the `docker-compose.yaml` file that you use to deploy Label Studio Enterprise, add the following environment variables in the `app` and `rqworkers` sections:

!!! attention "important" 
    Update:
    - `YOUR_AWS_ACCESS_KEY_ID`, `YOUR_AWS_SECRET_ACCESS_KEY` and `YOUR_AWS_ACCOUNT` with the credentials for the account created in step 1. 
    - `YOUR_AWS_REGION` with the AWS region that your EC2 instance exists in the following:
```
AWS_ACCESS_KEY_ID=YOUR_AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY=YOUR_AWS_SECRET_ACCESS_KEY
LS_LAMBDA_REGION_CUSTOM_METRICS=YOUR_AWS_REGION
LS_LAMBDA_ROLE_CUSTOM_METRICS=arn:aws:iam::YOUR_AWS_ACCOUNT:role/LSE_CustomMetricsExecuteRole
```

After you set up these permissions in your environment, you're ready to write your custom agreement metric and add it to Label Studio Enterprise:
1. [Write your custom agreement metric](#How-to-write-your-custom-agreement-metric).
2. [Add your custom agreement metric to Label Studio Enterprise](#Add-your-custom-agreement-metric-to-Label-Studio-Enterprise).

#### Deployed in EKS with an OIDC provider

If you deployed Label Studio Enterprise in Amazon Elastic Kubernetes Service (EKS) with OpenID Connect (OIDC) for identity and access management (IAM), do the following to finish setting up permissions for the custom agreement metric functionality:
1. Create an AWS IAM role named `LSE_ServiceAccountApp` following the steps to create a role to delegate permissions to an AWS service in the AWS Identity and Access Management documentation for [Creating a role for an AWS service (console)](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-service.html#roles-creatingrole-service-console).
2. When relevant, attach the [`LSE_AllowInteractLambda` policy](#Create-an-IAM-policy-to-grant-AWS-Lambda-permissions) to the `LSE_ServiceAccountApp` role. 
3. Update your helm `values.yaml` file to include the following map. Replace `YOUR_AWS_ACCOUNT` with your AWS account ID:
```yaml
app:
  serviceAccount:
    annotations: 
      eks.amazonaws.com/role-arn: arn:aws:iam::YOUR_AWS_ACCOUNT:role/LSE_ServiceAccountApp
```
4. [Restart your Helm release](install_enterprise_k8s.html#Restart-Label-Studio-Enterprise-using-Helm).
   
After you set up these permissions in your environment, you're ready to write your custom agreement metric and add it to Label Studio Enterprise:
1. [Write your custom agreement metric](#How-to-write-your-custom-agreement-metric).
2. [Add your custom agreement metric to Label Studio Enterprise](#Add-your-custom-agreement-metric-to-Label-Studio-Enterprise).

#### Deployed in EKS without an OIDC provider

If you deployed Label Studio Enterprise in Amazon Elastic Kubernetes Service (EKS) and are not using OpenID Connect (OIDC) for identity and access management (IAM), do the following to finish setting up permissions for the custom agreement metric functionality:
1. In the AWS console UI, go to **EKS > Clusters > YOUR_CLUSTER_NAME > Node Group**.
2. Select the name of **YOUR_NODE_GROUP** with Label Studio Enterprise deployed.
3. On the **Details** page, locate and select the option for **Node IAM Role ARN**.
4. Create the AWS IAM policy [`LSE_AllowInteractLambda`](#Create-an-IAM-policy-to-grant-AWS-Lambda-permissions).
5. [Restart your Helm release](install_enterprise_k8s.html#Restart-Label-Studio-Enterprise-using-Helm).
   
After you set up these permissions in your environment, you're ready to write your custom agreement metric and add it to Label Studio Enterprise:
1. [Write your custom agreement metric](#How-to-write-your-custom-agreement-metric).
2. [Add your custom agreement metric to Label Studio Enterprise](#Add-your-custom-agreement-metric-to-Label-Studio-Enterprise).

## Google Cloud Functions (on-prem deployments)

If you deploy Label Studio Enterprise on Google Cloud Platform (GCP), you can run custom agreement metrics on [Google Cloud Functions (2nd gen)](https://cloud.google.com/functions/docs/concepts/version-comparison) instead of AWS Lambda.

Before you start, make sure you have your GCP project ID and know how Label Studio Enterprise is deployed (Compute Engine VM, GKE with Workload Identity, or GKE without Workload Identity).

### Step 1: Enable the required GCP APIs

Label Studio Enterprise requires the following APIs to be enabled in your GCP project:

- [Cloud Functions API](https://console.cloud.google.com/apis/library/cloudfunctions.googleapis.com) (`cloudfunctions.googleapis.com`)
- [Cloud Build API](https://console.cloud.google.com/apis/library/cloudbuild.googleapis.com) (`cloudbuild.googleapis.com`)
- [Cloud Storage API](https://console.cloud.google.com/apis/library/storage.googleapis.com) (`storage.googleapis.com`)
- [Cloud Logging API](https://console.cloud.google.com/apis/library/logging.googleapis.com) (`logging.googleapis.com`)
- [Artifact Registry API](https://console.cloud.google.com/apis/library/artifactregistry.googleapis.com) (`artifactregistry.googleapis.com`)
- [Cloud Run API](https://console.cloud.google.com/apis/library/run.googleapis.com) (`run.googleapis.com`)

You can enable them using the `gcloud` CLI:
```bash
gcloud services enable \
  cloudfunctions.googleapis.com \
  cloudbuild.googleapis.com \
  storage.googleapis.com \
  logging.googleapis.com \
  artifactregistry.googleapis.com \
  run.googleapis.com \
  --project=YOUR_GCP_PROJECT
```

### Step 2: Create a runtime service account for Cloud Functions

Create a dedicated service account for your Cloud Functions to run as. It only needs the **Logs Writer** role.

1. Create the service account:
```bash
gcloud iam service-accounts create lse-custom-metrics-runtime \
  --display-name="LSE Custom Metrics Runtime" \
  --project=YOUR_GCP_PROJECT
```

2. Grant the Logs Writer role:
```bash
gcloud projects add-iam-policy-binding YOUR_GCP_PROJECT \
  --member="serviceAccount:lse-custom-metrics-runtime@YOUR_GCP_PROJECT.iam.gserviceaccount.com" \
  --role="roles/logging.logWriter"
```

You'll need this service account email (`lse-custom-metrics-runtime@YOUR_GCP_PROJECT.iam.gserviceaccount.com`) later for the `GCP_SERVICE_ACCOUNT_CUSTOM_METRICS` environment variable.

### Step 3: Grant permissions to the Label Studio Enterprise identity

The identity Label Studio Enterprise runs as needs to manage Cloud Functions and upload function source. This identity varies by deployment — your Compute Engine VM service account, a dedicated GCP service account bound via Workload Identity, or your GKE node pool service account. The deployment-specific sections below tell you which one to bind this role to.

Create a custom IAM role named `LSE_CustomMetricsDeployer` with the following permissions:

```yaml
title: LSE Custom Metrics Deployer
description: Allows Label Studio Enterprise to deploy and invoke custom metric Cloud Functions
stage: GA
includedPermissions:
  # Cloud Functions lifecycle
  - cloudfunctions.functions.create
  - cloudfunctions.functions.update
  - cloudfunctions.functions.delete
  - cloudfunctions.functions.get
  - cloudfunctions.functions.list
  # Source upload via generateUploadUrl
  - cloudfunctions.functions.generateUploadUrl
  # Source code download from GCS staging bucket
  - storage.objects.get
  - storage.objects.create
  # Cloud Functions invocation (2nd gen uses Cloud Run)
  - run.routes.invoke
  # View function execution logs
  - logging.logEntries.list
  - logging.logs.list
  # Act as the runtime service account
  - iam.serviceAccounts.actAs
```

You can create this role using the `gcloud` CLI:
```bash
gcloud iam roles create LSE_CustomMetricsDeployer \
  --project=YOUR_GCP_PROJECT \
  --title="LSE Custom Metrics Deployer" \
  --description="Allows Label Studio Enterprise to deploy and invoke custom metric Cloud Functions" \
  --permissions=cloudfunctions.functions.create,cloudfunctions.functions.update,cloudfunctions.functions.delete,cloudfunctions.functions.get,cloudfunctions.functions.list,cloudfunctions.functions.generateUploadUrl,storage.objects.get,storage.objects.create,run.routes.invoke,logging.logEntries.list,logging.logs.list,iam.serviceAccounts.actAs
```

Then bind it to the Label Studio Enterprise identity. Replace `LSE_IDENTITY` with the appropriate member (see the deployment-specific sections below):
```bash
gcloud projects add-iam-policy-binding YOUR_GCP_PROJECT \
  --member="LSE_IDENTITY" \
  --role="projects/YOUR_GCP_PROJECT/roles/LSE_CustomMetricsDeployer"
```

!!! note
    Instead of a single custom role, you can also assign the following predefined roles: `roles/cloudfunctions.developer`, `roles/run.invoker`, `roles/storage.objectAdmin`, `roles/logging.viewer`, and `roles/iam.serviceAccountUser`. The custom role above follows the principle of least privilege.

#### Deployed with Docker Compose on a Compute Engine VM

If you deployed Label Studio Enterprise using Docker Compose on a Compute Engine VM:

1. Make sure the VM runs as a service account that has the `LSE_CustomMetricsDeployer` custom role (or the equivalent predefined roles) bound at the project level. If you need to create one, run:
```bash
gcloud iam service-accounts create lse-app \
  --display-name="LSE Application" \
  --project=YOUR_GCP_PROJECT

gcloud projects add-iam-policy-binding YOUR_GCP_PROJECT \
  --member="serviceAccount:lse-app@YOUR_GCP_PROJECT.iam.gserviceaccount.com" \
  --role="projects/YOUR_GCP_PROJECT/roles/LSE_CustomMetricsDeployer"
```

2. Assign that service account to your Compute Engine instance.

3. In the `docker-compose.yaml` file you use to deploy Label Studio Enterprise, add the following environment variables to the `app` and `rqworkers` sections:

!!! attention "important"
    Replace `YOUR_GCP_PROJECT` with your GCP project ID and `YOUR_GCP_REGION` with your preferred region (for example, `us-central1`).

```
CUSTOM_METRIC_PROVIDER=gcp
GCP_PROJECT_CUSTOM_METRICS=YOUR_GCP_PROJECT
GCP_REGION_CUSTOM_METRICS=YOUR_GCP_REGION
GCP_SERVICE_ACCOUNT_CUSTOM_METRICS=lse-custom-metrics-runtime@YOUR_GCP_PROJECT.iam.gserviceaccount.com
```

!!! note
    On Compute Engine with a properly scoped service account, Label Studio uses [Application Default Credentials (ADC)](https://cloud.google.com/docs/authentication/application-default-credentials) automatically — no key file required.

#### Deployed in GKE with Workload Identity

If you're using GKE with [Workload Identity](https://cloud.google.com/kubernetes-engine/docs/concepts/workload-identity):

1. Create a GCP service account for Label Studio Enterprise (if you haven't already):
```bash
gcloud iam service-accounts create lse-app \
  --display-name="LSE Application" \
  --project=YOUR_GCP_PROJECT
```

2. Bind the `LSE_CustomMetricsDeployer` custom role to the service account:
```bash
gcloud projects add-iam-policy-binding YOUR_GCP_PROJECT \
  --member="serviceAccount:lse-app@YOUR_GCP_PROJECT.iam.gserviceaccount.com" \
  --role="projects/YOUR_GCP_PROJECT/roles/LSE_CustomMetricsDeployer"
```

3. Allow the Kubernetes service account to impersonate the GCP service account:
```bash
gcloud iam service-accounts add-iam-policy-binding \
  lse-app@YOUR_GCP_PROJECT.iam.gserviceaccount.com \
  --role="roles/iam.workloadIdentityUser" \
  --member="serviceAccount:YOUR_GCP_PROJECT.svc.id.goog[YOUR_NAMESPACE/YOUR_K8S_SERVICE_ACCOUNT]"
```

4. Update your Helm `values.yaml` to annotate the Kubernetes service account and set the required environment variables:
```yaml
app:
  serviceAccount:
    annotations:
      iam.gke.io/gcp-service-account: lse-app@YOUR_GCP_PROJECT.iam.gserviceaccount.com
  extraEnvironmentVars:
    CUSTOM_METRIC_PROVIDER: gcp
    GCP_PROJECT_CUSTOM_METRICS: YOUR_GCP_PROJECT
    GCP_REGION_CUSTOM_METRICS: YOUR_GCP_REGION
    GCP_SERVICE_ACCOUNT_CUSTOM_METRICS: lse-custom-metrics-runtime@YOUR_GCP_PROJECT.iam.gserviceaccount.com

rqworker:
  extraEnvironmentVars:
    CUSTOM_METRIC_PROVIDER: gcp
    GCP_PROJECT_CUSTOM_METRICS: YOUR_GCP_PROJECT
    GCP_REGION_CUSTOM_METRICS: YOUR_GCP_REGION
    GCP_SERVICE_ACCOUNT_CUSTOM_METRICS: lse-custom-metrics-runtime@YOUR_GCP_PROJECT.iam.gserviceaccount.com
```

5. Restart your Helm release.

#### Deployed in GKE without Workload Identity

If you're using GKE without Workload Identity, your pods authenticate as the GKE node's service account, so that's the identity you'll grant permissions to.

1. Identify the service account used by your node pool. In the GCP console, go to **Kubernetes Engine > Clusters > YOUR_CLUSTER > Node Pools > YOUR_NODE_POOL > Security** and note the service account.

2. Bind the `LSE_CustomMetricsDeployer` custom role to the node pool's service account:
```bash
gcloud projects add-iam-policy-binding YOUR_GCP_PROJECT \
  --member="serviceAccount:YOUR_NODE_SERVICE_ACCOUNT@YOUR_GCP_PROJECT.iam.gserviceaccount.com" \
  --role="projects/YOUR_GCP_PROJECT/roles/LSE_CustomMetricsDeployer"
```

3. Update your Helm `values.yaml` to set the required environment variables:
```yaml
app:
  extraEnvironmentVars:
    CUSTOM_METRIC_PROVIDER: gcp
    GCP_PROJECT_CUSTOM_METRICS: YOUR_GCP_PROJECT
    GCP_REGION_CUSTOM_METRICS: YOUR_GCP_REGION
    GCP_SERVICE_ACCOUNT_CUSTOM_METRICS: lse-custom-metrics-runtime@YOUR_GCP_PROJECT.iam.gserviceaccount.com

rqworker:
  extraEnvironmentVars:
    CUSTOM_METRIC_PROVIDER: gcp
    GCP_PROJECT_CUSTOM_METRICS: YOUR_GCP_PROJECT
    GCP_REGION_CUSTOM_METRICS: YOUR_GCP_REGION
    GCP_SERVICE_ACCOUNT_CUSTOM_METRICS: lse-custom-metrics-runtime@YOUR_GCP_PROJECT.iam.gserviceaccount.com
```

4. Restart your Helm release.


### Environment variable reference for GCP

| Variable | Description | Default |
| --- | --- | --- |
| `CUSTOM_METRIC_PROVIDER` | Set to `gcp` to use Google Cloud Functions instead of AWS Lambda. | `aws` |
| `GCP_PROJECT_CUSTOM_METRICS` | The GCP project ID where Cloud Functions are deployed. | (none, required) |
| `GCP_REGION_CUSTOM_METRICS` | The GCP region for Cloud Functions (e.g. `us-central1`, `europe-west1`). | `us-central1` |
| `GCP_SERVICE_ACCOUNT_CUSTOM_METRICS` | The email of the runtime service account that deployed Cloud Functions execute as. | (none, optional) |

### Cloud Function runtime configuration

Label Studio Enterprise deploys each custom metric Cloud Function with a fixed runtime configuration. These values are not user-configurable from the UI or environment variables and apply to every deployed function:

| Setting | Value |
| --- | --- |
| **Runtime** | Python 3.13 (`python313`) |
| **Entry point** | `agreement_handler` |
| **Memory** | 256 MB |
| **Request timeout** | 60 seconds |
| **Max instance count** | 1 |
| **Ingress** | Internal only (`ALLOW_INTERNAL_ONLY`) |

What this means for your metric code:

- Use Python 3.13–compatible code and imports.
- Keep the `agreement` function lightweight and stateless. Loading large models or running expensive computation can exceed the 256 MB / 60 s limits and fail scoring.
- Scoring requests are queued one-at-a-time per function. This is fine for background workers but means metrics aren't suited to interactive use.

##### Network connectivity

Custom metric functions are deployed with internal-only ingress, which affects how you reach them:

- **Same GCP project as Label Studio Enterprise:** Works out of the box. Internal traffic from your Compute Engine or GKE workloads is allowed by default.
- **Different VPC or project:** You'll need to set up internal connectivity first — Shared VPC, VPC peering, or a Serverless VPC Access connector — or scoring will fail.
- **Testing the function directly:** You can't invoke it from outside Google's network. `curl` from your laptop or the Cloud Console **Test** button will return `403 Forbidden`. To verify deployment, trigger scoring inside Label Studio Enterprise (for example, recalculate agreement on a project that uses the metric).

#### How Label Studio manages your functions

Label Studio creates one Cloud Function per project that uses a custom metric, reuses it across scoring runs, and resolves the function URL for you — there's no URL or function name to configure manually.

To find a function in the Google Cloud console:

1. Open **Cloud Functions** in your configured GCP project and region.
2. Look for functions named `custom-metric-<project_id>-<timestamp>`.
3. Open a function to view logs, deployment state, and the trigger URL.

##### Function lifecycle behavior

| Behavior | Description |
| --- | --- |
| **Deploy** | Creates a new function if no matching existing function is found. |
| **Update** | Updates the existing function when possible. |
| **Provider/metric change** | If you switch away from this metric/provider, Label Studio attempts to delete the previously managed GCP function. |
| **Runtime status in app** | The app reports states such as NotDeployed, Pending, Active, Failed, NotFound, Error. |
| **Discoverability and metadata** | Label Studio applies internal labels (metadata) to functions for project/org/host tracking. These labels are primarily for internal management and cloud-side filtering. The function HTTPS URL is resolved and cached automatically by Label Studio. |

!!! attention "important"
    Prefix and tags/labels are not currently exposed as user-facing fields in the UI for GCP. 
    
    Advanced users can only set such values through API-level metric parameters (if your organization exposes that workflow), not via standard settings screens.

---

### Source Guide Dashboard Annotator (`docs/source/guide/dashboard_annotator.md`)

---
title: Member performance dashboard
short: Member performance
tier: enterprise
type: guide
order: 0
order_enterprise: 175
meta_title: Member performance dashboard
meta_description: Use the performance dashboard to track user work and progress. 
section: "Analytics"
date: 2024-04-04 11:50:41
---


The performance dashboard provides metrics about a user's annotation and review activities over a period of time, including how many tasks a user has completed and how quickly. 




This can be useful in a number of ways: 

- **Resource management optimization**: If you have a project that is on a rigid timeline, you can determine which users have the quickest turnaround and assign them as necessary. 
- **Annotator payment management**: If you are using outside contractors or otherwise paying users based on their output, you can use this dashboard to help calculate or verify contractor payments. 
- **Reduce costs associated with internal analytic reports**: With this report, you no longer need to develop complex tools or workflows to track user performance. 

## Access the dashboard

Depending on your user role, you can access this dashboard in several ways:

* From the **Organization** page, select a user on the **Members** page. Then select **Member Performance** on the right. 

    
* From the **Home** page, click **Member Performance**. This will pre-select your own performance dashboard. 

    
* From the **Members** dashboard on a project, click the overflow menu next to a username in the **Annotation Summary** table. This will pre-select that user's annotator performance dashboard for that specific project.  

    
* From anywhere within the app, open the navigation menu and select **Analytics**. This will pre-select your own annotator performance dashboard. 

    

### Access by user role

| User role             | Access restrictions                                                                                  |
|-----------------------|------------------------------------------------------------------------------------------------------|
| **Owner** <br>and<br> **Admin**      | Can access from the navigation menu, the Organization page, the Home page, and the Members dashboard. <br> Can filter for all users. <br> Can filter for all workspaces. |
| **Manager**               | Can access from the navigation menu, the Home page and the Members dashboard. <br> Can only filter for users who are members of workspaces or projects in which they are also a member. <br> Can only filter for workspaces or projects in which they are a member. |
| **Reviewer** <br>and<br> **Annotator** | Can access from navigation menu and the Home page. <br> Can only see their own annotation and reviewer history. <br> Can only filter for workspaces or projects in which they are a member. |


## Export data

You can use the **Export** drop-down to export the following:

* **Report** - Download the information in the dashboard as CSV or JSON. 

* **Timeline** - Download a detailed timeline of all the user's annotation actions within the time frame, including when the began and submitted each annotation. 

* **Comments Added** - Download a CSV file with all of the comments that the user has left. 

## Filters

### Filter by project and workspace

Select one or more workspaces or individual projects. To see all projects, select **All Workspaces** at the top of the workspace filter. 

This does not include Personal Sandbox projects. 

### Filter by date range

You can select a date range to apply to the performance metrics. If you want to be precise, you can also include time. 

Note that annotation and review metrics are calculated on when they are **completed**, not when they **begin**. 

For example, say you select a date range of 2 days: Monday and Tuesday. If an annotator opens an annotation on Tuesday, begins working on it, and then submits it on Wednesday, then that annotation and the time they spent on it will **not** be included in your performance report.  

Similarly, if a reviewer opens an annotation on Tuesday but then doesn't accept it until Wednesday, that annotation is not counted in their total reviews or accepted annotation metrics. 

### Save views

You can save your member, date range, and project filters as custom views. Saved views makes it easier for you to group member and project combos you want to readily access. 

Note the following:

* You can save, update, and delete views. 
* Views are only visible to you (meaning they are not shared with other users within your organization).

!!! note
    If you click **Apply** after saving your date and project/workspace filters, your selections will propogate to other Analytics pages. You can reapply your view or incorporate those filters into a new or updated view. 

## Annotations

The **Annotations** tab shows information related to a user's annotation performance and history. 

The metrics are calculated from the following data:

* `last_action` -- The last action taken on an annotation. This can be submitted, updated, fixed and accepted, accepted, rejected. 
* `submitted_or_reviewed` -- Annotations that have a last action matching those listed above.
* `lead_times` -- The time spent with annotations that have a last action matching those listed above. Lead time includes idle time and is cumulative across multiple sessions. 

!!! note
    A dash (`--`) appears when there is no data. A zero (`0`) appears when there is data available and the value is zero.

### Annotation performance summaries

| Metric | Calculation | Description | 
| --- | --- | --- |
| **Total Submitted Annotations** | Sum of `submitted_or_reviewed` | The total number of annotations the user submitted during the selected time frame. <br /><br />This includes annotations that have been submitted or updated, either reviewed or not. It does not include annotations that have been skipped. <br /><br />Note that each annotation is only included in their submitted count once. Label Studio does not count the same annotation twice based if it is later updated. | 
| **Time Annotating** | Sum of `lead_times` | The total time spent annotating during the selected time frame. This is calculated based on annotations that meet the criteria for `submitted_or_reviewed`. <br /><br />All annotations have a lead time. The lead time reflects how much time a user spent labeling from the moment the task was opened until they click **Submit** or **Update**. This includes idle time and time spent updating previously submitted annotations. <br /><br />The total time does not include time spent on annotations that have not been submitted and/or updated. For example, it does not include time spent on drafts or time spent on skipped annotations. <br /><br />Lead time is cumulative across sessions. That means if they leave or log out, and then return to an annotation draft or a previously skipped annotation, then their earlier time spent on the annotation is included when calculating their lead time.  | 
| **Performance Score** | Calculated from reviewer actions | The Performance Score reflects the overall performance of annotators in terms of review actions (**Accept**, **Reject**, **Fix+Accept**). <br /><br />The calculation is as follows:<ul><li>Each annotation review action (**Accept**, **Reject**, **Fix+Accept**) contributes to the score.</li><li>The score is calculated by summing the scores of all review actions and dividing by the total number of review actions. For example: </li><ul><li>If an annotation is rejected twice and then accepted once, the Performance Score would be (0 + 0 + 1) / 3 = 33%.</li><li>If an annotation is rejected once and then fixed+accepted with an agreement score of 42%, the Performance Score would be (0 + 0.42) / 2 = 21%. The agreement here is measured between what the annotator submitted and what the reviewer submitted when fixing it.</li></ul></ul> | 

### Annotation charts



| Chart | Description | 
| --- | --- | 
| **Annotation States Over Time** | A summary of all annotations completed by the user, segmented by date over the selected time period.    | 
| **Time Spent Annotating** | The time spent annotating (the lead time), segmented by date over the selected time period. <br /><br />This includes two metrics: The total time spent annotating in hours, and the average time per annotation in minutes. |

!!! note
    The date and time are calculated based on when they completed the annotation (see `last_action` above), and not when they began their annotation. See [Filter by data range](#Filter-by-date-range)


### Annotator performance table

This is a detailed table with performance metrics for each user. 



Keep in mind that all metrics in the table, including task assignments, are [filtered by the time frame selected](#Filter-by-date-range). 

| Column | Description | 
| --- | --- | 
| **Assigned** | The number of tasks that have either been manually assigned to the user or have received a submitted annotation from the user.  | 
| **Pending** | The number of tasks manually assigned to the user, and for which they haven't submitted an annotation and have not skipped. |
| **Submitted** | Number of submitted annotations. This includes annotations that have and have not been reviewed. |
| **Skipped** | Number of skipped annotations. |
| **Performance Score** | The performance score is calculated based on review actions on submitted annotations. See [the table above](#Annotation-performance-summaries) for a complete description. <br /><br />If no annotations have been reviewed, this column is grayed out.  |
| **Accepted** |  Number of submitted annotations that have been accepted by reviewers.   | 
| **Fix + Accepted** | Number of submitted annotations that have been adjusted (fixed) by reviewers and then accepted. |
| **Rejected** | Number of submitted annotation that have been rejected by reviewers. |
| **Total Time** | The total time this user spent annotating (sum of `lead_times`).  See [the table above](#Annotation-performance-summaries) for a complete description. |
| **Average Time** | The average time spent on each submitted annotation (average of `lead_times`). |
| **Median Time** | The median time spent on each submitted annotation (median of `lead_times`). |


## Reviews

The **Reviews** tab shows information related to a user's activity reviewing other users' annotations.

!!! note
    A dash (`--`) appears when there is no data. A zero (`0`) appears when there is data available and the value is zero.

### Review performance summaries

| Metric | Description | 
| --- | --- | 
| **Total Reviews** | Total number of annotations reviewed by the user. This includes all annotations that they have accepted, rejected, or updated and then accepted.   | 
| **Time Reviewing** | The total time spent reviewing. This is calculated from when a user opens an annotation to when they either take action on it (such as approving it) or close it by moving away from the task. This also includes idle time when the window is in focus. |

!!! note
    Data collection for review time began on September 25, 2025 for cloud environments, or for 2.30 for on-prem deployments. If you filter for earlier dates, review time will not be calculated.

### Review charts



| Chart | Description | 
| --- | --- | 
| **Review States Over Time** | A summary of all reviews completed by the user, segmented by date over the selected time period.    | 
| **Time Spent Reviewing** | The time spent reviewing, segmented by date over the selected time period. <br /><br />This includes two metrics: The total time spent reviewing in hours, and the average review time per annotation in minutes.  |

!!! note
    The date and time are calculated based on when they either completed the review (such as by accepting it) or exited the annotation. See [Filter by data range](#Filter-by-date-range)

### Reviewer performance table

This is a detailed table with performance metrics for each user. 



Keep in mind that all metrics in the table, including task assignments, are [filtered by the time frame selected](#Filter-by-date-range).  

| Column | Description | 
| --- | --- | 
| **Annotations Reviewed** | The number of annotations that the user has reviewed. This includes all review actions (accept, reject, fix + accept).  | 
| **Pending** | The number of annotations that the user has been manually assigned as a reviewer and which have not yet received their review. |
| **Accepted** | Number of annotations the user has accepted. |
| **Fix + Accepted** | Number of annotations the user has updated and then accepted. |
| **Rejected** | Number of annotations the user has rejected.  |
| **Total Time** | The total time this user spent reviewing. See [the table above](#Review-performance-summaries) for a complete description. |
| **Average Time** | The average time spent reviewing each annotation. |
| **Median Time** | The median time spent reviewing each annotation. |

---

### Source Guide Dashboard Data Quality (`docs/source/guide/dashboard_data_quality.md`)

---
title: Data quality dashboard
short: Data quality dashboard
tier: enterprise
type: guide
order: 0
order_enterprise: 71
meta_title: Data quality dashboard
meta_description: Use the Data quality dashboard to analyze agreement, confusion, and dimension-level quality for a project.
section: "Project Management"
parent: "dashboards"
parent_enterprise: "dashboards"
date: 2026-06-29 00:00:00
---

> Investigate project data quality with agreement metrics, dimension-level comparisons, and confusion analysis.

The **Data quality** dashboard helps you understand how consistently tasks are labeled and where annotations or labels disagree. Use it to find ambiguous tasks, confusing labels, and dimensions that need clearer instructions or review.

For velocity and time spent, see the [Throughput dashboard](dashboard_throughput). For member-level performance, see the [Members dashboard](dashboard_members).

## Access the dashboard

From the project, open the **Dashboard** tab and select **Data quality** from the dashboard navigation.

The Data quality dashboard has two tabs:

- **Agreement analysis**: Review agreement metrics and confusion across dimensions.
- **Label distribution**: Compare annotation and prediction label counts by dimension.

## Agreement analysis

Use the **Agreement analysis** tab to understand where annotators, models, and ground truth annotations align or disagree.

### Average Task Agreement

The **Average Task Agreement** KPI card shows the average agreement score for tasks in the project.

Agreement is most useful when tasks have multiple annotations or a comparison source such as ground truth. Tasks with only one annotation do not provide an annotator-to-annotator agreement signal.

### Overall Agreement Charts

| Chart | Description |
|-------|-------------|
| **Task Agreement Distribution** | Histogram of task-level agreement scores.<br /><br /> More tasks in higher ranges indicate stronger labeling consistency; more tasks in lower ranges suggest ambiguous instructions, difficult examples, or labels that need clarification. Clicking on a bar opens up the tasks tasks in the Data Manager. |
| **Agreement by Dimension** | Agreement score per dimension.<br /><br /> Use it to identify which dimensions are driving disagreement issues. |

### Ground Truth Agreement

| Chart | Description |
|-------|-------------|
| **Top Confusion Pairs** | Table of labels with the most number of incorrect values applied according to the ground truth.<br /><br />Use it to find label pairs that need better examples, clearer definitions, or additional training. |
| **Confusion Matrix** | Heatmap comparing labels or values within a selected dimension. <br /><br />Use the dimension dropdown to choose which dimension to inspect (default: All dimensions). |

## Label distribution

Use the **Label distribution** tab to compare values from annotations and predictions for each dimension.

Each supported dimension has a card with:

- **Average Agreement**: Agreement for the dimension when there is enough data to compare annotations or predictions.
- **Label Distribution chart**: A grouped horizontal bar chart comparing values from annotations and predictions.
- **Label Distribution table**: Counts from annotations and predictions for each value, plus a **Total** row. Values in the **From Annotations** and **From Predictions** columns link to the Data Manager with filters applied for the selected dimension and value.

For more detail about supported dimensions and Label Distribution behavior, see the [Label Distribution dashboard](dashboard_distribution).

## Interpreting data quality issues

Low agreement or high confusion does not always mean poor annotator or model performance. It can also indicate:

- Ambiguous task data.
- Labels that overlap conceptually.
- Missing or unclear labeling instructions.
- A project configuration that requires higher annotation overlap or [additional review strategy](quality).

Use the Data quality dashboard together with the [Members dashboard](dashboard_members) and Data Manager to decide whether to update instructions, add examples, reassign tasks, or coach specific members.

---

### Source Guide Dashboard Distribution (`docs/source/guide/dashboard_distribution.md`)

---
title: Label distribution dashboard
short: Label distribution
tier: enterprise
type: guide
order: 0
order_enterprise: 175
meta_title: Label distribution dashboard
meta_description: Use the Label Distribution dashboard to compare annotation and prediction label distributions by project dimension.
section: "Analytics"
date: 2026-06-29 00:00:00
---

The **Analytics > Label Distribution** dashboard shows how annotation and prediction values are distributed across a project. Use it to identify imbalanced labels, compare annotator and model outputs, and drill into the tasks behind a label value.



## Access by user role

| User role | Access restrictions |
|-----------|---------------------|
| **Owner** and **Admin** | Can view the **Label Distribution** dashboard and filter for all projects. |
| **Manager** | Can view the **Label Distribution** dashboard, but can only filter for and view projects in which they are a member.  |
| **Reviewer** and **Annotator** | Cannot access the **Label Distribution** dashboard. <br/><br/>When they open **Analytics**, they are redirected to the [Member Performance dashboard](dashboard_annotator), where they can only see their own annotation and review history. |

## Label distribution by dimension

The dashboard groups results by labeling dimension. A dimension can come from a traditional labeling config control tag or from an Interface output, such as ReactCode or CustomInterface fields.

Each supported dimension has a card with:

- **Average Agreement**: Agreement for the dimension when there is enough data to compare annotations, predictions, or ground truth.
- **Label Distribution chart**: A grouped horizontal bar chart comparing values from annotations and predictions.
- **Label breakdown table**: Counts from annotations and predictions for each value.



## Label breakdown table

The label breakdown table shows each value for the selected dimension, plus a **Total** row.

| Column | Description |
|--------|-------------|
| **Label** | Label or value name. The final row shows the total across all values. |
| **From Annotations** | Number of times the value appears in annotations. |
| **From Predictions** | Number of times the value appears in predictions. |

Values in the **From Annotations** and **From Predictions** columns can link to the Data Manager with filters applied for the selected project, dimension, value, and source. Use these links to inspect the underlying tasks, annotations, or predictions.



## Supported dimensions

Label distribution supports dimensions with a finite set of values, including:

- Label controls such as `<Labels>`, `<RectangleLabels>`, and other labels-based controls.
- Choice controls such as `<Choices>`.
- Taxonomy controls with finite, nested choices.
- Enum-backed ReactCode and CustomInterface dimensions.
- Other finite scalar dimensions that can be represented as a known set of values.

Dimensions that do not have a finite set of values are not shown as label distributions. This includes free text, arbitrary numeric or date values, objects, arrays, and values that are not declared in the dimension schema.

!!! note
    If a dimension is unsupported, the dashboard should show that it is unsupported rather than rendering an empty chart. This helps distinguish "no data" from data that cannot be safely summarized as a label distribution.

---

### Source Guide Dashboard Members (`docs/source/guide/dashboard_members.md`)

---
title: Members dashboard
short: Members dashboard
tier: enterprise
type: guide
order: 0
order_enterprise: 72
meta_title: Members dashboard
meta_description: Use the Members dashboard to monitor annotator and reviewer performance, agreement, and quality metrics within a project.
section: "Project Management"
parent: "dashboards"
parent_enterprise: "dashboards"
date: 2026-06-17 00:00:00
---

> Get insight into the performance, agreement, and quality of members working on a project.

The Members dashboard gives you two views of your project's workforce: a **Performance** tab with separate tables for annotators and reviewers, and an **Agreement Matrix** tab for inter-annotator reliability.

This can be useful in a number of ways:

- **Track annotator and reviewer performance**: View key metrics like evaluation status, agreement, acceptance score, performance score, GT agreement, and time spent to identify who is aligned or needs support.
- **Monitor annotation and review progress**: See how many annotations have been submitted, accepted, rejected, or are still pending review for both annotators and reviewers.
- **Identify quality and consistency issues**: Use the Agreement Matrix tab to spot disagreements between members, then click into cells to view the specific tasks where they diverged.

For member metrics across projects and over time, see the [Member Performance dashboard](dashboard_annotator).

## Access the dashboard

From the project, open the **Dashboard** tab and select **Members** from the dashboard navigation.

The Data quality dashboard has two tabs:

- **Performance**: Review key metrics for individual annotators and reviewers in the project.
- **Agreement Matrix**: Compare pairwise agreement scores between members, models, and ground truth.



## Filter by participants

A participant filter is available in the top right of the dashboard. By default it shows **All participants**.

- Annotators are listed under **All Annotators**; if the project has model predictions, model versions appear under **All Model Versions**.
- Selecting specific participants narrows both the Performance tables and the Agreement Matrix to only those members.

## Performance

The **Performance** tab has a toggle to switch between:

- **Annotations**: metrics for each member's submission activity
- **Reviews**: metrics for each member's review activity

Use the **Export** button to download the currently visible table as a CSV.

### Annotation Summary

The **Annotation Summary** table shows annotation metrics for each project member. The table is sortable and includes a totals row at the bottom. Use the **Columns** picker to show or hide columns.

| Column | Description |
|--------|-------------|
| **Annotator** | Member name. Click to open this member's [Member Performance dashboard](dashboard_annotator) filtered to this project. |
| **ID** | Member's numeric user ID. |
| **Paused** | Toggle to manually pause or unpause an annotator. See [Pause an annotator](quality.html#Pause-an-annotator). |
| **Evaluation** | Shown when [Annotator Evaluation](project_settings_lse.html#annotator-eval) is enabled. Status for annotators and reviewers: **Passed**, **Failed**, **Not started**, or **In progress**. Hover for the metric, score or sample progress, and when they were last evaluated. You cannot use the **Paused** toggle to unpause someone paused by evaluation. |
| **Agreement** | Average agreement score with other annotators on tasks where multiple annotations exist. |
| **Assigned** | Number of tasks manually assigned to the member or containing a submitted annotation by the member. |
| **Pending** | Number of tasks manually assigned to the member that do not yet have a submitted or skipped annotation. |
| **Submitted** | Total annotations submitted by the member. Includes submitted, updated, and reviewed annotations. |
| **Skipped** | Number of tasks skipped by the member. |
| **Accepted** | Number of submitted annotations accepted by reviewers. |
| **Fix + Accepted** | Number of submitted annotations fixed then accepted by reviewers. |
| **Rejected** | Number of submitted annotations rejected by reviewers. |
| **Updated** | Shown when [Annotator Evaluation](project_settings_lse.html#annotator-eval) is enabled. Number of annotations whose last action is Updated, including updates before or after a review. |
| **Acceptance Score** | Accepted annotations (including Fix + Accepted) divided by total reviewed annotations. Uses the latest review outcome per annotation. |
| **Performance Score** | Overall annotation performance based on review outcomes (Accept, Reject, Fix+Accept). See [Annotation performance summaries](dashboard_annotator#Annotation-performance-summaries) for the calculation. |
| **Total Lead Time** | Total elapsed time from task open to submit across all annotations. Includes time spent updating. |
| **Avg Lead Time** | Average elapsed time per submitted annotation. |
| **Median Lead Time** | Median elapsed time per annotation. |
| **GT Agreement** | Agreement between this member's annotations and ground truth annotations on overlapping tasks. |
| **Prediction Agreement** | Agreement between this member's annotations and model predictions on overlapping tasks. |
| **Last Active** | Date and time of the member's most recent activity in the project. |

!!! note
    The following columns are hidden by default and can be shown using the **Columns** picker: **ID**, **Assigned**, **Pending**, **Median Lead Time**, **Prediction Agreement**.

### Review Summary

The **Review Summary** table shows review activity for each member who has performed reviews on the project.

| Column | Description |
|--------|-------------|
| **Reviewer** | Member name. Click to open this member's [Member Performance dashboard](dashboard_annotator) filtered to this project. |
| **ID** | Member's numeric user ID. |
| **Paused** | Toggle to pause or unpause a reviewer. |
| **Reviewed** | Total annotations reviewed. Sum of accepted, fix+accepted, and rejected. |
| **Pending** | Number of tasks manually assigned to the reviewer that have not yet been reviewed. |
| **Accepted** | Number of annotations accepted by the reviewer. |
| **Fix + Accept** | Number of annotations fixed then accepted by the reviewer. |
| **Rejected** | Number of annotations rejected by the reviewer. |
| **Avg time** | Average time spent reviewing an annotation. Includes time spent fixing annotations. |
| **Total time** | Total time spent reviewing. |
| **Median time** | Median time spent reviewing an annotation. |
| **Last active** | Date and time of the reviewer's most recent review activity. |

!!! note
    The following columns are hidden by default and can be shown using the **Columns** picker: **ID**, **Performance**, **Submitted tasks**.

!!! note
    Review time data collection began on September 25, 2025 for cloud environments, or version 2.30.0 for on-prem deployments. Filtering for earlier dates will not include review time.

### Performance by Dimension

The **Performance by Dimension** table appears below the Annotation Summary on the Annotations sub-tab. It breaks down each member's performance score per labeling dimension  configured in the project.

Use the **Compare with** dropdown to choose how scores are calculated:

- **Compare with GT** — scores reflect each member's agreement with ground truth annotations on overlapping tasks.
- **Compare with All Annotators** — scores reflect each member's agreement with other annotators on the same tasks.

An **Avg** column shows each member's overall average across all dimensions. A footer row shows the average per dimension across all members. Use the **Columns** picker to show or hide individual dimensions.

## Agreement Matrix 

The **Agreement Matrix** tab shows the **Member / Model Agreement Matrix**, a heatmap of pairwise agreement scores between all members (and models, if included via the participant filter) in the project.

Click on any cell to open the Data Manager filtered to tasks where both members annotated, so you can inspect disagreements directly.

!!! note
    Agreement in the Members dashboard reflects an agreement score between annotators, regardless of the [agreement methodology](stats#Methodology) configured for the project.

### Filters

- **Dimension**: Filter agreement by a specific labeling dimension (control tag). Defaults to **All dimensions**.
- **Label**: Filter to tasks where at least one annotation contains the specified label. Only available when a specific dimension is selected.

### Reading the matrix

Cells are color-coded by agreement band:

- **Green** — High (≥ 66%)
- **Yellow** — Moderate (33–66%)
- **Red** — Low (< 33%)

Hover over any cell to see the agreement percentage and the number of tasks where both members annotated. If a **Ground Truth** column is present, it shows each member's agreement against the project's [ground truth](ground_truths.html) annotations.

An **Avg** row at the bottom shows the average agreement per column across all members.

The participant filter (top right) applies to this tab — select specific annotators or model versions to narrow the matrix.

---

### Source Guide Dashboard Throughput (`docs/source/guide/dashboard_throughput.md`)

---
title: Throughput dashboard
short: Throughput dashboard
tier: enterprise
type: guide
order: 0
order_enterprise: 70
meta_title: Throughput dashboard
meta_description: Use the Throughput dashboard to monitor project velocity, remaining work, and time spent.
section: "Project Management"
parent: "dashboards"
parent_enterprise: "dashboards"
date: 2026-06-29 00:00:00
---

> Track how quickly work is moving through a project and where bottlenecks are forming.

The **Throughput** dashboard shows project velocity over time. Use it to monitor completed work, remaining work, and the time spent annotating and reviewing tasks.

For data quality-focused metrics, see the [Data quality dashboard](dashboard_data_quality). For member-level productivity and agreement, see the [Members dashboard](dashboard_members).

## Access the dashboard

From the project, open the **Dashboard** tab and select **Throughput** from the dashboard navigation.

## Filter by date range

Use the date range filter to update the KPI cards and charts for the selected time period.

If you include a precise time, the dashboard uses that timestamp when determining which task and work events fall within the range.

## KPI cards

The top of the dashboard shows summary cards for the selected date range.

| KPI | Description |
|-----|-------------|
| **Tasks Done** | Number of tasks that reached the [Done state](project_states#Task-states) during the selected date range.<br /><br /> **Avg Tasks Done / Day** sub-metric shows the average number of tasks completed per day in the selected date range. Calculated as Tasks Done / Days.   |
| **Annotations Submitted** | Total number of annotations submitted (manually or created from a prediction) during the selected date range. <br /><br /> **Annotations / Day** sub-metric shows the average number of annotations submitted per day in the selected date range. Calculated as Annotations Submitted / Days.|
| **Total Time Spent** | Total time spent annotating and reviewing. Note: Annotation time uses active annotation time, as opposed to [lead time](dashboard_annotator#Annotations).<br /><br /> The sub-metric shows the Total Time Spent on Done Tasks.
| **Avg Time / Task** | Average active time spent per task that has started work (ie. past the initial stage). Calculated as Total Time Spent / Tasks Started. <br /><br /> The sub-metric shows the Total Time Spent on Done Tasks / Tasks Done. |

!!! note
    Task state metrics are based on [project and task state management](project_states). If state tracking was enabled after some project work had already happened, earlier state transitions might not appear in the dashboard.

## Throughput charts

| Chart | Description |
|-------|-------------|
| **Tasks by State** | Stacked bar chart of the number of tasks in each state (Initial, Annotating, Needs Review, In Review, Done) over time. Use it to track task progress or identify bottlenecks.<br /><br /> Tasks start at the **Initial** state when they have been created but contain no annotations yet. Tasks that have completed the required number of annotations either move to the **Needs Review** or **Done** state, depending on whether your project settings require a review on the task. [Learn more about task state management](project_states#Task-states).|
| **Tasks Done / Day** | Number of tasks reaching the Done state per day. The same task may appear on multiple days if it is re-worked after reaching Done. Use it to track velocity and spot output changes after adjusting staffing, task instructions, review requirements, or project settings. |
| **Time Spent** | Time spent over the selected period, split between annotation and review. Use it to understand how effort is distributed across work types over time. |

---

### Source Guide Dashboards (`docs/source/guide/dashboards.md`)

---
title: Dashboards
short: Dashboards
tier: enterprise
type: guide
order: 0
order_enterprise: 135
meta_title: Dashboard in Label Studio
meta_description: Use the project Dashboard tab to monitor project throughput, quality, and member performance.
section: "Create & Manage Projects"
---

> Use project dashboards to monitor throughput, data quality, and member performance for a single project.

The project **Dashboard** tab gives managers, administrators, and owners a project-level view of how work is moving, where quality issues are appearing, and how members are performing.

Open a project and select **Dashboard** to access the project dashboards:

- **Throughput**: Track velocity, remaining work, and time spent.
- **Data quality**: Analyze agreement, dimension-level quality, and label confusion.
- **Members**: Review annotator, reviewer, and model performance within the project.

## Choose a project dashboard

| Dashboard | Use it to |
|-----------|-----------|
| [**Throughput**](dashboard_throughput) | Understand whether the project is moving at the expected pace, where work is blocked, and how annotation and review time are trending. |
| [**Data quality**](dashboard_data_quality) | Investigate agreement, confusion between labels, quality by dimension, and the tasks behind quality issues. |
| [**Members**](dashboard_members) | Compare annotator, reviewer, and model performance, including agreement, review outcomes, time spent, and per-dimension performance. |

## Access by user role

Project dashboards are available to users who can manage or monitor the project, such as Owners, Administrators, and Managers.

Annotators and Reviewers generally do not use the project Dashboard tab. For individual work history across projects, see the [Member Performance dashboard](dashboard_annotator).

## Filtering and drilldowns

Each project dashboard has filters that apply to the metrics on that page. Depending on the dashboard, you can filter by date range, task data, annotator, model, review outcome, ground truth status, labeling dimension, or label.

Many cards, charts, and tables link to the Data Manager with relevant filters applied. Use these drilldowns to move from a summary metric to the exact tasks, annotations, predictions, or labels behind it.

## Analytics dashboards

Analytics dashboards are available from the main **Analytics** navigation and can include data across projects, workspaces, or individual members depending on your role.

| Dashboard | Description |
|-----------|-------------|
| [**Projects overview dashboard**](dashboard_overview) | Monitor progress across multiple projects and workspaces. |
| [**Label Distribution dashboard**](dashboard_distribution) | Compare annotation and prediction value distributions for project dimensions. |
| [**Member Performance dashboard**](dashboard_annotator) | Review an individual member's annotation and review activity across projects. |

---

### Source Guide Decommission Minio (`docs/source/guide/decommission_minio.md`)

---
title: Decommission minio
short: Decommission minio
meta_title: Decommission minio
---


!!! warning
    Decommissioning Minio is a hazardous action because you may lose all your data uploaded using direct file upload. For more information or support, contact the [Heartex team](support@humansignal.com).

To decommission Minio, do the following:

## Configure Persistent Storage

To configure Persistent Storage, follow the instructions in [Set up persistent storage](persistent_storage.html).

## Upgrade to the latest available Release

1. Add `MINIO_MIGRATION: true` into `global.extraEnvironmentVars` section of your `ls-values.yaml` file:
```shell
global:
  extraEnvironmentVars:
    MINIO_MIGRATION: true
```

2. Redeploy a release by following the steps from [Upgrade Label Studio Enterprise](install_enterprise_k8s.html#Upgrade-Label-Studio-using-Helm). 

## Run migration script

1. Run shell in `lse-app` pod:
```shell
kubectl exec -ti deploy/<YOUR_RELEASE_NAME>-lse-app -c lse-app -- bash
```

2. Copy data from Minio to a persistent storage:
```shell
JSON_LOG=0 python3 $LSE_DIR/label_studio_enterprise/manage.py minio-migrate
```

## Ensure a successful data migration


!!! warning 
    Minio service will be deleted in the next step. 


- Try to run `Export` in your project.
- Ensure that this data appears in your `bucket/volume` by `project_id/export`.

!!! note 
    For more information or support, contact the [Heartex team](support@humansignal.com).

## Completely remove Minio service

1. Add the following map to your `ls-values.yaml` file:
```yaml
minio:
  enabled: false
```

2. Remove `MINIO_MIGRATION: true` from `global.extraEnvironmentVars` section of your `ls-values.yaml` file.

3. Redeploy a release by following steps from [Upgrade Label Studio Enterprise](install_enterprise_k8s.html#Upgrade-Label-Studio-using-Helm).

---

### Source Guide Email Setup (`docs/source/guide/email_setup.md`)

---
title: Set up email backend for Label Studio Enterprise
short: Set up email backend
tier: enterprise
type: guide
order: 0
order_enterprise: 90
meta_title: Email backends in Label Studio
section: "Install & Setup"

---

In Label Studio Enterprise, you can configure email backends to enable password reset via email and receive notifications. There are three available options for setting up email backends:
* Dummy console email backend - all emails will be printed in the app console.
* SMTP backend
* [Sendgrid backend](https://sendgrid.com/)


### Dummy Console Backend

The Dummy Console Email Backend is a simple option for testing purposes. When this backend is configured, all emails generated by the application will be printed in the application console, rather than being sent to recipients.

```bash
FROM_EMAIL = Label Studio <hello@labelstud.io>

# option 1: app console
EMAIL_BACKEND = 'django.core.mail.backends.console.EmailBackend'
```

### SMTP Backend

The SMTP (Simple Mail Transfer Protocol) Backend allows you to use a custom SMTP server to send emails. This provides flexibility in configuring email delivery options and can integrate with various email service providers.

More details can be found here: https://docs.djangoproject.com/en/4.2/topics/email/#smtp-backend. If your SMTP configuration is not compatible with certificate or hostname validation, we provide the
`label_studio.core.utils.mail.NoVerificationEmailBackend` which can be used instead of Django's default `smtp.EmailBackend`, but note that this option is less secure. See https://docs.djangoproject.com/en/5.0/releases/4.2/#miscellaneous for more information.

```bash
# SMTP server,
FROM_EMAIL=Label Studio <hello@labelstud.io>
EMAIL_BACKEND=django.core.mail.backends.smtp.EmailBackend
EMAIL_HOST=smtp.gmail.com
EMAIL_PORT=587
EMAIL_HOST_USER=USERNAME@gmail.com
EMAIL_HOST_PASSWORD=PASSWORD # It's your account password, or separate application password, if you have 2fa on your google account
EMAIL_USE_TLS=1
EMAIL_USE_SSL=0
# EMAIL_SSL_KEYFILE=/path/inside/of/ls/container/key  # optional
# EMAIL_SSL_CERTFILE=/path/inside/of/ls/container/cert  # optional
EMAIL_TIMEOUT=60
```

### Sendgrid Backend

The Sendgrid Backend utilizes the Sendgrid API to send emails. This option requires an active Sendgrid account and API key for authentication. The Sendgrid backend offers an easy-to-use and reliable email delivery service.

```bash
# option 2: Sendgrid
EMAIL_BACKEND=sendgrid_backend.SendgridBackend
SENDGRID_API_KEY=<SENDGRID-API-KEY>
```

---

### Source Guide Embed (`docs/source/guide/embed.md`)

---
title: Embed Label Studio - Beta 🧪
short: Embed Label Studio 🧪
type: guide
tier: enterprise
order: 409
order_enterprise: 409
meta_title: Embed Label Studio
meta_description: Overview information for embedding Label Studio Enterprise in an external application.
section: "Integrate & Extend"
---

Label Studio Embeds allow you to seamlessly integrate annotation and review functionality into your own applications. 

!!! note
    This feature is not available to all customers. Contact your [HumanSignal account manager](mailto:sales@humansignal.com) to enable.

Before you begin, ensure you have:
* Label Studio:
    * You must have the Owner role in Label Studio.
    * From **Organization > Settings > Access Token Settings**, ensure that at least one option is enabled.


* Your external application:
    * Access to your server backend (to generate signed tokens).
    * The URL where your embed will be hosted.



## Authorization

At a high level, the embed process is a secure handshake between your app and Label Studio:

1. A user logs into your external app.

2. Your backend signs a JWT with their `user_email`, `organization_id`, and `project_id` using your private key.

3. Your frontend initializes the Label Studio embed with `id`, `url`, and `token`.

4. Label Studio validates the JWT and issues its own internal token.


```mermaid
sequenceDiagram
    participant User
    participant YourApp as Your App (Frontend & Backend)
    participant LSE as Label Studio Enterprise

    User->>YourApp: Logs in
    YourApp->>YourApp: Backend signs JWT (user_email + organization_id + project_id)
    YourApp->>User: Sends JWT to frontend
    User->>LSE: Initializes Embed with (id, url, token)
    LSE->>LSE: Verifies JWT with Public Key
    LSE->>User: Issues Internal Token
    User->>LSE: Annotates, Reviews, Uses Embed
```

### Generate a JWT on your server

You must begin by generating a JSON web token (JWT) that authenticates Label Studio. 

You will also need to generate a base64-encoded public URL key, which you will add to Label Studio. 

JWT claims required:

```json
{
  "user_email": "user@example.com",
  "organization_id": 12345,
  "project_id": 67890,
  "iat": 1692445200,
  "exp": 1692452400
}
```

!!! note
    `exp`, `organization_id`, and `project_id` are required. Each accepts a positive integer or a string of digits. `exp` is a Unix timestamp in seconds and must fall within your organization's token lifetime (60 minutes by default). `project_id` scopes the embed to that project and must match the project in the embed URL. `iat` is optional.

**Notes:**

* You can find your organization ID from the **Organization** page in Label Studio.   
* The example below uses RS256 as a verification algorithm, but you can find a full list of supported algorithms from **Organization > Usage & License > Embedding**.   
* The public key must be stored in Label Studio in base64-url encoded format.



#### Example: Node.js (jsonwebtoken)

```javascript
const jwt = require("jsonwebtoken");
const fs = require("fs");

const privateKey = fs.readFileSync("./private.pem");

const token = jwt.sign(
  {
    user_email: "user@example.com",
    organization_id: 12345,
    project_id: 67890,
  },
  privateKey,
  { algorithm: "RS256", expiresIn: "1h" }
);

console.log(token);
```

## Configure embedding in Label Studio

From Label Studio, go to **Organization > Usage & License > Embedding** and complete the following fields:

| Field | Description  |
| :---- | :---- |
| **Supported domains** | List the domains of your external application where you will be adding an embedding.  |
| **Public Verification Key** | This must be a base64-url encoded key.  |
| **Public Verification Algorithms** | The algorithm you used when creating your key.  |
| **Embed SDK** | Add this to your application to initialize the embedding.   |

## Configure an embedding page in your external application

At a minimum, you need to add the **Embed SDK** script to an HTML page in your application. For example:

```xml 
<script src="https://app.humansignal.com/react-app/embed-sdk.js"></script>
```

But to maximize the usability of your embedding, you can also include additional Label Studio elements. See our [Frontend Reference](frontend_reference). 

Here is an example:

```
/* Detailed source-code truncated for AI context efficiency. */
```

## Troubleshooting

**Blank/white screen**

* Domain not listed in **Supported Domains** within Label Studio.  
* Misconfigured/missing public verification key  
* Misconfigured/missing public verification algorithms  
* Verification of the external JWT failed to verify one or more of the claims provided (invalid JWT).  
* Active organization of the user was different than specified in the JWT claims.  
* Your org has not enabled legacy API tokens or personal [access tokens](https://docs.humansignal.com/guide/access_tokens) (need at least one). 
* User with the provided `user_email` does not exist on `LabelStudioEnterprise`

**HTTP errors in SDK**

* 403 (permission errors)  
  * User is not a member of the Project  
  * Task was not assigned to a user, but attempted to load  
  * User did not have access to an annotation or prediction, but it attempted to load anyways

* 404  
  * Task was not found

* Other non-HTTP errors that caused a logical problem in the system.

---

### Source Guide Enterprise Features (`docs/source/guide/enterprise_features.md`)

---
title: "Enterprise features: Label Studio at scale"
short: Enterprise features
type: guide
tier: opensource
order: 35
order_enterprise: 0
section: "Discover & Learn"
meta_title: Label Studio Enterprise Features
meta_description: Overview of the features in Label Studio Enterprise.
---

This page provides an overview of the advanced capabilities available exclusively in [Label Studio Enterprise](https://humansignal.com/pricing/). These features extend the core functionality of Label Studio with enterprise-scale user management, security, quality assurance, and automation.



## Organizations and Workspaces

Label Studio Enterprise introduces a hierarchical structure for organizing projects and users through organizations and workspaces, which is not available in the community edition.

### Organizations

Organizations are the top-level entity in Label Studio Enterprise that contain workspaces, projects, and users. Each organization is isolated from others, providing complete data separation.

Key capabilities of organizations:

  - Organization-level user management
  - Isolated data and resources
  - Centralized billing and usage tracking
  - Custom branding options

### Workspaces

Workspaces provide a way to group related projects within an organization. They help structure projects by department, team, or use case.

Key capabilities of workspaces:

  - Group related projects together
  - Delegate management responsibilities
  - Control access at the workspace level
  - Share resources between projects in the same workspace

## Multi-tenant SaaS architecture

Organizations provide complete isolation between different Label Studio Enterprise SaaS customers, ensuring data segregation and independent user management. Workspaces within organizations group related projects by team, department, or product line. Label Studio Enterprise may also be deployed on-prem (see [Enterprise Deployment Options](#Enterprise-deployment-options)). 

## User management and role-based access control

### Role hierarchy and permissions

Label Studio Enterprise implements a comprehensive role-based access control (RBAC) system with five distinct roles, each providing specific capabilities and access levels. For information on how to assign users to roles, see [Manage user accounts.](https://docs.humansignal.com/guide/admin_manage_lse.html)

| Role      | Organization Access | Primary Capabilities                       | Project Management | Annotation | Review | User Management |
| :-------- | :------------------ | :----------------------------------------- | :----------------- | :--------- | :----- | :-------------- |
| Owner     | Full control        | Full system access, billing management     | All projects       | ✅          | ✅      | Full            |
| Admin     | Organization-wide   | User management, all organization settings | All projects       | ✅          | ✅      | Limited         |
| Manager   | Workspace-scoped    | Project creation, workspace management     | Assigned projects  | ✅          | ✅      | Project-level   |
| Reviewer  | Project-scoped      | Review annotations, manage quality control | View only          | ✅          | ✅      | None            |
| Annotator | Project-scoped      | Create and edit annotations                | View only          | ✅          | ❌      | None            |

```mermaid
graph TD
    subgraph Project Level
        Project1[Project 1]
        Project2[Project 2]
        Project3[Project 3]
        ReviewerRole[Reviewer Role]
        AnnotatorRole[Annotator Role]
        Project1 --> AnnotatorRole
        Project2 --> AnnotatorRole
        Project3 --> ReviewerRole
    end

    subgraph Workspace Level
        WorkspaceA[Workspace A]
        WorkspaceB[Workspace B]
        ManagerRole[Manager Role]
        WorkspaceA --> ManagerRole
        WorkspaceB --> ManagerRole
    end

    subgraph Organization Level
        Organization
        OwnerRole[Owner Role]
        AdminRole[Admin Role]
        Organization --> OwnerRole
        Organization --> AdminRole
    end

    Organization --> WorkspaceA
    Organization --> WorkspaceB
    WorkspaceA --> Project1
    WorkspaceA --> Project2
    WorkspaceB --> Project3
```

### Project-level roles

Project roles are assigned independently of organization roles. For example, a user can be an Annotator in one project and a Reviewer in another project, allowing for flexible permission management:

  - **Annotator:** Can view projects and assigned tasks and can create annotations
  - **Reviewer:** Can review annotations, accept or reject them, and provide feedback

### User lifecycle management

Enterprise features include comprehensive user account management capabilities:

  - **Invitation-based signup:** Control user onboarding through invitation links
  - **Bulk user operations:** Assign roles and update workspace memberships in bulk
  - **User activity monitoring:** Track annotation activity and system usage
  - **Account deactivation:** Preserve annotation history while removing access

## Authentication and identity management

Label Studio Enterprise provides comprehensive identity management capabilities for large-scale deployments, supporting integration with existing enterprise identity providers.

### SSO and SAML integration

The enterprise edition supports Single Sign-On through SAML 2.0 assertions, allowing integration with identity providers like Okta, Google SAML, Ping Identity, Microsoft Active Directory, and other SAML-compatible systems. For setup instructions, see the[ Authentication and SSO Configuration Guide](https://docs.humansignal.com/guide/auth_setup).

### SCIM 2.0 provisioning

System for Cross-domain Identity Management (SCIM) 2.0 enables automated user provisioning and deprovisioning from identity providers.

The SCIM integration supports:

  - **Create Users:** Automatic user provisioning with role assignments
  - **Update User Attributes:** Synchronize profile changes and role modifications
  - **Deactivate Users:** Remove access while preserving annotation history
  - **Push Groups:** Workspace and project membership management

For setup instructions, see the [SCIM 2.0 Integration Guide](https://docs.humansignal.com/guide/scim_setup).

### LDAP integration

Enterprise customers can integrate with LDAP directories for user authentication while maintaining local role management.

## Advanced security features

### Enterprise security architecture

Label Studio Enterprise includes comprehensive security features designed for enterprise compliance requirements. Enterprise security features include:

- Redis over TLS with client certificate authentication
- IP allowlisting and VPN access control for cloud storage  
 (OSS supports limited IP filtering only for GCP)
- Advanced audit logging
- SCIM 2.0 user provisioning and deprovisioning
- SAML-based Single Sign-On (SSO) integration
- Role-based access controls across organizations and workspaces
- Granular task and annotator guardrails (e.g. auto-pausing users, setting trust thresholds)
- Custom plugins and front-end customization with organization-level isolation
- Deployment support for fully air-gapped, on-prem environments
- Uptime SLA and security certifications (e.g. SOC 2, HIPAA readiness)

### Data security

  - **End-to-end TLS/SSL:** Secure communication between all components
  - **Audit Logging:** Comprehensive logging of all user activities

### Advanced cloud storage options 

Label Studio Enterprise supports a wider range of persistent storage options for scalable, production-grade deployments. These include:

* **AWS S3 IAM role access**: Configure secure access using IAM roles or static credentials
* **Google Cloud Storage with Workload Identity Federation (WIF)**: Use short-lived credentials to avoid hardcoded secrets

## Advanced labeling & evaluation workflows

### AI features and model integration

Label Studio Enterprise provides enhanced machine learning capabilities:

  - **AI Features Toggle:** Enable AI helper tools
  - **Prediction Score Display:** View prediction/confidence scores in the labeling interface
  - **Bulk Labeling:** Quickly label multiple tasks at once

### Automated active learning loops

Label Studio Enterprise supports active learning workflows that automatically update model predictions as labeling progresses.

After each annotation, Label Studio can send a webhook to a connected machine learning backend. The backend uses the new data to retrain the model `(fit())`, and Label Studio fetches updated predictions `(predict())` for the next task, creating a continuous feedback loop.

Key components:

  - **ML backend setup:** Register a model to receive training data and return predictions
  - **Prediction integration:** Display updated predictions during labeling
  - **Webhooks:** Trigger model retraining after annotations
  - **Task sampling:** Use prediction scores to prioritize which tasks to label
  - **Continuous loop:** As labeling continues, predictions improve with each iteration

### Prompts

For an additional fee, Label Studio Enterprise includes a built-in Prompts interface for working with language models as part of the labeling workflow.

Prompt templates allow users to define inputs, models, and outputs to send to connected LLMs. Prompts can be used to pre-label data, generate new tasks, or support review workflows.

The Prompts interface supports:

  - **Prompt Templates:** Define inputs, outputs, and model parameters
  - **Inline Generation:** Run prompts directly from the labeling interface
  - **Model Connections:** Connect to OpenAI, Anthropic, or custom endpoints via API
  - **Pre-labeling:** Populate fields using model-generated responses
  - **Task Creation:** Generate new tasks from prompt outputs

For more information, see the [Prompts Overview](https://docs.humansignal.com/guide/prompts_overview/).

### Customization with plugins

Label Studio Enterprise supports custom JavaScript plugins to extend and tailor the labeling interface for project-specific needs.

Plugins run automatically when users interact with annotation tasks. They can be used to validate annotations, add visual elements, call external services, or apply custom logic based on events in the labeling UI.

The plugin framework supports:

  - **Project-specific scripts:** Define custom plugins on a per-project basis
  - **Event-driven behavior:** Trigger logic on task load, annotation switch, and other actions
  - **UI extensions:** Add validations, overlays, or context-aware functionality
  - **External integration:** Connect to APIs or fetch reference data in real time
  - **Testing and preview:** Use the built-in panel to test plugins before deployment

To learn how organizations are using plugins in real-world workflows, see the blog post [Build Powerful Labeling Interfaces with Plugins](https://humansignal.com/blog/build-powerful-labeling-interfaces-with-plugins/).

## Quality workflows and analytics

Label Studio Enterprise enhances the annotation workflow with features designed for quality control and team collaboration, as well as analytics to optimize annotation and dataset quality.

### Quality control features

  - **Auto-validation**
  - **Review Workflows:** Configure review processes for annotations
  - **Inter-annotator agreement:** Measure agreement between annotators to ensure quality
  - **Annotator Performance Dashboard:** Track and analyze annotator quality to identity low performers, implement training and corrective actions
  - **Project Performance Dashboard:** View label distribution to ensure completeness
  - **Guardrails for low-trust annotators:** Set limits or pause annotators to prevent

### Inter-annotator agreement metrics and matrices

View real-time agreement metrics to take proactive action on quality. The Enterprise edition includes 30+ built-in agreement metrics for different annotation types, including:

* **Basic matching:** Default evaluation for each control tag type
* **Intersection over Union (IoU):** For bounding boxes and polygons
* **Edit distance algorithms:** For text transcription tasks
* **F1, Precision, Recall:** For classification and detection tasks

To learn how different team structures use agreement metrics to scale high-quality labeling, see the blog post [3 Annotation Team Playbooks for Scalable Labeling](https://labelstud.io/blog/the-spectrum-of-annotators-how-to-match-your-labeling-workflow-to-your-team/).

### Low-trust annotator guardrails

  * **Annotator pausing:** Manual and automatic pausing based on performance criteria
  * **Annotation limits:** Configure maximum annotations per user to prevent spam
 
### Project performance dashboards

The [project dashboard](https://docs.humansignal.com/guide/dashboards) provides a high-level view of project status over time, including metrics like tasks completed, annotations submitted, reviews performed, label distribution, and lead times. Users can filter by date ranges and reorder visualizations to focus on key indicators.

<img src="/images/FullDashboardPage.png" alt="" class="gif-border" />

### Annotator performance dashboards

Organization owners and administrators can view individual [annotator dashboards](https://docs.humansignal.com/guide/dashboard_annotator) to analyze task counts, average annotation times, review turnaround, and performance scoring. These dashboards help in managing resources, identifying top performers, and tracking quality without building custom analytics tools.

 

## Project Management

Label Studio Enterprise includes advanced project configuration options to support complex workflows across teams and workspaces.

### Enterprise-Only Project Settings

| Setting Category       | Option                                                                                                                                                                                        | Open Source | Enterprise |
| :--------------------- | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :---------- | :--------- |
| **General**            | **Project Name, Description, Color**                                                                                                                                                          | ✅           | ✅          |
|                        | **Workspace**                                                                                                                                                                                 | ❌           | ✅          |
|                        | **Proxy Credentials**                                                                                                                                                                         | ❌           | ✅          |
| **Labeling interface** | **Labeling configuration (XML tags)**                                                                                                                                                         | ✅           | ✅          |
| **Annotation**         | **Labeling Instructions (“Show before labeling”)**                                                                                                                                            | ✅           | ✅          |
|                        | **Distribute Labeling Tasks (Auto vs. Manual)**                                                                                                                                               | ❌           | ✅          |
|                        | **Task Reservation**                                                                                                                                                                          | ❌           | ✅          |
|                        | **Skip Queue behavior (Requeue, Ignore, etc.)**                                                                                                                                               | ❌           | ✅          |
|                        | **Annotating Options** <br />• **Show Skip button** <br />• **Allow empty annotations** <br />• **Reveal pre-annotations interactively** <br />• **Require skip comments**<br />• **Show Data Manager to annotators** | ❌           | ✅          |
| **Live Predictions**   | **Pre-label tasks via connected ML backend or Prompts**                                                                                                                                       | ✅           | ✅          |
| **Task Sampling**      | **Sequential, Uniform, Uncertainty Sampling**                                                                                                                                                 | ❌           | ✅          |
| **Review**             | **Review stream settings:** <br />• **Instructions** <br />• **Reviewing Options** <br />• **Reject Options** <br />• **Data Manager for reviewers** <br />• **Agreement column visibility**                          | ❌           | ✅          |
| **Quality**            | **Overlap of Annotations**                                                                                                                                                                    | ❌           | ✅          |
|                        | **Annotation Limit (per-user caps)**                                                                                                                                                          | ❌           | ✅          |
|                        | **Annotator Evaluation (ground truth or acceptance score & auto-pause)**                                                                                                                      | ❌           | ✅          |
|                        | **Task Agreement (metric, low-agreement strategy)**                                                                                                                                           | ❌           | ✅          |
|                        | **Custom label weights**                                                                                                                                                                      | ❌           | ✅          |
| **Members**            | **Project-level roles (Annotator/Reviewer per project)**                                                                                                                                      | ❌           | ✅          |
|                        | **Manage Members (add/remove users, assign project roles)**                                                                                                                                   | ❌           | ✅          |
| **Model**              | **Connect ML backends (fit on submit, interactive pre-annotations)**                                                                                                                          | ✅           | ✅          |
| **Predictions**        | **View/import pre-annotations**                                                                                                                                                               | ✅           | ✅          |
| **Cloud storage**      | **Source/Target connectors (S3, GCS, Azure, local)**                                                                                                                                          | ✅           | ✅          |
| **Webhooks**           | **Configure task/annotation webhooks**                                                                                                                                                        | ✅           | ✅          |
| **Danger Zone**        | **Delete Project, Drop All Tabs**                                                                                                                                                             | ✅           | ✅          |
|                        | **Reset Cache**                                                                                                                                                                               | ❌           | ✅          |


For details, view the [Enterprise Project Settings](https://docs.humansignal.com/guide/project_settings_lse).

## Annotation workflow features

### Bulk labeling

Label Studio Enterprise makes it easy to apply the same annotation to multiple tasks at once, ideal for quickly labeling filtered subsets or consistent repetitive data.

Key features:

* **Supports all major data types:** Works with images, text, audio, and video

* **Choose tasks in Data Manager:** Select multiple tasks via list or grid view, then apply bulk labeling

* **Role-based access:** Available to users with Reviewer, Manager, Admin, or Owner roles (Annotators can participate via plugins)

* **Supported control tags:** Choices, DateTime, Number, Rating, Taxonomy, TextArea


### Auto distribution
Label Studio Enterprise supports automated task assignment workflows—helping teams start labeling without manual setup.

When you enable **Auto distribution** under Annotation settings, these project behaviors apply:

* **Immediate task access:** Annotators can begin labeling as soon as the project is published
* **Configurable task overlap:** Use "Annotations per task minimum" to define how many annotators must label each task
* **Task reservation (lock timeout):** Define how long a task remains locked for an annotator before becoming available again
* **Skip queue behavior:** Choose whether skipped tasks should be requeued to others, ignored, or skipped until reassignable


These settings streamline labeling for large teams and support active learning workflows by routing tasks automatically and handling retries when tasks are skipped.

### Annotation results filtering

Label Studio Enterprise enhances task review and management with flexible filtering and sorting options in the Data Manager.

Key capabilities:
* **Field-level filtering:** Filter tasks by annotation outcomes, label values, region attributes, or reviewer actions
* **Task sorting:** Sort tasks by creation date, prediction confidence, or other metadata
* **Combined filters:** Apply multiple filters at once to narrow results for review, QA, or targeted labeling

## Collaboration features

Label Studio Enterprise supports rich collaboration capabilities to enhance communication, review workflows, and performance visibility across teams.

### Comments and notifications

Annotators and reviewers can leave threaded comments linked to specific regions or fields within an annotation, enabling contextual feedback and clarification. All collaborators can view and add comments, with permissions to resolve or delete based on their role. Notifications inform users of new comments, either as the first comment on an annotation or replies to ongoing discussions

### Deep linking

Users can generate direct links to specific annotations, regions, or fields, allowing reviewers or teammates to jump directly to the exact location needing attention. This improves communication efficiency and review precision.

## Enterprise deployment options

Label Studio Enterprise may be deployed on-prem for air-gapped environments. It supports flexible deployment options designed for high availability and scalability.

### Kubernetes deployment

Label Studio Enterprise can be deployed on Kubernetes clusters with advanced configuration options:

### Docker Compose deployment

For simpler enterprise deployments, Label Studio Enterprise supports Docker Compose with external PostgreSQL and Redis dependencies. Key configuration elements:

  * **Image repository:** `humansignal/label-studio-enterprise`
  * **License integration:** Volume-mounted license file
  * **Environment variables:** Database connections, Redis configuration, storage settings
  * **Worker queues:** Separate containers for different priority levels

### High availability configuration

Enterprise deployments support high availability configurations:

  * Multiple app replicas for fault tolerance
  * Configurable RQ worker replicas for different queues
  * Resource configuration for optimal performance
  * PostgreSQL and Redis with TLS support and high availability options

## Performance and scalability features

  * Storage Proxies
  * Optimized Database Queries: Improved performance for large projects
  * Scale-Out Architecture: Distribute workload across multiple pods
  * Redis for Background Tasks: Efficient queue system for handling asynchronous operations

## Whitelabeling

For an additional fee, Enterprise customers can customize the appearance of their Label Studio instance:

  * **Custom Branding:** Customize logos, colors, and styles
  * **Custom Domain:** Use your own domain for the Label Studio instance
  * **Login Page Redirects:** Configure custom login page redirects with the LOGIN\_PAGE\_URL variable

## Enterprise support & professional services

Enterprise customers receive:

  * 99.9% Uptime SLA: Production reliability guarantees
  * Premium Technical Support and SLAs: Priority technical support access
  * Customer Success Manager: Dedicated account management
  * Professional Services: Implementation and optimization assistance
  * Labeling Services: On-demand expert labeling services augment core team, fully integrated into the platform with full visibility and governance


See how Label Studio Enterprise can work at your organization and [contact sales](https://humansignal.com/contact-sales/).

---

### Source Guide Export (`docs/source/guide/export.md`)

---
title: Export annotations and data from Label Studio
short: Export annotations
type: guide
tier: all
order: 166
order_enterprise: 166
meta_title: Export Annotations
meta_description: Label Studio documentation for exporting data labeling annotations to use in machine learning models and data science projects.
section: "Import & Export"

---

At any point in your labeling project, you can export the annotations from Label Studio. 

Label Studio stores your annotations in a raw JSON format in the SQLite database backend, PostgreSQL database backend, or whichever cloud or database storage you specify as target storage. Cloud storage buckets contain one file per labeled task named `task_id.json`. For more information about syncing target storage, see [Cloud storage setup](storage.html).

Image annotations exported in JSON format use percentages of overall image size, not pixels, to describe the size and location of the bounding boxes. For more information, see [how to convert the image annotation units](#Units-of-image-annotations).

!!! note
    Some export formats export only the annotations and not the data from the task. For more information, see the [export formats supported by Label Studio](#Export-formats-supported-by-Label-Studio).

{% insertmd includes/annotation_ids.md %}

<div class="opensource-only">

### Export using the UI in Community Edition of Label Studio

Use the following steps to export data and annotations from the Label Studio UI. 

1. For a project, click **Export**.
2. Select an available export format.
3. Click **Export** to export your data.

!!! note
    1. The export will always include the annotated tasks, regardless of filters set on the tab. 
    2. Cancelled annotated tasks will be included in the exported result too.
    3. If you want to apply tab filters to the export, try creating [export snapshots using the SDK](https://api.labelstud.io/api-reference/api-reference/projects/exports/create).

### Export timeout in Community Edition

Exports from the Community Edition UI are generated **synchronously** as part of the request. Community Edition keeps deployment simple and does not run background export workers by default. Label Studio Enterprise supports background workers for asynchronous snapshot exports, which is better suited for large-scale projects. For large projects, the community's export can take longer than the timeout configured in your reverse proxy or ingress (often around **90 seconds**), which can result in 502/504 errors or an export timeout.

If you hit this limitation, you can still export your data using one of these options:

- **Export snapshots using the SDK**: See how to [export snapshots using the SDK](https://api.labelstud.io/api-reference/api-reference/projects/exports/create).
- **Export using the console command**: Use the [console command](#Export-using-console-command) to export your project directly from the machine running Label Studio.
- **Export in the UI at scale**: Label Studio Enterprise includes **background snapshot exports** in the UI for large datasets (see [Export snapshots using the UI](#Export-snapshots-using-the-UI)).

### Export using console command

Use the following command to export data and annotations.

```shell
label-studio export <project-id> <export-format> --export-path=<output-path>
```

To enable logs: 
```shell
DEBUG=1 LOG_LEVEL=DEBUG label-studio export <project-id> <export-format> --export-path=<output-path>
```

</div>

<div class="enterprise-only">

### Export snapshots using the UI

<img src="/images/lse-export-snapshots-ui.png" alt="" class="gif-border" />
<br>

In Label Studio Enterprise, create a snapshot of your data and annotations. Create a snapshot to export exactly what you want from your data labeling project. This delayed export method makes it easier to export large labeling projects from the Label Studio UI.  

1. Within a project in the Label Studio UI, click **Export**.
2. Click **Create New Snapshot**.
3. **Apply filters from tab ...**: Select **Default** from the drop-down list. 
4. (Optional) **Snapshot Name**: Enter a snapshot name to make it easier to find in the future. By default, export snapshots are named `PROJECT-NAME-at-YEAR-MM-DD-HH-MM`, where the time is in UTC.
5. **Include in the Snapshot…**: Choose which type of data you want to include in the snapshot. Select **All tasks**, **Only annotated** tasks, or **Only reviewed** tasks. 
6. **Drafts**: Choose whether to export the complete draft annotations (**Complete drafts**) for tasks, or only the IDs (**Only IDs**) of draft annotations, to indicate that drafts exist. 
7. **Predictions**: Choose whether to export the complete predictions (**Complete predictions**) for tasks, or only the IDs (**Only IDs**) of predictions to indicate that the tasks had predictions.
8. **Annotations**: Enable the types of annotations that you want to export. You can specify **Annotations**, **Ground Truth** annotations, and **Skipped** annotations. By default, only annotations are exported.
9. (Optional) Enable the **Remove user details** option to remove the user's details. 
10. Click **Create a Snapshot** to start the export process.
11. You see the list of snapshots available to download, with details about what is included in the snapshot, when it was created, and who created it. 
12. Click **Download** and select the export format that you want to use. Now, the snapshot file downloads to your computer. 

</div>

### Export using the Easy Export API

You can call the Label Studio API to export annotations. For a small labeling project, call the [export endpoint](https://api.labelstud.io/api-reference/api-reference/projects/exports/download-sync) to export annotations.


#### Export all tasks including tasks without annotations

Label Studio open source exports tasks with annotations only by default. If you want to easily export all tasks including tasks without annotations, you can call  the [Easy Export API](https://api.labelstud.io/api-reference/api-reference/projects/exports/download-sync) with query param `download_all_tasks=true`. For example:
```
curl -X GET https://localhost:8080/api/projects/{id}/export?exportType=JSON&download_all_tasks=true
``` 

If your project is large, you can use a [snapshot export](https://api.labelstud.io/api-reference/api-reference/projects/exports/create) to avoid timeouts in most cases. Snapshots include all tasks without annotations by default.


### Export snapshots using the Snapshot API 

For a large labeling project with hundreds of thousands of tasks, do the following:
1. Make a POST request to [create a new export file or snapshot](https://api.labelstud.io/api-reference/api-reference/projects/exports/create). The response includes an `id` for the created file.
2. [Check the status of the export file created](https://api.labelstud.io/api-reference/api-reference/projects/exports/get) using the `id` as the `export_pk`. 
3. Using the `id` from the created snapshot as the export primary key, or `export_pk`, make a GET request to [download the export file](https://api.labelstud.io/api-reference/api-reference/projects/exports/download).


## Export formats supported by Label Studio

Label Studio supports many common and standard formats for exporting completed labeling tasks. If you don't see a format that works for you, you can contribute one. For more information, see the [Label Studio Converter tool in our SDK repo](https://github.com/HumanSignal/label-studio-sdk/tree/master/src/label_studio_sdk/converter).

### ASR_MANIFEST

Export audio transcription labels for automatic speech recognition as the JSON manifest format expected by [NVIDIA NeMo models](https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/stable/core/core.html). Supports audio transcription labeling projects that use the `Audio` tag with the `TextArea` tag.

```json
{“audio_filepath”: “/path/to/audio.wav”, “text”: “the transcription”, “offset”: 301.75, “duration”: 0.82, “utt”: “utterance_id”, “ctm_utt”: “en_4156”, “side”: “A”}
```

### Brush labels to NumPy and PNG

Export your brush mask labels as NumPy 2d arrays and PNG images. Each label outputs as one image. Supports brush labeling image projects that use the `BrushLabels` tag.

### COCO

A popular machine learning format used by the [COCO dataset](http://cocodataset.org/#home) for object detection and image segmentation tasks. Supports bounding box and polygon image labeling projects that use the `BrushLabels`, `RectangleLabels`, `KeyPointLabels` (see note below), or `PolygonLabels` tags.


{% details <b>KeyPointLabels Export Support</b> %}

If using `KeyPointLabels`, you will need to add the following to your labeling config:

* At least one `<RectangleLabels>` option. You will use this as a parent bounding box for the keypoints. 
* Add a `model_index` to every `<Label>` inside your `<KeyPointLabels>` tag. The `model_index` value defines the order of the keypoint coordinates in the output array for YOLO. 

For example:

```xml
<View>
  <Image name="image" value="$image"/>

  <KeyPointLabels name="kp" toName="image">
    <Label value="nose" model_index="0"/>
    <Label value="eye" model_index="1"/>
    <Label value="tail" model_index="2"/>
  </KeyPointLabels>

  <RectangleLabels name="bbox" toName="image">
    <Label value="animal"/>
  </RectangleLabels>
</View>

```

After annotating, you must drag-and-drop each keypoint region under its corresponding rectangle region in the **Regions** panel. 

This establishes a parent–child hierarchy (via parentID), which is necessary for export. See the export examples below. 



**Export examples**

<div class="code-tabs">
  <div data-name="Keypoints in JSON">
```json
[
      {
        "result": [
          {
            "id": "17n06ubOJs",
            "type": "keypointlabels",
            "value": {
              "x": 6.675567423230974,
              "y": 20.597014925373134,
              "width": 0.26702269692923897,
              "keypointlabels": [
                "nose"
              ]
            },
            "origin": "manual",
            "to_name": "image",
            "parentID": "QHG4TBXuNC",
            "from_name": "kp",
            "image_rotation": 0,
            "original_width": 200,
            "original_height": 179
          },
          {
            "id": "QHG4TBXuNC",
            "type": "rectanglelabels",
            "value": {
              "x": 3.871829105473965,
              "y": 4.029850746268656,
              "width": 94.39252336448598,
              "height": 92.08955223880598,
              "rotation": 0,
              "rectanglelabels": [
                "animal"
              ]
            },
            "origin": "manual",
            "to_name": "image",
            "from_name": "bbox",
            "image_rotation": 0,
            "original_width": 200,
            "original_height": 179
          }
]
```
  </div>
  <div data-name="Keypoints in COCO">
```json
[
    {
      "id": 0,
      "image_id": 0,
      "category_id": 0,
      "segmentation": [],
      "bbox": [
        7.74365821094793,
        7.213432835820895,
        188.78504672897196,
        164.84029850746268
      ],
      "ignore": 0,
      "iscrowd": 0,
      "area": 31119.38345654903
    },
    {
      "id": 1,
      "image_id": 0,
      "category_id": 0,
      "keypoints": [
        13,
        37,
        2,
        33,
        33,
        2,
        167,
        24,
        2
      ],
      "num_keypoints": 3,
      "bbox": [
        13,
        24,
        154,
        13
      ],
      "iscrowd": 0
    }
]
```
  </div>
  <div data-name="Keypoints in YOLO">
```
0 0.5106809078771696 0.5007462686567165 0.9439252336448598 0.9208955223880598 0.06675567423230974 0.20597014925373133 2 0.1628838451268358 0.18507462686567164 2 0.8371161548731643 0.13134328358208955 2
```
  </div>
</div>
{% enddetails %}


### CoNLL2003

A popular format used for the [CoNLL-2003 named entity recognition challenge](https://www.clips.uantwerpen.be/conll2003/ner/). Supports text labeling projects that use the `Text` and `Labels` tags.

### CSV

Results are stored as comma-separated values with the column names specified by the values of the `"from_name"` and `"to_name"` fields in the labeling configuration. Supports all project types.

### JSON

List of items in [raw JSON format](#Label-Studio-JSON-format-of-annotated-tasks) stored in one JSON file. Use this format to export both the data and the annotations for a dataset. Supports all project types.

### JSON_MIN

List of items where only `"from_name", "to_name"` values from the [raw JSON format](#Label-Studio-JSON-format-of-annotated-tasks) are exported. Use this format to export the annotations and the data for a dataset, and no Label-Studio-specific fields. Supports all project types.

For example: 
```json
{
  "image": "https://htx-pub.s3.us-east-1.amazonaws.com/examples/images/nick-owuor-astro-nic-visuals-wDifg5xc9Z4-unsplash.jpg",
  "tag": [{
    "height": 10.458911419423693,
    "rectanglelabels": [
        "Moonwalker"
    ],
    "rotation": 0,
    "width": 12.4,
    "x": 50.8,
    "y": 5.869797225186766
  }]
}
```

### Pascal VOC XML

A popular XML-formatted task data is used for object detection and image segmentation tasks. Supports bounding box image labeling projects that use the `RectangleLabels` tag.

### spaCy 

Label Studio does not support exporting directly to spaCy binary format, but you can convert annotations exported from Label Studio to a format compatible with spaCy. You must have the spacy python package installed to perform this conversion. 

To transform Label Studio annotations into spaCy binary format, do the following:
1. Export your annotations to CONLL2003 format.
2. Open the downloaded file and update the first line of the exported file to add `O` on the first line:
```
-DOCSTART- -X- O O
```
3. From the command line, run spacy convert to convert the CoNLL-formatted annotations to spaCy binary format, replacing `/path/to/<filename>` with the path and file name of your annotations:

    spacy version 2:
    ```shell
    spacy convert /path/to/<filename>.conll -c ner
    ```
    spacy version 3:
    ```shell
    spacy convert /path/to/<filename>.conll -c conll . 
    ```

    For more information, see the spaCy documentation on [Converting existing corpora and annotations](https://spacy.io/usage/training#data-convert) on running spacy convert.

### TSV

Results are stored in a tab-separated tabular file with column names specified by `"from_name"` and `"to_name"` values in the labeling configuration. Supports all project types.

### YOLO

Export object detection annotations in the YOLOv3 and YOLOv4 format. Supports object detection labeling projects that use the `RectangleLabels`  and `KeyPointLabels` tags. 

!!! note
    If using KeyPointLabels, see the note under [COCO](#COCO).

{% insertmd includes/task_format.md %}

{% insertmd includes/image_units.md %}

## Manually convert JSON annotations to another format
You can run the [Label Studio converter tool](https://github.com/HumanSignal/label-studio-sdk/tree/master/src/label_studio_sdk/converter) on a directory or file of completed JSON annotations using the command line or Python to convert the completed annotations from Label Studio JSON format into another format. 

!!! note
    If you use versions of Label Studio earlier than 1.0.0, then this is the only way to convert your Label Studio JSON format annotations into another labeling format. 


## Access task data (images, audio, texts) outside of Label Studio for ML backends

Machine Learning backend uses data from tasks for predictions, and you need to download them on Machine Learning backend side. Label Studio provides tools for downloading of these resources, and they are located in label-studio-tools Python package. If you are using official Label Studio Machine Learning backend, label-studio-tools package is installed automatically with other requirements.

### Accessing task data from Label Studio instance

There are several ways of storing tasks resources (images, audio, texts, etc) in Label Studio:
- Cloud storages 
- External web links 
- Uploaded files
- Local files directory

Label Studio stores uploaded files in Project level structure. Each project has it's own folder for files.

You can use `label_studio_tools.core.utils.io.get_local_path` to get task data - it will transform path or URL from task data to local path.
In case of local path it will return full local path and download resource in case of using `download_resources` parameter.

Provide `Hostname` and `access_token` for accessing external resource.

### Accessing task data outside of Label Studio instance

You can use `label_studio_tools.core.utils.io.get_local_path` method to get data from outside machine for external links and cloud storages. 

!!! attention "important"
    Don't forget to provide credentials.

You can get data with `label_studio_tools.core.utils.io.get_local_path` in case if you mount same disk to your machine. If you mount same disk to external box 

Another way of accessing data is to use link from task and ACCESS_TOKEN ([see documentation for authentication](access_tokens)). Concatenate Label Studio hostname and link from task data. Then add access token to your request:

```json
curl -X GET http://localhost:8080/api/projects/ -H 'Authorization: Token {YOUR_TOKEN}'
```

### Frequently asked questions 

#### Question #1: I have made a request and received the following API responses: 
- No data was provided.
- 404 or 403 error code was returned. 

**Answer:**
First check the network access to your Label Studio instance when you send API requests. You can execute test curl request with sample data. 

#### Question #2: I tried to access files and received a `FileNotFound` error.

**Answer:**
1. Check that you have mounted the same disk as your Label Studio instance. Then check your files' existence in Label Studio instance first. 

2. Check `LOCAL_FILES_DOCUMENT_ROOT` environment variable in your Label Studio instance and add it to your accessing data script.


#### Question #3: How to modify order of categories for COCO and YOLO exports? 

Labels are sorted in alphabetical order, that is default behavior. If you want to modify that, please add **category** attribute in `<Label>` to modify that behaviour. For example: 
    
```xml
<Label value="abc" category="1" />
<Label value="def" category="2" />
```

---

### Source Guide Frontend (`docs/source/guide/frontend.md`)

---
title: Frontend builds
short: Frontend builds
tier: all
order: 0
order_enterprise: 0
hide_menu: true
meta_title: Customize User Interface
meta_description: Label Studio documentation for integrating the Label Studio frontend interface into your own machine learning or data labeling application workflow.
section: "Integrate & Extend"

---

!!! attention
    As of [Label Studio 1.11.0](https://github.com/HumanSignal/label-studio/releases/tag/1.11.0), the Label Studio frontend has been deprecated as a separate library and is no longer supported as a standalone distribution. For information about using the frontend library within Label Studio, see [the README](https://github.com/HumanSignal/label-studio/blob/develop/web/libs/editor/README.md).

    Rather than building a new labeling application from scratch, AI agents can build powerful labeling interfaces on top of Label Studio by combining the [XML labeling config builder skill](https://github.com/HumanSignal/create-xml-labeling-config-skill) with the [Label Studio SDK/CLI](https://api.labelstud.io/api-reference/introduction/getting-started). This lets agents generate optimized labeling configs directly from your specs and wire them into projects programmatically.

The [Label Studio Frontend](https://github.com/HumanSignal/label-studio-frontend) (LSF) is the main labeling interface distributed within Label Studio and as a separate package via NPM and Unpkg. You can integrate the LSF into your projects without Label Studio to provide data labeling capabilities to your users.

LSF can be customized and extended to fit your needs, and you can use a custom version of LSF standalone or in your LS installation. For more information see [Custom LSF + LS integration](#custom-lsf-in-label-studio).

LSF is located as a separate GitHub repository: https://github.com/HumanSignal/label-studio-frontend

<br>
<img src="/images/frontend/lsf-in-ls.jpg" class="gif-border">

<i>Figure 1: Label Studio Frontend </i>


## Installation

There are two ways to install the LSF as follows:

1. Using the package manager
2. Using the Unpkg CDN

### Using a package manager (recommended)
You can either use `npm` or `yarn` to install the LSF.

```bash
npm install heartexlabs@label-studio@latest --save
```

```bash
yarn add heartexlabs@label-studio@latest
```

### Using Unpkg.com CDN
```xhtml

<link href="https://unpkg.com/heartexlabs@label-studio@latest/build/static/css/main.css" rel="stylesheet">


<script src="https://unpkg.com/heartexlabs@label-studio@latest/build/static/js/main.js"></script>
```

## Frontend integration guide

The LSF can be used with Vanilla JS or with the framework of your choice. The following examples cover basic integration with Vanilla and React.

### Vanilla JS integration
You can use the Label Studio Frontend separately in your own projects by including it in your HTML page. Instantiate a new Label Studio object with a selector for the div that should become the editor.

To see all the available options for the initialization of LabelStudio object, see the [Label Studio Frontend](frontend_reference.html).

{% collapse "Using modern JS techniques (recommended)" %}

This guide assumes that you're using a bundler like Webpack or Rollup to assemble your JS bundles, and LSF is installed via a package manager.

In your HTML add the following code:
```xhtml
<div id="label-studio"></div>
```

Now to initialize the Label Studio Frontend, add the following code to your JS file:
``` js
import LabelStudio from 'heartexlabs@label-studio@latest';
import 'heartexlabs@label-studio@latest/build/static/css/main.css';

const labelStudio = new LabelStudio('label-studio', {
  config: `
    <View>
      <Image name="img" value="$image"></Image>
      <RectangleLabels name="tag" toName="img">
        <Label value="Hello"></Label>
        <Label value="World"></Label>
      </RectangleLabels>
    </View>
  `,
  interfaces: [
    "panel",
    "update",
    "controls",
    "side-column",
    "annotations:menu",
    "annotations:add-new",
    "annotations:delete",
    "predictions:menu"
  ],
  user: {
    pk: 1,
    firstName: "James",
    lastName: "Dean"
  },
  task: {
    annotations: [],
    predictions: [],
    id: 1,
    data: {
      image: "https://htx-pub.s3.us-east-1.amazonaws.com/examples/images/nick-owuor-astro-nic-visuals-wDifg5xc9Z4-unsplash.jpg"
    }
  }
});

labelStudio.on("labelStudioLoad", (LS) => {
  // Perform an action when Label Studio is loaded
  const c = LS.annotationStore.addAnnotation({
    userGenerate: true
  });
  LS.annotationStore.selectAnnotation(c.id);
});

labelStudio.on("submitAnnotation", (LS, annotation) => {
  // Retrieve an annotation in JSON format
  console.log(annotation.serializeAnnotation())
});
```
{% endcollapse %}

{% collapse "Using plain HTML and JS" %}

This technique is useful if you're not using a bundler or if you want to use the LSF in a static HTML page.

```xhtml

<link href="https://unpkg.com/heartexlabs@label-studio@latest/build/static/css/main.css" rel="stylesheet">

<div id="label-studio"></div>


<script src="https://unpkg.com/heartexlabs@label-studio@latest/build/static/js/main.js"></script>

<script>
const root = document.querySelector('#label-studio');
const labelStudio = new LabelStudio(root, {
  // all configuration options are the same
});
</script>
```
{% endcollapse %}

### React integration
LSF is flexible and can be used with a framework of choice. This guide covers the React integration but the same principles can be applied to other frameworks.

{% collapse "LSF with React" %}
Prepare new custom component:
```jsx
// components/LabelStudio.js
import {useRef} from 'react';

const LabelStudioReact = (props) => {
  const labelStudioContainerRef = useRef();
  const labelStudioRef = useRef();

  useEffect(() => {
    if (labelStudioContainerRef.current) {
      labelStudioRef.current = new LabelStudio(
        labelStudioContainerRef.current,
        props
      );
    }
  }, []);

  return (
    <div
      id="label-studio"
      ref={function(el) {
        labelStudioContainerRef.current = el
      }}
    />
  );
}
```

Use the component in your React application
```jsx
// App.js
import { render } from 'react-dom';
import LabelStudioReact from './components/LabelStudio';

const labelinConfig = `
  <View>
    <Image name="img" value="$image"></Image>
    <RectangleLabels name="tag" toName="img">
      <Label value="Hello"></Label>
      <Label value="World"></Label>
    </RectangleLabels>
  </View>
`

const task = {
  annotations: [],
  predictions: [],
  id: 1,
  data: {
    image: "https://htx-pub.s3.us-east-1.amazonaws.com/examples/images/nick-owuor-astro-nic-visuals-wDifg5xc9Z4-unsplash.jpg"
  }
}

const App = () => {
  return (
    <div class="app-root">
      <LabelStudioReact
        config={labelinConfig}
        task={task}
        interfaces={[
          "panel",
          "update",
          "controls",
          "side-column",
          "annotations:menu",
          "annotations:add-new",
          "annotations:delete",
          "predictions:menu"
        ]}
        user={{
          pk: 1,
          firstName: "James",
          lastName: "Dean"
        }}
      />
    </div>
  )
}

render(<App />, document.getElementById('root'));
```
{% endcollapse %}


## Frontend development

Refer to the [Frontend Reference](frontend_reference.html) when developing with Label Studio Frontend.

### Manual builds

If you want to build a new tag or change the behaviour of default components inside of LSF, then you need to go into the LSF repo and review the [Development part](https://github.com/HumanSignal/label-studio-frontend#development) of the README file. Making any changes requires that you have a good knowledge of React and Javascript.build.js `<branch-name-from-official-lsf-repo>`

### GitHub Artifacts

Use GitHub Artifacts to download a zip-formatted archive with LSF builds. Branches from the official LSF repo are built automatically and hosted on GitHub Artifacts.

See the [GitHub Actions for the LSF repository](https://github.com/HumanSignal/label-studio-frontend/actions) to access them.

You can also configure a GitHub token to obtain artifacts automatically:
```
export GITHUB_TOKEN=<token>
cd label-studio/frontend
yarn download:lsf <branch-name-from-official-lsf-repo>
```


<div class="opensource-only">

## Custom LSF in Label Studio

LS Frontend (LSF) with Backend (LSB) integration is similar what is described in the [Frontend integration guide](#Frontend-integration-guide). The JavaScript integration script is placed in [lsf-sdk.js](https://github.com/HumanSignal/label-studio/blob/develop/label_studio/static/js/lsf-sdk.js) in the Label Studio Backend. The main idea of this integration based on LSF callbacks.

Check out a quick guide on how to use custom LSF in Label Studio.

<iframe width="800" height="500" src="https://www.youtube.com/embed/QSGgiXie2SE" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>

### Prepare a custom LSF build
1. Make your custom LSF build by following these [instructions](https://github.com/HumanSignal/label-studio-frontend#development).
2. Finalize your development with `npm run build:module` to generate `main.css` and `main.js` files.
3. Confirm files appearance in the `build/static` folder.

### Integrate LSF into Label Studio
All frontend-related files are stored under `label-studio/frontend` directory. You will also find `dist/` folder there that contains the latest builds of the external frontend libraries.

Under `dist/` folder locate the `lsf/` folder and replace its contents with your custom LSF build.

!!! note
    Inside every folder under `dist/` you will find a `version.json` file. Do not modify or remove it. Its presence is required for the Label Studio to operate.

1. **Do not forget** to remove the old build from LSB:
    ```bash
    rm -rf label-studio/label_studio/frontend/dist/lsf/{js,css}
    ```

2. Copy build folder from LSF to LS:
    ```bash
    cp -r label-studio-frontend/build/static/{js,css} label-studio/label_studio/frontend/dist/lsf/
    ```

    If you installed LS as a pip package, replace `<env-path>/lib/python<version>/site-packages/label_studio/frontend/dist/lsf/`

3. Run the LS instance as usual. It is now using the new LSF build:
    ```bash
    label-studio start <your-project>
    ```

</div>

---

### Source Guide Frontend Reference (`docs/source/guide/frontend_reference.md`)

---
title: Frontend reference
short: Frontend
type: guide
tier: all
order: 416 
order_enterprise: 416
meta_title: Frontend Library Reference
meta_description: Reference documentation for implementing the Label Studio frontend into your own machine learning or data science application workflows.
section: "Integrate & Extend"
---

Label Studio frontend (LSF) includes several UI options and callbacks that you can use when implementing the frontend with a custom labeling backend, or when customizing the Label Studio interface.

!!! attention
    As of [Label Studio 1.11.0](https://github.com/HumanSignal/label-studio/releases/tag/1.11.0), the Label Studio frontend has been deprecated as a separate library and is no longer supported as a standalone distribution. For information about using the frontend library within Label Studio, see [the README](https://github.com/HumanSignal/label-studio/blob/develop/web/libs/editor/README.md). 

## Updates to LSF in version 1.0.0

!!! warning
    LSF version 1.0.0 is not compatible with earlier versions of Label Studio.

If you use LSF with a custom backend, you must make changes to the API callbacks that you use as follows:

| Callback in 0.9.1 and earlier | Renamed callback in 1.0.0 |
| ----------------------------- | ------------------------- |
| onSubmitCompletion            | onSubmitAnnotation        |
| onUpdateCompletion            | onUpdateAnnotation        |
| onDeleteCompletion            | onDeleteAnnotation        |

If you rely on specific formatting of Label Studio completed tasks, [Label Studio's annotation format](export.html#Raw-JSON-format-of-completed-tasks) has also been updated.

## Implement the Label Studio Frontend

```javascript
var labelStudio = new LabelStudio("editor", options);
```

The following options are recognized when initializing a Label Studio instance version 1.0.0.

## Options

### config

Default: `null`

Type data: `string`

XML-based configuration of the labeling interface. This configuration relies on the `data` field of the task.
See [Customizable Tags](/tags) for more information.

### interfaces

Default: `null`

Type data: `array`

Collection of UI elements to show. Available interfaces:

- `panel` - Enable navigaion panel for the current task with buttons: undo, redo and reset.
- `update` - Show a button to update the current task after submitting.
- `submit` - Show a button to submit or update the current annotation.
- `skip` - Show a button to skip the current task.
- `controls` - Enable panel with controls (`submit`, `update`, `skip`).
- `infobar` - A show button for information.
- `topbar` - A labeling interface that lists the top-level items in the Label Studio UI.
- `instruction` - A button for the [instructions](#description).
- `side-column` - Show a column on the left or right side of the Label Studio UI.
- `annotations:history` - A show button for annotation history.
- `annotations:tabs` - A show button for annotation tabs.
- `annotations:menu` - A show button for the annotation menu.
- `annotations:current` - A show button for the current annotation.
- `annotations:add-new` - A show button to add new annotations.
- `annotations:delete` - A show button to delete the current annotation.
- `annotations:view-all` - A show button to view all annotations.
- `predictions:tabs` - Show predictions tabs.
- `predictions:menu` - Show predictions menu.
- `auto-annotation` - Show auto annotations.
- `edit-history` - Show edit history.

### messages

Default: `null`

Type data: `object`

Messaging used for different actions

```javascript
{
  DONE: "Done!",
  NO_COMP_LEFT: "No more annotations",
  NO_NEXT_TASK: "No more data available for labeling",
  NO_ACCESS: "You don't have access to this task"
}
```

- `DONE` - Shown after the task is submitted to the server
- `NO_COMP_LEFT` - Shown if there are no more annotations
- `NO_NEXT_TASK` - No next task to load
- `NO_ACCESS` - Can't access the provided task

### description

Default: `No description`

Type data: `string`

Description of the current task.

### task

Task data

Default: `null`

Type data: `object`

```json
{
  id: 1,
  load: false
  },
  data: {
    text: "Labeling text..."
  },
  annotations: [],
  predictions: [],
}
```

#### id

Type data: `integer`

Default: `null`

#### data

#### annotations

Type data: `array`

Array of annotations. See the [annotation documentation](export.html#Raw-JSON-format-of-completed-tasks) for more information.

#### predictions

Type data: `array`

Array of predictions. Similar structure as completions or annotations. See the [annotation documentation](export.html#Raw-JSON-format-of-completed-tasks) and [guidance for importing predicted labels](predictions.html) for more information.

### user

User data

Type data: `object`

```json
{
  "pk": 1,
  "firstName": "Stanley",
  "lastName": "Kubrick"
}
```

#### pk

Type data: `number`

#### firstName

Type data: `string`

#### lastName

Type data: `string`

## Event system

LSF has a built-in event system that allows you to listen to events and trigger custom actions. You can subscribe to or unsubscribe from event at any time after the Label Studio instance is initialized.

### Using events

#### Subscribe to an event

```javascript
const callback = () => {
  console.log("Event triggered");
};
labelStudio.on("event", callback);
```

#### Unsubscribe from an event

```javascript
const callback = () => {
  console.log("Event triggered");
};
labelStudio.off("event", callback);
```

!!! note
    To be able to unsubscribe from an event, you must pass the same callback function reference to the `off` method.

## Available events

#### Top-level events

This events group contains top-level events. Those events are not related to any internal entities of the LSF.

### `labelStudioLoad`

Label Studio instance is loaded. 

!!! warning
    This event does not work in plugins.

**Event handler arguments**

| Argument      | Type     | Description              |
| ------------- | -------- | ------------------------ |
| `labelStudio` | `Object` | Instance of Label Studio |

### `storageInitialized`

The internal storage is initialized.

!!! warning
    This event does not work in plugins.

**Event handler arguments**

| Argument      | Type     | Description              |
| ------------- | -------- | ------------------------ |
| `labelStudio` | `Object` | Instance of Label Studio |

#### Task events

This events group contains events related to the task.

### `skipTask`

User clicked the "Skip" button.

**Event handler arguments**

| Argument      | Type     | Description                                 |
| ------------- | -------- | ------------------------------------------- |
| `labelStudio` | `Object` | Instance of Label Studio                    |
| `payload`     | `Object` | Additional data sent during the skip action |

### `unskipTask`

User clicked the "Cancel Skip" button.

| Argument      | Type     | Description              |
| ------------- | -------- | ------------------------ |
| `labelStudio` | `Object` | Instance of Label Studio |

### `nextTask`

User clicked the "Next" (chevron right) button.

| Argument       | Type      | Description                                    |
| -------------- | --------- | ---------------------------------------------- |
| `labelStudio`  | `Object`  | Instance of Label Studio                       |
| `taskId`       | `Number?` | ID of the next task in history                 |
| `annotationId` | `Number?` | ID of the annotation to select within the task |

### `prevTask`

User clicked the "Previous" (chevron left) button.

| Argument       | Type      | Description                                    |
| -------------- | --------- | ---------------------------------------------- |
| `labelStudio`  | `Object`  | Instance of Label Studio                       |
| `taskId`       | `Number?` | ID of the previous task in history             |
| `annotationId` | `Number?` | ID of the annotation to select within the task |

### `submitDraft`

Draft is sent to the server.

| Argument      | Type      | Description                      |
| ------------- | --------- | -------------------------------- |
| `labelStudio` | `Object`  | Instance of Label Studio         |
| `annotation`  | `Object`  | Current annotation               |
| `params`      | `Object?` | Extra params sent with the draft |

#### Annotation events

This events group contains events related to the annotation.

### `beforeSaveAnnotation`
Annotation is going to be saved as the result of the `submit` or `update` action. 
Returning `false` from this event will prevent saving the annotation.

| Argument         | Type     | Description                                                                                  |
|------------------|----------|----------------------------------------------------------------------------------------------|
| `labelStudio`    | `Object` | Instance of Label Studio                                                                     |
| `annotation`     | `Object` | Current annotation                                                                           |
| `payload`        | `Object` | Additional information                                                                       |
| `payload.event`  | `string` | Indicates which event is about to be executed (`submitAnnotation`, `updateAnnotation`, etc.) |


### `submitAnnotation`

Annotation is submitted.

| Argument      | Type     | Description              |
| ------------- | -------- | ------------------------ |
| `labelStudio` | `Object` | Instance of Label Studio |
| `annotation`  | `Object` | Current annotation       |

### `updateAnnotation`

Annotation is updated.

| Argument      | Type     | Description              |
| ------------- | -------- | ------------------------ |
| `labelStudio` | `Object` | Instance of Label Studio |
| `annotation`  | `Object` | Current annotation       |

### `selectAnnotation`

Annotation is selected.

| Argument              | Type      | Description                                   |
|-----------------------|-----------|-----------------------------------------------|
| `annotation`          | `Object`  | Current annotation                            |
| `previousAnnotation`  | `Object`  | Previous annotation                           |
| `payload`             | `Object?` | Additional information                        |
| `payload.fromViewAll` | `boolean` | `true` if ViewAll has just been switched off  |


### `deleteAnnotation`

Annotation is deleted.

| Argument      | Type     | Description              |
| ------------- | -------- | ------------------------ |
| `labelStudio` | `Object` | Instance of Label Studio |
| `annotation`  | `Object` | Current annotation       |

### `groundTruth`

Annotation is set as Ground Truth (the star button clicked).

| Argument         | Type      | Description                      |
| ---------------- | --------- | -------------------------------- |
| `store`          | `Object`  | Instance of Label Studio         |
| `labelStudio`    | `Object`  | Instance of Label Studio         |
| `params`         | `Object`  |                                  |
| `params.isDirty` | `Boolean` | `true` if annotation was changed |
| `params.entity`  | `Object`  | Current annotation               |

### `selectHistory`
Step in the annotation history is selected.

| Argument      | Type     | Description                 |
|---------------|----------|-----------------------------|
| `labelStudio` | `Object` | Instance of Label Studio    |
| `annotation`  | `Object` | Current annotation          |
| `historyItem` | `Object` | Current history item        |

#### Region events

This events group contains events related to the regions. Regions are the special entities used in segmentation tasks like image segmentation, audio segmentation, etc.

### `entityCreate`

Region is created.

| Argument | Type     | Description          |
| -------- | -------- | -------------------- |
| `region` | `Object` | Newly created region |

### `entityDelete`

Region is deleted.

| Argument | Type     | Description          |
| -------- | -------- | -------------------- |
| `region` | `Object` | Newly created region |

{% collapse "Callbacks (deprecated)" %}

Callbacks can be used to execute actions based on user interaction with the interface. For example, label-studio server uses callbacks to communicate with an API. Pass them along with other options when initiating the instance.

### onSubmitAnnotation

Type data: `function`

Called when the `submit` button is pressed. `ls` is label studio instance, `annotation` is the value of the current annotation.

#### Example

```javascript
onSubmitAnnotation: function(ls, annotation) {
  console.log(annotation)
}
```

### onUpdateAnnotation

Type data: `function`

Called when the `update` button is pressed. `ls` is label studio instance, `annotation` is the value of the current annotation.

#### Example

```javascript
onUpdateAnnotation: function(ls, annotation) {
  console.log(result)
}
```

### onDeleteAnnotation

Type data: `function`

Called when the `delete` button is pressed. `ls` is label studio instance, `annotation` is value of current annotation.

#### Example

```javascript
onDeleteAnnotation: function(ls, annotation) {
  console.log(result)
}
```

### onEntityCreate

Type data: `function`

Called when a new region gets labeled, for example, a new bbox is created. `region` is the object that was created.

#### Example

```javascript
onEntityCreate: function(region) {
  console.log(region)
}
```

### onEntityDelete

Type data: `function`

Called when an existing region gets deleted. `region` is the object itself.

#### Example

```javascript
onEntityDelete: function(region) {
  console.log(region)
}
```

### onSkipTask

Type data: `function`

Called when the `skip` button is pressed. `ls` is label studio instance.

#### Example

```javascript
onSkipTask: function(ls) {
  console.log(result)
}
```

### onLabelStudioLoad

Type data: `function`

Called when Label Studio has fully loaded and is ready for labeling. `ls` is the label studio instance

#### Example

```javascript
onLabelStudioLoad: function(ls) {
  console.log(result)
}
```

{% endcollapse %}

---

### Source Guide Get Started (`docs/source/guide/get_started.md`)

---
title: Label Studio overview
short: About Label Studio
type: guide
tier: all
order: 2
order_enterprise: 2
section: "Discover & Learn"
meta_title: Overview of Label Studio
meta_description: Get started with Label Studio by creating projects to label and annotate data for machine learning and data science models.
---

## What is Label Studio?

Label Studio is an open source data labeling tool that supports multiple projects, users, and data types in one platform. It allows you to do the following:

- Perform different types of labeling with many data formats.

- Integrate Label Studio with machine learning models to supply predictions for labels (pre-labels), or perform continuous active learning. See [Set up machine learning with your labeling process](ml).

<div class="enterprise-only">

- Use [Label Studio Enterprise as a cloud offering](https://humansignal.com/platform/).

</div>

<div class="opensource-only">

Label Studio is also available an [Enterprise cloud service](https://humansignal.com/platform/) with enhanced security (SSO, RBAC, SOC2), team management features, data discovery, analytics and reporting, and support SLAs. A [free trial is available](https://humansignal.com/free-trial) to get started quickly and explore the enterprise cloud product.

</div>

## Interface

<div class="opensource-only">

<br>
<center><i>Project List Screenshot</i></center>
<img class="make-intense-zoom" src="/images/terms/os/projects-min.png">
<br><br>
<center><i>Data Manager Screenshot</i></center>
<img class="make-intense-zoom" src="/images/terms/os/project--data-manager-min.png">
<br><br>
<center><i>Quick View Screenshot</i></center>
<img class="make-intense-zoom" src="/images/terms/os/project--data-manager--quick-view-min.png">

</div>

<div class="enterprise-only">

<br>
<center><i>Project List Screenshot</i></center>
<img class="make-intense-zoom" src="/images/terms/ent/workspace-min.png">
<br><br>
<center><i>Data Manager Screenshot</i></center>
<img class="make-intense-zoom" src="/images/terms/ent/project--data-manager-min.png">
<br><br>
<center><i>Quick View Screenshot</i></center>
<img class="make-intense-zoom" src="/images/terms/ent/project--data-manager--quick-view-min.png">

</div>



## Labeling workflow

Start and finish a labeling project with Label Studio by following these steps:

<div class="opensource-only">

1. [Install Label Studio](install.html).
2. [Start Label Studio](start.html).
3. [Create accounts for Label Studio](signup.html). Create an account to manage and set up labeling projects.
4. [Set up the labeling project](setup_project.html). Define the type of labeling to perform on the dataset and configure project settings.
5. [Set up the labeling interface](setup.html). Add the labels that you want annotators to apply and customize the labeling interface.
6. [Import data as labeling tasks](tasks.html).
7. [Label and annotate the data](labeling.html).
8. [Export the labeled data or the annotations](export.html).

</div>

<div class="enterprise-only">

1. [Create accounts for Label Studio](manage_users.html#Signup). Create an account to manage and set up labeling projects.
2. [Restrict access to the project](manage_users.html). Set up role-based access control. Only available in Label Studio Enterprise Edition.
3. [Set up the labeling project](setup_project.html). Define the type of labeling to perform on the dataset and configure project settings.
4. [Set up the labeling interface](setup.html). Add the labels that you want annotators to apply and customize the labeling interface.
5. [Import data as labeling tasks](tasks.html).
6. [Label and annotate the data](labeling.html).
7. [Review the annotated tasks](quality.html). Only available in Label Studio Enterprise Edition.
8. [Export the labeled data or the annotations](export.html).

</div>

## Label Studio citations

If you would like to cite Label Studio, you can add the following information to your references section:

```
@misc{Label Studio,
  title={{Label Studio}: Data labeling software},
  url={https://github.com/HumanSignal/label-studio},
  note={Open source software available from https://github.com/HumanSignal/label-studio},
  author={
    Maxim Tkachenko and
    Mikhail Malyuk and
    Andrey Holmanyuk and
    Nikolai Liubimov},
  year={2020-2025},
}
```

## Architecture

<div class="opensource-only">

!!! error Enterprise
    You can use any of the Label Studio components in your own tools, or customize them to suit your needs. Before customizing Label Studio extensively, you might want to review Label Studio Enterprise Edition to see if it already contains the relevant functionality you want to build. See [Label Studio Features](https://labelstud.io/guide/label_studio_compare.html) for more.

</div>

The component parts of Label Studio are available as modular extensible packages that you can integrate into your existing machine learning processes and tools.


| Module   | Technology      | Description      |
| --------------- | ----------------------------- | ------------------------------------------------------------- |
| [Label Studio main app](https://github.com/HumanSignal/label-studio/)     | Python and [Django](https://www.djangoproject.com/)    | The main app with most of the backend code for Label Studio; used to perform data labeling.    |
| [Label Studio frontend](frontend_reference)       | JavaScript web app using [React](https://reactjs.org/) and [MST](https://github.com/mobxjs/mobx-state-tree) | Located within the main app repo. `web/apps/labelstudio` acts as the central integration point for all frontend elements. `web/libs/editor` is the frontend library.              |
| Data Manager      | JavaScript web app using [React](https://reactjs.org/)     | Manage data and tasks for labeling. Located under `web/libs/datamanager` in the main app repo.                         |
| [Machine Learning Backends](https://github.com/HumanSignal/label-studio-ml-backend) | Python   | Predict data labels at various parts of the labeling process. |

<br>
<div style="margin:auto; text-align:center;"><img src="/images/ls-modules-scheme.png"/></div>

---

### Source Guide Glossary (`docs/source/guide/glossary.md`)

---
title: Label Studio terminology
short: Terminology
tier: all
type: guide
order: 16 
order_enterprise: 16
meta_title: Label Studio terminology
meta_description: A glossary of common terms seen throughout Label Studio. 
section: "Discover & Learn"
date: 2023-10-27 11:38:25
---

 The following table describes some terms you might encounter as you use Label Studio:


| Term   | Description   |
|--|-----|
| Annotations  | The output of a labeling task. Previously called "completions". The terms "annotations" and "labels" are frequently used interchangeably. |
| Bounding box | Region within an image. |
| Dataset | What you import into Label Studio, comprised of individual items, or labeling tasks. |
| Labels | What you add to each region while labeling a task in Label Studio.  |
| Label stream | When you click **Label All Tasks** from the Data Manager, you are working within the label stream. |
| Labeling configuration | The labeling configuration determines what annotators and reviewers will see. It is configured in the project settings. |
| Predictions, <br> Pre-annotations | Annotations in Label Studio format that machine learning models create for an unlabeled dataset. See [import pre-annotations](predictions.html)  |
| Relation   | A defined relationship between two labeled regions.  |
| Result | A label applied to a specific region as stored in an annotation or prediction. See [Label Studio JSON format of annotated tasks](export.html#Label-Studio-JSON-format-of-annotated-tasks). |
| Quick view   | The "quick view" is what you see when you click an individual item in the Data Manager to open it (different than viewing it in the "label stream").   |
| Record | Item in a dataset.   |
| Region | The portion of the task identified for labeling. For example, when working with text, this might be a specific span of text or field. For images, an example region is a bounding box. For text, an example region is a span of text. Often has a label assigned to it.         |
| Task  | When you upload data to Label Studio, each item in the dataset becomes a labeling *task*. A task is a distinct item from a dataset that is ready to be labeled, pre-annotated, or has already been annotated. For example: a text snippet, an image, or a video clip.  |
| Tags   | Configuration options to customize the labeling interface. See [more about tags](/tags).  |
| Templates | Example labeling configurations that you can use to specify the type of labeling that you're performing with your dataset. See [all available templates](/templates) |

---

### Source Guide Google Saml (`docs/source/guide/google_saml.md`)

---
title: Google SAML SSO Setup Example
short: Google SAML SSO with Label Studio Setup Example
tier: enterprise
order: 253
meta_title: Google SAML SSO with Label Studio Setup Example
meta_description: Label Studio Enterprise documentation for setting up Google as Identity Provider for SAML SSO Setup Example.
hide_sidebar: true
---

You can set up Label Studio to use Google's IdP for SSO. 

You will need the following permissions:

* Access to the [Google Admin console](https://admin.google.com/). Your account must have [super administrator privileges](https://support.google.com/a/answer/2405986#super_admin). 
* Your Label Studio user role must be Owner or Administrator. 


## Step 1: Get Label Studio ACS endpoints as a Service Provider (SP)

1. **Access Label Studio SSO settings:**
   - Log in to Label Studio with an account that has the Owner or Administrator role.
   - Click the menu in the upper left and select **Organization**.
   - Select **SSO & SAML** in the upper right.

2. **Copy SAML configuration**
   - From the **Details for Identity Provider** section, copy the following URLs: 
        - **Assertion Consumer Service (ACS) URL**
        - **Login URL**
        - **Logout URL**

    <img src="/images/google-saml/4-lse-saml.png" class="gif-border">

## Step 2: Configure Google Groups and Users

1. **Access Google Admin Console:**
   - Log in to the [Google Admin console](https://admin.google.com/).
   - Go to **Directory** > **Groups**.
 
    <img src="/images/google-saml/9-add-group.png" class="gif-border">

2. **Add a new group:**
   - Click **Create group** and follow the steps to create and save a new group. In this example, the group is named "TestGroup." 
 
    <img src="/images/google-saml/10-add-group-2.png" class="gif-border">

3. **Assign users to the new group:**
   - Go to **Directory** > **Users**.
   - Select the users you want to add to your new group.
   - Select **More options > Add selected users to groups**. 
 
    <img src="/images/google-saml/11-add-group-3.png" class="gif-border">


## Step 3: Configure Google as an Identity Provider (IdP)

1. **Access Google Admin console:**
   - Log in to the [Google Admin console](https://admin.google.com/).
   - Go to **Apps** > **Web and mobile apps**.

    <img src="/images/google-saml/1-web-apps.png" class="gif-border">


2. **Add a new SAML app:**
   - Select **Add App > Add custom SAML app**.
   - Enter a name for the app (e.g. "Label Studio") and click **Continue**.

    <img src="/images/google-saml/2-add-app.png" class="gif-border">

3. **Download IdP metadata:**
   - When given the option, download the IdP metadata file for your new app. This file contains all the necessary information to configure Label Studio.

    <img src="/images/google-saml/3-metadata.png" class="gif-border">
   
4. **Configure SAML settings:**
   - In the **Service Provider Details** section, enter the following:
     - **ACS URL**: Paste the ACS URL copied from Label Studio.
     - **Entity ID**: Use the same URL as the ACS URL.
     - **Start URL**: (Optional) You can leave this blank or use the **Login URL** from Label Studio.
     - **Signed Response**: Leave this unchecked.
   - Click **Continue**.
 
    <img src="/images/google-saml/5-acs-setup.png" class="gif-border">

5. **Map attributes:**
   - In the **Attribute Mapping** section, map the following attributes:
     - **Primary Email**: `Email`
     - **First Name**: `FirstName`
     - **Last Name**: `LastName`
   - Click **Finish**.

    <img src="/images/google-saml/6-atrributes-setup.png" class="gif-border">

## Step 4: Complete configuration in Label Studio

1. **Upload IdP Metadata:**
   - Return to the Label Studio SSO & SAML settings page.
   - Scroll to **Add metadata from Identity Provider**.
   - Upload the IdP metadata file you downloaded from Google Admin console.

    <img src="/images/google-saml/4-lse-saml-add-metadata.png" class="gif-border">

2. **Map Groups and Roles:**
   - Set up group mappings to roles and workspaces as needed. Ensure the group names match those sent by Google in the SAML response.

    <img src="/images/google-saml/14-lse-saml-role-group.png" class="gif-border">

3. **Save Configuration:**
   - Click **Save** to apply the SSO configuration.

## Step 5: Complete configuration in the Google Admin console

1. **Return to the Google Admin console**:
   - From the Google Admin console, go to **Apps** > **Web and mobile apps**.
   - Select the Label Studio app.

    <img src="/images/google-saml/12-assign-group-1.png" class="gif-border">

2. **Assign the Label Studio application to the group:**
    - Click the **User access** card. 
    - Search for and then assign the group to the app. 
    - Enable the **Service status** toggle. 

    <img src="/images/google-saml/13-assign-group-2.png" class="gif-border">
    

## Step 6: Test the configuration

**Log in using SSO:**
   - Navigate to the Label Studio login page.
   - Click the **SSO Login** button and enter your company domain if prompted.
   - You should be redirected to the Google SSO login page. After successful authentication, you will be redirected back to Label Studio.

<img src="/images/google-saml/15-lse-sso-login.png" class="gif-border">

## Additional notes

- If you encounter any issues, verify that the SAML attributes and URLs are correctly configured in both Label Studio and Google Admin Console.

For more detailed information, you can refer to the
- [Label Studio SSO setup guide](auth_setup)
- [Official Google Docs about SAML](https://support.google.com/a/answer/6087519?hl=en#zippy=%2Cstep-add-the-custom-saml-app)

---

### Source Guide Ground Truths (`docs/source/guide/ground_truths.md`)

---
title: Ground truth annotations
short: Ground truth annotations
tier: enterprise
type: guide
order: 0
order_enterprise: 305
meta_title: Ground truth annotations
meta_description: Set ground truth annotations in Label Studio Enterprise. 
section: "Review & Measure Quality"
date: 2024-09-30 13:57:28
---

A "ground truth" annotation is a verified, high-quality annotation that serves as the correct answer for a specific task. It acts as a benchmark to assess the accuracy of other annotations and the performance of machine learning models.

Label Studio Enterprise compares annotations from annotators and model predictions against the ground truth annotations for a task to calculate an accuracy score between 0 and 1.

!!! error Enterprise
    Ground truth annotations are only available in Label Studio Enterprise Edition. If you're using Label Studio Community Edition, see [Label Studio Features](https://labelstud.io/guide/label_studio_compare.html) to learn more.

## Mark an annotation as a ground truth

Open the task to review its annotations. Click the star icon at the bottom to set a specific annotation as the ground truth:



!!! note
    A task can only have one annotation set as the ground truth annotation for the task. If you set a new annotation for a task as a ground truth, the previous annotation for that task set as a ground truth is updated to no longer be a ground truth annotation.

## Set ground truths by user

To bulk update tasks to ensure that annotations from a specific annotator should be considered ground truth, first select the tasks you want to update. 

Then select **Actions >Set Ground Truths**. You can then select from a list of annotators. 



## Review existing ground truth annotations

You can identify which tasks already have a ground truth using the **Ground Truth** column in the Data Manager:



You can also use a filter to include or exclude tasks that have ground truths set. 

## View ground truths in the inter-annotator agreement popover

Hover over the **Agreement** column to view a pop-over showing inter-annotator agreement. Annotations that have been marked ground truth are indicated with a star icon ⭐️:

<img src="/images/review/iaa-popover.png" alt="" class="gif-border" style="max-width:705px" />


## Remove ground truth annotations

To remove a single ground truth annotation, simply open the task and click the star icon again to unset it. 

To remove multiple ground truth annotations, select the checkboxes next to the tasks in the Data Manager and then select **Actions > Delete Ground Truths**.

---

### Source Guide Helm Values (`docs/source/guide/helm_values.md`)

---
title: Available Helm values for Label Studio Helm Chart
short: Available Helm values
tier: all
type: guide
order: 72
order_enterprise: 74
meta_title: Available Helm values for Label Studio Helm Chart
meta_description: For cases when you want to customize your Label Studio Kubernetes deployment, review these available Helm values that you can set in your Helm chart.
section: "Install & Setup"
parent: "install_k8s"
parent_enterprise: "install_enterprise_k8s"
---


<style>
  td:first-child {
    max-width: 350px;
  }
  td:first-child code {
    white-space: normal;
    word-break: break-word;
    margin-left: 0;
    padding-left: 3px;
  }
  td:nth-child(3) {
    white-space: normal;
    word-break: break-word;
    max-width: 130px;
  }
  td:last-child {
    max-width: 150px;
  }
  td:last-child code {
    white-space: normal;
    word-break: break-all;
  }
</style>

<div class="opensource-only">

Refer to these tables with available Helm chart values for your `values.yaml` file
when configuring your Label Studio deployment on Kubernetes. See [Deploy Label Studio on Kubernetes](install_k8s) for more.

</div>

<div class="enterprise-only">

Refer to these tables with available Helm chart values for your `values.yaml` file
when configuring your Label Studio deployment on Kubernetes. See [Deploy Label Studio on Kubernetes](install_enterprise_k8s) for more.

</div>

## Global parameters

Global parameters for the Helm chart.

| Parameter                                                                   | Description                                                                                                                         | Default                    |
|-----------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------|----------------------------|
| `global.imagePullSecrets`                                                   | Global Docker registry secret names as an array                                                                                     | `[]`                       |
| `global.image.repository`                                                   | Image repository                                                                                                                    | `heartexlabs/label-studio` |
| `global.image.pullPolicy`                                                   | Image pull policy                                                                                                                   | `IfNotPresent`             |
| `global.image.tag`                                                          | Image tag (immutable tags are recommended)                                                                                          | `develop`                  |
| `global.pgConfig.host`                                                      | PostgreSQL hostname                                                                                                                 | `""`                       |
| `global.pgConfig.port`                                                      | PostgreSQL port                                                                                                                     | `5432`                     |
| `global.pgConfig.dbName`                                                    | PostgreSQL database name                                                                                                            | `""`                       |
| `global.pgConfig.userName`                                                  | PostgreSQL database user account                                                                                                    | `""`                       |
| `global.pgConfig.password.secretName`                                       | Name of an existing secret holding the password of PostgreSQL database user account                                                 | `""`                       |
| `global.pgConfig.password.secretKey`                                        | Key of an existing secret holding the password of PostgreSQL database user account                                                  | `""`                       |
| `global.pgConfig.ssl.pgSslMode`                                             | [PostgreSQL SSL mode](https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNECT-SSLMODE)                             | `""`                       |
| `global.pgConfig.ssl.pgSslSecretName`                                       | Name of an existing secret holding the ssl certificate for PostgreSQL host                                                          | `""`                       |
| `global.pgConfig.ssl.pgSslRootCertSecretKey`                                | Key of an existing secret holding the ssl certificate for PostgreSQL host                                                           | `""`                       |
| `global.pgConfig.ssl.pgSslCertSecretKey`                                    | Name of an existing secret holding the ssl certificate private key for PostgreSQL host                                              | `""`                       |
| `global.pgConfig.ssl.pgSslKeySecretKey`                                     | Key of an existing secret holding the ssl certificate private key for PostgreSQL host                                               | `""`                       |
| `global.redisConfig.host`                                                   | Redis connection string in a format: redis://[:password]@localhost:6379/1                                                           | `""`                       |
| `global.redisConfig.password.secretName`                                    | Name of an existing secret holding the password of Redis database                                                                   | `""`                       |
| `global.redisConfig.password.secretKey`                                     | Key of an existing secret holding the password of Redis database                                                                    | `""`                       |
| `global.redisConfig.ssl.redisSslCertReqs`                                   | Whether to validate the server public key or ignore it. Accepts (`""`, `"optional"`, `"required"`).                                 | `""`                       |
| `global.redisConfig.ssl.redisSslSecretName`                                 | Name of an existing secret holding the ssl certificate for Redis host                                                               | `""`                       |
| `global.redisConfig.ssl.redisSslCaCertsSecretKey`                           | Key of an existing secret holding the ssl certificate for Redis host                                                                | `""`                       |
| `global.redisConfig.ssl.redisSslCertFileSecretKey`                          | Name of an existing secret holding the ssl certificate private key for Redis host                                                   | `""`                       |
| `global.redisConfig.ssl.redisSslKeyFileSecretKey`                           | Key of an existing secret holding the ssl certificate private key for Redis host                                                    | `""`                       |
| `global.extraEnvironmentVars`	                                              | Key/value map of an extra Environment variables, for example, `PYTHONUNBUFFERED: 1`                                                 | `{}`                       |
| `global.extraEnvironmentSecrets`                                            | Key/value map of an extra Secrets                                                                                                   | `{}`                       |
| `global.persistence.enabled`                                                | Enable persistent storage. [See more about setting up persistent storage](https://labelstud.io/guide/persistent_storage.html)       | `true`                     |
| `global.persistence.type`                                                   | Persistent storage type                                                                                                             | `volume`                   |
| `global.persistence.config.s3.accessKey`                                    | Access key to use to access AWS S3                                                                                                  | `""`                       |
| `global.persistence.config.s3.secretKey`                                    | Secret key to use to access AWS S3                                                                                                  | `""`                       |
| `global.persistence.config.s3.accessKeyExistingSecret`                      | Existing Secret name to extract Access key from to access AWS S3                                                                    | `""`                       |
| `global.persistence.config.s3.accessKeyExistingSecretKey`                   | Existing Secret key to extract Access key from to access AWS S3                                                                     | `""`                       |
| `global.persistence.config.s3.secretKeyExistingSecret`                      | Existing Secret name to extract Access secret key from to access AWS S3                                                             | `""`                       |
| `global.persistence.config.s3.secretKeyExistingSecretKey`                   | Existing Secret key to extract Access secret key from to access AWS S3                                                              | `""`                       |
| `global.persistence.config.s3.region`                                       | AWS S3 region                                                                                                                       | `""`                       |
| `global.persistence.config.s3.bucket`                                       | AWS S3 bucket name                                                                                                                  | `""`                       |
| `global.persistence.config.s3.folder`                                       | AWS S3 folder name                                                                                                                  | `""`                       |
| `global.persistence.config.s3.urlExpirationSecs`                            | The number of seconds that a presigned URL is valid for                                                                             | `86400`                    |
| `global.persistence.config.s3.endpointUrl`                                  | Custom S3 URL to use when connecting to S3, including scheme                                                                        | `""`                       |
| `global.persistence.config.volume.storageClass`                             | StorageClass for Persistent Volume                                                                                                  | `""`                       |
| `global.persistence.config.volume.size`                                     | Persistent volume size                                                                                                              | `10Gi`                     |
| `global.persistence.config.volume.accessModes`                              | PVC Access mode                                                                                                                     | `[ReadWriteOnce]`          |
| `global.persistence.config.volume.annotations`	                             | Persistent volume additional annotations                                                                                            | `{}`                       |
| `global.persistence.config.volume.existingClaim`                            | Name of an existing PVC to use                                                                                                      | `""`                       |
| `global.persistence.config.volume.resourcePolicy`                           | PVC resource policy                                                                                                                 | `""`                       |
| `global.persistence.config.volume.annotations`                              | Persistent volume additional annotations                                                                                            | `{}`                       |
| `global.persistence.config.azure.storageAccountName`                        | Azure Storage Account Name to use to access Azure Blob Storage                                                                      | `""`                       |
| `global.persistence.config.azure.storageAccountKey`                         | Azure Storage Account Key to use to access Azure Blob Storage                                                                       | `""`                       |
| `global.persistence.config.azure.storageAccountNameExistingSecret`          | Existing Secret name to extract Azure Storage Account Name from to access Azure Blob Storage                                        | `""`                       |
| `global.persistence.config.azure.storageAccountNameExistingSecretKey`       | Existing Secret key to extract Azure Storage Account Name from to use to access Azure Blob Storage                                  | `""`                       |
| `global.persistence.config.azure.storageAccountKeyExistingSecret`           | Existing Secret name to extract Azure Storage Account Key from to access Azure Blob Storage                                         | `""`                       |
| `global.persistence.config.azure.storageAccountKeyExistingSecretKey`        | Existing Secret key to extract Azure Storage Account Key from to use to access Azure Blob Storage                                   | `""`                       |
| `global.persistence.config.azure.containerName`                             | Azure Storage container name                                                                                                        | `""`                       |
| `global.persistence.config.azure.folder`                                    | Azure Storage folder name                                                                                                           | `""`                       |
| `global.persistence.config.azure.urlExpirationSecs`                         | The number of seconds that a presigned URL is valid for                                                                             | `86400`                    |
| `global.persistence.config.gcs.projectID`                                   | GCP Project ID to use                                                                                                               | `""`                       |
| `global.persistence.config.gcs.applicationCredentialsJSON`                  | Service Account key to access GCS                                                                                                   | `""`                       |
| `global.persistence.config.gcs.applicationCredentialsJSONExistingSecret`    | Existing Secret name to extract Service Account Key from to access GCS                                                              | `""`                       |
| `global.persistence.config.gcs.applicationCredentialsJSONExistingSecretKey` | Existing Secret key to extract Service Account Key from to access GCS                                                               | `""`                       |
| `global.persistence.config.gcs.bucket`                                      | GCS bucket name                                                                                                                     | `""`                       |
| `global.persistence.config.gcs.folder`                                      | GCS folder name                                                                                                                     | `""`                       |
| `global.persistence.config.gcs.urlExpirationSecs`                           | The number of seconds that a presigned URL is valid for                                                                             | `86400`                    |
| `global.featureFlags`                                                       | Key/value map of Feature Flags                                                                                                      | `{}`                       |
| `global.envInjectSources`                                                   | List of file names of a shell scripts to load additional environment variables from. This is useful when using Vault Agent Injector | `[]`                       |
| `global.cmdWrapper`                                                         | Additional commands to run prior to starting App. Useful to run wrappers before startup command                                     | `""`                       |
| `global.customCaCerts`                                                      | List of file names of SSL certificates to add into trust chain                                                                      | `[]`                       |

## App parameters

Parameters specific to the `app` portion of the Label Studio deployment.

| Parameter                                      | Description                                                                                                          | Default                  |
|------------------------------------------------|----------------------------------------------------------------------------------------------------------------------|--------------------------|
| `app.args`                                     | Override default container args (useful when using custom images)	                                                   | `["label-studio-uwsgi"]` |
| `app.deploymentStrategy.type`                  | Deployment strategy type                                                                                             | `RollingUpdate`          |
| `app.replicas`                                 | Amount of app pod replicas                                                                                           | `1`                      |
| `app.NameOverride`                             | String to partially override release template name                                                                   | `""`                     |
| `app.FullnameOverride`                         | String to fully override release template name                                                                       | `""`                     |
| `app.resources.requests.memory`                | The requested memory resources for the App container                                                                 | `384Mi`                  |
| `app.resources.requests.cpu`                   | The requested cpu resources for the App container                                                                    | `250m`                   |
| `app.resources.limits.memory`                  | The memory resources limits for the App container                                                                    | `""`                     |
| `app.resources.limits.cpu`                     | The cpu resources limits for the App container                                                                       | `""`                     |
| `app.initContainer.resources.requests`         | Init container db-migrations resource requests                                                                       | `{}`                     |
| `app.initContainer.resources.limits`           | Init container db-migrations resource limits                                                                         | `{}`                     |
| `app.readinessProbe.enabled`                   | Enable redinessProbe                                                                                                 | `false`                  |
| `app.readinessProbe.path`                      | Path for reasinessProbe                                                                                              | `/version`               |
| `app.readinessProbe.failureThreshold`          | When a probe fails, Kubernetes will try failureThreshold times before giving up                                      | `2`                      |
| `app.readinessProbe.initialDelaySeconds`       | Number of seconds after the container has started before probe initiates                                             | `60`                     |
| `app.readinessProbe.periodSeconds`             | How often (in seconds) to perform the probe                                                                          | `10`                     |
| `app.readinessProbe.successThreshold`          | Minimum consecutive successes for the probe to be considered successful after having failed                          | `1`                      |
| `app.readinessProbe.timeoutSeconds`            | Number of seconds after which the probe times out                                                                    | `5`                      |
| `app.livenessProbe.enabled`                    | Enable livenessProbe                                                                                                 | `true`                   |
| `app.livenessProbe.path`                       | Path for livenessProbe                                                                                               | `/health`                |
| `app.livenessProbe.failureThreshold`           | When a probe fails, Kubernetes will try failureThreshold times before giving up                                      | `3`                      |
| `app.livenessProbe.initialDelaySeconds`        | Number of seconds after the container has started before probe initiates                                             | `60`                     |
| `app.livenessProbe.periodSeconds`              | How often (in seconds) to perform the probe                                                                          | `10`                     |
| `app.livenessProbe.successThreshold`           | Minimum consecutive successes for the probe to be considered successful after having failed                          | `1`                      |
| `app.livenessProbe.timeoutSeconds`             | Number of seconds after which the probe times out                                                                    | `5`                      |
| `app.extraEnvironmentVars`                     | A map of extra environment variables to set                                                                          | `{}`                     |
| `app.extraEnvironmentSecrets`                  | A map of extra environment secrets to set                                                                            | `{}`                     |
| `app.nodeSelector`                             | Labels for pod assignment, formatted as a multi-line string or YAML map                                              | `{}`                     |
| `app.annotations`                              | k8s annotations to attach to the app pods                                                                            | `{}`                     |
| `app.extraLabels`                              | extra k8s labels to attach                                                                                           | `{}`                     |
| `app.affinity`                                 | Affinity for pod assignment                                                                                          | `{}`                     |
| `app.tolerations`                              | Toleration settings for pod                                                                                          | `[]`                     |
| `app.nginx.resources.requests`                 | Nginx sidecar container: resource requests                                                                           | `{}`                     |
| `app.nginx.resources.limits`                   | Nginx sidecar container: resource limits                                                                             | `{}`                     |
| `app.dnsPolicy`                                | Pod DNS policy                                                                                                       | `ClusterFirst`           |
| `app.enableServiceLinks`                       | Service environment variables                                                                                        | `false`                  |
| `app.shareProcessNamespace`                    | Enable shared process namespace in a pod                                                                             | `false`                  |
| `app.automountServiceAccountToken`             | Automount service account token for the server service account                                                       | `true`                   |
| `app.serviceAccount.create`                    | Enable the creation of a ServiceAccount for app pod                                                                  | `true`                   |
| `app.serviceAccount.name`                      | Name of the created ServiceAccount                                                                                   |                          |
| `app.serviceAccount.annotations`               | Custom annotations for app ServiceAccount                                                                            | `{}`                     |
| `app.podSecurityContext.enabled`               | Enable pod Security Context                                                                                          | `true`                   |
| `app.podSecurityContext.fsGroup`               | Group ID for the pod                                                                                                 | `1001`                   |
| `app.containerSecurityContext.enabled`         | Enable container security context                                                                                    | `true`                   |
| `app.containerSecurityContext.runAsUser`       | User ID for the container                                                                                            | `1001`                   |
| `app.containerSecurityContext.runAsNonRoot`    | Avoid privilege escalation to root user                                                                              | `true`                   |
| `app.extraVolumes`                             | Array to add extra volumes                                                                                           | `[]`                     |
| `app.extraVolumeMounts`                        | Array to add extra mounts (normally used with extraVolumes)                                                          | `[]`                     |
| `app.topologySpreadConstraints`                | Topology Spread Constraints for pod assignment                                                                       | `[]`                     |
| `app.nginx.args`                               | Override default container args (useful when using custom images)	                                                   | `["nginx"]`              |
| `app.nginx.livenessProbe.enabled`              | Nginx sidecar container: Enable livenessProbe                                                                        | `true`                   |
| `app.nginx.livenessProbe.path`                 | Nginx sidecar container: path for livenessProbe                                                                      | `/nginx_health`          |
| `app.nginx.livenessProbe.failureThreshold`     | Nginx sidecar container: when a probe fails, Kubernetes will try failureThreshold times before giving up             | `2`                      |
| `app.nginx.livenessProbe.initialDelaySeconds`  | Nginx sidecar container: Number of seconds after the container has started before probe initiates                    | `60`                     |
| `app.nginx.livenessProbe.periodSeconds`        | Nginx sidecar container: How often (in seconds) to perform the probe                                                 | `5`                      |
| `app.nginx.livenessProbe.successThreshold`     | Nginx sidecar container: Minimum consecutive successes for the probe to be considered successful after having failed | `1`                      |
| `app.nginx.livenessProbe.timeoutSeconds`       | Nginx sidecar container: Number of seconds after which the probe times out                                           | `3`                      |
| `app.nginx.readinessProbe.enabled`             | Nginx sidecar container: Enable readinessProbe                                                                        | `true`                   |
| `app.nginx.readinessProbe.path`                | Nginx sidecar container: Path for readinessProbe                                                                     | `/version`               |
| `app.nginx.readinessProbe.failureThreshold`    | Nginx sidecar container: When a probe fails, Kubernetes will try failureThreshold times before giving up             | `2`                      |
| `app.nginx.readinessProbe.initialDelaySeconds` | Nginx sidecar container: Number of seconds after the container has started before probe initiates                    | `60`                     |
| `app.nginx.readinessProbe.periodSeconds`       | Nginx sidecar container: How often (in seconds) to perform the probe                                                 | `10`                     |
| `app.nginx.readinessProbe.successThreshold`    | Nginx sidecar container: Minimum consecutive successes for the probe to be considered successful after having failed | `1`                      |
| `app.nginx.readinessProbe.timeoutSeconds`      | Nginx sidecar container: Number of seconds after which the probe times out                                           | `5`                      |
| `app.service.type`                             | k8s service type                                                                                                     | `ClusterIP`              |
| `app.service.port`                             | k8s service port                                                                                                     | `80`                     |
| `app.service.targetPort`                       | k8s service target port                                                                                              | `8085`                   |
| `app.service.portName`                         | k8s service port name                                                                                                | `service`                |
| `app.service.annotations`	                     | Custom annotations for app service                                                                                   | `{}`                     |
| `app.service.sessionAffinity`                  | Custom annotations for app service                                                                                   | `None`                   |
| `app.service.sessionAffinityConfig`	           | Additional settings for the sessionAffinity                                                                          | `{}`                     |
| `app.ingress.enabled`                          | Set to true to enable ingress record generation	                                                                     | `false`                  |
| `app.ingress.className`                        | IngressClass that will be be used to implement the Ingress (Kubernetes 1.18+)                                        | `""`                     |
| `app.ingress.host`                             | Default host for the ingress resource	                                                                               | `""`                     |
| `app.ingress.path`                             | The Path to LabelStudio. You may need to set this to '/*' in order to use this with ALB ingress controllers.         | `/`                      |
| `app.ingress.extraPaths`                       | Extra paths to prepend to the host configuration                                                                     | `[]`                     |
| `app.ingress.tls`                              | TLS secrets definition                                                                                               | `[]`                     |
| `app.ingress.annotations`                      | Additional ingress annotations                                                                                       | `{}`                     |
| `app.rbac.create`                              | Specifies whether RBAC resources should be created for app service                                                   | `false`                  |
| `app.rbac.rules`                               | Custom RBAC rules to set for app service		                                                                           | `[]`                     |
| `app.contextPath`                              | Context path appended for health/readiness checks                                                                    | `/`                      |
| `app.cmdWrapper`                               | Additional commands to run prior to starting App. Useful to run wrappers before startup command                      | `""`                     |


## Rqworker parameters

Parameters specific to the `rqworkers` service of your Label Studio Enterprise deployment.

| Parameter                                        | Description                                                                                     | Default                                |
|--------------------------------------------------|-------------------------------------------------------------------------------------------------|----------------------------------------|
| `rqworker.enabled`                               | Enable rqworker pod                                                                             | `true`                                 |
| `rqworker.NameOverride`                          | String to partially override release template name                                              | `""`                                   |
| `rqworker.FullnameOverride`                      | String to fully override release template name                                                  | `""`                                   |
| `rqworker.deploymentStrategy.type`               | Deployment strategy type                                                                        | `Recreate`                             |
| `rqworker.extraEnvironmentVars`                  | A map of extra environment variables to set                                                     | `{}`                                   |
| `rqworker.extraEnvironmentSecrets`               | A map of extra environment secrets to set                                                       | `{}`                                   |
| `rqworker.nodeSelector`                          | labels for pod assignment, formatted as a multi-line string or YAML map                         | `{}`                                   |
| `rqworker.annotations`                           | k8s annotations to attach to the rqworker pods                                                  | `{}`                                   |
| `rqworker.extraLabels`                           | extra k8s labels to attach                                                                      | `{}`                                   |
| `rqworker.affinity`                              | Affinity for pod assignment                                                                     | `{}`                                   |
| `rqworker.tolerations`                           | Toleration settings for pod                                                                     | `[]`                                   |
| `rqworker.queues.high.replicas`                  | Rqworker queue "high" replicas amount                                                           | `1`                                    |
| `rqworker.queues.high.args`                      | Rqworker queue "high" launch arguments                                                          | `"high"`                               |
| `rqworker.queues.low.replicas`                   | Rqworker queue "low" replicas amount                                                            | `1`                                    |
| `rqworker.queues.low.args`                       | Rqworker queue "low" launch arguments                                                           | `"low"`                                |
| `rqworker.queues.default.replicas`               | Rqworker queue "default" replicas amount                                                        | `1`                                    |
| `rqworker.queues.default.args`                   | Rqworker queue "default" launch arguments                                                       | `"default"`                            |
| `rqworker.queues.critical.replicas`              | Rqworker queue "critical" replicas amount                                                       | `1`                                    |
| `rqworker.queues.critical.args`                  | Rqworker queue "critical" launch arguments                                                      | `"critical"`                           |
| `rqworker.queues.all.replicas`                   | Rqworker queue "all" replicas amount                                                            | `1`                                    |
| `rqworker.queues.all.args`                       | Rqworker queue "all" launch arguments                                                           | `"low", "default", "critical", "high"` |
| `rqworker.dnsPolicy`                             | Pod DNS policy                                                                                  | `ClusterFirst`                         |
| `rqworker.enableServiceLinks`                    | Service environment variables                                                                   | `false`                                |
| `rqworker.shareProcessNamespace`                 | Enable shared process namespace in a pod                                                        | `false`                                |
| `rqworker.automountServiceAccountToken`          | Automount service account token for the server service account                                  | `true`                                 |
| `rqworker.readinessProbe.enabled`                | Enable redinessProbe                                                                            | `false`                                |
| `rqworker.readinessProbe.path`                   | Path for reasinessProbe                                                                         | `/version`                             |
| `rqworker.readinessProbe.failureThreshold`       | When a probe fails, Kubernetes will try failureThreshold times before giving up                 | `2`                                    |
| `rqworker.readinessProbe.initialDelaySeconds`    | Number of seconds after the container has started before probe initiates                        | `60`                                   |
| `rqworker.readinessProbe.periodSeconds`          | How often (in seconds) to perform the probe                                                     | `5`                                    |
| `rqworker.readinessProbe.successThreshold`       | Minimum consecutive successes for the probe to be considered successful after having failed     | `1`                                    |
| `rqworker.readinessProbe.timeoutSeconds`         | Number of seconds after which the probe times out                                               | `3`                                    |
| `rqworker.livenessProbe.enabled`                 | Enable livenessProbe                                                                            | `false`                                |
| `rqworker.livenessProbe.path`                    | Path for livenessProbe                                                                          | `/health`                              |
| `rqworker.livenessProbe.failureThreshold`        | When a probe fails, Kubernetes will try failureThreshold times before giving up                 | `2`                                    |
| `rqworker.livenessProbe.initialDelaySeconds`     | Number of seconds after the container has started before probe initiates                        | `60`                                   |
| `rqworker.livenessProbe.periodSeconds`           | How often (in seconds) to perform the probe                                                     | `5`                                    |
| `rqworker.livenessProbe.successThreshold`        | Minimum consecutive successes for the probe to be considered successful after having failed     | `1`                                    |
| `rqworker.livenessProbe.timeoutSeconds`          | Number of seconds after which the probe times out                                               | `3`                                    |
| `rqworker.serviceAccount.create`                 | Enable the creation of a ServiceAccount for rqworker pod                                        | `true`                                 |
| `rqworker.serviceAccount.name`                   | Name of the created ServiceAccount                                                              | `""`                                   |
| `rqworker.podSecurityContext.enabled`            | Enable pod Security Context                                                                     | `true`                                 |
| `rqworker.podSecurityContext.fsGroup`            | Group ID for the pod                                                                            | `1001`                                 |
| `rqworker.containerSecurityContext.enabled`      | Enable container security context                                                               | `true`                                 |
| `rqworker.containerSecurityContext.runAsUser`    | User ID for the container                                                                       | `1001`                                 |
| `rqworker.containerSecurityContext.runAsNonRoot` | Avoid privilege escalation to root user                                                         | `true`                                 |
| `rqworker.serviceAccount.annotations`            | Custom annotations for app ServiceAccount                                                       | `{}`                                   |
| `rqworker.extraVolumes`                          | Array to add extra volumes                                                                      | `[]`                                   |
| `rqworker.extraVolumeMounts`                     | Array to add extra mounts (normally used with extraVolumes)                                     | `[]`                                   |
| `rqworker.topologySpreadConstraints`             | Topology Spread Constraints for pod assignment                                                  | `[]`                                   |
| `rqworker.rbac.create`                           | Specifies whether RBAC resources should be created for rqworker service                         | `false`                                |
| `rqworker.rbac.rules`                            | Custom RBAC rules to set for rqworker service		                                                 | `[]`                                   |
| `rqworker.cmdWrapper`                            | Additional commands to run prior to starting App. Useful to run wrappers before startup command | `""`                                   |

<div class="enterprise-only">

## Label Studio Enterprise parameters

| Parameter                                 | Description                                                                        | Default   |
|-------------------------------------------|------------------------------------------------------------------------------------|-----------|
| `enterprise.enabled`                      | Enable Enterprise features                                                         | `false`   |
| `enterprise.enterpriseLicense.secretName` | Name of an existing secret holding the Label Studio Enterprise license information | `""`      |
| `enterprise.enterpriseLicense.secretKey`  | Key of an existing secret holding the enterprise license information               | `license` |

</div>

## Sub-charts parameters

| Parameter                  | Description                                                                                             | Default       |
|----------------------------|---------------------------------------------------------------------------------------------------------|---------------|
| `postgresql.enabled`       | Enable Postgresql sub-chart                                                                             | `true`        |
| `postgresql.architecture`  | PostgreSQL architecture (standalone or replication)                                                     | `standalone`  |
| `postgresql.image.tag`     | PostgreSQL image tag                                                                                    | `13.8.0`      |
| `postgresql.auth.username` | Name for a custom user to create	                                                                       | `labelstudio` |
| `postgresql.auth.password` | Password for the custom user to create. Ignored if `auth.existingSecret` with key password is provided	 | `labelstudio` |
| `postgresql.auth.database` | Name for a custom database to create	                                                                   | `labelstudio` |
| `redis.enabled`            | Enable Redis sub-chart                                                                                  | `false`       |
| `redis.architecture`       | Redis architecture. Allowed values: `standalone` or `replication`	                                      | `standalone`  |
| `redis.auth.enabled`       | Enable password authentication	                                                                         | `false`       |

## Other parameters
| Parameter                 | Description                                      | Default         |
|---------------------------|--------------------------------------------------|-----------------|
| upgradeCheck.enabled      | Enable upgradecheck                              | `false`         |
| ci                        | Indicate that deployment running for CI purposes | `false`         |
| clusterDomain             | Kubernetes Cluster Domain                        | `cluster.local` |
| checkConfig.skipEnvValues | Skip validation for env variables                | `false`         |

## The `global.extraEnvironmentVars` usage

The `global.extraEnvironmentVars` section can be used to configure environment properties of Label Studio.

Any key value put under this section translates to environment variables used to control Label Studio's configuration. Every key is upper-cased before setting the environment variable.

An example:

```yaml
global:
  extraEnvironmentVars:
     PG_USER: labelstudio
```

!!! note
    If you are deploying to a production environment, you should set `SSRF_PROTECTION_ENABLED: true`. See [Secure Label Studio](security#Enable-SSRF-protection-for-production-environments). 

<div class="enterprise-only">

### Lower the seat limit of your license

Set `LICENSE_MAX_USERS_OVERRIDE` to use fewer seats than your license allows, for example to keep a staging deployment within a smaller budget:

```yaml
global:
  extraEnvironmentVars:
     LICENSE_MAX_USERS_OVERRIDE: 50
```

By default this variable is not set and the seat count from your license applies. It can only *lower* that count: if you set it higher than your licensed seats, or below `1`, the deployment fails to start. The license itself is not modified, so removing the variable and restarting restores the licensed seat count.

Because on-premise seats are counted across all organizations in the deployment, the override applies to the whole deployment rather than to a single organization. It also lowers the number of free View-Only seats when your license does not set them explicitly. See [Usage & License](admin_usage#Seats-in-use).

</div>

---

### Source Guide Hotkeys (`docs/source/guide/hotkeys.md`)

---
title: Hotkeys
short: Hotkeys
tier: all
type: guide
order: 138
order_enterprise: 138
meta_title: Hotkeys in Label Studio
section: "Create & Manage Projects"
parent: "labeling"
parent_enterprise: "labeling"
date: 2025-05-24 17:19:21
---

Use keyboard shortcuts, or hotkeys, to improve your labeling performance. 

## View out-of-the-box hotkeys

From the labeling editor, click the settings icon to see more details about hotkeys or to enable or disable hotkeys. 



!!! info Tip
    If `enter` is specified and you have a Mac keyboard, use `return` instead. Use `delete` instead of `backspace`, and `option` instead of `alt`.

## Customize global hotkeys from your account page

You can easily customize global hotkeys on from the [**Account & Settings > Hotkeys** page](user_account#Hotkeys). 



## Customize global hotkeys using environment variables

If you want to change the hotkeys used for specific actions, set the `EDITOR_KEYMAP` environment variable with valid JSON in your `.env` file or when starting Label Studio. For example, to change the keyboard shortcut used to submit an annotation to `shift` + `s`, set the environment variable as follows:
```
EDITOR_KEYMAP='{"annotation:submit":{"key": "shift+s","description": "My Custom Submit Hotkey!"}}'
```
This overwrites the existing hotkey mapping with your custom mapping. See [more about how to set environment variables](https://labelstud.io/guide/start#Set-environment-variables). 

Refer to the full list of customizable hotkeys in the [`keymap.json` file](https://github.com/HumanSignal/label-studio/blob/develop/web/libs/editor/src/core/settings/keymap.json) to update a different hotkey combination. 

You cannot use this environment variable to remove an existing or add a new keyboard shortcut. 

## Use the `<Shortcut>` tag

Use the [Shortcut tag](/tags/shortcut) to allow annotators to insert a predefined text snippet into their current annotation. 

They can do this by either clicking the shortcut button or by pressing its hotkey. 

!!! info Tip
    Shortcuts work inside control tags (e.g., `<TextArea>`, `<Labels>`)

For example, use the following to insert predefined text into a text area:

```html
<View>
  <Image name="image" value="$captioning"/>
  <Header value="Describe the image:"/>
  <TextArea name="caption" toName="image">
    
    <Shortcut alias="Car" value="CAR" hotkey="ctrl+1" background="#333333"/>
    
    <Shortcut alias="Airplane"   value="AIRPLANE"   hotkey="ctrl+2" background="#FFEE00"/>
  </TextArea>
</View>
```

---

### Source Guide Ingress Config (`docs/source/guide/ingress_config.md`)

---
title: Set up an ingress controller for Label Studio Kubernetes deployments
short: Set up an ingress controller
type: guide
tier: all
order: 70
order_enterprise: 70
meta_title: Set up an ingress controller for Label Studio Kubernetes Deployments
meta_description: Set up an ingress controller to manage load balancing and access to Label Studio Kubernetes deployments for your data science and machine learning projects.
section: "Install & Setup"
parent: "install_k8s"
parent_enterprise: "install_enterprise_k8s"
---

Set up an ingress controller to manage Ingress, the Kubernetes resource that exposes HTTP and HTTPS routes from outside your Kubernetes cluster to the services within the cluster, such as Label Studio rqworkers and others.  

Select the best option for your deployment:
- Ingress for Amazon Elastic Kubernetes Service (EKS)
- Ingress for Google Kubernetes Engine (GKE)
- Ingress for Microsoft Azure Kubernetes Service (AKS)
- Ingress using nginx (cloud-agnostic)
- Terminate TLS on the Load Balancer (cloud-agnostic)

Configure ingress before or after setting up [persistent storage](persistent_storage.html), but before you deploy Label Studio.

!!! note 
    You only need to set up an ingress controller if you plan to deploy Label Studio on Kubernetes. 

## Configure ingress for Amazon EKS

If you plan to deploy Label Studio onto Amazon EKS, configure ingress. 

1. Install the AWS Load Balancer Controller to install an ingress controller with default options. See the documentation for [AWS Load Balancer Controller](https://docs.aws.amazon.com/eks/latest/userguide/aws-load-balancer-controller.html) in the Amazon EKS user guide.
2. After installing the AWS Load Balancer Controller, configure SSL certificates using the AWS Certificate Manager (ACM). See [Requesting a public certificate](https://docs.aws.amazon.com/acm/latest/userguide/gs-acm-request-public.html) in the ACM user guide.
3. Update your `ls-values.yaml` file with the ingress details like the following example. Replace `"your_domain_name"` with your hostname.
```yaml
app:
  ingress:
    enabled: true
    path: /*
    host: "your_domain_name"
    className: alb
    annotations: 
      alb.ingress.kubernetes.io/scheme: internet-facing
      alb.ingress.kubernetes.io/target-type: ip
```

!!! note
    If you want to configure a certificate that you create in the ACM for the load balancer, add this annotation (updated for your certificate) to your `ls-values.yaml` file:  
```
alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:region:account-id:certificate/aaaa-bbbb-cccc
```

For more details about annotations that you can configure with ingress, see the guide on [Ingress annotations](https://kubernetes-sigs.github.io/aws-load-balancer-controller/latest/guide/ingress/annotations/) in the AWS Load Balancer Controller documentation on GitHub.

## Configure ingress for GKE

Google Kubernetes Engine (GKE) contains two pre-installed Ingress classes:
- The `gce` class deploys an external load balancer
- The `gce-internal` class deploys an internal load balancer

Label Studio is considered as an external service, so you want to use the `gce` class to deploy an external load balancer.

1. Update your `ls-values.yaml` file with the ingress details like the following example. Replace `"your_domain_name"` with your hostname.
```yaml
app:
  service:
    type: nodePort
  ingress:
    enabled: true
    path: /*
    host: "your_domain_name"
    className: gce
```

!!! note 
    You can also request Google-managed SSL certificates to use on the load balancer. See the details on [Using Google-managed SSL certificates](https://cloud.google.com/kubernetes-engine/docs/how-to/managed-certs) in the Google Kubernetes Engine how-to guide. If you use a managed certificate, add an annotation to your `ls-values.yaml` file like the following example, replacing `"managed-cert"` with your ManagedCertificate object name:
```yaml
"networking.gke.io/managed-certificates": "managed-cert"
```

For more details about annotations and ingress in GKE, see [Configuring Ingress for external load balancing](https://cloud.google.com/kubernetes-engine/docs/how-to/load-balance-ingress) in the Google Kubernetes Engine how-to guide.

## Configure ingress for Microsoft Azure Kubernetes Service

Configure ingress for Microsoft Azure Kubernetes Service (AKS).

1. Deploy an Application Gateway Ingress Controller (AGIC) using a new Application Gateway. See [How to Install an Application Gateway Ingress Controller (AGIC) Using a New Application Gateway](https://docs.microsoft.com/en-us/azure/application-gateway/ingress-controller-install-new) in the Microsoft Azure Ingress for AKS how-to guide. 
2. Update your `ls-values.yaml` file with the ingress details like the following example. Replace `"your_domain_name"` with your hostname.
```yaml
app:
  ingress:
    enabled: true
    host: "your_domain_name"
    className: azure/application-gateway
```

!!! note 
    You can create a self-signed certificate to use in AGIC. Follow the steps to [Create a self-signed certificate](https://docs.microsoft.com/en-us/azure/application-gateway/create-ssl-portal#create-a-self-signed-certificate) in the Microsoft Azure Networking Tutorial: Configure an application gateway with TLS termination using the Azure portal. 

For more details about using AGIC with Microsoft Azure, see [What is Application Gateway Ingress Controller?](https://docs.microsoft.com/en-us/azure/application-gateway/ingress-controller-overview) and [Annotations for Application Gateway Ingress Controller](https://docs.microsoft.com/en-us/azure/application-gateway/ingress-controller-annotations) in the Microsoft Azure Application Gateway documentation.

## Set up a cloud-agnostic ingress configuration

For advanced Kubernetes administrators, you can use the NGINX Ingress Controller to set up a cloud-agnostic ingress controller.

1. Deploy NGINX Ingress Controller following the relevant steps for your cloud deployment. See [Cloud deployments](https://kubernetes.github.io/ingress-nginx/deploy/#cloud-deployments) in the NGINX Ingress Controller Installation Guide. 
2. In order to terminate SSL certificates in the ingress controller, install cert-manager. See [Installation](https://cert-manager.io/docs/installation/) on the cert-manager documentation site.  
3. You must synchronize the ingress hosts with DNS. Install [ExternalDNS](https://github.com/kubernetes-sigs/external-dns#readme) and choose the relevant cloud provider for your deployment.
4. Finally, update your `ls-values.yaml` file with the ingress details like the following example. Replace `"your_domain_name"` with your hostname and `<CERTIFICATE_NAME>` with the name of the resource that you created with ExternalDNS.
```yaml
app:
  ingress:
    enabled: true
    host: "your_domain_name"
    className: nginx
    tls:
      - secretName: <CERTIFICATE_NAME>
        hosts:
          - "your_domain_name"
```

## Terminate TLS on the Load Balancer

If SSL termination is happening on a Load Balancer before traffic is forwarded to the Ingress, you'll need to set the LABEL_STUDIO_HOST environment variable.

Update your `ls-values.yaml` file with the LABEL_STUDIO_HOST environment variable like the following example. Replace `"your_domain_name"` with your hostname.
```yaml
global:
  extraEnvironmentVars:
    LABEL_STUDIO_HOST: https://your_domain_name
```


## Handle large file uploads with NGINX ingress

When uploading large files through an NGINX ingress controller, you might encounter a "413 Request Entity Too Large" error. This happens because NGINX has a default limit of 1MB on the size of client request body.

To handle large file uploads, you can configure the `proxy-body-size` annotation in your ingress configuration. Update your `ls-values.yaml` file to include this annotation:
```yaml
app:
  ingress:
    annotations:
      nginx.ingress.kubernetes.io/proxy-body-size: "200m"
```

If you're using a different ingress controller, consult the documentation for that specific controller to learn how to configure request body size limits. For example:

- For HAProxy Ingress Controller, use the `haproxy.org/client-max-body-size` annotation
- For Traefik, configure the `maxRequestBodyBytes` middleware
- For AWS ALB Ingress Controller, the maximum request body size is controlled by the ALB settings

The exact configuration will depend on your ingress controller type and version. Refer to your controller's documentation for the specific annotation or configuration needed.

---

### Source Guide Install (`docs/source/guide/install.md`)

---
title: Installation
type: guide
tier: opensource
order: 60
order_enterprise: 0
meta_title: Install and Upgrade Label Studio
meta_description: "Label Studio documentation: install and upgrade Label Studio with Docker, pip, and anaconda for your machine learning and data science projects."
section: "Install & Setup"
---

Install Label Studio on premises or in the cloud. Choose the installation method that works best for your environment:

- [Install with pip](#Install-using-pip)
- [Install with Docker](#Install-with-Docker)
- [Install on Ubuntu](#Install-on-Ubuntu)
- [Install from source](#Install-from-source)
- [Install with Anaconda](#Install-with-Anaconda)
- [Upgrade Label Studio](#Upgrade-Label-Studio)

Label Studio is also available an [enterprise product](https://heartex.com/), which you can explore instantly through a [free trial](https://humansignal.com/free-trial).

{% insertmd includes/deploy.md %}

### Web browser support

Label Studio is tested with the latest version of Google Chrome.

If using other web browsers, or older versions of supported web browsers, unexpected behavior could occur.

## Install prerequisite

Install Label Studio in a clean Python environment. We highly recommend using a virtual environment (venv or conda) to reduce the likelihood of package conflicts or missing packages.

## Install using pip

To install Label Studio with pip and a virtual environment, you need Python version 3.8 or later. Run the following:

```bash
python3 -m venv env
source env/bin/activate
python -m pip install label-studio
```

To install Label Studio with pip, you need Python version 3.8 or later. Run the following:

```bash
pip install label-studio
```

After you install Label Studio, start the server with the following command:

```bash
label-studio
```

The default web browser will automatically open at [http://localhost:8080](http://localhost:8080) with Label Studio. Please refer to [start Label Studio](start.html) for more options when starting Label Studio.

## Install using Homebrew

To install Label Studio with Brew, you need to have Brew package manager installed on your system. If you do not have Brew installed, please visit [brew.sh](https://brew.sh/) for installation instructions.

Execute the following command to add the Heartexlabs tap:

```bash
brew tap humansignal/tap
```

Execute the following command to install Label Studio:

```bash
brew install humansignal/tap/label-studio
```

After you install Label Studio, initiate the server using the following command:

```bash
label-studio
```

The default web browser will automatically open at [http://localhost:8080](http://localhost:8080) with Label Studio. Please refer to [start Label Studio](start.html) for more options when starting Label Studio.

!!! note
    If you've previously installed the brew tap from the now-deprecated organization name `heartexlabs/tap`, we've got good news. You don't have to worry about migrating immediately. The deprecated tap has been set up as a mirror for `humansignal/tap`. This ensures continuity and minimizes disruptions for existing users.

## Install with Docker

Label Studio is also available as a Docker container. Make sure you have [Docker](https://www.docker.com/) installed on your machine.

### Install with Docker on \*nix

To install and start Label Studio at [http://localhost:8080](http://localhost:8080), storing all labeling data in `./mydata` directory, run the following:

```bash
docker run -it -p 8080:8080 -v $(pwd)/mydata:/label-studio/data heartexlabs/label-studio:latest
```

!!! attention "important"
    As this is a non-root container, the mounted files and directories must have the proper permissions for the `UID 1001`.

### Install with Docker on Windows

Or for Windows, you have to modify the volumes paths set by `-v` option.

#### Override the default Docker install

You can override the default Docker install by appending new arguments.

In Windows Command Line (cmd):

```bash
docker run -it -p 8080:8080 -v %cd%/mydata:/label-studio/data heartexlabs/label-studio:latest label-studio --log-level DEBUG
```

In PowerShell:

```bash
docker run -it -p 8080:8080 -v ${PWD}/mydata:/label-studio/data heartexlabs/label-studio:latest label-studio --log-level DEBUG
```

### Build a local image with Docker

If you want to build a local image, run:

```bash
docker build -t heartexlabs/label-studio:latest .
```

### Run with Docker Compose

Use Docker Compose to serve Label Studio at `http://localhost:8080`. You must use Docker Compose version 1.25.0 or higher.

Start Label Studio:

```bash
docker-compose up -d
```

This starts Label Studio with a PostgreSQL database backend. You can also use a PostgreSQL database without Docker Compose. See [Set up database storage](storedata.html).

### Install Label Studio without internet access

Download label-studio docker image (host with internet access and docker):

```bash
docker pull heartexlabs/label-studio:latest
```

Export it as a tar archive:

```bash
docker save heartexlabs/label-studio:latest | gzip > label_studio_latest.tar.gz
```

Transfer it to another VM:

```bash
scp label_studio_latest.tar.gz <ANOTHER_HOST>:/tmp
```

SSH into `<ANOTHER_HOST>` and load the image from the archive:

```bash
docker load --input /tmp/label_studio_latest.tar.gz
```

Follow steps from [Install and Upgrade to run LS](install.html#Install-with-Docker).

## Install on Ubuntu

To install Label Studio on Ubuntu and run it in a virtual environment, run the following command:

```bash
python3 -m venv env
source env/bin/activate
sudo apt install python3.9-dev
python -m pip install label-studio
```

## Install from source

If you want to use nightly builds or extend the functionality, consider downloading the source code using Git and running Label Studio locally:

```bash
git clone https://github.com/HumanSignal/label-studio.git

# install dependencies
cd label-studio
pip install poetry
poetry install

# run db migrations
poetry run python label_studio/manage.py migrate

# collect static files
poetry run python label_studio/manage.py collectstatic

# start the server in development mode at http://localhost:8080
poetry run python label_studio/manage.py runserver

```

## Install with Anaconda

```bash
conda create --name label-studio
conda activate label-studio
conda install psycopg2  # required for LS 1.7.2 only
pip install label-studio
```

## Upgrade Label Studio

To upgrade to the latest version of Label Studio, reinstall or upgrade using pip.

```bash
pip install --upgrade label-studio
```

Migration scripts run when you upgrade to version 1.0.0 from version 0.9.1 or earlier.

To make sure an existing project gets migrated, when you [start Label Studio](start.html), run the following command:

```bash
label-studio start path/to/old/project
```

The most important change to be aware of is changes to rename "completions" to "annotations". See the [updated JSON format for completed tasks](export.html#Raw_JSON_format_of_completed_tasks).

If you customized the Label Studio Frontend, see the [Frontend reference guide](frontend_reference.html) for required updates to maintain compatibility with version 1.0.0.

---

### Source Guide Install Enterprise (`docs/source/guide/install_enterprise.md`)

---
title: Installation overview
short: Installation
tier: enterprise
type: guide
order: 0
order_enterprise: 60
meta_title: Installation overview for Label Studio Enterprise
meta_description: Overview of the components involved when installing Label Studio Enterprise as an on-prem deployment
section: "Install & Setup"
---


The two central components of Label Studio Enterprise are the main app and the RQ (Redis Queue) workers. An on-prem deployment typically also includes the following components:

* [**Identity provider**](auth_setup): An external IdP that connects to the main app to send a SAML or SCIM2.0 assertion to Label Studio. Identity providers help you manage user credentials and access. Common IdPs include Okta, Ping, Microsoft Active Directory. 

    While integrating an identity provider is recommended for managing user authentication and access control in a more scalable and secure manner, Label Studio Enterprise can be deployed with its own internal user management system if preferred. 
* [**Load balancer/ingress**](ingress_config): This serves as a traffic manager that directs incoming network requests to the main app.
* [**PostgresSQL database**](install_enterprise_k8s#Optional-set-up-TLS-for-PostgreSQL): This connects with the main app to store annotation metadata. It also connects with the RQ workers to store reports and pre-annotations.
* [**Redis**](install_enterprise_k8s#Optional-set-up-TLS-for-Redis): This works with the main app to perform background tasks such as computing statistical reports and predictions. It also communicates with the RQ workers to manage the background jobs queue. 
* [**Cloud storage**](persistent_storage): Cloud storage works with both the main app and the RQ workers to manage assets and data.
  * Persistent storage for assets such as avatars and snapshots. 
  * Import storage that passes data to Label Studio to be used in labeling tasks. 
  * Export storage for saving the exported annotation data as those tasks are labeled. 
* **Integrations**: For example, these can be a [custom ML backend](ml) that calculates pre-annotations, or any other third-party services that you integrate with Label Studio using [webhooks](webhooks).

---

### Source Guide Install Enterprise Docker (`docs/source/guide/install_enterprise_docker.md`)

---
title: Install Label Studio Enterprise On-premises using Docker Compose
short: Install using Docker
type: guide
tier: enterprise
order: 0
order_enterprise: 75
meta_title: Install Label Studio Enterprise on-premises using Docker
meta_description: Install, back up, and upgrade Label Studio Enterprise with Docker to create machine learning and data science projects on-premises.
section: "Install & Setup"
parent_enterprise: "install_enterprise"
---

Install Label Studio Enterprise on-premises if you need to meet strong privacy regulations, legal requirements, or want to manage a custom installation on your own infrastructure using Docker or public cloud. If you want to use a different installation method:
- You can use Kubernetes and Helm to deploy Label Studio Enterprise in the cloud. See [Deploy Label Studio Enterprise on Kubernetes](install_enterprise_k8s.html).
- You can run Label Studio Enterprise in an airgapped environment, and no data leaves your infrastructure. See [Install Label Studio Enterprise without public internet access](install_k8s_airgapped).

See [Secure Label Studio](security.html) for more details about security and hardening for Label Studio Enterprise.

!!! note
    Docker deployments are more suitable for proof-of-concept builds and small projects. For production environments and larger deployments, we strongly recommend using Kubernetes. Kubernetes deployments provide better scalability, reliability, and access to advanced features such as Prompts. See [Deploy Label Studio Enterprise on Kubernetes](install_enterprise_k8s.html) for more information.

To install Label Studio Community Edition, see [Install Label Studio](https://labelstud.io/guide/install). This page is specific to the Enterprise version of Label Studio.

!!! note
    On-prem deployments of Label Studio Enterprise are not supported for Academic licenses.  

{% insertmd includes/deploy.md %}

## Install Label Studio Enterprise using Docker

1. Log in to a Docker registry.
2. Add the license file.
3. Start the server using Docker Compose.

### Prerequisites
Make sure you have an authorization token to retrieve Docker images and a current license file. If you are a Label Studio Enterprise customer and do not have access, [email support](mailto:support@humansignal.com) or open a ticket through our [support portal](https://support.humansignal.com/) to receive an authorization token and a copy of your license file.

Make sure [Docker Compose](https://docs.docker.com/compose/install/) is installed on your system.

After you install Label Studio Enterprise, the app is automatically connected to the following running services:
- PostgresSQL (versions >=13)
- Redis (version >=6.0)

### Log in to a Docker registry

You must be authorized to access Label Studio Enterprise images. 

Set up the Docker login to retrieve the latest Docker image:
```bash
docker login --username heartexlabs
```
When prompted to enter the password, enter the token. If login succeeds, a `~/.docker/config.json` file is created with the authorization settings.  

!!! note 
    If you have default registries specified when logging into Docker, you might need to explicitly specify the registry: `docker login --username heartexlabs docker.io`.

### Add the license file 
After you retrieve the latest Label Studio Enterprise image, add the license file. You can't start the Docker image without a license file. 

1. Create a working directory called `label-studio-enterprise` and place the license file in it.
```bash
mkdir -p label-studio-enterprise
cd label-studio-enterprise
```
2. Move the license file, `license.txt`, to the `label-studio-enterprise` directory.

### Start using Docker Compose

To run Label Studio Enterprise in production, start it using [Docker compose](https://docs.docker.com/compose/). This configuration lets you connect Label Studio to external databases and services.

1. Create a file, `label-studio-enterprise/env.list` with the required environment variables:
```
/* Detailed source-code truncated for AI context efficiency. */
```

2. After you set all the environment variables, create the following `docker-compose.yml`:

```
/* Detailed source-code truncated for AI context efficiency. */
```

3. Run Docker Compose:

```bash
docker-compose up
```

!!! note 
    If you expose port 80, you must start Docker with `sudo`.

### Get the Docker image version

To check the version of the Label Studio Enterprise Docker image, use the [`docker ps`](https://docs.docker.com/engine/reference/commandline/ps/) command on the host. 

From the command line, run the following as root or using `sudo` and review the output:
```bash
$ docker ps
03b88eebdb65   heartexlabs/label-studio-enterprise:2.2.8-1   "uwsgi --ini deploy/…"   36 hours ago   Up 36 hours   0.0.0.0:80->8000/tcp   label-studio-enterprise_app_1
```
In this example output, the image column displays the Docker image and version number. The image `heartexlabs/label-studio-enterprise:2.2.8-1` is using the version `2.2.8-1`.

---

### Source Guide Install Enterprise K8s (`docs/source/guide/install_enterprise_k8s.md`)

---
title: Deploy Label Studio Enterprise on Kubernetes
short: Install using Kubernetes
tier: enterprise
type: guide
order: 0
order_enterprise: 69
meta_title: Deploy Label Studio Enterprise on Kubernetes
meta_description: Deploy Label Studio Enterprise on Kubernetes, such as on Amazon Elastic Container Service for Kubernetes, to create machine learning and data science projects in a scalable containerized environment. 
section: "Install & Setup"
parent_enterprise: "install_enterprise"

---

Deploy Label Studio Enterprise on a Kubernetes Cluster using Helm 3. You can use this Helm chart to set up Label Studio Enterprise for deployment onto a Kubernetes cluster and install, upgrade, and manage the application. 

Your Kubernetes cluster can be self-hosted or installed somewhere such as Amazon EKS. See the Amazon tutorial on how to [Deploy a Kubernetes Application with Amazon Elastic Container Service for Kubernetes](https://aws.amazon.com/getting-started/hands-on/deploy-kubernetes-app-amazon-eks/) for more about deploying an app on Amazon EKS.

<div class="enterprise-only">

!!! warning
    To install Label Studio Community Edition, see <a href="https://labelstud.io/guide/install_k8s.html">Deploy Label Studio on Kubernetes</a>. This page is specific to the Enterprise version of Label Studio.

</div>

!!! note
    On-prem deployments of Label Studio Enterprise are not supported for Academic licenses. 

This high-level architecture diagram that outlines the main components of a Label Studio Enterprise deployment.

<img src="/images/LSE_k8s_scheme.png"/>

!!! warning
    Label Studio Enterprise 2.2.9 decommissioned MinIO as a service.

Label Studio Enterprise runs on Python and uses rqworkers to perform additional tasks. Metadata and annotations are stored in a bundled version of PostgreSQL that functions as persistent storage. If you host Label Studio Enterprise in the cloud, use [persistent storage in the cloud](persistent_storage.html) instead of MinIO.

## Install Label Studio Enterprise on Kubernetes

If you want to install Label Studio Enterprise on Kubernetes and you have unrestricted access to the internet from your K8s cluster, follow these steps. 

1. Verify that you meet the [Required software prerequisites](#Required-software-prerequisites) and review the [capacity planning](#Capacity-planning) guidance.
2. [Prepare the Kubernetes cluster](#Prepare-the-Kubernetes-cluster).
3. [Add the Helm chart repository](#Add-the-Helm-chart-repository).
4. [Configure Kubernetes secrets](#Configure-Kubernetes-secrets)
5. [Configure persistent storage](persistent_storage.html). 
6. [Configure ingress](ingress_config.html).
7. [Configure a values.yaml file](#Configure-values-yaml).
8. (Optional) [Set up TLS for PostgreSQL](#Optional-set-up-TLS-for-PostgreSQL)
9. (Optional) [Set up TLS for Redis](#Optional-set-up-TLS-for-Redis)
10. [Use Helm to install Label Studio Enterprise on your Kubernetes cluster](#Use-Helm-to-install-Label-Studio-Enterprise-on-your-Kubernetes-cluster).

If you use a proxy to access the internet from your Kubernetes cluster, or it is airgapped from the internet, see how to [Install Label Studio Enterprise without public internet access](install_k8s_airgapped.html).

### Required software prerequisites

- **Kubernetes** — version 1.17 or higher
- **Helm** — version 3.6.3 or higher
- **Redis** — version 6.0.5 or higher
- **PostgreSQL** — version 14.0 or higher
- **Persistent storage** — PVC with ReadWriteMany access mode or S3-compatible object storage

Redis and PostgreSQL are mandatory components of Label Studio Enterprise and are required for it to be fully functional. This chart does not include or install Redis or PostgreSQL; you must have them preconfigured (for example, according to your company’s standards) and provide connection details in your Helm values.

**Persistent storage** must use one of the following: a Persistent Volume Claim (PVC) with **ReadWriteMany** access mode, or S3-compatible object storage. See [Set up persistent storage](persistent_storage.html) for configuration details.

This chart has been tested and confirmed to work with the [NGINX Ingress Controller](https://kubernetes.github.io/ingress-nginx/) and [cert-manager](https://cert-manager.io/docs/). See [Set up an ingress controller for Label Studio Kubernetes deployments](ingress_config.html) for more on ingress settings with Label Studio. 

Your Kubernetes cluster can be self-hosted or installed somewhere such as Amazon EKS. 

### Capacity planning

To plan the capacity of your Kubernetes cluster, refer to these guidelines. 

Label Studio Enterprise has the following default configurations for resource requests, resource limits, and replica counts:

<div class="enterprise-only">

```yaml
app:
  replicas: 1
  resources:
    requests:
      memory: 1024Mi
      cpu: 1000m
    limits:
      memory: 6144Mi
      cpu: 4000m

rqworker:
  queues:
    high:
      replicas: 1
    low:
      replicas: 1
    default:
      replicas: 4
    critical:
      replicas: 1
  resources:
    requests:
      memory: 256Mi
      cpu: 100m
    limits:
      memory: 1024Mi
      cpu: 1000m
```

</div>

Before you make changes to these values, familiarize yourself with the [Resource Management for Pods and Containers](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/) guidelines in the Kubernetes documentation. 

If you choose to make changes to these default settings, consider the following:

| For this case                               | Adjust this                                                                                                                           |
|---------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------|
| More than 10 concurrent annotators          | Adjust the requests and limits for `resources` in the `app` pod                                                                       |
| Increase fault tolerance                    | Increase the number of replicas of `app` and/or `rqworker` services                                                                   |
| Production deployment (replicas)            | Replicas equivalent or greater than the number of availability zones in your Kubernetes cluster                                       | 

#### RQ worker replicas

The `default` queue is the most extensive queue. It is recommended to use 4 times more replicas for the `default` queue compared to the other queues. The other queues (`critical`, `high`, `low`) can have the same number of replicas. You can start with 1 replica for each of them. 

### Prepare the Kubernetes cluster

Before installing Label Studio, prepare the Kubernetes cluster with [kubectl](https://kubernetes.io/docs/reference/kubectl/). 

Install Label Studio Enterprise and set up a PostgreSQL and Redis databases to store relevant Label Studio Enterprise configurations and annotations using the Helm chart. You must configure specific values for your deployment in a YAML file that you specify when installing using Helm.

### Add the Helm chart repository
Add the Helm chart repository to easily install and update Label Studio.

1. From the command line:
   ```shell
   helm repo add heartex https://charts.heartex.com/
   helm repo update heartex
   ```
2. If you want, check for available versions:
   ```shell
   helm search repo heartex/label-studio
   ```

<div class="enterprise-only">

### Configure Kubernetes secrets

1. Ensure that you have license key and Docker Hub credentials or request them from Heartex Team.
2. Create a key to pull the latest Label Studio Enterprise image from the Docker registry. From the command line of your cluster, run the following:
    ```shell
    kubectl create secret docker-registry heartex-pull-key \
        --docker-server=https://index.docker.io/v2/ \
        --docker-username=heartexlabs \
        --docker-password=<CUSTOMER_PASSWORD>
    ```
3. Create the Label Studio Enterprise license as a Kubernetes secret. You can specify it as a file or as a specific URL.
   From the command line, specify the license as a file:
   ```shell
   kubectl create secret generic lse-license --from-file=license=path/to/lic
   ```
   Or from the command line, specify the license as a URL:
   ```shell
   kubectl create secret generic lse-license --from-literal=license=https://lic.heartex.ai/db/<CUSTOMER_LICENSE_ID>
   ```
   
</div>

<div class="enterprise-only">

### Configure values.yaml 

You must configure a `values.yaml` file for your Label Studio Enterprise deployment. The following file contains default values for a minimal installation of Label Studio. This chart has been tested and confirmed to work with the [NGINX Ingress Controller](https://kubernetes.github.io/ingress-nginx/) and [cert-manager](https://cert-manager.io/docs/).

Example `values.yaml` file for a minimal installation of Label Studio Enterprise Enterprise:
```
/* Detailed source-code truncated for AI context efficiency. */
```

Adjust the included defaults to reflect your environment and copy these into a new file and save it as `ls-values.yaml`.


!!! note 
    For more complex configurations, you can create your own file based on the [list of all available Helm values](helm_values.html).

</div>

## Optional: Set up TLS for PostgreSQL
To configure Label Studio Enterprise to use TLS for end-client connections with PostgreSQL, do the following:

1. Enable TLS for your PostgreSQL instance and save Root TLS certificate, client certificate and its key for the next steps.
2. Create a Kubernetes secret with your certificates, replacing `<PATH_TO_CA>`, `<PATH_TO_CLIENT_CRT>` and `<PATH_TO_CLIENT_KEY>` with paths to your certificates:

```shell
kubectl create secret generic <YOUR_SECRET_NAME> --from-file=ca.crt=<PATH_TO_CA> --from-file=client.crt=<PATH_TO_CLIENT_CRT> --from-file=client.key=<PATH_TO_CLIENT_KEY>
```
3. Update your `ls-values.yaml` file with your newly-created Kubernetes secret:

!!! note 
    If `POSTGRE_SSL_MODE: verify-ca`, the server is verified by checking the certificate chain up to the root certificate stored on the client. If `POSTGRE_SSL_MODE: verify-full`, the server host name will be verified to make sure it matches the name stored in the server certificate. The SSL connection will fail if the server certificate cannot be verified. `verify-full` is recommended in most security-sensitive environments.

```yaml
global:
  pgConfig:
    ssl:
      pgSslMode: "verify-full"
      pgSslSecretName: "<YOUR_SECRET_NAME>"
      pgSslRootCertSecretKey: "ca.crt"
      pgSslCertSecretKey: "client.crt"
      pgSslKeySecretKey: "client.key"
```

4. Install or upgrade Label Studio Enterprise using Helm.

## Optional: Set up TLS for Redis
To configure Label Studio Enterprise to use TLS for end-client connections with Redis, do the following:

1. Enable TLS for your Redis instance and save Root TLS certificate, client certificate and its key for the next steps.
2. Create a Kubernetes secret with your certificates, replacing `<PATH_TO_CA>`, `<PATH_TO_CLIENT_CRT>` and `<PATH_TO_CLIENT_KEY>` with paths to your certificates:

```shell
kubectl create secret generic <YOUR_SECRET_NAME> --from-file=ca.crt=<PATH_TO_CA> --from-file=client.crt=<PATH_TO_CLIENT_CRT> --from-file=client.key=<PATH_TO_CLIENT_KEY>
```
3. Update your `ls-values.yaml` file with your newly-created Kubernetes secret:

!!! note 
    In the case if you are using self-signed certificates that host cannot verify you have to set `redisSslCertReqs` to `None`

```yaml
global:
  redisConfig:
    ssl:
      redisSslCertReqs: "required"
      redisSslSecretName: "<YOUR_SECRET_NAME>"
      redisSslCaCertsSecretKey: "ca.crt"
      redisSslCertFileSecretKey: "client.crt"
      redisSslKeyFileSecretKey: "client.key"
```

4. Install or upgrade Label Studio Enterprise using Helm.


## Optional: Set up username and password for Redis

Use one of these options to set a password and a username for Redis:

**1. Password via Kubernetes Secret**. Use this when:
* You want to avoid embedding credentials in `values.yaml`
* You already manage Secrets in your cluster
* You need a simple auth without multiple Redis users and you don't have username

```yaml
global:
  redisConfig:
    host: "redis://redis.example.com:6379/1"
    password:
      secretName: "my-redis-secret"   # Kubernetes Secret name
      secretKey: "redis-password"      # Key inside Secret
```

**2. Username + password in URL**. Use this when:
* Redis v.7 or later, and with ACL-enabled users
* You need a dedicated Redis user for permission scoping
* You need a quick, throwaway setup or local testing

```yaml
global:
  redisConfig:
    host: "redis://myuser:mypassword@redis.example.com:6379/1"
```

 **3. Username in environment variables + password in secret**. Use this when:
* Redis v.7 or later, and with ACL-enabled users
* You want to keep the password secret but still specify a username

```yaml
global:
  redisConfig:
    host: "redis://redis.example.com:6379/1"
    password:
      secretName: "my-redis-secret"   # Kubernetes Secret name
      secretKey: "redis-password"      # Key inside Secret
  extraEnvironmentVars:
    REDIS_USERNAME: "myuser"           # Injected into pod env
```

## Use Helm to install Label Studio Enterprise on your Kubernetes cluster

Use Helm to install Label Studio Enterprise on your Kubernetes cluster. Provide your custom resource definitions YAML file. Specify any environment variables that you need to set for your Label Studio Enterprise installation using the `--set` argument with the `helm install` command.

!!! note
    If you are deploying to a production environment, you should set the `SSRF_PROTECTION_ENABLED: true` environment variable. See [Secure Label Studio](security#Enable-SSRF-protection-for-production-environments).

From the command line, run the following:
```shell
helm install <RELEASE_NAME> heartex/label-studio -f ls-values.yaml
```

After installing, check the status of the Kubernetes pod creation:
```shell
kubectl get pods
```

## Restart Label Studio Enterprise using Helm

Restart your Helm release by doing the following from the command line:

1. Identify the &lt;RELEASE_NAME&gt; of the latest Label Studio Enterprise release:
```shell
helm list
```
2. Restart the rqworker for Label Studio:
```shell
kubectl rollout restart deployment/<RELEASE_NAME>-ls-rqworker
```
3. Restart the Label Studio Enterprise app:
```shell
kubectl rollout restart deployment/<RELEASE_NAME>-ls-app
```


## Uninstall Label Studio Enterprise using Helm

To uninstall Label Studio Enterprise using Helm, delete the configuration.

From the command line, run the following:
```shell
helm delete <RELEASE_NAME>
```

---

### Source Guide Install K8s (`docs/source/guide/install_k8s.md`)

---
title: Deploy Label Studio on Kubernetes
short: Install using Kubernetes
tier: opensource
type: guide
order: 69
order_enterprise: 0
meta_title: Deploy Label Studio on Kubernetes
meta_description: Deploy Label Studio on Kubernetes, such as on Amazon Elastic Container Service for Kubernetes, to create machine learning and data science projects in a scalable containerized environment.
section: "Install & Setup"
parent: "install"
---

Deploy Label Studio on a Kubernetes Cluster using Helm 3. You can use this Helm chart to set up Label Studio for deployment onto a Kubernetes cluster and install, upgrade, and manage the application.

Your Kubernetes cluster can be self-hosted or installed somewhere such as Amazon EKS. See the Amazon tutorial on how to [Deploy a Kubernetes Application with Amazon Elastic Container Service for Kubernetes](https://aws.amazon.com/getting-started/hands-on/deploy-kubernetes-app-amazon-eks/) for more about deploying an app on Amazon EKS.

<div class="opensource-only">

!!! warning
    To install Label Studio Enterprise Edition, see <a href="https://docs.humansignal.com/guide/install_enterprise_k8s">Deploy Label Studio Enterprise on Kubernetes</a>. This page is specific to the community version of Label Studio.

</div>

## Install Label Studio on Kubernetes

If you want to install Label Studio on Kubernetes and you have unrestricted access to the internet from your K8s cluster, follow these steps.

1. Verify that you meet the [Required software prerequisites](#Required-software-prerequisites) and review the [capacity planning](#Capacity-planning) guidance.
2. [Prepare the Kubernetes cluster](#Prepare-the-Kubernetes-cluster).
3. [Add the Helm chart repository](#Add-the-Helm-chart-repository).
4. (Optional) Set up [persistent storage](persistent_storage.html).
5. (Optional) Configure [ingress](ingress_config.html).
6. (Optional) Configure [values.yaml](helm_values.html).
7. (Optional) [Set up TLS for PostgreSQL](#Optional-set-up-TLS-for-PostgreSQL)
8. [Use Helm to install Label Studio on your Kubernetes cluster](#Use-Helm-to-install-Label-Studio-on-your-Kubernetes-cluster).

If you use a proxy to access the internet from your Kubernetes cluster, or it is airgapped from the internet, see how to [Install Label Studio without public internet access](/guide/install_k8s_airgapped.html).

### Required software prerequisites

- Kubernetes and kubectl version 1.17 or higher
- Helm version 3.6.3 or higher

This chart has been tested and confirmed to work with the [NGINX Ingress Controller](https://kubernetes.github.io/ingress-nginx/) and [cert-manager](https://cert-manager.io/docs/). See [Set up an ingress controller for Label Studio Kubernetes deployments](ingress_config.html) for more on ingress settings with Label Studio.

Your Kubernetes cluster can be self-hosted or installed somewhere such as Amazon EKS.

### Capacity planning

To plan the capacity of your Kubernetes cluster, refer to these guidelines.

Label Studio has the following default configurations for resource requests, resource limits, and replica counts:

```yaml
app:
  replicas: 1
  resources:
    requests:
      memory: 1024Mi
      cpu: 1000m
    limits:
      memory: 6144Mi
      cpu: 4000m
```

Before you make changes to these values, familiarize yourself with the [Resource Management for Pods and Containers](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/) guidelines in the Kubernetes documentation.

If you choose to make changes to these default settings, consider the following:

| For this case                      | Adjust this                                                                                     |
| ---------------------------------- | ----------------------------------------------------------------------------------------------- |
| More than 10 concurrent annotators | Adjust the requests and limits for `resources` in the `app` pod                                 |
| Increase fault tolerance           | Increase the number of replicas of `app` pod                                                    |
| Production deployment (replicas)   | Replicas equivalent or greater than the number of availability zones in your Kubernetes cluster |

### Prepare the Kubernetes cluster

Before installing Label Studio, prepare the Kubernetes cluster with [kubectl](https://kubernetes.io/docs/reference/kubectl/).

### Add the Helm chart repository

Add the Helm chart repository to easily install and update Label Studio.

1. From the command line:
   ```shell
   helm repo add heartex https://charts.heartex.com/
   helm repo update heartex
   ```
2. If you want, check for available versions:
   ```shell
   helm search repo heartex/label-studio
   ```

## Optional: set up TLS for PostgreSQL

To configure Label Studio to use TLS for end-client connections with PostgreSQL, do the following:

1. Enable TLS for your PostgreSQL instance and save Root TLS certificate, client certificate and its key for the next steps.
2. Create a Kubernetes secret with your certificates, replacing `<PATH_TO_CA>`, `<PATH_TO_CLIENT_CRT>` and `<PATH_TO_CLIENT_KEY>` with paths to your certificates:
```shell
kubectl create secret generic <YOUR_SECRET_NAME> --from-file=ca.crt=<PATH_TO_CA> --from-file=client.crt=<PATH_TO_CLIENT_CRT> --from-file=client.key=<PATH_TO_CLIENT_KEY>
```

3. Update your `ls-values.yaml` file with your newly-created Kubernetes secret:

!!! note
    If `POSTGRE_SSL_MODE: verify-ca`, the server is verified by checking the certificate chain up to the root certificate stored on the client. If `POSTGRE_SSL_MODE: verify-full`, the server host name will be verified to make sure it matches the name stored in the server certificate. The SSL connection will fail if the server certificate cannot be verified. `verify-full` is recommended in most security-sensitive environments.

```yaml
global:
  pgConfig:
    ssl:
      pgSslMode: "verify-full"
      pgSslSecretName: "<YOUR_SECRET_NAME>"
      pgSslRootCertSecretKey: "ca.crt"
      pgSslCertSecretKey: "client.crt"
      pgSslKeySecretKey: "client.key"
```

4. Install or upgrade Label Studio using Helm.

## Use Helm to install Label Studio on your Kubernetes cluster

Use Helm to install Label Studio on your Kubernetes cluster. Provide your custom resource definitions YAML file. Specify any environment variables that you need to set for your Label Studio installation using the `--set` argument with the `helm install` command.

!!! note
    If you are deploying to a production environment, you should set the `SSRF_PROTECTION_ENABLED: true` environment variable. See [Secure Label Studio](security#Enable-SSRF-protection-for-production-environments).

From the command line, run the following:

```shell
helm install <RELEASE_NAME> heartex/label-studio -f ls-values.yaml
```

After installing, check the status of the Kubernetes pod creation:

```shell
kubectl get pods
```

## Restart Label Studio using Helm

Restart your Helm release by doing the following from the command line:

1. Identify the &lt;RELEASE_NAME&gt; of the latest Label Studio release:
```shell
helm list
```

2. Restart the Label Studio app:
```shell
kubectl rollout restart deployment/<RELEASE_NAME>-ls-app
```

## Upgrade Label Studio using Helm

To upgrade Label Studio using Helm, do the following.

1. Determine the latest tag version of Label Studio and add/replace the following in your `ls-values.yaml` file:
   ```yaml
   global:
     image:
       tag: "20210914.154442-d2d1935"
   ```
2. After updating the values file, retrieve the latest updates for the Helm chart:
   ```shell
   helm repo update heartex
   ```
3. Run the following from the command line to upgrade your deployment:
   ```shell
   helm upgrade <RELEASE_NAME> heartex/label-studio -f ls-values.yaml
   ```
   If you want, you can specify a version from the command line:
   ```shell
   helm upgrade <RELEASE_NAME> heartex/label-studio -f ls-values.yaml --set global.image.tag=20210914.154442-d2d1935
   ```
   This command overrides the tag value stored in `ls-values.yaml`. You must update the tag value when you upgrade or redeploy your instance to avoid version downgrades.

## Uninstall Label Studio using Helm

To uninstall Label Studio using Helm, delete the configuration.

From the command line, run the following:

```shell
helm delete <RELEASE_NAME>
```

---

### Source Guide Install K8s Airgapped (`docs/source/guide/install_k8s_airgapped.md`)

---
title: Install Label Studio without public internet access
short: Airgapped Server
tier: all
type: guide
order: 71
order_enterprise: 73
meta_title: Install Label Studio without public internet access
meta_description: Install Label Studio without public internet access to create machine learning and data science projects in an airgapped environment. 
section: "Install & Setup"
parent: "install_k8s"
parent_enterprise: "install_enterprise_k8s"
---

If you need to install Label Studio on a server that blocks access to the internet using a proxy, or an airgapped server that does not allow outgoing connections to the internet, follow these steps:

- If you access the internet from your server using an HTTPS proxy, see [Install behind an HTTPS proxy](#Install-behind-an-HTTPS-proxy).
- If you do not have access to the internet from your server, or use a different proxy, see [Install without internet access or HTTPS proxy](#Install-without-internet-access-or-HTTPS-proxy).

### Install behind an HTTPS proxy

!!! note 
    If you are using a SOCKS proxy, Helm 3 does not support SOCKS proxies. See [Install without internet access or HTTPS proxy](#Install-without-internet-access-or-HTTPS-proxy).

If your organization uses an HTTPS proxy to manage access to the internet, do the following.

<div class="opensource-only">

1. Work with your network security team to whitelist `https://charts.heartex.com` so that you can access the Helm charts for deployment.
2. On the Label Studio server, set an environment variable with the HTTPS proxy address:
```shell
export HTTPS_PROXY=<your_proxy>
```
3. [Use Helm to install Label Studio on your Kubernetes cluster](install_k8s#Use-Helm-to-install-Label-Studio-on-your-Kubernetes-cluster).

</div>

<div class="enterprise-only">

1. Work with your network security team to whitelist `https://charts.heartex.com` so that you can access the Helm charts for deployment.
2. On the Label Studio server, set an environment variable with the HTTPS proxy address:
```shell
export HTTPS_PROXY=<your_proxy>
```
3. [Use Helm to install Label Studio on your Kubernetes cluster](install_enterprise_k8s#Use-Helm-to-install-Label-Studio-Enterprise-on-your-Kubernetes-cluster).

</div>

### Install without internet access or HTTPS proxy

If you can't access the internet using a proxy supported by Helm or at all, follow these steps to download the Helm charts necessary to deploy Label Studio on an airgapped Kubernetes cluster. 

!!! note
    You need the Label Studio credentials provided to you by your account manager to download the Helm charts.

1. Download the latest version of Label Studio Helm chart:
   ```shell
   helm repo add heartex https://charts.heartex.com/
   helm repo update heartex
   helm pull heartex/label-studio
   ```
2. Transfer the downloaded `tar.gz` archive to the host that has `kubectl` and `helm` installed.
3. Expand the `tar.gz` archive.
4. Install Label Studio, updating the path in the `helm` commands to reference the relative path of the folder where you expanded Label Studio Helm cahrt. For example, if you expanded the archive file in the current directory, run the following:
```shell
helm install ls ./label-studio -f ls-values.yaml
```

---

### Source Guide Install Prompts (`docs/source/guide/install_prompts.md`)

---
title: Install Prompts in an on-prem environment (optional)
short: Install Prompts
type: guide
tier: enterprise
order: 0
order_enterprise: 71
meta_title: Install Prompts
meta_description: Install Prompts in a Label Studio Enterprise on-prem environment
section: "Install & Setup"
parent: "install_k8s"
parent_enterprise: "install_enterprise_k8s"
---

Installing Prompts in an on-prem environment requires installing Adala, our data labeling agent microservice.

You only need to complete these steps if you want to use Prompts. For more information, see our [Prompts overview](prompts_overview).

!!! note
    Prompts are only supported for Kubernetes deployments. If you are using Docker, you will need to migrate to a Kubernetes deployment to use Prompts. See [Deploy Label Studio Enterprise on Kubernetes](install_enterprise_k8s.html) for more information.


## Prerequisites

- Kubernetes cluster **v1.24** or later
- Helm **v3.8.0** or later
- Docker CLI (for logging into Docker Hub)

## Resource requirements

Before installing, ensure your Kubernetes cluster can provide the following minimum resources for Adala:

| Resource | Requirement |
| --- | --- |
| CPU | 6 cores |
| Memory | 12 GB |

## 1. Authenticate to Docker Hub and validate access

You will need your Docker Hub username and password. If you do not have them, [request access from the HumanSignal team](mailto:support@humansignal.com).

Log in to DockerHub to access the private OCI repository:

```bash

docker login -u CUSTOMER_USERNAME
```

When prompted, enter your Docker Hub password.

Then verify your credentials and access:

```bash
helm pull oci://registry-1.docker.io/heartexlabs/adala
```

Expected output:

```bash
Pulled: registry-1.docker.io/heartexlabs/adala:X.X.X
Digest: sha256:***************************************************
```

## 2. Create a Kubernetes secret for image pulling

Create a Kubernetes secret to allow your cluster to pull private Adala images:

```bash
kubectl create secret docker-registry heartex-pull-key \
  --docker-server=https://index.docker.io/v2/ \
  --docker-username=CUSTOMER_USERNAME \
  --docker-password=CUSTOMER_PASSWORD
```

## 3. Prepare your custom values file

Create a file named `custom.values.yaml` with the following contents:

```yaml
adala-app:
  deployment:
    image:
      tag: 20260409.063233-main-6b619568
      pullSecrets:
        - heartex-pull-key
adala-worker:
  deployment:
    image:
      tag: 20260409.063233-main-6b619568
      pullSecrets:
        - heartex-pull-key
```

!!! note
    Replace the `image.tag` with the appropriate version if necessary.


## 4. Create a dedicated namespace for Adala

Create a dedicated namespace `prompt` for Adala:

```bash
kubectl create namespace prompt
```

## 5. Install the Adala Helm chart

Run the following command to install **Adala** using your custom values:

```bash
helm install lse oci://registry-1.docker.io/heartexlabs/adala --values custom.values.yaml
```

## 6. Validate that Adala is running

Check if all pods in the `prompt` namespace are in the **Running** or **Completed** state:

```bash
kubectl get pods -n prompt
```

You should see output where all pods have `STATUS` set to `Running`, for example:

```
NAME                                  READY   STATUS    RESTARTS       AGE
adala-adala-app-d4564ffd7-gtmhx       1/1     Running   0              100m
adala-adala-kafka-controller-0        1/1     Running   0              110m
adala-adala-kafka-controller-1        1/1     Running   0              111m
adala-adala-kafka-controller-2        1/1     Running   0              113m
adala-adala-redis-master-0            1/1     Running   0              125m
adala-adala-worker-5d87f97f76-mq952   1/1     Running   0              111m

```

If any pod is not running, you can investigate further:

```bash

kubectl describe pod <pod-name> -n prompt
```

or

```bash
kubectl logs <pod-name> -n prompt
```

## 7. Update the Label Studio `values.yaml` file

You will need to update the `global` section of your Label Studio Enterprise `values.yaml` file to include the following:

* Add the Adala endpoint, which will allow Label Studio to connect to Adala. 
* Add the Prompts feature flag, to enable Prompts visibility within Label Studio. 


```yaml
global:
  extraEnvironmentVars:
    PROMPTER_ADALA_URL: http://adala-adala-app.prompt:8000
  featureFlags:
    fflag_feat_all_dia_835_prompter_workflow_long: true
```

Note the following for `PROMPTER_ADALA_URL`:

- `prompt` is the namespace where Adala is installed.
- `adala-adala-app` is the name of the Adala service automatically created by the Helm release.
- Port `8000` is the default port where Adala listens.

After updating the values file, redeploy Label Studio to apply the changes.

## 8. Optional: Add a custom CA bundle

If your environment requires a custom CA bundle (for example, when communicating with an LLM provider behind a corporate proxy with a self-signed certificate), you must make the CA certificate available to the Adala pods.

Because Label Studio Enterprise and Adala (Prompts) are deployed in separate namespaces, Kubernetes secrets cannot be shared between them. Even if you already have a CA bundle secret in the Label Studio namespace, you must create a new secret in the `prompt` namespace:

```bash
kubectl create secret generic custom-ca-bundle \
  --from-file=ca.crt=my.cert \
  -n prompt
```

Replace `my.cert` with the path to your CA certificate file. The key name (`ca.crt`) must match the `key` referenced in the values file below.

Then add the following to your `custom.values.yaml` and re-run the `helm install`/`helm upgrade` command from step 5:

```yaml
adala-app:
  deployment:
    extraEnvVars:
      REQUESTS_CA_BUNDLE: /etc/ssl/custom/ca.crt
      SSL_CERT_FILE: /etc/ssl/custom/ca.crt
    extraVolumeMounts:
      - mountPath: /.cache
        name: cache-dir
      - mountPath: /etc/ssl/custom
        name: custom-ca
        readOnly: true
    extraVolumes:
      - name: cache-dir
        emptyDir: {}
      - name: custom-ca
        secret:
          secretName: custom-ca-bundle
          items:
            - key: ca.crt
              path: ca.crt

adala-worker:
  deployment:
    extraEnvVars:
      REQUESTS_CA_BUNDLE: /etc/ssl/custom/ca.crt
      SSL_CERT_FILE: /etc/ssl/custom/ca.crt
    extraVolumeMounts:
      - mountPath: /.cache
        name: cache-dir
      - mountPath: /etc/ssl/custom
        name: custom-ca
        readOnly: true
    extraVolumes:
      - name: cache-dir
        emptyDir: {}
      - name: custom-ca
        secret:
          secretName: custom-ca-bundle
          items:
            - key: ca.crt
              path: ca.crt
```

---

### Source Guide Install Requirements (`docs/source/guide/install_requirements.md`)

---
title: Requirements to Install and upgrade Label Studio
short: Requirements
tier: opensource
section: "Install & Setup"
order: 61 
meta_title: Requirements to Install and Upgrade
meta_description: "Label Studio documentation: Requirements to install and upgrade Label Studio." 
---

{% insertmd includes/deploy.md %}

## Install prerequisite

Install Label Studio in a clean Python environment. Heartex highly recommends using a virtual environment (venv or conda) to reduce the likelihood of package conflicts or missing packages.


## Install with pip

To install Label Studio with pip and a virtual environment, you need Python version 3.8 or later. Run the following:
```bash
python3 -m venv env
source env/bin/activate
python -m pip install label-studio
```

To install Label Studio with pip, you need Python version 3.8 or later. Run the following:
```bash
pip install label-studio
```

After you install Label Studio, start the server with the following command: 
```bash
label-studio
```
The default web browser opens automatically at [http://localhost:8080](http://localhost:8080) with Label Studio. See [start Label Studio](start.html) for more options when starting Label Studio.


## Install with Docker

Label Studio is also available as a Docker container. Make sure you have [Docker](https://www.docker.com/) installed on your machine.

### Install with Docker on *nix
To install and start Label Studio at [http://localhost:8080](http://localhost:8080), storing all labeling data in `./my_project` directory, run the following:
```bash
docker run -it -p 8080:8080 -v $(pwd)/mydata:/label-studio/data heartexlabs/label-studio:latest
```

!!! attention "important"
    As this is a non-root container, the mounted files and directories must have the proper permissions for the `UID 1001`.

### Install with Docker on Windows
Or for Windows, you have to modify the volumes paths set by `-v` option.

#### Override the default Docker install
You can override the default Docker install by appending new arguments.

In Windows Command Line (cmd):
```bash
docker run -it -p 8080:8080 -v %cd%/mydata:/label-studio/data heartexlabs/label-studio:latest label-studio --log-level DEBUG
```

In PowerShell:
```bash
docker run -it -p 8080:8080 -v ${PWD}/mydata:/label-studio/data heartexlabs/label-studio:latest label-studio --log-level DEBUG
```

### Build a local image with Docker
If you want to build a local image, run:
```bash
docker build -t heartexlabs/label-studio:latest .
```

### Run with Docker Compose
Use Docker Compose to serve Label Studio at `http://localhost:8080`. You must use Docker Compose version 1.25.0 or higher.

Start Label Studio:
```bash
docker-compose up -d
```

This starts Label Studio with a PostgreSQL database backend. You can also use a PostgreSQL database without Docker Compose. See [Set up database storage](storedata.html).

### Install Label Studio without internet access
Download label-studio docker image (host with internet access and docker):
```bash 
docker pull heartexlabs/label-studio:latest
```

Export it as a tar archive: 
```bash
docker save heartexlabs/label-studio:latest | gzip > label_studio_latest.tar.gz
```

Transfer it to another VM:
```bash
scp label_studio_latest.tar.gz <ANOTHER_HOST>:/tmp
```

SSH into <ANOTHER_HOST> and import the archive:
```bash
docker image import /tmp/label_studio_latest.tar.gz
```

Follow steps from [Install and Upgrade to run LS](install.html#Install-with-Docker).


## Install on Ubuntu

To install Label Studio on Ubuntu and run it in a virtual environment, run the following command:

```bash
python3 -m venv env
source env/bin/activate
sudo apt install python3.9-dev
python -m pip install label-studio
```

## Install from source

If you want to use nightly builds or extend the functionality, consider downloading the source code using Git and running Label Studio locally:

```bash
git clone https://github.com/HumanSignal/label-studio.git
cd label-studio
# Install all package dependencies
pip install -e .
# Run database migrations
python label_studio/manage.py migrate
# Start the server in development mode at http://localhost:8080
python label_studio/manage.py runserver
```

## Install with Anaconda

```bash
conda create --name label-studio
conda activate label-studio
pip install label-studio
```


## Upgrade Label Studio

To upgrade to the latest version of Label Studio, reinstall or upgrade using pip. 


```bash
pip install --upgrade label-studio
```

Migration scripts run when you upgrade to version 1.0.0 from version 0.9.1 or earlier. 

To make sure an existing project gets migrated, when you [start Label Studio](start.html), run the following command:

```bash
label-studio start path/to/old/project 
```

The most important change to be aware of is changes to rename "completions" to "annotations". See the [updated JSON format for completed tasks](export.html#Raw_JSON_format_of_completed_tasks). 

If you customized the Label Studio Frontend, see the [Frontend reference guide](frontend_reference.html) for required updates to maintain compatibility with version 1.0.0.

---

### Source Guide Install Troubleshoot (`docs/source/guide/install_troubleshoot.md`)

---
title: Troubleshoot installation issues
short: Troubleshooting
tier: opensource
type: guide
order: 81
order_enterprise: 0
meta_title: Troubleshoot Label Studio installation issues
meta_description: Tips for troubleshooting Label Studio installation issues
section: "Install & Setup"
parent: "install"
date: 2023-12-06 10:38:02
---

You might see errors when installing Label Studio. Follow these steps to resolve them.

## Run the latest version of Label Studio

Many bugs might be fixed in patch releases or maintenance releases. Make sure you're running the latest version of Label Studio by [upgrading](install#Upgrade-Label-Studio) your installation before you start Label Studio.

## Errors about missing packages

If you see errors about missing packages, install those packages and try to install Label Studio again. Make sure that you run Label Studio in a clean Python environment, such as a virtual environment.

For Windows users the default installation might fail to build the `lxml` package. Consider manually installing it from [the unofficial Windows binaries](https://www.lfd.uci.edu/~gohlke/pythonlibs/#lxml). If you are running Windows 64-bit with Python 3.8 or later, run `pip install lxml‑4.5.0‑cp38‑cp38‑win_amd64.whl` to install it.

## Errors from Label Studio

If you see any other errors during installation, try to rerun the installation.

```bash
pip install --ignore-installed label-studio
```

## OpenBLAS blas_thread_init: pthread_create failed for thread X of Y: Operation not permitted

Upgrade Docker Engine to the latest available version(>= [20.10.12](https://docs.docker.com/engine/release-notes/#201012)).

## PermissionError: [Errno 13] Permission denied: `/label-studio/data/media`

!!! warning
    Starting with Label Studio 1.7.0 release, the application run using a non-root docker user with ID `1001`.

You may already be aware that Docker containers generally operate with root privileges by default. This unrestricted container management permits actions like installing system packages, modifying configuration files, and binding privileged ports, which are all beneficial for development purposes. However, this can lead to significant risks when containers are deployed in a production environment.

This is because anyone who gains access to your root-running container could initiate undesirable processes, such as injecting malicious code. Running a process in your container as root also enables altering the user id (UID) or group id (GID) when launching the container, making your application more vulnerable.

It is advised to use non-root containers for the following reasons:

- Security: Non-root containers inherently offer better security. In the event of a container engine security issue, running the container as a non-privileged user will prevent malicious code from obtaining elevated permissions on the container host. For more information on Docker's security features, refer to this guide.

- Platform limitations: Some Kubernetes distributions such as [OpenShift](https://www.openshift.com/), execute containers with random UUIDs. This method is incompatible with root containers, which must always run using the root user's UUID. In these situations, only non-root container images will operate, making them a necessity.

Our [Dockerfile](https://github.com/HumanSignal/label-studio/blob/develop/Dockerfile) contains the line `USER 1001` which assigns a non-root user UID to the image, enabling the container to run as an unprivileged user. This implementation applies the security enhancements and other restrictions mentioned above to the container.

### File permissions for non-root user

By default, Label Studio container images operate as non-root. As a result, any directory requiring write access must be assigned to the root group (`GID 0`). This ensures that the arbitrary user (default `UID 1001`) can write to the directory, as this user is always part of the root group. To achieve this, simply set the ownership of the local directory to the root group (GID 0), which will be enough regardless of the UID:

```bash
mkdir mydata
sudo chown :0 mydata
```

!!! note

    If you want to learn more about non-root containers and Docker and Kubernetes security, check out the following articles:

    [Docker Security documentation](https://docs.docker.com/engine/security/security/)

    [Understanding how uid and gid work in Docker containers by Marc Campbell](https://medium.com/@mccode/understanding-how-uid-and-gid-work-in-docker-containers-c37a01d01cf)

    [Processes In Containers Should Not Run As Root](https://medium.com/@mccode/processes-in-containers-should-not-run-as-root-2feae3f0df3b)

    [Just say no to root (containers) by Daniel J. Walsh](https://opensource.com/article/18/3/just-say-no-root-containers)

    [Running a Docker container as a non-root user by Lucas Willson-Richter](https://medium.com/redbubble/running-a-docker-container-as-a-non-root-user-7d2e00f8ee15)

    [How to run a more secure non-root user container by Dan Wash](https://www.projectatomic.io/blog/2016/01/how-to-run-a-more-secure-non-root-user-container/)

---

### Source Guide Interfaces Agent (`docs/source/guide/interfaces-agent.md`)

---
title: Use the Label Studio agent to create an Interface
short: Create with agent
tier: enterprise
type: guide
order: 0
order_enterprise: 142
meta_title: Interfaces
meta_description: "Build custom labeling interfaces with React"
section: "Interfaces"
---

You can create Interfaces in two ways. Pick whichever you prefer:

* **Within Label Studio** - Use Label Studio's out-of-the-box Interface builder to prompt our agent to build and refine your Interface. 

* **Developed locally** - Install the [`create-interface-skill`](/skills/interface.html) and use an agent of your choice to vibe code an Interface that you can then import into Label Studio. For more information, see [Develop an Interface locally](interfaces-local).

!!! note
    These are two independent ways to build the same thing, and the resulting Interface is identical either way; only the authoring path differs.

    On Label Studio Cloud, both **Create with Agent** and [Develop Locally](interfaces-local) are available; choose whichever fits your workflow. **Create with Agent** is a Label Studio Cloud feature: on self-hosted or on-prem deployments it isn't available, so build your Interface with **Develop Locally** instead.

<iframe width="560" height="315" src="https://www.youtube.com/embed/pdcZ6HDwpiI?si=ZpeuyeKYx71q1vCR" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe>

To begin, select **Interfaces** in the main menu and then select **Create Interface > Create with Agent**. 

!!! info Tip
    If you want to start from an existing labeling UI rather than a blank slate, select the overflow menu next to any Interface and select **Duplicate**. From here you can edit the copy. 






The Interface builder is comprised of the following tabs:

| Tab | Description |
| --- | --- |
| [**Chat**](#Chat) | Provide instructions to the Label Studio agent to build out your custom interface. |
| [**Code**](#Code) | View and manually update the React component created during your building session. |
| [**Version navigator**](#Version-navigator) | As you iterate on the interface via chat, you can navigate between versions of the preview. |
| [**Preview**](#Preview) | An interactive preview of your interface. |
| [**Input/Output**](#Input-Output) | Customize the input data you want to see in the preview, and see the output you would achieve as you interact with the preview. |
| [**Config**](#Config) | Once you have a basic version of the Interface built, you can use this tab to modify the interface rather than iterating through the chat. |
| [**Grid Cell**](#Grid-Cell) | If you enabled **Build custom grid view** in the settings, this tab lists each grid cell. |

## Chat

The **Chat** tab is where you collaborate with the agent to build the Interface.

In the message box at the bottom, describe the labeling UI you want. Be specific about:

- What the annotator should see (text, image, dialogue, side-by-side documents, etc.).
- What they are being asked to do (pick a label, highlight spans, fill a form, rank options).
- Any domain constraints (label names, required fields, rules).


### Plan mode

In Plan mode, the agent will surface:

- **Question cards** — Multiple-choice questions the agent uses to narrow the design. Click the options that apply and submit.
- **A plan summary card** — The proposed design. You can **Approve** it, **Approve with edits** (tweak the plan text first), or **Revise** (send feedback and let the agent propose again).

Once the plan is approved, the agent switches to code generation.

<img src="/images/interfaces/plan-mode.png" style="max-width: 500px" alt="Screenshot of Plan mode">


### Settings

You can select whether you want to build a custom grid view for your Interface.

Grid view allows you add a custom summary that appears in the Data Manager when viewing tasks in Grid View. This summary can be edited when modifying the Interface from the project settings. 

## Code

The **Code** tab is where you can view and manually update the React component created during your building session.

## Version navigator

As you update the Interface, the preview will update to reflect the changes. You can move between versions of the Interface using the controls above the chat panel:

<img src="/images/interfaces/versions.png" style="max-width: 500px" alt="Screenshot of Versions">

Use this to:

- Step through every snapshot the agent has produced and every manual edit you've made.
- Roll the preview back to an earlier version if a later change broke something.
- Pick any earlier snapshot as the basis for the next prompt.

!!! info Tip
    Versions you navigate to stay editable — you can keep refining from any point in history.

## Preview

Every time the agent writes or edits code, the **Preview** on the right re-renders using the current sample task data. Click around in the preview exactly as a labeler would — if something looks wrong, describe the fix in chat and the agent will patch the code.

If the code has a syntax or runtime error, an error message appears in the preview pane so you know what's broken.

!!! info Tip
    As you interact with the preview, check the **Input/Output** tab to see a preview of the annotation output in JSON format. Ensure this matches what you would expect.  

## Input/Output

| Section | Description |
| --- | --- |
| **Input** | The input data that the Interface reads from. You can edit the example input to see how the Interface behaves against realistic data. |
| **Output** | The output data that the Interface produces. As you interact with the interface under the **Preview** tab, this section automatically updates to show the output in JSON format. |

## Config 

The **Config** tab allows you to adjust the Interface's annotation controls. For example, you can add or delete choices, change the colors of labels, adjust the display text of labels, and more. The changes you make will be reflected in the preview and the code. 

These values can also be updated in the **Chat** panel by describing the changes you want to make, and can also be updated later from the project settings.

## Grid Cell

The **Grid Cell** tab only appears if you have enabled **Build custom grid view** in the settings. It lists each grid cell in the Interface.

---

### Source Guide Interfaces Details (`docs/source/guide/interfaces-details.md`)

---
title: View Interface details
short: Interface details
tier: enterprise
type: guide
order: 0
order_enterprise: 145
meta_title: Interfaces
meta_description: "View, manage, and version custom labeling Interfaces"
section: "Interfaces"
---

The Interface details page is the home base for a saved Interface. Use it to inspect the Interface, edit it, manage versions, and create projects from it.

To open it, select **Interfaces** in the main menu and then select the Interface you want to view.



## Actions

The top bar of the Interface details page provides the following actions:

| Button | Description |
| --- | --- |
| [**Iterate with Agent**](interfaces-agent) | Open the interface editor with the agent. This will open a new tab with the interface editor and the agent. |
| [**Develop Locally**](interfaces-local) | Install the `create-interface-skill` and work with a local coding agent to develop your Interface. |


And from the overflow menu in the top right corner of the page:

| Action | Description |
| --- | --- |
| **Edit** | Edit the Interface name and description. |
| **Create Project** | Starts a new project pinned to the version currently selected in the [version navigator](#Versions). For more information, see [Use an Interface in a project](./interfaces-project.md). |
| **Duplicate** | Creates a new copy of the Interface. |
| **Delete** | Delete the Interface. This action is not available if the Interface is used by any projects. |

!!! info Tip
    Template Interfaces can only be used and edited by duplicating them first. 

## Preview

The **Preview** section renders the Interface against the current sample task data, exactly as a labeler will see it. Click through the controls to confirm the Interface behaves the way you expect.

!!! info Tip
    The preview always reflects the version currently selected in the [version navigator](#Versions). Switch versions to compare how the Interface behaved at different points in its history.

## Versions

A list of versions appears to the right of the **Preview** section.

Select a version to update the **Preview**, **Data I/O**, and **Code** sections to that version's source.

You can also manage versions from the overflow menu next to that version:

| Action | Description |
| --- | --- |
| **Add Description** | Add a description to the version. |
| **Unpublish** | When you create projects from an Interface, project managers can switch between interface versions from the context of the project. <br /><br />Unpublishing a version will hide it from the version list under **Project > Settings > Labeling Interface.**   |

<img src="/images/interfaces/version-overflow.png" style="max-width: 500px" alt="Screenshot of Version overflow menu">


## Data I/O

The **Data I/O** section provides information about the required input and the expected output of the Interface.

Use this section to confirm the Interface reads the task fields you plan to import and produces the annotation shape downstream workflows (exports, analytics, Prompts) expect.

!!! info Tip
    You can edit the example task data to see how the Interface behaves against realistic data.

## Projects

The **Projects** section lists every project currently using this Interface, along with the version each project is pinned to, the creation date, and the task count.

From this section you can:

- Click a project to open it directly.
- Click **Create Project** to start a new project from the Interface. The project is pinned to the version currently selected in the version navigator.

!!! info Tip
    Editing an Interface does not retroactively update existing projects. Each project keeps running against its pinned version until you upgrade it from the project's labeling settings. See [Use an Interface in a project](interfaces-project) for more information.

## Code

The **Code** section shows the read-only JSX source for the selected version. Use it to:

- Audit what the Interface actually does before attaching it to a project.
- Copy snippets to share with teammates or paste into a duplicate.
- Confirm the Interface exposes the optional helpers you expect (such as `paramsSchema`, `outputSchema`, or `GridCell`).

---

### Source Guide Interfaces Libraries (`docs/source/guide/interfaces-libraries.md`)

---
title: Use external libraries and services in an Interface
short: External libraries & APIs
tier: enterprise
type: guide
order: 0
order_enterprise: 146
meta_title: Use external libraries and network services in Label Studio Interfaces
meta_description: "How to call external APIs, load third-party packages, or bundle libraries directly into a Label Studio Interface, and which security setting each one needs."
section: "Interfaces"
---

An Interface runs as a **sandboxed module** inside the Label Studio editor. By default that sandbox has **no outbound network and cannot load third-party code**. This is a deliberate security posture that you can relax when your Interface needs it.

Common cases are small and specific: fetching predictions from your ML backend, formatting dates with `date-fns`, validating a payload with `zod`, pulling map tiles from a tile server, or drawing a chart with `d3`. When your Interface needs an external API or a third-party library, you enable exactly what you need in one of three ways.

The single most common point of confusion: **"call an external API" and "use an external package" are configured in different places.** Allow-listing a host in the wrong section silently does nothing. This page explains which is which.

## The three paths at a glance

| You want to… | Example | Where to enable |
|---|---|---|
| **Call an external HTTP service** from your Interface | `fetch()` your ML backend, a geocoder, an internal microservice, a tile server | **Organization > Settings > Interfaces → API origins** |
| **Load a third-party package from a CDN** at runtime | Pull `three.js` or `d3` from `cdn.jsdelivr.net` via a `<script>` tag | **Organization > Settings > Interfaces → Advanced: external scripts** |
| **Use a third-party package with no external network** | Bundle `three.js` *into* the Interface and publish it | Nothing to configure. The code is self-contained. |

## Choosing a path

- **Calling a service over HTTP?** → **API origins** (Path 1).
- **Need a third-party library and OK trusting a CDN?** → **Advanced: external scripts** (Path 2).
- **Need a third-party library with no external dependencies / air-gapped / strict security?** → **Bundle locally** (Path 3). *Preferred default.*

You can also combine them. A point-cloud Interface might **bundle `three.js` locally** (path 3) *and* **allow-list your tile/asset server** under **API origins** (path 1) so it can fetch the data to render.

!!! info Can I use a third-party library?
    Yes. Interface source code can't use `import` / `require`, because each Interface is one self-contained module rather than an ES module. That's a limit on **module syntax**; third-party code is still fine. Bring a library in by loading it from an allow-listed CDN (path 2) or by bundling it into the Interface (path 3).

---

## Path 1: Call an external API or service (API origins)

Use this when your Interface needs to talk to an HTTP service over `fetch`, `XHR`, or `WebSocket`. For example: sending a selected region to an ML backend for inference, geocoding an address, or pulling map tiles.

1. Go to **Organization > Settings > Interfaces → API origins**.
2. Add the exact origin (scheme **and** host), e.g. `https://api.example.com`. One host per entry.
3. Save. Your Interface's `fetch()` calls to that origin now succeed.

| Rule | Detail |
|---|---|
| Format | Full origin including scheme (`https://host`). |
| Wildcards | Not allowed (no `*` or `*.example.com`). List each host explicitly. |
| Your LS server | Always allowed automatically. Don't add it; only list third-party hosts. |
| Empty list | **No outbound network.** The Interface can only reach your Label Studio server. |

**API origins** governs **data requests only**. It does **not** let you load a `<script>` from that host. If you add a package CDN here expecting a library to load, nothing happens (use Path 2 for that).

!!! info Example
    To connect a custom ML backend to an Interface, you allow-list the backend's origin here. That is what lets the Interface send region payloads to it and render the predictions. No package is being installed; it's an API call.

---

## Path 2: Load a package from a CDN (Advanced: external scripts)

Use this when you want the convenience of pulling a library straight from a CDN at runtime and you **trust that host completely**.

1. Go to **Organization > Settings > Interfaces → Advanced: external scripts**.
2. Turn on **Allow external scripts / stylesheets**.
3. Add one or more script origins, e.g. `https://cdn.jsdelivr.net`.
4. In your Interface, load the library from that origin (for example, inject a `<script>` and use its global).

| Rule | Detail |
|---|---|
| Scheme | Must be `https://`. HTTP origins are rejected. |
| Format | Origin only (no paths, query strings, credentials, or fragments). |
| Wildcards | Not allowed. |
| Toggle | Origins are ignored unless the checkbox is on. Turning it off clears the list on save. |

This is a different setting from **API origins**. A host listed here can serve executable code to your Interface; a host listed under **API origins** cannot, and vice versa.

!!! warning
    Scripts loaded from these origins run with the Interface's **full privileges** and can read any task data the Interface is rendering. Only add hosts you trust completely. Most organizations should leave this section disabled and prefer **Path 3**.

---

## Path 3: Bundle a package locally (nothing to allow-list, no network)

**Recommended for locked-down, on-premise, or air-gapped deployments, and for any team that would rather not open an outbound origin or trust a CDN.**

Because an Interface is a **single self-contained module**, a third-party library can travel *inside* your Interface code. The library becomes part of what you publish: no CDN, no `<script>` tag, no allow-list entry, and no outbound network at label time.

**How to do it**

1. Get a **browser build that attaches to a variable**: a prebuilt single-file `dist`, or produce one with a bundler (for example, `esbuild --bundle --format=iife --global-name=MyLib`). Interface source is compiled but **not** bundled for you, so resolve the dependency into this one file first. Avoid an **ES-module** build: `import` / `export` statements aren't supported.
2. Include that code in your Interface source and reference the library through the variable/global it defines. Do **not** use `import` / `require` in your own code.
3. Validate, preview, and publish as usual (via the Interfaces editor, or `label-studio-sdk interface validate/preview/sync`). The library ships with the published Interface.

**What works well this way**

- Pure-computation and rendering libraries: date/number utilities, validation (e.g. `zod`), charting/geometry, and in-browser renderers.
- Standard browser rendering APIs: **Canvas and WebGL are available in the sandbox**, so libraries like `three.js` can render inside an Interface. (The *iframe capabilities* list governs device features such as camera and microphone, not 2D/3D rendering.)

**Constraints to keep in mind**

- **Use an IIFE / global build.** `import` / `export` statements aren't supported, and a build that relies on top-level `module.exports` or `require(...)` may not pass validation. You also can't `require()` an npm package at label time; bundle the dependency into the file rather than requiring it, and reference the variable/global it defines.
- The library must run **in-browser within the sandbox**: no Node.js APIs, no access to the parent page's `window`/`document`, no persistent `localStorage`/`sessionStorage`, and any network the library itself performs is still governed by **API origins** (Path 1).
- The bundled code becomes part of the Interface's stored body. There's no fixed size limit, but very large bundles add to load and parse time, so keep an eye on it.

---

## Worked example: rendering large point clouds

Suppose you need to review LiDAR point clouds and expect to use `three.js` to render them.

- **Air-gapped / strict security:** **bundle `three.js` locally** (Path 3). Nothing is allow-listed; the renderer ships inside the Interface. If the point-cloud data is fetched from a service, add that data host under **API origins** (Path 1).
- **CDN acceptable:** allow-list the CDN under **Advanced: external scripts** (Path 2) and load `three.js` from there.

Point-cloud **rendering** works because WebGL is available in the sandbox. The real limit is **data size and performance**: very large clouds stress browser/WebGL memory regardless of how the library is packaged, so test with representative data and down-sample or tile if needed.

!!! info
    A heavy third-party renderer already ships inside a stock Interface today: the **Document AI** template renders PDFs with **PDF.js** from its bundled code. This is a concrete precedent that large libraries run inside Interfaces, including ones that compile internally or offload work to a web worker.

---

## Troubleshooting

| Symptom | Likely cause | Fix |
|---|---|---|
| `fetch()` to my service is blocked | The host isn't in **API origins**, or it's mistakenly listed under **external scripts**. | Add the exact origin under **API origins** (Path 1). |
| My CDN `<script>` won't load, but I added the host to **API origins** | **API origins** governs data requests, not script loading. | Add the host under **Advanced: external scripts** and enable the toggle (Path 2). |
| "Interfaces can't use packages" | Misreading the no-`import` rule as a ban on all third-party code. | You can't `import` npm packages, but you can bundle them in (Path 3) or load them from an allow-listed CDN (Path 2). |
| Bundled library works in the app but fails local validation | The build relies on top-level `module.exports` / `require`. | Use an **IIFE / global** build (e.g. `esbuild --format=iife --global-name=…`) and reference the variable it defines. |
| Library expects Node APIs or the parent `window` | It isn't browser/sandbox-compatible. | Use a browser build; keep any network calls behind **API origins**. |

---

### Source Guide Interfaces Local (`docs/source/guide/interfaces-local.md`)

---
title: Develop an Interface locally
short: Develop locally
tier: enterprise
type: guide
order: 0
order_enterprise: 143
meta_title: Interfaces
meta_description: "Build and iterate on custom labeling Interfaces from your terminal using the Label Studio SDK CLI"
section: "Interfaces"
---

You can build or iterate on an Interface from your own editor and your own coding agent (Claude, Codex, or Cursor) instead of using the in-product agent. **This is a first-class option on any deployment (many teams prefer it), and it's the way to build Interfaces on self-hosted or on-prem (including air-gapped) deployments, where Create with Agent (a Label Studio Cloud feature) isn't available.**

You write JSX on disk, preview it live against your Label Studio instance, and sync the result back when you're ready to publish.

Developing locally is useful when you want to:

- Use a coding agent or IDE you already have set up.
- Work offline against a checked-out copy of an Interface, then push the changes back.
- Bring an existing React mockup into Label Studio.
- Run automated validation and Playwright scenarios against your Interface before publishing.

<iframe width="560" height="315" src="https://www.youtube.com/embed/-xOjbN4PAhk?si=CUtfpYCxYQc0lRmx" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe>

## 1. Install the label-studio-sdk package and check your setup

Before you begin, make sure you have:

- **The [`label-studio-sdk` Python package](https://github.com/HumanSignal/label-studio-sdk).** If you already have it installed, ensure you version 2.0.22 or later. 
   ```bash
   pip install --upgrade label-studio-sdk
   # or
   poetry add label-studio-sdk
   ```
- Node.js and npm available on your `PATH`.
- Your Label Studio Enterprise URL and an [API token](access_tokens).
- A coding agent installed (Claude, Codex, or Cursor) if you want agent assistance.

To begin, select **Interfaces** in the main menu and then select **Create Interface > Develop Locally**. 

!!! info Tip
    To iterate on an Interface that already exists in Label Studio, open the Interface details page and click **Develop Locally** in the top bar. The dialog respects the version currently selected in the [version navigator](interfaces-details#Versions).

## 2. Install the create-interface skill

Install the [`create-interface-skill`](/skills/interface.html) for your local coding agent. The skill includes the commands and conventions for working with Interfaces locally so your agent can author a valid Interface module.

In the **Develop Locally** modal, choose your agent and copy the install command:

| Agent | Install command |
| --- | --- |
| **Claude** | `npx skills add humansignal/create-interface-skill --skill create-interface-skill -g -a claude-code` |
| **Codex** | `npx skills add humansignal/create-interface-skill --skill create-interface-skill -g -a codex` |
| **Cursor** | `npx skills add humansignal/create-interface-skill --skill create-interface-skill -g -a cursor` |

Run the command in your terminal, then restart your agent so that it picks up the new skill. You only need to install the skill once per agent.

!!! info Tip
    You can skip this step if you'd rather write the Interface by hand. The skill is optional — it just gives your agent context.

## 3. Initialize the Interface

Use the SDK-provided command to initialize the Interface in a local directory:

```bash
export LABEL_STUDIO_URL="https://your-instance.humansignal.com"
export LABEL_STUDIO_API_KEY="<your token>"
label-studio-sdk interface init ./my-interface
cd ./my-interface
label-studio-sdk interface preview .
```

`interface init` scaffolds a starter `Screen.jsx`, `task.json`, and `scenarios.js` so you have a working baseline to build from.

!!! info Tip
    You can also pass `--lse-url` and `--token` to individual commands instead of exporting them. This is useful if you work against more than one Label Studio instance.


#### Start from an existing Interface

To iterate on an Interface that already exists in Label Studio, open the Interface details page and click **Develop Locally** in the top bar. The dialog respects the version currently selected in the [version navigator](interfaces-details#Versions).

Alternatively, you can use:
    
```bash
label-studio-sdk interface pull --id <interface-id> --version <version-id> ./my-interface
```
    
The dialog pre-fills the `--id` and `--version` for you. Without `--version`, `pull` grabs the latest version of the Interface, including any unpublished local drafts.

## 4. Check local setup

Run the following command to check your local setup:

```bash
label-studio-sdk interface doctor
```

The first validation or doctor run installs the Node validator dependencies into a user cache directory.

## 5. Start the live preview

Change into your Interface directory and run the preview command:

```bash
cd ./my-interface
label-studio-sdk interface preview .
```

This opens a **playground** in Label Studio (at the `interfaces/playground` URL) that is live-connected to your local environment. The playground watches your source files — every save reflects your latest changes in the preview in real time.

For example, if your Label Studio instance is at `https://your-instance.humansignal.com`, the playground will be at `https://your-instance.humansignal.com/interfaces/playground`.

## 6. Develop in your local environment

With the playground running, you can work in your familiar local tools — your IDE, terminal, or a coding agent on your machine — rather than editing in-product. Save your changes as you would for any other project, and the playground re-renders against the current sample task data.

## 7. Validate the Interface

Before syncing your changes, validate the Interface to catch any issues:

```bash
label-studio-sdk interface validate .
```

Static validation checks JSX compilation, the final module shape, schema exports, and runs smoke tests against `getResults` and `parseResults`. To also run any browser-driven Playwright scenarios you've defined in `scenarios.js`, add `--scenario`:

```bash
label-studio-sdk interface validate . --scenario scenarios.js
```

Validation also runs as part of `sync`, so you can skip this step if you'd rather catch issues at sync time. Use `--no-validate` on `sync` to bypass the check entirely.

## 8. Sync your changes back to Label Studio

Once your local code is ready, run the sync command to synchronize your local directory with the Label Studio instance:

| Mode | Sync command |
| --- | --- |
| **New Interface** | `label-studio-sdk interface sync . --title "My Interface" --workspace <id> --publish` |
| **Existing Interface** | `label-studio-sdk interface sync . --message "Describe the change"` |

Note the following:

* By default, `sync` creates an **unpublished local draft** so you can review the version in Label Studio before publishing. 

* Add `--publish` to publish the version immediately and make it available to new projects. The Develop Locally dialog includes `--publish` automatically when starting a new Interface from scratch and omits it when iterating on an existing one.

* For new Interfaces, the `--workspace` flag is optional. If you don't specify a workspace, the Interface will be created [in the **Shared** scope](interfaces#Interface-scope) (it can be used by projects in any workspace).

Useful flags for `sync`:

| Flag | Description |
| --- | --- |
| `--publish` | Publish the version immediately instead of leaving it as a draft. |
| `--message "..."` | Store a history message alongside the synced version. |
| `--dry-run` | Print the planned action without writing to the server. |
| `--workspace <id>` / `--workspace-title "..."` | Place a newly created Interface in a specific workspace. |
| `--force` | Upload even when the local source hash hasn't changed. |
| `--no-validate` | Skip the validation gate (compilation is still required). |

After a successful sync, the CLI writes a sidecar file (`Screen.jsx.ls-interface.json`) next to your source. 

The sidecar is keyed by Label Studio base URL and remembers the Interface ID, workspace, source version, and last pushed source hash, so future `sync`, `start`, and `open` commands don't need `--id`.

!!! info Tip
    Click **Refresh Versions** in the **Develop Locally** modal after a sync to see your new draft or published version appear on the Interface details page.

## Useful CLI commands

The full command reference is in the [**Interface CLI Guide**](https://github.com/HumanSignal/label-studio-sdk/blob/master/interface-cli.md). The most useful commands while developing locally are:

| Command | Description |
| --- | --- |
| `interface init <dir>` | Scaffold a starter Interface (`Screen.jsx`, `task.json`, `scenarios.js`). |
| `interface pull --id <id> <dir>` | Download an existing Interface to a local directory. Defaults to the latest version (including drafts) when `--version` isn't set. |
| `interface preview <dir>` | Open the live playground and watch the source for changes. |
| `interface validate <dir>` | Run static validation and (optionally) Playwright scenarios. |
| `interface sync <dir>` | Push local changes back to Label Studio. Creates a draft by default; add `--publish` to publish. |
| `interface start <dir>` | Sync the Interface, create a project that uses it, and open the project's Data tab. |
| `interface open <dir>` | Open the saved Interface in Label Studio using the local sidecar. |
| `interface doctor` | Check Node.js, npm, the API token, and connectivity. |

## Troubleshooting

| Error | What to do |
| --- | --- |
| `node is required for interface validation` | Install Node.js and make sure `node` is on your `PATH`. |
| `npm is required to install validator dependencies` | Install npm and re-run `interface doctor`. |
| `source did not compile` | Fix JSX syntax errors. Interface files are compiled as self-contained JSX snippets; `import` / `require` and bundler-specific syntax aren't supported. To use a third-party library, **pre-bundle it into the single source file** (or load it from an allow-listed host). See [Use external libraries and services](interfaces-libraries). |
| `Module did not return an object` | Make the final expression in the file an object literal wrapped in parentheses, for example `({ default: Screen })`. |
| `Missing getResults` or `Missing parseResults` | Warnings during static validation, but required for scenario validation. Add both functions when you need reliable save/load behavior. |
| `no sidecar entry` | Run `interface sync` against the target Label Studio URL, or pass `--id` to commands that support it. |
| `multiple workspaces titled ...` | Pass `--workspace` with the numeric workspace ID instead of `--workspace-title`. |

---

### Source Guide Interfaces Project (`docs/source/guide/interfaces-project.md`)

---
title: Create a project from an Interface
short: Interface projects
tier: enterprise
type: guide
order: 0
order_enterprise: 144
meta_title: Create projects from Interfaces
meta_description: "Create and update projects using Label Studio Interfaces"
section: "Interfaces"
---

Once you have created an Interface, you can create a project from it.

!!! note
    You can only use one Interface per project. However, you can customize the Interface and switch between versions in the project settings.


## Create a project from an Interface

* From the Interfaces list, click the overflow menu and select **Create Project**.
* From the Interface details page, select the **Projects** tab and click **Create Project**.

| Field | Description |
| --- | --- |
| **Workspace** | The workspace where the project will be created. You can only select workspaces for [shared Interfaces](interfaces#Interface-scope). |
| **Project Title** | The name of the project. |
| **Description** | A description of the project. |

## What each project inherits

The project owns its own copy of the Interface code, pinned to the version you selected. 

Note the following:

- **Later edits to the Interface don't retroactively change existing projects.** 

    A project keeps running its pinned version until someone explicitly changes versions. You can do this from the version selector under **Project > Settings > Labeling Interface**.
- **Configuration lives with the project.** 

    Any configurable options the Interface exposes (label names, colors, display text, which task data field holds the primary content, etc.) are stored per project. Editing one project's configuration does not affect any other project, even projects that share the same Interface and version.


## Configure the project's Interface

For projects created from Interfaces, you can adjust the UI from **Settings > Labeling Interface**. 

Adjustments you make from the project context will only affect the Interface for that project, and will not iterate the Interface version. 

If you want to iterate the Interface version, you can do so from the [Interface details page](interfaces-details). Click **Open Interface**. 

### Source Interface

From here you can see information about the Interface such as its name, creation date, and the version that you are using in the project. 

Use the drop-down to switch between versions.

### Interface configuration

From here you can configure labeling elements such as label names, colors, and display text.

---

### Source Guide Interfaces (`docs/source/guide/interfaces.md`)

---
title: Interfaces overview
short: Overview
tier: enterprise
type: guide
order: 0
order_enterprise: 141
meta_title: Interfaces
meta_description: "Build custom labeling interfaces with React"
section: "Interfaces"
---

The Interfaces builder allows you to build highly customized labeling UIs to use across your Label Studio projects. 

Instead of choosing a template and tuning [out-of-the-box XML tags](/tags), with Interfaces you describe the experience you want and Label Studio builds a fully interactive React-based labeling UI that you can preview, refine, save, and share across projects.

<iframe width="560" height="315" src="https://www.youtube.com/embed/pdcZ6HDwpiI?si=ZpeuyeKYx71q1vCR" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe>

!!! note Availability
    Interfaces are available in Label Studio Enterprise (including self-hosted) and Starter Cloud. If you're on a self-hosted or on-prem deployment and don't see Interfaces, contact your account manager to enable it for your organization.

    There are two ways to build an Interface:

    - **Create with Agent**: the in-product builder. Available on **Label Studio Cloud only**.
    - **[Develop Locally](interfaces-local)**: build with your own coding agent and sync the result to Label Studio. Available on any deployment.

    On Label Studio Cloud you can use either path, whichever you prefer. On self-hosted or on-prem deployments (including air-gapped), **Develop Locally is the way to build Interfaces**, since Create with Agent is a Cloud-only feature.

    Everything else about Interfaces (versioning, projects, exports) works the same across both paths. See [Create with the agent](interfaces-agent).

## When to use Interfaces

Interfaces are Label Studio's most powerful UI builder. Use them when:

* [Label Studio's XML tags](/tags) don't support your use case. For example, if you want to annotate data types that do not have a corresponding object tag (3D files, GEOTiff, DICOM, etc). 
* When you want to be able to vibe code your labeling UI either within Label Studio or using your own tools and agent outside of Label Studio. To work with a coding agent like Claude Code, Codex, or Cursor, install the [`create-interface-skill`](/skills/interface.html) — or see the full [agent skills overview](/skills/).
* When you need to be able to version your labeling UIs. 
* When you need more custom logic. 

For a full comparison between Interfaces and classic Label Studio tags, see [Interfaces vs. classic tags](#Interfaces-vs-classic-tags).


## Project and feature integration

Once saved, you can use the Interface to create new projects (subject to the workspace scope — see [Interface scope](#Interface-scope)). The project uses the Interface to render the labeling UI.

Because annotations are still stored as regular Label Studio results, standard review workflows and exports continue to work.

| Feature | Notes |
|---------|-------|
| **Prompts** | The Interface's output schema instructs Prompts how to turn an LLM's structured response into Label Studio annotations, so you can pre-label tasks and have humans review. |
| **Agreements** | You can view and explore agreement as you would with any other task. <br/>However, you cannot customize per-control-tag agreement metrics. |
| **Analytics** | You can use the dashboards to visualize annotation and review work as you would with any other Label Studio project. |
| **Project settings** | There is no change or restriction on project settings when configuring a project that has an Interface attached. |
| **Plugins** | Plugins are not supported with Interfaces. Instead, you can directly include plugin-type functionality within your Interface code. |
| **XML tags** | Interfaces do not incorporate classic XML tags and cannot be combined with a classic XML labeling interface.   |

## Interfaces vs. classic tags

| Feature | Classic XML tags | Interfaces |
| --- | --- | --- |
| Format | XML tag library | React / agent-generated code |
| How to build | Configure XML or template selection | Natural-language prompting via the Label Studio agent or coding agent with skills.md |
| Iteration speed | Moderate | Extremely fast (prompt → refine → ship) |
| Multi-modal support | Supported, but constrained | Fully flexible and supports data not supported by traditional XML tags (3D data, geotiff, etc) |
| Business logic | Implemented via plugins as separate code that hooks into the configuration | Written directly into the Interface component |
| AI automation | Connect models via ML backend | ML backend, native model integration, supported in Prompts for LLM-as-a-judge pre-labeling or evaluation with |
| Reusability across projects | Configuration is project-scoped; reuse means duplication and drift | Interfaces are first-class objects — port one to any project without copying |
| Versioning | Limited | Built-in version history with one-click rollback |
| Access control | Project-level | Workspace- or project-level access on the Interface itself |
| Backend integration | Yes | Yes  |
| Agent compatibility | No | Yes (Label Studio agent or select your own) |
| Guardrails | Tag library enforces compatibility | Agent runs separate from your production environment |



## Interface scope

There are template Interfaces, shared Interfaces, and Interfaces scoped to specific workspaces. 

Template Interfaces are the quickest way to get started; shared and workspace scopes let teams standardize on a shared set of labeling experiences.

| Scope        | Who can see it                          | Who can edit it                                  |
|--------------|-----------------------------------------|--------------------------------------------------|
| **Templates**   | Everyone in your organization          | No one — HumanSignal-maintained, read-only. However, you can duplicate a system interface and edit the copy. |
| **Shared**   | Everyone in your organization           | Anyone with the appropriate [permissions](#User-access)          |
| **Workspace**| Members of the chosen workspace only    | Owners and Admins, or Managers who are members of that workspace                        |

Note the following:

- **Template interfaces are read-only.** You cannot modify or delete a HumanSignal-provided template — duplicate it first and edit the copy.

- **Scope changes are structural.** Moving an Interface between shared and workspace scope can change who is able use it to create new projects.  


## Security

### User access

| Role | Access |
|------|--------|
| **Owners and Admins** | Can create and edit shared Interfaces and for any workspace |
| **Managers** | Can create and edit shared Interfaces and for any workspace in which they are a member |
| **Annotators and Reviewers** | Do not have access to the Interfaces feature |

### Admin controls

You can enable or disable the ability to create and edit Interfaces for your organization. For more information, see [Interfaces settings](admin-interfaces).

### Sandboxing

Interfaces run as sandboxed modules rendered inside the Label Studio editor shell. To keep them safe, portable, and upgrade-friendly, the runtime enforces a narrow contract:

- **React-only source, closed by default.** The code runs with `React` and the standard hooks (`useState`, `useRef`, `useEffect`, `useCallback`, `useMemo`) plus a small `getField` helper. You can't `import` npm packages in the source. You *can* still use a third-party library by **bundling it into the Interface** or by **loading it from a host you allow-list**. See [Use external libraries and services](interfaces-libraries).
- **Plain JavaScript with JSX.** No TypeScript, no type annotations, no `import` / `export` statements. The module is compiled and evaluated at runtime, and must expose a default component plus optional helpers (`getResults`, `parseResults`, `outputSchema`, `inputSchema`, `paramsSchema`, `GridCell`).
- **No network access by default.** Out of the box an Interface can only reach your Label Studio server. To let it call an external service (for example an ML backend), an Owner/Admin allow-lists that origin under **Organization > Settings > Interfaces → API origins**. See [Use external libraries and services](interfaces-libraries).
- **Limited persistence.** State lives only for the duration of a labeling session. Anything you want to keep has to come back out through the annotation results.

---

### Source Guide Label Studio Compare (`docs/source/guide/label_studio_compare.md`)

---
title: Compare Label Studio Editions
short: Compare editions
type: guide
tier: all
order: 28
order_enterprise: 28
section: "Discover & Learn"
meta_title: Label Studio Community and Enterprise Features
meta_description: Compare the features of Label Studio Community Edition with the paid versions so that you can choose the best option for your data labeling and annotation projects.
---

Label Studio is available to everyone as open source software (Label Studio Community Edition). There are also two paid editions: Starter Cloud and Enterprise. 

<a class="Button" href="#Feature-comparison" target="_blank" style="margin-bottom: 2em;">See full feature comparison</a>

## At-a-glance



<a class="Button" href="https://humansignal.com/" target="_blank" style="margin-bottom: 2em;">Learn about Enterprise</a>

## Feature comparison

<table>
<thead>
  <tr>
    <th>Functionality</th>
    <th>Community</th>
    <th>Starter Cloud</th>
    <th>Enterprise</th>
  </tr>
  </thead>
  <tr>
    <td colspan="4" style="text-align:center"><b>User Management</b></td>
  </tr>
  
  <tr>
    <td><b>Role-based workflows</b><br/><a href="https://docs.humansignal.com/guide/project_settings_lse#Annotation">Role-based automated workflows for annotators and reviewers.</a></td>
    <td style="text-align:center">❌</td>
    <td style="text-align:center">Limited</td>
    <td style="text-align:center">✅</td>
  </tr>
  <tr>
    <td><b>Role-based access control</b><br/><a href="https://docs.humansignal.com/guide/admin_roles">Role-based access control into workspaces and projects: Admin, Manager, Reviewer, and Annotator.</a></td>
    <td style="text-align:center">❌</td>
    <td style="text-align:center">✅</td>
    <td style="text-align:center">✅</td>
  </tr>
  <tr>
    <td><b>Customizable permissions</b><br/><a href="https://docs.humansignal.com/guide/admin_permissions">Fine-grained control over which roles can execute specific actions, such as inviting members, managing cloud storage, deleting tasks, and more.</a></td>
    <td style="text-align:center">❌</td>
    <td style="text-align:center">❌</td>
    <td style="text-align:center">✅</td>
  </tr>
  <tr>
    <td colspan="4" style="text-align:center"><b>Data Management</b></td>
  </tr>
  <tr>
    <td><b>Data management view</b><br/><a href="manage_data.html">View and manage datasets and tasks in a project through the Data Manager view.</a></td>
    <td style="text-align:center">✅</td>
    <td style="text-align:center">✅</td>
    <td style="text-align:center">✅</td>
  </tr>
  <tr>
    <td><b>Multiple data formats</b><br/><a href="https://labelstud.io/playground/"> Label any data type from text, images, audio, video, time series data to multimodality.</a></td>
    <td style="text-align:center">✅</td>
    <td style="text-align:center">✅</td>
    <td style="text-align:center">✅</td>
  </tr>
  <tr>
    <td><b>Import data</b><br/><a href="tasks.html">Reference data stored in your database, cloud storage buckets, or local storage and label it in the browser.</a></td>
    <td style="text-align:center">✅</td>
    <td style="text-align:center">✅</td>
    <td style="text-align:center">✅</td>
  </tr>
    <tr>
    <td><b>Advanced cloud storage integrations</b><br/><a href="storage.html">Databricks Unity Catalog, IAM for AWS S3, Azure Blob Storage with Service Principal, and WIF for Google Cloud Storage connections.</a></td>
    <td style="text-align:center">❌</td>
    <td style="text-align:center">❌</td>
    <td style="text-align:center">✅</td>
  </tr>
  <tr>
    <td><b>Import pre-annotated data</b><br/><a href="predictions.html">Import pre-annotated data (predictions) into Label Studio for further refinement and assessment.</a></td>
    <td style="text-align:center">✅</td>
    <td style="text-align:center">✅</td>
    <td style="text-align:center">✅</td>
  </tr>
  <tr>
    <td><b>Export data</b><br/><a href="export.html">Export annotations as common formats like JSON, COCO, Pascal VOC and others.</a></td>
    <td style="text-align:center">✅</td>
    <td style="text-align:center">✅</td>
    <td style="text-align:center">✅</td>
  </tr>
  <tr>
    <td><b>Sync data</b><br/><a href="storage.html">Synchronize new and labeled data between projects and your external data storage.</a></td>
    <td style="text-align:center">✅</td>
    <td style="text-align:center">✅</td>
    <td style="text-align:center">✅</td>
  </tr>
  <tr>
    <td><b>Chat</b><br/><a href="/tags/chat.html">Generate and label conversation data. Interact live with an LLM of your choice.</a></td>
    <td style="text-align:center">❌</td>
    <td style="text-align:center">Limited</td>
    <td style="text-align:center">✅</td>
  </tr>
  <tr>
    <td colspan="4" style="text-align:center"><b>Project Management</b></td>
  </tr>
  <tr>
    <td><b>Organize data in projects</b><br/><a href="setup_project.html">Projects to manage data labeling activities.</a></td>
    <td style="text-align:center">✅</td>
    <td style="text-align:center">✅</td>
    <td style="text-align:center">✅</td>
  </tr>
  <tr>
    <td><b>Organize projects in workspaces</b><br/><a href="https://docs.humansignal.com/guide/workspaces">Organizes related projects by team, department, or product. Users can only access workspaces they associated with. </a></td>
    <td style="text-align:center">❌</td>
    <td style="text-align:center">❌</td>
    <td style="text-align:center">✅</td>
  </tr>
  <tr>
    <td><b>Personal sandbox workspace</b><br/><a href="https://docs.humansignal.com/guide/workspaces">Personal sandbox workspace for project testing and experimentation. </a></td>
    <td style="text-align:center">❌</td>
    <td style="text-align:center">✅</td>
    <td style="text-align:center">✅</td>
  </tr>
  <tr>
    <td><b>Templates</b><br/><a href="/templates">Templates to set up data labeling projects faster.</a></td>
    <td style="text-align:center">✅</td>
    <td style="text-align:center">✅</td>
    <td style="text-align:center">✅</td>
  </tr>
  <tr>
    <td><b>AI assistant</b><br/><a href="https://docs.humansignal.com/guide/ask_ai">Use an LLM trained by HumanSignal to help you create and refine templates.</a></td>
    <td style="text-align:center">❌</td>
    <td style="text-align:center">✅</td>
    <td style="text-align:center">✅</td>
  </tr>
  <tr>
    <td><b>Project membership</b><br/><a href="https://docs.humansignal.com/guide/project_settings_lse#Members">Only users who are added as members to a project can view it. </a></td>
    <td style="text-align:center">❌</td>
    <td style="text-align:center">✅</td>
    <td style="text-align:center">✅</td>
  </tr>
  <tr>
    <td><b>Project-level roles</b><br/><a href="https://docs.humansignal.com/guide/project_settings_lse#Members">Annotator and Reviewer can be assigned to Annotator/Reviewer roles at a per-project level. </a></td>
    <td style="text-align:center">❌</td>
    <td style="text-align:center">✅</td>
    <td style="text-align:center">✅</td>
  </tr>
  <tr>
    <td><b>Project-level user settings</b><br/><a href="https://docs.humansignal.com/guide/project_settings_lse#Annotation">Multiple configuration options for how Annotators and Reviewers interact with tasks and what information they can see.</a></td>
    <td style="text-align:center">❌</td>
    <td style="text-align:center">Limited</td>
    <td style="text-align:center">✅</td>
  </tr>

<tr>
    <td colspan="4" style="text-align:center"><b>Data Labeling Workflows</b></td>
  </tr>
  <tr>
    <td><b>Assign tasks</b><br/><a href="https://docs.humansignal.com/guide/manage_data#Assign-annotators-to-tasks">Assign tasks to certain annotators or reviewers.</a></td>
    <td style="text-align:center">❌</td>
    <td style="text-align:center">✅</td>
    <td style="text-align:center">✅</td>
  </tr>
  <tr>
    <td><b>Automatically assign tasks</b><br/><a href="https://docs.humansignal.com/guide/setup_project#Configure-high-impact-settings">Set rules and automate how tasks are distributed to annotators.</a></td>
    <td style="text-align:center">❌</td>
    <td style="text-align:center">✅</td>
    <td style="text-align:center">✅</td>
  </tr>
  <tr>
    <td><b>Simplified interface for Annotators</b><br/>Annotator-specific labeling view that only shows assigned tasks.</td>
    <td style="text-align:center">❌</td>
    <td style="text-align:center">✅</td>
    <td style="text-align:center">✅</td>
  </tr>
  <tr>
    <td><b>Bulk labeling</b><br/><a href="https://docs.humansignal.com/guide/labeling_bulk">Classify data in batches.</a></td>
    <td style="text-align:center">❌</td>
    <td style="text-align:center">❌</td>
    <td style="text-align:center">✅</td>
  </tr>
   <tr>
    <td colspan="4" style="text-align:center"><b>Customization & Development</b></td>
  </tr>
 <tr>
    <td><b>Programmable & embeddable interfaces</b><br/><a href="https://docs.humansignal.com/tags/reactcode">Build fully customized labeling and evaluation interfaces with React. Embed into your own applications.</a></td>
    <td style="text-align:center">❌</td>
    <td style="text-align:center">❌</td>
    <td style="text-align:center">✅</td>
  </tr>
  <tr>
    <td><b>Tag library</b><br/><a href="/tags">Use our tag library to customize the labeling interface by modifying pre-built templates or by building your own templates.</a></td>
    <td style="text-align:center">✅</td>
    <td style="text-align:center">✅</td>
    <td style="text-align:center">✅</td>
  </tr>
  <tr>
    <td><b>White labeling</b><br/>Use your company colors and logo to give your team a consistent experience. (Additional cost)</td>
    <td style="text-align:center">❌</td>
    <td style="text-align:center">❌</td>
    <td style="text-align:center">✅</td>
  </tr>
  <tr>
    <td><b>Plugins</b><br/><a href="https://docs.humansignal.com/guide/plugins">Use JavaScript to further enhance and customize your labeling interface.</a></td>
    <td style="text-align:center">❌</td>
    <td style="text-align:center">❌</td>
    <td style="text-align:center">✅</td>
  </tr>
  <tr>
    <td><b>API/SDK & webhooks</b><br/><a href="api.html"> APIs, SDK, and webhooks for programmatically accessing and managing Label Studio.</a></td>
    <td style="text-align:center">✅</td>
    <td style="text-align:center">✅</td>
    <td style="text-align:center">✅</td>
  </tr>
  <tr>
    <td colspan="4" style="text-align:center"><b>Prompts</b></td>
  </tr>
   <tr>
    <td><b>Automated pre-labeling</b><br/><a href="https://docs.humansignal.com/guide/prompts_overview">Rapidly pre-label tasks using LLMs.</a></td>
    <td style="text-align:center">❌</td>
    <td style="text-align:center">Limited Trial</td>
    <td style="text-align:center">✅</td>
  </tr>
  <tr>
    <td><b>LLM fine-tuning and evaluation</b><br/><a href="https://docs.humansignal.com/guide/prompts_draft">Evaluate and fine-tune LLM prompts against a ground truth dataset.</a></td>
    <td style="text-align:center">❌</td>
    <td style="text-align:center">Limited Trial</td>
    <td style="text-align:center">✅</td>
  </tr>
  <tr>
    <td><b>Run benchmarks</b><br/><a href="https://docs.humansignal.com/guide/prompts_draft#Evaluation-results">Compare model outputs against ground truth or rubric criteria.</a></td>
    <td style="text-align:center">❌</td>
    <td style="text-align:center">Limited Trial</td>
    <td style="text-align:center">✅</td>
  </tr>
  <tr>
    <td colspan="4" style="text-align:center"><b>Machine Learning</b></td>
  </tr>
  <tr>
    <td><b>Custom ML backends</b><br/><a href="ml.html">Connect a machine learning model to the backend of a project.</a></td>
    <td style="text-align:center">✅</td>
    <td style="text-align:center">✅</td>
    <td style="text-align:center">✅</td>
  </tr>
  <tr>
    <td><b>Active learning loops</b><br/><a href="https://docs.humansignal.com/guide/active_learning.html">Accelerate labeling with active learning loops.</a></td>
    <td style="text-align:center">❌</td>
    <td style="text-align:center">❌</td>
    <td style="text-align:center">✅</td>
  </tr>
  <tr>
    <td><b>Predictions from connected models</b><br/><a href="ml.html#Pre-annotations-predictions">Automatically label and sort tasks by prediction score with the ML model backend.</a></td>
    <td style="text-align:center">✅</td>
    <td style="text-align:center">✅</td>
    <td style="text-align:center">✅</td>
  </tr>
  <tr>
    <td colspan="4" style="text-align:center"><b>Analytics and Reporting</b></td>
  </tr>
  <tr>
    <td><b>Project dashboards</b><br/><a href="https://docs.humansignal.com/guide/dashboards">Dashboards for monitoring project progress.</a></td>
    <td style="text-align:center">❌</td>
    <td style="text-align:center">❌</td>
    <td style="text-align:center">✅</td>
  </tr>
  <tr>
    <td><b>Annotator performance dashboards</b><br/><a href="https://docs.humansignal.com/guide/dashboard_annotator">Dashboards to review and monitor individual annotator performance.</a></td>
    <td style="text-align:center">❌</td>
    <td style="text-align:center">❌</td>
    <td style="text-align:center">✅</td>
  </tr>
  <tr>
    <td><b>Activity logs</b><br/><a href="https://docs.humansignal.com/guide/admin_logs">Activity logs for auditing annotation activity by project.</a></td>
    <td style="text-align:center">❌</td>
    <td style="text-align:center">❌</td>
    <td style="text-align:center">✅</td>
  </tr>
  <tr>
    <td><b>Annotation history</b><br/>View annotation history from the labeling interface.</td>
    <td style="text-align:center">❌</td>
    <td style="text-align:center">✅</td>
    <td style="text-align:center">✅</td>
  </tr>
  <tr>
    <td colspan="4" style="text-align:center"><b>Quality Workflows</b></td>
  </tr>
  <tr>
    <td><b>Assign reviewers</b><br/><a href="https://docs.humansignal.com/guide/quality.html">Assign reviewers to review, fix and update annotations.</a></td>
    <td style="text-align:center">❌</td>
    <td style="text-align:center">✅</td>
    <td style="text-align:center">✅</td>
  </tr>
  <tr>
    <td><b>Automatic task reassignment</b><br/><a href="https://docs.humansignal.com/guide/quality.html">Reassign tasks with low agreement scores to new annotators.</a></td>
    <td style="text-align:center">❌</td>
    <td style="text-align:center">❌</td>
    <td style="text-align:center">✅</td>
  </tr>
  <tr>
    <td><b>Agreement metrics</b><br/><a href="https://docs.humansignal.com/guide/stats.html">Define how annotator consensus is calculated using pre-defined agreement metrics.</a></td>
    <td style="text-align:center">❌</td>
    <td style="text-align:center">Limited</td>
    <td style="text-align:center">✅</td>
  </tr>
  <tr>
    <td><b>Custom agreement metrics</b><br/><a href="https://docs.humansignal.com/guide/custom_metric.html">Write your own custom agreement metric.</a></td>
    <td style="text-align:center">❌</td>
    <td style="text-align:center">❌</td>
    <td style="text-align:center">✅</td>
  </tr>
  <tr>
    <td><b>Comments and notifications</b><br/><a href="https://docs.humansignal.com/guide/comments_notifications.html">Team collaboration features like comments and notifications on annotation tasks.</a></td>
    <td style="text-align:center">❌</td>
    <td style="text-align:center">✅</td>
    <td style="text-align:center">✅</td>
  </tr>
  <tr>
    <td><b>Identify ground truths</b><br/><a href="https://docs.humansignal.com/guide/quality.html">Mark which annotations should be included in a Ground Truth dataset.</a></td>
    <td style="text-align:center">❌</td>
    <td style="text-align:center">✅</td>
    <td style="text-align:center">✅</td>
  </tr>
  <tr>
    <td><b>Overlap configuration</b><br/><a href="https://docs.humansignal.com/guide/project_settings_lse#Quality">Set how many annotators must label each sample.</a></td>
    <td style="text-align:center">❌</td>
    <td style="text-align:center">✅</td>
    <td style="text-align:center">✅</td>
  </tr>
  <tr>
    <td><b>Pause annotators</b><br/><a href="https://docs.humansignal.com/guide/quality#Pause-an-annotator">Pause an individual annotator's progress manually or based on pre-defined behaviors.</a></td>
    <td style="text-align:center">❌</td>
    <td style="text-align:center">❌</td>
    <td style="text-align:center">✅</td>
  </tr>
  <tr>
    <td><b>Annotation limits</b><br/><a href="https://docs.humansignal.com/guide/project_settings_lse#annotation-limit">Set a limit on how many annotations a user can submit within a project before their work is paused.</a></td>
    <td style="text-align:center">❌</td>
    <td style="text-align:center">❌</td>
    <td style="text-align:center">✅</td>
  </tr>
  <tr>
    <td><b>Annotator consensus matrices</b><br/><a href="https://docs.humansignal.com/guide/quality#Review-annotator-agreement-matrix">Matrices used to compare labeling results by different annotators.</a></td>
    <td style="text-align:center">❌</td>
    <td style="text-align:center">❌</td>
    <td style="text-align:center">✅</td>
  </tr>
  <tr>
    <td><b>Label distribution charts</b><br/><a href="https://docs.humansignal.com/guide/dashboard_distribution">Identify possible problems with your dataset distribution, such as an unbalanced dataset.</a></td>
    <td style="text-align:center">❌</td>
    <td style="text-align:center">❌</td>
    <td style="text-align:center">✅</td>
  </tr>
  <tr>
    <td colspan="4" style="text-align:center"><b>Security and Support </b></td>
  </tr>
  <tr>
    <td><b>SSO</b><br/><a href="https://docs.humansignal.com/guide/auth_setup.html">Secure access and authentication of users via SAML SSO or LDAP.</a></td>
    <td style="text-align:center">❌</td>
    <td style="text-align:center">❌</td>
    <td style="text-align:center">✅</td>
   </tr>
  <tr>
    <td><b>SOC2</b><br/><a href ="https://heartex.com/security"> SOC2-compliant hosted cloud service or on-premise availability</a></td>
    <td style="text-align:center">❌</td>
    <td style="text-align:center">❌</td>
    <td style="text-align:center">✅</td>
  </tr>
  <tr>
    <td><b>Support portal</b><br/>Access to a dedicated support portal.</td>
    <td style="text-align:center">❌</td>
    <td style="text-align:center">✅</td>
    <td style="text-align:center">✅</td>
  </tr>
  <tr>
    <td><b>Uptime SLA</b><br/>99.9% uptime SLA</td>
    <td style="text-align:center">❌</td>
    <td style="text-align:center">❌</td>
    <td style="text-align:center">✅</td>
  </tr>
  <tr>
    <td><b>Customer Success Manager</b><br/>Dedicated customer success manager to support onboarding, education, and escalations.</td>
    <td style="text-align:center">❌</td>
    <td style="text-align:center">❌</td>
    <td style="text-align:center">✅</td>
  </tr>
</table>

---

### Source Guide Labeling (`docs/source/guide/labeling.md`)

---
title: Labeling guide
tier: all 
type: guide
order: 134
order_enterprise: 134
meta_title: Label and annotate data
meta_description: Label and annotate data to create bounding boxes, label text spans, set up relations. Filter and sort project data for machine learning dataset creation.
section: "Create & Manage Projects"

---

Label and annotate your data with the open source data labeling tool, Label Studio. After you [set up your project](setup_project.html) and [labeling interface](setup.html) and [import your data](tasks.html), you can start labeling and annotating your data.

1. Open a project in Label Studio and optionally [filter or sort the data](manage_data.html#Filter-or-sort-project-data).    
2. Click **Label All Tasks** to [start labeling](#Start-labeling).
3. Use [keyboard shortcuts](#Use-keyboard-shortcuts-to-label-regions-faster) or your mouse to label the data and submit your annotations.
4. Follow the project instructions for labeling and deciding whether to skip tasks. 
5. Click the project name to return to the data manager.

You can also [collaborate with other annotators](#Label-with-collaborators) to improve the quality of your labeled data. 


## Start labeling

Some labeling tasks can be complicated to perform, for example, labeling that includes text, image, and audio data objects as part of one dataset and labeling task, or creating relations between annotations on a labeling task. This section includes guidance on how to perform more complex labeling tasks, such as labeling with relations, overlapping regions, selected tasks, or changing labels. 

### Choose which tasks to label
From a project, click **Label All Tasks** to start labeling all tasks. To label the tasks as they are filtered and sorted in the data manager, select **Label Tasks As Displayed** instead. 

You can also label a specific task in the **Quick View** or **Preview** by clicking it from the project data manager view, but you won't automatically see the next task in the labeling queue after submitting your annotations. 

You can also select the checkboxes next to specific tasks and then click **Label $n Tasks** to label the selected number of tasks. For example, select the checkboxes for 5 different tasks, then click **Label 5 Tasks** to label only those 5 tasks. 

!!! note
    When labeling tasks, you should not open the label stream (meaning to click **Label All Tasks**) simultaneously in two tabs. This could result in you receiving the same task twice, which can circumvent project settings that address annotator overlap. 

### Label a region in the data
Annotate a section of the data by adding a region. 

1. Select the label you want to apply to the region. For some configurations, you can skip this step.
2. Click the text, image, audio, or other data object to apply the label to the region. Your changes save automatically. 
3. Click **Submit** to submit the completed annotation and move on to the next task.

### Label overlapping regions
When you label with bounding boxes and other image segmentation tasks, or when you're highlighting text for NLP and NER labeling, you might want to label overlapping regions. To do this easily, hide labeled regions after you annotate them. 

1. Select the label that you want to apply to the region.
2. Draw the bounding box or highlight the text that you want to label. 
3. In the **Regions** or **Labels** sidebar, locate and select the region that you labeled and click the eye icon to hide the region.
4. Or press cmd or ctrl to draw over the existing **Regions** or **Labels** (NOTE: This do not work for Keypoints)
5. Select the next label that you want to apply to the overlapping region.
6. Draw the bounding box or highlight the text that you want to label.
7. Continue hiding and labeling regions until you've completed annotating the task. If you want, select the eye icon next to **Regions** to hide and then show all regions labeled on the task to confirm the end result.
8. Click **Submit** to submit the completed annotation and move on to the next task.

### Select multiple regions
You can select multiple regions while labeling to make changes to them together. 

1. After creating multiple regions, press `ctrl` and click each region that you want to select. You can select regions on the object that you're labeling or in the **Regions** sidebar. Select a range of regions in the **Regions** sidebar by clicking the first region in the list that you want to select and holding `Shift` while you click the last region in the list that you want to select. 
2. After selecting the regions you can apply a label to all selected regions or delete them. 
3. Click a selected region or press `u` to deselect it.

### Duplicate regions
You can duplicate a region to create many identically-sized polygons, rectangles, or ellipses. 

1. Select the region that you want to duplicate.
2. Press `ctrl + d` to duplicate the region, or the sequence of `ctrl + c` and `ctrl + v` to copy and paste the region. Duplicated regions appear in the exact location as the region being duplicated. If you're using a Mac keyboard, use `cmd` instead of `ctrl`. 
3. Click the arrow tool, then move and resize the newly-created region.

### Change the label
You can change the label of an existing region. 

1. Select the labeled region, for example a span, bounding box, image segment, audio region, or other region, on the object or using the **Regions** sidebar.
2. Select a new label. Your changes to the label save automatically. 
3. Click **Submit** to submit the completed annotation and move on to the next task.

### Delete a labeled region
You can delete labeled regions, such as bounding boxes or text spans, if needed.  

1. Select the labeled region on the object or in the **Regions** sidebar. 
2. Press the Backspace key, or go to the **Results** panel and remove the selected annotation.

### Delete an annotation
After labeling a task, you can delete the annotation by clicking the trash can button to delete an annotation on the task. If you haven't saved your annotation yet, click the `X` icon to reset the task and remove your labeling activity.

You can also delete all annotations on a task from the project page. See [Delete tasks or annotations](manage_data.html#Delete_tasks_or_annotations).

### Add relations between annotations
You can create relations between two results with both directions and labels. To add labels to directions, you must set up a labeling config with the relations tag. See more about [relations with labels](/tags/relations.html) in the Tags documentation.

1. Select the region for the annotation that you want to relate to another annotation. If you're creating a direction-based relation, select the first one first. 
2. In the **Regions** section of the **Results** sidebar, click the **Create Relation** button that looks like a hyperlink icon.
3. Select the second region for the annotation to complete the relation.

<br>
<img src="../images/relation.png" class="gif-border">
<i>Figure 1: Add relations between annotations </i>

After you relate two annotation regions, you can modify the relation in the **Relations** section of the **Results** sidebar. 
- To change the direction of the relation, click the direction button between the two related regions.
- To add labels to the direction arrow indicating the relation between two annotations, click the vertical ellipsis button next to the two related regions to add your predefined labels. You must have a [label configuration that includes relations](/tags/relations.html) to do this.

### Exiting a labeling flow 
When annotators are not finished with their annotation work but would like to exit their labeling flow to, for example, pause the [lead time](/guide/task_format.html#Relevant-JSON-property-descriptions) calculation on their annotations, they can click the drop-down menu next to **Submit** and then select **Submit and Exit** (or **Update and Exit**). This will submit the current annotation and redirect them to the Projects page. 

If they are not done with their annotation, then they can simply check to make sure their draft was saved in the history panel and then navigate to the Projects page to stop the [lead time](/guide/task_format.html#Relevant-JSON-property-descriptions) calculation from running. 

<img src="../images/submit-and-exit.png" class="gif-border">


## Label with collaborators

In both Label Studio and Label Studio Enterprise, you can label tasks with collaborators. Tasks are locked while someone performs annotations so that you don't accidentally overwrite the annotations of another annotator. After the other annotator finishes with the task, it can appear in your queue for labeling if the minimum annotations per task is set to more than one. By default, tasks only need to be annotated by one annotator. 

<div class="enterprise-only">

If you're using Label Studio Enterprise and want more than one annotator to annotate tasks, <a href="setup_project.html">update the project settings</a>. After you update the minimum annotations required per task, annotators can use the Label Stream workflow to label their tasks.  

</div>

If you want to label tasks more than once, even if the minimum annotations required is set to one, do the following:

To label tasks multiple times while the minimum annotations required is set to one, do the following:
1. In the data manager for the project, click a task to open the quick labeling view.
2. Click the `+` icon next to the task annotation ID to open an annotation tab. 
3. Label the task.
4. Click **Submit** to save your annotation.
5. Click the next task in the data manager to open the quick labeling view for that task and repeat steps 2-4.


## Perform ML-assisted labeling with interactive preannotations

If you have a machine learning backend set up to [get interactive preannotations](ml.html#Get-interactive-preannotations), you can choose whether to use those predictions while you label. 

1. After you start labeling, you can enable **Auto-Annotation** to see and use the smart option to assign a label to draw a shape, mask, or assign a keypoint. After using the smart option to draw on an image, or labeling a text or HTML span, the ML backend returns predictions.  
2. For image labeling, you can choose whether to **Auto accept annotation suggestions** after you enable auto-annotation. If you automatically accept annotation suggestions, regions show up automatically and are immediately created. If you don't automatically accept suggestions, the regions appear, but you can reject or approve them manually, either individually or all at once. Predicted text regions are automatically accepted.


## Customize the labeling interface 

Click the settings icon when labeling to configure the labeling interface to suit your labeling use case. 

For example, keep a label selected after creating a region, display labels on bounding boxes, polygons and other regions while labeling, and show line numbers for text labeling.


<img style="max-width: 600px" src='../images/lsf-settings.png' class="gif-border">
<br/>
<i>Figure 2: Customize the labeling interface  </i>

You can also modify the layout of the screen, hide or show predictions, annotations, or the results panel, and hide or show various controls and buttons.


## Advanced image labeling

If you want to perform advanced image labeling, follow these examples and guidance for assistance. 

### Add multiple types of regions to image annotations
You can add multiple types of regions to image annotations. You can add any of the following:
- Rectangles
- Ellipses
- Keypoints
- Polygons
- Brush masks

To add different types of regions to your image annotations, follow this example.

Create a custom template for your labeling interface using the following example:

```xml
<View>
  <Image name="image" value="$image" />
  <Rectangle name="rect" toName="image" />
  <Ellipse name="ellipse" toName="image" />
  <KeyPoint name="kp" toName="image" />
  <Polygon name="polygon" toName="image" />
  <Brush name="brush" toName="image" />
  <Choices name="choices" toName="image">
    <Choice value="yes"></Choice>
    <Choice value="no"></Choice>
  </Choices>
  <Labels name="labels" toName="image" fillOpacity="0.5" strokeWidth="5">
    <Label value="building" background="green"></Label>
    <Label value="vehicle" background="blue"></Label>
  </Labels>
</View>
```
This example makes rectangles, ellipses, polygons, keypoints, and brush masks available to the annotator, along with image classification choices of yes and no, and region labels of building and vehicle.

### Faster image labeling
You can add a rectangle or an ellipse to your image with just two clicks, or double click to create a polygon, rectangle, or ellipse. 

If you accidentally select a point on an image while creating a polygon, just double click to remove the erroneous point and continue creating the region. There must be at least three points on the polygon to be able to remove a point.

### Create regions without labels
When you're annotating images, you can create regions without applying labels. 

1. Create a custom template using the example provided above under [Add multiple types of regions to image annotations](#Add-multiple-types-of-regions-to-image-annotations).
2. Select which tool you want to use in the labeling toolbar. You only need to do this for the first task. 
3. Create a region by double-clicking or clicking and dragging to create a bounding box, or click the points necessary to construct a polygon.
4. Select the created region in the sidebar or on the image.
5. Select the label that you want to apply to the region.
6. Repeat these steps for any regions that you want to create.

This can be helpful for two-step labeling, where you want one annotator to create regions and another annotator to label the regions. 

### Erase brush mask labels
If you make a mistake when labeling with the brush mask, you can erase it. You must select a brush region in the sidebar before you can erase any part of it. 

If you want to completely remove a region and start over, delete the region instead of erasing it. Erasing a region does not delete it. 

## Outliner 

The **Outliner** is a replacement for the existing interface with a multiple-panel approach. This feature improves the user experience to fit more data in these two columns (**Outliner** and **Details**) and provides more options when working with regions. For example, Image segmentation is the only type of labeling that supports custom controls on the details panel. 

### Feature capabilities

Outliner is the area where you can see all the details about annotation, regions, and labeling history. It is split into two panels, named **Outliner** and **Details**. Figure 3 shows the appearance of the **Outliner** panel on the right of the image and **Details** panel on the left of the image. 

<br>
<img src="/images/outliner-details-panels-view.png" style="opacity: 0.8" class="gif-border"/>
<i>Figure 3: View of Outliner and Details panels view </i>
 
**Panel actions**

There are two panels which is the outliner for the regions list. These panels can be collapsed, moved, swapped to the sides, dragged, dropped, and expanded to provide a full-screen view. For example, if you navigate to label stream, you will have more space in this mode. 

The **Outliner** and **Details** panels are collapsible and detachable, so you can arrange them the way you want. Figure 4 shows the **Collapse details** button which allows you to collapse the **Details** panel and use the remaining portion of the screen. 

<br>
<img src="/images/collapse-details-button.png" style="opacity: 0.8" class="gif-border"/>
<i>Figure 4: Collapsed details button</i>

Now, the **Details** panel is collapsed, and you can expand it by clicking the **Expand Details** icon.
<br>
<img src="/images/details-collapsed-button-view.png" style="opacity: 0.8" class="gif-border"/>
<i>Figure 5: Details panel in collapsed view</i>

The details panel is displayed in expanded view to explore the feature. 

<br>
<img src="/images/details-panel-expanded-view.png" style="opacity: 0.8" class="gif-border"/>
<i>Figure 6: Details panel in expanded view</i>

To collapse the **Outliner** panel, click on the **Collapse outliner** icon. 

<br>
<img src="/images/collapse-outliner-button.png" style="opacity: 0.8" class="gif-border"/>
<i>Figure 7: Collapse outliner button</i>

Now, the **Outliner** panel is collapsed, and you can expand it by clicking the **Expand Details** icon.
<br>
<img src="/images/outliner-collapsed-view.png" style="opacity: 0.8" class="gif-border"/>
<i>Figure 8: Outliner panel in collapsed view</i>

The **Expand Details** icon expands the **Outliner** panel for further actions.

<br>
<img src="/images/expand-outliner-button.png" style="opacity: 0.8" class="gif-border"/>
<i>Figure 9: Outliner panel with expand outliner icon</i>

The Outliner panel is displayed in expanded view to explore the feature. 
<br>
<img src="/images/outliner-expanded-view.png" style="opacity: 0.8" class="gif-border"/>
<i>Figure 10: Outliner panel in expanded view</i>

These panels are detachable, so you can place them wherever you want on the screen, and you can resize them. 

<br>
<img src="/images/details-floating-dock.png" style="opacity: 0.8" class="gif-border"/>
<i>Figure 11: Details panel in floating dock view</i>
 
**Region editor** 

You can edit regions with a focus on labeling. You can zoom in and precise control over the numbers and dimensions of each region attribute. The image region details like height, rotation, and so on can be changed using the inputs in the **Details** panel. Now, you can collapse all panels and get the maximum working area. This feature design makes some actions clear in the product user interface.

<br>
<img src="/images/region-editor.png" style="opacity: 0.8" class="gif-border"/>
<i>Figure 12: Edit regions using different attributes</i>


!!! note 
    In the upcoming releases, video segmentation will be available to control keyframes, animations, and custom metadata. 

<div class="enterprise-only"> <p>The <b>Comments</b> box is available inside the outliner. </p></div>

<br>
<img src="/images/comments-box.png" style="opacity: 0.8" class="gif-border"/>
<i>Figure 13: Comments box </i>

!!! note
    Use Outliner to work with larger annotation tasks (many bounding boxes in one image, larger videos, and so on).

To add a region in the OCR transcription templates, draw a rectangle, and you can see a text box that appears in the **Outliner** panel to enter OCR text. The new functionality allows you to select a region and the **Details** panel changes. You can experience the following:

- See a text that is attached to the OCR case.
- Control the coordinates of the rectangle. For example, you can rotate, move, and resize the coordinates. 
- Lock the region to prevent it from moving. 
- Hiding/locking regions.
- Use the move tool to prevent the movement of regions. 
- Hide or show a selected region.

<br>
<div style="margin:auto; text-align:center;"><img src="/images/text-inside-outliner.png" style="opacity: 0.8" class="gif-border"/></div>
<i>Figure 13: Text inside outliner</i>

**Regions grouping or sorting**

You can group or sort regions using the following functionality from the drop-down list. 

- **Group Manually**: Manual regions grouping (drag-and-drop). When you select this option, you can drag and drop regions inside each other, create a tree structure, and group the regions together. 
- **Group by Tool**: When you select this option, the tools (rectangle, polygons) are added. You can use a template with all these tools to group regions by a tool that generates them.
- **Group by Label**: When you select this option and draw a region, now you will see a text label, and you can expand it. The text label shows only the text related to this functionality. 

!!! warning
    **Group by Label** and **Group by Tool** options does not work for multiple image scenario or multiple labeled type scenario; only the **Group Manually** option is available for this mode.  
    
You can also order your regions by **Order by Score** or **Order by Time** options. 

<br>
<img src="/images/group-by-label-sorted-by-score-expanded-dog-icon.png" style="opacity: 0.8" class="gif-border"/>
<i>Figure 14: Group by label using sorted by score</i>

<br>
<img src="/images/group-by-tool-sorted-by-score.png" style="opacity: 0.8" class="gif-border"/>
<i>Figure 15: Group by tool using sorted by score</i>

<br>
<img src="/images/group-by-label-sorted-by-score.png" style="opacity: 0.8" class="gif-border"/>
<i>Figure 16: Group by label using sorted by score</i>

<br>
<img src="/images/manual-grouping.png" style="opacity: 0.8" class="gif-border"/>
<i>Figure 17: Manual grouping</i>

**Prediction indication of the region**
This capability shows whether an ML model created the region or not.  

**Relations list**
One region belongs to another one. It is typically done for NLP type of text. For example, consider the city name as region 1 and district name as region 2. Now, in the relations list hierarchy you will place the region 2 under region 1. Relation lists can be unidirectional or bidirectional between one region and another. You can snap one region into another region. 

!!! note
    The relations list does not support multi-level listing for regions. 

**Draw bounding boxes over another**

!!! attention "important"
    When creating bounding box annotations for images, you must not start a bounding box inside another bounding box because it would just drag the first box. Instead, it would create a new bounding box in front of the image when the rectangle tool is selected.

To create bounding boxes on top you can use two ways:

1. Press the rectangle tool (R). 

!!! note 
    This option is not always visible.

2. Label a bounding box, and then label another bounding box inside it.

Now, you should be able to drag when the rectangle tool is not selected. Remember that when the rectangle tool is selected, you can make rectangles even if it is on top of another rectangle.

OR 

1. Press CTRL button and start drawing bounding box over another one. 

{% insertmd includes/annotation_ids.md %}

---

### Source Guide Labeling Bulk (`docs/source/guide/labeling_bulk.md`)

---
title: Bulk labeling
short: Bulk labeling
tier: enterprise
type: guide
order: 0
order_enterprise: 137
meta_title: Bulk Labeling
meta_description: Bulk labeling in Label Studio
section: "Create & Manage Projects"
parent: "labeling"
parent_enterprise: "labeling"
date: 2025-05-24 17:19:21
---

Bulk labeling is useful when you have a large set of tasks that share a common label or annotation that you want to apply in one go. For example:

* **Data classification**: Quickly label multiple tasks with the same category. 
* **Filtering and grouping**: Use filters to isolate tasks matching specific criteria (e.g., tasks you know are all “Approved”), then label them in bulk. 
* **Speed and consistency**: Avoid repetitive labeling for items that can confidently share a label.

## Supported data types

Bulk labeling supports all data types (e.g. images, text, audio, video). 

## Supported user roles

Only users who are in the Reviewer, Manager, Admin, or Owner role can use the bulk labeling actions available from the Data Manager. 

!!! info Tip
    Users in the Annotator role can still use [plugin-enabled bulk labeling actions](/plugins/bulk_labeling). 

## Supported labeling tasks

Bulk labeling supports classification tasks and text area input. 

You can use any combination of supported control tags, and all object tags (e.g. `<Image>`, `<Text>`, `<Video`) are supported. 

The following control tags can be assigned through the bulk labeling action:

* [Choices](/tags/choices.html)
* [DateTime](/tags/datetime.html)
* [Number](/tags/number.html)
* [Rating](/tags/rating.html)
* [Taxonomy](/tags/taxonomy.html)
* [TextArea](/tags/textarea.html)

If your labeling configuration includes control tags not listed above, the **Bulk labeling** action is not accessible. 
 

!!! note Limitations
    Note the following:
    * At this time, only global classification tasks are supported. This means that the control tag cannot have `perItem` or `perRegion` enabled. 
    * Lead time is not calculated for annotations performed through the bulk labeling option. 

## Bulk labeling panel 

When you perform bulk labeling, your options open in a side panel. Note that you can expand or collapse this panel as necessary by clicking the expand/collapse icons or by pressing `shift` + `.`.



## List view

You can perform bulk labeling from the list view in the Data Manager. This is especially useful if you want to use filters to label a subset of results. 

Select the checkboxes next to the tasks you want to annotate and then click the drop-down menu next to **Label *n* Tasks**. Select **Bulk label**. This opens a panel with your labeling options (see [above](#Bulk-labeling-panel)).  



Make your selections and click **Submit**, 

## Grid view

You can use the grid view to visually inspect and then select tasks for bulk annotation. 

To select tasks from the grid view, you can click the checkboxes next to them or open each and select tasks as you navigate through them. 



Once selected, click the drop-down menu next to **Label *n* Tasks**. Select **Bulk label**. This opens a panel with your labeling options (see [above](#Bulk-labeling-panel)).

!!! info Tip
    There are hotkeys for selecting the task, scrolling, zooming, and navigating between tasks. To see a list of the available hotkeys, click the help icon above the task:

---

### Source Guide Llms (`docs/source/guide/llms.txt`)

# Label Studio Documentation
> Label Studio is a data labeling platform for training and evaluating machine learning, AI and agentic systems. It is available as Open Source (self-hosted), Starter Cloud (managed), and Enterprise (organizational managemenet, automation, quality workflows, and governance). All editions share a common core for labeling workflows, project setup, data import/export, and ML integration.
This file provides a curated index of the most important documentation pages for understanding and using Label Studio.
Prefer:
- Markdown versions of pages over HTML
- Documentation that matches the user’s deployment model (Open Source vs Cloud vs Enterprise)
---
## How to choose the right docs
- Use **Open Source docs** also referred to as Community Edition for:
  - installation, deployment, Kubernetes, storage, upgrades
  - self-hosted environments and infrastructure questions for community edition
- Use **Enterprise docs** for:
  - organization management, permissions, security
  - workspaces, dashboards, analytics, quality, and automation workflows
  - SSO, LDAP, SCIM, and enterprise integrations
- Use **either** for:
  - projects, labeling interfaces, data import/export
  - annotation workflows and ML backends
If unsure:
- assume **Enterprise / Cloud** for hosted users
- assume **Open Source** for infrastructure or deployment questions
---
## Editions
- **Open Source**: self-hosted Label Studio with full control over deployment and infrastructure
- **Starter Cloud**: managed Label Studio with core functionality and reduced operational overhead
- **Enterprise**: advanced features for teams and organizations, including automation, analytics, security, and governance
---
## Core concepts (shared across all editions)
- [Projects](https://labelstud.io/guide/setup_project.md): Create and manage labeling projects
- [Labeling interface](https://labelstud.io/guide/setup.md): Configure annotation UI and schema
- [Tasks](https://labelstud.io/guide/tasks.md): Structure and manage labeling data
- [Import data](https://labelstud.io/guide/tasks.md): Ingest data into projects
- [Export annotations](https://labelstud.io/guide/export.md): Export labeled data
- [Machine learning integration](https://labelstud.io/guide/ml.md): Connect predictions and models
- [Data Manager](https://labelstud.io/guide/manage_data.md): Browse and filter tasks
---
## Open Source (self-hosted)
- [Open Source docs home](https://labelstud.io/guide/index.md)
- [Install Label Studio](https://labelstud.io/guide/install.md)
- [Kubernetes deployment](https://labelstud.io/guide/install_k8s.md)
- [Start Label Studio](https://labelstud.io/guide/start.md)
- [Upgrade Label Studio](https://labelstud.io/guide/upgrade_community.md)
- [Persistent storage](https://labelstud.io/guide/persistent_storage.md)
- [Database setup](https://labelstud.io/guide/storedata.md)
- [Security](https://labelstud.io/guide/security.md)
- [Storage integrations](https://labelstud.io/guide/storage.md)
---
## Enterprise / Cloud or On-Prem
- [Enterprise docs home](https://docs.humansignal.com/guide/index.md)
- [Workspaces](https://docs.humansignal.com/guide/workspaces.md)
- [Dashboards](https://docs.humansignal.com/guide/dashboards.md)
- [Analytics](https://docs.humansignal.com/guide/dashboard_overview.md)
- [Automation](https://docs.humansignal.com/guide/prompts_overview.md)
- [Review workflows](https://docs.humansignal.com/guide/quality.md)
- [Quality & agreement](https://docs.humansignal.com/guide/agreement_metrics.md)
- [User management](https://docs.humansignal.com/guide/manage_users.md)
- [Roles & permissions](https://docs.humansignal.com/guide/admin_roles.md)
- [SSO, LDAP, SCIM](https://docs.humansignal.com/guide/admin_auth.md)
- [Activity logs](https://docs.humansignal.com/guide/admin_logs.md)
---
## Developer & integrations
- [API reference](https://api.labelstud.io/)
- [Python SDK](https://docs.humansignal.com/guide/sdk.md)
- [Webhooks](https://docs.humansignal.com/guide/webhooks.md)
- [ML backend](https://labelstud.io/guide/ml_create.md)
- [Open source repository](https://github.com/HumanSignal/label-studio)
---
## Optional
- [Glossary](https://labelstud.io/guide/glossary.md)
- [Templates](https://labelstud.io/templates/index.md)
- [Enterprise On-Prem Release notes](https://docs.humansignal.com/guide/release_notes.md)
- [Community](https://community.labelstud.io/)

---

### Source Guide Manage Data (`docs/source/guide/manage_data.md`)

---
title: Use the Data Manager in projects
short: Data Manager
type: guide
tier: all
order: 122
order_enterprise: 122
meta_title: Use the Data Manager in projects
meta_description: Manage, filter, and sort project data for your labeling project.
section: "Create & Manage Projects"
parent: "manage_projects_lso"
parent_enterprise: "manage_projects" 
---

The Data Manager page is where you can view all your labeling tasks, sort and filter your data, import and export data, and perform various actions related to tasks. 

For information on setting up a project, see [Create and configure projects](setup_project). 

<div class="opensource-only">



</div>


<div class="enterprise-only">



</div>

Every row in the data manager represents a labeling task in your dataset.

<div class="enterprise-only">

## Task states

Each task progresses through a series of states as you take actions on it. For more information, see [Project and Task State Management](project_states). 

</div>

## Filter or sort project data

With filters and tabs, you can split data into different sections to be labeled by different annotators, or set up data in a particular order to perform labeling according to prediction score, or another value in your data.

When you filter or sort the data before you label it, you modify which tasks and the order of the tasks you see when labeling. While [task sampling](https://labelstud.io/guide/start#Set-up-task-sampling-for-your-project) affects the task order for an entire project and can't be changed, filtering and sorting tasks can be changed at any time. 

<img src="/images/data-manager-filters.png" class="gif-border">

<div class="opensource-only">

!!! error Enterprise
    In Label Studio Enterprise and Starter Cloud, you can use advanced filters against annotation results. For more information, see our [Enterprise documentation](https://docs.humansignal.com/guide/manage_data#Filter-annotation-results). 

</div>

<div class="enterprise-only">

### Filter annotation results

You can also filter on individual annotation results within a task:

<img src="/images/data-manager-filters-lse.png" class="gif-border" >

!!! note
    In Label Studio Community, the `Annotation results` filter is an unstructured text search across all annotations for the task, and the example above would not be achievable.

The following tags are supported:
- All `Labels` tags (ex. `Labels`, `ParagraphLabels`, ...)
- `Choices`
- `Taxonomy`
- `Rating`

**Known limitations:**
- [Taxonomies loaded using `apiUrl`](/templates/taxonomy) will not be detected.


</div>

### Example: Label new data first
Sort the data in your project by date to focus on labeling the newest data first.

1. In a project, update the **Order** of the data from the default to **Created at**.
2. Update the order of the items to be in ascending order, so the newest items appear first. 
3. Select **Label Tasks As Displayed** to start labeling tasks from newest to oldest. 

### Example: Sort by prediction score
You can sort the data in your project by prediction score if you upload [pre-annotated data](predictions.html) with prediction scores, or if your [machine learning backend](ml.html) produces prediction scores as part of the model output. 

1. In a project, update the **Order** of the data from the default to use the **Prediction score** field.
2. Update the order of the items in either ascending or descending order to label based on higher confidence or lower confidence predictions. 
3. Select **Label Tasks As Displayed** to start labeling tasks in prediction score order. 
You can also use [task sampling](https://labelstud.io/guide/start#Set-up-task-sampling-for-your-project) to use prediction score ordering.

### Example: Split a dataset using tabs and filters
If you want to label a large dataset, you might want to use tabs and filters to split it up into smaller sections, and assign different annotators to different tabs. You can't assign annotators to specific tasks in Label Studio Community Edition, but you can rename the tabs after specific annotators as a way to basically assign tasks using tabs.  

For example, you might split a dataset with 300 images into 3 different tabs, and have different annotators focus on each tab:
1. In a project, create a filter where the **ID** field **is between** the values "1" and "100". Click away from the filter to review filtered items the tab.
2. Click the vertical ellipsis for the tab and select **Rename**. Name it after a specific annotator that you want to focus on the items in that tab.
3. Click the **+** icon to create a new tab. Click the vertical ellipsis for the new tab and select **Rename** to name it after a second annotator.
4. On the new tab, create a filter where the **ID** field **is between** the values "101" and "200". Click away from the filter to review the filtered items on the tab.
5. Click the **+** icon to create a new tab. Click the vertical ellipsis for the new tab and select **Rename** to name it after a third annotator.
6. On the new tab, create a filter where the **ID** field **is between** the values "201" and "300". Click away from the filter to review the filtered items on the tab.
7. Any annotator can log in and navigate to the relevant tab for their work and click the **Label** button to start labeling the subset of tasks on their tab.

## Save filters as tabs

You can create tabs on the data manager to [split your dataset](#Example-Split-a-dataset-using-tabs-and-filters) for labeling, to separate tasks by status (annotated, predicted, unlabeled), or other reasons. 

Tabs that you create depend on your [labeling configuration setup](setup.html), because the labeling configuration defines the data fields available for filtering and sorting. 

!!! note 
    If you make changes to the labeling configuration, any tabs in your data manager are removed. Make sure to finish the project setup before setting up complex tabs in the Data Manager.

<div class="enterprise-only">

## Assign annotators to tasks

In Label Studio Enterprise, if you're an administrator or project manager, you can assign annotators to tasks in a specific project. After you [add the annotator to the project or workspace](setup_project.html#Add-members-to-a-project), assign the annotator to specific tasks. You must assign annotators to tasks if you're using [manual task distribution](setup_project.html#Set-up-task-distribution-for-labeling).

1. For a specific project, select tasks on the Data Manager.
2. Select the dropdown and choose **Assign Annotators**.
3. Select names of annotators and click the `>` arrow to assign them to the selected tasks.
4. Click **Assign**. 

If you want to bulk assign tasks, you can use filters to identify exactly the tasks that you want to assign, then select the top checkbox on the data manager to select all tasks shown and assign them to an annotator.

</div>


## Delete tasks or annotations
If you have duplicate tasks, or want to remove annotations, you can delete tasks and annotations from Label Studio.

1. In Label Studio UI, open the project you want to update.
2. Filter the Data Manager page to show only the data you want to delete. For example, specific annotations, or tasks annotated by a specific annotator. 
3. Select the checkboxes for the tasks or annotations that you want to delete.
4. Select the dropdown with the number of tasks, and choose **Delete tasks** or **Delete annotations**. 
5. Click **Ok** to confirm your action.

If you want to make changes to the labeling interface or perform a different type of data labeling, first select all the annotations for your dataset and delete the annotations.

<div class="enterprise-only">

## Agreement columns

The agreement columns in the Data Manager reflect consensus between annotators for a task. For more information on agreement and how it is calculated, see [Task agreement](stats).

You will see the following agreement columns in the Data Manager: 

* **Agreement** - The is the overall agreement for the task. 

    This is calculated as the mean agreement score between all control tags for a particular task. See [Overall agreement](stats#Overall-agreement).
* **[Control tag] agreement** - Each control tag has its own agreement score. 

    How control tag agreement is calculated depends on how your project is set up. See [Per-control-tag agreement](stats#Per-control-tag-agreement).



### Annotators and models

Click any agreement column to select specific annotators and models that you want to use for agreement calculation.



By default, all annotators (and not models) are selected for agreement calculation.

However, you can customize this to select a subset of annotators, models, or models and annotators to compare. 

For example, if you have 10 annotators and you select 3, the overall agreement score and the control tag agreement scores will be recalculated to reflect only your selections. 

!!! note
    You must select at least two items to compare. 

    Your selections will apply to all agreement columns in the Data Manager. You cannot select different annotators and models for different agreement columns.


### Ground truth match

If your project contains ground truth annotations, you can use this option to compare either a single annotator or a single model to ground truth annotations. 

Label Studio will apply whatever agreement metrics and methodology you have configured for your project, but will limit the calculation to the selected annotator or model and the annotations marked as ground truth.  

<img src="/images/review/agreement-dm-gt.png" class="gif-border" style="max-width:679px">

### Agreement popover

Click any agreement column to see a popover that has information about the metric and methodology used. 

If you are using **Pairwise** methodology, you will see a breakdown of agreement scores for the selected annotators and models.

<img src="/images/review/agreement-dm-popover.png" class="gif-border" style="max-width:600px">


</div>

---

### Source Guide Manage Projects (`docs/source/guide/manage_projects.md`)

---
title: Manage projects in Label Studio Enterprise
short: Manage projects
tier: enterprise
type: guide
order: 0
order_enterprise: 116
meta_title: Manage projects in Label Studio Enterprise
meta_description: A description of various project management actions available.
section: "Create & Manage Projects"
---

For more project actions, see:

* [Create a project](setup_project#Create-a-project)
* [Add members to a project](setup_project#Add-members-to-a-project)
* [Publish a project](setup_project#Publish-a-project)
* [Assign annotators to project tasks](manage_data#Assign-annotators-to-tasks)

You can use the **All Projects** page to view projects across all workspaces. 



## Search projects

Use the search box in the upper right. You can search by project name. 

* To search within a workspace, use the search field on the workspace page. 
* To search across all projects, including those within archived workspaces, go to **All Projects** to perform your search. 

The project search allows partial and case-insensitive matches.  

## Project states

Project states are reflected on the project page and in the top bar when viewing a project. For example, **Ready to Publish**, **Annotating**, **Done**. 

For more information about project states, see [Project and Task State Management](project_states). 


## Create a project template

A project template includes the following:

* Labeling configuration
* Most project settings

It does not include:

* Project tasks or imported data
* Project membership settings 
* Task assignments
* External storage settings

To create a project template:

1. In the project that you want to use as a template, open the **Settings**.
2. In the **General** tab for the project settings, click **Save as Template**.
3. Add a title and description for the project.
4. Click **Save**.

These templates then appear under **Label Studio Enterprise Templates** section in the Template Gallery in the project creation flow.

## Duplicate projects

When you duplicate a project, you can copy the following:

* Labeling configuration
* Most project settings
* Project membership settings
* Project tasks or uploaded data (optionally)

Duplication does not include:

* Task assignments
* Completed annotations
* External storage settings

To duplicate a project:

1. Click the overflow menu for the project and select **Duplicate project**:

    <img src="/images/project/project_menu_lse.png" style="max-width: 500px" alt="Screenshot">

2. Select the workspace in which you want the new project to be located. 
3. Enter a new name and (optionally) a description for the project.
4. Select whether you only want to duplicate project settings (including the labeling configuration), or if you also want to include tasks. 
5. Click **Duplicate**. 

You may need to refresh the page before you can see the new project. 

## Move projects between workspaces

1. Open the project and click **Settings** in the upper right.
2. Under the **General** tab, use the **Workspace** drop-down menu to select a new workspace.
3. Click **Save**.

## Publish a project 

You can hide projects from annotators so that you can fully configure the project before anyone can start labeling. When you're ready for annotators to start labeling, publish the project.

To publish a project, click the pause icon in the breadcrumb at the top of the page:



You can later unpublish the project by clicking the play icon in the breadcrumb.

## Pin projects

To pin a project, click the overflow menu for the project and select **Pin project**:

<img src="/images/project/project_menu_lse.png" style="max-width: 500px" alt="Screenshot">

Pinned projects are pinned to the top of the Projects page. 


## Delete projects 

To delete a project, open the project settings and select the **Danger Zone** page. From here you can access the **Delete Project** action. 

Deleting a project permanently removes all tasks, annotations, and project data from Label Studio.

---

### Source Guide Manage Projects Lso (`docs/source/guide/manage_projects_lso.md`)

---
title: Manage projects in Label Studio
short: Manage projects
tier: opensource
type: guide
order: 116
order_enterprise: 0
meta_title: Manage projects in Label Studio
section: "Create & Manage Projects"
date: 2024-04-03 22:21:21
---

!!! error Enterprise
    Label Studio Enterprise has additional actions available for project managers, such as organizing projects by workspace, defining project members, and automating workflows. For more information, see [Manage projects in Label Studio Enterprise](https://docs.humansignal.com/guide/manage_projects).

In Label Studio Community Edition, you have the following project management options:

* [Configuring settings for a project](project_settings)
* [Use the Data Manager to navigate labeling tasks](manage_data)

---

### Source Guide Manage Users (`docs/source/guide/manage_users.md`)

---
title: User management overview
short: User management 
tier: enterprise
type: guide
order: 0
order_enterprise: 366
meta_title: User management overview
meta_description: User management options available in Label Studio Enterprise
section: "Manage Your Organization"
---

User management can be a crucial component in your data annotation workflow. 

Data labeling often deals with sensitive or proprietary information, and controlling who has access to this data is vital to protect it from unauthorized access or misuse. By implementing robust user management, you can ensure that only authorized user accounts can access specific datasets. This is particularly important in environments that handle sensitive data, such as medical records or personal information, where data privacy regulations like GDPR or HIPAA might apply.

Furthermore, user management streamlines the workflow and enhances the efficiency of data labeling projects. Because different users can have distinct roles, you can customize your projects to so that each user has the appropriate tools and data access needed to perform their tasks effectively. This not only improves the accuracy and consistency of the data labeling process, but also minimizes the risk of errors or conflicts caused by unauthorized access or modifications. 

Label Studio Enterprise user management options include:

* **Role-Based Access Control (RBAC)**: Roles dictate the level of access that a users has and what actions they are able to complete. For more information about these roles, see [Roles and Permissions](admin_roles). For information about assigning roles, see [Manage user accounts](admin_manage_lse).

* **User workflows through roles**: You can also use roles to design annotation workflows for your users. For example, you can use the project settings to configure separate instructions to display to reviewers and annotators. For more information, see [Project setup](setup_project).

* **User Activity Monitoring**: Administrators can track user activities, such as who labeled what data and when, through an activity log. For more information, see [Activity logs](admin_logs). 

* **Projects and Workspaces**: You can control project visibility by using workspace and project membership. Users with the Manager, Annotator, or Reviewer role must be added as a member in order to view a workspace or project. Workspaces are used to group projects, and membership cascades down to projects within the workspace. For more information, see [Project setup](setup_project).

* **SSO/SAML and SCIM Integration**: For enterprises using Single Sign-On (SSO) or System for Cross-domain Identity Management (SCIM), Label Studio Enterprise allows mapping groups to roles at both the organization and project levels. For more information, see [SSO and SAML setup](auth_setup) and [SCIM2 setup](scim_setup). 

* **Customization and Programmability**: You can programmatically assign roles to users and manage user accounts through API endpoints, allowing for automation and integration with external systems. For more information, see our [API reference](/api/#tag/Users).

---

### Source Guide Ml (`docs/source/guide/ml.md`)

---
title: Integrate Label Studio into your machine learning pipeline
short: Machine learning integration
type: guide
tier: all
order: 251
order_enterprise: 251
meta_title: Integrate Label Studio into your machine learning pipeline
meta_description: Machine learning frameworks for integrating your model development pipeline seamlessly with your data labeling workflow.
section: "Machine Learning"
---

You can use an ML backend to integrate your model development pipeline with your data labeling workflow. There are several use cases, including: 

- **Pre-annotate/autolabel data:** Let ML/AI models predict labels autonomously, which can then be reviewed by human annotators.
- **Interactive labeling:** Integrate ML models into the platform to help humans label or annotate large datasets more efficiently and accurately.
- **Model evaluation and fine-tuning:** Labelers review and analyze the model outputs to assess model accuracy and optimize performance.

<div class="opensource-only">

!!! error enterprise
    Label Studio Enterprise features an out-of-the-box solution for LLM-assisted auto-labeling and model evaluation. Compare LLMs based on accuracy and cost. For more information, see [Prompts](https://docs.humansignal.com/guide/prompts_overview).

</div>

<div class="enterprise-only">

!!! info Tip
    Prompts is an out-of-the-box solution for LLM-assisted auto-labeling and model evaluation. For more information, see [Prompts](prompts_overview).

</div>

For example, for an image classification task, the model pre-selects an image class for data annotators to verify. For audio transcriptions, the model displays a transcription that data annotators can modify. 

Once a model is connected, the way it works is:

1. A user opens the task. 
2. Label Studio sends the request to ML backend. 
3. The ML backend responds with its prediction. 
4. The prediction is loaded into the Label Studio UI and shown to the annotator. 

If you just need to load static pre-annotated data into Label Studio, running an ML backend might be overkill for you. Instead, you can [import preannotated data](predictions).


!!! info Tip
    You can use [Label Studio Enterprise to build an automated active learning loop](https://docs.humansignal.com/guide/active_learning.html) with a machine learning model backend. If you use the open source Community Edition of Label Studio, you can manually sort tasks and retrieve predictions to mimic an active learning process.

## Set up an example ML backend

The Label Studio ML backend is an SDK that wraps your machine learning code and turns it into a web server. The web server can be connected to a running Label Studio instance to automate labeling tasks. We have provided a [library of example models](https://github.com/HumanSignal/label-studio-ml-backend?tab=readme-ov-file#models) that you can use in your own workflow, or extend and customize as needed. 

Ir you want to write your own model instead, see [Write your own ML backend](ml_create).

### Prerequisites

<div class="opensource-only">

* [Label Studio](install)
* [Docker Compose](https://docs.docker.com/compose/install/)

</div>

<div class="enterprise-only">

* [Label Studio](install_enterprise)
* [Docker Compose](https://docs.docker.com/compose/install/)

</div>

### Start the model

1. First, decide which [model](#Example-models) you want to use and check for required parameters (click the link for each model to see a full parameter list). 

    Set your parameters in the `docker-compose.yml` file that is located within the model directory. 

2. Then replace `{MODEL_NAME}` in the below command with the appropriate directory. 

    For example, if you are using the SAM backend with [SegmentAnything model](https://segment-anything.com/), the model name would be `segment_anything_model`, which matches the [directory name in the repo](https://github.com/HumanSignal/label-studio-ml-backend/tree/master/label_studio_ml/examples/segment_anything_model):

    ```bash
git clone https://github.com/HumanSignal/label-studio-ml-backend.git
cd label-studio-ml-backend/label_studio_ml/examples/segment_anything_model
docker-compose up
    ```

The model should begin running at `http://localhost:9090` (if you are using a Docker container, [see the note below](#localhost-and-Docker-containers)). You can verify this by clicking **Send Test Request** from the overflow menu next to the model or by using the following command: 

```bash
> curl http://localhost:9090
{"model_class":"SamMLBackend","status":"UP"}
```

<div class="opensource-only">

If you see any errors, see [Troubleshooting ML backends](troubleshooting#ML-backends) and the [Troubleshooting section in the README](https://github.com/HumanSignal/label-studio-ml-backend/tree/master/README.md#troubleshooting).

</div>

<div class="enterprise-only">

If you see any errors, see [Troubleshooting ML Backends & Predictions](https://support.humansignal.com/hc/en-us/sections/23627938255117-ML-Backend-Predictions) in the HumanSignal support center and see the [Troubleshooting section in the README](https://github.com/HumanSignal/label-studio-ml-backend/tree/master/README.md#troubleshooting).

</div>

#### localhost and Docker containers

`localhost` is a special domain name that loops back directly to your local environment. 

If you are running Label Studio in a Docker container, `localhost` loops back to the container itself, and not the machine that the container is hosted on. Docker provides a special domain as a workaround for this, docker.host.internal. If you're hosting Label Studio and your ML Backend inside of Docker, try using that domain instead of localhost (`http://host.docker.internal:9090`) or the internal IP address. 

### Connect the model to Label Studio

<div class="enterprise-only">

After you [create a project](setup_project), open the project settings and select [**Model**](project_settings_lse#Model). 

</div>

<div class="opensource-only">

After you [create a project](setup_project), open the project settings and select [**Model**](project_settings#Model). 

</div>

Click **Connect Model** and complete the following fields:

| Field | Description                                                                            |
| -------- | -------------------------------------------------------------------------------------- |
| **Name**   | Enter a name for the model.                        |
| **Backend URL**  | Enter a URL for the model. <br /><br />If you are following the steps above, this would be `http://localhost:9090`. If you are running Label Studio in a Docker container, [see the note above](#localhost-and-Docker-containers). |
| **Select authentication method**   | If a username and password are required to access the model, you can select **Basic Authentication** and enter them here.                                     |
| **Extra params**  | Enter any additional parameters you want to pass to the model.                                      |
| **Interactive preannotations**  | Enable this option to allow the model to assist with the labeling process by providing real-time predictions or suggestions as annotators work on tasks.  <br /><br />In other words, as you interact with data (for example, by drawing a region on an image, highlighting text, or asking an LLM a question), the ML backend receives this input and returns predictions based on it. For more information, see [Interactive pre-annotations](#Interactive-pre-annotations) below. |

!!! info Tip
    You can also [add an ML backend using the API](https://api.labelstud.io/api-reference/api-reference/ml/create). You will need the project ID and the machine learning backend URL. 

## Example models

The [ML backend repo](https://github.com/HumanSignal/label-studio-ml-backend) includes a variety of example models that you can experiment with and adapt for your own workflow. 

Some of them work without any additional configuration. Check the **Required parameters** column to see if you need to set any additional parameters. If the model has required parameters, you can set those parameters in `docker-compose.yml` within the model directory. 

- **Pre-annotation** column indicates if the model can be used for pre-annotation in Label Studio:  
  you can see pre-annotated data when opening the labeling page or after running predictions for a batch of data.
- **Interactive mode** column indicates if the model can be used for interactive labeling in Label Studio: see
  interactive predictions when performing actions on labeling page.
- **Training** column indicates if the model can be used for training in Label Studio: update the model state based the
  submitted annotations.

| MODEL_NAME                                                                                 | Description                                                                                                                               | Pre-annotation | Interactive mode | Training | Required parameters                           |
|--------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------|-----------------|------------------|----------|-----------------------------------------------|
| [segment_anything_model](https://github.com/HumanSignal/label-studio-ml-backend/tree/master/label_studio_ml/examples/segment_anything_model)                 | Image segmentation by [Meta](https://segment-anything.com/)                                                                               | ❌               | ✅                |   ❌       | None                                          |
| [llm_interactive](https://github.com/HumanSignal/label-studio-ml-backend/tree/master/label_studio_ml/examples/llm_interactive)                               | Prompt engineering with [OpenAI](https://platform.openai.com/), Azure LLMs.                                                               | ✅               | ✅                | ✅        | OPENAI_API_KEY                                |
| [grounding_dino](https://github.com/HumanSignal/label-studio-ml-backend/tree/master/label_studio_ml/examples/grounding_dino)                                 | Object detection with prompts. [Details](https://github.com/IDEA-Research/GroundingDINO)                                                  | ❌               | ✅                | ❌        | None                                          |
| [tesseract](https://github.com/HumanSignal/label-studio-ml-backend/tree/master/label_studio_ml/examples/tesseract)                                           | Interactive OCR. [Details](https://github.com/tesseract-ocr/tesseract)                                                                    | ❌               | ✅                | ❌        | None                                          |
| [easyocr](https://github.com/HumanSignal/label-studio-ml-backend/tree/master/label_studio_ml/examples/easyocr)                                               | Automated OCR. [EasyOCR](https://github.com/JaidedAI/EasyOCR)                                                                             | ✅               | ❌                | ❌        | None                                          |
| [spacy](https://github.com/HumanSignal/label-studio-ml-backend/tree/master/label_studio_ml/examples/spacy)                                                   | NER by [SpaCy](https://spacy.io/)                                                                                                         | ✅               | ❌                | ❌        | None                                          |
| [flair](https://github.com/HumanSignal/label-studio-ml-backend/tree/master/label_studio_ml/examples/flair)                                                   | NER by [flair](https://flairnlp.github.io/)                                                                                               | ✅               | ❌                | ❌        | None                                          |
| [bert_classifier](https://github.com/HumanSignal/label-studio-ml-backend/tree/master/label_studio_ml/examples/bert_classifier)                               | Text classification with [Huggingface](https://huggingface.co/transformers/v3.0.2/model_doc/auto.html#automodelforsequenceclassification) | ✅               | ❌                | ✅        | None                                          |
| [huggingface_llm](https://github.com/HumanSignal/label-studio-ml-backend/tree/master/label_studio_ml/examples/huggingface_llm)                               | LLM inference with [Hugging Face](https://huggingface.co/tasks/text-generation)                                                           | ✅               | ❌                | ❌        | None                                          |
| [huggingface_ner](https://github.com/HumanSignal/label-studio-ml-backend/tree/master/label_studio_ml/examples/huggingface_ner)                               | NER by [Hugging Face](https://huggingface.co/docs/transformers/en/tasks/token_classification)                                             | ✅               | ❌                | ✅        | None                                          |
| [nemo_asr](https://github.com/HumanSignal/label-studio-ml-backend/tree/master/label_studio_ml/examples/nemo_asr)                                             | Speech ASR by [NVIDIA NeMo](https://github.com/NVIDIA/NeMo)                                                                               | ✅               | ❌                | ❌        | None                                          |
| [mmdetection](https://github.com/HumanSignal/label-studio-ml-backend/tree/master/label_studio_ml/examples/mmdetection-3)                                     | Object Detection with [OpenMMLab](https://github.com/open-mmlab/mmdetection)                                                              | ✅               | ❌                | ❌        | None                                          |
| [sklearn_text_classifier](https://github.com/HumanSignal/label-studio-ml-backend/tree/master/label_studio_ml/examples/sklearn_text_classifier)               | Text classification with [scikit-learn](https://scikit-learn.org/stable/)                                                                 | ✅               | ❌                | ✅        | None                                          |
| [interactive_substring_matching](https://github.com/HumanSignal/label-studio-ml-backend/tree/master/label_studio_ml/examples/interactive_substring_matching) | Simple keywords search                                                                                                                    | ❌               | ✅                | ❌        | None                                          |
| [langchain_search_agent](https://github.com/HumanSignal/label-studio-ml-backend/tree/master/label_studio_ml/examples/langchain_search_agent)                 | RAG pipeline with Google Search and [Langchain](https://langchain.com/)                                                                   | ✅               | ✅                | ✅        | OPENAI_API_KEY, GOOGLE_CSE_ID, GOOGLE_API_KEY |

## Allow the ML backend to access Label Studio data

In most cases, you will need to set your environment variables to allow the ML backend access to the data in Label Studio. 

Label Studio tasks can have multiple sources of resource files:

* Direct `http` and `https` links.  
    Example: `task['data'] = {"image": "http://example.com/photo_1.jpg"}`

* Files that have been uploaded to Label Studio using the **Import** action.  
    Example: `task['data'] = {"image": "https://ls-instance/data/upload/42/photo_1.jpg"}`

* Files added through a [local storage connection](storage_local).  
    Example: `task['data'] = {"image": "https://ls-instance/data/local-files/?d=folder/photo_1.jpg"}`

* Files added through a [cloud storage](storage) (S3, GCS, Azure) connection.
    Example: `task['data'] = {"image": "s3://bucket/prefix/photo_1.jpg"}`

When Label Studio invokes the `predict(tasks)` method on an ML backend, it sends tasks containing data sub-dictionaries with links to resource files. 

Downloading files from direct `http` and `https` links (the first example above) is straightforward. However, the other three types (imported files, local storage files, and cloud storage files) are more complex. 

To address this, the ML backend utilizes the `get_local_path(url, task_id)` function from the `label_studio_tools` package (this package is pre-installed with `label-studio-ml-backend`):

```python
from label_studio_tools.core.utils.io import get_local_path

class MLBackend(LabelStudioMLBase)
  def predict(tasks):
    task = tasks[0]
    local_path = get_local_path(task['data']['image'], task_id=task['id'])
    with open(local_path, 'r') as f:
      f.read()
```

The `get_local_path()` function resolves URIs to URLs, then downloads and caches the file.

For this to work, you must specify the `LABEL_STUDIO_URL` and `LABEL_STUDIO_API_KEY` environment variables for your ML backend before using `get_local_path`. If you are using docker-compose.yml, these variables have to be added to the environment section. For example:

```yaml
services:
  ml-backend-1:
    container_name: ml-backend-1
    ...
    environment:
      # Specify the Label Studio URL and API key to access
      # uploaded, local storage and cloud storage files.
      # Do not use 'localhost' as it does not work within Docker containers.
      # Use prefix 'http://' or 'https://' for the URL always.
      # Determine the actual IP using 'ifconfig' (Linux/Mac) or 'ipconfig' (Windows).
      - LABEL_STUDIO_URL=http://192.168.42.42:8080/  # replace with your IP! 
      - LABEL_STUDIO_API_KEY=<your-label-studio-api-key>
```

Note the following:

* `LABEL_STUDIO_URL` must be accessible from the ML backend instance. 

* If you are running the ML backend in Docker, `LABEL_STUDIO_URL` can’t contain `localhost` or `0.0.0.0`. Use the full IP address instead, e.g. `192.168.42.42`. You can get this using the `ifconfig` (Unix) or `ipconfig` (Windows) commands.

* `LABEL_STUDIO_URL` must start either with `http://` or `https://`.

To find your `LABEL_STUDIO_API_KEY`, open Label Studio and go to your [user account page](user_account#Access-token). 

<div class="enterprise-only">

Note that your user must also have access to the project that you are connecting to ML backend.

</div>

## Model training

Training a model allows it to learn from submitted annotations and potentially improve its predictions for subsequent tasks. 

After you connect a model to Label Studio as a machine learning backend and annotate at least one task, you can start training the model. You can use automated or manual training. 

<div class="enterprise-only">

From the [**Model** page](project_settings_lse#Model) under project settings, select **Start Training** from the overflow menu next to the connected model. This manually initiates training. Use this action if you want to control when the model training occurs, such as after a specific number of annotations have been collected or at certain intervals.

</div>

<div class="opensource-only">

From the [**Model** page](project_settings#Model) under project settings, select **Start Training** from the overflow menu next to the connected model. This manually initiates training. Use this action if you want to control when the model training occurs, such as after a specific number of annotations have been collected or at certain intervals.

</div>

You can also initiate training programmatically using the following:

* From the API, specify the ID of the machine learning backend and run the following command: 
   ```
   curl -X POST http://localhost:8080/api/ml/{id}/train
   ```
  See [the Train API documentation](/api/#operation/api_ml_train_create) for more.
- [Trigger training with webhooks](ml_create#Trigger-training-with-webhooks).

Training logs appear in stdout and the console. 

To see more detailed logs, start the ML backend server with the `--debug` option. 

## Pre-annotations/predictions

!!! note
    The terms "predictions" and pre-annotations" are used interchangeably. 


### Get predictions from a model

After you connect a model to Label Studio, you can see model predictions in the labeling interface if the model is pre-trained, or right after it finishes [training](#Model-training). 

* To manually add predictions, go to the Data Manager, select the tasks you want to get predictions for, and then select **Actions > Retrieve predictions**.
* To automatically pre-label data with predictions, go to the project settings and enable **Annotation > Use predictions to prelabel tasks** and ensure that you have selected the appropriate model from the **Select which predictions or which model you want to use** drop-down menu. 


!!! note
    For a large dataset, the HTTP request to retrieve predictions might be interrupted by a timeout. If you want to **get all predictions** for all tasks in a dataset from connected machine learning backends, make a [POST call to the predictions endpoint of the Label Studio API](/api/#operation/api_predictions_create) for each task to prompt the machine learning backend to create predictions for the tasks. 

If you want to retrieve predictions manually for a list of tasks **using only an ML backend**, make a POST request to the `/predict` URL of your ML backend with a payload of the tasks that you want to see predictions for, formatted like the following example: 

```json
{
  "tasks": [
    {"data": {"text":"some text"}}
  ]
}
```

### Interactive pre-annotations

ML-assisted labeling with interactive pre-annotations works with image segmentation and object detection tasks using rectangles, ellipses, polygons, brush masks, and keypoints, as well as with HTML and text named entity recognition tasks. Your ML backend must support the type of labeling that you're performing, recognize the input that you create, and be able to respond with the relevant output for a prediction.

Either enable the **Interactive preannotations** option when adding a model, or use the **Edit** action from the project settings page to enable/disable this option. 

#### Smart tools

Smart tools duplicate the tools that you have set up to use through your labeling configuration (e.g. rectangles) to allow interaction with the ML backend. Smart tools are currently available for [`Rectangle`](/tags/rectangle), [`Ellipse`](/tags/ellipse), [`Polygon`](/tags/polygon), [`KeyPoint`](/tags/keypoint), and [`Brush`](/tags/brush) tags. 

Smart tools are dynamic and can change behavior or appearance based on the context, leveraging machine learning predictions. For example, smart tools can automatically detect and suggest annotations for objects in images, which annotators can then review and refine. This can significantly speed up the labeling process, especially when dealing with large datasets.

Smart tools use `context` to adapt their behavior based on the current state of the labeling environment and the specific needs of the task at hand. For instance, they might alter their functionality depending on the type of data being labeled or the particular region of interest within a task. When smart tools are enabled for a labeling task, they can interact with the ML backend by sending data to the `/predict` endpoint to receive predictions. These predictions are then used to provide interactive pre-annotations within the labeling interface. For more information, see [Support interactive pre-annotations in your ML backend](ml_create#Support-interactive-pre-annotations-in-your-ML-backend). 

For example, when an annotator starts labeling an image, the smart tool can send the image to the ML backend, which processes it and returns suggested annotations (like bounding boxes or segmentation masks) that the annotator can accept, reject, or refine. This process helps streamline the labeling workflow by providing a starting point for annotations based on the model's current understanding of the data.

To use smart tools:

* Smart tools appear by default if **Auto-annotation** is enabled in the labeling interface. 
* You can also update your labeling configuration to include the `smart="true"` option for the type of labeling you're performing. 
* If you *only* want the smart option to appear and don't want to perform manual labeling at all, use `smartOnly="true"`. 

For example: 

```<Brush name="brush" toName="img" smart="true" showInline="true"/>```

After you start labeling, enable **Auto-Annotation** to see and use the smart option to draw a shape, mask, or assign a keypoint. 
   
For image labeling, after you enable auto-annotation you can choose whether to **Auto accept annotation suggestions**. If you automatically accept annotation suggestions, regions show up automatically and are immediately created. If you don't automatically accept suggestions, the regions appear but you can reject or approve them manually, either individually or all at once.

<br/><img src="/images/predict-owl-region.gif" alt="" class="gif-border" width="800px" height="533px" />
   
### Delete predictions 

If you want to delete all predictions from Label Studio, you can do it using the Data Manager or the API:
- For a specific project, select the tasks that you want to delete predictions for and select **Delete predictions** from the drop-down menu.
- Using the API, run the following from the command line to delete the predictions for a specific project ID:
```
curl -H 'Authorization: Token <user-token-from-account-page>' -X POST \ 
 "<host>/api/dm/actions?id=delete_tasks_predictions&project=<id>"
```

### Choose which predictions to display to annotators

<div class="opensource-only">

You can choose which model or prediction set to display to annotators by default. This is available under the **Annotation > Live Predictions** in the project settings. 

</div>

<div class="enterprise-only">

You can choose which model or prediction set to display to annotators by default. This is available under the **Annotation > Live Predictions** in the project settings. 

</div>

Use the drop-down menu to select which model or predictions to use in your labeling workflow.

---

### Source Guide Ml Create (`docs/source/guide/ml_create.md`)

---
title: Write your own ML backend
short: Write your own ML backend
type: guide
tier: all
order: 254
order_enterprise: 254
meta_title: Write your own ML backend
meta_description: Set up your machine learning model to output and consume predictions in your data science and data labeling projects. 
section: "Machine Learning"

---

Use the Label Studio ML backend to integrate Label Studio with machine learning models. The Label Studio ML backend is an SDK that you can use to wrap your machine learning model code and turn it into a web server. The machine learning server uses [uWSGI](https://uwsgi-docs.readthedocs.io/en/latest/) and [supervisord](http://supervisord.org/), and handles background training jobs with [RQ](https://python-rq.org/).

Follow the steps below to wrap custom machine learning model code with the Label Studio ML SDK, or see [our library of example ML backends](ml_tutorials.html) to integrate with popular machine learning frameworks and tools such as [Huggingface's Transformers](https://huggingface.co/docs/transformers/index), [OpenAI](https://openai.com/), [Langchain](https://www.langchain.com/) and others. 

For information on using one of Label Studio's example backends, see [Set up an example ML backend](ml#Set-up-an-example-ML-backend). 

<div class="opensource-only">

!!! error enterprise
    Label Studio Enterprise features an out-of-the-box solution for LLM-assisted labeling. All you need to get set up is to connect one of our supported models. For more information, see [Prompts](https://docs.humansignal.com/guide/prompts_overview).

</div>

<div class="enterprise-only">

!!! info Tip
    Prompts is an out-of-the-box solution for LLM-assisted labeling. All you need to get set up is to connect one of our supported models. For more information, see [Prompts](prompts_overview).

</div>

For a video tutorial, see the following:

<iframe width="560" height="315" src="https://www.youtube.com/embed/cEwMNOlOUjI?si=sEPTUPOyyzZlA-n5" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe>


## 1. Install the ML backend repo

Download and install `label-studio-ml-backend` from the repository:

```bash
git clone https://github.com/HumanSignal/label-studio-ml-backend.git
cd label-studio-ml-backend/
pip install -e .
```

## 2. Create an empty ML backend

```bash
label-studio-ml create my_ml_backend
```

This creates the following directory structure, which you can modify to implement your own inference logic: 

```
my_ml_backend/
├── Dockerfile
├── .dockerignore
├── docker-compose.yml
├── model.py
├── _wsgi.py
├── README.md
├── requirements-base.txt
├── requirements-test.txt
├── requirements.txt
└── test_api.py
```

Where:

* `Dockerfile`, `docker-compose.yml` and `.dockerignore` are used to run the ML backend with Docker. 
* `model.py` is the main file where you can implement your own training and inference logic. 
* `_wsgi.py` is a helper file that is used to run the ML backend with Docker (you don't need to modify this). 
* `README.md` must contain instructions on how to run the ML backend. 
* `requirements.txt` is where you put your Python dependencies.
* `requirements_base.txt` and `requirements_test.txt` are basic dependencies (you don't need to modify this)
* `test_api.py` is where you put your model tests


## 3. Implement prediction logic

In your model directory, locate the `model.py` file (for example, `my_ml_backend/model.py`).

The `model.py` file contains a class declaration inherited from `LabelStudioMLBase`. This class provides wrappers for the API methods that are used by Label Studio to communicate with the ML backend. You can override the methods to implement your own logic:

```python
def predict(self, tasks, context, **kwargs):
    """Make predictions for the tasks."""
    return predictions
```

The `predict` method is used to make predictions for the tasks. It uses the following:

- `tasks`: [Label Studio tasks in JSON format](task_format)
- `context`: [Label Studio context in JSON format](#Support-interactive-pre-annotations-in-your-ML-backend) - for an interactive labeling scenario
- `predictions`: [Predictions array in JSON format](export#Raw-JSON-format-of-completed-tasks)

Once you implement the `predict` method, you can see predictions from the connected ML backend in Label Studio.

### Support interactive pre-annotations in your ML backend

If you want to support interactive pre-annotations in your machine learning backend, write an inference call using the `predict()` method. For an example that does this for text labeling projects, see [this code example for substring matching](https://github.com/HumanSignal/label-studio-ml-backend/tree/master/label_studio_ml/examples/interactive_substring_matching).

Complete the following steps:

1. Define an inference call with the `predict()` method as outlined above. The `predict()` method takes task data and context data:
  - The `tasks` parameter contains details about the task being pre-annotated. See [Label Studio tasks in JSON format](task_format).
  - The `context` parameter contains details about annotation actions performed in Label Studio, such as a text string highlighted sent in [Label Studio annotation results format](export#Label-Studio-JSON-format-of-annotated-tasks).
  
    `context` has the following properties. 
    - `annotation_id`: The annotation ID.
    - `draft_id`: The draft annotation ID.
    - `user_id`: The user ID.
    - `result`: This is the annotation result, but it includes an `is_positive: true` flag that can be changed by the user. For example, by pressing the **Alt** key and using keypoints to interact with the image in the UI. 
  
2. With the task and context data, construct a prediction using the data received from Label Studio. 
3. Return a result in the [Label Studio predictions format](predictions.html#Format-pre-annotations-for-Label-Studio), which varies depending on the type of labeling being performed.

Refer to the code example linked above for more details about how this might be performed for a NER labeling project. 

For more information about enabling pre-annotations, see [Interactive pre-annotations](ml#Interactive-pre-annotations).


## 4. Implement training logic (optional)

You can also implement the `fit` method to train your model. The `fit` method is typically used to train the model on the labeled data, although it can be used for any arbitrary operations that require data persistence (for example, storing labeled data in database, saving model weights, keeping LLM prompts history, etc).

By default, the `fit` method is called at any data action in Label Studio, like creating a new task or updating annotations. You can modify this behavior in using [Webhooks](webhooks).

To implement the `fit` method, you need to override the `fit` method in your `model.py` file:

```python
def fit(self, event, data, **kwargs):
    """Train the model on the labeled data."""
    old_model = self.get('old_model')
    # write your logic to update the model
    self.set('new_model', new_model)
```

with:

- `event`: The event type. This can be `'ANNOTATION_CREATED'`, `'ANNOTATION_UPDATED'`, etc.
- `data`: The payload received from the event (see the [Webhook event reference](webhook_reference)). 

Additionally, there are two helper methods that you can use to store and retrieve data from the ML backend:

- `self.set(key, value)` - Store data in the ML backend
- `self.get(key)` - Retrieve data from the ML backend

Both methods can be used elsewhere in the ML backend code, for example, in the `predict` method to get the new model weights.

### Trigger training with webhooks

Starting in version 1.4.1 of Label Studio, when you add an ML backend to your project, Label Studio creates a webhook to your ML backend to send an event every time an annotation is created or updated.

By default, the payload of the webhook event does not contain the annotation itself. You can either [modify the webhook event](webhooks) sent by Label Studio to send the full payload, or retrieve the annotation using the Label Studio API using the [get annotation by its ID endpoint](/api#operation/api_annotations_read), [SDK](sdk.html) using the [get task by ID method](https://labelstud.io/sdk/project.html#label_studio_sdk.project.Project.get_task), or by retrieving it from [target storage that you set up](storage) to store annotations.

See the [annotation webhook event reference](webhook_reference#Annotation-Created) for more details about the webhook event.


### Other methods and parameters

Other methods and parameters are available within the `LabelStudioMLBase` class:

- `self.label_interface` - Returns the Label Studio Label Interface object that contains all information about the labeling task.
- `self.model_version` - Returns the current model version.

## 4. Ensure the ML backend can access Label Studio data

If your data is stored in a cloud, local directory, or has been imported into Label Studio, you will need to set the `LABEL_STUDIO_URL` and `LABEL_STUDIO_API_KEY` environment variables. 

For more information, see [Allow the ML backend to access Label Studio data](ml#Allow-the-ML-backend-to-access-Label-Studio-data). 

## 5. Run the ML backend server

To run with Docker Compose: 

   ```bash
   docker-compose up
```

The ML backend server is available at `http://localhost:9090`. You can use this URL when [connecting the ML backend to Label Studio](ml#Connect-the-model-to-Label-Studio).

!!! note
    `localhost` is a special domain name that loops back directly to your local environment. In the instance of Docker-hosted containers, this loops back to the container itself, and not the machine the container is hosted on. Docker provides a special domain as a workaround for this, docker.host.internal. If you're hosting Label Studio and your ML Backend inside of Docker, try using that domain instead of localhost. (`http://host.docker.internal:9090`)


### Run without Docker

To run without docker (for example, for debugging purposes), you can use the following command:

```bash
pip install -r my_ml_backend
label-studio-ml start my_ml_backend
```

### Modify the host and port

To modify the host and port, use the following command line parameters:

```bash
label-studio-ml start my_ml_backend -p 9091 --host 0.0.0.0
```

### Test your ML backend

Modify the `my_ml_backend/test_api.py` to ensure that your ML backend works as expected.

## 6. Connect the ML backend to Label Studio

You can use the API or **Settings > Model**. For more information, see [Connect the model to Label Studio](ml#Connect-the-model-to-Label-Studio).

---

### Source Guide Model Providers (`docs/source/guide/model_providers.md`)

---
title: Model provider API keys for organizations
short: Model providers
tier: enterprise
type: guide
hide_menu: false
order: 0
order_enterprise: 361
meta_title: Model providers
meta_description: How to set up model providers for AI features in Label Studio. 
section: "Manage Your Organization"
parent_enterprise: "admin_settings"
date: 2025-02-18 12:03:59
---

To use certain AI features across your organization, you must first set up a model provider. You can set up model providers from **Organization > Settings**.

For example, if you want to interact with an LLM when using the [`<Chat>` tag](/tags/chat.html), you will first need to configure access to the model.


!!! note
    These models are not used with the Label Studio [AI Assistant](ask_ai). 

## About adding model providers

### User access 

Once added, these model providers can be used by anyone in your organization who is also using an AI-powered workflow. 
    
However, only users with access to the organization settings (Owner and Admins) can add and configure models. 

### Whitelisting network access

If you are restricting network access to your resource, you may need to whitelist HumanSignal IP addresses ([IP ranges on SaaS](saas#Outbound-Connections-IP-Addresses)) when configuring network security.

### Approaches

There are two approaches to adding a model provider API key. 

* In one scenario, you get one provider connection per organization, and this provides access to a set of whitelisted models. Examples include:

    * OpenAI
    * Vertex AI
    * Gemini
    * Anthropic

* In the second scenario, you add a separate API key per model. Examples include:

    * Azure OpenAI
    * Azure AI Foundry
    * Custom

{% insertmd includes/base_models.md %}

## Add a model provider

!!! note
    If you have already configured model providers to use with [Prompts](prompts_overview), those will automatically be added to your organization-level providers. 

{% insertmd includes/model_keys.md %}

---

### Source Guide Onboarding (`docs/source/guide/onboarding.md`)

---
title: Onboarding guides
short: Onboarding guides by role
type: guide
tier: enterprise
order: 0
order_enterprise: 500
section: "Onboarding"
---

These onboarding guides will help you get started with your role on the platform.

Label Studio Enterprise supports a collaborative workflow where team members work together to create high-quality training data. The platform is designed around two primary roles:

## For Annotators

If you're new to annotating tasks in Label Studio, start with the **[Annotator onboarding guide](onboarding_annotator)**. This guide covers:

* How to access and navigate Label Studio
* Finding and working on your assigned projects
* Step-by-step annotation workflows
* Reviewing your performance analytics
* Best practices for efficient and accurate annotation

Annotators are responsible for creating high-quality labels and annotations that train machine learning models. Your work is essential to building accurate, reliable AI systems.

## For Reviewers

If you're reviewing annotation tasks, check out the **[Reviewer onboarding guide](onboarding_reviewer)**. This guide includes:

* How to access the review interface
* Reviewing and evaluating annotation tasks
* Providing feedback to annotators
* Understanding review metrics and analytics
* Best practices for consistent, thorough reviews

Reviewers ensure quality and consistency across all annotations, serving as the quality assurance layer before tasks move to production.

---

### Source Guide Onboarding Annotator (`docs/source/guide/onboarding_annotator.md`)

---
title: How to Annotate in Label Studio
short: Annotator onboarding
type: guide
tier: enterprise
order: 503
order_enterprise: 503
parent: "onboarding"
parent_enterprise: "onboarding"
section: "Onboarding"
---

Welcome to the Label Studio platform!  

In this document, we will jump-start your understanding of data annotation.  Every project is used to train models based on its particular use case.  

There are a variety of use cases that can be implemented via Label Studio, including, but not limited to, pre-training, post-training, benchmarking, etc.

In essence, every task you annotate is a way to better your team’s model.  Whether you’re a doctorate-holding specialist or a research-savvy generalist, your knowledge and capabilities are needed to train models.

Usually, after you submit your task, Label Studio Enterprise sends off all submitted annotated tasks to reviewers, who verify if there are any errors.  Exact steps vary by team, but most workflows follow a pattern like: annotation → review and quality assurance → production.

During the quality assurance step, tasks are evaluated and graded. Tasks are graded based on how closely they align with the project’s protocols.

### What You Need

* **Computer**  
  While some teams may be flexible in the types of device that can be used, it is highly recommended that you use a laptop or desktop, as mobile devices often do not have enough processing power when a project involves larger files, nor ideal visibility of a lengthier task.

  Confirm with your team lead whether there are any restrictions on which device you can use. In some cases, your current company hardware may not be approved or preferred for labeling.

* **Project Protocols**  
  During your onboarding, your team lead should provide you with documentation regarding project workflows and steps.  Depending on the length and scope of your first days, a lead would often walk you through one or two tasks so that you would have some guidance.  Alternatively, you may be presented with video recordings for you to go through the protocols yourself.

* **Proper Account Permissions**  
  When your Label Studio account is created, your team lead needs to add you to the appropriate projects for you to work on.  Sometimes, you may be added onto more than one, so it is important that you ensure you have access to and are working on your highest priority project.

  The following view is how your home dashboard *should* look if you have a project assigned to you. This means your permissions are set and you’re ready to get to work\!

  

The following view would occur if you are missing the properly activated projects. If you have this view, please contact your lead.



## Walkthrough:

### How do I access Label Studio?

When you initially receive access to Label Studio, you will receive an email and a link to sign up. 

To access your company’s projects, use the information you provided at sign up to log in. (Location link may vary depending on whether your team works directly on Label Studio or in your company’s own integrated workspace.)  



### How Do I Find My Projects?

Your recent projects will be on your home page when you sign in.  You can choose to pin your priority projects onto this section.



Alternatively, click the hamburger menu at the top of your dashboard to open the side menu and view your Projects folder. If you click the folder and don’t see the correct project listed, your permissions may not be updated yet. Contact your team lead to resolve this.



### Are There Set Expectations for my Project?

Exact standards would be discussed during onboarding.  During that time, your lead normally conveys:

* Desired task output over a given time  
* Target scoring averages  
* Projected amount of time each task would take  
* Etc.

### How Do You Annotate?

1. Select the project you want to work on.

    

2. When you click “label,” you are brought to your first task.   
   
   

3. Annotate the task.  If, for whatever reason, you are unable to complete the task (e.g. not a subject matter expert, etc.,), click the “Skip” button.  If there is no “Skip” available for you, please consult your team lead on what would be best practices here.

4. Follow the protocol given to you by your lead. In this example, the annotator is selecting the car in the image. It’s possible that your project has been configured to allow a pop up of the annotation instructions, right in Label Studio. If this is true, you can see it by clicking on the circle with an i inside of it icon on the bottom of the annotation screen, to the left of the Submit button. 

5. Add or edit information in the side bar under the “Info”tab. You can also leave comments for your Reviewer or team leads in the “Comments” tab..  
   
   

6. After completing the flow of your task, click the “Submit” button to send your task in and progress to your next task. 

   There is also a right-hand dropdown on the button that allows you to submit the task and then exit out of the project.  However, to avoid accidental unfinished drafts, please reserve this option for only when you have completed the task and are exiting the project.  
   
   

### How can I Review my Annotation Stats?

1. Click on the hamburger menu button at the top of the LSE platform.

2. Click Analytics.  
   
   

3. Your overall stats would be included in this dashboard.  You can see your total submitted annotations, time spent annotating, and overall performance score.  The graphs also provide a good visual of your production flow over time.  If you want to adjust your date range, you can do so on the upper right-hand corner of the analytics dashboard.

   

4. If you choose, you can also export your analytics for your own records.

   

## Annotator Best Practices

#### Keep your project protocols on the side for easy reference

Keep your project protocols nearby for easy reference.  

Whether you print them out or keep them open on a second screen, having a "cheat sheet" of project steps and key notes is useful for both newcomers and experienced annotators.  

For those starting out, this makes it easier to follow the correct steps—especially when the workflow is nuanced or easy to misinterpret.  

For more experienced annotators, it reduces the need to flip back and forth when a detail is forgotten and helps preserve mental bandwidth during challenging tasks or longer shifts.

#### Research every minute detail, even ones that "sound right"

Especially for tasks with lengthy or nuanced responses, models may hallucinate.  Pre-annotations, or model derived answers, that seem accurate to a layperson may contain even small errors. As datasets become more complex, annotators should likewise be more meticulous. Inaccuracies may be amplified if a model trains on it, causing major setbacks to the models as a whole.

#### Proactively check analytics

Project standards should be announced during your onboarding. In the case of pivots, expectations also need to be adjusted.  During these instances, your team lead or project's point of contact would normally convey how many tasks should be done during an allotted period of time, the average time to be spent on each task, target vs average grading per task, and more.

Keep an eye out for these stats as they would allow you to self-monitor success rate and gauge where you may stand, in terms of project ideals. Additionally, recognizing where you may be falling short would inform you on where to improve.

#### Pace yourself

Empowered by your stats, make sure you keep cadence to meet the "ideal" analytics.  Treat your time and energy as a non-renewable resource. Mental exhaustion and the urge to rush to complete a task both can contribute to an increase in mistakes.

#### When possible, tackle the hardest/longest tasks at the beginning of the day

Some projects allow you to review your assigned tasks prior to working on them. If so, consider focusing your energy on the ones that take the most time and energy when you begin your work. This would allow you to more easily focus on complex tasks when you have less mental fatigue, resulting in potentially less careless errors. Of course, prior to doing this, consult your team lead to ensure this is allowable within the project protocol.

#### Track when your productivity dips, and how to get back on track

Depending on the project, how you work may fluctuate.  It may not necessarily always be the morning in which you are churning out the most annotations; it can be right after your coffee break or right before lunch. When that happens, experiment with different methods on how to get yourself back into the groove.

For some, saving "easier" tasks for these dips can be helpful, while pausing in the middle of a task for a stretch break enables others to jump right back in, quickly referring to their detailed notes to get back on track for the paused work.

#### Know when to flag something to a lead

If you hit a glitch, you don't always need to message your lead immediately. First, capture a few details so they can troubleshoot quickly: what you were doing, what happened, the task ID, and a screenshot (if possible). Issues are much easier to fix when they're clearly documented and reproducible.

The same applies to feedback and scoring. If something looks confusing or incorrect, note whether it seems like a one-time hiccup or a pattern across multiple tasks. Either way, bring it up, especially if it affects your scores or performance metrics. Recurring issues should be escalated sooner, since they may impact the whole project.

#### Document strategies that improve your efficiency

As you become comfortable in your project's protocols, note what types of methods work best for you.  There may be some tips that can be found in the documentation but they may not always be what is most efficient for you or for every task.  

Log which research techniques or websites have been dependable for tasks you fact-check.  This can serve as a "cheat sheet" you can click through, saving time from having to look for your resource each time you need it.

#### Build relationships with those who are more experienced or score higher than you

Whenever possible, connect with teammates who have more experience or consistently strong results. Many are happy to share practical tips that don't always make it into project documentation.

Experienced annotators have had time to refine their workflow and improve both speed and quality. Learning their approaches can help you build your own toolkit faster.

It also helps to have trusted peers you can sanity-check with, especially if those teammates also serve as reviewers.

---

### Source Guide Onboarding Reviewer (`docs/source/guide/onboarding_reviewer.md`)

---
title: How to Review Tasks in Label Studio
short: Reviewer onboarding
type: guide
tier: enterprise
order: 504
order_enterprise: 504
parent: "onboarding"
parent_enterprise: "onboarding"
section: "Onboarding"
---

Welcome to the reviewer tier! 

In this document, we’ll cover the essentials of reviewing annotation tasks. High-quality training data depends on high-quality review, so the goal is to approve work that is accurate, consistent, and aligned with the project’s guidelines.

Reviewers are the proofreaders and editors of human data, ensuring that what goes out is presentable and drives good future data back to the team. Reviewers tend to be held at a higher standard because your knowledge of the strategies and the protocols are essential in keeping the team streamlined.  

While an annotator is only responsible for the work they generate, reviewers are in charge of every task that comes into their queue, regardless of the original owner.

When there is only one review layer, after you review a task, it immediately goes to production.  This would mean a reviewer mistake can snowball into larger issues than an annotator mistake would.

In some projects, there are multiple layers of review. After you complete your review, a secondary review may serve as a final audit or provide the deciding vote when the annotator and first reviewer disagree on how a task should be labeled.

### What You Need

Most items that you used as an annotator can also be used for your reviewer work. However, there will be some additional materials and accesses you would need to get your work done.

* **Computer:** Please make sure to confirm with your team lead that your existing work computer is suitable for your target projects.

* **Reviewer Project Protocols**:When you move to the reviewer tier, your team lead or a veteran reviewer would likely provide you access to the separate workflows and project steps.  Usually, this would be accompanied by a live class or a recording to review independently.

* **Updated Account Permissions**:When you become a reviewer, your team lead needs to update your account, as permissions allow you to see different features on the same project as annotators.

## Frequently Asked Questions:

### How do I access Label Studio?

When you initially receive access to Label Studio, you will receive an email and a link to sign up. 

To access your company’s projects, use the information you provided at sign up to log in.  
(Location link may vary depending on whether your team works directly on Label Studio or in your company’s own integrated workspace.)



### How Do I Find My Projects?

Your recent projects will be on your home page when you sign in. You can choose to pin your priority projects onto this section.



Alternatively, click the hamburger menu at the top of your dashboard to open the side menu and view your Projects folder. If you click the folder and don’t see the correct project listed, your permissions may not be updated yet. Contact your team lead to resolve this.  



### How Do You Review? 

1. Select the project you want to work on.  Instead of a “Label” button like you would have for the annotator flow, you would have a “Review” button.  

   

2. Clicking the “Review” button would bring you immediately into a task to review.  

   

3. Alternatively, if you click directly onto the project name, you’d be able to look over the task queue in list format in the Data Manager.  This view also allows you to see who annotated a task, how many annotations are on any particular queued task, and if it has been approved.

   

4. Similarly to the annotation flow, be sure to follow the project protocols, provided by your team lead, on how to proceed with your reviewing.

   A reviewer flow may include:

   - fixing an annotation directly on the task. For example, the image below shows the reviewer adjusting the section on the image.

        

   - leaving a comment for the annotator to better understand any errors they may have made or the thought process of the reviewer. When leaving a comment, you can choose to optionally link your comment to a specific piece of the annotation by clicking on the box with a chat bubble in the bottom right corner, found here next to the “Add a comment” box. This is a great way to provide pointed feedback to annotators.   
   
        

5. Similarly to the annotation flow, you can select “Reject” or “Accept” at the bottom of your page.

   

6. After a task has been reviewed, the status of the task will be updated on the queue. If you, the reviewer, did not accept the annotations, it can be cycled back to the annotator to be fixed.

### How can I Review my Reviewer Stats?

1. Click on the hamburger menu button at the top of the LSE platform.

2. Click Analytics.  
   
   

3. Select “Reviews” at the upper left-hand corner of the dashboard.

   Your overall stats would be included in this dashboard. You can see your total submitted reviews, time spent reviewing, and ratio between “accepted”, “fixed+accepted”, and “rejected” tasks.  

   The graphs also provide a good visual of your production flow over time. If you want to adjust your date range, you can do so on the upper right-hand corner of the analytics dashboard.

   

4. If you choose, you can also export your analytics for your own records.

   


### What Should I Do When I Encounter Issues?

If you are encountering tech issues, be sure to log when you see them.  Depending on if it is a once-off or a recurring issue, gather information, such as which tasks (if task-specific) are being affected, and submit it to your team lead.  

Take screenshots and see if you can replicate the same issue on multiple tasks. Additionally, if possible, determine if it is an issue just for annotators or reviewers, or if it may be a data issue.

If you notice a specific annotator is having repetitive issues, you should likewise log any past instances, so that your lead can handle the situation with all relevant information in hand.

## Reviewer Best Practices

#### Approach every task as you would as an annotator

Though reviewing involves a different flow than an annotator, you should still evaluate the task as you would if you were annotating it from scratch. This means evaluating every statement of fact provided in a LLM response and verifying that the original annotator did, indeed, check for any hallucinations.  

Do NOT assume that, just because a task is a certain length or the previous worker has an SME in the topic, there were no mistakes made or issues overlooked.  In fact, you should be even more mindful as it is now your job to catch errors.  Human error occurs even to the best team members, whether due to oversight or fatigue.

#### Regularly refresh yourself on documentation for reviewing (especially if you've been a reviewer for a long time)

Just as you would as an annotator, keeping a copy of the project protocols next to you would make it easier to work. Usually team leads provide a separate set of instructions and documents to reviewers, as the workflow may vary, especially if the project involves edits or sendbacks.

Another reason you may want to regularly check the doc is when updates accompany project pivots. Your lead would likely update it with any shifts in instructions that may accompany the pivot, and it would serve as a reminder to not fall back into old workflows, even if accidentally.

#### Do not assume that everyone approaches a task the same as you

Especially when the original annotator is still correct and all of the information in a task's response is validated, do not penalize them for proceeding in a different way.  If the conclusion is still the same as yours, it does not detract from the quality of work.  In fact, modelsLLMs benefit from being provided with multiple ways to reach the same answer.  

The main exception to this rule is if a protocol is absolutely mandated by the higher ups (normally due to wanting to test out a specific logic process), but your lead would inform you of these cases.

#### Investigate if/when you see trends and log them

It is important to differentiate between a once-off error and a recurring issue.  While mistakes you'd encounter would likely tend to be human error (e.g. incorrect labeling, missed factuality checks, etc.,), it is possible for glitches to affect the data annotations themselves, from either a bug in the platform or an issue with the data set.  If it is not a consistent tech issue, be sure to log it for future reference and surface it with your team when you see it again.

Another possible trend can be annotators' recurring misunderstanding of a new guideline. In this case, consult with other reviewers and congregate the proper information so you can jointly bring this up to your lead.

Positive patterns should also be logged, like project scoring averages steadily increasing, as these factoids are helpful for team morale. Your lead may not always notice these as quickly, given they may be focused on other analytics at the time, but observed improvements would also be appreciated.

#### Identify possible gaps in documentation

While onboarding and protocol documents should provide the whole of the flow and instruction of the project, it is possible that there may be issues that higher-ups may not have foreseen. With an increase in complex queries for most data sets, nuance is often introduced, and can not necessarily be evaluated in a binary function.

For example, if a prompt mentions "a handful of ideas," there is a level of subjectivity introduced due to the usage of a word that has no direct denotation of numerical value.  Different data workers may interpret "handful" in a variety of ways. At this point, the lead would need to make an overall ruling; without it, reviewers may grade similar tasks inconsistently, causing confusion amongst the annotators. Letting a scenario like this go until your lead does an audit would cause more work for everyone.

#### Where possible, offer yourself as a resource

After you have settled in your reviewer duties, take initiative to answer questions for new team members, or offer to help spearhead upskill trainings or learning sessions. Being collaborative helps share project knowledge across the team and ensures that all members (annotators and reviewers) are consistent. 

This would show your lead that you're a team player and potentially ready for more interesting work or responsibilities. Assisting in making sure all annotators and reviewers are equally trained would have an additional side effect of making your work easier.

#### Provide helpful, applicable feedback specific to each task

Especially in the beginning of a project or when a new wave is onboarded, be detailed in what aspects need to be fixed, and why. It is a learning process and these types of comments foster improvement.

#### Be mindful of how your reviews may affect individual morale

Though leads should mainly focus on this, reviewers have a portion of responsibility here too.  Motivated team members make for more efficient workers, which would also likely reduce the amount of errors per task.  

Constructive criticism is necessary when errors are made, but framing it kindly (or even just neutrally) would reduce team discouragement and friction. Feedback is meant to improve the model as a whole, so, while annotators should learn to avoid taking it personally, reviewers should also take care in their language.

#### Know when/if to ask your Lead to be cycled back into annotators

Keep in mind that reviewing and annotating are two different workflows, so it is imperative to not perceive one flow to be "higher" or "more important" than the other.  Some workers have skills more suited for annotation, preferring it as the only work they have to worry about is their own tasks and research. If you find that you do not enjoy reviewing as much as annotating, feel free to let your lead know.

If you're feeling burn-out from constant reviewing, you can request to be temporarily put back into the annotating pool, as a difference in workflow may help "reset" how you work.

---

### Source Guide Organization (`docs/source/guide/organization.md`)

---
title: Organization management
short: Overview
tier: enterprise
type: guide
order: 0
order_enterprise: 351
meta_title: Organization management
meta_description: Brief overview of organization structures in Label Studio Enterprise. 
section: "Manage Your Organization"
date: 2024-02-16 15:44:07
---


To manage organization membership, use the **Organization** page in Label Studio. Only users with the Owner or Administrator role can access this page:

<img style="max-width: 235px" src="/images/admin/org_page.png" alt="Screenshot of org link"/>

## Create the initial organization account

When creating a new organization, you must sign up directly without an invite link. 

For on-prem deployments, use `/user/signup`. 

For SaaS deployments, use [`app.humansignal.com/`](https://app.humansignal.com/).

When you sign up for Label Studio Enterprise or Starter Cloud for the first time, an organization associated with your account is automatically created. You become the owner of that organization. People who join Label Studio from an invitation link or with an LDAP or SSO role join an existing organization.

!!! note
    There can only be one Owner per organization. If the user in control of the Owner account leaves, you will need to [open a support ticket](https://support.humansignal.com/hc/en-us/requests/new) to request that this role be reassigned. 

## Single or multiple organizations

If permitted by your Label Studio plan, you can create organizations in Label Studio to further separate access to data and projects. For example, you could create separate organizations to separate work and access between completely unrelated departments. If some departments might collaborate with each other on a project, you can use one organization for both and instead use workspaces to organize the projects that they might or might not be collaborating on.

For example, you might set up one of the following possible configurations:

- One organization for your company, with one workspace for the support department and another for the development team, with specific projects in each workspace for different types of customer requests.

  <img style="width:70%" src="/images/LSE/LSE-one-org-many-workspaces.jpg" alt="Diagram showing Label Studio with one organization with multiple workspaces and projects within each workspace."/>

- Multiple organizations, such as one for the customer claims department and another for the customer support department, with specific workspaces in each organization for specific types of insurance, such as home insurance claims and auto insurance claims, and specific projects in each workspace for types of claims, such as Accident Claims, Injury Claims, Natural Disaster Claims. The Customer support organization might have workspaces specific to the types of support queues, with projects for specific types of calls received.

  <img style="width:70%" src="/images/LSE/LSE-multiple-orgs-workspaces.jpg" alt="Diagram showing Label Studio with three organizations, each one with multiple workspaces and projects within each workspace."/>

  When you assign a user role to an organization member, they hold that role for all workspaces and projects for that organization.

!!! note
    Having users in multiple organizations may prevent your access to the [Plugins feature](/plugins/). 

Managers within an organization can see all workspaces in that organization, even if they don't have access to perform actions in them. Annotators and reviewers can only see projects, not workspaces.

If you have access to multiple organizations, use the **Organizations** drop-down to switch between the organizations that you are a member of:

<img style="max-width: 242px" src="/images/admin/org-switch.png" alt="Screenshot of org switcher"/>


## Change organization name

Users in the Owner or Admin role can use the API to rename their organization:

```curl
LSE_URL=<Replace with your URL>
ORG_ID=<Replace with your org ID>
TOKEN=<Replace with your token>
  curl -X PATCH "$LSE_URL/api/organizations/$ORG_ID" \ 
    -H "Authorization: Token $TOKEN" \
    -H "Content-Type: application/json" \
    -d '{"title": "New Org Name"}'
```

!!! note
    The example above uses the legacy token. To use your personal access token (JWT), use `-H "Authorization: Bearer $TOKEN"` instead.

---

### Source Guide Persistent Storage (`docs/source/guide/persistent_storage.md`)

---
title: Set up persistent storage
type: guide
tier: all
order: 87
order_enterprise: 87
meta_title: Set up persistent storage with Label Studio
meta_description: Configure persistent storage with Label Studio hosted in the cloud to store uploaded data such as task data, user images, and more.
section: "Install & Setup"
---

Set up persistent storage so that uploaded task data, user images, and other media are kept across restarts and available to all Label Studio components.

!!! note
    By default, Label Studio uses nginx to serve uploaded media. For persistent storage to work correctly, run nginx alongside Label Studio. This is the recommended setup and reduces load on the Label Studio server.

    If you use a minimal setup without nginx, you can set `USE_NGINX_FOR_UPLOADS=false` and `USE_NGINX_FOR_EXPORT_DOWNLOADS=false`. In that case, Label Studio serves all uploads and exports. This is not recommended: it increases load on the uwsgi workers and can cause outages when users work with large files.

Choose one of the following according to your deployment:

* **[Kubernetes with a PVC](#Set-up-Kubernetes-PVC)** — Use a Persistent Volume Claim (ReadWriteMany) for on-cluster storage.
* **[Amazon S3](#Set-up-Amazon-S3)** — For AWS or Docker Compose.
* **[Google Cloud Storage (GCS)](#Set-up-Google-Cloud-Storage)** — For Google Cloud or Docker Compose.
* **[Microsoft Azure Storage](#Set-up-Microsoft-Azure-Storage)** — For Azure or Docker Compose.

## Set up Kubernetes PVC

When you deploy Label Studio on Kubernetes, you can use a Persistent Volume Claim (PVC) for persistent storage instead of cloud object storage. The PVC **must** support **ReadWriteMany** access mode so that multiple pods (for example, the app and rqworker) can read and write the same volume.

Configure persistence in your `ls-values.yaml` using the `persistence.config.volume` section and set `accessModes` to `ReadWriteMany`:

```yaml
global:
  persistence:
    enabled: true
    type: volume
    config:
      volume:
        size: 50Gi
        accessModes:
          - ReadWriteMany
```

!!! note
    Your cluster must provide a StorageClass or default storage that supports `ReadWriteMany` (for example, NFS, EFS, Azure Files, or other network-backed storage). Standard block storage (e.g. many default cloud disks) typically supports only ReadWriteOnce and is not suitable when multiple pods need to mount the same PVC.

## Set up Amazon S3

Set up Amazon S3 as the persistent storage for Label Studio hosted in AWS or using Docker Compose.

### Create an S3 bucket

Start by [creating an S3 bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/create-bucket-overview.html) following the Amazon Simple Storage Service User Guide steps.

!!! note 
    If you want to secure the data stored in the S3 bucket at rest, you can [set up default server-side encryption for Amazon S3 buckets](https://docs.aws.amazon.com/AmazonS3/latest/userguide/bucket-encryption.html) following the steps in the Amazon Simple Storage Service User Guide.

### Configure CORS for the S3 bucket

!!! note 
    In the case if you're going to use direct file upload feature and store media files like audio, video, csv you should complete this step.

Set up Cross-Origin Resource Sharing (CORS) access to your bucket. See [Configuring cross-origin resource sharing (CORS)](https://docs.aws.amazon.com/AmazonS3/latest/userguide/enabling-cors-examples.html) in the Amazon S3 User Guide. Use or modify the following example:
```json
[
  {
    "AllowedHeaders": [
      "*"
    ],
    "AllowedMethods": [
      "GET",
      "PUT",
      "POST",
      "DELETE",
      "HEAD"
    ],
    "AllowedOrigins": [
      "*"
    ],
    "ExposeHeaders": [
      "x-amz-server-side-encryption",
      "x-amz-request-id",
      "x-amz-id-2"
    ],
    "MaxAgeSeconds": 3600
  }
]
```

### Configure the S3 bucket
After you create an S3 bucket, set up the necessary IAM permissions to grant Label Studio access to your bucket. There are four ways that you can manage access to your S3 bucket:
- Set up an **IAM role** with an OIDC provider (**recommended**).
- Use **access keys**.
- Set up an **IAM role** without an OIDC provider.
- Use **access keys with Docker Compose**.

Select the relevant tab and follow the steps for your desired option:

<div class="code-tabs">
  <div data-name="IAM role (OIDC)">

!!! note 
    To set up an IAM role using this method, you must have a configured and provisioned OIDC provider for your cluster. See [Create an IAM OIDC provider for your cluster](https://docs.aws.amazon.com/eks/latest/userguide/enable-iam-roles-for-service-accounts.html) in the Amazon EKS User Guide.


1. Follow the steps to [create an IAM role and policy for your service account](https://docs.aws.amazon.com/eks/latest/userguide/create-service-account-iam-policy-and-role.html) in the Amazon EKS User Guide.
2. Use the following IAM Policy, replacing `<YOUR_S3_BUCKET>` with the name of your bucket:
```json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::<YOUR_S3_BUCKET>"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:GetObject",
        "s3:DeleteObject"
      ],
      "Resource": [
        "arn:aws:s3:::<YOUR_S3_BUCKET>/*"
      ]
    }
  ]
}
```

3. Create an **IAM role as a Web Identity** using the cluster OIDC provider as the identity provider:
   - Create a new **Role** from your IAM Console.
   - Select the **Web identity** Tab.
   - In the **Identity Provider** drop-down, select the OpenID Connect provider URL of your EKS and `sts.amazonaws.com` as the Audience.
   - Attach the newly created permission to the Role and name it.
   - Retrieve the Role arn for the next step.
4. After you create an IAM role, add it as an annotation in your `ls-values.yaml` file.
   Optionally, you can choose a folder by specifying `folder` (default is `""` or omit this argument):
```yaml
global:
  persistence:
    enabled: true
    type: s3
    config:
      s3:
        bucket: "<YOUR_BUCKET_NAME>"
        region: "<YOUR_BUCKET_REGION>"
        folder: ""
app:
  serviceAccount:
    annotations:
      eks.amazonaws.com/role-arn: arn:aws:iam::<ACCOUNT_ID>:role/<ROLE_NAME_FROM_STEP_3>

rqworker:
  serviceAccount:
    annotations:
      eks.amazonaws.com/role-arn: arn:aws:iam::<ACCOUNT_ID>:role/<ROLE_NAME_FROM_STEP_3>
```

  </div>

  <div data-name="Access keys">

1. Create an IAM user with **Programmatic access**. See [Creating an IAM user in your AWS account](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_create.html) in the AWS Identity and Access Management User Guide.
2. When creating the user, for the **Set permissions** option, choose to **Attach existing policies directly**.
3. Select **Create policy** and attach the following policy, replacing `<YOUR_S3_BUCKET>` with the name of your bucket:
```json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::<YOUR_S3_BUCKET>"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:GetObject",
        "s3:DeleteObject"
      ],
      "Resource": [
        "arn:aws:s3:::<YOUR_S3_BUCKET>/*"
      ]
    }
  ]
}
```

4. After you create the user, save the username and access key somewhere secure.
5. Update your `ls-values.yaml` file with your newly-created access key ID and secret key as `<YOUR_ACCESS_KEY_ID>` and `<YOUR_SECRET_ACCESS_KEY>`.
   Optionally, you can choose a folder by specifying `folder` (default is `""` or omit this argument):
```yaml
global:
  persistence:
    enabled: true
    type: s3
    config:
      s3:
        accessKey: "<YOUR_ACCESS_KEY_ID>"
        secretKey: "<YOUR_SECRET_ACCESS_KEY>"
        bucket: "<YOUR_BUCKET_NAME>"
        region: "<YOUR_BUCKET_REGION>"
        folder: ""
```

!!! note 
    Optionally, you can use already existing Kubernetes secret and a key.


1. Create a Kubernetes secret with your AWS access keys:
```shell
kubectl create secret generic <YOUR_SECRET_NAME> --from-literal=accesskey=<YOUR_ACCESS_KEY_ID> --from-literal=secretkey=<YOUR_SECRET_ACCESS_KEY>
```
2. Update your `ls-values.yaml` file with your newly-created kubernetes secret:
```yaml
global:
  persistence:
    enabled: true
    type: s3
    config:
      s3:
        accessKeyExistingSecret: "<YOUR_SECRET_NAME>"
        accessKeyExistingSecretKey: "accesskey"
        secretKeyExistingSecret: "<YOUR_SECRET_NAME>"
        secretKeyExistingSecretKey: "secretkey"
        bucket: "<YOUR_BUCKET_NAME>"
        region: "<YOUR_BUCKET_REGION>"
```

  </div>

  <div data-name="IAM role (EKS node)">

To create an IAM role without using OIDC in EKS, follow these steps.

1. In the AWS console UI, go to **EKS > Clusters > `YOUR_CLUSTER_NAME` > Node Group**.
2. Select the name of `YOUR_NODE_GROUP` with Label Studio deployed.
3. On the **Details** page, locate and select the option for Node IAM Role ARN and choose to **Attach existing policies directly**.
3. Select **Create policy** and attach the following policy, replacing `<YOUR_S3_BUCKET>` with the name of your bucket:
```json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::<YOUR_S3_BUCKET>"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:GetObject",
        "s3:DeleteObject"
      ],
      "Resource": [
        "arn:aws:s3:::<YOUR_S3_BUCKET>/*"
      ]
    }
  ]
}
```

4. After you add an IAM policy, configure your `ls-values.yaml` file.
   Optionally, you can choose a folder by specifying `folder` (default is `""` or omit this argument):
```yaml
global:
  persistence:
    enabled: true
    type: s3
    config:
      s3:
        bucket: "<YOUR_BUCKET_NAME>"
        region: "<YOUR_BUCKET_REGION>"
        folder: ""
```

  </div>

  <div data-name="Docker Compose">

1. Create an IAM user with **Programmatic access**. See [Creating an IAM user in your AWS account](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_create.html) in the AWS Identity and Access Management User Guide.
2. When creating the user, for the **Set permissions** option, choose to **Attach existing policies directly**.
3. Select **Create policy** and attach the following policy, replacing `<YOUR_S3_BUCKET>` with the name of your bucket:
```json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::<YOUR_S3_BUCKET>"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:GetObject",
        "s3:DeleteObject"
      ],
      "Resource": [
        "arn:aws:s3:::<YOUR_S3_BUCKET>/*"
      ]
    }
  ]
}
```

4. After you create the user, save the username and access key somewhere secure.
5. Update your `env.list` file, replacing `<YOUR_ACCESS_KEY_ID>` and `<YOUR_SECRET_ACCESS_KEY>` with your newly-created access key ID and secret key. Optionally, you can specify a folder using `STORAGE_AWS_FOLDER` (default is `""` or omit this argument):
```shell
STORAGE_TYPE=s3
STORAGE_AWS_ACCESS_KEY_ID="<YOUR_ACCESS_KEY_ID>"
STORAGE_AWS_SECRET_ACCESS_KEY="<YOUR_SECRET_ACCESS_KEY>"
STORAGE_AWS_BUCKET_NAME="<YOUR_BUCKET_NAME>"
STORAGE_AWS_REGION_NAME="<YOUR_BUCKET_REGION>"
STORAGE_AWS_FOLDER=""
```

  </div>
</div>

## Set up Google Cloud Storage

Set up Google Cloud Storage (GCS) as the persistent storage for Label Studio hosted in Google Cloud Platform (GCP) or Docker Compose.

### Create a GCS bucket

1. Start by creating a bucket. See [Creating storage buckets](https://cloud.google.com/storage/docs/creating-buckets) in the Google Cloud Storage guide. For example, a bucket called `heartex-example-bucket-123456`.
2. When choosing the [access control method for the bucket](https://cloud.google.com/storage/docs/access-control), choose **uniform access control**.
3. Create an IAM Service Account. See [Creating and managing service accounts](https://cloud.google.com/iam/docs/creating-managing-service-accounts) in the Google Cloud Storage guide.
4. Select the predefined **Storage Object Admin** IAM role to add to the service account so that the account can create, access, and delete objects in the bucket.
5. Add a condition to the role that restricts the role to access only objects that belong to the bucket you created. You can add a condition in one of two ways:
    - Select **Add Condition** when setting up the service account IAM role, then use the **Condition Builder** to specify the following values:
      - Condition type: `Name`
      - Operator: `Starts with`
      - Value: `projects/_/buckets/heartex-example-bucket-123456`
    - Or, **use a Common Expression Language** (CEL) to specify an IAM condition. For example, set the following: `resource.name.startsWith('projects/_/buckets/heartex-example-bucket-123456')`. See [CEL for Conditions in Overview of IAM Conditions](https://cloud.google.com/iam/docs/conditions-overview#cel) in the Google Cloud Storage guide.

### Configure CORS for the GCS bucket

!!! note 
    In the case if you're going to use direct file upload feature and store media files like audio, video, csv you should complete this step.

Set up CORS access to your bucket. See [Configuring cross-origin resource sharing (CORS)](https://cloud.google.com/storage/docs/configuring-cors#configure-cors-bucket) in the Google Cloud User Guide. Use or modify the following example:
```shell
echo '[
   {
      "origin": ["*"],
      "method": ["GET","PUT","POST","DELETE","HEAD"],
      "responseHeader": ["Content-Type","Access-Control-Allow-Origin"],
      "maxAgeSeconds": 3600
   }
]' > cors-config.json
```

Replace `YOUR_BUCKET_NAME` with your actual bucket name in the following command to update CORS for your bucket:
```shell
gsutil cors set cors-config.json gs://YOUR_BUCKET_NAME
```

### Configure the GCS bucket

You can connect Label Studio to your GCS bucket using **Workload Identity** or **Access keys**.

After you create a bucket and set up IAM permissions, connect Label Studio to your GCS bucket. There are three ways that you can connect to your bucket:
- Use Workload Identity to allow workloads in GKE to access your GCS bucket by impersonating the service account you created (**recommended**).
- Create a service account key to use the service account outside Google Cloud.
- Create a service account key to use with Docker Compose.

<div class="code-tabs">
<div data-name="Workload Identity">

!!! note 
    Make sure that Workload Identity is enabled on your GKE cluster and that you meet the necessary prerequisites. See [Using Workload Identity](https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity) in the Google Kubernetes Engine guide.

1. Set up the following environment variables, specifying the service account you created as the `GCP_SA` variable, and replacing the other references in `<>` as needed:
```shell
GCP_SA=<Service-Account-You-Created>
APP_SA="serviceAccount:<GCP_PROJECT_ID>.svc.id.goog[<K8S_NAMESPACE>/<HELM_RELEASE_NAME>-lse-app]"
WORKER_SA="serviceAccount:<GCP_PROJECT_ID>.svc.id.goog[<K8S_NAMESPACE>/<HELM_RELEASE_NAME>-lse-rqworker]"
```

2. Create an IAM policy binding between the Kubernetes service account on your cluster and the GCS service account you created, allowing the K8s service account for the Label Studio app and the related rqworkers to impersonate the other service account. From the command line, run the following:
```shell
gcloud iam service-accounts add-iam-policy-binding ${GCP_SA} \
    --role roles/iam.workloadIdentityUser \
    --member "${APP_SA}"
gcloud iam service-accounts add-iam-policy-binding ${GCP_SA} \
    --role roles/iam.workloadIdentityUser \
    --member "${WORKER_SA}"
```

3. After binding the service accounts, update your `ls-values.yaml` file to include the values for the service account and other configurations. Update the `projectID`, `bucket`, and replace the`<GCP_SERVICE_ACCOUNT>` with the relevant values for your deployment.
   Optionally, you can choose a folder by specifying `folder` (default is `""` or omit this argument):
```yaml
global:
  persistence:
    enabled: true
    type: gcs
    config:
      gcs:
        projectID: "<YOUR_PROJECT_ID>"
        bucket: "<YOUR_BUCKET_NAME>"
        folder: ""
app:
  serviceAccount:
    annotations:
      iam.gke.io/gcp-service-account: "<GCP_SERVICE_ACCOUNT>"

rqworker:
  serviceAccount:
    annotations:
      iam.gke.io/gcp-service-account: "<GCP_SERVICE_ACCOUNT>"
```

</div>
<div data-name="Service Account Key">

You can use a service account key that you create, or if you already have a Kubernetes secret and key, follow [the steps below](#Use-an-existing-Kubernetes-secret-and-key) to use those.

#### Create a new service account key
1. Create a service account key from the UI and download the JSON. Follow the steps for [Creating and managing service account keys](https://cloud.google.com/iam/docs/creating-managing-service-account-keys) in the Google Cloud Identity and Access Management guide.

2. After downloading the JSON for the service account key, update or create references to the JSON, your projectID, and your bucket in your `ls-values.yaml` file.
   Optionally, you can choose a folder by specifying `folder` (default is `""` or omit this argument):
```yaml
global:
  persistence:
    enabled: true
    type: gcs
    config:
      gcs:
        projectID: "<YOUR_PROJECT_ID>"
        applicationCredentialsJSON: "<YOUR_JSON>"
        bucket: "<YOUR_BUCKET_NAME>"
        folder: ""
```

#### Use an existing Kubernetes secret and key

1. Create a Kubernetes secret with your GCS service account JSON file, replacing `<PATH_TO_JSON>` with the path to the service account JSON file:
```shell
kubectl create secret generic <YOUR_SECRET_NAME> --from-file=key_json=<PATH_TO_JSON>
```

2. Update your `ls-values.yaml` file with your newly-created Kubernetes secret:
```yaml
global:
   persistence:
      enabled: true
      type: gcs
      config:
         gcs:
            projectID: "<YOUR_PROJECT_ID>"
            applicationCredentialsJSONExistingSecret: "<YOUR_SECRET_NAME>"
            applicationCredentialsJSONExistingSecretKey: "key_json"
            bucket: "<YOUR_BUCKET_NAME>"
```

</div>
<div data-name="Docker Compose">

1. Create a service account key from the UI and download the JSON. Follow the steps for [Creating and managing service account keys](https://cloud.google.com/iam/docs/creating-managing-service-account-keys) in the Google Cloud Identity and Access Management guide.
2. After downloading the JSON for the service account key, update or create references to the JSON, your projectID, and your bucket in your `env.list` file.
   Optionally, you can choose a folder by specifying `STORAGE_GCS_FOLDER` (default is `""` or omit this argument):
```shell
STORAGE_TYPE=gcs
STORAGE_GCS_BUCKET_NAME="<YOUR_BUCKET_NAME>"
STORAGE_GCS_PROJECT_ID="<YOUR_PROJECT_ID>"
STORAGE_GCS_FOLDER=""
GOOGLE_APPLICATION_CREDENTIALS="/opt/heartex/secrets/key.json"
```

3. Place the downloaded JSON file from step 1 in the same directory as your `env.list` file.

4. Append the following entry in `docker-compose.yml` file as the path for `app.volumes`:
```yaml
- ./service-account-file.json:/opt/heartex/secrets/key.json:ro
```

</div>
</div>


## Set up Microsoft Azure Storage

Create a Microsoft Azure Storage container to use as persistent storage with Label Studio.

### Create a Storage container

1. Create an Azure storage account. See [Create a storage account](https://docs.microsoft.com/en-us/azure/storage/common/storage-account-create?tabs=azure-portal) in the Microsoft Azure product documentation.

!!! note 
    Make sure that you set **Stock Keeping Unit (SKU)** to `Premium_LRS` and the **kind** parameter to `BlockBlobStorage`. This configuration results in storage that uses solid state drives (SSDs) rather than standard hard disk drives (HDDs). If you set this parameter to an HDD-based storage option, your instance might be too slow and could malfunction.

2. Find the generated key in the **Storage accounts > Access keys** section in the [Azure Portal](https://portal.azure.com/) or by running the following command:
```shell
az storage account keys list --account-name=${STORAGE_ACCOUNT}
```

3. Create a storage container within your storage account by following the steps to [Upload, download, and list blobs with the Azure portal](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-quickstart-blobs-portal) in the Microsoft Azure product documentation, or run the following command:
```shell
az storage container create --name <YOUR_CONTAINER_NAME> \
          --account-name <YOUR_STORAGE_ACCOUNT> \
          --account-key "<YOUR_STORAGE_KEY>"
```

### Configure CORS for the Azure bucket

!!! note 
    In the case if you're going to use direct file upload feature and store media files like audio, video, csv you should complete this step.

Set up CORS access to your bucket. See [Configuring cross-origin resource sharing (CORS)](https://docs.microsoft.com/en-us/rest/api/storageservices/cross-origin-resource-sharing--cors--support-for-the-azure-storage-services#enabling-cors-for-azure-storage) in the Azure User Guide. Use or modify the following example:
```xml
<Cors>
    <CorsRule>
        <AllowedOrigins>*</AllowedOrigins>
        <AllowedMethods>GET,PUT,POST,DELETE,HEAD</AllowedMethods>
        <AllowedHeaders>x-ms-blob-content-type</AllowedHeaders>
        <ExposedHeaders>x-ms-*</ExposedHeaders>
        <MaxAgeInSeconds>3600</MaxAgeInSeconds>
    </CorsRule>
<Cors>
```

### Configure the Azure container

You can connect Label Studio to your Azure container using account keys in Kubernetes or account keys in Docker Compose. Choose the option relevant to your Label Studio deployment.

<div class="code-tabs">
  <div data-name="Kubernetes">

Update your `ls-values.yaml` file with the `YOUR_CONTAINER_NAME`, `YOUR_STORAGE_ACCOUNT`, and `YOUR_STORAGE_KEY` that you created.
Optionally, you can choose a folder by specifying `folder` (default is `""` or omit this argument):
```yaml
global:
  persistence:
    enabled: true
    type: azure
    config:
      azure:
        storageAccountName: "<YOUR_STORAGE_ACCOUNT>"
        storageAccountKey: "<YOUR_STORAGE_KEY>"
        containerName: "<YOUR_CONTAINER_NAME>"
        folder: ""
```

If you have an existing key, you can use that instead to create a Kubernetes secret.
1. Create a Kubernetes secret with your Azure access key:
```shell
kubectl create secret generic <YOUR_SECRET_NAME> --from-literal=storageaccountname=<YOUR_STORAGE_ACCOUNT> --from-literal=storageaccountkey=<YOUR_STORAGE_KEY>
```
2. Update your `ls-values.yaml` file with your newly-created Kubernetes secret:
```yaml
global:
   persistence:
      enabled: true
      type: azure
      config:
         azure:
            storageAccountNameExistingSecret: "<YOUR_SECRET_NAME>"
            storageAccountNameExistingSecretKey: "storageaccountname"
            storageAccountKeyExistingSecret: "<YOUR_SECRET_NAME>"
            storageAccountKeyExistingSecretKey: "storageaccountkey"
            containerName: "<YOUR_CONTAINER_NAME>"
```

  </div>

  <div data-name="Docker Compose">

Update your `env.list` file with the `YOUR_CONTAINER_NAME`, `YOUR_STORAGE_ACCOUNT`, and `YOUR_STORAGE_KEY` that you created.
Optionally, you can choose a folder by specifying `STORAGE_AZURE_FOLDER` (default is `""` or omit this argument):
```shell
STORAGE_TYPE=azure
STORAGE_AZURE_ACCOUNT_NAME="<YOUR_STORAGE_ACCOUNT>"
STORAGE_AZURE_ACCOUNT_KEY="<YOUR_STORAGE_KEY>"
STORAGE_AZURE_CONTAINER_NAME="<YOUR_CONTAINER_NAME>"
STORAGE_AZURE_FOLDER=""
```

  </div>
</div>

---

### Source Guide Pingone (`docs/source/guide/pingone.md`)

---
title: Ping Federate & Ping Identity SAML SSO Setup Example
short: Ping Federate & Ping Identity & PingOne SAML SSO Setup
tier: enterprise
order: 253
meta_title: Ping Federate & Ping Identity & PingOne SAML SSO Setup Example
meta_description: Label Studio Enterprise documentation for setting up Ping Federate & Ping Identity & PingOne SAML SSO Setup Example.
hide_sidebar: true
---

## PingOne Configuration

### Add new application 

1. Click blue circle with the `+` sign.  
<img src="/images/pingone/setup-1.png" class="gif-border">

2. Select `Manual Enter`, enter URLs from the Label Studio Enterprise SAML SSO page.
<img src="/images/pingone/setup-2.png" class="gif-border">

3. The result
<img src="/images/pingone/main.png" class="gif-border">


### Overview Tab

<img src="/images/pingone/overview.png" class="gif-border">
<br><br>

### Configuration Tab

<img src="/images/pingone/configuration-1.png" class="gif-border">
<br><br>

<img src="/images/pingone/configuration-2.png" class="gif-border">
<br><br>

### Attributes Tab

You should use this group value in mapping:
```
Expression: ${user.memberOfGroupNames == null ? '': #string.join(user.memberOfGroupNames, ',')}
```
<br>
<img src="/images/pingone/attributes.png" class="gif-border">
<br><br>

### Policies Tab

<img src="/images/pingone/policies.png" class="gif-border">
<br><br>

### Access Tab

<img src="/images/pingone/access.png" class="gif-border">

## Users & Groups

!!! warning
    All users should have at least one group, otherwise it will lead to the login error.
     

## Label Studio Enterprise Settings 

You can find LSE SAML SSO settings on the **Organization** page >> **SAML SSO**. You are able to map user roles and workspaces with `Groups` attribute.  
 
<br>
<img src="/images/pingone/saml-settings.png" class="gif-border">


## Login using SSO 

Now you are able to login using your SSO session, write your company domain on the SSO Login page:

<br>
<img src="/images/pingone/login-sso.png" class="gif-border">

Administrators can find the domain on the SAML SSO page:

<br>
<img src="/images/pingone/domain-settings.png" class="gif-border">

---

### Source Guide Plugins (`docs/source/guide/plugins.md`)

---
title: Plugins for projects
short: Plugins
tier: enterprise
type: guide
order: 0
order_enterprise: 140
meta_title: Plugins
meta_description: Use JavaScript to customize your labeling interface. 
section: "Create & Manage Projects"
date: 2024-07-30 10:39:03
---

You can extend your Labeling Configuration by implementing a custom JavaScript plugin.

Plugins (previously called "Custom Scripts") are defined in the project settings under the Labeling Interface section: 



!!! note
    Plugins are not available unless enabled. There are [important security considerations](#Security-notes-constraints-and-limitations) to understand before requesting access. To enable plugins for your organization, contact your account manager or go to **Project > Settings > Labeling Interface** and click **Request Access**. 

!!! error Enterprise
    Plugins are only available in Label Studio Enterprise. They are not available in Starter Cloud or Community editions. 

## Plugin examples

We provide out-of-the-box plugins that you can modify and use. You can also build your own custom plugins. 

For more information, see our [Plugins Gallery](/plugins) and [Customize and Build Your Own Plugins](/plugins/custom).

<a href="https://docs.humansignal.com/plugins/"><img src="/images/plugins/gallery.png" alt="Plugin gallery"></a>

## Use cases

Plugins allow you to enhance and tailor your labeling interface and workflow. For example:

* **General validation** – Plugins can be used to implement various validation checks to ensure the quality and consistency of annotations. 

    Examples: Data integrity checks, logical consistency checks, completeness checks. 

* **Incorporate dynamic elements** – Incorporate dynamic visualizations into the labeling interface to provide context or aid annotators in their tasks. 

    Examples: Charts or other data visualizations, insert custom options for tasks, real-time visual aids for annotations. 

* **Query external databases** – Enable dynamic querying of external databases to fetch data necessary for annotation tasks. 

    Examples: Retrieving contextual data, retrieving the latest data to populate choices

* **Workflow validations and customizations** – Use custom logic and validations to trigger subsequent workflows or actions based on the outcomes of the current annotation task.

    Examples: Feedback loops for annotators, conditional routing/assignment, progressive sampling, notifications regarding project progress. 

## Security notes, constraints, and limitations

Plugins are a powerful tool to help you fully customize your labeling workflow. In doing so you are running arbitrary JavaScript code on each annotator’s machine, which comes with certain risks. 

Because of this, you must opt-in before enabling plugins for your organization, and we urge you to use this feature with caution.

To enable plugins for your organization, you cannot have members that are in multiple organizations. This is enforced through application logic and is necessary for data security. The most common reason for this is when users have accounts in an expired free trial, but can also happen if you are using multiple organizations for project management or if you have an initial proof of concept or testing org. 


!!! note
    By default, only users who are in the Admin, Owner, or Manager role can access the project settings to view, add, and edit plugins. 
    
    You can optionally add a restriction so that only users in the Admin or Owner role can add or edit plugins. To request this, contact your account manager or [open a support ticket](mailto:support@humansignal.com).

---

### Source Guide Predictions (`docs/source/guide/predictions.md`)

---
title: Import pre-annotated data into Label Studio
short: Import pre-annotations
type: guide
tier: all
order: 163
order_enterprise: 163
meta_title: Import pre-annotated data into Label Studio
meta_description: Import predicted labels, predictions, pre-annotations, or pre-labels into Label Studio for your data labeling, machine learning, and data science projects.
section: "Import & Export"
parent: "tasks"
parent_enterprise: "tasks"
---

If you have predictions generated for your dataset from a model, either as pre-annotated tasks or pre-labeled tasks, you can import the predictions with your dataset into Label Studio for review and correction. Label Studio automatically displays the pre-annotations that you import on the Labeling page for each task. 

You can import pre-annotated tasks into Label Studio [using the UI](tasks.html#Import-data-from-the-Label-Studio-UI) or [using the API](/api#operation/projects_import_create). 

!!! note 
    To generate interactive pre-annotations with a machine learning model while labeling, see [Set up machine learning with Label Studio](ml.html).

<div class="opensource-only">

!!! error enterprise
    Label Studio Enterprise customers have access to a powerful tool called Prompts. With Prompts, you can leverage LLMs to automatically generate predictions for your Label Studio tasks. For more information, see [Prompts](https://docs.humansignal.com/guide/prompts_overview).

</div>

<div class="enterprise-only">

!!! info Tip
    With Prompts, you can leverage LLMs to automatically generate predictions for your Label Studio tasks. For more information, see [Prompts](prompts_overview).

</div>


## Prepare pre-annotations for Label Studio 

To import predicted labels into Label Studio, you must use the [Basic Label Studio JSON format](tasks.html#Basic-Label-Studio-JSON-format) and set up your tasks with the `predictions` JSON key. The Label Studio ML backend also outputs tasks in this format. Check this common video tutorial showing how to convert a submitted annotation to a prediction:

<iframe class="video-border" width="100%" height="400vh" src="https://www.youtube.com/embed/CyRe73VD4EE" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>

### JSON format for pre-annotations

Label Studio JSON format for pre-annotations must contain two sections:
- A `data` object which references the source of the data that the pre-annotations apply to. This can be a URL to an audio file, a pre-signed cloud storage link to an image, plain text, a reference to a CSV file stored in Label Studio, or something else. See how to [specify the data object](#Specify-the-data-object).
- A `predictions` array that contains the pre-annotation results for the different types of labeling. See how to [add results to the predictions array](#Add-results-to-the-predictions-array).

The JSON format for pre-annotations must match the labeling configuration used for your data labeling project. 

#### Specify the data object 
Use the `data` object to reference the `value` of the data specified by the [Object tag](/tags) in your labeling configuration. For example, the following excerpt of a time series labeling configuration:
```xml
...
  <TimeSeries name="ts" value="$csv" valueType="url">
      <Channel column="first_column"/>
  </TimeSeries>
...
```
This excerpt specifies `value="$csv"` in the TimeSeries Object tag. As a result, the data object for the pre-annotations JSON file for this labeling configuration must use "csv" to specify the location of the CSV data for the time series pre-annotations, like in the following example:

```json
[
  {
    "data": {
      "csv": "https://app.heartex.ai/samples/time-series.csv?time=None&values=first_column"
    },
    "predictions": [ ]
  }
]
```

#### Add results to the predictions array 

The `predictions` array also depends on the labeling configuration. Some pre-annotation fields are only relevant for certain types of labeling. The following table describes the JSON objects and arrays that exist for all pre-annotations: 

| JSON key                    | type   | description                                                                                                                                      |
|-----------------------------|--------|--------------------------------------------------------------------------------------------------------------------------------------------------|
| `predictions`               | array  | Contains all pre-annotations for a specific task.                                                                                                | 
| `predictions.model_version` | string | Specifies the model version that produced the prediction.                                                                                        |
| `predictions.result`        | array  | Contains all pre-annotated results for a specific task in a JSON object.                                                                         |
| `result.value`              | object | Contains details for a specific labeled region.                                                                                                  |
| `result.id`                 | string | Arbitrary string used to identify a labeled region. Must match related regions, such as in [OCR pre-annotations](#Import-OCR-pre-annotations).   |
| `result.from_name`          | string | String used to reference the labeling configuration `from_name` for the type of labeling being performed. Must match the labeling configuration. |
| `result.to_name`            | string | String used to reference the labeling configuration `to_name` for the type of labeling being performed. Must match the labeling configuration.   |
| `result.type`               | string | Specify the labeling tag for the type of labeling being performed. For example, a named entity recognition task has a type of `labels`.          |
| `result.readonly`           | bool   | readonly mode for a specific region                                                                                                              | 
| `result.hidden`             | bool   | default visibility (eye icon) for a specific region                                                                                              |

Other types of annotation contain specific fields. You can review the [examples on this page](#Specific-examples-for-pre-annotations), or review the [tag documentation for the Object and Control tags](/tags) in your labeling configuration labeling-specific `result` objects. For example, the [Audio tag](/tags/audio.html), [HyperText tag](/tags/hypertext.html), [Paragraphs tag](/tags/paragraphs.html), [KeyPointLabels](/tags/keypointlabels.html) and more all contain sample `result` JSON examples.

!!! info
    If you're generating pre-annotations for a [custom ML backend](ml_create.html), you can use the `self.parsed_label_config` variable to retrieve the labeling configuration for a project and generate pre-annotations. See the [custom ML backend](ml_create.html) documentation for more details.


### Predictions are read-only

Predictions cannot be modified and are always read-only.

If the "Show predictions to annotators in the Label Stream and Quick View" toggle in the project settings (Machine Learning section) is enabled, predictions will be automatically copied to newly created annotations. In Quick View mode, annotations can be created from predictions by using the special copy button on the editor tool panel: 
<img src="/images/quickview-copy-button.png" class="gif-border">


### Specific examples for pre-annotations

Refer to the following examples for sample pre-annotation formats:
- [Import bbox and choice pre-annotations for images](#Import-bbox-and-choice-pre-annotations-for-images)
- [Import pre-annotated rectangle, polygon, ellipse & keypoint regions without labels for images](#Import-pre-annotated-rectangle-polygon-ellipse-amp-keypoint-regions-without-labels-for-images)
- [Text pre-annotations with NER spans](#Import-span-pre-annotations-for-text)
- [Brush pre-annotations for segmentation with masks](#Import-brush-segmentation-pre-annotations-in-RLE-format)
- [OCR pre-annotations with bounding boxes, labels, and text transcriptions](#Import-OCR-pre-annotations)

To format pre-annotations for Label Studio not represented in these examples, refer to the sample results JSON for the relevant object and control tags for your labeling configuration, such as the [Audio tag](/tags/audio.html) for audio classification tasks. Each tag must be represented in the JSON pre-annotations format to render predictions in the Label Studio UI. Not all object and control tags list sample results JSON. 

You can also use the [Label Studio Playground](/playground) to preview the output JSON for a specific labeling configuration.


## Import bbox and choice pre-annotations for images

For example, import predicted **bounding box regions (rectangles)** and **choices** for tasks to determine whether an item in an image is an airplane or a car. 

For image pre-annotations, Label Studio expects the x, y, width, and height of image annotations to be provided in percentages of overall image dimension. See [Units for image annotations](predictions.html#Units_for_image_annotations) on this page for more about how to convert formats.

Use the following labeling configuration: 
```xml
<View>
  <Choices name="choice" toName="image" showInLine="true">
    <Choice value="Boeing" background="blue"/>
    <Choice value="Airbus" background="green" />
  </Choices>

  <RectangleLabels name="label" toName="image">
    <Label value="Airplane" background="green"/>
    <Label value="Car" background="blue"/>
  </RectangleLabels>

  <Image name="image" value="$image"/>
</View>
```

After you set up an example project, create example tasks that match the following format:

{% details <b>Click to expand the JSON example with predictions</b> %}
Save this example JSON as a file to import it into Label Studio, for example, `example_prediction_task.json`.

{% codeblock lang:json %}
[{
  "data": {
    "image": "/static/samples/sample.jpg" 
  },

  "predictions": [{
    "model_version": "one",
    "score": 0.5,
    "result": [
      {
        "id": "result1",
        "type": "rectanglelabels",        
        "from_name": "label", "to_name": "image",
        "original_width": 600, "original_height": 403,
        "image_rotation": 0,
        "value": {
          "rotation": 0,          
          "x": 4.98, "y": 12.82,
          "width": 32.52, "height": 44.91,
          "rectanglelabels": ["Airplane"]
        }
      },
      {
        "id": "result2",
        "type": "rectanglelabels",        
        "from_name": "label", "to_name": "image",
        "original_width": 600, "original_height": 403,
        "image_rotation": 0,
        "value": {
          "rotation": 0,          
          "x": 75.47, "y": 82.33,
          "width": 5.74, "height": 7.40,
          "rectanglelabels": ["Car"]
        }
      },
      {
        "id": "result3",
        "type": "choices",
        "from_name": "choice", "to_name": "image",
        "value": {
          "choices": ["Airbus"]
      }
    }]
  }]
}]
{% endcodeblock %}

In this example there are 3 results inside 1 prediction, or pre-annotation: 
- `result1` - the first bounding box
- `result2` - the second bounding box
- `result3` - choice selection
The prediction score applies to the entire prediction.

{% enddetails %}
<br/>

Import pre-annotated tasks into Label Studio [using the UI](tasks.html#Import-data-from-the-Label-Studio-UI) or [using the API](/api#operation/projects_import_create). 

In the Label Studio UI, the imported prediction for this task looks like the following: 
<img src="../images/predictions_loaded.png" alt="screenshot of the Label Studio UI showing an image of airplanes with bounding boxes covering each airplane." style="max-width: 600px;" class="gif-border">

## Import pre-annotated rectangle, polygon, ellipse & keypoint regions without labels for images  

If you want to import images with pre-annotated regions without labels assigned to them, follow this example.

Use the following labeling configuration: 
```xml
<View>
  <Image name="image" value="$image" zoom="true" zoomControl="true" rotateControl="false"/>

  <Rectangle name="rect" toName="image" showInline="false"/>
  <Polygon name="polygon" toName="image"/>
  <Ellipse name="ellipse" toName="image"/>
  <KeyPoint name="kp" toName="image"/>

  <Labels name="labels" toName="image" fillOpacity="0.5" strokeWidth="5">
    <Label value="Vehicle" background="green"/>
    <Label value="Building" background="blue"/>
    <Label value="Pavement" background="red"/>
  </Labels>
</View>
```

After you set up an example project, create example tasks that match the following format: 

{% details <b>Click to expand the JSON example</b> %}
Save this example JSON as a file to import it into Label Studio, for example, `example_prediction_task.json`.

{% codeblock lang:json %}
[{
  "data": {
    "image": "/static/samples/sample.jpg"
  },
  
  "predictions": [
    {
      "model_version": "one",
      "score": 0.5,
      "result": [
        {
          "original_width": 800,
          "original_height": 450,
          "image_rotation": 0,
          "value": {
            "x": 55.46,
            "y": 2.36,
            "width": 35.86,
            "height": 46.9,
            "rotation": 0
          },
          "id": "ABC",
          "from_name": "rect",
          "to_name": "image",
          "type": "rectangle"
        },
        {
          "original_width": 800,
          "original_height": 450,
          "image_rotation": 0,
          "value": {
            "points": [
              [
                20.93,
                28.90
              ],
              [
                25.86,
                64.69
              ],
              [
                38.40,
                62.79
              ],
              [
                34.13,
                27.48
              ]
            ]
          },
          "id": "GHI",
          "from_name": "polygon",
          "to_name": "image",
          "type": "polygon"
        },
        {
          "original_width": 800,
          "original_height": 450,
          "image_rotation": 0,
          "value": {
            "x": 8.4,
            "y": 20.14,
            "radiusX": 4,
            "radiusY": 7.10,
            "rotation": 0
          },
          "id": "JKL",
          "from_name": "ellipse",
          "to_name": "image",
          "type": "ellipse"
        },
        {
          "original_width": 800,
          "original_height": 450,
          "image_rotation": 0,
          "value": {
            "x": 38.40,
            "y": 34.21,
            "width": 1.0
          },
          "id": "DEF",
          "from_name": "rect",
          "to_name": "image",
          "type": "keypoint"
        }
      ]
    }
  ]
}]
{% endcodeblock %}

In this example there are 3 regions inside 1 result field for a prediction, or pre-annotation: 
- region `ABC` - a rectangle bounding box
- region `DEF` - a second rectangle bounding box
- region `GHI` - a polygonal segmentation
- region `JKL` - an ellipse
None of the regions have labels applied. The labeling configuration must use the `Rectangle` tag instead of the `RectangleLabels` tag to support this type of prediction. Even though the labeling configuration for this example has a `Labels` tag, the predictions do not need to specify labels for the regions. 

{% enddetails %}
<br/>

## Import span pre-annotations for text 

In this example, import pre-annotations for text using the [named entity recognition template](/templates/named_entity.html) (NER):
```xml
<View>
  <Labels name="label" toName="text">
    <Label value="Person"></Label>
    <Label value="Organization"></Label>
    <Label value="Fact"></Label>
    <Label value="Money"></Label>
    <Label value="Date"></Label>
    <Label value="Time"></Label>
    <Label value="Ordinal"></Label>
    <Label value="Percent"></Label>
    <Label value="Product"></Label>
    <Label value="Language"></Label>
    <Label value="Location"></Label>
  </Labels>
  <Text name="text" value="$text"></Text>
</View>
```

This example JSON file contains two tasks, each with two sets of pre-annotations from different models. The first task also contains prediction scores for each NER span: 

{% details <b>Click to expand the NER JSON example</b> %}
Save this example JSON as a file, for example: `example_preannotated_ner_tasks.json`.

{% codeblock lang:json %}
[
  {
    "data": {
      "text": "All that changed when he was 27 and he came to Jerusalem. It was the weekend of both Easter and Passover, and the city was flooded with tourists."
    },
    "predictions": [
      {
        "model_version": "one",
        "score": 0.5,
        "result": [
          {
            "id": "abc",
            "from_name": "label",
            "to_name": "text",
            "type": "labels",
            "value": {
              "start": 29,
              "end": 31,
              "score": 0.70,
              "text": "27",
              "labels": [
                "Date"
              ]
            }
          },
          {
            "id": "def",
            "from_name": "label",
            "to_name": "text",
            "type": "labels",
            "value": {
              "start": 47,
              "end": 56,
              "score": 0.65,
              "text": "Jerusalem",
              "labels": [
                "Location"
              ]
            }
          },
          {
            "id": "ghi",
            "from_name": "label",
            "to_name": "text",
            "type": "labels",
            "value": {
              "start": 65,
              "end": 76,
              "score": 0.95,
              "text": "the weekend",
              "labels": [
                "Date"
              ]
            }
          },
          {
            "id": "jkl",
            "from_name": "label",
            "to_name": "text",
            "type": "labels",
            "value": {
              "start": 85,
              "end": 91,
              "score": 0.50,
              "text": "Easter",
              "labels": [
                "Date"
              ]
            }
          }
        ]
      },
      {
        "model_version": "two",
        "score": 0.42,
        "result": [
          {
            "id": "mno",
            "from_name": "label",
            "to_name": "text",
            "type": "labels",
            "value": {
              "start": 29,
              "end": 31,
              "score": 0.55,
              "text": "27",
              "labels": [
                "Date"
              ]
            }
          },
          {
            "id": "pqr",
            "from_name": "label",
            "to_name": "text",
            "type": "labels",
            "value": {
              "start": 47,
              "end": 56,
              "score": 0.40,
              "text": "Jerusalem",
              "labels": [
                "Location"
              ]
            }
          },
          {
            "id": "stu",
            "from_name": "label",
            "to_name": "text",
            "type": "labels",
            "value": {
              "start": 65,
              "end": 76,
              "score": 0.32,
              "text": "the weekend",
              "labels": [
                "Time"
              ]
            }
          },
          {
            "id": "vwx",
            "from_name": "label",
            "to_name": "text",
            "type": "labels",
            "value": {
              "start": 85,
              "end": 91,
              "score": 0.22,
              "text": "Easter",
              "labels": [
                "Location"
              ]
            }
          },
          {
            "id": "yz1",
            "from_name": "label",
            "to_name": "text",
            "type": "labels",
            "value": {
              "start": 96,
              "end": 104,
              "score": 0.96,
              "text": "Passover",
              "labels": [
                "Date"
              ]
            }
          }
        ]
      }
    ]
  },
  {
    "data": {
      "text": " Each journal was several inches thick and bound in leather. On one page are drawn portraits of Sunny in a flowery, Easter dress and sun hat. On another page are hundreds of sketches of leaves that Niyati saw in her yard."
    },
    "predictions": [
      {
        "model_version": "one",
        "result": [
          {
            "id": "ab2",
            "from_name": "label",
            "to_name": "text",
            "type": "labels",
            "value": {
              "start": 18,
              "end": 32,
              "text": "several inches",
              "labels": [
                "Product"
              ]
            }
          },
          {
            "id": "cd3",
            "from_name": "label",
            "to_name": "text",
            "type": "labels",
            "value": {
              "start": 64,
              "end": 67,
              "text": "one",
              "labels": [
                "Percent"
              ]
            }
          },
          {
            "id": "ef4",
            "from_name": "label",
            "to_name": "text",
            "type": "labels",
            "value": {
              "start": 96,
              "end": 101,
              "text": "Sunny",
              "labels": [
                "Person"
              ]
            }
          },
          {
            "id": "gh5",
            "from_name": "label",
            "to_name": "text",
            "type": "labels",
            "value": {
              "start": 162,
              "end": 170,
              "text": "hundreds",
              "labels": [
                "Percent"
              ]
            }
          },
          {
            "id": "ij6",
            "from_name": "label",
            "to_name": "text",
            "type": "labels",
            "value": {
              "start": 198,
              "end": 204,
              "text": "Niyati",
              "labels": [
                "Person"
              ]
            }
          }
        ]
      },
      {
        "model_version": "two",
        "result": [
          {
            "id": "kl7",
            "from_name": "label",
            "to_name": "text",
            "type": "labels",
            "value": {
              "start": 18,
              "end": 32,
              "text": "several inches",
              "labels": [
                "Fact"
              ]
            }
          },
          {
            "id": "mn8",
            "from_name": "label",
            "to_name": "text",
            "type": "labels",
            "value": {
              "start": 64,
              "end": 67,
              "text": "one",
              "labels": [
                "Percent"
              ]
            }
          },
          {
            "id": "op9",
            "from_name": "label",
            "to_name": "text",
            "type": "labels",
            "value": {
              "start": 96,
              "end": 101,
              "text": "Sunny",
              "labels": [
                "Time"
              ]
            }
          },
          {
            "id": "qr10",
            "from_name": "label",
            "to_name": "text",
            "type": "labels",
            "value": {
              "start": 116,
              "end": 122,
              "text": "Easter",
              "labels": [
                "Location"
              ]
            }
          },
          {
            "id": "st11",
            "from_name": "label",
            "to_name": "text",
            "type": "labels",
            "value": {
              "start": 162,
              "end": 170,
              "text": "hundreds",
              "labels": [
                "Money"
              ]
            }
          },
          {
            "id": "uv12",
            "from_name": "label",
            "to_name": "text",
            "type": "labels",
            "value": {
              "start": 198,
              "end": 204,
              "text": "Niyati",
              "labels": [
                "Person"
              ]
            }
          }
        ]
      }
    ]
  }
]
{% endcodeblock %}
{% enddetails %}

Import pre-annotated tasks into Label Studio [using the UI](tasks.html#Import-data-from-the-Label-Studio-UI) or [using the API](/api#operation/projects_import_create). In the Label Studio UI, the imported prediction for the first task looks like the following:

<center><img src="../images/predictions_loaded_text.png" alt="screenshot of the Label Studio UI showing the text with highlighted text labels and prediction scores visible." style="max-width: 100%;"></center>

You can sort the prediction scores for each labeled region using the **Regions** pane options. 


## Import brush segmentation pre-annotations in RLE format

If you want to import pre-annotations for brush mask image segmentation using the [BrushLabels tag](/tags/brushlabels.html), you must convert the masks to RLE format first. The [Label Studio Converter](https://github.com/HumanSignal/label-studio-converter) package has some helper functions for this. See the following for common conversion cases and guidance.

Install Label Studio Converter:
```
pip install -U label-studio-converter
```

Import brush tools:
```python
from label_studio_converter import brush
```

- Convert masks to RLE. This expects an `np.array` with `shape=[image_width, image_height]` and `dtype=np.uint8`. Add the following to your python code to perform the conversion:
    ```python
    mask = (np.random.random([10, 20]) * 255).astype(np.uint8)  # just a random 2D mask
    mask = (mask > 128).astype(np.uint8) * 255  # better to threshold, it reduces output annotation size
    rle = brush.mask2rle(mask)  # mask image in RLE format 
    ```
  
- To convert OpenCV contours, use 
[`brush.contour2rle(contours, contour_id, img_width, img_height)`](https://github.com/HumanSignal/label-studio-converter/blob/master/label_studio_converter/brush.py#L310).

- To convert an image from path (jpg, png. bmp), use 
[`brush.image2rle(path)`](https://github.com/HumanSignal/label-studio-converter/blob/master/label_studio_converter/brush.py#L343).

- To prepare the pre-annotation, use 
[`brush.image2annotation(path, label_name, from_name, to_name, ground_truth=False, model_version=None, score=None)`](https://github.com/HumanSignal/label-studio-converter/blob/master/label_studio_converter/brush.py#L361)

For more assistance, review this [example code creating a Label Studio task with pre-annotations](https://github.com/HumanSignal/label-studio-converter/blob/master/tests/test_brush.py#L11) for brush labels.

## Import OCR pre-annotations 

Import pre-annotations for optical character recognition (OCR), such as output from [tesseract like in this example blog post](https://labelstud.io/blog/improve-ocr-quality-for-receipt-processing-with-tesseract-and-label-studio). 

In this example, import pre-annotations for OCR tasks using the [OCR template](/templates/optical_character_recognition.html):

```xml
<View>
  <Image name="image" value="$ocr"/>
  <Labels name="label" toName="image">
    <Label value="Text" background="green"/>
    <Label value="Handwriting" background="blue"/>
  </Labels>
  <Rectangle name="bbox" toName="image" strokeWidth="3"/>
  <Polygon name="poly" toName="image" strokeWidth="3"/>
  <TextArea name="transcription" toName="image" editable="true" perRegion="true" required="true" maxSubmissions="1" rows="5" placeholder="Recognized Text" displayMode="region-list"/>
</View>
```

This example JSON contains one task with three results dictionaries, one for each type of tag in the labeling configuration: Rectangle, Labels, and TextArea: 

{% details <b>Click to expand the JSON example</b> %}
Save this example JSON as a file to import it into Label Studio, for example, `example_prediction_task.json`.

{% codeblock lang:json %}
{
   "data": {
      "ocr": "/data/upload/receipt_00523.png"
   },
   "predictions": [
      {
         "model_version": "best_ocr_model_1_final",
         "result": [
            {
               "original_width": 864,
               "original_height": 1296,
               "image_rotation": 0,
               "value": {
                  "x": 48.9333,
                  "y": 61.3336,
                  "width": 9.73333,
                  "height": 2.8446,
                  "rotation": 0
               },
               "id": "bb1",
               "from_name": "bbox",
               "to_name": "image",
               "type": "rectangle"
            },
            {
               "original_width": 864,
               "original_height": 1296,
               "image_rotation": 0,
               "value": {
                  "x": 48.9333,
                  "y": 61.3336,
                  "width": 9.7333,
                  "height": 2.8446,
                  "rotation": 0,
                  "labels": [
                     "Text"
                  ]
               },
               "id": "bb1",
               "from_name": "label",
               "to_name": "image",
               "type": "labels"
            },
            {
               "original_width": 864,
               "original_height": 1296,
               "image_rotation": 0,
               "value": {
                  "x": 48.9333,
                  "y": 61.3336,
                  "width": 9.7333,
                  "height": 2.8446,
                  "rotation": 0,
                  "text": [
                     "TOTAL"
                  ]
               },
               "id": "bb1",
               "from_name": "transcription",
               "to_name": "image",
               "type": "textarea"
            }
         ],
         "score": 0.89
      }
   ]
}
{% endcodeblock %}
{% enddetails %}

This example JSON also includes a prediction score for the task. The IDs for each rectangle result match the label assigned to the region and the text area transcription for the region. 

Import pre-annotated tasks into Label Studio [using the UI](tasks.html#Import-data-from-the-Label-Studio-UI) or [using the API](/api#operation/projects_import_create).

!!! note 
    The image data in this example task references an uploaded file, identified by the source_filename assigned by Label Studio after uploading the image. The best way to reference image data is using presigned URLs for images stored in cloud storage, or absolute paths to image data stored in local storage and added to Label Studio by [syncing storage](storage.html). 


### Troubleshooting pre-annotations

<div class="opensource-only">

See [Troubleshooting pre-annotations](troubleshooting#Pre-annotations).

</div>

<div class="enterprise-only">

See [Troubleshooting ML Backends & Predictions](https://support.humansignal.com/hc/en-us/sections/23627938255117-ML-Backend-Predictions) in the HumanSignal support center.

</div>

---

### Source Guide Project Components (`docs/source/guide/project_components.md`)

---
title: Project components
short: Project components
# type: guide  # this page is not ready, so remove it from everywhere
# tier: all
order: 118
meta_title: Project components
meta_description: "Label Studio Documentation for Project components."
section: "Project & Team Management"
---

This page describes the general project components in the Label Studio UI.

## Labeling interface

The Label Studio interface allows you to label and annotate your data. You can start labeling and annotating your data only after you setup your project, labeling interface, and import your data. For more information, see the [Label and annotate data](labeling.html#Start-labeling) page.

## Annotation

Annotation is the process of labeling data on images. The image may contain humans, vehicles, any objects to make it recognizable for machines. Annotations are of various types, and it can be used to educate machines about the presence of various objects in the world. For more information, see the [Label and annotate data](labeling.html#Start-labeling) page.

<div class="enterprise-only">

## Review

After multiple labelers have annotated tasks, review their output to validate the quality of the results. For more information, see the [Review annotations in Label Studio](quality.html) page.

## Members

In the Label Studio Enterprise version, you can add members to a specific workspace or add members to a specific project within a workspace. For more information, see [Add members to a project](setup_project.html#Add-members-to-a-project) page.

</div>

## Machine Learning

Use the Label Studio ML backend to integrate Label Studio with machine learning models. For more information, see [Integrate Label Studio into your machine learning pipeline](ml.html) page.

## Cloud Storage

You can add source storage connections to sync data from an external source to a Label Studio project, and add target storage connections to sync annotations from Label Studio to external storage. For more information, see [Sync data from external storage](storage.html) page.

## Webhook

Webhooks in Label Studio allows you to set up integrations that subscribe to certain events which occur inside Label Studio. When an event is triggered, Label Studio sends an HTTP POST request to the configured webhook URL. For more information, see [Set up webhooks in Label Studio](webhooks.html) page.

---

### Source Guide Project Settings (`docs/source/guide/project_settings.md`)

---
title: Project settings
short: Project settings
tier: opensource
type: guide
order: 119
order_enterprise: 0
meta_title: Project settings
meta_description: Brief descriptions of all the options available when configuring the project settings
section: "Create & Manage Projects"
parent: "manage_projects_lso"
date: 2024-02-06 22:28:27
---

!!! error Enterprise
    There are many more possible controls and configurations available for Label Studio Enterprise users. For more information on those options, see [Project settings in Label Studio Enterprise](https://docs.humansignal.com/guide/project_settings_lse). 

## General

Use these settings to specify some basic information about the project. 

| Field          | Description    |
| ------------- | ------------ |
| **Project Name** | Enter a name for the project. |
| **Description**       | Enter a description for the project. |
| **Color**      | You can select a color for the project. The project is highlighted with this color when viewing the Projects page. |
| **Task Sampling**     | <ul><li><strong>Sequential sampling</strong>–Tasks are shown to annotators in the same order that they appear on the Data Manager</li><li><strong>Random sampling</strong>–Tasks are shown in random order.</li></ul> |


## Labeling interface

The labeling interface is the central configuration point for projects. This determines how tasks are presented to annotators. 

For information on setting up the labeling interface, see [Labeling configuration](setup). 

## Annotation

<dl>

<dt>Labeling Instructions</dt>

<dd>

Specify instructions to show the users as they annotate task. This field accepts HTML formatting. 

Enable **Show before labeling** to display a pop-up message to users when they enter the label stream. 

If disabled, users will need to click the **Show instructions** action at the bottom of the labeling interface. 

</dd>

<dt>Live Predictions</dt>

<dd>

If you have an ML backend or model connected, you can use this setting to determine whether tasks should be pre-labeled using predictions from the model. For more information, see [Integrate Label Studio into your machine learning pipeline](ml). 

Use the drop-down menu to select the predictions source. For example, you can select a [connected model](#Model) or a set of [predictions](#Predictions). 

</dd>

</dl>

## Model

Click **Connect Model** to connect a machine learning (ML) backend to your project. For more information on connecting a model, see [Machine learning integration](ml).

You have the following configuration options:

| Field          | Description    |
| ------------- | ------------ |
| **Start model training on annotation submission**         | Triggers the connected ML backend to start the training process each time an annotation is created or updated. <br /><br />This is part of an [active learning loop](https://docs.humansignal.com/guide/active_learning) where the model can be continuously improved as new annotations are added to the dataset. When this setting is enabled, the ML backend's `fit()` method is called, allowing the model to learn from the most recent annotations and potentially improve its predictions for subsequent tasks.   |
| [**Interactive preannotations**](ml#interactive-pre-annotations)         | (Available when creating or editing a model connection)<br /><br />Enable this option to allow the model to assist with the labeling process by providing real-time predictions or suggestions as annotators work on tasks.  <br /><br />In other words, as you interact with data (for example, by drawing a region on an image, highlighting text, or asking an LLM a question), the ML backend receives this input and returns predictions based on it.   |


And the following actions are available from the overflow menu next to a connected model:

| Action          | Description    |
| ------------- | ------------ |
| **Start Training**         | Manually initiate training. Use this action if you want to control when the model training occurs, such as after a specific number of annotations have been collected or at certain intervals.  |
| **Send Test Request**         | (Available from the overflow menu next to the connected model)<br /><br />Use this for troubleshooting and sending a test resquest to the connected model.   |
| **Edit**         | Edit the model name, URL, and parameters. For more information, see [Connect a model to Label Studio](ml#Connect-a-model-to-Label-Studio). |
| **Delete**         | Remove the connection to the model. |

## Predictions

From here you can view predictions that have been imported or generated when executing the **Retrieve Predictions** action from the Data Manager. For more information, see [Import pre-annotated data into Label Studio](predictions). 

## Cloud storage

This is where you connect Label Studio to a cloud storage provider:

* **Source Cloud Storage**--This is where the source data for your project is saved. When you sync your source storage, Label Studio retrieves data to be annotated. 
* **Target Cloud Storage**--This is where your annotations are saved. When you sync your target storage, annotations are sent from Label Studio to the target storage location. 

For more information, see [Sync data from external storage](storage). 

## Webhooks

You can use webhooks to integration third-party applications. For more information, see [Set up webhooks in Label Studio](webhooks) and our [integrations directory](https://labelstud.io/integrations/).

## Danger Zone

From here, you can access actions that result in data loss, and should be used with caution. 

* **Drop All Tabs**

    If the Data Manager is not loading, dropping all Data Manager tabs can help.
* **Delete Project**

    Deleting a project permanently removes all tasks, annotations, and project data from Label Studio.

---

### Source Guide Project Settings Lse (`docs/source/guide/project_settings_lse.md`)

---
title: Project settings
short: Project settings
tier: enterprise
type: guide
order: 0
order_enterprise: 119
meta_title: Project settings
meta_description: Brief descriptions of all the options available when configuring the project settings
section: "Create & Manage Projects"
parent: "manage_projects"
parent_enterprise: "manage_projects"
date: 2024-02-06 22:28:14
---

!!! error Enterprise
    Many settings are only available in Label Studio Enterprise Edition. If you're using Label Studio Community Edition, see [Label Studio Features](label_studio_compare) to learn more.

!!! error Starter Cloud
    If you see an Enterprise badge: <span class="badge"></span>
    
    This means the setting is not available in Label Studio Starter Cloud. 


## General

Use these settings to specify some basic information about the project. 

| Field          | Description    |
| ------------- | ------------ |
| **Workspace**         | Select a [workspace](workspaces) for the project. |
| **Project Name** | Enter a name for the project. |
| **Description**       | Enter a description for the project. |
| **Color**      | You can select a color for the project. The project is highlighted with this color when viewing the Projects page. |
| **Proxy Credentials**     | Enter proxy credentials. These might be necessary if your task data is protected with basic HTTP access authentication.<br><br> For example, if your Label Studio instance needs to access the internet through a corporate proxy server. |

## Labeling interface

The labeling interface is the central configuration point for projects. This determines how tasks are presented to annotators. 

For information on setting up the labeling interface, see [Labeling configuration](setup). 

## Members

Use this section to add and remove project members. You can only configure project members for public projects (meaning the project is not in your Personal Sandbox workspace). 

##### Add members

Click **Add Members** to open a window where you can search for and add members to the project. Depending on your organization's permission settings, you may also be able to invite new users to an organization and automatically add them to the project. 

!!! info Tip
    If you are in the Admin or Owner role, you can bulk assign users to workspaces and projects from the **Organization > Members** page. 

##### User roles and membership

| Role | Membership |
| ------------- | ------------ |
| **Admins and Owners** | Cannot be added or removed from a public project. These users have access to all projects, regardless of state, unless the project is in a Personal Sandbox workspace. |
| **Managers** | Must be added to a project or parent workspace to have access, but once added will have access to the project regardless of state. |
| **Annotators and Reviewers** | Must be added to a project or parent workspace to have access, but cannot access projects until they are published. |

##### Inherited members

Managers, Reviewers, and Annotators who are added as members at the [workspace level](workspaces#Add-or-remove-workspace-members) are automatically granted membership to any projects within the workspace. 

Admins and Owners have inherited membership because they have access to all public projects within an organizations. 

##### Annotator roles 

The annotator role is the most constrained role in Label Studio, and by default annotators can only access the labeling stream where they see tasks that are ready for labeling. 

* If you have [Automatic distribution](#distribute-tasks) enabled, users with the Annotator role are automatically assigned tasks when they are added as members. Similarly, by default, project members with the Reviewer role are able to begin reviewing annotations once the tasks are labeled. 
* If you have [Manual distribution](#distribute-tasks) enabled, you need to add users with the Annotator role as project members before you can assign them to tasks. And if you have [**Review only manually assigned tasks**](#reviewing-options) enabled, the users with the Reviewer role must also be project members before they can be assigned to tasks. 

##### Project-level roles

For users in the Annotator or Reviewer role (set at the organization level), you can change their role from project to project as needed. 

For example, if Heidi Opossum is granted the Annotator role at the organization level, but you would like her to work as a Reviewer in "Project 1", you can set her project-level role to Reviewer.    

To assign a project-level role, first add the person to your project. Once added, you can use the drop-down menu to change their role:




!!! note
    This is only available for users who have the Annotator or Reviewer role applied at the organization level. Users with Manager, Administrator, and Owner role cannot have their permissions downgraded to Annotator or Reviewer on a per-project basis. 

## Annotation

Use these settings to configure what options annotators will see and how their labeling tasks are assigned. 

<dl>

<dt>Annotation Instructions</dt>

<dd>

Specify instructions to show the annotators. 

This field accepts HTML formatting, including iframes (click **Preview** to check your formatting).

!!! note
    If you are using HTML formatting and want to include CSS styles, use a `<Style>` block as inline styles will be removed when saving. 

Enable **Show before labeling** to display a pop-up message to annotators when they enter the label stream. If disabled, users will need to click the **Show instructions** action at the bottom of the labeling interface. 

The instructions do not pop-up when opening tasks individually from the Data Manager (Quick View). 

</dd>

<dt id="distribute-tasks">Task Assignment</dt>

<dd>

Select how you want to distribute tasks to annotators for labeling. 

| Field          | Description    |
| ------------- | ------------ |
| **Automatic**         | Annotators are automatically assigned to tasks, and the option to manually assign annotators is disabled. Automatic assignments are distributed to all users with the Annotator role who are project [members](#Members) <br /><br />You can further define the automatic assignment workflow in the [**Quality** settings](#Quality). <br /><br />When Automatic distribution is used, you will also have additional configuration options throughout the project settings, such as task ordering, task reservation, and many settings under **Quality**. |
| **Manual** | [Manually assign](manage_data#Assign-annotators-to-tasks) tasks to annotators and manage their workload directly. Annotators will be able to start on tasks once you've assigned them. |

</dd>

<dt id="task-ordering">Task Ordering Method</dt>

<dd>

*Only available when using Automatic task assignment.* 

Select the order in which tasks are presented to annotators.  

| Field          | Description    |
| ------------- | ------------ |
| **By Task ID** | Tasks are shown to annotators in ascending order by ID. |
| **Random** | Tasks are shown in random order.  |
| **Uncertainty**         | This option is for when you are using a machine learning backend and want to employ [active learning](active_learning). Active learning mode continuously trains and reviews predictions from a connected machine learning model, allowing the model to improve iteratively as new annotations are created.<br /><br />When Uncertainty Sampling is enabled, Label Studio strategically selects tasks with the least confident, or most uncertain, prediction scores from your model. The goal is to minimize the amount of data that needs to be labeled while maximizing the performance of the model. |


</dd>

<dt id="lock-tasks">Task Reservation</dt>

<dd>

*Only available when using Automatic task assignment.* 

Control how long tasks are reserved for annotators. 

Task reservation ensures that the tasks an annotator starts working on are temporarily reserved by them, preventing other annotators from accessing the same tasks until the reservation expires. This helps manage task allocation and keeps project progress efficient. 

!!! note

    Task reservation takes overlap ([**Quality > Annotations per task**](#overlap)) into consideration. For example, if your overlap is `2`, then two annotators can reserve a task simultaneously. 

    When [**Task Assignment**](#distribute-tasks) is set to **Manual**, the Task Reservation setting is hidden because it does not apply. 

A task reservation is put in place as soon as an annotator opens the task. The reservation remains until one of the following happens (whichever happens first):

* The task reservation time expires. 
* The annotator submits the task. 
* The annotator skips the task, and the project **Skip Queue** setting is either **Requeue skipped tasks to others** or **Ignore skipped**.

**Recommended reservation time** 

By default, the task reservation time is 1440 minutes (1 day). This is the maximum time allowed for task reservations.

When setting a reservation time, you should aim to allow a little above the max amount of time it should take to complete a task. 

* **Notes about allowing too much time**

    If you allow too much time for task reservation, you could risk some users becoming blocked. 

    For example, say you have multiple annotators working on a project. Your minimum annotation overlap is set to `2`. 

    If some of your annotators move through their labeling queue looking for the easiest tasks to complete, they could inadvertently leave a large number of tasks locked. Depending on the size of the project and how many annotators you have working, this could result in some annotators unable to continue their work until the task reservation time expires. 

* **Notes about allowing too little time**

    If you set the reservation time too low, you may find that you have many tasks that exceed your minimum overlap setting. 

    For example, say you have multiple annotators working on a project. Your minimum annotation overlap is set to `2`. 
    
    Two annotators begin working on a task and it takes them both 15 minutes to complete, but your reservation time is 10 minutes. This means that after 10 minutes, another annotator can also begin working on that task - resulting in 3 annotations on the task rather than 2 (your minimum annotator overlap).

    You can help avoid this by selecting **Enforce strict overlap** under [**Quality > Overlap of Annotations**](#overlap).

</dd>

<dt id="annotating-options">Annotation Options</dt>

<dd>

Configure additional settings for annotators. 

| Field          | Description    |
| ------------- | ------------ |
| **Allow empty annotations** | This determines whether annotators can submit a task without making any annotations on it. If enabled, annotators can submit a task even if they haven't added any labels or regions, resulting in an empty annotation. |
| **Show Data Manager to annotators** | When disabled, annotators can only enter the label stream. When enabled, annotators can access the Data Manager. The tasks that are available to them depend on the how task are assigned: <ul><li>Automatic task assignment: Annotators can only see tasks that they have already completed or have created a draft for.</li><li>Manual task assignment: Annotators can only see the tasks that they have been assigned.</li></ul>Note that some information is still hidden from annotators and they can only view a subset of the Data Manager columns. For example, they cannot see columns such as Annotators, Agreement, Reviewers, and more. |
| **Show only columns used in labeling configuration to Annotators** | (Only available if the previous setting is enabled)<br><br /> Hide unused Data Manager columns from Annotators. <br><br />Unused Data Manager columns are columns that contain data that is not being used in the labeling configuration. <br><br />For example, you may include meta or system data that you want to view as part of a project, but you don't necessarily want to expose that data to Annotators. |


</dd>

<dt id="skip">Task Skipping</dt>

<dd>

Configure settings related to the **Skip** action in the labeling stream. 

| Field          | Description    |
| ------------- | ------------ |
| **Allow skipping tasks**         | Use this to show or hide the **Skip** action for annotators. |
| **Require comment to skip** | When enabled, annotators are required to leave a comment when skipping a task. |

!!! info Tip
    You can configure individual tasks to be unskippable in the JSON source for the task. For more information, see [Individual unskippable tasks](skip#Individual-unskippable-tasks)

</dd>


<dt id="skip-queue">Skip Queue</dt>

<dd>

Select how you want to handle skipped tasks. To disallow skipping tasks, you can hide the **Skip** action under the **Task Skipping** section above.

<table>
<thead>
    <tr>
      <th>Field</th>
      <th>Description</th>
    </tr>
</thead>
<tr>
<td>

**Requeue skipped tasks back to the annotator**
</td>
<td>

If an annotator skips a task, the task is moved to the bottom of their queue. They see the task again as they reach the end of their queue. 

If the annotator exits the label stream without labeling the skipped task, and then later re-enters the label stream, whether they see the task again depends on how task assignments are set up. 

* Automatic task assignment: Whether they see the task again depends on if other annotators have since completed the task. If the task is still incomplete when the annotator re-enters the labeling stream, they can update label and re-submit the task. 
* Manual task assignment: The annotator will continue to see the skipped task until it is completed.  

Skipped tasks are not marked as completed, and affect the Overall Project Progress calculation visible from the project Dashboard. (Meaning that the progress for a project that has skipped tasks will be less than 100%.)  

</td>
</tr>
<tr>
<td>

**Requeue skipped tasks to others**
</td>
<td>

*Only applies when using Automatic task assignment.*

If an annotator skips a task, the task is removed from their queue and automatically assigned to a different annotator.

After skipping the task and completing their labeling queue, the annotator cannot return to the skipped task.  

If there are no other annotators assigned to the task, or if all annotators skip the task, then the task remains unfinished. 

Skipped tasks are not marked as completed, and affect the Overall Project Progress calculation visible from the project Dashboard. (Meaning that the progress for a project that has skipped tasks will be less than 100%.) 

Note that if you select this option before switching to Manual mode, this option stays selected and behaves the same as **Ignore skipped**.

</td>
</tr>
<tr>
<td>

**Ignore skipped**
</td>
<td>

How this setting works depends on your task assignment method. 

* Automatic task assignment: If an annotator skips a task, the task is removed from the annotator's queue. 

    If task overlap (as defined in [**Quality > Annotations per task**](#overlap)) is set to 1, then the skipped task is marked as Completed and is not seen again by an annotator. However, if the overlap is greater than 1, then the task is shown to other annotators until the minimum annotations are reached. 

* Manual task assignment: If the annotator skips a task, it is removed from their queue. But other annotators assigned to the task will still see it in their queue.  

For both assignment methods, **Ignore skipped** treats skipped tasks differently when it comes to calculating progress. 

Unlike the other skip queue options, in this case skipped tasks do not adversely affect the Overall Project Progress calculation visible from the project dashboard. (Meaning that the progress for a project that has skipped tasks can still be 100%, assuming all tasks are otherwise completed.)

</td>
</tr>
</table>

</dd>

<dt id="predictions">Task Pre-Labeling</dt>

<dd>

| Field          | Description    |
| ------------- | ------------ |
| **Use predictions to pre-label tasks** | If you have an ML backend or model connected, or if you're using [Prompts](prompts_overview) to generate predictions, you can use this setting to determine whether tasks should be pre-labeled using predictions. For more information, see [Integrate Label Studio into your machine learning pipeline](ml) and [Generate predictions from a prompt](prompts_predictions).  |
| **Model or predictions to use** | Use the drop-down menu to select the predictions source. For example, you can select a [connected model](#Model) or a set of [predictions](#Predictions). |
| **Reveal pre-annotations interactively** | When enabled, pre-annotation regions (such as bounding boxes or text spans) are not automatically displayed to the annotator. Instead, annotators can draw a selection rectangle to reveal pre-annotation regions within that area. This allows annotators to first review the image or text without being influenced by the model’s predictions. Pre-annotation regions must have the attribute `"hidden": true`. <br /><br />This feature is particularly useful when there are multiple low-confidence regions that you prefer not to display all at once to avoid clutter. |

</dd>

</dl>

## Review

Use these settings to configure what options reviewers will see. 

<dl>

<dt>Instructions</dt>

<dd>

Specify instructions to show the reviewers. This field accepts HTML formatting. 

Enable **Show before reviewing** to display a pop-up message to reviewers when they enter the label stream. If disabled, users will need to click the **Show instructions** action at the bottom of the labeling interface.  

</dd>

<dt id="reviewing-options">Reviewing Options</dt>

<dd>

Use this section to configure:

* How many annotations need to be accepted/rejected for the task to be considered reviewed (move from the **Needs Review** state to the **Done** state).

* How many annotations need to be submitted for a task to enter the **Needs Review** state.

* Whether a reviewer can only see tasks to which they've been assigned.

For more information about states, see [Project states](project_states).

!!! note
    Moving from **Needs Review** to **Done** influences the following:

    * **Review stream**: Tasks that are **Done** are not shown in the review stream (what reviewers see when they click **Review All Tasks**).
    * **Data Manager**: The value shown in the **Reviewed** column. 
    * **Export**: Which tasks are included when you want to only include reviewed tasks in your export snapshot.
    * **Dashboards**: Various  counts and metric related to reviews. 

<table>
<thead>
    <tr>
      <th>Field</th>
      <th>Description</th>
    </tr>
</thead>
<tr>
<td>

**Task is reviewed after at least one accepted annotation**
</td>
<td>

For eligible tasks (tasks that enter the **Needs Review** state) in which multiple annotators submitted annotations, the reviewer only needs to accept one annotation to consider the task done. 

</td>
</tr>
<tr>
<td>

**Task is reviewed after all annotations are reviewed**
</td>
<td>

For eligible tasks (tasks that enter the **Needs Review** state) in which multiple annotators submitted annotations, the reviewer needs to either accept or reject **all** annotations to consider the task done.

</td>
</tr>
<tr>
<td>

**Review only manually assigned tasks**
</td>
<td>

If enabled, a reviewer can only see tasks to which they've been assigned.

This also means that a task can progress from the **Annotating** state directly to the **Done** state if this is enabled and no reviewers are assigned. 

</td>
</tr>
<tr>
<td>

**Show only finished tasks in the review stream**
</td>
<td>

When enabled, a task is considered ready for review only when it has achieved its annotation requirement.   

If your project is using [auto distribution](#distribute-tasks), then this means the task has met the [**Annotations per task** threshold](#overlap). 

If your project is using [manual distribution](#distribute-tasks), then this means all assigned annotators have submitted an annotation. 

Note that in most cases, skipped tasks do not contribute towards meeting the minimum.  

</td>
</tr>
</table>

</dd>

<dt id="review-sampling">Review Sampling <span class="badge"></span></dt>

<dd>

Determine how many eligible tasks need to be reviewed. 

"Eligible" tasks are first defined by whether you have enabled **Show only finished tasks in the review stream**:

* If enabled, then eligible tasks only include those tasks that have been completed by all required annotators. 
* If not enabled, as soon as a task has at least one submitted annotation, it is eligible for review. 

Eligible tasks enter the **Needs Review** state, which means they are included in the review stream.  

You can use the review sampling settings to configure whether tasks can skip the **Needs Review** state and go straight to **Done**. 

!!! note
    The percentages set under **Review Sampling** are applied as probabilistic sampling rates, not exact quotas. 
    
    For example, setting **Basic Sampling** to 50% does not guarantee that exactly half of eligible tasks will be reviewed — the actual share will vary around 50%, with less variance the more tasks your project has.

<table>
<thead>
    <tr>
      <th style="width: 20%;">Field</th>
      <th>Description</th>
    </tr>
</thead>
<tr>
<td>

**Basic Sampling**
</td>
<td>

Configure the percentage of eligible tasks that enter the **Needs Review** state.<br/><br/>For example, if you set this to 80%, then reviewers will see 80% of eligible tasks in the review stream. The remaining 20% of tasks will go straight to the **Done** state and skip the review stream. 

</td>
</tr>
<tr>
<td>

**Agreement-based Sampling** 
</td>
<td>

Use [agreement scores](stats) to determine which and how many tasks enter the **Needs Review** state vs. moving straight to the **Done** state.  
<ul><li><b>Agreement threshold</b>: The agreement score threshold you will use to partition the the <b>High agreement</b>/<b>Low agreement</b> sampling rates.</li>
<li><b>Agreement source</b>: The source of the agreement score used for the agreement threshold. You can select the overall task agreement or the control tag-level agreement score. For more information, see <a href="stats#Overall-vs-per-control-tag-agreement">Overall vs. per-control-tag agreement</a>.</li>
<li><b>Low agreement</b>: The percentage of tasks with agreement scores that fall below the <b>Agreement threshold</b> that you want to enter the <b>Needs Review</b> state.</li>
<li><b>High agreement</b>: The percentage of tasks with agreement scores that fall above the <b>Agreement threshold</b> that you want to enter the <b>Needs Review</b> state.</li></ul>

</td>
</tr>
</table>

</dd>

<dt id="task-ordering">Task Ordering</dt>

<dd>

Select the order in which reviewers see tasks in the review stream.

<table>
<thead>
    <tr>
      <th style="width: 20%;">Field</th>
      <th>Description</th>
    </tr>
</thead>
<tr>
<td>

**By Task ID**
</td>
<td>

Tasks are ordered by their numeric ID (ascending). Annotation order within a task remains stable.

</td>
</tr>
<tr>
<td>

**Random** <br/>
<span class="badge"></span>
</td>
<td>

Tasks are shown in randomized task order while preserving the stable order of annotations within each task. 

</td>
</tr>
</table>

</dd>

<dt id="reject-options">Reject Options</dt>

<dd>

Select which reject options are available to reviewers, and which one is the default. 

Use the **Available** column to choose the options. At least one option must be available. Use the **Default** column to choose the option that is applied when a reviewer clicks **Reject** without opening the menu. 

<table>
<thead>
    <tr>
      <th>Field</th>
      <th>Description</th>
    </tr>
</thead>
<tr>
<td>

**Reject and do not require rework**
</td>
<td>

The annotation is rejected and is not sent back to the annotator for rework. 

This is the default for new projects.

</td>
</tr>
<tr>
<td>

**Reject and return to annotator**
</td>
<td>

The annotation is rejected and sent back to the original annotator for rework. The annotator sees the rejected annotation in their labeling stream and can update it. The updated annotation is then shown to reviewers again in the review stream. 

</td>
</tr>
<tr>
<td>

**Reject and pass to another annotator**
</td>
<td>

The annotation is rejected and the task is sent to a different annotator for a new annotation. 

The rejected annotation no longer counts toward the project's [overlap](#overlap), which opens one slot on the task for another annotator. The original annotator can no longer update the rejected annotation. The annotation, its rejection, and the time spent on it remain in the Data Manager and in dashboards.

This option is only available when [**Task Assignment**](#distribute-tasks) is set to **Automatic**.

</td>
</tr>
</table>

When only one option is available, reviewers see a single **Reject** button that applies it. When more than one option is available, reviewers can click **Reject** to apply the default option, or open the reject menu to choose another option:

* **No Rework** -- Reject without sending for rework.
* **Return to Annotator** -- Reject and send back to the original annotator for rework.
* **Pass to Another Annotator** -- Reject and send to a different annotator.



For example, a reviewer might return an annotation that is nearly correct and only needs a small change. For an annotation with numerous errors, they might pass the task to another annotator instead. 

The same options are available when reviewers reject annotations in bulk from the Data Manager. 

Changing the available options only affects future rejections. Annotations that were already rejected keep the outcome that the reviewer chose. For example, annotations that were returned to an annotator can still be updated by that annotator after you make **Reject and return to annotator** unavailable. 

Separately, you can enable **Reviewers must leave a comment on reject** to require reviewers to add a comment before they can reject an annotation from the review stream. This does not apply to bulk rejections from the Data Manager.

</dd>

<dt id="data-manager">Data Manager</dt>

<dd>

Configure what Data Manager features are available to reviewers.

<table>
<thead>
    <tr>
      <th>Field</th>
      <th>Description</th>
    </tr>
</thead>
<tr>
<td>

**Show the Data Manager to reviewers**
</td>
<td>

When disabled, reviewers can only enter the review stream. When enabled, reviewers can access the Data Manager, where they can select which tasks to review. Some information is still hidden from reviewers and they can only view a subset of the Data Manager columns.

</td>
</tr>
<tr>
<td>

**Show unused task data columns to reviewers in the Data Manager**
</td>
<td>

If reviewers can view the Data Manager, this setting will hide unused columns from them.

Unused Data Manager columns are columns that contain data that is not being used in the labeling configuration.

For example, you may include meta or system data that you want to view as part of a project, but you don’t necessarily want to expose that data to reviewers.

</td>
</tr>
<tr>
<td>

**Show agreement to reviewers in the Data Manager**
</td>
<td>

If reviewers can view the Data Manager, this setting controls whether they can access the **Agreement** column.

</td>
</tr>
</table>

</dd>


## Quality

Use these settings to determine task completeness and agreement metrics. 

<dl>

<dt id="overlap">Overlap of Annotations</dt>

<dd>

!!! note
    Overlap settings only apply when the project is using Automatic distribution mode. If you are using Manual distribution mode, all tasks must be manually assigned - meaning that you are also manually determining overlap.  

By default, each task only needs to be annotated by one annotator. If you want multiple annotators to annotate the same tasks, increase the **Annotations per task**.

<table>
<thead>
    <tr>
      <th>Field</th>
      <th>Description</th>
    </tr>
</thead>
<tr>
<td>

**Annotations per task**
</td>
<td>

The number of distinct annotations you want to allow per task. 

Note that in certain situations, this may be exceeded. For example, if there are long-standing drafts within a project or you have a very low [task reservation](#lock-tasks) time. 

Also note that only annotations created by distinct users count towards the overlap. For example, if the overlap is `2` and a user creates and submits two annotations on a single task (which can be done in Quick View), the overlap threshold will not be reached until another user submits an annotation. 

!!! note
    Setting annotations per task above 20 may impact loading performance in the Data Manager.

</td>
</tr>
<tr>
<td>

**Annotations per task coverage**
</td>
<td>

Only available if **Annotations per task** is ≥ 2. 

This setting controls the percentage of the project tasks for which the overlap is enforced. 

For example, if you want all tasks to be annotated by at least 2 annotators:

- Set the number of annotations to **2**
- Enforce the overlap for 100% of tasks.

If you want half of the tasks to be annotated by at least 3 people:

- Set the number of annotations to **3**
- Enforce the overlap for 50% of tasks.

</td>
</tr>
<tr>
<td>

**Show tasks with overlap first** 
</td>
<td>

If your overlap enforcement is less than 100% (meaning that only some tasks require multiple annotators), then the tasks that *do* require multiple annotations are shown first. <br /><br />If your overlap is 100%, then this setting has no effect.

Note that if enabled, this setting supersedes what you specified under [**Annotations > Task Ordering Method**](#task-ordering).

</td>
</tr>
<tr>
<td>

**Enforce strict overlap limit** 
</td>
<td>

This setting strictly enforces your **Annotations per task** limit.<br /><br />If you do not enable this setting, you may see tasks where the number of annotations exceed your **Annotations per task** limit. This typically happens when you set a low [task reservation time](#lock-tasks), meaning that task locks expire before annotators submit their tasks. This allows other annotators to access and then submit the task, potentially resulting in an excess of annotations.<br /><br />

When enabled, if an annotator tries to submit a task after the limit has been reached, they will receive an error message stating **Annotation Overlap Reached**. Their draft will be saved but they will be unable to submit. <br /><br />

Note that enforcement only applies when the user submitting the annotation is in the Annotator role. All other roles are exempt. 


</td>
</tr>
</table> 

</dd>

<dt id="annotation-limit">Tasks Per Annotator Limit <span class="badge"></span></dt>

<dd>

Set limits on how many tasks each individual user can annotate. This can be useful if you are concerned with preventing any potential bias that might arise from a small set of power users completing a majority of project tasks. 

When an annotator reaches their limit, they will see a notification telling them that they have been paused. This is a flexible pause: the annotator cannot begin any new annotation work, but they keep access to the project so that they can update annotations that reviewers have rejected. For more information, see [Full and flexible pauses](quality#Full-and-flexible-pauses). 

When **Limit tasks per annotator** is enabled, you will see the following options:

| Field          | Description    |
| ------------- | ------------ |
| **Limit by Number of Tasks**    | Set a specific number of tasks each annotator is able to complete before their progress is paused.   |
| **Limit by Percentage of Tasks** | Calculate the number of tasks each annotator is able to complete as a percentage of tasks within the project.  |

You can set one or both values. Annotators will be paused when they reach whichever limit is smaller.  

To unpause annotators:

* If you increase the limits, previously paused annotators will regain access. 
* If you are using a percentage-based limit and you add more tasks to a project, previously paused annotators will regain access. 

For more information about pausing annotators, including how to manually pause specific annotators, see [Pause an annotator](quality#Pause-an-annotator).

!!! note
    Pauses are enforced for users in Annotator and Reviewer roles. 
    
    Because this is a flexible pause, it only stops new annotation work. So, for example, if a Reviewer is also annotating tasks and they hit the annotation limit, they can continue reviewing other people's annotations. 

    Users in the Manager, Administrator, or Owner role are unaffected by the task limit.

</dd>

<dt id="annotator-eval">Annotator Evaluation<span class="badge"></span></dt>

<dd>

!!! note
    Annotator Evaluation settings are only available when the project is configured to [automatically assign tasks](#distribute-tasks). If you are using Manual distribution, this section will not appear in your project settings.
    
    If you switch a project from Automatic to Manual distribution, annotator evaluation is automatically disabled.

Evaluate annotators against [ground truths](ground_truths) or against their **acceptance score** from review outcomes. You can then automatically pause annotators who do not meet the threshold you set.

Use ground truth evaluation when the project has verified labels to score against. Use acceptance score when you want to evaluate annotators from review outcomes instead, including projects that do not have ground truth.

The **Members** dashboard shows each annotator's evaluation status. If you do not enable pausing, scores are still calculated and can be reviewed there. See [Members dashboard](dashboard_members).

!!! note 
    When you evaluate against ground truth, those tasks are not constrained by the [annotator overlap](#overlap). For example, if you set overlap to `2`, but you have 10 annotators, all 10 will still be able to add annotations to ground truth tasks. 

!!! info Tip
    You can specify that ground truth tasks should be unskippable by adding `"allow_skip": false` as part of the JSON task definition that you import to your project. For more information, see [Individual unskippable tasks](skip#Individual-unskippable-tasks)

<table>
<thead>
    <tr>
      <th>Field</th>
      <th>Description</th>
    </tr>
</thead>
<tr>
<td>

**Evaluation** 
</td>
<td>

Select how annotators should be evaluated:

* **Don't evaluate annotators**: Evaluation is turned off.
* **Evaluate against ground truth**: Require annotators to complete ground truth tasks. Overlap is ignored for these tasks.
* **Evaluate against acceptance score (accepted / reviewed)**: Score annotators by how often their submitted annotations are accepted during review. This includes annotations that were fixed and then accepted.

Acceptance score is calculated as (accepted + fix and accepted) / reviewed. The score uses the latest review outcome for each annotation. If an annotator updates a rejected annotation, that rejection still counts until a reviewer records a new verdict.

</td>
</tr>
<tr>
<td>

**Onboarding evaluation**
</td>
<td>

Only available when evaluating against ground truth.

When annotators enter the labeling stream, they are first presented with tasks that have a ground truth annotation. This ensures that annotators meet your evaluation standards before progressing through the remaining project tasks. 

Use the counter to determine how many ground truth tasks should be presented first before the annotator progresses through the remaining project tasks. 

Set this counter to zero if you want to skip onboarding and only use continuous evaluation. 
</td>
</tr>
<tr>
<td>

**Continuous evaluation**
</td>
<td>

Only available when evaluating against ground truth.

Annotators are presented with ground truth tasks in the order that is configured under [**Task Ordering Method**](#task-ordering). 

To have all ground truths presented as part of continuous evaluation, set the **Onboarding evaluation** counter to zero and set this number equal to the number of ground truth tasks in your project. 

You can also use a combination of both, so that annotators see a subset of ground truths immediately, and then are presented the remaining ground truths periodically as they progress through the project (depending on your task ordering method). 

</td>
</tr>
<tr>
<td>

**Number of tasks for evaluation** / **Minimum reviewed annotations before evaluation**
</td>
<td>

The minimum sample an annotator must complete before they can fail evaluation (and be paused, if pausing is enabled).

When evaluating against ground truth, this is the number of ground truth tasks they must complete. For example, if you set this to `10`, even if the annotator gets every single task wrong, they will not be paused until after they have completed 10 ground truth tasks.

When evaluating against acceptance score, this is the number of **reviewed** annotations they must have. Evaluation does not start until reviewers have recorded that many verdicts.

</td>
</tr>
<tr>
<td>

**Score required to pass evaluation** / **Minimum acceptance score**
</td>
<td>

The percent threshold an annotator must meet to pass evaluation.

When evaluating against ground truth, this is the agreement score threshold compared to ground truth annotations. How agreement is calculated depends on what you select in the [**Agreement** section](#task-agreement).

When evaluating against acceptance score, this is the minimum acceptance score. Annotators below this threshold fail evaluation.

</td>
</tr>
<tr>
<td>

**Pause annotator on failed evaluation** 
</td>
<td>

Determines whether annotators should be paused if they do not meet the required score. If they fail to meet the score, they are paused and cannot submit additional annotations.

If you do not enable pausing, evaluation still runs and the results can be reviewed in the [Members dashboard](dashboard_members).

</td>
</tr>
</table> 

You can see each annotator's evaluation status and who is paused from the **Members** page. 

When users are paused as part of the annotator evaluation workflow, you cannot manually unpause them from the Members pause toggle. They are unpaused automatically if their score recovers. You can also relax the evaluation settings for the project, for example by increasing the minimum sample or changing the score threshold.

If your project requeues rejected annotations, paused annotators can still receive and update their rejected work from the labeling stream and the Data Manager. When you evaluate against acceptance score, updating a rejected annotation does not lift the pause by itself; a reviewer has to record a new verdict that improves the score. For more information, see [Recover from a flexible pause](quality#Recover-from-a-flexible-pause).

For more information about pausing annotators, including how to manually pause specific annotators, see [Pause an annotator](quality#Pause-an-annotator).

!!! note
    Pauses are enforced for users in Annotator and Reviewer roles.  
    
    Because this is a flexible pause, it only stops new annotation work. So, for example, if a Reviewer is also annotating tasks and they fail evaluation, they can continue reviewing other people's annotations. 

    Users in the Manager, Administrator, or Owner role are unaffected by evaluation requirements. 

</dd>

<dt id="task-agreement">Agreement <span class="badge"></span></dt>

<dd>

When multiple annotators are labeling a task, the task agreement reflects how much agreement there is between annotators. 

For example, if 10 annotators review a task and only 2 select the same choice, then that task would have a low agreement score.  

You can customize how task agreement is calculated and how it should affect the project workflow. For more information, see [Task agreement](stats). 

!!! error Enterprise
    Label Studio Starter Cloud only supports the **Pairwise** methodology. Each control tag uses the [default built-in metric](agreement_metrics#Default-metric-reference) for agreement calculation.

<table>
<thead>
    <tr>
      <th>Field</th>
      <th>Description</th>
    </tr>
</thead>
<tr>
<td>

**Methodology**

</td>
<td>

Methodology to use for calculating task agreement. 

* **Consensus**: Consensus measures *"What percentage of annotators chose the most common answer?"*
* **Pairwise**: Pairwise measures *"What is the average agreement score across all pairs of annotators?"*

For more information, see [Task agreement - methodology](stats#Methodology).

</td>
</tr>
<tr>
<td>

**Built-in Metrics vs Custom**

</td>
<td>

Select whether you want to use the built-in metrics or custom metrics for agreement.

For more information, see [Built-in agreement metrics reference](agreement_metrics) and [Custom agreement metrics](custom_metric).

</td>
</tr>
<tr>
<td>

**Overall Agreement**

</td>
<td>

Configure how overall agreement is calculated by setting the weight for each control tag.

For more information, see [Configure weight for the overall agreement](stats#Configure-weight-for-the-overall-agreement).


</td>
</tr>
<tr>
<td>

**Agreement Columns**

</td>
<td>

Configure how agreement is calculated for each control tag.

For more information, see [Configure agreement for each control tag](stats#Configure-agreement-for-each-control-tag).

</td>
</tr>
</table>

</dd>

<dt id="low-agreement">Low Agreement Resolution <span class="badge"></span></dt>

<dd>

!!! note
    Low agreement resolution settings are only available when the project is configured to [automatically assign tasks](#distribute-tasks). If you are using Manual distribution, this section will not appear in your project settings.
    
    If you switch a project from Automatic to Manual distribution, low agreement resolution is automatically disabled.

Resolve tasks with low agreement scores by automatically assigning additional annotators to the task. 

<table>
<thead>
    <tr>
      <th>Field</th>
      <th>Description</th>
    </tr>
</thead>
<tr>
<td>

**Assign additional annotator**

</td>
<td>
Enable this option to automatically assign an additional annotator to any tasks that have a low agreement score. 

This will ensure that the task is not marked complete until 1) it meets the required [overlap](#overlap) and 2) a minimum agreement score is achieved (this is specified below).

Note that to see this setting, the project must be set up with [automatic task assignments](#distribute-tasks).

</td>
</tr>
<tr>
<td>

**Agreement threshold**

</td>
<td>

Enter the agreement score that a task must meet before it can be considered complete.

</td>
</tr>
<tr>
<td>

**Maximum additional annotators**

</td>
<td>

Enter a maximum number of annotators that can be automatically assigned to the task. 

Annotators are assigned one at a time until the agreement threshold is achieved. 

</td>
</tr>
</table>

!!! note
    When configuring **Maximum additional annotators**, be mindful of the number of annotators available in your project. If you have fewer annotators available than the sum of [**Annotations per task**](#overlap) + **Maximum additional annotators**, you might encounter a scenario in which a task with a low agreement score cannot be marked complete.

</dd>
</dl>

## Model

Click **Connect Model** to connect a machine learning (ML) backend to your project. For more information on connecting a model, see [Machine learning integration](ml).

You have the following configuration options:

| Field          | Description    |
| ------------- | ------------ |
| **Start model training on annotation submission**         | Triggers the connected ML backend to start the training process each time an annotation is created or updated. <br /><br />This is part of an [active learning loop](active_learning) where the model can be continuously improved as new annotations are added to the dataset. When this setting is enabled, the ML backend's `fit()` method is called, allowing the model to learn from the most recent annotations and potentially improve its predictions for subsequent tasks.   |
| [**Interactive preannotations**](ml#interactive-pre-annotations)         | (Available when creating or editing a model connection)<br /><br />Enable this option to allow the model to assist with the labeling process by providing real-time predictions or suggestions as annotators work on tasks.  <br /><br />In other words, as you interact with data (for example, by drawing a region on an image, highlighting text, or asking an LLM a question), the ML backend receives this input and returns predictions based on it.   |


And the following actions are available from the overflow menu next to a connected model:

| Action          | Description    |
| ------------- | ------------ |
| **Start Training**         | Manually initiate training. Use this action if you want to control when the model training occurs, such as after a specific number of annotations have been collected or at certain intervals.  |
| **Send Test Request**         | (Available from the overflow menu next to the connected model)<br /><br />Use this for troubleshooting and sending a test resquest to the connected model.   |
| **Edit**         | Edit the model name, URL, and parameters. For more information, see [Connect a model to Label Studio](ml#Connect-a-model-to-Label-Studio). |
| **Delete**         | Remove the connection to the model. |

## Predictions

From here you can view predictions that have been imported, generated with [Prompts](prompts_predictions), or generated when executing the **Batch Predictions** action from the Data Manager. For more information on using predictions, see [Import pre-annotated data into Label Studio](predictions). 

To remove predictions from the project, click the overflow menu next to the predictions set and select **Delete**.  

To determine which predictions are show to annotators, use the [**Annotation > Live Predictions** section](#Annotation). 

## Cloud storage

This is where you connect Label Studio to a cloud storage provider:

* **Source Cloud Storage**--This is where the source data for your project is saved. When you sync your source storage, Label Studio retrieves data to be annotated. 
* **Target Cloud Storage**--This is where your annotations are saved. When you sync your target storage, annotations are sent from Label Studio to the target storage location. 

For more information, see [Sync data from external storage](storage). 


## Webhooks

You can use webhooks to integration third-party applications. For more information, see [Set up webhooks in Label Studio](webhooks) and our [integrations directory](https://labelstud.io/integrations/).

## Danger Zone

From here, you can access actions that result in data loss, and should be used with caution. 

* **Reset Cache** 

    Reset the labeling cache. This can help in situations where you are seeing validation errors concerning certain labels -- but you know that those labels do not exist. 
* **Drop All Tabs**

    If the Data Manager is not loading, dropping all Data Manager tabs can help.
* **Delete Project**

    Deleting a project permanently removes all tasks, annotations, and project data from Label Studio.

---

### Source Guide Project States (`docs/source/guide/project_states.md`)

---
title: Project and task state management
short: Project and task states
type: guide
tier: enterprise
order: 105
order_enterprise: 105
section: "Create & Manage Projects"
---

Project and tasks move through a series of states as they progress from their initial created state to completion. 

!!! note
    Note that state change history tracking did not start until state management was implemented for your organization. For most Label Studio Cloud organizations, state management was implemented in February 2026. 

## Project states



Projects progress through the following states:

```mermaid
stateDiagram-v2
    state "Initial" as CREATED
    state "Setup In Progress" as SETUP_IN_PROGRESS
    state "Ready To Publish" as READY_TO_PUBLISH
    state "Published" as PUBLISHED
    state "Annotating" as ANNOTATION_IN_PROGRESS
    state "Needs Review" as NEEDS_REVIEW
    state "In Review" as REVIEW_IN_PROGRESS
    state "Done" as COMPLETED

    CREATED --> SETUP_IN_PROGRESS: Add config or tasks
    SETUP_IN_PROGRESS --> READY_TO_PUBLISH: Add config and tasks
    READY_TO_PUBLISH --> PUBLISHED: Publish project
    PUBLISHED --> ANNOTATION_IN_PROGRESS: First annotation started
    ANNOTATION_IN_PROGRESS --> NEEDS_REVIEW: All tasks annotated
    NEEDS_REVIEW --> REVIEW_IN_PROGRESS: First review started
    REVIEW_IN_PROGRESS --> COMPLETED: All reviews done
    ANNOTATION_IN_PROGRESS --> COMPLETED: No review required

    class CREATED neutral
    class SETUP_IN_PROGRESS cantaloupe
    class READY_TO_PUBLISH cantaloupe
    class PUBLISHED grape
    class ANNOTATION_IN_PROGRESS grape
    class NEEDS_REVIEW cantaloupe
    class REVIEW_IN_PROGRESS plum
    class COMPLETED kale
```

| State | Description |
|-----------|---------|
| **Initial** | You have created the project, but you have not created tasks or configured the labeling interface yet. |
| **Setup In Progress** | You have created the project and you have either created tasks or configured the labeling interface, but you have not done both. <br/><br/>If your project is in a Personal Sandbox, it cannot move beyond this state until you move it into a shared workspace.  |
| **Ready to Publish** | Your project has tasks and you have configured the labeling interface, but you have not published the project yet. |
| **Published** | Project is published, but does not have any annotations yet. |
| **Annotating** | At least one task has at least one submitted annotation, meaning annotation work has started. |
| **Needs Review** | All tasks have been annotated by the required number of users, but there are still tasks that need to be reviewed. <br/><br/>Whether your project enters this state depends on your project settings and task states. See [Additional notes](#Additional-notes) below. |
| **In Review** | All tasks have received at least one review, but reviews have not yet been completed for all tasks. <br/><br/>Whether your project enters this state depends on your project settings and task states. See [Additional notes](#Additional-notes) below. |
| **Done** | All tasks have been completed. |



## Task states



```mermaid
stateDiagram-v2
    state "Initial" as CREATED
    state "Annotating" as ANNOTATION_IN_PROGRESS
    state "Needs Review" as NEEDS_REVIEW
    state "In Review" as REVIEW_IN_PROGRESS
    state "Done" as COMPLETED

    CREATED --> ANNOTATION_IN_PROGRESS: First annotation created
    ANNOTATION_IN_PROGRESS --> NEEDS_REVIEW: Annotations complete -<br>review required
    ANNOTATION_IN_PROGRESS --> COMPLETED: Annotations complete -<br>review not required
    NEEDS_REVIEW --> REVIEW_IN_PROGRESS: First review started
    REVIEW_IN_PROGRESS --> COMPLETED: All reviews done
    REVIEW_IN_PROGRESS --> ANNOTATION_IN_PROGRESS: Annotation rejected<br>(returned or passed)

    class CREATED neutral
    class ANNOTATION_IN_PROGRESS grape
    class NEEDS_REVIEW cantaloupe
    class REVIEW_IN_PROGRESS plum
    class COMPLETED kale
```

| State | Description | 
|----------|------------|
| **Initial** | Task has been created but does not have any annotations yet. |
| **Annotating** | First annotation has been submitted, or a rejected annotation has been [returned to the annotator or passed to another annotator](project_settings_lse#reject-options). <br/><br/>Whether the task lingers in this state long enough for it to be reflected in the Data Manager depends on your project settings. See [Additional notes](#Additional-notes) below.|
| **Needs Review** | The required number of annotations have been completed and the task is ready for review. <br/><br/>Whether the task enters this state depends on your project settings. See [Additional notes](#Additional-notes) below. |
| **In Review** | First review has been created for any annotation within the task.  <br/><br/>Whether the task enters this state depends on your project settings. See [Additional notes](#Additional-notes) below.|
| **Done** | All required annotations and reviews have been completed. |

!!! info Tip
    You can click on the state in the Data Manager to view a history of state changes:

    <img src="/images/project/state-history.png" style="max-width: 450px" alt="Screenshot">


## Additional notes

### Project settings and states

Several project settings affect how and if a project/task enters certain states. 

##### Annotating state

The **Annotating** state for tasks is influenced by [**Quality > Overlap of Annotations**](project_settings_lse#overlap). 

If your overlap is `1`, the task immediately progresses to the next state as soon as the first annotation is submitted. 

If your overlap is greater than `1`, the task will sit in the **Annotating** state until you've reached the required number of annotations. 


##### Needs Review state

There are several settings that can cause tasks (and by extension, the project) to skip the **Needs Review** state:

* If you have [**Review > Reviewing Options > Review only manually assigned tasks**](project_settings_lse#reviewing-options) enabled, but have not assigned any reviewers. 
* If you have your [**Review Sampling**](project_settings_lse#review-sampling) settings configured in a way that allows some or all tasks to bypass the **Needs Review** state. 



##### In Review state

The **In Review** state for tasks (and by extension, the project) only happens when:

* Tasks have multiple annotations (meaning an overlap greater than 1). 
* The [**Review > Reviewing Options > Task is reviewed when all annotations are reviewed**](project_settings_lse#reviewing-options) setting is enabled. 

By default, the overall task is considered reviewed if any one of its annotations have been been accepted/rejected. In these situations, the task will move from **Needs Review** to **Done** as soon as one review is submitted. 

If you enable **Task is reviewed when all annotations are reviewed**, the task moves from **Needs Review** to **In Review** after the first review has been submitted, and then **Done** after the final review is submitted. 

### Task state rollup into project state

Once your project reaches the **Published** state, its subsequent states are determined by the state of the tasks within the project. 

The project adopts the **least advanced** state of the tasks.

For example, if you have 10 tasks:
* 1 task is in the **Annotating** state
* 6 tasks are in the **In Review** state
* 3 tasks are in the **Done** state  
  
Your project will be in the **Annotating** state.

##### Initial state exception

The exception is the **Initial** state. For the purposes of the project state rollup, **Initial** tasks are calculated as **Annotating** tasks. 

For example, if you have 10 tasks:

* 5 tasks are in the **Initial** state
* 5 tasks are in the **In Review** state

Your project will be in the **Annotating** state, even though you do not have any tasks in the **Annotating** state.

### API values

If you are using the [SDK](https://api.labelstud.io/api-reference/introduction/getting-started), the API values for each state are as follows. 

Projects:


| Project state | API value | 
|-----------|----------|
| **Initial** | `CREATED` | 
| **Setup In Progress** | `SETUP_IN_PROGRESS` | 
| **Ready to Publish** | `READY_TO_PUBLISH` | 
| **Published** | `PUBLISHED` | 
| **Annotating** | `ANNOTATION_IN_PROGRESS` | 
| **Needs Review** | `NEEDS_REVIEW` | 
| **In Review** | `REVIEW_IN_PROGRESS` | 
| **Done** | `COMPLETED` | 

Tasks:

| Task state | API value | 
|----------|------------|
| **Initial** | `CREATED` | 
| **Annotating** | `ANNOTATION_IN_PROGRESS` | 
| **Needs Review** | `NEEDS_REVIEW` | 
| **In Review** | `REVIEW_IN_PROGRESS` | 
| **Done** | `COMPLETED` |

---

### Source Guide Prompts Create (`docs/source/guide/prompts_create.md`)

---
title: Create a Prompt
short: Create a Prompt
tier: enterprise
type: guide
order: 0
order_enterprise: 228
meta_title: Create a Prompt
meta_description: How to create a Prompt
section: Prompts
date: 2024-06-11 16:53:16
---


## Prerequisites

* An [API key](prompts_keys) for your LLM. 
* A project that meets the [criteria noted below](#Create-a-Prompt). 

## Create a Prompt

From the Prompts page, click **Create Prompt** in the upper right and then complete the following fields:

<div class="noheader rowheader">

| | |
| --- | --- |
| Name | Enter a name for the Prompt. |
| Description | Enter a description for the Prompt.  |
| Target Project| Select the project you want to use. If you don't have any eligible projects, you will see an error message. <br>See the note below. <br><br>When you select a project, additional information about the labeling config appears. This includes the classes that will be used when applying the prompt.  |

</div>

!!! note Eligible projects
    Target projects must meet the following criteria:
    * You must have access to the project. If you are in the Manager role, you need to be added to the project to have access. 
    * The project cannot be located in your Personal Sandbox workspace. 
    * While projects connected to an ML backend will still appear in the list of eligible projects, we do not recommend using Prompts with an ML backend as this can interfere with how accuracy and score are calculated when evaluating the prompt.

---

### Source Guide Prompts Draft (`docs/source/guide/prompts_draft.md`)

---
title: Draft and run prompts
short: Draft and run prompts
tier: enterprise
type: guide
order: 0
order_enterprise: 231
meta_title: Draft your Prompt
meta_description: Create and evaluate an LLM prompt
section: Prompts
date: 2024-06-12 14:09:09
---

With your [Prompt created](prompts_create), you can begin drafting your prompt content to run against baseline tasks.

## Draft a prompt and generate predictions


1. Select your base model. 

    The models that appear depend on the [API keys](prompts_keys) that you have configured for your organization. 
    
2. In the **Prompt** field, enter your prompt. Keep in mind the following:
    * You must include the text variables. These appear directly above the prompt field. (In the demo below, this is the `review` variable.) Click the text variable name to insert it into the prompt. 
    * Although not strictly required, you should provide definitions for each class to ensure prediction accuracy and to help [add context](#Add-context). 

!!! info "Tip"
    You can generate an initial draft by simply adding the text variables and then [clicking **Enhance Prompt**](#Enhance-prompt). 

3. Select your baseline:
   * **All Project Tasks** - Generate predictions for all tasks in the project. Depending on the size of your project, this might take some time to process. This does not generate an accuracy score for the prompt. 
   
        See the [Bootstrapping projects with prompts](prompts_overview#Bootstrapping-projects-with-Prompts) use case.
   * **Sample Tasks** - Generate predictions for the first 20 tasks in the project. This does not generate an accuracy score for the prompt. 
   
        See the [Bootstrapping projects with prompts](prompts_overview#Bootstrapping-projects-with-Prompts) use case.
   * **Ground Truths** - Generate predictions and a prompt accuracy score for all tasks with ground truth annotations. This option is only available if your project has ground truth annotations. 
   
        See the [Auto-labeling with Prompts](prompts_overview#Auto-labeling-with-Prompts) use case and the [Prompt evaluation and fine-tuning](prompts_overview#Prompt-evaluation-and-fine-tuning). 
4. If this is your first version of the prompt or you want to adjust the current version, click **Save**. 

    If you want to create a new version of the prompt so that you can compare evaluations between versions, click the drop-down menu next to **Save** and select **Save As**. 
5. Click **Evaluate** (if running against a ground truth baseline) or **Run**. 

!!! warning
    When you click **Evaluate** or **Run**, you will create predictions for each task in the baseline you selected and overwrite any previous predictions you generated with this prompt. 
    
    Evaluating your Prompts can result in multiple predictions on your tasks: if you have multiple Prompts for one Project, or if you click both **Evaluate**/**Run** and **Get Predictions for All Tasks from a Prompt**, you will see multiple predictions for tasks in the Data Manager. 

<br><br>
<video src="../images/prompts/prompts.mp4" controls="controls" style="max-width: 800px;" class="gif-border" />

## Evaluation results

When you evaluate a prompt, you will see the following metrics:

<table>
<thead>
    <tr>
      <th>Metric</th>
      <th>Tasks</th>
      <th>Config</th>
      <th>Description</th>
    </tr>
</thead>
<tr>
<td>

**Overall accuracy**
</td>
<td>

Ground Truths

</td>
<td>

All labeling configs

</td>
<td>

A measure of how many predictions are correct when measured against the ground truth. 

For example, if there are 10 ground truths and your Prompt's predictions match 7 of them, then the overall accuracy would be `0.70`. 

</td>
</tr>
<tr>
<td>

**Outputs**
</td>
<td>

All task types

</td>
<td>

All labeling configs

</td>
<td>

Number of tasks evaluated.

</td>
</tr>
<tr>
<td>

**F1 Score**
</td>
<td>

Ground Truths

</td>
<td>

Single `<Choices>`

Single `<Labels>`

</td>
<td>

The [F1 score](https://en.wikipedia.org/wiki/F-score) is a metric to assess a machine learning model's accuracy. This is measured in terms of **precision** and **recall** as follows:

`F1 = 2 * (precision * recall) / (precision + recall)`

* **Precision**: The proportion of correct positive predictions out of all positive predictions. This is a measure of the quality of the predictions, and looks at how many predictions are correctly aligned with ground truths. 

    This metric is useful for limiting false positives. It answers the question: "How often were the positive predictions correct?" or "How much can I trust a positive prediction?" In other words, it measures how good the model is at not making false predictions.

* **Recall**: The proportion of correct positive predictions out of all actual positives. This is a measure of a model's ability to find all positive predictions, and is especially useful in situations in which multiple labels are possible.

     This metric is useful for maximizing the number positive predictions that you get back. It answers the question "How many of the actual positives were successfully identified?" In other words, it measures how good the model is at not missing things.

Note that a "positive" prediction denotes either a 'positive' label (like a checkbox), or the presence of a particular choice/label in the prediction.

</td>
</tr>
<tr>
<td>

**Inference cost**
</td>
<td>

All task types

</td>
<td>

All labeling configs

</td>
<td>

The cost to run the prompt based on the number of tokens required. 

</td>
</tr>
</table>

### Classification reports

Click **Expand** to view classification reports for the Prompt. Thee reports tell you how many times each class was identified. This is available for the following tags:

`Choices`  
`Labels`  
`Pairwise`  
`Rating`

## Enhance prompt

You can use **Enhance Prompt** to help you construct and auto-refine your prompts. 

At minimum, you need to insert the text variable first. (Click the text variable name to insert it into the prompt. These appear above the prompts field). 

From the **Enhance Prompt** window you will need to select the **Teacher Model** that you want to use to write your prompt. As you auto-refine your prompt, you'll get the following:

* A new prompt displayed next to the previous prompt. 
* An explanation of the changes made. 
* The estimated cost spent auto-refining your prompt. 



**How it works** 

The **Task Subset** is used as the context when auto-refining the prompt. If you have ground truth data available, that will serve as the task subset. Otherwise, a sample of up to to 10 project tasks are used. 

Auto-refinement applies your initial prompt and the Teacher Model to generate predictions on the task subset (which will be ground truth tasks or a sample dataset). If applicable, predictions are then compared to the ground truth for accuracy. 

Your Teacher Model evaluates the initial prompt’s predictions against the ground truth (or sample task output) and identifies areas for improvement. It then suggests a refined prompt, aimed at achieving closer alignment with the desired outcomes.


## Drafting effective prompts

For a comprehensive guide to drafting prompts, see [The Prompt Report: A Systematic Survey of Prompting Techniques](https://arxiv.org/abs/2406.06608) or OpenAI's guide to [Prompt Engineering](https://platform.openai.com/docs/guides/prompt-engineering). 

### Text placement

When you place your text variable in the prompt (`review` in the demo above), this placeholder will be replaced by the actual text.

Depending on the length and complexity of your text, inserting it into the middle of another sentence or thought could potentially confuse the LLM. 

For example, instead of "*Classify `text` as one of the following:*", try to structure it as something like, "*Given the following text: `text`. Classify this text as one of the following:*." 

### Define your objective 

The first step to composing an effective prompt is to clearly define the task you want to accomplish. Your prompt should explicitly state that the goal is to classify the given text into predefined categories. This sets clear expectations for the model. For instance, instead of a vague request like "Analyze this text," you should say, "Classify the following text into categories such as 'spam' or 'not spam'." Clarity helps the model understand the exact task and reduces ambiguity in the responses.

### Add context

Context is crucial in guiding the model towards accurate classification. Providing background information or examples can significantly enhance the effectiveness of the prompt. For example, if you are classifying customer reviews, include a brief description of what constitutes a positive, negative, or neutral review. You could frame it as, "Classify the following customer review as 'positive,' 'negative,' or 'neutral.' A positive review indicates customer satisfaction, a negative review indicates dissatisfaction, and a neutral review is neither overly positive nor negative." This additional context helps the model align its responses with your specific requirements.

### Specificity 

Specificity in your prompt enhances the precision of the model's output. This includes specifying the format you want for the response, any particular keywords or phrases that are important, and any other relevant details. For instance, "Please classify the following text and provide the category in a single word: 'positive,' 'negative,' or 'neutral.'" By being specific, you help ensure that the model's output is consistent and aligned with your expectations.

---

### Source Guide Prompts Examples (`docs/source/guide/prompts_examples.md`)

---
title: Prompts examples
short: Examples
tier: enterprise
type: guide
order: 0
order_enterprise: 236
meta_title: Prompts examples
meta_description: Example use cases for Prompts
section: Prompts
date: 2025-01-15 12:11:22
---

## Example project types

### Text classification  

Text classification is the process of assigning predefined categories or labels to segments of text based on their content. This involves analyzing the text and determining which category or label best describes its subject, sentiment, or purpose. The goal is to organize and categorize textual data in a way that makes it easier to analyze, search, and utilize. 

Text classification labeling tasks are fundamental in many applications, enabling efficient data organization, improving searchability, and providing valuable insights through data analysis. Some examples include:

* **Spam Detection**: Classifying emails as "spam" or "ham" (not spam). 
* **Sentiment Analysis**: Categorizing user reviews as "positive," "negative," or "neutral."
* **Topic Categorization**: Assigning articles to categories like "politics," "sports," "technology," etc.
* **Support Ticket Classification**: Labeling customer support tickets based on the issue type, such as "billing," "technical support," or "account management."
* **Content Moderation**: Identifying and labeling inappropriate content on social media platforms, such as "offensive language," "hate speech," or "harassment."

### Named entity recognition (NER)

A Named Entity Recognition (NER) labeling task involves identifying and classifying named entities within text. For example, people, organizations, locations, dates, and other proper nouns. The goal is to label these entities with predefined categories that make the text easier to analyze and understand. NER is commonly used in tasks like information extraction, text summarization, and content classification.

For example, in the sentence "Heidi Opossum goes grocery shopping at Aldi in Miami" the NER task would involve identifying "Aldi" as a place or organization, "Heidi Opossum" as a person (even though, to be precise, she is an iconic opossum), and "Miami" as a location. Once labeled, this structured data can be used for various purposes such as improving search functionality, organizing information, or training machine learning models for more complex natural language processing tasks.

NER labeling is crucial for industries such as finance, healthcare, and legal services, where accurate entity identification helps in extracting key information from large amounts of text, improving decision-making, and automating workflows.

Some examples include:

* **News and Media Monitoring**: Media organizations use NER to automatically tag and categorize entities such as people, organizations, and locations in news articles. This helps in organizing news content, enabling efficient search and retrieval, and generating summaries or reports. 
* **Intelligence and Risk Analysis**: By extracting entities such as personal names, organizations, IP addresses, and financial transactions from suspicious activity reports or communications, organizations can better assess risks and detect fraud or criminal activity.
* **Specialized Document Review**: Once trained, NER can help extract industry-specific key entities for better document review, searching, and classification. 
* **Customer Feedback and Product Review**: Extract named entities like product names, companies, or services from customer feedback or reviews. This allows businesses to categorize and analyze feedback based on specific products, people, or regions, helping them make data-driven improvements.

### Text summarization

Text summarization involves condensing large amounts of information into concise, meaningful summaries. 

Models can be trained or fine-tuned to recognize essential information within a document and generate summaries that retain the core ideas while omitting less critical details. This capability is especially valuable in today’s information-heavy landscape, where professionals across various fields are often overwhelmed by the sheer volume of text data.

Some examples include:

* **Customer Support and Feedback Analysis**: Companies receive vast volumes of customer support tickets, reviews, and feedback that are often repetitive or lengthy. Auto-labeling can help summarize these inputs, focusing on core issues or themes, such as “billing issues” or “technical support.” 
* **News Aggregation and Media Monitoring**: News organizations and media monitoring platforms need to process and distribute news stories efficiently. Auto-labeling can summarize articles while tagging them with labels like “politics,” “economy,” or “health,” making it easier for users to find relevant stories.
* **Document Summarization**: Professionals often need to quickly understand the key points in lengthy contracts, research papers, and files.
* **Educational Content Summarization**: EEducators and e-learning platforms need to distill complex material into accessible summaries for students. Auto-labeling can summarize key topics and categorize them under labels like “concept,” “example,” or “important fact.”


### Image captioning and classification

Image captioning involves applying descriptive text for images. This has valuable applications across industries, particularly where visual content needs to be systematically organized, analyzed, or made accessible. 

You can also use Prompts to automatically categorizing images into predefined classes or categories, ensuring consistent labeling of large image datasets.

Some examples include:

* **E-commerce Product Cataloging**: Online retailers often deal with thousands of product images that require captions describing their appearance, features, or categories.

* **Digital Asset Management (DAM)**: Companies managing large libraries of images, such as marketing teams, media organizations, or creative agencies, can use auto-labeling to caption, tag, and classify their assets.

* **Content Moderation and Analysis**: Platforms that host user-generated content can employ image captioning to analyze and describe uploaded visuals. This helps detect inappropriate content, categorize posts (e.g., "Outdoor landscape with a sunset"), and surface relevant content to users. You may also want to train a model to classify image uploads into categories such as “safe,” “explicit,” or “spam.”

* **Accessibility for Visually Impaired Users**: Image captioning is essential for making digital content more accessible to visually impaired users by providing descriptive alt-text for images on websites, apps, or documents. For instance, an image of a cat playing with yarn might generate the caption, "A fluffy orange cat playing with a ball of blue yarn."

!!! info Tip
    You can use the `valueList` parameter to include a series of images within each task. See [this template](/templates/multi-page-document-annotation) for an example of `valueList` in use. 

## Example Walkthroughs

### Autolabel image captions 

This example demonstrates how to set up Prompts to predict image captions.

1. [Create a new label studio project](setup_project) by importing image data via [cloud storage](storage). 

    If you’d like to, you can generate a dataset to test the process using [https://data.heartex.net](https://data.heartex.net).

!!! note
    Prompts does not currently support image data uploaded as raw images. Only image references (HTTP URIs to images) or images imported via cloud storage are supported. 

2. Create a [label config](setup) for image captioning (or Ask AI to create one for you), for example:

```xml
<View>
  <Image name="image" value="$image"/>
  <Header value="Describe the image:"/>
  <TextArea name="caption" toName="image" placeholder="Enter description here..."
            rows="5" maxSubmissions="1"/>
</View>
```
3. Navigate to **Prompts** from the sidebar, and [create a prompt](prompts_create) for the project

    If you have not yet set up API the keys you want to use, do that now: [API keys](prompts_keys). 

4. Add the instruction you’d like to provide the LLM to caption your images. For example:

    *Explain the contents of the following image: `{image}`*

!!! note
    Ensure you include `{image}` in your instructions. Click `image` above the instruction field to insert it. 

!!! info Tip
    You can also automatically generate the instructions using the [**Enhance Prompt** action](prompts_draft#Enhance-prompt). Before you can use this action, you must at least add the variable name `{image}` and then click **Save**. 

    

5. Run the prompt! View predictions to accept or correct.

    You can [read more about evaluation metrics](prompts_draft#Evaluation-results) and ways to assess your prompt performance. 

!!! info Tip
    You can change the subset of data being used (e.g. only data with Ground Truth annotations, or a small sample of records). 

    

6. Accept the [predictions as annotations](prompts_predictions#Create-annotations-from-predictions)!


### Evaluate LLM outputs for toxicity

This example demonstrates how to set up Prompts to evaluate if the LLM-generated output text is classified as harmful, offensive, or inappropriate.

1. [Create a new label studio project](setup_project) by importing text data of LLM-generated outputs. 

    For example: you can use the [jigsaw_toxicity](https://huggingface.co/datasets/tasksource/jigsaw_toxicity) dataset as an example. See [the appendix](#Appendix-Generate-dataset) for how you can pre-process and (optionally) downsample this dataset to use with this guide.

2. Create a [label config](setup) for toxicity detection (or Ask AI to create one for you), for example:

```xml
<View>
  <Header value="Comment" />
  <Text name="comment" value="$comment_text"/>
  
  <Header value="Toxic" size="3"/>
  <Choices name="toxic" toName="comment" choice="single" showInline="true">
    <Choice value="Yes" alias="1"/>
    <Choice value="No" alias="0"/>
  </Choices>
  <Header value="Severely Toxic" size="3"/>
  <Choices name="severe_toxic" toName="comment" choice="single" showInline="true">
    <Choice value="Yes" alias="1"/>
    <Choice value="No" alias="0"/>
  </Choices>
  <Header value="Insult" size="3"/>
  <Choices name="insult" toName="comment" choice="single" showInline="true">
    <Choice value="Yes" alias="1"/>
    <Choice value="No" alias="0"/>
  </Choices>
  <Header value="Threat" size="3"/>
  <Choices name="threat" toName="comment" choice="single" showInline="true">
    <Choice value="Yes" alias="1"/>
    <Choice value="No" alias="0"/>
  </Choices>
  <Header value="Obscene" size="3"/>
  <Choices name="obscene" toName="comment" choice="single" showInline="true">
    <Choice value="Yes" alias="1"/>
    <Choice value="No" alias="0"/>
  </Choices>
  <Header value="Identity Hate" size="3"/>
  <Choices name="identity_hate" toName="comment" choice="single" showInline="true">
    <Choice value="Yes" alias="1"/>
    <Choice value="No" alias="0"/>
  </Choices>
  
  <Header value="Reasoning" size="3"/>
  <TextArea name="reasoning" toName="comment" editable="true" placeholder="Provide reasoning for your choices here..."/>
</View>
```

3. Navigate to **Prompts** from the sidebar, and [create a prompt](prompts_create) for the project

    If you have not yet set up API the keys you want to use, do that now: [API keys](prompts_keys). 

4. Add the instruction you’d like to provide the LLM to best evaluate the text. For example:

    *Determine whether the following text falls into any of the following categories (for each, provide a "0" for False and "1" for True):*
    
    *toxic, severe_toxic, insult, threat, obscene, and identity_hate.*

    *Comment:*

    *`{comment_text}`*


!!! note
    Ensure you include `{comment_text}` in your instructions. Click `comment_text` above the instruction field to insert it. 

!!! info Tip
    You can also automatically generate the instructions using the [**Enhance Prompt** action](prompts_draft#Enhance-prompt). Before you can use this action, you must at least add the variable name `{comment_text}` and then click **Save**. 

    

5. Run the prompt! View predictions to accept or correct.

    You can [read more about evaluation metrics](prompts_draft#Evaluation-results) and ways to assess your prompt performance. 

!!! info Tip
    You can change the subset of data being used (e.g. only data with Ground Truth annotations, or a small sample of records). 

    

6. Accept the [predictions as annotations](prompts_predictions#Create-annotations-from-predictions)! 

#### Appendix: Preprocess jigsaw toxicity dataset

Download the jigsaw_toxicity dataset, then downsample/format using the following script (modify the `INPUT_PATH` and `OUTPUT_PATH` to suit your needs):

```python
import pandas as pd
import json


def gen_task(row):
    labels = [
        {
            "from_name": field,
            "to_name": "comment",
            "type": "choices",
            "value": {"choices": [str(int(row._asdict()[field]))]},
        }
        for field in [
            "toxic",
            "severe_toxic",
            "insult",
            "threat",
            "obscene",
            "identity_hate",
        ]
    ]
    return {
        "data": {"comment_text": row.comment_text},
        "annotations": [
            {
                "result": labels,
                "ground_truth": True,
                "was_cancelled": False,
            }
        ],
    }


INPUT_PATH = "/Users/pakelley/Downloads/Jigsaw Toxicity Train.csv"
OUTPUT_PATH = "/Users/pakelley/Downloads/toxicity-sample-ls-format.json"

df = pd.read_csv(INPUT_PATH)
sample = df.sample(n=100)
label_studio_tasks = [gen_task(row) for row in sample.itertuples()]
with open(OUTPUT_PATH, "w") as f:
    json.dump(label_studio_tasks, f)
```

If you choose to, you could also easily change how many records to use (or use the entire dataset by removing the sample step). 

### Generate Synthetic Q&A Datasets

#### Overview

Synthetic datasets are datasets artificially generated rather than being collected from real-world observations. They encode characteristics similar to real data, but allow for scaling up data diversity or volume gaps in general purpose application, such as model training and evaluation. Synthetic datasets also work well in enhancing AI systems that have unbound human context as inputs and output, such as chatbot question and answers, test datasets for evaluation, and rich knowledge datasets for contextual retrieval. LLMs are particularly effective for generating synthetic datasets for these use cases, and allow you to enhance your AI system’s performance by creating diversity to learn from. 

#### Example

Let’s expand on the Q&A use case above with an example demonstrating how to use Prompts to generate synthetic user prompts for a chatbot RAG system. Given a dataset of chatbot answers, we’ll generate some questions that could return each answer.
 

1. [Create a new label studio project](setup_project) by importing chunks of text that would be meaningful answers from a chatbot.

    You can use a preprocessed sample of the [SQuAD](https://huggingface.co/datasets/rajpurkar/squad) dataset as an example. See [the appendix](#Appendix-Preprocess-SQuAD-Q-A-dataset) for how this was generated.

2. Create a [label config](setup) for question generation (or Ask AI to create one for you), for example:

```xml
<View>
  <Header value="Context" />
  <Text name="context" value="$context" />
  <Header value="Answer" />
  <Text name="answer" value="$answer" />

  <Header value="Questions" />
  <TextArea name="question1" toName="context" 
            placeholder="Enter question 1" 
            rows="2"
            maxSubmissions="1" />

  <TextArea name="question2" toName="context" 
            placeholder="Enter question 2" 
            rows="2"
            maxSubmissions="1" />

  <TextArea name="question3" toName="context" 
            placeholder="Enter question 3" 
            rows="2"
            maxSubmissions="1" />
</View>
```

3. Navigate to **Prompts** from the sidebar, and [create a prompt](prompts_create) for the project

    If you have not yet set up API the keys you want to use, do that now: [API keys](prompts_keys). 

4. Add instructions to create 3 questions:

    *Using the "context" below as context, come up with 3 questions ("question1", "question2", and "question3") for which the appropriate answer would be the "answer" below:*

    *Context:*

    *---*

    *{context}*

    *---*

    *Answer:*

    *---*

    *{answer}*

    *---*


!!! note
    Ensure you include `{answer}` and `{context}` in your instructions. Click `answer`/`context` above the instruction field to insert them. 

!!! info Tip
    You can also automatically generate the instructions using the [**Enhance Prompt** action](prompts_draft#Enhance-prompt). Before you can use this action, you must at least add a variable name (e.g. `{context}` or `{answer}`) and then click **Save**. 

    
    
5. Run the Prompt! View predictions to accept or correct.

    You can [read more about evaluation metrics](prompts_draft#Evaluation-results) and ways to assess your prompt performance. 

!!! info Tip
    You can change the subset of data being used (e.g. only data with Ground Truth annotations, or a small sample of records). 

    

6. Accept the [predictions as annotations](prompts_predictions#Create-annotations-from-predictions)!

#### Appendix: Preprocess SQuAD Q&A dataset

This downloads the SQuAD dataset from Huggingface and formats it for use in Label Studio.

```python
import pandas as pd
import json

OUTPUT_PATH = "/Users/pakelley/Downloads/qna-sample-ls-format.json"
N_SAMPLES = 100

splits = {'train': 'plain_text/train-00000-of-00001.parquet', 'validation': 'plain_text/validation-00000-of-00001.parquet'}
df = pd.read_parquet("hf://datasets/rajpurkar/squad/" + splits["train"])

sample = df.sample(n=N_SAMPLES)

sample['answer'] = sample['answers'].map(lambda item: item['text'][0])
label_studio_tasks = [{"context": row.context, "answer": row.answer} for row in sample.itertuples()]
with open(OUTPUT_PATH, "w") as f:
    json.dump(label_studio_tasks, f)
```

If you choose to, you could also easily change how many records to use (or use the entire dataset by removing the sample step).

---

### Source Guide Prompts Keys (`docs/source/guide/prompts_keys.md`)

---
title: Model provider API keys for Prompts
short: API keys
tier: enterprise
type: guide
order: 0
order_enterprise: 229
meta_title: Model provider API keys
meta_description: Add API keys to use with Prompts
section: Prompts
date: 2024-06-11 16:53:16
---

There are two approaches to adding a model provider API key. 

* In one scenario, you get one provider connection per organization, and this provides access to a set of whitelisted models. Examples include:

    * OpenAI
    * Vertex AI
    * Gemini
    * Anthropic

* In the second scenario, you add a separate API key per model. Examples include:

    * Azure OpenAI
    * Azure AI Foundry
    * Custom

!!! note
        If you are restricting network access to your resource, you may need to whitelist HumanSignal IP addresses ([IP ranges on SaaS](saas#Outbound-Connections-IP-Addresses)) when configuring network security.


Once a model is added via the API key, anyone in the organization who has access to the Prompts feature can select the associated models when executing their prompt. 

You can see what API keys you have and add new ones by clicking **API Keys** in the top right of the Prompts page to open the **Model Provider API Keys** window:



{% insertmd includes/base_models.md %} 

Once you have added a key, all supported models will appear in the base model drop-down when you [draft your prompt](prompts_draft).

## Add a model provider

!!! note
    Models you configure here will also be available in your [organization-level providers](model_providers). 

{% insertmd includes/model_keys.md %}

---

### Source Guide Prompts Predictions (`docs/source/guide/prompts_predictions.md`)

---
title: Generate predictions from a prompt
short: Generate predictions from a prompt
tier: enterprise
type: guide
order: 0
order_enterprise: 234
meta_title: Generate predictions
meta_description: Generate predictions to auto-label project tasks
section: Prompts
date: 2024-06-13 21:54:22
---

You can continue to save and evaluate your prompts until you reach your desired level of accuracy. When your prompt is ready, you can use it to generate predictions for all tasks within your project. 

!!! note
    When you click **Evaluate**, you will create predictions for each task in the baseline you selected and overwrite any previous predictions you generated with this prompt. 
    
    Evaluating your Prompts can result in multiple predictions on your tasks: if you have multiple Prompts for one Project, or if you click both **Evaluate** and **Get Predictions for All Tasks from a Prompt**, you will see multiple predictions for tasks in the Data Manager. 


## Generate predictions for all tasks

When you are confident in your prompt, click **Get Predictions for All Tasks**. This may take some time to complete depending on how many tasks you have in your project. 

<video src="../images/prompts/predictions.mp4" controls="controls" style="max-width: 800px;" class="gif-border" />

!!! note 
    Predictions are not cumulative when using this action. If you get predictions for all tasks, change the prompt, and then generate predictions for all tasks again, you will overwrite your previous predictions and each task will only reflect the most recent prediction. 

Once complete, you can return to the project and open the Data Manager. Use the **Total predictions per task** column to confirm that each task has at least one prediction:



## Remove predictions

If you prematurely generated predictions or want to use a new prompt, simply select all tasks and select **Actions > Delete Predictions**. To only remove predictions from certain models or model versions, use [the **Predictions** page in the project settings](project_settings_lse#Predictions). 

## Create annotations from predictions

Once you have your predictions in place, you might still want to convert them to annotations (depending on your workflow and your desired outcome). 

You can review predictions by opening tasks. The predictions are listed under the model name and are grayed out: 




From the Data Manager, select all the tasks you want to label and then select **Actions > Create Annotations from Predictions**. You are asked to select the model and version you want to use.

---

### Source Guide Quality (`docs/source/guide/quality.md`)

---
title: Review annotations in Label Studio
short: Review annotations
tier: enterprise
type: guide
order: 0
order_enterprise: 304
meta_title: Review annotation quality in Label Studio
meta_description: In data labeling projects, start evaluating annotator performance against ground truth annotations, predictions, and other annotator's annotations.
section: "Review & Measure Quality"
---

After multiple labelers have annotated tasks, review their output to validate the quality of the results. You can also perform this task after a model has predicted labels for tasks in your dataset. To configure the settings for reviewing annotations, see [Set up review settings for your project](setup_project.html#Set-up-review-settings-for-your-project).

The annotation review workflow is only available in Label Studio Enterprise Edition. If you're using Label Studio Community Edition, see <a href="https://labelstud.io/guide/label_studio_compare.html">Label Studio Features</a> to learn more.

See the following video for an overview of reviewer workflows: 

<iframe width="560" height="315" src="https://www.youtube.com/embed/dhBrphE7PHo?si=YMRI-omwxoQFuhma" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe>


## Why review annotations?

Data labeling is a crucial step for training many machine learning models, and it's essential to review annotations to make sure that only the highest quality data is used to train your machine learning model. If you don't review the quality of labeled data, weak annotations might be used when training your model and degrade overall model performance.

## Review annotated tasks

After you [assign reviewers to tasks](#Assign-reviewers-to-tasks), they can review annotated tasks. Administrators and project managers can review tasks at any time, without being added to a project.

1. Reviewers can click **Review Annotations** for a specific project, then click **Review All Tasks** on the Data Manager to start reviewing tasks. Administrators and project managers can click tasks from the Data Manager or **Explore Review**.
2. Review the first task and annotation. 

    By default, you view the tasks in the order in which they were presented to the annotator. If you want to change the order that you review tasks, see [Choose what to review](#Choose-what-to-review). 
    
    - If the annotation is correct, click **Accept**. 
    
    - If the annotation is mostly correct, you can correct it by selecting a different option, changing the selected region, moving the bounding box, or whichever makes sense for the type of label you're reviewing. After correcting the annotation, click **Fix & Accept**. 
    - If the annotation is incorrect, or you don't want to attempt to correct it yourself, click **Reject** to reject the annotation. 
    
        Clicking **Reject** applies the project's default reject option. If the project has more than one reject option available, you can open the reject menu to choose how to handle the rejection: 
        
        - **No Rework** -- The annotation is rejected and is not sent back for rework. 
        - **Return to Annotator** -- The annotation is sent back to the original annotator so that they can update it. 
        - **Pass to Another Annotator** -- The task is sent to a different annotator for a new annotation. The original annotator can no longer update the rejected annotation. 
        
        Which options you see depends on the project's [**Reject Options**](project_settings_lse#reject-options) settings. 

3. Continue reviewing annotated tasks until you've reviewed all annotated tasks. Click **Data Manager** to return to the list of tasks for the project.

!!! info Tip
    If there are multiple annotations, you can select the tab of each annotation by annotator and result ID to view them separately. The [annotation result ID](labeling.html#How-Label-Studio-saves-results-in-annotations) is different from the task ID visible in the left menu. To see annotations side-by-side, you can click the task in the Data Manager and view a grid of annotations in the task preview mode.

### Choose what to review

You can review tasks in random order, or order tasks in the project data manager in different ways, depending on your use case:

- Order tasks by annotator, to review annotations and assess individual annotator performance at the same time.
- Order tasks by agreement, to review annotations with more uncertainty among annotators first.
- Order tasks by model confidence score, to review the annotations that a machine learning model was less certain about first.

#### Task ordering and limiting (Enterprise)

For Label Studio Enterprise projects, you can configure the review stream ordering and limit the review slice:

- Task Ordering: Choose between By Task ID (sequential) and Random. Random preserves the order of annotations within each task but randomizes task order.
- Task Limit (%): When Random is selected, optionally limit the percentage of project tasks available for review. The limit is applied after any filters such as “Show only finished tasks.” When reviewers start a review session from a Data Manager selection (selected items), the limit is bypassed for that session.

See **Settings > Review > Task Ordering** and **Task Limit** for configuration details.

### Navigate between tasks or annotations

You can now navigate back through the review stream in the same path as moving forward when `Task is reviewed after all annotations are reviewed` option is set. The go back (`<`)functionality takes you back through the same set of annotations as it had moving forward.

1. As an **ADMINISTRATOR**, log in to the [Label Studio app](https://app.heartex.com/projects/?page=1).

2. Navigate to **All Projects** page, if needed.

3. Open any project.

4. Observe that you are taken to **Data Manager**.

5. Navigate to **Settings** >> **Review**.

6. Select `Task is reviewed after all annotations are reviewed` option.

7. Click **SAVE** button.

8. Navigate back to Data Manager.

9. Make sure to have several tasks set up with exactly two annotations each, but no reviews.

10. Click **REVIEW ALL TASKS** button.

11. Observe that you are taken to Review Stream.

12. Note the task or annotation that you are on.

13. Complete review for this annotation.

14. Observe that you are taken to same task (second annotation).

15. Repeat steps 12 to 14 for couple more times.

16. Navigate back through the Review Stream using the go back (`<`) button.

!!! note
    Confirm that you are not taken through the same path that you have come through moving forward.

<br>
<div style="margin:auto; text-align:center;"><img src="/images/go-back-reviewstream.png" style="opacity: 0.8"/></div>
<i>Figure 1: Go back to the previous task. </i>

### Assign reviewers to tasks

As an administrator or project manager, you can assign reviewers to tasks, or people with access can review tasks on an ad hoc basis. Anyone who is assigned to a task or who completes a review of a task appears in the **Reviewers** column on the Data Manager. You must first [add a reviewer to the project](setup_project.html#Add-members-to-a-project) or [add members to the project workspace](manage_users.html#Add-or-remove-members-to-a-workspace) before you can assign them as a reviewer.

1. For a specific project, select tasks on the **Data Manager**.
2. Select the tasks dropdown and select **Assign Reviewers**.
3. Select names of reviewers and click the `>` arrow to assign them to the selected tasks.
4. Click **Assign**.

You can assign reviewers to multiple tasks at once, but you cannot remove reviewers from multiple tasks at once.

## Review project quality dashboards

Use project dashboards to review annotator activity, agreement, and label distribution. For a project, click **Dashboard** to view project-level dashboards.

If you don't see an annotator's activity reflected on a dashboard, make sure they have been added as a member to the project.

### Review label distribution

Use the [Label Distribution dashboard](dashboard_distribution) to see how labels and dimension values are distributed across annotations and predictions. Use this dashboard to identify possible problems with your dataset distribution, such as labels that are overrepresented in annotated data.

For example, if you're developing a dataset of OCR images, and 90% of your tasks have **Text** labels and 10% have **Handwriting** labels, you might want to increase the number of images of handwriting in your dataset, to improve the eventual accuracy of a machine learning model trained on this dataset.

## Pause an annotator

For organizations with a large number of annotators, it might prove useful to pause an annotator's progress. This might be helpful for annotators that are performing poorly or exhibiting behavior that might indicate they have automated their work (bot behavior). 

### Full and flexible pauses

How much access a paused user keeps depends on why they were paused: 

| Pause type          | Description    |
| ------------- | ------------ |
| **Full pause** | Applies when a user is [paused manually](#Manually-pause-an-annotator) or by a [behavior-based trigger](#Behavior-based-triggers). <br><br />The user loses access to the project entirely. They cannot annotate, review, comment, or make changes to their own existing annotations. |
| **Flexible pause** | Applies when a user is paused by the [Tasks Per Annotator Limit](project_settings_lse#annotation-limit) or by [Annotator Evaluation](project_settings_lse#annotator-eval). <br><br />The user keeps access to the project so that they can work through the rejected annotations that are returned to them. They cannot begin any new annotation work. For more information, see [Recover from a flexible pause](#Recover-from-a-flexible-pause). |

A pause only restricts the type of work that triggered it. Annotation limits and annotator evaluation both measure annotation work, so a flexible pause stops new annotation work only. If a paused user is also a reviewer, they can continue reviewing. Only a full pause stops a user from reviewing. 

!!! note
    A flexible pause acts as a full pause when the project's [**Reject Options**](project_settings_lse#reject-options) do not include **Reject and return to annotator**. In this case, rejected annotations are never returned to the annotator, so there is nothing for them to recover. 

### Manually pause an annotator

You can manually pause annotators from the Members dashboard in a project. This action is only available next to users in the Annotator and Reviewer roles:



Manually pausing a user is always a full pause. When a user is paused, the following occurs:

* They immediately see a message informing them that they have been paused. 

    
* Their progress within their current task is saved as a draft, but they cannot make any further changes.   
* When they click **Go Back**, they are returned to the Projects page. If they attempt to re-enter the project, they are shown the error message above. 

!!! info Tip

    If you hover over the **Paused** indicator, you can see the message that was shown to the user when they were paused. If a user was manually paused, it also shows who initiated the action.  

    

### Automatically pause annotators 

#### Annotation Limit settings

You can use **Settings > Quality > Annotation Limit** to set limits on how many tasks an annotator is able to complete before they are paused. For more information, see [Annotation Limit](project_settings_lse#annotation-limit). 

#### Annotator Evaluation settings

You can use **Settings > Quality > Annotator Evaluation** to automatically pause annotators who do not meet a ground truth agreement or acceptance score threshold. For more information, see [Annotator Evaluation](project_settings_lse#annotator-eval).

#### Behavior-based triggers

If you have [plugins](plugins) enabled, you can automatically pause an annotator based on certain behaviors and then customize the message that appears on their screen. 

For more information, see [Plugins - Spam and Bot Detection](/plugins/pause_annotator).

### Recover from a flexible pause

When a user is under a flexible pause, they keep access to the project so that they can correct the annotations that reviewers have rejected. Getting those annotations accepted is how they work their way out of the pause. 

Users under a flexible pause can do the following:

* Re-enter the project. If the project has **Show Data Manager to annotators** enabled, they can use the Data Manager. Otherwise, they can only use the labeling stream. For more information, see [Annotation Options](project_settings_lse#annotating-options).
* Update their own rejected annotations. In the labeling stream, they are only served their rejected annotations, and they cannot skip them. 
* Add comments, so that they can respond to reviewer feedback. 

They cannot begin any new annotation work, and they cannot update annotations that have not been rejected. 

Once they have updated all of their rejected annotations, they see a message telling them that no further tasks are available until more of their work has been reviewed. 

!!! note
    Updating a rejected annotation does not lift the pause by itself. The pause remains in place until the user meets the project requirements again. For how each pause is lifted, see [Tasks Per Annotator Limit](project_settings_lse#annotation-limit) and [Annotator Evaluation](project_settings_lse#annotator-eval).


## Verify model and annotator performance

To verify the performance of specific annotators, review the [Members dashboard](dashboard_members) for a specific project. If you don't see an annotator's activity reflected, make sure they have been added as a member to the project.

### Review annotator performance

For each project, use the [Members dashboard](dashboard_members) to learn more about annotators, reviewers, models, review outcomes, and overall agreement.

Discover who has worked on the project, how many annotations and reviews they have completed, how much time they spent, and how their work compares against other annotators, ground truth annotations, or model predictions.

Review a table to see the following for each annotator:

- The total agreement for one annotator with other annotators. See more about [how annotator agreement is calculated](stats.html).
- The number of tasks assigned, pending, submitted, or skipped.
- The number of tasks that they skipped.
- The outcome of reviews for the annotations they performed.
- Their performance score and acceptance score.
- Their annotator evaluation status, if [Annotator Evaluation](project_settings_lse#annotator-eval) is enabled.
- Their average, median, and total time spent.
- The agreement of their annotations with the ground truth annotations, if there are any.
- The agreement of their annotations with predicted annotations, if there are any.

### Review annotator agreement matrix

You can also review overall annotator agreement on a more individual basis with the [Members dashboard](dashboard_members) agreement matrix.

Review the agreement matrix to understand which annotator's annotations consistently agree with or don't agree with other annotator's annotations. You can filter the matrix by dimension and label, or include model versions when predictions are available. See more about [how annotator agreement is calculated](stats).

To see the specific annotations contributing to the agreement, do the following:

1. Open the **Data Manager** for the project.
2. Locate a task annotated by the different annotators that you want to compare.
3. Click the task to open the task preview.
4. Click each annotation tab to compare how the different annotations differ. The initials of each annotator appears in the tab header with the annotation ID.

### Review agreement distribution across tasks

Use the [Data quality dashboard](dashboard_data_quality) to review the distribution of agreement percentages across project tasks. The more tasks with higher agreement, the higher quality your dataset is likely to be. Clusters of lower agreement scores might mean that some tasks are confusing, difficult to label consistently, or need clearer instructions.

## Review annotations against ground truth annotations

Define ground truth annotations in a Label Studio project. Use ground truth annotations to assess the quality of your annotated dataset. Review ground truths to make sure that annotators are accurately labeling data at the start of the project, and continually throughout the lifecycle of the training dataset creation.

Label Studio Enterprise compares annotations from annotators and model predictions against the ground truth annotations for a task to calculate an accuracy score between 0 and 1.

!!! note
    Ground truth annotations are only available in Label Studio Enterprise Edition. If you're using Label Studio Community Edition, see [Label Studio Features](https://labelstud.io/guide/label_studio_compare.html) to learn more.

## Define ground truth annotations for a project

Set specific task annotations as ground truth annotations from the **Data Manager** page for a project.

1. Locate the task that you want to set a ground truth annotation for.
2. Click the task to preview all annotations for the task as tabs. If an annotation is already set as a ground truth, the annotation ID in the tab has a yellow star next to it.
3. In the annotation sidebar for the task, click the star icon next to the annotation ID to set the annotation result as a ground truth.

A task can only have one annotation set as the ground truth annotation for the task. If you set a new annotation for a task as a ground truth, the previous annotation for that task set as a ground truth is updated to no longer be a ground truth annotation.

## Manage ground truth annotations for a project

Review and modify the ground truth annotations for a project.

### Review existing ground truth annotations

You can adjust the Data Manager columns to show whether a task has any annotations set as ground truth annotations.

1. On the **Data Manager**, select the **Columns** drop-down menu.
2. Select the checkbox for **Ground Truth**. A column appears with a star icon and true or false values listed for tasks, indicating whether a ground truth annotation has been set for a task.
3. Click **Columns** again to close the menu.

You can also filter the Data Manager to show only tasks with ground truth annotations so that you can review them.

1. On the **Data Manager**, select the **Filters** drop-down menu.
2. Click **+ Add Filter** and select where **Ground Truth** is **yes**. The Data Manager updates to show only tasks with ground truth annotations set.
3. Click **Filters** again to close the menu.

### Remove ground truth annotations

To remove ground truth annotations,

1. When viewing the data manager for a project, select the checkboxes next to annotated tasks.
2. In the selected tasks dropdown menu, select **Delete ground truths**. This does not delete the annotation, but changes the status of the ground truth setting for the annotation to false.

You can also remove ground truths when you annotate a task.

1. When labeling a task, create an annotation or select an existing one.
2. Click the star icon to unset the annotation result as a ground truth.

---

--- METRICS ---
- Files Extracted: 96
- Estimated Token Budget: ~203780 tokens
- Canonical Reference: https://codewiki.google/github.com/HumanSignal/label-studio