# Technical Documentation: microsoft/presidio > ℹ️ **Provenance:** Multi-Tier Ingestion · [CodeWiki Reference](https://codewiki.google/github.com/microsoft/presidio) ### Index (`docs/index.md`)
Inky, the Presidio mascot

Presidio

by Data Privacy Stack

Presidio _(Origin from Latin praesidium ‘protection, garrison’)_ helps to ensure sensitive data is properly managed and governed. It provides fast **_identification_** and **_anonymization_** modules for private entities in text and images such as credit card numbers, names, locations, social security numbers, bitcoin wallets, US phone numbers, financial data and more. !!! info "Project transition update" Presidio has transitioned to a community-owned project. Read the full announcement [here](project_transition.md). ## Goals - Allow organizations to preserve privacy in a simpler way by democratizing de-identification technologies and introducing transparency in decisions. - Embrace extensibility and customizability to a specific business need. - Facilitate both fully automated and semi-automated PII de-identification flows on multiple platforms. ## How it works ## Main features 1. **Predefined** or **custom PII recognizers** leveraging _Named Entity Recognition_, _regular expressions_, _rule based logic_ and _checksum_ with relevant context in multiple languages. 2. Options for connecting to external PII detection models. 3. Multiple usage options, **from Python or PySpark workloads through Docker to Kubernetes**. 4. **Customizability** in PII identification and anonymization. 5. Module for **redacting PII text in images**. !!! warning "Warning" Presidio can help identify sensitive/PII data in un/structured text. However, because it is using automated detection mechanisms, there is no guarantee that Presidio will find all sensitive information. Consequently, additional systems and protections should be employed. ## Demo Link to demo: ## Presidio's modules 1. [Presidio analyzer](analyzer/index.md): PII identification in text 2. [Presidio anonymizer](anonymizer/index.md): De-identify detected PII entities using different operators 3. [Presidio image redactor](image-redactor/index.md): Redact PII entities from images using OCR and PII identification 4. [Presidio structured](structured/index.md): PII identification in structured/semi-structured data ## Installing Presidio 1. [Supported Python Versions](installation.md#supported-python-versions) 2. [Using pip](installation.md#using-pip) 3. [Using Docker](installation.md#using-docker) 4. [From source](installation.md#install-from-source) 5. [Migrating from V1 to V2](presidio_V2.md) ## Running Presidio 1. [Samples for running Presidio via code](samples/index.md) 2. [Running Presidio as an HTTP service](samples/docker/index.md) 3. [Setting up a development environment](development.md) 4. [Perform PII identification using presidio-analyzer](analyzer/index.md) 5. [Perform PII de-identification using presidio-anonymizer](anonymizer/index.md) 6. [Perform PII identification and redaction in images using presidio-image-redactor](image-redactor/index.md) 7. [Example deployments](samples/deployments/index.md) --- ## Support - Before you submit an issue, please go over the documentation. For general discussions, please use the [GitHub repo's discussion board](https://github.com/data-privacy-stack/presidio/discussions). - If you have a usage question, found a bug or have a suggestion for improvement, please file a [GitHub issue](https://github.com/data-privacy-stack/presidio/issues). - For other matters, please email [presidio@dataprivacystack.org](mailto:presidio@dataprivacystack.org). --- ### Analyzer Index (`docs/analyzer/index.md`) # Presidio Analyzer The Presidio analyzer is a Python based service for detecting PII entities in text. During analysis, it runs a set of different _PII Recognizers_, each one in charge of detecting one or more PII entities using different mechanisms. Presidio analyzer comes with a set of predefined recognizers, but can easily be extended with other types of custom recognizers. Predefined and custom recognizers leverage regex, Named Entity Recognition and other types of logic to detect PII in unstructured text. ## Installation see [Installing Presidio](../installation.md). ## Getting started === "Python" Once the Presidio-analyzer package is installed, run this simple analysis script: ```python from presidio_analyzer import AnalyzerEngine # Set up the engine, loads the NLP module (spaCy model by default) and other PII recognizers analyzer = AnalyzerEngine() # Call analyzer to get results results = analyzer.analyze(text="My phone number is 212-555-5555", entities=["PHONE_NUMBER"], language='en') print(results) ``` === "As an HTTP server" You can run presidio analyzer as an http server using either python runtime or using a docker container. #### Using docker container ```sh cd presidio-analyzer docker run -p 5002:3000 presidio-analyzer ``` #### Using python runtime !!! note "Note" This requires the Presidio Github repository to be cloned. ```sh cd presidio-analyzer python app.py curl -d '{"text":"John Smith drivers license is AC432223", "language":"en"}' -H "Content-Type: application/json" -X POST http://localhost:3000/analyze ``` ## Main concepts Presidio analyzer is a set of tools that are used to detect entities in text. The main object in Presidio Analyzer is the `AnalyzerEngine`. In the following section we'll describe the main concepts in Presidio Analyzer. This simplified class diagram shows the main classes in Presidio Analyzer: ``` /* Detailed source-code truncated for AI context efficiency. */ ``` ### `RecognizerResult` A `RecognizerResult` holds the type and span of a PII entity. ### `EntityRecognizer` An entity recognizer is an object in Presidio that is responsible for detecting entities in text. An entity recognizer can be a rule-based recognizer, a machine learning model, or a combination of both. ### `PatternRecognizer` A `PatternRecognizer` is a type of entity recognizer that uses regular expressions to detect entities in text. One can create new `PatternRecognizer` objects by providing a list of regular expressions, context words, validation and invalidation logic and additional parameters that facilitate the detection of entities. ### `AnalyzerEngine` The `AnalyzerEngine` is the main object in Presidio Analyzer that is responsible for detecting entities in text. The `AnalyzerEngine` can be configured in various ways to fit the specific needs of the user. ### `RecognizerRegistry` The `RecognizerRegistry` is a registry that contains all the entity recognizers that are available in Presidio. The `AnalyzerEngine` uses the `RecognizerRegistry` to detect entities in text. ### `NlpEngine` An NLP Engine is an object that holds the NLP model that is used by the `AnalyzerEngine` to parse the input text and extract different features from it, such as tokens, lemmas, entities, and more. Note that Named Entity Recognition (NER) models can be added in two ways to Presidio: One is through the `NlpEngine` object, and the other is through a new `EntityRecognizer` object. By creating a Named Entity Recognition model through the `NlpEngine`, the named entities will be available to the different modules in Presidio. Furthermore, the `NlpEngine` object supports a batch mode (i.e., processing multiple texts at once) which allows for faster processing of large amounts of text. It is possible to mix multiple NER models in Presidio, for instance, one model as the `NlpEngine` and others as additional `EntityRecognizer` objects. Presidio has an off-the-shelf support for multiple NLP packages, such as spaCy, stanza, and huggingface. The simplest way to integrate a model from these packages is through the `NlpEngine`. More information on this [can be found in the NlpEngine documentation](customizing_nlp_models.md). The samples gallery has several examples of leveraging NER models as new `EntityRecognizer` objects. For example, [flair](../samples/python/flair_recognizer.py) and [spanmarker](../samples/python/span_marker_recognizer.py). For a detailed flow of Named Entities within presidio, see the diagram [in this document](nlp_engines/transformers.md#how-ner-results-flow-within-presidio). ### `Context Aware Enhancer` The `ContextAwareEnhancer` is a module that enhances the detection of entities by using the context of the text. The `ContextAwareEnhancer` can be used to improve the detection of entities that are dependent on the context of the text, such as dates, locations, and more. The default implementation is the `LemmaContextAwareEnhancer` which uses the lemmas of the tokens in the text to enhance the detection of entities. Note that it's possible (and sometimes recommended) to create custom `ContextAwareEnhancer` objects to fit the specific needs of the user, for example if the context should support more than one word, which is currently not supported by the default Lemma based enhancer. More information on this can be found [in this sample](../samples/python/customizing_presidio_analyzer.ipynb). ## Creating PII recognizers Presidio analyzer can be easily extended to support additional PII entities. See [this tutorial on adding new PII recognizers](adding_recognizers.md) for more information. ## Multi-language support Presidio can be used to detect PII entities in multiple languages. Refer to the [multi-language support](languages.md) for more information. ## Filtering recognizers by country Predefined recognizers in Presidio are tagged with a `country_code` so deployments can scope detection to a specific set of countries (e.g. US-only, EU-only) without manually enumerating recognizer classes. See the [Filtering recognizers by country](filtering_by_country.md) guide for usage and the rules for tagging custom recognizers. ## Outputting the analyzer decision process Presidio analyzer has a built in mechanism for tracing each decision made. This can be useful when attempting to understand a specific PII detection. For more info, see the [decision process](decision_process.md) documentation. ## Supported entities For a list of the current supported entities: [Supported entities](../supported_entities.md). ## API reference Follow the [API Spec](https://data-privacy-stack.github.io/presidio/api-docs/api-docs.html#tag/Analyzer) for the Analyzer REST API reference details and [Analyzer Python API](../api/analyzer_python.md) for Python API reference --- ### Anonymizer Index (`docs/anonymizer/index.md`) # Presidio Anonymizer The Presidio anonymizer is a Python based module for anonymizing detected PII text entities with desired values. Presidio anonymizer supports both anonymization and deanonymization by applying different operators. Operators are built-in text manipulation classes which can be easily extended. The Presidio-Anonymizer package contains both `Anonymizers` and `Deanonymizers`. - *Anonymizers* are used to replace a PII entity text with some other value by applying a certain operator (e.g. replace, mask, redact, encrypt) - *Deanonymizers* are used to revert the anonymization operation. (e.g. to decrypt an encrypted text). ## Installation see [Installing Presidio](../installation.md). ## Getting started === "Python" Simple example: ```python from presidio_anonymizer import AnonymizerEngine from presidio_anonymizer.entities import RecognizerResult, OperatorConfig # Initialize the engine: engine = AnonymizerEngine() # Invoke the anonymize function with the text, # analyzer results (potentially coming from presidio-analyzer) and # Operators to get the anonymization output: result = engine.anonymize( text="My name is Bond, James Bond", analyzer_results=[ RecognizerResult(entity_type="PERSON", start=11, end=15, score=0.8), RecognizerResult(entity_type="PERSON", start=17, end=27, score=0.8), ], operators={"PERSON": OperatorConfig("replace", {"new_value": "BIP"})}, ) print(result) ``` This example takes the output of the `AnonymizerEngine` containing an encrypted PII entity, and decrypts it back to the original text: ```python from presidio_anonymizer import DeanonymizeEngine from presidio_anonymizer.entities import OperatorResult, OperatorConfig # Initialize the engine: engine = DeanonymizeEngine() # Invoke the deanonymize function with the text, anonymizer results and # Operators to define the deanonymization type. result = engine.deanonymize( text="My name is S184CMt9Drj7QaKQ21JTrpYzghnboTF9pn/neN8JME0=", entities=[ OperatorResult(start=11, end=55, entity_type="PERSON"), ], operators={"DEFAULT": OperatorConfig("decrypt", {"key": "WmZq4t7w!z%C&F)J"})}, ) print(result) ``` === "As an HTTP server" You can run presidio anonymizer as an http server using either python runtime or using a docker container. #### Using docker container ```sh cd presidio-anonymizer docker run -p 5001:3000 presidio-anonymizer ``` #### Using python runtime !!! note "Note" This requires the Presidio Github repository to be cloned. ``` /* Detailed source-code truncated for AI context efficiency. */ ``` ## Main concepts The following class diagram shows a simplified view of the main classes in Presidio Anonymizer: ``` /* Detailed source-code truncated for AI context efficiency. */ ``` - The **AnonymizerEngine** is the main class in Presidio that is responsible for anonymizing PII entities in text. It uses the results from the **AnalyzerEngine** to perform the anonymization. - The **DeanonymizerEngine** is a class in Presidio that is responsible for deanonymizing text that has been anonymized by the **AnonymizerEngine**, given that the operation is reversible (e.g. encryption). - An **Operator** is an object in Presidio that is responsible for performing the anonymization operation on a PII entity. Presidio provides several built-in operators, such as **Replace**, **Redact**, and **Encrypt**, and allows users to create custom operators. - The **BatchAnonymizerEngine** is a class in Presidio that is responsible for anonymizing PII entities in a batch of texts. It uses the **AnonymizerEngine** to perform the anonymization on each text in the batch. ([see more here](../samples/python/batch_processing.ipynb)). ## Built-in operators | Operator type | Operator name | Description | Parameters | |---------------|---------------|---------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | Anonymize | replace | Replace the PII with desired value | `new_value`: replaces existing text with the given value.
If `new_value` is not supplied or empty, default behavior will be: e.g: | | Anonymize | redact | Remove the PII completely from text | None | | Anonymize | hash | Hashes the PII text using salted hashing for security | `hash_type`: sets the type of hashing. Can be either `sha256` or `sha512`. The default is `sha256`.
`salt`: Optional salt for reproducible hashing. If not provided, a random salt is generated per entity to prevent brute-force attacks. To maintain referential integrity across records/calls, provide a consistent salt (see example below). | | Anonymize | mask | Replace the PII with a given character | `chars_to_mask`: the amount of characters out of the PII that should be replaced.
`masking_char`: the character to be replaced with.
`from_end`: Whether to mask the PII from it's end. | | Anonymize | encrypt | Encrypt the PII using a given key | `key`: a cryptographic key used for the encryption. | | Anonymize | custom | Replace the PII with the result of the function executed on the PII | `lambda`: lambda to execute on the PII data. The lambda return type must be a string. | | Anonymize | surrogate_ahds | Generate realistic, medically-appropriate surrogates using Azure Health Data Services de-identification service surrogation | `endpoint`: AHDS endpoint (optional, uses AHDS_ENDPOINT env var)
`entities`: List of entities detected by analyzer
`input_locale`: Input locale (default: "en-US")
`surrogate_locale`: Surrogate locale (default: "en-US")
Requires: `pip install presidio-anonymizer[ahds]` | | Anonymize | keep | Preserver the PII unmodified | None | | Deanonymize | decrypt | Decrypt the encrypted PII in the text using the encryption key | `key`: a cryptographic key used for the encryption is also used for the decryption. | !!! note "Note" When performing anonymization, if anonymizers map is empty or "DEFAULT" key is not stated, the default anonymization operator is "replace" for all entities. The replacing value will be the entity type e.g.: ### Hash operator with salt for referential integrity !!! warning "BREAKING CHANGE in Hash Operator" **Starting from version 2.2.361, the hash operator uses random salt by default for security.** **Impact:** - Hash outputs are different from previous versions - Same PII value gets **different hashes** across different entities and calls (unless you provide a salt) - This prevents brute-force and dictionary attacks but breaks backward compatibility **Action Required:** - If you need the same hash for the same value (referential integrity), you **must** explicitly provide a `salt` parameter - If you're upgrading from a previous version and need to match existing hashes, this is not possible without the original salt !!! note: Note Presidio does not store or maintain stateful sessions. For referential integrity across records or calls, users must securely manage and provide their own salt. The hash operator uses random salt by default for maximum security, which means the same PII value will get different hashes in different entities or calls. If you need referential integrity (same values getting the same hash), you must provide a consistent salt: ```python import secrets from presidio_anonymizer import AnonymizerEngine from presidio_anonymizer.entities import RecognizerResult, OperatorConfig # Generate a secure salt once (store this securely for reuse across calls) salt = secrets.token_hex(32) # 64 character hex string (256 bits) engine = AnonymizerEngine() text = "John Doe called John Doe yesterday" analyzer_results = [ RecognizerResult(start=0, end=8, score=0.8, entity_type="PERSON"), RecognizerResult(start=16, end=24, score=0.8, entity_type="PERSON"), ] # Use the same salt for all anonymizations to maintain referential integrity result = engine.anonymize( text, analyzer_results, {"DEFAULT": OperatorConfig("hash", {"salt": salt})} ) # Both "John Doe" instances will have the same hash print(result.text) ``` !!! warning "Security Considerations" - **Discard the salt after processing**: The salt should be discarded once all data is anonymized to prevent re-identification if leaked - Store salt only if you need to process additional data later for referential integrity (increases re-identification risk) - If you must store salt, use secure storage (e.g., key vault or secrets manager) with strict access controls - Use a salt of at least 128 bits (16 bytes) - enforced by the operator - Never include the salt in anonymized output - For maximum security without referential integrity needs, omit the salt parameter to use random per-entity salts ## Handling overlaps between entities As the input text could potentially have overlapping PII entities, there are different anonymization scenarios: - **No overlap (single PII)**: When there is no overlap in spans of entities, Presidio Anonymizer uses a given or default anonymization operator to anonymize and replace the PII text entity. - **Full overlap of PII entity spans**: When entities have overlapping substrings, the PII with the higher score will be taken. Between PIIs with identical scores, the selection is arbitrary. - **One PII is contained in another**: Presidio Anonymizer will use the PII with the larger text even if it's score is lower. - **Partial intersection**: Presidio Anonymizer will anonymize each individually and will return a concatenation of the anonymized text. For example: For the text ``` I'm George Washington Square Park. ``` Assuming one entity is `George Washington` and the other is `Washington State Park` and assuming we're using the default anonymizer, the result would be: ``` I'm . ``` ### Additional examples for overlapping PII scenarios Text: ``` My name is Inigo Montoya. You Killed my Father. Prepare to die. BTW my number is: 03-232323. ``` Results: - **No overlaps**: Assuming only `Inigo` is recognized as NAME: ``` My name is Montoya. You Killed my Father. Prepare to die. BTW my number is: 03-232323. ``` - **Full overlap**: Assuming the number is recognized as PHONE_NUMBER with score of 0.7 and as SSN with score of 0.6, the higher score would count: ``` My name is Inigo Montoya. You Killed my Father. Prepare to die. BTW my number is: < PHONE_NUMBER>. ``` - **One PII is contained is another**: Assuming Inigo is recognized as FIRST_NAME and Inigo Montoya was recognized as NAME, the larger one will be used: ``` My name is . You Killed my Father. Prepare to die. BTW my number is: 03-232323. ``` - **Partial intersection**: Assuming the number 03-2323 is recognized as a PHONE_NUMBER but 232323 is recognized as SSN: ``` My name is Inigo Montoya. You Killed my Father. Prepare to die. BTW my number is: < PHONE_NUMBER>. ``` ## Creating a new `operator` Presidio anonymizer can be easily extended to support additional operators. See [this tutorial on adding new operators](adding_operators.md) for more information. ## API reference Follow the [API Spec](https://data-privacy-stack.github.io/presidio/api-docs/api-docs.html#tag/Anonymizer) for the Anonymizer REST API reference details and [Anonymizer Python API](../api/anonymizer_python.md) for Python API reference. --- ### Evaluation Index (`docs/evaluation/index.md`) # Evaluating PII detection with Presidio ## Why evaluate PII detection? No de-identification system is perfect. It is important to evaluate the performance of a PII detection system for your specific use case. This evaluation can help you understand where the system makes mistakes and how to iteratively improve the detection mechanisms, which recognizers and models to use, and how to configure them. ## Common evaluation metrics The most common evaluation metrics are [`precision`, `recall`](), and [`Fβ score`](), which is a combination of precision and recall. These metrics are calculated based on the number of true positives, false positives, and false negatives. For every use case, the false positive and false negative rates should be balanced to achieve the desired level of accuracy. - Precision measures the proportion of true positive results among the positive results: `TP / (TP + FP)`. - Recall measures the proportion of true positive results among the actual positives: `TP / (TP + FN)`. - Fβ score is a weighted harmonic mean of precision and recall: `(1 + β^2) * (precision * recall) / (β^2 * precision + recall)`. [Click here for more definitions](https://en.wikipedia.org/wiki/Precision_and_recall#Definition). !!! note "Note" In PII detection, recall is often more important than precision, as we'd like to avoid missing any PII. In such cases, we recommend to use the β=2 score, which gives more importance to recall. ## How to evaluate PII detection with Presidio Presidio provides a set of tools to evaluate the performance of the PII detection system. In addition, it provides simple data generation tools to help you create a dataset for evaluation. ### Evaluating the Presidio Analyzer using Presidio-Research Presidio-Research is a python package with a set of tools that help you evaluate the performance of the Presidio Analyzer. To get started, follow the instructions in the [Presidio-Research repository](https://github.com/data-privacy-stack/presidio-research). The easiest way to get started is by reviewing the notebooks: - [Notebook 1](https://github.com/data-privacy-stack/presidio-research/blob/master/notebooks/1_Generate_data.ipynb): Shows how to use the PII data generator. - [Notebook 2](https://github.com/data-privacy-stack/presidio-research/blob/master/notebooks/2_PII_EDA.ipynb): Shows a simple analysis of the PII dataset. - [Notebook 3](https://github.com/data-privacy-stack/presidio-research/blob/master/notebooks/3_Split_by_pattern_number.ipynb): Provides tools to split the dataset into train/test/validation sets while avoiding leakage due to the same pattern appearing in multiple folds (only applicable for synthetically generated data). - [Notebook 4](https://github.com/data-privacy-stack/presidio-research/blob/master/notebooks/4_Evaluate_Presidio_Analyzer.ipynb): Shows how to use the evaluation tools to evaluate how well Presidio detects PII. Note that this is using the vanilla Presidio, and the results aren't very accurate. - [Notebook 5](https://github.com/data-privacy-stack/presidio-research/blob/master/notebooks/5_Evaluate_Custom_Presidio_Analyzer.ipynb): Shows how one can configure Presidio to detect PII much more accurately, and boost the f score in ~30%. For more information and advanced usage, refer to the [Presidio-Research repository](https://github.com/data-privacy-stack/presidio-research). ### Evaluating DICOM redaction with Presidio Image Redactor See [Evaluating DICOM redaction](../image-redactor/evaluating_dicom_redaction.md) for more information. For a full demonstration, see the [evaluation notebook](../samples/python/example_dicom_redactor_evaluation.ipynb). --- ### Image Redactor Index (`docs/image-redactor/index.md`) # Presidio Image Redactor ***Please notice, this package is still in beta and not production ready.*** ## Description The Presidio Image Redactor is a Python based module for detecting and redacting PII text entities in images. This module may also be used on medical DICOM images. The `DicomImageRedactorEngine` class may be used to redact text PII present as pixels in DICOM images. !!! note "Note" This class only redacts pixel data and does not scrub text PII which may exist in the DICOM metadata. We highly recommend using the DICOM image redactor engine to redact text from images BEFORE scrubbing metadata PII.* ## Installation Pre-requisites: - Install [Tesseract OCR](https://github.com/tesseract-ocr/tesseract#installing-tesseract) by following the instructions on how to install it for your operating system. !!! attention "Attention" For best performance, please use the most up-to-date version of Tesseract OCR. Presidio was tested with **v5.2.0**. === "Using pip" !!! note "Note" Consider installing the Presidio python packages on a virtual environment like venv or conda. To get started with Presidio-image-redactor, download the package and the `en_core_web_lg` spaCy model: ```sh pip install presidio-image-redactor python -m spacy download en_core_web_lg ``` === "Using Docker" !!! note "Note" This requires Docker to be installed. [Download Docker](https://docs.docker.com/get-docker/). ```sh # Download image from GitHub Container Registry docker pull ghcr.io/data-privacy-stack/presidio-image-redactor # Run the container with the default port docker run -d -p 5003:3000 ghcr.io/data-privacy-stack/presidio-image-redactor:latest ``` === "From source" First, clone the Presidio repo. [See here for instructions](../installation.md#install-from-source). Then, build the presidio-image-redactor container: ```sh cd presidio-image-redactor docker build . -t presidio/presidio-image-redactor ``` ## Getting started (standard image types) === "Python" Once the Presidio-image-redactor package is installed, run this simple script: ```python from PIL import Image from presidio_image_redactor import ImageRedactorEngine # Get the image to redact using PIL lib (pillow) image = Image.open("./docs/image-redactor/ocr_text.png") # Initialize the engine engine = ImageRedactorEngine() # Redact the image with pink color redacted_image = engine.redact(image, (255, 192, 203)) # Optional: Redact the image and return redacted regions redacted_image, bboxes = engine.redact_and_return_bbox(image, (255, 192, 203)) # save the redacted image redacted_image.save("new_image.png") # uncomment to open the image for viewing # redacted_image.show() ``` === "As an HTTP server" You can run presidio image redactor as an http server using either python runtime or using a docker container. #### Using docker container ```sh cd presidio-image-redactor docker run -p 5003:3000 presidio-image-redactor ``` #### Using python runtime !!! note "Note" This requires the Presidio Github repository to be cloned. ```sh cd presidio-image-redactor python app.py # use ocr_test.png as the image to redact, and 255 as the color fill. # out.png is the new redacted image received from the server. curl -XPOST "http://localhost:3000/redact" -H "content-type: multipart/form-data" -F "image=@ocr_test.png" -F "data=\"{'color_fill':'255'}\"" > out.png ``` Python script example can be found under: /presidio/e2e-tests/tests/test_image_redactor.py ## Getting started (DICOM images) === "Python" Once the Presidio-image-redactor package is installed, run this simple script: ```python import pydicom from presidio_image_redactor import DicomImageRedactorEngine # Set input and output paths input_path = "path/to/your/dicom/file.dcm" output_dir = "./output" # Initialize the engine engine = DicomImageRedactorEngine() # Option 1: Redact from a loaded DICOM image dicom_image = pydicom.dcmread(input_path) redacted_dicom_image = engine.redact(dicom_image, fill="contrast") # Option 2: Redact from a loaded DICOM image and return redacted regions redacted_dicom_image, bboxes = engine.redact_and_return_bbox(dicom_image, fill="contrast") # Option 3: Redact from DICOM file and save redacted regions as json file engine.redact_from_file(input_path, output_dir, padding_width=25, fill="contrast", save_bboxes=True) # Option 4: Redact from directory and save redacted regions as json files ocr_kwargs = {"ocr_threshold": 50} engine.redact_from_directory("path/to/your/dicom", output_dir, fill="background", save_bboxes=True, ocr_kwargs=ocr_kwargs) ``` ## Getting started using the document intelligence OCR engine Presidio offers two engines for OCR based PII removal. The first is the default engine which uses Tesseract OCR. The second is the Document Intelligence OCR engine which uses Azure's Document Intelligence service, which requires an Azure subscription. The following sections describe how to setup and use the Document Intelligence OCR engine. You will need to register with Azure to get an API key and endpoint. Perform the steps in the "Prerequisites" section of [this page](https://learn.microsoft.com/en-us/azure/ai-services/document-intelligence/quickstarts/get-started-sdks-rest-api). Once your resource deploys, copy your endpoint and key values and save them for the next step. The most basic usage of the engine can be setup like the following in python ``` diOCR = DocumentIntelligenceOCR(endpoint="", key="") ``` If your environment uses Azure Identity instead of API keys, pass an Azure SDK credential that implements `TokenCredential`, such as `DefaultAzureCredential`. For example, install `azure-identity` and use `DefaultAzureCredential`: ``` from azure.identity import DefaultAzureCredential diOCR = DocumentIntelligenceOCR( endpoint="", credential=DefaultAzureCredential(), ) ``` The DocumentIntelligenceOCR can also attempt to pull your endpoint and key values from environment variables. ``` export DOCUMENT_INTELLIGENCE_ENDPOINT= export DOCUMENT_INTELLIGENCE_KEY= ``` ### Document Intelligence Model Support There are numerous document processing models available, and currently we only support the most basic usage of the model. For an overview of the functionalities offered by Document Intelligence, see [this page](https://learn.microsoft.com/en-us/azure/ai-services/document-intelligence/concept-model-overview). Presidio offers only word-level processing on the result for PII redaction purposes, as all prebuilt document models support this interface. Different models support additional structured support for tables, paragraphs, key-value pairs, fields and other types of metadata in the response. Additional metadata can be sent to the Document Intelligence API call, such as pages, locale, and features, which are documented [here](https://learn.microsoft.com/en-us/python/api/azure-ai-formrecognizer/azure.ai.formrecognizer.documentanalysisclient?view=azure-python#azure-ai-formrecognizer-documentanalysisclient-begin-analyze-document). You are encouraged to test each model to see which fits best to your use case. #### Creating an image redactor engine in Python ``` diOCR = DocumentIntelligenceOCR() ia_engine = ImageAnalyzerEngine(ocr=diOCR) my_engine = ImageRedactorEngine(image_analyzer_engine=ia_engine) ``` #### Testing Document Intelligence Follow the steps of [running the tests](../development.md#running-tests) The test suite has a series of tests which are only exercised when the appropriate environment variables are populated. To run the test suite, to test the DocumentIntelligenceOCR engine, call the tests like this: ``` export DOCUMENT_INTELLIGENCE_ENDPOINT= export DOCUMENT_INTELLIGENCE_KEY= pytest ``` ### Evaluating de-identification performance If you are interested in evaluating the performance of the DICOM de-identification against ground truth labels, please see the [evaluating DICOM de-identification page](./evaluating_dicom_redaction.md). ### Side note for Windows If you are using a Windows machine, you may run into issues if file paths are too long. Unfortunately, this is not rare when working with DICOM images that are often nested in directories with descriptive names. To avoid errors where the code may not recognize a path as existing due to the length of the characters in the file path, please [enable long paths on your system](https://learn.microsoft.com/en-us/answers/questions/293227/longpathsenabled.html). ### DICOM Data Citation The DICOM data used for unit and integration testing for `DicomImageRedactorEngine` are stored in this repository with permission from the original dataset owners. Please see the dataset information as follows: > Rutherford, M., Mun, S.K., Levine, B., Bennett, W.C., Smith, K., Farmer, P., Jarosz, J., Wagner, U., Farahani, K., Prior, F. (2021). A DICOM dataset for evaluation of medical image de-identification (Pseudo-PHI-DICOM-Data) [Data set]. The Cancer Imaging Archive. DOI: ## API reference the [API Spec](https://data-privacy-stack.github.io/presidio/api-docs/api-docs.html#tag/Image-redactor) for the Image Redactor REST API reference details and [Image Redactor Python API](../api/image_redactor_python.md) for Python API reference --- ### Learn Presidio Index (`docs/learn_presidio/index.md`) # Learn Presidio Presidio is a suite of tools for detecting and de-identifying PII in text, images, and structured data. The recommended place to start is to follow the [tutorial](../tutorial/index.md) which will guide you through the process of setting up and using Presidio. To learn about the different concepts in Presidio, visit the [concepts page](concepts.md). To go deeper into each component, visit the relevant docs: - For the Presidio Analyzer, visit the [Analyzer docs](../analyzer/index.md). - For Presidio Anonymizer, visit the [Anonymizer docs](../anonymizer/index.md). - For Presidio Image Redactor, visit the [Image Redactor docs](../image-redactor/index.md). - For Presidio structured, visit the [Structured docs](../structured/index.md). The following diagrams provide a high level understanding of the Presidio components: ## Analyzer [](../analyzer/index.md) ## Anonymizer [](../anonymizer/index.md) ## Image Redactor ### Standard Image Types [](../image-redactor/index.md) ### DICOM Images [](../image-redactor/index.md) --- ### Recipes Index (`docs/recipes/index.md`) # Recipes Gallery Welcome to the Presidio Recipes Gallery! This section provides curated, end-to-end examples demonstrating how to customize Presidio for specific data privacy and de-identification scenarios. ## What are Recipes? Recipes are comprehensive, reproducible examples tailored to common data domains and use cases. Each recipe goes beyond basic documentation to provide: 1. **Real-world Context**: Focused on specific scenarios like financial chatbot conversations, clinical notes, REST API logs, or multilingual content 2. **Synthetic Data Generation**: Methods for creating realistic test data that mimics your production environment 3. **Performance Benchmarks**: Evaluation metrics (precision, recall, F₂ score, and latency) across different Presidio configurations 4. **Progressive Complexity**: Examples ranging from out-of-the-box usage to advanced customization with transformers, LLMs, or hybrid approaches ## Why Use Recipes? While Presidio's documentation covers the fundamentals, recipes bridge the gap between generic examples and production-ready implementations. They help you: - **Evaluate Performance**: Understand Presidio's accuracy and speed for your specific domain before deployment - **Customize Effectively**: Learn which recognizers, models, and configurations work best for different data types - **Compare Approaches**: See side-by-side comparisons of different implementation strategies - **Reduce Development Time**: Start with a working example close to your use case instead of building from scratch ## Recipe Structure Each recipe typically includes: - **Scenario Description**: The domain and data type - **Data Synthesis**: Methods for generating test data using [Presidio Evaluator](https://github.com/data-privacy-stack/presidio-research/blob/master/notebooks/1_Generate_data.ipynb) or custom methods - **Configuration**: Presidio setup with any custom recognizers or models - **Evaluation**: Performance metrics (precision, recall, F₂ score, latency) - **Implementation**: Jupyter notebook or Python scripts showing the end-to-end flow (see [example](https://github.com/data-privacy-stack/presidio-research/blob/master/notebooks/4_Evaluate_Presidio_Analyzer.ipynb)) - **Key Findings**: When to use this approach and trade-offs to consider For complex flows, consider breaking into multiple notebooks or scripts for better organization. ## Available Recipes Currently, the recipes gallery is being built. Check back soon for recipes covering: - **Financial Domain**: Chat conversations, transaction logs, customer service interactions - **Healthcare Domain**: Clinical notes, patient records, medical reports - **Retail/E-commerce**: Customer data, order information, support tickets - **Enterprise**: REST API logs, database exports, internal communications - **Multilingual**: Examples for Spanish, French, German, and other languages ## Recipe Performance Table (Coming Soon) We're developing a comprehensive benchmark table that will show Presidio's performance across different domains and implementation levels. The table will include: | Domain / Scenario | Out-of-the-box (spaCy) | Augmented (+ custom recognizers) | Custom Model (ML/Transformer) | Hybrid "Best-Effort" (ensemble/LLM) | |-------------------|------------------------|----------------------------------|--------------------------------|-------------------------------------| | **Financial (Chatbot)** | Coming Soon | Coming Soon | Coming Soon | Coming Soon | | **Medical (Clinical Notes)** | Coming Soon | Coming Soon | Coming Soon | Coming Soon | | **Retail (JSON REST)** | Coming Soon | Coming Soon | Coming Soon | Coming Soon | | **Multilingual (Spanish)** | Coming Soon | Coming Soon | Coming Soon | Coming Soon | Each cell will contain: - **P** = Precision - **R** = Recall - **F₂** = F₂ score (recall-weighted F-score) - **Latency** = Average processing time per sample (milliseconds) - **Notebook** = Link to interactive Jupyter notebook ## How to Use a Recipe 1. **Browse the recipes** to find one matching your domain or use case 2. **Review the notebook** to understand the approach and results 3. **Run the notebook** in your environment to reproduce the results 4. **Adapt the configuration** to your specific data and requirements 5. **Evaluate performance** on your own test dataset 6. **Deploy** the configuration that best meets your accuracy and performance needs ## Contributing a Recipe We welcome community contributions! See our [contribution guidelines](CONTRIBUTING.md) for details. **Reference Examples:** - [Evaluate Presidio Analyzer](https://github.com/data-privacy-stack/presidio-research/blob/master/notebooks/4_Evaluate_Presidio_Analyzer.ipynb) - Complete end-to-end evaluation workflow - [Generate Synthetic Data](https://github.com/data-privacy-stack/presidio-research/blob/master/notebooks/1_Generate_data.ipynb) - Presidio Evaluator data generator Follow the pattern: **Data Synthesis** → **Configuration** → **Evaluation** For complex flows, break into multiple notebooks or scripts. Focus on getting working code first - we'll help refine documentation during review. ## Related Resources - [Presidio Samples](../samples/index.md): Additional usage examples and integration patterns - [Tutorial Series](../tutorial/index.md): Step-by-step guide to Presidio features - [Best Practices for Developing Recognizers](../analyzer/developing_recognizers.md): Deep dive into creating custom PII recognizers - [Presidio Research Repository](https://github.com/data-privacy-stack/presidio-research): Evaluation tools and research datasets - [FAQ](../faq.md): Common questions about improving detection accuracy ## Questions or Feedback? If you have questions about recipes or suggestions for new scenarios to cover, please: - Open an issue on [GitHub](https://github.com/data-privacy-stack/presidio/issues) - Email us at [presidio@dataprivacystack.org](mailto:presidio@dataprivacystack.org) - Join the discussion in our [community channels](../community.md) --- **Note**: The recipes gallery demonstrates Presidio's flexibility and customization capabilities. The goal is to show that Presidio is designed to be adapted to your specific needs, not used as a one-size-fits-all solution. Each recipe illustrates best practices for customization in different contexts. --- ### Samples Index (`docs/samples/index.md`) # Samples | Topic | Data Type |Resource | Sample | | :---------- |:--------------------------------------| :---------------------------------| :---------------------------------------------------------------------------------------------------------------------------------------------- | | Usage | Text | Python Notebook | [Presidio Basic Usage Notebook](python/presidio_notebook.ipynb) | | Usage | Text | Python Notebook | [Customizing Presidio Analyzer](python/customizing_presidio_analyzer.ipynb) | | Usage | Text | Python Notebook | [Configuring The NLP engine](python/ner_model_configuration.ipynb) | | Usage | Semi-structured | Python Notebook | [Analyzing structured / semi-structured data in batch](python/batch_processing.ipynb)| | Usage | Text | Python Notebook | [Encrypting and Decrypting identified entities](python/encrypt_decrypt.ipynb)| | Usage | Text | Python Notebook | [Getting the identified entity value using a custom Operator](python/getting_entity_values.ipynb)| | Usage | text | Python Notebook | [Anonymizing known values](python/Anonymizing%20known%20values.ipynb) | Usage | Images | Python Notebook | [Redacting Text PII from DICOM images](python/example_dicom_image_redactor.ipynb) | Usage | Images | Python Notebook | [Using an allow list with image redaction](python/image_redaction_allow_list_approach.ipynb) | Usage | PDF | Python Notebook | [Annotating PII in a PDF](python/example_pdf_annotation.ipynb) | Usage | Images | Python Notebook | [Plot custom bounding boxes](python/plot_custom_bboxes.ipynb) | Usage | Text | Python Notebook | [Integrating with external services](python/integrating_with_external_services.ipynb) | | Usage | Text | Python file | [Remote Recognizer](python/example_remote_recognizer.py) | | Usage | Structured | Python Notebook | [Presidio Structured Basic Usage Notebook](python/example_structured.ipynb) | | Usage | Text | Python file | [Azure AI Language as a Remote Recognizer](python/text_analytics/index.md) | | Usage | Text | Python file | [Azure Health Data Services de-identification Service as a Remote Recognizer](python/ahds/index.md) | | Usage | Text | Python file | [AHDS Surrogate Example](python/ahds/example_ahds_surrogate.py) | | Usage | Text | Python file | [Language Model-based PII/PHI Detection using LangExtract](python/langextract/index.md) | | Usage | CSV | Python file | [Analyze and Anonymize CSV file](https://github.com/data-privacy-stack/presidio/blob/main/docs/samples/python/process_csv_file.py) | | Usage | Text | Python | [Using Flair as an external PII model](https://github.com/data-privacy-stack/presidio/blob/main/docs/samples/python/flair_recognizer.py)| | Usage | Text | Python file | [Using Span Marker as an external PII model](https://github.com/data-privacy-stack/presidio/blob/main/docs/samples/python/span_marker_recognizer.py)| | Usage | Text | Python file | [Using Transformers as an external PII model](python/transformers_recognizer/index.md)| | Usage | Text | Python file | [Pseudonymization (replace PII values using mappings)](python/pseudonymization.ipynb)| | Usage | Text | Python file | [Passing a lambda as a Presidio anonymizer using Faker](python/example_custom_lambda_anonymizer.py)| | Usage | Text | Python file | [Synthetic data generation with OpenAI](python/synth_data_with_openai.ipynb)| | Usage | Text | Python file | [Keeping some entities from being anonymized](python/keep_entities.ipynb)| | Usage | Text | LiteLLM Proxy | [PII Masking LLM calls across Anthropic/Gemini/Bedrock/Azure, etc.](docker/litellm.md)| | Usage | Text | AISIX AI Gateway | [Protect LLM requests with Presidio and AISIX AI Gateway](docker/aisix.md)| | Usage | Text | Python Notebook | [YAML based no-code configuration](python/no_code_config.ipynb) | | Usage | Text | Python file | [Using GLiNER within Presidio](python/gliner.md) | | Usage | | REST API (postman) | [Presidio as a REST endpoint](docker/index.md)| | Deployment | | App Service | [Presidio with App Service](deployments/app-service/index.md)| | Deployment | | Kubernetes | [Presidio with Kubernetes](deployments/k8s/index.md)| | Deployment | | Spark/Azure Databricks | [Presidio with Spark](deployments/spark/index.md)| | Deployment | | OpenTelemetry/Grafana | [Redacting Telemetry with Presidio](deployments/redacting-telemetry/index.md)| | Deployment | | Data Protection toolkit for OpenAI | [Data Protection toolkit for OpenAI](deployments/openai-anonymaztion-and-deanonymaztion-best-practices/index.md)| | Deployment | | Spark with Microsoft Fabric | [Presidio with Fabric](fabric/index.md)| | Deployment | | Azure Data Factory with App Service | [ETL for small dataset](deployments/data-factory/presidio-data-factory.md#option-1-presidio-as-an-http-rest-endpoint) | | Deployment | | Azure Data Factory with Databricks | [ETL for large datasets](deployments/data-factory/presidio-data-factory.md#option-2-presidio-on-azure-databricks) | | ADF Pipeline | | Azure Data Factory | [Add Presidio as an HTTP service to your Azure Data Factory](deployments/data-factory/presidio-data-factory-template-gallery-http.md) | | ADF Pipeline | | Azure Data Factory | [Add Presidio on Databricks to your Azure Data Factory](deployments/data-factory/presidio-data-factory-template-gallery-databricks.md) | | Demo | | Streamlit app | [Create a simple demo app using Streamlit](python/streamlit/index.md) --- ### Structured Index (`docs/structured/index.md`) # Presidio structured ## Description The Presidio structured package is a flexible and customizable framework designed to identify and protect structured sensitive data. This tool extends the capabilities of Presidio, focusing on structured data formats such as tabular formats and semi-structured formats (JSON). It leverages the detection capabilities of Presidio-Analyzer to identify columns or keys containing personally identifiable information (PII), and establishes a mapping between these column/keys names and the detected PII entities. Following the detection, Presidio-Anonymizer is used to apply de-identification techniques to each value in columns identified as containing PII, ensuring the sensitive data is appropriately protected. Note that sensitive data might not be automatically detected in some cases. Consequently, additional systems and protections should be employed. ## Installation ### As a python package To install the `presidio-structured` package, run the following command: ```sh pip install presidio-structured ``` #### Getting started ##### Example 1: Anonymizing DataFrames ```python import pandas as pd from presidio_structured import StructuredEngine, PandasAnalysisBuilder from presidio_anonymizer.entities import OperatorConfig from faker import Faker # optionally using faker as an example # Initialize the engine with a Pandas data processor (default) pandas_engine = StructuredEngine() # Create a sample DataFrame sample_df = pd.DataFrame({'name': ['John Doe', 'Jane Smith'], 'email': ['john.doe@example.com', 'jane.smith@example.com']}) # Generate a tabular analysis which describes PII entities in the DataFrame. tabular_analysis = PandasAnalysisBuilder().generate_analysis(sample_df) # Define anonymization operators fake = Faker() operators = { "PERSON": OperatorConfig("replace", {"new_value": "REDACTED"}), "EMAIL_ADDRESS": OperatorConfig("custom", {"lambda": lambda x: fake.safe_email()}) } # Anonymize DataFrame anonymized_df = pandas_engine.anonymize(sample_df, tabular_analysis, operators=operators) print(anonymized_df) ``` ##### Example 2: Anonymizing JSON Data ``` /* Detailed source-code truncated for AI context efficiency. */ ``` A more detailed sample can be found here: - #### Selection Strategy for Entity Detection in Tabular Data - **Most Common (default):** Identifies the most frequently occurring PII entity in a data column or field. - **Highest Confidence:** Selects PII entities based on the highest confidence scores, irrespective of their occurrence frequency. - **Mixed:** Combines the strengths of both the above strategies. It selects the entity with the highest confidence score if that score exceeds a specified threshold (controlled by `mixed_strategy_threshold`); otherwise, it defaults to the most common entity. ##### Usage Specify the `selection_strategy` and optionally the `mixed_strategy_threshold` in the `generate_analysis()` method: ```python # Generate a tabular analysis using the most common strategy tabular_analysis = PandasAnalysisBuilder().generate_analysis(sample_df) # Generate a tabular analysis using the highest confidence strategy tabular_analysis = PandasAnalysisBuilder().generate_analysis(sample_df, selection_strategy="highest_confidence") # Generate a tabular analysis using the mixed strategy tabular_analysis = PandasAnalysisBuilder().generate_analysis(sample_df, selection_strategy="mixed", mixed_strategy_threshold=0.75) ``` #### Future work - Improve support for datasets with mixed free-text and structure data (e.g. some columns contain free text) - Add support for the detection of sensitive column names - PySpark implementation - Integration of additional anonymization techniques such as K-Anonymity and Differential Privacy. Contributions are welcome! Please refer to the [Contributing Guide](https://github.com/data-privacy-stack/presidio/blob/main/CONTRIBUTING.md). #### More information - [API documentation](../api/structured_python.md) - [Sample code](../samples/python/example_structured.ipynb) - [Join the discussion](https://github.com/data-privacy-stack/presidio/discussions?discussions_q=structured) - [Relevant issues on GitHub](https://github.com/data-privacy-stack/presidio/issues?q=is%3Aissue+label%3Astructured-data) --- ### Tutorial Index (`docs/tutorial/index.md`) # Tutorial: Customization in Presidio This tutorial covers different customization use cases to: 1. Adapt Presidio to detect new types of PII entities. 2. Adapt Presidio to detect PII entities in a new language. 3. Embed new types of detection modules into Presidio, to improve the coverage of the service. 4. Operate on identified entities: simple de-identification, custom operators and encryption. ## Table of contents - [Getting started](00_getting_started.md) - [Deny-list based recognizers](01_deny_list.md) - [Regex based PII recognition](02_regex.md) - [Rule based logic recognizer](03_rule_based.md) - [Supporting new models and languages](05_languages.md) - [Calling an external service for PII detection](04_external_services.md) - [Using context words](06_context.md) - [Tracing the decision process](07_decision_process.md) - [Loading recognizers from file](08_no_code.md) - [Ad-Hoc recognizers](09_ad_hoc.md) - [Simple anonymization](10_simple_anonymization.md) - [Custom anonymization](11_custom_anonymization.md) - [Encryption/Decryption](12_encryption.md) --- ### Recipes German Language Support README (`docs/recipes/german-language-support/README.md`) # German Language Support > **Domain**: Healthcare, Legal, Finance, General > **Data Type**: German-language free text (medical documents, contracts, invoices, ID documents) > **Goal**: Detect German PII and sensitive identifiers using Presidio's built-in German recognizers with a bilingual spaCy NLP engine ## Overview **Domain**: Healthcare / Legal / Finance **Data Type**: German-language documents **Goal**: Detect German PII — including healthcare identifiers, tax numbers, official document numbers, and vehicle plates — using the German spaCy model alongside the English model so that bilingual (EN + DE) documents are handled correctly. This recipe provides: - `spacy_en_de.yaml` — a ready-to-use NLP engine configuration that loads both `en_core_web_lg` and `de_core_news_md` - An overview of all German-specific recognizers available in Presidio ## Quick Start ### Prerequisites ```bash pip install presidio-analyzer presidio-anonymizer python -m spacy download en_core_web_lg python -m spacy download de_core_news_md ``` ### Sample Data ```python sample_text = """ Sehr geehrter Herr Müller, Ihre Krankenversicherungsnummer (KVNR): A123456780 Steuer-IdNr.: 86095742719 Arztnummer (LANR): 123456601 Betriebsstättennummer (BSNR): 021234568 USt-IdNr.: DE136695976 Führerscheinnummer: BO12345678A Reisepassnummer: C01234565 """ ``` ### Basic Configuration ```python from presidio_analyzer import AnalyzerEngine from presidio_analyzer.nlp_engine import NlpEngineProvider from presidio_anonymizer import AnonymizerEngine # Load the bilingual EN + DE spaCy configuration provider = NlpEngineProvider(conf_file="spacy_en_de.yaml") nlp_engine = provider.create_engine() analyzer = AnalyzerEngine( nlp_engine=nlp_engine, supported_languages=["en", "de"], ) anonymizer = AnonymizerEngine() # Analyze German text results = analyzer.analyze(text=sample_text, language="de") anonymized = anonymizer.anonymize(text=sample_text, analyzer_results=results) print(anonymized.text) ``` ## Approach Presidio ships pattern-based recognizers for 13 German entity types (see table below). Each recognizer targets a single entity, uses `\b`-anchored regex patterns with base confidence between 0.2 and 0.5, and relies on: 1. **Context words** (German terminology near the match) to boost confidence 2. **Check digit validation** where the official specification defines one (KVNR, Rentenversicherungsnummer, Steuer-IdNr., LANR) The spaCy `de_core_news_md` model adds named-entity recognition for PERSON, LOCATION, and ORGANIZATION on top of the pattern recognizers. ### Supported German Entities | Entity | Name | Check digit | |--------|------|-------------| | `DE_TAX_ID` | Steueridentifikationsnummer | ✅ ISO 7064 Mod 11,10 | | `DE_TAX_NUMBER` | Steuernummer (Länder format) | – | | `DE_SOCIAL_SECURITY` | Rentenversicherungsnummer | ✅ DRV algorithm | | `DE_HEALTH_INSURANCE` | Krankenversicherungsnummer (KVNR) | ✅ GKV-Spitzenverband | | `DE_PASSPORT` | Reisepassnummer | – | | `DE_ID_CARD` | Personalausweisnummer | – | | `DE_KFZ` | Kfz-Kennzeichen | – | | `DE_PLZ` | Postleitzahl | – | | `DE_HANDELSREGISTER` | Handelsregisternummer | – | | `DE_LANR` | Lebenslange Arztnummer | ✅ KBV weights algorithm | | `DE_BSNR` | Betriebsstättennummer | – | | `DE_VAT_ID` | Umsatzsteuer-Identifikationsnummer | – | | `DE_FUEHRERSCHEIN` | Führerscheinnummer (post-2013) | – | ## Results Formal evaluation against a labelled German dataset has not yet been performed. To benchmark this recipe follow the [Presidio Research evaluation workflow](https://github.com/data-privacy-stack/presidio-research/blob/master/notebooks/4_Evaluate_Presidio_Analyzer.ipynb): 1. Generate synthetic German text with the [data generator](https://github.com/data-privacy-stack/presidio-research/blob/master/notebooks/1_Generate_data.ipynb) 2. Configure the analyzer with `spacy_en_de.yaml` 3. Run the evaluator and report precision / recall / F₂ / latency **Precision**: TBD **Recall**: TBD **F₂ Score**: TBD **Latency**: TBD ### Key Findings - Recognizers with check digit validation (KVNR, RVNR, Steuer-IdNr., LANR) achieve very low false-positive rates even on ambiguous digit strings. - Recognizers without a checksum (BSNR, PLZ, KFZ) rely heavily on context words; setting `score_threshold=0.5` when no context is present is recommended. - The `DE_PLZ` (postal code) and `DE_KFZ` (vehicle plate) recognizers overlap with generic patterns; use the `entities` parameter to restrict detection when only specific entity types are needed. ## Tips for Others - **Set `score_threshold`** to 0.4–0.5 for production use to filter out low-confidence pattern-only matches from context-free digit strings. - **Use the `entities` parameter** to limit detection to the entity types relevant to your domain (e.g. only healthcare identifiers in clinical notes). - **Pre-2013 Führerschein** numbers use locally defined, non-standardized formats that are not covered by `DE_FUEHRERSCHEIN`; handle them with a custom recognizer if needed. - **DE_TELEMATIK_ID** was evaluated and rejected as a generic recognizer: the format (`\d{1,2}-`) is too ambiguous for reliable free-text detection. --- **Author**: MvdB **Date**: 2026-03-19 --- ### Samples Deployments Index (`docs/samples/deployments/index.md`) # Example deployments - [Azure App Service](app-service/index.md) - [Kubernetes](k8s/index.md) - [Spark/Azure Databricks](spark/index.md) - [Azure Data Factory](data-factory/index.md) - [Data Protection toolkit for OpenAI](openai-anonymaztion-and-deanonymaztion-best-practices/index.md) --- ### Samples Docker Index (`docs/samples/docker/index.md`) # Using Presidio in Docker ## Description Presidio can expose REST endpoints for each service using Flask and Docker. Follow the [installation guide](https://github.com/data-privacy-stack/presidio/blob/V2/docs/installation.md#using-docker) to learn how to install and run presidio-analyzer and presidio-anonymizer using docker. ## Postman collection This repository contains a postman collection with sample REST API request for each service. Follow [this tutorial](https://learning.postman.com/docs/getting-started/importing-and-exporting-data/#importing-data-into-postman) to learn how to export the sample requests into postman 1. Download Presidio Analyzer postman requests 2. Download Presidio Anonymizer postman requests ## Sample API Calls ### Simple Text Analysis ```curl curl -X POST http://localhost:5002/analyze -H "Content-type: application/json" --data "{ \"text\": \"John Smith drivers license is AC432223\", \"language\" : \"en\"}" ``` ### Simple Text Anonymization ```curl curl -X POST http://localhost:5001/anonymize -H "Content-type: application/json" --data "{\"text\": \"hello world, my name is Jane Doe. My number is: 034453334\", \"analyzer_results\": [{\"start\": 24, \"end\": 32, \"score\": 0.8, \"entity_type\": \"NAME\"}, { \"start\": 48, \"end\": 57, \"score\": 0.95,\"entity_type\": \"PHONE_NUMBER\" }], \"anonymizers\": {\"DEFAULT\": { \"type\": \"replace\", \"new_value\": \"ANONYMIZED\" },\"PHONE_NUMBER\": { \"type\": \"mask\", \"masking_char\": \"*\", \"chars_to_mask\": 4, \"from_end\": true }}}" ``` --- ### Samples Fabric Index (`docs/samples/fabric/index.md`) # Presidio with Fabric This folder contains guides and samples for running Presidio in Microsoft Fabric notebooks with Spark for scalable PII detection and anonymization. ## Contents 1. [Environment Setup](env_setup.md) - How to set up your Fabric environment with the required dependencies. 2. [Presidio and Spark Notebook](./artifacts/presidio_and_spark.ipynb) - The sample notebook demonstrating PII detection and anonymization using Presidio with Spark. 3. [Sample Data](./fabric_sample_data.csv) - Example CSV data for testing the PII detection and anonymization workflow. ## Overview Microsoft Fabric provides a unified analytics platform, and these samples demonstrate how to leverage Presidio's PII detection and anonymization capabilities at scale using Fabric's Spark processing. The integration enables data engineers and analysts to: - Detect PII in large datasets using distributed Spark processing - Anonymize sensitive information in a scalable manner - Enhance data privacy compliance for analytics workloads ## What You'll Accomplish By the end of this guide, you'll have a working Presidio implementation in Fabric that can: - Automatically detect various types of PII in your data at scale - Transform sensitive data while preserving its analytical value - Scale to handle larger datasets through Spark's distributed processing - Integrate with your existing Fabric data workflows You can easily incorporate this solution into your Fabric pipelines for scheduled runs and integrated workflows, or extend it with custom detection rules and anonymization methods. ## Results Preview Here's what you'll achieve when running the sample notebook: ### PII Detection Results Quickly identify what types of sensitive information exist in your data, helping you understand your privacy exposure: ### Anonymized Data Transform your data by masking or replacing sensitive information while keeping its structure intact: ### Scale Testing See how the solution performs with larger datasets— for understanding how it will handle your production workloads: ### Delta Table Output Save your anonymized data directly to a Delta table, making it immediately available for downstream analytics while maintaining privacy: --- ### Samples Deployments App Service Index (`docs/samples/deployments/app-service/index.md`) # Deploy presidio services to an Azure App Service Presidio containers can be hosted on an [Azure App Service](https://docs.microsoft.com/en-us/azure/app-service/). Azure App Service provides a managed production environment, which supports docker containers and devops optimizations. It is a global scale service with built in security and compliance features that fits multiple cloud workloads. The presidio team uses Azure App Service for both its development environment and the presidio demo website. ## Deploy Presidio services to Azure Use the following button to deploy presidio services to your Azure subscription. [](https://portal.azure.com/#create/Microsoft.Template/uri/https%3A%2F%2Fraw.githubusercontent.com%2Fdata-privacy-stack%2Fpresidio%2Fmain%2Fdocs%2Fsamples%2Fdeployments%2Fapp-service%2Fpresidio-services.json) ## Deploy using command-line script The following script can be used alternatively to the ARM template deployment above. It sets up the same components which are required for each of the presidio services (analyzer and anonymizer) as the template. ## Basic setup ```bash RESOURCE_GROUP= APP_SERVICE_NAME= LOCATION= APP_SERVICE_SKU= IMAGE_NAME=ghcr.io/data-privacy-stack/presidio-analyzer:latest # the following parameters are only required if you build and deploy your own containers from a private registry ACR_USER_NAME= ACR_USER_PASSWORD= # create the resource group az group create --name $RESOURCE_GROUP # create the app service plan az appservice plan create --name $APP_SERVICE_NAME-plan --resource-group $RESOURCE_GROUP \ --is-linux --location $LOCATION --sku $APP_SERVICE_SKU # create the web app using the official presidio images az webapp create --name $APP_SERVICE_NAME --plan $APP_SERVICE_NAME-plan \ --resource-group $RESOURCE_GROUP -i $IMAGE_NAME # or alternatively, if building presidio and deploying from a private container registry az webapp create --name $APP_SERVICE_NAME --plan $APP_SERVICE_NAME-plan \ --resource-group $RESOURCE_GROUP -i $IMAGE_NAME -s $ACR_USER_NAME -w $ACR_USER_PASSWORD ``` ## Blocking network access Use the following script to restrict network access for a specific ip such as your computer, a front-end website or an API management. ```bash FRONT_END_IP_RANGE=[front end ip range] az webapp config access-restriction add --resource-group $RESOURCE_GROUP --name $APP_SERVICE_NAME \ --rule-name 'Front-end allow rule' --action Allow --ip-address $FRONT_END_IP_RANGE --priority 100 ``` Further network isolation, using virtual networks, is possible using an Isolated tier of Azure App Service. ## Configure App Service Logging ### Logging to the App Service File System ```bash az webapp log config --name $APP_SERVICE_NAME --resource-group $RESOURCE_GROUP \ --application-logging filesystem --detailed-error-messages true \ --docker-container-logging filesystem --level information ``` ### Logging to Log Analytics Workspace ```bash LOG_ANALYTICS_WORKSPACE_RESROUCE_GROUP= LOG_ANALYTICS_WORKSPACE_NAME= # create a log analytics workspace az monitor log-analytics workspace create --resource-group $LOG_ANALYTICS_WORKSPACE_RESROUCE_GROUP --workspace-name $LOG_ANALYTICS_WORKSPACE_NAME # query the log analytics workspace id LOG_ANALYTICS_WORKSPACE_ID=$(az monitor log-analytics workspace show --resource-group $LOG_ANALYTICS_WORKSPACE_RESROUCE_GROUP --workspace-name $LOG_ANALYTICS_WORKSPACE_NAME --query id -o tsv) # query the app service id APP_SERVICE_ID=$(az monitor log-analytics workspace show --resource-group $RESOURCE_GROUP --name $APP_SERVICE_NAME --query id -o tsv) # create the diagnostics settings az monitor diagnostic-settings create --name $APP_SERVICE_NAME-diagnostics --resource / $APP_SERVICE_ID --logs '[{"category": "AppServicePlatformLogs","enabled": true}, {"category": "AppServiceConsoleLogs", "enabled": true}]' --metrics '[{"category": "AllMetrics","enabled": true}]' --workspace $LOG_ANALYTICS_WORKSPACE_ID ``` ## Using an ARM template Alternatively, you can use the provided ARM template which can deploy either both or any of the presidio services. Note that while Log Analytics integration with Azure App Service is in preview, the ARM template deployment will not create a Log Analytics resource or configure the diagnostics settings from the App Service to a Log Analytics workspace. To deploy the app services using the provided ARM template, fill in the provided values.json file with the required values and run the following script. ```bash az deployment group create --resource-group $RESOURCE_GROUP --template-file presidio-services.json --parameters @values.json ``` --- ### Samples Deployments Data Factory Index (`docs/samples/deployments/data-factory/index.md`) # Anonymize PII entities with Azure Data Factory You can build data anonymization ETL pipelines using Azure Data Factory (ADF) and Presidio. This section provides instructions on how to leverage presidio in Azure Data Factory: 1. [Complete samples](presidio-data-factory.md) - Setup Azure Data Factory and Presidio in a single deployment and run the samples. 2. [Using Azure Data Factory Template Gallery to anonymize text files](presidio-data-factory-template-gallery-http.md) - If you already have an instance of Azure Data Factory and would like to use the built in data-anonymization-with-presidio template to anonymize text files. 3. [Using Azure Data Factory Template Gallery to anonymize datasets](presidio-data-factory-template-gallery-databricks.md) - If you already have an instance of Azure Data Factory and would like to use the built in data-anonymization-with-presidio template to anonymize csv datasets. --- ### Samples Deployments K8s Index (`docs/samples/deployments/k8s/index.md`) # Deploy presidio to Kubernetes You can install Presidio locally using [KIND](https://github.com/kubernetes-sigs/kind), as a service in [Kubernetes](https://kubernetes.io/) or [AKS](https://docs.microsoft.com/en-us/azure/aks/intro-kubernetes). - [Deploy locally using KIND](#deploy-locally-with-kind) - [Deploy with Kubernetes](#presidio-as-a-service-with-kubernetes) - [Prerequisites](#prerequisites) - [Step by Step Deployment with customizable parameters](#step-by-step-deployment-with-customizable-parameters) ## Deploy locally with KIND [KIND (Kubernetes IN Docker)](https://github.com/kubernetes-sigs/kind). 1. Install [Docker](https://docs.docker.com/install/). 2. Clone Presidio. 3. Run the following script, which will use KIND (Kubernetes emulation in Docker) ```sh cd docs/samples/deployments/k8s/deployment/ ./run-with-kind.sh ``` 4. Wait and verify all pods are running: ```sh kubectl get pod -n presidio ``` 5. Port forwarding of HTTP requests to the API micro-service will be done automatically. In order to run manual: ```sh kubectl port-forward 8080:8080 -n presidio ``` ## Presidio As a Service with Kubernetes ### Prerequisites 1. A Kubernetes 1.18+ cluster with [RBAC](https://kubernetes.io/docs/reference/access-authn-authz/rbac/) enabled. If you are using [AKS](https://docs.microsoft.com/en-us/azure/aks/intro-kubernetes) RBAC is enabled by default. !!! note: Note Note the pod's resource requirements (CPU and memory) and plan the cluster accordingly. 2. [kubectl](https://kubernetes.io/docs/reference/kubectl/overview/) installed. Verify you can communicate with the cluster by running: ```sh kubectl version ``` 3. Local [helm](https://helm.sh/) client. 4. **Optional** - Container Registry - such as [ACR](https://docs.microsoft.com/en-us/azure/container-registry/container-registry-intro). Only needed if you are using your own Presidio images and not the default ones from [GitHub Container Registry](https://github.com/orgs/data-privacy-stack/packages). 5. Recent presidio repo is cloned on your local machine. ### Step by step deployment with customizable parameters 1. Install [Helm](https://github.com/kubernetes/helm) with [RBAC](https://github.com/kubernetes/helm/blob/master/docs/rbac.md#tiller-and-role-based-access-control). 2. Optional - Ingress controller for presidio API, e.g., [NGINX](https://docs.microsoft.com/en-us/azure/aks/ingress-tls). > **NOTE**: > Presidio is deployed with an ingress controller by default, and uses `nginx` as `ingress.class`. To change > this behavior, deploy the helm chart with `ingress.enabled=false`. 3. Deploy from `/docs/samples/deployments/k8s/charts/presidio` ```sh # Choose a namespace and ensure it is created NAMESPACE=presidio # Choose the tag from ghcr.io/data-privacy-stack, e.g. `latest` TAG=latest # Choose a name for the deployment NAME= # Use Helm to install all required components helm install $NAME . --set tag=$TAG --namespace $NAMESPACE # If you have your own images in a separate ACR, run DOCKER_REGISTRY= helm install $NAME . --set registry=$DOCKER_REGISTRY,tag=$TAG --namespace $NAMESPACE ``` --- ### Samples Deployments Openai Anonymaztion And Deanonymaztion Best Practices Index (`docs/samples/deployments/openai-anonymaztion-and-deanonymaztion-best-practices/index.md`) # Invisio - Data Protection toolkit for OpenAI ### Sample Overview In this sample, we will demonstrate best practices for creating meaningful conversations with OpenAI while ensuring that all Personally Identifiable Information (PII) is anonymized before being sent to the OpenAI API. Additionally, we will cover the process of de-anonymizing the data upon receiving the response. ### Problem Statement The primary challenge we aim to address is: how can we ensure that the responses processed by OpenAI, which are also anonymized, remain meaningful to the user? To tackle this issue, we provide a reversible and repeatable ***AI Design Win*** which handles the anonymization process for sensitive data. In other words, we implemented a de-anonymization middleware that maps the anonymized entities, allowing us to de-anonymize the data effectively upon receiving it back. --- ### Core Concept The core concept of the toolkit revolves around sessions, which encapsulate the anonymization context. This context defines how each Personally Identifiable Information (PII) entity is anonymized. By maintaining consistency within the same session, the toolkit ensures that the same PII is always anonymized in the same manner. Moreover, it facilitates the de-anonymization of data as needed. This repository serves as a comprehensive sample that you can use as a foundation and extend according to your specific requirements. The toolkit is particularly useful in "Chat with your data" scenarios within enterprises, ensuring that personally identifiable information (PII) present in corporate documents is not exposed to users during queries. The following sections will describe the required infrastructure and code to achieve the above functionality. ### Workflow Example The following is the desired output. We can input PII data anonymized to the LLM input and return the replaced entities back to the original in the response back to the user. - User Input: "Hello world, my name is Jane Doe. My number is: 034453334" - Anonymized LLM Input: "Hello world, my name is [PERSON]. My number is: [PHONE_NUMBER]" - LLM Response: "Hey [PERSON], nice to meet you!" - De-anonymized User Response: "Hey Jane Doe, nice to meet you!" ### Demo Client Application The [demo client application](src/client_app/README.md) showcases the capabilities of the Anonymization Toolkit within the context of an "Anonymous Chat with a Large Language Model (LLM)" use case. In this scenario, the toolkit anonymizes Personally Identifiable Information (PII) in the messages sent to the LLM and subsequently de-anonymizes the responses received from it. This approach ensures that the LLM never has access to actual PII, while still facilitating meaningful conversations. ### Architecture and Infrastructure The solution consists of the following components: - Api (Deployed in AKS) - Client - Redis ## Code ### Anonymization First we need to setup our `Presidio` engines to aid with analysis, anonymization and deanonymization. The following code can exist as part of a service available to the appropriate api endpoints. ```python configuration = { "nlp_engine_name": "spacy", "models": [ {"lang_code": "en", "model_name": "en_core_web_lg"}, {"lang_code": "nl", "model_name": "nl_core_news_sm"}, {"lang_code": "es", "model_name": "es_core_news_sm"}, ], } provider = NlpEngineProvider(nlp_configuration=configuration) nlp_engine = provider.create_engine() self.analyzer = AnalyzerEngine( nlp_engine=nlp_engine, supported_languages=["en", "nl", "es"] ) self.anonymizer = AnonymizerEngine() self.anonymizer.add_anonymizer(InstanceCounterAnonymizer) self.deanonymizer = DeanonymizeEngine() self.deanonymizer.add_deanonymizer(InstanceCounterDeanonymizer) ``` `InstanceCounterAnonymizer` and `InstanceCounterDeanonymizer` taken from [these examples](https://data-privacy-stack.github.io/presidio/samples/python/pseudonymization/). We anonymize data with the following: ``` /* Detailed source-code truncated for AI context efficiency. */ ``` and deanonymize with: ``` /* Detailed source-code truncated for AI context efficiency. */ ``` The above code would be called with: ```python # anonymize example anonymized_text, new_entity_mappings = presidio_service.anonymize_text( request.session_id, request.text, request.language, entity_mappings ) # deanonymize example deanonymized_text = presidio_service.deanonymize_text( request.session_id, request.text, entity_mappings ) ``` Where: - `request.session_id` - A unique session identifier, e.g. guid. - `request.text` - The text to be anonymized or deanonymized. - `request.language` - The language code to analyze with. Note this requires the appropriate language model to be setup during the configuration. See code above. Example codes, `en`, `es`, `nl`. - `entity_mappings` - A dictionary with the structure of: ```python {'LOCATION': {'Redmond': ''}, 'PERSON': {'Joe Bloggs': ''}} ``` - Each item is the entity type and the value contains each instance of the entity and its replacement. - This dictionary is optional. On the first request of a session we wont have any mappings, subsequent requests can then build upon this dictionary to maintain knowledge of previous request mappings. ### Repository structure - [src/api](./src/api) - The toolkit API, which is responsible for performing anonymization and deanonymization, as well as managing user sessions in a persistent storage. - [src/client_app/](src/client_app) - The demo client app, which enables users to engage in an anonymous chat with an LLM. The app utilizes the toolkit API to hide sensitive data from the LLM. It can be run either in a terminal or as a website. - [infrastructure/](infrastructure) - Bicep definition for Azure infrastructure required to support deployment to a Kubernetes (k8s) cluster. - [deployments/](deployments) - The manifests necessary to deploy the Toolkit API and the Demo to a k8s cluster. - [spikes/](spikes) - Experimental code, such as a Streamlit chat application that can interact with the toolkit API in a similar manner to the demo client app. --- ### Samples Deployments Redacting Telemetry Index (`docs/samples/deployments/redacting-telemetry/index.md`) # Redacting Telemetry with Presidio **A practical example of redacting PII in logs using Presidio.** > ⚠️ This is a proof-of-concept demonstration, not a production-ready solution. ## What This Demo Shows This sample demonstrates **client-side PII masking** before logs are exported to observability platforms like Grafana/Loki. The application calls Presidio to detect and mask PII (names, emails, SSNs, phone numbers) before logging. ## Considerations and Out of Scope This demo intentionally does NOT cover: - Production deployment patterns - Configurations, NLP model selection, or custom entity recognizers - Reliability & Performance optimization - Alternative architectures (sidecar, (otel-)collector-side masking, et cetera) **Architecture:** ``` Application → Presidio (mask PII) → OpenTelemetry Collector → Loki/Tempo → Grafana ``` ## Quick Start ### Prerequisites - Docker and Docker Compose ### Run the Demo ```bash cd docs/samples/deployments/redacting-telemetry docker compose up -d # Wait ~60 seconds for Presidio to download models # Then open Grafana ``` **Access Grafana:** http://localhost:3000 Navigate to **Dashboards → PII Redaction Demo** to see: - **Unredacted Logs Panel**: Raw PII (names, emails, SSNs) - **Redacted Logs Panel**: PII replaced with types (``, ``, ``) The dashboard auto-refreshes every 5 seconds. ## What's Running | Service | Purpose | Port | | ----------------------- | ------------------------------- | ---------- | | **presidio-analyzer** | Detects PII using NLP | 5002 | | **presidio-anonymizer** | Masks detected PII | 5001 | | **pii-demo-app** | FastAPI app generating PII logs | 8000 | | **otel-collector** | Receives and forwards telemetry | 4317, 4318 | | **loki** | Log aggregation | 3100 | | **tempo** | Distributed tracing | 3200 | | **grafana** | Visualization | 3000 | ## How It Works The demo app (`app/main.py`) generates logs with PII, then calls Presidio to mask them: ```python from presidio_client import mask_pii # Original log unredacted = "User registered: John Doe, SSN: 123-45-6789, Email: john@example.com" logger.info(f"[UNREDACTED] {unredacted}") # Redacted log redacted = mask_pii(unredacted) # Calls Presidio logger.info(f"[REDACTED] {redacted}") # Output: [REDACTED] User registered: , SSN: , Email: ``` The `presidio_client.py` makes two HTTP calls: 1. **Analyze**: Detect PII entities 2. **Anonymize**: Replace with placeholders ## Use in Your Application ### Python Example Copy `app/presidio_client.py` to your project: ```python from presidio_client import mask_pii # Before logging sensitive data sensitive = f"Customer {name} called from {phone}" redacted = mask_pii(sensitive) logger.info(redacted) # "Customer called from " ``` ## Cleanup ```bash docker compose down # Stop services docker compose down -v # Stop and delete data ``` --- ### Samples Deployments Spark Index (`docs/samples/deployments/spark/index.md`) # Anonymize PII using Presidio on Spark You can leverages presidio to perform data anonymization as part of spark notebooks. The following sample uses [Azure Databricks](https://docs.microsoft.com/en-us/azure/databricks/) and simple text files hosted on [Azure Blob Storage](https://docs.microsoft.com/en-us/azure/storage/blobs/). However, it can easily change to fit any other scenario which requires PII analysis or anonymization as part of spark jobs. **Note** that this code works for Databricks runtime 8.1 (spark 3.1.1) and the libraries described [here](https://docs.microsoft.com/en-us/azure/databricks/release-notes/runtime/8.1). ## The basics of working with Presidio in Spark A typical use case of Presidio in Spark is transforming a text column in a data frame, by anonymizing its content. The following code sample, a part of [transform presidio notebook](./notebooks/01_transform_presidio.py), is the basis of the e2e sample which uses Azure Databricks as the Spark environment. ```python anonymized_column = "value" # name of column to anonymize analyzer = AnalyzerEngine() anonymizer = AnonymizerEngine() # broadcast the engines to the cluster nodes broadcasted_analyzer = sc.broadcast(analyzer) broadcasted_anonymizer = sc.broadcast(anonymizer) # define a pandas UDF function and a series function over it. def anonymize_text(text: str) -> str: analyzer = broadcasted_analyzer.value anonymizer = broadcasted_anonymizer.value analyzer_results = analyzer.analyze(text=text, language="en") anonymized_results = anonymizer.anonymize( text=text, analyzer_results=analyzer_results, operators={ "DEFAULT": OperatorConfig("replace", {"new_value": ""}) }, ) return anonymized_results.text def anonymize_series(s: pd.Series) -> pd.Series: return s.apply(anonymize_text) # define a the function as pandas UDF anonymize = pandas_udf(anonymize_series, returnType=StringType()) # apply the udf anonymized_df = input_df.withColumn( anonymized_column, anonymize(col(anonymized_column)) ) ``` ## Pre-requisites If you do not have an instance of Azure Databricks, follow through with the following steps to provision and setup the required infrastructure. If you do have a Databricks workspace and a cluster you wish to configure to run Presidio, jump over to the [Configure an existing cluster](#configure-an-existing-cluster) section. ### Deploy Infrastructure Provision the Azure resources by running the following script. ``` bash export RESOURCE_GROUP=[resource group name] export STORAGE_ACCOUNT_NAME=[storage account name] export STORAGE_CONTAINER_NAME=[blob container name] export DATABRICKS_WORKSPACE_NAME=[databricks workspace name] export DATABRICKS_SKU=[basic/standard/premium] export LOCATION=[location] # Create the resource group az group create --name $RESOURCE_GROUP --location $LOCATION # Use ARM template to build the resources and get back the workspace URL deployment_response=$(az deployment group create -g $RESOURCE_GROUP --template-file ./docs/samples/deployments/spark/arm-template/databricks.json --parameters location=$LOCATION workspaceName=$DATABRICKS_WORKSPACE_NAME storageAccountName=$STORAGE_ACCOUNT_NAME containerName=$STORAGE_CONTAINER_NAME) export DATABRICKS_WORKSPACE_URL=$(echo $deployment_response | jq -r ".properties.outputs.workspaceUrl.value") export DATABRICKS_WORKSPACE_ID=$(echo $deployment_response | jq -r ".properties.outputs.workspaceId.value") ``` ### Setup Databricks The following script will setup a new cluster in the databricks workspace and prepare it to run presidio anonymization jobs. Once finished, the script will output an access key which you can use when working with databricks cli. ``` bash sh ./scripts/configure_databricks.sh ``` ### Configure an existing cluster Only follow through with the steps in this section if you have an existing databricks workspace and clsuter you wish to configure to run presidio. If you've followed through with the "Deploy Infrastructure" and "Setup Databricks" sections you do not have to run the script in this section. #### Set up secret scope and secrets for storage account Add an Azure Storage account key to secret scope. ``` bash STORAGE_PRIMARY_KEY=[Primary key of storage account] databricks secrets create-scope --scope storage_scope --initial-manage-principal users databricks secrets put --scope storage_scope --key storage_account_access_key --string-value "$STORAGE_PRIMARY_KEY" ``` #### Upload or update cluster init scripts Presidio libraries are loaded to the cluster on init. Upload the cluster setup script or add its content to the existing cluster's init script. ```bash databricks fs cp "./setup/startup.sh" "dbfs:/FileStore/dependencies/startup.sh" ``` Setup the cluster to run the [init script](https://docs.microsoft.com/en-us/azure/databricks/clusters/configure#init-scripts). #### Upload presidio notebooks ```bash databricks workspace import_dir "./notebooks" "/notebooks" --overwrite ``` #### Update cluster environment Add the following [environment variables](https://docs.microsoft.com/en-us/azure/databricks/clusters/configure#environment-variables) to your databricks cluster: ```bash "STORAGE_MOUNT_NAME": "/mnt/files" "STORAGE_CONTAINER_NAME": [Blob container name] "STORAGE_ACCOUNT_NAME": [Storage account name] ``` #### Mount the storage container Run the notebook 00_setup to mount the storage account to databricks. ## Running the sample ### Configure Presidio transformation notebook From Databricks workspace, under notebooks folder, open the provided 01_transform_presidio notebook and attach it to the cluster preisidio_cluster. Run the first code-cell and note the following parameters on the top end of the notebook (notebook widgets) and set them accordingly * Input File Format - text (selected). * Input path - a folder on the container where input files are found. * Output Folder - a folder on the container where output files will be written to. * Column to Anonymize - value (selected). ### Run the notebook Upload a text file to the blob storage input folder, using any preferred method ([Azure Portal](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-quickstart-blobs-portal), [Azure Storage Explorer](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-quickstart-blobs-storage-explorer), [Azure CLI](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-quickstart-blobs-cli)). ```bash az storage blob upload --account-name $STORAGE_ACCOUNT_NAME --container $STORAGE_CONTAINER_NAME --file ./[file name] --name input/[file name] ``` Run the notebook cells, the output should be csv files which contain two columns, the original file name, and the anonymized content of that file. --- ### Samples Python Ahds Index (`docs/samples/python/ahds/index.md`) # Azure Health Data Services de-identification Integration ## Introduction The Azure Health Data Services (AHDS) de-identification Service is a cloud-based service that provides advanced natural language processing over raw text. One of its main functions includes Named Entity Recognition (NER), which has the ability to identify different entities in text and categorize them into pre-defined classes or types. This document will demonstrate Presidio integration with the AHDS De-Identification Service. ## Supported entity categories in the Azure Health Data Services de-identification API Azure Health Data Services de-identification supports multiple PII entity categories. The Azure Health Data Services de-identification service runs a predictive model to identify and categorize named entities from an input document. The service's latest version includes the ability to detect personal (PII) and health (PHI) information. A list of all supported entities can be found in the [official documentation](https://learn.microsoft.com/en-us/azure/healthcare-apis/deidentification/overview). ## Prerequisites To use AHDS De-Identification with Presidio, an Azure De-Identification Service resource should first be created under an Azure subscription. Follow the [official documentation](https://learn.microsoft.com/en-us/azure/healthcare-apis/deidentification/quickstart) for instructions. The endpoint, generated once the resource is created, will be used when integrating with AHDS De-Identification, using a Presidio remote recognizer. ### Authentication Setup The integration uses a secure-by-default authentication approach: **Production Mode (Default)**: Uses a restricted credential chain (EnvironmentCredential, WorkloadIdentityCredential, ManagedIdentityCredential) **Development Mode**: Set `ENV=development` to use DefaultAzureCredential for local development with Azure CLI: ```bash export ENV=development az login ``` For more details, see the [AHDS Integration Authentication documentation](../../../ahds_integration.md#authentication). ## Azure Health Data Services de-identification Recognizer [The implementation of a `AzureHealthDeid` recognizer can be found here](https://github.com/data-privacy-stack/presidio/blob/main/presidio-analyzer/presidio_analyzer/predefined_recognizers/third_party/ahds_recognizer.py). ## How to integrate Azure Health Data Services de-identification into Presidio 1. Install the package with the ahds extra: ```sh pip install "presidio-analyzer[ahds]" ``` 2. Define environment variables `AHDS_ENDPOINT` 3. Add the `AzureHealthDeidRecognizer` to the recognizer registry: ```python from presidio_analyzer import AnalyzerEngine from presidio_analyzer.predefined_recognizers import AzureHealthDeidRecognizer ahds = AzureHealthDeidRecognizer() analyzer = AnalyzerEngine() analyzer.registry.add_recognizer(ahds) analyzer.analyze(text="My email is email@email.com", language="en") ``` --- **See also:** For a full surrogate integration example, see [example_ahds_surrogate.py](example_ahds_surrogate.py) --- ### Samples Python Langextract Index (`docs/samples/python/langextract/index.md`) # Language Model-based PII/PHI Detection (Experimental Feature) ## Introduction Presidio supports language model-based PII/PHI detection for flexible entity recognition using language models (LLMs, SLMs, etc.). This approach enables detection of both: - **PII (Personally Identifiable Information)**: Names, emails, phone numbers, SSN, credit cards, etc. - **PHI (Protected Health Information)**: Medical records, health identifiers, etc. (The default approach uses [LangExtract](https://github.com/google/langextract) under the hood to integrate with language model providers.) ## Entity Detection Capabilities Unlike pattern-based recognizers, language model-based detection is flexible and depends on: - The language model being used - The prompt description provided - The few-shot examples configured The default configuration includes examples for common PII/PHI entities such as PERSON, EMAIL_ADDRESS, PHONE_NUMBER, US_SSN, CREDIT_CARD, MEDICAL_LICENSE, and more. **You can customize the prompts and examples to detect any entity types relevant to your use case**. For the default entity mappings and examples, see the [default configuration](https://github.com/data-privacy-stack/presidio/blob/main/presidio-analyzer/presidio_analyzer/conf/langextract_config_ollama.yaml). ## Supported Language Model Providers Presidio supports the following language model providers through LangExtract: 1. **Azure OpenAI** - Cloud-based Azure OpenAI Service (GPT-4o, GPT-4, GPT-3.5-turbo, etc.) 2. **Ollama** - Local language model deployment (open-source models like Gemma, Llama, etc.) ## Choosing Between Azure OpenAI and Ollama | Feature | Azure OpenAI | Ollama | |---------|--------------|--------| | **Deployment** | Cloud (Azure) | Local (on-premises) | | **Cost** | Pay-per-use (tokens) | Free (hardware required) | | **Models** | GPT-4o, GPT-4, GPT-3.5-turbo | Open-source (Gemma, Llama, etc.) | | **Privacy** | Microsoft Azure compliance | Complete data control | | **Setup** | Azure Portal + API key/Managed Identity | Docker/local installation | | **Authentication** | API Key or Managed Identity (RBAC) | None (local) | | **Best For** | Production, enterprise compliance | Development, on-premises requirements | **Recommendations**: - **Use Azure OpenAI** for production workloads requiring enterprise security, compliance (HIPAA, SOC 2, etc.), and managed infrastructure - **Use Ollama** for local development, testing, or when data must stay on-premises ## Language Model-based Recognizer Implementation Presidio provides a hierarchy of recognizers for language model-based PII/PHI detection: - **`LMRecognizer`**: Abstract base class for all language model recognizers (LLMs, SLMs, etc.) - **`LangExtractRecognizer`**: Abstract base class for LangExtract library integration (model-agnostic) - **`AzureOpenAILangExtractRecognizer`**: Concrete implementation for Azure OpenAI Service - [Implementation](https://github.com/data-privacy-stack/presidio/blob/main/presidio-analyzer/presidio_analyzer/predefined_recognizers/third_party/azure_openai_langextract_recognizer.py) - **`BasicLangExtractRecognizer`**: Concrete implementation where ModelConfig is configured from YAML (supporting Ollama, OpenAI, Gemini, and other providers) - [Implementation](https://github.com/data-privacy-stack/presidio/blob/main/presidio-analyzer/presidio_analyzer/predefined_recognizers/third_party/basic_langextract_recognizer.py) --- ## Using Azure OpenAI (Cloud Models) Azure OpenAI provides cloud-based access to OpenAI models (GPT-4o, GPT-4, GPT-3.5-turbo) with enterprise security and compliance features. ### Prerequisites 1. **Install Presidio with LangExtract support**: ```sh pip install "presidio-analyzer[langextract]" ``` 2. **Set up Ollama** You have two options to set up Ollama: **Option 1: Docker Compose** (recommended for CPU) This option requires Docker to be installed on your system. **Where to run:** From the root presidio directory (where `docker-compose.yml` is located) ```bash docker compose up -d ollama docker exec presidio-ollama-1 ollama pull qwen2.5:1.5b docker exec presidio-ollama-1 ollama list ``` **Platform differences:** - **Linux/Mac**: Commands above work as-is - **Windows**: Use PowerShell or CMD, commands are the same If you don't have Docker installed: - Linux: Follow [Docker installation guide](https://docs.docker.com/engine/install/) - Mac: Install [Docker Desktop for Mac](https://docs.docker.com/desktop/install/mac-install/) - Windows: Install [Docker Desktop for Windows](https://docs.docker.com/desktop/install/windows-install/) **Option 2: Native installation** (recommended for GPU acceleration) Follow the [official LangExtract Ollama guide](https://github.com/google/langextract?tab=readme-ov-file#using-local-llms-with-ollama). After installation, pull and run the model: ```bash ollama pull qwen2.5:1.5b ollama run qwen2.5:1.5b ``` > This option provides better performance with GPU acceleration (e.g., on Mac with Metal Performance Shaders or systems with NVIDIA GPUs). > The model must be pulled and run before using the recognizer. The default model is `qwen2.5:1.5b`. 3. **Configuration** (optional): Create your own `ollama_config.yaml` or use the [default configuration](https://github.com/data-privacy-stack/presidio/blob/main//presidio-analyzer/presidio_analyzer/conf/langextract_config_ollama.yaml) ### Usage **Option 1: Enable in configuration file** Enable the recognizer in [`default_recognizers.yaml`](https://github.com/data-privacy-stack/presidio/blob/main/presidio-analyzer/presidio_analyzer/conf/default_recognizers.yaml): ```yaml - name: BasicLangExtractRecognizer enabled: true # Change from false to true ``` Then load the analyzer using this modified configuration file: ```python from presidio_analyzer import AnalyzerEngine from presidio_analyzer.recognizer_registry import RecognizerRegistryProvider # Point to your modified default_recognizers.yaml with Ollama enabled provider = RecognizerRegistryProvider( conf_file="/path/to/your/modified/default_recognizers.yaml" ) registry = provider.create_recognizer_registry() # Create analyzer with the registry that includes Ollama recognizer analyzer = AnalyzerEngine(registry=registry, supported_languages=["en"]) # Analyze text - Ollama recognizer will participate in detection results = analyzer.analyze(text="My email is john.doe@example.com", language="en") ``` **Option 2: Add programmatically** ```python from presidio_analyzer import AnalyzerEngine from presidio_analyzer.predefined_recognizers.third_party.basic_langextract_recognizer import BasicLangExtractRecognizer analyzer = AnalyzerEngine() analyzer.registry.add_recognizer(BasicLangExtractRecognizer()) results = analyzer.analyze(text="My email is john.doe@example.com", language="en") ``` !!! note "Note" The recognizer is disabled by default in `default_recognizers.yaml` to avoid requiring Ollama for basic Presidio usage. Enable it when you have Ollama set up and running. ### Custom Configuration To use a custom configuration file: ```python analyzer.registry.add_recognizer( BasicLangExtractRecognizer(config_path="/path/to/custom_config.yaml") ) ``` ### Configuration Options The `langextract_config_ollama.yaml` file supports the following options: - **`model_id`**: The model to use. - **`provider.name`**: The model provider (eg, ollama, openai) - **`provider.kwargs`**: kwargs to pass to the model provider (eg, `model_url` for Ollama, `base_url` for OpenAI) - **`provider.extract_params`**: Extraction parameters (eg, `use_schema_constraints`, `fence_output`, `temperature`) - **`provider.language_model_params`**: Parameters for the model itself (eg, `timeout`, `num_ctx`) - **`supported_entities`**: PII/PHI entity types to detect - **`entity_mappings`**: Map LangExtract entity classes to Presidio entity names - **`min_score`**: Minimum confidence score (default: `0.5`) See the [configuration file](https://github.com/data-privacy-stack/presidio/blob/main/presidio-analyzer/presidio_analyzer/conf/ollama_config.yaml) for all options. ## Troubleshooting **ConnectionError: "Ollama server not reachable"** - Ensure Ollama is running: `docker ps` or check `http://localhost:11434` - Verify the `model_url` in your configuration matches your Ollama server address **RuntimeError: "Model 'qwen2.5:1.5b' not found"** - Pull the model: `docker exec -it presidio-ollama-1 ollama pull qwen2.5:1.5b` - Or for manual setup: `ollama pull qwen2.5:1.5b` - Verify the model name matches the `model_id` in your configuration --- ## Using Azure OpenAI (Cloud Models) Azure OpenAI provides cloud-based access to OpenAI models (GPT-4o, GPT-4, GPT-3.5-turbo) with enterprise security and compliance features. ### Prerequisites 1. **Install Presidio with LangExtract support**: ```sh pip install "presidio-analyzer[langextract]" ``` This installs `langextract` with OpenAI support, including the OpenAI Python SDK and Azure Identity libraries. 2. **Azure Subscription**: Create one at [azure.microsoft.com](https://azure.microsoft.com) 3. **Azure OpenAI Resource**: - Create an Azure OpenAI resource in [Azure Portal](https://portal.azure.com) - Request access if needed (some regions require approval) - Deploy a model and note the **deployment name** you choose (e.g., "gpt-4", "my-gpt-deployment") 4. **Optional: Download config file** (only if customizing entities/prompts): ```sh # On macOS/Linux/PowerShell: wget https://raw.githubusercontent.com/data-privacy-stack/presidio/main/presidio-analyzer/presidio_analyzer/conf/langextract_config_azureopenai.yaml # Or download manually from: # https://github.com/data-privacy-stack/presidio/blob/main/presidio-analyzer/presidio_analyzer/conf/langextract_config_azureopenai.yaml ``` ### Authentication Options Azure OpenAI supports multiple authentication methods with flexible configuration: #### Option 1: Direct Parameters (Recommended for Most Users) Simplest approach - pass credentials and deployment name as parameters: ```python from presidio_analyzer import AnalyzerEngine from presidio_analyzer.predefined_recognizers import AzureOpenAILangExtractRecognizer # Initialize with deployment name and credentials azure_openai = AzureOpenAILangExtractRecognizer( model_id="gpt-4", # Your Azure deployment name azure_endpoint="https://your-resource.openai.azure.com/", api_key="your-api-key-here", api_version="2024-02-15-preview" # Optional ) analyzer = AnalyzerEngine() analyzer.registry.add_recognizer(azure_openai) results = analyzer.analyze( text="My email is john.doe@example.com and my phone is 555-123-4567", language="en" ) ``` #### Option 2: Environment Variables Use environment variables for credentials: ```python import os from presidio_analyzer import AnalyzerEngine from presidio_analyzer.predefined_recognizers import AzureOpenAILangExtractRecognizer # Set environment variables os.environ["AZURE_OPENAI_ENDPOINT"] = "https://your-resource.openai.azure.com/" os.environ["AZURE_OPENAI_API_KEY"] = "your-api-key-here" # Initialize with just deployment name azure_openai = AzureOpenAILangExtractRecognizer( model_id="gpt-4" # Your Azure deployment name ) analyzer = AnalyzerEngine() analyzer.registry.add_recognizer(azure_openai) results = analyzer.analyze( text="My email is john.doe@example.com and my phone is 555-123-4567", language="en" ) ``` #### Option 3: Managed Identity (Production) **More secure** - No API keys in code, uses Azure RBAC: ```python import os from presidio_analyzer import AnalyzerEngine from presidio_analyzer.predefined_recognizers import AzureOpenAILangExtractRecognizer # Set endpoint (no API key = uses managed identity) os.environ["AZURE_OPENAI_ENDPOINT"] = "https://your-resource.openai.azure.com/" # Initialize without API key (uses managed identity) azure_openai = AzureOpenAILangExtractRecognizer( model_id="gpt-4" # Your Azure deployment name ) analyzer = AnalyzerEngine() analyzer.registry.add_recognizer(azure_openai) results = analyzer.analyze( text="Patient John Smith has SSN 123-45-6789", language="en" ) ``` **Managed Identity Authentication Flow** (Production): When `api_key` is not provided, the provider automatically uses `ChainedTokenCredential` which tries credentials in order: 1. **EnvironmentCredential** - Service principal from environment variables 2. **WorkloadIdentityCredential** - Azure Kubernetes Service workload identity 3. **ManagedIdentityCredential** - Azure VM/App Service managed identity For local development, set `ENV=development` to use `DefaultAzureCredential` instead: ```python import os os.environ["ENV"] = "development" os.environ["AZURE_OPENAI_ENDPOINT"] = "https://your-resource.openai.azure.com/" # AZURE_OPENAI_API_KEY not set - uses DefaultAzureCredential in development mode # (includes Azure CLI, VS Code, etc.) azure_openai = AzureOpenAILangExtractRecognizer() ``` **Setup Managed Identity**: 1. Enable managed identity on your Azure resource (VM, App Service, Container Instance, etc.) 2. Grant the managed identity **"Cognitive Services OpenAI User"** role on your Azure OpenAI resource 3. No API keys needed - authentication is automatic See [Azure Managed Identity documentation](https://learn.microsoft.com/en-us/azure/active-directory/managed-identities-azure-resources/overview) for details. ### Configuration File (Optional) The configuration file is **optional** for basic usage. You only need it to customize: - Supported entity types - Entity mappings (LangExtract → Presidio) - Prompts and examples - Detection parameters For basic usage, just pass `model_id` as a parameter (see examples above). **When you need a custom config:** 1. **Download** the default config: ```sh wget https://raw.githubusercontent.com/data-privacy-stack/presidio/main/presidio-analyzer/presidio_analyzer/conf/langextract_config_azureopenai.yaml ``` 2. **Customize** entities, prompts, or other settings in the file 3. **Use** the customized config: ```python recognizer = AzureOpenAILangExtractRecognizer( model_id="gpt-4", # Can override config's model_id config_path="./custom_config.yaml", azure_endpoint="...", api_key="..." ) ``` **Configuration Reference**: The config file contains two main sections: **`lm_recognizer` section** (LLM recognizer settings): - `supported_entities`: List of PII/PHI entity types to detect - `labels_to_ignore`: Entity labels to skip during processing - `enable_generic_consolidation`: Whether to consolidate unknown entities to GENERIC_PII_ENTITY - `min_score`: Minimum confidence score threshold (0.0-1.0) **`langextract` section** (LangExtract-specific settings): - `model.model_id`: Azure OpenAI deployment name (e.g., "gpt-4o", "gpt-4", "gpt-35-turbo") - `model.temperature`: Model temperature for generation (null = use model default) - `prompt_file`: Path to custom prompt template file - `examples_file`: Path to few-shot examples file - `entity_mappings`: Map LangExtract entity classes to Presidio entity names See the [full config file](https://github.com/data-privacy-stack/presidio/blob/main/presidio-analyzer/presidio_analyzer/conf/langextract_config_azureopenai.yaml) for details. --- ## Using Ollama (Local Models) --- ### Samples Python Streamlit Index (`docs/samples/python/streamlit/index.md`) # Simple demo website for Presidio Here's a simple app, written in pure Python, to create a demo website for Presidio. The app is based on the [streamlit](https://streamlit.io/) package. A live version can be found here: https://huggingface.co/spaces/presidio/presidio_demo ## Requirements 1. Clone the repo and move to the `docs/samples/python/streamlit` folder 2. Install dependencies (preferably in a virtual environment) ```sh pip install -r requirements.txt ``` > Note: This would install additional packages such as `transformers` and `flair` which are not mandatory for using Presidio. 3. *Optional*: Update the `analyzer_engine` and `anonymizer_engine` functions for your specific implementation (in `presidio_helpers.py`). 4. Start the app: ```sh streamlit run presidio_streamlit.py ``` 5. Consider adding an `.env` file with the following environment variables, for further customizability: ```sh TA_KEY=YOUR_TEXT_ANALYTICS_KEY TA_ENDPOINT=YOUR_TEXT_ANALYTICS_ENDPOINT OPENAI_TYPE="Azure" #or "openai" OPENAI_KEY=YOUR_OPENAI_KEY OPENAI_API_VERSION = "2023-05-15" AZURE_OPENAI_ENDPOINT=YOUR_AZURE_OPENAI_AZURE_OPENAI_ENDPOINT AZURE_OPENAI_DEPLOYMENT=text-davinci-003 ALLOW_OTHER_MODELS=true #true if the user could download new models ``` ## Output Output should be similar to this screenshot: --- ### Samples Python Text Analytics Index (`docs/samples/python/text_analytics/index.md`) # Azure AI Language Integration ## Introduction Azure Text Analytics is a cloud-based service that provides advanced natural language processing over raw text. One of its main functions includes Named Entity Recognition (NER), which has the ability to identify different entities in text and categorize them into pre-defined classes or types. This document will demonstrate Presidio integration with Azure Text Analytics. ## Supported entity categories in the Text Analytics API Azure AI Language supports multiple PII entity categories. The Azure AI Laguage service runs a predictive model to identify and categorize named entities from an input document. The service's latest version includes the ability to detect personal (PII) and health (PHI) information. A list of all supported entities can be found in the [official documentation](https://docs.microsoft.com/en-us/azure/cognitive-services/text-analytics/named-entity-types?tabs=personal). ## Prerequisites To use Azure AI Language with Preisido, an Azure AI Language resource should first be created under an Azure subscription. Follow the [official documentation](https://docs.microsoft.com/en-us/azure/cognitive-services/text-analytics/how-tos/text-analytics-how-to-call-api?tabs=synchronous#create-a-text-analytics-resource) for instructions. The key and endpoint, generated once the resource is created, will be used when integrating with Text Analytics, using a Presidio Text Analytics recognizer. ## Azure AI Language Recognizer [The implementation of a `AzureAILanguage` recognizer can be found here](https://github.com/data-privacy-stack/presidio/blob/main/presidio-analyzer/presidio_analyzer/predefined_recognizers/third_party/azure_ai_language.py). ## How to integrate Azure AI Language into Presidio 1. Install the package with the azure-ai-language extra: ```sh pip install "presidio-analyzer[azure-ai-language]" ``` 2. Define environment varibles `AZURE_AI_KEY` and `AZURE_AI_ENDPOINT` 3. Add the `AzureAILanguageRecognizer` to the recognizer registry: ```python from presidio_analyzer import AnalyzerEngine from presidio_analyzer.predefined_recognizers import AzureAILanguageRecognizer azure_ai_language = AzureAILanguageRecognizer() analyzer = AnalyzerEngine() analyzer.registry.add_recognizer(azure_ai_language) analyzer.analyze(text="My email is email@email.com", language="en") ``` --- ### Samples Python Transformers Recognizer Index (`docs/samples/python/transformers_recognizer/index.md`) # Add a Transformers model based EntityRecognizer !!! note "Note" This example demonstrates how to create a **Presidio Recognizer**. To integrate a transformers model as a **Presidio NLP Engine**, see [this documentation](../../../analyzer/nlp_engines/transformers.md). We allow these two options, as a user might want to have multiple NER models running in parallel. In this case, one can create multiple `EntityRecognizer` instances, each serving a different model. If you only plan to use one NER model, consider creating a [`TransformersNlpEngine`](../../../analyzer/nlp_engines/transformers.md) instead of the [`TransformersRecognizer`](https://github.com/data-privacy-stack/presidio/blob/main/docs/samples/python/transformers_recognizer/transformer_recognizer.py) described in this document. When initializing the `TransformersRecognizer`, choose from the following options: 1. A string referencing an uploaded model to HuggingFace. See the different available options for models [here](https://huggingface.co/models?pipeline_tag=token-classification&sort=downloads). 2. Initialize your own `TokenClassificationPipeline` instance using your custom transformers model and use it for inference. 3. Provide the path to your own local custom trained model. !!! note "Note" For each combination of model & dataset, it is recommended to create a configuration object which includes setting necessary parameters for getting the correct results. Please reference this [configuration.py](https://github.com/data-privacy-stack/presidio/blob/main/configuration.py) file for examples. ## Example Code This example code uses a `TransformersRecognizer` for NER, and removes the default `SpacyRecognizer`. In order to be able to use spaCy features such as lemmas, we introduce the small (and faster) `en_core_web_sm` model. [link to full TransformersRecognizer code](https://github.com/data-privacy-stack/presidio/blob/main/docs/samples/python/transformers_recognizer/transformer_recognizer.py) ```python from presidio_analyzer import AnalyzerEngine, RecognizerRegistry from presidio_analyzer.nlp_engine import NlpEngineProvider import spacy model_path = "obi/deid_roberta_i2b2" supported_entities = BERT_DEID_CONFIGURATION.get( "PRESIDIO_SUPPORTED_ENTITIES") transformers_recognizer = TransformersRecognizer(model_path=model_path, supported_entities=supported_entities) # This would download a large (~500Mb) model on the first run transformers_recognizer.load_transformer(**BERT_DEID_CONFIGURATION) # Add transformers model to the registry registry = RecognizerRegistry() registry.add_recognizer(transformers_recognizer) registry.remove_recognizer("SpacyRecognizer") # Use small spacy model, for faster inference. if not spacy.util.is_package("en_core_web_sm"): spacy.cli.download("en_core_web_sm") nlp_configuration = { "nlp_engine_name": "spacy", "models": [{"lang_code": "en", "model_name": "en_core_web_sm"}], } nlp_engine = NlpEngineProvider(nlp_configuration=nlp_configuration).create_engine() analyzer = AnalyzerEngine(registry=registry, nlp_engine=nlp_engine) sample = "My name is John and I live in NY" results = analyzer.analyze(sample, language="en", return_decision_process=True, ) print("Found the following entities:") for result in results: print(result, '----', sample[result.start:result.end]) ``` --- ### Samples Deployments Openai Anonymaztion And Deanonymaztion Best Practices Src Api README (`docs/samples/deployments/openai-anonymaztion-and-deanonymaztion-best-practices/src/api/README.md`) # API Readme ## Install requirements ```bash cd src/api pip install -r requirements.txt ``` ### Set environment variables Add `.env` file in the `/src/api` folder: ```bash cp .env.sample .env ``` Then edit this configuration according to your setup. ## Run locally ```bash uvicorn main:app --host 0.0.0.0 --port 8080 ``` ## Run as a docker container ```bash docker build -t api . docker run -d -p 8080:80 --env-file .env api ``` --- ### Samples Deployments Openai Anonymaztion And Deanonymaztion Best Practices Src Client App README (`docs/samples/deployments/openai-anonymaztion-and-deanonymaztion-best-practices/src/client_app/README.md`) # Demo client for Anonymizer API Have an anonymous chat with the LLM right in your terminal or on a web page. ## Prerequisites ### Install requirements ```bash cd src/client_app pip install -r requirements.txt ``` ### Create .env file ```bash cp .env.sample .env ``` Then edit the configuration in the `.env` file, for example: - add your OpenAI API settings if you'd like to chat with an LLM - add anonymizer/deanonymizer API endpoints ## Running the chat ### Chat with LLM in the terminal ```bash python client.py ``` ### Chat with yourself in the terminal ```bash python client.py --mode manual ``` ### Chat with LLM on a web page ```bash python serve.py ``` ### Dockerize chatting on a web page Build container: ```bash docker build -t client . ``` And run it, e.g like this: ```bash docker run -p 8081:8081 --env-file .env client ``` **Note**: anonymizer API (API_URL env variable) should not point to localhost, otherwise the docker container won't be able to access it. --- ### Ahds Integration (`docs/ahds_integration.md`) # Azure Health Data Services De-identification Service Integration Presidio supports integration with Azure Health Data Services (AHDS) de-identification service for both entity recognition and anonymization with realistic surrogate generation. ## Resources - [Azure Health Data Services Documentation](https://learn.microsoft.com/azure/healthcare-apis/deidentification/) - [Azure Health Deidentification Python SDK](https://pypi.org/project/azure-health-deidentification/) ## Overview The AHDS de-identification service integration provides two main capabilities: 1. **AHDS Recognizer** (in presidio-analyzer): Detects PHI entities using the Azure Health Data Services de-identification service 2. **AHDS Surrogate Operator** (in presidio-anonymizer): Replaces PHI entities with realistic surrogates using the de-identification service ## Benefits of Surrogation - **Maintains Data Utility**: Preserves structure and format for downstream analytics - **Realistic Healthcare Context**: Generates medically plausible names, dates, and identifiers - **Consistent Cross-References**: Same entity gets same surrogate throughout document - **Format Preservation**: Maintains original formatting and linguistic patterns ## Installation ### For AHDS Recognizer ```bash pip install "presidio-analyzer[ahds]" ``` ### For AHDS Surrogate Operator ```bash pip install "presidio-anonymizer[ahds]" ``` ## Prerequisites - [Azure Health Data Services de-identification service endpoint](https://learn.microsoft.com/azure/healthcare-apis/deidentification/quickstart) - [Azure role-based access control configured](https://learn.microsoft.com/azure/healthcare-apis/deidentification/manage-access-rbac) - Environment variables: - `AHDS_ENDPOINT`: Your AHDS de-identification service endpoint ## Complete Workflow Example ```python import os from presidio_analyzer import AnalyzerEngine from presidio_anonymizer import AnonymizerEngine from presidio_anonymizer.entities import OperatorConfig # Set your AHDS endpoint os.environ["AHDS_ENDPOINT"] = "https://your-ahds-endpoint.api.eus001.deid.azure.com" # Initialize engines analyzer = AnalyzerEngine() anonymizer = AnonymizerEngine() # Medical text with PHI text = "Patient John Doe was seen by Dr. Smith on 2024-01-15 for diabetes treatment." # Step 1: Detect entities using AHDS recognizer analyzer_results = analyzer.analyze( text=text, entities=["PATIENT", "DOCTOR", "DATE"], language="en" ) # Step 2: Anonymize using AHDS surrogate generation result = anonymizer.anonymize( text=text, analyzer_results=analyzer_results, operators={ "DEFAULT": OperatorConfig("surrogate", { "entities": analyzer_results, "input_locale": "en-US", "surrogate_locale": "en-US" }) }, ) print(f"Original: {text}") print(f"Anonymized: {result.text}") # Output: "Patient Michael Johnson was seen by Dr. Brown on 1987-06-23 for diabetes treatment." ``` **Note:** This example uses the Azure Health Data Services de-identification service surrogation, which provides superior data utility by generating realistic, medically-appropriate surrogates while maintaining document structure and relationships. ## Configuration Options ### AHDS Surrogate Operator Parameters - `endpoint`: AHDS de-identification service endpoint (optional, uses `AHDS_ENDPOINT` env var) - `entities`: List of entities detected by analyzer - `input_locale`: Input locale (default: "en-US") - `surrogate_locale`: Surrogate locale (default: "en-US") ## Authentication The AHDS de-identification service integration uses Azure authentication with a secure-by-default approach: ### Production Mode (Default) By default, the integration uses a restricted credential chain that only includes: 1. **EnvironmentCredential**: Service Principal via environment variables 2. **WorkloadIdentityCredential**: Workload Identity (Kubernetes) 3. **ManagedIdentityCredential**: Managed Identity (Azure services) This credential chain is more secure as it excludes interactive browser logins and developer credentials, making it suitable for production deployments. ### Development Mode For local development, you can enable `DefaultAzureCredential` by setting the environment variable: ```bash export ENV=development ``` In development mode, the integration uses `DefaultAzureCredential`, which supports additional authentication methods: 1. Environment variables (Service Principal) 2. Workload Identity 3. Managed Identity 4. Azure CLI (`az login`) 5. Azure PowerShell 6. Visual Studio/VS Code credentials 7. Interactive browser login ### Local Development Setup For local development with Azure CLI authentication: ```bash # Set development mode export ENV=development # Login with Azure CLI az login # Set your AHDS endpoint export AHDS_ENDPOINT="https://your-ahds-endpoint.api.eus001.deid.azure.com" # Now you can run Presidio with AHDS integration python your_script.py ``` ### Production Deployment Recommendations For production deployments, we recommend: 1. **Do not set ENV=development** (use default production mode) 2. **Use Managed Identity** when running on Azure services (AKS, App Service, Functions, etc.) 3. **Use Workload Identity** for Kubernetes deployments 4. **Use Service Principal** with environment variables for other scenarios: ```bash export AZURE_CLIENT_ID="" export AZURE_CLIENT_SECRET="" export AZURE_TENANT_ID="" ``` ### Environment Variables Summary - `AHDS_ENDPOINT`: Your AHDS de-identification service endpoint (required) - `ENV`: Set to `development` for local dev, omit for production (optional, default: production mode) ## Troubleshooting ### Common Issues 1. **ModuleNotFoundError**: Install the AHDS optional dependencies ```bash pip install "presidio-analyzer[ahds]" "presidio-anonymizer[ahds]" ``` 2. **Authentication errors**: Ensure Azure credentials are properly configured ```bash az login # For local development ``` 3. **Endpoint not found**: Verify the `AHDS_ENDPOINT` environment variable is set correctly ### Testing without AHDS The AHDS operators gracefully handle missing dependencies and will be skipped if not available. Tests will be automatically skipped if the `AHDS_ENDPOINT` environment variable is not set. ## See Also - [Presidio Analyzer](../analyzer/index.md) - [Presidio Anonymizer](../anonymizer/index.md) - [Azure Health Data Services de-identification service documentation](https://learn.microsoft.com/azure/healthcare-apis/deidentification/) --- ### Api (`docs/api.md`) # Presidio API Api reference for Presidio's main python modules - [Presidio analyzer Python API reference](api/analyzer_python.md) - [Presidio anonymizer Python API reference](api/anonymizer_python.md) - [Presidio image redactor Python API reference](api/image_redactor_python.md) - [Presidio structured Python API reference](api/structured_python.md) --- ### Build Release (`docs/build_release.md`) # Build and release process Presidio uses GitHub Actions workflows to validate, build, release, and deliver official artifacts. ## Workflows The following workflows are maintained as part of the Presidio development process: - [CI](https://github.com/data-privacy-stack/presidio/blob/main/.github/workflows/ci.yml) - triggered for pull requests and merges to the main branch. - Linting - Dependency review - Unit tests - Package builds - Container builds and E2E tests - [Release](https://github.com/data-privacy-stack/presidio/blob/main/.github/workflows/release.yml) - manually triggered for official releases. - Publishes Python packages to [PyPI](https://pypi.org/search/?q=presidio) - Publishes container images to [GitHub Container Registry](https://github.com/orgs/data-privacy-stack/packages) - Creates a draft GitHub release ## Container image publishing Official Presidio container images are now published to GitHub Container Registry (GHCR) under the Data Privacy Stack organization: - [`ghcr.io/data-privacy-stack/presidio-analyzer`](https://github.com/orgs/data-privacy-stack/packages/container/package/presidio-analyzer) - [`ghcr.io/data-privacy-stack/presidio-anonymizer`](https://github.com/orgs/data-privacy-stack/packages/container/package/presidio-anonymizer) - [`ghcr.io/data-privacy-stack/presidio-image-redactor`](https://github.com/orgs/data-privacy-stack/packages/container/package/presidio-image-redactor) The legacy Microsoft Container Registry images at `mcr.microsoft.com/presidio-*` are no longer updated. Existing MCR tags can still be pulled for older deployments, but users should update image references to GHCR before upgrading. For example: ```sh docker pull ghcr.io/data-privacy-stack/presidio-analyzer:latest ``` For production deployments, avoid relying on the moving `latest` tag. Pin the release tag shown on the relevant [GitHub Packages page](https://github.com/orgs/data-privacy-stack/packages), for example: ```sh docker pull ghcr.io/data-privacy-stack/presidio-analyzer: ``` If an existing deployment uses `mcr.microsoft.com/presidio-analyzer:latest`, treat it as a legacy reference and replace it with either `ghcr.io/data-privacy-stack/presidio-analyzer:latest` for quick testing or a pinned GHCR release tag for production. Docker registries do not provide a portable way to redirect `docker pull mcr.microsoft.com/presidio-analyzer:latest` to GHCR or show a deprecation warning during `docker pull`. To reduce confusion, each GitHub release includes the GHCR image names, the GHCR images include OCI metadata labels pointing to the source repository, package page, and installation guide, and the MCR tags are documented as legacy references. ## PyPI publishing with OIDC The release workflow uses OIDC (OpenID Connect) trusted publishing to PyPI, which eliminates the need to manage PyPI API tokens. This requires: 1. **PyPI configuration**: Each package (`presidio-analyzer`, `presidio-anonymizer`, and the other Presidio packages) must be configured on PyPI to trust the GitHub repository and workflow. 2. **GitHub workflow permissions**: The workflow uses `pypa/gh-action-pypi-publish@release/v1` with `id-token: write`. 3. **No PyPI secrets**: No PyPI API tokens need to be stored as GitHub secrets. Benefits of OIDC: - Enhanced security through short-lived tokens - No manual token management - Automatic token rotation - Audit trail of publishing activities ## GHCR publishing permissions The container publishing jobs authenticate to GHCR with the workflow `GITHUB_TOKEN` by default and require `packages: write` permissions. If the workflow is run from a repository outside the `data-privacy-stack` organization during the transition, set the `GHCR_TOKEN` secret and optional `GHCR_USERNAME` variable to credentials that can publish packages to the Data Privacy Stack organization, or run the release from the migrated repository. --- ### Community (`docs/community.md`) # Presidio eco-system This section collects different resources developed with Presidio. ## Resources | Resource | Description | | ------ | ------ | | [HashiCorp Vault Operator](https://github.com/sahajsoft/presidio-vault) | A library that allows to integrate Presidio with HashiCorp Vault for anonymization and deanonymization. | | [Rasa bot framework](https://rasa.com/docs/rasa/pii-management/) | Use Presidio to de-identify chat bot messages in Rasa. | | [LangChain](https://python.langchain.com/v0.1/docs/guides/productionization/safety/presidio_data_anonymization/) | De-identification and reversible anonymization within LangChain. | | [LlamaIndex](https://python.langchain.com/v0.1/docs/guides/productionization/safety/presidio_data_anonymization) | De-identification and reversible anonymization within LlamaIndex. | | [LiteLLM](https://docs.litellm.ai/docs/proxy/guardrails/pii_masking_v2) | Integrate Presidio into LiteLLM | | [Guardrails-ai](https://www.guardrailsai.com/docs/examples/check_for_pii) | Use Presidio as an LLM guardrails using the Guardrails AI suite. | | [Presidio in LLMGuard](https://llm-guard.com/input_scanners/anonymize) | Integrate Presidio into LLM Guard - The Security Toolkit for LLM Interactions. | | [Privy](https://blog.px.dev/detect-pii/) | Integrate Presidio into Privy. | | [Huggingface](https://huggingface.co/blog/presidio-pii-detection) | Automatic PII detection on Huggingface datasets. | | [KNIME](https://hub.knime.com/knime/extensions/org.knime.python.features.presidio/latest/org.knime.python3.nodes.extension.ExtensionNodeSetFactory$DynamicExtensionNodeFactory:290c90e1) | Use Presidio within the KNIME framework. | | [OpenMetadata](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/profiler/auto-pii-tagging) | Auto PII tagging for Sensitive/NonSensitive at the column level. | | [dataiku](https://doc.dataiku.com/dss/latest/generative-ai/pii-detection.html) | PII detection in the LLM Mesh can detect various forms of PII in your prompts and queries, and either block or redact the queries. | | [Obsei](https://github.com/obsei/obsei) | Obsei is an open-source, low-code, AI powered automation tool | | [data-describe](https://github.com/data-describe/data-describe) | data-describe is a Python toolkit for Exploratory Data Analysis (EDA). It aims to accelerate data exploration and analysis by providing automated and polished analysis widgets. | | [Azure Search Power Skills](https://github.com/Azure-Samples/azure-search-power-skills) | Power Skills are a collection of useful functions to be deployed as custom skills for Azure Cognitive Search. The skills can be used as templates or starting points for your own custom skills, or they can be deployed and used as they are if they happen to meet your requirements. | | [DataOps for the Modern Data Warehouse](https://github.com/Azure-Samples/modern-data-warehouse-dataops) | Contains numerous code samples and artifacts on how to apply DevOps principles to data pipelines built according to the [Modern Data Warehouse (MDW)](https://azure.microsoft.com/en-au/solutions/architecture/modern-data-warehouse/) architectural pattern on [Microsoft Azure](https://azure.microsoft.com/en-au/). | | [Extending Power BI with Python and R](https://github.com/PacktPublishing/Extending-Power-BI-with-Python-and-R) | Code repository for [Extending Power BI with Python and R](https://www.packtpub.com/product/extending-power-bi-with-python-and-r/9781801078207), published by Packt. | | [HebSafeHarbor](https://github.com/8400TheHealthNetwork/HebSafeHarbor) | Clinical notes anonymization in Hebrew. | | [Presidio GitHub Action](https://github.com/marketplace/actions/presidio-action) | GitHub Action that analyzes text for PII entities with the Presidio framework. | * Please create a PR if you're interested in adding your tool to this list. --- ### Design (`docs/design.md`) # Presidio design ## Analyzer [](analyzer/index.md) ## Anonymizer [](anonymizer/index.md) ## Image Redactor ### Standard Image Types [](image-redactor/index.md) ### DICOM Images [](image-redactor/index.md) --- ### Development (`docs/development.md`) # Setting Up a Development Environment ## Getting started ### Cloning the repo To create a local copy of Presidio repository, follow [Github instructions](https://docs.github.com/en/github/creating-cloning-and-archiving-repositories/cloning-a-repository) on how to clone a project using git. The project is structured so that: - Each Presidio service has a designated directory: - The service logic. - Tests, both unit and integration. - Serving it as an HTTP service (found in app.py). - Python Packaging setup script (setup.py). - In the project root directory, you will find common code for using, serving and testing Presidio as a cluster of services, as well as CI/CD pipelines codebase and documentation. ### Setting up uv [uv](https://docs.astral.sh/uv/) is the Python package manager used to manage dependencies and virtual environments for Presidio services (the CI installs dependencies the same way). Follow these steps when starting to work on a Presidio service with uv: 1. Install uv - Using the standalone installer (you can inspect [install.sh](https://astral.sh/uv/install.sh) before running it) ```sh curl -LsSf https://astral.sh/uv/install.sh | sh ``` - Using Homebrew (in MacOS) ``` brew install uv ``` Additional installation instructions for uv: 2. Have uv create a virtualenv for the project and install all requirements in the pyproject.toml, including dev requirements. For example, in the `presidio-analyzer` folder, run: ``` uv sync --all-extras --group dev ``` Add `--locked` to install the exact locked dependency graph (matching CI behavior). 3. Run all tests: ``` uv run pytest ``` 4. To run arbitrary scripts within the virtual env, start the command with `uv run`. For example: 1. `uv run ruff check` 2. `uv run pip freeze` 3. `uv run python -m spacy download en_core_web_lg` Command 3 downloads the default spacy model needed for Presidio Analyzer. #### Alternatively, [activate the virtual environment](https://docs.astral.sh/uv/pip/environments/#using-a-virtual-environment) and run the commands without the `uv run` prefix. ### Development guidelines - A Github issue suggesting the change should be opened prior to a PR. - All contributions should be documented, tested and linted. Please verify that all tests and lint checks pass successfully before proposing a change. - To make the linting process easier, you can use [pre-commit hooks](#automatically-format-code-and-check-for-code-styling) to verify and automatically format code upon a git commit - In order for a pull request to be accepted, the CI (containing unit tests, e2e tests and linting) needs to succeed, in addition to approvals from two maintainers. - PRs should be small and solve/improve one issue at a time. If you have multiple suggestions for improvement, please open multiple PRs. ### Local build process After modifying presidio codebase, you might want to build presidio cluster locally, and run tests to spot regressions. The recommended way of doing so is using docker-compose (bundled with 'Docker Desktop' for Windows and Mac systems, more information can be found [here](https://docs.docker.com/compose/install/)). Once installed, to start presidio cluster with all of its services in HTTP mode, run from the project root: ```bash docker-compose up --build -d ``` !!! note "Note" Building for the first time might take some time, mainly on downloading the default spacy models. To validate that the services were built and started successfully, and to see the designated port for each, use docker-compose ps: ```bash >docker-compose ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 6d5a258d19c2 presidio-anonymizer "./entrypoint.sh" 6 minutes ago Up 6 minutes 0.0.0.0:5001->5001/tcp presidio_presidio-anonymizer_1 9aad2b68f93c presidio-analyzer "./entrypoint.sh" 2 days ago Up 6 minutes 0.0.0.0:5002->5001/tcp presidio_presidio-analyzer_1 1448dfb3ec2b presidio-image-redactor "./entrypoint.sh" 2 seconds ago Up 2 seconds 0.0.0.0:5003->5001/tcp presidio_presidio-image-redactor_1 ``` Edit docker-compose.yml configuration file to change the default ports. Starting part of the cluster, or one service only, can be done by stating its image name as argument for docker-compose. For example for analyzer service: ```bash docker-compose up --build -d presidio-analyzer ``` ### Testing We strive to have a full test coverage in Presidio, and expect every pull request to include tests. In each service directory, a 'test' directory can be found. In it, both unit tests, for testing single files or classes, and integration tests, for testing integration between the service components, or integration with external packages. #### Basic conventions For tests to be consistent and predictable, we use the following basic conventions: 1. Treat tests as production code. Keep the tests concise and readable, with descriptive namings. 2. Assert on one behavior at a time in each test. 3. Test names should follow a pattern of `test_when_[condition_to_test]_then_[expected_behavior]`. For example: `test_given_an_unknown_entity_then_anonymize_uses_defaults`. 4. Use [test doubles and mocks](https://docs.pytest.org/en/stable/monkeypatch.html) when writing unit tests. Make less use of them when writing integration tests. #### Running tests Presidio uses the [pytest](http://doc.pytest.org/) framework for testing. See the pytest [documentation](https://docs.pytest.org/en/latest/contents.html) for more information. Running the tests locally can be done in two ways: 1. Using cli, from each service directory, run: ```sh uv run pytest ``` 2. Using your IDE. See configuration examples for [JetBrains PyCharm / IntelliJ IDEA](https://www.jetbrains.com/help/pycharm/creating-run-debug-configuration-for-tests.html) and [Visual Studio Code](https://code.visualstudio.com/docs/python/testing) #### End-to-end tests Since Presidio services can function as HTTP servers, Presidio uses an additional end-to-end (e2e) testing layer to test their REST APIs. This e2e test framework is located under 'e2e-tests' directory. In it, you can also find test scenarios testing the integration between Presidio services through REST API. These tests should be annotated with 'integration' pytest marker `@pytest.mark.integration`, while tests calling a single service API layer should be annotated with 'api' pytest marker `@pytest.mark.api`. Running the e2e-tests locally can be done in two ways: 1. Using cli, from e2e-tests directory, run: On Mac / Linux / WSL: ```sh # Create a virtualenv named presidio-e2e (needs to be done only on the first run) python -m venv presidio-e2e # Activate the virtualenv source presidio-e2e/bin/activate # Install e2e-tests requirements using pip pip install -r requirements.txt # Run pytest pytest # Deactivate the virtualenv deactivate ``` On Windows CMD / Powershell: ```shell # Create a virtualenv named presidio-e2e (needs to be done only on the first run) py -m venv presidio-e2e # Activate the virtualenv presidio-e2e\Scripts\activate # Install e2e-tests requirements using pip pip install -r requirements.txt # Run pytest pytest # Deactivate the virtualenv deactivate ``` 2. Using your IDE See references in the section above. !!! note "Note" The e2e tests require a Presidio cluster to be up, for example using the containerized cluster with docker-compose. ### Linting Presidio services are PEP8 compliant and continuously enforced on style guide issues during the build process using `ruff`, in turn running `flake8` and other linters. Running ruff locally, using `uv run ruff check`, you can check for those issues prior to committing a change. Ruff runs linters in addition to the basic `flake8` functionality, Presidio uses linters as part as ruff such as: - _pep8-naming_: To check that variable names are PEP8 compliant. - _flake8-docstrings_: To check that docstrings are compliant. ### Automatically format code and check for code styling To make the linting process easier, you can use pre-commit hooks to verify and automatically format code upon a git commit, using `ruff-format`: 1. [Install pre-commit package manager locally.](https://pre-commit.com/#install) 2. From the project's root, enable pre-commit, installing git hooks in the `.git/` directory by running: `pre-commit install`. 3. Commit non PEP8 compliant code will cause commit failure and automatically format your code using, as well as checking code formatting using `ruff` ```sh [INFO] Initializing environment for https://github.com/astral-sh/ruff-pre-commit. [INFO] Installing environment for https://github.com/astral-sh/ruff-pre-commit. [INFO] Once installed this environment will be reused. [INFO] This may take a few minutes... ruff.....................................................................Passed ruff-format..............................................................Failed - hook id: ruff-format - files were modified by this hook 5 files reformatted, 4 files left unchanged ``` 4. Committing again will finish successfully, with a well-formatted code. --- ### Faq (`docs/faq.md`) # Frequently Asked Questions (FAQ) - [General](#general) - [What is Presidio?](#what-is-presidio) - [Why was Presidio created?](#why-was-presidio-created) - [Who owns and maintains Presidio?](#who-owns-and-maintains-presidio) - [What is the difference between Presidio and different PII detection services like Azure AI Language, Azure Health Data Services, and Amazon Comprehend?](#what-is-the-difference-between-presidio-and-different-pii-detection-services-like-azure-ai-language-and-amazon-comprehend) - [Using Presidio](#using-presidio) - [How can I start using Presidio?](#how-can-i-start-using-presidio) - [What are the main building blocks in Presidio?](#what-are-the-main-building-blocks-in-presidio) - [Customizing Presidio](#customizing-presidio) - [How can Presidio be customized to my needs?](#how-can-presidio-be-customized-to-my-needs) - [What NLP frameworks does Presidio support?](#what-nlp-frameworks-does-presidio-support) - [Can Presidio be used for Pseudonymization?](#can-presidio-be-used-for-pseudonymization) - [Does Presidio work on structured/tabular data?](#does-presidio-work-on-structuredtabular-data) - [Improving detection accuracy](#improving-detection-accuracy) - [What can I do if Presidio does not detect some of the PII entities in my data (False Negatives)?](#what-can-i-do-if-presidio-does-not-detect-some-of-the-pii-entities-in-my-data-false-negatives) - [What can I do if Presidio falsely detects text as PII entities (False Positives)?](#what-can-i-do-if-presidio-falsely-detects-text-as-pii-entities-false-positives) - [How can I evaluate the performance of my Presidio instance?](#how-can-i-evaluate-the-performance-of-my-presidio-instance) - [Deployment](#deployment) - [How can I deploy Presidio into my environment?](#how-can-i-deploy-presidio-into-my-environment) - [Contributing](#contributing) - [How can I contribute to Presidio?](#how-can-i-contribute-to-presidio) - [How can I report security vulnerabilities?](#how-can-i-report-security-vulnerabilities) ## General ### What is Presidio? Presidio (Origin from Latin praesidium ‘protection, garrison’) helps to ensure sensitive data is properly managed and governed. It provides fast identification and anonymization modules for private entities in text and images. It is fully customizable and pluggable, can be adapted to your needs and be deployed into various environments. !!! note "Note" Presidio is a library or SDK rather than a service. It is meant to be customized to the user's or organization's specific needs. !!! warning "Warning" Presidio can help identify sensitive/PII data in un/structured text. However, because it is using automated detection mechanisms, there is no guarantee that Presidio will find all sensitive information. Consequently, additional systems and protections should be employed. ### Why was Presidio created? By developing Presidio, our goals are: 1. Allow organizations to preserve privacy in a simpler way by democratizing de-identification technologies and introducing transparency in decisions. 2. Embrace extensibility and customizability to a specific business need. 3. Facilitate both fully automated and semi-automated PII de-identification flows on multiple platforms. ### Who owns and maintains Presidio? Presidio is a community-governed open source project under the [Data Privacy Stack](https://github.com/data-privacy-stack) organization. It was originally created at Microsoft and has since transitioned to an independent, vendor-neutral project maintained by contributors and volunteers from across the community. See the [project transition update](project_transition.md) for more details. !!! note "Note" Presidio is not an official product of any company and comes with no warranty or SLA. Usage terms are defined in the [repository's license](https://github.com/data-privacy-stack/presidio/blob/main/LICENSE). ### What is the difference between Presidio and different PII detection services like Azure AI Language, Azure Health Data Services, and Amazon Comprehend? In a nutshell, Presidio is a library which is meant to be customized, whereas different SaaS tools for PII detection have less customization capabilities. Most of these SaaS offerings use dedicated ML models and other logic for PII detection and often have better entity coverage or accuracy than Presidio. Based on our internal research, leveraging Presidio in parallel to 3rd party PII detection services like Azure AI Language can bring optimal results mainly when the data in hand has entity types or values not supported by the 3rd party service. ([see example here](https://data-privacy-stack.github.io/presidio/samples/python/text_analytics/)). ## Using Presidio ### How can I start using Presidio? 1. Check out the [installation docs](https://data-privacy-stack.github.io/presidio/installation/). 2. Take a look at the [different samples](https://data-privacy-stack.github.io/presidio/samples/). 3. Try the [demo website](https://huggingface.co/spaces/presidio/presidio_demo). ### What are the main building blocks in Presidio? Presidio is a suite built of several packages and building blocks: 1. [Presidio Analyzer](https://data-privacy-stack.github.io/presidio/analyzer/): a package for detecting PII entities in natural language. 2. [Presidio Anonymizer](https://data-privacy-stack.github.io/presidio/anonymizer/): a package for manipulating PII entities in text (e.g. remove, redact, hash, encrypt). 3. [Presidio Image Redactor](https://data-privacy-stack.github.io/presidio/image-redactor/): A package for detecting PII entities in image using OCR. 4. [Presidio Structured](https://data-privacy-stack.github.io/presidio/structured/): A package for detecting PII entities in structured/semi-structured data. 5. A set of sample deployments as Python packages or Docker containers for Kubernetes, Azure Data Factory, Spark and more. ## Customizing Presidio ### How can Presidio be customized to my needs? Users can customize Presidio in multiple ways: 1. Create new or updated PII recognizers ([docs](https://data-privacy-stack.github.io/presidio/analyzer/adding_recognizers/)). 2. Adapt Presidio to new languages ([docs](https://data-privacy-stack.github.io/presidio/analyzer/languages/)). 3. Leverage state of the art Named Entity Recognition models ([docs](https://data-privacy-stack.github.io/presidio/analyzer/customizing_nlp_models/)). 4. Add new types of anonymizers ([docs](https://data-privacy-stack.github.io/presidio/anonymizer/adding_operators/)). 5. Create PII analysis and anonymization pipelines on different environments using Docker or Python ([samples](https://data-privacy-stack.github.io/presidio/samples/)). And more. ### What NLP frameworks does Presidio support? Presidio supports spaCy version 3+ for Named Entity Recognition, tokenization, lemmatization and more. We also support [Stanza](https://stanfordnlp.github.io/stanza/) using the [spacy-stanza](https://spacy.io/universe/project/spacy-stanza) package, and it is further possible to create PII recognizers leveraging other frameworks like [transformers](https://huggingface.co/transformers/usage.html#named-entity-recognition) or [Flair](https://github.com/flairNLP/flair). For more information, see the [docs](https://data-privacy-stack.github.io/presidio/analyzer/customizing_nlp_models/). ### Can Presidio be used for Pseudonymization? Pseudonymization is a de-identification technique in which the real data is replaced with fake data in a reversible way. Since there are various ways and approaches for this, we provide a simple [sample](https://github.com/data-privacy-stack/presidio/blob/main/docs/samples/python/example_custom_lambda_anonymizer.py) which can be extended for more sophisticated usage. If you have a question or a request on this topic, please open an issue on the repo. ### Does Presidio work on structured/tabular data? [Presidio-structured](https://data-privacy-stack.github.io/presidio/structured/) is a capability in Presidio for detecting PII entities in structured/semi-structured data. It scans datasets for PII using Presidio Analyzer, and supports the redaction of text, cells, or columns in a tabular dataset. ## Improving detection accuracy ### What can I do if Presidio does not detect some of the PII entities in my data (False Negatives)? Presidio comes loaded with several PII recognizers (see [list here](https://data-privacy-stack.github.io/presidio/supported_entities/)), however its main strength lies in its customization capabilities to new entities, specific datasets, file types, languages or use cases. ### What can I do if Presidio falsely detects text as PII entities (False Positives)? Some PII recognizers are less specific than others. A driver's license number, for example, could be any 9-digit number. While Presidio leverages context words and other logic to improve the detection quality, it could still falsely detect non-entity values as PII entities. In order to avoid false positives, one could try to: 1. Change the acceptance threshold, which defines what is the minimum confidence value for a detected entity to be returned. 2. Remove unnecessary PII recognizers, if the dataset does not contain these entities. 3. Update/replace the logic of specific recognizers to better suit a specific dataset or use case. 4. Replace PII recognizers with those coming from 3rd party services. Every PII identification logic would have its errors, and there is a trade-off between false positives (falsely detected text) and false negatives (PII entities which are not detected). ### How can I evaluate the performance of my Presidio instance? In addition to Presidio, we maintain a repo focused on evaluation of models and PII recognizers [here](https://github.com/data-privacy-stack/presidio-research). It also features a simple PII data generator. ## Deployment ### How can I deploy Presidio into my environment? The main Presidio modules (analyzer, anonymizer, image-redactor) can be used both as a Python package and as a dockerized REST API. See the [different deployment samples](https://data-privacy-stack.github.io/presidio/samples/) for example deployments. !!! warning "Authentication and Authorization" Presidio API endpoints do not include built-in authentication by design. The containers are intentionally kept lean to allow flexibility for different deployment scenarios. Authentication and authorization should be implemented at a separate infrastructure layer (e.g., an API gateway, reverse proxy, or service mesh) according to your specific use case and security requirements. It is strongly recommended not to expose Presidio services directly to untrusted networks without an authentication layer in front of them. ## Contributing ### How can I contribute to Presidio? First, review the [contribution guidelines](https://github.com/data-privacy-stack/presidio/blob/main/CONTRIBUTING.md), and feel free to reach out by opening an issue, posting a discussion or emailing us at ### How can I report security vulnerabilities? Please see the [security information](https://github.com/data-privacy-stack/presidio/blob/main/SECURITY.md). --- ### Getting Started (`docs/getting_started.md`) # Getting started with Presidio The core functionality in Presidio is to detect PII in text. Presidio further contains a set of tools that build on top of text PII detection, for example in images, structured data, JSON and more. - For a quickstart for PII detection and de-identification in text [click here](getting_started/getting_started_text.md). - For a quickstart for PII detection and de-identification in images [click here](getting_started/getting_started_images.md). - For a quickstart for PII detection and de-identification in structured and semi-structured data [click here](getting_started/getting_started_structured.md). --- ### Installation (`docs/installation.md`) # Installing Presidio ## Description This document describes the installation of the entire Presidio suite using `pip` (as Python packages) or using `Docker` (As containerized services). ## Using pip !!! note "Note" Consider installing the Presidio python packages in a virtual environment like [venv](https://docs.python.org/3/tutorial/venv.html) or [conda](https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html). ### Supported Python Versions Presidio is supported for the following python versions: * 3.10 * 3.11 * 3.12 * 3.13 * 3.14 ### PII anonymization on text For PII anonymization on text, install the `presidio-analyzer` and `presidio-anonymizer` packages with at least one NLP engine (`spaCy`, `transformers` or `stanza`): ===+ "spaCy (default)" ``` pip install presidio_analyzer pip install presidio_anonymizer python -m spacy download en_core_web_lg ``` === "Transformers" ``` pip install "presidio_analyzer[transformers]" pip install presidio_anonymizer python -m spacy download en_core_web_sm ``` !!! note "Note" When using a transformers NLP engine, Presidio would still use spaCy for other capabilities, therefore a small spaCy model (such as en_core_web_sm) is required. Transformers models would be loaded lazily. To pre-load them, see: [Downloading a pre-trained model](./analyzer/nlp_engines/transformers.md#downloading-a-pre-trained-model) === "Stanza" ``` pip install "presidio_analyzer[stanza]" pip install presidio_anonymizer ``` !!! note "Note" Stanza models would be loaded lazily. To pre-load them, see: [Downloading a pre-trained model](./analyzer/nlp_engines/spacy_stanza.md#download-the-pre-trained-model). ### GPU acceleration (optional) For GPU acceleration, install the appropriate dependencies for your hardware: - **Linux with NVIDIA GPU**: `pip install "spacy[cuda12x]"` (or the version matching your CUDA installation) - **macOS with Apple Silicon**: MPS is detected automatically, no additional dependencies required. For detailed GPU setup, verification, and troubleshooting, see [GPU Acceleration](./analyzer/nlp_engines/gpu_usage.md). ### PII redaction in images For PII redaction in images 1. Install the `presidio-image-redactor` package: ```sh pip install presidio_image_redactor # Presidio image redactor uses the presidio-analyzer # which requires a spaCy language model: python -m spacy download en_core_web_lg ``` 2. Install an OCR engine. The default version uses the [Tesseract OCR Engine](https://github.com/tesseract-ocr/tesseract). More information on installation can be found [here](image-redactor/index.md#installation). ## Using Docker Presidio can expose REST endpoints for each service using Flask and Docker. To download the Presidio Docker containers, run the following command: !!! note "Note" This requires Docker to be installed. [Download Docker](https://docs.docker.com/get-docker/). !!! important "Container registry moved to GitHub Packages" New Presidio container releases are published to [GitHub Container Registry](https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-container-registry) under the [Data Privacy Stack packages organization](https://github.com/orgs/data-privacy-stack/packages). The legacy Microsoft Container Registry images at `mcr.microsoft.com/presidio-*` are no longer updated. If you previously used `mcr.microsoft.com/presidio-analyzer:latest`, switch to `ghcr.io/data-privacy-stack/presidio-analyzer:latest`; for production, prefer pinning an explicit release tag from the package page. ### For PII anonymization in text For PII detection and anonymization in text, the `presidio-analyzer` and `presidio-anonymizer` modules are required. ```sh # Download Docker images docker pull ghcr.io/data-privacy-stack/presidio-analyzer docker pull ghcr.io/data-privacy-stack/presidio-anonymizer # Run containers with default ports docker run -d -p 5002:3000 ghcr.io/data-privacy-stack/presidio-analyzer:latest docker run -d -p 5001:3000 ghcr.io/data-privacy-stack/presidio-anonymizer:latest ``` ### For PII redaction in images For PII detection in images, the `presidio-image-redactor` is required. ```sh # Download Docker image docker pull ghcr.io/data-privacy-stack/presidio-image-redactor # Run container with the default port docker run -d -p 5003:3000 ghcr.io/data-privacy-stack/presidio-image-redactor:latest ``` Once the services are running, their APIs are available. API reference and example calls can be found [here](api.md). ## Install from source To install Presidio from source, first clone the repo: * using HTTPS ```sh git clone https://github.com/data-privacy-stack/presidio.git ``` * Using SSH ```sh git clone git@github.com:data-privacy-stack/presidio.git ``` Then, build the containers locally. !!! note "Note" Presidio uses [docker-compose](https://docs.docker.com/compose/) to manage the different Presidio containers. From the root folder of the repo: ```sh docker-compose up --build ``` Alternatively, you can build and run individual services. For example, for the `presidio-anonymizer` service: ```sh docker build ./presidio-anonymizer -t presidio/presidio-anonymizer ``` And run: ```sh docker run -d -p 5001:5001 presidio/presidio-anonymizer ``` --- For more information on developing locally, refer to the [setting up a development environment](development.md) section. --- ### Presidio V2 (`docs/presidio_V2.md`) # Presidio Revamp (aka V2) As of March 2021, Presidio underwent a revamp to a new version referred to as **V2**. The main changes introduced in **V2** are: 1. gRPC replaced with HTTP to allow more customizable APIs and easier debugging 2. Focus on the Analyzer and Anonymizer services. 1. Presidio Anonymizer is now Python-based and pip-installable. 2. Presidio Analyzer does not use templates and external recognizer store. 3. Image Redactor (formerly presidio-image-anonymizer) is in early beta and is Python based and pip installable. 4. Other services are deprecated and may be migrated over time to **V2** with the help of the community. 3. Improved documentation, sample code, and build workflows. 4. Format-Preserving Encryption replaced with Advanced Encryption Standard (AES) ## V1 Availability Version V1 (legacy) is still available for download. To continue using the previous version: - For docker containers, use tag=v1 - For python packages, download version < 2 (e.g. pip install presidio-analyzer==0.95) !!! note "Note" The legacy V1 code base will continue to be available under branch [V1](https://github.com/data-privacy-stack/presidio/tree/V1) but will no longer be officially supported. ## API Changes The move from gRPC to HTTP-based APIs included changes to the API requests. 1. Changed payload format – moving from structured objects to JSON. 2. Removed templates from the API, including flattening the JSON structure. 3. Using snake_case instead of camelCase . Below is a detailed outline of all changes made to the Analyzer and Anonymizer. ### Analyzer API Changes #### Legacy json request (gRPC) ```json { "text": "My phone number is 212-555-5555", "AnalyzeTemplateId": "1234", "AnalyzeTemplate": { "Fields": [ { "Name": "PHONE_NUMBER", "MinScore": "0.5" } ], "AllFields": true, "Description": "template description", "CreateTime": "template creation time", "ModifiedTime": "template modification time", "Language": "fr", "ResultsScoreThreshold": 0.5 } } ``` #### V2 json request (HTTP) ```json { "text": "My phone number is 212-555-5555", "entities": ["PHONE_NUMBER"], "language": "en", "correlation_id": "213", "score_threshold": 0.5, "trace": true, "return_decision_process": true } ``` ### Anonymizer API Changes #### Legacy json request (gRPC) ```json { "text": "hello world, my name is Jane Doe. My number is: 034453334", "template": { "description": "DEPRECATED", "create_time": "DEPRECATED", "modified_time": "DEPRECATED", "default_transformation": { "replace_value": {...}, "redact_value": {...}, "hash_value": {...}, "mask_value": {...}, "fpe_value": {...} }, "field_type_transformations": [ { "fields": [ { "name": "FIRST_NAME", "min_score": "0.2" } ], "transformation": { "replace_value": {...}, "redact_value": {...}, "hash_value": {...}, "mask_value": {...}, "fpe_value": {...} } } ], "analyze_results": [ { "text": "Jane", "field": { "name": "FIRST_NAME", "min_score": "0.5" }, "location": { "start": 24, "end": 32, "length": 6 }, "score": 0.8 } ] } } ``` #### V2 json request (HTTP) ```json { "text": "hello world, my name is Jane Doe. My number is: 034453334", "anonymizers": { "DEFAULT": { "type": "replace", "new_value": "val" }, "PHONE_NUMBER": { "type": "mask", "masking_char": "*", "chars_to_mask": 4, "from_end": true } }, "analyzer_results": [ { "start": 24, "end": 32, "score": 0.8, "entity_type": "NAME" }, { "start": 24, "end": 28, "score": 0.8, "entity_type": "FIRST_NAME" }, { "start": 29, "end": 32, "score": 0.6, "entity_type": "LAST_NAME" }, { "start": 48, "end": 57, "score": 0.95, "entity_type": "PHONE_NUMBER" } ] } ``` **Specific for each anonymization type:** | Anonymization name | Legacy format (V1) | New json format (V2) | | ------------------ | --------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------ | | Replace | 
string newValue = 1;
| 
{ "new_value": "VALUE" }
| | Redact | NONE | NONE | | Mask | 
string maskingCharacter = 1;
int32 charsToMask = 2; 
bool fromEnd = 3;
| 
{
 "chars_to_mask": 10,
 "from_end": true,
 "masking_char": "\*" 
}
| | Hash | NONE | 
{"hash_type": "VALUE"}
| | FPE (now Encrypt) | 
string key = 3t6w9z$C&F)J@NcR;
int32 tweak = D8E7920AFA330A73
| 
{"key": "3t6w9z$C&F)J@NcR"}
| !!! note "Note" The V2 API is continuously evolving. please [follow the change log](https://github.com/data-privacy-stack/presidio/blob/main/CHANGELOG.md) for updates. --- ### Project Transition (`docs/project_transition.md`) # Presidio has transitioned to a community-owned project We want to share an important update regarding the future of Presidio. The Presidio project has completed its transition from a Microsoft-owned project to an independent, community-governed open source project under the GitHub organization Data Privacy Stack. Its official URL is: Microsoft supported this transition and the continued success of the Presidio project as an open source community initiative. Our goal remains to ensure Presidio serves the privacy engineering and data protection community as an open, transparent, and vendor-neutral project. ## What this means * Presidio continues to be open source under the MIT license. * Existing functionality, APIs, and documentation remain available. * The project is maintained by contributors and volunteers from the community. * The project is not owned or operated by a commercial entity. * The focus on privacy, responsible AI, and data protection tooling remains unchanged. ## Microsoft and Azure users This transition does not change the experience for users of Presidio, including organizations using Presidio alongside Microsoft and Azure services. Integrations, dependencies, and usage patterns continue to work as expected. ## Why this change Over the years, Presidio has grown beyond its original scope and has been adopted by organizations, researchers, and developers across industries. Moving to an independent community initiative positions the project for long-term sustainability, broader collaboration, and open governance. ## What to expect next Now that the migration is complete: * Repositories, documentation links, and package references have moved to the new organization. Please update your references. * Legacy GitHub URLs and documentation links redirect to the new location. * New Docker image releases are published to [GitHub Container Registry](https://github.com/orgs/data-privacy-stack/packages) under the Data Privacy Stack organization. Legacy Microsoft Container Registry (MCR) images remain available for older tags but are no longer updated; update `mcr.microsoft.com/presidio-*` image references to `ghcr.io/data-privacy-stack/presidio-*`. * Contribution and governance processes are now community-driven. * The Technical Steering Committee (TSC) includes maintainers and contributors from across the community. ## Maintainers and contributors Maintainers and contributors continue to actively drive the project. We welcome new contributors who care about privacy-preserving technologies and responsible data handling. Details regarding contribution guidelines and Docker image publishing are available in the repository documentation. Thank you to everyone who supported Presidio through this transition and helped make it a valuable resource for the privacy engineering community. --- ### Requirements Docs (`docs/requirements-docs.txt`) # Dependencies for building the Presidio documentation with Zensical, the # successor to Material for MkDocs. Build with scripts/zensical_build.py, which # pre-converts the notebook samples and emits a generated zensical.yml. See that # script's module docstring for the full flow. # # pip install -r docs/requirements-docs.txt # python scripts/zensical_build.py # build to ./site # python scripts/zensical_build.py serve # local preview # # Zensical does not run MkDocs plugins, so mkdocs / mkdocs-material / # mkdocs-jupyter are intentionally absent. nbconvert renders the notebooks and # mkdocstrings-python + the presidio packages back the API reference pages. zensical nbconvert mkdocstrings-python pymdown-extensions markdown pyyaml presidio_analyzer presidio_anonymizer presidio_image_redactor presidio_structured pygments>=2.10 black --- ### Supported Entities (`docs/supported_entities.md`) # PII entities supported by Presidio Presidio contains predefined recognizers for PII entities. This page describes the different entities Presidio can detect and the method Presidio employs to detect those. In addition, Presidio allows you to add custom entity recognizers. For more information, refer to the [adding new recognizers documentation](analyzer/adding_recognizers.md). ## List of supported entities ### Global |Entity Type | Description | Detection Method | | --- | --- | --- | |CREDIT_CARD |A credit card number is between 12 to 19 digits. |Pattern match and checksum| |CRYPTO|A Crypto wallet number. Currently only Bitcoin address is supported|Pattern match, context and checksum| |DATE_TIME|Absolute or relative dates or periods or times smaller than a day.|Pattern match and context| |EMAIL_ADDRESS|An email address identifies an email box to which email messages are delivered|Pattern match, context and RFC-822 validation| |IBAN_CODE|The International Bank Account Number (IBAN) is an internationally agreed system of identifying bank accounts across national borders to facilitate the communication and processing of cross border transactions with a reduced risk of transcription errors.|Pattern match, context and checksum| |IP_ADDRESS|An Internet Protocol (IP) address (either IPv4 or IPv6).|Pattern match and context| |MAC_ADDRESS| A Media Access Control (MAC) address is a unique identifier assigned to network interfaces for communications on the physical network segment.|Pattern match and context| |NRP|A person’s Nationality, religious or political group.|Custom logic and context| |LOCATION|Name of politically or geographically defined location (cities, provinces, countries, international regions, bodies of water, mountains|Custom logic and context| |PERSON|A full person name, which can include first names, middle names or initials, and last names.|Custom logic and context| |PHONE_NUMBER|A telephone number. The `PhoneRecognizer` can be extended programmatically for country-specific detection by configuring `supported_regions` and `supported_entity` (e.g. Philippines: `PhoneRecognizer(supported_regions=["PH"], supported_entity="PH_MOBILE_NUMBER")`, Turkey: `PhoneRecognizer(supported_regions=["TR"], supported_entity="TR_PHONE_NUMBER")`)|Custom logic, pattern match and context| |MEDICAL_LICENSE|Common medical license numbers.|Pattern match, context and checksum| |URL|A URL (Uniform Resource Locator), unique identifier used to locate a resource on the Internet|Pattern match, context and top level url validation| |UUID|A Universally Unique Identifier (UUID) in the standard 8-4-4-4-12 hyphenated hexadecimal format. Covers RFC 4122 versions 1-5 and RFC 9562 versions 6-8; the nil UUID (all zeros) is excluded as a non-identifying sentinel.|Pattern match, validation of version/variant nibbles, and context| ### USA |Entity Type|Description|Detection Method| |--- |--- |--- | |US_BANK_NUMBER|A US bank account number is between 8 to 17 digits.|Pattern match and context| |US_DRIVER_LICENSE|A US driver license according to |Pattern match and context| |US_ITIN | US Individual Taxpayer Identification Number (ITIN). Nine digits that start with a "9" and contain a "7" or "8" as the 4 digit.|Pattern match and context| |US_CLAIM_NUMBER|A US healthcare claim identifier used in billing and claims processing.|Pattern match, context enhancement, and entity threshold| |US_HEALTH_INSURANCE_MEMBER_ID|A US health insurance member or subscriber identifier printed on an insurance card. Healthcare or insurance context increases detection confidence.|Pattern match, context enhancement, and entity threshold| |US_MBI|A US Medicare Beneficiary Identifier (MBI) with 11 alphanumeric characters.|Pattern match and context| |US_NPI|A US National Provider Identifier (NPI) is a 10-digit number issued to healthcare providers by CMS under HIPAA.|Pattern match, context and checksum| |US_PASSPORT |A US passport number with 9 digits.|Pattern match and context| |US_PRESCRIPTION_NUMBER|A US prescription or pharmacy order identifier.|Pattern match, context enhancement, and entity threshold| |US_PRIOR_AUTHORIZATION_NUMBER|A US prior authorization identifier used for treatment or drug approval requests.|Pattern match, context enhancement, and entity threshold| |US_PROVIDER_TAX_ID|A US provider organization tax identifier (TIN/EIN) used in healthcare billing workflows.|Pattern match, context enhancement, and entity threshold| |US_REFERRAL_NUMBER|A US healthcare referral identifier, including specialty or infusion referral numbers.|Pattern match, context enhancement, and entity threshold| |US_SSN|A US Social Security Number (SSN) with 9 digits.|Pattern match and context| ### UK |Entity Type|Description|Detection Method| |--- |--- |--- | |UK_DRIVING_LICENCE|UK driving licence number issued by the DVLA. A 16-character alphanumeric string encoding surname, date of birth, and initials.|Pattern match, context and validation| |UK_NHS|A UK NHS number is 10 digits.|Pattern match, context and checksum| |UK_NINO|UK [National Insurance Number](https://en.wikipedia.org/wiki/National_Insurance_number) is a unique identifier used in the administration of National Insurance and tax.|Pattern match and context| |UK_PASSPORT|A UK passport number consists of 2 letters followed by 7 digits (e.g., AB1234567), used in passports issued from 2015 onwards.|Pattern match and context| |UK_POSTCODE|A UK [postcode](https://en.wikipedia.org/wiki/Postcodes_in_the_United_Kingdom) is a 5-8 character alphanumeric code used by the Royal Mail for mail sorting.|Pattern match and context| |UK_VEHICLE_REGISTRATION|A UK vehicle registration number (number plate) in current (2001+), prefix (1983-2001), or suffix (1963-1983) format.|Pattern match, context and validation| ### Spain |Entity Type|Description|Detection Method| |--- |--- |--- | |ES_NIF | A Spanish NIF number (Personal tax ID).|Pattern match, context and checksum| |ES_NIE | A Spanish NIE number (Foreigners ID card).|Pattern match, context and checksum| |ES_PASSPORT | A Spanish passport number.|Pattern match and context| ### Italy |Entity Type|Description|Detection Method| |--- |--- |--- | |IT_FISCAL_CODE| An Italian personal identification code. |Pattern match, context and checksum| |IT_DRIVER_LICENSE| An Italian driver license number.|Pattern match and context| |IT_VAT_CODE| An Italian VAT code number |Pattern match, context and checksum| |IT_PASSPORT|An Italian passport number.|Pattern match and context| |IT_IDENTITY_CARD|An Italian identity card number. |Pattern match and context| ### Poland |Entity Type|Description|Detection Method| |--- |--- |--- | |PL_PESEL|Polish PESEL number|Pattern match, context and checksum| ### Singapore |FieldType|Description|Detection Method| |--- |--- |--- | |SG_NRIC_FIN| A National Registration Identification Card | Pattern match and context | |SG_UEN| A Unique Entity Number (UEN) is a standard identification number for entities registered in Singapore. | Pattern match, context, and checksum | ### Australia |FieldType|Description|Detection Method| |--- |--- |--- | |AU_ABN| The Australian Business Number (ABN) is a unique 11 digit identifier issued to all entities registered in the Australian Business Register (ABR). | Pattern match, context, and checksum | |AU_ACN| An Australian Company Number is a unique nine-digit number issued by the Australian Securities and Investments Commission to every company registered under the Commonwealth Corporations Act 2001 as an identifier. | Pattern match, context, and checksum | |AU_TFN| The tax file number (TFN) is a unique identifier issued by the Australian Taxation Office to each taxpaying entity | Pattern match, context, and checksum | |AU_MEDICARE| Medicare number is a unique identifier issued by Australian Government that enables the cardholder to receive a rebates of medical expenses under Australia's Medicare system| Pattern match, context, and checksum | ### India | FieldType | Description |Detection Method| |------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------|--- | | IN_PAN | The Indian Permanent Account Number (PAN) is a unique 12 character alphanumeric identifier issued to all business and individual entities registered as Tax Payers. | Pattern match, context | | IN_AADHAAR | Indian government issued unique 12 digit individual identity number | Pattern match, context, and checksum | | IN_VEHICLE_REGISTRATION | Indian government issued transport (govt, personal, diplomatic, defence) vehicle registration number | Pattern match, context, and checksum | | IN_VOTER | Indian Election Commission issued 10 digit alpha numeric voter id for all indian citizens (age 18 or above) | Pattern match, context | | IN_PASSPORT | Indian Passport Number | Pattern match, Context | | IN_GSTIN | The Indian Goods and Services Tax Identification Number (GSTIN) is a 15-character identifier with state code (01-37), PAN, registration number, 'Z', and checksum. | Pattern match, context, and validation | ### Finland | FieldType | Description | Detection Method | |------------|---------------------------------------------------------------------------------------------------------|------------------------------------------| | FI_PERSONAL_IDENTITY_CODE | The Finnish Personal Identity Code (Henkilötunnus) is a unique 11 character individual identity number. | Pattern match, context and custom logic. | ### Korea | FieldType | Description | Detection Method | |------------|---------------------------------------------------------------------------------------------------------|------------------------------------------| | KR_DRIVER_LICENSE | The Korean driver license number is a 12-digit number. | Pattern match, context and custom logic. | | KR_FRN | The Korean Foreigner Registration Number (FRN) is a 13-digit number. | Pattern match, context and custom logic. | | KR_PASSPORT| The Korean Passport Number | Pattern match, context. | | KR_BRN | The Korean Business Registration Number (BRN) is a 10-digit number assigned to business entities for taxation purposes. | Pattern match, context and custom logic. | | KR_RRN | The Korean Resident Registration Number (RRN) is a 13-digit number issued to all Korean residents. | Pattern match, context and custom logic. | ### Nigeria | FieldType | Description | Detection Method | |------------|---------------------------------------------------------------------------------------------------------|------------------------------------------| | NG_NIN | The Nigerian National Identification Number (NIN) is a unique 11-digit number issued by the National Identity Management Commission (NIMC). | Pattern match, context, and checksum | | NG_VEHICLE_REGISTRATION | Nigerian vehicle registration plate number in the current format (2011+): 3 letters (LGA code), 3 digits (serial), 2 letters (year/batch). | Pattern match and context | ### Philippines | FieldType | Description | Detection Method | |------------|---------------------------------------------------------------------------------------------------------|------------------------------------------| | PH_PASSPORT | Philippine Passport Number. Alphanumeric format, usually 1 letter followed by 7 digits and 1 letter, or 2 letters followed by 7 digits (e.g., P1234567A, EB1234567). | Pattern match and context | | PH_TIN | Philippines Taxpayer Identification Number (TIN). 9 or 12-digit number issued by the Bureau of Internal Revenue (BIR). | Pattern match, context, and checksum | ### Canada |FieldType|Description|Detection Method| |--- |--- |--- | |CA_SIN|A Canadian Social Insurance Number (SIN) is a 9-digit number issued by Employment and Social Development Canada (ESDC) to administer government programs. The last digit is a Luhn check digit. SINs starting with 0 or 8 are reserved and not issued.|Pattern match, context, and checksum| |CA_POSTAL_CODE|A Canadian postal code in the standard A1A 1A1 format (letter-digit-letter, space, digit-letter-digit). The letters D, F, I, O, Q, U are never used in any letter position; W and Z are additionally excluded from the first letter position.|Pattern match and context| ### Sweden | FieldType | Description | Detection Method | |------------|---------------------------------------------------------------------------------------------------------|------------------------------------------| | SE_ORGANISATIONSNUMMER | The Swedish Organisations ID Number is a unique 10-digit number issued to all Swedish organisations. | Pattern match, context, and checksum. | | SE_PERSONNUMMER | The Swedish Personal ID Number is a unique 10/12-digit number issued to all Swedish residents. The recognizer also supports Samordningsnummer (coordination numbers) issued to individuals who are not (yet) registered residents but need a Swedish identifier (e.g., temporary workers, students). | Pattern match, context, and checksum. | ### South Africa | FieldType | Description | Detection Method | |------------|---------------------------------------------------------------------------------------------------------|------------------------------------------| | ZA_ID_NUMBER | The South African identity number is a 13-digit identifier in the `YYMMDDSSSSCAZ` format, where the trailing digit is validated with the Luhn algorithm. | Pattern match, context, and checksum. | | ZA_PASSPORT | The South African passport number is a 9-character identifier with prefix letter A, D, M, or T followed by 8 digits. | Pattern match, context, and validation. | | ZA_INCOME_TAX_NUMBER | The South African SARS income tax reference number is a 10-digit numeric identifier, commonly starting with 0, 1, 2, 3, or 9. | Pattern match, context, and validation. | | ZA_DRIVER_LICENSE | The South African eNaTIS driver's licence number is an alphanumeric identifier of 10–14 characters. | Pattern match, context, and validation. | | ZA_VAT_NUMBER | The South African VAT registration number is a 10-digit identifier starting with 4. | Pattern match, context, and validation. | | ZA_COMPANY_REGISTRATION | The South African CIPC company registration number uses modern `YYYY/NNNNNN/NN` format or legacy prefixed formats such as CK. | Pattern match, context, and validation. | | ZA_TRAFFIC_REGISTER_NUMBER | The South African eNaTIS traffic register number is a 13-digit identifier for foreigners and organisations, disambiguated from ZA_ID_NUMBER via validation. | Pattern match, context, and validation. | | ZA_LICENSE_PLATE | The South African vehicle licence plate uses provincial formats such as compact suffix forms (e.g. KD93GKGP) or spaced layouts (e.g. DK 28 LF GP). | Pattern match, context, and validation. | | ZA_MOBILE_NUMBER | The South African mobile (cellular) number uses a 9-digit national significant number with country code +27 or domestic trunk prefix 0 (primarily 06x, 07x, and cellular 08x ranges). | `phonenumbers` match, line-type filter, and context. | | ZA_TELEPHONE_NUMBER | The South African telephone number covers geographic landlines (01x–05x) and non-mobile service lines such as toll-free (080), sharecall (086), and VoIP (087). | `phonenumbers` match, line-type filter, and context. | ### Thai | FieldType | Description | Detection Method | |------------|---------------------------------------------------------------------------------------------------------|------------------------------------------| | TH_TNIN | The Thai National ID Number (TNIN) is a unique 13-digit number issued to all Thai residents. | Pattern match, context and custom logic. | ### Turkey | FieldType | Description | Detection Method | |------------|---------------------------------------------------------------------------------------------------------|------------------------------------------| | TR_NATIONAL_ID | The Turkish National Identification Number (TCKN) is a unique 11-digit number issued to all Turkish citizens. | Pattern match, context and checksum. | | TR_LICENSE_PLATE | Turkish vehicle license plate (plaka): 2-digit province code (01–81), 1–3 letters (A–Z, excluding Q, W, X), and 2–4 digits. Standard civilian format only. Legal basis: KTK Madde 23. | Pattern match, context and province code validation. | ### Philippines | FieldType | Description | Detection Method | |------------|---------------------------------------------------------------------------------------------------------|------------------------------------------| | PH_UMID | Philippine Unified Multi-Purpose ID (UMID) / Common Reference Number (CRN). 12-digit format (e.g., 0111-1234567-8). Acts as a master ID connecting SSS, GSIS, PhilHealth, and Pag-IBIG. Disabled by default. | Pattern match and context | ### Germany | Entity Type | Description | Detection Method | | --- | --- | --- | | DE_TAX_ID | German Steueridentifikationsnummer (Steuer-IdNr.): unique 11-digit personal tax identification number issued by the Bundeszentralamt für Steuern. Legal basis: §§ 139a–139e AO. | Pattern match, context and checksum (ISO 7064 Mod 11, 10) | | DE_TAX_NUMBER | German Steuernummer: tax number assigned by the local Finanzamt, in ELSTER unified 13-digit format or state-specific slash-separated formats. Legal basis: § 139a AO. | Pattern match and context | | DE_PASSPORT | German Reisepassnummer: 9-character alphanumeric document number using ICAO Doc 9303 character set. Legal basis: Passgesetz (PassG) § 4. | Pattern match and context | | DE_ID_CARD | German Personalausweisnummer: 9-character alphanumeric document number (nPA since Nov 2010) or legacy T+8-digit format. Legal basis: Personalausweisgesetz (PAuswG). | Pattern match and context | | DE_SOCIAL_SECURITY | German Rentenversicherungsnummer (RVNR): 12-character identifier encoding birth date, surname initial, serial and check digit. Legal basis: § 147 SGB VI. | Pattern match, context and checksum (DRV algorithm) | | DE_HEALTH_INSURANCE | German Krankenversicherungsnummer (KVNR): 10-character identifier (1 letter + 9 digits) printed on the elektronische Gesundheitskarte (eGK). Legal basis: § 290 SGB V. Special category: health data (DSGVO Art. 9). | Pattern match, context and checksum (GKV-Spitzenverband algorithm) | | DE_KFZ | German KFZ-Kennzeichen (vehicle registration plate): district code (1–3 letters), identifier (1–2 letters), and 1–4 digits, optionally with E (electric) or H (historic) suffix. Legal basis: Fahrzeug-Zulassungsverordnung (FZV) § 8. | Pattern match and context | | DE_HANDELSREGISTER | German Handelsregisternummer: commercial register number with HRA (sole traders / partnerships) or HRB (corporations) prefix followed by 1–6 digits. HRA entries directly identify natural persons (sole traders). Legal basis: §§ 9, 14 HGB, HRV. | Pattern match and context | | DE_PLZ | German Postleitzahl (postal code): 5-digit code in the range 01001–99998. Constitutes personal data in combination with other address fields (DSGVO Art. 4 Nr. 1). **High false-positive risk** – only reliable with address-context words present; base confidence is 0.05. Legal basis: DSGVO Art. 4 Nr. 1. | Pattern match and context (context required for actionable results) | ### Medical / Clinical Detected using the `MedicalNERRecognizer` (requires the `transformers` extra). Uses the [blaze999/Medical-NER](https://huggingface.co/blaze999/Medical-NER) model by default. |Entity Type | Description | Detection Method | | --- | --- | --- | |MEDICAL_DISEASE_DISORDER | A disease or disorder (e.g. diabetes, hypertension). | NER model (HuggingFace transformers) | |MEDICAL_MEDICATION | A medication or drug name (e.g. metformin, aspirin). | NER model (HuggingFace transformers) | |MEDICAL_THERAPEUTIC_PROCEDURE | A therapeutic or diagnostic procedure (e.g. surgery, MRI). | NER model (HuggingFace transformers) | |MEDICAL_CLINICAL_EVENT | A clinical event (e.g. admission, discharge). | NER model (HuggingFace transformers) | |MEDICAL_BIOLOGICAL_ATTRIBUTE | A biological attribute or measurement (e.g. blood pressure, BMI). | NER model (HuggingFace transformers) | |MEDICAL_BIOLOGICAL_STRUCTURE | A biological or anatomical structure (e.g. liver, left ventricle). | NER model (HuggingFace transformers) | |MEDICAL_FAMILY_HISTORY | A family medical history reference. | NER model (HuggingFace transformers) | |MEDICAL_HISTORY | A patient medical history reference. | NER model (HuggingFace transformers) | ## Adding a custom PII entity See [this documentation](analyzer/adding_recognizers.md) for instructions on how to add a new Recognizer for a new type of PII entity. ## Complementing Presidio with Azure AI Language PII [Azure AI Language PII](https://learn.microsoft.com/en-us/azure/ai-services/language-service/personally-identifiable-information/overview) is a cloud-based service that provides Natural Language Processing (NLP) features for detecting PII in text. A list of supported entities by Azure AI Language PII [can be found here](https://learn.microsoft.com/en-us/azure/ai-services/language-service/personally-identifiable-information/concepts/entity-categories). To add Azure AI language into Presidio, [see this sample](samples/python/text_analytics/index.md#how-to-integrate-azure-ai-language-into-presidio). ## Complementing Presidio with Azure Health Data Services PHI Azure Health Data Services PHI is a cloud-based service that provides Natural Language Processing (NLP) features for detecting PHI in text. A list of supported entities by Azure Health Data Services PHI [can be found here](https://learn.microsoft.com/en-us/azure/healthcare-apis/deidentification/overview). To add Azure AI language into Presidio, [see this sample](samples/python/ahds/index.md#how-to-integrate-azure-health-data-services-dei-dentification-into-presidio). ### Connecting to 3rd party PII detectors See [this documentation](analyzer/adding_recognizers.md#creating-a-remote-recognizer) for instructions on how to implement an external PII detector for a new or existing type of PII entity. --- ### Text Anonymization (`docs/text_anonymization.md`) # Text anonymization Presidio's features two main modules for anonymization PII in text: - [Presidio analyzer](analyzer/index.md): Identification of PII in text - [Presidio anonymizer](anonymizer/index.md): De-identify detected PII entities using different operators In most cases, we would run the Presidio analyzer to detect where PII entities exist, and then the Presidio anonymizer to remove those using specific operators (such as redact, replace, hash or encrypt) This figure presents the overall flow in high level: - The [Presidio Analyzer](analyzer/index.md) holds multiple recognizers, each one capable of detecting specific PII entities. These recognizers leverage regular expressions, deny lists, checksum, rule based logic, Named Entity Recognition ML models and context from surrounding words. - The [Presidio Anonymizer](anonymizer/index.md) holds multiple operators, each one can be used to anonymize the PII entity in a different way. Additionally, it can be used to de-anonymize an already anonymized entity (For example, decrypt an encrypted entity) --- ### Analyzer Adding Recognizers (`docs/analyzer/adding_recognizers.md`) # Supporting detection of new types of PII entities Presidio can be extended to support detection of new types of PII entities, and to support additional languages. These PII recognizers could be added **via code** or **ad-hoc as part of the request**. ## Introduction to recognizer development Entity recognizers are Python objects capable of detecting one or more entities in a specific language. In order to extend Presidio's detection capabilities to new types of PII entities, these `EntityRecognizer` objects should be added to the existing list of recognizers. ## Types of recognizer classes in Presidio The following class diagram shows the different types of recognizer families Presidio contains. - The `EntityRecognizer` is an abstract class for all recognizers. - The `RemoteRecognizer` is an abstract class for calling external PII detectors. See more info [here](#creating-a-remote-recognizer). - The abstract class `LocalRecognizer` is implemented by all recognizers running within the Presidio-analyzer process. - The `PatternRecognizer` is an class for supporting regex and deny-list based recognition logic, including validation (e.g., with checksum) and context support. See an example [here](#simple-example). ## Extending the analyzer for additional PII entities 1. Create a new class based on `EntityRecognizer`. 2. Add the new recognizer to the recognizer registry so that the `AnalyzerEngine` can use the new recognizer during analysis. ### Simple example For simple recognizers based on regular expressions or deny-lists, we can leverage the provided `PatternRecognizer`: ```python from presidio_analyzer import PatternRecognizer titles_recognizer = PatternRecognizer(supported_entity="TITLE", deny_list=["Mr.","Mrs.","Miss"]) ``` Calling the recognizer itself: ```python titles_recognizer.analyze(text="Mr. Schmidt", entities="TITLE") ``` Adding it to the list of recognizers: ```python from presidio_analyzer import AnalyzerEngine, RecognizerRegistry registry = RecognizerRegistry() registry.load_predefined_recognizers() # Add the recognizer to the existing list of recognizers registry.add_recognizer(titles_recognizer) # Set up analyzer with our updated recognizer registry analyzer = AnalyzerEngine(registry=registry) # Run with input text text="His name is Mr. Jones" results = analyzer.analyze(text=text, language="en") print(results) ``` Alternatively, we can add the recognizer directly to the existing registry: ```python from presidio_analyzer import AnalyzerEngine analyzer = AnalyzerEngine() analyzer.registry.add_recognizer(titles_recognizer) results = analyzer.analyze(text=text, language="en") print(results) ``` For pattern based recognizers, it is possible to change the regex flags, either for one recognizer or for all. For one recognizer, use the `global_regex_flags` parameter in the `PatternRecognizer` constructor. For all recognizers, use the `global_regex_flags` parameter in the `RecognizerRegistry` constructor: ```python from presidio_analyzer import AnalyzerEngine, RecognizerRegistry import regex as re registry = RecognizerRegistry(global_regex_flags=re.DOTALL | re.MULTILINE | re.IGNORECASE) engine = AnalyzerEngine(registry=registry) engine.analyze(...) ``` ### Creating a new `EntityRecognizer` in code To create a new recognizer via code: 1. Create a new Python class which implements [LocalRecognizer](https://github.com/data-privacy-stack/presidio/blob/main/presidio-analyzer/presidio_analyzer/local_recognizer.py). (`LocalRecognizer` implements the base [EntityRecognizer](https://github.com/data-privacy-stack/presidio/blob/main/presidio-analyzer/presidio_analyzer/entity_recognizer.py) class) This class has the following functions: i. load: load a model / resource to be used during recognition ```python def load(self) ``` ii. analyze: The main function to be called for getting entities out of the new recognizer: ```python def analyze(self, text, entities, nlp_artifacts) ``` Notes: 1. Each recognizer has access to different NLP assets such as tokens, [lemmas](https://en.wikipedia.org/wiki/Lemma_(morphology)), and more. These are given through the `nlp_artifacts` parameter. Refer to the [source code](https://github.com/data-privacy-stack/presidio/blob/main/presidio-analyzer/presidio_analyzer/entity_recognizer.py) for more information. 2. The `analyze` method should return a list of [RecognizerResult](https://github.com/data-privacy-stack/presidio/blob/main/presidio-analyzer/presidio_analyzer/recognizer_result.py). 2. Add it to the recognizer registry using `registry.add_recognizer(my_recognizer)`. For more examples, see the [Customizing Presidio Analyzer](../samples/python/customizing_presidio_analyzer.ipynb) jupyter notebook. ### Creating a remote recognizer A remote recognizer is an `EntityRecognizer` object interacting with an external service. The external service could be a 3rd party PII detection service or a custom service deployed in parallel to Presidio. [Sample implementation of a `RemoteRecognizer`](../samples/python/example_remote_recognizer.py). In this example, an external PII detection service exposes two APIs: `detect` and `supported_entities`. The class implemented here, `ExampleRemoteRecognizer`, uses the `requests` package to call the external service via HTTP. In this code snippet, we simulate the external PII detector by using the Presidio analyzer. In reality, we would adapt this code to fit the external PII detector we have in hand. For an example of integrating a `RemoteRecognizer` with Presidio-Analyzer, see [this example](../samples/python/integrating_with_external_services.ipynb). ### Creating pre-defined recognizers Once a recognizer is created, it can either be added to the `RecognizerRegistry` via the `add_recognizer` method, or it could be added into the list of predefined recognizers. To add a recognizer to the list of pre-defined recognizers: 1. Clone the repo. 2. Create a file containing the new recognizer Python class. 3. Add the recognizer to the `recognizers` in the [`default_recognizers`](https://github.com/data-privacy-stack/presidio/blob/main/presidio-analyzer/presidio_analyzer/conf/default_recognizers.yaml) config. Details of recognizer parameters are given [Here](./recognizer_registry_provider.md#the-recognizer-parameters). 4. If the recognizer is country-specific (file lives under `predefined_recognizers/country_specific//`), declare its country by setting the class-level `COUNTRY_CODE = ""` attribute. This makes the recognizer participate in the country filter — see [Filtering recognizers by country](./filtering_by_country.md). Generic / locale-agnostic recognizers leave `COUNTRY_CODE` unset (defaults to `None`). 5. Optional: Update documentation (e.g., the [supported entities list](../supported_entities.md)). ### Azure AI Language recognizer On how to integrate Presidio with Azure AI Language PII detection service, and a sample for a Text Analytics Remote Recognizer, refer to the [Azure Text Analytics Integration document](../samples/python/text_analytics/index.md). ### Azure Health Data Services (AHDS) de-identification recognizer On how to integrate Presidio with AHDS De-Identification Protected Health Information (PHI) detection service, and a sample for a ADHS Remote Recognizer, refer to the [AHDS de-Identification Integration document](../samples/python/ahds/index.md). ### Language Model-based PII/PHI detection recognizer Presidio supports language model-based entity detection using LLMs and SLMs for flexible PII/PHI recognition. The current implementation uses LangExtract with Ollama (local models). For full setup instructions and usage examples, see the [Language Model-based PII/PHI Detection guide](../samples/python/langextract/index.md). ### Creating ad-hoc recognizers In addition to recognizers in code, it is possible to create ad-hoc recognizers via the Presidio Analyzer API for regex and deny-list based logic. These recognizers, in JSON form, are added to the `/analyze` request and are only used in the context of this request. - The json structure for a regex ad-hoc recognizer is the following: ```json { "text": "John Smith drivers license is AC432223. Zip code: 10023", "language": "en", "ad_hoc_recognizers":[ { "name": "Zip code Recognizer", "supported_language": "en", "patterns": [ { "name": "zip code (weak)", "regex": "(\\b\\d{5}(?:\\-\\d{4})?\\b)", "score": 0.01 } ], "context": ["zip", "code"], "supported_entity":"ZIP" } ] } ``` - The json structure for a deny-list based recognizers is the following: ```json { "text": "Mr. John Smith's drivers license is AC432223", "language": "en", "ad_hoc_recognizers":[ { "name": "Mr. Recognizer", "supported_language": "en", "deny_list": ["Mr", "Mr.", "Mister"], "supported_entity":"MR_TITLE" }, { "name": "Ms. Recognizer", "supported_language": "en", "deny_list": ["Ms", "Ms.", "Miss", "Mrs", "Mrs."], "supported_entity":"MS_TITLE" } ] } ``` In both examples, the `/analyze` request is extended with a list of `ad_hoc_recognizers`, which could be either `patterns`, `deny_list` or both. Additional examples can be found in the [OpenAPI spec](../api-docs/api-docs.html). ### Reading pattern recognizers from YAML Recognizers can be loaded from a YAML file, which allows users to add recognition logic without writing code. An example YAML file can be found [here](https://github.com/data-privacy-stack/presidio/blob/main/presidio-analyzer/presidio_analyzer/conf/default_recognizers.yaml). Once the YAML file is created, it can be loaded into the `RecognizerRegistry` instance. This example creates a `RecognizerRegistry` holding only the recognizers in the YAML file: ``` python from presidio_analyzer import AnalyzerEngine from presidio_analyzer.recognizer_registry import RecognizerRegistryProvider recognizer_registry_conf_file = "./analyzer/recognizers-config.yml" provider = RecognizerRegistryProvider( conf_file=recognizer_registry_conf_file ) registry = provider.create_recognizer_registry() analyzer = AnalyzerEngine(registry=registry) results = analyzer.analyze(text="My name is Morris", language="en") print(results) ``` This example adds the new recognizers to the predefined recognizers in Presidio: ``` python from presidio_analyzer import AnalyzerEngine, RecognizerRegistry yaml_file = "recognizers.yaml" registry = RecognizerRegistry() registry.load_predefined_recognizers() registry.add_recognizers_from_yaml(yaml_file) analyzer = AnalyzerEngine() analyzer.analyze(text="Mr. and Mrs. Smith", language="en") ``` Further reading: 1. [PII detection in different languages](languages.md). 1. [Customizing the NLP model](customizing_nlp_models.md). 1. [Best practices for developing PII recognizers](developing_recognizers.md). 1. [Code samples for customizing Presidio Analyzer with new recognizers](../samples/python/customizing_presidio_analyzer.ipynb). --- ### Analyzer Analyzer Engine Provider (`docs/analyzer/analyzer_engine_provider.md`) # Configuring the Analyzer Engine from file Presidio uses `AnalyzerEngineProvider` to load `AnalyzerEngine` configuration from file. Configuration can be loaded in three different ways: ## Using a single file Create an `AnalyzerEngineProvider` using a single configuration file and set its path to `analyzer_engine_conf_file`, then create `AnalyzerEngine` based on it: ```python from presidio_analyzer import AnalyzerEngine, AnalyzerEngineProvider analyzer_conf_file = "./analyzer/analyzer-config-all.yml" provider = AnalyzerEngineProvider( analyzer_engine_conf_file=analyzer_conf_file ) analyzer = provider.create_engine() results = analyzer.analyze(text="My name is Morris", language="en") print(results) ``` An example configuration file: ```yaml supported_languages: - en default_score_threshold: 0 nlp_configuration: nlp_engine_name: spacy models: - lang_code: en model_name: en_core_web_lg - lang_code: es model_name: es_core_news_md ner_model_configuration: model_to_presidio_entity_mapping: PER: PERSON PERSON: PERSON LOC: LOCATION LOCATION: LOCATION GPE: LOCATION ORG: ORGANIZATION DATE: DATE_TIME TIME: DATE_TIME NORP: NRP low_confidence_score_multiplier: 0.4 low_score_entity_names: - ORGANIZATION - ORG default_score: 0.85 recognizer_registry: global_regex_flags: 26 recognizers: - name: CreditCardRecognizer supported_languages: - en supported_entity: IT_FISCAL_CODE type: predefined score_thresholds: default: 0.4 CREDIT_CARD: 0.7 - name: ItFiscalCodeRecognizer type: predefined ``` The configuration file contains the following parameters: - `supported_languages`: A list of supported languages that the analyzer will support. - `default_score_threshold`: A score that determines the minimal threshold for detection. - `nlp_configuration`: Configuration given to the NLP engine which will detect the PIIs and extract features for the downstream logic. - `recognizer_registry`: All the recognizers that will be used by the analyzer. Each recognizer entry can define `score_thresholds`, using `default` as its fallback and entity names for overrides. !!! note "Defining confidence thresholds" Presidio supports multiple types of score thresholds: default ones (for the analyzer and each recognizer), per-request thresholds, and per-entity thresholds. Supplying `analyzer.analyze(score_threshold=...)` bypasses recognizer-level thresholds for that request and applies the supplied threshold to every result. The order of importance is: `analyzer.analyze(score_threshold=...)` > an entity specific threshold > a recognizer default threshold (`default`) > the Presidio Analyzer `default_score_threshold`. !!! note "Note" `supported_languages` must be identical to the same field in recognizer_registry ## Using multiple files Create an `AnalyzerEngineProvider` using three different configuration files for each of the following components: - Analyzer - NLP Engine - Recognizer Registry !!! note "Note" Each of these parameters is optional and in case it's not set, the default configuration will be used. ```python from presidio_analyzer import AnalyzerEngine, AnalyzerEngineProvider analyzer_conf_file = "./analyzer/analyzer-config.yml" nlp_engine_conf_file = "./analyzer/nlp-config.yml" recognizer_registry_conf_file = "./analyzer/recognizers-config.yml" provider = AnalyzerEngineProvider( analyzer_engine_conf_file=analyzer_conf_file, nlp_engine_conf_file=nlp_engine_conf_file, recognizer_registry_conf_file=recognizer_registry_conf_file, ) analyzer = provider.create_engine() results = analyzer.analyze(text="My name is Morris", language="en") print(results) ``` The structure of the configuration files is as follows: - Analyzer engine configuration file: ```yaml supported_languages: - en default_score_threshold: 0 ``` - NLP engine configuration file structure is examined thoroughly in the [Customizing the NLP model](customizing_nlp_models.md) section. - Recognizer registry configuration file structure is examined thoroughly in the [Customizing recognizer registry from file](recognizer_registry_provider.md) section. ## Using the default configuration Create an `AnalyzerEngineProvider` without any parameters. This will load the default configuration: ```python from presidio_analyzer import AnalyzerEngine, AnalyzerEngineProvider provider = AnalyzerEngineProvider().create_engine() results = provider.analyze(text="My name is Morris", language="en") print(results) ``` The default configuration of `AnalyzerEngine` is defined in the following files: - [Analyzer Engine](https://github.com/data-privacy-stack/presidio/blob/main/presidio-analyzer/presidio_analyzer/conf/default_analyzer.yaml) - [NLP Engine](https://github.com/data-privacy-stack/presidio/blob/main/presidio-analyzer/presidio_analyzer/conf/default.yaml) - [Recognizer Registry](https://github.com/data-privacy-stack/presidio/blob/main/presidio-analyzer/presidio_analyzer/conf/default_recognizers.yaml) ## Enabling and disabling recognizers In general, recognizers that are not added to the configuration would not be created, with one exception. ### Enabling/Disabling the NLP recognizer One exception to this is the recognizer which extracts the `NlpEngine` entities (e.g. `SpacyRecognizer` when the `NlpEngine` is `SpacyNlpEngine`; `TransformersRecognizer` when the engine is `TransformersNlpEngine` and `StanzaRecognizer` when the engine is `StanzaNlpEngine`). Recognizers (including the NLP recognizer) could be disabled by defining `enabled=false` in the YAML configuration. For example: ```yaml recognizer_registry: global_regex_flags: 26 recognizers: - name: SpacyRecognizer type: predefined enabled: false - name: CreditCardRecognizer type: predefined enabled: true supported_languages: - en default_score_threshold: 0.7 nlp_configuration: nlp_engine_name: spacy models: - lang_code: en model_name: en_core_web_lg ``` In this example, the `SpacyRecognizer` is disabled, and the `CreditCardRecognizer` is enabled, resulting in only the `CREDIT_CARD` PII entity to be returned if detected. ### Adding context words in YAML recognizers Recognizers defined in YAML can also include a `context` field. When used with `AnalyzerEngine` and a context enhancer, these words boost the score if they appear near the detected entity. Example: ```yaml recognizers: - name: "Date of Birth Recognizer" supported_entity: "DATE_TIME" supported_language: "en" patterns: - name: "DOB without slashes" regex: "((19|20)\\d{2}(0[1-9]|1[0-2])(0[1-9]|[12]\\d|3[01]))" score: 0.8 context: - DOB ``` ``` /* Detailed source-code truncated for AI context efficiency. */ ``` --- ### Analyzer Customizing Nlp Models (`docs/analyzer/customizing_nlp_models.md`) # Customizing the NLP engine in Presidio Analyzer Presidio uses NLP engines for two main tasks: NER based PII identification, and feature extraction for downstream rule based logic (such as leveraging context words for improved detection). While Presidio comes with an open-source model (the `en_core_web_lg` model from spaCy), additional NLP models and frameworks could be plugged in, either public or proprietary. These models can be trained or downloaded from existing NLP frameworks like [spaCy](https://spacy.io/usage/models), [Stanza](https://github.com/stanfordnlp/stanza) and [transformers](https://github.com/huggingface/transformers). In addition, other types of NLP frameworks [can be integrated into Presidio](developing_recognizers.md#machine-learning-ml-based-or-rule-based). ## Setting up a custom NLP model - [spaCy or stanza](nlp_engines/spacy_stanza.md) - [transformers](nlp_engines/transformers.md) ## Configure Presidio to use the new model Configuration can be done in two ways: - **Via code**: Create an `NlpEngine` using the `NlpEnginerProvider` class, and pass it to the `AnalyzerEngine` as input: ```python from presidio_analyzer import AnalyzerEngine, RecognizerRegistry from presidio_analyzer.nlp_engine import NlpEngineProvider # Create configuration containing engine name and models configuration = { "nlp_engine_name": "spacy", "models": [{"lang_code": "es", "model_name": "es_core_news_md"}, {"lang_code": "en", "model_name": "en_core_web_lg"}], } # Create NLP engine based on configuration provider = NlpEngineProvider(nlp_configuration=configuration) nlp_engine_with_spanish = provider.create_engine() # Pass the created NLP engine and supported_languages to the AnalyzerEngine analyzer = AnalyzerEngine( nlp_engine=nlp_engine_with_spanish, supported_languages=["en", "es"] ) # Analyze in different languages results_spanish = analyzer.analyze(text="Mi nombre es Morris", language="es") print(results_spanish) results_english = analyzer.analyze(text="My name is Morris", language="en") print(results_english) ``` - **Via configuration**: Set up the models which should be used in the [default `conf` file](https://github.com/data-privacy-stack/presidio/blob/main/presidio-analyzer/presidio_analyzer/conf/default.yaml). An example Conf file: ```yaml nlp_engine_name: spacy models: - lang_code: en model_name: en_core_web_lg - lang_code: es model_name: es_core_news_md ner_model_configuration: labels_to_ignore: - O model_to_presidio_entity_mapping: PER: PERSON LOC: LOCATION ORG: ORGANIZATION AGE: AGE ID: ID DATE: DATE_TIME low_confidence_score_multiplier: 0.4 low_score_entity_names: - ID - ORG ``` The `ner_model_configuration` section contains the following parameters: - `labels_to_ignore`: A list of labels to ignore. For example, `O` (no entity) or entities you are not interested in returning. - `model_to_presidio_entity_mapping`: A mapping between the transformers model labels and the Presidio entity types. - `low_confidence_score_multiplier`: A multiplier to apply to the score of entities with low confidence. - `low_score_entity_names`: A list of entity types to apply the low confidence score multiplier to. The [default conf file](https://github.com/data-privacy-stack/presidio/blob/main/presidio-analyzer/presidio_analyzer/conf/default.yaml) is read during the default initialization of the `AnalyzerEngine`. Alternatively, the path to a custom configuration file can be passed to the `NlpEngineProvider`: ```python from presidio_analyzer import AnalyzerEngine, RecognizerRegistry from presidio_analyzer.nlp_engine import NlpEngineProvider LANGUAGES_CONFIG_FILE = "./docs/analyzer/languages-config.yml" # Create NLP engine based on configuration file provider = NlpEngineProvider(conf_file=LANGUAGES_CONFIG_FILE) nlp_engine_with_spanish = provider.create_engine() # Pass created NLP engine and supported_languages to the AnalyzerEngine analyzer = AnalyzerEngine( nlp_engine=nlp_engine_with_spanish, supported_languages=["en", "es"] ) # Analyze in different languages results_spanish = analyzer.analyze(text="Mi nombre es David", language="es") print(results_spanish) results_english = analyzer.analyze(text="My name is David", language="en") print(results_english) ``` In this examples we: a. create an `NlpEngine` holding two spaCy models (one in English: `en_core_web_lg` and one in Spanish: `es_core_news_md`). b. define the `supported_languages` parameter accordingly. c. pass requests in each of these languages. !!! note "Note" Presidio can currently use one NER model per language via the `NlpEngine`. If multiple are required, consider wrapping NER models as additional recognizers ([see sample here](https://github.com/data-privacy-stack/presidio/blob/main/docs/samples/python/example_remote_recognizer.py)). ## Leverage frameworks other than spaCy, Stanza and transformers for ML based PII detection In addition to the built-in spaCy/Stanza/transformers capabilities, it is possible to create new recognizers which serve as interfaces to other models. For more information: - [Remote recognizer documentation](adding_recognizers.md#creating-a-remote-recognizer) and [samples](../samples/python/integrating_with_external_services.ipynb). - [Flair recognizer example](../samples/python/flair_recognizer.py) For considerations for creating such recognizers, see the [best practices for adding ML recognizers documentation](developing_recognizers.md#machine-learning--ml--based-or-rule-based). --- ### Analyzer Decision Process (`docs/analyzer/decision_process.md`) # The Presidio-analyzer decision process ## Background Presidio-analyzer's decision process exposes information on why a specific PII was detected. Such information could contain: - Which recognizer detected the entity - Which regex pattern was used - Interpretability mechanisms in ML models - Which context words improved the score - Confidence scores before and after each step And more. ## Usage The decision process can be leveraged in two ways: 1. Presidio-analyzer can log its decision process into a designated logger, which allows you to investigate a specific api request, by exposing a `correlation-id` as part of the api response headers. 2. The decision process can be returned as part of the `/analyze` response. ### Getting the decision process as part of the response The decision process result can be added to the response. To enable it, call the `analyze` method with `return_decision_process` set as True. For example: === "HTTP" ```sh curl -d '{ "text": "John Smith drivers license is AC432223", "language": "en", "return_decision_process": true}' -H "Content-Type: application/json" -X POST http://localhost:3000/analyze ``` === "Python" ```python from presidio_analyzer import AnalyzerEngine # Set up the engine, loads the NLP module (spaCy model by default) # and other PII recognizers analyzer = AnalyzerEngine() # Call analyzer to get results results = analyzer.analyze(text='My phone number is 212-555-5555', entities=['PHONE_NUMBER'], language='en', return_decision_process=True) # Get the decision process results for the first result print(results[0].analysis_explanation) ``` ### Logging the decision process Logging of the decision process is turned off by default. To turn it on, create the `AnalyzerEngine` object with `log_decision_process=True`. For example: ```python from presidio_analyzer import AnalyzerEngine # Set up the engine, loads the NLP module (spaCy model by default) # and other PII recognizers analyzer = AnalyzerEngine(log_decision_process=True) # Call analyzer to get results results = analyzer.analyze(text='My phone number is 212-555-5555', entities=['PHONE_NUMBER'], language='en', correlation_id="xyz") ``` The decision process logs will be written to standard output. Note that it is possible to define a `correlation-id` which is the trace identification. It will help you to query the stdout logs. The id can be retrieved from each API response header: `x-correlation-id`. By having the traces written into the `stdout` it's very easy to configure a monitoring solution to ease the process of reading processing the tracing logs in a distributed system. ## Examples For the a request with the following text: ```text My name is Bart Simpson, my Credit card is: 4095-2609-9393-4932, my phone is 425 8829090 ``` The following traces will be written to log, with this format: `[Date Time][decision_process][Log Level][Unique Correlation ID][Trace Message]` ``` /* Detailed source-code truncated for AI context efficiency. */ ``` ## Writing custom decision process for a recognizer When creating new PII recognizers, it is possible to add information about the recognizer's decision process. This information will be traced or returned to the user, depending on the configuration. For example, the [spacy_recognizer.py](https://github.com/data-privacy-stack/presidio/blob/main/presidio-analyzer/presidio_analyzer/predefined_recognizers/nlp_engine_recognizers/spacy_recognizer.py) implements a custom trace as follows: ```python SPACY_DEFAULT_EXPLANATION = "Identified as {} by Spacy's Named Entity Recognition" def build_spacy_explanation(recognizer_name, original_score, entity): explanation = AnalysisExplanation( recognizer=recognizer_name, original_score=original_score, textual_explanation=SPACY_DEFAULT_EXPLANATION.format(entity)) return explanation ``` The `textual_explanation` field in `AnalysisExplanation` class allows you to add your own custom text into the final trace which will be written. !!! note "Note" These traces leverage the Python `logging` mechanisms. In the default configuration, A `StreamHandler` is used to write these logs to `sys.stdout`. !!! warning "Warning" Decision-process traces explain why PIIs were detected, but not why they were not detected! --- ### Analyzer Developing Recognizers (`docs/analyzer/developing_recognizers.md`) # Recognizers Development - Best Practices and Considerations Recognizers are the main building blocks in Presidio. Each recognizer is in charge of detecting one or more entities in one or more languages. Recognizers define the logic for detection, as well as the confidence a prediction receives and a list of words to be used when context is leveraged. ## Implementation Considerations ### Accuracy Each recognizer, regardless of its complexity, could have false positives and false negatives. When adding new recognizers, we try to balance the effect of each recognizer on the entire system. A recognizer with many false positives would affect the system's usability, while a recognizer with many false negatives might require more work before it can be integrated. For reproducibility purposes, it is be best to note how the recognizer's accuracy was tested, and on which datasets. For tools and documentation on evaluating and analyzing recognizers, refer to the [presidio-research GitHub repository](https://github.com/data-privacy-stack/presidio-research). !!! note "Note" When contributing recognizers to the Presidio OSS, new predefined recognizers should be added to the [supported entities list](../supported_entities.md), and follow the [contribution guidelines](https://github.com/data-privacy-stack/presidio/blob/main/CONTRIBUTING.md). ### Performance Make sure your recognizer doesn't take too long to process text. Anything above 100ms per request with 100 tokens is probably not good enough. ### Environment When adding new recognizers that have 3rd party dependencies, make sure that the new dependencies don't interfere with Presidio's dependencies. In the case of a conflict, one can create an isolated model environment (outside the main presidio-analyzer process) and implement a [`RemoteRecognizer`](https://github.com/data-privacy-stack/presidio/blob/main/presidio-analyzer/presidio_analyzer/remote_recognizer.py) on the presidio-analyzer side to interact with the model's endpoint. ## Recognizer Types Generally speaking, there are three types of recognizers: ### Deny Lists A deny list is a list of words that should be removed during text analysis. For example, it can include a list of titles (`["Mr.", "Mrs.", "Ms.", "Dr."]` to detect a "Title" entity.) See [this documentation](adding_recognizers.md) on adding a new recognizer. The [`PatternRecognizer`](https://github.com/data-privacy-stack/presidio/blob/main/presidio-analyzer/presidio_analyzer/pattern_recognizer.py) class has built-in support for a deny-list input. ### Pattern Based Pattern based recognizers use regular expressions to identify entities in text. See [this documentation](adding_recognizers.md) on adding a new recognizer via code. The [`PatternRecognizer`](https://github.com/data-privacy-stack/presidio/blob/main/presidio-analyzer/presidio_analyzer/pattern_recognizer.py) class should be extended. See some examples here: !!! example "Examples" Examples of pattern based recognizers are the [`CreditCardRecognizer`](https://github.com/data-privacy-stack/presidio/blob/main/presidio-analyzer/presidio_analyzer/predefined_recognizers/generic/credit_card_recognizer.py) and [`EmailRecognizer`](https://github.com/data-privacy-stack/presidio/blob/main/presidio-analyzer/presidio_analyzer/predefined_recognizers/generic/email_recognizer.py). ### Machine Learning (ML) Based or Rule-Based Many PII entities are undetectable using naive approaches like deny-lists or regular expressions. In these cases, we would wish to utilize a Machine Learning model capable of identifying entities in free text, or a rule-based recognizer. #### ML: Utilize SpaCy, Stanza or Transformers Presidio currently uses [spaCy](https://spacy.io/) as a framework for text analysis and Named Entity Recognition (NER), and [stanza](https://stanfordnlp.github.io/stanza/) and [huggingface transformers](https://huggingface.co/docs/transformers/index) as an alternative. To avoid introducing new tools, it is recommended to first try to use `spaCy`, `stanza` or `transformers` over other tools if possible. `spaCy` provides decent results compared to state-of-the-art NER models, but with much better computational performance. `spaCy`, `stanza` and `transformers` models could be trained from scratch, used in combination with pre-trained embeddings, or be fine-tuned. In addition to those, it is also possible to use other ML models. In that case, a new `EntityRecognizer` should be created. See an example using [Flair here](https://github.com/data-privacy-stack/presidio/blob/main/docs/samples/python/flair_recognizer.py). #### Apply Custom Logic In some cases, rule-based logic provides reasonable ways for detecting entities. The Presidio `EntityRecognizer` API allows you to use `spaCy` extracted features like lemmas, part of speech, dependencies and more to create your logic. When integrating such logic into Presidio, a class inheriting from the [`EntityRecognizer`](https://github.com/data-privacy-stack/presidio/blob/main/presidio-analyzer/presidio_analyzer/entity_recognizer.py) should be created. !!! attention "Considerations for selecting one option over another" - Accuracy. - Ease of integration. - Runtime considerations (For example if the new model requires a GPU). - 3rd party dependencies of the new model vs. the existing `presidio-analyzer` package. --- ### Analyzer Filtering By Country (`docs/analyzer/filtering_by_country.md`) # Filtering recognizers by country Many Presidio deployments only care about a subset of the world's PII — e.g. a US-only healthcare app, or an EU-only data pipeline. Loading every predefined country-specific recognizer in those scenarios means more false positives, more memory, and more noise in the analyzer output. `RecognizerRegistry.load_predefined_recognizers` accepts an optional `countries` argument that narrows the country-specific recognizers to the subset you care about, while always preserving locale-agnostic recognizers (credit cards, emails, URLs, IBAN, dates, crypto, NER, NLP engine recognizers, third-party integrations, …). ## Quick example ```python from presidio_analyzer import AnalyzerEngine, RecognizerRegistry registry = RecognizerRegistry() registry.load_predefined_recognizers(languages=["en"], countries=["us", "uk"]) analyzer = AnalyzerEngine(registry=registry) ``` The registry above contains all locale-agnostic recognizers (e.g. credit card, email, URL) plus US- and UK-specific recognizers (US SSN, UK NHS, UK NINO, …) — but excludes German, Spanish, Indian, Italian, Korean, Polish, Singaporean, Swedish, Thai, Turkish, and other country recognizers. The same filter can be set globally in YAML, mirroring how `supported_languages` works: ```yaml supported_languages: ["en"] supported_countries: ["us", "uk"] recognizers: # ... ``` When the registry is built from a YAML file, this top-level `supported_countries` is applied automatically inside the loader; you don't need to pass `countries=` again from Python. ## How the filter works A recognizer can be tagged with a country in two ways, reconciled at construction time: - **Class-level `COUNTRY_CODE`** — the canonical declaration for predefined recognizers (`UsSsnRecognizer.COUNTRY_CODE = "us"`, …). - **Constructor `country_code=` kwarg** — for *custom* recognizers without a subclass, or YAML `type: custom` entries that flow through `PatternRecognizer.from_dict`. This is the path used by the no-code/YAML route. If both are set, they must agree (case-insensitively); a constructor value that conflicts with `COUNTRY_CODE` raises `ValueError` so a predefined Polish-tax-ID recognizer can't be silently re-tagged as British. The filter then reads the resolved tag through two instance methods on `EntityRecognizer`: - `recognizer.country_code()` returns the lower-cased ISO code, or `None` if the recognizer is locale-agnostic. - `recognizer.is_country_specific()` is the named predicate equivalent to `country_code() is not None`. To introspect a class without instantiating, read the `COUNTRY_CODE` ClassVar directly (e.g. `UsSsnRecognizer.COUNTRY_CODE`); the methods themselves are instance methods so they can also surface constructor-tagged custom recognizers. The filter has a single rule: | Recognizer's `country_code()` | `countries=None` | `countries=["us"]` | `countries=["us", "uk"]` | `countries=[]` | | --- | --- | --- | --- | --- | | `None` (locale-agnostic) | ✅ kept | ✅ kept | ✅ kept | ✅ kept | | `"us"` | ✅ kept | ✅ kept | ✅ kept | ❌ dropped | | `"uk"` | ✅ kept | ❌ dropped | ✅ kept | ❌ dropped | | `"de"` | ✅ kept | ❌ dropped | ❌ dropped | ❌ dropped | Code comparison is **case-insensitive**: `"US"`, `"Us"`, `"us"` all work. ## Declaring a country on your own recognizers There are three equivalent paths, pick whichever fits your code shape. All three end up at the same `self._country_code` and are filtered the same way. ### 1. Subclass with a class-level `COUNTRY_CODE` (preferred for shared code) This is the path used by every predefined country-specific recognizer in Presidio (`UsSsnRecognizer`, `UkNinoRecognizer`, `EsNifRecognizer`, …). Best when your recognizer lives in code and you want the country tag visible at the class definition for review: ```python from presidio_analyzer import Pattern, PatternRecognizer class BrCpfRecognizer(PatternRecognizer): """Recognize Brazilian CPF numbers.""" COUNTRY_CODE = "br" PATTERNS = [ Pattern("BR CPF (medium)", r"\b\d{3}\.\d{3}\.\d{3}-\d{2}\b", 0.5), ] def __init__(self): super().__init__( supported_entity="BR_CPF", patterns=self.PATTERNS, supported_language="pt", ) ``` `BrCpfRecognizer().country_code()` returns `"br"` and the registry's country filter treats it the same as predefined country-specific recognizers. ### 2. Constructor `country_code=` (preferred for one-offs) If you don't want to subclass — e.g. you're constructing a one-off `PatternRecognizer` for an Armenian national ID — pass `country_code` to the constructor: ```python from presidio_analyzer import Pattern, PatternRecognizer am_recognizer = PatternRecognizer( supported_entity="AM_NATIONAL_ID", name="AmNationalIdRecognizer", patterns=[Pattern("AM 10-digit", r"\b\d{10}\b", 0.5)], country_code="am", ) assert am_recognizer.country_code() == "am" ``` This also covers `PatternRecognizer.from_dict({"country_code": "am", ...})`, so any tooling that round-trips recognizer dicts (logs, serialization, recognizer-store backends, …) preserves the tag. ### 3. YAML `type: custom` with `country_code:` (no-code path) YAML custom-recognizer entries forward `country_code` straight through to `PatternRecognizer.__init__` via `from_dict`, so the no-code path gets the same tagging without writing a single line of Python: ```yaml recognizers: - name: AmNationalIdRecognizer type: custom supported_entity: AM_NATIONAL_ID supported_languages: - language: en country_code: am patterns: - name: "AM 10-digit" regex: '\b\d{10}\b' score: 0.5 ``` Loading this registry with `supported_countries: ["am"]` (or the Python-side `countries=["am"]`) keeps the recognizer; filtering to a different country drops it. ### Combining paths If you subclass *and* pass `country_code=` to the constructor, the two must agree — passing a different value raises `ValueError`. This means a predefined `UsSsnRecognizer` can never be silently re-tagged via the constructor, and a YAML `country_code:` on a `type: predefined` entry is cross-checked against the class attribute at load time (see the "Annotating predefined recognizers in YAML" section below). ## Annotating predefined recognizers in YAML When you wire up the registry from a YAML file (the no-code path), you can — and should — also declare the country tag next to each predefined country-specific entry. This is purely advisory metadata that documents the entry for human readers; the class-level `COUNTRY_CODE` remains the source of truth, and the loader cross-checks the two and refuses to load on a mismatch: ```yaml recognizers: - name: UsSsnRecognizer supported_languages: - en type: predefined country_code: us # must match UsSsnRecognizer.COUNTRY_CODE - name: UkNinoRecognizer supported_languages: - en type: predefined country_code: uk - name: CreditCardRecognizer type: predefined # no country_code: locale-agnostic ``` If you change one without the other, the registry will raise `ValueError` at load time naming both the YAML value and the class attribute, so the misconfiguration is fixable from the error message alone. ## When to leave `COUNTRY_CODE` unset Some PII patterns are not anchored to a single country and should remain locale-agnostic: - Credit card numbers, email addresses, URLs, MAC addresses, crypto wallet addresses, dates, phone numbers (PhoneNumbers cover multiple countries via `phonenumbers`), … - IBAN — geographically European-ish but not anchored to one country. - Generic NER models (spaCy, Stanza, transformer-based recognizers). Leave `COUNTRY_CODE` unset (it inherits the default `None` from `EntityRecognizer`) for these. They will always be loaded and the country filter will not touch them. ## Backwards compatibility The filter is fully backwards compatible: 1. **Default behavior is unchanged.** Calling `load_predefined_recognizers()` without `countries` (or with `countries=None`) loads every predefined recognizer exactly as before. 2. **Untagged custom recognizers continue to work.** Any recognizer you have already deployed that does not declare `COUNTRY_CODE` is treated as locale-agnostic and is never filtered out — regardless of the requested countries. The filter only excludes recognizers that have explicitly opted in via the class attribute. 3. **Predefined recognizers are pre-tagged.** Every recognizer under `presidio_analyzer.predefined_recognizers.country_specific.` ships with its `COUNTRY_CODE` set to the appropriate ISO code, so `countries=["us"]` works on a fresh install with no further setup. ## Debugging If you pass `countries=["br"]` and your custom Brazilian recognizer disappears, the most likely cause is that the recognizer hasn't declared `COUNTRY_CODE = "br"`. Two diagnostic helpers: - `registry.get_country_codes()` returns the sorted list of country codes currently represented in the registry. If `"br"` is missing, no recognizer is tagged. - The filter logs a `WARNING` whenever a requested country matches no recognizer in the input set, with a hint to set `country_code` on custom recognizers. ```python import logging logging.getLogger("presidio-analyzer").setLevel(logging.WARNING) registry = RecognizerRegistry() registry.load_predefined_recognizers(countries=["br"]) # WARNING ... Country filter: no recognizer matched country_code='br'. # If you have custom recognizers for 'br', set country_code='br' # on them to include them in country-filtered loads. print(registry.get_country_codes()) # ['au', 'ca', 'de', 'es', 'fi', 'in', 'it', 'kr', 'ng', 'pl', 'se', # 'sg', 'th', 'tr', 'uk', 'us'] ``` --- ### Analyzer Languages (`docs/analyzer/languages.md`) # PII detection in different languages Presidio supports PII detection in multiple languages. In its default configuration, it contains recognizers and models for English. To extend Presidio to detect PII in an additional language, these modules require modification: 1. The `NlpEngine` containing the NLP model which performs tokenization, lemmatization, Named Entity Recognition and other NLP tasks. 2. PII recognizers (different `EntityRecognizer` objects) should be adapted or created. !!! note "Note" While different detection mechanisms such as regular expressions are language agnostic, the context words used to increase the PII detection confidence aren't. Consider updating the list of context words for each recognizer to leverage context words in additional languages. ## Table of contents - [Configuring the NLP Engine](#configuring-the-nlp-engine) - [Set up language specific recognizers](#set-up-language-specific-recognizers) - [Automatically install NLP models into the Docker container](#automatically-install-nlp-models-into-the-docker-container) ### Configuring the NLP Engine Presidio's NLP engine can be adapted to support multiple languages and frameworks (such as *spaCy, Stanza* and *transformers*). Configuring the NLP engine for a new language or NLP framework is done by downloading or using a model trained on a different language, and providing a configuration. See the [NLP model customization documentation](customizing_nlp_models.md) for details on how to configure models for new languages. ### Set up language specific recognizers Recognizers are language dependent either by their logic or by the context words used while scanning the surrounding of a detected entity. As these context words are used to increase score, they should be in the expected input language. Consider updating the context words of existing recognizers or add new recognizers to support new languages. Each recognizer can support one language. For example: ```python from presidio_analyzer import AnalyzerEngine, RecognizerRegistry from presidio_analyzer.predefined_recognizers import EmailRecognizer from presidio_analyzer.nlp_engine import NlpEngineProvider LANGUAGES_CONFIG_FILE = "./docs/analyzer/languages-config.yml" # Create NLP engine based on configuration file provider = NlpEngineProvider(conf_file=LANGUAGES_CONFIG_FILE) nlp_engine_with_spanish = provider.create_engine() # Setting up an English Email recognizer: email_recognizer_en = EmailRecognizer(supported_language="en", context=["email", "mail"]) # Setting up a Spanish Email recognizer email_recognizer_es = EmailRecognizer(supported_language="es", context=["correo", "electrónico"]) registry = RecognizerRegistry() # Add recognizers to registry registry.add_recognizer(email_recognizer_en) registry.add_recognizer(email_recognizer_es) # Set up analyzer with our updated recognizer registry analyzer = AnalyzerEngine( registry=registry, supported_languages=["en","es"], nlp_engine=nlp_engine_with_spanish) analyzer.analyze(text="My name is David", language="en") ``` Link to LANGUAGES_CONFIG_FILE=[languages-config.yml](https://github.com/data-privacy-stack/presidio/blob/main/docs/analyzer/languages-config.yml) ### Automatically install NLP models into the Docker container When packaging the code into a Docker container, NLP models are automatically installed. To define which models should be installed, update the [conf/default.yaml](https://github.com/data-privacy-stack/presidio/blob/main/presidio-analyzer/presidio_analyzer/conf/default.yaml) file. This file is read during the `docker build` phase and the models defined in it are installed automatically. For `transformers` based models, the configuration [can be found here](https://github.com/data-privacy-stack/presidio/blob/main/presidio-analyzer/presidio_analyzer/conf/transformers.yaml). A docker file supporting transformers models [can be found here](https://github.com/data-privacy-stack/presidio/blob/main/presidio-analyzer/Dockerfile.transformers). --- ### Analyzer Recognizer Registry Provider (`docs/analyzer/recognizer_registry_provider.md`) # Customizing recognizer registry from file To load recognizers from file, use `RecognizerRegistryProvider` to instantiate the recognizer registry and then pass it through to the analyzer engine: ```python from presidio_analyzer import AnalyzerEngine from presidio_analyzer.recognizer_registry import RecognizerRegistryProvider recognizer_registry_conf_file = "./analyzer/recognizers-config.yml" provider = RecognizerRegistryProvider( conf_file=recognizer_registry_conf_file ) registry = provider.create_recognizer_registry() analyzer = AnalyzerEngine(registry=registry) results = analyzer.analyze(text="My name is Morris", language="en") print(results) ``` ## Configuration file structure ```yaml global_regex_flags: 26 supported_languages: - en recognizers: ... ``` The configuration file consists of two parts: - `global_regex_flags`: regex flags to be used in regex matching (see [regex flags](https://docs.python.org/3/library/re.html#flags)). - `supported_languages`: A list of supported languages that the registry will support. - `recognizers`: a list of recognizers to be loaded by the recognizer registry. This list consists of two different types of recognizers: - Predefined: A set of already defined recognizer classes in presidio. This includes all recognizers defined in the codebase (along with user defined recognizers) that inherit from EntityRecognizer. - Custom: custom created pattern recognizers that are created based on the fields provided in the configuration file. !!! note "Note" supported_languages must be identical to the same field in analyzer_engine ## Recognizer list The recognizer list comprises of both the predefined and custom recognizers, for example: ```yaml ... - name: CreditCardRecognizer supported_languages: - language: en context: [credit, card, visa, mastercard, cc, amex, discover, jcb, diners, maestro, instapayment] - language: es context: [tarjeta, credito, visa, mastercard, cc, amex, discover, jcb, diners, maestro, instapayment] - language: it - language: pl type: predefined score_thresholds: default: 0.4 CREDIT_CARD: 0.7 - name: UsBankRecognizer supported_languages: - en type: predefined - name: MedicalLicenseRecognizer type: predefined - name: ExampleCustomRecognizer patterns: - name: "zip code (weak)" regex: "(\\b\\d{5}(?:\\-\\d{4})?\\b)" score: 0.01 supported_languages: - language: en context: [zip, code] - language: es context: [código, postal] supported_entity: "ZIP" type: custom enabled: true - name: "TitlesRecognizer" supported_language: "en" supported_entity: "TITLE" deny_list: [Mr., Mrs., Ms., Miss, Dr., Prof.] deny_list_score: 1 - name: "HuggingFace NER" type: "predefined" class_name: "HuggingFaceNerRecognizer" model_name: "dslim/bert-base-NER" supported_languages: - en supported_entities: ["PERSON", "LOCATION", "ORGANIZATION"] aggregation_strategy: "simple" device: "cpu" ``` ### The recognizer parameters - `supported_languages`: A list of supported languages that the analyzer will support. In case this field is missing, a recognizer will be created for each supported language provided to the `AnalyzerEngine`. In addition to the language code, this field also contains a list of context words, which increases confidence in the detection in case it is found in the surroundings of a detected entity (as seen in the credit card example above). - `type`: this could be either predefined or custom. As this is optional, if not stated otherwise, the default type is custom. - `name`: Different per the type of the recognizer. For predefined recognizers, this is the class name as defined in presidio, while for custom recognizers, it will be set as the name of the recognizer. - `patterns`: a list of objects of type `Pattern` that contains a name, score and regex that define matching patterns. - `enabled`: enables or disables the recognizer. - `supported_entity`: the detected entity associated by the recognizer. - `deny_list`: A list of words to detect, in case the recognizer uses a predefined list of words. - `deny_list_score`: confidence score for a term identified using a deny-list. If omitted, defaults to `1.0` (previously this silently defaulted to `0.0` when loaded through `RecognizerRegistryProvider`, which caused deny-list matches to be filtered out by any positive `score_threshold`). - `score_thresholds`: optional score thresholds for this recognizer. Use `default` as the recognizer-wide threshold and entity names for overrides. Note that supplying `analyzer_engine.analyze(score_threshold=...)` bypasses recognizer-level thresholds for that request. The precedence is: Presidio Analyzer analyzer.analyze(score_threshold=...) > an entity specific threshold > a recognizer default threshold (`default`) > the Presidio Analyzer `default_score_threshold`. - `text_chunker`: configures how long texts are split for NER recognizers (`GLiNERRecognizer`, `HuggingFaceNerRecognizer`). Accepts a dict with `chunker_type` and params. Available types: `character` (default) and `tokenizer` (uses the model's tokenizer for accurate token-based splitting). Example: ```yaml - name: GLiNERRecognizer type: predefined model_name: urchade/gliner_multi_pii-v1 text_chunker: chunker_type: tokenizer # max_tokens omitted: auto-derived from the model's tokenizer and # reduced to reserve room for special tokens ([CLS]/[SEP]). Set it # explicitly only if you account for those special tokens yourself. overlap_tokens: 32 ``` !!! tip "Configuration Tip: Agglutinative languages (e.g., Korean)" If spaCy-based NER produces noisy or redundant results, you can disable `SpacyRecognizer` and use `HuggingFaceNerRecognizer` as an alternative. Note that `HuggingFaceNerRecognizer` bypasses the spaCy tokenizer alignment mechanism. ```yaml - name: "SpacyRecognizer" type: "predefined" class_name: "SpacyRecognizer" supported_languages: ["ko"] enabled: false ``` ## Using a custom registry file with the Presidio Analyzer server The Presidio Analyzer server (the `presidio-analyzer` Docker image) selects its recognizer registry through the `RECOGNIZER_REGISTRY_CONF_FILE` environment variable. It defaults to the bundled `presidio_analyzer/conf/default_recognizers.yaml`; point it at your own YAML file (mounted into the container) to load a modified registry without rebuilding the image. The server reads `ANALYZER_CONF_FILE` and `NLP_CONF_FILE` the same way. !!! note "Note" `supported_languages` in the registry file must match the same field in the analyzer configuration file. ### Example: enabling a pattern-based country recognizer for `en` Some predefined recognizers are only registered for their native language. `ItFiscalCodeRecognizer`, for example, is registered with `supported_languages: [it]` in `default_recognizers.yaml`, so the default image — which loads the `en` NLP model and `supported_languages: [en]` — cannot detect Italian fiscal codes. Because the recognizer is pattern-based, it does not depend on the Italian NLP model and can be enabled for `en` by overriding the registry file: 1. Copy the default registry file and change the recognizer's `supported_languages` (leave `country_code` unchanged — it must match the class declaration and is only the country tag, not a language): ```yaml - name: ItFiscalCodeRecognizer supported_languages: - en type: predefined country_code: it ``` 2. Run the analyzer with the file mounted and the environment variable set: ```bash docker run --rm -p 3000:3000 \ -v $(pwd)/my_recognizers.yaml:/app/my_recognizers.yaml \ -e RECOGNIZER_REGISTRY_CONF_FILE=/app/my_recognizers.yaml \ ghcr.io/data-privacy-stack/presidio-analyzer: ``` Requests with `entities: ["IT_FISCAL_CODE"]` are now served for `language: en`. To restrict or extend which country-specific recognizers are loaded in general, see [Filtering recognizers by country](filtering_by_country.md). --- ### Anonymizer Adding Operators (`docs/anonymizer/adding_operators.md`) # Supporting new types of PII operators Operators are the presidio-anonymizer actions over the text. There are two types of operators: - Anonymize (e.g., hash, replace, redact, encrypt, mask) - Deanonymize (e.g., decrypt) Presidio anonymizer can be easily extended to support additional anonymization and deanonymization methods (called Operators). ## Extending presidio-anonymizer for additional PII operators 1. Create new python class implementing the abstract [Operator](https://github.com/data-privacy-stack/presidio/blob/main/presidio-anonymizer/presidio_anonymizer/operators/operator.py) class. 2. Implement the methods: - `operate` - gets the data and returns a new text expected to replace the old one. - `validate` - validate the parameters entered for the anonymizer exists and valid. - `operator_name` - this method helps to automatically load the existing anonymizers. - `operator_type` - either Anonymize or Deanonymize. Will be mapped to the proper engine. 3. Call the `AnonymizerEngine.add_anonymizer` method to add a new operator to the anonymizer. Alternatively, call the `DeanonymizeEngine.add_deanonymizer` method to add a new deanonymizer. See a detailed example [here](../samples/python/pseudonymization.ipynb). --- ### Api Analyzer Python (`docs/api/analyzer_python.md`) # Presidio Analyzer API Reference ## Objects at the top of the presidio-analyzer package ::: presidio_analyzer.AnalyzerEngine handler: python ::: presidio_analyzer.analyzer_engine_provider.AnalyzerEngineProvider handler: python ::: presidio_analyzer.analysis_explanation.AnalysisExplanation handler: python ::: presidio_analyzer.recognizer_result.RecognizerResult handler: python ## Batch modules ::: presidio_analyzer.batch_analyzer_engine.BatchAnalyzerEngine handler: python ::: presidio_analyzer.dict_analyzer_result.DictAnalyzerResult handler: python ## Recognizers and patterns ::: presidio_analyzer.entity_recognizer.EntityRecognizer handler: python ::: presidio_analyzer.local_recognizer.LocalRecognizer handler: python ::: presidio_analyzer.pattern.Pattern handler: python ::: presidio_analyzer.pattern_recognizer.PatternRecognizer handler: python ::: presidio_analyzer.remote_recognizer.RemoteRecognizer handler: python ## Recognizer registry modules ::: presidio_analyzer.recognizer_registry.RecognizerRegistry handler: python ::: presidio_analyzer.recognizer_registry.RecognizerRegistryProvider handler: python ## Context awareness modules ::: presidio_analyzer.context_aware_enhancers ## NLP Engine modules ::: presidio_analyzer.nlp_engine handler: python ## Predefined Recognizers ::: presidio_analyzer.predefined_recognizers ## Misc ::: presidio_analyzer.analyzer_request.AnalyzerRequest handler: python --- ### Api Anonymizer Python (`docs/api/anonymizer_python.md`) # Presidio Anonymizer API Reference ::: presidio_anonymizer handler: python --- ### Api Image Redactor Python (`docs/api/image_redactor_python.md`) # Presidio Image Redactor API Reference ::: presidio_image_redactor handler: python options: docstring_style: sphinx --- ### Getting Started Getting Started Images (`docs/getting_started/getting_started_images.md`) # Getting started with image de-identification with Presidio Presidio provides a simple way to de-identify image data by detecting and anonymizing personally identifiable information (PII). This guide shows you how to get started with image de-identification using Presidio's Python packages. Presidio has two main modules for image de-identification: General purpose, and specifically for DICOM (medical) images. ## Simple flow - Python package === "Anonymize PII in images" 1. Install presidio-image-redactor ```sh pip install presidio-image-redactor ``` 2. Redact PII from image ```py from presidio_image_redactor import ImageRedactorEngine from PIL import Image image = Image.open(path_to_image_file) redactor = ImageRedactorEngine() redactor.redact(image=image) ``` === "Redact text PII in DICOM images" 1. Install presidio-image-redactor ```sh pip install presidio-image-redactor ``` 2. Redact text PII from DICOM image ```py import pydicom from presidio_image_redactor import DicomImageRedactorEngine # Set input and output paths input_path = "path/to/your/dicom/file.dcm" output_dir = "./output" # Initialize the engine engine = DicomImageRedactorEngine() # Option 1: Redact from a loaded DICOM image dicom_image = pydicom.dcmread(input_path) redacted_dicom_image = engine.redact(dicom_image, fill="contrast") # Option 2: Redact from DICOM file engine.redact_from_file(input_path, output_dir, padding_width=25, fill="contrast") # Option 3: Redact from directory engine.redact_from_directory("path/to/your/dicom", output_dir, padding_width=25, fill="contrast") ``` --- ## Simple flow - Docker container Presidio provides a Docker containers that you can use to de-identify image data. 1. Download Docker image ```sh docker pull ghcr.io/data-privacy-stack/presidio-image-redactor ``` 2. Run container ```sh docker run -d -p 5003:3000 ghcr.io/data-privacy-stack/presidio-image-redactor:latest ``` 3. Use the API ```sh curl -XPOST "http://localhost:5003/redact" -H "content-type: multipart/form-data" -F "image=@img.png" -F "data=\"{'color_fill':'255'}\"" > out.png ``` ## Read more - [Installing Presidio](../installation.md) - [PII detection in images](../image-redactor/index.md) - [Samples](../samples/index.md) - [Python API reference - Image Redactor](../api/image_redactor_python.md) --- ### Getting Started Getting Started Structured (`docs/getting_started/getting_started_structured.md`) # Getting started with structured and semi-structured de-identification with Presidio Presidio-structured is a package built on top of Presidio that provides a simple way to de-identify structured and semi-structured data by detecting and anonymizing personally identifiable information (PII). Presidio-structured supports the detection and anonymization of PII in tables (e.g. Pandas DataFrames or SQL tables) and semi-structured data (e.g. JSON). !!! warning "Warning" **Alpha**: This package is currently in alpha, meaning it is in its early stages of development. Features and functionality may change as the project evolves. ## Simple flow - structured data ```python import pandas as pd from presidio_structured import StructuredEngine, PandasAnalysisBuilder from presidio_anonymizer.entities import OperatorConfig from faker import Faker # optionally using faker as an example # Initialize the engine with a Pandas data processor (default) pandas_engine = StructuredEngine() # Create a sample DataFrame sample_df = pd.DataFrame({'name': ['John Doe', 'Jane Smith'], 'email': ['john.doe@example.com', 'jane.smith@example.com']}) # Generate a tabular analysis which detects the PII entities in the DataFrame. tabular_analysis = PandasAnalysisBuilder().generate_analysis(sample_df) # Define anonymization operators fake = Faker() operators = { "PERSON": OperatorConfig("replace", {"new_value": "REDACTED"}), "EMAIL_ADDRESS": OperatorConfig("custom", {"lambda": lambda x: fake.safe_email()}) } # Anonymize DataFrame anonymized_df = pandas_engine.anonymize(sample_df, tabular_analysis, operators=operators) print(anonymized_df) ``` ## Read more - [Presidio structured documentation](../structured/index.md) - [Presidio structured sample notebook](../samples/python/example_structured.ipynb) --- ### Getting Started Getting Started Text (`docs/getting_started/getting_started_text.md`) # Getting started with text de-identification with Presidio Presidio provides a simple way to de-identify text data by detecting and anonymizing personally identifiable information (PII). This guide shows you how to get started with text de-identification using Presidio's Python packages. Note that Presidio can leverage different NLP packages to analyze text data. The default engine is based on `spaCy`, but you can [also use others](../analyzer/customizing_nlp_models.md). This guide shows two examples: one using `spaCy` and the other using `transformers`. ## Simple flow - Python package Using Presidio's modules as Python packages to get started: ===+ "Anonymize PII in text (Default spaCy model)" 1. Install Presidio ```sh pip install presidio-analyzer pip install presidio-anonymizer python -m spacy download en_core_web_lg ``` 2. Analyze + Anonymize ```py from presidio_analyzer import AnalyzerEngine from presidio_anonymizer import AnonymizerEngine text="My phone number is 212-555-5555" # Set up the engine, loads the NLP module (spaCy model by default) # and other PII recognizers analyzer = AnalyzerEngine() # Call analyzer to get results results = analyzer.analyze(text=text, entities=["PHONE_NUMBER"], language='en') print(results) # Analyzer results are passed to the AnonymizerEngine for anonymization anonymizer = AnonymizerEngine() anonymized_text = anonymizer.anonymize(text=text,analyzer_results=results) print(anonymized_text) ``` === "Anonymize PII in text (transformers)" 1. Install Presidio ```sh pip install "presidio-analyzer[transformers]" pip install presidio-anonymizer python -m spacy download en_core_web_sm ``` 2. Analyze + Anonymize ```py from presidio_analyzer import AnalyzerEngine from presidio_analyzer.nlp_engine import TransformersNlpEngine from presidio_anonymizer import AnonymizerEngine text = "My name is Don and my phone number is 212-555-5555" # Define which transformers model to use model_config = [{"lang_code": "en", "model_name": { "spacy": "en_core_web_sm", # use a small spaCy model for lemmas, tokens etc. "transformers": "dslim/bert-base-NER" } }] nlp_engine = TransformersNlpEngine(models=model_config) # Set up the engine, loads the NLP module (spaCy model by default) # and other PII recognizers analyzer = AnalyzerEngine(nlp_engine=nlp_engine) # Call analyzer to get results results = analyzer.analyze(text=text, language='en') print(results) # Analyzer results are passed to the AnonymizerEngine for anonymization anonymizer = AnonymizerEngine() anonymized_text = anonymizer.anonymize(text=text, analyzer_results=results) print(anonymized_text) ``` !!! tip "Tip: Downloading models" If not available, the transformers model and the spacy model would be downloaded on the first call to the `AnalyzerEngine`. To pre-download, see [this doc](../analyzer/nlp_engines/transformers.md#downloading-a-pre-trained-model). === "GPU Acceleration (Optional)" For GPU acceleration, install the appropriate dependencies for your hardware: - **Linux with NVIDIA GPU**: cupy-cuda12x (or the version matching your CUDA installation) - **macOS with Apple Silicon**: MPS (Metal Performance Shaders) is currently not supported. The analyzer will use CPU for PyTorch operations. ## Simple flow - Docker container Presidio provides Docker containers that you can use to de-identify text data. Each module, analyzer, and anonymizer, has its own Docker container. New releases are available on [GitHub Container Registry](https://github.com/orgs/data-privacy-stack/packages); the legacy Microsoft Container Registry images are no longer updated. 1. Download Docker images ```sh docker pull ghcr.io/data-privacy-stack/presidio-analyzer docker pull ghcr.io/data-privacy-stack/presidio-anonymizer ``` 2. Run containers ```sh docker run -d -p 5002:3000 ghcr.io/data-privacy-stack/presidio-analyzer:latest docker run -d -p 5001:3000 ghcr.io/data-privacy-stack/presidio-anonymizer:latest ``` 3. Use the API ```sh curl -X POST http://localhost:5002/analyze \ -H "Content-Type: application/json" \ -d '{ "text": "My phone number is 555-123-4567.", "language": "en" }' curl -X POST http://localhost:5001/anonymize -H "Content-Type: application/json" -d ' { "text": "My phone number is 555-123-4567", "anonymizers": { "PHONE_NUMBER": { "type": "replace", "new_value": "--Redacted phone number--" } }, "analyzer_results": [ { "start": 19, "end": 31, "score": 0.95, "entity_type": "PHONE_NUMBER" } ]}' ``` ## Read more - [Installing Presidio](../installation.md) - [PII detection in text](../analyzer/index.md) - [PII anonymization in text](../anonymizer/index.md) - [Tutorial](../tutorial/index.md) - [Samples](../samples/index.md) - [Python API reference - Analyzer](../api/analyzer_python.md) - [Python API reference - Anonymizer](../api/anonymizer_python.md) - [REST API reference](../api-docs/api-docs.html) --- ### Image Redactor Evaluating Dicom Redaction (`docs/image-redactor/evaluating_dicom_redaction.md`) # Evaluating DICOM de-identification ## Table of Contents * [Introduction](#introduction) * [Ground truth format](#ground-truth-format) * [Creating ground truth files](#creating-ground-truth-files) * [Evaluating de-identification performance](#evaluating-de-identification-performance) ## Introduction We can evaluate the performance of the `DicomImageRedactorEngine` DICOM de-identification by using the `DicomImagePiiVerifyEngine`. The evaluation results consist of: * Image with bounding boxes identifying detected Personal Health Information (PHI) * All positives (True Positives and False Positives) * Precision * Recall ## Ground truth format Ground truth labels are stored as `.json` files containing filename as the highest level keys. Each filename object consists of an item for each individual entity. ```json { "your/dicom/dir/file_0.dcm": [ { "label": "DAVIDSON", "left": 25, "top": 25, "width": 241, "height": 37 }, { "label": "DOUGLAS", "left": 287, "top": 25, "width": 230, "height": 36 }, { "label": "[M]", "left": 535, "top": 25, "width": 60, "height": 45 }, { "label": "01.09.2012", "left": 613, "top": 26, "width": 226, "height": 35 }, { "label": "06.16.1976", "left": 170, "top": 72, "width": 218, "height": 35 } ], "your/dicom/dir/file_1.dcm": [ ... ] } ``` *Return to the [Table of Contents](#table-of-contents)* ## Creating ground truth files The `DicomImagePiiVerifyEngine` class can be used to assist in ground truth label generation. Use the following code snippet to generate the verification image, OCR results, and NER (analyzer) results. ```python import pydicom from presidio_image_redactor import DicomImagePiiVerifyEngine # Initialize engine dicom_engine = DicomImagePiiVerifyEngine() # Choose your file to create ground truth for filename = "path/to/your/file.dcm" instance = pydicom.dcmread(filename) padding_width = 25 # Get OCR and NER results verification_image, ocr_results, analyzer_results = dicom_engine.verify_dicom_instance(instance, padding_width) ``` By looking at the output of `verify_dicom_instance`, we can create a ground truth labels json. Save `analyzer_results` as a json file and then perform the following 1. Group the results into a new item with the file name set as the key. 2. For each item in this group: a. Remove the "entity_type" field and value. b. Add a new "label" field with the value set to the ground truth text PHI with matching coordinate as you can see in the formatted OCR results and verification image. Then check that your ground truth json contains all the text PHI you can visually confirm in the DICOM image. If something is not detected by the OCR or NER, you will need to manually add the item yourself. Pixel position and size data can be obtained using any labeling software or imaging processing software (e.g., MS Paint) on the verification image. > *Note: When manually specifying pixel position, make sure to account for any padding introduced in the OCR process (default padding added is 25 pixels).* ### Example Let's say we ran the above code block and see the following for `ocr_results` and `analyzer_results`. ```json // OCR Results [ { "left": 25, "top": 25, "width": 241, "height": 37, "conf": 95.833916, "label": "DAVIDSON" }, { "left": 287, "top": 25, "width": 230, "height": 36, "conf": 93.292221, "label": "DOUGLAS" } ] // Analyzer Results [ { "entity_type": "PERSON", "score": 1.0, "left": 25, "top": 25, "width": 241, "height": 37 }, { "entity_type": "PERSON", "score": 1.0, "left": 287, "top": 25, "width": 230, "height": 36 } ] ``` Looking at the position and size values of the ground truth and detected text from the analyzer results, we can see that the first item in the analyzer results is likely "DAVIDSON" and the second is likely "DOUGLAS". With this, we set our ground truth json to the following: ```json // Ground truth json { "path/to/your/file.dcm": [ { "label": "DAVIDSON", "left": 25, "top": 25, "width": 241, "height": 37 }, { "label": "DOUGLAS", "left": 287, "top": 25, "width": 230, "height": 36 } ] } ``` *Return to the [Table of Contents](#table-of-contents)* ## Evaluating de-identification performance The `DicomImagePiiVerifyEngine` can be used to evaluate DICOM de-identification performance. ```python # Load ground truth for one file with open(gt_path) as json_file: all_ground_truth = json.load(json_file) ground_truth = all_ground_truth[file_of_interest] # Select your DICOM instance instance = pydicom.dcmread(file_of_interest) # Evaluate the DICOM de-identification performance _, eval_results = dicom_engine.eval_dicom_instance(instance, ground_truth) ``` You can also set optional arguments to see the effect of padding width, ground-truth matching tolerance, and OCR confidence threshold (e.g., `ocr_kwargs={"ocr_threshold": 50}`). For a full demonstration, please see the [evaluation notebook](../samples/python/example_dicom_redactor_evaluation.ipynb). *Return to the [Table of Contents](#table-of-contents)* --- ### Learn Presidio Concepts (`docs/learn_presidio/concepts.md`) # Concepts in Presidio ## High-level concepts | Concept | Definition | Learn More | |--------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------| | **Entity** | An *entity* is a span of text that can be used to directly identify an individual. For example, a phone number, email address, or social security number. In Presidio, an entity is represented by a **RecognizerResult** object. | [Analyzer concepts](../analyzer/index.md#main-concepts) | | **Context** | *Context* is defined as the surrounding text of an entity. Context can be used to provide additional information about the entity which can be used to improve the detection accuracy. | [Analyzer concepts](../analyzer/index.md#main-concepts) | | **Recognizer** | A *recognizer* is an object that is responsible for detecting entities in text. Recognizers can be rule-based, machine learning-based, or a combination of both. The Presidio Analyzer orchestrates multiple recognizers to detect PII entities in text. The main objects in Presidio that implement PII detection logic are the **EntityRecognizer** and **PatternRecognizer**. | [Analyzer concepts](../analyzer/index.md#main-concepts) | | **Analyzer** | The Presidio `AnalyzerEngine` is responsible for orchestrating the PII detection using various recognizers.| [Analyzer concepts](../analyzer/index.md#main-concepts) | | **Predefined recognizer** | A recognizer that already exists in Presidio | [Predefined recognizers](../supported_entities.md) | | **Custom recognizer** | A recognizer that is added by the user | [Adding recognizers](../analyzer/adding_recognizers.md) | | **ad-hoc recognizer** | A recognizer that is added to the request itself, rather than to the list of recognizers loaded within Presidio | [ad-hoc recognizers](../analyzer/adding_recognizers.md#creating-ad-hoc-recognizers) | | **Deny list** | A list of terms that should always be identified as PII | [denylist tutorial](../tutorial/01_deny_list.md) | | **Allow list** | A list of terms that should not be identified as PII | [allowlist tutorial](../tutorial/13_allow_list.md) | ## Main objects in Presidio | Concept | Definition | Learn More | |--------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------| | **EntityRecognizer** | An **EntityRecognizer** is an object in Presidio that is responsible for detecting entities in text. An entity recognizer can be rule-based, a machine learning model, or a combination of both. | [Analyzer concepts](../analyzer/index.md#main-concepts) | | **RecognizerResult** | A **RecognizerResult** holds the type and span of a PII entity. | [Analyzer concepts](../analyzer/index.md#main-concepts) | | **RecognizerRegistry** | The **RecognizerRegistry** is a class in Presidio that is responsible for holding the various recognizers used by the **AnalyzerEngine**. | [link](../analyzer/index.md#main-concepts) | | **NlpEngine** | The **NlpEngine** is an interface that defines the methods for processing text. Presidio provides several implementations of the **NlpEngine**, such as **SpacyNlpEngine**, **TransformersNlpEngine**, and **StanzaNlpEngine**. | [Analyzer concepts](../analyzer/index.md#main-concepts) | | **AnalyzerEngine** | The **AnalyzerEngine** is the main class in Presidio that is responsible for orchestrating the PII detection in text. It uses an **NlpEngine** to process the text and a **RecognizerRegistry** to hold the different recognizers. | [Analyzer concepts](../analyzer/index.md#main-concepts) | | **BatchAnalyzerEngine** | The **BatchAnalyzerEngine** is a class in Presidio that is responsible for detecting PII entities in a batch of texts. It uses the **AnalyzerEngine** to process each text in the batch. | [Batch processing sample](../samples/python/batch_processing.ipynb) | | **AnonymizerEngine** | The **AnonymizerEngine** is the main class in Presidio that is responsible for anonymizing PII entities in text. It uses the results from the **AnalyzerEngine** to perform the anonymization. | [Anonymizer concepts](../anonymizer/index.md#main-concepts) | | **DeanonymizerEngine** | The **DeanonymizerEngine** is a class in Presidio that is responsible for deanonymizing text that has been anonymized by the **AnonymizerEngine**, given that the operation is reversible (e.g. encryption). | [Anonymizer concepts](../anonymizer/index.md#main-concepts) | | **Operator** | An **Operator** is an object in Presidio that is responsible for performing the anonymization operation on a PII entity. Presidio provides several built-in operators, such as **Replace**, **Redact**, and **Encrypt**, and allows users to create custom operators. | [Anonymizer concepts](../anonymizer/index.md#main-concepts) | | **BatchAnonymizerEngine** | The **BatchAnonymizerEngine** is a class in Presidio that is responsible for anonymizing PII entities in a batch of texts. It uses the **AnonymizerEngine** to perform the anonymization on each text in the batch. | [Sample](../samples/python/batch_processing.ipynb) | | **ImageRedactorEngine** | The **ImageRedactorEngine** is a class in Presidio that is responsible for redacting PII entities in images. It leverages the **AnalyzerEngine** to detect PII entities in the text extracted from the images. | [Image redaction docs](../image-redactor/index.md) | | **StructuredEngine** | The **StructuredEngine** is a class in Presidio that is responsible for detecting PII entities in structured data. It uses the **AnalyzerEngine** to detect PII entities in the text fields of the structured data. | [Image redaction docs](../structured/index.md) | ## Evaluation concepts | Concept | Definition | Learn More | |--------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------| | **Precision** | **Precision** is a metric that measures the proportion of true positive results among the positive results. In the context of PII detection, precision measures the proportion of correctly identified PII entities among all the entities identified by the system. | [Evaluation docs](../evaluation/index.md) | | **Recall** | **Recall** is a metric that measures the proportion of true positive results among the actual positive results. In the context of PII detection, recall measures the proportion of correctly identified PII entities among all the PII entities present in the text.| [Evaluation docs](../evaluation/index.md) | --- ### Recipes CONTRIBUTING (`docs/recipes/CONTRIBUTING.md`) # Contributing to Presidio Recipes Gallery Thank you for your interest in contributing! Recipes help others learn how to customize Presidio for specific domains. ## What Makes a Good Recipe? A good recipe should: 1. **Focus on a Specific Domain**: Target a well-defined scenario (e.g., "Financial chatbot logs", "Clinical notes", "Customer support tickets") 2. **Be Reproducible**: Include working code that others can run 3. **Follow the Research Pattern**: Build upon the end-to-end approach from [presidio-research](https://github.com/data-privacy-stack/presidio-research), showing: - **Data Synthesis**: How to generate or obtain test data - **Configuration**: Your Presidio setup with any custom recognizers - **Evaluation**: Metrics (precision, recall, F₂) showing performance 4. **Keep It Simple**: Focus on getting a working example first. Detailed documentation can come later. ## Recipe Format ### Follow Presidio Research Examples Your recipe should follow the end-to-end evaluation approach from [presidio-research](https://github.com/data-privacy-stack/presidio-research). **Key reference notebooks:** - [Evaluate Presidio Analyzer](https://github.com/data-privacy-stack/presidio-research/blob/master/notebooks/4_Evaluate_Presidio_Analyzer.ipynb) - Complete evaluation workflow example - [Generate Synthetic Data](https://github.com/data-privacy-stack/presidio-research/blob/master/notebooks/1_Generate_data.ipynb) - Using Presidio Evaluator data generator ### Option 1: Single Jupyter Notebook (Recommended for Simple Cases) For straightforward examples, use one notebook: ``` your-recipe-name/ ├── recipe.ipynb # Main notebook with data synthesis → evaluation └── README.md # Brief overview (optional) ``` ### Option 2: Multiple Files (Recommended for Complex Flows) For complex scenarios, break into separate files: ``` your-recipe-name/ ├── 1_generate_data.ipynb # Data synthesis (use Presidio Evaluator or custom) ├── 2_configure.ipynb # Presidio setup with custom recognizers ├── 3_evaluate.ipynb # Run evaluation and analysis └── README.md # Overview and instructions ``` Or as Python scripts: ``` your-recipe-name/ ├── generate_data.py # Generate test data ├── configure.py # Presidio setup ├── evaluate.py # Run evaluation └── README.md # Overview ``` ### Required Components Your recipe should include: - **Data Synthesis**: Generate synthetic data using [Presidio Evaluator](https://github.com/data-privacy-stack/presidio-research/blob/master/notebooks/1_Generate_data.ipynb) or your own method - **Presidio Configuration**: Show your setup (default, custom recognizers, or custom models) - **Evaluation**: Measure and report precision, recall, F₂ score, latency - **Key Findings**: Brief summary of results and when to use this approach ## Quick Contribution Steps 1. **Fork the repo** and create a new branch 2. **Create your recipe folder** under `docs/recipes/your-recipe-name/` 3. **Use the template**: Copy `template.md` or start with a notebook 4. **Focus on working code first**: Don't worry about making it perfect 5. **Submit a PR**: We'll help refine it during review ## Evaluation Metrics Include at minimum: - **Precision**: Percentage of detected entities that were correct - **Recall**: Percentage of actual PII that was detected - **F₂ Score**: Recall-weighted F-score (emphasizes catching all PII) - **Latency**: Average processing time per sample ## Examples to Learn From **Presidio Research Notebooks** (recommended starting point): - [Evaluate Presidio Analyzer](https://github.com/data-privacy-stack/presidio-research/blob/master/notebooks/4_Evaluate_Presidio_Analyzer.ipynb) - Complete end-to-end evaluation example - [Generate Synthetic Data](https://github.com/data-privacy-stack/presidio-research/blob/master/notebooks/1_Generate_data.ipynb) - Presidio Evaluator data generator - [Other presidio-research notebooks](https://github.com/data-privacy-stack/presidio-research/tree/master/notebooks) - Additional examples and tools **Additional Resources:** - [Presidio Samples](../samples/index.md): Integration patterns and usage examples ## Questions? - Open an issue with the `recipe` label - Email [presidio@dataprivacystack.org](mailto:presidio@dataprivacystack.org) - Tag @omri374 in your PR for guidance ## License By contributing, you agree to license your work under the MIT License. --- ### Recipes Template (`docs/recipes/template.md`) # Recipe Template: [Recipe Name] > **Note**: This is a simplified template for contributing recipes. Focus on providing a working example - detailed documentation can come later. ## Overview **Domain**: [e.g., Healthcare, Finance, Retail] **Data Type**: [e.g., Clinical notes, Chat logs, API responses] **Goal**: [What you're trying to detect and anonymize] ## Quick Start ### Sample Data ```python # Example data that demonstrates the scenario sample_text = """ [Your example text with PII] """ ``` ### Basic Configuration ```python from presidio_analyzer import AnalyzerEngine from presidio_anonymizer import AnonymizerEngine # Your Presidio configuration analyzer = AnalyzerEngine() anonymizer = AnonymizerEngine() # Analyze and anonymize results = analyzer.analyze(text=sample_text, language="en") anonymized = anonymizer.anonymize(text=sample_text, analyzer_results=results) print(anonymized.text) ``` ## Approach Describe your approach in a few sentences. What recognizers, models, or techniques did you use? ## Results **Precision**: [0.XX] **Recall**: [0.XX] **F₂ Score**: [0.XX] **Latency**: [XX ms/sample] ### Key Findings - [Main insight 1] - [Main insight 2] - [Main insight 3] ## End-to-End Example ### Recommended Approach: Follow Presidio Research Patterns Provide an end-to-end example following the patterns from [presidio-research](https://github.com/data-privacy-stack/presidio-research): **Reference notebooks:** - [Evaluate Presidio Analyzer](https://github.com/data-privacy-stack/presidio-research/blob/master/notebooks/4_Evaluate_Presidio_Analyzer.ipynb) - Complete evaluation workflow - [Generate Synthetic Data](https://github.com/data-privacy-stack/presidio-research/blob/master/notebooks/1_Generate_data.ipynb) - Using Presidio Evaluator data generator **Structure your example:** 1. **Data Synthesis**: Generate synthetic data using [Presidio Evaluator](https://github.com/data-privacy-stack/presidio-research/blob/master/notebooks/1_Generate_data.ipynb) or your own method 2. **Configuration**: Set up Presidio with your custom recognizers or models 3. **Evaluation**: Measure precision, recall, and F₂ score 4. **Analysis**: Discuss results and trade-offs ### Format Options - **Single Jupyter Notebook**: Best for simple, focused examples (recommended for most recipes) - **Multiple Files**: For complex flows, break into separate notebooks or Python scripts: ``` your-recipe-name/ ├── 1_generate_data.ipynb # Data synthesis ├── 2_configure.ipynb # Presidio setup ├── 3_evaluate.ipynb # Evaluation └── README.md # Overview ``` ## Tips for Others - [Tip 1: When this approach works best] - [Tip 2: Common pitfalls to avoid] - [Tip 3: How to adapt to similar scenarios] --- **Author**: [Your Name] **Date**: [YYYY-MM-DD] --- ### Tutorial 00 Getting Started (`docs/tutorial/00_getting_started.md`) # Getting started ## Installation First, let's install presidio using `pip`. For detailed documentation, see the [installation docs](https://data-privacy-stack.github.io/presidio/installation). Install from PyPI: ```sh pip install presidio_analyzer pip install presidio_anonymizer python -m spacy download en_core_web_lg ``` ## Simple flow A simple call to Presidio Analyzer: ```python from presidio_analyzer import AnalyzerEngine text = "His name is Mr. Jones and his phone number is 212-555-5555" analyzer = AnalyzerEngine() analyzer_results = analyzer.analyze(text=text, language="en") print(analyzer_results) ``` Next, we'll go over ways to customize Presidio to specific needs by adding PII recognizers, using context words, NER models and more. --- ### Tutorial 01 Deny List (`docs/tutorial/01_deny_list.md`) # Example 1: Deny-list based PII recognition In this example, we will pass a short list of tokens which should be marked as PII if detected. First, let's define the tokens we want to treat as PII. In this case it would be a list of titles: ```python titles_list = [ "Sir", "Ma'am", "Madam", "Mr.", "Mrs.", "Ms.", "Miss", "Dr.", "Professor", ] ``` Second, let's create a `PatternRecognizer` which would scan for those titles, by passing a `deny_list`: ```python from presidio_analyzer import PatternRecognizer titles_recognizer = PatternRecognizer(supported_entity="TITLE", deny_list=titles_list) ``` At this point we can call our recognizer directly: ```python from presidio_analyzer import PatternRecognizer text1 = "I suspect Professor Plum, in the Dining Room, with the candlestick" result = titles_recognizer.analyze(text1, entities=["TITLE"]) print(f"Result:\n {result}") ``` Finally, let's add this new recognizer to the list of recognizers used by the Presidio `AnalyzerEngine`: ```python from presidio_analyzer import AnalyzerEngine analyzer = AnalyzerEngine() analyzer.registry.add_recognizer(titles_recognizer) ``` When initializing the `AnalyzerEngine`, Presidio loads all available recognizers, including the `NlpEngine` used to detect entities, and extract tokens, lemmas and other linguistic features. Let's run the analyzer with the new recognizer in place: ```python results = analyzer.analyze(text=text1, language="en") ``` ```python print("Results:") print(results) ``` As expected, both the name "Plum" and the title were identified as PII: ```python print("Identified these PII entities:") for result in results: print(f"- {text1[result.start:result.end]} as {result.entity_type}") ``` --- ### Tutorial 02 Regex (`docs/tutorial/02_regex.md`) # Example 2: Regular-expressions based PII recognition Another simple recognizer we can add is based on regular expressions. Let's assume we want to be extremely conservative and treat any token which contains a number as PII. ```python from presidio_analyzer import Pattern, PatternRecognizer # Define the regex pattern in a Presidio `Pattern` object: numbers_pattern = Pattern(name="numbers_pattern", regex=r"\d+", score=0.5) # Define the recognizer with one or more patterns number_recognizer = PatternRecognizer( supported_entity="NUMBER", patterns=[numbers_pattern] ) ``` Testing the recognizer itself: ```python text2 = "I live in 510 Broad st." numbers_result = number_recognizer.analyze(text=text2, entities=["NUMBER"]) print("Result:") print(numbers_result) ``` It's important to mention that recognizers are likely to have errors, both false-positive and false-negative, which would impact the entire performance of Presidio. Consider testing each recognizer on a representative dataset prior to integrating it into Presidio. For more info, see the [best practices for developing recognizers documentation](https://data-privacy-stack.github.io/presidio/analyzer/developing_recognizers/). --- ### Tutorial 03 Rule Based (`docs/tutorial/03_rule_based.md`) # Example 3: Rule based logic recognizer Taking the numbers recognizer one step further, let's say we also would like to detect numbers within words, e.g. "Number One". We can leverage the underlying `spaCy` token attributes, or write our own logic to detect such entities. Notes: - In this example we would create a new class, which implements [`EntityRecognizer`](https://github.com/data-privacy-stack/presidio/blob/main/presidio-analyzer/presidio_analyzer/entity_recognizer.py), the basic recognizer in Presidio. This abstract class requires us to implement the `load` method and `analyze` method. - Each recognizer accepts an object of type `NlpArtifacts`, which holds pre-computed attributes on the input text. A new recognizer should have this structure: ```python from typing import List from presidio_analyzer import EntityRecognizer, RecognizerResult from presidio_analyzer.nlp_engine import NlpArtifacts class MyRecognizer(EntityRecognizer): def load(self) -> None: """No loading is required.""" pass def analyze( self, text: str, entities: List[str], nlp_artifacts: NlpArtifacts ) -> List[RecognizerResult]: """ Logic for detecting a specific PII """ pass ``` For example, detecting numbers in either numerical or alphabetic (e.g. Forty five) form: ```python from typing import List from presidio_analyzer import EntityRecognizer, RecognizerResult from presidio_analyzer.nlp_engine import NlpArtifacts class NumbersRecognizer(EntityRecognizer): expected_confidence_level = 0.7 # expected confidence level for this recognizer def load(self) -> None: """No loading is required.""" pass def analyze( self, text: str, entities: List[str], nlp_artifacts: NlpArtifacts ) -> List[RecognizerResult]: """ Analyzes test to find tokens which represent numbers (either 123 or One Two Three). """ results = [] # iterate over the spaCy tokens, and call `token.like_num` for token in nlp_artifacts.tokens: if token.like_num: result = RecognizerResult( entity_type="NUMBER", start=token.idx, end=token.idx + len(token), score=self.expected_confidence_level, ) results.append(result) return results # Instantiate the new NumbersRecognizer: new_numbers_recognizer = NumbersRecognizer(supported_entities=["NUMBER"]) ``` Since this recognizer requires the `NlpArtifacts`, we would have to call it as part of the `AnalyzerEngine` flow: ```python from presidio_analyzer import AnalyzerEngine text3 = "Roberto lives in Five 10 Broad st." analyzer = AnalyzerEngine() analyzer.registry.add_recognizer(new_numbers_recognizer) numbers_results2 = analyzer.analyze(text=text3, language="en") print("Results:") print("\n".join([str(res) for res in numbers_results2])) ``` The analyzer was able to pick up both numeric and alphabetical numbers, including other types of PII entities from other recognizers (PERSON in this case). --- ### Tutorial 04 External Services (`docs/tutorial/04_external_services.md`) # Example 4: Calling an external service/framework for PII detection In a similar way to example 3, we can write logic to call external services for PII detection. There are two types of external services we support: 1. Remote services such as a PII detection model hosted somewhere. In this case, the recognizer would do the actual REST request and translate the results to a list of `RecognizerResult`. 2. Calling PII models from other frameworks, such as [transformers](https://huggingface.co/transformers/usage.html#named-entity-recognition) or [Flair](https://github.com/flairNLP/flair). ## Calling a remote service 1. [Documentation on remote recognizers](https://data-privacy-stack.github.io/presidio/analyzer/adding_recognizers/#creating-a-remote-recognizer). 2. [A sample implementation of a remote recognizer](https://github.com/data-privacy-stack/presidio/blob/main/docs/samples/python/example_remote_recognizer.py). ## Calling a model in a different framework - [This example](https://github.com/data-privacy-stack/presidio/blob/main/docs/samples/python/flair_recognizer.py) shows a Presidio wrapper for a Flair model. - Using a similar approach, we could create wrappers for HuggingFace models, Conditional Random Fields or any other framework. --- ### Tutorial 05 Languages (`docs/tutorial/05_languages.md`) # Example 5: Supporting new models and languages Two main parts in Presidio handle the text, and should be adapted if a new language is required: 1. The `NlpEngine` containing the NLP model which performs tokenization, lemmatization, Named Entity Recognition and other NLP tasks. 2. The different PII recognizers (`EntityRecognizer` objects) should be adapted or created. ## Adapting the NLP engine As its internal NLP engine, Presidio supports both spaCy and Stanza. Make sure you download the required models from spacy/stanza prior to using them. More details [here](https://data-privacy-stack.github.io/presidio/analyzer/languages/#configuring-the-nlp-engine). For example, to download the Spanish medium spaCy model: `python -m spacy download es_core_news_md` In this example we will configure Presidio to use spaCy as its underlying NLP framework, with NLP models in English and Spanish: ```python from presidio_analyzer import AnalyzerEngine from presidio_analyzer.nlp_engine import NlpEngineProvider # import spacy # spacy.cli.download("es_core_news_md") # Create configuration containing engine name and models configuration = { "nlp_engine_name": "spacy", "models": [ {"lang_code": "es", "model_name": "es_core_news_md"}, {"lang_code": "en", "model_name": "en_core_web_lg"}, ], } # Create NLP engine based on configuration provider = NlpEngineProvider(nlp_configuration=configuration) nlp_engine_with_spanish = provider.create_engine() # Pass the created NLP engine and supported_languages to the AnalyzerEngine analyzer = AnalyzerEngine( nlp_engine=nlp_engine_with_spanish, supported_languages=["en", "es"] ) # Analyze in different languages results_spanish = analyzer.analyze(text="Mi nombre es Morris", language="es") print("Results from Spanish request:") print(results_spanish) results_english = analyzer.analyze(text="My name is Morris", language="en") print("Results from English request:") print(results_english) ``` [See this documentation](https://data-privacy-stack.github.io/presidio/analyzer/languages/) for more details on setting up additional NLP models and languages. ## Using external models/frameworks Some languages are not supported by spaCy/Stanza/huggingface, or have very limited support in those. In this case, other frameworks could be leveraged. (see [example 4](04_external_services.md) for more information). Since Presidio requires a spaCy model to be passed, we propose to use a simple spaCy pipeline such as `en_core_web_sm` as the NLP engine's model, and a recognizer calling an external framework/service as the Named Entity Recognition (NER) model. --- ### Tutorial 06 Context (`docs/tutorial/06_context.md`) # Example 6: Leveraging context words Presidio has an internal mechanism for leveraging context words. This mechanism would increase the detection confidence of a PII entity in case a specific word appears before or after it. Furthermore, it is possible to create your own context enhancer, if you require a different logic for identifying context terms. The default context-aware enhancer in Presidio is the `LemmaContextAwareEnhancer` which compares each recognizer's context terms with the lemma of each token in the sentence. In this example we would first implement a zip code recognizer without context, and then add context to see how the confidence changes. Zip regex patterns (essentially 5 digits) are very weak, so we would want the initial confidence to be low, and increased with the existence of context words. ## Example: Adding context words support to recognizers First, let's create a simple `US_ZIP_CODE` recognizer: ```python from presidio_analyzer import ( Pattern, PatternRecognizer, RecognizerRegistry, AnalyzerEngine, ) # Define the regex pattern regex = r"(\b\d{5}(?:\-\d{4})?\b)" # very weak regex pattern zipcode_pattern = Pattern(name="zip code (weak)", regex=regex, score=0.01) # Define the recognizer with the defined pattern zipcode_recognizer = PatternRecognizer( supported_entity="US_ZIP_CODE", patterns=[zipcode_pattern] ) registry = RecognizerRegistry() registry.add_recognizer(zipcode_recognizer) analyzer = AnalyzerEngine(registry=registry) # Test results = analyzer.analyze(text="My zip code is 90210", language="en") print(f"Result:\n {results}") ``` So this is working, but would catch any 5 digit string. This is why we set the score to 0.01. Let's use context words to increase score: ```python from presidio_analyzer import PatternRecognizer # Define the recognizer with the defined pattern and context words zipcode_recognizer_w_context = PatternRecognizer( supported_entity="US_ZIP_CODE", patterns=[zipcode_pattern], context=["zip", "zipcode"], ) ``` When creating an ```AnalyzerEngine``` we can provide our own context enhancement logic by passing it to ```context_aware_enhancer``` parameter. ```AnalyzerEngine``` will create ```LemmaContextAwareEnhancer``` by default if not passed, which will enhance score of each matched result if its recognizer holds context words and the lemma of those words are found in the surroundings of the matched entity. Creating the `AnalyzerEngine` and adding the new recognizer: ```python from presidio_analyzer import AnalyzerEngine, RecognizerRegistry registry = RecognizerRegistry() registry.add_recognizer(zipcode_recognizer_w_context) analyzer = AnalyzerEngine(registry=registry) # Test results = analyzer.analyze(text="My zip code is 90210", language="en") print("Result:") print(results) ``` The confidence score is now 0.4, instead of 0.01, since the ```LemmaContextAwareEnhancer``` default context similarity factor is 0.35 and default minimum score with context similarity is 0.4. We can change that by passing other values to the ```context_similarity_factor``` and ```min_score_with_context_similarity``` parameters of the ```LemmaContextAwareEnhancer``` object. For example: ```python from presidio_analyzer import AnalyzerEngine, RecognizerRegistry from presidio_analyzer.context_aware_enhancers import LemmaContextAwareEnhancer context_aware_enhancer = LemmaContextAwareEnhancer( context_similarity_factor=0.45, min_score_with_context_similarity=0.4 ) registry = RecognizerRegistry() registry.add_recognizer(zipcode_recognizer_w_context) analyzer = AnalyzerEngine( registry=registry, context_aware_enhancer=context_aware_enhancer ) # Test results = analyzer.analyze(text="My zip code is 90210", language="en") print("Result:") print(results) ``` The confidence score is now 0.46 because it got enhanced from 0.01 with 0.45 and is more than the minimum of 0.4. In addition to surrounding words, additional context words could be passed on the request level. This is useful when there is context coming from metadata such as column names or a specific user input. In the following example, notice how the "zip" context word doesn't appear in the text but still enhances the confidence score from 0.01 to 0.4: ```python from presidio_analyzer import AnalyzerEngine, RecognizerRegistry, PatternRecognizer # Define the recognizer with the defined pattern and context words zipcode_recognizer = PatternRecognizer( supported_entity="US_ZIP_CODE", patterns=[zipcode_pattern], context=["zip", "zipcode"], ) registry = RecognizerRegistry() registry.add_recognizer(zipcode_recognizer) analyzer = AnalyzerEngine(registry=registry) # Test with an example record having a column name which could be injected as context record = {"column_name": "zip", "text": "My code is 90210"} result = analyzer.analyze( text=record["text"], language="en", context=[record["column_name"]] ) print("Result:") print(result) ``` --- ### Tutorial 07 Decision Process (`docs/tutorial/07_decision_process.md`) # Example 7: Tracing the decision process Presidio-analyzer's decision process exposes information on why a specific PII was detected. Such information could contain: - Which recognizer detected the entity - Which regex pattern was used - Interpretability mechanisms in ML models - Which context words improved the score - Confidence scores before and after each step And more. For more information, refer to the [decision process documentation](https://data-privacy-stack.github.io/presidio/analyzer/decision_process/). Let's use the decision process output to understand how the zip code value was detected: ```python from presidio_analyzer import AnalyzerEngine import pprint analyzer = AnalyzerEngine() results = analyzer.analyze( text="My zip code is 90210", language="en", return_decision_process=True ) decision_process = results[0].analysis_explanation pp = pprint.PrettyPrinter() print("Decision process output:\n") pp.pprint(decision_process.__dict__) ``` When developing new recognizers, one can add information to this explanation and extend it with additional findings. --- ### Tutorial 08 No Code (`docs/tutorial/08_no_code.md`) # No code configuration No-code configuration can be helpful in three scenarios: 1. There's an existing set of regular expressions / deny-lists that should be leveraged within Presidio. 2. As a simple way to configure which recognizers to enable and disable, and how to configure the NLP engine. 3. For team members interested in changing the configuration without writing code. In this example, we'll show how to create a no-code configuration in Presidio. We start by creating YAML configuration files that are based on the default ones. The default configuration files for Presidio can be found here: - [Analyzer configuration](https://github.com/data-privacy-stack/presidio/blob/main/presidio-analyzer/presidio_analyzer/conf/default_analyzer.yaml) - [Recognizer registry configuration](https://github.com/data-privacy-stack/presidio/blob/main/presidio-analyzer/presidio_analyzer/conf/default_recognizers.yaml) - [NLP engine configuration](https://github.com/data-privacy-stack/presidio/blob/main/presidio-analyzer/presidio_analyzer/conf/default.yaml) Alternatively, one can create one configuration file for all three components. In this example, we'll tweak the configuration to reduce the number of predefined recognizers to only a few, and add a new custom one. We'll also adjust the context words to support the detection of a different language (Spanish). ```python import yaml import json import tempfile from pprint import pprint from presidio_analyzer import AnalyzerEngineProvider ``` In this example we're going to create the yaml as a string for illustration purposes, but the more common scenario is to create these YAML files and load them into the `PresidioAnalyzerProvider`. ## Defining the configuration in YAML format ### General Analyzer parameters ([default file](https://github.com/data-privacy-stack/presidio/blob/main/presidio-analyzer/presidio_analyzer/conf/default_analyzer.yaml)) ```python analyzer_config_yaml = """ supported_languages: - en - es default_score_threshold: 0.4 """ ``` ### Recognizer Registry parameters ([default file](https://github.com/data-privacy-stack/presidio/blob/main/presidio-analyzer/presidio_analyzer/conf/default_recognizers.yaml)) ``` /* Detailed source-code truncated for AI context efficiency. */ ``` Each recognizer can set a `default` threshold and entity-specific overrides in `score_thresholds`. Supplying `analyze(score_threshold=...)` bypasses recognizer-level thresholds for that request. When omitted, precedence is an entity override, the recognizer default, then the analyzer's `default_score_threshold`. ### NLP Engine parameters ([default file](https://github.com/data-privacy-stack/presidio/blob/main/presidio-analyzer/presidio_analyzer/conf/default.yaml)) ```python nlp_engine_yaml = """ nlp_configuration: nlp_engine_name: transformers models: - lang_code: en model_name: spacy: en_core_web_sm transformers: StanfordAIMI/stanford-deidentifier-base - lang_code: es model_name: spacy: es_core_news_sm transformers: MMG/xlm-roberta-large-ner-spanish ner_model_configuration: labels_to_ignore: - O aggregation_strategy: first # "simple", "first", "average", "max" stride: 16 alignment_mode: expand # "strict", "contract", "expand" model_to_presidio_entity_mapping: PER: PERSON PERSON: PERSON LOC: LOCATION LOCATION: LOCATION GPE: LOCATION ORG: ORGANIZATION ORGANIZATION: ORGANIZATION NORP: NRP AGE: AGE ID: ID EMAIL: EMAIL PATIENT: PERSON STAFF: PERSON HOSP: ORGANIZATION PATORG: ORGANIZATION DATE: DATE_TIME TIME: DATE_TIME PHONE: PHONE_NUMBER HCW: PERSON HOSPITAL: LOCATION FACILITY: LOCATION VENDOR: ORGANIZATION MISC: ID low_confidence_score_multiplier: 0.4 low_score_entity_names: - ID """ ``` ## Creating the analyzer engine and running it ### Create a unified YAML file and save it as a temp file ```python full_config = f"{analyzer_config_yaml}\n{recognizer_registry_config_yaml}\n{nlp_engine_yaml}" with tempfile.NamedTemporaryFile(mode='w+', delete=False, suffix='.yaml') as temp_file: # Write the YAML string to the temp file temp_file.write(full_config) temp_file_path = temp_file.name ``` ### Pass the YAML file to `AnalyzerEngineProvider` to create an `AnalyzerEngine` instance ```python analyzer_engine = AnalyzerEngineProvider(analyzer_engine_conf_file=temp_file_path).create_engine() ``` ### Print the loaded configuration for both languages ```python for lang in ("en", "es"): pprint(f"Supported entities for {lang}:") print("\n") pprint(analyzer_engine.get_supported_entities(lang), compact=True) print(f"\nLoaded recognizers for {lang}:") pprint([rec.name for rec in analyzer_engine.registry.get_recognizers(lang, all_fields=True)], compact=True) print("\n") print(f"\nLoaded NER models:") pprint(analyzer_engine.nlp_engine.models) ``` ## Run two requests, one in English and one in Spanish ```python es_text = "Hola, me llamo David Johnson y soy originalmente de Liverpool. Mi número de tarjeta de crédito es 4095260993934932" analyzer_engine.analyze(es_text, language="es") ``` ```python en_text = "Hi, my name is David Johnson and I'm originally from Liverpool. My credit card number is 4095260993934932" analyzer_engine.analyze(en_text, language="en") ``` --- ### Tutorial 09 Ad Hoc (`docs/tutorial/09_ad_hoc.md`) # Example 9: Ad-hoc recognizers In addition to recognizers in code or in a YAML file, it is possible to create ad-hoc recognizers via the Presidio Analyzer API for regex and deny-list based logic. These recognizers, in JSON form, are added to the `/analyze` request and are only used in the context of this request. !!! note "Note" These ad-hoc recognizers could be useful if Presidio is already deployed, but requires additional detection logic to be added. - The json structure for a *regex ad-hoc recognizer* is the following: ```json { "text": "John Smith drivers license is AC432223. Zip code: 10023", "language": "en", "ad_hoc_recognizers":[ { "name": "Zip code Recognizer", "supported_language": "en", "patterns": [ { "name": "zip code (weak)", "regex": "(\\b\\d{5}(?:\\-\\d{4})?\\b)", "score": 0.01 } ], "context": ["zip", "code"], "supported_entity":"ZIP" } ] } ``` - The json structure for *deny-list based recognizers* is the following: ```json { "text": "Mr. John Smith's drivers license is AC432223", "language": "en", "ad_hoc_recognizers":[ { "name": "Mr. Recognizer", "supported_language": "en", "deny_list": ["Mr", "Mr.", "Mister"], "supported_entity":"MR_TITLE" }, { "name": "Ms. Recognizer", "supported_language": "en", "deny_list": ["Ms", "Ms.", "Miss", "Mrs", "Mrs."], "supported_entity":"MS_TITLE" } ] } ``` In both examples, the `/analyze` request is extended with a list of `ad_hoc_recognizers`, which could be either `patterns`, `deny_list` or both. Example call to the `/analyze` service: ``` json { "text": "John Smith drivers license is AC432223 and the zip code is 12345", "language": "en", "return_decision_process": false, "correlation_id": "123e4567-e89b-12d3-a456-426614174000", "score_threshold": 0.6, "entities": [ "US_DRIVER_LICENSE", "ZIP" ], "trace": false, "ad_hoc_recognizers": [ { "name": "Zip code Recognizer", "supported_language": "en", "patterns": [ { "name": "zip code (weak)", "regex": "(\\b\\d{5}(?:\\-\\d{4})?\\b)", "score": 0.01 } ], "context": [ "zip", "code" ], "supported_entity": "ZIP" } ] } ``` For more examples of deny-list recognizers, see [this sample](../samples/python/Anonymizing%20known%20values.ipynb). --- ### Tutorial 10 Simple Anonymization (`docs/tutorial/10_simple_anonymization.md`) # Example 10: Simple anonymization Once we have the identified PII entities, we can perform different de-identification operations on them. For more information on the supported operators, see [the anonymizer documentation](../anonymizer/index.md). The anonymizer requires a configuration specifying the requested operation on each entity type. There's also a default operator which replaces a PII entity with the entity type name. Each operator has a unique configuration with the parameters needed to perform the operation (redact, hash, mask, replace, encrypt etc.) Here's an simple example of using presidio-anonymizer: ```python from presidio_anonymizer import AnonymizerEngine from presidio_anonymizer.entities import RecognizerResult # Analyzer output analyzer_results = [ RecognizerResult(entity_type="PERSON", start=11, end=15, score=0.8), RecognizerResult(entity_type="PERSON", start=17, end=27, score=0.8), ] # Initialize the engine: engine = AnonymizerEngine() # Invoke the anonymize function with the text, # analyzer results (potentially coming from presidio-analyzer) and # Operators to get the anonymization output: result = engine.anonymize( text="My name is Bond, James Bond", analyzer_results=analyzer_results ) print("De-identified text") print(result.text) ``` To introduce additional operators, we can pass an `OperatorConfig`. In this example we: 1. Mask the last 12 chars of a `PHONE_NUMBER` entity and replace them with `*` 2. Redact a `TITLE` entity 3. Replace all other entities with the string ``. Defining the operators: ```python # Define anonymization operators operators = { "DEFAULT": OperatorConfig("replace", {"new_value": ""}), "PHONE_NUMBER": OperatorConfig( "mask", { "type": "mask", "masking_char": "*", "chars_to_mask": 12, "from_end": True, }, ), "TITLE": OperatorConfig("redact", {}), } ``` Full example: ```python from pprint import pprint import json from presidio_anonymizer import AnonymizerEngine from presidio_anonymizer.entities import OperatorConfig, RecognizerResult # Analyzer output analyzer_results = [ RecognizerResult(entity_type="PERSON", start=11, end=15, score=0.8), RecognizerResult(entity_type="PERSON", start=17, end=27, score=0.8), ] text_to_anonymize = "My name is Bond, James Bond" anonymizer = AnonymizerEngine() # Define anonymization operators operators = { "DEFAULT": OperatorConfig("replace", {"new_value": ""}), "PHONE_NUMBER": OperatorConfig( "mask", { "type": "mask", "masking_char": "*", "chars_to_mask": 12, "from_end": True, }, ), "TITLE": OperatorConfig("redact", {}), } anonymized_results = anonymizer.anonymize( text=text_to_anonymize, analyzer_results=analyzer_results, operators=operators ) print(f"text: {anonymized_results.text}") print("detailed result:") pprint(json.loads(anonymized_results.to_json())) ``` --- ### Tutorial 11 Custom Anonymization (`docs/tutorial/11_custom_anonymization.md`) # Example 11: Custom anonymization Presidio-anonymizer can accept arbitrary operations to perform on identified entities. These operations can be passed in the form of a lambda function. In the following example, we use fake values to perform pseudonymization. First, let's look at the operator: ```python from faker import Faker from presidio_anonymizer.entities import OperatorConfig fake = Faker() # Create faker function (note that it has to receive a value) def fake_name(x): return fake.name() # Create custom operator for the PERSON entity operators = {"PERSON": OperatorConfig("custom", {"lambda": fake_name})} ``` Full example: ```python from presidio_anonymizer import AnonymizerEngine from presidio_anonymizer.entities import OperatorConfig, EngineResult, RecognizerResult from faker import Faker fake = Faker() # Create faker function (note that it has to receive a value) def fake_name(x): return fake.name() # Create custom operator for the PERSON entity operators = {"PERSON": OperatorConfig("custom", {"lambda": fake_name})} # Analyzer output analyzer_results = [RecognizerResult(entity_type="PERSON", start=11, end=18, score=0.8)] text_to_anonymize = "My name is Raphael and I like to fish." anonymizer = AnonymizerEngine() anonymized_results = anonymizer.anonymize( text=text_to_anonymize, analyzer_results=analyzer_results, operators=operators ) print(anonymized_results.text) ``` This is a simple example, but here are some examples for more advanced anonymization options: - Identify the gender and create a random value from the same gender (e.g., Laura -> Pam) - Identifying the date pattern and perform date shift (01-01-2020 -> 05-01-2020) - Identify the age and bucket by decade (89 -> 80-90) --- ### Tutorial 12 Encryption (`docs/tutorial/12_encryption.md`) # Example 12: Encryption and decryption This sample shows how to use Presidio Anonymizer built-in functionality, to encrypt and decrypt identified entities. The encryption is using AES cypher in CBC mode and requires a cryptographic key as an input for both the encryption and the decryption. ## Set up imports ```python from presidio_anonymizer import AnonymizerEngine, DeanonymizeEngine from presidio_anonymizer.entities import ( RecognizerResult, OperatorResult, OperatorConfig, ) ``` ## Define a cryptographic key (for both encryption and decryption) ```python crypto_key = "WmZq4t7w!z%C&F)J" ``` ## Presidio Anonymizer: Encrypt ```python engine = AnonymizerEngine() # Invoke the anonymize function with the text, # analyzer results (potentially coming from presidio-analyzer) # and an 'encrypt' operator to get an encrypted anonymization output: anonymize_result = engine.anonymize( text="My name is James Bond", analyzer_results=[ RecognizerResult(entity_type="PERSON", start=11, end=21, score=0.8), ], operators={"PERSON": OperatorConfig("encrypt", {"key": crypto_key})}, ) anonymize_result ``` The output contains both the anonymized text, as well as the location of the encrypted entities. This is useful as we would need to decrypt only the entities and not the full text: ```python # Fetch the anonymized text from the result. anonymized_text = anonymize_result.text # Fetch the anonynized entities from the result. anonymized_entities = anonymize_result.items ``` ## Presidio Anonymizer: Decrypt ```python # Initialize the engine: engine = DeanonymizeEngine() # Invoke the deanonymize function with the text, anonymizer results # and a 'decrypt' operator to get the original text as output. deanonymized_result = engine.deanonymize( text=anonymized_text, entities=anonymized_entities, operators={"DEFAULT": OperatorConfig("decrypt", {"key": crypto_key})}, ) deanonymized_result ``` ## Alternatively, call the Decrypt operator directly ```python from presidio_anonymizer.operators import Decrypt # Fetch the encrypted entity value from the previous stage encrypted_entity_value = anonymize_result.items[0].text # Restore the original entity value Decrypt().operate(text=encrypted_entity_value, params={"key": crypto_key}) ``` --- ### Tutorial 13 Allow List (`docs/tutorial/13_allow_list.md`) # Example 13: Allow-list to exclude words from being identified as PII In this example, we will define a list of tokens that should not be marked as PII even if we want to identify others of that kind. In this example, we will pass a short list of tokens which should not be marked as PII even if detected by one of the recognizers. ```python websites_list = [ "bing.com", "microsoft.com" ] ``` We will use the built in recognizers that include the `URLRecognizer` and the NLP model `EntityRecognizer` and see the default functionality if we don't specify any list of words for the detector to allow to keep in the text. ```python from presidio_analyzer import AnalyzerEngine text1 = "My favorite website is bing.com, his is microsoft.com" analyzer = AnalyzerEngine() result = analyzer.analyze(text = text1, language = 'en') print(f"Result: \n {result}") ``` To specify an allow list we just pass a list of values we want to keep as a parameter to call to `analyze`. Now we can see that in the results, `bing.com` is no longer being recognized as a PII item, only `microsoft.com` is still recognized since we did include it in the allow list. ```python result = analyzer.analyze(text = text1, language = 'en', allow_list = ["bing.com"] ) print(f"Result:\n {result}") ``` --- ### Analyzer Nlp Engines Gpu Usage (`docs/analyzer/nlp_engines/gpu_usage.md`) # GPU Acceleration Presidio supports GPU acceleration for NLP models, which can significantly improve performance when processing large volumes of text. GPU is automatically detected and used when available—no code changes required. ## Environment Variable Override You can explicitly control device selection using the `PRESIDIO_DEVICE` environment variable. Any valid PyTorch device string is accepted: ```bash # Force CPU usage export PRESIDIO_DEVICE=cpu # Force GPU usage export PRESIDIO_DEVICE=cuda # Use a specific GPU export PRESIDIO_DEVICE=cuda:0 export PRESIDIO_DEVICE=cuda:1 # Or when running Docker containers docker run -e PRESIDIO_DEVICE=cuda ghcr.io/data-privacy-stack/presidio-analyzer:latest ``` !!! note "Priority" When `PRESIDIO_DEVICE` is set, it takes priority over automatic detection. If not set (or empty), Presidio will auto-detect available hardware. ## Prerequisites ### Hardware Requirements | Hardware | Requirement | |----------|-------------| | **NVIDIA GPU** | CUDA Toolkit 11.x or 12.x | | **Apple Silicon** | MPS (Metal Performance Shaders) on M1/M2/M3 | | **CPU** | Automatic fallback when GPU is unavailable | ### Software Dependencies === "NVIDIA GPU" Install the appropriate CUDA library matching your CUDA version: ```bash pip install "spacy[cuda12x]" # For CUDA 12.x # or pip install "spacy[cuda11x]" # For CUDA 11.x ``` !!! warning "Silent Fallback" If `cupy` is not installed or the version mismatches your CUDA driver, spaCy will silently fall back to CPU. Always verify your setup using the methods below. === "Apple Silicon" No additional dependencies required. MPS is detected automatically. ## Verifying GPU Usage To check if GPU is available and being used: ```python import torch if torch.cuda.is_available(): print(f"GPU available: CUDA ({torch.cuda.get_device_name(0)})") elif hasattr(torch.backends, 'mps') and torch.backends.mps.is_available(): print("GPU available: MPS (Apple Silicon)") else: print("Using CPU") ``` You can also monitor GPU usage during processing: ```bash # For NVIDIA GPUs watch -n 1 nvidia-smi # For Apple Silicon sudo powermetrics --samplers gpu_power ``` ## GPU-Enabled NLP Engines | Engine | GPU Support | |--------|-------------| | **TransformersNlpEngine** | ✅ Full | | **GLiNER** | ✅ Full | | **Stanza** | ✅ Full | | **spaCy (transformer models)** | ✅ Full | | **spaCy (standard models)** | ⚠️ Not recommended | !!! warning "Standard spaCy Models" GPU is recommended for Transformers, Stanza, and GLiNER workloads. Standard spaCy models (e.g., `en_core_web_lg`) may be slower on GPU due to data transfer overhead. ## Troubleshooting ### GPU not detected 1. Check CUDA is installed: ```bash nvidia-smi nvcc --version ``` 2. Verify cupy can see your GPU: ```python import cupy as cp print(cp.cuda.runtime.getDeviceCount()) # Should be > 0 ``` 3. Reinstall cupy with correct CUDA version: ```bash pip uninstall cupy cupy-cuda11x cupy-cuda12x pip install cupy-cuda12x # Match your CUDA version ``` ### Out of memory errors If you see `RuntimeError: CUDA out of memory`: - Use a smaller transformer model (e.g., `dslim/bert-base-NER` instead of larger models) - Split longer texts into smaller chunks before processing - Reduce the maximum sequence length if using Hugging Face transformers ### CPU fallback Presidio automatically uses CPU if: - No GPU is detected - CUDA or cupy are not installed - GPU initialization fails This ensures code portability across different environments. ## Additional Resources - [spaCy: GPU Usage](https://spacy.io/usage/embeddings-transformers#gpu) - [NVIDIA CUDA Installation Guide](https://docs.nvidia.com/cuda/cuda-installation-guide-linux/) - [cupy Documentation](https://docs.cupy.dev/en/stable/install.html) Related Presidio documentation: - [Transformer Models](transformers.md) - [spaCy and Stanza Models](spacy_stanza.md) - [Customizing NLP Models](../customizing_nlp_models.md) --- ### Analyzer Nlp Engines Spacy Stanza (`docs/analyzer/nlp_engines/spacy_stanza.md`) # spaCy/Stanza NLP engine Presidio can be loaded with pre-trained or custom models coming from spaCy or Stanza. ## Using a public pre-trained spaCy/Stanza model ### Download the pre-trained model To replace the default model with a different public model, first download the desired spaCy/Stanza NER models. - To download a new model with spaCy: ```sh python -m spacy download es_core_news_md ``` In this example we download the medium size model for Spanish. - To download a new model with Stanza: ```python import stanza stanza.download("en") # where en is the language code of the model. ``` For the available models, follow these links: [spaCy](https://spacy.io/usage/models), [stanza](https://stanfordnlp.github.io/stanza/available_models.html#available-ner-models). !!! tip "Tip" For Person, Location and Organization detection, it could be useful to try out the transformers based models (e.g. `en_core_web_trf`) which uses a more modern deep-learning architecture, but is generally slower than the default `en_core_web_lg` model. ### Configure Presidio to use the pre-trained model Once created, see [the NLP configuration documentation](../customizing_nlp_models.md#Configure-Presidio-to-use-the-new-model) for more information. ## How NER results flow within Presidio This diagram describes the flow of NER results within Presidio, and the relationship between the `SpacyNlpEngine` component and the `SpacyRecognizer` component: ```mermaid sequenceDiagram AnalyzerEngine->>SpacyNlpEngine: Call engine.process_text(text)
to get model results SpacyNlpEngine->>spaCy: Call spaCy pipeline spaCy->>SpacyNlpEngine: return entities and other attributes Note over SpacyNlpEngine: Map entity names to Presidio's,
update scores,
remove unwanted entities
based on NerModelConfiguration SpacyNlpEngine->>AnalyzerEngine: Pass NlpArtifacts
(Entities, lemmas, tokens, scores etc.) Note over AnalyzerEngine: Call all recognizers AnalyzerEngine->>SpacyRecognizer: Pass NlpArtifacts Note over SpacyRecognizer: Extract PII entities out of NlpArtifacts SpacyRecognizer->>AnalyzerEngine: Return List[RecognizerResult] ``` ## Training your own model !!! note "Note" A labeled dataset containing text and labeled PII entities is required for training a new model. For more information on model training and evaluation for Presidio, see the [Presidio-Research Github repository](https://github.com/data-privacy-stack/presidio-research). To train your own model, see these links on spaCy and Stanza: - [Train your own spaCy model](https://spacy.io/usage/training). - [Train your own Stanza model](https://stanfordnlp.github.io/stanza/training.html). Once models are trained, they should be installed locally in the same environment as Presidio Analyzer. ## Using a previously loaded spaCy pipeline If the app is already loading an existing spaCy NLP pipeline, it can be re-used to prevent presidio from loading it again by extending the relevant engine. ```python from presidio_analyzer import AnalyzerEngine from presidio_analyzer.nlp_engine import SpacyNlpEngine import spacy # Create a class inheriting from SpacyNlpEngine class LoadedSpacyNlpEngine(SpacyNlpEngine): def __init__(self, loaded_spacy_model): super().__init__() self.nlp = {"en": loaded_spacy_model} # Load a model a-priori nlp = spacy.load("en_core_web_sm") # Pass the loaded model to the new LoadedSpacyNlpEngine loaded_nlp_engine = LoadedSpacyNlpEngine(loaded_spacy_model = nlp) # Pass the engine to the analyzer analyzer = AnalyzerEngine(nlp_engine = loaded_nlp_engine) # Analyze text analyzer.analyze(text="My name is Bob", language="en") ``` --- ### Analyzer Nlp Engines Transformers (`docs/analyzer/nlp_engines/transformers.md`) # Transformers based Named Entity Recognition models Presidio's `TransformersNlpEngine` consists of a spaCy pipeline which encapsulates a Huggingface Transformers model instead of the spaCy NER component: Presidio leverages other types of information from spaCy such as tokens, lemmas and part-of-speech. Therefore the pipeline returns both the NER model results as well as results from other pipeline components. ## How NER results flow within Presidio This diagram describes the flow of NER results within Presidio, and the relationship between the `TransformersNlpEngine` component and the `TransformersRecognizer` component: ```mermaid sequenceDiagram AnalyzerEngine->>TransformersNlpEngine: Call engine.process_text(text)
to get model results TransformersNlpEngine->>spaCy: Call spaCy pipeline spaCy->>transformers: call NER model transformers->>spaCy: get entities spaCy->>TransformersNlpEngine: return transformers entities
+ spaCy attributes Note over TransformersNlpEngine: Map entity names to Presidio's,
update scores,
remove unwanted entities
based on NerModelConfiguration TransformersNlpEngine->>AnalyzerEngine: Pass NlpArtifacts
(Entities, lemmas, tokens, scores etc.) Note over AnalyzerEngine: Call all recognizers AnalyzerEngine->>TransformersRecognizer: Pass NlpArtifacts Note over TransformersRecognizer: Extract PII entities out of NlpArtifacts TransformersRecognizer->>AnalyzerEngine: Return List[RecognizerResult] ``` ## Adding a new model As the underlying transformers model, you can choose from either a public pretrained model or a custom model. ### Using a public pre-trained transformers model #### Downloading a pre-trained model To download the desired NER model from HuggingFace: ```python import transformers from huggingface_hub import snapshot_download from transformers import AutoTokenizer, AutoModelForTokenClassification transformers_model = # e.g. "obi/deid_roberta_i2b2" snapshot_download(repo_id=transformers_model) # Instantiate to make sure it's downloaded during installation and not runtime AutoTokenizer.from_pretrained(transformers_model) AutoModelForTokenClassification.from_pretrained(transformers_model) ``` Then, also download a spaCy pipeline/model: ```sh python -m spacy download en_core_web_sm ``` ### Configuring the NER pipeline Once the models are downloaded, one option to configure them is to create a YAML configuration file. Note that the configuration needs to contain both a `spaCy` pipeline name and a transformers model name. In addition, different configurations for parsing the results of the transformers model can be added. The NER model configuration can be done in a YAML file or in Python: #### Configuring the NER pipeline via code Example configuration in Python: ```python # Transformer model config model_config = [ {"lang_code": "en", "model_name": { "spacy": "en_core_web_sm", # for tokenization, lemmatization "transformers": "StanfordAIMI/stanford-deidentifier-base" # for NER } }] # Entity mappings between the model's and Presidio's mapping = dict( PER="PERSON", LOC="LOCATION", ORG="ORGANIZATION", AGE="AGE", ID="ID", EMAIL="EMAIL", DATE="DATE_TIME", PHONE="PHONE_NUMBER", PERSON="PERSON", LOCATION="LOCATION", GPE="LOCATION", ORGANIZATION="ORGANIZATION", NORP="NRP", PATIENT="PERSON", STAFF="PERSON", HOSP="LOCATION", PATORG="ORGANIZATION", TIME="DATE_TIME", HCW="PERSON", HOSPITAL="LOCATION", FACILITY="LOCATION", VENDOR="ORGANIZATION", ) labels_to_ignore = ["O"] ner_model_configuration = NerModelConfiguration( model_to_presidio_entity_mapping=mapping, alignment_mode="expand", # "strict", "contract", "expand" aggregation_strategy="max", # "simple", "first", "average", "max" labels_to_ignore = labels_to_ignore) transformers_nlp_engine = TransformersNlpEngine( models=model_config, ner_model_configuration=ner_model_configuration) # Transformer-based analyzer analyzer = AnalyzerEngine( nlp_engine=transformers_nlp_engine, supported_languages=["en"] ) ``` #### Creating a YAML configuration file Once the models are downloaded, one option to configure them is to create a YAML configuration file. Note that the configuration needs to contain both a `spaCy` pipeline name and a transformers model name. In addition, different configurations for parsing the results of the transformers model can be added. Example configuration (in YAML): ```yaml nlp_engine_name: transformers models: - lang_code: en model_name: spacy: en_core_web_sm transformers: StanfordAIMI/stanford-deidentifier-base ner_model_configuration: labels_to_ignore: - O aggregation_strategy: max # "simple", "first", "average", "max" stride: 16 alignment_mode: expand # "strict", "contract", "expand" model_to_presidio_entity_mapping: PER: PERSON LOC: LOCATION ORG: ORGANIZATION AGE: AGE ID: ID EMAIL: EMAIL PATIENT: PERSON STAFF: PERSON HOSP: ORGANIZATION PATORG: ORGANIZATION DATE: DATE_TIME PHONE: PHONE_NUMBER HCW: PERSON HOSPITAL: LOCATION VENDOR: ORGANIZATION low_confidence_score_multiplier: 0.4 low_score_entity_names: - ID ``` ##### Calling the new model Once the configuration file is created, it can be used to create a new `TransformersNlpEngine`: ```python from presidio_analyzer import AnalyzerEngine, RecognizerRegistry from presidio_analyzer.nlp_engine import NlpEngineProvider # Create configuration containing engine name and models conf_file = PATH_TO_CONF_FILE # Create NLP engine based on configuration provider = NlpEngineProvider(conf_file=conf_file) nlp_engine = provider.create_engine() # Pass the created NLP engine and supported_languages to the AnalyzerEngine analyzer = AnalyzerEngine( nlp_engine=nlp_engine, supported_languages=["en"] ) results_english = analyzer.analyze(text="My name is Morris", language="en") print(results_english) ``` #### Explaining the configuration options - `model_name.spacy` is a name of a spaCy model/pipeline, which would wrap the transformers NER model. For example, `en_core_web_sm`. - The `model_name.transformers` is the full path for a huggingface model. Models can be found on [HuggingFace Models Hub](https://huggingface.co/models?pipeline_tag=token-classification). For example, `obi/deid_roberta_i2b2` The `ner_model_configuration` section contains the following parameters: - `labels_to_ignore`: A list of labels to ignore. For example, `O` (no entity) or entities you are not interested in returning. - `aggregation_strategy`: The strategy to use when aggregating the results of the transformers model. - `stride`: The value is the length of the window overlap in transformer tokenizer tokens. - `alignment_mode`: The strategy to use when aligning the results of the transformers model to the original text. - `model_to_presidio_entity_mapping`: A mapping between the transformers model labels and the Presidio entity types. - `low_confidence_score_multiplier`: A multiplier to apply to the score of entities with low confidence. - `low_score_entity_names`: A list of entity types to apply the low confidence score multiplier to. !!! note "Defining the entity mapping" To be able to create the `model_to_presidio_entity_mapping` dictionary, it is advised to check which classes the model is able to predict. This can be found on the huggingface hub site for the model in some cases. In other, one can check the model's `config.json` under `id2label`. For example, for `bert-base-NER-uncased`, it can be found here: . Note that most NER models add a prefix to the class (e.g. `B-PER` for class `PER`). When creating the mapping, do not add the prefix. See more information on parameters on the [spacy-huggingface-pipelines Github repo](https://github.com/explosion/spacy-huggingface-pipelines#token-classification). Once created, see [the NLP configuration documentation](../customizing_nlp_models.md#Configure-Presidio-to-use-the-new-model) for more information. ### Training your own model !!! note "Note" A labeled dataset containing text and labeled PII entities is required for training a new model. For more information on model training and evaluation for Presidio, see the [Presidio-Research Github repository](https://github.com/data-privacy-stack/presidio-research). To train your own model, see this tutorial: [Train your own transformers model](https://huggingface.co/docs/transformers/training). ### Using a transformers model as an `EntityRecognizer` In addition to the approach described in this document, one can decide to integrate a transformers model as a recognizer. We allow these two options, as a user might want to have multiple NER models running in parallel. In this case, one can create multiple `EntityRecognizer` instances, each serving a different model, instead of one model used in an `NlpEngine`. [See this sample](../../samples/python/transformers_recognizer/index.md) for more info on integrating a transformers model as a Presidio recognizer and not as a Presidio `NLPEngine`. --- ### Samples Docker Aisix (`docs/samples/docker/aisix.md`) # Protect LLM requests with Presidio and AISIX AI Gateway [AISIX](https://github.com/api7/aisix) can call a self-hosted Presidio Analyzer and Anonymizer as an input guardrail before forwarding a request to an LLM provider. This sample configures two outcomes: - email addresses are replaced with an entity placeholder; - US Social Security numbers are blocked before the provider is called. ```text Application --> AISIX --> LLM provider | +--> Presidio Analyzer --> Presidio Anonymizer ``` ## Prerequisites - Docker - an OpenAI API key, or an OpenAI-compatible endpoint and matching AISIX provider configuration - `curl` The commands below use AISIX 1.2.0 and Presidio 2.2.364. ## Start Presidio Create a private Docker network and run both Presidio services: ```bash docker network create aisix-presidio docker run -d --name presidio-analyzer --network aisix-presidio \ -p 127.0.0.1:5002:3000 \ ghcr.io/data-privacy-stack/presidio-analyzer:2.2.364 docker run -d --name presidio-anonymizer --network aisix-presidio \ -p 127.0.0.1:5001:3000 \ ghcr.io/data-privacy-stack/presidio-anonymizer:2.2.364 ``` These services do not authenticate callers. Keep them on a private network in production instead of publishing their ports. Wait for both services to report healthy before continuing: ```bash until curl -fsS http://127.0.0.1:5002/health >/dev/null; do sleep 5; done until curl -fsS http://127.0.0.1:5001/health >/dev/null; do sleep 2; done ``` Confirm that the Analyzer detects an email address: ```bash curl -sS -X POST http://127.0.0.1:5002/analyze \ -H 'Content-Type: application/json' \ -d '{"text":"Contact alice@example.com","language":"en"}' ``` The response includes an `EMAIL_ADDRESS` result with its character offsets and confidence score. ## Configure AISIX Create `config.yaml`: ```yaml resources_file: /etc/aisix/resources.yaml proxy: addr: "0.0.0.0:3000" admin: enabled: false ``` Create `resources.yaml`: ```yaml _format_version: "1" provider_keys: - display_name: openai-main provider: openai api_key: ${OPENAI_API_KEY} models: - display_name: presidio-demo provider: openai model_name: gpt-4o-mini provider_key: openai-main api_keys: - display_name: local-demo key_env: CALLER_API_KEY allowed_models: ["presidio-demo"] guardrails: - name: presidio-pii-policy enabled: true hook_point: input enforcement_mode: block fail_open: false kind: presidio analyzer_url: http://presidio-analyzer:3000 anonymizer_url: http://presidio-anonymizer:3000 entities: - type: EMAIL_ADDRESS action: mask - type: US_SSN action: block default_action: mask operator: replace language: en guardrail_attachments: - guardrail_id: presidio-pii-policy scope_type: env priority: 100 ``` Export credentials and start the gateway on the same Docker network: ```bash export OPENAI_API_KEY="YOUR_PROVIDER_KEY" export CALLER_API_KEY="sk-local-presidio-demo" docker run -d --name aisix-presidio-gateway \ --platform linux/amd64 \ --network aisix-presidio \ -v "$(pwd)/config.yaml:/etc/aisix/config.yaml:ro" \ -v "$(pwd)/resources.yaml:/etc/aisix/resources.yaml:ro" \ -e OPENAI_API_KEY -e CALLER_API_KEY \ -p 127.0.0.1:3000:3000 \ ghcr.io/api7/aisix:1.2.0 ``` The guardrail must have an attachment. A configured guardrail with no matching attachment does not inspect traffic. ## Verify masking and blocking Send a request containing an email address: ```bash curl -sS -X POST http://127.0.0.1:3000/v1/chat/completions \ -H "Authorization: Bearer ${CALLER_API_KEY}" \ -H 'Content-Type: application/json' \ -d '{ "model": "presidio-demo", "messages": [ {"role": "user", "content": "Contact alice@example.com about the order"} ] }' ``` AISIX asks the Analyzer to locate the email and the Anonymizer to replace it. The provider receives `Contact about the order`; the model's response remains provider-dependent. Now send a value configured with the `block` action: ```bash curl -sSi -X POST http://127.0.0.1:3000/v1/chat/completions \ -H "Authorization: Bearer ${CALLER_API_KEY}" \ -H 'Content-Type: application/json' \ -d '{ "model": "presidio-demo", "messages": [ {"role": "user", "content": "My SSN is 987-65-4321"} ] }' ``` The request returns `422 Unprocessable Entity` with a `content_filter` error, and AISIX does not call the model provider. This example is fail-closed. If the Analyzer is unavailable, AISIX returns a `422` `guardrail_unavailable` error instead of forwarding uninspected content. Set `fail_open: true` only when availability is more important than preventing unscreened text from reaching the provider. ## Clean up ```bash docker rm -f aisix-presidio-gateway presidio-analyzer presidio-anonymizer docker network rm aisix-presidio ``` For entity selection, anonymization operators, output guardrails, streaming behavior, and deployment guidance, see the [AISIX Presidio guardrail documentation](https://docs.api7.ai/ai-gateway/traffic-controls/guardrails/presidio). --- ### Samples Docker Litellm (`docs/samples/docker/litellm.md`) # LiteLLM (OpenAI Proxy) with Presidio Run Presidio PII Masking across Anthropic/Gemini/Bedrock/etc. calls with [LiteLLM](https://github.com/BerriAI/litellm) [👉 **Refer to LiteLLM Docs for detailed guide**](https://docs.litellm.ai/docs/proxy/guardrails/pii_masking_v2) **Flow:** App <-> `LiteLLM Proxy + Presidio PII Masking` <-> LLM Provider ## Prerequisites - Run `pip install 'litellm[proxy]'` [Docs](https://docs.litellm.ai/docs/proxy/quick_start) - Setup [Presidio Docker](https://data-privacy-stack.github.io/presidio/installation/#using-docker) ## Quick Start ### Step 1. Add to env ```bash export PRESIDIO_ANALYZER_API_BASE="http://localhost:5002" export PRESIDIO_ANONYMIZER_API_BASE="http://localhost:5001" export OPENAI_API_KEY="sk-..." ``` ### Step 2. Set Presidio as a callback in config.yaml ```yaml model_list: - model_name: my-openai-model ### RECEIVED MODEL NAME ### litellm_params: # all params accepted by litellm.completion() - https://docs.litellm.ai/docs/completion/input model: gpt-3.5-turbo ### MODEL NAME sent to `litellm.completion()` ### litellm_settings: callbacks = ["presidio"] ``` ### Step 3. Start proxy ```bash litellm --config /path/to/config.yaml ``` This will mask the input going to the llm provider ## Output parsing LLM responses can sometimes contain the masked tokens. For presidio 'replace' operations, LiteLLM can check the LLM response and replace the masked token with the user-submitted values. Just set `litellm.output_parse_pii = True`, to enable this. ```yaml litellm_settings: output_parse_pii: true ``` **Expected Flow: ** 1. User Input: "hello world, my name is Jane Doe. My number is: 034453334" 2. LLM Input: "hello world, my name is [PERSON]. My number is: [PHONE_NUMBER]" 3. LLM Response: "Hey [PERSON], nice to meet you!" 4. User Response: "Hey Jane Doe, nice to meet you!" ## Ad-hoc recognizers Send ad-hoc recognizers to presidio `/analyze` by passing a json file to the proxy [**Example** ad-hoc recognizer](https://github.com/BerriAI/litellm/blob/b69b7503db5aa039a49b7ca96ae5b34db0d25a3d/litellm/proxy/hooks/example_presidio_ad_hoc_recognizer.json#L4) ```yaml litellm_settings: callbacks: ["presidio"] presidio_ad_hoc_recognizers: "./hooks/example_presidio_ad_hoc_recognizer.json" ``` You can see this working, when you run the proxy: ```bash litellm --config /path/to/config.yaml --debug ``` Make a chat completions request, example: ```bash { "model": "azure-gpt-3.5", "messages": [{"role": "user", "content": "John Smith AHV number is 756.3026.0705.92. Zip code: 1334023"}] } ``` And search for any log starting with `Presidio PII Masking`, example: ```bash Presidio PII Masking: Redacted pii message: AHV number is . Zip code: ``` ## Turn on/off per key LiteLLM lets you create [virtual keys](https://docs.litellm.ai/docs/proxy/virtual_keys) for calling the proxy. You can use these to control model access, set budgets, track usage, etc. Turn off PII masking for a given key. Do this by setting `permissions: {"pii": false}`, when generating a key. ```shell curl --location 'http://0.0.0.0:4000/key/generate' \ --header 'Authorization: Bearer sk-1234' \ --header 'Content-Type: application/json' \ --data '{ "permissions": {"pii": false} }' ``` ## Turn on/off per request The proxy supports 2 request-level PII controls: - *no-pii*: Optional(bool) - Allow user to turn off pii masking per request. - *output_parse_pii*: Optional(bool) - Allow user to turn off pii output parsing per request. [**Output Parsing**](#output-parsing) ### Usage **Step 1. Create key with pii permissions** Set `allow_pii_controls` to true for a given key. This will allow the user to set request-level PII controls. ```bash curl --location 'http://0.0.0.0:4000/key/generate' \ --header 'Authorization: Bearer my-master-key' \ --header 'Content-Type: application/json' \ --data '{ "permissions": {"allow_pii_controls": true} }' ``` **Step 2. Turn off pii output parsing** ```python import os from openai import OpenAI client = OpenAI( # This is the default and can be omitted api_key=os.environ.get("OPENAI_API_KEY"), base_url="http://0.0.0.0:4000" ) chat_completion = client.chat.completions.create( messages=[ { "role": "user", "content": "My name is Jane Doe, my number is 8382043839", } ], model="gpt-3.5-turbo", extra_body={ "content_safety": {"output_parse_pii": False} } ) ``` **Step 3: See response** ```bash { "id": "chatcmpl-8c5qbGTILZa1S4CK3b31yj5N40hFN", "choices": [ { "finish_reason": "stop", "index": 0, "message": { "content": "Hi [PERSON], what can I help you with?", "role": "assistant" } } ], "created": 1704089632, "model": "gpt-35-turbo", "object": "chat.completion", "system_fingerprint": null, "usage": { "completion_tokens": 47, "prompt_tokens": 12, "total_tokens": 59 }, "_response_ms": 1753.426 } ``` ## Turn on for logging only Only apply PII Masking before logging to Langfuse, etc. Not on the actual llm api request / response. This is currently only applied for - `/chat/completion` requests - on 'success' logging 1. Setup config.yaml ```yaml litellm_settings: presidio_logging_only: true model_list: - model_name: gpt-3.5-turbo litellm_params: model: gpt-3.5-turbo api_key: os.environ/OPENAI_API_KEY ``` 2. Start proxy ```bash litellm --config /path/to/config.yaml ``` 3. Test it! ```bash curl -X POST 'http://0.0.0.0:4000/chat/completions' \ -H 'Content-Type: application/json' \ -H 'Authorization: Bearer sk-1234' \ -D '{ "model": "gpt-3.5-turbo", "messages": [ { "role": "user", "content": "Hi, my name is Jane!" } ] }' ``` **Expected Logged Response** ``` Hi, my name is ! ``` --- ### Samples Fabric Env Setup (`docs/samples/fabric/env_setup.md`) ## Environment Setup for Presidio in Fabric The spaCy model can be downloaded from here: [English · spaCy Models Documentation](https://spacy.io/models/en#en_core_web_lg) ### 1. Requirements - **Fabric workspace** with sufficient permissions to create and manage custom environments. - **Lakehouse** access for uploading large models or data files. ### 2. Configure Spark Pool Make sure to create (or select) a valid Spark pool that you can attach to your Fabric environment. ### 3. Create a New Environment 1. In your Fabric workspace, go to **Settings** and select **New Environment**. 2. Provide a name (e.g., `presidio-env`) and choose the appropriate Python version. 3. Configure any required settings (e.g., pinned versions, advanced options). ### 4. Add Dependencies 1. Under **Public Library**, add the essential libraries: - `presidio-analyzer` - `presidio-anonymizer` - `spacy` For Microsoft Fabric compatibility, use the versions verified by this sample: - `presidio-analyzer==2.2.357` - `presidio-anonymizer==2.2.357` - `spacy==3.8.4` Newer Presidio package versions may introduce dependency conflicts with Fabric notebook utilities such as `mssparkutils`. 2. For **smaller SpaCy models** (like `en_core_web_md` < 300MB), you can include them directly in this environment. ### 5. Upload a Large SpaCy Model If you want to use `en_core_web_lg` (which typically exceeds 300MB): 1. Upload the `.whl` file to your Lakehouse (or any location accessible by Spark). 2. You will install it **within the notebook** rather than from this environment. ### 6. Compute Configure your compute, make sure to use the pool configured before ### 7. Review & Save - Confirm your chosen libraries appear under the **Custom Library** or **Public Library** tabs. - Click **Save** to finalize your environment setup. ### 8. Run the Sample Notebook 1. Open the [`presidio_and_spark.ipynb`](./artifacts/presidio_and_spark.ipynb) notebook. 2. When opening your notebook, ensure you pick the **custom environment** you created. 3. Confirm you have selected the valid **Spark pool** you configured earlier. 4. Within the notebook, if you're using a large SpaCy model, install it using: ```python # Please update the path to your model path. %pip install /lakehouse/default/Files/presidio/models/en_core_web_lg-3.8.0-py3-none-any.whl --- ### Samples Python Gliner (`docs/samples/python/gliner.md`) # Using GLiNER within Presidio ## What is GLiNER GLiNER is a Named Entity Recognition (NER) model capable of identifying any entity type using a bidirectional transformer encoder (BERT-like). It provides a practical alternative to traditional NER models, which are limited to predefined entities, and Large Language Models (LLMs) that, despite their flexibility, are costly and large for resource-constrained scenarios. Paper: [GLiNER: Generalist Model for Named Entity Recognition using Bidirectional Transformer](https://arxiv.org/abs/2311.08526) Since GLiNER takes as input both the sentence/text and entity types, it can be used for zero-shot named entity recognition. This means that it can recognize entities that were not seen during training. ## PII Detection with GLiNER GLiNER has a trained PII detection model: 🔍 [`urchade/gliner_multi_pii-v1`](https://huggingface.co/urchade/gliner_multi_pii-v1) *(Apache 2.0)* This model is capable of recognizing various types of *personally identifiable information* (PII), including but not limited to these entity types: `person`, `organization`, `phone number`, `address`, `passport number`, `email`, `credit card number`, `social security number`, `health insurance id number`, `date of birth`, `mobile phone number`, `bank account number`, `medication`, `cpf`, `driver's license number`, `tax identification number`, `medical condition`, `identity card number`, `national id number`, `ip address`, `email address`, `iban`, `credit card expiration date`, `username`, `health insurance number`, `registration number`, `student id number`, `insurance number`, `flight number`, `landline phone number`, `blood type`, `cvv`, `reservation number`, `digital signature`, `social media handle`, `license plate number`, `cnpj`, `postal code`, `passport_number`, `serial number`, `vehicle registration number`, `credit card brand`, `fax number`, `visa number`, `insurance company`, `identity document number`, `transaction number`, `national health insurance number`, `cvc`, `birth certificate number`, `train ticket number`, `passport expiration date`, and `social_security_number`. ## Using GLiNER with Presidio Presidio has a built-in `EntityRecognizer` for GLiNER: `GLiNERRecognizer`. This recognizer can be used to detect PII entities in text using the GLiNER model. ### Installation To use GLiNER with Presidio, you need to install the `presidio-analyzer` with the `gliner` extra: ```bash pip install 'presidio-analyzer[gliner]' ``` ### Example ```python from presidio_analyzer import AnalyzerEngine from presidio_analyzer.nlp_engine import NlpEngineProvider from presidio_analyzer.predefined_recognizers import GLiNERRecognizer # Load a small spaCy model as we don't need spaCy's NER nlp_engine = NlpEngineProvider( nlp_configuration={ "nlp_engine_name": "spacy", "models": [{"lang_code": "en", "model_name": "en_core_web_sm"}], } ) # Create an analyzer engine analyzer_engine = AnalyzerEngine(nlp_engine=nlp_engine.create_engine()) # Define and create the GLiNER recognizer entity_mapping = { "person": "PERSON", "name": "PERSON", "organization": "ORGANIZATION", "location": "LOCATION" } gliner_recognizer = GLiNERRecognizer( model_name="urchade/gliner_multi_pii-v1", entity_mapping=entity_mapping, flat_ner=False, multi_label=True, map_location="cpu", ) # Add the GLiNER recognizer to the registry analyzer_engine.registry.add_recognizer(gliner_recognizer) # Remove the spaCy recognizer to avoid NER coming from spaCy analyzer_engine.registry.remove_recognizer("SpacyRecognizer") # Analyze text results = analyzer_engine.analyze( text="Hello, my name is Rafi Mor, I'm from Binyamina and I work at Microsoft. ", language="en" ) print(results) ``` ## Text Chunking By default, GLiNERRecognizer splits long texts into character-based chunks (250 chars, 50 overlap). You can customize this via `text_chunker`: **From Python:** ```python from presidio_analyzer.chunkers import CharacterBasedTextChunker gliner_recognizer = GLiNERRecognizer( model_name="urchade/gliner_multi_pii-v1", entity_mapping=entity_mapping, text_chunker=CharacterBasedTextChunker(chunk_size=400, chunk_overlap=60), ) ``` **From YAML (using tokenizer-based chunking):** ```yaml - name: GLiNERRecognizer type: predefined model_name: urchade/gliner_multi_pii-v1 text_chunker: chunker_type: tokenizer overlap_tokens: 32 ``` The `tokenizer` chunker uses the model's own tokenizer (resolved automatically at load time) to split text by token count, respecting the model's token limit instead of approximating with character counts. `max_tokens` is omitted above so it is auto-derived from the model's tokenizer and reduced to reserve room for special tokens (e.g. `[CLS]`/`[SEP]`). Set it explicitly only if you account for those special tokens yourself, otherwise chunks may overflow the model's real input limit and be truncated. ## ONNX Runtime Support GLiNERRecognizer supports using ONNX Runtime as a backend, which provides better CPU compatibility and can prevent crashes on older CPUs without AVX2 instruction set support (e.g., Intel Sandy Bridge). ### Using ONNX Runtime Backend To use ONNX Runtime with GLiNER: ```python from presidio_analyzer.predefined_recognizers import GLiNERRecognizer # Enable ONNX Runtime backend gliner_recognizer = GLiNERRecognizer( model_name="urchade/gliner_multi_pii-v1", entity_mapping=entity_mapping, load_onnx_model=True, # Enable ONNX Runtime map_location="cpu", ) ``` **Benefits of using ONNX Runtime:** - Works on older CPUs without AVX2 support - Prevents "Illegal instruction (core dumped)" crashes on incompatible hardware - Can provide better performance on certain CPU architectures **Note:** Make sure `onnxruntime` is installed when using this feature. It's included in the `gliner` extra dependencies. --- ### Samples Deployments Data Factory Presidio Data Factory Template Gallery Databricks (`docs/samples/deployments/data-factory/presidio-data-factory-template-gallery-databricks.md`) # Anonymize PII entities in datasets using Azure Data Factory template and Presidio on Databricks This sample uses the built in [data anonymization template](https://github.com/Azure/Azure-DataFactory/tree/main/templates/Data%20Anonymization%20with%20Presidio%20on%20Databricks) of Azure Data Factory (which is a part of the Template Gallery) to copy a csv dataset from one location to another, while anonymizing PII data from a text column in the dataset. It leverages the code for using [Presidio on Azure Databricks](../spark/index.md) to call Presidio as a Databricks notebook job in the Azure Data Factory (ADF) pipeline to transform the input dataset before merging the results to an Azure Blob Storage. **Note that** this solution is capable of transforming large datasets. For smaller, text based input you may want to work with the [Data Anonymization with Presidio as an HTTP service](./presidio-data-factory-template-gallery-http.md) template which offers an easier deployment for Presidio. The sample deploys the following Azure Services: * Azure Storage - The target storage account where data will be persisted. * Azure Databricks - Host presidio to anonymize the data. Additionally you should already have an instance of Azure Data Factory which hosts and orchestrates the transformation pipeline and a storage account which holds the source files. ## About this Solution Template This template gets the files from your source file-based store. It then anonymizes the content and uploads each of them to the destination store. The template contains three activities: * **AnonymizeSource** runs the presidio notebook on the input file to create an output folder with csv parts. * **MergeAnonymizedToTarget** merges the csv parts from databricks output folder to a single csv on the target storage container. * **DeleteAnonymized** deletes the temporary output folder. The template defines four parameters: * **SourceStore_Location** is the container name of your source store where you want to move files from (STORAGE_CONTAINER_NAME). * **DestinationStore_Name** is the container name in the target storage account which is provisioned by the ARM template. * **SourceFile_Name** is the name of the input file. * **TextColumn_Name** is the name of the column to be anonymized. ## How to use this Solution Template To use this template you should first setup the required infrastructure for the sample to run, then setup the template in Azure Data Factory. ### Setup Presidio Provision and setup the databricks cluster by following the Deploy and Setup steps in [presidio-spark sample](../spark/index.md#Pre-requisites). Take a note of the authentication token and do not follow the "Running the sample" steps. ### Setup Azure Data Factory 1. Go to the Data anonymization with Presidio on Databricks template. Select the AnonymizedCSV connection (Azure Storage) and select "New" from the drop down menu. 2. Name the service "PresidioStorage" and select the storage account that was created in the previous steps from your subscription. Note that Target source was also selected as the sample uses the same storage account for both source and target. 3. Select the Anonymize Source connection (Databricks) and select "New" from the drop down menu. 4. Name the service "PresidioDatabricks" and select the Azure Databricks workspace that was created in the previous steps from your subscription. Follow through the steps to input the authentication token which was generated in the [previous step](../spark/index.md#Setup-Databricks), or create a new one by following [this guide](https://docs.microsoft.com/en-us/azure/databricks/dev-tools/api/latest/authentication). Select presidio_cluster to run the job. 5. Select Use this template tab 6. You'll see the pipeline, as in the following example: 7. Upload a csv file to the storage container and select Debug, enter the Parameters, and then select Finish. The parameters are the container where you want to move files from, the container name where you want to move the anonymized files to, the csv file name and the name of a text column in the csv file. 8. Review the result. --- ### Samples Deployments Data Factory Presidio Data Factory Template Gallery Http (`docs/samples/deployments/data-factory/presidio-data-factory-template-gallery-http.md`) # Anonymize PII entities in text using Azure Data Factory template and Presidio as an HTTP service This sample uses the built in [data anonymization template](https://github.com/Azure/Azure-DataFactory/tree/main/templates/Data%20Anonymization%20with%20Presidio%20as%20an%20HTTP%20service) of Azure Data Factory which is a part of the Template Gallery to move a set of text files from one location to another while anonymizing their content. It leverages the code for using [Presidio on Azure App Service](../app-service/index.md) to call Presidio as an HTTP REST endpoint in the Azure Data Factory (ADF) pipeline while parsing and storing each file as an Azure Blob Storage. **Note that** given the solution architecture which call presidio services using HTTP, this sample should be used for up to 5000 files, each up to 200KB in size. The restrictions are based on ADF lookup-activity which is used to iterate the files in the storage container (up to 5000 records), and having Presidio as an HTTP endpoint with text being sent over network to be anonymized. For larger sets please work with the [Data Anonymization with Presidio on Databricks](./presidio-data-factory-template-gallery-databricks.md) template. The sample deploys the following Azure Services: * Azure KeyVault - Holds the access keys for Azure Storage to avoid having keys and secrets in the code. * Azure Storage - The target storage account where data will be persisted. * Azure App Service - Host presidio to anonymize the data. Additionally you should already have an instance of Azure Data Factory which host and orchestrate the transformation pipeline and a storage account which holds the source files. ## About this Solution Template This template gets the files from your source file-based store. It then anonymizes the content and uploads each of them to the destination store. The template contains eight activities: * **GetMetadata** gets the list of objects including the files and subfolders from your folder on source store. It will not retrieve the objects recursively. * **Filter** filter the objects list from GetMetadata activity to select the files only. * **GetSASToken** gets the target storage account SAS token from the Azure Key Vault. * **ForEach** gets the file list from the Filter activity and then iterates over the list and passes each file to the Anonymization activities. * **LoadFileContent** loads the content of a text file to an ADF Lookup. * **PresidioAnalyze** sends the text content to Presidio Analyzer. * **PresidioAnonymize** sends the text and the analysis response to Presidio Anonymizer to get the anonymized text. * **UploadBlob** uses Azure Blob Storage REST API to upload the anonymized text to the target container. The template defines four parameters: * **SourceStore_Location** is the container name of your source store where you want to move files from. * **DestinationStore_Name** is the name of the target storage account which is provisioned by the ARM template. * **DestinationStore_Location** is the container name of your destination store where you want to move files to. it has a default value of a container which was created during provisioning of the ARM template (presidio). * **KeyVault_Name** is the name of the Azure Key Vault which is provisioned by the ARM template. * **Analyzer_Url** is the URL for the Analyzer App Service which is provisioned by the ARM template. * **Anonymizer_Url** is the URL for the Anonymizer App Service which is provisioned by the ARM template. ## How to use this Solution Template To use this template you should first setup the required infrastructure for the sample to run, then setup the template in Azure Data Factory. ### Setup Presidio Create the Azure App Service, the storage accounts and an Azure Key Vault by clicking the Deploy-to-Azure button, or by running the following script to provision the [provided ARM template](./arm-templates/azure-deploy-adf-template-gallery-http.json). [](https://portal.azure.com/#create/Microsoft.Template/uri/https%3A%2F%2Fraw.githubusercontent.com%2Fdata-privacy-stack%2Fpresidio%2Fmain%2Fdocs%2Fsamples%2Fdeployments%2Fdata-factory%2Farm-templates%2Fazure-deploy-adf-template-gallery-http.json) ```bash RESOURCE_GROUP=[Name of resource group] LOCATION=[location of resources] az group create --name $RESOURCE_GROUP --location $LOCATION az deployment group create -g $RESOURCE_GROUP --template-file ./arm-templates/azure-deploy-adf-template-gallery-http.json ``` Note that: * A SAS token keys is created and read from Azure Storage and then imported to Azure Key Vault. Using ARM template built in [functions](https://docs.microsoft.com/en-us/azure/azure-resource-manager/templates/template-functions): [listAccountSas](https://docs.microsoft.com/en-us/rest/api/storagerp/storageaccounts/listaccountsas). This token is time limited. * An access policy grants the Azure Data Factory managed identity access to the Azure Key Vault. You should provide your ADF client principal ID by following [this guide](https://docs.microsoft.com/en-us/azure/data-factory/data-factory-service-identity). ### Setup Azure Data Factory 1. Go to the Data anonymization with Presidio as an HTTP service template. Select existing connection or create a New connection to your source file store where you want to move files from. Be aware that DataSource_Folder and DataSource_File are reference to the same connection of your source file store. 2. Select Use this template tab 3. You'll see the pipeline, as in the following example: 4. Select Debug, enter the Parameters, and then select Finish. The parameters are the container where you want to move files from and the container path where you want to move the anonymized files to. 5. Review the result. --- ### Samples Deployments Data Factory Presidio Data Factory (`docs/samples/deployments/data-factory/presidio-data-factory.md`) # Anonymize PII entities in an Azure Data Factory ETL Pipeline The following samples showcase two scenarios which use Azure Data Factory (ADF) to move a set of JSON objects from an online location to an Azure Storage while anonymizing their content. The first sample leverages the code for using [Presidio on Azure App Service](../app-service/index.md) to call Presidio as an HTTP REST endpoint in the ADF pipeline while parsing and storing each file as an Azure Blob Storage. The second sample leverage the code for using [Presidio on spark](../spark/index.md) to run over a set of files on an Azure Blob Storage to anonymize their content, in the case of having a large data set that requires the scale of databricks. The samples deploy and use the following Azure Services: * Azure Data Factory - Host and orchestrate the transformation pipeline. * Azure KeyVault - Holds the access keys for Azure Storage to avoid having keys and secrets in the code. * Azure Storage - Persistence layer of this sample. * Azure Databricks/ Azure App Service - Host presidio to anonymize the data. The input file used by the samples is hosted on [presidio-research](https://github.com/data-privacy-stack/presidio-research/) repository. It is setup as a variable on the provided ARM template and used by Azure Data Factory as the input source. ## Option 1: Presidio as an HTTP REST endpoint By using Presidio as an HTTP endpoint, the user can select which infrastructure best suits their requirements. in this sample, Presidio is deployed to an Azure App Service, but other deployment targets can be used, such as [kubernetes](../k8s/index.md). ### Deploy the ARM template Create the Azure App Service and the ADF pipeline by clicking the Deploy-to-Azure button, or by running the following script to provision the [provided ARM template](./arm-templates/azure-deploy-adf-app-service.json). [](https://portal.azure.com/#create/Microsoft.Template/uri/https%3A%2F%2Fraw.githubusercontent.com%2Fdata-privacy-stack%2Fpresidio%2Fmain%2Fdocs%2Fsamples%2Fdeployments%2Fdata-factory%2Farm-templates%2Fazure-deploy-adf-app-service.json) ```bash RESOURCE_GROUP=[Name of resource group] LOCATION=[location of resources] az group create --name $RESOURCE_GROUP --location $LOCATION az deployment group create -g $RESOURCE_GROUP --template-file ./arm-templates/azure-deploy-adf-app-service.json ``` Note that: * A SAS token keys is created and read from Azure Storage and then imported to Azure Key Vault. Using ARM template built in [functions](https://docs.microsoft.com/en-us/azure/azure-resource-manager/templates/template-functions): [listAccountSas](https://docs.microsoft.com/en-us/rest/api/storagerp/storageaccounts/listaccountsas). * An access policy grants the Azure Data Factory managed identity access to the Azure Key Vault by using ARM template [reference](https://docs.microsoft.com/en-us/azure/azure-resource-manager/templates/template-functions-resource?tabs=json#reference) function to the Data Factory object and acquire its identity.principalId property. This is enabled by setting the data factory ARM resource's identity attribute to managed identity (SystemAssigned). ### About this Solution Template This template gets a collection of JSON documents from a file on GitHub. It then extracts one of the text fields of the document, anonymizes the content and uploads it as a text file to the destination store. The template contains seven activities: * **GetDataSet**- Copy the dataset from GitHub to the first folder on the Azure Storage blob container (/dataset). * **LoadSet**- Loads the dataset into the Azure Data Factory memory for processing in a for-each loop. * **GetSASToken** - Get the SAS token from Azure Key Vault. This will be used later for writing to the blob container. * **SaveBlobs** - Is a For-Each loop activity. It includes a clause which is executed for each document in the array. * **PresidioAnalyze** - Sends the text to presidio analyzer endpoint. * **PresidioAnonymize** - Sends the response from presidio analyzer to presidio anonymizer endpoint. * **UploadBlob** - Saves the anonymized response from presidio to a randomly named text file on the target Azure Blob Storage. ## Option 2: Presidio on Azure Databricks By using Presidio as a Notebook step in ADF, we allow Databricks to scale presidio according to the cluster capabilities and the input dataset. Using presidio as a native python package in pyspark can unlock more analysis and de-identification scenarios. ### Pre-requisite - Deploy Azure Databricks Provision and setup the databricks cluster by following the steps in [presidio-spark sample](../spark/index.md#Deploy-Infrastructure). Note the output key and export it as DATABRICKS_TOKEN environment variable. ### Deploy the ARM template Create the rest of the services by running the following script which uses the [provided ARM template](./arm-templates/azure-deploy-adf-databricks.json). ```bash RESOURCE_GROUP=[Name of resource group] LOCATION=[location of resources] DATABRICKS_HOST=https://$DATABRICKS_WORKSPACE_URL DATABRICKS_CLUSTER_ID=$(databricks clusters get --cluster-name presidio_cluster | jq -r .cluster_id) DATABRICKS_NOTEBOOK_LOCATION="/notebooks/01_transform_presidio" az deployment group create -g $RESOURCE_GROUP --template-file ./arm-templates/azure-deploy-adf-databricks.json --parameters Databricks_accessToken=$DATABRICKS_TOKEN Databricks_clusterId=$DATABRICKS_CLUSTER_ID Databricks_notebookLocation=$DATABRICKS_NOTEBOOK_LOCATION Databricks_workSpaceUrl=$DATABRICKS_HOST AzureBlobStorage_accountName=$STORAGE_ACCOUNT_NAME AzureBlobStorage_cotainerName=$STORAGE_CONTAINER_NAME ``` Note that: Two keys are read from Azure Storage and imported to Azure Key Vault, the account Access Token and a SAS token, using ARM template built in [functions](https://docs.microsoft.com/en-us/azure/azure-resource-manager/templates/template-functions): [listAccountSas](https://docs.microsoft.com/en-us/rest/api/storagerp/storageaccounts/listaccountsas) and [listKeys](https://docs.microsoft.com/en-us/rest/api/storagerp/storageaccounts/listkeys). ### About this Solution Template This template gets a collection of JSON documents from a file on GitHub. It then extracts one of the text fields of the document and saves it to a text file on a temporary folder in the storage account (un-anonymized content). It then runs a spark notebook job that anonymizes the content of the files in that folder and saves the result as csv files on the destination store. The template contains seven activities: * **GetDataSet** - Copy the dataset from GitHub to the first folder on the Azure Storage blob container (/dataset). * **GetSASToken** - Get the SAS token from Azure Key Vault. This will be used later for writing to the blob container. * **LoadSet** -  Loads the dataset into the Azure Data Factory memory for processing in a for-each loop. * **SaveBlobs** - Is a For-Each loop activity. It includes a clause which is executed for each document in the array. * **UploadBlob** - Saves the text file on a temporary container on the target Azure Blob Storage * **GetSecret** - Get the storage account secret from Azure Key Vault. This will be used later for accessing the blob container from databricks * **Presidio-Anonymize** - Is a databricks spark job which runs presidio on the temporary storage container. the result of this job is a new container (/output) with csv files that contain the anonymized text. --- ### Samples Python Streamlit Demo Text (`docs/samples/python/streamlit/demo_text.txt`) Here are a few example sentences we currently support: Hello, my name is David Johnson and I live in Maine. My credit card number is 4095-2609-9393-4932 and my crypto wallet id is 16Yeky6GMjeNkAiNcBY7ZhrLoMSgg1BoyZ. On September 18 I visited microsoft.com and sent an email to test@presidio.site, from the IP 192.168.0.1. My passport: 191280342 and my phone number: (212) 555-1234. This is a valid International Bank Account Number: IL150120690000003111111 . Can you please check the status on bank account 954567876544? Kate's social security number is 078-05-1126. Her driver license? it is 1234567A. --- ### Samples Python Streamlit Requirements (`docs/samples/python/streamlit/requirements.txt`) presidio-analyzer[transformers] presidio-analyzer[stanza] presidio-anonymizer streamlit>=1.37.0 streamlit-tags pandas>=1.0.4 python-dotenv st-annotated-text torch>=2.8.0 flair>=0.15.0 openai azure-ai-textanalytics --- ### Samples Deployments Redacting Telemetry App Requirements (`docs/samples/deployments/redacting-telemetry/app/requirements.txt`) fastapi>=0.129.0 uvicorn>=0.30.0 requests>=2.31.0 opentelemetry-api>=1.21.0 opentelemetry-sdk>=1.21.0 opentelemetry-exporter-otlp-proto-grpc>=1.21.0 --- ### Samples Deployments Openai Anonymaztion And Deanonymaztion Best Practices Docs Sample For Presidio Pr Anonymization Toolkit Sample (`docs/samples/deployments/openai-anonymaztion-and-deanonymaztion-best-practices/docs/sample_for_presidio_pr/anonymization_toolkit_sample.md`) # Anonymization Toolkit Sample This sample introduces a toolkit designed to: - Anonymize sensitive data before sending it to large language models (LLMs) - De-anonymize the received data while maintaining confidentiality. The toolkit is particularly useful in "Chat with your data" scenarios within enterprises, ensuring that personally identifiable information (PII) present in corporate documents is not exposed to users during queries. The following sections will describe the required infrastructure and code to achieve the above functionality. ## Target Audience The toolkit serves enterprises that: Require PII to remain internal while leveraging external OpenAI APIs, and Aim to safeguard PII from being displayed post-OpenAI processing. ## Workflow Example The following is the desired output. We can input PII data anonymized to the LLM input and return the replaced entities back to the original in the response back to the user. - User Input: "Hello world, my name is Jane Doe. My number is: 034453334" - Anonymized LLM Input: "Hello world, my name is [PERSON]. My number is: [PHONE_NUMBER]" - LLM Response: "Hey [PERSON], nice to meet you!" - De-anonymized User Response: "Hey Jane Doe, nice to meet you!" ## Infrastructure The solution consists of the following components: - Api - Client - Redis TODO complete this section ## Code ### Anonymization First we need to setup our `Presidio` engines to aid with analysis, anonymization and deanonymization. The following code can exist as part of a service available to the appropriate api endpoints. ```python configuration = { "nlp_engine_name": "spacy", "models": [ {"lang_code": "en", "model_name": "en_core_web_lg"}, {"lang_code": "nl", "model_name": "nl_core_news_sm"}, {"lang_code": "es", "model_name": "es_core_news_sm"}, ], } provider = NlpEngineProvider(nlp_configuration=configuration) nlp_engine = provider.create_engine() self.analyzer = AnalyzerEngine( nlp_engine=nlp_engine, supported_languages=["en", "nl", "es"] ) self.anonymizer = AnonymizerEngine() self.anonymizer.add_anonymizer(InstanceCounterAnonymizer) self.deanonymizer = DeanonymizeEngine() self.deanonymizer.add_deanonymizer(InstanceCounterDeanonymizer) ``` `InstanceCounterAnonymizer` and `InstanceCounterDeanonymizer` taken from [these examples](https://data-privacy-stack.github.io/presidio/samples/python/pseudonymization/). We anonymize data with the following: ``` /* Detailed source-code truncated for AI context efficiency. */ ``` and deanonymize with: ``` /* Detailed source-code truncated for AI context efficiency. */ ``` The above code would be called with: ```python # anonymize example anonymized_text, new_entity_mappings = presidio_service.anonymize_text( request.session_id, request.text, request.language, entity_mappings ) # deanonymize example deanonymized_text = presidio_service.deanonymize_text( request.session_id, request.text, entity_mappings ) ``` Where: - `request.session_id` - A unique session identifier, e.g. guid. - `request.text` - The text to be anonymized or deanonymized. - `request.language` - The language code to analyze with. Note this requires the appropriate language model to be setup during the configuration. See code above. Example codes, `en`, `es`, `nl`. - `entity_mappings` - A dictionary with the structure of: ```python {'LOCATION': {'Redmond': ''}, 'PERSON': {'Joe Bloggs': ''}} ``` - Each item is the entity type and the value contains each instance of the entity and its replacement. - This dictionary is optional. On the first request of a session we wont have any mappings, subsequent requests can then build upon this dictionary to maintain knowledge of previous request mappings. --- ### Samples Deployments Openai Anonymaztion And Deanonymaztion Best Practices Src Api Requirements (`docs/samples/deployments/openai-anonymaztion-and-deanonymaztion-best-practices/src/api/requirements.txt`) fastapi[standard]==0.129.0 uvicorn[standard]==0.30.6 presidio-analyzer==2.2.355 presidio-anonymizer==2.2.355 pydantic==2.9.2 redis==5.0.8 --- ### Samples Deployments Openai Anonymaztion And Deanonymaztion Best Practices Src Client App Requirements (`docs/samples/deployments/openai-anonymaztion-and-deanonymaztion-best-practices/src/client_app/requirements.txt`) ConfigArgParse==1.7 openai==1.37.1 python-dotenv==1.2.2 requests>=2.32.4 textual[syntax]==0.79.1 textual-serve==1.1.1 --- ### Samples Deployments K8s Charts Presidio Templates NOTES (`docs/samples/deployments/k8s/charts/presidio/templates/NOTES.txt`) presidio is now installed! To find out about your newly configured system, run: $ helm status {{ .Release.Name }} To test your new Presidio service, forward the ports: the Analyzer publishes a service at port 8080. To use the Analyzer, you first need to forward this port to your local IP address: $ kubectl port-forward -n presidio $(kubectl get pod -n presidio -l app=demo-presidio-analyzer -o jsonpath='{.items[0].metadata.name}') 8080:8080 --- --- METRICS --- - Files Extracted: 91 - Estimated Token Budget: ~97221 tokens - Canonical Reference: https://codewiki.google/github.com/microsoft/presidio