promptflow Documentation

Technical Documentation: microsoft/promptflow

ℹ️ Provenance: Multi-Tier Ingestion · CodeWiki Reference

Overview (README.md) Copy Section

Prompt flow

[!WARNING]
Prompt flow in Microsoft Foundry and Azure Machine Learning will be retired on April 20, 2027. Prompt flow is no longer recommended for new development. Migrate existing Prompt flow applications and deployments to Microsoft Agent Framework before April 20, 2027.

Prompt flow container images are no longer receiving updates, including security and package updates. This applies to Prompt flow runtime images, including promptflow-runtime, promptflow-runtime-stable, and promptflow-python.

After April 20, 2027, Prompt flow, including the web authoring experience in Microsoft Foundry and Azure Machine Learning, the VS Code extensions, and related Prompt flow container images, will no longer be supported or available.

If your application depends on Prompt flow deployments or runtime images, plan to move those workloads to supported alternatives such as Microsoft Agent Framework before the retirement date. For migration guidance, see the Prompt flow migration guide and migration code samples.

Welcome to join us to make prompt flow better by
participating discussions,
opening issues,
submitting PRs.

Prompt flow is a suite of development tools designed to streamline the end-to-end development cycle of LLM-based AI applications, from ideation, prototyping, testing, evaluation to production deployment and monitoring. It makes prompt engineering much easier and enables you to build LLM apps with production quality.

With prompt flow, you will be able to:

• Create and iteratively develop flow
  • Create executable flows that link LLMs, prompts, Python code and other tools together.
  • Debug and iterate your flows, especially tracing interaction with LLMs with ease.
• Evaluate flow quality and performance
  • Evaluate your flow's quality and performance with larger datasets.
  • Integrate the testing and evaluation into your CI/CD system to ensure quality of your flow.
• Streamlined development cycle for production
  • Deploy your flow to the serving platform you choose or integrate into your app's code base easily.
  • (Optional but highly recommended) Collaborate with your team by leveraging the cloud version of Prompt flow in Azure AI.

Installation Copy Chapter

To get started quickly, you can use a pre-built development environment. Click the button below to open the repo in GitHub Codespaces, and then continue the readme!

If you want to get started in your local environment, first install the packages:

Ensure you have a python environment, python>=3.9, <=3.11 is recommended.

pip install promptflow promptflow-tools

Quick Start ⚡ Copy Chapter

Create a chatbot with prompt flow

Run the command to initiate a prompt flow from a chat template, it creates folder named my_chatbot and generates required files within it:

pf flow init --flow ./my_chatbot --type chat

Setup a connection for your API key

For OpenAI key, establish a connection by running the command, using the openai.yaml file in the my_chatbot folder, which stores your OpenAI key (override keys and name with --set to avoid yaml file changes):

pf connection create --file ./my_chatbot/openai.yaml --set api_key=<your_api_key> --name open_ai_connection

For Azure OpenAI key, establish the connection by running the command, using the azure_openai.yaml file:

pf connection create --file ./my_chatbot/azure_openai.yaml --set api_key=<your_api_key> api_base=<your_api_base> --name open_ai_connection

Chat with your flow

In the my_chatbot folder, there's a flow.dag.yaml file that outlines the flow, including inputs/outputs, nodes, connection, and the LLM model, etc

Note that in the chat node, we're using a connection named open_ai_connection (specified in connection field) and the gpt-35-turbo model (specified in deployment_name field). The deployment_name filed is to specify the OpenAI model, or the Azure OpenAI deployment resource.

Interact with your chatbot by running: (press Ctrl + C to end the session)

pf flow test --flow ./my_chatbot --interactive

Core value: ensuring "High Quality” from prototype to production

Explore our 15-minute tutorial that guides you through prompt tuning ➡ batch testing ➡ evaluation, all designed to ensure high quality ready for production.

Next Step! Continue with the Tutorial 👇 section to delve deeper into prompt flow.

Tutorial 🏃‍♂️ Copy Chapter

Prompt flow is a tool designed to build high quality LLM apps, the development process in prompt flow follows these steps: develop a flow, improve the flow quality, deploy the flow to production.

Develop your own LLM apps Copy Section

VS Code Extension

We also offer a VS Code extension (a flow designer) for an interactive flow development experience with UI.

You can install it from the visualstudio marketplace.

Deep delve into flow development

Getting started with prompt flow: A step by step guidance to invoke your first flow run.

Learn from use cases Copy Section

Tutorial: Chat with PDF: An end-to-end tutorial on how to build a high quality chat application with prompt flow, including flow development and evaluation with metrics.

More examples can be found here. We welcome contributions of new use cases!

Setup for contributors Copy Section

If you're interested in contributing, please start with our dev setup guide: dev_setup.md.

Next Step! Continue with the Contributing 👇 section to contribute to prompt flow.

Contributing Copy Chapter

This project welcomes contributions and suggestions. Most contributions require you to agree to a
Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us
the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.

When you submit a pull request, a CLA bot will automatically determine whether you need to provide
a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions
provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct.
For more information see the Code of Conduct FAQ or
contact [email protected] with any additional questions or comments.

Trademarks Copy Chapter

This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft
trademarks or logos is subject to and must follow
Microsoft's Trademark & Brand Guidelines.
Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship.
Any use of third-party trademarks or logos are subject to those third-party's policies.

Code of Conduct Copy Chapter

This project has adopted the
Microsoft Open Source Code of Conduct.
For more information see the
Code of Conduct FAQ
or contact [email protected]
with any additional questions or comments.

Data Collection Copy Chapter

The software may collect information about you and your use of the software and
send it to Microsoft if configured to enable telemetry.
Microsoft may use this information to provide services and improve our products and services.
You may turn on the telemetry as described in the repository.
There are also some features in the software that may enable you and Microsoft
to collect data from users of your applications. If you use these features, you
must comply with applicable law, including providing appropriate notices to
users of your applications together with a copy of Microsoft's privacy
statement. Our privacy statement is located at
https://go.microsoft.com/fwlink/?LinkID=824704. You can learn more about data
collection and use in the help documentation and our privacy statement. Your
use of the software operates as your consent to these practices.

Telemetry Configuration Copy Section

Telemetry collection is on by default.

To opt out, please run pf config set telemetry.enabled=false to turn it off.

License Copy Chapter

Copyright (c) Microsoft Corporation. All rights reserved.

Licensed under the MIT license.

README (`docs/README.md`) Copy Section

Promptflow documentation contribute guidelines

This folder contains the source code for prompt flow documentation site.

This readme file will not be included in above doc site. It keeps a guide for promptflow documentation contributors.

Content Copy Chapter

Below is a table of important doc pages.

Writing tips Copy Chapter

0. Reach the doc source repository by clicking Edit this page on any page.

1. Please use :::{admonition} for experimental feature or notes, and admonition with dropdown for the Limitation Part.

2. Please use ::::{tab-set} to group your sdk/cli example, and put the cli at first. Use :sync: to sync multiple tables .

3. If you are unclear with the above lines, refer to get started to see the usage.

4. Add gif: Use ScreenToGif to record your screen, edit and save as a gif.

5. Reach more element style at Sphinx Design Components.

Preview your changes Copy Chapter

Local build: We suggest using local build at the beginning, as it's fast and efficiency.

Please refer to How to build doc site locally.

FAQ Copy Chapter

Adding image in doc Copy Section

Please use markdown syntax `` to reference image, because the relative path of image will be changed after sphinx build, and image placed in html tags can not be referenced when build.

Draw flow chart in doc Copy Section

We recommend using the mermaid, learn more from the mermaid syntax doc

• Recommend to install vscode extension to preview graph in vscode.

Reference Copy Chapter

• md-and-rst
• sphinx-quickstart

Index (`docs/index.md`) Copy Section

myst:
html_meta:
"description lang=en": "Prompt flow Doc"
"google-site-verification": "rEZN-2h5TVqEco07aaMpqNcDx4bjr2czx1Hwfoxydrg"

html_theme.sidebar_secondary.remove: true Copy Chapter

:::{admonition} Prompt Flow Retirement Notice
:class: warning
Prompt Flow feature development ended on April 20, 2026. The feature will be fully retired on April 20, 2027. On the retirement date, Prompt Flow enters read-only mode. Your existing flows will continue to operate until that date.

Recommended action: Migrate your Prompt Flow workloads to Microsoft Agent Framework before April 20, 2027.
:::

Prompt flow

Prompt flow is a suite of development tools designed to streamline the end-to-end development cycle of LLM-based AI applications, from ideation, prototyping, testing, evaluation to production deployment and monitoring. It makes prompt engineering much easier and enables you to build LLM apps with production quality.

With prompt flow, you will be able to:

• Create flows that link LLMs, prompts, Python code and other tools together in a executable workflow.
• Debug and iterate your flows, especially tracing interaction with LLMs with ease.
• Evaluate your flows, calculate quality and performance metrics with larger datasets.
• Integrate the testing and evaluation into your CI/CD system to ensure quality of your flow.
• Deploy your flows to the serving platform you choose or integrate into your app's code base easily.
• (Optional but highly recommended) Collaborate with your team by leveraging the cloud version of Prompt flow in Azure AI.

Welcome to join us to make prompt flow better by
participating discussions,
opening issues,
submitting PRs.

This documentation site contains guides for prompt flow sdk, cli and vscode extension users.

:grid-columns: 1 2 2 2
- header: "🚀 Quick Start"
  content: "
    Quick start and end-to-end tutorials.<br/><br/>
    - [Getting started with prompt flow](how-to-guides/quick-start.md)<br/>
    - [E2E development tutorial: chat with PDF](https://github.com/microsoft/promptflow/blob/main/examples/tutorials/e2e-development/chat-with-pdf.md)<br/>
    - Find more: [tutorials & samples](tutorials/index.md)<br/>
  "

- header: "📒 How-to Guides"
  content: "
    Articles guide user to complete a specific task in prompt flow.<br/><br/>
    - [Tracing](how-to-guides/tracing/index.md)<br/>
    - [Develop a flow](how-to-guides/develop-a-flex-flow/index.md)<br/>
    - [Run and evaluate a flow](how-to-guides/run-and-evaluate-a-flow/index.md)<br/>
    - [Deploy a flow](how-to-guides/deploy-a-flow/index.md)<br/>
  "

:grid-columns: 1 2 2 2
- header: "📑 Concepts"
  content: "
    Introduction of key concepts of prompt flow.<br/><br/>
    - [Flows](concepts/concept-flows.md)<br/>
    - [Tools](concepts/concept-tools.md)<br/>
    - [Connections](concepts/concept-connections.md)<br/>
    - [Design principles](concepts/design-principles.md)<br/>
  "


- header: "🔍 Reference"
  content: "
    Reference provides technical information about prompt flow API.<br/><br/>
    - Command line Interface reference: [pf](reference/pf-command-reference.md)<br/>
    - Python library reference: [promptflow](reference/python-library-reference/promptflow-core/promptflow.rst)<br/>
    - Tool reference: [LLM Tool](reference/tools-reference/llm-tool.md), [Python Tool](reference/tools-reference/python-tool.md), [Prompt Tool](reference/tools-reference/prompt-tool.md)<br/>
    - ChangeLog: [ChangeLog](reference/changelog/promptflow.md)<br/>
  "

:hidden:
:maxdepth: 1
how-to-guides/index

:hidden:
:maxdepth: 1
tutorials/index

:hidden:
:maxdepth: 2
concepts/index

:hidden:
:maxdepth: 1
reference/index

:hidden:
:maxdepth: 1
cloud/index

:hidden:
:maxdepth: 1
integrations/index

Cloud Index (`docs/cloud/index.md`) Copy Section

Cloud

Prompt flow streamlines the process of developing AI applications based on LLM, easing prompt engineering, prototyping, evaluating, and fine-tuning for high-quality products.

Transitioning to production, however, typically requires a comprehensive LLMOps process, LLMOps is short for large language model operations. This can often be a complex task, demanding high availability and security, particularly vital for large-scale team collaboration and lifecycle management when deploying to production.

To assist in this journey, we've introduced Azure AI, a cloud-based platform tailored for executing LLMOps, focusing on boosting productivity for enterprises.

• Private data access and controls
• Collaborative development
• Automating iterative experimentation and CI/CD
• Deployment and optimization
• Safe and Responsible AI

Transitioning from local to cloud (Azure AI) Copy Chapter

In prompt flow, You can develop your flow locally and then seamlessly transition to Azure AI. Here are a few scenarios where this might be beneficial:

For more resources on Azure AI, visit the cloud documentation site: Build AI solutions with prompt flow.

:caption: Flow
:maxdepth: 2
azureai/manage-flows
azureai/run-promptflow-in-azure-ai
azureai/create-run-with-compute-session
azureai/use-flow-in-azure-ml-pipeline
azureai/generate-test-data-cloud.md

:caption: Deployment
:maxdepth: 2
azureai/deploy-to-azure-appservice

:caption: FAQ
:maxdepth: 2
azureai/faq
azureai/consume-connections-from-azure-ai
azureai/runtime-change-log.md

Concepts Index (`docs/concepts/index.md`) Copy Section

Concepts

In this section, you will learn the basic concepts of prompt flow.

:maxdepth: 1

concept-flows
concept-tools
concept-connections
concept-variants
design-principles

How To Guides Index (`docs/how-to-guides/index.md`) Copy Section

How-to Guides

Simple and short articles grouped by topics, each introduces a core feature of prompt flow and how you can use it to address your specific use cases.

:caption: Quick start
:maxdepth: 1
quick-start
installation/index

:caption: Tracing
:maxdepth: 1
tracing/index

:caption: Prompty
:maxdepth: 1
develop-a-prompty/index

:caption: Flow
:maxdepth: 1
develop-a-flex-flow/index
develop-a-dag-flow/index
execute-flow-as-a-function
chat-with-a-flow/index
run-and-evaluate-a-flow/index

:caption: Tool
:maxdepth: 1
develop-a-tool/index

:caption: Deployment
:maxdepth: 1
deploy-a-flow/index
enable-streaming-mode

:caption: FAQ
:maxdepth: 1
faq
set-promptflow-configs
manage-connections
tune-prompts-with-variants

Integrations Index (`docs/integrations/index.md`) Copy Section

Integrations

The Integrations section contains documentation on custom extensions created by the community that expand prompt flow's capabilities.
These include tools that enrich flows, as well as tutorials on innovative ways to use prompt flow.

:maxdepth: 1

tools/index
llms/index

Reference Index (`docs/reference/index.md`) Copy Section

Reference

Current stable version:

• promptflow:
  
  
  • promptflow-tracing:
    
    
  • promptflow-core:
    
    
  • promptflow-devkit:
    
    
  • promptflow-azure:
    
    
• promptflow-tools:
  
  

:caption: Command Line Interface
:maxdepth: 1

pf-command-reference.md
pfazure-command-reference.md

:caption: Python Library ChangeLog
:maxdepth: 1

changelog/promptflow

:caption: Python Library Reference
:maxdepth: 1

python-library-reference/promptflow-tracing/promptflow
python-library-reference/promptflow-core/promptflow
python-library-reference/promptflow-devkit/promptflow
python-library-reference/promptflow-azure/promptflow
python-library-reference/promptflow-rag/promptflow
python-library-reference/promptflow-evals/promptflow

:caption: Tool Reference
:maxdepth: 1

tools-reference/llm-tool
tools-reference/llm-vision-tool
tools-reference/prompt-tool
tools-reference/python-tool
tools-reference/serp-api-tool
tools-reference/faiss_index_lookup_tool
tools-reference/vector_db_lookup_tool
tools-reference/embedding_tool
tools-reference/open_model_llm_tool
tools-reference/openai-gpt-4v-tool
tools-reference/contentsafety_text_tool
tools-reference/aoai-gpt4-turbo-vision
tools-reference/rerank-tool

:caption: YAML Schema
:maxdepth: 1

flow-yaml-schema-reference.md
run-yaml-schema-reference.md

:caption: Trace Specification
:maxdepth: 1

trace-span-spec-reference.md

Tutorials Index (`docs/tutorials/index.md`) Copy Section

Tutorials

This section contains a collection of flow samples and step-by-step tutorials.

Learn more: Try out more promptflow examples.

How To Guides Chat With A Flow Index (`docs/how-to-guides/chat-with-a-flow/index.md`) Copy Section

Chat with a flow

:::{admonition} Experimental feature
This is an experimental feature, and may change at any time. Learn more.
:::

Prompt flow provides the chat window feature to facilitate an interactive chat experience within a local environment.
You can engage in conversation with the flow and view its responses directly within the chat window.

Initiating a chat window Copy Chapter

There are two methods to open a chat window: executing the prompt flow CLI command or clicking the
Open test chat page button when viewing a flow YAML file in the Prompt flow VS Code extension.

::::{tab-set}
:::{tab-item} CLI
:sync: CLI

The following CLI command allows you to trigger a chat window.

pf flow test --flow . --ui

Running the above command will yield the following example output:

Starting prompt flow service...
...
You can begin chat flow on http://127.0.0.1:**/v1.0/ui/chat?flow=***

The browser page corresponding to the chat URL will automatically open and direct the user to a chat page
corresponding to the passed flow:

:::

:::{tab-item} VS Code Extension
:sync: VSC

Click the Open test chat page button while viewing a flow YAML file in the Prompt flow VS Code extension, and you
will be directed to the chat page.

:::
::::

How To Guides Deploy A Flow Index (`docs/how-to-guides/deploy-a-flow/index.md`) Copy Section

Deploy a flow

A flow can be deployed to multiple platforms, such as a local development service, Docker container, Kubernetes cluster, etc.

:grid-columns: 1 2 2 3
- image: ../../media/how-to-guides/local.png
  content: "<center><b>Development server</b></center>"
  website: deploy-using-dev-server.html

- image: ../../media/how-to-guides/docker.png
  content: "<center><b>Docker</b></center>"
  website: deploy-using-docker.html
  
- image: ../../media/how-to-guides/kubernetes.png
  content: "<center><b>Kubernetes</b></center>"
  website: deploy-using-kubernetes.html

We also provide guides to deploy to cloud, such as azure app service:

:grid-columns: 1 2 2 3

- image: ../../media/how-to-guides/appservice.png
  content: "<center><b>Azure App Service</b></center>"
  website: ../../cloud/azureai/deploy-to-azure-appservice.html

We are working on more official deployment guides for other hosting providers, and welcome user submitted guides.

:maxdepth: 1
:hidden:

deploy-using-dev-server
deploy-using-docker
deploy-using-kubernetes
distribute-flow-as-executable-app

How To Guides Develop A Dag Flow Index (`docs/how-to-guides/develop-a-dag-flow/index.md`) Copy Section

Develop a dag flow

LLM apps can be defined as Directed Acyclic Graphs (DAGs) of function calls. These DAGs are flows in prompt flow.

A DAG flow in prompt flow is a DAG of functions (we call them tools). These functions/tools connected via input/output dependencies and executed based on the topology by prompt flow executor.

A flow is represented as a YAML file and can be visualized with our Prompt flow for VS Code extension. Here is an example flow.dag.yaml:

Please refer to our examples and guides in this section to learn how to write a DAG flow.

Note:

• promptflow also support user develop a a flow using code. learn more on comparasion of these two flow concepts.

:maxdepth: 1

quick-start
init-and-test-a-flow
develop-standard-flow
develop-chat-flow
develop-evaluation-flow
add-conditional-control-to-a-flow
process-image-in-flow
referencing-external-files-or-folders-in-a-flow

How To Guides Develop A Flex Flow Index (`docs/how-to-guides/develop-a-flex-flow/index.md`) Copy Section

Develop a flow

:::{admonition} Experimental feature
This is an experimental feature, and may change at any time. Learn more.
:::

You can create LLM apps using a Python function or class as the entry point, which encapsulating your app logic. You can directly test or run these entries with pure code experience.

In PromptFlow, these functions or classes are referred to as flow or flex flow.

Alternatively, you can define a flow.flex.yaml that points to these entries (entry:function_name or entry:ClassName). This enables testing, running, or viewing traces via the Promptflow VS Code Extension.

Our examples should give you a good idea on how to write flows.

Note:

• The term Flex is a shorthand for flexible, indicating its adaptability to most scenarios with minimal adjustments.
• PromptFlow also supports the development of a dag flow. learn more on comparasion of these two flow concepts.

:maxdepth: 1

function-based-flow
class-based-flow
input-output-format
model-config
connection-support

How To Guides Develop A Prompty Index (`docs/how-to-guides/develop-a-prompty/index.md`) Copy Section

Develop a prompty

:::{admonition} Experimental feature
This is an experimental feature, and may change at any time. Learn more.
:::

Promptflow introduces the prompty feature, it is designed to simplify the development of prompt templates for customers.

Create a prompty Copy Chapter

Prompty specification Copy Section

In promptflow, file bearing the .prompty extension is recognized as a prompty. This unique file type facilitates the development of prompt template.

Prompty is a markdown file, this front matter, structured in YAML, encapsulates a series of metadata fields pivotal for defining the model’s configuration and the inputs for the prompty.
After this front matter is the prompt template, articulated in the Jinja format.

Fields in the front matter:

---
name: Basic Prompt
description: A basic prompt that uses the GPT-3 chat API to answer questions
model:
  api: chat
  configuration:
    type: azure_openai
    azure_deployment: gpt-35-turbo
    connection: azure_open_ai_connection
  parameters:
    max_tokens: 128
    temperature: 0.2
inputs:
  first_name:
    type: string
  last_name:
    type: string
  question:
    type: string
sample:
  first_name: John
  last_name: Doe
  question: Who is the most famous person in the world?
---
system:
You are an AI assistant who helps people find information.
As the assistant, you answer questions briefly, succinctly,
and in a personable manner using markdown and even add some personal flair with appropriate emojis.

# Safety
- You **should always** reference factual statements to search results based on [relevant documents]
- Search results based on [relevant documents] may be incomplete or irrelevant. You do not make assumptions
# Customer
You are helping {{first_name}} {{last_name}} to find answers to their questions.
Use their name to address them in your responses.

user:
{{question}}

Load a prompty Copy Chapter

prompty are designed with flexibility in mind, allowing users to override the default model configuration during the loading process.

::::{tab-set}
:::{tab-item} Azure OpenAI
:sync: Azure OpenAI

---
name: Basic Prompt
description: A basic prompt that uses the GPT-3 chat API to answer questions
model:
  api: chat
  configuration:
    type: azure_openai
    azure_deployment: gpt-35-turbo
    api_key: ${env:AZURE_OPENAI_API_KEY}
    api_version: ${env:AZURE_OPENAI_API_VERSION}
    azure_endpoint: ${env:AZURE_OPENAI_ENDPOINT}
  parameters:
    max_tokens: 128
    temperature: 0.2
inputs:
  first_name:
    type: string
  last_name:
    type: string
  question:
    type: string
sample:
  first_name: John
  last_name: Doe
  question: Who is the most famous person in the world?
---
system:
You are an AI assistant who helps people find information.
As the assistant, you answer questions briefly, succinctly,
and in a personable manner using markdown and even add some personal flair with appropriate emojis.

# Safety
- You **should always** reference factual statements to search results based on [relevant documents]
- Search results based on [relevant documents] may be incomplete or irrelevant. You do not make assumptions
# Customer
You are helping {{first_name}} {{last_name}} to find answers to their questions.
Use their name to address them in your responses.

user:
{{question}}

Users can specify alternative parameters or utilize environment variables to adjust the model settings. The format ${env:ENV_NAME} is used to reference environment variables.

• Using a dictionary

  python Copy

  from promptflow.core import Prompty
  
  # Load prompty with dict override
  override_model = {
      "configuration": {
          "api_key": "${env:AZURE_OPENAI_API_KEY}",
          "api_version": "${env:AZURE_OPENAI_API_VERSION}",
          "azure_endpoint": "${env:AZURE_OPENAI_ENDPOINT}"
      },
      "parameters": {"max_tokens": 512}
  }
  prompty = Prompty.load(source="path/to/prompty.prompty", model=override_model)

• Using AzureOpenAIModelConfiguration:

  python Copy

  from promptflow.core import Prompty, AzureOpenAIModelConfiguration
  
  # Load prompty with AzureOpenAIModelConfiguration override
  configuration = AzureOpenAIModelConfiguration(
      azure_deployment="gpt-3.5-turbo",
      api_key="${env:AZURE_OPENAI_API_KEY}",
      api_version="${env:AZURE_OPENAI_API_VERSION}",
      azure_endpoint="${env:AZURE_OPENAI_ENDPOINT}"
  )
  override_model = {
      "configuration": configuration,
      "parameters": {"max_tokens": 512}
  }
  prompty = Prompty.load(source="path/to/prompty.prompty", model=override_model)

:::

:::{tab-item} OpenAI
:sync: OpenAI

---
name: Basic Prompt
description: A basic prompt that uses the GPT-3 chat API to answer questions
model:
  api: chat
  configuration:
    type: openai
    model: gpt-3.5-turbo
    api_key: ${env:OPENAI_API_KEY}
    base_url: ${env:OPENAI_BASE_URL}
  parameters:
    max_tokens: 128
    temperature: 0.2
inputs:
  first_name:
    type: string
  last_name:
    type: string
  question:
    type: string
sample:
  first_name: John
  last_name: Doe
  question: Who is the most famous person in the world?
---
system:
You are an AI assistant who helps people find information.
As the assistant, you answer questions briefly, succinctly,
and in a personable manner using markdown and even add some personal flair with appropriate emojis.

# Safety
- You **should always** reference factual statements to search results based on [relevant documents]
- Search results based on [relevant documents] may be incomplete or irrelevant. You do not make assumptions
# Customer
You are helping {{first_name}} {{last_name}} to find answers to their questions.
Use their name to address them in your responses.

user:
  {{question}}

Users can specify alternative parameters or utilize environment variables to adjust the model settings. The format ${env:ENV_NAME} is used to reference environment variables.

• Using a dictionary

  python Copy

  from promptflow.core import Prompty
  
  # Load prompty with dict override
  override_model = {
      "configuration": {
          "api_key": "${env:OPENAI_API_KEY}",
          "base_url": "${env:OPENAI_BASE_URL}",
      },
      "parameters": {"max_tokens": 512}
  }
  prompty = Prompty.load(source="path/to/prompty.prompty", model=override_model)

• Using OpenAIModelConfiguration

  python Copy

  from promptflow.core import Prompty, OpenAIModelConfiguration
  
  # Load prompty with OpenAIModelConfiguration override
  configuration = OpenAIModelConfiguration(
      model="gpt-35-turbo",
      base_url="${env:OPENAI_BASE_URL}",
      api_key="${env:OPENAI_API_KEY}",
  )
  override_model = {
      "configuration": configuration,
      "parameters": {"max_tokens": 512}
  }
  prompty = Prompty.load(source="path/to/prompty.prompty", model=override_model)

:::
::::

Execute a prompty Copy Chapter

Promptflow offers versatile methods for executing a prompty to meet the needs of customers in different scenarios.

Direct function call Copy Section

Once loaded, the Prompty object can be invoked directly as a function, returning the content of the first choice in the LLM response.

from promptflow.core import Prompty

prompty_obj = Prompty.load(source="path/to/prompty.prompty")
result = prompty_obj(first_name="John", last_name="Doh", question="What is the capital of France?")

Testing prompty Copy Section

Flow test

Execute and test your Prompty with inputs or a sample file.
::::{tab-set}
:::{tab-item} CLI
:sync: CLI

# Test prompty with default inputs
pf flow test --flow path/to/prompty.prompty

# Test prompty with specified inputs
pf flow test --flow path/to/prompty.prompty --inputs first_name=John last_name=Doh question="What is the capital of France?"

# Test prompty with sample file
pf flow test --flow path/to/prompty.prompty --inputs path/to/sample.json

A trace link will be provided in the terminal to visualize the internal execution details for this command.
For Prompty, users can find the generated prompt, LLM request parameters, and other information in the trace UI. Learn more.

:::

:::{tab-item} SDK
:sync: SDK

from promptflow.client import PFClient

pf = PFClient()

# Test prompty with specified inputs
result = pf.test(flow="path/to/prompty.prompty", inputs={"first_name": "John", "last_name": "Doh", "question": "What is the capital of France?"})

# Test prompty with sample file
result = pf.test(flow="path/to/prompty.prompty", inputs="path/to/sample.json")

:::
::::

Test with interactive mode

Promptflow CLI also provides an interactive chat session for testing chat flows.

pf flow test --flow path/to/prompty.prompty --interactive

---
name: Basic Prompt With Chat History
description: A basic prompt that uses the GPT-3 chat API to answer questions
model:
  api: chat
  configuration:
    type: azure_open_ai
    azure_deployment: gpt-35-turbo
    connection: azure_open_ai_connection

  parameters:
    max_tokens: 128
    temperature: 0.2
inputs:
  first_name:
    type: string
  last_name:
    type: string
  question:
    type: string
   chat_history:
    type: list
sample:
  first_name: John
  last_name: Doe
  question: Who is the most famous person in the world?
  chat_history: [ { "role": "user", "content": "what's the capital of France?" }, { "role": "assistant", "content": "Paris" } ]
---
system:
You are an AI assistant who helps people find information.
As the assistant, you answer questions briefly, succinctly,
and in a personable manner using markdown and even add some personal flair with appropriate emojis.

# Safety
- You **should always** reference factual statements to search results based on [relevant documents]
- Search results based on [relevant documents] may be incomplete or irrelevant. You do not make assumptions
# Customer
You are helping {{first_name}} {{last_name}} to find answers to their questions.
Use their name to address them in your responses.

Here is a chat history you had with the user:
{% for item in chat_history %}
   {{item.role}}: {{item.content}}
{% endfor %}

user:
  {{question}}

Terminal outputs:

Batch run prompty Copy Section

::::{tab-set}
:::{tab-item} CLI
:sync: CLI

To execute a batch run of a Prompty in Promptflow, you can use the following commands:

pf run create --flow path/to/prompty.prompty --data path/to/inputs.jsonl

:::

:::{tab-item} SDK
:sync: SDK

To execute a batch run of a Prompty in Promptflow, you can use the following SDK:

from promptflow.client import PFClient

pf = PFClient() 
# create run
prompty_run = pf.run(
    flow="path/to/prompty.prompty",
    data="path/to/inputs.jsonl",
)
pf.stream(prompty_run)

:::
::::

When executing a batch run, Promptflow provides a trace UI to visualize the internal execution details of the run. This feature allows you to track the execution details of each line in the data file, including the prompt and LLM request parameters. Learn more.

For example, after starting the Prompt flow service, you might see output like this in your terminal:

Prompt flow service has started...
You can view the traces from local: http://127.0.0.1:49240/v1.0/ui/traces/?#run=prompty_variant_0_20240424_152808_282517
[2024-04-24 15:28:12,597][promptflow._sdk._orchestrator.run_submitter][INFO] - Submitting run prompty_variant_0_20240424_152808_282517, log path: .promptflow\.runs\prompty_variant_0_20240424_152808_282517\logs.txt

The trace UI will record the execution details of each line in the data file, providing a comprehensive view of the batch run’s performance and outcomes.

:maxdepth: 1
:hidden:

prompty-output-format
use-prompty-in-flow

How To Guides Develop A Tool Index (`docs/how-to-guides/develop-a-tool/index.md`) Copy Section

Develop a tool

We provide guides on how to develop a tool and use it.

:maxdepth: 1
:hidden:

create-and-use-tool-package
add-a-tool-icon
add-category-and-tags-for-tool
use-file-path-as-tool-input
customize_an_llm_tool
create-cascading-tool-inputs
create-your-own-custom-strong-type-connection
create-dynamic-list-tool-input

How To Guides Installation Index (`docs/how-to-guides/installation/index.md`) Copy Section

Installation

The prompt flow SDK and CLI are under active development, with regular stable releases on PyPI:

. Refer to SDK Changelog for release history and upcoming features.

This guide outlines the Promptflow SDK and CLI installation process.

Create a virtual environment (optional) Copy Chapter

When installing prompt flow locally, we recommend using a virtual environment for the installation. This ensures
that the dependencies for prompt flow are isolated from the rest of your system. Please ensure you have a working
python environment (python>=3.9,<4.0), a new virtual environment is preferred.

::::{tab-set}
:::{tab-item} venv
:sync: venv

To create and activate:

python3 -m venv pf
source pf/bin/activate

To deactivate later, run:

deactivate

:::

:::{tab-item} Conda
:sync: Conda
Install Conda if you have not already.
To create and activate:

conda create -n pf python=3.11
conda activate pf

To deactivate later, run:

conda deactivate

:::

::::

Install prompt flow Copy Chapter

Run below command to install the latest version of the promptflow.

# Install the latest stable version
pip install promptflow --upgrade

After developing your flow locally, you can seamlessly transition to Azure AI and interact with your flow in the cloud.
Use the following command to install the latest version of prompt flow to work with flows in Azure.

# Install the latest stable version
pip install promptflow[azure] --upgrade

Promptflow sub packages Copy Chapter

Prompt flow consists of several sub-packages, each is designed to provide specific functionalities.

Verify installation Copy Chapter

To verify the installation, run the following command to check the version of prompt flow installed.

pf --version

Running the above command will yield the following example output:

{
  "promptflow": "1.10.1",
  "promptflow-core": "1.10.1",
  "promptflow-devkit": "1.10.1",
  "promptflow-tracing": "1.10.1"
}

:maxdepth: 1
:hidden:

standalone

How To Guides Run And Evaluate A Flow Index (`docs/how-to-guides/run-and-evaluate-a-flow/index.md`) Copy Section

Run and evaluate a flow

After you have developed and tested the flow in init and test a flow, this guide will help you learn how to run a flow with a larger dataset and then evaluate the flow you have created.

Create a batch run Copy Chapter

Since you have run your flow successfully with a small set of data, you might want to test if it performs well in large set of data, you can run a batch test and check the outputs.

A bulk test allows you to run your flow with a large dataset and generate outputs for each data row, and the run results will be recorded in local db so you can use pf commands to view the run results at anytime. (e.g. pf run list)

Let's create a run with flow web-classification. It is a flow demonstrating multi-class classification with LLM. Given an url, it will classify the url into one web category with just a few shots, simple summarization and classification prompts.

To begin with the guide, you need:

• Git clone the sample repository(above flow link) and set the working directory to /examples/flows/.
• Make sure you have already created the necessary connection.
  ::::{tab-set}

:::{tab-item} CLI
:sync: CLI

Create the run with flow and data, can add --stream to stream the run.

pf run create --flow standard/web-classification --data standard/web-classification/data.jsonl --column-mapping url='${data.url}' --stream

Note column-mapping is a mapping from flow input name to specified values, see more details in Use column mapping.

You can also name the run by specifying --name my_first_run in above command, otherwise the run name will be generated in a certain pattern which has timestamp inside.

With a run name, you can easily view or visualize the run details using below commands:

pf run show-details -n my_first_run

pf run visualize -n my_first_run

More details can be found with pf run --help

:::
:::{tab-item} SDK
:sync: SDK

from promptflow.client import PFClient

# Please protect the entry point by using `if __name__ == '__main__':`,
# otherwise it would cause unintended side effect when promptflow spawn worker processes.
# Ref: https://docs.python.org/3/library/multiprocessing.html#the-spawn-and-forkserver-start-methods
if __name__ == "__main__":
  # PFClient can help manage your runs and connections.
  pf = PFClient()

  # Set flow path and run input data
  flow = "standard/web-classification" # set the flow directory
  data= "standard/web-classification/data.jsonl" # set the data file

  # create a run, stream it until it's finished
  base_run = pf.run(
      flow=flow,
      data=data,
      stream=True,
      # map the url field from the data to the url input of the flow
      column_mapping={"url": "${data.url}"},
  )

# get the inputs/outputs details of a finished run.
details = pf.get_details(base_run)
details.head(10)

# visualize the run in a web browser
pf.visualize(base_run)

Feel free to check Promptflow Python Library Reference for all SDK public interfaces.

:::

:::{tab-item} VS Code Extension
:sync: VS Code Extension
Use the code lens action on the top of the yaml editor to trigger batch run

Click the bulk test button on the top of the visual editor to trigger flow test.

:::

::::

We also have a more detailed documentation Manage runs demonstrating how to manage your finished runs with CLI, SDK and VS Code Extension.

Evaluate your flow Copy Chapter

You can use an evaluation method to evaluate your flow. The evaluation methods are also flows which use Python or LLM etc., to calculate metrics like accuracy, relevance score. Please refer to Develop evaluation flow to learn how to develop an evaluation flow.

In this guide, we use eval-classification-accuracy flow to evaluate. This is a flow illustrating how to evaluate the performance of a classification system. It involves comparing each prediction to the groundtruth and assigns a Correct or Incorrect grade, and aggregating the results to produce metrics such as accuracy, which reflects how good the system is at classifying the data.

Run evaluation flow against run Copy Section

::::{tab-set}

:::{tab-item} CLI
:sync: CLI

Evaluate the finished flow run

After the run is finished, you can evaluate the run with below command, compared with the normal run create command, note there are two extra arguments:

• column-mapping: A mapping from flow input name to specified data values. Reference here for detailed information.
• run: The run name of the flow run to be evaluated.

More details can be found in Use column mapping.

pf run create --flow evaluation/eval-classification-accuracy --data standard/web-classification/data.jsonl --column-mapping groundtruth='${data.answer}' prediction='${run.outputs.category}' --run my_first_run --stream

Same as the previous run, you can specify the evaluation run name with --name my_first_eval_run in above command.

You can also stream or view the run details with:

pf run stream -n my_first_eval_run  # same as "--stream" in command "run create"
pf run show-details -n my_first_eval_run
pf run show-metrics -n my_first_eval_run

Since now you have two different runs my_first_run and my_first_eval_run, you can visualize the two runs at the same time with below command.

pf run visualize -n "my_first_run,my_first_eval_run"

A web browser will be opened to show the visualization result.

:::

:::{tab-item} SDK
:sync: SDK

Evaluate the finished flow run

After the run is finished, you can evaluate the run with below command, compared with the normal run create command, note there are two extra arguments:

• column-mapping: A dictionary represents sources of the input data that are needed for the evaluation method. The sources can be from the flow run output or from your test dataset.
  • If the data column is in your test dataset, then it is specified as ${data.<column_name>}.
  • If the data column is from your flow output, then it is specified as ${run.outputs.<output_name>}.
• run: The run name or run instance of the flow run to be evaluated.

More details can be found in Use column mapping.

from promptflow.client import PFClient

# PFClient can help manage your runs and connections.
pf = PFClient()

# set eval flow path
eval_flow = "evaluation/eval-classification-accuracy"
data= "standard/web-classification/data.jsonl"

# run the flow with existing run
eval_run = pf.run(
    flow=eval_flow,
    data=data,
    run=base_run,
    column_mapping={  # map the url field from the data to the url input of the flow
      "groundtruth": "${data.answer}",
      "prediction": "${run.outputs.category}",
    }
)

# stream the run until it's finished
pf.stream(eval_run)

# get the inputs/outputs details of a finished run.
details = pf.get_details(eval_run)
details.head(10)

# view the metrics of the eval run
metrics = pf.get_metrics(eval_run)
print(json.dumps(metrics, indent=4))

# visualize both the base run and the eval run
pf.visualize([base_run, eval_run])

A web browser will be opened to show the visualization result.

:::

:::{tab-item} VS Code Extension
:sync: VS Code Extension

There are actions to trigger local batch runs. To perform an evaluation you can use the run against "existing runs" actions.

:::

::::

Next steps Copy Chapter

Learn more about:

• Tune prompts with variants
• Deploy a flow
• Manage runs
• Python library reference

:maxdepth: 1
:hidden:

use-column-mapping
manage-runs

How To Guides Tracing Index (`docs/how-to-guides/tracing/index.md`) Copy Section

Tracing

:::{admonition} Experimental feature
This is an experimental feature, and may change at any time. Learn more.
:::

Traces records specific events or the state of an application during execution. It can include data about function calls, variable values, system events and more. Traces help break down an application's components into discrete inputs and outputs, which is crucial for debugging and understanding an application. You can learn more from here on traces.

Prompt flow provides the trace feature to enable user to trace LLM call or function, and LLM frameworks like LangChain and AutoGen, following OpenTelemetry specification.

Installing the package Copy Chapter

pip install promptflow-tracing

Instrumenting user's code Copy Chapter

Enable trace for LLM calls Copy Section

Let's start with the simplest example, add single line code start_trace() to enable trace for LLM calls in your application.

from openai import OpenAI
from promptflow.tracing import start_trace

# instrument OpenAI
start_trace()

client = OpenAI()

completion = client.chat.completions.create(
  model="gpt-4",
  messages=[
    {"role": "system", "content": "You are a poetic assistant, skilled in explaining complex programming concepts with creative flair."},
    {"role": "user", "content": "Compose a poem that explains the concept of recursion in programming."}
  ]
)

print(completion.choices[0].message)

Then OpenAI is instrumented, and as prompt flow follows OpenTelemetry specification, user can fully leverage the OpenTelemetry knowledge to use these traces during the OpenAI calls.

Trace on any function Copy Section

A more common scenario is the application has complicated code structure, and developer would like to add trace on critical path that they would like to debug and monitor.

See the math_to_code example on how to use @trace.

Execute below command will get an URL to display the trace records and trace details of each test.

from promptflow.tracing import trace

# trace your function
@trace
def code_gen(client: AzureOpenAI, question: str) -> str:
    sys_prompt = (
        "I want you to act as a Math expert specializing in Algebra, Geometry, and Calculus. "
        "Given the question, develop python code to model the user's question. "
        "Make sure only reply the executable code, no other words."
    )
    completion = client.chat.completions.create(
        model=os.getenv("CHAT_DEPLOYMENT_NAME", "gpt-35-turbo"),
        messages=[
            {
                "role": "system",
                "content": sys_prompt,
            },
            {"role": "user", "content": question},
        ],
    )
    raw_code = completion.choices[0].message.content
    result = code_refine(raw_code)
    return result

python math_to_code.py

Trace LLM and frameworks Copy Chapter

Prompt flow tracing works not only for general LLM application, but also for more frameworks like autogen and langchain. Besides basic tracing capability, prompt flow also provides several trace toolkits that can improve the tracing experience (e.g., trace UI for visualization).

1. Example: Add trace for LLM

2. Example: Add trace for Autogen

3. Example: Add trace for Langchain

:maxdepth: 1
:hidden:

trace-ui
manage

Integrations Llms Index (`docs/integrations/llms/index.md`) Copy Section

Alternative LLMs

This section provides tutorials on incorporating alternative large language models into prompt flow.

:maxdepth: 1
:hidden:

Integrations Tools Index (`docs/integrations/tools/index.md`) Copy Section

Custom Tools

This section contains documentation for custom tools created by the community to extend Prompt flow's capabilities for specific use cases. These tools are developed following the guide on Creating and Using Tool Packages. They are not officially maintained or endorsed by the Prompt flow team. For questions or issues when using a tool, please use the support contact link in the table below.

Tool Package Index Copy Chapter

The table below provides an index of custom tool packages. The columns contain:

• Package Name: The name of the tool package. Links to the package documentation.
• Description: A short summary of what the tool package does.
• Owner: The creator/maintainer of the tool package.
• Support Contact: Link to contact for support and reporting new issues.

:maxdepth: 1
:hidden:

azure-ai-language-tool
llmlingua-prompt-compression-tool

Concepts Concept Connections (`docs/concepts/concept-connections.md`) Copy Section

In prompt flow, you can utilize connections to securely manage credentials or secrets for external services.

Connections

Connections are for storing information about how to access external services like LLMs: endpoint, api keys etc.

• In your local development environment, the connections are persisted in your local machine with keys encrypted.
• In Azure AI, connections can be configured to be shared across the entire workspace. Secrets associated with connections are securely persisted in the corresponding Azure Key Vault, adhering to robust security and compliance standards.

Prompt flow provides a variety of pre-built connections, including Azure OpenAI, OpenAI, etc. These pre-built connections enable seamless integration with these resources within the built-in tools. Additionally, you have the flexibility to create custom connection types using key-value pairs, empowering them to tailor the connections to their specific requirements, particularly in Python tools.

By leveraging connections in prompt flow, you can easily establish and manage connections to external APIs and data sources, facilitating efficient data exchange and interaction within their AI applications.

Next steps Copy Chapter

• Create connections

Concepts Concept Flows (`docs/concepts/concept-flows.md`) Copy Section

While how LLMs work may be elusive to many developers, how LLM apps work is not - they essentially involve a series of calls to external services such as LLMs/databases/search engines, or intermediate data processing, all glued together.

Flows

Flex flow Copy Chapter

You can create LLM apps using a Python function or class as the entry point, which encapsulating your app logic. You can directly test or run these with pure code experience. Or you can define a flow.flex.yaml that points to these entries, which enables testing, running, or viewing traces via the Promptflow VS Code Extension.

Our examples should also give you an idea how to write flex flows.

DAG flow Copy Chapter

Thus LLM apps can be defined as Directed Acyclic Graphs (DAGs) of function calls. These DAGs are flows in prompt flow.

A DAG flow in prompt flow is a DAG of functions (we call them tools). These functions/tools connected via input/output dependencies and executed based on the topology by prompt flow executor.

A flow is represented as a YAML file and can be visualized with our Prompt flow for VS Code extension. Here is an example flow.dag.yaml:

Please refer to our examples to learn how to write a DAG flow.

When to use Flex or DAG flow Copy Chapter

Dag flow provides a UI-friendly way to develop your LLM app, which has the following benefits:

• Low code: user can drag-and-drop in UI to create a LLM app.
• DAG Visualization: user can easily understand the logic structure of the app with DAG view.

Flex flow provides a code-friendly way to develop your LLM app, which has the following benefits:

• Quick start: Users can quickly test with a simple prompt, then customize with python code with Tracing visualization UI.
• More advanced orchestration: Users can write complex flow with Python built-in control operators (if-else, foreach) or other 3rd party / open-source library.
• Easy onboard from other platforms: user might already onboard platforms like langchain and sematic kernel with existing code. User can easily onboard promptflow with a few code changes.

Flow types Copy Chapter

Prompt flow examples organize flows by three categories:

• Standard flow or Chat flow: these two are for you to develop your LLM application. The primary difference between the two lies in the additional support provided by the "Chat Flow" for chat applications. For instance, you can define chat_history, chat_input, and chat_output for your flow. The prompt flow, in turn, will offer a chat-like experience (including conversation history) during the development of the flow. Moreover, it also provides a sample chat application for deployment purposes.
• Evaluation flow is for you to test/evaluate the quality of your LLM application (standard/chat flow). It usually run on the outputs of standard/chat flow, and compute some metrics that can be used to determine whether the standard/chat flow performs well. E.g. is the answer accurate? is the answer fact-based?

Flex flow examples:

• basic: A basic standard flow define using function entry that calls Azure OpenAI with connection info stored in environment variables.
• chat-basic: A basic chat flow defined using class entry. It demonstrates how to create a chatbot that can remember previous interactions and use the conversation history to generate next message.
• eval-code-quality: A example flow defined using function entry which shows how to evaluate the quality of code snippet.

DAG flow examples:

• basic: A basic standard flow using custom python tool that calls Azure OpenAI with connection info stored in environment variables.
• chat-basic: This example shows how to create a basic chat flow. It demonstrates how to create a chatbot that can remember previous interactions and use the conversation history to generate next message.
• eval-basic:

Next steps Copy Chapter

• Quick start
• Initialize and test a flow
• Run and evaluate a flow
• Tune prompts using variants

Concepts Concept Tools (`docs/concepts/concept-tools.md`) Copy Section

Tools are the fundamental building blocks of a flow.

Each tool is an executable unit, basically a function to performs various tasks including but not limited to:

• Accessing LLMs for various purposes
• Querying databases
• Getting information from search engines
• Pre/post processing of data

Tools

Prompt flow provides 3 basic tools:

• LLM: The LLM tool allows you to write custom prompts and leverage large language models to achieve specific goals, such as summarizing articles, generating customer support responses, and more.
• Python: The Python tool enables you to write custom Python functions to perform various tasks, such as fetching web pages, processing intermediate data, calling third-party APIs, and more.
• Prompt: The Prompt tool allows you to prepare a prompt as a string for more complex use cases or for use in conjunction with other prompt tools or python tools.

More tools Copy Chapter

Our partners also contributes other useful tools for advanced scenarios, here are some links:

• Vector DB Lookup: vector search tool that allows users to search top k similar vectors from vector database.
• Faiss Index Lookup: querying within a user-provided Faiss-based vector store.

Custom tools Copy Chapter

You can create your own tools that can be shared with your team or anyone in the world.
Learn more on Create and Use Tool Package

Next steps Copy Chapter

For more information on the available tools and their usage, visit the our reference doc.

Concepts Concept Variants (`docs/concepts/concept-variants.md`) Copy Section

With prompt flow, you can use variants to tune your prompt. In this article, you'll learn the prompt flow variants concept.

Variants

A variant refers to a specific version of a tool node that has distinct settings. Currently, variants are supported only in the LLM tool. For example, in the LLM tool, a new variant can represent either a different prompt content or different connection settings.

Suppose you want to generate a summary of a news article. You can set different variants of prompts and settings like this:

By utilizing different variants of prompts and settings, you can explore how the model responds to various inputs and outputs, enabling you to discover the most suitable combination for your requirements.

Benefits of using variants Copy Chapter

• Enhance the quality of your LLM generation: By creating multiple variants of the same LLM node with diverse prompts and configurations, you can identify the optimal combination that produces high-quality content aligned with your needs.
• Save time and effort: Even slight modifications to a prompt can yield significantly different results. It's crucial to track and compare the performance of each prompt version. With variants, you can easily manage the historical versions of your LLM nodes, facilitating updates based on any variant without the risk of forgetting previous iterations. This saves you time and effort in managing prompt tuning history.
• Boost productivity: Variants streamline the optimization process for LLM nodes, making it simpler to create and manage multiple variations. You can achieve improved results in less time, thereby increasing your overall productivity.
• Facilitate easy comparison: You can effortlessly compare the results obtained from different variants side by side, enabling you to make data-driven decisions regarding the variant that generates the best outcomes.

Next steps Copy Chapter

• Tune prompts with variants

Concepts Design Principles (`docs/concepts/design-principles.md`) Copy Section

Design principles

When we started this project, LangChain already became popular esp. after the ChatGPT launch. One of the questions we’ve been asked is what’s the difference between prompt flow and LangChain. This article is to elucidate the reasons for building prompt flow and the deliberate design choices we have made. To put it succinctly, prompt flow is a suite of development tools for you to build LLM apps with a strong emphasis of quality through experimentations, not a framework - which LangChain is.

While LLM apps are mostly in exploration stage, Microsoft started in this area a bit earlier and we’ve had the opportunity to observe how developers are integrating LLMs into existing systems or build new applications. These invaluable insights have shaped the fundamental design principles of prompt flow.

1. Expose the prompts vs. hiding them Copy Chapter

The core essence of LLM applications lies in the prompts themselves, at least for today. When developing a reasonably complex LLM application, the majority of development work should be “tuning” the prompts (note the intentional use of the term "tuning," which we will delve into further later on). Any framework or tool trying to help in this space should focus on making prompt tuning easier and more straightforward. On the other hand, prompts are very volatile, it's unlikely to write a single prompt that can work across different models or even different version of same models. Building a successful LLM-based application, you have to understand every prompt introduced, so that you can tune it when necessary. LLM is simply not powerful or deterministic enough that you can use a prompt written by others like you use libraries in traditional programming languages.

In this context, any design that tries to provide a smart function or agent by encapsulating a few prompts in a library is unlikely to yield favorable results in real-world scenarios. And hiding prompts inside a library’s code base only makes it’s hard for people to improve or tailor the prompts to suit their specific needs.

Prompt flow, being positioned as a tool, refrains from wrapping any prompts within its core codebase. The only place you will see prompts are our sample flows, which are, of course, available for adoption and utilization. Every prompt should be authored and controlled by the developers themselves, rather than relying on us.

2. A new way of work Copy Chapter

LLMs possess remarkable capabilities that enable developers to enhance their applications without delving deep into the intricacies of machine learning. In the meantime, LLMs make these apps more stochastic, which pose new challenges to application development. Merely asserting "no exception" or "result == x" in gated tests is no longer sufficient. Adopting a new methodology and employing new tools becomes imperative to ensure the quality of LLM applications — an entirely novel way of working is required.

At the center of this paradigm shift is evaluation, a term frequently used in machine learning space, refers to the process of assessing the performance and quality of a trained model. It involves measuring how well the model performs on a given task or dataset, which plays a pivotal role in understanding the model's strengths, weaknesses, and overall effectiveness. Evaluation metrics and techniques vary depending on the specific task and problem domain. Some common metrics include accuracy, precision and recall, you probably already familiar with. Now the LLM apps share similarities with machine learning models, they requires an evaluation-centric approach integrated into the development workflow, with a robust set of metrics and evaluation forming the foundation for ensuring the quality of LLM applications.

Prompt flow offers a range of tools to streamline the new way of work:

• Develop your evaluation program as Evaluation flow to calculate metrics for your app/flow, learn from our sample evaluation flows.
• Iterate on your application flow and run evaluation flows via the SDK/CLI, allowing you to compare metrics and choose the optimal candidate for release. These iterations include trying different prompts, different LLM parameters like temperature etc. - this is referred as “tuning” process earlier, or sometime referred as experimentation.
• Integrate the evaluation into your CI/CD pipeline, aligning the assertions in your gated tests with the selected metrics.

Prompt flow introduces two conceptual components to facilitate this workflow:

• Evaluation flow: a flow type that indicates this flow is not for deploy or integrate into your app, it’s for evaluating an app/flow performance.
• Run: every time you run your flow with data, or run an evaluation on the output of a flow, a Run object is created to manage the history and allow for comparison and additional analysis.

While new concepts introduce additional cognitive load, we firmly believe they hold greater importance compared to abstracting different LLM APIs or vector database APIs.

3. Optimize for “visibility” Copy Chapter

There are quite some interesting application patterns emerging because of LLMs, like Retrieval Augmented Generation (RAG), ReAct and more. Though how LLMs work may remain enigmatic to many developers, how LLM apps work is not - they essentially involve a series of calls to external services such as LLMs, databases, and search engines, all glued together. Architecturally there isn’t much new, patterns like RAG and ReAct are both straightforward to implement once a developer understands what they are - plain Python programs with API calls to external services can totally serve the purpose effectively.

By observing many internal use cases, we learned that deeper insight into the detail of the execution is critical. Establishing a systematic method for tracking interactions with external systems is one of design priority. Consequently, We adopted an unconventional approach - prompt flow has a YAML file describing how function calls (we call them Tools) are executed and connected into a Directed Acyclic Graph (DAG).

This approach offers several key benefits, primarily centered around enhanced visibility:

1. During development, your flow can be visualized in an intelligible manner, enabling clear identification of any faulty components. As a byproduct, you obtain an architecturally descriptive diagram that can be shared with others.
2. Each node in the flow has it’s internal detail visualized in a consistent way.
3. Single nodes can be individually run or debugged without the need to rerun previous nodes.

The emphasis on visibility in prompt flow's design helps developers to gain a comprehensive understanding of the intricate details of their applications. This, in turn, empowers developers to engage in effective troubleshooting and optimization.

Despite there're some control flow features like "activate-when" to serve the needs of branches/switch-case, we do not intend to make Flow itself Turing-complete. If you want to develop an agent which is fully dynamic and guided by LLM, leveraging Semantic Kernel together with prompt flow would be a favorable option.

Dev Dev Setup (`docs/dev/dev_setup.md`) Copy Section

Dev Setup

Set up process Copy Chapter

Select either Conda or Poetry to set up your development environment.

1. Conda environment setup

• First create a new conda environment. Please specify python version as 3.9/3.10/3.11.
  conda create -n <env_name> python=3.9.
• Activate the env you created.
• In root folder, run python scripts/dev-setup/main.py to install the packages and dependencies; if you are using Visual Studio Code, it is recommended to add --vscode (which is python scripts/dev-setup/main.py --vscode) to enable VS Code to recognize the packages.

2. Poetry environment setup

• Install poetry. Please specify python version as 3.9/3.10/3.11.
• Each folder under src (except the promptflow folder) is a separate package, so you need to install the dependencies for each package.
  • poetry install -C promptflow-core -E <extra> --with dev,test
  • poetry install -C promptflow-devkit -E <extra> --with dev,test
  • poetry install -C promptflow-azure -E <extra> --with dev,test

How to run tests Copy Chapter

Set up your secrets Copy Section

dev-connections.json.example is a template about connections provided in src/promptflow. You can follow these steps to refer to this template to configure your connection for the test cases:

1. cd ./src/promptflow
2. Run the command cp dev-connections.json.example connections.json;
3. Replace the values in the json file with your connection info;
4. Set the environment PROMPTFLOW_CONNECTIONS='connections.json';

After above setup process is finished. You can use pytest command to run test, for example in root folder you can:

Run tests via command Copy Section

1. Conda environment

• Run all tests under a folder: pytest src/promptflow/tests -v, pytest src/promptflow-devkit/tests -v
• Run a single test: pytest src/promptflow/tests/promptflow_test/e2etests/test_executor.py::TestExecutor::test_executor_basic_flow -v

2. Poetry environment: there is limitation for running tests in src/promptflow folder, you can only run tests under other package folders.

• for example: under the target folder promptflow-devkit, you can run poetry run pytest tests/sdk_cli_test -v

Run tests in VSCode Copy Section

Conda environment

1. Set up your python interperter

• Open the Command Palette (Ctrl+Shift+P) and select Python: Select Interpreter.

• Select existing conda env which you created previously.

2. Set up your test framework and directory

• Open the Command Palette (Ctrl+Shift+P) and select Python: Configure Tests.

• Select pytest as test framework.

• Select Root directory as test directory.

3. Exclude specific test folders.

You can exclude specific test folders if you don't have some extra dependency to avoid VS Code's test discovery fail.
For example, if you don't have azure dependency, you can exclude sdk_cli_azure_test.
Open .vscode/settings.json, write "--ignore=src/promptflow/tests/sdk_cli_azure_test" to "python.testing.pytestArgs".

4. Click the Run Test button on the left

Run tests in pycharm Copy Section

1. Set up your pycharm python interpreter

2. Select existing conda env which you created previously

3. Run test, right-click the test name to run, or click the green arrow button on the left.

Poetry environment

VSCode could pick up the correct environment automatically if you open vscode/pycharm under the package folders.

There are some limitations currently, intellisense may not work properly in poetry environment.

PyCharm behaves differently from VSCode, it will automatically picks up the correct environment.

How to write docstring Copy Chapter

A clear and consistent API documentation is crucial for the usability and maintainability of our codebase. Please refer to API Documentation Guidelines to learn how to write docstring when developing the project.

How to write tests Copy Chapter

• Put all test data/configs under src/promptflow/tests/test_configs.
• Write unit tests:
  • Flow run: src/promptflow/tests/sdk_cli_test/unittest/
  • Flow run in azure: src/promptflow/tests/sdk_cli_azure_test/unittest/
• Write e2e tests:
  • Flow run: src/promptflow/tests/sdk_cli_test/e2etests/
  • Flow run in azure: src/promptflow/tests/sdk_cli_azure_test/e2etests/
• Test file name and the test case name all start with test_.
• A basic test example, see test_connection.py.

Test structure Copy Section

Tests are under corresponding source folder, and test_configs are shared among different test folders:

• src/promptflow/
  • test_configs/
    • connections/
    • datas/
    • flows/
    • runs/
    • wrong_flows/
    • wrong_tools/
• src/promptflow-core/
  • tests/
    • core/ # Basic test with promptflow-core installed.
      • e2etests/
      • unittests/
    • azureml-serving/ # Test with promptflow-core[azureml-serving] installed.
      • e2etests/
      • unittests/
    • executor-service/ # Test with promptflow-core[executor-service] installed.
      • e2etests/
      • unittests/
• src/promptflow-devkit/
  • tests/
    • sdk_cli_tests/
      • e2etests/
      • unittests/
• src/promptflow-azure/
  • tests/
    • sdk_cli_azure_test/
      • e2etests/
      • unittests/

Principal #1: Put the tests in the same folder as the code they are testing, to ensure code can work within minor environment requirements.

For example, you write code requires basic promptflow-core package, then put the tests in promptflow-core/tests/core, DO NOT put it in the promptflow-devkit or promptflow-azure.

Principal #2: Setup separate workflow for tests with extra-requires.

For example, you want to test promptflow-core[azureml-serving], then add a new test folder promptflow-core/tests/azureml-serving to test the azure related code,
and add new test steps and environment setup step into promptflow-core-test.yml for that folder. DO NOT update the environment of promptflow-core basic test directly.

Record and replay tests Copy Section

Please refer to Replay End-to-End Tests to learn how to record and replay tests.

Dev Documentation Guidelines (`docs/dev/documentation_guidelines.md`) Copy Section

Promptflow Reference Documentation Guide

Overview Copy Chapter

This guide describes how to author Python docstrings for promptflow public interfaces. See our doc site at Promptflow API reference documentation.

Principles Copy Chapter

• Coverage: Every public object must have a docstring. For private objects, docstrings are encouraged but not required.
• Style: All docstrings should be written in Sphinx style noting all types and if any exceptions are raised.
• Relevance: The documentation is up-to-date and relevant to the current version of the product.
• Clarity: The documentation is written in clear, concise language that is easy to understand.
• Consistency: The documentation has a consistent format and structure, making it easy to navigate and follow.

How to write the docstring Copy Chapter

First please read through Sphinx style to have a basic understanding of sphinx style docstring.

Write class docstring Copy Section

Let's start with a class example:

from typing import Dict, Optional, Union
from promptflow.client import PFClient

class MyClass:
    """One-line summary of the class.

    More detailed explanation of the class. May include below notes, admonitions, code blocks.

    .. note::

        Here are some notes to show, with a nested python code block:

        .. code-block:: python

            from promptflow import MyClass, PFClient
            obj = MyClass(PFClient())

    .. admonition:: [Title of the admonition]

        Here are some admonitions to show.

    :param client: Descrition of the client.
    :type client: ~promptflow.PFClient
    :param param_int: Description of the parameter.
    :type param_int: Optional[int]
    :param param_str: Description of the parameter.
    :type param_str: Optional[str]
    :param param_dict: Description of the parameter.
    :type param_dict: Optional[Dict[str, str]]
    """
    def __init__(
        client: PFClient,
        param_int: Optional[int] = None,
        param_str: Optional[str] = None,
        param_dict: Optional[Dict[str, str]] = None,
    ) -> None:
        """No docstring for __init__, it should be written in class definition above."""
        ...

Notes:

1. One-line summary is required. It should be clear and concise.

2. Detailed explanation is encouraged but not required. This part may or may not include notes, admonitions and code blocks.

   • The format like .. note:: is called directive. Directives are a mechanism to extend the content of reStructuredText. Every directive declares a block of content with specific role. Start a new line with .. directive_name:: to use the directive.
   • The directives used in the sample(note/admonition/code-block) should be enough for basic usage of docstring in our project. But you are welcomed to explore more Directives.

3. Parameter description and type is required.

   • A pair of :param [ParamName]: and :type [ParamName]: is required.
   • If the type is a promptflow public class, use the full path to the class and prepend it with a "~". This will create a link when the documentation is rendered on the doc site that will take the user to the class reference documentation for more information.
     text Copy

     :param client: Descrition of the client.
     :type client: ~promptflow.PFClient

   • Use Union/Optional when appropriate in function declaration. And use the same annotaion after :type [ParamName]:
     text Copy

     :type param_int: Optional[int]

4. For classes, include docstring in definition only. If you include a docstring in both the class definition and the constructor (init method) docstrings, it will show up twice in the reference docs.

5. Constructors (def __init__) should return None, per PEP 484 standards.

6. To create a link for promptflow class on our doc site. ~promptflow.xxx.MyClass alone only works after :type [ParamName] and :rtype:. If you want to achieve the same effect in docstring summary, you should use it with :class::

   python Copy

   """
   An example to achieve link effect in summary for :class:`~promptflow.xxx.MyClass`
   For function, use :meth:`~promptflow.xxx.my_func`
   """

7. There are some tricks to highlight the content in your docstring:

   • Single backticks (`): Single backticks are used to represent inline code elements within the text. It is typically used to highlight function names, variable names, or any other code elements within the documentation.
   • Double backticks(``): Double backticks are typically used to highlight a literal value.

8. If there are any class level constants you don't want to expose to doc site, make sure to add _ in front of the constant to hide it.

Write function docstring Copy Section

from typing import Optional

def my_method(param_int: Optional[int] = None) -> int:
    """One-line summary

    Detailed explanations.

    :param param_int: Description of the parameter.
    :type param_int: int
    :raises [ErrorType1]: [ErrorDescription1]
    :raises [ErrorType2]: [ErrorDescription2]
    :return: Description of the return value.
    :rtype: int
    """
    ...

In addition to class docstring notes:

1. Function docstring should include return values.
   • If return type is promptflow class, we should also use ~promptflow.xxx.[ClassName].
2. Function docstring should include exceptions that may be raised in this function.
   • If exception type is PromptflowException, use ~promptflow.xxx.[ExceptionName]
   • If multiple exceptions are raised, just add new lines of :raises, see the example above.

How to build doc site locally Copy Chapter

You can build the documentation site locally to preview the final effect of your docstring on the rendered site. This will provide you with a clear understanding of how your docstring will appear on our site once your changes are merged into the main branch.

1. Setup your dev environment, see dev_setup for details. Sphinx will load all source code to process docstring.
   • Skip this step if you just want to build the doc site without reference doc, but do remove -WithReferenceDoc from the command in step 3.
2. Install langchain package since it is used in our code but not covered in dev_setup.
3. Open a powershell, activate the conda env and navigate to <repo-root>/scripts/docs , run doc_generation.ps1:
   pwsh Copy

   cd scripts\docs
   .\doc_generation.ps1 -WithReferenceDoc -WarningAsError

   • For the first time you execute this command, it will take some time to install sphinx dependencies. After the initial installation, next time you can add param -SkipInstall to above command to save some time for dependency check.
4. Check warnings/errors in the build log, fix them if any, then build again.
5. Open scripts/docs/_build/index.html to preview the local doc site.

Additional comments Copy Chapter

• Utilities: The autoDocstring VSCode extension or GitHub Copilot can help autocomplete in this style for you.

• Advanced principles

  • Accuracy: The documentation accurately reflects the features and functionality of the product.
  • Completeness: The documentation covers all relevant features and functionality of the product.
  • Demonstration: Every docstring should include an up-to-date code snippet that demonstrates how to use the product effectively.

References Copy Chapter

• AzureML v2 Reference Documentation Guide
• Azure SDK for Python documentation guidelines
• How to document a Python API

Dev Replay E2e Test (`docs/dev/replay-e2e-test.md`) Copy Section

Replay end-to-end tests

• This document introduces replay tests for those located in sdk_cli_azure_test and sdk_cli_test.
• The primary purpose of replay tests is to avoid the need for credentials, Azure workspaces, OpenAI tokens, and to directly test prompt flow behavior.
• Although there are different techniques behind recording/replaying, there are some common steps to run the tests in replay mode.
• The key handle of replay tests is the environment variable PROMPT_FLOW_TEST_MODE.

How to run tests in replay mode Copy Chapter

After cloning the full repo and setting up the proper test environment following dev_setup.md, run the following command in the root directory of the repo:

1. If you have changed/affected tests in sdk_cli_test : Copy or rename the file dev-connections.json.example to connections.json in the same folder.
   • There are some python package version requirements for running the replay/record tests. It needs pydantic >= 2.0.0.
2. In your Python environment, set the environment variable PROMPT_FLOW_TEST_MODE to 'replay' and run the test(s).

These tests should work properly without any real connection settings.

Test modes Copy Chapter

There are 3 representative values of the environment variable PROMPT_FLOW_TEST_MODE

• live: Tests run against the real backend, which is the way traditional end-to-end tests do.
• record: Tests run against the real backend, and network traffic will be sanitized (filter sensitive and unnecessary requests/responses) and recorded to local files (recordings).
• replay: There is no real network traffic between SDK/CLI and the backend, tests run against local recordings.

Supported modules Copy Chapter

• promptflow-devkit
• promptflow-azure

Update test recordings Copy Chapter

To record a test, don’t forget to clone the full repo and set up the proper test environment following dev_setup.md:

1. Ensure you have installed dev version of promptflow-recording package.
   • If it is not installed, run pip install -e src/promptflow-recording in the root directory of the repo.
2. Prepare some data.
   • If you have changed/affected tests in sdk_cli_test: Copy or rename the file dev-connections.json.example to connections.json in the same folder.
   • If you have changed/affected tests in sdk_cli_azure_test: prepare your Azure ML workspace, make sure your Azure CLI logged in, and set the environment variable PROMPT_FLOW_SUBSCRIPTION_ID, PROMPT_FLOW_RESOURCE_GROUP_NAME, PROMPT_FLOW_WORKSPACE_NAME and PROMPT_FLOW_RUNTIME_NAME (if needed) pointing to your workspace.
3. Record the test.
   • Specify the environment variable PROMPT_FLOW_TEST_MODE to 'record'. If you have a .env file, we recommend specifying it there. Here is an example .env file. Then, just run the test that you want to record.
4. Once the test completed.
   • If you have changed/affected tests in sdk_cli_azure_test: There should be one new YAML file located in Azure recording folder, containing the network traffic of the test.
   • If you have changed/affected tests in sdk_cli_test: There may be changes in the folder Local recording folder. Don’t worry if there are no changes, because similar LLM calls may have been recorded before.

Techniques behind replay test Copy Chapter

Sdk_cli_azure_test Copy Section

End-to-end tests for pfazure aim to test the behavior of the PromptFlow SDK/CLI as it interacts with the service. This process can be time-consuming, error-prone, and require credentials (which are unavailable to pull requests from forked repositories); all of these go against our intention for a smooth development experience.

Therefore, we introduce replay tests, which leverage VCR.py to record all required network traffic to local files and replay during tests. In this way, we avoid the need for credentials, speed up, and stabilize the test process.

Sdk_cli_test Copy Section

sdk_cli_test often doesn’t use a real backend. It will directly invokes LLM calls from localhost. Thus the key target of replay tests is to avoid the need for OpenAI tokens. If you have OpenAI / Azure OpenAI tokens yourself, you can try recording the tests. Record Storage will not record your own LLM connection, but only the inputs and outputs of the LLM calls.

There are also limitations. Currently, recorded calls are:

• AzureOpenAI calls
• OpenAI calls
• tool name "fetch_text_content_from_url" and tool name "my_python_tool"

Dev Rotate Aoai Keys (`docs/dev/rotate_aoai_keys.md`) Copy Section

Here is a checklist to rotate the AOAI keys:

1. Go to the well known URL of the AOAI service.
2. Check the secondary keys (This is the key used in the following days).
3. Change promptflow-eastus2euap Connections
4. Also Change promptflow-eastus Connections
5. Save the key in the well known key vault.
6. Save the key in the github secrets, to mask the key.
7. Rotate the DEPRECATED the leaked key in the AOAI service.

How To Guides Enable Streaming Mode (`docs/how-to-guides/enable-streaming-mode.md`) Copy Section

Use streaming endpoints deployed from prompt flow

In prompt flow, you can deploy flow as REST endpoint for real-time inference.

When consuming the endpoint by sending a request, the default behavior is that the online endpoint will keep waiting until the whole response is ready, and then send it back to the client. This can cause a long delay for the client and a poor user experience.

To avoid this, you can use streaming when you consume the endpoints. Once streaming enabled, you don't have to wait for the whole response ready. Instead, the server will send back the response in chunks as they are generated. The client can then display the response progressively, with less waiting time and more interactivity.

This article will describe the scope of streaming, how streaming works, and how to consume streaming endpoints.

Create a streaming enabled flow Copy Chapter

If you want to use the streaming mode, you need to create a flow that has a node that produces a string generator as the flow’s output. A string generator is an object that can return one string at a time when requested. You can use the following types of nodes to create a string generator:

• LLM node: This node uses a large language model to generate natural language responses based on the input.
  jinja Copy

  {# Sample prompt template for LLM node #}
  
  # system:
  You are a helpful assistant.
  
  # user:
  {{question}}

• Python tools node: This node allows you to write custom Python code that can yield string outputs. You can use this node to call external APIs or libraries that support streaming. For example, you can use this code to echo the input word by word:
  python Copy

  from promptflow.core import tool
  
  # Sample code echo input by yield in Python tool node
  
  @tool
  def my_python_tool(paragraph: str) -> str:
      yield "Echo: "
      for word in paragraph.split():
          yield word + " "

In this guide, we will use the "Chat with Wikipedia" sample flow as an example. This flow processes the user’s question, searches Wikipedia for relevant articles, and answers the question with information from the articles. It uses streaming mode to show the progress of the answer generation.

Deploy the flow as an online endpoint Copy Chapter

To use the streaming mode, you need to deploy your flow as an online endpoint. This will allow you to send requests and receive responses from your flow in real time.

Follow this guide to deploy your flow as an online endpoint.

[!NOTE]

You can follow this document to deploy an online endpoint.
Please deploy with runtime environment version later than version 20230816.v10.
You can check your runtime version and update runtime in the run time detail page.

Understand the streaming process Copy Chapter

When you have an online endpoint, the client and the server need to follow specific principles for content negotiation to utilize the streaming mode:

Content negotiation is like a conversation between the client and the server about the preferred format of the data they want to send and receive. It ensures effective communication and agreement on the format of the exchanged data.

To understand the streaming process, consider the following steps:

• First, the client constructs an HTTP request with the desired media type included in the Accept header. The media type tells the server what kind of data format the client expects. It's like the client saying, "Hey, I'm looking for a specific format for the data you'll send me. It could be JSON, text, or something else." For example, application/json indicates a preference for JSON data, text/event-stream indicates a desire for streaming data, and */* means the client accepts any data format.

  [!NOTE]

  If a request lacks an Accept header or has empty Accept header, it implies that the client will accept any media type in response. The server treats it as */*.

• Next, the server responds based on the media type specified in the Accept header. It's important to note that the client may request multiple media types in the Accept header, and the server must consider its capabilities and format priorities to determine the appropriate response.

  • First, the server checks if text/event-stream is explicitly specified in the Accept header:
    • For a stream-enabled flow, the server returns a response with a Content-Type of text/event-stream, indicating that the data is being streamed.
    • For a non-stream-enabled flow, the server proceeds to check for other media types specified in the header.
  • If text/event-stream is not specified, the server then checks if application/json or */* is specified in the Accept header:
    • In such cases, the server returns a response with a Content-Type of application/json, providing the data in JSON format.
  • If the Accept header specifies other media types, such as text/html:
    • The server returns a 424 response with a PromptFlow runtime error code UserError and a runtime HTTP status 406, indicating that the server cannot fulfill the request with the requested data format.

    Note: Please refer handle errors for details.

• Finally, the client checks the Content-Type response header. If it is set to text/event-stream, it indicates that the data is being streamed.

Let’s take a closer look at how the streaming process works. The response data in streaming mode follows the format of server-sent events (SSE).

The overall process works as follows:

0. The client sends a message to the server. Copy Section

POST https://<your-endpoint>.inference.ml.azure.com/score
Content-Type: application/json
Authorization: Bearer <key or token of your endpoint>
Accept: text/event-stream

{
    "question": "Hello",
    "chat_history": []
}

[!NOTE]

The Accept header is set to text/event-stream to request a stream response.

1. The server sends back the response in streaming mode. Copy Section

HTTP/1.1 200 OK
Content-Type: text/event-stream; charset=utf-8
Connection: close
Transfer-Encoding: chunked

data: {"answer": ""}

data: {"answer": "Hello"}

data: {"answer": "!"}

data: {"answer": " How"}

data: {"answer": " can"}

data: {"answer": " I"}

data: {"answer": " assist"}

data: {"answer": " you"}

data: {"answer": " today"}

data: {"answer": " ?"}

data: {"answer": ""}

Note that the Content-Type is set to text/event-stream; charset=utf-8, indicating the response is an event stream.

The client should decode the response data as server-sent events and display them incrementally. The server will close the HTTP connection after all the data is sent.

Each response event is the delta to the previous event. It is recommended for the client to keep track of the merged data in memory and send them back to the server as chat history in the next request.

2. The client sends another chat message, along with the full chat history, to the server. Copy Section

POST https://<your-endpoint>.inference.ml.azure.com/score
Content-Type: application/json
Authorization: Bearer <key or token of your endpoint>
Accept: text/event-stream

{
    "question": "Glad to know you!",
    "chat_history": [
        {
            "inputs": {
                "question": "Hello"
            },
            "outputs": {
                "answer": "Hello! How can I assist you today?"
            }
        }
    ]
}

3. The server sends back the answer in streaming mode. Copy Section

HTTP/1.1 200 OK
Content-Type: text/event-stream; charset=utf-8
Connection: close
Transfer-Encoding: chunked

data: {"answer": ""}

data: {"answer": "Nice"}

data: {"answer": " to"}

data: {"answer": " know"}

data: {"answer": " you"}

data: {"answer": " too"}

data: {"answer": "!"}

data: {"answer": " Is"}

data: {"answer": " there"}

data: {"answer": " anything"}

data: {"answer": " I"}

data: {"answer": " can"}

data: {"answer": " help"}

data: {"answer": " you"}

data: {"answer": " with"}

data: {"answer": "?"}

data: {"answer": ""}

4. The chat continues in a similar way. Copy Section

Handle errors Copy Chapter

The client should check the HTTP response code first. See this table for common error codes returned by online endpoints.

If the response code is "424 Model Error", it means that the error is caused by the model’s code. The error response from a PromptFlow model always follows this format:

{
  "error": {
    "code": "UserError",
    "message": "Media type text/event-stream in Accept header is not acceptable. Supported media type(s) - application/json",
  }
}

• It is always a JSON dictionary with only one key "error" defined.
• The value for "error" is a dictionary, containing "code", "message".
• "code" defines the error category. Currently, it may be "UserError" for bad user inputs and "SystemError" for errors inside the service.
• "message" is a description of the error. It can be displayed to the end user.

How to consume the server-sent events Copy Chapter

Consume using Python Copy Section

In this sample usage, we are using the SSEClient class. This class is not a built-in Python class and needs to be installed separately. You can install it via pip:

pip install sseclient-py

A sample usage would like:

import requests
from sseclient import SSEClient
from requests.exceptions import HTTPError

try:
    response = requests.post(url, json=body, headers=headers, stream=stream)
    response.raise_for_status()

    content_type = response.headers.get('Content-Type')
    if "text/event-stream" in content_type:
        client = SSEClient(response)
        for event in client.events():
            # Handle event, i.e. print to stdout
    else:
        # Handle json response

except HTTPError:
    # Handle exceptions

Consume using JavaScript Copy Section

There are several libraries to consume server-sent events in JavaScript. Here is one of them as an example.

A sample chat app using Python Copy Chapter

Here is a sample chat app written in Python.
(Click here to view the source code.)

Advance usage - hybrid stream and non-stream flow output Copy Chapter

Sometimes, you may want to get both stream and non-stream results from a flow output. For example, in the “Chat with Wikipedia” flow, you may want to get not only LLM’s answer, but also the list of URLs that the flow searched. To do this, you need to modify the flow to output a combination of stream LLM’s answer and non-stream URL list.

In the sample "Chat With Wikipedia" flow, the output is connected to the LLM node augmented_chat. To add the URL list to the output, you need to add an output field with the name url and the value ${get_wiki_url.output}.

The output of the flow will be a non-stream field as the base and a stream field as the delta. Here is an example of request and response.

0. The client sends a message to the server. Copy Section

POST https://<your-endpoint>.inference.ml.azure.com/score
Content-Type: application/json
Authorization: Bearer <key or token of your endpoint>
Accept: text/event-stream
{
    "question": "When was ChatGPT launched?",
    "chat_history": []
}

1. The server sends back the answer in streaming mode. Copy Section

HTTP/1.1 200 OK
Content-Type: text/event-stream; charset=utf-8
Connection: close
Transfer-Encoding: chunked

data: {"url": ["https://en.wikipedia.org/w/index.php?search=ChatGPT", "https://en.wikipedia.org/w/index.php?search=GPT-4"]}

data: {"answer": ""}

data: {"answer": "Chat"}

data: {"answer": "G"}

data: {"answer": "PT"}

data: {"answer": " was"}

data: {"answer": " launched"}

data: {"answer": " on"}

data: {"answer": " November"}

data: {"answer": " "}

data: {"answer": "30"}

data: {"answer": ","}

data: {"answer": " "}

data: {"answer": "202"}

data: {"answer": "2"}

data: {"answer": "."}

data: {"answer": " \n\n"}

...

data: {"answer": "PT"}

data: {"answer": ""}

2. The client sends another chat message, along with the full chat history, to the server. Copy Section

POST https://<your-endpoint>.inference.ml.azure.com/score
Content-Type: application/json
Authorization: Bearer <key or token of your endpoint>
Accept: text/event-stream
{
    "question": "When did OpenAI announce GPT-4? How long is it between these two milestones?",
    "chat_history": [
        {
            "inputs": {
                "question": "When was ChatGPT launched?"
            },
            "outputs": {
                "url": [
                    "https://en.wikipedia.org/w/index.php?search=ChatGPT",
                    "https://en.wikipedia.org/w/index.php?search=GPT-4"
                ],
                "answer": "ChatGPT was launched on November 30, 2022. \n\nSOURCES: https://en.wikipedia.org/w/index.php?search=ChatGPT"
            }
        }
    ]
}

3. The server sends back the answer in streaming mode. Copy Section

HTTP/1.1 200 OK
Content-Type: text/event-stream; charset=utf-8
Connection: close
Transfer-Encoding: chunked

data: {"url": ["https://en.wikipedia.org/w/index.php?search=Generative pre-trained transformer ", "https://en.wikipedia.org/w/index.php?search=Microsoft "]}

data: {"answer": ""}

data: {"answer": "Open"}

data: {"answer": "AI"}

data: {"answer": " released"}

data: {"answer": " G"}

data: {"answer": "PT"}

data: {"answer": "-"}

data: {"answer": "4"}

data: {"answer": " in"}

data: {"answer": " March"}

data: {"answer": " "}

data: {"answer": "202"}

data: {"answer": "3"}

data: {"answer": "."}

data: {"answer": " Chat"}

data: {"answer": "G"}

data: {"answer": "PT"}

data: {"answer": " was"}

data: {"answer": " launched"}

data: {"answer": " on"}

data: {"answer": " November"}

data: {"answer": " "}

data: {"answer": "30"}

data: {"answer": ","}

data: {"answer": " "}

data: {"answer": "202"}

data: {"answer": "2"}

data: {"answer": "."}

data: {"answer": " The"}

data: {"answer": " time"}

data: {"answer": " between"}

data: {"answer": " these"}

data: {"answer": " two"}

data: {"answer": " milestones"}

data: {"answer": " is"}

data: {"answer": " approximately"}

data: {"answer": " "}

data: {"answer": "3"}

data: {"answer": " months"}

data: {"answer": ".\n\n"}

...

data: {"answer": "Chat"}

data: {"answer": "G"}

data: {"answer": "PT"}

data: {"answer": ""}

How To Guides Execute Flow As A Function (`docs/how-to-guides/execute-flow-as-a-function.md`) Copy Section

Execute flow as a function

:::{admonition} Experimental feature
This is an experimental feature, and may change at any time. Learn more.
:::

Overview Copy Chapter

Promptflow allows you to load a flow and use it as a function in your code.
This feature is useful when building a service on top of a flow, reference here for a simple example service with flow function consumption.

Load and invoke the flow function Copy Chapter

To use the flow-as-function feature, you first need to load a flow using the load_flow function.
Then you can consume the flow object like a function by providing key-value arguments for it.

f = load_flow("../../examples/flows/standard/web-classification/")
f(url="sample_url")

Config the flow with context Copy Chapter

You can overwrite some flow configs before flow function execution by setting flow.context.

Load flow as a function with in-memory connection override Copy Section

By providing a connection object to flow context, flow won't need to get connection in execution time, which can save time when for cases where flow function need to be called multiple times.

from promptflow.entities import AzureOpenAIConnection

connection_obj = AzureOpenAIConnection(
    name=conn_name,
    api_key=api_key,
    api_base=api_base,
    api_type="azure",
    api_version=api_version,
)
# no need to create the connection object.  
f.context = FlowContext(
    connections={"classify_with_llm": {"connection": connection_obj}}
)

Local flow as a function with flow inputs override Copy Section

By providing overrides, the original flow dag will be updated in execution time.

f.context = FlowContext(
    # node "fetch_text_content_from_url" will take inputs from the following command instead of from flow input
    overrides={"nodes.fetch_text_content_from_url.inputs.url": sample_url},
)

Note, the overrides are only doing YAML content replacement on original flow.dag.yaml.
If the flow.dag.yaml become invalid after overrides, validation error will be raised when executing.

Load flow as a function with streaming output Copy Section

After set streaming in flow context, the flow function will return an iterator to stream the output.

f = load_flow(source="../../examples/flows/chat/chat-basic/")
f.context.streaming = True
result = f(
    chat_history=[
        {
            "inputs": {"chat_input": "Hi"},
            "outputs": {"chat_output": "Hello! How can I assist you today?"},
        }
    ],
    question="How are you?",
)


answer = ""
# the result will be a generator, iterate it to get the result
for r in result["answer"]:
    answer += r

Reference our sample for usage.

Flow with multiple overrides Copy Chapter

Note: the flow context configs may affect each other in some cases. For example, using connection & overrides to override same node.
The behavior is undefined for those scenarios. Pleas avoid such usage.

# overriding `classify_with_llm`'s connection and inputs in the same time will lead to undefined behavior.
f.context = FlowContext(
    connections={"classify_with_llm": {"connection": connection_obj}},
    overrides={"nodes.classify_with_llm.inputs.url": sample_url}
)

Next steps Copy Chapter

Learn more about:

• Flow as a function sample
• Deploy a flow

How To Guides Faq (`docs/how-to-guides/faq.md`) Copy Section

Frequency asked questions (FAQ)

General Copy Chapter

Stable vs experimental Copy Section

Prompt flow provides both stable and experimental features in the same SDK.

OpenAI 1.x support Copy Section

Please use the following command to upgrade promptflow for openai 1.x support:

pip install promptflow>=1.1.0
pip install promptflow-tools>=1.0.0

Note that the command above will upgrade your openai package a version later than 1.0.0,
which may introduce breaking changes to custom tool code.

Reach OpenAI migration guide for more details.

Promptflow 1.8.0 upgrade guide Copy Section

Before upgrading to promptflow version 1.8.0 or later, it's important to first uninstall any existing installations of promptflow and its sub-packages.
This ensures a clean installation of the new version without any conflicts.

pip uninstall -y promptflow promptflow-core promptflow-devkit promptflow-azure # uninstall promptflow and its sub-packages
pip install 'promptflow>=1.8.0' # install promptflow version 1.8.0 or later

Reason for 'pip install promptflow>=1.8.0' or 'pf upgrade' directly not working:

promptflow package has been split into multiple packages. When installing promptflow, you will get the following packages:

• promptflow:
  • promptflow-tracing: Tracing capability for promptflow.
  • promptflow-core: Core functionality to run flow.
  • promptflow-devkit: Development kit for promptflow.
  • promptflow-azure: Azure extra requires(promptflow[azure]) for promptflow to integrate with Azure.

When upgrading promptflow from an existing version to promptflow 1.8.0, pip will remove the old promptflow after installing promptflow subpackages,
which caused subpackage files got wrongly removed.

Troubleshooting Copy Chapter

Connection creation failed with StoreConnectionEncryptionKeyError Copy Section

Connection creation failed with StoreConnectionEncryptionKeyError: System keyring backend service not found in your operating system. See https://pypi.org/project/keyring/ to install requirement for different operating system, or 'pip install keyrings.alt' to use the third-party backend.

This error raised due to keyring can't find an available backend to store keys.
For example macOS Keychain and Windows Credential Locker
are valid keyring backends.

To resolve this issue, install the third-party keyring backend or write your own keyring backend, for example:
pip install keyrings.alt

For more detail about keyring third-party backend, please refer to 'Third-Party Backends' in keyring.

Pf visualize show error: "tcgetpgrp failed: Not a tty" Copy Section

If you are using WSL, this is a known issue for webbrowser under WSL; see this issue for more information. Please try to upgrade your WSL to 22.04 or later, this issue should be resolved.

If you are still facing this issue with WSL 22.04 or later, or you are not even using WSL, please open an issue to us.

Installed tool not appearing in VSCode Extension tool list Copy Section

After installing a tool package via pip install [tool-package-name], the new tool may not immediately appear in the tool list within the VSCode Extension, as shown below:

This is often due to outdated cache. To refresh the tool list and make newly installed tools visible:

1. Open the VSCode Extension window.

2. Bring up the command palette by pressing "Ctrl+Shift+P".

3. Type and select the "Developer: Reload Webviews" command.

4. Wait a moment for the tool list refreshing.

Reloading clears the previous cache and populates the tool list with any newly installed tools. So that the missing tools are now visible.

Set logging level Copy Section

Promptflow uses logging module to log messages. You can set logging level via environment variable PF_LOGGING_LEVEL, valid values includes CRITICAL, ERROR, WARNING, INFO, DEBUG, default to INFO.
Below is the serving logs after setting PF_LOGGING_LEVEL to DEBUG:

Compare to the serving logs with WARNING level:

Set logging format Copy Section

Promptflow uses the following log format and datetime format by default:

• Log format: %(asctime)s %(process)7d %(name)-18s %(levelname)-8s %(message)s
• Datetime format: %Y-%m-%d %H:%M:%S %z

You can customize the log format with the PF_LOG_FORMAT environment variable, and the datetime format with PF_LOG_DATETIME_FORMAT. These variables can also be defined in a flow yaml file or set directly in the environment.

Set environment variables Copy Section

Currently, promptflow supports the following environment variables:

PF_WORKER_COUNT

Effective for batch run only, count of parallel workers in batch run execution.

The default value is 4 (was 16 when promptflow<1.4.0)

Please take the following points into consideration when changing it:

1. The concurrency should not exceed the total data rows count. Otherwise, the execution may slow down due to additional time spent on process startup and shutdown.

2. High parallelism may cause the underlying API call to reach the rate limit of your LLM endpoint. In which case you can decrease the PF_WORKER_COUNT or increase the rate limit. Please refer to this doc on quota management. Then you can refer to this expression to set up the concurrency.

PF_WORKER_COUNT <= TPM * duration_seconds / token_count / 60

TPM: token per minute, capacity rate limit of your LLM endpoint

duration_seconds: single flow run duration in seconds

token_count: single flow run token count

For example, if your endpoint TPM (token per minute) is 50K, the single flow run takes 10k tokens and runs for 30s, pls do not set up PF_WORKER_COUNT bigger than 2. This is a rough estimation. Please also consider collboaration (teammates use the same endpoint at the same time) and tokens consumed in deployed inference endpoints, playground and other cases which might send request to your LLM endpoints.

PF_BATCH_METHOD

Valid for batch run only. Optional values: 'spawn', 'fork'.

spawn

1. The child processes will not inherit resources of the parent process, therefore, each process needs to reinitialize the resources required for the flow, which may use more system memory.

2. Starting a process is slow because it will take some time to initialize the necessary resources.

fork

1. Use the copy-on-write mechanism, the child processes will inherit all the resources of the parent process, thereby using less system memory.

2. The process starts faster as it doesn't need to reinitialize resources.

Note: Windows only supports spawn, Linux and macOS support both spawn and fork.

How to configure environment variables

1. Configure environment variables in flow.dag.yaml. Example:

inputs: []
    outputs: []
    nodes: []
    environment_variables:
      PF_WORKER_COUNT: 2
      PF_BATCH_METHOD: "spawn"
      MY_CUSTOM_SETTING: my_custom_value

2. Specify environment variables when submitting runs.

::::{tab-set}
:::{tab-item} CLI
:sync: CLI

Use this parameter: --environment-variable to specify environment variables.
Example: --environment-variable PF_WORKER_COUNT="2" PF_BATCH_METHOD="spawn".

:::

:::{tab-item} SDK
:sync: SDK

Specify environment variables when creating run. Example:

pf = PFClient(
        credential=credential,
        subscription_id="<SUBSCRIPTION_ID>",
        resource_group_name="<RESOURCE_GROUP>",
        workspace_name="<AML_WORKSPACE_NAME>",
    )

    flow = "web-classification"
    data = "web-classification/data.jsonl"

    environment_variables = {"PF_WORKER_COUNT": "2", "PF_BATCH_METHOD": "spawn"}

    # create run
    base_run = pf.run(
        flow=flow,
        data=data,
        environment_variables=environment_variables,
    )

:::

:::{tab-item} VS Code Extension
:sync: VS Code Extension

VSCode Extension supports specifying environment variables only when submitting batch runs.
Specify environment variables in batch_run_create.yaml. Example:

name: flow_name
    display_name: display_name
    flow: flow_folder
    data: data_file
    column_mapping:
        customer_info: <Please select a data input>
        history: <Please select a data input>
    environment_variables:
        PF_WORKER_COUNT: "2"
        PF_BATCH_METHOD: "spawn"

:::

::::

Priority

The environment variables specified when submitting runs always takes precedence over the environment variables in the flow.dag.yaml file.

How To Guides Manage Connections (`docs/how-to-guides/manage-connections.md`) Copy Section

Manage connections

Connection helps securely store and manage secret keys or other sensitive credentials required for interacting with LLM (Large Language Models) and other external tools, for example, Azure Content Safety.

:::{note}
This document target for manage connections locally. To use Azure AI connections locally, refer to this guide.
:::

Connection types Copy Chapter

There are multiple types of connections supported in promptflow, which can be simply categorized into strong type connection and custom connection. The strong type connection includes AzureOpenAIConnection, OpenAIConnection, etc. The custom connection is a generic connection type that can be used to store custom defined credentials.

We are going to use AzureOpenAIConnection as an example for strong type connection, and CustomConnection to show how to manage connections.

Create a connection Copy Chapter

:::{note}
If you are using WSL or other OS without default keyring storage backend, you may encounter StoreConnectionEncryptionKeyError, please refer to FAQ for the solutions.
:::

::::{tab-set}
:::{tab-item} CLI
:sync: CLI
Each of the strong type connection has a corresponding yaml schema, the example below shows the AzureOpenAIConnection yaml:

$schema: https://azuremlschemas.azureedge.net/promptflow/latest/AzureOpenAIConnection.schema.json
name: azure_open_ai_connection
type: azure_open_ai
api_key: "<to-be-replaced>"
api_base: "https://<name>.openai.azure.com/"
api_type: "azure"
api_version: "2023-03-15-preview"

The custom connection yaml will have two dict fields for secrets and configs, the example below shows the CustomConnection yaml:

$schema: https://azuremlschemas.azureedge.net/promptflow/latest/CustomConnection.schema.json
name: custom_connection
type: custom
configs:
  endpoint: "<your-endpoint>"
  other_config: "other_value"
secrets:  # required
  my_key: "<your-api-key>"

After preparing the yaml file, use the CLI command below to create them:

# Override keys with --set to avoid yaml file changes
pf connection create -f <path-to-azure-open-ai-connection> --set api_key=<your-api-key>
# Create the custom connection
pf connection create -f <path-to-custom-connection> --set configs.endpoint=<endpoint> secrets.my_key=<your-api-key>

The expected result is as follows if the connection created successfully.

:::

:::{tab-item} SDK
:sync: SDK
Using SDK, each connection type has a corresponding class to create a connection. The following code snippet shows how to import the required class and create the connection:

from promptflow.client import PFClient
from promptflow.entities import AzureOpenAIConnection, CustomConnection

# Get a pf client to manage connections
pf = PFClient()

# Initialize an AzureOpenAIConnection object
connection = AzureOpenAIConnection(
    name="my_azure_open_ai_connection", 
    api_key="<your-api-key>", 
    api_base="<your-endpoint>",
    api_version="2023-03-15-preview"
)

# Create the connection, note that api_key will be scrubbed in the returned result
result = pf.connections.create_or_update(connection)
print(result)

# Initialize a custom connection object
connection = CustomConnection(
    name="my_custom_connection", 
    # Secrets is a required field for custom connection
    secrets={"my_key": "<your-api-key>"},
    configs={"endpoint": "<your-endpoint>", "other_config": "other_value"}
)

# Create the connection, note that all secret values will be scrubbed in the returned result
result = pf.connections.create_or_update(connection)
print(result)

:::

:::{tab-item} VS Code Extension
:sync: VSC

On the VS Code primary sidebar > prompt flow pane. You can find the connections pane to manage your local connections. Click the "+" icon on the top right of it and follow the popped out instructions to create your new connection.

:::
::::

Update a connection Copy Chapter

::::{tab-set}
:::{tab-item} CLI
:sync: CLI
The commands below show how to update existing connections with new values:

# Update an azure OpenAI connection with a new api base
pf connection update -n my_azure_open_ai_connection --set api_base='new_value'
# Update a custom connection
pf connection update -n my_custom_connection --set configs.other_config='new_value'

:::

:::{tab-item} SDK
:sync: SDK
The code snippet below shows how to update existing connections with new values:

# Update an azure OpenAI connection with a new api base
connection = pf.connections.get(name="my_azure_open_ai_connection")
connection.api_base = "new_value"
connection.api_key = "<original-key>"  # secrets are required when updating connection using sdk
result = pf.connections.create_or_update(connection)
print(connection)
# Update a custom connection
connection = pf.connections.get(name="my_custom_connection")
connection.configs["other_config"] = "new_value"
connection.secrets = {"key1": "val1"}  # secrets are required when updating connection using sdk
result = pf.connections.create_or_update(connection)
print(connection)

:::

:::{tab-item} VS Code Extension
:sync: VSC

On the VS Code primary sidebar > prompt flow pane. You can find the connections pane to manage your local connections. Right click the item of the connection list to update or delete your connections.

:::
::::

List connections Copy Chapter

::::{tab-set}
:::{tab-item} CLI
:sync: CLI
List connection command will return the connections with json list format, note that all secrets and api keys will be scrubbed:

pf connection list

:::

:::{tab-item} SDK
:sync: SDK
List connection command will return the connections object list, note that all secrets and api keys will be scrubbed:

from promptflow.client import PFClient
# Get a pf client to manage connections
pf = PFClient()
# List and print connections
connection_list = pf.connections.list()
for connection in connection_list:
    print(connection)

:::

:::{tab-item} VS Code Extension
:sync: VSC

:::
::::

Delete a connection Copy Chapter

::::{tab-set}
:::{tab-item} CLI
:sync: CLI
Delete a connection with the following command:

pf connection delete -n <connection_name>

:::

:::{tab-item} SDK
:sync: SDK
Delete a connection with the following code snippet:

from promptflow.client import PFClient

# Get a pf client to manage connections
pf = PFClient()
# Delete the connection with specific name
pf.connections.delete(name="my_custom_connection")

:::

:::{tab-item} VS Code Extension
:sync: VSC

On the VS Code primary sidebar > prompt flow pane. You can find the connections pane to manage your local connections. Right click the item of the connection list to update or delete your connections.

:::
::::

Load from environment variables Copy Chapter

With promptflow>=1.8.0, user is able to load a connection object from os environment variables with <ConnectionType>.from_env func.
Note that the connection object will NOT BE CREATED to local database.

Supported types are as follows:

For example, with OPENAI_API_KEY set to environment, an OpenAIConnection object can be loaded with OpenAIConnection.from_env().

Authenticate with Microsoft Entra ID Copy Chapter

Microsoft Entra ID is a cloud-based identity and access management service that enables your employees access external resources.

Some promptflow connection types supports connection authentication with Microsoft Entra ID.

Next steps Copy Chapter

• Reach more detail about connection concepts.
• Try the connection samples.
• Consume connections from Azure AI.

How To Guides Quick Start (`docs/how-to-guides/quick-start.md`) Copy Section

Quick start

This guide will walk you through the first steps of the prompt flow code-first experience.

Prerequisite - To make the most of this tutorial, you'll need:

• Python programming knowledge

Learning Objectives - Upon completing this tutorial, you should know how to:

• Setup your python environment to run prompt flow
• Create a flow using a prompt and python function
• Test the flow using your favorite experience: CLI, SDK or UI.

Installation Copy Chapter

Install promptflow package to start.

pip install promptflow

Learn more about installation.

Create your first flow Copy Chapter

Model a LLM call with a prompty Copy Section

Create a Prompty file to help you trigger one LLM call.

---
name: Minimal Chat
model:
  api: chat
  configuration:
    type: azure_openai
    azure_deployment: gpt-35-turbo
  parameters:
    temperature: 0.2
    max_tokens: 1024
inputs:
  question:
    type: string
sample:
  question: "What is Prompt flow?"
---

system:
You are a helpful assistant.

user:
{{question}}

Prompty is a markdown file. The front matter structured in YAML, encapsulates a series of metadata fields pivotal for defining the model’s configuration and the inputs for the prompty. After this front matter is the prompt template, articulated in the Jinja format.
See more details in Develop a prompty.

Create a flow Copy Section

Create a python function which is the entry of a flow.

import os

from dotenv import load_dotenv
from pathlib import Path
from promptflow.tracing import trace
from promptflow.core import Prompty

BASE_DIR = Path(__file__).absolute().parent

@trace
def chat(question: str = "What's the capital of France?") -> str:
    """Flow entry function."""

    if "OPENAI_API_KEY" not in os.environ and "AZURE_OPENAI_API_KEY" not in os.environ:
        # load environment variables from .env file
        load_dotenv()

    prompty = Prompty.load(source=BASE_DIR / "chat.prompty")
    # trigger a llm call with the prompty obj
    output = prompty(question=question)
    return output

Flow can be a python function or class or a yaml file describing a DAG which encapsulates your LLM application logic.
Learn more on the flow concept and how to Develop a flow.

See the full example of this python file in: Minimal Chat.

Test the flow Copy Chapter

Test the flow with your favorite experience: CLI, SDK or UI.

::::{tab-set}

:::{tab-item} CLI
:sync: CLI

pf is the CLI command you get when you install the promptflow package. Learn more about features of the pf CLI in the reference doc.

pf flow test --flow flow:chat --inputs question="What's the capital of France?"

You will get some output like the following in your terminal.

Prompt flow service has started...
You can view the trace detail from the following URL:
http://127.0.0.1:51330/v1.0/ui/traces/?#collection=chat-minimal&uiTraceId=0x49382bbe30664f747348a8ae9dc8b954

The capital of France is Paris

If you click the trace URL printed, you will see a trace UI which helps you understand the actual LLM call that happened behind the scenes.

:::

:::{tab-item} SDK
:sync: SDK

Call the chat function with your question. Assume you have a flow.py file with the following content.

if __name__ == "__main__":
    from promptflow.tracing import start_trace

    start_trace()

    result = chat("What's the capital of France?")
    print(result)

Run the script with python flow.py, and you will get some output like below:

Prompt flow service has started...
You can view the trace detail from the following URL:
http://127.0.0.1:51330/v1.0/ui/traces/?#collection=chat-minimal&uiTraceId=0x49382bbe30664f747348a8ae9dc8b954

The capital of France is Paris

If you click the trace URL printed, you will see a trace UI which helps you understand the actual LLM call that happened behind the scenes.

:::

:::{tab-item} UI
:sync: VS Code Extension

Start test chat ui with below command.

pf flow test --flow flow:chat --ui

The command will open a browser page like below:

See more details of this topic in Chat with a flow.

Click the "View trace" button to see a trace UI which helps you understand the actual LLM call that happened behind the scenes.

:::

::::

Next steps Copy Chapter

Learn more on how to:

• Tracing: details on how tracing works.
• Develop a prompty: details on how to develop prompty.
• Develop a flow: details on how to develop a flow using a python function or class.
• Develop a DAG flow: details on how to develop a flow using friendly DAG UI.

And you can also check our Tutorials, especially:

• Tutorial: Chat with PDF: An end-to-end tutorial on how to build a high quality chat application with prompt flow, including flow development and evaluation with metrics.

How To Guides Set Promptflow Configs (`docs/how-to-guides/set-promptflow-configs.md`) Copy Section

Set promptflow configs

:::{admonition} Experimental feature
This is an experimental feature, and may change at any time. Learn more.
:::

Promptflow supports setting global configs to avoid passing the same parameters to each command. The global configs are stored in a yaml file, which is located at ~/.promptflow/pf.yaml by default.

Additionally, promptflow supports setting configs to a specified path, and these configs will only take effect for the Promptflow program when the working directory is the specified path or its subdirectories.
The configs are stored in a yaml file, which is located at <config_folder>/pf.yaml.

The config file is shared between promptflow extension and sdk/cli. Promptflow extension controls each config through UI, so the following sections will show how to set global configs using promptflow cli.

Set config Copy Chapter

Set global config

pf config set <config_name>=<config_value>

For example:

pf config set connection.provider="azureml://subscriptions/<your-subscription>/resourceGroups/<your-resourcegroup>/providers/Microsoft.MachineLearningServices/workspaces/<your-workspace>"

Setting the config for a specific path

pf config set <config_name>=<config_value> --path <config_folder>

For example:

pf config set connection.provider="azureml://subscriptions/<your-subscription>/resourceGroups/<your-resourcegroup>/providers/Microsoft.MachineLearningServices/workspaces/<your-workspace>" --path .

Show config Copy Chapter

The following command will get all configs and show them as json format:

pf config show

After running the above config set command, show command will return the following result:

{
  "connection": {
    "provider": "azureml://subscriptions/<your-subscription>/resourceGroups/<your-resourcegroup>/providers/Microsoft.MachineLearningServices/workspaces/<your-workspace>"
  }
}

Supported configs Copy Chapter

connection.provider Copy Section

The connection provider, default to "local". There are 3 possible provider values.

local

Set connection provider to local with connection.provider=local.

Connections will be saved locally. PFClient(or pf connection commands) will manage local connections. Consequently, the flow will be executed using these local connections.

full azure machine learning workspace resource id

Set connection provider to a specific workspace with:

connection.provider=azureml://subscriptions/<your-subscription>/resourceGroups/<your-resourcegroup>/providers/Microsoft.MachineLearningServices/workspaces/<your-workspace>

When get or list connections, PFClient(or pf connection commands) will return workspace connections, and flow will be executed using these workspace connections.
Secrets for workspace connection will not be shown by those commands, which means you may see empty dict {} for custom connections.

:::{note}
Command create, update and delete are not supported for workspace connections, please manage it in workspace portal, Azure AI Studio, az ml cli or azure-ai-ml sdk.
:::

azureml

In addition to the full resource id, you can designate the connection provider as "azureml" with connection.provider=azureml. In this case,
promptflow will attempt to retrieve the workspace configuration by searching .azureml/config.json from the current directory, then progressively from its parent folders. So it's possible to set the workspace configuration for different flow by placing the config file in the project folder.

The expected format of the config file is as follows:

{
  "workspace_name": "<your-workspace-name>",
  "resource_group": "<your-resource-group>",
  "subscription_id": "<your-subscription-id>"
}

service.host Copy Section

The promptflow service host, default to "127.0.0.1". You can set the service host with service.host=<your-host>.

For example:

pf config set service.host="0.0.0.0"

💡 Tips
In addition to the CLI command line setting approach, we also support setting this connection provider through the VS Code extension UI. Click here to learn more.

How To Guides Tune Prompts With Variants (`docs/how-to-guides/tune-prompts-with-variants.md`) Copy Section

Tune prompts using variants

:::{admonition} Experimental feature
This is an experimental feature, and may change at any time. Learn more.
:::

To better understand this part, please read Quick start and Run and evaluate a flow first.

What is variant and why should we care Copy Chapter

In order to help users tune the prompts in a more efficient way, we introduce the concept of variants which can help you test the model’s behavior under different conditions, such as different wording, formatting, context, temperature, or top-k, compare and find the best prompt and configuration that maximizes the model’s accuracy, diversity, or coherence.

Create a run with different variant node Copy Chapter

In this example, we use the flow web-classification, its node summarize_text_content has two variants: variant_0 and variant_1. The difference between them is the inputs parameters:

...
nodes:
- name: summarize_text_content
  use_variants: true
...
node_variants:
  summarize_text_content:
    default_variant_id: variant_0
    variants:
      variant_0:
        node:
          type: llm
          source:
            type: code
            path: summarize_text_content.jinja2
          inputs:
            deployment_name: text-davinci-003
            max_tokens: '128'
            temperature: '0.2'
            text: ${fetch_text_content_from_url.output}
          provider: AzureOpenAI
          connection: open_ai_connection
          api: completion
          module: promptflow.tools.aoai
      variant_1:
        node:
          type: llm
          source:
            type: code
            path: summarize_text_content__variant_1.jinja2
          inputs:
            deployment_name: text-davinci-003
            max_tokens: '256'
            temperature: '0.3'
            text: ${fetch_text_content_from_url.output}
          provider: AzureOpenAI
          connection: open_ai_connection
          api: completion
          module: promptflow.tools.aoai

You can check the whole flow definition in flow.dag.yaml.

Now we will create a variant run which uses node summarize_text_content's variant variant_1.
Assuming you are in working directory /examples/flows/standard

::::{tab-set}

:::{tab-item} CLI
:sync: CLI

Note we pass --variant to specify which variant of the node should be running.

pf run create --flow web-classification --data web-classification/data.jsonl --variant '${summarize_text_content.variant_1}' --column-mapping url='${data.url}' --stream --name my_first_variant_run

:::

:::{tab-item} SDK
:sync: SDK

from promptflow.client import PFClient

pf = PFClient()  # get a promptflow client
flow = "web-classification"
data= "web-classification/data.jsonl"

# use the variant1 of the summarize_text_content node.
variant_run = pf.run(
    flow=flow,
    data=data,
    variant="${summarize_text_content.variant_1}",  # use variant 1.
    column_mapping={"url": "${data.url}"},
)

pf.stream(variant_run)

:::

:::{tab-item} VS Code Extension
:sync: VS Code Extension

:::

::::

After the variant run is created, you can evaluate the variant run with a evaluation flow, just like you evalute a standard flow run.

Next steps Copy Chapter

Learn more about:

• Run and evaluate a flow
• Deploy a flow
• Prompt flow in Azure AI

Reference Flow Yaml Schema Reference (`docs/reference/flow-yaml-schema-reference.md`) Copy Section

Flow YAML Schema

:::{admonition} Experimental feature
This is an experimental feature, and may change at any time. Learn more.
:::

The source JSON schema can be found at Flow.schema.json

YAML syntax Copy Chapter

Flow input Copy Section

Flow output Copy Section

Nodes Copy Section

Nodes is a set of node which is a dictionary with following fields. Below, we only show the common fields of a single node using built-in tool.

Node variants Copy Section

Node variants is a dictionary containing variants definition for nodes with variants with their respective node names as dictionary keys.
Below, we explore the variants for a single node.

Examples Copy Chapter

Flow examples are available in the GitHub repository.

• basic
• web-classification
• chat-basic
• chat-with-pdf
• eval-basic

Reference Pf Command Reference (`docs/reference/pf-command-reference.md`) Copy Section

pf

Manage prompt flow resources with the prompt flow CLI.

pf flow Copy Chapter

Manage promptflow flow flows.

pf flow init Copy Section

Initialize a prompt flow directory.

pf flow init [--flow]
             [--entry]
             [--function]
             [--prompt-template]
             [--type]
             [--yes]

Examples

Create a flow folder with code, prompts and YAML specification of the flow.

pf flow init --flow <path-to-flow-direcotry>

Create an evaluation prompt flow

pf flow init --flow <path-to-flow-direcotry> --type evaluation

Create a flow in existing folder

pf flow init --flow <path-to-existing-folder> --entry <entry.py> --function <function-name> --prompt-template <path-to-prompt-template.md>

Optional Parameters

--flow

The flow name to create.

--entry

The entry file name.

--function

The function name in entry file.

--prompt-template

The prompt template parameter and assignment.

--type

The initialized flow type.
accepted value: standard, evaluation, chat

--yes --assume-yes -y

Automatic yes to all prompts; assume 'yes' as answer to all prompts and run non-interactively.

pf flow test Copy Section

Test the prompt flow or flow node.

pf flow test --flow
             [--inputs]
             [--node]
             [--variant]
             [--debug]
             [--interactive]
             [--verbose]
             [--ui]
             [--collection]

Examples

Test the flow.

pf flow test --flow <path-to-flow-directory>

Test the flow from json file.

pf flow test --flow <path-to-flow-directory> --inputs inputs.json

Test the flow with first line from jsonl file.

pf flow test --flow <path-to-flow-directory> --inputs inputs.jsonl

Test the flow with input values.

pf flow test --flow <path-to-flow-directory> --inputs data_key1=data_val1 data_key2=data_val2

Test the flow with specified variant node.

pf flow test --flow <path-to-flow-directory> --variant '${node_name.variant_name}'

Test the single node in the flow.

pf flow test --flow <path-to-flow-directory> --node <node_name>

Debug the single node in the flow.

pf flow test --flow <path-to-flow-directory> --node <node_name> --debug

Chat in the flow.

pf flow test --flow <path-to-flow-directory> --node <node_name> --interactive

Chat in the chat window.

pf flow test --flow <path-to-flow-directory> --ui

Test the flow while log traces to a specific collection.

pf flow test --flow <path-to-flow-directory> --collection <collection>

Required Parameter

--flow

The flow directory to test.

Optional Parameters

--inputs

Input data for the flow. Example: --inputs data1=data1_val data2=data2_val

--node

The node name in the flow need to be tested.

--variant

Node & variant name in format of ${node_name.variant_name}.

--debug

Debug the single node in the flow.

--interactive

Start a interactive chat session for chat flow.

--verbose

Displays the output for each step in the chat flow.

--ui

The flag to start an interactive chat experience in local chat window.

pf flow validate Copy Section

Validate the prompt flow and generate a flow.tools.json under .promptflow. This file is required when using flow as a component in a Azure ML pipeline.

pf flow validate --source
                 [--debug]
                 [--verbose]

Examples

Validate the flow.

pf flow validate --source <path-to-flow>

Required Parameter

--source

The flow source to validate.

pf flow build Copy Section

Build a flow for further sharing or deployment.

pf flow build --source
              --output
              --format
              [--variant]
              [--verbose]
              [--debug]

Examples

Build a flow as docker, which can be built into Docker image via docker build.

pf flow build --source <path-to-flow> --output <output-path> --format docker

Build a flow as docker with specific variant.

pf flow build --source <path-to-flow> --output <output-path> --format docker --variant '${node_name.variant_name}'

Required Parameter

--source

The flow or run source to be used.

--output

The folder to output built flow. Need to be empty or not existed.

--format

The format to build flow into

Optional Parameters

--variant

Node & variant name in format of ${node_name.variant_name}.

--verbose

Show more details for each step during build.

--debug

Show debug information during build.

pf flow serve Copy Section

Serving a flow as an endpoint.

pf flow serve --source
              [--port]
              [--host]
              [--environment-variables]
              [--verbose]
              [--debug]
              [--skip-open-browser]
              [--engine]

Examples

Serve flow as an endpoint.

pf flow serve --source <path-to-flow>

Serve flow as an endpoint with specific port and host.

pf flow serve --source <path-to-flow> --port <port> --host <host> --environment-variables key1="`${my_connection.api_key}`" key2="value2"

Serve flow as an endpoint with specific port, host, environment-variables and fastapi serving engine.

pf flow serve --source <path-to-flow> --port <port> --host <host> --environment-variables key1="`${my_connection.api_key}`" key2="value2" --engine fastapi

Required Parameter

--source

The flow or run source to be used.

Optional Parameters

--port

The port on which endpoint to run.

--host

The host of endpoint.

--environment-variables

Environment variables to set by specifying a property path and value. Example: --environment-variable key1="`${my_connection.api_key}`" key2="value2". The value reference to connection keys will be resolved to the actual value, and all environment variables specified will be set into os.environ.

--verbose

Show more details for each step during serve.

--debug

Show debug information during serve.

--skip-open-browser

Skip opening browser after serve. Store true parameter.

--engine

Switch python serving engine between flask amd fastapi, default to flask.

pf connection Copy Chapter

Manage prompt flow connections.

pf connection create Copy Section

Create a connection.

pf connection create --file
                     [--name]
                     [--set]

Examples

Create a connection with YAML file.

pf connection create -f <yaml-filename>

Create a connection with YAML file with override.

pf connection create -f <yaml-filename> --set api_key="<api-key>"

Create a custom connection with .env file; note that overrides specified by --set will be ignored.

pf connection create -f .env --name <name>

Required Parameter

--file -f

Local path to the YAML file containing the prompt flow connection specification.

Optional Parameters

--name -n

Name of the connection.

--set

Update an object by specifying a property path and value to set. Example: --set property1.property2=.

pf connection update Copy Section

Update a connection.

pf connection update --name
                     [--set]

Example

Update a connection.

pf connection update -n <name> --set api_key="<api-key>"

Required Parameter

--name -n

Name of the connection.

Optional Parameter

--set

Update an object by specifying a property path and value to set. Example: --set property1.property2=.

pf connection show Copy Section

Show details of a connection.

pf connection show --name

Required Parameter

--name -n

Name of the connection.

pf connection list Copy Section

List all the connection.

pf connection list

pf connection delete Copy Section

Delete a connection.

pf connection delete --name

Required Parameter

--name -n

Name of the connection.

pf run Copy Chapter

Manage prompt flow runs.

pf run create Copy Section

Create a run.

pf run create [--file]
              [--flow]
              [--data]
              [--column-mapping]
              [--run]
              [--variant]
              [--stream]
              [--environment-variables]
              [--connections]
              [--set]
              [--source]
              [--resume-from] # require promptflow>=1.8.0, and original run created with promptflow>=1.8.0

Examples

Create a run with YAML file.

pf run create -f <yaml-filename>

Create a run with YAML file and replace another data in the YAML file.

pf run create -f <yaml-filename> --data <path-to-new-data-file-relative-to-yaml-file>

Create a run from flow directory and reference a run.

pf run create --flow <path-to-flow-directory> --data <path-to-data-file> --column-mapping groundtruth='${data.answer}' prediction='${run.outputs.category}' --run <run-name> --variant '${summarize_text_content.variant_0}' --stream

Create a run from an existing run record folder.

pf run create --source <path-to-run-folder>

Create a run by specifying the resume_from. (Require promptflow>=1.8.0, and original run created with promptflow>=1.8.0)

Succeeded line result of the original run will be reused, only remaining/failed lines will be run.

pf run create --resume-from <original-run-name>

pf run create --resume-from <original-run-name> --name <new-run-name> --set display_name='A new run' description='my run description' tags.Type=Test

Optional Parameters

--file -f

Local path to the YAML file containing the prompt flow run specification; can be overwritten by other parameters. Reference here for YAML schema.

--flow

Local path to the flow directory. If --file is provided, this path should be relative path to the file.

--data

Local path to the data file. If --file is provided, this path should be relative path to the file.

--column-mapping

Inputs column mapping, use ${data.xx} to refer to data columns, use ${run.inputs.xx} to refer to referenced run's data columns, and ${run.outputs.xx} to refer to run outputs columns.

--run

Referenced flow run name. For example, you can run an evaluation flow against an existing run. For example, "pf run create --flow evaluation_flow_dir --run existing_bulk_run".

--variant

Node & variant name in format of ${node_name.variant_name}.

--stream -s

Indicates whether to stream the run's logs to the console.
default value: False

--environment-variables

Environment variables to set by specifying a property path and value. Example:
--environment-variable key1='${my_connection.api_key}' key2='value2'. The value reference
to connection keys will be resolved to the actual value, and all environment variables
specified will be set into os.environ.

--connections

Overwrite node level connections with provided value.
Example: --connections node1.connection=test_llm_connection node1.deployment_name=gpt-35-turbo

--set

Update an object by specifying a property path and value to set.
Example: --set property1.property2=<value>.

--source

Local path to the existing run record folder.

pf run update Copy Section

Update a run metadata, including display name, description and tags.

pf run update --name
              [--set]

Example

Update a run

pf run update -n <name> --set display_name="<display-name>" description="<description>" tags.key="value"

Required Parameter

--name -n

Name of the run.

Optional Parameter

--set

Update an object by specifying a property path and value to set. Example: --set property1.property2=.

pf run stream Copy Section

Stream run logs to the console.

pf run stream --name

Required Parameter

--name -n

Name of the run.

pf run list Copy Section

List runs.

pf run list [--all-results]
            [--archived-only]
            [--include-archived]
            [--max-results]

Optional Parameters

--all-results

Returns all results.
default value: False

--archived-only

List archived runs only.
default value: False

--include-archived

List archived runs and active runs.
default value: False

--max-results -r

Max number of results to return. Default is 50.
default value: 50

pf run show Copy Section

Show details for a run.

pf run show --name

Required Parameter

--name -n

Name of the run.

pf run show-details Copy Section

Preview a run's input(s) and output(s).

pf run show-details --name

Required Parameter

--name -n

Name of the run.

pf run show-metrics Copy Section

Print run metrics to the console.

pf run show-metrics --name

Required Parameter

--name -n

Name of the run.

pf run visualize Copy Section

Visualize a run in the browser.

pf run visualize --names

Required Parameter

--names -n

Name of the runs, comma separated.

pf run archive Copy Section

Archive a run.

pf run archive --name

Required Parameter

--name -n

Name of the run.

pf run restore Copy Section

Restore an archived run.

pf run restore --name

Required Parameter

--name -n

Name of the run.

pf tool Copy Chapter

Manage promptflow tools.

pf tool init Copy Section

Initialize a tool directory.

pf tool init [--package]
             [--tool]
             [--set]

Examples

Creating a package tool from scratch.

pf tool init --package <package-name> --tool <tool-name>

Creating a package tool with extra info.

pf tool init --package <package-name> --tool <tool-name> --set icon=<icon-path> category=<tool-category> tags="{'<key>': '<value>'}"

Creating a package tool from scratch.

pf tool init --package <package-name> --tool <tool-name>

Creating a python tool from scratch.

pf tool init --tool <tool-name>

Optional Parameters

--package

The package name to create.

--tool

The tool name to create.

--set

Set extra information about the tool, like category, icon and tags. Example: --set =.

pf tool list Copy Section

List all tools in the environment.

pf tool list [--flow]

Examples

List all package tool in the environment.

pf tool list

List all package tool and code tool in the flow.

pf tool list --flow <path-to-flow-direcotry>

Optional Parameters

--flow

The flow directory.

pf tool validate Copy Section

Validate tool.

pf tool validate --source

Examples

Validate single function tool.

pf tool validate -–source <package-name>.<module-name>.<tool-function>

Validate all tool in a package tool.

pf tool validate -–source <package-name>

Validate tools in a python script.

pf tool validate --source <path-to-tool-script>

Required Parameter

--source

The tool source to be used.

pf config Copy Chapter

Manage config for current user.

pf config set Copy Section

Set prompt flow configs for current user, configs will be stored at ~/.promptflow/pf.yaml.

pf config set

Examples

Connection provider

Set connection provider to Azure ML workspace or Azure AI project for current user.

pf config set connection.provider="azureml://subscriptions/<subscription-id>/resourceGroups/<resource-group-name>/providers/Microsoft.MachineLearningServices/workspaces/<workspace-or-project-name>"

Tracing

Set trace destination to Azure ML workspace or Azure AI project.

pf config set trace.destination="azureml://subscriptions/<subscription-id>/resourceGroups/<resource-group-name>/providers/Microsoft.MachineLearningServices/workspaces/<workspace-or-project-name>"

Only log traces to local.

pf config set trace.destination="local"

Disable tracing feature.

pf config set trace.destination="none"

pf config show Copy Section

Show prompt flow configs for current user.

pf config show

Examples

Show prompt flow for current user.

pf config show

pf service Copy Chapter

Manage prompt flow service.

pf service start Copy Section

Start the prompt flow service.

pf service start [--port]
                 [--force]
                 [--debug]

Examples

Prompt flow will try to start the service on the default port 23333. If the port is already taken, prompt flow will
sequentially probe new ports, incrementing by one each time. Prompt flow retains the port number for future reference
and will utilize it for subsequent service startups.

pf service start

Forcefully start the prompt flow service. If the port is already in use, the existing service will be terminated and
restart a new service

pf service start --force

Start the prompt flow service with a specified port. If the port is already taken, prompt flow will raise an error
unless forcefully start the service with the --force flag. Upon availability, prompt flow retains the port number for
future reference and will utilize it for subsequent service startups.

pf service start --port 65553

Start prompt flow service in foreground, displaying debug level logs directly in the terminal.

pf service start --debug

Optional Parameters

--port -p

The designated port of the prompt flow service and port number will be remembered if port is available.

--force

Force restart the existing service if the port is used.

--debug

Start prompt flow service in foreground, displaying debug level logs directly in the terminal.

pf service stop Copy Section

Stop prompt flow service.

pf service stop [--debug]

Example

Stop prompt flow service.

pf service stop

Optional Parameter

--debug

The flag to turn on debug mode for cli.

pf service status Copy Section

Display the started prompt flow service info.

pf service status

pf upgrade Copy Chapter

Upgrade prompt flow CLI.

Examples Copy Section

Upgrade prompt flow without prompt and run non-interactively.

pf upgrade --yes

pf trace Copy Chapter

Manage prompt flow traces.

pf trace delete Copy Section

Delete traces.

pf trace delete [--run]
                [--collection]
                [--started-before]  # should combine with `collection`

Examples

Delete traces comes from a specific run.

pf trace delete --run <run-name>

Delete traces in a specific collection.

pf trace delete --collection <collection>

Delete traces in a specific collection started before a specific time.

# `started-before` should be in ISO 8601 format
pf trace delete --collection <collection> --started-before '2024-03-19T15:17:23.807563'

Autocomplete Copy Chapter

To activate autocomplete features for the pf CLI you need to add the following snippet to your ~/.bashrc or ~/.zshrc:

source <promptflow_package_install_root>/pf.completion.sh

Reference Pfazure Command Reference (`docs/reference/pfazure-command-reference.md`) Copy Section

pfazure

:::{admonition} Experimental feature
This is an experimental feature, and may change at any time. Learn more.
:::

Manage prompt flow resources on Azure with the prompt flow CLI.

pfazure flow Copy Chapter

Manage flows.

pfazure flow create Copy Section

Create a flow in Azure AI from a local flow folder.

pfazure flow create [--flow]
                    [--set]
                    [--subscription]
                    [--resource-group]
                    [--workspace-name]

Parameters

--flow

Local path to the flow directory.

--set

Update an object by specifying a property path and value to set.

• display_name: Flow display name that will be created in remote. Default to be flow folder name + timestamp if not specified. e.g. "--set display_name=<display_name>".
• type: Flow type. Default to be "standard" if not specified. Available types are: "standard", "evaluation", "chat". e.g. "--set type=<type>".
• description: Flow description. e.g. "--set description=<description>."
• tags: Flow tags. e.g. "--set tags.key1=value1 tags.key2=value2."

--subscription

Subscription id, required when there is no default value from az configure.

--resource-group -g

Resource group name, required when there is no default value from az configure.

--workspace-name -w

Workspace name, required when there is no default value from az configure.

pfazure flow update Copy Section

Update a flow's metadata, such as display name, description and tags.

pfazure flow update --flow
                    [--set]
                    [--subscription]
                    [--resource-group]
                    [--workspace-name]

Parameters

--flow

The flow name on azure. It's a guid that can be found from 2 ways:

• After creating a flow to azure, it can be found in the printed message in "name" attribute.
• Open a flow in azure portal, the guid is in the url. e.g. https://ml.azure.com/prompts/flow/{workspace-id}/{flow-name}/xxx

--set

Update an object by specifying a property path and value to set.

• display_name: Flow display name. e.g. "--set display_name=<display_name>".
• description: Flow description. e.g. "--set description=<description>."
• tags: Flow tags. e.g. "--set tags.key1=value1 tags.key2=value2."

--subscription

Subscription id, required when there is no default value from az configure.

--resource-group -g

Resource group name, required when there is no default value from az configure.

--workspace-name -w

Workspace name, required when there is no default value from az configure.

pfazure flow list Copy Section

List remote flows on Azure AI.

pfazure flow list [--max-results]
                  [--include-others]
                  [--type]
                  [--output]
                  [--archived-only]
                  [--include-archived]
                  [--subscription]
                  [--resource-group]
                  [--workspace-name]
                  [--output]

Parameters

--max-results -r

Max number of results to return. Default is 50, upper bound is 100.

--include-others

Include flows created by other owners. By default only flows created by the current user are returned.

--type

Filter flows by type. Available types are: "standard", "evaluation", "chat".

--archived-only

List archived flows only.

--include-archived

List archived flows and active flows.

--output -o

Output format. Allowed values: json, table. Default: json.

--subscription

Subscription id, required when there is no default value from az configure.

--resource-group -g

Resource group name, required when there is no default value from az configure.

--workspace-name -w

Workspace name, required when there is no default value from az configure.

pfazure run Copy Chapter

Manage prompt flow runs.

pfazure run create Copy Section

Create a run.

pfazure run create [--file]
                   [--flow]
                   [--data]
                   [--column-mapping]
                   [--run]
                   [--variant]
                   [--stream]
                   [--environment-variables]
                   [--connections]
                   [--resume-from] # require promptflow>=1.8.0
                   [--set]
                   [--subscription]
                   [--resource-group]
                   [--workspace-name]

Parameters

--file -f

Local path to the YAML file containing the prompt flow run specification; can be overwritten by other parameters. Reference here for YAML schema.

--flow

The flow source to create the run. It could be:

• Local path to the flow directory.
  bash Copy

  pfazure run create --flow <path-to-flow-directory> --data <path-to-data-file> --column-mapping <key-value-pair>

• The flow name on azure with a prefix azureml:. Flow name is a guid that can be found from 2 ways:
  • After creating a flow to azure, it can be found in the printed message in "name" attribute.
  • Open a flow in azure portal, the guid is in the url. e.g. https://ml.azure.com/prompts/flow/{workspace-id}/{flow-name}/xxx
  bash Copy

  pfazure run create --flow azureml:<flow-name> --data <path-to-data-file> --column-mapping <key-value-pair>

--data

Local path to the data file or remote data. e.g. azureml:name:version.

--column-mapping

Inputs column mapping, use ${data.xx} to refer to data columns, use ${run.inputs.xx} to refer to referenced run's data columns, and ${run.outputs.xx} to refer to run outputs columns.

--run

Referenced flow run name. For example, you can run an evaluation flow against an existing run. For example, "pfazure run create --flow evaluation_flow_dir --run existing_bulk_run --column-mapping url='${data.url}'".

--variant

Node & variant name in format of ${node_name.variant_name}.

--stream -s

Indicates whether to stream the run's logs to the console.
default value: False

--environment-variables

Environment variables to set by specifying a property path and value. Example:
--environment-variable key1='${my_connection.api_key}' key2='value2'. The value reference
to connection keys will be resolved to the actual value, and all environment variables
specified will be set into os.environ.

--connections

Overwrite node level connections with provided value.
Example: --connections node1.connection=test_llm_connection node1.deployment_name=gpt-35-turbo

--resume-from

Create a run resume from an existing run. (Require promptflow>=1.8.0)
Example: --resume-from <run_name>

--set

Update an object by specifying a property path and value to set.
Example: --set property1.property2=<value>.

--subscription

Subscription id, required when there is no default value from az configure.

--resource-group -g

Resource group name, required when there is no default value from az configure.

--workspace-name -w

Workspace name, required when there is no default value from az configure.

pfazure run list Copy Section

List runs in a workspace.

pfazure run list [--archived-only]
                 [--include-archived]
                 [--max-results]
                 [--subscription]
                 [--resource-group]
                 [--workspace-name]

Parameters

--archived-only

List archived runs only.
default value: False

--include-archived

List archived runs and active runs.
default value: False

--max-results -r

Max number of results to return. Default is 50, upper bound is 100.
default value: 50

--subscription

Subscription id, required when there is no default value from az configure.

--resource-group -g

Resource group name, required when there is no default value from az configure.

--workspace-name -w

Workspace name, required when there is no default value from az configure.

pfazure run show Copy Section

Show details for a run.

pfazure run show --name
                 [--subscription]
                 [--resource-group]
                 [--workspace-name]

Parameters

--name -n

Name of the run.

--subscription

Subscription id, required when there is no default value from az configure.

--resource-group -g

Resource group name, required when there is no default value from az configure.

--workspace-name -w

Workspace name, required when there is no default value from az configure.

pfazure run stream Copy Section

Stream run logs to the console.

pfazure run stream --name
                   [--timeout]
                   [--subscription]
                   [--resource-group]
                   [--workspace-name]

Parameters

--name -n

Name of the run.

--timeout

Timeout in seconds. If the run stays in the same status and produce no new logs in a period longer than the timeout value, the stream operation will abort. Default value is 600 seconds

--subscription

Subscription id, required when there is no default value from az configure.

--resource-group -g

Resource group name, required when there is no default value from az configure.

--workspace-name -w

Workspace name, required when there is no default value from az configure.

pfazure run show-details Copy Section

Show a run details.

pfazure run show-details --name
                         [--subscription]
                         [--resource-group]
                         [--workspace-name]

Parameters

--name -n

Name of the run.

--subscription

Subscription id, required when there is no default value from az configure.

--resource-group -g

Resource group name, required when there is no default value from az configure.

--workspace-name -w

Workspace name, required when there is no default value from az configure.

pfazure run show-metrics Copy Section

Show run metrics.

pfazure run show-metrics --name
                         [--subscription]
                         [--resource-group]
                         [--workspace-name]

Parameters

--name -n

Name of the run.

--subscription

Subscription id, required when there is no default value from az configure.

--resource-group -g

Resource group name, required when there is no default value from az configure.

--workspace-name -w

Workspace name, required when there is no default value from az configure.

pfazure run visualize Copy Section

Visualize a run.

pfazure run visualize --name
                      [--subscription]
                      [--resource-group]
                      [--workspace-name]

Parameters

--name -n

Name of the run.

--subscription

Subscription id, required when there is no default value from az configure.

--resource-group -g

Resource group name, required when there is no default value from az configure.

--workspace-name -w

Workspace name, required when there is no default value from az configure.

pfazure run archive Copy Section

Archive a run.

pfazure run archive --name
                    [--subscription]
                    [--resource-group]
                    [--workspace-name]

Parameters

--name -n

Name of the run.

--subscription

Subscription id, required when there is no default value from az configure.

--resource-group -g

Resource group name, required when there is no default value from az configure.

--workspace-name -w

Workspace name, required when there is no default value from az configure.

pfazure run restore Copy Section

Restore a run.

pfazure run restore --name
                    [--subscription]
                    [--resource-group]
                    [--workspace-name]

Parameters

--name -n

Name of the run.

--subscription

Subscription id, required when there is no default value from az configure.

--resource-group -g

Resource group name, required when there is no default value from az configure.

--workspace-name -w

Workspace name, required when there is no default value from az configure.

pfazure run update Copy Section

Update a run's metadata, such as display name, description and tags.

pfazure run update --name
                    [--set display_name="<value>" description="<value>" tags.key="<value>"]
                    [--subscription]
                    [--resource-group]
                    [--workspace-name]

Examples

Set display name, description and tags:

pfazure run update --name <run_name> --set display_name="<value>" description="<value>" tags.key="<value>"

Parameters

--name -n

Name of the run.

--set

Set meta information of the run, like display_name, description or tags. Example: --set =.

--subscription

Subscription id, required when there is no default value from az configure.

--resource-group -g

Resource group name, required when there is no default value from az configure.

--workspace-name -w

Workspace name, required when there is no default value from az configure.

pfazure run download Copy Section

Download a run's metadata, such as input, output, snapshot and artifact. After the download is finished, you can use pf run create --source <run-info-local-folder> to register this run as a local run record, then you can use commands like pf run show/visualize to inspect the run just like a run that was created from local flow.

pfazure run download --name
                    [--output]
                    [--overwrite]
                    [--subscription]
                    [--resource-group]
                    [--workspace-name]

Examples

Download a run data to local:

pfazure run download --name <name> --output <output-folder-path>

Parameters

--name -n

Name of the run.

--output -o

Output folder path to store the downloaded run data. Default to be ~/.promptflow/.runs if not specified

--overwrite

Overwrite the existing run data if the output folder already exists. Default to be False if not specified

--subscription

Subscription id, required when there is no default value from az configure.

--resource-group -g

Resource group name, required when there is no default value from az configure.

--workspace-name -w

Workspace name, required when there is no default value from az configure.

Reference Run Yaml Schema Reference (`docs/reference/run-yaml-schema-reference.md`) Copy Section

Run YAML Schema

:::{admonition} Experimental feature
This is an experimental feature, and may change at any time. Learn more.
:::

The source JSON schema can be found at Run.schema.json

YAML syntax Copy Chapter

Resources Schema Copy Section

Identity Schema Copy Section

Examples Copy Chapter

Run examples are available in the GitHub repository.

• basic
• web-classification
• flow-with-additional-includes

Run with identity examples Copy Section

# default value
identity:
  type: user_identity

# use workspace primary UAI
identity:
  type: managed

# use specified client_id's UAI
identity:
  type: managed
  client_id: xxx

Reference Trace Span Spec Reference (`docs/reference/trace-span-spec-reference.md`) Copy Section

Trace Span Specification

:::{admonition} Experimental feature
This is an experimental feature, and may change at any time. Learn more.
:::

This document outlines the design of Prompt flow spans, detailing what information is traced and how it is structured.

Introduction Copy Chapter

The span, as you may know, is the fundamental unit of the trace system, representing a unit that captures execution information in the Prompt flow system. Spans are nested together in a parent-child relationship and paired together by link relationships, providing developers and users with a comprehensive view of the application’s execution process.

By adhering to these specifications, we ensure transparency and consistency in our tracing system.

The UI interprets the captured spans and presents them in a user-friendly manner. Understanding the fields and contracts defined within the spans is essential for effectively utilizing Prompt flow or integrating its components.

OpenTelemetry Span Basics Copy Chapter

A typical span object contains below information:

Span in Prompt flow Copy Chapter

In Prompt flow, we define several span types, and the system automatically creates spans with execution information in designated attributes and events.

These span types share common attributes and events, which we refer to as standard attributes and events. Let’s explore these common elements before diving into the specifics of each span type.

Common Attributes and Events Copy Section

Attributes

Each span in Prompt flow is enriched with a set of standard attributes that provide essential information about the span's context and purpose. The following table outlines these attributes:

[1]: Cumulative token counts are propagated up the span hierarchy, ensuring each span reflects the total token count of all LLM executions within its scope.

Events

In Prompt flow, events emitted by the Prompt flow framework follow the format below

• event MUST has attributes
• event attributes MUST contain a key named payload, which refers to the data carried within an event.
• event attributes payload MUST be a JSON string that represent an object.

Span Types Specification Copy Section

Within the Prompt flow system, we have delineated several distinct span types to cater to various execution units. Each span type is designed to capture specific execution information, complementing the standard attributes and events. Currently, our system includes the following span types: LLM, Function, LangChain, Flow, Embedding and Retrieval.

Beyond the standard attributes and events, each span type possesses designated fields to store pertinent information unique to its role within the system. These specialized attributes and events ensure that all relevant data is meticulously traced and available for analysis.

LLM

The LLM (Large Language Model) span captures detailed execution information from calls to large language models.

Note: OpenTelemetry currently defines several LLM-related span attributes and events as semantic conventions. We plan to align with these conventions in the future. For more information, visit Semantic Conventions for GenAI operations.

Function

The Function span is a versatile default span within Prompt flow, designed to capture a wide range of general function execution information.

[1]: Template formatting is a process by resolving prompt template into prompt message, this process can happen within a function that invokes LLM call.

Flow

The Flow span encapsulates the execution details of a flow within Prompt flow.

Embedding

The Embedding span is dedicated to recording the details of embedding calls within Prompt flow.

Retrieval

The Retrieval span type is specifically designed to encapsulate the execution details of a retrieval task within Prompt flow.

Cloud Azureai Consume Connections From Azure Ai (`docs/cloud/azureai/consume-connections-from-azure-ai.md`) Copy Section

Consume connections from Azure AI

For a smooth development flow that transitions from cloud (Azure AI) to local environments, you can directly utilize the connection already established on the cloud by setting the connection provider to "Azure AI connections".

:::{note}
Promptflow is not able to manage connections from Azure AI, please manage it in workspace portal, Azure AI Studio, az ml cli or azure-ai-ml sdk.
:::

You can set the connection provider using the following steps:

1. Navigate to the connection list in the VS Code primary sidebar.

2. Click on the ... (more options icon) at the top and select the Set connection provider option.

3. Choose one of the "Azure AI connections" provider types that you wish to use. Click to learn more about the differences between the connection providers.

   1. If you choose "Azure AI Connections - for current working directory", then you need to specify the cloud resources in the config.json file within the project folder.

   2. If you choose "Azure AI Connections - for this machine", specify the cloud resources in the connection string. You can do this in one of two ways:

      (1) Input connection string in the input box above.
      For example azureml://subscriptions/<your-subscription>/resourceGroups/<your-resourcegroup>/providers/Microsoft.MachineLearningServices/workspaces/<your-workspace>

      (2) Follow the wizard to set up your config step by step.

4. Once the connection provider is set, the connection list will automatically refresh, displaying the connections retrieved from the selected provider.

Note:

1. You need to have a project folder open to use the "Azure AI connections - for current working directory" option.
2. Once you change the connection provider, it will stay that way until you change it again and save the new setting.

Different connection providers Copy Chapter

Currently, we support three types of connections:

Next steps Copy Chapter

• Set global configs on connection.provider.
• Manage connections on local.

Cloud Azureai Create Run With Compute Session (`docs/cloud/azureai/create-run-with-compute-session.md`) Copy Section

Create run with compute session

:::{admonition} Experimental feature
This is an experimental feature, and may change at any time. Learn more.
:::

A prompt flow compute session provides computing resources that are required for the application to run, including a Docker image that contains all necessary dependency packages. This reliable and scalable compute session environment enables prompt flow to efficiently execute its tasks and functions for a seamless user experience.

If you're a new user, we recommend that you use the compute session (preview). You can easily customize the environment by adding packages in the requirements.txt file in flow.dag.yaml in the flow folder.

Create a run with compute session Copy Chapter

::::{tab-set}
:::{tab-item} CLI
:sync: CLI

pfazure run create --flow path/to/flow --data path/to/data --stream

:::

:::{tab-item} SDK
:sync: SDK

from promptflow.azure import PFClient


pf = PFClient(
    credential=credential,
    subscription_id="<SUBSCRIPTION_ID>",  # this will look like xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
    resource_group_name="<RESOURCE_GROUP>",
    workspace_name="<AML_WORKSPACE_NAME>",
)
pf.run(
    flow=flow,
    data=data,
)

:::
::::

Specify pip requirements for compute session Copy Chapter

If requirements.txt exists in the same folder with flow.dag.yaml.
The dependencies in it will be automatically installed for compute session.

You can also specify which requirements file to use in flow.dag.yaml like this:

$schema: https://azuremlschemas.azureedge.net/promptflow/latest/Flow.schema.json
environment:
  python_requirements_txt: path/to/requirement/file
...

Reference Flow YAML Schema for details.

Customize compute session Copy Chapter

In compute session case, you can also specify the instance type, if you don't specify the instance type, Azure Machine Learning chooses an instance type (VM size) based on factors like quota, cost, performance and disk size, learn more about serverless compute.

$schema: https://azuremlschemas.azureedge.net/promptflow/latest/Run.schema.json
flow: <path_to_flow>
data: <path_to_flow>/data.jsonl

column_mapping:
  url: ${data.url}

# define instance type only work for compute session.
resources:
  instance_type: <instance_type>

::::{tab-set}
:::{tab-item} CLI
:sync: CLI

pfazure run create --file run.yml

:::

:::{tab-item} SDK
:sync: SDK

from promptflow.client import load_run

run = load_run(source="run.yml")
pf = PFClient(
    credential=credential,
    subscription_id="<SUBSCRIPTION_ID>",  # this will look like xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
    resource_group_name="<RESOURCE_GROUP>",
    workspace_name="<AML_WORKSPACE_NAME>",
)
pf.runs.create_or_update(
    run=run
)

:::
::::

Next steps Copy Chapter

• Try the example here.

Cloud Azureai Deploy To Azure Appservice (`docs/cloud/azureai/deploy-to-azure-appservice.md`) Copy Section

Deploy to Azure App Service

Azure App Service is an HTTP-based service for hosting web applications, REST APIs, and mobile back ends.
The scripts (deploy.sh for bash and deploy.ps1 for powershell) under this folder are here to help deploy the docker image to Azure App Service.

This example demos how to deploy web-classification deploy a flow using Azure App Service.

Build a flow as docker format app Copy Chapter

Use the command below to build a flow as docker format app:

pf flow build --source ../../flows/standard/web-classification --output dist --format docker

Note that all dependent connections must be created before building as docker.

Deploy with Azure App Service Copy Chapter

The two scripts will do the following things:

1. Create a resource group if not exists.
2. Build and push the image to docker registry.
3. Create an app service plan with the given sku.
4. Create an app with specified name, set the deployment container image to the pushed docker image.
5. Set up the environment variables for the app.

::::{tab-set}
:::{tab-item} Bash
Example command to use bash script:

bash deploy.sh --path dist -i <image_tag> --name my-app-23d8m -r <docker registry> -g <resource_group>

See the full parameters by bash deploy.sh -h.
:::
:::{tab-item} PowerShell
Example command to use powershell script:

.\deploy.ps1 -Path dist -i <image_tag> -n my-app-23d8m -r <docker registry> -g <resource_group>

See the full parameters by .\deploy.ps1 -h.
:::
::::

Note that the name will produce a unique FQDN as AppName.azurewebsites.net.

View and test the web app Copy Chapter

The web app can be found via azure portal

After the app created, you will need to go to https://portal.azure.com/ find the app and set up the environment variables
at (Settings>Configuration) or (Settings>Environment variables), then restart the app.

The app can be tested by sending a POST request to the endpoint or browse the test page.
::::{tab-set}
:::{tab-item} Bash

curl https://<name>.azurewebsites.net/score --data '{"url":"https://play.google.com/store/apps/details?id=com.twitter.android"}' -X POST  -H "Content-Type: application/json"

:::
:::{tab-item} PowerShell

Invoke-WebRequest -URI https://<name>.azurewebsites.net/score -Body '{"url":"https://play.google.com/store/apps/details?id=com.twitter.android"}' -Method POST  -ContentType "application/json"

:::
:::{tab-item} Test Page
Browse the app at Overview and see the test page:

:::
::::

Tips:

• Reach deployment logs at (Deployment>Deployment Central) and app logs at (Monitoring>Log stream).
• Reach advanced deployment tools at (Development Tools>Advanced Tools).
• Reach more details about app service at Azure App Service.

Next steps Copy Chapter

• Try the example here.

Cloud Azureai Faq (`docs/cloud/azureai/faq.md`) Copy Section

Frequency asked questions (FAQ)

How to create feedback for azure portal experience Copy Chapter

For azure portal experience related issues, please create feedback on azure portal.

Token expired when run pfazure cmd Copy Chapter

If hit error "AADSTS700082: The refresh token has expired due to inactivity." when running pfazure cmd, it's caused by local cached token expired. Please clear the cached token under "%LOCALAPPDATA%/.IdentityService/msal.cache". Then run below command to login again:

az login

Cloud Azureai Generate Test Data Cloud (`docs/cloud/azureai/generate-test-data-cloud.md`) Copy Section

How to generate test data in cloud based on documents

This guide will help you learn how to generate test data on Azure AI, so that you can integrate the created flow and process a large amount of data.

Prerequisites Copy Chapter

1. Go through local test data generation guide and prepare your test data generation flow.

2. Go to the example_gen_test_data folder and run command pip install -r requirements_cloud.txt to prepare local environment.

3. Prepare cloud environment.

   • Navigate to file conda.yml.
   • For specific document file types, you may need to install extra packages:
     • .docx - pip install docx2txt
     • .pdf - pip install pypdf
     • .ipynb - pip install nbconvert

     !Note: We use llama index SimpleDirectoryReader to load documents. For the latest information on required packages, please check here.

4. Prepare Azure AI resources in cloud.

   • An Azure AI ML workspace - Create workspace resources you need to get started with Azure AI.
   • A compute target - Learn more about compute cluster.

5. Create cloud AzureOpenAI or OpenAI connection

6. Prepare test data generation setting.

   • Navigate to example_gen_test_data folder.
   • Prepare config.yml by copying config.yml.example.
   • Fill in configurations in the config.yml by following inline comment instructions.

Generate test data at cloud Copy Chapter

For handling larger test data, you can leverage the PRS component to run flow in cloud.

• Navigate to example_gen_test_data folder.

• After configuration, run the following command to generate the test data set:

  bash Copy

  python -m generate-test-data.run --cloud

• The generated test data will be a data asset which can be found in the output of the last node. You can register this data asset for future use.

Cloud Azureai Manage Flows (`docs/cloud/azureai/manage-flows.md`) Copy Section

Manage flows

This documentation will walk you through how to manage your flow with CLI and SDK on Azure AI.
The flow examples in this guide come from examples/flows/standard.

In general:

• For CLI, you can run pfazure flow --help in the terminal to see help messages.
• For SDK, you can refer to Promptflow Python Library Reference and check promptflow.azure.PFClient.flows for more flow operations.

:::{admonition} Prerequisites

• Refer to the prerequisites in Quick start.
• Use the az login command in the command line to log in. This enables promptflow to access your credentials.
  :::

Let's take a look at the following topics:

• Manage flows
  • Create a flow
  • List flows

Create a flow Copy Chapter

::::{tab-set}
:::{tab-item} CLI
:sync: CLI

To set the target workspace, you can either specify it in the CLI command or set default value in the Azure CLI.
You can refer to Quick start for more information.

To create a flow to Azure from local flow directory, you can use

# create the flow 
pfazure flow create --flow <path-to-flow-folder> 

# create the flow with metadata
pfazure flow create --flow <path-to-flow-folder> --set display_name=<display-name> description=<description> tags.key1=value1

After the flow is created successfully, you can see the flow summary in the command line.

:::

:::{tab-item} SDK
:sync: SDK

1. Import the required libraries

from azure.identity import DefaultAzureCredential, InteractiveBrowserCredential
# azure version promptflow apis
from promptflow.azure import PFClient

2. Get credential

try:
    credential = DefaultAzureCredential()
    # Check if given credential can get token successfully.
    credential.get_token("https://management.azure.com/.default")
except Exception as ex:
    # Fall back to InteractiveBrowserCredential in case DefaultAzureCredential not work
    credential = InteractiveBrowserCredential()

3. Get a handle to the workspace

# Get a handle to workspace
pf = PFClient(
    credential=credential,
    subscription_id="<SUBSCRIPTION_ID>",  # this will look like xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
    resource_group_name="<RESOURCE_GROUP>",
    workspace_name="<AML_WORKSPACE_NAME>",
)

4. Create the flow

# specify flow path
flow = "./web-classification"

# create flow to Azure
flow = pf.flows.create_or_update(
    flow=flow,  # path to the flow folder
    display_name="my-web-classification",  # it will be "web-classification-{timestamp}" if not specified
    type="standard",  # it will be "standard" if not specified
)

:::

::::

On Azure portal, you can see the created flow in the flow list.

And the flow source folder on file share is Users/<alias>/promptflow/<flow-display-name>:

Note that if the flow display name is not specified, it will default to the flow folder name + timestamp. (e.g. web-classification-11-13-2023-14-19-10)

List flows Copy Chapter

::::{tab-set}
:::{tab-item} CLI
:sync: CLI

List flows with default json format:

pfazure flow list --max-results 1

:::

:::{tab-item} SDK
:sync: SDK

# reuse the pf client created in "create a flow" section
flows = pf.flows.list(max_results=1)

:::

::::

Cloud Azureai Run Promptflow In Azure Ai (`docs/cloud/azureai/run-promptflow-in-azure-ai.md`) Copy Section

Run prompt flow in Azure AI

Assuming you have learned how to create and run a flow following Quick start. This guide will walk you through the main process of how to submit a promptflow run to Azure AI.

Benefits of use Azure AI comparison to just run locally:

• Designed for team collaboration: Portal UI is a better fix for sharing & presentation your flow and runs. And workspace can better organize team shared resources like connections.
• Enterprise Readiness Solutions: prompt flow leverages Azure AI's robust enterprise readiness solutions, providing a secure, scalable, and reliable foundation for the development, experimentation, and deployment of flows.

Prerequisites Copy Chapter

1. An Azure account with an active subscription - Create an account for free
2. An Azure AI ML workspace - Create workspace resources you need to get started with Azure AI.
3. A python environment, python=3.9 or higher version like 3.10 is recommended.
4. Install promptflow with extra dependencies and promptflow-tools.

pip install promptflow[azure] promptflow-tools

5. Clone the sample repo and check flows in folder examples/flows.

git clone https://github.com/microsoft/promptflow.git

Create necessary connections Copy Chapter

Connection helps securely store and manage secret keys or other sensitive credentials required for interacting with LLM and other external tools for example Azure Content Safety.

In this guide, we will use flow web-classification which uses connection open_ai_connection inside, we need to set up the connection if we haven't added it before.

Please go to workspace portal, click Prompt flow -> Connections -> Create, then follow the instruction to create your own connections. Learn more on connections.

Submit a run to workspace Copy Chapter

Assuming you are in working directory /examples/flows/standard/

::::{tab-set}

:::{tab-item} CLI
:sync: CLI

Use az login to login so promptflow can get your credential.

az login

Submit a run to workspace.

pfazure run create --subscription <my_sub> -g <my_resource_group> -w <my_workspace> --flow web-classification --data web-classification/data.jsonl --stream

Default subscription/resource-group/workspace

Note --subscription, -g and -w can be omitted if you have installed the Azure CLI and set the default configurations.

az account set --subscription <my-sub>
az configure --defaults group=<my_resource_group> workspace=<my_workspace>

Specify run name and view a run

You can also name the run by specifying --name my_first_cloud_run in the run create command, otherwise the run name will be generated in a certain pattern which has timestamp inside.

With a run name, you can easily stream or view the run details using below commands:

pfazure run stream -n my_first_cloud_run  # same as "--stream" in command "run create"
pfazure run show-details -n my_first_cloud_run
pfazure run visualize -n my_first_cloud_run

More details can be found in CLI reference: pfazure

:::

:::{tab-item} SDK
:sync: SDK

1. Import the required libraries

from azure.identity import DefaultAzureCredential, InteractiveBrowserCredential
# azure version promptflow apis
from promptflow.azure import PFClient

2. Get credential

try:
    credential = DefaultAzureCredential()
    # Check if given credential can get token successfully.
    credential.get_token("https://management.azure.com/.default")
except Exception as ex:
    # Fall back to InteractiveBrowserCredential in case DefaultAzureCredential not work
    credential = InteractiveBrowserCredential()

3. Get a handle to the workspace

# Get a handle to workspace
pf = PFClient(
    credential=credential,
    subscription_id="<SUBSCRIPTION_ID>",  # this will look like xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
    resource_group_name="<RESOURCE_GROUP>",
    workspace_name="<AML_WORKSPACE_NAME>",
)

4. Submit the flow run

# load flow
flow = "web-classification"
data = "web-classification/data.jsonl"

# create run
base_run = pf.run(
    flow=flow,
    data=data,
)

pf.stream(base_run)

5. View the run info

details = pf.get_details(base_run)
details.head(10)

pf.visualize(base_run)

:::

::::

View the run in workspace Copy Chapter

At the end of stream logs, you can find the portal_url of the submitted run, click it to view the run in the workspace.

Run snapshot of the flow with additional includes Copy Section

Flows that enabled additional include files can also be submitted for execution in the workspace. Please note that the specific additional include files or folders will be uploaded and organized within the Files folder of the run snapshot in the cloud.

Next steps Copy Chapter

Learn more about:

• CLI reference: pfazure

Cloud Azureai Runtime Change Log (`docs/cloud/azureai/runtime-change-log.md`) Copy Section

Change log of default runtime image

Runtime image Copy Chapter

In Azure Machine Learning prompt flow, runtime provides the environment to execute flows. The default runtime includes a pre-built Docker image, which contains all necessary dependent packages.

Pull image Copy Section

The image can be pulled by specifying a runtime version and executing the following command:

docker pull mcr.microsoft.com/azureml/promptflow/promptflow-runtime-stable:<runtime_version>

Check image version Copy Section

You can check the runtime image version from the flow execution log:

Change log Copy Chapter

Default runtime image is continuously updated, and here we record the new features and fixed bugs of each image version.

20251211.v2 Copy Section

New features

• NA

Bugs fixed

• Upgrade pip==25.3 go==1.24.9 mlflow>=3.6.0 to fix vulnerabilities

20251014.v1 Copy Section

New features

• NA

Bugs fixed

• fix GNU C Library and dpkg vulnerabilities in Ubuntu Image

20250831.v1 Copy Section

New features

• NA

Bugs fixed

• Update Ubuntu Image 22.04:latest to fix several security issues in Perl and SQLite

20250806.v3 Copy Section

New features

• NA

Bugs fixed

• Upgrade aiohttp>=3.12.14 to fix vulnerability

20250728.v1 Copy Section

New features

• NA

Bugs fixed

• Update Ubuntu Image 22.04 to fix several security issues in libssh and Git

20250220.v2 Copy Section

New features

• Update promptflow-tools to 1.6.0

Bugs fixed

• Pin marshmallow to ensure compatibility with older versions of promptflow for customer use.

20250131.v2 Copy Section

Promptflow version 1.17.1 or higher is required due to a known issue with marshmallow.

New features

• Update promptflow-tools to 1.5.0
• Update promptflow to 1.17.2

Bugs fixed

NA

20250109.v2 Copy Section

New features

• Update promptflow to 1.17.0
• PromptFlow-vectordb package update : To support langchain 0.3.x

Bugs fixed

• Update golang crypto, net to fix vulnerabilities
• Pin aiohttp in runtime image to fix vulnerability

20241214.v1 Copy Section

Do not use this version due to known issues and bugs.

New features

NA

Bugs fixed

• Update langchain-community>=0.3.0 to fix vulnerability
• Update mlflow>=2.16.0 to fix vulnerability

20241126.v3 Copy Section

New features

• Upgrade promptflow version to 1.16.2

Bugs fixed

NA

20241114.v2 Copy Section

New features

NA

Bugs fixed

• Update langchain-community>=0.2.19 to fix vulnerability

20241031.v1 Copy Section

New features

• Upgrade promptflow version to 1.16.1

Bugs fixed

NA

20241021.v1 Copy Section

New features

NA

Bugs fixed

• Upgrade langchain>=0.2.5 and langchain-community>=0.2.5,<0.3.0,!=0.2.14 to fix vulnerability

20240918.v2 Copy Section

New features

NA

Bugs fixed

NA

20240909.v2 Copy Section

New features

NA

Bugs fixed

NA

20240829.v1 Copy Section

New features

NA

Bugs fixed

• Add uvicorn dependency explicitly

20240814.v1 Copy Section

New features

NA

Bugs fixed

• Fix non serializable search results

20240731.v1 Copy Section

New features

• Add Reranker tool with Cohere accessibility to Prompt flow

Bugs fixed

• Fix run id duplicate check
• Upgrade langchain version in promptflow-vectordb-sdk

20240709.v1 Copy Section

New features

• cosmos db vector db support
• Build and publish promptflow-parallel

Bugs fixed

• Add aad auth check
• Fix metadata parser error

20240619.v2 Copy Section

New features

NA

Bugs fixed

• Add @blob_error_handling_decorator for cancelling_run and get_run_status in async storage
• Use dataplane api to list embedding deployments
• Fix KeyError when get run meet UserAuthenticationValidationError.

20240529.v1 Copy Section

New features

• [BYOI] Implement PostgreSQL
• [Lookup tool] Implement BYO-Weaviate Index

Bugs fixed

• The destination of _download_fileshare_directory should be str
• Passing DefaultAzureCredential into as_langchain_vectorstore
• Upgrade Werkzeug to 3.0.3.

20240515.v1 Copy Section

New features

• runtime support batch run prompty
• Support init_kwargs for multi-container c# batch run
• Upgrade gunicorn and mlflow

Bugs fixed

• Fix and avoid FlowExecutor in PythonApiExecutorProxy

20240511.v1 Copy Section

New features

• Add support for more huggingface models

Bugs fixed

• _download_azure_file_share should return str
• When ingress call gunicorn met timeout, we should not return 200.
• Pin azureml-rag version to 0.2.30.2
• Upgrade golang.org/x/net from 0.17.0 to 0.23.0 to fix vulnerability

20240429.v8 Copy Section

New features

• Remove preview tag from Index Lookup
• Support init_kwargs in c# batch run in cloud

Bugs fixed

• Pass init_kwargs to executor in multi container case
• Fix error of span.attributes.flow_log_path in multi-container.
• Add duplicated key in SystemLogContext to avoid breaking when LogContext does not have it
• Add clarification to migration notebook and small fixes
• [Lookup tool] Update list pinecone index endpoint
• Handle case where flow_logs_folder is None

20240424.v1 Copy Section

New features

• Support download working_dir from fileshare.
• Enable line log for batch run
• Add prompt_tokens/completion_tokens to run properties
• Support C# batch run in multi-container runtime
• Implement ESIndex in common index lookup
• Handle metadata processing better in Index Lookup
• implement mongodb support

Bugs fixed

• Remove Promptflow-evals publish.
• [Lookup tool] Add workspace name in get()
• changes to utils.py were accidentally left out of previous PR.

20240411.v4 Copy Section

New features

• Add 'detail' parameter on 'Azure OpenAI GPT-4 Turbo with Vision" tool and "OpenAI GPT-4V" tool.

Bugs fixed

• Resolve an intermittent ImportError that arose while loading the package tool.
• Upgrade langchain>=0.1.1 and langchain-core>=0.1.30 to fix vulnerability.

20240403.v2 Copy Section

New features

NA

Bugs fixed

NA

20240326.v2 Copy Section

New features

• Support environment variables for C# batch run.

Bugs fixed

NA

20240319.v1 Copy Section

New features

NA

Bugs fixed

NA

20240313.v1 Copy Section

New features

NA

Bugs fixed

• Fix an issue where calling a flow with a flow function would result in failure.
• Improve error handling by categorizing errors as user errors when a run is archived prior to being processed by the runtime.

20240306.v5 Copy Section

New features

• Support "seed" parameter for built-in LLM tools and GPT-4V tools.

Bugs fixed

• Handle ClientAuthenticationError properly.
• Fix appending blob exceeded size limit error by truncating debug info.

20240228.v3 Copy Section

New features

• Support async flow test for long running jobs.

Bugs fixed

• Fix bug when collecting package tools.

20240222.v3 Copy Section

New features

• Added support for executing C# batch runs in eager mode.
• Introduced the ability to specify the number of workers for batch runs.
• Implemented functionality to define a timeout for batch runs.

Bugs fixed

NA

20240205.v2 Copy Section

New features

NA