BentoML Documentation

File: README.md

Unified Model Serving Framework

🍱 Build model inference APIs and multi-model serving systems with any open-source or custom AI models. 👉 Join our forum!

What is BentoML?

BentoML is a Python library for building online serving systems optimized for AI apps and model inference.

🍱 Easily build APIs for Any AI/ML Model. Turn any model inference script into a REST API server with just a few lines of code and standard Python type hints.
🐳 Docker Containers made simple. No more dependency hell! Manage your environments, dependencies and model versions with a simple config file. BentoML automatically generates Docker images, ensures reproducibility, and simplifies how you deploy to different environments.
🧭 Maximize CPU/GPU utilization. Build high performance inference APIs leveraging built-in serving optimization features like dynamic batching, model parallelism, multi-stage pipeline and multi-model inference-graph orchestration.
👩‍💻 Fully customizable. Easily implement your own APIs or task queues, with custom business logic, model inference and multi-model composition. Supports any ML framework, modality, and inference runtime.
🚀 Ready for Production. Develop, run and debug locally. Seamlessly deploy to production with Docker containers or BentoCloud.

Getting started

Install BentoML:

text

# Requires Python≥3.9
pip install -U bentoml

Define APIs in a service.py file.

python

import bentoml

@bentoml.service(
    image=bentoml.images.Image(python_version="3.11").python_packages("torch", "transformers"),
)
class Summarization:
    def __init__(self) -> None:
        import torch
        from transformers import pipeline

        device = "cuda" if torch.cuda.is_available() else "cpu"
        self.pipeline = pipeline('summarization', device=device)

    @bentoml.api(batchable=True)
    def summarize(self, texts: list[str]) -> list[str]:
        results = self.pipeline(texts)
        return [item['summary_text'] for item in results]

💻 Run locally

Install PyTorch and Transformers packages to your Python virtual environment.

bash

pip install torch transformers  # additional dependencies for local run

Run the service code locally (serving at http://localhost:3000 by default):

bash

bentoml serve

You should expect to see the following output.

text

[INFO] [cli] Starting production HTTP BentoServer from "service:Summarization" listening on http://localhost:3000 (Press CTRL+C to quit)
[INFO] [entry_service:Summarization:1] Service Summarization initialized

Now you can run inference from your browser at http://localhost:3000 or with a Python script:

python

import bentoml

with bentoml.SyncHTTPClient('http://localhost:3000') as client:
    summarized_text: str = client.summarize([bentoml.__doc__])[0]
    print(f"Result: {summarized_text}")

🐳 Deploy using Docker

Run bentoml build to package necessary code, models, dependency configs into a Bento - the standardized deployable artifact in BentoML:

bash

bentoml build

Ensure Docker is running. Generate a Docker container image for deployment:

bash

bentoml containerize summarization:latest

Run the generated image:

bash

docker run --rm -p 3000:3000 summarization:latest

☁️ Deploy on BentoCloud

BentoCloud provides compute infrastructure for rapid and reliable GenAI adoption. It helps speed up your BentoML development process leveraging cloud compute resources, and simplify how you deploy, scale and operate BentoML in production.

bash

# After signup, run the following command to create an API token:
bentoml cloud login

# Deploy from current directory:
bentoml deploy

For detailed explanations, read the Hello World example.

Examples

LLMs: Llama 3.2, Mistral, DeepSeek Distil, and more.
Image Generation: Stable Diffusion 3 Medium, Stable Video Diffusion, Stable Diffusion XL Turbo, ControlNet, and LCM LoRAs.
Embeddings: SentenceTransformers and ColPali
Audio: ChatTTS, XTTS, WhisperX, Bark
Computer Vision: YOLO and ResNet
Advanced examples: Function calling, LangGraph, CrewAI

Check out the full list for more sample code and usage.

Advanced topics

See Documentation for more tutorials and guides.

Community

Get involved and join our Community Forum 💬, where thousands of AI/ML engineers help each other, contribute to the project, and talk about building AI products.

To report a bug or suggest a feature request, use
GitHub Issues.

Contributing

There are many ways to contribute to the project:

Report bugs and "Thumbs up" on issues that are relevant to you.
Investigate issues and review other developers' pull requests.
Contribute code or documentation to the project by submitting a GitHub pull request.
Check out the Contributing Guide and Development Guide to learn more.
Share your feedback and discuss roadmap plans in our forum.

Thanks to all of our amazing contributors!

Usage tracking and feedback

The BentoML framework collects anonymous usage data that helps our community improve the product. Only BentoML's internal API calls are being reported. This excludes any sensitive information, such as user code, model data, model names, or stack traces. Here's the code used for usage tracking. You can opt-out of usage tracking by the --do-not-track CLI option:

bash

bentoml [command] --do-not-track

Or by setting the environment variable:

bash

export BENTOML_DO_NOT_TRACK=True

License

Apache License 2.0

File: docs/source/build-with-bentoml/observability/index.rst

============= Observability

This section explains observability in BentoML.

.. grid:: 1 2 2 2
:gutter: 3
:margin: 0
:padding: 3 4 0 0

text

.. grid-item-card:: :doc:`/build-with-bentoml/observability/monitoring-and-data-collection`
    :link: /build-with-bentoml/observability/monitoring-and-data-collection
    :link-type: doc

    Learn how to implement monitoring and collect inference data in BentoML.

.. grid-item-card:: :doc:`/build-with-bentoml/observability/logging`
    :link: /build-with-bentoml/observability/logging
    :link-type: doc

    Learn how to configure logging in BentoML.

.. grid-item-card:: :doc:`/build-with-bentoml/observability/metrics`
    :link: /build-with-bentoml/observability/metrics
    :link-type: doc

    Learn how to configure metrics in BentoML.

.. grid-item-card:: :doc:`/build-with-bentoml/observability/tracing`
    :link: /build-with-bentoml/observability/tracing
    :link-type: doc

    Learn how to configure tracing in BentoML.

.. toctree::
:maxdepth: 1
:titlesonly:
:hidden:

text

monitoring-and-data-collection
logging
metrics
tracing

File: docs/source/build-with-bentoml/observability/logging.rst

======= Logging

BentoML provides a built-in logging system to provide comprehensive insights into the operation of your BentoML Services. It implements the OpenTelemetry <https://opentelemetry.io/docs/>_ standard to propagate critical information throughout the HTTP call stack for detailed debugging and analysis.

This document provides guidance on configuring logging in BentoML, including managing server-side logs and customizing logging for BentoML as a library.

Server logging

Server logging is enabled by default in BentoML. After a Service starts, every request made to the server is logged with detailed information. BentoML structures the logs to provide a clear and concise overview of each request, formatted as follows:

.. code-block:: bash

text

timestamp [LEVEL] [component] ClientIP:ClientPort (scheme,method,path,type,length) (status,type,length) Latency (trace,span,sampled,service.name)

An example of a log message for a request processed by BentoML might look like this:

.. code-block:: bash

text

2024-04-13T02:03:49+0000 [INFO] [entry_service:Summarization:1] 44.xxx.xxx.104:7181 (scheme=http,method=GET,path=/docs.json,type=,length=) (status=200,type=application/json,length=5543) 1.972ms (trace=7589d361df3e8ad3f0a71acb44d150be,span=07ef3bc1685d067c,sampled=0,service.name=Summarization)

This log entry provides detailed information about the request, including the client IP and port, request method, path, payload type and length, response status, response content type and length, request latency, and OpenTelemetry identifiers.

BentoML's logging system is fully compatible with the OpenTelemetry standard. The server log contains several OpenTelemetry parameters that are useful for correlating logs back to specific requests or operations.

trace: Identifies a trace, which consists of one or multiple spans that represent a single request flowing through multiple Services. See Traces <https://opentelemetry.io/docs/concepts/signals/traces/>_ for details.
span: Identifies an individual span within a trace. Each span represents a specific operation or a unit of work within the trace, such as a single HTTP request. See Spaces <https://opentelemetry.io/docs/concepts/signals/traces/#spans>_ for details.
sampled: Indicates whether a trace is being sampled, namely whether or not the trace data should be recorded. If sampling is enabled (usually denoted as 1 for sampled and 0 for not sampled), only a subset of traces is captured, which helps manage data volume and reduce performance overhead. See Sampling <https://opentelemetry.io/docs/concepts/sampling/>_ for details.

Configure logging
^^^^^^^^^^^^^^^^^

You can configure server logging in your Service definition by using the logging parameter in the @bentoml.service decorator.

.. code-block:: python

text

import bentoml

@bentoml.service(logging={
    "access": {
        "enabled": True,
        "request_content_length": True,
        "request_content_type": True,
        "response_content_length": True,
        "response_content_type": True,
        "skip_paths": ["/metrics", "/healthz", "/livez", "/readyz"],
        "format": {
            "trace_id": "032x",
            "span_id": "016x"
        }
    }
})
class MyService:
    # Service implementation

Available logging parameters to provide control over what data is logged and how it is formatted:

enabled: Enables or disables logging.
request_content_length: Logs the size of the request body.
request_content_type: Logs the content type of the request.
response_content_length: Logs the size of the response body.
response_content_type: Logs the content type of the response.
skip_paths: Specifies route paths that should be excluded from logging.
format: Customizes the logging format of OpenTelemetry trace identifiers.
- trace_id: Logs the trace identifier in a specified format, such as 032x.
- span_id: Logs the span identifier in a specified format, such as 016x.

To configure other logs, use the default Python logging configuration <https://docs.python.org/3/howto/logging.html>_. All BentoML logs are logged under the bentoml namespace.

Library logging

When you use BentoML as a library for your Python application, it does not configure any logs, without any specific handlers, formatters, or filters. This means that without additional configuration, the logging output from BentoML would follow the Python root logger's settings, which by default logs messages at the WARNING level and higher (including ERROR and CRITICAL).

To capture more detailed logs from BentoML, especially at the DEBUG or INFO levels, you must explicitly set up and register a log handler to the bentoml namespace. Here is a simple example of how to do this:

.. code-block:: python

text

import logging

# Create a stream handler
ch = logging.StreamHandler()

# Set a format for the handler
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
ch.setFormatter(formatter)

# Get the BentoML logger
bentoml_logger = logging.getLogger("bentoml")

# Add the handler to the BentoML logger
bentoml_logger.addHandler(ch)

# Set the desired logging level (e.g., DEBUG)
bentoml_logger.setLevel(logging.DEBUG)

.. note::

text

When starting a Service using ``bentoml serve``, the command forks ``service.py`` into a child process. Handlers that involve file operations, such as log rotation (``RotatingFileHandler`` or ``TimedRotatingFileHandler``), are not supported within the Service definition. For more information, see `the Python Logging Cookbook <https://docs.python.org/3/howto/logging-cookbook.html#logging-to-a-single-file-from-multiple-processes>`_.

File: docs/source/build-with-bentoml/observability/metrics.rst

======= Metrics

Metrics are important measurements that provide insights into the usage and performance of :doc:Services . BentoML provides a set of default metrics for performance analysis while you can also define custom metrics with Prometheus <https://prometheus.io/>_.

In this document, you will:

Learn and configure the default metrics in BentoML
Create custom metrics for BentoML Services
Use Prometheus <https://prometheus.io/>_ to scrape metrics
Create a Grafana <https://grafana.com/>_ dashboard to visualize metrics

Understand metrics

You can access metrics via the metrics endpoint of a BentoML Service. This endpoint is enabled by default and outputs metrics that Prometheus can scrape to monitor your Services continuously.

Default metrics
^^^^^^^^^^^^^^^

BentoML automatically collects a set of default metrics for each Service. These metrics are tracked across different dimensions to provide detailed visibility into Service operations:

.. list-table::
:widths: 30 10 60
:header-rows: 1

- Name
- Type
- Dimension
- bentoml_service_request_in_progress
- Gauge
- endpoint, runner_name, service_name, service_version
- bentoml_service_request_total
- Counter
- endpoint, service_name, runner_name, service_version, http_response_code
- bentoml_service_request_duration_seconds_sum, bentoml_service_request_duration_seconds_count, bentoml_service_request_duration_seconds_bucket
- Histogram
- endpoint, service_name, runner_name, service_version, http_response_code
- bentoml_service_adaptive_batch_size_sum, bentoml_service_adaptive_batch_size_count, bentoml_service_adaptive_batch_size_bucket
- Histogram
- method_name, service_name, runner_name, service_version, worker_index

request_in_progress: The number of requests that are currently being processed by a Service.
request_total: The total number of requests that a Service has processed.
request_duration_seconds: The time taken to process requests, including the total sum of request processing time, count of requests processed, and distribution across specified duration buckets.
adaptive_batch_size: The adaptive batch sizes used during Service execution, which is relevant for optimizing performance in batch processing scenarios. You need to enable :doc:adaptive batching </get-started/adaptive-batching> to collect this metric.

Metric types
^^^^^^^^^^^^

BentoML supports all metric types provided by Prometheus.

Gauge: A metric that represents a single numerical value that can arbitrarily go up and down.
Counter: A cumulative metric that only increases, useful for counting total requests.
Histogram: Tracks the number of observations and the sum of the observed values in configurable buckets, allowing you to calculate averages, percentiles, and so on.
Summary: Similar to Histogram but provides a total count of observations and a sum of observed values.

For more information, see the Prometheus documentation <https://prometheus.io/docs/concepts/metric_types/>_.

Dimensions
^^^^^^^^^^

Dimensions tracked for the default BentoML metrics include:

endpoint: The specific API endpoint being accessed.
runner_name: The name of the running Service handling the request.
service_name: The name of the Bento Service handling the request.
service_version: The version of the Service.
http_response_code: The HTTP response code of the request.
worker_index: The worker instance that is running the inference.

Configure default metrics

To customize how metrics are collected and reported in BentoML, use the metrics parameter within the @bentoml.service decorator:

.. code-block:: python

text

@bentoml.service(metrics={
    "enabled": True,
    "namespace": "custom_namespace",
})
class MyService:
    # Service implementation

enabled: This option is enabled by default. When enabled, you can access the metrics through the metrics endpoint of a BentoML Service.
namespace: Follows the labeling convention <https://prometheus.io/docs/practices/naming/#metric-and-label-naming>_ of Prometheus. The default namespace is bentoml_service, which covers most use cases.

Customize the duration bucket size
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

You can customize the duration bucket size <https://prometheus.io/docs/practices/histograms/#count-and-sum-of-observations>_ of request_duration_seconds in the following two ways:

Manual bucket definition. Specify explicit steps using buckets:

.. code-block:: python

text

@bentoml.service(metrics={
      "enabled": True,
      "namespace": "bentoml_service",
      "duration": {
          "buckets": [0.1, 0.2, 0.5, 1, 2, 5, 10]
      }
  })
  class MyService:
      # Service implementation

Exponential bucket generation. Automatically generate exponential buckets with any given min, max and factor values.
- min: The lower bound of the smallest bucket in the histogram.
- max: The upper bound of the largest bucket in the histogram.
- factor: Determines the exponential growth rate of the bucket sizes. Each subsequent bucket boundary is calculated by multiplying the previous boundary by the factor.
.. code-block:: python
text
```
@bentoml.service(metrics={
      "enabled": True,
      "namespace": "bentoml_service",
      "duration": {
          "min": 0.1,
          "max": 10,
          "factor": 1.2
      }
  })
  class MyService:
      # Service implementation
```

.. note::

text

- ``duration.min``, ``duration.max`` and ``duration.factor`` are mutually exclusive with ``duration.buckets``.
- ``duration.factor`` must be greater than 1 to ensure each subsequent bucket is larger than the previous one.
- The buckets for the ``adaptive_batch_size`` Histogram are calculated based on the ``max_batch_size`` defined. The bucket sizes start at 1 and increase exponentially up to the ``max_batch_size`` with a factor of 2.

By default, BentoML provides optimized histogram buckets ranging from 5ms to 180s, carefully distributed to monitor both fast API calls and long-running LLM/GenAI inference requests. The buckets are strategically placed to cover key latency ranges: fast API calls (5ms-50ms), regular API calls (100ms-1s), long API calls (2.5s-10s), and LLM inference (30s-180s).

Create custom metrics

You can define and use custom metrics of Counter, Histogram, Summary, and Gauge within your BentoML Service using the prometheus_client API.

Prerequisites
^^^^^^^^^^^^^

Install the Prometheus Python client <https://github.com/prometheus/client_python>_ package.

.. code-block:: bash

text

pip install prometheus-client

Define custom metrics
^^^^^^^^^^^^^^^^^^^^^

To define custom metrics, use the metric classes from the prometheus_client module and set the following parameters as needed:

name: A unique string identifier for the metric.
documentation: A description of what the metric measures.
labelnames: A list of strings defining the labels to apply to the metric. Labels add dimensions to the metric, which are useful for querying and aggregation purposes. When you record a metric, you specify the labels in the format <metric_object>.labels(<label_name>='<label_value>').<metric_function>. Once you define a label for a metric, all instances of that metric must include that label with some value.

The value of a label can also be dynamic, meaning it can change based on the context of the tracked metric. For example, you can use a label to log the version of model serving predictions, and this version label can change as you update the model.
buckets: A Histogram-specific parameter which defines the boundaries for Histogram buckets, useful for categorizing measurement ranges. The list should end with float('inf') to capture all values that exceed the highest defined boundary. See the Prometheus documentation on Histogram <https://prometheus.io/docs/practices/histograms/>_ for more details.

.. tab-set::

.. tab-item:: Histogram

text

.. code-block:: python

      import bentoml
      from prometheus_client import Histogram

      # Define Histogram metric
      inference_duration_histogram = Histogram(
          name="inference_duration_seconds",
          documentation="Time taken for inference",
          labelnames=["endpoint"],
          buckets=(
            0.005, 0.01, 0.025, 0.05,         # Fast API calls (5ms - 50ms)
            0.1, 0.25, 0.5, 1.0,              # Regular API calls (100ms - 1s)
            2.5, 5.0, 10.0,                   # Long API calls (2.5s - 10s)
            30.0, 60.0, 120.0, 180.0,         # LLM models (30s - 180s)
            float("inf"),
          ),
      )

      @bentoml.service
      class HistogramService:
          def __init__(self) -> None:
              # Initialization code

          @bentoml.api
          def infer(self, text: str) -> str:
              # Track the metric
              inference_duration_histogram.labels(endpoint='summarize').observe(512)
              # Implementation logic

.. tab-item:: Counter

text

.. code-block:: python

      import bentoml
      from prometheus_client import Counter

      # Define Counter metric
      inference_requests_counter = Counter(
          name="inference_requests_total",
          documentation="Total number of inference requests",
          labelnames=["endpoint"],
      )

      @bentoml.service
      class CounterService:
          def __init__(self) -> None:
              # Initialization code

          @bentoml.api
          def infer(self, text: str) -> str:
              # Track the metric
              inference_requests_counter.labels(endpoint='summarize').inc()  # Increment the counter by 1
              # Implementation logic

.. tab-item:: Summary

text

.. code-block:: python

      import bentoml
      from prometheus_client import Summary

      # Define Summary metric
      response_size_summary = Summary(
          name="response_size_bytes",
          documentation="Response size in bytes",
          labelnames=["endpoint"],
      )

      @bentoml.service
      class SummaryService:
          def __init__(self) -> None:
              # Initialization code

          @bentoml.api
          def infer(self, text: str) -> str:
              # Track the metric
              response_size_summary.labels(endpoint='summarize').observe(512)
              # Implementation logic

.. tab-item:: Gauge

text

.. code-block:: python

      import bentoml
      from prometheus_client import Gauge

      # Define Gauge metric
      in_progress_gauge = Gauge(
          name="in_progress_requests",
          documentation="In-progress inference requests",
          labelnames=["endpoint"],
      )

      @bentoml.service
      class GaugeService:
          def __init__(self) -> None:
              # Initialization code

          @bentoml.api
          def infer(self, text: str) -> str:
              # Track the metric
              in_progress_gauge.labels(endpoint='summarize').inc()  # Increment by 1
              in_progress_gauge.labels(endpoint='summarize').dec()  # Decrement by 1
              # Implementation logic

For more information on prometheus_client, see the Prometheus Python client library documentation <https://prometheus.github.io/client_python/>_.

An example with custom metrics
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

The following service.py file contains a custom Histogram and a Counter metric to measure the inference time and track the total number of requests.

.. code-block:: python

text

from __future__ import annotations
import bentoml
from prometheus_client import Histogram, Counter
from transformers import pipeline
import time

# Define the metrics
request_counter = Counter(
    name='summary_requests_total',
    documentation='Total number of summarization requests',
    labelnames=['status']
)

inference_time_histogram = Histogram(
    name='inference_time_seconds',
    documentation='Time taken for summarization inference',
    labelnames=['status'],
    buckets=(0.1, 0.2, 0.5, 1, 2, 5, 10, float('inf'))  # Example buckets
)

EXAMPLE_INPUT = "Breaking News: In an astonishing turn of events, the small town of Willow Creek has been taken by storm as local resident Jerry Thompson's cat, Whiskers, performed what witnesses are calling a 'miraculous and gravity-defying leap.' Eyewitnesses report that Whiskers, an otherwise unremarkable tabby cat, jumped a record-breaking 20 feet into the air to catch a fly. The event, which took place in Thompson's backyard, is now being investigated by scientists for potential breaches in the laws of physics. Local authorities are considering a town festival to celebrate what is being hailed as 'The Leap of the Century."

@bentoml.service(
    resources={"cpu": "2"},
    traffic={"timeout": 10},
)
class Summarization:
    def __init__(self) -> None:
        self.pipeline = pipeline('summarization')

    @bentoml.api
    def summarize(self, text: str = EXAMPLE_INPUT) -> str:
        start_time = time.time()
        try:
            result = self.pipeline(text)
            summary_text = result[0]['summary_text']
            # Capture successful requests
            status = 'success'
        except Exception as e:
            # Capture failures
            summary_text = str(e)
            status = 'failure'
        finally:
            # Measure how long the inference took and update the histogram
            inference_time_histogram.labels(status=status).observe(time.time() - start_time)
            # Increment the request counter
            request_counter.labels(status=status).inc()

        return summary_text

Run this Service locally:

.. code-block:: bash

text

bentoml serve

Make sure you have sent some requests to the summarize endpoint, then view the custom metrics by running the following command. You need to replace inference_time_seconds and summary_requests_total with your own metric names.

.. code-block:: bash

text

curl -X 'GET' 'http://localhost:3000/metrics' -H 'accept: */*' | grep -E 'inference_time_seconds|summary_requests_total'

Expected output:

.. code-block:: bash

text

# HELP summary_requests_total Total number of summarization requests
# TYPE summary_requests_total counter
summary_requests_total{status="success"} 12.0
# HELP inference_time_seconds Time taken for summarization inference
# TYPE inference_time_seconds histogram
inference_time_seconds_sum{status="success"} 51.74311947822571
inference_time_seconds_bucket{le="0.1",status="success"} 0.0
inference_time_seconds_bucket{le="0.2",status="success"} 0.0
inference_time_seconds_bucket{le="0.5",status="success"} 0.0
inference_time_seconds_bucket{le="1.0",status="success"} 0.0
inference_time_seconds_bucket{le="2.0",status="success"} 0.0
inference_time_seconds_bucket{le="5.0",status="success"} 12.0
inference_time_seconds_bucket{le="10.0",status="success"} 12.0
inference_time_seconds_bucket{le="+Inf",status="success"} 12.0
inference_time_seconds_count{status="success"} 12.0

Use Prometheus to scrape metrics

You can integrate Prometheus to scrape and visualize both default and custom metrics from your BentoML Service.

Install Prometheus <https://prometheus.io/docs/prometheus/latest/installation/>_.
Create a Prometheus configuration file <https://prometheus.io/docs/prometheus/latest/configuration/configuration/>_ to define scrape jobs. Here is an example that scrapes metrics every 5 seconds from a BentoML Service.

.. code-block:: yaml
:caption: prometheus.yml

global:
scrape_interval: 5s
evaluation_interval: 15s

scrape_configs:
- job_name: prometheus
metrics_path: "/metrics" # The metrics endpoint of the BentoML Service
static_configs:
- targets: ["0.0.0.0:3000"] # The address where the BentoML Service is running
Make sure you have a BentoML Service running, then start Prometheus in a different terminal session using the configuration file you created:

.. code-block:: bash
text
```
./prometheus --config.file=/path/to/the/file/prometheus.yml
```
Once Prometheus is running, access its web UI by visiting http://localhost:9090 in your web browser. This interface allows you to query and visualize the metrics collected from your BentoML Service.
Use PromQL expressions <https://prometheus.io/docs/prometheus/latest/querying/basics/>_ to query and visualize metrics. For example, to get the 99th percentile of request durations to the /encode endpoint over the last minute, use:

.. code-block:: bash
text
```
histogram_quantile(0.99, rate(bentoml_service_request_duration_seconds_bucket{endpoint="/encode"}[1m]))
```
.. image:: ../../_static/img/build-with-bentoml/observability/metrics/prome-ui-bentoml.png
:alt: Prometheus UI for BentoML metrics

Create a Grafana dashboard

Grafana is an analytics platform that allows you to create dynamic and informative dashboards <https://grafana.com/grafana/dashboards/>_ to visualize BentoML metrics. Do the following to create a Grafana dashboard.

Install Grafana <https://grafana.com/docs/grafana/latest/setup-grafana/installation/>_.
By default, Grafana runs on port 3000, which conflicts with BentoML's default port. To avoid this, change Grafana's default port. For example:

.. code-block:: bash
text
```
sudo nano /etc/grafana/grafana.ini
```
Find the [http] section and change http_port to a free port like 4000:

.. code-block:: bash
text
```
;http_port = 3000  # Change it to a port of your choice and uncomment the line by removing the semicolon
 http_port = 4000
```
Save the file and restart Grafana to apply the change:

.. code-block:: bash
text
```
sudo systemctl restart grafana-server
```
Access the Grafana web UI at http://localhost:4000/ (use your own port). Log in with the default credentials (admin/admin).
In the Grafana search box at the top, enter Data sources and add Prometheus as an available option. In Connection, set the URL to the address of your running Prometheus instance, such as http://localhost:9090. Save the configuration and test the connection to ensure Grafana can retrieve data from Prometheus.

.. image:: ../../_static/img/build-with-bentoml/observability/metrics/grafana-bentoml-1.png
:alt: Add Prometheus in Grafana
With Prometheus configured as a data source, you can create a new dashboard. Start by adding a panel and selecting a metric to visualize, such as bentoml_service_request_duration_seconds_bucket. Grafana offers a wide array of visualization options, from simple line graphs to more complex representations like heatmaps or gauges.

.. image:: ../../_static/img/build-with-bentoml/observability/metrics/grafana-bentoml-2.png
:alt: Grafana UI for BentoML metrics

For detailed instructions on dashboard creation and customization, read the Grafana documentation <https://grafana.com/docs/grafana/latest/dashboards/>_.

File: docs/source/build-with-bentoml/observability/monitoring-and-data-collection.rst

========== Monitoring

Data-centric AI is a paradigm that positions data as the cornerstone of AI systems. This approach emphasizes the importance of data quality and relevance, suggesting that the effectiveness of AI models can be primarily determined by the data they are trained on and interact with.

BentoML fully embraces this paradigm by offering APIs to implement a data-centric workflow, making it straightforward to collect inference data, monitor models, and ship the data to various destinations (for example, local storage, cloud services, and any OTLP supported tools) for your AI project.

This document explains how to implement monitoring and collect inference data in BentoML.

Benefits of model monitoring and data collection

Implementing an effective data collection and model monitoring strategy with BentoML offers the following benefits:

Key business metric monitoring: Track crucial statistical metrics to gauge the impact of your models on business objectives.
Early detection of data drift: Identify shifts in input data distributions early, helping decide when model retraining is necessary.
Quality assurance: Implement quality assurance for previously untracked metrics, such as model performance, accuracy, and degradation over time.
Enhanced interoperability: Promote collaboration between data science and operations teams, streamlining the process of training and iterating on models.
Informed decision-making: Provide insights into model behavior in production, informing future model improvements and iterations.

Built-in monitoring endpoints

BentoML Services automatically expose several built-in endpoints for health monitoring:

/livez: A liveness probe endpoint that checks if the Service is still alive and needs to restart.
/readyz: A readiness probe endpoint that indicates if the Service is ready to accept traffic.
/healthz: An alias of /livez.

These endpoints return a 200 status code and an empty response body when the Service is healthy.

After a Service starts, you can call these endpoints and get their status codes:

.. code-block:: python

import requests

base_url = "http://localhost:3000"

print(requests.get(f"{base_url}/livez").status_code)
print(requests.get(f"{base_url}/readyz").status_code)
print(requests.get(f"{base_url}/healthz").status_code)

You can customize the liveness and readiness endpoint paths using the endpoints argument. This is useful when your Service delegates traffic to an external process or application that exposes its own health check endpoints with non-standard paths.

.. code-block:: python

Use the @bentoml.service decorator:

@bentoml.service(endpoints={"livez": "/health", "readyz": "/ready"})
class MyService:
...

Or use the Service constructor:

MyService = bentoml.Service(
name="my_service",
cmd=["./bin/http"],
endpoints={"livez": "/health", "readyz": "/ready"}
)

Implement monitoring

In BentoML, you use the bentoml.monitor context manager to log data related to model inference. It allows you to specify a monitoring session where you can log various data types. This ensures that logging is structured and organized, making it easier to analyze the data later.

The following is an example of implementing monitoring in the Summarization Service in :doc:/get-started/hello-world.

.. code-block:: python

text

from __future__ import annotations
import bentoml
from transformers import pipeline

EXAMPLE_INPUT = "Breaking News: In an astonishing turn of events, the small town of Willow Creek has been taken by storm as local resident Jerry Thompson's cat, Whiskers, performed what witnesses are calling a 'miraculous and gravity-defying leap.' Eyewitnesses report that Whiskers, an otherwise unremarkable tabby cat, jumped a record-breaking 20 feet into the air to catch a fly. The event, which took place in Thompson's backyard, is now being investigated by scientists for potential breaches in the laws of physics. Local authorities are considering a town festival to celebrate what is being hailed as 'The Leap of the Century.'"

@bentoml.service(
    resources={"cpu": "2"},
    traffic={"timeout": 10},
)
class Summarization:
    def __init__(self) -> None:
        self.pipeline = pipeline('summarization')

    @bentoml.api
    def summarize(self, text: str = EXAMPLE_INPUT) -> str:
        # Use bentoml.monitor as a context manager
        with bentoml.monitor("text_summarization") as mon:
            # Log the input data
            mon.log(text, name="input_text", role="original_text", data_type="text")

            result = self.pipeline(text)
            summary_text = result[0]['summary_text']

            # Log the output data
            mon.log(summary_text, name="summarized_text", role="prediction", data_type="text")

            return summary_text

When you enter the bentoml.monitor context, you instantiate a monitoring session uniquely identified by a name ("text_summarization" in this example), which helps you categorize and retrieve the logged data for specific monitoring tasks.

Within the bentoml.monitor context, you use the log method to record individual data points, which requires several parameters to describe the data being logged:

data: The actual data value you want to log. This could be an input/output parameter value (the input text and output summary_text in this example).
name: A string identifier for the data point, which will be displayed in logs. This helps label the data in your logs, making it clear what each logged value represents.
role: The role of the data in model inference. Default roles include:
- "feature": Indicates that the logged data is an input feature of the model.
- "prediction": Indicates that the logged data is a prediction made by the model.
- "target": Indicates that the logged data is the target or label.
You can set a custom role like original_text in the example, which will also be logged.
data_type: The type of the data. Default data types include:
- "numerical": For quantitative data.
- "categorical": For discrete values representing categories.
- "numerical_sequence": For sequences or lists of numerical values.
You can set a custom data_type like text in the example, which will also be logged.

View request and schema logs

When the Service starts, BentoML exports the request and schema logs to the default directory monitoring/<your_monitor_name>, which contains the data and schema subdirectories.

The input and output data is stored in the data directory, including the corresponding timestamp and a unique request ID. To view the real-time data logs, run:

.. code-block:: bash

text

$ tail -f monitoring/text_summarization/data/*.log

{"input_text": "Breaking News: In an astonishing turn of events, the small town of Willow Creek has been taken by storm as local resident Jerry Thompson's cat, Whiskers, performed what witnesses are calling a 'miraculous and gravity-defying leap.' Eyewitnesses report that Whiskers, an otherwise unremarkable tabby cat, jumped a record-breaking 20 feet into the air to catch a fly. The event, which took place in Thompson's backyard, is now being investigated by scientists for potential breaches in the laws of physics. Local authorities are considering a town festival to celebrate what is being hailed as 'The Leap of the Century.'", "summarized_text": " Whiskers, an otherwise unremarkable tabby cat, jumped a record-breaking 20 feet into the air to catch a fly . The event is now being investigated by scientists for potential breaches in the laws of physics . Local authorities are considering a town festival to celebrate what is being hailed as 'The Leap of the Century'", "timestamp": "2024-03-05T03:33:59.490137", "request_id": "14642743634293743168"}
{"input_text": "Breaking News: In an astonishing turn of events, the small town of Willow Creek has been taken by storm as local resident Jerry Thompson's cat, Whiskers, performed what witnesses are calling a 'miraculous and gravity-defying leap.' Eyewitnesses report that Whiskers, an otherwise unremarkable tabby cat, jumped a record-breaking 20 feet into the air to catch a fly. The event, which took place in Thompson's backyard, is now being investigated by scientists for potential breaches in the laws of physics. Local authorities are considering a town festival to celebrate what is being hailed as 'The Leap of the Century.'", "summarized_text": " Whiskers, an otherwise unremarkable tabby cat, jumped a record-breaking 20 feet into the air to catch a fly . The event is now being investigated by scientists for potential breaches in the laws of physics . Local authorities are considering a town festival to celebrate what is being hailed as 'The Leap of the Century'", "timestamp": "2024-03-05T03:41:49.870589", "request_id": "7485759375304577245"}

The schema information is stored in the schema directory.

.. code-block:: bash

text

$ cat monitoring/text_summarization/schema/*.log

{"meta_data": {"bento_name": "", "bento_version": "not available"}, "columns": [{"name": "input_text", "role": "original_text", "type": "text"}, {"name": "summarized_text", "role": "prediction", "type": "text"}]}

BentoML logs request and schema data to rotating files. This means that new log files are created periodically or when the current log file reaches a certain size, older files are archived based on the default retention policy. You can customize the behavior by :ref:using a configuration file .

The actual data points are logged as JSON objects, providing a structured format for storing multiple records. This format is widely supported and can be easily ingested into various data analysis tools or databases for further processing.

Ship the collected data

BentoML provides a general monitoring data collection API. It allows you to transmit collected data to various destinations such as data warehouses, analytics pipelines, or specialized monitoring and drift detection solutions, all without requiring any modifications to your existing codebase.

The following table outlines the available targets for shipping monitoring data, the monitoring types (read the following sections for details), and additional notes.

.. list-table::
:widths: 33 33 34
:header-rows: 1

text

- - Destination
  - Monitoring type
  - Note
- - ``./monitoring/<name>/data/xxx.log``
  - ``default``
  - Logs are stored locally by default.
- - Cloud and monitoring services (Amazon S3, Azure Blob, Datadog, Elasticsearch, InfluxDB, Google BigQuery, Kafka, etc.)
  - ``otlp`` + deployed Fluent Bit
  - For more output options and configurations, see `Fluent Bit Outputs <https://docs.fluentbit.io/manual/pipeline/outputs>`_.
- - Any OTLP supported tools
  - ``otlp``
  - Useful for environments where direct file access is restricted, like AWS Lambda.
- - Arize
  - ``bentoml_plugins.arize.ArizeMonitor``
  - Ensure API keys and space keys are configured correctly.

.. _through-log-files:

Through log files
^^^^^^^^^^^^^^^^^

Writing monitoring data to log files is the most common way of data collection in BentoML, which is compatible with popular logging tools such as Fluent Bit <https://fluentbit.io/>, Filebeat <https://www.elastic.co/beats/filebeat>, and Logstash <https://www.elastic.co/logstash/>_. You can customize the monitoring configuration using the @bentoml.service decorator.

.. code-block:: python

text

...
@bentoml.service(
    resources={"cpu": "2"},
    traffic={"timeout": 10},
    monitoring={
        "enabled": True,
        "type": "default",
        "options": {
            "log_config_file": "path/to/log_config.yaml",
            "log_path": "monitoring"
        }
    }
)
class Summarization:
    # Service implementation code

Available fields for monitoring:

enabled: Whether monitoring is enabled for the Service. Setting it to True allows BentoML to collect and log data based on the specified configurations.
type: Specifies the type of monitoring system to use. The value default means the use of BentoML's built-in monitoring system, which collects data and logs it to files as shown in the previous section.
options: A dictionary that allows you to customize the monitoring setup.
- log_config_file: Specifies the path to a custom logging configuration file in YAML, which specifies logging behavior, such as log rotation policies, handlers, log formats, and log levels. The logging parameters should be set according to the Python logging module's configuration schema <https://docs.python.org/3/library/logging.html>_. If not provided, BentoML uses the default logging configuration, which is suitable for most use cases.
  
  Here is an example configuration file, which outputs log messages to a stream:
  
  .. code-block:: yaml
  text
```
version: 1
disable_existing_loggers: false
loggers:
  bentoml_monitor_data:
    level: INFO
    handlers: [bentoml_monitor_data]
    propagate: false
  bentoml_monitor_schema:
    level: INFO
    handlers: [bentoml_monitor_schema]
    propagate: false
handlers:
  bentoml_monitor_data:
    class: logging.StreamHandler
    stream: "ext://sys.stdout"
    level: INFO
    formatter: bentoml_json
  bentoml_monitor_schema:
    class: logging.StreamHandler
    stream: "ext://sys.stdout"
    level: INFO
    formatter: bentoml_json
formatters:
  bentoml_json:
    class: pythonjsonlogger.jsonlogger.JsonFormatter
    format: "()"
    validate: false
```
- log_path: Defines the directory where monitoring logs will be stored, which is relative to the Service's running location. It defaults to monitoring.

For deployments using :doc:the OCI-compliant image </get-started/packaging-for-deployment>, you can persist log files by mounting the specified log directory (monitoring in the example) to a volume. This ensures that your monitoring data is retained across container restarts and redeployments.

In Kubernetes, you can persist and ship logs by mounting the log directory and using a Fluent Bit <https://fluentbit.io/>_ DaemonSet or a sidecar container. This allows the collected log files to be automatically forwarded to your designated monitoring system or data warehouse, ensuring that your monitoring data is centralized and accessible for analysis and alerting.

Through an OTLP endpoint
^^^^^^^^^^^^^^^^^^^^^^^^

In scenarios where you can't directly access log files, such as when using AWS Lambda, as it doesn't support log files, BentoML supports exporting monitoring data to an external telemetry system using the OpenTelemetry Protocol (OTLP).

.. note::

text

Some log collectors like Fluent Bit also support OTLP input.

Below is an example of setting up OTLP for a BentoML Service:

.. code-block:: python

text

...
@bentoml.service(
    resources={"cpu": "2"},
    traffic={"timeout": 10},
    monitoring={
      "enabled": True,
      "type": "otlp",
      "options": {
        "endpoint": "http://localhost:5000",
        "insecure": True,
        "credentials": null,
        "headers": null,
        "timeout": 10,
        "compression": null,
        "meta_sample_rate": 1.0
      }
    }
)
class Summarization:
    # Service implementation code

Available parameters:

endpoint: Specifies the URL of the telemetry system's OTLP receiver. Data collected by BentoML will be sent to this endpoint.
insecure: A Boolean flag that specifies whether to disable transport security for the connection with the OTLP endpoint. Setting this to True means an insecure connection, which is typical for local or development environments.
credentials: If your OTLP endpoint requires authentication, you can use this parameter to provide credentials such as tokens or certificates. If set to null, it means that no authentication is required.
headers: Additional headers that may be required by the OTLP endpoint, useful for passing tokens or other necessary information.
timeout: Defines the maximum duration (in seconds) that BentoML will wait for a response from the OTLP endpoint before timing out.
compression: Specifies the type of compression to use when sending data. This can help reduce bandwidth usage. Supported values include gzip or none.
meta_sample_rate: Determines the sampling rate for sending metadata to the endpoint. A value of 1.0 means that all metadata is sent, while lower values reduce the frequency, sending only a percentage of the collected metadata.

For more information, see the OTLP documentation <https://opentelemetry.io/docs/specs/otel/protocol/exporter/>_.

Plugins and third-party monitoring data collectors
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

BentoML also supports plugins and third-party monitoring data collectors. You can create a custom monitoring data collector and publish it as a Python package. Unlike the built-in collector, which is more protocol specific for general use cases, plugins could be more platform-specific.

To use a plugin, you need to install it in the runtime environment for deployment. For details, see :doc:/build-with-bentoml/runtime-environment.

Arize AI
""""""""

For end-to-end solutions for data and model monitoring, BentoML works with Arize AI <https://arize.com/docs/>_ to provide a plugin for Arize. If you don't want to deploy a pipeline by yourself but still need data and model monitoring, Arize AI is a good choice. It provides a unified platform for data scientists, data engineers, and ML engineers to monitor, analyze, and debug ML models in production.

To use this plugin, make sure you have installed it first:

.. code-block:: bash

text

pip install bentoml-plugins-arize

In the @bentoml.service decorator, add the space_key and api_key to connect to your Arize account.

.. code-block:: python

text

...
@bentoml.service(
    resources={"cpu": "2"},
    traffic={"timeout": 10},
    monitoring={
      "enabled": True,
      "type": "bentoml_plugins.arize.ArizeMonitor",
      "options": {
        "space_key": <your_space_key>,
        "api_key": <your_api_key>
      }
    }
)
class Summarization:
  # Service implementation code

For more information about available Arize parameters, see the Arize documentation <https://docs.arize.com/arize/api-reference/python-sdk/arize.log>_.

For deployment, the plugin should also be installed in your runtime environment:

.. code-block:: python

text

import bentoml

my_image = bentoml.images.Image(python_version='3.11') \
    .python_packages("bentoml-plugins-arize") # Add this plugin

@bentoml.service(image=my_image)
class MyService:
    # Service implementation

File: docs/source/build-with-bentoml/observability/tracing.rst

======= Tracing

Tracing offers a continuous view of an application's behavior, tracking the flow of data as they move through different components of the application. BentoML provides built-in tracing support through OpenTelemetry, allowing users to export traces with Zipkin <https://zipkin.io/>, Jaeger <https://www.jaegertracing.io/>, and OTLP exporter <https://opentelemetry.io/>_.

This document explains how to configure tracing in BentoML.

Why do you need tracing?

Debugging models and services in production can be challenging. Traditional logging approaches often fall short when tracking issues across multiple services or components. Additionally, logs alone might not provide enough context or information to identify the root cause of a problem.

Tracing, on the other hand, offers several key benefits:

Comprehensive visibility: Tracing provides a holistic view of an application's flow, allowing developers to follow requests as they move through various components.
Performance monitoring: Tracing helps identify performance bottlenecks by showing where time is being spent within the application.
Cross-service correlation: Tracing helps track requests across multiple services, which is useful in distributed systems and microservice architectures.

Install dependencies

Before configuring tracing in BentoML, install the corresponding packages.

.. tab-set::

text

.. tab-item:: Zipkin

    .. code-block:: bash

    	pip install "bentoml[tracing-zipkin]"

.. tab-item:: Jaeger

    .. code-block:: bash

    	pip install "bentoml[tracing-jaeger]"

.. tab-item:: OTLP

    .. code-block:: bash

    	pip install "bentoml[tracing-otlp]"

Configure exporters

You configure a tracing exporter for a BentoML Service using the tracing field within the @bentoml.service decorator, where you define some common configurations, followed by the specific configurations of each exporter type.

Common configurations are shared values among the exporters, which means they will be applied to the corresponding exporter set via exporter_type. Here is an example:

.. code-block:: python

text

import bentoml

@bentoml.service(
    resources={"cpu": "2"},
    traffic={"timeout": 10},
    tracing={
        # Common configurations
        "exporter_type": "jaeger",
        "sample_rate": 1.0,
        "timeout": 5,
        "max_tag_value_length": 256,
        "excluded_urls": "readyz",
        "jaeger": {
            # Specific configurations of the exporter
    }
)
class MyService:
   # Service implementation code

Available fields in tracing:

exporter_type: The tracing exporter used for the Service. Supported values are jaeger, zipkin, and otlp.
sample_rate: By default, no traces will be collected. Set sample_rate to your desired fraction to start collecting them.
timeout: A timeout for the exporter, which waits for each batch export.
max_tag_value_length: A maximum length for string attribute values.
excluded_urls: Exclude routes from tracing. This parameter can be either a comma-separated string of routes, or a list of strings.

.. note::

text

BentoML implements OpenTelemetry APIs, which means OpenTelemetry environment variables will take precedence over the configurations set in the ``@bentoml.service`` decorator.

Zipkin
^^^^^^

When using Zipkin, BentoML only supports its V2 protocol. If you are reporting to an OpenZipkin server directly, add the URL path /api/v2/spans to the server address.

Here is an example of using Zipkin as the tracing exporter:

.. code-block:: python

text

import bentoml

@bentoml.service(
    resources={"cpu": "2"},
    traffic={"timeout": 60},
    tracing={
        "exporter_type": "zipkin",
        "sample_rate": 1.0,
        "zipkin": {
            "endpoint": "http://localhost:9411/api/v2/spans",
            "local_node_ipv4": "192.168.0.1",
            "local_node_ipv6": "2001:db8::c001",
            "local_node_port": 31313,
        }
   }
)
class MyService:
    # Service implementation code

Available fields for Zipkin, which are passed through the OpenTelemetry Zipkin Exporter <https://opentelemetry-python.readthedocs.io/en/latest/exporter/zipkin/zipkin.html>_:

endpoint: The endpoint URL where Zipkin spans are sent.
local_node_ipv4: The local node's IPv4 address.
local_node_ipv6: The local node's IPv6 address.
local_node_port: The local node's port.

With your BentoML Service configured, run Zipkin <https://zipkin.io/pages/quickstart>_ before starting the Service. For example, run the latest Zipkin Docker image as below:

.. code-block:: bash

text

docker run -d -p 9411:9411 openzipkin/zipkin

Start your BentoML Service and send some requests to it. You can then visit the Zipkin UI at http://localhost:9411/ to view the traces:

.. image:: ../../_static/img/build-with-bentoml/observability/tracing/zipkin-ui-tracing-bentoml.png
:alt: Zipkin UI for BentoML traces

Jaeger
^^^^^^

BentoML allows you to use Jaeger to collect traces over the Thrift protocol.

.. note::

text

When it is not feasible to deploy Jaeger Agent next to the application, for example, when the application code is running as a Lambda function, a collector can be configured to send spans using Thrift over HTTP. If both agent and collector are configured, the exporter sends traces only to the collector to eliminate the duplicate entries.

Here is an example of using Jaeger as the tracing exporter:

.. code-block:: python

text

import bentoml

@bentoml.service(
    resources={"cpu": "2"},
    traffic={"timeout": 60},
    tracing={
        "exporter_type": "jaeger",
        "sample_rate": 1.0,
        "jaeger": {
            "protocol": "thrift",
            "collector_endpoint": "http://localhost:14268/api/traces?format=jaeger.thrift",
            "thrift": {
                "agent_host_name": "localhost",
                "agent_port": 6831,
                "udp_split_oversized_batches": True,
            }
   	}
    }
)
class MyService:
    # Service implementation code

Available fields for Jaeger:

protocol: The protocol used for Jaeger.
collector_endpoint: The URL for the Jaeger collector endpoint.
thrift: A dictionary containing the specific configurations for the Thrift protocol.
- agent_host_name: The hostname of the Jaeger agent to which traces are sent.
- agent_port: The port on which the Jaeger agent listens for incoming traces.
- udp_split_oversized_batches: When it is set to True, the oversized batch will be split into smaller batches over the UDP max packet size (default: 65000) if the given buffer is larger than the max packet size:
.. math::

\text{packets} \triangleq \left\lceil \frac{\text{len}(\text{buff})}{\text{max_packet_size}} \right\rceil

With your BentoML Service configured, run Jaeger before starting the Service. For example, you can run the Jaeger all-in-one <https://www.jaegertracing.io/docs/1.57/getting-started/>_ Docker image as below:

.. code-block:: bash

text

docker run --rm --name jaeger \
  -e COLLECTOR_ZIPKIN_HOST_PORT=:9411 \
  -p 6831:6831/udp \
  -p 6832:6832/udp \
  -p 5778:5778 \
  -p 16686:16686 \
  -p 4317:4317 \
  -p 4318:4318 \
  -p 14250:14250 \
  -p 14268:14268 \
  -p 14269:14269 \
  -p 9411:9411 \
  jaegertracing/all-in-one:1.57

Start your BentoML Service and send some requests to it. You can then visit the Jaeger UI at http://localhost:16686/ to view the traces:

.. image:: ../../_static/img/build-with-bentoml/observability/tracing/jaeger-ui-tracing-bentoml.png
:alt: Jaeger UI for BentoML traces

OTLP exporter
^^^^^^^^^^^^^

BentoML supports the OTLP exporter for easy integration with an OpenTelemetry traces receiver. Currently, BentoML only supports the HTTP protocol for the OTLP exporter, which uses Protobuf to send traces over HTTP.

Here is an example of using the OTLP exporter:

.. code-block:: python

text

import bentoml

@bentoml.service(
    resources={"cpu": "2"},
    traffic={"timeout": 60},
    tracing={
        "exporter_type": "otlp",
        "sample_rate": 1.0,
        "otlp": {
            "protocol": "http",
            "endpoint": "http://localhost:4318/v1/traces",
            "http": {
                "certificate_file": "/path/to/cert.pem",
                "headers": {
                    "Keep-Alive": "timeout=5, max=1000",
                },
            },
        }
    }
)
class MyService:
...

Available fields for OTLP:

protocol: The protocol used for OTLP.
endpoint: The endpoint URL where OTLP spans are sent, namely the address of the OpenTelemetry receiver.
http: A dictionary containing specific configurations for the HTTP protocol.
- certificate_file: The path to the certificate file for secure HTTP communication.
- headers: A dictionary containing custom HTTP headers to include in the trace requests.

To start an OTLP exporter, you generally need an OpenTelemetry Collector running that can receive, process, and export telemetry data (traces, metrics, logs, etc.) to the desired backend. You can find the latest OpenTelemetry Collector releases on its GitHub repository <https://github.com/open-telemetry/opentelemetry-collector-releases>_ or read its Quick Start guide <https://opentelemetry.io/docs/collector/quick-start/>_.

The quickest way is to run the latest Docker image directly and output the tracing information to a separate file:

.. code-block:: bash

text

docker run \
  -p 127.0.0.1:4318:4318 \
  otel/opentelemetry-collector:0.100.0 \
  2>&1 | tee collector-output.txt

In a separate terminal, start your BentoML Service with otlp tracing configured and send some requests to it. You can then view the details of traces collected and processed by the OpenTelemetry Collector.

.. code-block:: bash
:caption: collector-output.txt

text

...
Resource SchemaURL:
Resource attributes:
     -> telemetry.sdk.language: Str(python)
     -> telemetry.sdk.name: Str(opentelemetry)
     -> telemetry.sdk.version: Str(1.20.0)
     -> service.name: Str(Summarization)
     -> service.instance.id: Int(1)
     -> service.version: Str(not available)
ScopeSpans #0
ScopeSpans SchemaURL:
InstrumentationScope opentelemetry.instrumentation.asgi 0.41b0
Span #0
    Trace ID       : 55f76a6172e5ef154bef3024cb7244a4
    Parent ID      : 80ebf4a29195d793
    ID             : 60bb910352f8279d
    Name           : POST /summarize http send
    Kind           : Internal
    Start time     : 2024-05-10 06:14:04.614485867 +0000 UTC
    End time       : 2024-05-10 06:14:04.614835736 +0000 UTC
    Status code    : Unset
    Status message :
Attributes:
     -> http.status_code: Int(200)
     -> type: Str(http.response.start)
Span #1
    Trace ID       : 55f76a6172e5ef154bef3024cb7244a4
    Parent ID      : 80ebf4a29195d793
    ID             : 027802ac3a807e5a
    Name           : POST /summarize http send
    Kind           : Internal
    Start time     : 2024-05-10 06:14:04.615378986 +0000 UTC
    End time       : 2024-05-10 06:14:04.615539025 +0000 UTC
    Status code    : Unset
    Status message :
Attributes:
     -> type: Str(http.response.body)
Span #2
    Trace ID       : 55f76a6172e5ef154bef3024cb7244a4
    Parent ID      :
    ID             : 80ebf4a29195d793
    Name           : POST /summarize
    Kind           : Server
    Start time     : 2024-05-10 06:13:59.653244649 +0000 UTC
    End time       : 2024-05-10 06:14:04.615805503 +0000 UTC
    Status code    : Unset
    Status message :
Attributes:
     -> http.scheme: Str(http)
     -> http.host: Str(xx.x.xxx.x:3000)
     -> net.host.port: Int(3000)
     -> http.flavor: Str(1.1)
     -> http.target: Str(/summarize)
     -> http.url: Str(http://xx.x.xxx.x:3000/summarize)
     -> http.method: Str(POST)
     -> http.server_name: Str(xxx.xxx.xx.xxx:3000)
     -> http.user_agent: Str(Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36)
     -> net.peer.ip: Str(xxx.xxx.xx.xx)
     -> net.peer.port: Int(59482)
     -> http.status_code: Int(200)
	{"kind": "exporter", "data_type": "traces", "name": "debug"}
2024-05-10T06:14:06.875Z	info	MetricsExporter	{"kind": "exporter", "data_type": "metrics", "name": "debug", "resource metrics": 1, "metrics": 22, "data points": 22}
2024-05-10T06:14:06.875Z	info	ResourceMetrics #0
...

.. note::

text

OTLP does not include a UI component. To visualize OTLP data, you need to use a backend that supports OTLP and provides a UI, such as Jaeger or Zipkin.

File: docs/source/build-with-bentoml/snippets/grpc/additional_setup.rst

Since there is no easy way to add additional proto files, we will have to clone some
repositories and copy the proto files into our project:

:github:protocolbuffers/protobuf - the official repository for the Protocol Buffers. We will need protobuf files that live under src/google/protobuf:

.. code-block:: bash

» mkdir -p thirdparty && cd thirdparty
» git clone --depth 1 https://github.com/protocolbuffers/protobuf.git

:github:bentoml/bentoml - We need the service.proto under bentoml/grpc/ to build the client; therefore, we will perform
a sparse checkout <https://github.blog/2020-01-17-bring-your-monorepo-down-to-size-with-sparse-checkout/>_ to only checkout bentoml/grpc directory:

.. code-block:: bash

» mkdir bentoml && pushd bentoml
» git init
» git remote add -f origin https://github.com/bentoml/BentoML.git
» git config core.sparseCheckout true
» cat <|.git/info/sparse-checkout
src/bentoml/grpc
EOT
» git pull origin main && mv src/bentoml/grpc .
» popd

File: docs/source/build-with-bentoml/snippets/grpc/grpc_tools.rst

.. tab-set::

.. tab-item:: gRPCurl

text

We will use :github:`fullstorydev/grpcurl` to send a CURL-like request to the gRPC BentoServer.

  Note that we will use `docker <https://docs.docker.com/get-docker/>`_ to run the ``grpcurl`` command.

  .. tab-set::

     .. tab-item:: MacOS/Windows
        :sync: __fullstorydev_macwin

        .. code-block:: bash

           » docker run -i --rm \
                          fullstorydev/grpcurl -d @ -plaintext host.docker.internal:3000 \
                          bentoml.grpc.v1.BentoService/Call <<EOT
           {
              "apiName": "classify",
              "ndarray": {
                 "shape": [1, 4],
                 "floatValues": [5.9, 3, 5.1, 1.8]
              }
           }
           EOT

     .. tab-item:: Linux
        :sync: __fullstorydev_linux

        .. code-block:: bash

           » docker run -i --rm \
                          --network=host \
                          fullstorydev/grpcurl -d @ -plaintext 0.0.0.0:3000 \
                          bentoml.grpc.v1.BentoService/Call <<EOT
           {
              "apiName": "classify",
              "ndarray": {
                 "shape": [1, 4],
                 "floatValues": [5.9, 3, 5.1, 1.8]
              }
           }
           EOT

.. tab-item:: gRPCUI

text

We will use :github:`fullstorydev/grpcui` to send request from a web browser.

  Note that we will use `docker <https://docs.docker.com/get-docker/>`_ to run the ``grpcui`` command.

  .. tab-set::

     .. tab-item:: MacOS/Windows
        :sync: __fullstorydev_macwin

        .. code-block:: bash

           » docker run --init --rm \
                          -p 8080:8080 fullstorydev/grpcui -plaintext host.docker.internal:3000

     .. tab-item:: Linux
        :sync: __fullstorydev_linux

        .. code-block:: bash

           » docker run --init --rm \
                          -p 8080:8080 \
                          --network=host fullstorydev/grpcui -plaintext 0.0.0.0:3000

  Proceed to http://127.0.0.1:8080 in your browser and send a test request from the web UI.

File: docs/source/build-with-bentoml/asgi.rst

======================= Mount ASGI applications

ASGI (Asynchronous Server Gateway Interface) <https://asgi.readthedocs.io/en/latest/>_ is a spiritual successor to WSGI (Web Server Gateway Interface), designed to provide a standard interface between async-capable Python web servers, frameworks, and applications. ASGI supports asynchronous request handling, allowing multiple requests to be processed at the same time, making it suitable for real-time web applications, such as WebSockets, long polling, and more.

BentoML's server runs the Service API in an ASGI web serving layer, exposing REST endpoints for inference APIs (for example, POST /summarize) and common infrastructure APIs (for example, GET /metrics) for monitoring. This ASGI-native web serving layer allows for direct mounting of existing ASGI applications, enabling them to serve side-by-side with BentoML Services.

This document explains how to mount an ASGI application onto a BentoML Service.

Why should you mount an ASGI application

Mounting an ASGI application, such as one built with FastAPI, onto a BentoML Service can be advantageous for several reasons:

Extended functionality: It enables you to extend your machine learning services with additional web functionalities, such as custom APIs for data processing, user management, or serving static files and web user interfaces, which are not directly related to model serving.
Custom authentication and authorization: By integrating an ASGI application, you can implement advanced authentication and authorization mechanisms, tailoring security measures to the specific needs of your application.
API documentation: With tools like FastAPI, you automatically get interactive API documentation, making it easier for end-users to understand and interact with the APIs.

Integrate BentoML with ASGI frameworks

BentoML offers seamless integration with different ASGI frameworks, allowing you to serve ML models alongside custom web application logic, such as asynchronous operations, real-time data processing, and complex web application functionalities.

When integrating the ASGI frameworks, you use bentoml.get_current_service() to retrieve the current BentoML Service instance. It is useful when you need to access the BentoML Service instance from within ASGI application routes or when injecting the Service instance into ASGI application routes using dependency injection patterns.

See the following examples for details.

FastAPI
^^^^^^^

FastAPI <https://fastapi.tiangolo.com/>_ is a web framework for building APIs that is built on top of ASGI, allowing it to handle asynchronous requests. To integrate a FastAPI application with a BentoML Service, you can define a FastAPI route either inside or outside the Service as below.

.. note::

Make sure you have installed FastAPI by running pip install fastapi. See the FastAPI documentation <https://fastapi.tiangolo.com/tutorial/>_ to learn more.

.. code-block:: python

text

from fastapi import FastAPI, Depends
import bentoml

app = FastAPI()

@bentoml.service
@bentoml.asgi_app(app, path="/v1")
class MyService:
    name = "MyService"

    @app.get('/hello')
    def hello(self):  # Inside service class, use `self` to access the service
        return f"Hello {self.name}"

@app.get("/hello1")
async def hello(service: MyService = Depends(bentoml.get_current_service)):
    # Outside service class, use `Depends` to get the service
    return f"Hello {service.name}"

Specifically, do the following to mount FastAPI:

Create a FastAPI application with FastAPI().
Use the @bentoml.asgi_app decorator to mount the FastAPI application to the BentoML Service, enabling them to be served together. Set the path parameter to :ref:customize the prefix path <customize-prefix-path>.
Define a FastAPI route inside or outside the Service class using @app.get("/<route-name>").
- Inside the class: Use self to access the Service instance's attributes and methods.
- Outside the class: Use FastAPI's dependency injection system (Depends) to inject the BentoML Service instance into the route function. In the code above, the hello1 route uses Depends(bentoml.get_current_service) to inject the MyService instance, allowing the route to access the Service's attributes and methods.
Within the FastAPI route, add your desired implementation logic. This example returns a greeting message using the Service's name.

.. note::

text

In addition to ``get``, you can use the other operations like ``post``, ``put``, and ``delete``. See `the FastAPI documentation <https://fastapi.tiangolo.com/tutorial/first-steps/>`_ to learn more.

.. dropdown:: Design choice: Inside vs. Outside

text

Accessing the BentoML Service instance both inside and outside the Service class offers flexibility in how you structure and interact with your Service logic and dependencies. The differences in accessing the BentoML Service instance in these contexts primarily relate to scope and the intended use cases.

Inside the Service class

- Direct access: Within the class defining a BentoML Service, you have direct access to ``self``, which represents the instance of the Service. This allows you to directly access its attributes and methods without injecting any dependency. It's the most straightforward way to use the Service's functionality from within its own definition.
- Contextual use: Accessing the Service instance inside the class is typical for defining the Service's internal logic, such as setting up endpoints, performing operations with the model, and handling requests directly related to the Service's primary functionality.

Outside the Service class

- Dependency injection: Accessing the BentoML Service instance outside the class typically requires dependency injection mechanisms, such as the ``Depends`` function in FastAPI. This approach is necessary when you want to use the Service instance in other parts of your project.
- Modular and decoupled design: This approach allows different components of your BentoML project to interact with the Service without being tightly integrated into its class definition. For example, your ML logic can be encapsulated within the BentoML Service, while other aspects, such as custom authentication, supplementary data processing, or additional REST endpoints, can be managed externally yet still interact with the Service as needed.

The following is a more practical example of mounting FastAPI onto the Summarization Service in :doc:/get-started/hello-world. It defines two additional endpoints with FastAPI by accessing the Service from inside and outside the class respectively.

.. code-block:: python

text

from __future__ import annotations
import bentoml
from transformers import pipeline
from fastapi import FastAPI, Depends

EXAMPLE_INPUT = "Breaking News: In an astonishing turn of events, the small town of Willow Creek has been taken by storm as local resident Jerry Thompson's cat, Whiskers, performed what witnesses are calling a 'miraculous and gravity-defying leap.' Eyewitnesses report that Whiskers, an otherwise unremarkable tabby cat, jumped a record-breaking 20 feet into the air to catch a fly. The event, which took place in Thompson's backyard, is now being investigated by scientists for potential breaches in the laws of physics. Local authorities are considering a town festival to celebrate what is being hailed as 'The Leap of the Century."

# Create a FastAPI app instance
app = FastAPI()

@bentoml.service(
    resources={"cpu": "2"},
    traffic={"timeout": 10},
)
@bentoml.asgi_app(app, path="/v1")
class Summarization:
    def __init__(self) -> None:
        self.pipeline = pipeline('summarization')

    # Define a name attribute
    name = "MyService"

    # The original Service API endpoint for text summarization
    @bentoml.api
    def summarize(self, text: str = EXAMPLE_INPUT) -> str:
        result = self.pipeline(text)
        return result[0]['summary_text']

    # Access the Service instance inside the class
    @app.get("/hello-inside")
    def hello(self):
        # Add other logic here if needed
        return f"Hello {self.name}. You can access the Service instance inside the class."

# Access the Service instance outside the class
@app.get("/hello-outside")
async def hello(service: MyService = Depends(bentoml.get_current_service)):
    # Add other logic here if needed
    return f"Hello {service.name}. You can access the Service instance outside the class."

After you start the BentoML Service, which is accessible at http://localhost:3000 <http://localhost:3000/>_, you can find two additional endpoints hello-inside and hello-outside exposed.

.. image:: ../../_static/img/build-with-bentoml/asgi/two-asgi-fastapi-routes.png
:alt: Two API endpoints defined in BentoML

By sending a GET request, you can receive the corresponding output from both endpoints.

FastAPI route inside the Service class:

.. image:: ../../_static/img/build-with-bentoml/asgi/inside-the-class.png
:alt: FastAPI route inside the BentoML Service class

FastAPI route outside the Service class:

.. image:: ../../_static/img/build-with-bentoml/asgi/outside-the-class.png
:alt: FastAPI route outside the BentoML Service class

Quart
^^^^^

Quart <https://quart.palletsprojects.com/en/latest/index.html>_ is an asynchronous web framework for Python that enables you to use async/await features in your web applications to handle large volumes of concurrent connections.

The following is an example of integrating Quart with BentoML.

.. note::

text

Make sure you have installed Quart by running ``pip install quart``. See `the Quart documentation <https://quart.palletsprojects.com/en/latest/tutorials/installation.html>`_ to learn more.

.. code-block:: python

text

from quart import Quart

app = Quart(__name__)

@app.get("/hello")
async def hello_world():
    service = bentoml.get_current_service()
    return f"Hello, {service.name}"

@bentoml.service
@bentoml.asgi_app(app, path="/v1")
class MyService:
    name = "MyService"

Specifically, do the following to mount Quart:

Create a Quart application with Quart().
Use the @bentoml.asgi_app decorator to mount the Quart application to the BentoML Service, enabling them to be served together. Set the path parameter to :ref:customize the prefix path <customize-prefix-path>.
Define a Quart route outside the Service class using @app.get(/"<route-name>"). Use bentoml.get_current_service() to inject the MyService instance, allowing the route to access the Service's attributes and methods.
Within the Quart route, add your desired implementation logic. This example returns a greeting message using the Service's name.

.. note::

text

In addition to ``get``, you can use the other operations like ``post``, ``put``, and ``delete``. See `the Quart documentation <https://quart.palletsprojects.com/en/latest/tutorials/index.html>`_ to learn more.

The following is a more practical example of mounting Quart onto the Summarization Service in :doc:/get-started/hello-world. It defines an additional endpoint hello.

.. code-block:: python

text

from __future__ import annotations
import bentoml
from transformers import pipeline
from quart import Quart

EXAMPLE_INPUT = "Breaking News: In an astonishing turn of events, the small town of Willow Creek has been taken by storm as local resident Jerry Thompson's cat, Whiskers, performed what witnesses are calling a 'miraculous and gravity-defying leap.' Eyewitnesses report that Whiskers, an otherwise unremarkable tabby cat, jumped a record-breaking 20 feet into the air to catch a fly. The event, which took place in Thompson's backyard, is now being investigated by scientists for potential breaches in the laws of physics. Local authorities are considering a town festival to celebrate what is being hailed as 'The Leap of the Century."

# Create a Quart app instance
app = Quart(__name__)

@app.get("/hello")
async def hello_world():
    service = bentoml.get_current_service()
    # Add other logic here if needed
    return f"Hello, {service.name}"

@bentoml.service(
    resources={"cpu": "2"},
    traffic={"timeout": 10},
)
@bentoml.asgi_app(app, path="/v1")
class Summarization:
    def __init__(self) -> None:
        self.pipeline = pipeline('summarization')

    # Define a name attribute
    name = "MyService"

    # The original Service API endpoint for text summarization
    @bentoml.api
    def summarize(self, text: str = EXAMPLE_INPUT) -> str:
        result = self.pipeline(text)
        return result[0]['summary_text']

After you start the BentoML Service, which is accessible at http://localhost:3000 <http://localhost:3000/>_, you can interact with the exposed endpoint hello. For example:

.. code-block:: bash

text

$ curl http://localhost:3000/v1/hello

Hello, MyService

.. note::

text

Unlike FastAPI, Quart does not natively support the OpenAPI specification, so the endpoint is not displayed on the Swagger UI. You can use other ways to communicate with it, such as ``curl``.

.. _customize-prefix-path:

Customize the prefix path

When mounting an ASGI tool onto a BentoML Service, it is possible to customize the route path by setting a prefix. This is useful for organizing your API endpoints and simplifying routing and namespace management.

To set a prefix path, simply set the path parameter in the decorator @bentoml.asgi_app. Here is a FastAPI example:

.. code-block:: python

text

from fastapi import FastAPI, Depends
import bentoml

app = FastAPI()

@bentoml.service
@bentoml.asgi_app(app, path="/fastapi") # Add the prefix here
class MyService:
    name = "MyService"

    @app.get('/hello')  # This endpoint should be requested via "/fastapi/hello"
    def hello(self):
        return f"Hello {self.name}"

By specifying path="/fastapi", the entire FastAPI application is served under this prefix. This means all the routes defined within the FastAPI application will be accessible under /fastapi. In this example, after you start this BentoML Service, you should interact with the /fastapi/hello endpoint.

Add custom ASGI middleware

add_asgi_middleware is an API provided by BentoML to apply custom ASGI middleware <https://asgi.readthedocs.io/en/latest/specs/main.html>_. Middleware functions as a layer that processes requests and responses, allowing you to manipulate them or execute additional actions based on specific conditions. It is commonly used for implementing security measures and custom headers, managing CORS, compressing responses, and more.

Example usage:

.. code-block:: python

text

from __future__ import annotations
import bentoml
from transformers import pipeline

from starlette.middleware.trustedhost import TrustedHostMiddleware

@bentoml.service(
    resources={"cpu": "2"},
    traffic={"timeout": 10},
)
class Summarization:
    def __init__(self) -> None:
        self.pipeline = pipeline('summarization')

    @bentoml.api
    def summarize(self, text: str) -> str:
        result = self.pipeline(text)
        return result[0]['summary_text']

# Add TrustedHostMiddleware to ensure the Service only accepts requests from certain hosts
Summarization.add_asgi_middleware(TrustedHostMiddleware, allowed_hosts=['example.com', '*.example.com'])

This example ensures that the Summarization Service only accepts requests from specified hosts and prevents host header attacks. You can then interact with the Service by manually specifying the Host header in requests:

.. code-block:: bash

text

curl -H "Host: example.com" http://localhost:3000

.. note::

text

Alternatively, you can edit your ``hosts`` file to map ``example.com`` to ``127.0.0.1`` (localhost) and then access ``http://example.com:3000/``.

While add_asgi_middleware is used to add middleware to the ASGI application that BentoML uses to serve the APIs, @bentoml.asgi_app is used to integrate the entire ASGI application into the BentoML Service. This is suitable for adding complete web applications like FastAPI or Quart applications that come with their routing logic, directly alongside your BentoML Service.

The middleware added via add_asgi_middleware applies to the entire ASGI application, including both the BentoML Service and any mounted ASGI applications. This ensures consistent processing of all requests across the application, whether they target BentoML Services or other components.

File: docs/source/build-with-bentoml/clients.rst

==================== Call an API endpoint

BentoML provides a client implementation that allows you to make synchronous and asynchronous requests to BentoML :doc:Services .

This document explains how to use :doc:BentoML clients </reference/bentoml/client>.

Client types

Depending on your requirements, you can create a BentoML client object using the following classes.

bentoml.SyncHTTPClient: Defines a synchronous client, suitable for straightforward, blocking operations where your application waits for the response before proceeding.
bentoml.AsyncHTTPClient: Defines an asynchronous client, suitable for non-blocking operations, allowing your application to handle other tasks while waiting for responses.

Create a client

Suppose your BentoML Service has an endpoint named summarize that takes a string text as input and returns a summarized version of the text as below.

.. code-block:: python

text

class Summarization:
    def __init__(self) -> None:
        # Load model into pipeline
        self.pipeline = pipeline('summarization')

    @bentoml.api
    def summarize(self, text: str) -> str:
        result = self.pipeline(text)
        return result[0]['summary_text']

After you start the Summarization Service, you can create the following clients by specifying the server address.

.. tab-set::

text

.. tab-item:: Synchronous

    .. code-block:: python

        import bentoml

        client = bentoml.SyncHTTPClient('http://localhost:3000')
        summarized_text: str = client.summarize(text="Your long text to summarize")
        print(summarized_text)

        # Close the client to release resources
        client.close()

.. tab-item:: Asynchronous

    .. code-block:: python

        import asyncio
        import bentoml

        async def async_client_operation():
            async with bentoml.AsyncHTTPClient('http://localhost:3000') as client:
                summarized_text: str = await client.summarize(text="Your long text to summarize")
                print(summarized_text)

        asyncio.run(async_client_operation())

In the above synchronous and asynchronous clients, requests are sent to the summarize endpoint of the Service hosted at http://localhost:3000. The BentoML client implementation supports methods corresponding to the Service APIs and they should be called with the same arguments (text in this example) as defined in the Service. These methods are dynamically created based on the Service's endpoints, providing a direct mapping to the Service's functionality.

In this example, the summarize method on the client is directly mapped to the summarize method in the Summarization Service. The data passed to the summarize method (text="Your long text to summarize") conforms to the expected input of the Service.

If you define :ref:root input <root-input> for an API endpoint, use only the positional argument in your client without specifying the parameter name.

.. note::

text

If you deploy your Service to BentoCloud, you can get the client of a Deployment by using ``get_client()`` or ``get_async_client()``. For more information, see :ref:`scale-with-bentocloud/deployment/call-deployment-endpoints:interact with the Deployment`.

Use a context manager
^^^^^^^^^^^^^^^^^^^^^

To enhance resource management and reduce the risk of connection leaks, we recommend you create a client within a context manager as below.

.. tab-set::

text

.. tab-item:: Synchronous

    .. code-block:: python

        import bentoml

        with bentoml.SyncHTTPClient('http://localhost:3000') as client:
            summarized_text: str = client.summarize(text="Your long text to summarize")
            print(summarized_text)

.. tab-item:: Asynchronous

    .. code-block:: python

        import bentoml

        async with bentoml.AsyncHTTPClient('http://localhost:3000') as client:
            summarized_text: str = await client.summarize(text="Your long text to summarize")
            print(summarized_text)

Check Service readiness

Before making calls to specific Service methods, you can use the is_ready method of the client to check if the Service is ready to handle requests. This ensures that your API calls are made only when the Service is up and running.

.. code-block:: python

text

import bentoml

client = bentoml.SyncHTTPClient('http://localhost:3000')
if client.is_ready():
    summarized_text: str = client.summarize(text="Your long text to summarize.")
    print("Summarized text:", summarized_text)
else:
    print("Service is not ready")

client.close()

Alternatively, use the server_ready_timeout parameter to specify the maximum duration in seconds the client will wait for the BentoML Service to become ready before timing out. This is useful during the initial connection to a Service that might be starting up. If the Service does not become ready within the specified timeout, the client will raise a timeout exception.

.. code-block:: python

text

import bentoml

client = bentoml.SyncHTTPClient(
  'http://localhost:3000',
  server_ready_timeout=60  # Wait up to 60 seconds for the Service to be ready
)
summarized_text: str = client.summarize(text="Your long text to summarize")
print(summarized_text)

client.close()

.. _call-a-task-endpoint:

Call a task endpoint

You can create clients to interact with Services defined with :ref:task endpoints by submitting inputs and then asynchronously checking for results at a later time. This is particularly useful for scenarios where the client does not need to actively wait for the task to complete. For more information, see :doc:/get-started/async-task-queues.

Input and output

BentoML clients support handling different input and output types.

JSON
^^^^

You can easily handle JSONable data input and JSON output with BentoML's HTTP clients, which are designed to seamlessly serialize and deserialize JSON data.

For input, when you send data that can be serialized to JSON (for example, dictionaries, lists, strings, and numbers), you simply pass it as arguments to the client method corresponding to your Service API.

The following code comes from the Service SentenceEmbedding of this example project <https://github.com/bentoml/BentoSentenceTransformers>_, which accepts JSONable input (lists in this case).

.. code-block:: python

text

import typing as t

@bentoml.service
class SentenceEmbedding:
    ...

    @bentoml.api
    def encode(self, sentences: t.List[str] = SAMPLE_SENTENCES) -> np.ndarray:
    ...

To create a client to handle JSONable input for Services like SentenceEmbedding:

.. code-block:: python

text

import bentoml
import typing as t

client = bentoml.SyncHTTPClient("http://localhost:3000")

# Specify the sentences for the request
sentences_list: t.List[str] = [
    "The sun dips below the horizon, painting the sky orange.",
    "A gentle breeze whispers through the autumn leaves.",
    "The moon casts a silver glow on the tranquil lake.",
    # Add more if necessary
]

# Make the request using the Service endpoint
result = client.encode(sentences=sentences_list)

# Print the result
print(f"Encoded sentences result: {result}")

client.close()

For output, when a BentoML Service returns JSON data, the client automatically deserializes this JSON into a Python data structure (like a dictionary or a list, depending on the JSON structure).

The following code comes from the Service WhisperX of this example project <https://github.com/bentoml/BentoWhisperX>_, which returns JSONable output (dictionaries in this case).

.. code-block:: python

text

import typing as t
from pathlib import Path

@bentoml.service
class WhisperX:
    ...

    @bentoml.api
    def transcribe(self, audio_file: Path) -> t.Dict:
    ...

To create a client to handle JSONable output for Services like WhisperX:

.. code-block:: python

text

import bentoml
import typing as t

client = bentoml.SyncHTTPClient('http://localhost:3000')

# Set the audio URL
audio_url = 'https://example.org/female.wav'

# The response is expected to be a dictionary
response: t.Dict = client.transcribe(audio_file=audio_url)

print(response)

.. tip::

text

You can print specific values of keys from the JSON response. For example, the Service ``WhisperX`` returns the following and you can output the text of the first segment:

.. code-block:: python

    response = {
        "segments": [
            {
                "start": 0.009,
                "end": 2.813,
                "text": " The Hispaniola was rolling scuppers under in the ocean swell.",
                "words": [
                    {"word": "The", "start": 0.009, "end": 0.069, "score": 0.0},
                    {"word": "Hispaniola", "start": 0.109, "end": 0.81, "score": 0.917},
                    # Other words omitted...
                ],
            },
            # Other segments omitted...
        ],
        "word_segments": [
            {"word": "The", "start": 0.009, "end": 0.069, "score": 0.0},
            {"word": "Hispaniola", "start": 0.109, "end": 0.81, "score": 0.917},
            # Other words omitted...
        ],
    }

    # Print the text of the first segment
    # Add the following line to your client code
    print("Segment text:", response["segments"][0]["text"])

Files
^^^^^

BentoML clients support a variety of file types, such as images and generic binary files.

For file input, you pass a Path object pointing to the file. The client handles the file reading and sends it as part of the request. For file output, the client provides the output as a Path object. You can use this Path object to access, read, or process the file.

The following code snippet comes from the :doc:/examples/controlnet example, which accepts and returns an image file.

.. code-block:: python

text

import PIL
from PIL.Image import Image as PIL_Image

@bentoml.service
class ControlNet:
    ...

    @bentoml.api
    async def generate(self, image: PIL_Image, params: Params) -> PIL_Image:
    ...

To create a client to handle file input and output for Services like ControlNet:

.. code-block:: python

text

import bentoml
from pathlib import Path

client = bentoml.SyncHTTPClient("http://localhost:3000")

# Specify the image path and other parameters for the request
image_path: Path = Path("/path/to/example-image.png")
params = {
    "prompt": "A young man walking in a park, wearing jeans.",
    "negative_prompt": "ugly, disfigured, ill-structure, low resolution",
    "controlnet_conditioning_scale": 0.5,
    "num_inference_steps": 25
}

# Make the request using the Service endpoint
result_path: Path = client.generate(
    image=image_path,
    params=params,
)

print(f"Generated file saved at: {result_path}")

client.close()

You can also use URLs as the input as below:

.. code-block:: python

text

import bentoml
from pathlib import Path

client = bentoml.SyncHTTPClient("http://localhost:3000")

# Specify the image URL and other parameters for the request
image_url = 'https://example.org/1.png'
# The remaining code is the same
...

Streaming
^^^^^^^^^

You can add streaming logic to a BentoML client, which is especially useful when dealing with large amounts of data or real-time data feeds. Streamed output is returned a generator or async generator, depending on the client type.

.. tab-set::

text

.. tab-item:: Synchronous

    For synchronous streaming, ``SyncHTTPClient`` uses a Python generator to output data as it is received from the stream.

    .. code-block:: python

        import bentoml

        client = bentoml.SyncHTTPClient("http://localhost:3000")
        for data_chunk in client.stream_data():
            # Process each chunk of data as it arrives
            process_data(data_chunk)

        client.close()

        def process_data(data_chunk):
            # Add processing logic
            print("Processing data chunk:", data_chunk)
            # Add more logic here to handle the data chunk

.. tab-item:: Asynchronous

    For asynchronous streaming, ``AsyncHTTPClient`` uses an async generator. This allows for asynchronous iteration over the streaming data.

    .. code-block:: python

        import bentoml

        async with bentoml.AsyncHTTPClient("http://localhost:3000") as client:
            async for data_chunk in client.stream_data():
                # Process each chunk of data as it arrives
                await process_data_async(data_chunk)

        async def process_data_async(data_chunk):
            # Add processing logic
            print("Processing data chunk asynchronously:", data_chunk)
            # Add more complex asynchronous processing here
            await some_async_operation(data_chunk)

Authorization

When working with BentoML Services that require authentication, you can authorize clients (SyncHTTPClient and AsyncHTTPClient) using a token. This token, typically a JWT (JSON Web Token) or some other form of API key, is used to ensure that the client is allowed to access the specified BentoML Service. The token is included in the HTTP headers of each request made by the client, allowing the server to validate the client's credentials.

To authorize a client, you pass the token as an argument during initialization.

.. code-block:: python

text

import bentoml

client = bentoml.SyncHTTPClient('http://localhost:3000', token='your_token_here')
summarized_text: str = client.summarize(text="Your long text to summarize")
print(summarized_text)

client.close()

Error handling

Handling errors, checking for error code and messages, and implementing retries are important for reliable client-server communication. Here are some strategies and examples on error handling and retries.

Basics
^^^^^^

When interacting with a BentoML Service, errors like network issues, Service downtime, or invalid input, may occur. Proper error handling allows your client to respond gracefully to these issues.

You can use try and except blocks to catch exceptions that may occur during the request:

.. code-block:: python

text

import bentoml
from bentoml.exceptions import BentoMLException

client = bentoml.SyncHTTPClient('http://localhost:3000')

try:
    summarized_text: str = client.summarize(text="Your long text to summarize.")
    print(summarized_text)
except BentoMLException as e:
    print(f"An error occurred: {e}")
finally:
    client.close()

When catching exceptions, it's useful to examine specific error code or messages to determine the cause of the failure. This can guide the retry logic or inform you about the issue more precisely.

Implement retry logic
^^^^^^^^^^^^^^^^^^^^^

Retrying failed requests can help overcome temporary issues like network disruptions or service unavailability. When implementing retries, consider exponential backoff to avoid overwhelming the server or the network.

Here's a simple example of implementing retries with exponential backoff.

.. code-block:: python

text

import time
from bentoml.exceptions import BentoMLException
import bentoml

def retry_request(client, max_retries=3, backoff_factor=2):
    for attempt in range(max_retries):
        try:
            summarized_text: str = client.summarize(text="Your long text to summarize.")
            return summarized_text
        except BentoMLException as e:
            print(f"Attempt {attempt+1}: An error occurred: {e}")
            time.sleep(backoff_factor ** attempt)
    print("Max retries reached. Giving up.")

client = bentoml.SyncHTTPClient('http://localhost:3000')

try:
    response = retry_request(client)
    if response:
        print(response)
finally:
    client.close()

Docker & Self-Hosted Deployment Verified OCI

File: README.md Copy Chapter

Unified Model Serving Framework Copy Chapter

What is BentoML? Copy Chapter

Getting started Copy Chapter

💻 Run locally Copy Section

🐳 Deploy using Docker Copy Section

☁️ Deploy on BentoCloud Copy Section

Examples Copy Chapter

Advanced topics Copy Chapter

Community Copy Chapter

Contributing Copy Section

Usage tracking and feedback Copy Section

License Copy Section

File: docs/source/build-with-bentoml/observability/index.rst Copy Chapter

============= Observability

File: docs/source/build-with-bentoml/observability/logging.rst Copy Chapter

======= Logging

Server logging Copy Chapter

Library logging Copy Chapter

File: docs/source/build-with-bentoml/observability/metrics.rst Copy Chapter

======= Metrics

Understand metrics Copy Chapter

Configure default metrics Copy Chapter

Create custom metrics Copy Chapter

Use Prometheus to scrape metrics Copy Chapter

Create a Grafana dashboard Copy Chapter

File: docs/source/build-with-bentoml/observability/monitoring-and-data-collection.rst Copy Chapter

========== Monitoring

Benefits of model monitoring and data collection Copy Chapter

Built-in monitoring endpoints Copy Chapter

Use the @bentoml.service decorator:

Or use the Service constructor:

Implement monitoring Copy Chapter

View request and schema logs Copy Chapter

Ship the collected data Copy Chapter

File: docs/source/build-with-bentoml/observability/tracing.rst Copy Chapter

======= Tracing

Why do you need tracing? Copy Chapter

Install dependencies Copy Chapter

Configure exporters Copy Chapter

File: docs/source/build-with-bentoml/snippets/grpc/additional_setup.rst Copy Chapter

File: docs/source/build-with-bentoml/snippets/grpc/grpc_tools.rst Copy Chapter

File: docs/source/build-with-bentoml/asgi.rst Copy Chapter

======================= Mount ASGI applications

Why should you mount an ASGI application Copy Chapter

Integrate BentoML with ASGI frameworks Copy Chapter

Customize the prefix path Copy Chapter

Add custom ASGI middleware Copy Chapter

File: docs/source/build-with-bentoml/clients.rst Copy Chapter

==================== Call an API endpoint

Client types Copy Chapter

Create a client Copy Chapter

Check Service readiness Copy Chapter

Call a task endpoint Copy Chapter

Input and output Copy Chapter

Authorization Copy Chapter

Error handling Copy Chapter

Explore Related Technologies & Topics

File: README.md

Unified Model Serving Framework

What is BentoML?

Getting started

💻 Run locally

🐳 Deploy using Docker

☁️ Deploy on BentoCloud

Examples

Advanced topics

Community

Contributing

Usage tracking and feedback

License

File: docs/source/build-with-bentoml/observability/index.rst

File: docs/source/build-with-bentoml/observability/logging.rst

Server logging

Library logging

File: docs/source/build-with-bentoml/observability/metrics.rst

Understand metrics

Configure default metrics

Create custom metrics

Use Prometheus to scrape metrics

Create a Grafana dashboard

File: docs/source/build-with-bentoml/observability/monitoring-and-data-collection.rst

Benefits of model monitoring and data collection

Built-in monitoring endpoints

Implement monitoring

View request and schema logs

Ship the collected data

File: docs/source/build-with-bentoml/observability/tracing.rst

Why do you need tracing?

Install dependencies

Configure exporters

File: docs/source/build-with-bentoml/snippets/grpc/additional_setup.rst

File: docs/source/build-with-bentoml/snippets/grpc/grpc_tools.rst

File: docs/source/build-with-bentoml/asgi.rst

Why should you mount an ASGI application

Integrate BentoML with ASGI frameworks

Customize the prefix path

Add custom ASGI middleware

File: docs/source/build-with-bentoml/clients.rst

Client types

Create a client

Check Service readiness

Call a task endpoint

Input and output

Authorization

Error handling