## File: README.md ## Unified Model Serving Framework 🍱 Build model inference APIs and multi-model serving systems with any open-source or custom AI models. 👉 [Join our forum](https://forum.modular.com/c/bento/31)! [](https://github.com/bentoml/BentoML?tab=Apache-2.0-1-ov-file) [](https://github.com/bentoml/bentoml/releases) [](https://github.com/bentoml/BentoML/actions/workflows/ci.yml?query=branch%3Amain) [](https://twitter.com/bentomlai) ## What is BentoML? BentoML is a Python library for building online serving systems optimized for AI apps and model inference. - **🍱 Easily build APIs for Any AI/ML Model.** Turn any model inference script into a REST API server with just a few lines of code and standard Python type hints. - **🐳 Docker Containers made simple.** No more dependency hell! Manage your environments, dependencies and model versions with a simple config file. BentoML automatically generates Docker images, ensures reproducibility, and simplifies how you deploy to different environments. - **🧭 Maximize CPU/GPU utilization.** Build high performance inference APIs leveraging built-in serving optimization features like dynamic batching, model parallelism, multi-stage pipeline and multi-model inference-graph orchestration. - **👩‍💻 Fully customizable.** Easily implement your own APIs or task queues, with custom business logic, model inference and multi-model composition. Supports any ML framework, modality, and inference runtime. - **🚀 Ready for Production.** Develop, run and debug locally. Seamlessly deploy to production with Docker containers or [BentoCloud](https://www.bentoml.com/). ## Getting started Install BentoML: ``` # Requires Python≥3.9 pip install -U bentoml ``` Define APIs in a `service.py` file. ```python import bentoml @bentoml.service( image=bentoml.images.Image(python_version="3.11").python_packages("torch", "transformers"), ) class Summarization: def __init__(self) -> None: import torch from transformers import pipeline device = "cuda" if torch.cuda.is_available() else "cpu" self.pipeline = pipeline('summarization', device=device) @bentoml.api(batchable=True) def summarize(self, texts: list[str]) -> list[str]: results = self.pipeline(texts) return [item['summary_text'] for item in results] ``` ### 💻 Run locally Install PyTorch and Transformers packages to your Python virtual environment. ```bash pip install torch transformers # additional dependencies for local run ``` Run the service code locally (serving at http://localhost:3000 by default): ```bash bentoml serve ``` You should expect to see the following output. ``` [INFO] [cli] Starting production HTTP BentoServer from "service:Summarization" listening on http://localhost:3000 (Press CTRL+C to quit) [INFO] [entry_service:Summarization:1] Service Summarization initialized ``` Now you can run inference from your browser at http://localhost:3000 or with a Python script: ```python import bentoml with bentoml.SyncHTTPClient('http://localhost:3000') as client: summarized_text: str = client.summarize([bentoml.__doc__])[0] print(f"Result: {summarized_text}") ``` ### 🐳 Deploy using Docker Run `bentoml build` to package necessary code, models, dependency configs into a Bento - the standardized deployable artifact in BentoML: ```bash bentoml build ``` Ensure [Docker](https://docs.docker.com/) is running. Generate a Docker container image for deployment: ```bash bentoml containerize summarization:latest ``` Run the generated image: ```bash docker run --rm -p 3000:3000 summarization:latest ``` ### ☁️ Deploy on BentoCloud [BentoCloud](https://www.bentoml.com) provides compute infrastructure for rapid and reliable GenAI adoption. It helps speed up your BentoML development process leveraging cloud compute resources, and simplify how you deploy, scale and operate BentoML in production. [Sign up for BentoCloud](https://cloud.bentoml.com/signup) for personal access; for enterprise use cases, [contact our team](https://www.bentoml.com/contact). ```bash # After signup, run the following command to create an API token: bentoml cloud login # Deploy from current directory: bentoml deploy ``` For detailed explanations, read the [Hello World example](https://docs.bentoml.com/en/latest/get-started/hello-world.html). ## Examples - LLMs: [Llama 3.2](https://github.com/bentoml/BentoVLLM/tree/main/llama3.2-11b-vision-instruct), [Mistral](https://github.com/bentoml/BentoVLLM/tree/main/ministral-8b-instruct-2410), [DeepSeek Distil](https://github.com/bentoml/BentoVLLM/tree/main/deepseek-r1-distill-llama3.1-8b-tool-calling), and more. - Image Generation: [Stable Diffusion 3 Medium](https://github.com/bentoml/BentoDiffusion/tree/main/sd3-medium), [Stable Video Diffusion](https://github.com/bentoml/BentoDiffusion/tree/main/svd), [Stable Diffusion XL Turbo](https://github.com/bentoml/BentoDiffusion/tree/main/sdxl-turbo), [ControlNet](https://github.com/bentoml/BentoDiffusion/tree/main/controlnet), and [LCM LoRAs](https://github.com/bentoml/BentoDiffusion/tree/main/lcm). - Embeddings: [SentenceTransformers](https://github.com/bentoml/BentoSentenceTransformers) and [ColPali](https://github.com/bentoml/BentoColPali) - Audio: [ChatTTS](https://github.com/bentoml/BentoChatTTS), [XTTS](https://github.com/bentoml/BentoXTTS), [WhisperX](https://github.com/bentoml/BentoWhisperX), [Bark](https://github.com/bentoml/BentoBark) - Computer Vision: [YOLO](https://github.com/bentoml/BentoYolo) and [ResNet](https://github.com/bentoml/BentoResnet) - Advanced examples: [Function calling](https://github.com/bentoml/BentoFunctionCalling), [LangGraph](https://github.com/bentoml/BentoLangGraph), [CrewAI](https://github.com/bentoml/BentoCrewAI) Check out the [full list](https://docs.bentoml.com/en/latest/examples/overview.html) for more sample code and usage. ## Advanced topics - [Model composition](https://docs.bentoml.com/en/latest/get-started/model-composition.html) - [Workers and model parallelization](https://docs.bentoml.com/en/latest/build-with-bentoml/parallelize-requests.html) - [Adaptive batching](https://docs.bentoml.com/en/latest/get-started/adaptive-batching.html) - [GPU inference](https://docs.bentoml.com/en/latest/build-with-bentoml/gpu-inference.html) - [Distributed serving systems](https://docs.bentoml.com/en/latest/build-with-bentoml/distributed-services.html) - [Concurrency and autoscaling](https://docs.bentoml.com/en/latest/scale-with-bentocloud/scaling/autoscaling.html) - [Model loading and Model Store](https://docs.bentoml.com/en/latest/build-with-bentoml/model-loading-and-management.html) - [Observability](https://docs.bentoml.com/en/latest/build-with-bentoml/observability/index.html) - [BentoCloud deployment](https://docs.bentoml.com/en/latest/get-started/cloud-deployment.html) See [Documentation](https://docs.bentoml.com) for more tutorials and guides. ## Community Get involved and join our [Community Forum 💬](https://forum.modular.com/c/bento/31), where thousands of AI/ML engineers help each other, contribute to the project, and talk about building AI products. To report a bug or suggest a feature request, use [GitHub Issues](https://github.com/bentoml/BentoML/issues/new/choose). ### Contributing There are many ways to contribute to the project: - Report bugs and "Thumbs up" on [issues](https://github.com/bentoml/BentoML/issues) that are relevant to you. - Investigate [issues](https://github.com/bentoml/BentoML/issues) and review other developers' [pull requests](https://github.com/bentoml/BentoML/pulls). - Contribute code or [documentation](https://docs.bentoml.com/en/latest/index.html) to the project by submitting a GitHub pull request. - Check out the [Contributing Guide](https://github.com/bentoml/BentoML/blob/main/CONTRIBUTING.md) and [Development Guide](https://github.com/bentoml/BentoML/blob/main/DEVELOPMENT.md) to learn more. - Share your feedback and discuss roadmap plans in our [forum](https://forum.modular.com/c/bento/31). Thanks to all of our amazing contributors! ### Usage tracking and feedback The BentoML framework collects anonymous usage data that helps our community improve the product. Only BentoML's internal API calls are being reported. This excludes any sensitive information, such as user code, model data, model names, or stack traces. Here's the [code](https://github.com/bentoml/BentoML/blob/main/src/bentoml/_internal/utils/analytics/usage_stats.py) used for usage tracking. You can opt-out of usage tracking by the `--do-not-track` CLI option: ```bash bentoml [command] --do-not-track ``` Or by setting the environment variable: ```bash export BENTOML_DO_NOT_TRACK=True ``` ### License [Apache License 2.0](https://github.com/bentoml/BentoML/blob/main/LICENSE) --- ## File: docs/source/build-with-bentoml/observability/index.rst ============= Observability ============= This section explains observability in BentoML. .. grid:: 1 2 2 2 :gutter: 3 :margin: 0 :padding: 3 4 0 0 .. grid-item-card:: :doc:`/build-with-bentoml/observability/monitoring-and-data-collection` :link: /build-with-bentoml/observability/monitoring-and-data-collection :link-type: doc Learn how to implement monitoring and collect inference data in BentoML. .. grid-item-card:: :doc:`/build-with-bentoml/observability/logging` :link: /build-with-bentoml/observability/logging :link-type: doc Learn how to configure logging in BentoML. .. grid-item-card:: :doc:`/build-with-bentoml/observability/metrics` :link: /build-with-bentoml/observability/metrics :link-type: doc Learn how to configure metrics in BentoML. .. grid-item-card:: :doc:`/build-with-bentoml/observability/tracing` :link: /build-with-bentoml/observability/tracing :link-type: doc Learn how to configure tracing in BentoML. .. toctree:: :maxdepth: 1 :titlesonly: :hidden: monitoring-and-data-collection logging metrics tracing --- ## File: docs/source/build-with-bentoml/observability/logging.rst ======= Logging ======= BentoML provides a built-in logging system to provide comprehensive insights into the operation of your BentoML Services. It implements the `OpenTelemetry `_ standard to propagate critical information throughout the HTTP call stack for detailed debugging and analysis. This document provides guidance on configuring logging in BentoML, including managing server-side logs and customizing logging for BentoML as a library. Server logging -------------- Server logging is enabled by default in BentoML. After a Service starts, every request made to the server is logged with detailed information. BentoML structures the logs to provide a clear and concise overview of each request, formatted as follows: .. code-block:: bash timestamp [LEVEL] [component] ClientIP:ClientPort (scheme,method,path,type,length) (status,type,length) Latency (trace,span,sampled,service.name) An example of a log message for a request processed by BentoML might look like this: .. code-block:: bash 2024-04-13T02:03:49+0000 [INFO] [entry_service:Summarization:1] 44.xxx.xxx.104:7181 (scheme=http,method=GET,path=/docs.json,type=,length=) (status=200,type=application/json,length=5543) 1.972ms (trace=7589d361df3e8ad3f0a71acb44d150be,span=07ef3bc1685d067c,sampled=0,service.name=Summarization) This log entry provides detailed information about the request, including the client IP and port, request method, path, payload type and length, response status, response content type and length, request latency, and OpenTelemetry identifiers. BentoML's logging system is fully compatible with the OpenTelemetry standard. The server log contains several OpenTelemetry parameters that are useful for correlating logs back to specific requests or operations. - ``trace``: Identifies a trace, which consists of one or multiple spans that represent a single request flowing through multiple Services. See `Traces `_ for details. - ``span``: Identifies an individual span within a trace. Each span represents a specific operation or a unit of work within the trace, such as a single HTTP request. See `Spaces `_ for details. - ``sampled``: Indicates whether a trace is being sampled, namely whether or not the trace data should be recorded. If sampling is enabled (usually denoted as ``1`` for sampled and ``0`` for not sampled), only a subset of traces is captured, which helps manage data volume and reduce performance overhead. See `Sampling `_ for details. Configure logging ^^^^^^^^^^^^^^^^^ You can configure server logging in your Service definition by using the ``logging`` parameter in the ``@bentoml.service`` decorator. .. code-block:: python import bentoml @bentoml.service(logging={ "access": { "enabled": True, "request_content_length": True, "request_content_type": True, "response_content_length": True, "response_content_type": True, "skip_paths": ["/metrics", "/healthz", "/livez", "/readyz"], "format": { "trace_id": "032x", "span_id": "016x" } } }) class MyService: # Service implementation Available logging parameters to provide control over what data is logged and how it is formatted: - ``enabled``: Enables or disables logging. - ``request_content_length``: Logs the size of the request body. - ``request_content_type``: Logs the content type of the request. - ``response_content_length``: Logs the size of the response body. - ``response_content_type``: Logs the content type of the response. - ``skip_paths``: Specifies route paths that should be excluded from logging. - ``format``: Customizes the logging format of OpenTelemetry trace identifiers. - ``trace_id``: Logs the trace identifier in a specified format, such as ``032x``. - ``span_id``: Logs the span identifier in a specified format, such as ``016x``. To configure other logs, use the `default Python logging configuration `_. All BentoML logs are logged under the ``bentoml`` namespace. Library logging --------------- When you use BentoML as a library for your Python application, it does not configure any logs, without any specific handlers, formatters, or filters. This means that without additional configuration, the logging output from BentoML would follow the Python root logger's settings, which by default logs messages at the WARNING level and higher (including ERROR and CRITICAL). To capture more detailed logs from BentoML, especially at the ``DEBUG`` or ``INFO`` levels, you must explicitly set up and register a log handler to the ``bentoml`` namespace. Here is a simple example of how to do this: .. code-block:: python import logging # Create a stream handler ch = logging.StreamHandler() # Set a format for the handler formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s') ch.setFormatter(formatter) # Get the BentoML logger bentoml_logger = logging.getLogger("bentoml") # Add the handler to the BentoML logger bentoml_logger.addHandler(ch) # Set the desired logging level (e.g., DEBUG) bentoml_logger.setLevel(logging.DEBUG) .. note:: When starting a Service using ``bentoml serve``, the command forks ``service.py`` into a child process. Handlers that involve file operations, such as log rotation (``RotatingFileHandler`` or ``TimedRotatingFileHandler``), are not supported within the Service definition. For more information, see `the Python Logging Cookbook `_. --- ## File: docs/source/build-with-bentoml/observability/metrics.rst ======= Metrics ======= Metrics are important measurements that provide insights into the usage and performance of :doc:`Services `. BentoML provides a set of default metrics for performance analysis while you can also define custom metrics with `Prometheus `_. In this document, you will: - Learn and configure the default metrics in BentoML - Create custom metrics for BentoML Services - Use `Prometheus `_ to scrape metrics - Create a `Grafana `_ dashboard to visualize metrics Understand metrics ------------------ You can access metrics via the ``metrics`` endpoint of a BentoML Service. This endpoint is enabled by default and outputs metrics that Prometheus can scrape to monitor your Services continuously. Default metrics ^^^^^^^^^^^^^^^ BentoML automatically collects a set of default metrics for each Service. These metrics are tracked across different dimensions to provide detailed visibility into Service operations: .. list-table:: :widths: 30 10 60 :header-rows: 1 * - Name - Type - Dimension * - ``bentoml_service_request_in_progress`` - Gauge - ``endpoint``, ``runner_name``, ``service_name``, ``service_version`` * - ``bentoml_service_request_total`` - Counter - ``endpoint``, ``service_name``, ``runner_name``, ``service_version``, ``http_response_code`` * - ``bentoml_service_request_duration_seconds_sum``, ``bentoml_service_request_duration_seconds_count``, ``bentoml_service_request_duration_seconds_bucket`` - Histogram - ``endpoint``, ``service_name``, ``runner_name``, ``service_version``, ``http_response_code`` * - ``bentoml_service_adaptive_batch_size_sum``, ``bentoml_service_adaptive_batch_size_count``, ``bentoml_service_adaptive_batch_size_bucket`` - Histogram - ``method_name``, ``service_name``, ``runner_name``, ``service_version``, ``worker_index`` - ``request_in_progress``: The number of requests that are currently being processed by a Service. - ``request_total``: The total number of requests that a Service has processed. - ``request_duration_seconds``: The time taken to process requests, including the total sum of request processing time, count of requests processed, and distribution across specified duration buckets. - ``adaptive_batch_size``: The adaptive batch sizes used during Service execution, which is relevant for optimizing performance in batch processing scenarios. You need to enable :doc:`adaptive batching ` to collect this metric. Metric types ^^^^^^^^^^^^ BentoML supports all metric types provided by Prometheus. - ``Gauge``: A metric that represents a single numerical value that can arbitrarily go up and down. - ``Counter``: A cumulative metric that only increases, useful for counting total requests. - ``Histogram``: Tracks the number of observations and the sum of the observed values in configurable buckets, allowing you to calculate averages, percentiles, and so on. - ``Summary``: Similar to Histogram but provides a total count of observations and a sum of observed values. For more information, see `the Prometheus documentation `_. Dimensions ^^^^^^^^^^ Dimensions tracked for the default BentoML metrics include: - ``endpoint``: The specific API endpoint being accessed. - ``runner_name``: The name of the running Service handling the request. - ``service_name``: The name of the Bento Service handling the request. - ``service_version``: The version of the Service. - ``http_response_code``: The HTTP response code of the request. - ``worker_index``: The worker instance that is running the inference. Configure default metrics ------------------------- To customize how metrics are collected and reported in BentoML, use the ``metrics`` parameter within the ``@bentoml.service`` decorator: .. code-block:: python @bentoml.service(metrics={ "enabled": True, "namespace": "custom_namespace", }) class MyService: # Service implementation - ``enabled``: This option is enabled by default. When enabled, you can access the metrics through the ``metrics`` endpoint of a BentoML Service. - ``namespace``: Follows the `labeling convention `_ of Prometheus. The default namespace is ``bentoml_service``, which covers most use cases. Customize the duration bucket size ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ You can customize the `duration bucket size `_ of ``request_duration_seconds`` in the following two ways: - **Manual bucket definition**. Specify explicit steps using ``buckets``: .. code-block:: python @bentoml.service(metrics={ "enabled": True, "namespace": "bentoml_service", "duration": { "buckets": [0.1, 0.2, 0.5, 1, 2, 5, 10] } }) class MyService: # Service implementation - **Exponential bucket generation**. Automatically generate exponential buckets with any given ``min``, ``max`` and ``factor`` values. - ``min``: The lower bound of the smallest bucket in the histogram. - ``max``: The upper bound of the largest bucket in the histogram. - ``factor``: Determines the exponential growth rate of the bucket sizes. Each subsequent bucket boundary is calculated by multiplying the previous boundary by the factor. .. code-block:: python @bentoml.service(metrics={ "enabled": True, "namespace": "bentoml_service", "duration": { "min": 0.1, "max": 10, "factor": 1.2 } }) class MyService: # Service implementation .. note:: - ``duration.min``, ``duration.max`` and ``duration.factor`` are mutually exclusive with ``duration.buckets``. - ``duration.factor`` must be greater than 1 to ensure each subsequent bucket is larger than the previous one. - The buckets for the ``adaptive_batch_size`` Histogram are calculated based on the ``max_batch_size`` defined. The bucket sizes start at 1 and increase exponentially up to the ``max_batch_size`` with a factor of 2. By default, BentoML provides optimized histogram buckets ranging from 5ms to 180s, carefully distributed to monitor both fast API calls and long-running LLM/GenAI inference requests. The buckets are strategically placed to cover key latency ranges: fast API calls (5ms-50ms), regular API calls (100ms-1s), long API calls (2.5s-10s), and LLM inference (30s-180s). Create custom metrics --------------------- You can define and use custom metrics of ``Counter``, ``Histogram``, ``Summary``, and ``Gauge`` within your BentoML Service using the ``prometheus_client`` API. Prerequisites ^^^^^^^^^^^^^ Install the `Prometheus Python client `_ package. .. code-block:: bash pip install prometheus-client Define custom metrics ^^^^^^^^^^^^^^^^^^^^^ To define custom metrics, use the metric classes from the ``prometheus_client`` module and set the following parameters as needed: - ``name``: A unique string identifier for the metric. - ``documentation``: A description of what the metric measures. - ``labelnames``: A list of strings defining the labels to apply to the metric. Labels add dimensions to the metric, which are useful for querying and aggregation purposes. When you record a metric, you specify the labels in the format ``.labels(='').``. Once you define a label for a metric, all instances of that metric must include that label with some value. The value of a label can also be dynamic, meaning it can change based on the context of the tracked metric. For example, you can use a label to log the version of model serving predictions, and this version label can change as you update the model. - ``buckets``: A Histogram-specific parameter which defines the boundaries for Histogram buckets, useful for categorizing measurement ranges. The list should end with ``float('inf')`` to capture all values that exceed the highest defined boundary. See the Prometheus documentation on `Histogram `_ for more details. .. tab-set:: .. tab-item:: Histogram .. code-block:: python import bentoml from prometheus_client import Histogram # Define Histogram metric inference_duration_histogram = Histogram( name="inference_duration_seconds", documentation="Time taken for inference", labelnames=["endpoint"], buckets=( 0.005, 0.01, 0.025, 0.05, # Fast API calls (5ms - 50ms) 0.1, 0.25, 0.5, 1.0, # Regular API calls (100ms - 1s) 2.5, 5.0, 10.0, # Long API calls (2.5s - 10s) 30.0, 60.0, 120.0, 180.0, # LLM models (30s - 180s) float("inf"), ), ) @bentoml.service class HistogramService: def __init__(self) -> None: # Initialization code @bentoml.api def infer(self, text: str) -> str: # Track the metric inference_duration_histogram.labels(endpoint='summarize').observe(512) # Implementation logic .. tab-item:: Counter .. code-block:: python import bentoml from prometheus_client import Counter # Define Counter metric inference_requests_counter = Counter( name="inference_requests_total", documentation="Total number of inference requests", labelnames=["endpoint"], ) @bentoml.service class CounterService: def __init__(self) -> None: # Initialization code @bentoml.api def infer(self, text: str) -> str: # Track the metric inference_requests_counter.labels(endpoint='summarize').inc() # Increment the counter by 1 # Implementation logic .. tab-item:: Summary .. code-block:: python import bentoml from prometheus_client import Summary # Define Summary metric response_size_summary = Summary( name="response_size_bytes", documentation="Response size in bytes", labelnames=["endpoint"], ) @bentoml.service class SummaryService: def __init__(self) -> None: # Initialization code @bentoml.api def infer(self, text: str) -> str: # Track the metric response_size_summary.labels(endpoint='summarize').observe(512) # Implementation logic .. tab-item:: Gauge .. code-block:: python import bentoml from prometheus_client import Gauge # Define Gauge metric in_progress_gauge = Gauge( name="in_progress_requests", documentation="In-progress inference requests", labelnames=["endpoint"], ) @bentoml.service class GaugeService: def __init__(self) -> None: # Initialization code @bentoml.api def infer(self, text: str) -> str: # Track the metric in_progress_gauge.labels(endpoint='summarize').inc() # Increment by 1 in_progress_gauge.labels(endpoint='summarize').dec() # Decrement by 1 # Implementation logic For more information on ``prometheus_client``, see the `Prometheus Python client library documentation `_. An example with custom metrics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ The following ``service.py`` file contains a custom Histogram and a Counter metric to measure the inference time and track the total number of requests. .. code-block:: python from __future__ import annotations import bentoml from prometheus_client import Histogram, Counter from transformers import pipeline import time # Define the metrics request_counter = Counter( name='summary_requests_total', documentation='Total number of summarization requests', labelnames=['status'] ) inference_time_histogram = Histogram( name='inference_time_seconds', documentation='Time taken for summarization inference', labelnames=['status'], buckets=(0.1, 0.2, 0.5, 1, 2, 5, 10, float('inf')) # Example buckets ) EXAMPLE_INPUT = "Breaking News: In an astonishing turn of events, the small town of Willow Creek has been taken by storm as local resident Jerry Thompson's cat, Whiskers, performed what witnesses are calling a 'miraculous and gravity-defying leap.' Eyewitnesses report that Whiskers, an otherwise unremarkable tabby cat, jumped a record-breaking 20 feet into the air to catch a fly. The event, which took place in Thompson's backyard, is now being investigated by scientists for potential breaches in the laws of physics. Local authorities are considering a town festival to celebrate what is being hailed as 'The Leap of the Century." @bentoml.service( resources={"cpu": "2"}, traffic={"timeout": 10}, ) class Summarization: def __init__(self) -> None: self.pipeline = pipeline('summarization') @bentoml.api def summarize(self, text: str = EXAMPLE_INPUT) -> str: start_time = time.time() try: result = self.pipeline(text) summary_text = result[0]['summary_text'] # Capture successful requests status = 'success' except Exception as e: # Capture failures summary_text = str(e) status = 'failure' finally: # Measure how long the inference took and update the histogram inference_time_histogram.labels(status=status).observe(time.time() - start_time) # Increment the request counter request_counter.labels(status=status).inc() return summary_text Run this Service locally: .. code-block:: bash bentoml serve Make sure you have sent some requests to the ``summarize`` endpoint, then view the custom metrics by running the following command. You need to replace ``inference_time_seconds`` and ``summary_requests_total`` with your own metric names. .. code-block:: bash curl -X 'GET' 'http://localhost:3000/metrics' -H 'accept: */*' | grep -E 'inference_time_seconds|summary_requests_total' Expected output: .. code-block:: bash # HELP summary_requests_total Total number of summarization requests # TYPE summary_requests_total counter summary_requests_total{status="success"} 12.0 # HELP inference_time_seconds Time taken for summarization inference # TYPE inference_time_seconds histogram inference_time_seconds_sum{status="success"} 51.74311947822571 inference_time_seconds_bucket{le="0.1",status="success"} 0.0 inference_time_seconds_bucket{le="0.2",status="success"} 0.0 inference_time_seconds_bucket{le="0.5",status="success"} 0.0 inference_time_seconds_bucket{le="1.0",status="success"} 0.0 inference_time_seconds_bucket{le="2.0",status="success"} 0.0 inference_time_seconds_bucket{le="5.0",status="success"} 12.0 inference_time_seconds_bucket{le="10.0",status="success"} 12.0 inference_time_seconds_bucket{le="+Inf",status="success"} 12.0 inference_time_seconds_count{status="success"} 12.0 Use Prometheus to scrape metrics -------------------------------- You can integrate Prometheus to scrape and visualize both default and custom metrics from your BentoML Service. 1. `Install Prometheus `_. 2. Create `a Prometheus configuration file `_ to define scrape jobs. Here is an example that scrapes metrics every 5 seconds from a BentoML Service. .. code-block:: yaml :caption: `prometheus.yml` global: scrape_interval: 5s evaluation_interval: 15s scrape_configs: - job_name: prometheus metrics_path: "/metrics" # The metrics endpoint of the BentoML Service static_configs: - targets: ["0.0.0.0:3000"] # The address where the BentoML Service is running 3. Make sure you have a BentoML Service running, then start Prometheus in a different terminal session using the configuration file you created: .. code-block:: bash ./prometheus --config.file=/path/to/the/file/prometheus.yml 4. Once Prometheus is running, access its web UI by visiting ``http://localhost:9090`` in your web browser. This interface allows you to query and visualize the metrics collected from your BentoML Service. 5. Use `PromQL expressions `_ to query and visualize metrics. For example, to get the 99th percentile of request durations to the ``/encode`` endpoint over the last minute, use: .. code-block:: bash histogram_quantile(0.99, rate(bentoml_service_request_duration_seconds_bucket{endpoint="/encode"}[1m])) .. image:: ../../_static/img/build-with-bentoml/observability/metrics/prome-ui-bentoml.png :alt: Prometheus UI for BentoML metrics Create a Grafana dashboard -------------------------- Grafana is an analytics platform that allows you to create dynamic and informative `dashboards `_ to visualize BentoML metrics. Do the following to create a Grafana dashboard. 1. `Install Grafana `_. 2. By default, Grafana runs on port ``3000``, which conflicts with BentoML's default port. To avoid this, change Grafana's default port. For example: .. code-block:: bash sudo nano /etc/grafana/grafana.ini Find the ``[http]`` section and change ``http_port`` to a free port like ``4000``: .. code-block:: bash ;http_port = 3000 # Change it to a port of your choice and uncomment the line by removing the semicolon http_port = 4000 3. Save the file and restart Grafana to apply the change: .. code-block:: bash sudo systemctl restart grafana-server 4. Access the Grafana web UI at ``http://localhost:4000/`` (use your own port). Log in with the default credentials (``admin``/``admin``). 5. In the Grafana search box at the top, enter ``Data sources`` and add Prometheus as an available option. In **Connection**, set the URL to the address of your running Prometheus instance, such as ``http://localhost:9090``. Save the configuration and test the connection to ensure Grafana can retrieve data from Prometheus. .. image:: ../../_static/img/build-with-bentoml/observability/metrics/grafana-bentoml-1.png :alt: Add Prometheus in Grafana 6. With Prometheus configured as a data source, you can create a new dashboard. Start by adding a panel and selecting a metric to visualize, such as ``bentoml_service_request_duration_seconds_bucket``. Grafana offers a wide array of visualization options, from simple line graphs to more complex representations like heatmaps or gauges. .. image:: ../../_static/img/build-with-bentoml/observability/metrics/grafana-bentoml-2.png :alt: Grafana UI for BentoML metrics For detailed instructions on dashboard creation and customization, read the `Grafana documentation `_. --- ## File: docs/source/build-with-bentoml/observability/monitoring-and-data-collection.rst ========== Monitoring ========== Data-centric AI is a paradigm that positions data as the cornerstone of AI systems. This approach emphasizes the importance of data quality and relevance, suggesting that the effectiveness of AI models can be primarily determined by the data they are trained on and interact with. BentoML fully embraces this paradigm by offering APIs to implement a data-centric workflow, making it straightforward to collect inference data, monitor models, and ship the data to various destinations (for example, local storage, cloud services, and any OTLP supported tools) for your AI project. This document explains how to implement monitoring and collect inference data in BentoML. Benefits of model monitoring and data collection ------------------------------------------------ Implementing an effective data collection and model monitoring strategy with BentoML offers the following benefits: - **Key business metric monitoring:** Track crucial statistical metrics to gauge the impact of your models on business objectives. - **Early detection of data drift:** Identify shifts in input data distributions early, helping decide when model retraining is necessary. - **Quality assurance:** Implement quality assurance for previously untracked metrics, such as model performance, accuracy, and degradation over time. - **Enhanced interoperability:** Promote collaboration between data science and operations teams, streamlining the process of training and iterating on models. - **Informed decision-making:** Provide insights into model behavior in production, informing future model improvements and iterations. Built-in monitoring endpoints ----------------------------- BentoML Services automatically expose several built-in endpoints for health monitoring: - ``/livez``: A liveness probe endpoint that checks if the Service is still alive and needs to restart. - ``/readyz``: A readiness probe endpoint that indicates if the Service is ready to accept traffic. - ``/healthz``: An alias of ``/livez``. These endpoints return a ``200`` status code and an empty response body when the Service is healthy. After a Service starts, you can call these endpoints and get their status codes: .. code-block:: python import requests base_url = "http://localhost:3000" print(requests.get(f"{base_url}/livez").status_code) print(requests.get(f"{base_url}/readyz").status_code) print(requests.get(f"{base_url}/healthz").status_code) You can customize the liveness and readiness endpoint paths using the ``endpoints`` argument. This is useful when your Service delegates traffic to an external process or application that exposes its own health check endpoints with non-standard paths. .. code-block:: python # Use the @bentoml.service decorator: @bentoml.service(endpoints={"livez": "/health", "readyz": "/ready"}) class MyService: ... # Or use the Service constructor: MyService = bentoml.Service( name="my_service", cmd=["./bin/http"], endpoints={"livez": "/health", "readyz": "/ready"} ) Implement monitoring -------------------- In BentoML, you use the ``bentoml.monitor`` context manager to log data related to model inference. It allows you to specify a monitoring session where you can log various data types. This ensures that logging is structured and organized, making it easier to analyze the data later. The following is an example of implementing monitoring in the Summarization Service in :doc:`/get-started/hello-world`. .. code-block:: python from __future__ import annotations import bentoml from transformers import pipeline EXAMPLE_INPUT = "Breaking News: In an astonishing turn of events, the small town of Willow Creek has been taken by storm as local resident Jerry Thompson's cat, Whiskers, performed what witnesses are calling a 'miraculous and gravity-defying leap.' Eyewitnesses report that Whiskers, an otherwise unremarkable tabby cat, jumped a record-breaking 20 feet into the air to catch a fly. The event, which took place in Thompson's backyard, is now being investigated by scientists for potential breaches in the laws of physics. Local authorities are considering a town festival to celebrate what is being hailed as 'The Leap of the Century.'" @bentoml.service( resources={"cpu": "2"}, traffic={"timeout": 10}, ) class Summarization: def __init__(self) -> None: self.pipeline = pipeline('summarization') @bentoml.api def summarize(self, text: str = EXAMPLE_INPUT) -> str: # Use bentoml.monitor as a context manager with bentoml.monitor("text_summarization") as mon: # Log the input data mon.log(text, name="input_text", role="original_text", data_type="text") result = self.pipeline(text) summary_text = result[0]['summary_text'] # Log the output data mon.log(summary_text, name="summarized_text", role="prediction", data_type="text") return summary_text When you enter the ``bentoml.monitor`` context, you instantiate a monitoring session uniquely identified by a name (``"text_summarization"`` in this example), which helps you categorize and retrieve the logged data for specific monitoring tasks. Within the ``bentoml.monitor`` context, you use the ``log`` method to record individual data points, which requires several parameters to describe the data being logged: - ``data``: The actual data value you want to log. This could be an input/output parameter value (the input ``text`` and output ``summary_text`` in this example). - ``name``: A string identifier for the data point, which will be displayed in logs. This helps label the data in your logs, making it clear what each logged value represents. - ``role``: The role of the data in model inference. Default roles include: - ``"feature"``: Indicates that the logged data is an input feature of the model. - ``"prediction"``: Indicates that the logged data is a prediction made by the model. - ``"target"``: Indicates that the logged data is the target or label. You can set a custom ``role`` like ``original_text`` in the example, which will also be logged. - ``data_type``: The type of the data. Default data types include: - ``"numerical"``: For quantitative data. - ``"categorical"``: For discrete values representing categories. - ``"numerical_sequence"``: For sequences or lists of numerical values. You can set a custom ``data_type`` like ``text`` in the example, which will also be logged. View request and schema logs ---------------------------- When the Service starts, BentoML exports the request and schema logs to the default directory ``monitoring/``, which contains the ``data`` and ``schema`` subdirectories. The input and output data is stored in the ``data`` directory, including the corresponding timestamp and a unique request ID. To view the real-time data logs, run: .. code-block:: bash $ tail -f monitoring/text_summarization/data/*.log {"input_text": "Breaking News: In an astonishing turn of events, the small town of Willow Creek has been taken by storm as local resident Jerry Thompson's cat, Whiskers, performed what witnesses are calling a 'miraculous and gravity-defying leap.' Eyewitnesses report that Whiskers, an otherwise unremarkable tabby cat, jumped a record-breaking 20 feet into the air to catch a fly. The event, which took place in Thompson's backyard, is now being investigated by scientists for potential breaches in the laws of physics. Local authorities are considering a town festival to celebrate what is being hailed as 'The Leap of the Century.'", "summarized_text": " Whiskers, an otherwise unremarkable tabby cat, jumped a record-breaking 20 feet into the air to catch a fly . The event is now being investigated by scientists for potential breaches in the laws of physics . Local authorities are considering a town festival to celebrate what is being hailed as 'The Leap of the Century'", "timestamp": "2024-03-05T03:33:59.490137", "request_id": "14642743634293743168"} {"input_text": "Breaking News: In an astonishing turn of events, the small town of Willow Creek has been taken by storm as local resident Jerry Thompson's cat, Whiskers, performed what witnesses are calling a 'miraculous and gravity-defying leap.' Eyewitnesses report that Whiskers, an otherwise unremarkable tabby cat, jumped a record-breaking 20 feet into the air to catch a fly. The event, which took place in Thompson's backyard, is now being investigated by scientists for potential breaches in the laws of physics. Local authorities are considering a town festival to celebrate what is being hailed as 'The Leap of the Century.'", "summarized_text": " Whiskers, an otherwise unremarkable tabby cat, jumped a record-breaking 20 feet into the air to catch a fly . The event is now being investigated by scientists for potential breaches in the laws of physics . Local authorities are considering a town festival to celebrate what is being hailed as 'The Leap of the Century'", "timestamp": "2024-03-05T03:41:49.870589", "request_id": "7485759375304577245"} The schema information is stored in the ``schema`` directory. .. code-block:: bash $ cat monitoring/text_summarization/schema/*.log {"meta_data": {"bento_name": "", "bento_version": "not available"}, "columns": [{"name": "input_text", "role": "original_text", "type": "text"}, {"name": "summarized_text", "role": "prediction", "type": "text"}]} BentoML logs request and schema data to rotating files. This means that new log files are created periodically or when the current log file reaches a certain size, older files are archived based on the default retention policy. You can customize the behavior by :ref:`using a configuration file `. The actual data points are logged as JSON objects, providing a structured format for storing multiple records. This format is widely supported and can be easily ingested into various data analysis tools or databases for further processing. Ship the collected data ----------------------- BentoML provides a general monitoring data collection API. It allows you to transmit collected data to various destinations such as data warehouses, analytics pipelines, or specialized monitoring and drift detection solutions, all without requiring any modifications to your existing codebase. The following table outlines the available targets for shipping monitoring data, the monitoring types (read the following sections for details), and additional notes. .. list-table:: :widths: 33 33 34 :header-rows: 1 - - Destination - Monitoring type - Note - - ``./monitoring//data/xxx.log`` - ``default`` - Logs are stored locally by default. - - Cloud and monitoring services (Amazon S3, Azure Blob, Datadog, Elasticsearch, InfluxDB, Google BigQuery, Kafka, etc.) - ``otlp`` + deployed Fluent Bit - For more output options and configurations, see `Fluent Bit Outputs `_. - - Any OTLP supported tools - ``otlp`` - Useful for environments where direct file access is restricted, like AWS Lambda. - - Arize - ``bentoml_plugins.arize.ArizeMonitor`` - Ensure API keys and space keys are configured correctly. .. _through-log-files: Through log files ^^^^^^^^^^^^^^^^^ Writing monitoring data to log files is the most common way of data collection in BentoML, which is compatible with popular logging tools such as `Fluent Bit `_, `Filebeat `_, and `Logstash `_. You can customize the monitoring configuration using the ``@bentoml.service`` decorator. .. code-block:: python ... @bentoml.service( resources={"cpu": "2"}, traffic={"timeout": 10}, monitoring={ "enabled": True, "type": "default", "options": { "log_config_file": "path/to/log_config.yaml", "log_path": "monitoring" } } ) class Summarization: # Service implementation code Available fields for ``monitoring``: - ``enabled``: Whether monitoring is enabled for the Service. Setting it to ``True`` allows BentoML to collect and log data based on the specified configurations. - ``type``: Specifies the type of monitoring system to use. The value ``default`` means the use of BentoML's built-in monitoring system, which collects data and logs it to files as shown in the previous section. - ``options``: A dictionary that allows you to customize the monitoring setup. - ``log_config_file``: Specifies the path to a custom logging configuration file in YAML, which specifies logging behavior, such as log rotation policies, handlers, log formats, and log levels. The logging parameters should be set according to `the Python logging module's configuration schema `_. If not provided, BentoML uses the default logging configuration, which is suitable for most use cases. Here is an example configuration file, which outputs log messages to a stream: .. code-block:: yaml version: 1 disable_existing_loggers: false loggers: bentoml_monitor_data: level: INFO handlers: [bentoml_monitor_data] propagate: false bentoml_monitor_schema: level: INFO handlers: [bentoml_monitor_schema] propagate: false handlers: bentoml_monitor_data: class: logging.StreamHandler stream: "ext://sys.stdout" level: INFO formatter: bentoml_json bentoml_monitor_schema: class: logging.StreamHandler stream: "ext://sys.stdout" level: INFO formatter: bentoml_json formatters: bentoml_json: class: pythonjsonlogger.jsonlogger.JsonFormatter format: "()" validate: false - ``log_path``: Defines the directory where monitoring logs will be stored, which is relative to the Service's running location. It defaults to ``monitoring``. For deployments using :doc:`the OCI-compliant image `, you can persist log files by mounting the specified log directory (``monitoring`` in the example) to a volume. This ensures that your monitoring data is retained across container restarts and redeployments. In Kubernetes, you can persist and ship logs by mounting the log directory and using a `Fluent Bit `_ DaemonSet or a sidecar container. This allows the collected log files to be automatically forwarded to your designated monitoring system or data warehouse, ensuring that your monitoring data is centralized and accessible for analysis and alerting. Through an OTLP endpoint ^^^^^^^^^^^^^^^^^^^^^^^^ In scenarios where you can't directly access log files, such as when using AWS Lambda, as it doesn't support log files, BentoML supports exporting monitoring data to an external telemetry system using the OpenTelemetry Protocol (OTLP). .. note:: Some log collectors like Fluent Bit also support OTLP input. Below is an example of setting up OTLP for a BentoML Service: .. code-block:: python ... @bentoml.service( resources={"cpu": "2"}, traffic={"timeout": 10}, monitoring={ "enabled": True, "type": "otlp", "options": { "endpoint": "http://localhost:5000", "insecure": True, "credentials": null, "headers": null, "timeout": 10, "compression": null, "meta_sample_rate": 1.0 } } ) class Summarization: # Service implementation code Available parameters: - ``endpoint``: Specifies the URL of the telemetry system's OTLP receiver. Data collected by BentoML will be sent to this endpoint. - ``insecure``: A Boolean flag that specifies whether to disable transport security for the connection with the OTLP endpoint. Setting this to ``True`` means an insecure connection, which is typical for local or development environments. - ``credentials``: If your OTLP endpoint requires authentication, you can use this parameter to provide credentials such as tokens or certificates. If set to ``null``, it means that no authentication is required. - ``headers``: Additional headers that may be required by the OTLP endpoint, useful for passing tokens or other necessary information. - ``timeout``: Defines the maximum duration (in seconds) that BentoML will wait for a response from the OTLP endpoint before timing out. - ``compression``: Specifies the type of compression to use when sending data. This can help reduce bandwidth usage. Supported values include ``gzip`` or ``none``. - ``meta_sample_rate``: Determines the sampling rate for sending metadata to the endpoint. A value of ``1.0`` means that all metadata is sent, while lower values reduce the frequency, sending only a percentage of the collected metadata. For more information, see `the OTLP documentation `_. Plugins and third-party monitoring data collectors ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ BentoML also supports plugins and third-party monitoring data collectors. You can create a custom monitoring data collector and publish it as a Python package. Unlike the built-in collector, which is more protocol specific for general use cases, plugins could be more platform-specific. To use a plugin, you need to install it in the runtime environment for deployment. For details, see :doc:`/build-with-bentoml/runtime-environment`. Arize AI """""""" For end-to-end solutions for data and model monitoring, BentoML works with `Arize AI `_ to provide a plugin for Arize. If you don't want to deploy a pipeline by yourself but still need data and model monitoring, Arize AI is a good choice. It provides a unified platform for data scientists, data engineers, and ML engineers to monitor, analyze, and debug ML models in production. To use this plugin, make sure you have installed it first: .. code-block:: bash pip install bentoml-plugins-arize In the ``@bentoml.service`` decorator, add the ``space_key`` and ``api_key`` to connect to your Arize account. .. code-block:: python ... @bentoml.service( resources={"cpu": "2"}, traffic={"timeout": 10}, monitoring={ "enabled": True, "type": "bentoml_plugins.arize.ArizeMonitor", "options": { "space_key": , "api_key": } } ) class Summarization: # Service implementation code For more information about available Arize parameters, see `the Arize documentation `_. For deployment, the plugin should also be installed in your runtime environment: .. code-block:: python import bentoml my_image = bentoml.images.Image(python_version='3.11') \ .python_packages("bentoml-plugins-arize") # Add this plugin @bentoml.service(image=my_image) class MyService: # Service implementation --- ## File: docs/source/build-with-bentoml/observability/tracing.rst ======= Tracing ======= Tracing offers a continuous view of an application's behavior, tracking the flow of data as they move through different components of the application. BentoML provides built-in tracing support through OpenTelemetry, allowing users to export traces with `Zipkin `_, `Jaeger `_, and `OTLP exporter `_. This document explains how to configure tracing in BentoML. Why do you need tracing? ------------------------ Debugging models and services in production can be challenging. Traditional logging approaches often fall short when tracking issues across multiple services or components. Additionally, logs alone might not provide enough context or information to identify the root cause of a problem. Tracing, on the other hand, offers several key benefits: - **Comprehensive visibility**: Tracing provides a holistic view of an application's flow, allowing developers to follow requests as they move through various components. - **Performance monitoring**: Tracing helps identify performance bottlenecks by showing where time is being spent within the application. - **Cross-service correlation**: Tracing helps track requests across multiple services, which is useful in distributed systems and microservice architectures. Install dependencies -------------------- Before configuring tracing in BentoML, install the corresponding packages. .. tab-set:: .. tab-item:: Zipkin .. code-block:: bash pip install "bentoml[tracing-zipkin]" .. tab-item:: Jaeger .. code-block:: bash pip install "bentoml[tracing-jaeger]" .. tab-item:: OTLP .. code-block:: bash pip install "bentoml[tracing-otlp]" Configure exporters ------------------- You configure a tracing exporter for a BentoML Service using the ``tracing`` field within the ``@bentoml.service`` decorator, where you define some common configurations, followed by the specific configurations of each exporter type. Common configurations are shared values among the exporters, which means they will be applied to the corresponding exporter set via ``exporter_type``. Here is an example: .. code-block:: python import bentoml @bentoml.service( resources={"cpu": "2"}, traffic={"timeout": 10}, tracing={ # Common configurations "exporter_type": "jaeger", "sample_rate": 1.0, "timeout": 5, "max_tag_value_length": 256, "excluded_urls": "readyz", "jaeger": { # Specific configurations of the exporter } ) class MyService: # Service implementation code Available fields in ``tracing``: - ``exporter_type``: The tracing exporter used for the Service. Supported values are ``jaeger``, ``zipkin``, and ``otlp``. - ``sample_rate``: By default, no traces will be collected. Set ``sample_rate`` to your desired fraction to start collecting them. - ``timeout``: A timeout for the exporter, which waits for each batch export. - ``max_tag_value_length``: A maximum length for string attribute values. - ``excluded_urls``: Exclude routes from tracing. This parameter can be either a comma-separated string of routes, or a list of strings. .. note:: BentoML implements OpenTelemetry APIs, which means OpenTelemetry environment variables will take precedence over the configurations set in the ``@bentoml.service`` decorator. Zipkin ^^^^^^ When using Zipkin, BentoML only supports its V2 protocol. If you are reporting to an OpenZipkin server directly, add the URL path ``/api/v2/spans`` to the server address. Here is an example of using Zipkin as the tracing exporter: .. code-block:: python import bentoml @bentoml.service( resources={"cpu": "2"}, traffic={"timeout": 60}, tracing={ "exporter_type": "zipkin", "sample_rate": 1.0, "zipkin": { "endpoint": "http://localhost:9411/api/v2/spans", "local_node_ipv4": "192.168.0.1", "local_node_ipv6": "2001:db8::c001", "local_node_port": 31313, } } ) class MyService: # Service implementation code Available fields for Zipkin, which are passed through the `OpenTelemetry Zipkin Exporter `_: - ``endpoint``: The endpoint URL where Zipkin spans are sent. - ``local_node_ipv4``: The local node's IPv4 address. - ``local_node_ipv6``: The local node's IPv6 address. - ``local_node_port``: The local node's port. With your BentoML Service configured, `run Zipkin `_ before starting the Service. For example, run the latest Zipkin Docker image as below: .. code-block:: bash docker run -d -p 9411:9411 openzipkin/zipkin Start your BentoML Service and send some requests to it. You can then visit the Zipkin UI at ``http://localhost:9411/`` to view the traces: .. image:: ../../_static/img/build-with-bentoml/observability/tracing/zipkin-ui-tracing-bentoml.png :alt: Zipkin UI for BentoML traces Jaeger ^^^^^^ BentoML allows you to use Jaeger to collect traces over the Thrift protocol. .. note:: When it is not feasible to deploy Jaeger Agent next to the application, for example, when the application code is running as a Lambda function, a collector can be configured to send spans using Thrift over HTTP. If both agent and collector are configured, the exporter sends traces only to the collector to eliminate the duplicate entries. Here is an example of using Jaeger as the tracing exporter: .. code-block:: python import bentoml @bentoml.service( resources={"cpu": "2"}, traffic={"timeout": 60}, tracing={ "exporter_type": "jaeger", "sample_rate": 1.0, "jaeger": { "protocol": "thrift", "collector_endpoint": "http://localhost:14268/api/traces?format=jaeger.thrift", "thrift": { "agent_host_name": "localhost", "agent_port": 6831, "udp_split_oversized_batches": True, } } } ) class MyService: # Service implementation code Available fields for Jaeger: - ``protocol``: The protocol used for Jaeger. - ``collector_endpoint``: The URL for the Jaeger collector endpoint. - ``thrift``: A dictionary containing the specific configurations for the Thrift protocol. - ``agent_host_name``: The hostname of the Jaeger agent to which traces are sent. - ``agent_port``: The port on which the Jaeger agent listens for incoming traces. - ``udp_split_oversized_batches``: When it is set to ``True``, the oversized batch will be split into smaller batches over the UDP max packet size (default: ``65000``) if the given buffer is larger than the max packet size: .. math:: \text{packets} \triangleq \left\lceil \frac{\text{len}(\text{buff})}{\text{max_packet_size}} \right\rceil With your BentoML Service configured, run Jaeger before starting the Service. For example, you can run the Jaeger `all-in-one `_ Docker image as below: .. code-block:: bash docker run --rm --name jaeger \ -e COLLECTOR_ZIPKIN_HOST_PORT=:9411 \ -p 6831:6831/udp \ -p 6832:6832/udp \ -p 5778:5778 \ -p 16686:16686 \ -p 4317:4317 \ -p 4318:4318 \ -p 14250:14250 \ -p 14268:14268 \ -p 14269:14269 \ -p 9411:9411 \ jaegertracing/all-in-one:1.57 Start your BentoML Service and send some requests to it. You can then visit the Jaeger UI at ``http://localhost:16686/`` to view the traces: .. image:: ../../_static/img/build-with-bentoml/observability/tracing/jaeger-ui-tracing-bentoml.png :alt: Jaeger UI for BentoML traces OTLP exporter ^^^^^^^^^^^^^ BentoML supports the OTLP exporter for easy integration with an OpenTelemetry traces receiver. Currently, BentoML only supports the HTTP protocol for the OTLP exporter, which uses Protobuf to send traces over HTTP. Here is an example of using the OTLP exporter: .. code-block:: python import bentoml @bentoml.service( resources={"cpu": "2"}, traffic={"timeout": 60}, tracing={ "exporter_type": "otlp", "sample_rate": 1.0, "otlp": { "protocol": "http", "endpoint": "http://localhost:4318/v1/traces", "http": { "certificate_file": "/path/to/cert.pem", "headers": { "Keep-Alive": "timeout=5, max=1000", }, }, } } ) class MyService: ... Available fields for OTLP: - ``protocol``: The protocol used for OTLP. - ``endpoint``: The endpoint URL where OTLP spans are sent, namely the address of the OpenTelemetry receiver. - ``http``: A dictionary containing specific configurations for the HTTP protocol. - ``certificate_file``: The path to the certificate file for secure HTTP communication. - ``headers``: A dictionary containing custom HTTP headers to include in the trace requests. To start an OTLP exporter, you generally need an OpenTelemetry Collector running that can receive, process, and export telemetry data (traces, metrics, logs, etc.) to the desired backend. You can find the latest OpenTelemetry Collector releases on its `GitHub repository `_ or `read its Quick Start guide `_. The quickest way is to run the latest Docker image directly and output the tracing information to a separate file: .. code-block:: bash docker run \ -p 127.0.0.1:4318:4318 \ otel/opentelemetry-collector:0.100.0 \ 2>&1 | tee collector-output.txt In a separate terminal, start your BentoML Service with ``otlp`` tracing configured and send some requests to it. You can then view the details of traces collected and processed by the OpenTelemetry Collector. .. code-block:: bash :caption: `collector-output.txt` ... Resource SchemaURL: Resource attributes: -> telemetry.sdk.language: Str(python) -> telemetry.sdk.name: Str(opentelemetry) -> telemetry.sdk.version: Str(1.20.0) -> service.name: Str(Summarization) -> service.instance.id: Int(1) -> service.version: Str(not available) ScopeSpans #0 ScopeSpans SchemaURL: InstrumentationScope opentelemetry.instrumentation.asgi 0.41b0 Span #0 Trace ID : 55f76a6172e5ef154bef3024cb7244a4 Parent ID : 80ebf4a29195d793 ID : 60bb910352f8279d Name : POST /summarize http send Kind : Internal Start time : 2024-05-10 06:14:04.614485867 +0000 UTC End time : 2024-05-10 06:14:04.614835736 +0000 UTC Status code : Unset Status message : Attributes: -> http.status_code: Int(200) -> type: Str(http.response.start) Span #1 Trace ID : 55f76a6172e5ef154bef3024cb7244a4 Parent ID : 80ebf4a29195d793 ID : 027802ac3a807e5a Name : POST /summarize http send Kind : Internal Start time : 2024-05-10 06:14:04.615378986 +0000 UTC End time : 2024-05-10 06:14:04.615539025 +0000 UTC Status code : Unset Status message : Attributes: -> type: Str(http.response.body) Span #2 Trace ID : 55f76a6172e5ef154bef3024cb7244a4 Parent ID : ID : 80ebf4a29195d793 Name : POST /summarize Kind : Server Start time : 2024-05-10 06:13:59.653244649 +0000 UTC End time : 2024-05-10 06:14:04.615805503 +0000 UTC Status code : Unset Status message : Attributes: -> http.scheme: Str(http) -> http.host: Str(xx.x.xxx.x:3000) -> net.host.port: Int(3000) -> http.flavor: Str(1.1) -> http.target: Str(/summarize) -> http.url: Str(http://xx.x.xxx.x:3000/summarize) -> http.method: Str(POST) -> http.server_name: Str(xxx.xxx.xx.xxx:3000) -> http.user_agent: Str(Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36) -> net.peer.ip: Str(xxx.xxx.xx.xx) -> net.peer.port: Int(59482) -> http.status_code: Int(200) {"kind": "exporter", "data_type": "traces", "name": "debug"} 2024-05-10T06:14:06.875Z info MetricsExporter {"kind": "exporter", "data_type": "metrics", "name": "debug", "resource metrics": 1, "metrics": 22, "data points": 22} 2024-05-10T06:14:06.875Z info ResourceMetrics #0 ... .. note:: OTLP does not include a UI component. To visualize OTLP data, you need to use a backend that supports OTLP and provides a UI, such as Jaeger or Zipkin. --- ## File: docs/source/build-with-bentoml/snippets/grpc/additional_setup.rst Since there is no easy way to add additional proto files, we will have to clone some repositories and copy the proto files into our project: 1. :github:`protocolbuffers/protobuf` - the official repository for the Protocol Buffers. We will need protobuf files that live under ``src/google/protobuf``: .. code-block:: bash » mkdir -p thirdparty && cd thirdparty » git clone --depth 1 https://github.com/protocolbuffers/protobuf.git 2. :github:`bentoml/bentoml` - We need the ``service.proto`` under ``bentoml/grpc/`` to build the client; therefore, we will perform a `sparse checkout `_ to only checkout ``bentoml/grpc`` directory: .. code-block:: bash » mkdir bentoml && pushd bentoml » git init » git remote add -f origin https://github.com/bentoml/BentoML.git » git config core.sparseCheckout true » cat <|.git/info/sparse-checkout src/bentoml/grpc EOT » git pull origin main && mv src/bentoml/grpc . » popd --- ## File: docs/source/build-with-bentoml/snippets/grpc/grpc_tools.rst .. tab-set:: .. tab-item:: gRPCurl We will use :github:`fullstorydev/grpcurl` to send a CURL-like request to the gRPC BentoServer. Note that we will use `docker `_ to run the ``grpcurl`` command. .. tab-set:: .. tab-item:: MacOS/Windows :sync: __fullstorydev_macwin .. code-block:: bash » docker run -i --rm \ fullstorydev/grpcurl -d @ -plaintext host.docker.internal:3000 \ bentoml.grpc.v1.BentoService/Call <`_ to run the ``grpcui`` command. .. tab-set:: .. tab-item:: MacOS/Windows :sync: __fullstorydev_macwin .. code-block:: bash » docker run --init --rm \ -p 8080:8080 fullstorydev/grpcui -plaintext host.docker.internal:3000 .. tab-item:: Linux :sync: __fullstorydev_linux .. code-block:: bash » docker run --init --rm \ -p 8080:8080 \ --network=host fullstorydev/grpcui -plaintext 0.0.0.0:3000 Proceed to http://127.0.0.1:8080 in your browser and send a test request from the web UI. --- ## File: docs/source/build-with-bentoml/asgi.rst ======================= Mount ASGI applications ======================= `ASGI (Asynchronous Server Gateway Interface) `_ is a spiritual successor to WSGI (Web Server Gateway Interface), designed to provide a standard interface between async-capable Python web servers, frameworks, and applications. ASGI supports asynchronous request handling, allowing multiple requests to be processed at the same time, making it suitable for real-time web applications, such as WebSockets, long polling, and more. BentoML's server runs the Service API in an ASGI web serving layer, exposing REST endpoints for inference APIs (for example, ``POST /summarize``) and common infrastructure APIs (for example, ``GET /metrics``) for monitoring. This ASGI-native web serving layer allows for direct mounting of existing ASGI applications, enabling them to serve side-by-side with BentoML Services. This document explains how to mount an ASGI application onto a BentoML Service. Why should you mount an ASGI application ---------------------------------------- Mounting an ASGI application, such as one built with FastAPI, onto a BentoML Service can be advantageous for several reasons: - **Extended functionality**: It enables you to extend your machine learning services with additional web functionalities, such as custom APIs for data processing, user management, or serving static files and web user interfaces, which are not directly related to model serving. - **Custom authentication and authorization**: By integrating an ASGI application, you can implement advanced authentication and authorization mechanisms, tailoring security measures to the specific needs of your application. - **API documentation**: With tools like FastAPI, you automatically get interactive API documentation, making it easier for end-users to understand and interact with the APIs. Integrate BentoML with ASGI frameworks -------------------------------------- BentoML offers seamless integration with different ASGI frameworks, allowing you to serve ML models alongside custom web application logic, such as asynchronous operations, real-time data processing, and complex web application functionalities. When integrating the ASGI frameworks, you use ``bentoml.get_current_service()`` to retrieve the current BentoML Service instance. It is useful when you need to access the BentoML Service instance from within ASGI application routes or when injecting the Service instance into ASGI application routes using dependency injection patterns. See the following examples for details. FastAPI ^^^^^^^ `FastAPI `_ is a web framework for building APIs that is built on top of ASGI, allowing it to handle asynchronous requests. To integrate a FastAPI application with a BentoML Service, you can define a FastAPI route either inside or outside the Service as below. .. note:: Make sure you have installed FastAPI by running ``pip install fastapi``. See `the FastAPI documentation `_ to learn more. .. code-block:: python from fastapi import FastAPI, Depends import bentoml app = FastAPI() @bentoml.service @bentoml.asgi_app(app, path="/v1") class MyService: name = "MyService" @app.get('/hello') def hello(self): # Inside service class, use `self` to access the service return f"Hello {self.name}" @app.get("/hello1") async def hello(service: MyService = Depends(bentoml.get_current_service)): # Outside service class, use `Depends` to get the service return f"Hello {service.name}" Specifically, do the following to mount FastAPI: 1. Create a FastAPI application with ``FastAPI()``. 2. Use the ``@bentoml.asgi_app`` decorator to mount the FastAPI application to the BentoML Service, enabling them to be served together. Set the ``path`` parameter to :ref:`customize the prefix path `. 3. Define a FastAPI route inside or outside the Service class using ``@app.get("/")``. - Inside the class: Use ``self`` to access the Service instance's attributes and methods. - Outside the class: Use FastAPI's dependency injection system (``Depends``) to inject the BentoML Service instance into the route function. In the code above, the ``hello1`` route uses ``Depends(bentoml.get_current_service)`` to inject the ``MyService`` instance, allowing the route to access the Service's attributes and methods. 4. Within the FastAPI route, add your desired implementation logic. This example returns a greeting message using the Service's name. .. note:: In addition to ``get``, you can use the other operations like ``post``, ``put``, and ``delete``. See `the FastAPI documentation `_ to learn more. .. dropdown:: Design choice: Inside vs. Outside Accessing the BentoML Service instance both inside and outside the Service class offers flexibility in how you structure and interact with your Service logic and dependencies. The differences in accessing the BentoML Service instance in these contexts primarily relate to scope and the intended use cases. Inside the Service class - Direct access: Within the class defining a BentoML Service, you have direct access to ``self``, which represents the instance of the Service. This allows you to directly access its attributes and methods without injecting any dependency. It's the most straightforward way to use the Service's functionality from within its own definition. - Contextual use: Accessing the Service instance inside the class is typical for defining the Service's internal logic, such as setting up endpoints, performing operations with the model, and handling requests directly related to the Service's primary functionality. Outside the Service class - Dependency injection: Accessing the BentoML Service instance outside the class typically requires dependency injection mechanisms, such as the ``Depends`` function in FastAPI. This approach is necessary when you want to use the Service instance in other parts of your project. - Modular and decoupled design: This approach allows different components of your BentoML project to interact with the Service without being tightly integrated into its class definition. For example, your ML logic can be encapsulated within the BentoML Service, while other aspects, such as custom authentication, supplementary data processing, or additional REST endpoints, can be managed externally yet still interact with the Service as needed. The following is a more practical example of mounting FastAPI onto the Summarization Service in :doc:`/get-started/hello-world`. It defines two additional endpoints with FastAPI by accessing the Service from inside and outside the class respectively. .. code-block:: python from __future__ import annotations import bentoml from transformers import pipeline from fastapi import FastAPI, Depends EXAMPLE_INPUT = "Breaking News: In an astonishing turn of events, the small town of Willow Creek has been taken by storm as local resident Jerry Thompson's cat, Whiskers, performed what witnesses are calling a 'miraculous and gravity-defying leap.' Eyewitnesses report that Whiskers, an otherwise unremarkable tabby cat, jumped a record-breaking 20 feet into the air to catch a fly. The event, which took place in Thompson's backyard, is now being investigated by scientists for potential breaches in the laws of physics. Local authorities are considering a town festival to celebrate what is being hailed as 'The Leap of the Century." # Create a FastAPI app instance app = FastAPI() @bentoml.service( resources={"cpu": "2"}, traffic={"timeout": 10}, ) @bentoml.asgi_app(app, path="/v1") class Summarization: def __init__(self) -> None: self.pipeline = pipeline('summarization') # Define a name attribute name = "MyService" # The original Service API endpoint for text summarization @bentoml.api def summarize(self, text: str = EXAMPLE_INPUT) -> str: result = self.pipeline(text) return result[0]['summary_text'] # Access the Service instance inside the class @app.get("/hello-inside") def hello(self): # Add other logic here if needed return f"Hello {self.name}. You can access the Service instance inside the class." # Access the Service instance outside the class @app.get("/hello-outside") async def hello(service: MyService = Depends(bentoml.get_current_service)): # Add other logic here if needed return f"Hello {service.name}. You can access the Service instance outside the class." After you start the BentoML Service, which is accessible at `http://localhost:3000 `_, you can find two additional endpoints ``hello-inside`` and ``hello-outside`` exposed. .. image:: ../../_static/img/build-with-bentoml/asgi/two-asgi-fastapi-routes.png :alt: Two API endpoints defined in BentoML By sending a ``GET`` request, you can receive the corresponding output from both endpoints. FastAPI route inside the Service class: .. image:: ../../_static/img/build-with-bentoml/asgi/inside-the-class.png :alt: FastAPI route inside the BentoML Service class FastAPI route outside the Service class: .. image:: ../../_static/img/build-with-bentoml/asgi/outside-the-class.png :alt: FastAPI route outside the BentoML Service class Quart ^^^^^ `Quart `_ is an asynchronous web framework for Python that enables you to use async/await features in your web applications to handle large volumes of concurrent connections. The following is an example of integrating Quart with BentoML. .. note:: Make sure you have installed Quart by running ``pip install quart``. See `the Quart documentation `_ to learn more. .. code-block:: python from quart import Quart app = Quart(__name__) @app.get("/hello") async def hello_world(): service = bentoml.get_current_service() return f"Hello, {service.name}" @bentoml.service @bentoml.asgi_app(app, path="/v1") class MyService: name = "MyService" Specifically, do the following to mount Quart: 1. Create a Quart application with ``Quart()``. 2. Use the ``@bentoml.asgi_app`` decorator to mount the Quart application to the BentoML Service, enabling them to be served together. Set the ``path`` parameter to :ref:`customize the prefix path `. 3. Define a Quart route outside the Service class using ``@app.get(/"")``. Use ``bentoml.get_current_service()`` to inject the ``MyService`` instance, allowing the route to access the Service's attributes and methods. 4. Within the Quart route, add your desired implementation logic. This example returns a greeting message using the Service's name. .. note:: In addition to ``get``, you can use the other operations like ``post``, ``put``, and ``delete``. See `the Quart documentation `_ to learn more. The following is a more practical example of mounting Quart onto the Summarization Service in :doc:`/get-started/hello-world`. It defines an additional endpoint ``hello``. .. code-block:: python from __future__ import annotations import bentoml from transformers import pipeline from quart import Quart EXAMPLE_INPUT = "Breaking News: In an astonishing turn of events, the small town of Willow Creek has been taken by storm as local resident Jerry Thompson's cat, Whiskers, performed what witnesses are calling a 'miraculous and gravity-defying leap.' Eyewitnesses report that Whiskers, an otherwise unremarkable tabby cat, jumped a record-breaking 20 feet into the air to catch a fly. The event, which took place in Thompson's backyard, is now being investigated by scientists for potential breaches in the laws of physics. Local authorities are considering a town festival to celebrate what is being hailed as 'The Leap of the Century." # Create a Quart app instance app = Quart(__name__) @app.get("/hello") async def hello_world(): service = bentoml.get_current_service() # Add other logic here if needed return f"Hello, {service.name}" @bentoml.service( resources={"cpu": "2"}, traffic={"timeout": 10}, ) @bentoml.asgi_app(app, path="/v1") class Summarization: def __init__(self) -> None: self.pipeline = pipeline('summarization') # Define a name attribute name = "MyService" # The original Service API endpoint for text summarization @bentoml.api def summarize(self, text: str = EXAMPLE_INPUT) -> str: result = self.pipeline(text) return result[0]['summary_text'] After you start the BentoML Service, which is accessible at `http://localhost:3000 `_, you can interact with the exposed endpoint ``hello``. For example: .. code-block:: bash $ curl http://localhost:3000/v1/hello Hello, MyService .. note:: Unlike FastAPI, Quart does not natively support the OpenAPI specification, so the endpoint is not displayed on the Swagger UI. You can use other ways to communicate with it, such as ``curl``. .. _customize-prefix-path: Customize the prefix path ------------------------- When mounting an ASGI tool onto a BentoML Service, it is possible to customize the route path by setting a prefix. This is useful for organizing your API endpoints and simplifying routing and namespace management. To set a prefix path, simply set the ``path`` parameter in the decorator ``@bentoml.asgi_app``. Here is a FastAPI example: .. code-block:: python from fastapi import FastAPI, Depends import bentoml app = FastAPI() @bentoml.service @bentoml.asgi_app(app, path="/fastapi") # Add the prefix here class MyService: name = "MyService" @app.get('/hello') # This endpoint should be requested via "/fastapi/hello" def hello(self): return f"Hello {self.name}" By specifying ``path="/fastapi"``, the entire FastAPI application is served under this prefix. This means all the routes defined within the FastAPI application will be accessible under ``/fastapi``. In this example, after you start this BentoML Service, you should interact with the ``/fastapi/hello`` endpoint. Add custom ASGI middleware -------------------------- ``add_asgi_middleware`` is an API provided by BentoML to apply `custom ASGI middleware `_. Middleware functions as a layer that processes requests and responses, allowing you to manipulate them or execute additional actions based on specific conditions. It is commonly used for implementing security measures and custom headers, managing CORS, compressing responses, and more. Example usage: .. code-block:: python from __future__ import annotations import bentoml from transformers import pipeline from starlette.middleware.trustedhost import TrustedHostMiddleware @bentoml.service( resources={"cpu": "2"}, traffic={"timeout": 10}, ) class Summarization: def __init__(self) -> None: self.pipeline = pipeline('summarization') @bentoml.api def summarize(self, text: str) -> str: result = self.pipeline(text) return result[0]['summary_text'] # Add TrustedHostMiddleware to ensure the Service only accepts requests from certain hosts Summarization.add_asgi_middleware(TrustedHostMiddleware, allowed_hosts=['example.com', '*.example.com']) This example ensures that the ``Summarization`` Service only accepts requests from specified hosts and prevents host header attacks. You can then interact with the Service by manually specifying the ``Host`` header in requests: .. code-block:: bash curl -H "Host: example.com" http://localhost:3000 .. note:: Alternatively, you can edit your ``hosts`` file to map ``example.com`` to ``127.0.0.1`` (localhost) and then access ``http://example.com:3000/``. While ``add_asgi_middleware`` is used to add middleware to the ASGI application that BentoML uses to serve the APIs, ``@bentoml.asgi_app`` is used to integrate the entire ASGI application into the BentoML Service. This is suitable for adding complete web applications like FastAPI or Quart applications that come with their routing logic, directly alongside your BentoML Service. The middleware added via ``add_asgi_middleware`` applies to the entire ASGI application, including both the BentoML Service and any mounted ASGI applications. This ensures consistent processing of all requests across the application, whether they target BentoML Services or other components. --- ## File: docs/source/build-with-bentoml/clients.rst ==================== Call an API endpoint ==================== BentoML provides a client implementation that allows you to make synchronous and asynchronous requests to BentoML :doc:`Services `. This document explains how to use :doc:`BentoML clients `. Client types ------------ Depending on your requirements, you can create a BentoML client object using the following classes. - ``bentoml.SyncHTTPClient``: Defines a synchronous client, suitable for straightforward, blocking operations where your application waits for the response before proceeding. - ``bentoml.AsyncHTTPClient``: Defines an asynchronous client, suitable for non-blocking operations, allowing your application to handle other tasks while waiting for responses. Create a client --------------- Suppose your BentoML Service has an endpoint named ``summarize`` that takes a string ``text`` as input and returns a summarized version of the text as below. .. code-block:: python class Summarization: def __init__(self) -> None: # Load model into pipeline self.pipeline = pipeline('summarization') @bentoml.api def summarize(self, text: str) -> str: result = self.pipeline(text) return result[0]['summary_text'] After you start the ``Summarization`` Service, you can create the following clients by specifying the server address. .. tab-set:: .. tab-item:: Synchronous .. code-block:: python import bentoml client = bentoml.SyncHTTPClient('http://localhost:3000') summarized_text: str = client.summarize(text="Your long text to summarize") print(summarized_text) # Close the client to release resources client.close() .. tab-item:: Asynchronous .. code-block:: python import asyncio import bentoml async def async_client_operation(): async with bentoml.AsyncHTTPClient('http://localhost:3000') as client: summarized_text: str = await client.summarize(text="Your long text to summarize") print(summarized_text) asyncio.run(async_client_operation()) In the above synchronous and asynchronous clients, requests are sent to the ``summarize`` endpoint of the Service hosted at ``http://localhost:3000``. The BentoML client implementation supports methods corresponding to the Service APIs and they should be called with the same arguments (``text`` in this example) as defined in the Service. These methods are dynamically created based on the Service's endpoints, providing a direct mapping to the Service's functionality. In this example, the ``summarize`` method on the client is directly mapped to the ``summarize`` method in the ``Summarization`` Service. The data passed to the ``summarize`` method (``text="Your long text to summarize"``) conforms to the expected input of the Service. If you define :ref:`root input ` for an API endpoint, use only the positional argument in your client without specifying the parameter name. .. note:: If you deploy your Service to BentoCloud, you can get the client of a Deployment by using ``get_client()`` or ``get_async_client()``. For more information, see :ref:`scale-with-bentocloud/deployment/call-deployment-endpoints:interact with the Deployment`. Use a context manager ^^^^^^^^^^^^^^^^^^^^^ To enhance resource management and reduce the risk of connection leaks, we recommend you create a client within a context manager as below. .. tab-set:: .. tab-item:: Synchronous .. code-block:: python import bentoml with bentoml.SyncHTTPClient('http://localhost:3000') as client: summarized_text: str = client.summarize(text="Your long text to summarize") print(summarized_text) .. tab-item:: Asynchronous .. code-block:: python import bentoml async with bentoml.AsyncHTTPClient('http://localhost:3000') as client: summarized_text: str = await client.summarize(text="Your long text to summarize") print(summarized_text) Check Service readiness ----------------------- Before making calls to specific Service methods, you can use the ``is_ready`` method of the client to check if the Service is ready to handle requests. This ensures that your API calls are made only when the Service is up and running. .. code-block:: python import bentoml client = bentoml.SyncHTTPClient('http://localhost:3000') if client.is_ready(): summarized_text: str = client.summarize(text="Your long text to summarize.") print("Summarized text:", summarized_text) else: print("Service is not ready") client.close() Alternatively, use the ``server_ready_timeout`` parameter to specify the maximum duration in seconds the client will wait for the BentoML Service to become ready before timing out. This is useful during the initial connection to a Service that might be starting up. If the Service does not become ready within the specified timeout, the client will raise a timeout exception. .. code-block:: python import bentoml client = bentoml.SyncHTTPClient( 'http://localhost:3000', server_ready_timeout=60 # Wait up to 60 seconds for the Service to be ready ) summarized_text: str = client.summarize(text="Your long text to summarize") print(summarized_text) client.close() .. _call-a-task-endpoint: Call a task endpoint -------------------- You can create clients to interact with Services defined with :ref:`task ` endpoints by submitting inputs and then asynchronously checking for results at a later time. This is particularly useful for scenarios where the client does not need to actively wait for the task to complete. For more information, see :doc:`/get-started/async-task-queues`. Input and output ---------------- BentoML clients support handling different input and output types. JSON ^^^^ You can easily handle JSONable data input and JSON output with BentoML's HTTP clients, which are designed to seamlessly serialize and deserialize JSON data. For input, when you send data that can be serialized to JSON (for example, dictionaries, lists, strings, and numbers), you simply pass it as arguments to the client method corresponding to your Service API. The following code comes from the Service ``SentenceEmbedding`` of `this example project `_, which accepts JSONable input (lists in this case). .. code-block:: python import typing as t @bentoml.service class SentenceEmbedding: ... @bentoml.api def encode(self, sentences: t.List[str] = SAMPLE_SENTENCES) -> np.ndarray: ... To create a client to handle JSONable input for Services like ``SentenceEmbedding``: .. code-block:: python import bentoml import typing as t client = bentoml.SyncHTTPClient("http://localhost:3000") # Specify the sentences for the request sentences_list: t.List[str] = [ "The sun dips below the horizon, painting the sky orange.", "A gentle breeze whispers through the autumn leaves.", "The moon casts a silver glow on the tranquil lake.", # Add more if necessary ] # Make the request using the Service endpoint result = client.encode(sentences=sentences_list) # Print the result print(f"Encoded sentences result: {result}") client.close() For output, when a BentoML Service returns JSON data, the client automatically deserializes this JSON into a Python data structure (like a dictionary or a list, depending on the JSON structure). The following code comes from the Service ``WhisperX`` of this `example project `_, which returns JSONable output (dictionaries in this case). .. code-block:: python import typing as t from pathlib import Path @bentoml.service class WhisperX: ... @bentoml.api def transcribe(self, audio_file: Path) -> t.Dict: ... To create a client to handle JSONable output for Services like ``WhisperX``: .. code-block:: python import bentoml import typing as t client = bentoml.SyncHTTPClient('http://localhost:3000') # Set the audio URL audio_url = 'https://example.org/female.wav' # The response is expected to be a dictionary response: t.Dict = client.transcribe(audio_file=audio_url) print(response) .. tip:: You can print specific values of keys from the JSON response. For example, the Service ``WhisperX`` returns the following and you can output the text of the first segment: .. code-block:: python response = { "segments": [ { "start": 0.009, "end": 2.813, "text": " The Hispaniola was rolling scuppers under in the ocean swell.", "words": [ {"word": "The", "start": 0.009, "end": 0.069, "score": 0.0}, {"word": "Hispaniola", "start": 0.109, "end": 0.81, "score": 0.917}, # Other words omitted... ], }, # Other segments omitted... ], "word_segments": [ {"word": "The", "start": 0.009, "end": 0.069, "score": 0.0}, {"word": "Hispaniola", "start": 0.109, "end": 0.81, "score": 0.917}, # Other words omitted... ], } # Print the text of the first segment # Add the following line to your client code print("Segment text:", response["segments"][0]["text"]) Files ^^^^^ BentoML clients support a variety of file types, such as images and generic binary files. For file input, you pass a ``Path`` object pointing to the file. The client handles the file reading and sends it as part of the request. For file output, the client provides the output as a ``Path`` object. You can use this ``Path`` object to access, read, or process the file. The following code snippet comes from the :doc:`/examples/controlnet` example, which accepts and returns an image file. .. code-block:: python import PIL from PIL.Image import Image as PIL_Image @bentoml.service class ControlNet: ... @bentoml.api async def generate(self, image: PIL_Image, params: Params) -> PIL_Image: ... To create a client to handle file input and output for Services like ``ControlNet``: .. code-block:: python import bentoml from pathlib import Path client = bentoml.SyncHTTPClient("http://localhost:3000") # Specify the image path and other parameters for the request image_path: Path = Path("/path/to/example-image.png") params = { "prompt": "A young man walking in a park, wearing jeans.", "negative_prompt": "ugly, disfigured, ill-structure, low resolution", "controlnet_conditioning_scale": 0.5, "num_inference_steps": 25 } # Make the request using the Service endpoint result_path: Path = client.generate( image=image_path, params=params, ) print(f"Generated file saved at: {result_path}") client.close() You can also use URLs as the input as below: .. code-block:: python import bentoml from pathlib import Path client = bentoml.SyncHTTPClient("http://localhost:3000") # Specify the image URL and other parameters for the request image_url = 'https://example.org/1.png' # The remaining code is the same ... Streaming ^^^^^^^^^ You can add streaming logic to a BentoML client, which is especially useful when dealing with large amounts of data or real-time data feeds. Streamed output is returned a generator or async generator, depending on the client type. .. tab-set:: .. tab-item:: Synchronous For synchronous streaming, ``SyncHTTPClient`` uses a Python generator to output data as it is received from the stream. .. code-block:: python import bentoml client = bentoml.SyncHTTPClient("http://localhost:3000") for data_chunk in client.stream_data(): # Process each chunk of data as it arrives process_data(data_chunk) client.close() def process_data(data_chunk): # Add processing logic print("Processing data chunk:", data_chunk) # Add more logic here to handle the data chunk .. tab-item:: Asynchronous For asynchronous streaming, ``AsyncHTTPClient`` uses an async generator. This allows for asynchronous iteration over the streaming data. .. code-block:: python import bentoml async with bentoml.AsyncHTTPClient("http://localhost:3000") as client: async for data_chunk in client.stream_data(): # Process each chunk of data as it arrives await process_data_async(data_chunk) async def process_data_async(data_chunk): # Add processing logic print("Processing data chunk asynchronously:", data_chunk) # Add more complex asynchronous processing here await some_async_operation(data_chunk) Authorization ------------- When working with BentoML Services that require authentication, you can authorize clients (``SyncHTTPClient`` and ``AsyncHTTPClient``) using a token. This token, typically a JWT (JSON Web Token) or some other form of API key, is used to ensure that the client is allowed to access the specified BentoML Service. The token is included in the HTTP headers of each request made by the client, allowing the server to validate the client's credentials. To authorize a client, you pass the token as an argument during initialization. .. code-block:: python import bentoml client = bentoml.SyncHTTPClient('http://localhost:3000', token='your_token_here') summarized_text: str = client.summarize(text="Your long text to summarize") print(summarized_text) client.close() Error handling -------------- Handling errors, checking for error code and messages, and implementing retries are important for reliable client-server communication. Here are some strategies and examples on error handling and retries. Basics ^^^^^^ When interacting with a BentoML Service, errors like network issues, Service downtime, or invalid input, may occur. Proper error handling allows your client to respond gracefully to these issues. You can use ``try`` and ``except`` blocks to catch exceptions that may occur during the request: .. code-block:: python import bentoml from bentoml.exceptions import BentoMLException client = bentoml.SyncHTTPClient('http://localhost:3000') try: summarized_text: str = client.summarize(text="Your long text to summarize.") print(summarized_text) except BentoMLException as e: print(f"An error occurred: {e}") finally: client.close() When catching exceptions, it's useful to examine specific error code or messages to determine the cause of the failure. This can guide the retry logic or inform you about the issue more precisely. Implement retry logic ^^^^^^^^^^^^^^^^^^^^^ Retrying failed requests can help overcome temporary issues like network disruptions or service unavailability. When implementing retries, consider exponential backoff to avoid overwhelming the server or the network. Here's a simple example of implementing retries with exponential backoff. .. code-block:: python import time from bentoml.exceptions import BentoMLException import bentoml def retry_request(client, max_retries=3, backoff_factor=2): for attempt in range(max_retries): try: summarized_text: str = client.summarize(text="Your long text to summarize.") return summarized_text except BentoMLException as e: print(f"Attempt {attempt+1}: An error occurred: {e}") time.sleep(backoff_factor ** attempt) print("Max retries reached. Giving up.") client = bentoml.SyncHTTPClient('http://localhost:3000') try: response = retry_request(client) if response: print(response) finally: client.close()