# Technical Documentation: ArroyoSystems/arroyo > ℹ️ **Provenance:** Multi-Tier Ingestion · [CodeWiki Reference](https://codewiki.google/github.com/ArroyoSystems/arroyo) ### Overview (`README.md`)

Getting started | Docs | Discord | Website

[Arroyo](https://arroyo.dev) is a distributed stream processing engine written in Rust, designed to efficiently perform stateful computations on streams of data. Unlike traditional batch processing, streaming engines can operate on both bounded and unbounded sources, emitting results as soon as they are available. In short: Arroyo lets you ask complex questions of high-volume real-time data with subsecond results. ## Features 🦀 SQL streaming pipelines 🚀 Scales up to millions of events per second 🪟 Stateful operations including windows and joins 🔥State checkpointing for fault-tolerance and recovery of pipelines 🕒 Time-oriented stream processing via the [Dataflow model](https://www.oreilly.com/radar/the-world-beyond-batch-streaming-101/) 🔌 A wide variety of [connectors](https://doc.arroyo.dev/connectors), including Kafka and Iceberg ## Use cases Some example use cases include: * Detecting fraud and security incidents * Real-time product and business analytics * Real-time ingestion into your data warehouse or data lake * Real-time ML feature generation ## Why Arroyo There are already a number of existing streaming engines out there, including [Apache Flink](https://flink.apache.org), [Spark Streaming](https://spark.apache.org/docs/latest/streaming-programming-guide.html), and [Kafka Streams](https://kafka.apache.org/documentation/streams/). Why create a new one? * _Serverless operations_: Arroyo pipelines are designed to run in modern cloud environments, supporting seamless scaling, recovery, and rescheduling * _High performance SQL_: SQL is a first-class concern, with consistently excellent performance * _Designed for non-experts_: Arroyo cleanly separates the pipeline APIs from its internal implementation. You don't need to be a streaming expert to build real-time data pipelines. ## Installing Arroyo ships as a single binary. You can install it locally on MacOS using Homebrew ```shellsession brew install arroyosystems/tap/arroyo ``` or on MacOS or Linux with this script: ```shellsession curl -LsSf https://arroyo.dev/install.sh | sh ``` or you can download a binary for your platform from the [releases page](https://github.com/ArroyoSystems/arroyo/releases). Once you have Arroyo installed, start a cluster with ```shellsession $ arroyo cluster ``` You can also run a cluster in Docker, with ```shellsession docker run -p 5115:5115 \ ghcr.io/arroyosystems/arroyo:latest ``` Then, load the Web UI at http://localhost:5115. For a more in-depth guide, see the [getting started guide](https://doc.arroyo.dev/getting-started). Once you have Arroyo running, follow the [tutorial](https://doc.arroyo.dev/tutorial/first-pipeline/) to create your first real-time pipeline. ## Cloudflare Pipelines If you don't want to self-host, Arroyo is available as a fully-managed solution on the Cloudflare Developer Platform: [Cloudflare Pipelines](https://developers.cloudflare.com/pipelines/), now available in beta. Currently, stateless pipelines ingesting into R2 are supported, and we'll be expanding to stateful pipelines in the near future. ## Developing Arroyo We love contributions from the community! See the [developer setup](https://doc.arroyo.dev/developing/dev-setup) guide to get started, and reach out to the team on [discord](https://discord.gg/cjCr5rVmyR) or create an issue. ## Community * [Discord](https://discord.gg/cjCr5rVmyR) — support and project discussion * [GitHub issues](https://github.com/ArroyoSystems/arroyo/issues) — bugs and feature requests * [Arroyo Blog](https://arroyo.dev/blog) — updates from the Arroyo team --- ### Overview (`k8s/arroyo/README.md`) # Arroyo This is the official Helm chart for deploying [Arroyo](https://github.com/ArroyoSystems/arroyo), a distributed stream processing engine, on Kubernetes. Arroyo supports Kubernetes as both a _scheduler_ (for running Arroyo pipeline tasks) and as as a deploy target for the Arroyo control plane. This is the easiest way to get a production quality Arroyo cluster running. See the [docs](https://doc.arroyo.dev/deployment/kubernetes) for full information on how to use this helm chart. Each version of the helm chart is associated by default with a particular release of Arroyo. The latest release is [0.10.3](https://www.arroyo.dev/blog/arroyo-0-10-0). ## Quickstart To quickly get up and running with Arroyo on a local kubernetes cluster (e.g. minikube), first create the following configuration file: _values.yaml_ ```yaml artifactUrl: "/tmp/arroyo/artifacts" checkpointUrl: "/tmp/arroyo/checkpoints" volumes: - name: data hostPath: path: /tmp/arroyo type: DirectoryOrCreate volumeMounts: - name: data mountPath: /tmp/arroyo ``` (Note that if running on a distributed kubernetes cluster, you will need some sort of shared storage for checkpoints and artifacts, for example an object store like S3 or GCS. See the [deployment](https://doc.arroyo.dev/deployment/overview) docs for more details.) Then you can create the cluster with ``` $ helm install arroyo arroyo/arroyo -f values.yaml ``` Once all of the pods start up (which may take a few minutes if you're deploying Postgres) you should be able to access the Arroyo UI by running ``` $ open "http://$(kubectl get service/arroyo-api -o jsonpath='{.spec.clusterIP}')" ``` or if that doesn't work, by proxying the service: ``` $ kubectl port-forward service/arroyo-api 5115:80 ``` and opening http://localhost:5115. ## Configuration The Helm chart provides a number of options, which can be inspected by running ``` $ helm show values arroyo/arroyo ``` The most important options are: - `postgresql.deploy`: Whether to deploy a new Postgres instance. If set to `false`, the chart will expect a Postgres instance to be available with the connection settings determined by `postgresql.externalDatabase` configurations (by default: postgres://arroyo:arroyo@localhost:5432/arroyo). - `prometheus.deploy`: Whether to deploy a new Prometheus instance. If set to `false`, the chart will expect a Prometheus instance to be availble at the URL determined by `prometheus.endpoint` (by default: http://localhost:9090). Prometheus is not required for operation of the system, but is needed for the Arroyo UI metrics to function. By default Arroyo expects you to have a scrape interval of 5s. If you have a higher scrape interval (for example, the default 1m) you will need to update the `prometheus.queryRate` configuration to at least 4x the scrape interval. - `s3.bucket` and `s3.region`: Configures the s3 bucket and region that will be used for storing artifacts and checkpoints. If using s3, the pods will need to have access to the bucket. - `outputDir`: Alternatively, you may configure the pods to write artifacts and checkpoints to a local directory when running a local Kubernetes cluster. You will additionally need to configure `volumes` and `volumeMounts` to make this directory available on all of the pods. The helm chart can be configured either via a `values.yaml` file or via command line arguments. See the [Helm documentation](https://helm.sh/docs/intro/using_helm/#customizing-the-chart-before-installing) for more details. ## Help If you have any questions or need help, please feel free to reach out on our [Discord server](https://discord.gg/cjCr5rVmyR). --- --- METRICS --- - Files Extracted: 2 - Estimated Token Budget: ~2194 tokens - Canonical Reference: https://codewiki.google/github.com/ArroyoSystems/arroyo