## File: README.md # Redpanda Console – A UI for Data Streaming [](https://hub.docker.com/r/redpandadata/console/tags) Redpanda Console (previously known as Kowl) is a web application that helps you manage and debug your Kafka/Redpanda workloads effortlessly. https://user-images.githubusercontent.com/23424570/220130537-7b0b8596-0a06-4132-90a5-1be3c169e4f4.mp4 ## Features - **Message viewer:** Explore your topics' messages in our message viewer through ad-hoc queries and dynamic filters. Find any message you want using JavaScript functions to filter messages. Supported encodings are: JSON, Avro, Protobuf, CBOR, XML, MessagePack, Text and Binary (hex view). The used encoding (except Protobuf and CBOR) is recognized automatically. - **Consumer groups:** List all your active consumer groups along with their active group offsets, edit group offsets (by group, topic or partition) or delete a consumer group. - **Topic overview:** Browse through the list of your Kafka topics, check their configuration, space usage, list all consumers who consume a single topic or watch partition details (such as low and high water marks, message count, ...), embed topic documentation from a git repository and more. - **Cluster overview:** List available brokers, their space usage, rack id, health, configuration and other information to get a high level overview of your brokers in your cluster. - **Security**: Create, list or edit Kafka ACLs and SASL-SCRAM users. - **Schema Registry:** List and manage all aspects of your Avro, Protobuf or JSON schemas within your schema registry. - **Kafka connect:** Manage connectors from multiple connect clusters, patch configs, view their current state or restart tasks. - **Redpanda Transforms:** Manage and monitor data transforms deployed in your Redpanda cluster. ## Code Quality The frontend uses [react-doctor](https://github.com/aidenybai/react-doctor) to statically analyze React components for performance anti-patterns. Run `bun run doctor` in the `frontend/` directory for a local scan. React Doctor also runs automatically in CI on every push and pull request. ## Getting Started ### Prerequisites - Redpanda or any Kafka deployment (v1.0.0+) compatible - A web browser to access the Console, which can be deployed using Docker builds or as a single binary across all platforms (amd64 / arm64) ### Installing We offer pre built docker images for RP Console, binary builds and a Helm chart to make the installation as comfortable as possible for you. Please take a look at our dedicated [Installation documentation](https://docs.redpanda.com/current/get-started/quick-start/). ### Quick Start Do you just want to test RP Console against one of your Kafka clusters without spending too much time on the test setup? Here are some docker commands that allow you to run it locally against an existing Redpanda or Kafka cluster: #### Redpanda/Kafka is running locally Since Console runs in its own container (which has its own network scope), we have to use host.docker.internal as a bootstrap server. That DNS resolves to the host system's ip address. However since the brokers send a list of all brokers' DNS when a client has connected, you have to make sure your advertised listener is connected accordingly, e.g.: `PLAINTEXT://host.docker.internal:9092` ```shell docker run -p 8080:8080 -e KAFKA_BROKERS=host.docker.internal:9092 docker.redpanda.com/redpandadata/console:latest ``` Docker supports the `--network=host` option only on Linux. So Linux users use `localhost:9092` as an advertised listener and use the host network namespace instead. Console will then be run as it would be executed on the host machine. ```shell docker run --network=host -p 8080:8080 -e KAFKA_BROKERS=localhost:9092 docker.redpanda.com/redpandadata/console:latest ``` #### Kafka is running remotely Protected via SASL_SSL and trusted certificates (e.g. Confluent Cloud): ```shell docker run -p 8080:8080 -e KAFKA_BROKERS=pkc-4r000.europe-west1.gcp.confluent.cloud:9092 -e KAFKA_TLS_ENABLED=true -e KAFKA_SASL_ENABLED=true -e KAFKA_SASL_USERNAME=xxx -e KAFKA_SASL_PASSWORD=xxx docker.redpanda.com/redpandadata/console:latest ``` #### I don't have a running Kafka cluster to test against We maintain a docker-compose file that launches Redpanda and Console: [/docs/local](./docs/local). # Community [Slack](https://redpanda.com/slack) is the main way the community interacts with one another in real time :) [GitHub Issues](https://github.com/redpanda-data/console/issues) can be used for issues, questions and feature requests. [Code of conduct](https://github.com/redpanda-data/redpanda/blob/dev/CODE_OF_CONDUCT.md) code of conduct for the community [Contributing docs](./CONTRIBUTING.md) --- ## File: backend/pkg/proto/embed/protobuf/google/type/README.md # Google Common Types This package contains definitions of common types for Google APIs. All types defined in this package are suitable for different APIs to exchange data, and will never break binary compatibility. They should have design quality comparable to major programming languages like Java and C#. NOTE: Some common types are defined in the package `google.protobuf` as they are directly supported by Protocol Buffers compiler and runtime. Those types are called Well-Known Types. ## Java Utilities A set of Java utilities for the Common Types are provided in the `//java/com/google/type/util/` package. --- ## File: docs/features/hosting.md --- title: Hosting path: /docs/features/hosting --- # Hosting Want to host Console under a sub-path (instead of `/`)? Then this is the document you're looking for. ## Config Entries There are 3 config entries that should cover most scenarios. As you can see in [docs/config/console.yaml](../config/console.yaml) the config entries can be found in the `server:` group. - `basePath` --- The sub-path under which Console will be hosted. If you have a proxy in front of Console that sets the `X-Forwarded-Prefix` header and you didn't disable the `setBasePathFromXForwardedPrefix` setting (enabled by default), then you don't need to set this. Default: not set / empty string - `setBasePathFromXForwardedPrefix` --- If true, Console will check the `X-Forwarded-Prefix` header on incoming requests, and if the header is present its value will be used as the path prefix. That means if a request has `X-Forwarded-Prefix` set, the value set in the `basePath` setting will be ignored. Default: `true` - `stripPrefix` --- **Short Version** If you're hosting Console on a sub-path and are using a reverse-proxy like Traefik with its "StripPrefix" middleware enabled, disable this setting! **Long Version** Some proxies (like Traefik with its "StripPrefix" middleware) can remove a prefix from the URL path of an request before forwarding it. This can lead to a situation where both the proxy and Console will try to remove a prefix which *could* lead to issues. If a prefix is set/used it **must** be removed at some point before reaching Console's internal routing. We recommend that only **one** part of the stack removes the prefix (even though a scenario where this is a problem is unlikely). So if you're using Traefik (or any other proxy that modifies the request path / URL) you should either set `stripPrefix: false` in Console, or configure the proxy so it doesn't modify the path of a request. See the example below for a setup where the double remocal is a problem. Default: `true` ## Example For the curious, here is an example scenario that, albeit pretty contrived, should illustrate nicely how double removal could be a problem: > - Traefik configured to route `/topics` to Console and enabled "StripPrefix" middleware > - Console configured with the default settings > - A user types the following address into their browser `example.com/topics/topics/example-topic` > - Traefik sees the incoming request path `/topics/topics/example-topic`, sees that the prefix matches and removes it so the path becomes `/topics/example-topic`, and finally sets `/topics` in the `X-Forwarded-Prefix` header > - The request reaches Console, which sees that `X-Forwarded-Prefix` is set and `setBasePathFromXForwardedPrefix` is true. So it tries to remove what it thinks is the prefix with the request path `/topics/example-topic` now becoming `/example-topic` > - Console tries to find a handler for the route `/example-topic` but of course it won't find one. --- ## File: docs/features/kafka-connect.md --- title: Kafka Connect path: /docs/features/kafka-connect --- # Kafka Connect Console provides a user interface that enables you to manage multiple Kafka connect clusters via a user interface. You can inspect the configured connectors, configure them, restart/pause/resume connectors and also delete them if desired. If you have more than one cluster configured some requests (such as listing connectors) will query all configured connect clusters and Console will aggregate the results so that you can see them in one spot. ## Configuration Below sample configuration can be put at the root level. For each cluster you have to provide at least a unique name, the HTTP address of the cluster, and the authentication settings if required. All available configuration options can be found in the [reference config](/docs/config/console.yaml). ```yaml connect: enabled: true clusters: - name: datawarehouse # Required field, will be used as identifier in the frontend url: http://dwh-connect.mycompany.com:8083 tls: enabled: false # Trusted certs are still allowed by default username: admin # password: # Set via flag --connect.clusters.0.password=secret - name: analytics # Required field, will be used as identifier in the frontend url: http://analytics.mycompany.com:8083 # No auth configured on that cluster, hence no username/password set ``` --- ## File: docs/features/protobuf.md --- title: Protobuf Deserialization path: /docs/features/protobuf --- # Protobuf Deserialization If you have one or more topics with Protobuf serialized messages you can configure Redpanda Console to deserialize the binary content into JSON, so that it will be human readable and can also be used in the filter engine like a JavaScript object. To deserialize the binary content, Console needs access to the protobuf schema definitions. These can be provided through multiple sources: **Schema Sources:** - **Confluent Schema Registry** - Embedded schema IDs in message payload - **Buf Schema Registry (BSR)** - Schema metadata in Kafka record headers - **Local Filesystem** - .proto files from local directories - **Git Repository** - .proto files from a Git repo (cloned and periodically synced) **Topic Mappings:** - **Required for**: Local filesystem and Git repository sources - **Not required for**: Schema Registry and BSR (metadata is self-contained in messages) All providers can be used together, allowing you to support multiple serialization formats across different topics. ## Preparation ### Schema Registry Unlike the other providers the schema registry does not require you to setup mappings that define what topics use which proto types. Instead this information is inferred from the messages and the schema registry will be consulted to find the right prototype for deserialization. The protobuf deserializer will use the same schema registry client that is configured under `kafka.schemaRegistry`. ```yaml kafka: schemaRegistry: enabled: true urls: ["https://my-schema-registry.com"] username: console password: redacted # Or set via flags/env variable serde: protobuf: enabled: true schemaRegistry: enabled: true # This tells the proto service to consider the schema registry when deserializing messages refreshInterval: 5m # How often the compiled proto schemas in the cache should be updated ``` ### Buf Schema Registry (BSR) Redpanda Console supports deserializing Protobuf messages that use [Buf Schema Registry (BSR)](https://buf.build/docs/bsr) for schema management. Unlike Confluent Schema Registry, BSR clients store message type and commit information in Kafka record headers rather than embedding schema IDs in the message payload. **Key Differences:** - **No topic mappings required** - Schema information is stored in record headers - **Plain protobuf payload** - No magic bytes or schema ID prefix - **Headers-based metadata** - Message type and version stored in `buf.registry.value.schema.message` and `buf.registry.value.schema.commit` headers **Configuration:** ```yaml serde: protobuf: enabled: true bufSchemaRegistry: enabled: true url: "https://buf.build" # or your BSR instance URL token: "your-bsr-auth-token" ``` **Environment Variables:** ```bash export SERDE_PROTOBUF_BUFSCHEMAREGISTRY_ENABLED=true export SERDE_PROTOBUF_BUFSCHEMAREGISTRY_URL=https://buf.build export SERDE_PROTOBUF_BUFSCHEMAREGISTRY_TOKEN=your-bsr-auth-token ``` **Required Record Headers:** For BSR deserialization to work, Kafka records must include these headers: - `buf.registry.value.schema.message` - Fully qualified message name (e.g., `com.example.v1.Order`) - `buf.registry.value.schema.commit` - BSR commit ID for the schema version **Caching:** The BSR client implements intelligent caching to minimize API calls: - Successfully fetched descriptors are cached for 1 hour - Failed lookups are cached for 1 minute to avoid repeated failures - Automatic cleanup every 30 minutes to prevent memory growth ### Local Filesystem Put all of your required .proto files into a git repository. It doesn't matter in what directory you put them, but the files' extension must be `.proto`. You can configure Redpanda Console to search one or more paths for proto files: ```yaml kafka: protobuf: enabled: true mappings: [] - topicName: xy valueProtoType: fake_model.Order # You can specify the proto type for the record key and/or value (just one will work too) keyProtoType: package.Type fileSystem: enabled: true refreshInterval: 5m # 5min is the default refresh interval paths: - /etc/protos ``` ### Git repository If you want to provide the files via a git repository, put all of your required .proto files in there. It doesn't matter in what directory as Console will search for all files with the file extension `.proto` in your repository up to a directory depth of 5 levels. All files with other file extensions will be ignored. ```yaml kafka: protobuf: enabled: true mappings: [] - topicName: xy valueProtoType: fake_model.Order # You can specify the proto type for the record key and/or value (just one will work too) keyProtoType: package.Type # importPaths is a list of paths from which to import Proto files into Redpanda Console. # Paths are relative to the root directory. # The `git` configuration must be enabled to use this feature. importPaths: [] git: enabled: true refreshInterval: 5m url: https://github.com/redpanda-data/owlshop-protos.git basicAuth: enabled: true username: token # Uses an API token via basic auth password: redacted ``` ### Imports In order to support imports all prototypes will first be registered in a proto registry so that your imports can be resolved. Therefore you have to make sure that all imported proto types are part of the repository. Standard types (such as Google's timestamp type) are included by default so that you don't need to worry about these. ## Configuration ### Topic mappings In the configuration you need to provide a list of all mappings between the Kafka Topic (key/value) and the Prototype that shall be used to deserialize the payload. Let's assume you have a Kafka topic called `address-v1` and the respective `address.proto` file in your repository which looks like this: ```proto syntax = "proto3"; package fake_models; option go_package = "pkg/protobuf"; message Address { int32 version = 1; string id = 2; message Customer { string customer_id = 1; string customer_type = 2; } } ``` The required configuration would look like this ```yaml mappings: - topicName: address-v1 valueProtoType: fake_model.Address # The full prototype URL is required # keyProtoType: The key is a plain string in Kafka, hence we don't have a prototype for the record's key ``` ### Full configuration Take a look at the reference config how to configure Protobuf in Console: [/docs/config/console.yaml](https://github.com/redpanda-data/console/blob/master/docs/config/console.yaml) --- ## File: docs/features/topic-documentation.md --- title: Topic Documentation path: /docs/features/topic-documentation --- # Topic Documentation You can embed your topic's documentation into the Redpanda Console user interface by providing access to a GitHub repository that hosts your documentation files in Markdown format. ## Integrating topic documentation into Redpanda Console Redpanda Console clones the provided GitHub repository, recursively iterates through all directories in the repository (up to a max depth of 5) and stores all `.md` files it finds in memory. The "Documentation" tab in the frontend will show the markdown of the file matching the name of the Kafka topic. | Path/Filename | Kafka Topic Name | Matches | | -------------------- | ---------------- | ------------------ | | ordersv2.md | orders-v2 | :x: | | Orders-v2.md | orders-v2 | :x: | | orders-v2.md | orders-v2 | :white_check_mark: | | /orders/orders-v2.md | orders-v2 | :white_check_mark: | ## Config In addition to the repository URL and branch, you usually need to configure authentication credentials so that you can access private repositories. Redpanda Console supports SSH as well as basic auth. If neither is specified you could still pull publicly accessible repositories. Following is the configuration: ```yaml console: topicDocumentation: enabled: true # Git is where the topic documentation can come from, in the future there might be additional git: enabled: true url: https://github.com/redpanda-data/redpanda-examples branch: main baseDirectory: console/topic-documentation # How often Console shall pull the repository to look for new files. Set 0 to disable periodic pulls refreshInterval: 1m # Basic Auth # If you want to use GitHub's personal access tokens use `token` as username and pass the token as password basicAuth: enabled: false username: token password: # This can be set via the via the --owl.topic-documentation.git.basic-auth.password flag as well # SSH Auth # You can either pass the private key file directly via flag or yaml config or refer to a mounted key file ssh: enabled: false username: privateKey: # This can be set via the via the --owl.topic-documentation.git.ssh.private-key flag as well privateKeyFilepath: passphrase: # This can be set via the via the --owl.topic-documentation.git.ssh.passphrase flag as well ``` --- ## File: docs/installation.md --- title: Installation & Deployment path: /docs/installation --- # Installation & Deployment This page describes how to install Redpanda Console on your platform. This documentation targets the installation of Redpanda Console without Redpanda or Apache Kafka. Redpanda maintains documentation that is written for the installation of Redpanda Platform which includes Redpanda Console here: https://docs.redpanda.com/docs/deploy/. ## Docker images Docker images are available on [Dockerhub](https://hub.docker.com/r/redpandadata/console/tags). We comply with semantic versioning and every new release is pushed to Dockerhub. Additionally we publish a new docker image after each push to any branch to a separate [Dockerhub repository](https://hub.docker.com/r/redpandadata/console-unstable/tags). ## Binaries Precompiled binaries for different platforms are attached to each release. ## Deployment While RP Console was written to be deployed in Kubernetes it does not have any specific dependencies. You can run it with any other container orchestrator or on bare metal. ### Hosting and Proxies (path rewriting) If you want to host Console under a sub path (e.g. `domain.com/some/sub/path` instead of `domain.com/`) take a look at [HTTP Path Rewrites](https://docs.redpanda.com/docs/manage/console/http-path-rewrites/) for more information. ### Kubernetes You can use the Redpanda Console Helm chart to deploy Redpanda Console into your Kubernetes cluster. To install the Helm repository and chart, run: ```bash helm repo add redpanda 'https://charts.redpanda.com/' helm repo update helm install redpanda/console -f myvalues.yaml ``` We maintain a [Helm chart](https://github.com/redpanda-data/helm-charts/tree/main/charts/console) which makes it easy to deploy Console on Kubernetes. Please refer to this repo for further documentation and/or help. If you want to create your own manifest you can browse through the Helm chart linked above. ### Configuration Console can be configured using environment variables and a YAML config. To use YAML you need to set the path to the YAML config either using the flag `-config.filepath` or by setting the env variable `CONFIG_FILEPATH`. Additionally, you can use flags for all sensitive input (such as Kafka passwords). All available flags along with a reference YAML config are described below. If your Kafka cluster requires TLS authentication you can make certificates available via volume mounts. Afterwards you need to configure the paths to your certificates in the YAML config as shown in the reference config (see section [config files](#config-files)). #### Environment variables Everything that can be configured via YAML can also be configured using env variables. Due to the complexity of the config we recommend using YAML configs for most deployments, but you can also use env variables which might be handy to start Console within a few secs by executing a one line cli statement. All environment variable keys are generated by using the YAML config names as described in the [reference config](/docs/config/console.yaml). #### Flags In general we only use flags to specify the path to your config file which contains all the actual configuration. Because Console requires sensitive information such as credentials, we offer further flags so that you don't need to put them into your YAML configuration. This way the YAML config can be pushed into your repository while sensitive information can be mounted during the deployment process. | Argument | Description | Default | | --- | --- | --- | | --config.filepath | Path to the config file | (No default) | | --kafka.sasl.password | SASL Password | (No default) | | --kafka.sasl.gssapi.password | Kerberos password if auth type user auth is used | (No default) | | --kafka.tls.passphrase | Passphrase to decrypt the TLS key (leave empty for unencrypted key files) | (No default) | #### Config Files All available settings which can be configured in your YAML configs can be found here and should document themselves: [/docs/config/console.yaml](https://github.com/redpanda-data/console/blob/master/docs/config/console.yaml) --- ## File: docs/menu.md - **Getting Started** - [Overview](./README.md) - [Installation](./installation.md) - **Features** - [Hosting](./features/hosting.md) - [Kafka Connect](./features/kafka-connect.md) - [Topic Documentation](./features/topic-documentation.md) - [Protobuf](./features/protobuf.md) --- ## File: docs/README.md --- title: Documentation path: /docs --- # Documentation In this folder you'll find documentation about how to configure Redpanda Console. If there are still open questions after reading this documentation don't hesitate to submit an issue. - Getting started - [Installation](./installation.md) - [Helm Chart](https://github.com/redpanda-data/helm-charts) - [Terraform Module](https://github.com/cloudhut/terraform-modules) - Features - [Hosting](./features/hosting.md) - [Kafka Connect](./features/kafka-connect.md) - [Topic Documentation](./features/topic-documentation.md) - [Protobuf](./features/protobuf.md) - Reference Config - [console.yaml](./config/console.yaml) --- ## File: licenses/README.md # FAQ There are 2 licenses for Redpanda. BSL covers our core and RCL (Redpanda Community License) which covers enterprise features. 1. [BSL](https://github.com/redpanda-data/redpanda/blob/dev/licenses/bsl.md): The intent as we mentioned in our [blog post](https://redpanda.com/blog/open-source/) is to allow any use of the none enterprise features Redpanda unless you want to offer Redpanda as a service. The license will at some point in the future (change date) become Apache 2. 2. [RCL](https://github.com/redpanda-data/redpanda/blob/dev/licenses/rcl.md): Redpanda Community License - is intended to allow you to use enterprise features that you pay for. We thank MariaDB and CockroachDB for pioneering the use of BSL for storage systems. It gave us a path to build an infrastructure company in the age of the hyperclouds. ## How to recreate the third party licenses files Third party licenses audit files are generated on demand, the steps to do so are documented below. 1. backend run `task backend:licenses:third-party` 2. frontend: * `cd frontend` * `bunx license-checker --csv --production > ../licenses/third_party_js.csv`