console Documentation

File: README.md Copy Chapter

Redpanda Console – A UI for Data Streaming

Redpanda Console (previously known as Kowl) is a web application that helps you manage and debug your Kafka/Redpanda workloads effortlessly.

https://user-images.githubusercontent.com/23424570/220130537-7b0b8596-0a06-4132-90a5-1be3c169e4f4.mp4

Features Copy Chapter

• Message viewer: Explore your topics' messages in our message viewer through ad-hoc queries and dynamic filters. Find any message you want using JavaScript functions to filter messages. Supported encodings are: JSON, Avro, Protobuf, CBOR, XML, MessagePack, Text and Binary (hex view). The used encoding (except Protobuf and CBOR) is recognized automatically.
• Consumer groups: List all your active consumer groups along with their active group offsets, edit group offsets (by group, topic or partition) or delete a consumer group.
• Topic overview: Browse through the list of your Kafka topics, check their configuration, space usage, list all consumers who consume a single topic or watch partition details (such as low and high water marks, message count, ...), embed topic documentation from a git repository and more.
• Cluster overview: List available brokers, their space usage, rack id, health, configuration and other information to get a high level overview of your brokers in your cluster.
• Security: Create, list or edit Kafka ACLs and SASL-SCRAM users.
• Schema Registry: List and manage all aspects of your Avro, Protobuf or JSON schemas within your schema registry.
• Kafka connect: Manage connectors from multiple connect clusters, patch configs, view their current state or restart tasks.
• Redpanda Transforms: Manage and monitor data transforms deployed in your Redpanda cluster.

Code Quality Copy Chapter

The frontend uses react-doctor to statically analyze React components for performance anti-patterns. Run bun run doctor in the frontend/ directory for a local scan. React Doctor also runs automatically in CI on every push and pull request.

Getting Started Copy Chapter

Prerequisites Copy Section

• Redpanda or any Kafka deployment (v1.0.0+) compatible
• A web browser to access the Console, which can be deployed using Docker builds or as a single binary across all platforms (amd64 / arm64)

Installing Copy Section

We offer pre built docker images for RP Console, binary builds and a Helm chart to make the installation as comfortable as possible for you. Please take a look at our dedicated Installation documentation.

Quick Start Copy Section

Do you just want to test RP Console against one of your Kafka clusters without spending too much time on the test setup? Here are some docker commands that allow you to run it locally against an existing Redpanda or Kafka cluster:

Redpanda/Kafka is running locally

Since Console runs in its own container (which has its own network scope), we have to use host.docker.internal as a bootstrap server. That DNS resolves to the host system's ip address. However since the brokers send a list of all brokers' DNS when a client has connected, you have to make sure your advertised listener is connected accordingly, e.g.: PLAINTEXT://host.docker.internal:9092

docker run -p 8080:8080 -e KAFKA_BROKERS=host.docker.internal:9092 docker.redpanda.com/redpandadata/console:latest

Docker supports the --network=host option only on Linux. So Linux users use localhost:9092 as an advertised listener and use the host network namespace instead. Console will then be run as it would be executed on the host machine.

docker run --network=host -p 8080:8080 -e KAFKA_BROKERS=localhost:9092 docker.redpanda.com/redpandadata/console:latest

Kafka is running remotely

Protected via SASL_SSL and trusted certificates (e.g. Confluent Cloud):

docker run -p 8080:8080 -e KAFKA_BROKERS=pkc-4r000.europe-west1.gcp.confluent.cloud:9092 -e KAFKA_TLS_ENABLED=true -e KAFKA_SASL_ENABLED=true -e KAFKA_SASL_USERNAME=xxx -e KAFKA_SASL_PASSWORD=xxx docker.redpanda.com/redpandadata/console:latest

I don't have a running Kafka cluster to test against

We maintain a docker-compose file that launches Redpanda and Console: /docs/local.

Community

Slack is the main way the community interacts with one another in real time :)

GitHub Issues can be used for issues, questions and feature requests.

Code of conduct code of conduct for the community

Contributing docs

File: backend/pkg/proto/embed/protobuf/google/type/README.md Copy Chapter

Google Common Types

This package contains definitions of common types for Google APIs.
All types defined in this package are suitable for different APIs to
exchange data, and will never break binary compatibility. They should
have design quality comparable to major programming languages like
Java and C#.

NOTE: Some common types are defined in the package google.protobuf
as they are directly supported by Protocol Buffers compiler and
runtime. Those types are called Well-Known Types.

Java Utilities Copy Chapter

A set of Java utilities for the Common Types are provided in the
//java/com/google/type/util/ package.

File: docs/features/hosting.md Copy Chapter

title: Hosting path: /docs/features/hosting Copy Chapter

Hosting

Want to host Console under a sub-path (instead of /)? Then this is the document you're looking for.

Config Entries Copy Chapter

There are 3 config entries that should cover most scenarios.
As you can see in docs/config/console.yaml the config entries can be found in the server: group.

• `basePath` Copy Chapter

  The sub-path under which Console will be hosted.
  If you have a proxy in front of Console that sets the X-Forwarded-Prefix header and you didn't disable the setBasePathFromXForwardedPrefix setting (enabled by default), then you don't need to set this.

  Default: not set / empty string

• `setBasePathFromXForwardedPrefix` Copy Chapter

  If true, Console will check the X-Forwarded-Prefix header on incoming requests, and if the header is present its value will be used as the path prefix.
  That means if a request has X-Forwarded-Prefix set, the value set in the basePath setting will be ignored.

  Default: true

• `stripPrefix` Copy Chapter

  Short Version
  If you're hosting Console on a sub-path and are using a reverse-proxy like Traefik with its "StripPrefix" middleware enabled, disable this setting!

  Long Version
  Some proxies (like Traefik with its "StripPrefix" middleware) can remove a prefix from the URL path of an request before forwarding it. This can lead to a situation where both the proxy and Console will try to remove a prefix which could lead to issues.
  If a prefix is set/used it must be removed at some point before reaching Console's internal routing. We recommend that only one part of the stack removes the prefix (even though a scenario where this is a problem is unlikely).
  So if you're using Traefik (or any other proxy that modifies the request path / URL) you should either set stripPrefix: false in Console, or configure the proxy so it doesn't modify the path of a request.

  See the example below for a setup where the double remocal is a problem.

  Default: true

Example Copy Chapter

For the curious, here is an example scenario that, albeit pretty contrived, should illustrate nicely how double removal could be a problem:

• Traefik configured to route /topics to Console and enabled "StripPrefix" middleware
• Console configured with the default settings
• A user types the following address into their browser example.com/topics/topics/example-topic
• Traefik sees the incoming request path /topics/topics/example-topic, sees that the prefix matches and removes it so the path becomes /topics/example-topic, and finally sets /topics in the X-Forwarded-Prefix header
• The request reaches Console, which sees that X-Forwarded-Prefix is set and setBasePathFromXForwardedPrefix is true. So it tries to remove what it thinks is the prefix with the request path /topics/example-topic now becoming /example-topic
• Console tries to find a handler for the route /example-topic but of course it won't find one.

File: docs/features/kafka-connect.md Copy Chapter

title: Kafka Connect path: /docs/features/kafka-connect Copy Chapter

Kafka Connect

Console provides a user interface that enables you to manage multiple Kafka connect clusters
via a user interface. You can inspect the configured connectors, configure them, restart/pause/resume
connectors and also delete them if desired. If you have more than one cluster configured some requests
(such as listing connectors) will query all configured connect clusters and Console will aggregate the results
so that you can see them in one spot.

Configuration Copy Chapter

Below sample configuration can be put at the root level. For each cluster you have to provide at
least a unique name, the HTTP address of the cluster, and the authentication settings if required.
All available configuration options can be found in the reference config.

connect:
  enabled: true
  clusters:
    - name: datawarehouse # Required field, will be used as identifier in the frontend
      url: http://dwh-connect.mycompany.com:8083
      tls:
        enabled: false # Trusted certs are still allowed by default
      username: admin
      # password: # Set via flag --connect.clusters.0.password=secret
    - name: analytics # Required field, will be used as identifier in the frontend
      url: http://analytics.mycompany.com:8083
      # No auth configured on that cluster, hence no username/password set

File: docs/features/protobuf.md Copy Chapter

title: Protobuf Deserialization path: /docs/features/protobuf Copy Chapter

Protobuf Deserialization

If you have one or more topics with Protobuf serialized messages you can configure Redpanda Console to deserialize
the binary content into JSON, so that it will be human readable and can also be used in the filter engine
like a JavaScript object.

To deserialize the binary content, Console needs access to the protobuf schema definitions. These can be provided through multiple sources:

Schema Sources:

• Confluent Schema Registry - Embedded schema IDs in message payload
• Buf Schema Registry (BSR) - Schema metadata in Kafka record headers
• Local Filesystem - .proto files from local directories
• Git Repository - .proto files from a Git repo (cloned and periodically synced)

Topic Mappings:

• Required for: Local filesystem and Git repository sources
• Not required for: Schema Registry and BSR (metadata is self-contained in messages)

All providers can be used together, allowing you to support multiple serialization formats across different topics.

Preparation Copy Chapter

Schema Registry Copy Section

Unlike the other providers the schema registry does not require you to setup mappings that define
what topics use which proto types. Instead this information is inferred from the messages and
the schema registry will be consulted to find the right prototype for deserialization.

The protobuf deserializer will use the same schema registry client that is configured under
kafka.schemaRegistry.

kafka:
  schemaRegistry:
    enabled: true
    urls: ["https://my-schema-registry.com"]
    username: console
    password: redacted # Or set via flags/env variable
serde:
  protobuf:
    enabled: true
    schemaRegistry:
      enabled: true # This tells the proto service to consider the schema registry when deserializing messages
      refreshInterval: 5m # How often the compiled proto schemas in the cache should be updated

Buf Schema Registry (BSR) Copy Section

Redpanda Console supports deserializing Protobuf messages that use Buf Schema Registry (BSR) for schema management.
Unlike Confluent Schema Registry, BSR clients store message type and commit information in Kafka record headers rather than embedding
schema IDs in the message payload.

Key Differences:

• No topic mappings required - Schema information is stored in record headers
• Plain protobuf payload - No magic bytes or schema ID prefix
• Headers-based metadata - Message type and version stored in buf.registry.value.schema.message and buf.registry.value.schema.commit headers

Configuration:

serde:
  protobuf:
    enabled: true
    bufSchemaRegistry:
      enabled: true
      url: "https://buf.build"  # or your BSR instance URL
      token: "your-bsr-auth-token"

Environment Variables:

export SERDE_PROTOBUF_BUFSCHEMAREGISTRY_ENABLED=true
export SERDE_PROTOBUF_BUFSCHEMAREGISTRY_URL=https://buf.build
export SERDE_PROTOBUF_BUFSCHEMAREGISTRY_TOKEN=your-bsr-auth-token

Required Record Headers:

For BSR deserialization to work, Kafka records must include these headers:

• buf.registry.value.schema.message - Fully qualified message name (e.g., com.example.v1.Order)
• buf.registry.value.schema.commit - BSR commit ID for the schema version

Caching:

The BSR client implements intelligent caching to minimize API calls:

• Successfully fetched descriptors are cached for 1 hour
• Failed lookups are cached for 1 minute to avoid repeated failures
• Automatic cleanup every 30 minutes to prevent memory growth

Local Filesystem Copy Section

Put all of your required .proto files into a git repository. It doesn't matter in what directory you put them,
but the files' extension must be .proto. You can configure Redpanda Console to search one or more paths for
proto files:

kafka:
  protobuf:
    enabled: true
    mappings: []
      - topicName: xy
        valueProtoType: fake_model.Order # You can specify the proto type for the record key and/or value (just one will work too)
        keyProtoType: package.Type
    fileSystem:
      enabled: true
      refreshInterval: 5m # 5min is the default refresh interval
      paths:
        - /etc/protos

Git repository Copy Section

If you want to provide the files via a git repository, put all of your required .proto files in there.
It doesn't matter in what directory as Console will search for all files with the file extension .proto
in your repository up to a directory depth of 5 levels. All files with other file extensions will be ignored.

kafka:
  protobuf:
    enabled: true
    mappings: []
      - topicName: xy
        valueProtoType: fake_model.Order # You can specify the proto type for the record key and/or value (just one will work too)
        keyProtoType: package.Type
    # importPaths is a list of paths from which to import Proto files into Redpanda Console.
    # Paths are relative to the root directory.
    # The `git` configuration must be enabled to use this feature.
    importPaths: []
    git:
      enabled: true
      refreshInterval: 5m
        url: https://github.com/redpanda-data/owlshop-protos.git
      basicAuth:
        enabled: true
        username: token # Uses an API token via basic auth
        password: redacted

Imports Copy Section

In order to support imports all prototypes will first be registered in a proto registry so that your
imports can be resolved. Therefore you have to make sure that all imported proto types are part of
the repository. Standard types (such as Google's timestamp type) are included by default so that you
don't need to worry about these.

Configuration Copy Chapter

Topic mappings Copy Section

In the configuration you need to provide a list of all mappings between the Kafka Topic (key/value)
and the Prototype that shall be used to deserialize the payload. Let's assume you have a Kafka topic
called address-v1 and the respective address.proto file in your repository which looks like this:

syntax = "proto3";
package fake_models;

option go_package = "pkg/protobuf";

message Address {
  int32 version = 1;
  string id = 2;
  message Customer {
    string customer_id = 1;
    string customer_type = 2;
  }
}

The required configuration would look like this

mappings:
  - topicName: address-v1
    valueProtoType: fake_model.Address # The full prototype URL is required
    # keyProtoType: The key is a plain string in Kafka, hence we don't have a prototype for the record's key

Full configuration Copy Section

Take a look at the reference config how to configure Protobuf in Console: /docs/config/console.yaml

File: docs/features/topic-documentation.md Copy Chapter

title: Topic Documentation path: /docs/features/topic-documentation Copy Chapter

Topic Documentation

You can embed your topic's documentation into the Redpanda Console user interface by providing access to a GitHub
repository that hosts your documentation files in Markdown format.

Integrating topic documentation into Redpanda Console Copy Chapter

Redpanda Console clones the provided GitHub repository, recursively iterates through all directories in the
repository (up to a max depth of 5) and stores all .md files it finds in memory.
The "Documentation" tab in the frontend will show the markdown of the file matching the name of the Kafka topic.

Config Copy Chapter

In addition to the repository URL and branch, you usually need to configure authentication credentials so
that you can access private repositories. Redpanda Console supports SSH as well as basic auth.
If neither is specified you could still pull publicly accessible repositories.

Following is the configuration:

console:
  topicDocumentation:
    enabled: true
    # Git is where the topic documentation can come from, in the future there might be additional
    git:
      enabled: true
        url: https://github.com/redpanda-data/redpanda-examples
        branch: main
        baseDirectory: console/topic-documentation 
      # How often Console shall pull the repository to look for new files. Set 0 to disable periodic pulls
      refreshInterval: 1m
      # Basic Auth
      # If you want to use GitHub's personal access tokens use `token` as username and pass the token as password
      basicAuth:
        enabled: false
        username: token
        password: #  This can be set via the via the --owl.topic-documentation.git.basic-auth.password flag as well
      # SSH Auth
      # You can either pass the private key file directly via flag or yaml config or refer to a mounted key file
      ssh:
        enabled: false
        username:
        privateKey: # This can be set via the via the --owl.topic-documentation.git.ssh.private-key flag as well
        privateKeyFilepath:
        passphrase: # This can be set via the via the --owl.topic-documentation.git.ssh.passphrase flag as well

File: docs/installation.md Copy Chapter

title: Installation & Deployment path: /docs/installation Copy Chapter

Installation & Deployment

This page describes how to install Redpanda Console on your platform. This documentation targets the installation
of Redpanda Console without Redpanda or Apache Kafka. Redpanda maintains documentation that is written
for the installation of Redpanda Platform which includes Redpanda Console here: https://docs.redpanda.com/docs/deploy/.

Docker images Copy Chapter

Docker images are available on Dockerhub. We comply with semantic versioning and every new release is pushed to Dockerhub.
Additionally we publish a new docker image after each push to any branch to a separate Dockerhub repository.

Binaries Copy Chapter

Precompiled binaries for different platforms are attached to each release.

Deployment Copy Chapter

While RP Console was written to be deployed in Kubernetes it does not have any specific dependencies.
You can run it with any other container orchestrator or on bare metal.

Hosting and Proxies (path rewriting) Copy Section

If you want to host Console under a sub path (e.g. domain.com/some/sub/path instead of domain.com/)
take a look at HTTP Path Rewrites for more information.

Kubernetes Copy Section

You can use the Redpanda Console Helm chart to deploy Redpanda Console into your Kubernetes cluster.
To install the Helm repository and chart, run:

helm repo add redpanda 'https://charts.redpanda.com/' 
helm repo update
helm install redpanda/console -f myvalues.yaml

We maintain a Helm chart which makes
it easy to deploy Console on Kubernetes. Please refer to this repo for further documentation and/or help.

If you want to create your own manifest you can browse through the Helm chart linked above.

Configuration Copy Section

Console can be configured using environment variables and a YAML config. To use YAML you need to set the path to
the YAML config either using the flag -config.filepath or by setting the env variable CONFIG_FILEPATH.
Additionally, you can use flags for all sensitive input (such as Kafka passwords).
All available flags along with a reference YAML config are described below.

If your Kafka cluster requires TLS authentication you can make certificates available via volume mounts.
Afterwards you need to configure the paths to your certificates in the YAML config as shown in the
reference config (see section config files).

Environment variables

Everything that can be configured via YAML can also be configured using env variables.
Due to the complexity of the config we recommend using YAML configs for most deployments,
but you can also use env variables which might be handy to start Console within a few secs
by executing a one line cli statement.

All environment variable keys are generated by using the YAML config names as described
in the reference config.

Flags

In general we only use flags to specify the path to your config file which contains
all the actual configuration. Because Console requires sensitive information such as
credentials, we offer further flags so that you don't need to put them into your
YAML configuration. This way the YAML config can be pushed into your repository
while sensitive information can be mounted during the deployment process.

Config Files

All available settings which can be configured in your YAML configs can be found here and should document themselves:

/docs/config/console.yaml

File: docs/menu.md Copy Chapter

• Getting Started
  • Overview
  • Installation
• Features
  • Hosting
  • Kafka Connect
  • Topic Documentation
  • Protobuf

File: docs/README.md Copy Chapter

title: Documentation path: /docs Copy Chapter

Documentation

In this folder you'll find documentation about how to configure Redpanda Console.
If there are still open questions after reading this documentation don't hesitate to submit an issue.

• Getting started
  • Installation
  • Helm Chart
  • Terraform Module
• Features
  • Hosting
  • Kafka Connect
  • Topic Documentation
  • Protobuf
• Reference Config
  • console.yaml

File: licenses/README.md Copy Chapter

FAQ

There are 2 licenses for Redpanda. BSL covers our core and RCL (Redpanda Community License)
which covers enterprise features.

1. BSL:
   The intent as we mentioned in our blog post
   is to allow any use of the none enterprise features Redpanda unless you want to
   offer Redpanda as a service. The license will at some point in the future (change date) become Apache 2.

2. RCL:
   Redpanda Community License - is intended to allow you to use enterprise features
   that you pay for.

We thank MariaDB and CockroachDB for pioneering the use of BSL for storage systems.
It gave us a path to build an infrastructure company in the age of the hyperclouds.

How to recreate the third party licenses files Copy Chapter

Third party licenses audit files are generated on demand, the steps to do so are documented below.

1. backend run task backend:licenses:third-party
2. frontend:
   • cd frontend
   • bunx license-checker --csv --production > ../licenses/third_party_js.csv