pulsar Documentation

1. Project Overview & Quickstart (apache/pulsar) Copy Chapter

File: README.md Copy Chapter

Pulsar is a distributed pub-sub messaging platform with a very
flexible messaging model and an intuitive client API.

Learn more about Pulsar at https://pulsar.apache.org

Main features Copy Chapter

• Horizontally scalable (Millions of independent topics and millions
  of messages published per second)
• Strong ordering and consistency guarantees
• Low latency durable storage
• Topic and queue semantics
• Load balancer
• Designed for being deployed as a hosted service:
  • Multi-tenant
  • Authentication
  • Authorization
  • Quotas
  • Support mixing very different workloads
  • Optional hardware isolation
• Keeps track of consumer cursor position
• REST API for provisioning, admin and stats
• Geo replication
• Transparent handling of partitioned topics
• Transparent batching of messages

Repositories Copy Chapter

This repository is the main repository of Apache Pulsar. Pulsar PMC also maintains other repositories for
components in the Pulsar ecosystem, including connectors, adapters, and other language clients.

• Pulsar Core

Helm Chart Copy Section

• Pulsar Helm Chart

Ecosystem Copy Section

• Pulsar Adapters

Clients Copy Section

• .NET/C# Client
• C++ Client
• Go Client
• NodeJS Client
• Python Client
• Reactive Java Client

Dashboard & Management Tools Copy Section

• Dekaf UI
• Pulsar Manager

Website Copy Section

• Pulsar Site

CI/CD Copy Section

• Pulsar CI

Archived/Halted Copy Section

• Pulsar Connectors (bundled as pulsar-io)
• Pulsar Translation
• Ruby Client

Pulsar Runtime Java Version Recommendation Copy Chapter

Note:

When using Java versions, it is recommended to
use a recent version of a particular Java release (17 or 21)
with the most recent bug fixes and security patches.
For example, the JVM bug JDK-8351933 can cause stability issues in
Pulsar.
JDK-8351933 was fixed in Java 17.0.17+ and Java 21.0.8+.
Pulsar Docker images come with the most recent Java version at the time of release.

When using the Pulsar Java client, it is recommended to set specific system properties and JVM options to allow optimal performance.

pulsar ver >= 4.1 and master branch Copy Section

Docker image Java runtime: 21

3.3 <= pulsar ver <= 4.0 Copy Section

Docker image Java runtime: 21

2.10 <= pulsar ver <= 3.0 Copy Section

Docker image Java runtime: 17

2.8 <= pulsar ver <= 2.10 Copy Section

pulsar ver < 2.8 Copy Section

Build Pulsar Copy Chapter

The quick start below covers the essentials; see CONTRIBUTING.md → Building
for the full build and lint commands, and ARCHITECTURE.md → Build infrastructure
for the Gradle build infrastructure and how to change build files.

Requirements Copy Section

• JDK

  Pulsar Version	JDK Version
  master	JDK 21 or JDK 25
  4.0+	JDK 21
  2.11 +	JDK 17
  2.8 / 2.9 / 2.10	JDK 11
  2.7 -	JDK 8

• zip

There is also a guide for setting up the tooling for building Pulsar.

Note:

This project includes a Gradle Wrapper so no separate Gradle installation is needed.
Use ./gradlew on Linux/macOS and gradlew.bat on Windows.

For a better developer experience, install Gradle command-line completion (gradle-completion installation) for bash and zsh shells.

Build Copy Section

./gradlew assemble                                                       # compile and assemble
./gradlew :pulsar-client-original:test --tests "ConsumerBuilderImplTest" # run a single test
bin/pulsar standalone                                                    # run a standalone service
./gradlew quickCheck                                                     # license headers + checkstyle, no compile
./gradlew sanityCheck                                                    # quickCheck + compile main/test (pre-PR)

For the full build, lint, test, and PR workflow — test groups, integration tests, Personal CI, and PR
conventions — see CONTRIBUTING.md. For the module map and the Gradle build
infrastructure see ARCHITECTURE.md, and for coding conventions see
CODING.md. AI coding agents should start from AGENTS.md.

Check https://pulsar.apache.org for documentation and examples.

Build custom docker images Copy Chapter

The commands used in the Apache Pulsar release process can be found in the release process documentation.

Here are some general instructions for building custom docker images:

• Java 21 is the recommended JDK version since branch-4.0.

The following command builds the docker image apachepulsar/pulsar:latest:

./gradlew docker

After the image is built, it can be tagged and pushed to your custom repository. Here's an example of a bash script that tags the docker image with the current version and git revision and pushes it to localhost:32000/apachepulsar.

image_repo_and_project=localhost:32000/apachepulsar
pulsar_version=$(src/get-pulsar-version.sh)
gitrev=$(git rev-parse HEAD | colrm 10)
tag="${pulsar_version}-${gitrev}"
echo "Using tag $tag"
docker tag apachepulsar/pulsar:latest ${image_repo_and_project}/pulsar:$tag
docker push ${image_repo_and_project}/pulsar:$tag

Setting up your IDE Copy Chapter

Read https://pulsar.apache.org/contribute/setup-ide for setting up IntelliJ IDEA or Eclipse for developing Pulsar.

Documentation Copy Chapter

Note:

For how to make contributions to Pulsar documentation, see Pulsar Documentation Contribution Guide.

Contact Copy Chapter

Mailing lists

• The mailing lists are the primary contact for the Apache Pulsar project.
• Your email to the mailing list might be placed in a moderation queue if you haven't joined the mailing list by subscribing before posting.

Note

Please note that security-related issues or concerns should not be reported in public channels.
Follow the instructions in the Security Policy to contact the ASF Security Team and the Apache Pulsar PMC directly.

Slack

Pulsar slack channel at https://apache-pulsar.slack.com/

You can self-register at https://communityinviter.com/apps/apache-pulsar/apache-pulsar

Security Policy Copy Chapter

See the Pulsar security policy for the full policy. In short: never disclose a suspected vulnerability publicly (issue, PR, or discussion) before a fix is released, and a human must verify and take responsibility for a report — autonomous agents must not file security reports on their own.

Reporting a security vulnerability Copy Section

Report privately by email to the Apache Security Team at [email protected]; you may also copy the Apache Pulsar PMC's private list, [email protected]. See the Pulsar security policy for more detail.

Checking exposure to an already-public CVE Copy Section

An already-public CVE is not a private disclosure. Search PRs and issues by the CVE id at https://github.com/apache/pulsar/pulls (including closed ones), then ask via a GitHub issue or the [email protected] mailing list if needed. Pulsar's supported versions are listed at https://pulsar.apache.org/contribute/release-policy/; upgrade to a supported version to receive security updates. See the Pulsar security policy.

License Copy Chapter

Licensed under the Apache License, Version 2.0: http://www.apache.org/licenses/LICENSE-2.0

Crypto Notice Copy Chapter

This distribution includes cryptographic software. The country in which you currently reside may have restrictions on the import, possession, use, and/or re-export to another country, of encryption software. BEFORE using any encryption software, please check your country's laws, regulations and policies concerning the import, possession, or use, and re-export of encryption software, to see if this is permitted. See The Wassenaar Arrangement for more information.

The U.S. Government Department of Commerce, Bureau of Industry and Security (BIS), has classified this software as Export Commodity Control Number (ECCN) 5D002.C.1, which includes information security software using or performing cryptographic functions with asymmetric algorithms. The form and manner of this Apache Software Foundation distribution makes it eligible for export under the License Exception ENC Technology Software Unrestricted (TSU) exception (see the BIS Export Administration Regulations, Section 740.13) for both object code and source code.

The following provides more details on the included cryptographic software: Pulsar uses the SSL library from Bouncy Castle written by http://www.bouncycastle.org.

File: deployment/kubernetes/helm/README.md Copy Chapter

The Helm Chart is moved to pulsar-helm-chart.

Read Deploying Pulsar on Kubernetes for more details.

File: deployment/kubernetes/README.md Copy Chapter

This directory contains the Kubernetes services definitions for all
the components required to do a complete Pulsar deployment.

Refer to Kubernetes.md document for instructions on
how to deploy Pulsar on a Kubernetes cluster.

File: docker/README.md Copy Chapter

Apache Pulsar Docker Images

The Apache Pulsar community produces the apachepulsar/pulsar docker image with each official release.

Since the 2.10.0 release, these docker images run as an unnamed, non-root user that is also part of the root group, by
default. This was done to increase container security. The user is part of the root group to ensure that the container
image can easily run on OpenShift and to ensure that the Pulsar process can write to configuration files.

Development Copy Chapter

You can build and test the docker image on your own machine by running the ./build.sh script in this directory.
Note that you first must build the project in order to have the right dependencies in your local environment.

Building Derivative Custom Images Copy Chapter

If you want to use a connector, you can easily build an image with a curated list of connectors based on the official
Apache Pulsar image. For more information please see the
Pulsar Connectors repository.

Troubleshooting non-root containers Copy Chapter

Troubleshooting is harder because the docker image runs as a non-root user. For example, a non-root user won't be able
to download arbitrary utilities. There are several ways to troubleshoot.

One option is to build a custom docker image that includes your preferred debugging tools. Here is an example of adding
some tools to an existing docker image.

FROM apachepulsar/pulsar:2.10.0

# Switch to root user to download tools
USER 0

# Install your preferred utilities
RUN apt-get update \
     && apt-get install -y vim net-tools unzip \
     && apt-get clean \
     && rm -rf /var/lib/apt/lists/*
     
# Assuming you still want to run as a non root user by default
USER 10000

The remaining debug options depend on your environment. For example, if you have access to the host running your
container, you might be able to use the docker exec command to shell into the container. By using the --user
argument, you can run as the root user.

If you're running your container on kubernetes, you can override the container's default user by setting the pod's
securityContext.

Bitnami provides a helpful guide here: https://engineering.bitnami.com/articles/running-non-root-containers-on-openshift.html.

File: docker-compose/kitchen-sink/README.MD Copy Chapter

Apache Pulsar Docker Compose

In this folder path, run the following command to start the cluster.

docker-compose up

This command starts individual containers for the following services:

• ZooKeeper (3)
• BookKeeper (3)
• Pulsar (3)
• Proxy (1)
• WebSocket (1)
• Function (1)

Motivation Copy Chapter

Most of what is available with Docker and Apache Pulsar uses the standalone version of Apache Pulsar, which is fantastic. But, users need to take care of all of the complexity of Apache Pulsar.

This aims to create a Docker implementation that allows each individual part to be exposed to learn how it all fits together.

URLs Copy Chapter

Assume that localhost addresses for the following endpoints are available:

• broker-admin - http://localhost:8080: access the broker REST interface

• broker-service-url - pulsar//:locahost:6650: broker service URL for use with producers and consumers

You can shut it down with the following command in the folder path:

docker-compose down

You may need to delete the ./data folder created by the Pulsar Manager.

Notes Copy Chapter

Docker is notorious for being difficult to manage the startup order of containers. In Pulsar, startup order is very significant. This may cause some containers to fail and have to be restarted.

Generally speaking, everything should be started up successfully in about 2 minutes.

If you plan to use this in production, more care should be taken on restart policies, logging, configuration, etc.

File: grafana/README.md Copy Chapter

Grafana Dashboards Copy Chapter

In the dashboards directory, you can find sample grafana dashboards for several Pulsar components.

File: microbench/README.md Copy Chapter

Microbenchmarks for Apache Pulsar

This module contains microbenchmarks for Apache Pulsar.

Running the benchmarks Copy Chapter

The benchmarks are written using JMH. To compile & run the benchmarks, use the following command:

# Compile everything including the shaded microbenchmarks jar
./gradlew :microbench:shadowJar

# run the benchmarks using the standalone shaded jar in any environment
java -jar microbench/build/libs/microbench-*-benchmarks.jar

Running specific benchmarks Copy Section

Display help:

java -jar microbench/build/libs/microbench-*-benchmarks.jar -h

Listing all benchmarks:

java -jar microbench/build/libs/microbench-*-benchmarks.jar -l

Running specific benchmarks:

java -jar microbench/build/libs/microbench-*-benchmarks.jar ".*BenchmarkName.*"

Running specific benchmarks with machine-readable output and saving the output to a file:

java -jar microbench/build/libs/microbench-*-benchmarks.jar -rf json -rff jmh-result-$(date +%s).json ".*BenchmarkName.*" | tee jmh-result-$(date +%s).txt

The jmh-result-*.json file can be used to visualize the results using JMH Visualizer.

Checking what benchmarks match the pattern:

java -jar microbench/build/libs/microbench-*-benchmarks.jar ".*BenchmarkName.*" -lp

Profiling benchmarks with async-profiler:

Set LIBASYNCPROFILER_PATH to the path of the async-profiler library.

Corretto JDK ships with async-profiler (asprof binary and libasyncProfiler dynamic library)

LIBASYNCPROFILER_PATH=$(ls $JAVA_HOME/lib/libasyncProfiler.*)

Alternatively, download async-profiler from https://github.com/async-profiler/async-profiler/releases and install to ~/async-profiler directory.

Mac OS example:

LIBASYNCPROFILER_PATH=$HOME/async-profiler/lib/libasyncProfiler.dylib

Linux example:

# macos example
LIBASYNCPROFILER_PATH=$HOME/async-profiler/lib/libasyncProfiler.dylib

Then run the benchmarks with the -prof argument:

java -jar microbench/build/libs/microbench-*-benchmarks.jar -prof async:libPath=$LIBASYNCPROFILER_PATH\;output=flamegraph\;dir=profile-results ".*BenchmarkName.*"

When profiling on Mac OS, you might need to add \;event=itimer to the -prof argument since it's the only async profiler CPU sampling engine that supports Mac OS. The default value for event is cpu.

It's possible to add options to the async-profiler that aren't supported by the JMH async-profiler plugin. This can be done by adding rawCommand option to the -prof argument. This example shows how to add all (new in Async Profiler 4.1), jfrsync (record JFR events such as garbage collection) and cstack=vmx options.

java -jar microbench/build/libs/microbench-*-benchmarks.jar -prof async:libPath=$LIBASYNCPROFILER_PATH\;output=jfr\;dir=profile-results\;rawCommand=all,jfrsync,cstack=vmx ".*BenchmarkName.*"

File: pip/README.md Copy Chapter

Pulsar Improvement Proposal (PIP)

What is a PIP? Copy Chapter

The PIP is a "Pulsar Improvement Proposal" and it's the mechanism used to propose changes to the Apache Pulsar codebases.

The changes might be in terms of new features, large code refactoring, changes to APIs.

In practical terms, the PIP defines a process in which developers can submit a design doc, receive feedback and get the "go ahead" to execute.

What is the goal of a PIP? Copy Section

There are several goals for the PIP process:

1. Ensure community technical discussion of major changes to the Apache Pulsar codebase.

2. Provide clear and thorough design documentation of the proposed changes. Make sure every Pulsar developer will have enough context to effectively perform a code review of the Pull Requests.

3. Use the PIP document to serve as the baseline on which to create the documentation for the new feature.

4. Have greater scrutiny to changes are affecting the public APIs (as defined below) to reduce chances of introducing breaking changes or APIs that are not expressing an ideal semantic.

It is not a goal for PIP to add undue process or slow-down the development.

When is a PIP required? Copy Section

• Any new feature for Pulsar brokers or client
• Any change to the public APIs (Client APIs, REST APIs, Plugin APIs)
• Any change to the wire protocol APIs
• Any change to the API of Pulsar CLI tools (eg: new options)
• Any change to the semantic of existing functionality, even when current behavior is incorrect.
• Any large code change that will touch multiple components
• Any changes to the metrics (metrics endpoint, topic stats, topics internal stats, broker stats, etc.)
• Any change to the configuration

When is a PIP *not* required? Copy Section

• Bug-fixes
• Simple enhancements that won't affect the APIs or the semantic
• Small documentation changes
• Small website changes
• Build scripts changes (except: a complete rewrite)

Who can create a PIP? Copy Section

Any person willing to contribute to the Apache Pulsar project is welcome to create a PIP.

How does the PIP process work? Copy Chapter

A PIP proposal can be in these states:

1. DRAFT: (Optional) This might be used for contributors to collaborate and to seek feedback on an incomplete version of the proposal.

2. DISCUSSION: The proposal has been submitted to the community for feedback and approval.

3. ACCEPTED: The proposal has been accepted by the Pulsar project.

4. REJECTED: The proposal has not been accepted by the Pulsar project.

5. IMPLEMENTED: The implementation of the proposed changes have been completed and everything has been merged.

6. RELEASED: The proposed changes have been included in an official
   Apache Pulsar release.

The process works in the following way:

1. Fork https://github.com/apache/pulsar repository (Using the fork button on GitHub).
2. Clone the repository, and on it, copy the file pip/TEMPLATE.md and name it pip-xxx.md. The number xxx should be the next sequential number after the last contributed PIP. You view the list of contributed PIPs (at any status) as a list of Pull Requests having a "PIP" label. Use the link here as shortcut.
3. Write the proposal following the section outlined by the template and the explanation for each section in the comment it contains (you can delete the comment once done).
   • If you need diagrams, avoid attaching large files. You can use MermaidJS as simple language to describe many types of diagrams.
4. Create GitHub Pull request (PR). The PR title should be [improve][pip] PIP-xxx: {title}, where the xxx match the number given in previous step (file-name). Replace {title} with a short title to your proposal.
   Validate again that your number does not collide, by step (2) numbering check.
5. The author(s) will email the [email protected] mailing list to kick off a discussion, using subject prefix [DISCUSS] PIP-xxx: {PIP TITLE}. The discussion will happen in broader context either on the mailing list or as general comments on the PR. Many of the discussion items will be on particular aspect of the proposal, hence they should be as comments in the PR to specific lines in the proposal file.
6. Update file with a link to the discussion on the mailing. You can obtain it from Apache Pony Mail.
7. Based on the discussion and feedback, some changes might be applied by authors to the text of the proposal. They will be applied as extra commits, making it easier to track the changes.
8. Once some consensus is reached, there will be a vote to formally approve the proposal. The vote will be held on the [email protected] mailing list, by
   sending a message using subject [VOTE] PIP-xxx: {PIP TITLE}. Make sure to include a link to the PIP PR in the body of the message.
   Make sure to update the PIP with a link to the vote. You can obtain it from Apache Pony Mail.
   Everyone is welcome to vote on the proposal, though only the vote of the PMC members will be considered binding.
   The requirement is to have at least one binding +1 vote from a lazy majority if no binding -1 votes have been cast on the PIP.
   The vote should stay open for at least 48 hours.
9. When the vote is closed, if the outcome is positive, ask a PMC member (using voting thread on mailing list) to merge the PR.
10. If the outcome is negative, please close the PR (with a small comment that the close is a result of a vote).

All the future implementation Pull Requests that will be created, should always reference the PIP-XXX in the commit log message and the PR title.
It is advised to create a master GitHub issue to formulate the execution plan and track its progress.

Example Copy Section

• Eve ran into some issues with the client metrics - she needed a metric which was missing.
• She read the code a bit, and has an idea what metrics she wishes to add.
• She summarized her idea and direction in an email to the DEV mailing list (she located it on
  Discussions section on the website.
• She didn't get any response from the community, so she joined the next
  community meeting. There Matteo Merli and Asaf helped
  setup a channel in Slack to brainstorm the idea and meet on Zoom with a few Pulsar contributors (e.g. Lari and Tison).
• Once Eve had a good enough context, and good design outline, she opened a new branch in her Pulsar repository, duplicated
  TEMPLATE.md and created pip-xxx.MD (the number she will take later).
• She followed the template and submitted the pip as a new PR to pulsar repository.
• Once the PR was created, she modified the version to match the rules described at step 2, both for PR title and file name.
• She sent an email to the DEV mailing list, titled "[DISCUSS] PIP-123: Adding metrics for ..." , described shortly in the
  email what the PIP was about and gave a link.
• She got no response for anyone for 2 weeks, so she nudged the people that helped
  her brainstorm (e.g. Lary and Tison) and pinged in #dev that she needs more reviewers.
• Once she got 3 reviews from PMC members and the community had at least a few days from the moment
  the PR was announceed on DEV, she sent a vote email to the DEV mailing list titled
  "[VOTE] PIP-123: Adding metrics for ...".
• She nudged the reviewers to reply with a binding vote, waited for 2-3 days, and then
  concluded the vote by sending a reply tallying up the binding and non-binding votes.
• She updated the PIP with links to discuss and vote emails, and then asked a PMC member
  who voted +1, to merge (using GitHub mentionon the PR).

List of PIPs Copy Chapter

Current PIPs Table of Contents Copy Section

The following table lists all current PIPs in this directory, sorted by PIP number:

Additional Information Copy Section

1. You can view all PIPs (besides the historical ones) as the list of Pull Requests having title starting with [improve][pip] PIP-. Here is the link for it.
   • Merged PR means the PIP was accepted.
   • Closed PR means the PIP was rejected.
   • Open PR means the PIP was submitted and is in the process of discussion.
2. All PIP files in the pip folder follow the naming convention pip-xxx.md where xxx is the PIP number.

2. Official Technical Reference & Guides (apache/pulsar-site) Copy Chapter

Apache Pulsar Website and Documentation

This repository contains the Apache Pulsar website source code and Apache Pulsar documentation.

Apache Pulsar is an open-source, distributed messaging and streaming platform built for the cloud.

Quick start: Updating documentation and previewing changes Copy Chapter

Make changes to the documentation in the docs directory.

To preview the changes, run the following command (prerequisites):

./preview.sh

This command starts a local web server on port 3000 and opens a browser window pointing to the website.

Updating versioned docs Copy Section

When your documentation changes apply to existing supported versions, you should update both the versioned documentation in the versioned_docs directory and the documentation in the docs directory.

# List all supported major.minor.x versions
./scripts/docs-tool.sh supported_versions

After committing the changes for the docs directory, you can use the docs-tool to apply the changes to the versioned docs. This tool is a wrapper around git diff and patch. If the patch is not applied correctly, you will have to manually apply the changes to the versioned docs.

./scripts/docs-tool.sh apply_changes_to_versioned_docs

Markdown placeholders Copy Chapter

Markdown files under docs/ and versioned_docs/version-*/ are run through
two preprocessors at build time:

• @pulsar:version@, @pulsar:rpm:client@, @pulsar:apidoc:python@, etc. —
  version-aware tokens. The values come from versions.json, site-baseurls.js,
  and data/release-*.js. See src/config/pulsarVariables.ts for the full list.
• {@inject:javadoc:Name:org/...}, {@inject:endpoint|GET|/admin/...} — link
  expansion for Javadoc, GitHub, REST endpoints. See
  src/server/markdownPreprocessors/inject.ts.

To reference the same values from React components, import from
@site/src/config/pulsarVariables — no placeholder is needed in .js/.tsx.

More information Copy Chapter

• Pulsar Website contribution guide
  • How to update data-driven pages on the website
• Pulsar Documentation contribution guide
  • Updating documentation
  • Previewing content

Website not getting updated? Copy Chapter

If the https://pulsar.apache.org website doesn't get updated, check the latest GitHub Actions workflow run for the main branch. If it succeeded, the problem could be that the git history for the asf-site-next branch is too large and needs to be truncated. Here are the commands to perform that task:

git clone --depth 1 -b asf-site-next https://github.com/apache/pulsar-site pulsar-site-static
cd pulsar-site-static
git checkout --orphan asf-site-next-truncated
git add -A
git commit -m "History truncated"
git push -f origin HEAD:asf-site-next

Contact information Copy Chapter

• For documentation improvements, submit an issue to the site repository.
• For Pulsar bug reports or any improvements, submit an issue to the main repository.
• Subscribe to the [email protected] mailing list and start a discussion.
• Ask on the #dev channel on Pulsar Slack (join)