sarama Documentation

1. Project Overview & Quickstart (IBM/sarama) Copy Chapter

File: README.md Copy Chapter

sarama

Sarama is an MIT-licensed Go client library for Apache Kafka.

Getting started Copy Chapter

• API documentation and examples are available via pkg.go.dev.
• Mocks for testing are available in the mocks subpackage.
• The examples directory contains more elaborate example applications.
• The tools directory contains command line tools that can be useful for testing, diagnostics, and instrumentation.

You might also want to look at the Frequently Asked Questions.

Compatibility and API stability Copy Chapter

Sarama provides a "2 releases + 2 months" compatibility guarantee: we support
the two latest stable releases of Kafka and Go, and we provide a two month
grace period for older releases. However, older releases of Kafka are still likely to work.

Sarama follows semantic versioning and provides API stability via the standard Go
module version numbering scheme.

A changelog is available here.

Contributing Copy Chapter

• Get started by checking our contribution guidelines.
• Read the Sarama wiki for more technical and design details.
• The Kafka Protocol Specification contains a wealth of useful information.
• For more general issues, there is a google group for Kafka client developers.
• If you have any questions, just ask!

File: examples/interceptors/README.md Copy Chapter

Interceptors Example

It creates a Producer interceptor to produce some OpenTelemetry spans and also modifies
the intercepted message to include some headers.

conf.Producer.Interceptors = []sarama.ProducerInterceptor{xxx}

Run the example Copy Chapter

• go run main.go trace_interceptor.go.
• or `go build and pass different parameters.

go build && ./interceptors --h
Usage of ./interceptors:
  -brokers string
        The Kafka brokers to connect to, as a comma separated list (default "localhost:9092")
  -topic string
        The Kafka topic to use (default "default_topic")

App will output OpenTelemetry spans for every intercepted message, i.e:

/* Detailed source-code truncated for AI context efficiency. */

Check the produced intercepted messages Copy Chapter

Check that messages have some headers added by the interceptor:

kafkacat -Cb localhost:9092 -t default_topic -f '\n- %s\nheaders: %h'

headers: trace_id=235b3424775d8b2f9bf21e458496f447,span_id=50da1552c105e712,message_id=50da1552c105e712
- test message 1/2 from kafka-client-go-test at 2020-08-11T11:36:01-07:00
headers: trace_id=2c4210c1d6c2ebe758eb41cbc95a0478,span_id=046bfc6d6db17ed7,message_id=046bfc6d6db17ed7
- test message 2/2 from kafka-client-go-test at 2020-08-11T11:36:01-07:00
% Reached end of topic default_topic [0] at offset 444

File: examples/http_server/README.md Copy Chapter

HTTP server example

This HTTP server example shows you how to use the AsyncProducer and SyncProducer, and how to test them using mocks. The server simply sends the data of the HTTP request's query string to Kafka, and send a 200 result if that succeeds. For every request, it will send an access log entry to Kafka as well in the background.

If you need to know whether a message was successfully sent to the Kafka cluster before you can send your HTTP response, using the SyncProducer is probably the simplest way to achieve this. If you don't care, e.g. for the access log, using the AsyncProducer will let you fire and forget. You can send the HTTP response, while the message is being produced in the background.

One important thing to note is that both the SyncProducer and AsyncProducer are thread-safe. Go's http.Server handles requests concurrently in different goroutines, but you can use a single producer safely. This will actually achieve efficiency gains as the producer will be able to batch messages from concurrent requests together.

File: examples/exactly_once/README.md Copy Chapter

Exactly-Once example

This example shows you how to use the Sarama transactional producer to ensure exacly-once paradigm with Kafka transaction. The example simply starts consuming the given Kafka topics and produce the consumed message to another topic including current message offset in procuder transaction.

$ go run main.go -brokers="127.0.0.1:9092" -topics="sarama" -destination-topic="destination-sarama" -group="example"

To ensure transactional-id uniqueness it implement some ProducerProvider that build a producer using current message topic-partition.
You can also toggle (pause/resume) the consumption by sending SIGUSR1.

File: examples/consumergroup/README.md Copy Chapter

Consumergroup example

This example shows you how to use the Sarama consumer group consumer. The example simply starts consuming the given Kafka topics and logs the consumed messages.

$ go run main.go -brokers="127.0.0.1:9092" -topics="sarama" -group="example"

You can also toggle (pause/resume) the consumption by sending SIGUSR1

File: examples/sasl_scram_client/README.md Copy Chapter

Example command line:

./sasl_scram_client -brokers localhost:9094 -username foo -passwd a_password -topic topic_name -tls -algorithm [sha256|sha512]

File: examples/txn_producer/README.md Copy Chapter

Transactional producer example

This example shows you how to use the Sarama transactional producer. The example simply starts some goroutine that produce endlesslly on associated topic.

$ go run main.go -brokers="127.0.0.1:9092" -topic "sarama" -producers 10 -records-number 10000

To ensure transactional-id uniqueness it implement some ProducerProvider that build a producer appending an integer that grow when producer is created.
You can also see record-rate each 5s printing on stdout.

File: examples/README.md Copy Chapter

Sarama examples

This folder contains example applications to demonstrate the use of Sarama. For code snippet examples on how to use the different types in Sarama, see Sarama's API documentation on pkg.go.dev

HTTP server

http_server is a simple HTTP server uses both the sync producer to produce data as part of the request handling cycle, as well as the async producer to maintain an access log. It also uses the mocks subpackage to test both.

Interceptors

Basic example to use a producer interceptor that produces OpenTelemetry spans and add some headers for each intercepted message.

Transactional Producer

txn_producer Basic example to use a transactional producer that produce on some topic within a Kafka transaction. To ensure transactional-id uniqueness it implement some ProducerProvider that build a producer appending an integer that grow when producer is created.

Exacly-once transactional paradigm

exactly_once Basic example to use a transactional producer that produce consumed message from some topics within a Kafka transaction. To ensure transactional-id uniqueness it implement some ProducerProvider that build a producer using current message topic-partition.

Load-aware sticky consumer

consumer_load_aware demonstrates the SubscriptionUserDataBalanceStrategy interface: a LoadAwareSticky strategy wraps the built-in sticky assignor and injects a fresh load sample (CPU%, in-flight count) into each JoinGroup's subscription metadata.

Alter partition reassignments

alter_partition_reassignments shows how to raise the replication factor of an existing topic with ClusterAdmin.AlterPartitionReassignments, including how to construct the per-partition replica list and how to poll ListPartitionReassignments until the move completes.

File: internal/toxiproxy/README.md Copy Chapter

toxiproxy

A minimal client implementation to setup proxies and toxics in toxiproxy as used in the FV suite.
We have our own minimal client implementation to avoid having to pull in the toxiproxy repo which carries a number of transitive dependencies.

File: mocks/README.md Copy Chapter

sarama/mocks

The mocks subpackage includes mock implementations that implement the interfaces of the major sarama types.
You can use them to test your sarama applications using dependency injection.

The following mock objects are available:

• Consumer, which will create PartitionConsumer mocks.
• AsyncProducer
• SyncProducer

The mocks allow you to set expectations on them. When you close the mocks, the expectations will be verified,
and the results will be reported to the *testing.T object you provided when creating the mock.

2. Official Technical Reference & Guides (IBM/IBM.github.io) Copy Chapter

Portal for IBM @ GitHub

This repository is rendered online at http://ibm.github.io, it contains a listing of repositories that are open source and maintained by IBM teams.

Adding a new repo to the listing Copy Section

In order to have your repository show up at http://ibm.github.io, a minor change to orgs.js is required.

• To add a single repository add a new entry to orgs.js, specify the Github organization name and the repository name (separate them with a /), and set the type to repo, an example can be seen below:

{
      "name": "RuntimeTools/appmetrics",
      "type": "repo"
  }

• To add all the repositories in a Github organization add a new entry to orgs.js, specify the Github organization name, and set the type to org, an example can be seen below:

{
      "name": "IBMResilient",
      "type": "org"
  }

To test changes locally Copy Section

From within the top level folder of the cloned repository run:

$ python -m http.server {port}

For example: python -m http.server 8000 -> Open the following URL in a browser:

http://localhost:8000/

Quick Git tutorial Copy Section

1. Clone the repository and checkout a new branch

$ git clone https://github.com/IBM/ibm.github.io
$ git checkout -b branch_name

2. Update the files you'd like to change
3. Push the changes upstream

$ git add file1 file2
$ git commit -m "add your commit message here"
$ git push origin branch_name

4. View your branch in Github and create a Pull Request