## 1. Project Overview & Quickstart (IBM/sarama)

## File: README.md

# sarama

[](https://pkg.go.dev/github.com/IBM/sarama)
[](https://securityscorecards.dev/viewer/?uri=github.com/IBM/sarama)
[](https://www.bestpractices.dev/projects/7996)

Sarama is an MIT-licensed Go client library for [Apache Kafka](https://kafka.apache.org/).

## Getting started

- API documentation and examples are available via [pkg.go.dev](https://pkg.go.dev/github.com/IBM/sarama).
- Mocks for testing are available in the [mocks](./mocks) subpackage.
- The [examples](./examples) directory contains more elaborate example applications.
- The [tools](./tools) directory contains command line tools that can be useful for testing, diagnostics, and instrumentation.

You might also want to look at the [Frequently Asked Questions](https://github.com/IBM/sarama/wiki/Frequently-Asked-Questions).

## Compatibility and API stability

Sarama provides a "2 releases + 2 months" compatibility guarantee: we support
the two latest stable releases of Kafka and Go, and we provide a two month
grace period for older releases. However, older releases of Kafka are still likely to work.

Sarama follows semantic versioning and provides API stability via the standard Go
[module version numbering](https://go.dev/doc/modules/version-numbers) scheme.

A changelog is available [here](CHANGELOG.md).

## Contributing

- Get started by checking our [contribution guidelines](https://github.com/IBM/sarama/blob/main/CONTRIBUTING.md).
- Read the [Sarama wiki](https://github.com/IBM/sarama/wiki) for more technical and design details.
- The [Kafka Protocol Specification](https://kafka.apache.org/protocol.html) contains a wealth of useful information.
- For more general issues, there is [a google group](https://groups.google.com/forum/#!forum/kafka-clients) for Kafka client developers.
- If you have any questions, just ask!

---

## File: examples/interceptors/README.md

# Interceptors Example

It creates a *Producer* interceptor to produce some [OpenTelemetry](https://github.com/open-telemetry/opentelemetry-go/) spans and also modifies
the intercepted message to include some headers.

``` go
conf.Producer.Interceptors = []sarama.ProducerInterceptor{xxx}
```

## Run the example
- `go run main.go trace_interceptor.go`.
- or `go build and pass different parameters.
``` sh
go build && ./interceptors --h
Usage of ./interceptors:
  -brokers string
        The Kafka brokers to connect to, as a comma separated list (default "localhost:9092")
  -topic string
        The Kafka topic to use (default "default_topic")
```

App will output OpenTelemetry spans for every intercepted message, i.e:

```
/* Detailed source-code truncated for AI context efficiency. */
```

## Check the produced intercepted messages

Check that messages have some headers added by the interceptor:
``` sh
kafkacat -Cb localhost:9092 -t default_topic -f '\n- %s\nheaders: %h'
```

```
headers: trace_id=235b3424775d8b2f9bf21e458496f447,span_id=50da1552c105e712,message_id=50da1552c105e712
- test message 1/2 from kafka-client-go-test at 2020-08-11T11:36:01-07:00
headers: trace_id=2c4210c1d6c2ebe758eb41cbc95a0478,span_id=046bfc6d6db17ed7,message_id=046bfc6d6db17ed7
- test message 2/2 from kafka-client-go-test at 2020-08-11T11:36:01-07:00
% Reached end of topic default_topic [0] at offset 444
```

---

## File: examples/http_server/README.md

# HTTP server example

This HTTP server example shows you how to use the AsyncProducer and SyncProducer, and how to test them using mocks. The server simply sends the data of the HTTP request's query string to Kafka, and send a 200 result if that succeeds. For every request, it will send an access log entry to Kafka as well in the background.

If you need to know whether a message was successfully sent to the Kafka cluster before you can send your HTTP response, using the `SyncProducer` is probably the simplest way to achieve this. If you don't care, e.g. for the access log, using the `AsyncProducer` will let you fire and forget. You can send the HTTP response, while the message is being produced in the background.

One important thing to note is that both the `SyncProducer` and `AsyncProducer` are **thread-safe**. Go's `http.Server` handles requests concurrently in different goroutines, but you can use a single producer safely. This will actually achieve efficiency gains as the producer will be able to batch messages from concurrent requests together.

---

## File: examples/exactly_once/README.md

# Exactly-Once example

This example shows you how to use the Sarama transactional producer to ensure exacly-once paradigm with Kafka transaction. The example simply starts consuming the given Kafka topics and produce the consumed message to another topic including current message offset in procuder transaction.

```bash
$ go run main.go -brokers="127.0.0.1:9092" -topics="sarama" -destination-topic="destination-sarama" -group="example"
```

To ensure transactional-id uniqueness it implement some ***ProducerProvider*** that build a producer using current message topic-partition.
You can also toggle (pause/resume) the consumption by sending SIGUSR1.

---

## File: examples/consumergroup/README.md

# Consumergroup example

This example shows you how to use the Sarama consumer group consumer. The example simply starts consuming the given Kafka topics and logs the consumed messages.

```bash
$ go run main.go -brokers="127.0.0.1:9092" -topics="sarama" -group="example"
```

You can also toggle (pause/resume) the consumption by sending SIGUSR1

---

## File: examples/sasl_scram_client/README.md

Example command line:

```./sasl_scram_client -brokers localhost:9094 -username foo -passwd a_password -topic topic_name -tls -algorithm [sha256|sha512]```

---

## File: examples/txn_producer/README.md

# Transactional producer example

This example shows you how to use the Sarama transactional producer. The example simply starts some goroutine that produce endlesslly on associated topic.

```bash
$ go run main.go -brokers="127.0.0.1:9092" -topic "sarama" -producers 10 -records-number 10000
```

To ensure transactional-id uniqueness it implement some ***ProducerProvider*** that build a producer appending an integer that grow when producer is created.
You can also see record-rate each 5s printing on stdout.

---

## File: examples/README.md

# Sarama examples

This folder contains example applications to demonstrate the use of Sarama. For code snippet examples on how to use the different types in Sarama, see [Sarama's API documentation on pkg.go.dev](https://pkg.go.dev/github.com/IBM/sarama)

#### HTTP server

[http_server](./http_server) is a simple HTTP server uses both the sync producer to produce data as part of the request handling cycle, as well as the async producer to maintain an access log. It also uses the [mocks subpackage](https://pkg.go.dev/github.com/IBM/sarama/mocks) to test both.

#### Interceptors

Basic example to use a producer interceptor that produces [OpenTelemetry](https://github.com/open-telemetry/opentelemetry-go/) spans and add some headers for each intercepted message.

#### Transactional Producer

[txn_producer](./txn_producer) Basic example to use a transactional producer that produce on some topic within a Kafka transaction. To ensure transactional-id uniqueness it implement some **_ProducerProvider_** that build a producer appending an integer that grow when producer is created.

#### Exacly-once transactional paradigm

[exactly_once](./exactly_once) Basic example to use a transactional producer that produce consumed message from some topics within a Kafka transaction. To ensure transactional-id uniqueness it implement some **_ProducerProvider_** that build a producer using current message topic-partition.

#### Load-aware sticky consumer

[consumer_load_aware](./consumer_load_aware) demonstrates the `SubscriptionUserDataBalanceStrategy` interface: a `LoadAwareSticky` strategy wraps the built-in sticky assignor and injects a fresh load sample (CPU%, in-flight count) into each JoinGroup's subscription metadata.

#### Alter partition reassignments

[alter_partition_reassignments](./alter_partition_reassignments) shows how to raise the replication factor of an existing topic with `ClusterAdmin.AlterPartitionReassignments`, including how to construct the per-partition replica list and how to poll `ListPartitionReassignments` until the move completes.

---

## File: internal/toxiproxy/README.md

# toxiproxy

A minimal client implementation to setup proxies and toxics in toxiproxy as used in the FV suite.
We have our own minimal client implementation to avoid having to pull in the toxiproxy repo which carries a number of transitive dependencies.

---

## File: mocks/README.md

# sarama/mocks

The `mocks` subpackage includes mock implementations that implement the interfaces of the major sarama types.
You can use them to test your sarama applications using dependency injection.

The following mock objects are available:

- [Consumer](https://pkg.go.dev/github.com/IBM/sarama/mocks#Consumer), which will create [PartitionConsumer](https://pkg.go.dev/github.com/IBM/sarama/mocks#PartitionConsumer) mocks.
- [AsyncProducer](https://pkg.go.dev/github.com/IBM/sarama/mocks#AsyncProducer)
- [SyncProducer](https://pkg.go.dev/github.com/IBM/sarama/mocks#SyncProducer)

The mocks allow you to set expectations on them. When you close the mocks, the expectations will be verified,
and the results will be reported to the `*testing.T` object you provided when creating the mock.

## 2. Official Technical Reference & Guides (IBM/IBM.github.io)

# Portal for IBM @ GitHub

This repository is rendered online at [http://ibm.github.io](http://ibm.github.io), it contains a listing of repositories that are open source and maintained by IBM teams.

### Adding a new repo to the listing

In order to have your repository show up at [http://ibm.github.io](http://ibm.github.io), a minor change to [orgs.js](js/orgs.js) is required.

* To add a single repository add a new entry to [orgs.js](js/orgs.js), specify the Github organization name and the repository name (separate them with a `/`), and set the `type` to `repo`, an example can be seen below:

```
  {
      "name": "RuntimeTools/appmetrics",
      "type": "repo"
  }
```

* To add all the repositories in a Github organization add a new entry to [orgs.js](js/orgs.js), specify the Github organization name, and set the `type` to `org`, an example can be seen below:

```
  {
      "name": "IBMResilient",
      "type": "org"
  }
```

### To test changes locally

From within the top level folder of the cloned repository run:

```
$ python -m http.server {port}
```

For example: `python -m http.server 8000` -> Open the following URL in a browser:

```
http://localhost:8000/
```

### Quick Git tutorial

1. Clone the repository and checkout a new branch

```
$ git clone https://github.com/IBM/ibm.github.io
$ git checkout -b branch_name
```

2. Update the files you'd like to change
3. Push the changes upstream

```
$ git add file1 file2
$ git commit -m "add your commit message here"
$ git push origin branch_name
```

4. View your branch in Github and create a Pull Request