File: README.md
- Overview
- Where to Start
- Confluent Cloud
- Stream Processing
- Data Pipelines
- Confluent Platform
- Build Your Own
- Additional Demos
Overview
This is a curated list of demos that showcase Apache Kafka® event stream processing on the Confluent Platform, an event stream processing platform that enables you to process, organize, and manage massive amounts of streaming data across cloud, on-prem, and serverless deployments.
Where to start
The best demo to start with is cp-demo which spins up a Kafka event streaming application using ksqlDB for stream processing, with many security features enabled, in an end-to-end streaming ETL pipeline with a source connector pulling from live data and a sink connector connecting to Elasticsearch and Kibana for visualizations.cp-demo also comes with a tutorial and is a great configuration reference for Confluent Platform.
Confluent Cloud
There are many examples from full end-to-end demos that create connectors, streams, and KSQL queries in Confluent Cloud, to resources that help you build your own demos.
You can find the documentation and instructions for all Confluent Cloud demos at https://docs.confluent.io/platform/current/tutorials/examples/ccloud/docs/ccloud-demos-overview.html
| Demo | Local | Docker | Description |
|---|---|---|---|
| Confluent CLI | Y | N | Fully automated demo interacting with your Confluent Cloud cluster using the Confluent CLI |
| Clients in Various Languages to Cloud | Y | N | Client applications, showcasing producers and consumers, in various programming languages connecting to Confluent Cloud |
| Cloud ETL | Y | N | Fully automated cloud ETL solution using Confluent Cloud connectors (AWS Kinesis, Postgres with AWS RDS, GCP GCS, AWS S3, Azure Blob) and fully-managed ksqlDB |
| ccloud-stack | Y | N | Creates a fully-managed stack in Confluent Cloud, including a new environment, service account, Kafka cluster, KSQL app, Schema Registry, and ACLs. The demo also generates a config file for use with client applications. |
| On-Prem Kafka to Cloud | N | Y | Module 2 of Confluent Platform demo (cp-demo) with a playbook for copying data between the on-prem and Confluent Cloud clusters |
| DevOps for Apache Kafka® with Kubernetes and GitOps | N | N | Simulated production environment running a streaming application targeting Apache Kafka on Confluent Cloud using Kubernetes and GitOps |
Stream Processing
| Demo | Local | Docker | Description |
|---|---|---|---|
| Clickstream | N | Y | Automated version of the ksqlDB clickstream demo |
| Kafka Tutorials | Y | Y | Collection of common event streaming use cases, with each tutorial featuring an example scenario and several complete code solutions |
| Microservices ecosystem | N | Y | Microservices orders Demo Application integrated into the Confluent Platform |
Data Pipelines
| Demo | Local | Docker | Description |
|---|---|---|---|
| Clients in Various Languages | Y | N | Client applications, showcasing producers and consumers, in various programming languages |
| Connect and Kafka Streams | Y | N | Demonstrate various ways, with and without Kafka Connect, to get data into Kafka topics and then loaded for use by the Kafka Streams API |
Confluent Platform
| Demo | Local | Docker | Description |
|---|---|---|---|
| Avro | Y | N | Client applications using Avro and Confluent Schema Registry |
| CP Demo | N | Y | Confluent Platform demo (cp-demo) with a playbook for Kafka event streaming ETL deployments |
| Kubernetes | N | Y | Demonstrations of Confluent Platform deployments using the Confluent Operator |
| Multi Datacenter | N | Y | Active-active multi-datacenter design with two instances of Confluent Replicator copying data bidirectionally between the datacenters |
| Multi-Region Clusters | N | Y | Multi-Region clusters (MRC) with follower fetching, observers, and replica placement |
| Quickstart | Y | Y | Automated version of the Confluent Quickstart: for Confluent Platform on local install or Docker, community version, and Confluent Cloud |
| Role-Based Access Control | Y | Y | Role-based Access Control (RBAC) provides granular privileges for users and service accounts |
| Replicator Security | N | Y | Demos of various security configurations supported by Confluent Replicator and examples of how to implement them |
Build Your Own
As a next step, you may want to build your own custom demo or test environment.
We have several resources that launch just the services in Confluent Cloud or on prem, with no pre-configured connectors, data sources, topics, schemas, etc.
Using these as a foundation, you can then add any connectors or applications.
You can find the documentation and instructions for these "build-your-own" resources at https://docs.confluent.io/platform/current/tutorials/build-your-own-demos.html.
Additional Demos
Here are additional GitHub repos that offer an incredible set of nearly a hundred other Apache Kafka demos.
They are not maintained on a per-release basis like the demos in this repo, but they are an invaluable resource.
confluentinc/demo-scene, the most popular demos include:
File: ccloud/beginner-cloud/README.md
Confluent CLI
Overview
You can use Confluent CLI to interact with your Confluent Cloud cluster.
Documentation
You can find the documentation and instructions for running this Confluent CLI example at https://docs.confluent.io/platform/current/tutorials/examples/ccloud/docs/beginner-cloud.html
File: ccloud/ccloud-stack/README.md
Overview
The ccloud-stack utility creates a stack of fully managed services in Confluent Cloud.
It is a quick way to create fully managed resources in Confluent Cloud, which you can then use for learning and building other demos.
Do not use this in a production environment.
The script uses the Confluent CLI to dynamically do the following in Confluent Cloud:
- Create a new environment
- Create a new service account
- Create a new Kafka cluster and associated credentials
- Enable Schema Registry and associated credentials
- Create a new ksqlDB app and associated credentials
- Create ACLs with wildcard for the service account
- Generate a local configuration file with all above connection information, useful for other demos/automation
Documentation
You can find the documentation and instructions for running ccloud-stack at https://docs.confluent.io/platform/current/tutorials/examples/ccloud/docs/ccloud-stack.html
File: ccloud/docs/includes/connect-worker-override-policy.rst
Finally, configure the |kconnect| worker to allow overrides, because |crep| needs to override the default behavior of the |kconnect| worker's embedded producer.
.. sourcecode:: bash
connector.client.config.override.policy=All
File: ccloud/docs/includes/connect-worker-to-destination-ccloud.rst
Set the management topics to replication factor of 3 as required by |ccloud|.
.. literalinclude:: config/connect-ccloud-destination.delta
:lines: 1-4
The |kconnect| worker's admin client requires connection information to the destination |ccloud|.
.. literalinclude:: config/connect-ccloud-destination.delta
:lines: 6-10
The |kconnect| worker's embedded producer requires connection information to the destination |ccloud|.
.. literalinclude:: config/connect-ccloud-destination.delta
:lines: 12-15
If you are using |c3| and doing stream monitoring then the embedded producer's monitoring interceptors require connection information to the destination |ccloud|.
.. literalinclude:: config/connect-ccloud-destination.delta
:lines: 22-25
File: ccloud/docs/includes/connect-worker-to-origin-ccloud.rst
Set the management topics to replication factor of 3 as required by |ccloud|.
.. literalinclude:: config/connect-ccloud-origin.delta
:lines: 1-4
The |kconnect| worker's admin client requires connection information to the origin |ccloud|.
.. literalinclude:: config/connect-ccloud-origin.delta
:lines: 6-10
The |kconnect| worker's embedded producer requires connection information to the origin |ccloud|.
.. literalinclude:: config/connect-ccloud-origin.delta
:lines: 12-15
If you are using |c3| and doing stream monitoring then the embedded producer's monitoring interceptors require connection information to the origin |ccloud|.
.. literalinclude:: config/connect-ccloud-origin.delta
:lines: 22-25
File: ccloud/docs/includes/connect-worker-to-origin-onprem.rst
The |kconnect| worker is backed to the origin on-premises |ak| cluster, so set the replication factor required for the origin on-premises cluster:
.. sourcecode:: bash
replication.factor=
config.storage.replication.factor=
offset.storage.replication.factor=
status.storage.replication.factor=
The origin on-premises |ak| cluster can have a varied set of security features enabled, but for simplicity in this example we show no security configurations, just PLAINTEXT (see this page <https://docs.confluent.io/kafka-connect-replicator/current/configuration_options.html>__ for more |crep| security configuration options).
The |kconnect| worker’s admin client requires connection information to the on-premises cluster.
.. sourcecode:: bash
bootstrap.servers=
File: ccloud/docs/includes/replicator-connect-worker-to-destination.rst
The destination cluster is your |ccloud| cluster, and |crep| needs to know how to connect to it.
Use the prefix dest. to set these configuration parameters.
.. sourcecode:: bash
Confluent Replicator license topic must have replication factor set to 3 for |ccloud|
confluent.topic.replication.factor=3
New user topics that |crep-full| creates must have replication factor set to 3 for |ccloud|
dest.topic.replication.factor=3
If your deployment has |c3| end-to-end streams monitoring setup to gather data in |ccloud|, then you also need to setup the Confluent Monitoring Interceptors to send data to your |ccloud| cluster, which also requires appropriate connection information set for the embedded consumer with the prefix src.consumer.
.. sourcecode:: bash
src.consumer.interceptor.classes=io.confluent.monitoring.clients.interceptor.MonitoringConsumerInterceptor
src.consumer.confluent.monitoring.interceptor.bootstrap.servers=
src.consumer.confluent.monitoring.interceptor.security.protocol=SASL_SSL
src.consumer.confluent.monitoring.interceptor.sasl.mechanism=PLAIN
src.consumer.confluent.monitoring.interceptor.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="