## File: README.md * [Overview](#overview) * [Where to Start](#where-to-start) * [Confluent Cloud](#confluent-cloud) * [Stream Processing](#stream-processing) * [Data Pipelines](#data-pipelines) * [Confluent Platform](#confluent-platform) * [Build Your Own](#build-your-own) * [Additional Demos](#additional-demos) # Overview This is a curated list of demos that showcase Apache Kafka® event stream processing on the Confluent Platform, an event stream processing platform that enables you to process, organize, and manage massive amounts of streaming data across cloud, on-prem, and serverless deployments. # Where to start The best demo to start with is [cp-demo](https://github.com/confluentinc/cp-demo) which spins up a Kafka event streaming application using ksqlDB for stream processing, with many security features enabled, in an end-to-end streaming ETL pipeline with a source connector pulling from live data and a sink connector connecting to Elasticsearch and Kibana for visualizations. `cp-demo` also comes with a tutorial and is a great configuration reference for Confluent Platform. # Confluent Cloud There are many examples from full end-to-end demos that create connectors, streams, and KSQL queries in Confluent Cloud, to resources that help you build your own demos. You can find the documentation and instructions for all Confluent Cloud demos at [https://docs.confluent.io/platform/current/tutorials/examples/ccloud/docs/ccloud-demos-overview.html](https://docs.confluent.io/platform/current/tutorials/examples/ccloud/docs/ccloud-demos-overview.html?utm_source=github&utm_medium=demo&utm_campaign=ch.examples_type.community_content.top) | Demo | Local | Docker | Description | ------------------------------------------ | ----- | ------ | --------------------------------------------------------------------------- | [Confluent CLI](ccloud/beginner-cloud/README.md#confluent-cloud-cli) | Y | N | Fully automated demo interacting with your Confluent Cloud cluster using the Confluent CLI | [Clients in Various Languages to Cloud](clients/cloud/README.md) | [Y](clients/cloud/README.md) | N | Client applications, showcasing producers and consumers, in various programming languages connecting to [Confluent Cloud](https://www.confluent.io/confluent-cloud/?utm_source=github&utm_medium=demo&utm_campaign=ch.examples_type.community_content.top) | [Cloud ETL](cloud-etl/README.md) | [Y](cloud-etl/README.md) | N | Fully automated cloud ETL solution using Confluent Cloud connectors (AWS Kinesis, Postgres with AWS RDS, GCP GCS, AWS S3, Azure Blob) and fully-managed ksqlDB | [ccloud-stack](ccloud/ccloud-stack/README.md) | Y | N | Creates a fully-managed stack in Confluent Cloud, including a new environment, service account, Kafka cluster, KSQL app, Schema Registry, and ACLs. The demo also generates a config file for use with client applications. | [On-Prem Kafka to Cloud](https://docs.confluent.io/current/tutorials/cp-demo/docs/index.html?utm_source=github&utm_medium=demo&utm_campaign=ch.examples_type.community_content.top) | N | [Y](https://docs.confluent.io/current/tutorials/cp-demo/docs/index.html?utm_source=github&utm_medium=demo&utm_campaign=ch.examples_type.community_content.top) | Module 2 of [Confluent Platform demo](https://docs.confluent.io/current/tutorials/cp-demo/docs/index.html?utm_source=github&utm_medium=demo&utm_campaign=ch.examples_type.community_content.top) (`cp-demo`) with a playbook for copying data between the on-prem and Confluent Cloud clusters | [DevOps for Apache Kafka® with Kubernetes and GitOps](https://docs.confluent.io/platform/current/tutorials/streaming-ops/index.html) | N | N | Simulated production environment running a streaming application targeting Apache Kafka on Confluent Cloud using Kubernetes and GitOps # Stream Processing | Demo | Local | Docker | Description | ------------------------------------------ | ----- | ------ | --------------------------------------------------------------------------- | [Clickstream](clickstream/README.md) | N | [Y](https://docs.confluent.io/platform/current/tutorials/examples/clickstream/docs/index.html#clickstream-demo?utm_source=github&utm_medium=demo&utm_campaign=ch.examples_type.community_content.top) | Automated version of the [ksqlDB clickstream demo](https://docs.confluent.io/platform/current/tutorials/examples/clickstream/docs/index.html#clickstream-demo?utm_source=github&utm_medium=demo&utm_campaign=ch.examples_type.community_content.top) | [Kafka Tutorials](https://kafka-tutorials.confluent.io?utm_source=github&utm_medium=demo&utm_campaign=ch.examples_type.community_content.top) | [Y](https://kafka-tutorials.confluent.io?utm_source=github&utm_medium=demo&utm_campaign=ch.examples_type.community_content.top) | [Y](https://kafka-tutorials.confluent.io?utm_source=github&utm_medium=demo&utm_campaign=ch.examples_type.community_content.top) | Collection of common event streaming use cases, with each tutorial featuring an example scenario and several complete code solutions | [Microservices ecosystem](microservices-orders/README.md) | N | [Y](microservices-orders/README.md) | [Microservices orders Demo Application](https://github.com/confluentinc/kafka-streams-examples/tree/5.2.2-post/src/main/java/io/confluent/examples/streams/microservices) integrated into the Confluent Platform # Data Pipelines | Demo | Local | Docker | Description | ------------------------------------------ | ----- | ------ | --------------------------------------------------------------------------- | [Clients in Various Languages](clients/cloud/README.md) | [Y](clients/cloud/README.md) | N | Client applications, showcasing producers and consumers, in various programming languages | [Connect and Kafka Streams](connect-streams-pipeline/README.md) | [Y](connect-streams-pipeline/README.md) | N | Demonstrate various ways, with and without Kafka Connect, to get data into Kafka topics and then loaded for use by the Kafka Streams API # Confluent Platform | Demo | Local | Docker | Description | ------------------------------------------ | ----- | ------ | --------------------------------------------------------------------------- | [Avro](clients/README.md) | [Y](clients/README.md) | N | Client applications using Avro and Confluent Schema Registry | [CP Demo](https://github.com/confluentinc/cp-demo) | N | [Y](https://github.com/confluentinc/cp-demo) | [Confluent Platform demo](https://docs.confluent.io/platform/current/tutorials/cp-demo/docs/index.html?utm_source=github&utm_medium=demo&utm_campaign=ch.examples_type.community_content.top) (`cp-demo`) with a playbook for Kafka event streaming ETL deployments | [Kubernetes](kubernetes/README.md) | N | [Y](kubernetes/README.md) | Demonstrations of Confluent Platform deployments using the [Confluent Operator](https://docs.confluent.io/operator/current/overview.html?utm_source=github&utm_medium=demo&utm_campaign=ch.examples_type.community_content.top) | [Multi Datacenter](multi-datacenter/README.md) | N | [Y](multi-datacenter/README.md) | Active-active multi-datacenter design with two instances of Confluent Replicator copying data bidirectionally between the datacenters | [Multi-Region Clusters](multiregion/README.md) | N | [Y](multiregion/README.md) | Multi-Region clusters (MRC) with follower fetching, observers, and replica placement | [Quickstart](cp-quickstart/README.md) | [Y](cp-quickstart/README.md) | [Y](cp-quickstart/README.md) | Automated version of the [Confluent Quickstart](https://docs.confluent.io/platform/current/quickstart/index.html?utm_source=github&utm_medium=demo&utm_campaign=ch.examples_type.community_content.top): for Confluent Platform on local install or Docker, community version, and Confluent Cloud | [Role-Based Access Control](security/rbac/README.md) | Y | Y | Role-based Access Control (RBAC) provides granular privileges for users and service accounts | [Replicator Security](replicator-security/README.md) | N | [Y](replicator-security/README.md) | Demos of various security configurations supported by Confluent Replicator and examples of how to implement them # Build Your Own As a next step, you may want to build your own custom demo or test environment. We have several resources that launch just the services in Confluent Cloud or on prem, with no pre-configured connectors, data sources, topics, schemas, etc. Using these as a foundation, you can then add any connectors or applications. You can find the documentation and instructions for these "build-your-own" resources at [https://docs.confluent.io/platform/current/tutorials/build-your-own-demos.html](https://docs.confluent.io/platform/current/tutorials/build-your-own-demos.html?utm_source=github&utm_medium=demo&utm_campaign=ch.examples_type.community_content.top). # Additional Demos Here are additional GitHub repos that offer an incredible set of nearly a hundred other Apache Kafka demos. They are not maintained on a per-release basis like the demos in this repo, but they are an invaluable resource. * [Learn: Apache Kafka Demos and Examples](https://developer.confluent.io/demos-examples/) * [confluentinc/demo-scene](https://github.com/confluentinc/demo-scene/blob/master/README.md), the most popular demos include: * [Workshop: Apache Kafka and ksqlDB in Action: Let’s Build a Streaming Data Pipeline!](https://github.com/confluentinc/demo-scene/tree/master/build-a-streaming-pipeline/workshop) * [Introduction to ksqlDB](https://github.com/confluentinc/demo-scene/tree/master/introduction-to-ksqldb) * [Kafka Connect Zero to Hero](https://github.com/confluentinc/demo-scene/tree/master/kafka-connect-zero-to-hero) * [vdesabou/kafka-docker-playground](https://github.com/vdesabou/kafka-docker-playground/blob/master/README.md) --- ## File: ccloud/beginner-cloud/README.md # Confluent CLI ## Overview You can use [Confluent CLI](https://docs.confluent.io/confluent-cli/current/install.html) to interact with your [Confluent Cloud](https://www.confluent.io/confluent-cloud/) cluster. # Documentation You can find the documentation and instructions for running this Confluent CLI example at [https://docs.confluent.io/platform/current/tutorials/examples/ccloud/docs/beginner-cloud.html](https://docs.confluent.io/platform/current/tutorials/examples/ccloud/docs/beginner-cloud.html?utm_source=github&utm_medium=demo&utm_campaign=ch.examples_type.community_content.beginner-cloud) --- ## File: ccloud/ccloud-stack/README.md # Overview The `ccloud-stack` utility creates a stack of fully managed services in Confluent Cloud. It is a quick way to create fully managed resources in Confluent Cloud, which you can then use for learning and building other demos. Do not use this in a production environment. The script uses the Confluent CLI to dynamically do the following in Confluent Cloud: * Create a new environment * Create a new service account * Create a new Kafka cluster and associated credentials * Enable Schema Registry and associated credentials * Create a new ksqlDB app and associated credentials * Create ACLs with wildcard for the service account * Generate a local configuration file with all above connection information, useful for other demos/automation # Documentation You can find the documentation and instructions for running `ccloud-stack` at [https://docs.confluent.io/platform/current/tutorials/examples/ccloud/docs/ccloud-stack.html](https://docs.confluent.io/platform/current/tutorials/examples/ccloud/docs/ccloud-stack.html?utm_source=github&utm_medium=demo&utm_campaign=ch.examples_type.community_content.clients-ccloud) --- ## File: ccloud/docs/includes/connect-worker-override-policy.rst Finally, configure the |kconnect| worker to allow overrides, because |crep| needs to override the default behavior of the |kconnect| worker's embedded producer. .. sourcecode:: bash connector.client.config.override.policy=All --- ## File: ccloud/docs/includes/connect-worker-to-destination-ccloud.rst Set the management topics to replication factor of 3 as required by |ccloud|. .. literalinclude:: config/connect-ccloud-destination.delta :lines: 1-4 The |kconnect| worker's admin client requires connection information to the destination |ccloud|. .. literalinclude:: config/connect-ccloud-destination.delta :lines: 6-10 The |kconnect| worker's embedded producer requires connection information to the destination |ccloud|. .. literalinclude:: config/connect-ccloud-destination.delta :lines: 12-15 If you are using |c3| and doing stream monitoring then the embedded producer's monitoring interceptors require connection information to the destination |ccloud|. .. literalinclude:: config/connect-ccloud-destination.delta :lines: 22-25 --- ## File: ccloud/docs/includes/connect-worker-to-origin-ccloud.rst Set the management topics to replication factor of 3 as required by |ccloud|. .. literalinclude:: config/connect-ccloud-origin.delta :lines: 1-4 The |kconnect| worker's admin client requires connection information to the origin |ccloud|. .. literalinclude:: config/connect-ccloud-origin.delta :lines: 6-10 The |kconnect| worker's embedded producer requires connection information to the origin |ccloud|. .. literalinclude:: config/connect-ccloud-origin.delta :lines: 12-15 If you are using |c3| and doing stream monitoring then the embedded producer's monitoring interceptors require connection information to the origin |ccloud|. .. literalinclude:: config/connect-ccloud-origin.delta :lines: 22-25 --- ## File: ccloud/docs/includes/connect-worker-to-origin-onprem.rst The |kconnect| worker is backed to the origin on-premises |ak| cluster, so set the replication factor required for the origin on-premises cluster: .. sourcecode:: bash replication.factor= config.storage.replication.factor= offset.storage.replication.factor= status.storage.replication.factor= The origin on-premises |ak| cluster can have a varied set of security features enabled, but for simplicity in this example we show no security configurations, just PLAINTEXT (see `this page `__ for more |crep| security configuration options). The |kconnect| worker’s admin client requires connection information to the on-premises cluster. .. sourcecode:: bash bootstrap.servers= --- ## File: ccloud/docs/includes/replicator-connect-worker-to-destination.rst The destination cluster is your |ccloud| cluster, and |crep| needs to know how to connect to it. Use the prefix ``dest.`` to set these configuration parameters. .. sourcecode:: bash # Confluent Replicator license topic must have replication factor set to 3 for |ccloud| confluent.topic.replication.factor=3 # New user topics that |crep-full| creates must have replication factor set to 3 for |ccloud| dest.topic.replication.factor=3 If your deployment has |c3| end-to-end streams monitoring setup to gather data in |ccloud|, then you also need to setup the Confluent Monitoring Interceptors to send data to your |ccloud| cluster, which also requires appropriate connection information set for the embedded consumer with the prefix ``src.consumer``. .. sourcecode:: bash src.consumer.interceptor.classes=io.confluent.monitoring.clients.interceptor.MonitoringConsumerInterceptor src.consumer.confluent.monitoring.interceptor.bootstrap.servers= src.consumer.confluent.monitoring.interceptor.security.protocol=SASL_SSL src.consumer.confluent.monitoring.interceptor.sasl.mechanism=PLAIN src.consumer.confluent.monitoring.interceptor.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="" password="";