Kafka Ingestion Example
Streams data from Kafka and writes to a data sink
Kafka organizes streaming data into topics which are subdivided by numeric independent partitions. This example walks through creating a simple Java Kafka producer that publishes data into a Kafka broker to be consumed by the kdb Insights Stream Processor.
Use TLS in production
It is recommended for production deployments that Kafka be configured with TLS encryption. For a guide on setting up TLS with Kafka and the KX Stream Processor, refer to the Kafka with TLS guide.
This example is broken down into multiple parts for setting up, running, and adding TLS.
- For information on setting up a simple Kafka broker cluster, see the setup guide.
- To see a pipeline configured to ingest from, see the ingestion example.
- To learn more about creating and configuring TLS certificates, see the TLS setup guide.
- To enable TLS ingestion, see ingesting with TLS.