Skip to content

Streams

A Stream is kdb Insights Enterprise deployment of the Reliable Transport (or RT) microservice. It ensures the reliable streaming of messages into and within kdb Insights Enterprise. It is designed to satisfy both high availability and high performance requirements.

Messages are sent to the stream by a publisher and are replicated, merged and sequenced before being passed to any consumers, for example the Stream Processor and the Storage Manager.

In kdb Insights Enterprise all data written to the database goes via a Stream which the Stream Processor publishes to and the Storage Manager subscribes to. This helps to ensure no data is lost as part of the writedown process.

When using the build a database wizard a stream is automatically created for the writedown process and you can choose to create a second stream for any external publishers that use the kdb Insights SDKs to publish data into kdb Insights Enterprise.

See here for more details on RT in general.

Video Tutorial

Set up

  1. Give the Stream a name.

  2. Define the Sub Topic for the external publisher to subscribe too. This represents the stream id as defined here

  3. Optionally update resource requirements for the stream.

  4. Complete optional advanced properties.

  5. Submit.

Screenshot

item description
Stream Name Name of the stream used in the UI.
Sub Topic This is the stream id for an external publisher to subscribe to. For more details on how to publish data to a Stream see here.
Externally Facing (Advanced) Check to enable if streaming data is to accessible to kdb Insights Enterprise components through RT.

Resourcing

CPU

item description
Minimum CPU Minimum CPU required - 1 is for one core, 0.5 is half time for one core - must be greater than 0.1; defaults to 0.1.
Maximum CPU Maximum CPU available and must be less than 12. Process is throttled when maximum CPU usage is reached; defaults to 1.

Memory Size

item description
Minimum Memory Minimum memory to allocate to the stream and always available, must be greater than 1MB; defaults to 128 MB
Maximum Memory Maximum memory to allocate to the stream; once reached, an out-of-memory error will return, must be less than 20,480MB; defaults to 512 MB.

Volume Size

Define the data volume size of the stream; limit how much data the stream can store; defaults to 20Gi.