A stream is the kdb Insights Enterprise deployment of a Reliable Transport (or RT) cluster. Streams come in two forms. Internal streams, which are required to process data between kdb Insights components within a deployment, and external streams, which connect an external data source to the database and provide highly available, high performance messaging.
In the kdb Insights Enterprise all data written to the database goes via a stream. External streams can be connected to a Stream Processor and transformed, or they can be directly connected to the database.
Configuration in YAML
This guide discusses how to configure a stream using the kdb Insights Enterprise user interface. Streams can also be configured using YAML configuration file. YAML configured streams can be deployed using the kdb Insights CLI
Managing internal streams
Internal streams are created automatically when creating a new database. Simply create a new database via the component tree and navigate to the 'Stream Settings' tab.
By default, a stream will be given the same name as the database within which it resides. This can be changed by toggling the Advanced Settings switch; a name field will appear. This name is required to be globally unique across all streams in all databases. This name is restricted to alpha numeric characters and dashes for name based data routing.
Once the stream has been configured, click the "Submit" button to save the settings.
Streams are responsible for moving data between components within kdb Insights. They ensure a deterministic order of events from multiple producers in one unified log of the event sequence. The throughput and latency of the system is dependent on the compute resources allocated to a stream compared to how much data is moving through it. The compute resources for a stream allows you to tune your system for your desired throughput.
|Minimum CPU||The minimum amount of virtual CPU cycles to allocate to this stream. This value is measured in fractions of CPU cores which is either represented as
|Maximum CPU||The limit of CPU available to for this stream to consume. During burst operations, this process may consume more than the limit for a short period of time. This is a best effort governance of the amount of compute used.|
|Minimum Memory||The minimum amount of RAM to reserve for this process. RAM is measured in bytes using multiples of 1024 (the
|Maximum Memory||The maximum amount of RAM to reserve for this process. If the maximum RAM is consumed by this stream, it will be evicted and restarted to free up resources.|
|Stream Size||This is the amount of disk allocated to the stream to hold in-flight data. By default, this will hold
Managing external streams
The External Stream section allows you to create an externally-facing ingress to the RT cluster for ingesting data from outside the kdb Insights cluster using an external publishing client such as a feedhandler. To enable an external stream, click on the Enable External Subscription checkbox.
By default, the name of the external stream will be your database name with a
-e suffix. This will also be the default value for the External Topic. To change either of these values, with the "Enable External Subscription" box checked, turn on the "Advanced Settings" toggle. The two fields should appear and be editable. External streams will use the same computation settings as the internal stream configured above.
|Name||This name is restricted to the DNS characters (alpha numeric and dashes) as it is used for data routing.|
|External Topic||The external topic field is used by external publishers as the