Stream configuration

Streams move and sequence data and messages between components within kdb Insights. kdb Insights includes Reliable Transport (RT) as the primary stream bus. Custom streams can also be used, but they must comply with the RT interface.

Configuration

In kdb Insights Enterprise, all streams use Reliable Transport to move data. In this mode, streams are configured under the spec.elements.sequencer key of the assembly.

User interface configuration

This guide discusses configuration using YAML files. If you are using kdb Insights Enterprise, you can configure your system using the kdb Insights user interface

Sequencer

The sequencer field under elements allows you to optionally define multiple Sequencer instances within the Assembly.

The operator will have defaults set for sequencer at install time, these cover target ports and image details.

Under they key sequencer each Sequencer instance can be defined under its own key, representing the instance name.

spec:
...
  elements:
...
   sequencer:
      north:
        size: 3
        external: true
        externalNodePort: true
        useInternalLBAnnotations: false
        topicConfig:
          subTopic: "data"

key	type	required	description	default	validation
`size`	`integer`	`false`	Size of the StatefulSet to be deployed.	`3`	Limited to `3`
`external`	`boolean`	`true`	External facing Sequencer, setting true enables External IP.	`"false"`
`externalNodePort`	`boolean`	`true`	Use node port type for externally facing Sequencer service.	`"false"`
`useInternalLBAnnotations`	`boolean`	`false`	When enabled will set Service annotations to create an Internal LoadBalancer the external service.	`"true"`
`image`	`object`	`false`	Image details for container.
`env`	`list`	`false`	List of environment variables.
`args`	`string[]`	`false`	Command line arguments to be passed to container.
`topicConfig`	`object`	`false`	Sequencer Topic Configurations See Sequencer Topics Config.
`volume`	`object`	`false`	RT Sequencer directory paths. See RT Volume.
`topicConfigDir`	`string`	`false`	Location of RT 'pull' directory.	`"/config/topics/"`	`^[\/]+[a-zA-Z0-9\/-_]*$`
`volumeMounts`	`list`	`false`	List of standard Kubernetes Volume Mount definitions. Volume must be present in `spec.volumes`.
`k8sPolicy`	`object`	`false`	Kubernetes Pod configurations. See Kubernetes policy for more details.
`archiver`	`object`	`false`	Sequencer Archiver.

Topic config

The topicConfig object allows the topic configuration to be set for a Sequencer. This is the topic subscribed to by the sequencer and published.

spec:
...
  elements:
...
    sequencer:
      south:
        external: false
        subTopic: "north"
      north:
        external: true
        topicConfig:
          subTopic: "data"
          topic: "north"

key	type	required	description	default	validation
`topicPrefix`	`string`	`false`	Prefix to apply to Service name for Sequencer. Prefix followed by a required "-".	`"rt-"`	`^[a-z0-9]+[a-z0-9-]*[a-z0-9]+-+$`
`subTopic`	`string`	`false`	Topic for Sequencer to subscribe to.		`^[a-z0-9]+[a-z0-9-]*[a-z0-9]+$`
`topic`	`string`	`false`	Topic for generated by Sequencer.		`^[a-z0-9]+[a-z0-9-]*[a-z0-9]+$`
`externalTopicPrefix`	`string`	`false`	Prefix to apply to Service name for Sequencer. Prefix followed by a required "-".		`^[a-z0-9]+[a-z0-9-]*[a-z0-9]+-+$`

Sequencer volume

The volume object allows you to configure the Sequencers RT log volume. This is the volume container the sequencer logs for state, subscribing and publishing topics.

spec:
...
  elements:
...
    sequencer:
      south:
        volume:
          mountPath: "/s/"
          subPaths:
            in: "in"
            out: "out"
            cp: "state"
          size: "20Gi"

key	type	required	description	default	validation
`mountPath`	`string`	`false`	Mount location of volume.	`"/s/"`	`^[\/]+[a-zA-Z0-9\/-_]*$`
`accessModes`	`string[]`	`false`	Requested Kubernetes access modes for PVC.
`storageClass`	`string`	`false`	Kubernetes Storage Class.
`size`	`string`	`false`	Kubernetes Storage size request.	`"20Gi"`
`subPaths`	`object`	`false`	Sub directories under Mount location.
`subPaths.in`	`string`	`false`	Location of RT 'in' sub directory.	`"in"`	`^[a-zA-Z0-9-_]+$`
`subPaths.out`	`string`	`false`	Location of RT 'out' sub directory.	`"out"`	`^[a-zA-Z0-9-_]+$`
`subPaths.cp`	`string`	`false`	Location of RT 'cp' sub directory.	`"state"`	`^[a-zA-Z0-9-_]+$`

Archiver

Each Sequencer has the option to enable and Archiver deployment. This Archiver deployment is used for clearing the Sequencers log file, based on log size or age.

spec:
...
  elements:
...
    sequencer:
      south:
        archiver:
          retentionDuration: 10080
          maxLogSize: 5

key	type	required	description	validation
`retentionDuration`	`integer`	`false`	Log retention in minutes
`maxLogSize`	`string`	`false`	Maximum log size	`^([+-]?[0-9.]+)([eEinukmgtpKMGTP][-+]?[0-9])$`
`maxDiskUsagePercent`	`integer`	`false`	Max disk utilization