Skip to content

Stream configuration

Streams move and sequence data and messages between components within kdb Insights. kdb Insights includes Reliable Transport (RT) as the primary stream bus. Custom streams can also be used, but they must comply with the RT interface.

Configuration

In kdb Insights Enterprise, all streams use Reliable Transport to move data. In this mode, streams are configured under the spec.elements.sequencer key of the assembly.

User interface configuration

This guide discusses configuration using YAML files. If you are using kdb Insights Enterprise, you can configure your system using the kdb Insights user interface

Sequencer

The sequencer field under elements allows you to optionally define multiple Sequencer instances within the Assembly.

The operator will have defaults set for sequencer at install time, these cover target ports and image details.

Under they key sequencer each Sequencer instance can be defined under its own key, representing the instance name.

spec:
...
  elements:
...
   sequencer:
      north:
        size: 3
        external: true
        externalNodePort: true
        useInternalLBAnnotations: false
        topicConfig:
          subTopic: "data"
key type required description default validation
size integer false Size of the StatefulSet to be deployed. 3 Limited to 3
external boolean true External facing Sequencer, setting true enables External IP. "false"
externalNodePort boolean true Use node port type for externally facing Sequencer service. "false"
useInternalLBAnnotations boolean false When enabled will set Service annotations to create an Internal LoadBalancer the external service. "true"
image object false Image details for container.
env list false List of environment variables.
args string[] false Command line arguments to be passed to container.
topicConfig object false Sequencer Topic Configurations See Sequencer Topics Config.
volume object false RT Sequencer directory paths. See RT Volume.
topicConfigDir string false Location of RT 'pull' directory. "/config/topics/" ^[\/]+[a-zA-Z0-9\/-_]*$
volumeMounts list false List of standard Kubernetes Volume Mount definitions. Volume must be present in spec.volumes.
k8sPolicy object false Kubernetes Pod configurations.
archiver object false Sequencer Archiver.

Topic config

The topicConfig object allows the topic configuration to be set for a Sequencer. This is the topic subscribed to by the sequencer and published.

spec:
...
  elements:
...
    sequencer:
      south:
        external: false
        subTopic: "north"
      north:
        external: true
        topicConfig:
          subTopic: "data"
          topic: "north"
key type required description default validation
topicPrefix string false Prefix to apply to Service name for Sequencer. Prefix followed by a required "-". "rt-" ^[a-z0-9]+[a-z0-9-]*[a-z0-9]+-+$
subTopic string false Topic for Sequencer to subscribe to. ^[a-z0-9]+[a-z0-9-]*[a-z0-9]+$
topic string false Topic for generated by Sequencer. ^[a-z0-9]+[a-z0-9-]*[a-z0-9]+$
externalTopicPrefix string false Prefix to apply to Service name for Sequencer. Prefix followed by a required "-". ^[a-z0-9]+[a-z0-9-]*[a-z0-9]+-+$

Sequencer volume

The volume object allows you to configure the Sequencers RT log volume. This is the volume container the sequencer logs for state, subscribing and publishing topics.

spec:
...
  elements:
...
    sequencer:
      south:
        volume:
          mountPath: "/s/"
          subPaths:
            in: "in"
            out: "out"
            cp: "state"
          size: "20Gi"
key type required description default validation
mountPath string false Mount location of volume. "/s/" ^[\/]+[a-zA-Z0-9\/-_]*$
accessModes string[] false Requested Kubernetes access modes for PVC.
storageClass string false Kubernetes Storage Class.
size string false Kubernetes Storage size request. "20Gi"
subPaths object false Sub directories under Mount location.
subPaths.in string false Location of RT 'in' sub directory. "in" ^[a-zA-Z0-9-_]+$
subPaths.out string false Location of RT 'out' sub directory. "out" ^[a-zA-Z0-9-_]+$
subPaths.cp string false Location of RT 'cp' sub directory. "state" ^[a-zA-Z0-9-_]+$

Archiver

Each Sequencer has the option to enable and Archiver deployment. This Archiver deployment is used for clearing the Sequencers log file, based on log size or age.

spec:
...
  elements:
...
    sequencer:
      south:
        archiver:
          retentionDuration: 10080
          maxLogSize: 5
key type required description default validation
retentionDuration integer false Log retention in minutes
maxLogSize string false Maximum log size ^([+-]?[0-9.]+)([eEinukmgtpKMGTP]*[-+]?[0-9]*)$
maxDiskUsagePercent integer false Max disk utilization