Skip to content

Key processes

This page describes the key KX Sensors processes, grouped by function, and how they interact.

Data ingestion and storage

  • SDL (Sensor Data Loader): Ingests sensor data and performs any data enrichment, transformation, or adjustment to master data that may be required at the point of ingestion.
  • MDL (Master Data Loader): Updates master data in response to SDL activity or mutating transactions received via GW (Gateway).
  • SM (Storage Manager): Subscribes to data from SDL and persists it at scheduled intervals within a day (EOI) and at the end of day (EOD).
  • DBW (Database Writer): Moves intraday data from memory to a single on-disk partition, triggered either by SM or by RDB in response to memory pressure. DBW performs both end-of-interval and end-of-day write-downs. For more detail, see DBW/Z and EOX processing.

Data that DBW writes down moves through three storage tiers, from newest to oldest:

  • RDB (Real-Time Database): Provides access to recent sensor data held in memory, which may reflect sensor activity for a period ranging from minutes to days depending on configuration. Data older than the configured age is eligible for purging, and any data remaining in RDB is purged at end of day.
  • IDB (Intraday Database): Only required if the daily volume of sensor data is too large to fit in RDB's memory (for example, the manufacturing vertical). Provides access to intraday data that's been purged from RDB due to a recurring interval or memory-pressure write-down, after it's written to disk but before it's persisted to HDB at end of day.
  • HDB (Historical Database): Provides access to partitioned sensor data that's been persisted to disk as part of an end-of-day event, by mounting the partitioned on-disk database.

Query processing

  • GW (Gateway): Determines the data services required to fulfill an asynchronous client query or API — any combination of RDB, IDB, HDB, SDL, or MDL — and directs the query or API to a free RP for execution.
  • RP (Response Process): Sends the client query or API to the data services recruited by GW, aggregates the result, and returns it to the requesting client.
  • SGW (Synchronous Gateway): Handles synchronous API requests, mainly for the CDL (Calculation Data Loader). Lets a client string together sequences of APIs and save the results.

Messaging

EMS (Enterprise Messaging System) is the general term for the KX Sensors messaging layer. Two implementations are available: RT for multi-node deployments, and TP+ for single-node deployments.

  • RT (Reliable Transport): Manages the streaming of messages using the Raft consensus algorithm. Its main components are replicators, sequencers, and an archiver.
  • RTO (RT Orchestrator): Responsible for starting and retrieving monitoring and health information for each RT cluster.
  • MQ (Message Queue): Provides a reliable asynchronous messaging service that performs message routing and delivery to KXS clients.

Discovery and coordination

  • DISC (Discovery): Receives registration information as processes start in a KX Sensors instance and disseminates that information so processes can discover each other. Discovery uses the open-source etcd as a high-availability data store.
  • etcd: An open-source, distributed, consistent key-value store for shared configuration, service discovery, and scheduler coordination across distributed systems or clusters of machines.
  • qetcd: Serves as a proxy between q processes and etcd, since q processes can't communicate with etcd directly. One instance of qetcd runs on every node.

Monitoring

  • Mon (Monitoring): Collects real-time statistics about system operations and performance for core processes such as SDL, DBW, EMS, and RDB.
  • MonRDB: Aggregates data from Mon for performance monitoring queries.

Next steps