DBW/Z (Database Writer) and EOX processing¶
This page describes the data ingestion path leading into DBW, and how DBW/Z handles EOI and EOD processing.
Data ingestion path¶
The following example shows the single use case of sensor readings data being published from an external client to SDL before being processed by the DBs and DBW.
DBW/Z and EOX processing¶
Database Writer is responsible for moving intraday data from memory to one or more on-disk locations, as well as for performing the end-of-interval (EOI) and end-of-day (EOD) archival to disk for the IDB and HDB. It's designed both to ensure that no data is lost and to minimize the recovery latency for any process requiring real-time data.
EOI is triggered either periodically by SM or in response to memory pressure in RDB. Each EOI write-down generates one IDB partition per interval. Configure the interval with the eoiFreqMins parameter in systemParams.yaml. By default, SM signals a periodic EOI at the interval boundary. To delay the signal by a fixed amount of time after the interval boundary, set eoiOffset in systemParams.yaml. On service startup, SM completes its end-of-interval procedure before publishing an EOI signal. DBW then creates new start-of-interval snapshots for all tables.
| Parameter | Description | Default |
|---|---|---|
eodInterval |
During this one- or two-second interval just before EOD, KX Sensors publishes all records as delta records; DBW later re-interprets these delta records as base records if possible. The purpose of this interval is to avoid a potential race condition: SDL publishes a record as a base table record, but SM initiates its EOD processing before that record reaches the Transit Portal. Note! If you deactivate this option (that is, set the value to zero), certain delta records could be published as base table records. |
0D00:00:01 |
shardName |
Template for naming database shards. Table sharding is designed to improve write-down (EOD/EOI) performance. | {TABLE}{SHARD}, for example kxsReading0, kxsReading1, kxsReading2, etc. |
eodMaxCollapseKeys |
The maximum number of unique keys for a partitioned collapsed table to process at one time during the end-of-day process. | 100000 |
eodPeachLevel |
The level at which to apply parallelism when moving data from IDB to the appropriate HDB partition during the end-of-day process. If set to table, tables are processed in parallel; if set to part (partition), tables are processed sequentially while their partitions are processed in parallel.Note! The part option is only recommended when your schema contains a single very large table that receives data for multiple days during any 24-hour period. |
table |