Skip to content

Late Data Queries

This page explains how to handle late-arriving data in kdb Insights, including setup tips and common issues to avoid.

The kdb Insights database supports the persisting and querying of late and out of order data. This is done through configuration and coordination between the Storage Manager and the Data Access Process.

Core concepts

Late data

Timeseries data is considered late if the timestamp in the partition column prtnCol for the table is less than the current time of the system. For example, if at 12:00pm we ingest timeseries data with a partition column time of 11:45am, that data is 15 minutes late.

Out of order data

Out of order data refers to receiving data in a non-monotonically increasing order. For example, if we received three timeseries data updates with timestamps of 11:00am, 10:00am, and 12:00pm respectively, that data is out of order.

Temporal purview

Temporal purview refers to the period of time that a DAP is responsible for servicing queries for. This is represented as a range of timestamps that is inclusive on the start time, and exclusive on the end time.

See purviews.

Base table

For partitioned data the "main" data is located in the main table within the global namespace. For an RDB this is an in-memory table, but for local DAPs the partitioned table in global namespace is an on-disk table which was written down by the Storage Manager.

In-memory table

For local DAPs (IDB and HDB), in addition to the on-disk base table, there is an in-memory table which stores updates and new data, within the temporal purview, which have not yet been written to disk by an EOX event (EOI for IDB, and EOD for HDB). This is used to ensure queries to this DAP are able to fully satisfy the request with the most up to date information. The in-memory table can always be accessed with .da.getTableMem.

In-memory delta table

Local DAPs (IDB and HDB) have a second in-memory table. It is used to store in-purview updates that occur between the start and end of an EOX event. This is because those updates can be queried but cannot be written to disk during the current EOX event. This table can be accessed with .da.getTableDelta.

.kxi.selectTable

A UDA helper function provided by the data access process to abstract the location of data from the query. It is a function which understands the data access configuration and data model and is able to correctly interact with base, in-memory, and in-memory delta tables when running a query.

_prtnEnd

Signal from Storage Manager that an EOX event has started. Received via the data stream. Details of the signal are used to determine if the event is an EOI or an EOD.

_reload

Signal received from Storage Manager via IPC that an EOX event has finished. Details of the event are used to determine if the event was an EOI or an EOD.

Configuring late data

Timeseries data is considered late if the timestamp in the prtnCol partition column is earlier than the current system time. For example, if timeseries data is ingested at 12:00 PM with a partition column time of 11:45 AM, that data is 15 minutes late.

By default, kdb Insights Enterprise turns on late data for packages.

Disable on specific databases

Late data handling adds overheads. When late data is enabled, the IDB and HDB need extra RAM as they must store in-purview data in memory until the next EOX event. If your data is never late and memory is limited, this memory and processing overhead is unnecessary and there is no benefit to enabling late data as the code path and behavior is unchanged if disabled.

The Storage Manager and all corresponding DAPs need to have the same late data configuration. When a DAP connects to the Storage Manager, it checks both have the same late data setting.

Disabling late data

If you choose to disable late data where it was previously enabled, be aware that some data may still be held in HDB DAP memory. Since the delete API only targets on-disk data, you should perform a manual EOD operation to flush all in-memory data to disk before disabling late data.

Disable late data across all databases

Update the values.yaml as follows to ensure all databases have the KXI_LATE_DATA environment variable applied to all Storage Manager and DAP processes:

kxi-operator:
  config:
    sm:
      env:
        - name: KXI_LATE_DATA
          value: "false"
    dap:
      env:
        - name: KXI_LATE_DATA
          value: "false"

Note

Set KXI_LATE_DATA to "false" to disable it; set to "true" or omit the variable to enable it.

Contact your administrator, and refer to Applying configuration changes, for details on how to add to the values.yaml file.

Note

Setting KXI_LATE_DATA takes precedence over Setting it in a package.

Inject at deploy time

If deploying a database via the CLI, you can disable late data as follows:

kxi pm deploy mypackage --env dap:KXI_LATE_DATA=false

Note

Setting KXI_LATE_DATA takes precedence over manually editing a package.

Edit the package

If you only want late data to be disabled in specific databases you can edit the package's shard YAML file, setting lateData: false on the Storage Manager and all DAP instances:

sm:
  lateData: false
daps:
  instances:
    rdb:
      lateData: false
      mountName: RDB
    idb:
      lateData: false
      mountName: IDB
    hdb:
      lateData: false
      mountName: HDB

Object storage tier

If you are using an object storage tier, the Storage Manager does not support writedown of late data to it, making late data effectively unqueryable after the next EOD for that tier. In this case, do not enable late data as this can cause issues or data gaps.

Data flow

When late data is enabled, all DAPs subscribe to the stream, and filter timeseries data based on their current or expected purview and store it in memory. DAPs continue ingesting and storing relevant records in memory until the receipt of the _prtnEnd signal. When this is received the DAPs react differently depending on their dapType. An RDB DAP does not do anything special when it receives the _prtnEnd signal.

For local DAPs like the IDB and HDB, they first determine what kind of signal the _prtnEnd is for, either end of interval or end of day. If the signal is relevant to the DAP, it extends the data it filters for and direct updates to the in-memory delta table.

On the _reload signal, the local DAP purges its in-memory table and moves any updates from the delta table into the in-memory table. Its new purview is reported to the kdb Insights Resource Coordinator and it continues filtering for data within its purview.

Object Tier

The kdb Insights Storage Manager does not currently support ingestion and persistence of late data that would hit the object tier."

Back to top