Skip to content

Data Access Interface

The following section describes the interface for DA's communication with either kdb Insights Storage Manager, or a custom equivalent that is responsible for data writedown in an application.

Storage registration

The data access process will invoke .sm.api.register passing its mount name, a requested timeout (timespan) for acknowledging Reload signals (null if acknowledgment is not expected), and a callback function. Data Access (DA) can find a storage manager process either through discovery, looking for a process with a serviceName of SM, or by reaching out to the endpoints defined in assembly config at smEndpoints. Through this connection DA will receive and process the Reload signal.

Reload signal

This signal is sent only to DA processes that need to reload their mounted database. For the "stream mount" (RDB-like DA processes) this means purging old table records from memory

  • Format:
    • Invoked in a DA process (via qIPC) lambda (specified at DA registration) which takes a dictionary:
      • common entries for all DA processes:
        • ts (timestamp) - time when the migration (e.g. EOI or EOD) processing was started by SM
        • minTS (timestamp) - inclusive purview start time (for stream mount: 1 ns + maxTS of the next mount)
      • for non-stream mounted DA processes (IDB/HDB)
        • maxTS (timestamp) - inclusive purview end time
      • for "stream mounted" DA processes (RDB):
        • startTS, endTS (timestamp) - parameters of the corresponding _prntEnd signal
        • pos(long) - stream position of the corresponding _prntEnd signal

If DA has registered with the storage manager using non-null reload timeout parameter, the callback function specified at registration sends the acknowledgments of Reload signal processing completion:

{[d]
    ... / Process the reload signal (load the new DB etc)
    neg[.z.w](`.sm.api.reloadComplete;d`ts)
    }

DA-SG interface

The following section describes the interface for DA's communication with either kdb Insights Service Gateway, or a custom equivalent that is responsible for managing client query requests in an application.

Resource Coordinator registration

  • The data access process will invoke .sgrc.registerDAP passing its current availability and its purview as arguments. Data Access Processes (DAPs) can find the resource coordinator process either through discovery, looking for a process with a serviceName of KXI-SG-RC with a metadata key name matching the name defined in the DA's rcName element config. If discovery is not being used, then the DA will reach out to the endpoints defined in assembly config at rcEndpoints. Through this connection DA will send availability and purview updates by invoking .sgrc.updDapStatus with arguments of availability and its current purview.

On the registration call the resource coordinator can trigger .da.registrationErr to state whether the registration was successful. On TYPE, MISMATCH, or DOMAIN errors DA will log the error and not retry registering with that process, as this is usually caused by a configuration issue.

API execution

When a gateway invokes an API on a data access process it does so by calling .da.execute with three arguments.

  • Format:

    • Invoked in a DA process (via qIPC) lambda .da.reload which takes a dictionary with keys:

      • api - Name of API to call.
      • hdr - Header dictionary.
      • args - Dictionary of arguments to call API with. Missing arguments will be replaced with (::) when calling API.

    • Response will be a two element list, where the first element is a response dictionary indicating success or failure of the request, and the second element is the payload from the API itself which can vary depending on the API.

SG architecture

Two types of GW architectures are supported, symmetric and asymmetric, and they are must be set in the element config of the data access process under sgArch. This setting controls how the data access process responds on an API request that calls .da.execute.

If sgArch is set to symmetric then the response from .da.execute will be returned directly to the gateway upon completion, using .z.w and it will invoke .sgagg.onPartial, which must be defined in the gateway that DA is sending its payload to.

If sgArch is set to asymmetric, then the gateway must set an agg key in the hdr argument of .da.execute defining the endpoint where DA should send the payload. Upon completing execution of the API, DA will send the response header and payload to the process defined in agg and then call .sgagg.onPartial, which must be defined on the aggregator.

Custom purviews

User can establish custom purview keys in a process by defining adding to the labels to the assembly. These labels were become associated with the data access process and get pushed anywhere purviews are pushed.

DA-RT interface

Data Access uses the following interface to interact with Reliable Transport service (or a custom tickerplant):

  • .rt.sub[topic;position] - subscribe to a topic (string) starting from a given position (long) in the stream
  • .rt.pub[topic] - register as a publisher for a topic
  • .rt.push[message] - publish a message (arbitrary type; a custom tickerplant is likely to support a pair (table name;table data))
  • .rt.upd[message;position] - callback for receiving a message

The subscription topic can be set with the RT_SOURCE environment variable or the assembly. If using the assembly a stream bus needs to be set with the appropriate topic. Example below where the topic is called dataStream:

bus:
  stream:
    protocol: custom
    nodes: tp:5000
    topic: dataStream

For a custom tickerplant, this interface needs to be implemented in a client library (q file), which will be loaded at startup from the location specified in KXI_RT_LIB environment variable. For the basic tickerplant implementation, available at https://github.com/KxSystems/kdb-tick, see the deployment example for the interface code to load into the services.

Low Memory Mode

If a historical Data Access Process' ratio of memory used to memory available (e.g., pod memory limit) exceeds pctMemThreshold then it may enter a low memory mode, where it will stop ingesting records late-arriving (records with a timestamp within the purview of the historical DAP) until the next EOD event that allows it to clear its cache. During this time it may still return results from APIs such as getData if the configuration allowPartialResults is set to true. These responses will have an AC code of MEMORY. If allowPartialResults is set to false, than queries against that DAP will cause the API to fail, until the memory situation is resolved by the next EOD.