Data Access Interface
The following section describes the interface for DA's communication with either kdb Insights Storage Manager, or a custom equivalent that is responsible for data writedown in an application.
Storage registration
The data access process will invoke .sm.api.register
passing its mount name, a requested timeout (timespan) for acknowledging Reload
signals (null if acknowledgment is not expected), and a callback function. Data Access (DA
) can find a storage manager process either through discovery, looking for a process with a serviceName
of SM
, or by reaching out to the endpoints defined in assembly config at smEndpoints
. Through this connection DA
will receive and process the Reload
signal.
Reload signal
This signal is sent only to DA processes that need to reload their mounted database. For the "stream mount" (RDB-like DA processes) this means purging old table records from memory
- Format:
- Invoked in a DA process (via qIPC) lambda (specified at DA registration) which takes a dictionary:
- common entries for all DA processes:
ts
(timestamp) - time when the migration (e.g. EOI or EOD) processing was started by SMminTS
(timestamp) - inclusive purview start time (for stream mount: 1 ns +maxTS
of the next mount)
- for non-stream mounted DA processes (IDB/HDB)
maxTS
(timestamp) - inclusive purview end time
- for "stream mounted" DA processes (RDB):
startTS
,endTS
(timestamp) - parameters of the corresponding_prntEnd
signalpos
(long) - stream position of the corresponding_prntEnd
signal
- common entries for all DA processes:
- Invoked in a DA process (via qIPC) lambda (specified at DA registration) which takes a dictionary:
If DA
has registered with the storage manager using non-null reload timeout parameter, the callback function specified at registration sends the acknowledgments of Reload
signal processing completion:
{[d]
... / Process the reload signal (load the new DB etc)
neg[.z.w](`.sm.api.reloadComplete;d`ts)
}
DA-SG interface
The following section describes the interface for DA's communication with either kdb Insights Service Gateway, or a custom equivalent that is responsible for managing client query requests in an application.
Resource Coordinator registration
- The data access process will invoke
.sgrc.registerDAP
passing its currentavailability
and itspurview
as arguments. Data Access Processes (DAPs
) can find the resource coordinator process either through discovery, looking for a process with aserviceName
ofKXI-SG-RC
with a metadata keyname
matching the name defined in the DA'srcName
element config. If discovery is not being used, then the DA will reach out to the endpoints defined in assembly config atrcEndpoints
. Through this connectionDA
will send availability and purview updates by invoking.sgrc.updDapStatus
with arguments ofavailability
and its currentpurview
.
On the registration call the resource coordinator can trigger .da.registrationErr
to state whether the registration was successful. On TYPE, MISMATCH, or DOMAIN errors DA will log the error and not retry registering with that process, as this is usually caused by a configuration issue.
API execution
When a gateway invokes an API on a data access process it does so by calling .da.execute
with three arguments.
-
Format:
-
Invoked in a DA process (via qIPC) lambda
.da.reload
which takes a dictionary with keys:api
- Name of API to call.hdr
- Header dictionary.args
- Dictionary of arguments to call API with. Missing arguments will be replaced with(::)
when calling API.
-
Response will be a two element list, where the first element is a response dictionary indicating success or failure of the request, and the second element is the payload from the API itself which can vary depending on the API.
-
SG architecture
Two types of GW architectures are supported, symmetric
and asymmetric
, and they are must be set in the element config of the data access process under sgArch
. This setting controls how the data access process responds on an API request that calls .da.execute
.
If sgArch
is set to symmetric
then the response from .da.execute
will be returned directly to the gateway upon completion, using .z.w
and it will invoke .sgagg.onPartial
, which must be defined in the gateway that DA is sending its payload to.
If sgArch
is set to asymmetric
, then the gateway must set an agg
key in the hdr
argument of .da.execute
defining the endpoint where DA should send the payload. Upon completing execution of the API, DA will send the response header and payload to the process defined in agg
and then call .sgagg.onPartial
, which must be defined on the aggregator.
Custom purviews
User can establish custom purview keys in a process by defining adding to the labels to the assembly. These labels were become associated with the data access process and get pushed anywhere purviews are pushed.
DA-RT interface
Data Access uses the following interface to interact with Reliable Transport service (or a custom tickerplant):
.rt.sub[topic;position]
- subscribe to a topic (string) starting from a given position (long) in the stream.rt.pub[topic]
- register as a publisher for a topic.rt.push[message]
- publish a message (arbitrary type; a custom tickerplant is likely to support a pair (table name;table data)).rt.upd[message;position]
- callback for receiving a message
The subscription topic can be set with the RT_SOURCE
environment variable or the assembly. If using the assembly a stream
bus needs to be set with the appropriate topic. Example below where the topic is called dataStream
:
bus:
stream:
protocol: custom
nodes: tp:5000
topic: dataStream
For a custom tickerplant, this interface needs to be implemented in a client library (q file), which will be loaded at startup from the location specified in KXI_RT_LIB
environment variable. For the basic tickerplant implementation, available at https://github.com/KxSystems/kdb-tick, see the deployment example for the interface code to load into the services.
Low Memory Mode
If a historical Data Access Process' ratio of memory used to memory available (e.g., pod memory limit) exceeds pctMemThreshold
then it may enter a low memory mode, where it will stop ingesting records late-arriving (records with a timestamp within the purview of the historical DAP) until the next EOD event that allows it to clear its cache. During this time it may still return results from APIs such as getData
if the configuration allowPartialResults
is set to true. These responses will have an AC
code of MEMORY
. If allowPartialResults
is set to false, than queries against that DAP will cause the API to fail, until the memory situation is resolved by the next EOD.