Skip to content

Introduction

The KX Insights data access services handle the functions of accessing the data efficiently through routing and APIs. This is done through two microservices which can be used together or independently, the Service Gateway (SG) and the Data Access Processes (DAPs).

Data Access Processes provide read-only access to all data stored in a kdb+ database through a REST and q API, a distributed SQL API, and a QSQL API. DAPs react to data control messages (typically sent by the KX Insights Storage Manager in order to synchronize temporal purviews (semantic labels and the range of timeseries data in available in a given DAP) in order to distribute timeseries queries across multiple DAPs.

The Service Gateway provides a unified access point for data that is spread across multiple DAPs, providing request queueing, query routing and response aggregation capabilities.

At a high level, the Service Gateway component is responsible for:

  • Accepting and validating service requests
  • Identifying the locale for service execution based on metadata discernable from a combination of previously known configuration information, dynamic system topology information, and data contained in the request instance itself (such as its arguments and potentially other aspects of accompanying out-of-band information)
  • Selecting one or more DAPs to execute the request
  • Shepherding the request to and triggering execution on the selected DAPs
  • Triggering any aggregation of partial results from multiple DAPs
  • Shepherding results back to original caller

Architecture

A high-level architecture of the relation between the Service Gateway and the Data Access Processes is depicted below, which also introduces the three internal services that make up the Service Gateway: the Gateway (GW), Resource Coordinator (RC), and Aggregtor (Agg).

process description
Client Client process (external). Connects to Gateway (GW) process to issue API calls.
GW Gateway (internal). Receives API requests from clients, forwards to appropriate DAPs, returns results.
RC Resource Coordinator (internal). Makes routing decisions based on DAP data purviews and availability.
Agg Aggregator (internal). Aggregates responses from DAPs.
DAP Data Access Process (external). Has access to data.

process diagram

Unless otherwise specified, the arrows in the diagram represent asynchronous q IPC communication between processes. Query flow is as follows.

arrow description
1 Client makes API request to a GW. This can be sync or async.
2 GW sends request to the appropriate DAPs.
3a Each DAP, upon completion, returns response to the Aggregator designated for the request.
3b Each DAP, upon completion, notifies the RC that it is available to service requests.
4 GW sends response back to client.

See the Service Gateway Interface for detailed information on interfacing with the SG service.

Query routing

One of the key features of the SG is its routing capabilities. It uses its configuration and the DAPs data purviews to route requests exactly where it needs to. Every request specifies a temporal range and the labels of interest. The SG partitions the request across DAPs that can satisfy it, aggregates the results from each, and returns the consolidated response to the client that made the call.