Skip to content

Service Gateway Introduction

kdb Insights Service Gateway (SG) provides a unified access point for data that is spread across multiple processes, hereto referred to as Data Access Processes (DAPs). DAPs can be user-defined q processes or processes within the kdb Insights Data Access microservice.

The Service Gateway component supports users who wish to invoke a publicly exposed API, by providing the protocols and interfaces to compose and issue the request, and by overseeing all activities related to the fulfilment of the request until a response can be presented to the caller (even if indicating that request was unable to be fulfilled). This component also supports users who wish to offer such a service, by providing similar interfaces, protocols and configuration definitions to make this service available to other components in the system.

Data in the system is dispersed across many different locations. The set of processes with access to particular data is located at some arbitrarily distant positions in the system architecture. Over time, that set of processes will change, since processes will come and go, and data will drift through various storage tiers. Similarly, the ability to perform certain activities may not be defined (exclusively) by locale of data, but also by the locale of the runtime logic required to perform the activity.

The Service Gateway component and associated elements abstracts away this complexity by defining concepts and interfaces by which a service provider can describe the functional and data access requirements which must be met in order to satisfy the request, including mechanisms to allow the details and context of a particular request instance to contribute to the expression of these requirements.

At a high level, the Service Gateway component is responsible for:

  • Accepting and validating service requests
  • Identifying the locale for service execution based on metadata discernable from a combination of previously known configuration information, dynamic system topology information, and data contained in the request instance itself (notably its arguments and potentially other aspects of accompanying out-of-band information)
  • Selecting one or more specific processes to execute the request
  • Shepherding the request to and triggering execution on the selected target process(es)
  • Triggering any aggregation of partial results from multiple execution processes where appropriate
  • Shepherding results back to original caller

Architecture

The Service Gateway (SG) microservice provides a unified access point for data that is spread across multiple processes, hereto referred to as Data Access Processes (DAPs). DAPs can be user-defined q processes or processes within the kdb Insights Data Access microservice. Data may be distributed across DAPs temporally (e.g. RDB, IDB, HDB) or along user-defined labels (e.g. region, asset class) or a combination thereof. The SG provides a portal for making data retrieval API calls, routes to the appropriate DAPs, aggregates the results, and returns to the client process that issued the request. The process architecture is as follows (note: only arrows representing communication with external processes are shown):

process diagram

process description
Client Client process (external). Connects to Gateway (GW) process to issue API calls.
GW Gateway (internal). Receives API requests from clients, forwards to appropriate DAPs, returns results.
RC Resource Coordinator (internal). Makes routing decisions based on DAP data purviews and availability.
Agg Aggregator (internal). Aggregates responses from DAPs.
DAP Data Access Process (external). Has access to data.

Unless otherwise specified, the arrows in the diagram represent asynchronous q IPC communication between processes. Query flow is as follows.

arrow description
1 Client makes API request to a GW. This can be sync or async.
2 GW sends request to the appropriate DAPs.
3a Each DAP, upon completion, returns response to the Aggregator designated for the request.
3b Each DAP, upon completion, notifies the RC that it is available to service requests.
4 GW sends response back to client.

See the "Interface" page for detailed information on interfacing with the SG components.

Routing

One of the key features of the SG is its routing capabilities. It uses the DAPs data purviews (see "interface") to route requests exactly where it needs to. While explained in greater detail in the "interface" page, a DAP's purview is a description of the data that it provides (both temporally and along user-defined "labels"). Every request specifies a temporal range and the labels of interest. The SG partitions the request across DAPs that can satisfy it, aggregates the results from each, and returns the consolidated response to the client that made the call.

On the Aggregator

With the exception of the pre-configured APIs (see "API" page), the only supported aggregation function is raze. This will be expanded upon in future versions.