Skip to content

kdb Insights Gateway Microservice

This document provides a high-level overview of the kdb Insights Gateway (GW) microservice, as well a description of the various APIs for interfacing with it.

Query flow

The Gateway microservice is structured as follows, with arrows representing IPC triggered from an API request from a client.

process diagram

Note 1: Processes outside the grey box are not part of the GW microservice, but represent processes that may interact with the GW microservice. Arrows within the grey box are internal messages within the microservice, and arrows crossing the grey box boundary represent external entrypoints for interfacing with the microservice.

Note 2: The GW process is a Java application, whereas other processes are q processes. All communication is done via async q IPC.

Below is a brief description of the message flow to execute an API call. More details (including API documentation) are provided below:

arrow description
1 External client makes request to the Gateway process. Includes a callback.
2 GW requests resources from the RC to service the request.
3 The RC returns a list of DAPs that are available to service the request along with the modified args for each (see .svcRC.getResources for details) and the Aggregator chosen to aggregate the partial responses from the DAP. In the event that the full request cannot be satisfied with the currently available DAPs, portions of the request are queued, and the RC sends DAPs to the GW as they become available (see Arrow 5b).
4 GW sends to DAPs provided allocated to it.
5a DAPs process request and send their partial results to the Aggregator.
5b DAPs send message to the RC indicating that they are available to service another request.
6a Once all partial results are in, the Aggregator aggregates them into a single response and sends it back to the GW.
6b Once the response is sent to the GW, the Aggregator notifies the RC or completion.
7 GW invokes callback on the client, completing the request.

Client interface

Client / GW

The client makes API requests by opening a connection to one of the GW process and asynchronously issuing an API request. Note that the connection must be maintained on the client-side (i.e. hopen) as the GW responds on this connection. APIs are invoked by asynchronously passing a list consisting of the following elements to the GW.

  • apiName - symbol - Name of the API to invoke.
  • args - dictionary - Named arguments to the API. At minimum, these must include:
    • startTS - timestamp - Start time (inclusive).
    • endTS - timestamp - End time (exclusive).
    • Any other assembly taxonomy values.
  • callback - symbol|function - (Name of) function to be invoked on asynchronous return of the query. This is a dyadic function that takes two arguments:
    • header - dictionary - Repsonse header.
    • payload - any - Response payload. Type is dependent on the API invoked.
  • opts - dictionary - Optional parameters, which can augment the default request header. Note that key-value pairs included are guaranteed to return in the response header on completion of the request. Hence this can be used to store request correlators. Note also that this paramater is optional.

Example:

h:open`:gwHost:gwPort;

/ Without optional fields. //! nyi
neg[h](`myAPI;`startTS`endTS`region`arg1`arg2!(-0Wp;0Wp;`amer;1;"abc");`myCallbackFn);

/ With optional header fields.
neg[h](`myAPI;`startTS`endTS`region`arg1`arg2!(-0Wp;0Wp;`amer;1;"abc");`myCallbackFn;`logCorr`appCorr!("request_1";10));
Upon completion, the callback function is invoked with the response header and the response payload.

GW / DAP

In order to integrate with the GW microservice, the DAP must do the following (all functions are documented below):

  • Register with the Resource Coordinator (can be found through discovery). This can be done by opening a connection to the Resource Coordinator (hopen) and invoking .svcRC.registerDAP with its data purview. The purview can be updated by calling .svcRC.updDapStatus.
  • Execute APIs invoked by the client. The GW asynchronously invokes the following function, which must be defined:

    • .svcDA.execute - Executes an API.
      • apiName - symbol - API name.
      • hdr - dictionary - Request header.
      • args - dictionary - Named API arguments.

    In response to this, the DAP must:

    • Send a response header and a response payload to the correct Aggregator by invoking .svcAgg.onPartial on the host/port specified by hdr.agg.
    • Send a message to the Resource Coordinator indicating that it is available to do more work by invoking .svcRC.onPartial on the Resource Coordinator with an appropriate response header.

    Note that the DAP is expected to respond even in the event of an error. Proper error handling should be implemented in order to guarantee this.

APIs

DAP to RC
  • .svcRC.registerDAP - Registers DAP as an API target to service requests.
    • host - symbol - DAP host.
    • port - int - DAP port.
    • avail - boolean - Flag indicating if the DAP is available to service requests or not.
    • purview - dictionary - DAP purview. Contains the following keys:
      • ver - long - Purview version number.
      • startTS - timestamp - Start time covered by DAP (inclusive).
      • endTS - timestamp - End time covered by DAP (exclusive).
      • All other taxonomy values defined in the assembly.
  • .svcRC.updDapStatus - Updates current target status.
    • avail - boolean - Flag indicating if the DAP is available to service requests or not.
    • purview - dictionary - DAP purview. Keys include:

      • ver - long - Purview version number.
      • startTS - timestamp - Start time covered by DAP (inclusive).
      • endTS - timestamp - End time covered by DAP (exclusive).
      • All other taxonomy values defined in the assembly.

      Note that certain subsets of the purview keys are allowed, for efficiency. The allowed combinations are:

      • All the keys specified above.
      • Only ver, startTS, and endTS, if no other purview dimensions have changed.
      • None, if the purview hasn't changed, and we only want to update availability.
      • .svcRC.onPartial - Informs the RC that we are available for more work following an API request.
        • hdr - dictionary - Response header. Must contain, at minimum, the following keys:
      • rc - byte - Response code.
      • ac - byte - Application code.
      • All other keys present in the request header.

      Additionally, the DAP may use this notification to inform the RC that it was unable to send its response to the Aggregator. This is done by including the following key in the response header:

      • sendErr - any - Value is irrelevant.

      If the sendErr key is present in the response header and hdr.rc is non-zero, the RC attempts to notify the Aggregator/GW of the failure in order to issue a response to the client.

DAP to Agg
  • .svcAgg.onPartial - Forwards this DAPs partial results to the Aggregator for aggregation. This function should be executed on the process pointed to by agg in the request header (this request header value is a symbol of the form `:host:port).
    • hdr - dictionary - Response header. Must contain, at minimum, the following keys:
      • rc - short - Response code.
      • ac - short - Application code.
      • All other keys present in the request header.
    • payload - any - Response payload.

The various GW microservice components communicate with each other using the following interfaces.