Skip to content

Interface

There are two types of processes that can interact with the SG: client processes making API calls, and data access processes (DAPs) servicing requests (e.g. RDB/IDB/HDB).

Client

To make an API call to the SG from a q process, do the following: 1. hopen a connection to a GW process. 2. Issue an API call by passing a 4-element list with the following elements:

  • apiName - symbol - API name.
  • args - dictionary - Named arguments to the API. At minimum, these must include:
    • startTS - timestamp - Start time (inclusive).
    • endTS - timestamp - End time (exclusive).
    • Any other assembly label values.
  • callback -symbol - Dyadic callback function to be invoked on request completion in the async case (see below).
  • opts - dictionary - Optional out-of-band information to accompany the request (see "header" page). This MUST be an untyped dictionary. For no opts, use (0#`)!().

API requests are done synchronously or asynchronously. Upon completion of the request, the GW uses the hopened handle to invoke the callback function with the following arguments:

  • hdr - dictionary - Response header (see "header" page).
  • payload - any - Response payload.

Example:

h:hopen`:gwHost:gwPort;
myCallbackFn:{[hdr;payload] ...};

/ Async -- `myCallBackFn` will be invoked.
neg[h](`myAPI;`startTS`endTS`region`assetClass`arg1`arg2!(-0Wp;0Wp;`amer;`equity;1;"abc");`myCallbackFn;`logCorr`appCorr!("request_1";10));

/ Sync -- response is a two-element list: (hdr;payload).
res:h(`myAPI;`startTS`endTS`region`assetClass`arg1`arg2!(-0Wp;0Wp;`amer;`equity;1;"abc");`myCallbackFn;`logCorr`appCorr!("request_1";10));
res 0 / Header
res 1 / Payload

Moreover, note that, whether the API itself uses the argument, all API calls must include startTS, endTS, and all labels in the SG's assembly definition. You may choose to target one or multiple labels and the SG will route the request to DAPs covering all specified labels (the Cartesian product of them). For example. if the labels of our assembly were:

labels:
  region: amer,emea,apac
  assetClass: equity,futures

All API calls need to include startTS, endTS, region and asssetClass. Example:

/ Multiple regions, one asset class.
args:`startTS`endTS`region`assetClass`other`args!(-0Wp;0Wp;`amer`emea;`futures;1;0b)

The SG will route the request to DAPs covering `amer`futures and `emea`futures.

Discovery

If your client process uses KX Insights Discovery, Gateway processes can be found via discovery. The following sample code will select all active GWs:

sn:`$"KXI-SG-GW" / GW service name
select from .com_kx_sd.getServices[]where serviceName=sn,status=`UP

REST

Optionally, API requests for the pre-configured APIs (see "API" page) can be made via the REST interface if it is configured to run (see "Configuration") and the DAPs implement the APIs in question (i.e. you are using the KX Insights Data Access microservcie, or your own custom DAPs with your implementations of the APIs -- see "API" page for details).

DAP

In order to integrate with the GW microservice, a DAP must do the following (all functions are documented below and a sample example is provided):

  • Register with the Resource Coordinator. This can be done by opening a connection to the Resource Coordinator (hopen) and invoking .sgrc.registerDAP with its data purview (see Purview)). The purview can be updated by calling .sgrc.updDapStatus.
  • Execute APIs invoked by the client. The GW asynchronously invokes the following function on the DAP, which must be defined:
  • .da.execute - Executes an API.
    • apiName - symbol - API name.
    • hdr - dictionary - Request header (see "header" page).
    • args - dictionary - Named API arguments.

In response to this, the DAP must:

  • Send a response header and a response payload to the correct Aggregator by invoking .sgagg.onPartial on the host/port specified by hdr.agg.
  • Send a message to the Resource Coordinator indicating that it is available to do more work by invoking .sgrc.onPartial on the Resource Coordinator with an appropriate response header.

Note

The DAP is expected to respond even in the event of an error. Proper error handling should be implemented in order to guarantee this.

If using the KX Insights Data Access microservice, all of the above is automatically implemented; the DA only needs to be configured with its purview and the name of the SG that it should connect to (see "Data Access" configuration).

Purview

The purview describes the portion of the data each individual DAP offers and it is the means by which the SG makes its routing decisions. The purview is a dictionay with the following keys:

  • ver - long - Purview version number (used for keeping track of purview changes, not routing)
  • startTS - timestamp - Start time covered by DAP (inclusive).
  • endTS - timestamp - End time covered by DAP (exclusive).
  • All other label values defined in the assembly (symbols).

DAPs register and update their purviews with the RC using the APIs described below. Every request must include all the purview keys (except ver). The RC chooses DAPs that service the request and splits up the arguments as necessary. For example, suppose our SG assembly labels were:

labels:
  city: toronto,montreal,vancouver
  sensorType: gas,electric

Then every request must include startTS and endTS and one or multiple of each city and sensorType. Likewise, every DAP must register itself with startTS and endTS, and one city and one sensorType. Suppose the DAPs registered with the following purviews (ver omitted):

dap city sensorType startTS endTS
dap1 toronto gas -0Wp 0Wp
dap2 toronto electric -0Wp 0Wp
dap3 montreal gas -0Wp 2021.06.01D
dap3 montreal gas 2021.05.01D 0Wp
dap4 montreal electric -0Wp 0Wp
dap5 vancouver gas -0Wp 0Wp
dap6 vancouver electric -0Wp 0Wp

If a request came in for `startTS`endTS`city`sensorType!(2021.05.10D;2021.06.15D;`toronto`montreal;`gas), the RC would split up the request as send to the DAPs as follows:

dap args
dap1 `startTS`endTS`city`sensorType!(2021.05.10D;2021.06.15D;`toronto;`gas)
dap3 `startTS`endTS`city`sensorType!(2021.05.10D;2021.06.01D;`montreal;`gas)
dap4 `startTS`endTS`city`sensorType!(2021.06.01D;2021.06.15D;`montreal;`gas)

Things to note:

  • Any additional arguments (not related to purview) are sent unchanged to all DAPs participating in the request.
  • The SG will not send the same portion of a request to two different DAPs if they overlap.
  • Once the SG enlists a DAP to service request, it is marked unavailable until it reports that it has completed (.sgrc.onPartial). If no DAPs are available to service all or part of a request, the SG sends out what it can, and enqueues the portions that can't be completed until a DAP registers or updates itself as being able to satisfy (wholly or partially) the outstanding portion of the request.

On the Aggregator

With the exception of the pre-configured APIs (see "API" page), the only supported aggregation function is raze. This will be expanded upon in future versions. Thus, all API results returned to the aggregator should be "raze-able" (i.e. raze(res 0; res 1;...)).

Discovery

The DAP can use discovery to find the RC it should report to. If the DAP uses discovery, the following sample code can be used to discover the RC:

name:`mySG; / "name" in SG assembly definition
sn:`$"KXI-SG-RC"; / RC service name
select from .com_kx_sd.getServices[]where serviceName=sg,status=`UP,name=metadata@\:`gwa

Note that the uniqueness of the RC process guarantees that there should only be one result from this select.

APIs

Below are the processes a DAP can connect to and the APIs that it can call.

  • Resource Coordinator
    • .sgrc.registerDAP - Registers DAP as an API target to service requests.
      • host - symbol - DAP host.
      • port - int - DAP port.
      • avail - boolean - Flag indicating if the DAP is available to service requests or not.
      • purview - dictionary - See Purview.
    • .sgrc.updDapStatus - Updates current target status.
      • avail - boolean - Flag indicating if the DAP is available to service requests or not.
      • purview - dictionary - See Purview.
      • Note that certain subsets of the purview keys are allowed, for efficiency. The allowed combinations are:
    • All the keys specified above.
    • Only ver, startTS, and endTS, if no other purview dimensions have changed.
    • None, if the purview hasn't changed, and we only want to update availability.
    • .sgrc.onPartial - Informs the RC that we are available for more work following an API request.
      • hdr - dictionary - Response header (See "header" page). Must contain, at minimum, the following keys:
        • rc - byte - Response code.
        • ac - byte - Application code.
        • All other keys present in the request header.

Additionally, the DAP may use this notification to inform the RC that it was unable to send its response to the Aggregator. This is done by including the following key in the response header: - sendErr - any - Value is irrelevant.

If the sendErr key is present in the response header and hdr.rc is non-zero, the RC attempts to notify the Aggregator/GW of the failure in order to issue a response to the client.

  • Aggregator
    • .sgagg.onPartial - Forwards this DAPs partial results to the Aggregator for aggregation. This function should be executed on the process pointed to by agg in the request header (this request header value is a symbol of the form `:host:port).
      • hdr - dictionary - Response header (see "header" page). Must contain, at minimum, the following keys:
        • rc - short - Response code.
        • ac - short - Application code.
        • All other keys present in the request header.
      • payload - any - Response payload.

Example

Suppose the SG assembly file was

name: mySG
labels:
  region: amer,emea,apac
  assetClass: equity,futures
elements:
  rc:
    host: rcHost
    port: 1234

The following is a very simple example of a DAP implementation.

//
// Purview - a DAP covers one possible combination of the SG's labels, and some temporal
// interval. For example, this could be an RDB that covers all of today.
//
purview:`ver`region`assetClass`startTS`endTS!(1;`amer;`equity;"p"$.z.D;0Wp)

rc:hopen`:rcHost:1234 / Connect to the RC (alternatively, RC can be found via discovery)
neg[rc](`.sgrc.registerDAP;.z.h;system"p";1b;purview) / Register with the RC


//
// Purview can be updated. For example, at the start of an EOD, a DAP may wish to
// announce itself as unavailable until the EOD ends. Upon EOD completion, it can
// roll its date (and its version number) forward and announce itself as available
// again.
//
onEodStart:{[]
    neg[rc](`.sgrc.updDapStatus;0b;()!());
    }

onEodEnd:{[]
    purview[`ver]+:1; / Roll our version number forward
    purview[`startTS]+:1D; / Roll the date forward
    neg[rc](`.sgrc.updDapStatus;1b;`ver`startTS#purview)
    }

//
// API execution. This function gets invoked by the GW on an API request. The DAP is
// obligated to respond to the Agg with its result, and the RC with an update that
// it available for more work. It should do this even in the event of an error.
//
// Note the name and signature of the function are important.
//
.da.execute:{[apiName;hdr;args]
    //
    // The hdr includes, among other things, `pvVer`, which is the version number
    // the RC used to make its selection. If the number is out of date, we should
    // NOT proceed as normal, since we may no longer cover the portion of the
    // request that the SG needs from us. We should error or trigger a retry
    // (Note: automatic retries NYI).
    //
    res:$[purview[`ver]=hdr`pvVer;
        @[apiName;args;`execErr]; / Run the API as normal
        `pvErr]; / Purview versions don't match

    //
    // The Agg and RC expect a response header with, at minimum:
    //  - the keys that, and
    //  - response and application codes (where 0 represents OK/no errors).
    //
    // Note: Error mapping in the Agg NYI.
    //
    respHdr:hdr,`rc`ac!
        $[res~`execErr;
            `rc`ac!10 10h; / General error code
        res~`pvErr;
            `rc`ac!13 30h; / Version mismatch error
        / OK
            `rc`ac!0 0h]; / OK

    //
    // Send response to the Agg. It is good practice to check whether the send was
    // successful. In the event that the send failed, we can notify the RC who
    // can complete the request on our behalf.
    //
    // Note, the Agg we are to report to is in hdr`agg.
    //
    sent:.[;(hdr`agg;respHdr;agg);0b]{
        h:hopen x; / Open connection to agg
        neg[h](`.sgagg.onPartial;y;z); / Send response
        hclose h; / Alternatively, cache this for future responses
        1b / Success
        }

    //
    // Notify the RC of completion so that it can select us for more work.
    // If the send to the Agg failed, we can use this opportunity to inform
    // the RC of the failure with the `sendErr` key in the response header.
    //
    if[not sent;respHdr[`sendErr]:1b]; / Include sendErr (note: value is arbitrary)
    neg[rc](`.sgrc.onPartial;respHdr); / Notify RC of completion
    }

//
// Implementation of APIs.
//
...