Skip to content

Service Gateway Configuration

In its most basic form, the SG is a set of Docker images that are combined using minimal configuration. Below is an explanation of the images required, what configuration parameters need to be defined, and an some example configurations.

Images

There is one image per process type in the SG. The SG architecture allows for multiple of any process. The table below summarizes the process types

process description image
GW Receives requests and responds to requests. kxi-sg-gw
RC Routes requests to DAPs. kxi-sg-rc
Agg Aggregates responses from DAPs. kxi-sg-agg

In addition, the SG can optionally use KX Insights Discovery in order for processes to discover and connect with each other seamlessly (see "Discovery"). Images required are as follows.

process description image
sidecar Discovery sidecar (for the RC). kxi_sidecar
discovery Discovery client. Configure one, which all processes seamlessly connect to. kxi-eureka-discovery
proxy Discovery proxy. discovery_proxy

Environment variables

The GW microservice relies on certain environment variables to be defined in the containers. The variables are described below.

variable required containers description
KXI_NAME Yes RC, Agg Unique Process name.
KXI_PORT Yes RC, Agg Port.
KXI_LOG_FORMAT No RC, Agg, sidecar Message format (see "Logging" page).
KXI_LOG_DEST No RC, Agg, sidecar Endpoints (see "Logging" page).
KXI_LOG_LEVELS No all Component routing (see "Logging" page).
DISCOVERY_PROXY No GW Discovery proxy address (not required if not using discovery).
GATEWAY_QIPC_PORT Yes GW Gateway port for QIPC traffic.
GATEWAY_HTTP_PORT Yes GW Gateway port for HTTP traffic.
KXI_CONFIG_FILE Yes sidecar Discovery configuration file (see "Discovery" page).
KXI_CUSTOM_FILE No Agg File containing custom code to load into Agg process.
KXI_SG_TIMEOUT No RC, Agg Default request timeout in milliseconds (default is 30,000).
KXI_SG_DISC No RC Multi-RC discovery mode (see Multi-RC.
KXI_SG_RC_ADDR Yes GW, Agg host:port of RC process to connect to.
KXI_SG_CONN_TIMEOUT No RC, Agg Timeout on connection open.
KDB_LICENSE_B64 Yes all KDB licence.
KXI_SG_BUFFER_INITIAL No GW Initial size in bytes to allocate per connected aggregator. Default 2MB
KXI_SG_BUFFER_RETAIN No GW Maximum size in bytes to hold on to per connected aggregator. Default 2MB
KXI_AUTH_DISABLED No GW Set KXI_AUTH_DISABLED=0 to use the gateway with Keycloak.
KXI_OIDC_JSON_PATH No GW OIDC JSON used for mapping Keycloak authentication endpoints.
KXI_SG_MAX_RETRY No RC Maximum number of retries the RC is willing to do for retryable errors.
KXI_ALLOWED_SBX_APIS No RC Comma-delimited list of sandbox APIs to allow in non-sandbox RCs (ex: ".kxi.sql,.kxi.qsql").

See example section below.

Multi RC

Multiple RCs can work together spread request load. The means by which RCs discover each other is specified by the KXI_SG_DISC environment variable. Currently supported discovery methods are

  • kubernetes

    RCs discover each other via the Kubernetes control plane (this discovery method only works if using Kubernetes). This mode allows for dynamic scalability, but at requires polling the control plane to discover new RCs as they come up. The following annotations must be added to the pod's metadata in order for RC containers to properly identify each other:

    kind: Pod
metadata:
annotations:
    kxi-kdisc: "${rc_container_name}: sc=KXI-SG-RC port=${rc_port}"

    where rc_container_name is the .spec.containers.name of the RC container and rc_port is the RC containers port (this should match the KXI_PORT environment variable).

    Note also that the RC pod(s)'s Service Account need the necessary privileges to call the Kubernetes control plane API. This can be achieved by running the following command:

    kubectl create rolebinding default-view --clusterrole=view --serviceaccount=$NAMESPACE:default --namespace=$NAMESPACE

    where $NAMESPACE is your cluster's namespace.

Kubernetes secrets

To setup image pull secrets, and licence secrets:

Create a new secret named kx-repo-access.

kubectl create secret docker-registry kx-repo-access \
    --docker-username=<username> \
    --docker-password=<password> \
    --docker-server=registry.dl.kx.com

Create and apply new secret named kx-license-info.

$ cat kx-license-secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: kx-license-info
type: Opaque
stringData:
  license: <base64 license string>

$ kubectl apply -f kx-license-secret.yaml

Custom file

The default aggregation for any API is raze. However, the Agg processes will load the q file pointed to by the KXI_CUSTOM_FILE environment variable. In this file, you can: * define/register custom aggregation functions, * define which aggregation functions to invoke for which APIs, * define metadata pertaining to the aggregation function (which is subsequently query-able using the .kxi.getMeta API, see API page for details).

Note that while SG only supports loading a single file, you can load other files from within this file using \l (allowing you to control load order). The current working directory (pwd) at load time is the base directory of the file.

This can be combined with the Data Access microservice (which allows custom function definitions in the DAPs) to create full custom API support within KX Insights (see "Data Access" for details).

Note: It's recommended to avoid .sg* namespaces to avoid colliding with SG functions.

Aggregation functions can be anything. Their input is a list of the output of the corresponding API defined in the DAPs, and its output can be anything.

To define which aggregation function to use for an API, use the .sgagg.registerAggFn API, whose signature is as follows. * aggFn - symbol - Aggregation function name. * metadata - list|string|dictionary - Aggregation function metadata (see "SAPI - Metadata" documentation). * apis - symbol|symbol[] - API(s) for which this should be the default aggregation function.

Aggregation function MUST be registered with .sgagg.registerAggFn in order to be invoke-able by the Aggregator. See Custom file example below for an example.

Example

Below is a sample configuration in two flavours: docker-compose and Kubernetes. Note: variables ${...} are user-defined and based on your local directory structure/file names. Sections/lines marked Optional are optional. Note in particular:

  • ${registry}: Image registry from which you download images.
  • ${*_version}: Version of the image. Format is #.#.#.

Docker-compose

docker-compose example

Kubernetes

Kubernetes example

RC discovery sidecar

Config file (see "Discovery" for more details):

{
    "connection": ":sgrc:5050",
    "frequencySecs": 5,
    "discovery":
    {
        "registry": ":proxy:4000",
        "adaptor": "discEurekaAdaptor.q",
        "heartbeatSecs": 30,
        "leaseExpirySecs": 90
    }
}

Discovery proxy config

Proxy config file (see "Discovery" for more details):

{
    "discovery":
    {
        "registry": ":eureka:8761",
        "adaptor": "discEurekaAdaptor.q"
    }
}

Custom file example

The Agg process can load a custom code file, wherein you can define custom aggregation functions and define API to aggregation function mappings for APIs. Below is an example file, as some example API calls that exercise the custom code.

// Sample Agg custom file.

// Can load other files within this file. Note that the current directory
// is the directory of this file (in this example: /opt/kx/custom).
\l subFolder/otherFile1.q
\l subFolder/otherFile2.q

//
// @desc An override to the default ping aggregation function. Instead of doing a raze,
// we just take the min (so true indicates all targets successful).
//
// @param res   {boolean[]} Results from the DAPs.
//
// @return      {boolean}   Min of all DAP results.
//
pingAggOverride:{[res]
    min res
    }


//
// @desc Agg function that does a plus join on a list of tables.
//
// @param tbls  {table[]}   List plus-joinable tables.
//
// @return      {table}     Plus join.
//
pjAgg:{[tbls]
    (pj/)tbls
    }


//
// @desc Agg function that does an average daily count by sym.
//
// @param tbls  {table[]}   List of tables with ``` `sym`date`cnt``` columns.
//
// @return      {table}     Average count by sym
//
avAgg:{[tbls]
    res:select sum cnt by sym,date from raze 0!'tbls; / Join common dates
    select avg cnt by sym from res / Average
    }


//
// In order to be usable, aggregation functions MUST be registered with the Agg process. When registering,
// one can also set the aggregation function as the default aggregation function for one or more APIs.
// For example, Suppose we had an API defined in the DAPs that peforms a "count by" operation on a table:
//
// countBy:{[table;startTS;endTS;byCols]
//     ?[table;enlist(within;`realTime;(startTS;endTS-1));{x!x,:()}byCols;enlist[`cnt]!enlist(count;`i)]
//     }
//
// We can then register our aggregations functions thusly:
//
.sgagg.registerAggFn[`pingAggOverride;
    .sapi.metaDescription["Custom override to .kxi.ping"],
    .sapi.metaParams[`name`type`description!(`res;1h;"List of booleans indicating ping was successful")],
    .sapi.metaReturn`type`description!(-1h;"The worst of all results"];
    `$()
    ]

.sgagg.registerAggFn[`pjAgg;
    .sapi.metaDescription["Plus join aggregation"],
    .sapi.metaParams[`name`type`description!(`tbls;0h;"Tables received from DAPs")],
    .sapi.metaReturn`type`description!(98h;"The plus join (over) of the tables");
    `countBy]; // Register as default aggregation function for this API

.sgagg.registerAggFn[`avAgg;
    .sapi.metaDescription["Average join aggregation"],
    .sapi.metaParams[`name`type`description!(`tbls;0h;"Tables received from DAPs")],
    .sapi.metaReturn`type`description!(98h;"The average join (over) of the tables");
    `$()
    ]

//
// Note also that an aggregation function can be default aggregation function for multiple APIs. E.g.
//  .sgagg.registerAggFn[`myAggFn;();`api1`api2`api3]
//

Calls to the GW can now reference these new agg functions. Note that agg function mappings can always be overridden with the aggFn key in the request header.

q)h:hopen`:gwHost:5678 // Connect to GW

// Call ping without specifying the agg function. Defaults to raze.
q)last h(`.kxi.ping;`region`assetClass`startTS`endTS!(`amer`emea`apac;`equity;-0Wp;0Wp);`;(0#`)!())
111b

// Call with an override; we'll now just get the min value.
q)last h(`.kxi.ping;`region`assetClass`startTS`endTS!(`amer`emea`apac;`equity;-0Wp;0Wp);`;``aggFn!("";`pingAggOverride))
1b

// Call to our countBy API without specifying the aggregation function. Defaults to `pjAgg` as specified
// in the Agg's mapping.
q)last h(`countBy;`region`assetClass`startTS`endTS`table`byCols!(`amer;`equity;-0Wp;0Wp;`trade;`sym);(0#`)!())
sym    | cnt
-------| ------
AAPL.N | 103212
MSFT.OQ| 100213
..

// Call to our countBy API and specify an aggFn override.
q)last h(`countBy;`region`assetClass`startTS`endTS`table`byCols!(`amer;`equity;-0Wp;0Wp;`trade;`sym`date);``aggFn!("";`avAgg))
sym    | cnt
-------| ------
AAPL.N | 344.5
MSFT.OQ| 301.22
..

Tuning the log level

The kxi-sg-gw log level can be configured with `KXI_LOG_LEVELS as above.

At runtime, the log level can be dynamically adjusted RESTfully by POSTing to the /log endpoint.

curl -X POST "https://${HOSTNAME}/log?level=debug

Garbage Collection

To configure garbage collection for the aggregator and resource-coordinator, set the garbage collection flag.

The default garbage collection mode is deferred.

For the service-gateway, message buffers are allocated per connected aggregator.

Upon connection, each thread is allocated KXI_SG_BUFFER_INITIAL bytes. Message buffers will expand to fit any response up to the 2GB limit. After returning a response, the buffers will shrink back to KXI_SG_BUFFER_RETAIN bytes.

By default the KXI_SG_BUFFER_INITIAL and the KXI_SG_BUFFER_RETAIN are 2MB.