Skip to content

Configuration

In its most basic form, the SG is a set of Docker images that are combined using minimal configuration. Below is an explanation of the images required, what configuration parameters need to be defined, and an some example configurations.

Images

There is one image per process type in the SG. The SG architecture allows for multiple GWs to be run in parallel (clients can load balance as desired), and multiple Aggs (the SG automatically load balances requests between them). Note, however, that there can only be one RC process. The table below summarizes the required images.

process number required image
GW many Yes kxi-sg-gw
RC 1 Yes kxi-sg-rc
Agg many Yes kxi-sg-agg
REST many No kxi-sg-rest
REST proxy many (1 per REST) No kxi-sg-rest-proxy

In addition, the SG can optionally use KX Insights Discovery in order for processes to discover and connect with each other seamlessly (see "Discovery"). Images required are as follows.

process description image
sidecar Discovery sidecar (for the RC). kxi_sidecar
discovery Discovery client. Configure one, which all processes seamlessly connect to. kxi-eureka-discovery
proxy Discovery proxy. discovery_proxy

Assembly

The assembly configuration is a yaml file that defines the SG's purview, i.e. what data the DAPs below it offer. Assemblies are used in all KX Insights microservices (see "Assembly"), but the SG in particular only needs to define the following fields.

field required description
name Yes GW assembly name. This can be used in by DAPs to discover the SG's RC (see "Interface" page).
description No Description of the assembly.
labels Yes Labels (i.e. dimensions) along which the data is partitioned in the DAPs the GW is responsible for, and possible values (see Labels).
elements Yes RC connection details (see Elements).

See Labels/Elements or Example for example assembly yaml configurations. The assembly yaml file must be included in the Docker container.

Labels

Labels are used to define the SG's purview. That is, the data that it grants access to. Labels come in two flavours: * User defined labels. Values are comma-delimited lists of values in the GW's purview. * startTS/endTS. These are timestamps denoting the earliest/latest time (as per the DAPs temporal partition column) the GW is responsible for. Neither, either, or both of these values may be defined. If ommitted, startTS defaults to -0Wp and endTS defaults to 0Wp.

Below are some examples.

Example 1

Neither startTS nor endTS are defined. They default to -0Wp and 0Wp, respectively.

labels:
    region: amer,emea,apac
    assetClass: fx,equity
Example 2

endTS is defined, but startTS is not (this can be appropriate for a historical-only DB).

labels:
    sensorType: gas,electric,water
    clientType: residential,commercial
    billing: weekly,monthly
    endTS: 2021.01.01D

Elements

The elements tag is used to define RC connection details for all processes within the microservice. The elements section contains one tag: rc, which itself contains two tags: host and port, which are the host and port of the RC process, respectively. Example:

elements:
  rc:
    host: rcHost
    port: 1234

Environment variables

The GW microservice relies on certain environment variables to be defined in the containers. The variables are described below.

variable required containers description
KXI_NAME Yes RC, Agg, REST proxy Process name.
KXI_PORT Yes RC, Agg, REST proxy Port.
KXI_LOG_FORMAT No RC, Agg, REST proxy, sidecar Message format (see "Logging" page).
KXI_LOG_DEST No RC, Agg, REST proxy, sidecar Endpoints (see "Logging" page).
KXI_LOG_LEVELS No RC, Agg, REST proxy, sidecar Component routing (see "Logging" page).
KXI_ASSEMBLY_FILE Yes RC, Agg, GW, REST proxy Assembly yaml.file.
DISCOVERY_PROXY No GW Discovery proxy address (not required if not using discovery).
GATEWAY_PORT Yes GW Gateway port.
KXI_CONFIG_FILE Yes sidecar Discovery configuration file (see "Discovery" page).
KXI_GW Yes REST proxy Gateway host:port to connect to.
KXI_CUSTOM_FILE No Agg File containing custom code to load into Agg process.

In addition to environment variables, the REST container requires command line arguments at startup:

--port <REST process port> --proxy <REST proxy host> <REST proxy port>

See example section below.

Custom file

The default aggregation for any API is raze. However, the Agg processes will load the q file pointed to by the KXI_CUSTOM_FILE environment variable. In this file, you can * define custom aggregation functions, and * define the aggregation functions to use for each API. Note that while SG only supports loading a single file, you can load other files from within this file using \l (allowing you to control load order). The current working directory (pwd) at load time is the base directory of the file.

This can be combined with the Data Access microservice (which allows custom function definitions in the DAPs) to create full custom API support within KX Insights (see "Data Access" for details).

Note: It's recommended to avoid .sg* namespaces to avoid colliding with SG functions.

Aggregation functions can be anything. Their input is a list of the output of the corresponding API defined in the DAPs, and its output can be anything.

To define which aggregation function to use for an API, use the .sgagg.addAggFn API, whose signature is as follows. * api - symbol - API name. * aggFn - symbol - Agg function to associate to that API.

Note that .sgagg.addAggFn throws an aggFnMapType error if either parameter is not a symbol. Morevoer, aggregation function can always be overridden using the aggFn key in the request header (see "Header" page).

See Custom file example below for an example.

Example

Below is a sample configuration. We use a docker-compose yaml, but this can be adapted to other formats. Note: variables ${...} are user-defined and based on your local directory structure/file names. Sections/lines marked Optional are optional.

Docker-compose yaml file:

#
# Optional: Create volumes to include licence/configuration in the containers.
#
x-vols:
    volumes:
    - ${kx_licence_dir}:/opt/kx/lic
    - ${cfg_dir}:/opt/kx/cfg
    - ${custom_dir}:/opt/kx/custom # Optional mount for loading custom code

#
# Optional: Create a network for processes to communicate.
#
x-kxnet: &kxnet
    networks:
    - kx

networks:
    kx:
    name: kx
    driver: bridge

#
# Services.
#
services:
    #
    # Resource Coordinator.
    #
    sg_rc:
    image: kxi-sg-rc:0.8.0
    environment:
        - KXI_NAME=sg_rc
        - KXI_PORT=5050
        - KXI_LOG_FORMAT=text # Optional
        - KXI_LOG_LEVELS=default:debug # Optional
        - KXI_ASSEMBLY_FILE=/opt/app/cfg/${assembly_file_yaml}
    <<: *vols # Optional
    <<: *kxnet # Optional

    #
    # Optional: Resource Coordinator sidecar. Only required if using discovery, otherwise, may be omitted.
    #
    rc_sidecar:
    image: kxi_sidecar:0.8.0
    environment:
        - KXI_CONFIG_FILE=/opt/kx/cfg/${rc_sidecar_config_json}
        - KXI_LOG_LEVELS=default:debug # Optional
    <<: *vols # Optional
    <<: *kxnet # Optional

    #
    # Aggregator. Note we only have one here, but multiple can be configured.
    #
    sg_agg:
    image: kxi-sg-agg:0.8.0
    environment:
        - KXI_NAME=sg_agg
        - KXI_PORT=5060
        - KXI_LOG_FORMAT=text # Optional
        - KXI_LOG_LEVELS=default:debug # Optional
        - KXI_ASSEMBLY_FILE=/opt/app/cfg/${assembly_file_yaml}
        - KXI_CUSTOM_FILE=/opt/kx/custom/${custom_agg_code}.q # Optional
    ports:
        - 5060-5069:5060
    # Optional: deploy multiple replicas.
    deploy:
        mode: replicated
        replicas: 3
    <<: *vols # Optional
    <<: *kxnet # Optional

    #
    # Gateway.
    #
    sg_gw:
    image: kxi-sg-gw:0.8.0
    environment:
        - DISCOVERY_PROXY=http://proxy:4000 # Optional, only if using discovery
        - KXI_ASSEMBLY_FILE=/opt/app/cfg/${assembly_file_yaml}
        - GATEWAY_PORT=5040
    ports:
        - 5040-5049:5040
    # Optional: deploy multiple replicas.
    deploy:
        mode: replicated
        replicas: 5
    <<: *vols # Optional
    <<: *kxnet # Optional

    #
    # Optional: REST. Can have multiple of these.
    #
    sg_rest:
    image: kxi-sg-rest:0.8.0
    command: --port 5030 --proxy sg_rest_proxy 5031
    ports:
        - 5030:5030
    <<: *kxnet # Optional

    #
    # Optional: REST proxy. Need one per REST container.
    #
    sg_rest_proxy:
    image: kxi-sg-rest
    environment:
        - KXI_NAME=rest_proxy
        - KXI_PORT=5031
        - KXI_GW=sg_gw:5040
        - KXI_LOG_FORMAT=text # Optional
        - KXI_LOG_LEVELS=default:trace # Optional
        - KXI_ASSEMBLY_FILE=/opt/kx/cfg/{assembly_file_yaml}

    #
    # Optional: Eureka Service Discovery Registry. Only required if using discovery, otherwise, may be omitted.
    #
    eureka:
    image: kxi-eureka-discovery:0.8.0
    ports:
        - 9000:8761

    #
    # Optional: Discovery proxy. Only required if using discovery, otherwise, may be omitted.
    #
    proxy:
    image: discovery_proxy:0.8.0
    ports:
    - 4000:4000
    environment:
    - KXI_CONFIG_FILE=/opt/app/cfg/${proxy_config_json}
    command: -p 4000

Assembly:

name: myServiceGatway
description: Example SG assembly.
labels:
    region: amer,emea,apac
    assetClass: fx,equity,futures
elements:
    rc:
    host: sg_rc
    port: 5050

The RC discovery sidecar

Config file (see "Discovery" for more details):

{
    "connection": ":sg_rc:5050",
    "frequencySecs": 5,
    "discovery":
    {
        "registry": ":proxy:4000",
        "adaptor": "discEurekaAdaptor.q",
        "heartbeatSecs": 30,
        "leaseExpirySecs": 90
    }
}

Custom file example

The Agg process can load a custom code file, wherein you can define custom aggregation functions and define API to aggregation function mappings for APIs. Below is an example file, as some example API calls that exercise the custom code.

// Sample Agg custom file.

// Can load other files within this file. Note that the current directory
// is the directory of this file (in this example: /opt/kx/custom).
\l subFolder/otherFile1.q
\l subFolder/otherFile2.q

//
// @desc An override to the default ping aggregation function. Instead of doing a raze,
// we just take the min (so true indicates all targets successful).
//
// @param res   {boolean[]} Results from the DAPs.
//
// @return      {boolean}   Min of all DAP results.
//
pingAggOverride:{[res]
    min res 
    }


//
// @desc Agg function that does a plus join on a list of tables.
//
// @param tbls  {table[]}   List plus-joinable tables.
//
// @return      {table}     Plus join.
//
pjAgg:{[tbls]
    (pj/)tbls
    }


//
// @desc Agg function that does an average daily count by sym.
//
// @param tbls  {table[]}   List of tables with ``` `sym`date`cnt``` columns.
//
// @return      {table}     Average count by sym
//
avAgg:{[tbls]
    res:select sum cnt by sym,date from raze 0!'tbls; / Join common dates
    select avg cnt by sym from res / Average
    }


//
// Suppose we had an API defined in the DAPs that peforms a "count by" operation on a table:
//
// countBy:{[table;startTS;endTS;byCols]
//     ?[table;enlist(within;`realTime;(startTS;endTS-1));{x!x,:()}byCols;enlist[`cnt]!enlist(count;`i)]
//     }
//
// Then we can specify one of our aggregation functions be the default aggregation for this API.
//
.sgagg.addAggFn[`countBy;`pjAgg]

Calls to the GW can now reference these new agg functions. Note that agg function mappings can always be overridden with the aggFn key in the request header.

q)h:hopen`:gwHost:5678 / Connect to GW

// Call ping without specifying the agg function. Defaults to raze.
q)last h(`.kxi.ping;`region`assetClass`startTS`endTS!(`amer`emea`apac;`equity;-0Wp;0Wp);`;(0#`)!())
111b

// Call with an override; we'll now just get the min value.
q)last h(`.kxi.ping;`region`assetClass`startTS`endTS!(`amer`emea`apac;`equity;-0Wp;0Wp);`;``aggFn!("";`pingAggOverride))
1b

// Call to our countBy API without specifying the aggregation function. Defaults to `pjAgg` as specified
// in the Agg's mapping.
q)last h(`countBy;`region`assetClass`startTS`endTS`table`byCols!(`amer;`equity;-0Wp;0Wp;`trade;`sym);(0#`)!())
sym    | cnt
-------| ------
AAPL.N | 103212
MSFT.OQ| 100213
..

// Call to our countBy API and specify an aggFn override.
q)last h(`countBy;`region`assetClass`startTS`endTS`table`byCols!(`amer;`equity;-0Wp;0Wp;`trade;`sym`date);``aggFn!("";`avAgg))
sym    | cnt
-------| ------
AAPL.N | 344.5
MSFT.OQ| 301.22
..