Skip to content

Service Gateway configuration

In its most basic form, the SG is a set of Docker images that are combined using minimal configuration. Below is an explanation of the images required, the configuration parameters to be defined, and some example configurations.

Images

There is one image per process type in the SG. The SG architecture allows for multiple gateways to be run in parallel (clients can load balance as desired) and multiple Aggs (the SG automatically load balances requests between them). Note, however, that there can only be one RC process.

Required images:

process number required image
GW many Yes kxi-sg-gw
RC 1 Yes kxi-sg-rc
Agg many Yes kxi-sg-agg
REST many No kxi-sg-rest
REST proxy many (1 per REST) No kxi-sg-rest-proxy

In addition, the SG can optionally use KX Insights Discovery for processes to discover and connect with each other seamlessly.

Required images:

process description image
sidecar Discovery sidecar (for the RC). kxi_sidecar
discovery Discovery client. Configure one, which all processes seamlessly connect to. kxi-eureka-discovery
proxy Discovery proxy. discovery_proxy

Assembly

The assembly configuration is a YAML file that defines the SG’s purview, i.e. what data the DAPs below it offer. Assemblies are used in all KX Insights microservices, but the SG in particular only needs to define the following fields:

field required description
name Yes GW assembly name. This can be used in by DAPs to discover the SG's RC (see "Interface" page).
description No Description of the assembly.
labels Yes Labels (i.e. dimensions) along which the data is partitioned in the DAPs the GW is responsible for, and possible values (see Labels).
elements Yes RC connection details (see Elements).

See Labels/Elements, or Example for example assembly YAML configurations. Include the assembly YAML file in the Docker container.

Labels

Labels are used to define the SG's purview, that is, the data it grants access to. There are two kinds:

  • User defined: values are comma-delimited lists of values in the GW’s purview
  • startTS or endTS: timestamps denoting the earliest or latest time (as per the DAP’s temporal partition column) the GW is responsible for. Either or both may omitted: startTS defaults to -0Wp and endTS to 0Wp.

Some examples:

Neither startTS nor endTS is defined. They default to -0Wp and 0Wp, respectively.

labels:
  region: amer,emea,apac
  assetClass: fx,equity

endTS is defined, but startTS is not. (This might be appropriate for a historical-only DB.)

labels:
  sensorType: gas,electric,water
  clientType: residential,commercial
  billing: weekly,monthly
  endTS: 2021.01.01D

Elements

The elements tag is used to define:

  • RC connection details for all processes within the microservice, specified as host and port
  • default timeout value in milliseconds for requests; optional, default 30,000 ms

Example:

elements:
  rc:
    host: rcHost
    port: 1234
  timeout: 30000

Environment variables

The GW microservice relies on certain environment variables to be defined in the containers.

variable required containers description
KXI_NAME Yes RC, Agg, REST proxy Process name.
KXI_PORT Yes RC, Agg, REST proxy Port.
KXI_LOG_FORMAT No RC, Agg, REST proxy, sidecar Message format (see "Logging" page).
KXI_LOG_DEST No RC, Agg, REST proxy, sidecar Endpoints (see "Logging" page).
KXI_LOG_LEVELS No RC, Agg, REST proxy, sidecar Component routing (see "Logging" page).
KXI_ASSEMBLY_FILE Yes RC, Agg, GW, REST proxy Assembly yaml.file.
DISCOVERY_PROXY No GW Discovery proxy address (not required if not using discovery).
GATEWAY_PORT Yes GW Gateway port.
KXI_CONFIG_FILE Yes sidecar Discovery configuration file (see "Discovery" page).
KXI_GW Yes REST proxy Gateway host:port to connect to.
KXI_CUSTOM_FILE No Agg File containing custom code to load into Agg process.

The REST container also requires command-line arguments at startup:

--port <REST process port> --proxy <REST proxy host> <REST proxy port>

Example

Custom file

The default aggregation for any API is raze. However, the Agg processes will load the q file pointed to by the KXI_CUSTOM_FILE environment variable. In this file, you can:

  • define/register custom aggregation functions
  • define which aggregation functions to invoke for which APIs
  • define metadata pertaining to the aggregation function, which is subsequently queryable using the .kxi.getMeta API
Loading multiple scripts

While SG supports loading only a single file, you can load others from within this file using \l, allowing you to control load order. The current working directory (pwd) at load time is the base directory of the file.

This can be combined with the Data Access microservice (which allows custom function definitions in the DAPs) to create full custom API support within KX Insights.

Avoid .sg* namespaces lest they collide with SG functions.

An aggregation function can be anything. Its argument is a list, the output of the corresponding API defined in the DAPs, and its result can be anything.

Register an aggregation function with .sgagg.registerAggFn to make it available to the Aggregator.

.sgagg.registerAggFn[aggFn;metadata;apis]

Where

  • aggFn is the aggregation function name (symbol)
  • metadata is the aggregation function metadata (list, string, or dictionary)
  • apis is the API/s for which this should be the default aggregation function (symbol vector)

Custom file example

Example

We use a docker-compose.yaml; this can be adapted for other formats.

Variables ${...} are user-defined and based on your local directory structure and file names. Sections/lines marked Optional are optional. Note in particular:

  • ${registry}: Registry from which you download images, e.g. "gcr.io/cloudpak"
  • ${*-version}: Version of the image, formatted as #.#.#

docker-compose.yaml:

#
# Optional: Create volumes to include licence/configuration in the containers.
#
x-vols:
  volumes:
  - ${kx_licence_dir}:/opt/kx/lic
  - ${cfg_dir}:/opt/kx/cfg
  - ${custom_dir}:/opt/kx/custom # Optional mount for loading custom code

#
# Optional: Create a network for processes to communicate.
#
x-kxnet: &kxnet
  networks:
  - kx

networks:
  kx:
    name: kx
    driver: bridge

#
# Services.
#
services:
  #
  # Resource Coordinator.
  #
  sg_rc:
    image: ${registry}/kxi-sg-rc:${sg-version}
    environment:
      - KXI_NAME=sg_rc
      - KXI_PORT=5050
      - KXI_LOG_FORMAT=text # Optional
      - KXI_LOG_LEVELS=default:debug # Optional
      - KXI_ASSEMBLY_FILE=/opt/kx/cfg/${assembly_file_yaml}
      - KDB_LICENSE_B64=${kdb-licence}
    <<: *vols # Optional
    <<: *kxnet # Optional

  #
  # Optional: Resource Coordinator sidecar. Only required if using discovery, otherwise, may be omitted.
  #
  rc_sidecar:
    image: ${registry}/kxi_sidecar:${sidecar-version}
    environment:
      - KXI_CONFIG_FILE=/opt/kx/cfg/${rc_sidecar_config_json}
      - KXI_LOG_FORMAT=text # Optional
      - KXI_LOG_LEVELS=default:debug # Optional
      - KDB_LICENSE_B64=${kdb-licence}
    <<: *vols # Optional
    <<: *kxnet # Optional

  #
  # Aggregator. Note we only have one here, but multiple can be configured.
  #
  sg_agg:
    image: ${registry}/kxi-sg-agg:${sg-version}
    environment:
      - KXI_NAME=sg_agg
      - KXI_PORT=5060
      - KXI_LOG_FORMAT=text # Optional
      - KXI_LOG_LEVELS=default:debug # Optional
      - KXI_ASSEMBLY_FILE=/opt/kx/cfg/${assembly_file_yaml}
      - KXI_CUSTOM_FILE=/opt/kx/custom/${custom_agg_code}.q # Optional
      - KDB_LICENSE_B64=${kdb-licence}
    ports:
      - 127.0.0.1::5060
    # Optional: deploy multiple replicas.
    deploy:
      mode: replicated
      replicas: 3
    <<: *vols # Optional
    <<: *kxnet # Optional

  #
  # Gateway.
  #
  sg_gw:
    image: ${registry}/kxi-sg-gw:${sg-version}
    environment:
      - DISCOVERY_PROXY=http://proxy:4000 # Optional, only if using discovery
      - KXI_ASSEMBLY_FILE=/opt/kx/cfg/${assembly_file_yaml}
      - GATEWAY_QIPC_PORT=5040
      - GATEWAY_HTTP_PORT=8080
    ports:
      - 127.0.0.1::5040
      - 127.0.0.1::8080
    # Optional: deploy multiple replicas.
    deploy:
      mode: replicated
      replicas: 5
    <<: *vols # Optional
    <<: *kxnet # Optional

  #
  # Optional: Eureka Service Discovery Registry. Only required if using discovery, otherwise, may be omitted.
  #
  eureka:
    image: ${registry}/kxi-eureka-discovery:${eureka-version}
    ports:
      - 9000:8761

  #
  # Optional: Discovery proxy. Only required if using discovery, otherwise, may be omitted.
  #
  proxy:
    image: ${registry}/kxi-discovery-proxy:${discovery-version}
    ports:
      - 4000:4000
    environment:
      - KXI_CONFIG_FILE=/opt/kx/cfg/${proxy_config_json}
      - KDB_LICENSE_B64=${kdb-licence}
    command: -p 4000

Assembly:

name: myServiceGatway
description: Example SG assembly.
labels:
  region: amer,emea,apac
  assetClass: fx,equity,futures
elements:
  rc:
    host: sg_rc
    port: 5050

The RC discovery sidecar config file:

{
    "connection": ":sg_rc:5050",
    "frequencySecs": 5,
    "discovery":
    {
        "registry": ":proxy:4000",
        "adaptor": "discEurekaAdaptor.q",
        "heartbeatSecs": 30,
        "leaseExpirySecs": 90
    }
}

[Proxy config file[(../service-discovery/README.md):

{
    "discoveryProxy":
    {
        "registry": ":eureka:8761",
        "adaptor": "discEurekaAdaptor.q"
    }
}

Custom file example

The Agg process can load a custom code file, wherein you can define custom aggregation functions and define API-to-aggregation function mappings for APIs.

An example file, and API calls that exercise the custom code:

// Sample Agg custom file.

// Can load other files within this file. Note that the current directory
// is the directory of this file (in this example: /opt/kx/custom).
\l subFolder/otherFile1.q
\l subFolder/otherFile2.q

//
// @desc An override to the default ping aggregation function. 
// Instead of doing a raze, we just take the min 
// (so true indicates all targets successful).
//
// @param res   {boolean[]} Results from the DAPs.
//
// @return      {boolean}   Min of all DAP results.
//
pingAggOverride:{[res]
  min res }


//
// @desc Agg function that does a plus join on a list of tables.
//
// @param tbls  {table[]}   List plus-joinable tables.
//
// @return      {table}     Plus join.
//
pjAgg:{[tbls]
  (pj/)tbls }


//
// @desc Agg function that does an average daily count by sym.
//
// @param tbls  {table[]}   List of tables with ``` `sym`date`cnt``` columns.
//
// @return      {table}     Average count by sym
//
avAgg:{[tbls]
  res:select sum cnt by sym,date from raze 0!'tbls; / Join common dates
  select avg cnt by sym from res } / Average

//
// To be usable, aggregation functions MUST be registered with the Agg process. 
// When registering, one can also set the aggregation function as the default 
// aggregation function for one or more APIs. 
// For example, suppose we had an API defined in the DAPs 
// that performs a "count by" operation on a table:
//
// countBy:{[table;startTS;endTS;byCols]
//   ?[table;
//     enlist(within;`realTime;(startTS;endTS-1));
//     {x!x,:()}byCols;
//     enlist[`cnt]!enlist(count;`i)] }
//
// We can then register our aggregations functions:
//
.sgagg.registerAggFn[`pingAggOverride;
  .sapi.metaDescription["Custom override to .kxi.ping"],
    .sapi.metaParams[`name`type`description!
      (`res;1h;"List of booleans indicating ping was successful")],
    .sapi.metaReturn`type`description!(-1h;"The worst of all results"];
  `$() ]

// Register default aggregation function for this API
.sgagg.registerAggFn[`pjAgg;
  .sapi.metaDescription["Plus join aggregation"],
    .sapi.metaParams[`name`type`description!
      (`tbls;0h;"Tables received from DAPs")],
    .sapi.metaReturn`type`description!(98h;"The plus join (over) of the tables");
  `countBy] 

.sgagg.registerAggFn[`avAgg;
  .sapi.metaDescription["Average join aggregation"],
    .sapi.metaParams[`name`type`description!
      (`tbls;0h;"Tables received from DAPs")],
    .sapi.metaReturn`type`description!
      (98h;"The average join (over) of the tables");
  `$() ]

An aggregation function can be the default for multiple APIs.

.sgagg.registerAggFn[`myAggFn;();`api1`api2`api3]

Calls to the GW can now reference these new aggregation functions.

Note that aggregation function mappings can always be overridden with the aggFn key in the request header.

q)h:hopen`:gwHost:5678 // Connect to GW

Call ping without specifying the agg function. (Defaults to raze.)

q)args:`region`assetClass`startTS`endTS!(`amer`emea`apac;`equity;-0Wp;0Wp)
q)last h(`.kxi.ping;args;`;(0#`)!())
111b

Call with an override; we’ll now just get the minimum value.

q)args:`region`assetClass`startTS`endTS!(`amer`emea`apac;`equity;-0Wp;0Wp)
q)last h(`.kxi.ping;args;`;``aggFn!("";`pingAggOverride))
1b

Call to our countBy API without specifying the aggregation function. Defaults to pjAgg as specified in the agg’s mapping.

q)k:`region`assetClass`startTS`endTS`table`byCols
q)v:(`amer;`equity;-0Wp;0Wp;`trade;`sym)
q)last h(`countBy;k!v;(0#`)!())
sym    | cnt
-------| ------
AAPL.N | 103212
MSFT.OQ| 100213
..

Call to our countBy API and specify an aggFn override.

q)k:`region`assetClass`startTS`endTS`table`byCols
q)v:(`amer;`equity;-0Wp;0Wp;`trade;`sym`date)
q)last h(`countBy;k!v;``aggFn!("";`avAgg))
sym    | cnt
-------| ------
AAPL.N | 344.5
MSFT.OQ| 301.22
..