Customizing the FSI Accelerator
It is possible to customize an FSI accelerator by unpacking and updating the contents of the package. Let's go through some examples using a generic fsi-data-assembly.
Overlaying a Schema with Custom Columns
The fsi-data-assembly
package comes with a set of base schemas. If you'd like to append additional custom columns to these schemas, carry out the following:
Unpack fsi-data-assembly
Unpack the fsi-data-assembly
package.
> kxi package unpack fsi-data-assembly-1.0.0.kxi
>
> ls
fsi-data-assembly fsi-data-assembly-1.0.0.kxi
>
Add a Patch
Add an overlay patch to fsi-data-assembly
. In this case we're calling the patch "FuturesOverlay".
> kxi package add --to fsi-data-assembly patch --name FuturesOverlay
Writing fsi-data-assembly/manifest.yaml
>
>
> ls fsi-data-assembly/patches/
FuturesOverlay.yaml
>
>
> cat fsi-data-assembly/patches/FuturesOverlay.yaml
kind: Package
apiVersion: pakx/v1
metadata:
name: target
spec:
manifest: {}
databases: []
pipelines: []
router: null
deployment_config: null
>
Add in Additional Columns to the Overlay yaml
Overlay the base FutQuotes
table in fsi-data-assembly/databases/fsi-data-assembly-db/schema.yaml
with some additional columns. Do this by updating FuturesOverlay.yaml to the following:
kind: Package
apiVersion: pakx/v1
metadata:
name: target
spec:
databases:
- name: fsi-data-assembly-db
tables:
schemas:
- name: FutQuote
columns:
- name: contractID
type: string
- name: openInterest
type: long
- name: settlePrice
type: float
In this example 3 new columns are being added to the FutQuotes
table, contractID
, openInterest
and settlePrice
.
Overlay the fsi-data-assembly Package
Apply the overlay to the package using the kxi package overlay
command.
> cd fsi-data-assembly/
>
> kxi package overlay . patches/FuturesOverlay.yaml
Writing /tmp/artifacts/fsi-data-assembly/manifest.yaml
>
Re-package
Re-package the fsi-data-assembly
package. It's recommended that the package is renamed during repackaging so that it can be distinguished as a custom package. In this example it's renamed to client-assembly
.
> kxi package packit fsi-data-assembly --tag --package-name client-assembly --version 1.0.0
Refreshing package before packing...
Writing fsi-data-assembly/manifest.yaml
Creating package from fsi-data-assembly
Package created: /tmp/artifacts/client-assembly-1.0.0.kxi
client-assembly-1.0.0.kxi
>
Deploy
Finally, deploy the client-assembly
package with the new changes. See How to Deploy the FSI Accelerator for these steps.
Adding a New Schema
If you'd like to add a new schema to fsi-data-assembly
, carry out the following:
Unpack fsi-data-assembly
Unpack the fsi-data-assembly
package.
> kxi package unpack fsi-data-assembly-1.0.0.kxi
>
> ls
fsi-data-assembly fsi-data-assembly-1.0.0.kxi
>
Add a Patch
Add an overlay patch to fsi-data-assembly
. In this case we're calling the patch "FIOverlay".
> kxi package add --to fsi-data-assembly patch --name FIOverlay
Writing fsi-data-assembly/manifest.yaml
>
>
> ls fsi-data-assembly/patches/
FIOverlay.yaml
>
>
> cat fsi-data-assembly/patches/FIOverlay.yaml
kind: Package
apiVersion: pakx/v1
metadata:
name: target
spec:
manifest: {}
databases: []
pipelines: []
router: null
deployment_config: null
>
Add the New Schema to the Overlay yaml
Add a new schema to the FIOverlay.yaml file. In this example the new table added is called FIQuote
. Do this by updating FIOverlay.yaml to the following:
kind: Package
apiVersion: pakx/v1
metadata:
name: target
spec:
databases:
- name: fsi-data-assembly-db
tables:
schemas:
- name: FIQuote
columns:
- name: eventTimestamp
type: timestamp
attrDisk: parted
attrOrd: parted
- name: instrumentID
type: symbol
- name: bidPrice
type: float
- name: bidSize
type: long
- name: askPrice
type: float
- name: askSize
type: long
- name: bidYield
type: float
- name: askYield
type: float
- name: srcSys
type: symbol
description: Fixed Income Quote Schema
prtnCol: eventTimestamp
sortColsDisk:
- eventTimestamp
sortColsMem:
- eventTimestamp
sortColsOrd:
- eventTimestamp
type: partitioned
Overlay the fsi-data-assembly Package
Apply the overlay to the package using the kxi package overlay
command.
> cd fsi-data-assembly/
>
> kxi package overlay . patches/FIOverlay.yaml
Writing /tmp/artifacts/fsi-data-assembly/manifest.yaml
>
Re-package
Re-package the fsi-data-assembly
package. It's recommended that the package is renamed during repackaging so that it can be distinguished as a custom package. In this example it's renamed to client-assembly
.
> kxi package packit fsi-data-assembly --tag --package-name client-assembly --version 1.0.0
Refreshing package before packing...
Writing fsi-data-assembly/manifest.yaml
Creating package from fsi-data-assembly
Package created: /tmp/artifacts/client-assembly-1.0.0.kxi
client-assembly-1.0.0.kxi
>
Deploy
Finally, deploy the client-assembly
package with the new changes. See How to Deploy the FSI Accelerator for these steps.
Adding Custom Configuration or Code
Custom configuration can be added to load before the fsi-lib
package. This could be necessary to customize getStats analytics or getBars aggregations, for example.
This involves two steps: - Create a package in order to deploy the configuration - Add the package name and version to the KXI_PACKAGES environment variable
Creating a Package
To create a package for the custom configuration/code follow the steps in these articles:
It is only necessary to include a single q file in the package with the required configuration/code. This could be in a src/sp.q
file, which is loaded in the package init.q
file, as suggested in the above links, or the default init.q
file in the package could be updated to add the configuration.
Adding the Package to KXI_PACKAGES
Add the package name and version number to the KXI_PACKAGES environment variable that should be added under every aggregator in the values.yaml. To ensure the new package containing configuration loads before fsi-lib
, place it first in the list of packages. This should be done before installing kdb Insights Enterprise Pre-requisites.
aggregator:
env:
KXI_PACKAGES: your-package-name:1.0.0,fsi-lib:1.0.0
If kdb Insights Enterprise is already installed you can edit the stateful set of the aggregators to update KXI_PACKAGES.
You may also want to add your package name to KXI_PACKAGES on the resource coordinators if any of your configuration or code applies to resource coordinator functionality. This can be accomplished in the same way as for the aggregator.
The new configuration package should be pushed to your kdb Insights Enterprise install along with the fsi-lib
and fsi-data-assembly
packages Push the packages.
Example
Initialize Package
Initialize a new package using kxi package init
. In this example the package is called client-lib
.
> kxi package init client-lib
Creating package at location: client-lib
Writing client-lib/manifest.yaml
>
> ls
client-lib
>
> ls client-lib/
init.q manifest.yaml
>
Create client-da.q File
Create a src
directory.
> mkdir client-lib/src
>
> ls client-lib/
init.q manifest.yaml src
>
Within that src
directory create a file called client-da.q
and add the following code to register an example API in the Data Access Processes:
.client.testAPI:{[args]
startDate:"d"$args`startTS;
endDate:"d"$args`endTS;
//Drop null inbound args
select from FutQuote where date>=startDate, date<=endDate
};
.da.registerAPI[`.client.testAPI;
.sapi.metaDescription["Test API"],
.sapi.metaReturn[`type`description!(98h;"Results table")],
.sapi.metaMisc[enlist[`safe]!enlist 1b] // allow retries in the event of failure
];
Create client-agg.q
Create a file called client-agg.q
and add the following:
.client.join:{[returns]
if[@[get;`.debug.active;0b];.debug.clientjoin:(`time`returns)!(.z.p;returns)];
.sapi.ok {$[0h=type x;raze;{x}]} returns
};
.sgagg.registerAggFn[`.client.join;
.sapi.metaDescription["join test API results"],
.sapi.metaParam[`name`type`description!(`returns;0h;"tables of results")],
.sapi.metaReturn`type`description!(98h;"join of the tables");
enlist `client.testAPI
];
For more on creating and registering custom APIs see Custom APIs.
Update manifest.yaml
Add the following entrypoints to the manifest.yaml:
aggregator: src/client-agg.q
data-access: src/client-da.q
Package client-lib
> kxi package packit client-lib --tag
Refreshing package before packing...
Writing client-lib/manifest.yaml
Creating package from client-lib
Package created: /tmp/artifacts/client-lib-0.0.1.kxi
client-lib-0.0.1.kxi
>
Update fsi-data-assembly to Load client-lib
In the fsi-data-assembly
package, update the KXI_PACKAGES
env in fsi-data-assembly/databases/shards/fsi-data-assembly-db-shard.yaml
to the following:
- name: KXI_PACKAGES
value: client-lib:0.0.1,fsi-lib:1.0.0
Update Aggregator KXI_PACKAGES Variable
As discussed here, the KXI_PACKAGES
env needs to be updated to this in the aggregator:
aggregator:
env:
KXI_PACKAGES: client-lib:0.0.1,fsi-lib:1.0.0
Update Resource Coordinator KXI_PACKAGES Variable
As discussed here, you may want to update the KXI_PACKAGES
env on the resource coordinator:
resource-coordinator:
env:
KXI_PACKAGES: client-lib:0.0.1,fsi-lib:1.0.0
Push the Package
> kxi package push --force client-lib/0.0.1
{
"client-lib": [
{
"version": "0.0.1",
"_status": "InstallationStatus.SUCCESS"
}
]
}
>
.client.testAPI REST call
Given the fsi-data-assembly
package has been deployed (or a customized renamed version of fsi-data-assembly
as recommended here), generate a token for REST call:
export INSIGHTS_TOKEN=$(curl --header "Content-Type: application/x-www-form-urlencoded" -d "grant_type=client_credentials&client_id=$INSIGHTS_CLIENT_ID&client_secret=$INSIGHTS_CLIENT_SECRET" "${INSIGHTS_URL}/auth/realms/insights/protocol/openid-connect/token" | jq -r '.access_token');
and then make the REST call using the custom API .client.testAPI
:
curl -X POST "${INSIGHTS_URL}/servicegateway/client/testAPI" \
-H "Content-Type: application/json" \
-H "Accepted: application/json" \
-H "Authorization: Bearer $INSIGHTS_TOKEN" \
--data "$(jq -n \
'{
startTS : "2023.04.19D00:00:00.000000000",
endTS : "2023.04.20D00:00:00.000000000"
}' | jq -cr .)"