API Configuration

Some features of the API calls either must have, or can have configuration to be available for use. This page documents the configuration that can be set for these API parameters.

Depending on the feature, this is either configured by a q variable loaded into the system, or by populating a reference table in the assembly.

Configuration defined using q variables

These q variables can be packaged up in a custom package. Refer to the packaging documentation.

Alternatively, these variables can be edited in the existing fsi-lib package itself. This should be done before deploying any assemblies. To use this method, follow these steps:

• unpack the fsi-lib package
• edit the src/fsi.config.q file and update the variables to the desired values
• re-package the fsi-lib package
• push the fsi-lib package
• bounce the aggregators and resource-coordinators

Commands to unpack and re-package a package are found here.

Commands to push a package can be found in the packaging documentation.

idCol / idList

If no configuration is provided, and idCol is not provided in the API call, it looks for an instrumentID column. If not found and no configuration is provided, it uses the leftmost symbol type column. If no symbol is found it then looks for guids, then strings and then any type that is not temporal.

`.api.cfg.IdCols` can be appended to in a client specified q library loaded into the DAPs and aggregator. This is a two column table, or table names and idCols.

For example, some out of the box definitions are:

.api.cfg.IdCols:flip `table`idCol ! flip (
    (`Quote;                `instrumentID);
    (`Trade;                `instrumentID);
    (`Order;                `instrumentID);
    (`Contract;             `contractID);
    (`CanCor;               `instrumentID);
    ...

timeCol time windowing

When a timeCol that is not the default partition column is provided, more partitions must be scanned to find data within the given time window. Doing this is less performant. Therefore the buffer for how many days either side of the provided window are checked is configurable.

By default it is 7 days.

If the use case is to query on non partition time columns, but the timestamps are guaranteed to be in the same partition, then this can be set to 0 for optimal performance.

If timeCol is not provided in query then this makes no difference to performance regardless of the setting.

.api.cfg.timeColQueryBuffer:1

Configuration defined using reference table population

idMapping

The idMapping functionality depends on a populated IDMap table. This is a reference table that exists in all DAPs. The content should be as follows:

Header	Type	Description
queryIdentifier	symbol	The identifier that is provided in the API call
mappedIdentifier	symbol (list)	The underlying identifier(s) that is queried
alwaysMap	boolean	For this querySymbol, always do idMapping regardless of API flag
returnAsMappedIdentifier	boolean	Return in the data as the underlying identifier, rather than the querySymbol

The corresponding yaml for this table is:

- name: IDMap
  columns:
  - description: Identifier that will be provided on query
    name: queryIdentifier
    type: symbol
  - description: Identifier in the source data
    name: mappedIdentifier
    type: symbols
  - description: Flag for if this is always mapped or done with the API flag
    name: alwaysMap
    type: boolean
  - description: Flag for if to returning data in source ID format
    name: returnAsMappedIdentifier
    type: boolean
  description: Identifier mapping reference table
  type: splayed

filterRule

Similarly to idMapping, the filterRule functionality depends on a reference table called FilterRules of format:

Column	Type	Description
tableName	symbol	Name of the table that is filtered
tradedExchange	symbol	Name of the exchange in the table
condition	string	The condition to map against
include	symbol[]	List of filter names that include this condition/exchange combination

The corresponding yaml for this table is:

- name: FilterRules
  columns:
  - description: Table name that the filter applies to
    name: tableName
    type: symbol
  - description: Exchange that this condition is found on
    name: tradedExchange
    type: symbol
  - description: The condition code to match
    name: condition
    type: string
  - description: Filter rules that include this condition
    name: include
    type: symbols
  description: Filter rule configuration reference table
  type: splayed

Symbology resolution and continuity

To use the symbology resolution and continuity arguments (idTS and idSymbology) in the fsi-lib API, configure the symbology of the underlying ticker in the queried database. Set the symbology by using the FSI_DATA_SYMBOLOGY environment variable.

The default value is DTKR, which represents the database ticker.

If a query doesn't include idTS or idSymbology, the query doesn't use the FSI_DATA_SYMBOLOGY environment variable.

For DB Service, set FSI_DATA_SYMBOLOGY in the DAP section of the Docker Compose file. For kdb Insights Enterprise, set it in the DAP section of the database shard YAML file.

Symbology reference data

To use idTS and idSymbology, create and populate a Symbology reference table on every DAP. Use the following table format:

Column	Type	Description
oid	integer	Identifies the security by its internal object identifier (OID)
source_exchange	symbol	The exchange associated with the source symbol for this mapping period, specified only if the mapping differs by exchange
start	timestamp	Start of the time period during which this symbol mapping was valid
end	timestamp	End of the time period during which this symbol mapping was valid
mapped_symbol	symbol	The destination symbol name this OID maps to in the target symbology (e.g. a Bloomberg ticker or CUSIP)
dest_exchange	symbol	The exchange associated with the destination symbol for this mapping period
symbology	symbol	The destination symbology (e.g. Bloomberg Symbol, Ticker, CUSIP) for symbol translation

The following YAML defines the table:

- name: Symbology
  columns:
  - description: Internal integer symbology (OID) identifying the security.
    name: oid
    type: int
  - description: The exchange associated with the source symbol for this mapping period, specified only if the mapping itself differs by exchange.
    name: source_exchange
    type: symbol
  - description: Start of the time period (GMT) during which this symbol mapping was valid.
    name: start
    type: timestamp
  - description: End of the time period (GMT) during which this symbol mapping was valid.
    name: end
    type: timestamp
  - description: The destination symbol name this OID maps to in the target symbology (e.g. a Bloomberg ticker or CUSIP).
    name: mapped_symbol
    type: symbol
  - description: The exchange associated with the destination symbol for this mapping period.
    name: dest_exchange
    type: symbol
  - description: The destination symbology (target symbol-naming convention, e.g. Bloomberg Symbol, Ticker, CUSIP) for symbol translation.
    name: symbology
    type: symbol
  description: Symbol mapping reference data, translating internal OID symbology to external symbologies (e.g. Bloomberg, CUSIP, ticker) over time.
  type: basic

Example: symbology

Meta (former Facebook) changed its ticker from FB to META on June 9, 2022. Trades from before the change are stored under FB, and trades from after the change are stored under META.

Query META over the change without idSymbology. Only the days when the ticker was META are returned:

args:(!) . flip (
    (`table;`Trade);
    (`idList;enlist `META);
    (`startTS;2022.06.07D);
    (`endTS;2022.06.11D)
    );

noSym:last h(`.fsi.getTicks;args;`;()!())
select n:count i by instrumentID from noSym

instrumentID| n
------------| -----
META        | 10284

Add idSymbology to look the security up in the Symbology table and return the data stored under every ticker it has had. sourceId shows the ticker each row was stored under:

args:(!) . flip (
    (`table;`Trade);
    (`idList;enlist `META);
    (`startTS;2022.06.07D);
    (`endTS;2022.06.11D);
    (`idSymbology;`DTKR);
    (`columns;`eventTimestamp`instrumentID`price`sourceId)
    );

withSym:last h(`.fsi.getTicks;args;`;()!())
select n:count i by instrumentID,sourceId from withSym

instrumentID sourceId| n
---------------------| -----
META         FB      | 9594
META         META    | 10284

All rows are returned under the one id, META.

You can set idSymbology to any value in the symbology column of the Symbology table. For example, the following query uses a Bloomberg identifier:

args:(!) . flip (
    (`table;`Trade);
    (`idList;enlist `$"META US Equity");
    (`startTS;2022.06.07D);
    (`endTS;2022.06.11D);
    (`idSymbology;`BSYM)
    );

bbg:last h(`.fsi.getTicks;args;`;()!())
select n:count i by instrumentID from bbg

instrumentID  | n
--------------| -----
META US Equity| 19878

Corporate actions

Use the applyCorpActions argument to apply corporate action adjustments. This argument supports the following corporate action types:

• cashDividend
• split
• spinoff
• stockDividend
• securitySplice

To add another corporate action type, update the .api.corpActions.map dictionary in fsi-lib/src/api.corpActions.config.q. This dictionary maps the values accepted by the API to the adjustment types in the underlying reference data:

.api.corpActions.map:(!) . flip (
        (`cashDividend;         `CASH_DIVIDEND);
        (`split;                `SPLIT);
        (`spinoff;              `SPINOFF);
        (`stockDividend;        `STOCK_DIVIDEND);
        (`securitySplice;       `SECURITY_SPLICE)
        );

If you omit adjustPrice or adjustSize, or set either argument to 1b (boolean true), the function adjusts every column with price or size in its name, respectively.

To configure different default columns for a specific table, update the .api.corpActions.tableFields dictionary in fsi-lib/src/api.corpActions.config.q. The dictionary includes default columns for the Trade and Quote tables:

.api.corpActions.tableFields:1!flip `name`adjustPrice`adjustSize ! flip (
                  (`Trade          ;enlist `price;enlist `size);
                  (`Quote          ;`ask`bid     ;`ask_volume`bid_volume)
    );

When you use applyCorpActions with getStats, aggregations that can run on an individual DAP are applied to the on-disk data before the corporate action adjustments. This behavior prevents the DAP from loading potentially large amounts of data into memory to apply the adjustments before aggregation.

To override this behavior and apply corporate action adjustments before the DAP aggregates the data, set the FSI_CORPACTION_ON_AGG_DATA environment variable to FALSE. For DB Service, set the variable in the DAP section of the Docker Compose file. For kdb Insights Enterprise, set it in the DAP section of the database shard YAML file.

Corporate action reference data

To use applyCorpActions, create and populate a CorpActions reference table on every DAP. Use the following table format:

Column	Type	Description
oid	integer	Internal integer symbology (OID) identifying the security this corporate action applies to
exchange	symbol	The exchange on which the security traded
time	timestamp	The effective date-time from which this corporate action's adjustment applies
mltp_adj	float	Multiplicative adjustment factor - the ratio by which historical prices and/or sizes are scaled to account for this corporate action
add_adj	float	Additive adjustment factor - the fixed amount added to historical prices to account for this corporate action
adj_type	symbol	Type of corporate action - such as SPLIT, STOCK_DIVIDEND, CASH_DIVIDEND and SPINOFF; custom, user-defined adjustment type names are also permitted

The following YAML defines the table:

- name: CorpActions
  columns:
  - description: Internal integer symbology (OID) identifying the security this corporate action applies to.
    name: oid
    type: int
  - description: The exchange on which the security traded.
    name: exchange
    type: symbol
  - description: The effective date-time (GMT) from which this corporate action's adjustment applies.
    name: time
    type: timestamp
  - description: Multiplicative adjustment factor - the ratio by which historical prices and/or sizes are scaled to account for this corporate action.
    name: mltp_adj
    type: float
  - description: Additive adjustment factor - the fixed amount added to historical prices to account for this corporate action.
    name: add_adj
    type: float
  - description: Type of corporate action - such as SPLIT, STOCK_DIVIDEND, CASH_DIVIDEND and SPINOFF; custom, user-defined adjustment type names are also permitted.
    name: adj_type
    type: symbol
  description: Corporate actions reference data, used to adjust historical prices and/or sizes for splits, dividends, spin-offs and similar events.
  type: basic

Example: corporate actions

Apple split 4-for-1 on 31 August 2020. First, query the trades without applying corporate action adjustments:

args:(!) . flip (
    (`table;`Trade);
    (`idList;enlist `AAPL);
    (`startTS;2020.08.27D);
    (`endTS;2020.09.02D);
    (`columns;`eventTimestamp`instrumentID`price`volume)
    );

res:last h(`.fsi.getTicks;args;`;()!())
3#res

eventTimestamp                instrumentID price volume
-------------------------------------------------------
2020.08.27D13:30:00.000476000 AAPL         508.5 203
2020.08.27D13:30:00.000499000 AAPL         508.5 203
2020.08.27D13:30:00.001073000 AAPL         508.5 673

The price is about 500 before the split. Add applyCorpActions to adjust earlier prices and sizes for the split. adjustSize names the column that holds the trade size:

args:(!) . flip (
    (`table;`Trade);
    (`idList;enlist `AAPL);
    (`startTS;2020.08.27D);
    (`endTS;2020.09.02D);
    (`columns;`eventTimestamp`instrumentID`price`volume);
    (`applyCorpActions;enlist `split);
    (`adjustSize;`volume)
    );

adjRes:last h(`.fsi.getTicks;args;`;()!())
3#adjRes

eventTimestamp                instrumentID price   volume
---------------------------------------------------------
2020.08.27D13:30:00.000476000 AAPL         127.125 812
2020.08.27D13:30:00.000499000 AAPL         127.125 812
2020.08.27D13:30:00.001073000 AAPL         127.125 2692

Prices before 31 August are divided by 4 and sizes multiplied by 4, so the whole period is on the same scale. Prices and sizes on and after 31 August are unchanged.

applyCorpActions takes a list of corporate action types, so several can be applied at once, for example (`applyCorpActions; `split`cashDividend).