ICE Consolidated Feed Handler

Getting Started

Get started with one of the following:

Settings

ice

Note

Key terms specific to the ICE Consolidated Feed architecture are highlighted. For more detailed information on these terms please refer to ICE's documentation.

Setting	Command line	Environment Variable	Type	Description
loggingFile	`--logging.file`	`KXFEED_ICE_LOGGING_FILE`	string	Specifies the log file path for general log messages from the KX feed handler
apiLoggingFile	`--ice.api.log.file`	`KXFEED_ICE_API_LOG_FILE`	string	Specifies the log file path for log messages specifically from the ICE API.
multithreaded	`--ice.multithread`	`KXFEED_ICE_MULTITHREAD`	boolean	Determines if the ICE API creates multiple threads to handle multiple *CSP* connections.
primaryConnection	`--ice.primary.connection`	`KXFEED_ICE_PRIMARY_CONNECTION`	string	Specifies the primary connection addresses to the ICE feed in the format `ip1:port1,ip2:port2`.
backupConnection	`--ice.backup.connection`	`KXFEED_ICE_BACKUP_CONNECTION`	string	Specifies the backup connection addresses to the ICE feed, in the format `ip1:port1,ip2:port2`.The number of backup connections must match the number of primary connections.
subscriptionList	`--ice.subscription.list`	`KXFEED_ICE_SUBSCRIPTION_LIST`	string	Specifies the list of instruments to subscribe to. The special syntax for this field is discussed in Subscription List Syntax.
logMsg	`--ice.log.msg`	`KXFEED_ICE_LOG_MESSAGE`	boolean	Indicates whether to log incoming messages from the ICE feed.
token2typeLookup	`--ice.token2type.lookup`	`KXFEED_ICE_TOKEN2TYPE_LOOKUP`	string	Specifies the file path to a JSON file containing key-value pairs that map tokens to field names, for example: `{"1":"EVENT.DATETIME,DateTime","2":"EVENT.DATETIME.IND,Integer"}`.
tradeTableTokens	`--ice.trade.fields`	`KXFEED_ICE_TRADE_FIELDS`	string	Specifies the list of tokens to capture in the trade message.
quoteTableTokens	`--ice.quote.fields`	`KXFEED_ICE_QUOTE_FIELDS`	string	Specifies the list of tokens to capture in the quote message.
tradeMessageIdentifiers	`--ice.trade.message.identifiers`	`KXFEED_ICE_TRADE_MESSAGE_IDENTIFIERS`	string	Specifies a list of comma-separated tokens used to identify trade messages (e.g., `17,18`). Messages containing these tokens are treated as trade messages. Other messages are published to the quote table unless marked with event type *REFRESH*, refer to the `refreshTable` setting).
enableFiltertokens	`--ice.enable.filtertokens`	`KXFEED_ICE_ENABLE_FILTERTOKENS`	boolean	Enables *ICE token filtration* to ensure only tokens specified in `tradeTableTokens` and `quoteTableTokens` are published by the ICE feed to the KX ICE Consolidated Feed Handler. This uses ICE's `SELECTUSERFILTERTOKENS` API.
refreshTable	`--ice.refresh.table`	`KXFEED_ICE_REFRESH_TABLE`	boolean	If set to true, messages with event type *REFRESH* are published to a table called refresh. Otherwise, these messages are ignored.
restServer	`--rest.server`	`REST_SERVER`	boolean	Enables or disables sending *ICE commands* via REST. If enabled, a REST server starts on port 8080 on the host where the KX feed handler runs. Only `POST` requests are accepted. Example requests include: `curl -X POST localhost:8080 -d "SUBSCRIBEWILDCARD,ENUM_SRC_ID:326,SYMBOL_TICKER:{^[a-zA-Z].*}"` `curl -X POST localhost:8080 -d "SUBSCRIBE,ENUM_SRC_ID:326,SYMBOL_TICKER:IBM"` `curl -X POST localhost:8080 -d "UNSUBSCRIBE,ENUM_SRC_ID:558,SYMBOL_TICKER:IBM"`
sourceTimeZone	`--ice.source.time.zone`	`KXFEED_ICE_SOURCE_TIME_ZONE`	string	Specifies a list of key-value pairs mapping *ICE source IDs* to specific time zones. The KX feed handler normalizes all time formats to `UTC` based on the source time zone of each message. Use the format `source1:TZ1;source2:TZ2`. Time zones must be in POSIX time format for example: `EST5EDT,M3.2.0,M11.1.0`.
dbConfigFile	`--db.config.file`	`KXFEED_ICE_DB_CONFIG_FILE`	string	Specifies the path to the database connection configuration file. If publishing data directly to a q-process, configure this by setting `"isDirectConnect":1`.

rt

Info

The following settings apply only to kdb Insights Enterprise deployments.

Settings for configuring the RT publisher within the KX ICE Consolidated Feed Handler.

Field	Type	Description
sinkName	string	The name of the RT stream to publish to.
logLevel	string	The logging verbosity of the RT publisher (INFO, WARN, ERROR, OFF).
logPath	string	The mount path of the PVC where the RT publisher stores local log files.
volCapacity	string	The size of the PVC that the RT publisher uses to store local files.

Persistence

Note

These settings are only relevant for kdb Insights Enterprise deployments.

Field	Type	Description
enabled	boolean	Enable or disable persistent storage, claimed via PVCs.
useLocalValues	boolean	Override global `accessMode` and `storageClass`.
storageClass	string	The storage class to apply to the PVC.
storageSize	string	The requested size of the volume.
accessModes	string	A list of desired access modes for the PVC.

Resources

Note

These settings are only relevant for kdb Insights Enterprise deployments.

Field	Description
limits	The maximum amount of resources that can be requested.
requests	The minimum amount of resources that must be requested.
For more information, refer to the RT publisher documentation.
## Tokens

The ICE Consolidated Feed uses a protocol called the CTF wire protocol when sending data to downstream subscribers. Within this protocol, each field published by the ICE Consolidated Feed is assigned a distinct integer value called a token. The KX ICE Consolidated Feed Handler receives these tokens to identify fields in incoming messages. The token2typeLookup configuration maps each token to the correct field name.

Constructing the Trade and Quote Schemas

The KX ICE Consolidated Feed Handler publishes two primary tables: the trade table and the quote table. Both tables share a set of base fields included in every new message. These fields are:

msgType: The ICE message event type.
sym: Symbol ticker corresponding to the instrument for which an event has occurred.
received: The time the message was received by the KX ICE Consilidated Feed Handler. This time is generated by the feed handler and does not come from the ICE Consolidated Feed.
srcID: ID that identifies the source of the message, for example, London Stock Exchange Domestic Level 1 data is on Source ID 755. Refer to ICE's documentation for more information.
permission: Permission ID for the event. Refer to ICE's documentation for more details.

All other fields published by the KX ICE Consolidated Feed Handler are specified by the user through the tradeTableTokens and quoteTableTokens configuration settings. The tokens listed in these settings are appended to the schemas of their respective tables.

To determine which messages belong in each table, the KX ICE Consolidated Feed Handler uses the tradeMessageIdentifiers setting. In this setting, you specify tokens that indicate a message is a trade message. For example, token 18 corresponds to the field TRADE.DATETIME, which typically appears only in trade messages, making it a good identifier for trade messages. Messages that do not contain at least one token from tradeMessageIdentifiers are classified as quote messages, unless they have the event type REFRESH. In such cases, the message is either published to the refresh table or discarded, depending on the refreshTable setting.

Refresh Table

The KX ICE Consolidated Feed Handler publishes a special table called the refresh table. This table captures ICE refresh messages, which are periodically published by the ICE Consolidated Feed to correct data.

The schema for the refresh table is a union of the trade and quote schemas. This ensures that all corrections made to tokens associated with the trade or quote table are captured in the refresh table.

You can enable or disable the refresh table using the refreshTable configuration setting. If the refreshTable is turned off, refresh messages are discarded.

For more information about refresh messages, refer to ICE's documentation.

Subscription List Syntax

The subscriptionList setting uses the format COMMAND1,PARM1:VAL1,PARM2:VAL2,PARM3:VAL3;COMMAND2,PARM1:VAL1,PARM2:VAL2,PARM3:VAL3;.... Multiple commands can be sent in a single setting.

Examples

Subscribe to a specific source and symbol:

SUBSCRIBE,ENUM_SRC_ID:558,SYMBOL_TICKER:IBM

Subscribe using a wildcard for symbols:

SUBSCRIBEWILDCARD,ENUM_SRC_ID:326,SYMBOL_TICKER:{^[a-zA-Z].*}

Send multiple commands together using a semicolon (;) separator:

SUBSCRIBE,ENUM_SRC_ID:558,SYMBOL_TICKER:IBM;SUBSCRIBEWILDCARD,ENUM_SRC_ID:326,SYMBOL_TICKER:{^[a-zA-Z].*}

Supported Commands

SUBSCRIBE, UNSUBSCRIBE, QUERYDEPTH, QUERYSNAP, QUERYWILDCARD, QUERYDEPTHANDSUBSCRIBE, QUERYSNAPANDSUBSCRIBE, SELECTUSERFILTERTOKENS, SUBSCRIBEWILDCARD, UNSUBSCRIBEWILDCARD

Supported Parameters

CUSIP, SEDOL, ISIN, ENUM_SRC_ID, ENUM_SRC_UNDERLYING_ID, SYMBOL_TICKER, SYMBOL_UNDERLYING_TICKER, CONFLATION, CTF_TOKEN_NAME, CTF_TOKEN_NUM, SYMBOL_BLOOMBERG_TICKER, SYMBOL_ESIGNAL_TICKER, PRODUCT_ROOT, DEPTH_TYPE, USER_NAME, CONFLATION_INTERVAL, CTF_FILTER_ID, QUERY_REF_TAG

For more information on these commands and parameters, refer to ICE's documentation.