Skip to content

ICE Consolidated Feed Handler

Getting Started

Get started with one of the following:

Settings

ice

Note

Key terms specific to the ICE Consolidated Feed architecture are highlighted. For more detailed information on these terms please refer to ICE's documentation.

Setting Command line Environment Variable Type Description
loggingFile --logging.file KXFEED_ICE_LOGGING_FILE string Specifies the log file path for general log messages from the KX feed handler
apiLoggingFile --ice.api.log.file KXFEED_ICE_API_LOG_FILE string Specifies the log file path for log messages specifically from the ICE API.
multithreaded --ice.multithread KXFEED_ICE_MULTITHREAD boolean Determines if the ICE API creates multiple threads to handle multiple CSP connections.
primaryConnection --ice.primary.connection KXFEED_ICE_PRIMARY_CONNECTION string Specifies the primary connection addresses to the ICE feed in the format ip1:port1,ip2:port2.
backupConnection --ice.backup.connection KXFEED_ICE_BACKUP_CONNECTION string Specifies the backup connection addresses to the ICE feed, in the format ip1:port1,ip2:port2.The number of backup connections must match the number of primary connections.
subscriptionList --ice.subscription.list KXFEED_ICE_SUBSCRIPTION_LIST string Specifies the list of instruments to subscribe to. The special syntax for this field is discussed in Subscription List Syntax.
logMsg --ice.log.msg KXFEED_ICE_LOG_MESSAGE boolean Indicates whether to log incoming messages from the ICE feed.
token2typeLookup --ice.token2type.lookup KXFEED_ICE_TOKEN2TYPE_LOOKUP string Specifies the file path to a JSON file containing key-value pairs that map tokens to field names, for example: {"1":"EVENT.DATETIME,DateTime","2":"EVENT.DATETIME.IND,Integer"}.
tradeTableTokens --ice.trade.fields KXFEED_ICE_TRADE_FIELDS string Specifies the list of tokens to capture in the trade message.
quoteTableTokens --ice.quote.fields KXFEED_ICE_QUOTE_FIELDS string Specifies the list of tokens to capture in the quote message.
tradeMessageIdentifiers --ice.trade.message.identifiers KXFEED_ICE_TRADE_MESSAGE_IDENTIFIERS string Specifies a list of comma-separated tokens used to identify trade messages (e.g., 17,18). Messages containing these tokens are treated as trade messages. Other messages are published to the quote table unless marked with event type REFRESH, refer to the refreshTable setting).
enableFiltertokens --ice.enable.filtertokens KXFEED_ICE_ENABLE_FILTERTOKENS boolean Enables ICE token filtration to ensure only tokens specified in tradeTableTokens and quoteTableTokens are published by the ICE feed to the KX ICE Consolidated Feed Handler. This uses ICE's SELECTUSERFILTERTOKENS API.
refreshTable --ice.refresh.table KXFEED_ICE_REFRESH_TABLE boolean If set to true, messages with event type REFRESH are published to a table called refresh. Otherwise, these messages are ignored.
restServer --rest.server REST_SERVER boolean Enables or disables sending ICE commands via REST. If enabled, a REST server starts on port 8080 on the host where the KX feed handler runs. Only POST requests are accepted. Example requests include:
  • curl -X POST localhost:8080 -d "SUBSCRIBEWILDCARD,ENUM_SRC_ID:326,SYMBOL_TICKER:{^[a-zA-Z].*}"
  • curl -X POST localhost:8080 -d "SUBSCRIBE,ENUM_SRC_ID:326,SYMBOL_TICKER:IBM"
  • curl -X POST localhost:8080 -d "UNSUBSCRIBE,ENUM_SRC_ID:558,SYMBOL_TICKER:IBM"
sourceTimeZone --ice.source.time.zone KXFEED_ICE_SOURCE_TIME_ZONE string Specifies a list of key-value pairs mapping ICE source IDs to specific time zones. The KX feed handler normalizes all time formats to UTC based on the source time zone of each message. Use the format source1:TZ1;source2:TZ2. Time zones must be in POSIX time format for example: EST5EDT,M3.2.0,M11.1.0.
dbConfigFile --db.config.file KXFEED_ICE_DB_CONFIG_FILE string Specifies the path to the database connection configuration file. If publishing data directly to a q-process, configure this by setting "isDirectConnect":1.

rt

Info

The following settings apply only to kdb Insights Enterprise deployments.

Settings for configuring the RT publisher within the KX ICE Consolidated Feed Handler.

Field Type Description
sinkName string The name of the RT stream to publish to.
logLevel string The logging verbosity of the RT publisher (INFO, WARN, ERROR, OFF).
logPath string The mount path of the PVC where the RT publisher stores local log files.
volCapacity string The size of the PVC that the RT publisher uses to store local files.

Persistence

Note

These settings are only relevant for kdb Insights Enterprise deployments.

Field Type Description
enabled boolean Enable or disable persistent storage, claimed via PVCs.
useLocalValues boolean Override global accessMode and storageClass.
storageClass string The storage class to apply to the PVC.
storageSize string The requested size of the volume.
accessModes string A list of desired access modes for the PVC.

Resources

Note

These settings are only relevant for kdb Insights Enterprise deployments.

Field Description
limits The maximum amount of resources that can be requested.
requests The minimum amount of resources that must be requested.
For more information, refer to the RT publisher documentation.
## Tokens

The ICE Consolidated Feed uses a protocol called the CTF wire protocol when sending data to downstream subscribers. Within this protocol, each field published by the ICE Consolidated Feed is assigned a distinct integer value called a token. The KX ICE Consolidated Feed Handler receives these tokens to identify fields in incoming messages. The token2typeLookup configuration maps each token to the correct field name.

Constructing the Trade and Quote Schemas

The KX ICE Consolidated Feed Handler publishes two primary tables: the trade table and the quote table. Both tables share a set of base fields included in every new message. These fields are:

  • msgType: The ICE message event type.
  • sym: Symbol ticker corresponding to the instrument for which an event has occurred.
  • received: The time the message was received by the KX ICE Consilidated Feed Handler. This time is generated by the feed handler and does not come from the ICE Consolidated Feed.
  • srcID: ID that identifies the source of the message, for example, London Stock Exchange Domestic Level 1 data is on Source ID 755. Refer to ICE's documentation for more information.
  • permission: Permission ID for the event. Refer to ICE's documentation for more details.

All other fields published by the KX ICE Consolidated Feed Handler are specified by the user through the tradeTableTokens and quoteTableTokens configuration settings. The tokens listed in these settings are appended to the schemas of their respective tables.

To determine which messages belong in each table, the KX ICE Consolidated Feed Handler uses the tradeMessageIdentifiers setting. In this setting, you specify tokens that indicate a message is a trade message. For example, token 18 corresponds to the field TRADE.DATETIME, which typically appears only in trade messages, making it a good identifier for trade messages. Messages that do not contain at least one token from tradeMessageIdentifiers are classified as quote messages, unless they have the event type REFRESH. In such cases, the message is either published to the refresh table or discarded, depending on the refreshTable setting.

Refresh Table

The KX ICE Consolidated Feed Handler publishes a special table called the refresh table. This table captures ICE refresh messages, which are periodically published by the ICE Consolidated Feed to correct data.

The schema for the refresh table is a union of the trade and quote schemas. This ensures that all corrections made to tokens associated with the trade or quote table are captured in the refresh table.

You can enable or disable the refresh table using the refreshTable configuration setting. If the refreshTable is turned off, refresh messages are discarded.

For more information about refresh messages, refer to ICE's documentation.

Subscription List Syntax

The subscriptionList setting uses the format COMMAND1,PARM1:VAL1,PARM2:VAL2,PARM3:VAL3;COMMAND2,PARM1:VAL1,PARM2:VAL2,PARM3:VAL3;.... Multiple commands can be sent in a single setting.

Examples

Subscribe to a specific source and symbol:

SUBSCRIBE,ENUM_SRC_ID:558,SYMBOL_TICKER:IBM

Subscribe using a wildcard for symbols:

SUBSCRIBEWILDCARD,ENUM_SRC_ID:326,SYMBOL_TICKER:{^[a-zA-Z].*}

Send multiple commands together using a semicolon (;) separator:

SUBSCRIBE,ENUM_SRC_ID:558,SYMBOL_TICKER:IBM;SUBSCRIBEWILDCARD,ENUM_SRC_ID:326,SYMBOL_TICKER:{^[a-zA-Z].*}

Supported Commands

SUBSCRIBE, UNSUBSCRIBE, QUERYDEPTH, QUERYSNAP, QUERYWILDCARD, QUERYDEPTHANDSUBSCRIBE, QUERYSNAPANDSUBSCRIBE, SELECTUSERFILTERTOKENS, SUBSCRIBEWILDCARD, UNSUBSCRIBEWILDCARD

Supported Parameters

CUSIP, SEDOL, ISIN, ENUM_SRC_ID, ENUM_SRC_UNDERLYING_ID, SYMBOL_TICKER, SYMBOL_UNDERLYING_TICKER, CONFLATION, CTF_TOKEN_NAME, CTF_TOKEN_NUM, SYMBOL_BLOOMBERG_TICKER, SYMBOL_ESIGNAL_TICKER, PRODUCT_ROOT, DEPTH_TYPE, USER_NAME, CONFLATION_INTERVAL, CTF_FILTER_ID, QUERY_REF_TAG

For more information on these commands and parameters, refer to ICE's documentation.