ICE Consolidated Feed Handler
Getting Started
Get started with one of the following:
Settings
ice
Note
Key terms specific to the ICE Consolidated Feed architecture are highlighted. For more detailed information on these terms please refer to ICE's documentation.
Setting | Command line | Environment Variable | Type | Description |
---|---|---|---|---|
loggingFile | --logging.file |
KXFEED_ICE_LOGGING_FILE |
string | Specifies the log file path for general log messages from the KX feed handler |
apiLoggingFile | --ice.api.log.file |
KXFEED_ICE_API_LOG_FILE |
string | Specifies the log file path for log messages specifically from the ICE API. |
multithreaded | --ice.multithread |
KXFEED_ICE_MULTITHREAD |
boolean | Determines if the ICE API creates multiple threads to handle multiple CSP connections. |
primaryConnection | --ice.primary.connection |
KXFEED_ICE_PRIMARY_CONNECTION |
string | Specifies the primary connection addresses to the ICE feed in the format ip1:port1,ip2:port2 . |
backupConnection | --ice.backup.connection |
KXFEED_ICE_BACKUP_CONNECTION |
string | Specifies the backup connection addresses to the ICE feed, in the format ip1:port1,ip2:port2 .The number of backup connections must match the number of primary connections. |
subscriptionList | --ice.subscription.list |
KXFEED_ICE_SUBSCRIPTION_LIST |
string | Specifies the list of instruments to subscribe to. The special syntax for this field is discussed in Subscription List Syntax. |
logMsg | --ice.log.msg |
KXFEED_ICE_LOG_MESSAGE |
boolean | Indicates whether to log incoming messages from the ICE feed. |
token2typeLookup | --ice.token2type.lookup |
KXFEED_ICE_TOKEN2TYPE_LOOKUP |
string | Specifies the file path to a JSON file containing key-value pairs that map tokens to field names, for example: {"1":"EVENT.DATETIME,DateTime","2":"EVENT.DATETIME.IND,Integer"} . |
tradeTableTokens | --ice.trade.fields |
KXFEED_ICE_TRADE_FIELDS |
string | Specifies the list of tokens to capture in the trade message. |
quoteTableTokens | --ice.quote.fields |
KXFEED_ICE_QUOTE_FIELDS |
string | Specifies the list of tokens to capture in the quote message. |
tradeMessageIdentifiers | --ice.trade.message.identifiers |
KXFEED_ICE_TRADE_MESSAGE_IDENTIFIERS |
string | Specifies a list of comma-separated tokens used to identify trade messages (e.g., 17,18 ). Messages containing these tokens are treated as trade messages. Other messages are published to the quote table unless marked with event type REFRESH, refer to the refreshTable setting). |
enableFiltertokens | --ice.enable.filtertokens |
KXFEED_ICE_ENABLE_FILTERTOKENS |
boolean | Enables ICE token filtration to ensure only tokens specified in tradeTableTokens and quoteTableTokens are published by the ICE feed to the KX ICE Consolidated Feed Handler. This uses ICE's SELECTUSERFILTERTOKENS API. |
refreshTable | --ice.refresh.table |
KXFEED_ICE_REFRESH_TABLE |
boolean | If set to true, messages with event type REFRESH are published to a table called refresh. Otherwise, these messages are ignored. |
restServer | --rest.server |
REST_SERVER |
boolean | Enables or disables sending ICE commands via REST. If enabled, a REST server starts on port 8080 on the host where the KX feed handler runs. Only POST requests are accepted. Example requests include:
|
sourceTimeZone | --ice.source.time.zone |
KXFEED_ICE_SOURCE_TIME_ZONE |
string | Specifies a list of key-value pairs mapping ICE source IDs to specific time zones. The KX feed handler normalizes all time formats to UTC based on the source time zone of each message. Use the format source1:TZ1;source2:TZ2 . Time zones must be in POSIX time format for example: EST5EDT,M3.2.0,M11.1.0 . |
dbConfigFile | --db.config.file |
KXFEED_ICE_DB_CONFIG_FILE |
string | Specifies the path to the database connection configuration file. If publishing data directly to a q-process, configure this by setting "isDirectConnect":1 . |
rt
Info
The following settings apply only to kdb Insights Enterprise deployments.
Settings for configuring the RT publisher within the KX ICE Consolidated Feed Handler.
Field | Type | Description |
---|---|---|
sinkName | string | The name of the RT stream to publish to. |
logLevel | string | The logging verbosity of the RT publisher (INFO, WARN, ERROR, OFF). |
logPath | string | The mount path of the PVC where the RT publisher stores local log files. |
volCapacity | string | The size of the PVC that the RT publisher uses to store local files. |
Persistence
Note
These settings are only relevant for kdb Insights Enterprise deployments.
Field | Type | Description |
---|---|---|
enabled | boolean | Enable or disable persistent storage, claimed via PVCs. |
useLocalValues | boolean | Override global accessMode and storageClass . |
storageClass | string | The storage class to apply to the PVC. |
storageSize | string | The requested size of the volume. |
accessModes | string | A list of desired access modes for the PVC. |
Resources
Note
These settings are only relevant for kdb Insights Enterprise deployments.
Field | Description |
---|---|
limits | The maximum amount of resources that can be requested. |
requests | The minimum amount of resources that must be requested. |
For more information, refer to the RT publisher documentation. | |
## Tokens |
The ICE Consolidated Feed uses a protocol called the CTF wire protocol when sending data to downstream subscribers. Within this protocol, each field published by the ICE Consolidated Feed is assigned a distinct integer value called a token
. The KX ICE Consolidated Feed Handler receives these tokens
to identify fields in incoming messages. The token2typeLookup
configuration maps each token
to the correct field name.
Constructing the Trade and Quote Schemas
The KX ICE Consolidated Feed Handler publishes two primary tables: the trade
table and the quote
table. Both tables share a set of base fields included in every new message. These fields are:
- msgType: The ICE message event type.
- sym: Symbol ticker corresponding to the instrument for which an event has occurred.
- received: The time the message was received by the KX ICE Consilidated Feed Handler. This time is generated by the feed handler and does not come from the ICE Consolidated Feed.
- srcID: ID that identifies the source of the message, for example, London Stock Exchange Domestic Level 1 data is on Source ID 755. Refer to ICE's documentation for more information.
- permission: Permission ID for the event. Refer to ICE's documentation for more details.
All other fields published by the KX ICE Consolidated Feed Handler are specified by the user through the tradeTableTokens
and quoteTableTokens
configuration settings. The tokens listed in these settings are appended to the schemas of their respective tables.
To determine which messages belong in each table, the KX ICE Consolidated Feed Handler uses the tradeMessageIdentifiers
setting. In this setting, you specify tokens that indicate a message is a trade message. For example, token 18
corresponds to the field TRADE.DATETIME
, which typically appears only in trade messages, making it a good identifier for trade messages. Messages that do not contain at least one token from tradeMessageIdentifiers
are classified as quote
messages, unless they have the event type REFRESH
. In such cases, the message is either published to the refresh table or discarded, depending on the refreshTable
setting.
Refresh Table
The KX ICE Consolidated Feed Handler publishes a special table called the refresh
table. This table captures ICE refresh messages, which are periodically published by the ICE Consolidated Feed to correct data.
The schema for the refresh
table is a union of the trade
and quote
schemas. This ensures that all corrections made to tokens associated with the trade
or quote
table are captured in the refresh
table.
You can enable or disable the refresh
table using the refreshTable
configuration setting. If the refreshTable
is turned off, refresh messages are discarded.
For more information about refresh messages, refer to ICE's documentation.
Subscription List Syntax
The subscriptionList
setting uses the format COMMAND1,PARM1:VAL1,PARM2:VAL2,PARM3:VAL3;COMMAND2,PARM1:VAL1,PARM2:VAL2,PARM3:VAL3;...
. Multiple commands can be sent in a single setting.
Examples
Subscribe to a specific source and symbol:
SUBSCRIBE,ENUM_SRC_ID:558,SYMBOL_TICKER:IBM
Subscribe using a wildcard for symbols:
SUBSCRIBEWILDCARD,ENUM_SRC_ID:326,SYMBOL_TICKER:{^[a-zA-Z].*}
Send multiple commands together using a semicolon (;) separator:
SUBSCRIBE,ENUM_SRC_ID:558,SYMBOL_TICKER:IBM;SUBSCRIBEWILDCARD,ENUM_SRC_ID:326,SYMBOL_TICKER:{^[a-zA-Z].*}
Supported Commands
SUBSCRIBE, UNSUBSCRIBE, QUERYDEPTH, QUERYSNAP, QUERYWILDCARD, QUERYDEPTHANDSUBSCRIBE, QUERYSNAPANDSUBSCRIBE, SELECTUSERFILTERTOKENS, SUBSCRIBEWILDCARD, UNSUBSCRIBEWILDCARD
Supported Parameters
CUSIP, SEDOL, ISIN, ENUM_SRC_ID, ENUM_SRC_UNDERLYING_ID, SYMBOL_TICKER, SYMBOL_UNDERLYING_TICKER, CONFLATION, CTF_TOKEN_NAME, CTF_TOKEN_NUM, SYMBOL_BLOOMBERG_TICKER, SYMBOL_ESIGNAL_TICKER, PRODUCT_ROOT, DEPTH_TYPE, USER_NAME, CONFLATION_INTERVAL, CTF_FILTER_ID, QUERY_REF_TAG
For more information on these commands and parameters, refer to ICE's documentation.