FIX Feed Handler

Getting Started

Get started with one of the following:

Supported FIX Versions

The KX FIX Feed Handler supports FIX versions 4.0 - 5.0.

Settings

fix

Setting	Command line	Environment Variable	Type	Description
sessionFile	`--fix.session.file`	`KXFEED_FIX_FIX_SESSION_FILE`	string	The settings for each FIX session are configured in this file. More details on this config below.
eventToTableFile	`--fix.eventTableMapping.file`	`KXFEED_FIX_FIX_EVENT2TABLE_FILE`	string	This file maps each table in `dbSchemaFile` to a list of message types (FIX tag 35). The message types are separated by ';' e.g. "V;W;X". The KX FIX Feed Handler uses this mapping to determine which table to use when it receives a new FIX message. Any FIX message types not mapped to a table are ignored.
rejectMessageTypes	`--fix.reject.message_types`	`KXFEED_FIX_FIX_REJECT_MESSAGE_TYPES`	string	List of reject message types, separated by ','. A warning is logged when sending or receiving these message types.
maxNesting	`--fix.max.nesting`	`KXFEED_FIX_FIX_MAX_NESTING`	string	Comma separated key-value pairs where message type is the key and max nesting is the value e.g "J:2,X:3". Max nesting is the maximum depth of repeating groups expected within a FIX message for a particular message type. By default this is set to 1 for all message types, meaning at most 1 level of repeating groups is expected.
messageLogRotationSize	`--fix.log.rotation_size`	`KXFEED_FIX_FIX_LOG_ROTATION_SIZE`	string	Size in bytes. The contents of FIX.*.messages.current.log are deleted when this file size is reached.
messageLogCheckInterval	`--fix.log.check_interval`	`KXFEED_FIX_FIX_LOG_CHECK_INTERVAL`	string	How often (in seconds) to check the size of the message logs FIX.*.messages.current.log.
dbSchemaFile	`--db.schema.file`	`KXFEED_FIX_DB_SCHEMA_FILE`	string	XML file that defines the tables to be published downstream. Discussed further in Schema Config.
dbConfigFile	`--db.config.file`	`KXFEED_FIX_DB_CONFIG_FILE`	string	The path to the database connection configuration file. If publishing data directly to a q-process, configure this by setting `"isDirectConnect":1`.
pubDelay	`--db.pub_delay`	`KXFEED_FIX_DB_PUB_DELAY`	string	Publish delay in milliseconds. Enables message batching to reduce IO. Set to 100 milliseconds by default.
logFile	`--logging.file`	`KXFEED_FIX_LOGGING_FILE`	string	The application log file path for general log messages from the KX FIX feed handler.
logLevel	`--logging.level`	`KXFEED_FIX_LOGGING_LEVEL`	string	Level of logging from the KX FIX feed handler. Options are `trace`, `debug`, `info`, `warning`, `error` and `fatal` (from lowest to highest severity).
logAutoFlush	`--logging.auto_flush`	`KXFEED_FIX_LOGGING_AUTO_FLUSH`	boolean	If true, log messages are immediately written to the log file. When false, log messages are only written once the message buffer becomes full.

rt

Info

The following settings apply only to kdb Insights Enterprise deployments.

Settings for configuring the RT publisher within the KX FIX Feed Handler.

Field	Type	Description
sinkName	string	The name of the RT stream to publish to.
logLevel	string	The logging verbosity of the RT publisher (INFO, WARN, ERROR, OFF).
logPath	string	The mount path of the PVC where the RT publisher stores local log files.
volCapacity	string	The size of the PVC that the RT publisher uses to store local files.

persistence

Info

These settings are only relevant for kdb Insights Enterprise deployments.

Field	Type	Description
enabled	boolean	Enable or disable persistent storage, claimed through PVCs.
useLocalValues	boolean	Override global `accessMode` and `storageClass`.
storageClass	string	The storage class to apply to the PVC.
storageSize	string	The requested size of the volume.
accessModes	string	A list of desired access modes for the PVC.

resources

Info

These settings are only relevant for kdb Insights Enterprise deployments.

Field	Description
limits	The maximum amount of resources that can be requested.
requests	The minimum amount of resources that must be requested.

For more information, refer to the RT publisher documentation.

service

Info

These settings are only relevant for kdb Insights Enterprise deployments.

Field	Description
create	Create a Kubernetes service to expose ports listed in ports. Overrode to false if no ports are listed.
annotations	Annotations to add to the service.
name	The name of the service to use. If not set and create is true, a name is generated using the helm chart full name template.
type	Kubernetes service type - ClusterIP, NodePort, LoadBalancer.

ports

Info

Only relevant for kdb Insights Enterprise deployments.

List of ports to expose in the KX FIX Feed Handler. Important when ConnectionType=acceptor in the FIX session config.

configMap

Info

These settings are only relevant for kdb Insights Enterprise deployments.

Field	Description
create	Create a config map to mount configs in the `config` directory of the KX FIX Feed Handler helm chart. If using an external config map set this to false.
mount	Mount the named config map on the path `/opt/kx/kxfeed_activ/config`.
annotations	Annotations to add to the config map.
name	The name of the config map to use. If not set, a name is generated using the helm chart full name template.
files	List of files in the `config` directory that the config map should mount.

FIX Session Config

Those familiar with QuickFIX should be familiar with the FIX session config. The full documentation of all the settings can be found on the QuickFIX documentation site under Configuration.

In the following sections, we go through the most relevant settings for the KX FIX Feed Handler.

Establishing a FIX Session

When establishing a FIX session with a counterparty, the first consideration is whether the KX FIX Feed Handler or the counterparty is going to be the initiator of the connection.

If the KX FIX Feed Handler is the initiator then ConnectionType must be set to initiator in the session config, and a SocketConnectHost and SocketConnectPort must be specified for the KX FIX Feed Handler to connect to.

If the counterparty is the initiator, then the ConnectionType must be set to acceptor in the session config, and a SocketAcceptPort must be specified for the counterparty to connect to.

Other settings that must be set are:

BeginString indicating the FIX Version of the session.
SenderCompID for the ID of the KX FIX Feed Handler.
TargetCompID for the ID of the counterparty.

What exactly SenderCompID and TargetCompID need to be set to is normally dictated by the counterparty.

Note

The KX FIX Feed Handler can only operate exclusively as either an initiator or an acceptor, even in a multi-session setup. ConnectionType must always be specified in the [DEFAULT] section of the session config.

Once the connection to the counterparty has been established, a logon message (FIX message type 35=A) is sent to the counterparty by the KX FIX Feed Handler (and vice versa). After a successful logon, the KX FIX Feed Handler is ready to receive data from the counterparty.

Multiple FIX Sessions

In the scenario where there are multiple counterparties that the KX FIX Feed Handler needs to connect to, you can define multiple [SESSION] sections in the session config. Here is an example:

[SESSION]
SenderCompID=KX1
TargetCompID=SERVER1
SocketConnectHost=host1
SocketConnectPort=1235
[SESSION]
SenderCompID=KX2
TargetCompID=SERVER2
SocketConnectHost=host2
SocketConnectPort=1236
[SESSION]
SenderCompID=KX3
TargetCompID=SERVER3
SocketConnectHost=host3
SocketConnectPort=1237
[SESSION]
SenderCompID=KX4
TargetCompID=SERVER4
SocketConnectHost=host4
SocketConnectPort=1238

Each session must have a unique SenderCompID and TargetCompID.

Each session is started and managed on their own CPU thread.

Finding the FIX Message Logs

Every message that the KX FIX Feed Handler sends and receives is logged to a file that matches the regex FIX.*.messages.current.log. Each FIX session has its own FIX message log, with the files being distinguished by the SenderCompID and TargetCompID in the file name.

The KX FIX Feed Handler writes these files to the directory specified in the FileLogPath setting.

Other useful files written to this directory are the FIX event logs, which are also created for each session and match the regex FIX.*.event.current.log.

Managing Sequence Numbers

The sequence number (FIX tag 34) of every message that the KX FIX Feed Handler sends and receives is logged to a file that matches the regex FIX.*.seqnums. Each FIX session has its own sequence number log, with the files being distinguished by the SenderCompID and TargetCompID in the file name.

The KX FIX Feed Handler writes these files to the directory specified in the FileStorePath setting.

The contents of the sequence number log file look something like this:

0000000004 : 0000000005

The value on the left hand side of the : is the sender sequence number. This indicates the sequence number of the last message the KX FIX Feed Handler sent to the counterparty. The behavior if this sequence number becomes out of sync with the counterparty is largely dictated by the counterparty. Often a reject FIX message is sent back to the KX FIX Feed Handler with a message resembling "Received message sequence number too low" or "Received message sequence number too high". Resolving this involves communicating with the counterparty to find out the correct sequence number and then manually updating the sequence number file with the correct value.

The value on the right hand side of the : is the target sequence number. This indicates the sequence number of the last message the KX FIX Feed Handler received from the counterparty. If the KX FIX Feed Handler detects that the target sequence number it has in the log file is less than the sequence number from the latest message sent by the counterparty, it triggers a FIX resend request message. How this resend request is responded to depends on the counterparty.

To avoid having to worry about sequence numbers becoming out of sync, set ResetOnLogon=Y in the session config. This resets both sequence numbers to 0000000001 every time a fresh logon is initiated between the KX FIX Feed Handler and the counterparty. Note that this is only an option if the counterparty is also resetting its sequence numbers on every fresh logon too, which is not always possible.

FIX Data Dictionary

The FIX data dictionary is a XML definition file that defines the structure of each FIX message type, primarily the fields and repeating groups it contains, along with the tag numbers associated with each field. The path to this file is set in the session config with the setting DataDictionary.

If the counterparty sends messages that contain repeating groups, then the KX FIX Feed Handler requires that a DataDictionary is specified. Here is an example of a repeating group definition in a data dictionary:

<group name='NoOrders' required='Y'>
 <field name='ClOrdID' required='Y' />
 <field name='CumQty' required='Y' />
 <field name='OrdStatus' required='Y' />
 <field name='LeavesQty' required='Y' />
 <field name='CxlQty' required='Y' />
 <field name='AvgPx' required='Y' />
 <field name='OrdRejReason' required='N' />
 <field name='Text' required='N' />
 <field name='EncodedTextLen' required='N' />
 <field name='EncodedText' required='N' />
</group>

A NUMINGROUP field at the start of every block of repeating groups in a FIX message is mandatory. This field tells the KX FIX Feed Handler how many repeating groups are in the message. For the above repeating group definition, NoOrders is the NUMINGROUP field.

The list of fields contained within the <group> XML tags are the fields that are to be repeated. The first field in the list is known as the delimiter field. In order for repeating groups in a FIX message to be processed correctly, the first field in each repeating group must match the field listed first in the repeating group definition. In the above example, the delimiter field is ClOrdID.

If the counterparty does not send any messages with repeating groups, then the DataDictionary setting is not required and UseDataDictionary can be set to N.

QuickFIX provides example data dictionaries for each FIX version here.

Schema Config

The schema configuration defines the structure of the tables that are published downstream by the KX FIX Feed Handler. Here's an example of a table definition:

<table>
  <name>Trade</name>
  <columns>
    <column>
      <name>received</name>
      <type>TimestampAtom</type>
    </column>
    <column>
      <name>sent</name>
      <type>TimestampAtom</type>
      <fid>52</fid>
    </column>
    <column>
      <name>msgType</name>
      <type>SymbolAtom</type>
      <fid>35</fid>
    </column>
    <column>
      <name>senderCompID</name>
      <type>SymbolAtom</type>
      <fid>49</fid>
    </column>
    <column>
      <name>targetCompID</name>
      <type>SymbolAtom</type>
      <fid>56</fid>
    </column>
    <column>
      <name>tradeReportID</name>
      <type>CharList</type>
      <fid>571</fid>
    </column>
    <column>
      <name>previouslyReported</name>
      <type>CharAtom</type>
      <fid>570</fid>
    </column>
    <column>
      <name>sym</name>
      <type>SymbolAtom</type>
      <fid>55</fid>
    </column>
    <column>
      <name>lastQty</name>
      <type>FloatAtom</type>
      <fid>32</fid>
    </column>
    <column>
      <name>lastPx</name>
      <type>FloatAtom</type>
      <fid>31</fid>
    </column>
    <column>
      <name>tradeDate</name>
      <type>DateAtom</type>
      <fid>75</fid>
    </column>
    <column>
      <name>transactTime</name>
      <type>TimestampAtom</type>
      <fid>60</fid>
    </column>
    <column>
      <name>noSides</name>
      <type>IntAtom</type>
      <fid>552</fid>
    </column>
    <column>
      <name>side</name>
      <type>CharAtom</type>
      <fid>54</fid>
    </column>
    <column>
      <name>orderID</name>
      <type>CharList</type>
      <fid>37</fid>
    </column>
  </columns>
</table>

Each table is defined by a list of columns. Each column has the following attributes:

name: The name of the column.
type: The kdb+ type of the column. The KX FIX Feed Handler supports the following types:
- SymbolAtom: A kdb+ symbol. Also known as an interned string.
- FloatAtom: A kdb+ float.
- IntAtom: A kdb+ int.
- CharAtom: A kdb+ char.
- DateAtom: A kdb+ date.
- TimestampAtom: A kdb+ timestamp.
- CharList: A kdb+ string.
fid: The FIX tag value to fill the column with. If the tag value specified is not present on a message, the column is filled with a null value that matches the type of the type attribute.

The received column is a special column in a table definition. It does not require a fid to be specified. Instead, when the KX FIX Feed Handler sees that a table definition has a column with the name received it fills that column with the current timestamp each time it receives a new message for that table.

Repeating Groups

The KX FIX Feed Handler extracts repeating groups from messages based on the message type definitions in the data dictionary.

A row of data is published downstream for every repeating group contained within a message.

In some cases there may be a message type that contain repeating groups, but you do not want to publish any of the repeating group fields, nor do you want multiple rows published for that message type. In this case, you can set the maxNesting setting to 0 for that message type, which causes the KX FIX Feed Handler to ignore the repeating groups in the message.

For message types that contain nested repeating groups, maxNesting must be set equal to the maximum depth of those nested groups within the message for them to be processed correctly.

Info

Maximum depth here is defined as the total number of layers of repeating groups. For example, in the simple repeating group case, a message contains a single layer of repeating groups and thus has a maximum depth of 1. In the nested repeating group case, a message contains a layer of repeating groups and the repeating groups within that layer contains another layer of repeating groups and so on. This continues until the n^th layer (i.e. the last layer) is reached, thus giving us a maximum depth of n.

Performance Tips

See below some tips that can help optimize the performance of the KX FIX Feed Handler:

Keep tables small: Ensure each table in the schema config has only essential columns for faster message processing.
Assign each message type to 1 table only: Message types that are assigned to multiple tables in the eventToTableFile take longer to process. Avoid this if possible.
CPU Cores: For the most optimal performance, the number of CPU cores available to the KX FIX Feed Handler should match the number of FIX sessions + 3. This helps avoid CPU contention between threads.
Batching: Setting a longer pubDelay can lead to better performance, particularly in cases where a high volume of small individual FIX messages are being sent to the KX FIX Feed Handler. However, be mindful that this increases the overall memory usage of the KX FIX Feed Handler, since table rows are queued in memory during the delay.
Max nestings: Set the maxNesting of each message type to the minimal possible depth.
Turn off log auto flushing: Setting logAutoFlush to false turns off automatic flushing of the buffer each time a message is logged, reducing the frequency at which the KX FIX Feed Handler writes to the log file.

Common Errors

Below are some common errors that you may encounter:

Tag specified without a value: Received a FIX message with an empty tag e.g. 35=.
Tag appears more than once: FIX message contains a tag more than once and that tag is not contained within a repeating group. This error can sometimes occur with a tag that is expected to be in a repeating group if the format of the FIX message does not match the definition in the FIX data dictionary.
Incorrect data format for value: Tag value type does match the expected type as defined in the FIX data dictionary.
the option * is required but missing: Indicates that a required setting has not been set.
file does not exist for the option *: Indicates that the file path of a required setting does not exist.
Header Fields out of order: Indicates the header fields of the FIX message are out of order or there are header fields missing. Must match the order in the data dictionary.
Expected BodyLength=293, Received BodyLength=131: Body length (i.e. FIX tag 9) of the FIX message is not correct.
Expected CheckSum=213, Received CheckSum=2: Check sum (i.e. FIX tag 10) of the FIX message is not correct.
Required tag missing: Indicates the received message is missing a tag that is required in the data dictionary for the specified message type (tag 35).
failed to publish kdb+ data - sessionID:FIX.4.2:CLIENT->SERVER, seqnum_start:22, seqnum_end:65: Indicates that the KX FIX Feed Handler failed to publish the kdb+ data downstream. seqnum_start and seqnum_end correspond to the range of sequence numbers that were affected.