Skip to content

ACTIV Market Data Feed Handler

This page provides an overview of the KX ACTIV Market Data Feed Handler and lists its settings.

The ACTIV Feed Handler streams data from ACTIV Financial's ACTIV One Platform (AOP) into a kdb+, kdb Insights SDK, or kdb Insights Enterprise installation. It subscribes to the symbols you specify, converts the ACTIV updates into kdb+ tables, and publishes them downstream. In kdb Insights deployments it publishes to kdb Insights Reliable Transport (RT), the messaging layer that carries data between components.

Get started

Get started with one of the following:

Settings

This section lists the settings for:

activ

The following table describes the settings for the feed handler. You can provide each setting on the command line, as an environment variable, or in a settings file. In the kdb Insights Enterprise Helm chart, the settings the chart exposes are under the activ key in values.yaml. The Setting column shows the name used there.

Setting Command line Environment variable Type Default Description
logFile --logging.file KXFEED_ACTIV_LOGGING_FILE string The application log file path for general log messages from the feed handler.
logLevel --logging.level KXFEED_ACTIV_LOGGING_LEVEL string info Level of logging from the feed handler. Options are trace, debug, info, warning, error, and fatal.
logAutoFlush --logging.auto_flush KXFEED_ACTIV_LOGGING_AUTO_FLUSH boolean false Flush the log buffers after every message.
logRotationSize --logging.rotation_size KXFEED_ACTIV_LOGGING_ROTATION_SIZE integer 10485760 Rotate the log file when it reaches this size, in bytes.
host --aop.host KXFEED_ACTIV_AOP_HOST string Hostname of the ACTIV server.
userId --aop.userid KXFEED_ACTIV_AOP_USERID string User ID for the ACTIV server.
pass --aop.password KXFEED_ACTIV_AOP_PASSWORD string Password for the ACTIV server.
subscriptionFile --aop.subscription.file KXFEED_ACTIV_AOP_SUBSCRIPTION_FILE string Path to the file listing what to subscribe to. See Subscriptions.
eventToTableFile --aop.eventTableMapping.file KXFEED_ACTIV_AOP_EVENT2TABLE_FILE string Path to the file that maps ACTIV event types to tables. See Mapping configuration.
timeout --aop.timeout KXFEED_ACTIV_AOP_TIMEOUT integer 30000 ACTIV server connection timeout, in milliseconds.
enableQueries --aop.enable_queries KXFEED_ACTIV_AOP_ENABLE_QUERIES boolean false Allow subscription entries to be ACTIV queries instead of plain symbols. Queries can use wildcards, for example symbol=GOOGL*. Each entry is a complete query. Separate entries with ;, for example symbol=SHEL.L;symbol=POW.TO, or put each entry on its own line in the subscription file. See Subscriptions. The Helm chart sets this to true.
dbSchemaFile --db.schema.file KXFEED_ACTIV_DB_SCHEMA_FILE string Required. Path to the database schema file. See Schema considerations.
dbConfigFile --db.config.file KXFEED_ACTIV_DB_CONFIG_FILE string Required. Path to the database connection configuration file. See Database connection.
dbSendSchema --db.send_schema KXFEED_ACTIV_DB_SEND_SCHEMA boolean false Create the tables in the target kdb+ process when the feed handler starts. Applies only to direct connections to a kdb+ process.
pubDelay --db.pub_delay KXFEED_ACTIV_DB_PUB_DELAY integer 100 Publish delay in milliseconds. The feed handler caches updates and publishes them in batches at this interval. Set to 0 to turn batching off.
batchQueueSize --batch.queue_size KXFEED_ACTIV_BATCH_QUEUE_SIZE integer 1000 Size of the queue that the worker threads share to hand batches to the publisher thread.
ThreadsCount --threads.count KXFEED_ACTIV_THREADS_COUNT integer 1 Number of worker threads. The feed handler distributes symbols to the workers in round-robin order. The Helm chart sets this to 2.

Settings file

You can also place the settings in a file and pass it with -c or --config (or KXFEED_ACTIV_CONFIG). You can also pass the file path as the last argument, for example kxfeed_activ settings.cfg. The template cfg/settings.cfg shows the layout:

[logging]
file=/var/log/kxfeed_activ.log
level=info
auto_flush=true

[aop]
host=
userid=
password=
timeout=10000
subscription.file=/etc/kxfeed_activ/symbols.txt
eventTableMapping.file=/etc/kxfeed_activ/eventToTable.json
enable_queries=false

[db]
config.file=/etc/kxfeed_activ/kxfeed_config.json
schema.file=/etc/kxfeed_activ/schema.xml

If you set the same setting in more than one place, the command line overrides environment variables, and environment variables override the settings file.

Tip

With the default pubDelay of 100, the feed handler caches updates and publishes them every 100 ms. If several updates for one instrument arrive within the interval, they are published together in a single message. Set pubDelay to 0 to publish each update as it arrives.

rt

Info

The following settings apply only to kdb Insights Enterprise deployments.

The following table describes the settings for configuring the RT publisher within the feed handler.

Field Type Description
sinkName string The name of the RT stream to publish to. Set in the stream section.
logLevel string The logging verbosity of the RT publisher (INFO, WARN, ERROR, OFF).
logPath string The mount path of the Persistent Volume Claim (PVC) where the RT publisher stores local log files.
volCapacity string The size of the PVC that the RT publisher uses to store local files.
configURL string Optional URL of the kdb Insights Enterprise Information Service (the service that tells clients where the RT endpoints are), used to discover the RT endpoints.
replicaCountFeatureFlag boolean Monitor the number of RT replicas the publisher connects to.
streamSuffix string Optional suffix appended to the RT stream name to support proxy wildcards, for example .namespace.svc.

persistence

Info

The following settings apply only to kdb Insights Enterprise deployments.

The following table describes the settings for configuring persistence within the feed handler.

Field Type Description
enabled boolean Enable or disable persistent storage, claimed through PVCs.
useLocalValues boolean Override global accessMode and storageClass.
storageClass string The storage class to apply to the PVC.
storageSize string The requested size of the volume.
accessModes string The access modes to request for the PVC.

resources

Info

The following settings apply only to kdb Insights Enterprise deployments.

Field Description
limits The maximum resources the feed handler pod can request.
requests The minimum resources the pod requests.

The disk space, memory, and CPU required depend on the volume of data you expect to ingest. For more information, refer to the RT publisher documentation.

configMap

Info

The following settings apply only to kdb Insights Enterprise deployments.

Field Description
create Create a ConfigMap to mount the configs in the config directory of the feed handler Helm chart. Set to false to use an external ConfigMap.
mount Mount the named ConfigMap on the path /opt/kx/kxfeed_activ/config.
annotations Annotations to add to the ConfigMap.
name The name of the ConfigMap to use. If you don't set it, the chart generates a name from its full name template.
files List of files in the config directory that the ConfigMap mounts.

Helm chart settings

Info

The following settings apply only to kdb Insights Enterprise deployments.

The activ key in values.yaml also contains the following settings. You rarely need to change them.

Setting Description
tmpConfigDir Directory in the container where the startup script writes the working copies of the configuration files, the generated RT configuration, and symbols.txt, which it builds from subscription each time the feed handler starts.
dbConfigFile Path to the database connection configuration file. The startup script updates a copy of this file in tmpConfigDir with the RT settings.
subscriptionFile Path to the subscription file. The feed handler reads the symbols from this file. The chart default is /opt/kx/kxfeed_activ/config/SymbolHJ.txt, which isn't the file the startup script writes. To use the subscription value, set subscriptionFile to symbols.txt in tmpConfigDir, for example /tmp/kxfeed_activ/symbols.txt.

The chart also has an encryption section, where enabled is set to true when the cluster uses Istio, and the podSecurityContext and securityContext sections for the pod security settings.

Database connection

The dbConfigFile controls where the feed handler publishes. It is a JSON file with one of three modes. If you set both isDirectConnect and isTickerPlantConnect to 1, the feed handler uses the direct connection. If you set neither, or leave both out, it publishes to RT.

Mode Set Publishes to
RT (default) "isDirectConnect": 0 An RT stream. Used by the kdb Insights SDK and kdb Insights Enterprise deployments.
Direct kdb+ "isDirectConnect": 1 A single kdb+ process. Used by the standalone deployment.
Ticker plant "isTickerPlantConnect": 1 A kdb+ ticker plant.

For RT, set configUrl to the RT configuration file or the information service URL:

{
    "configUrl": "file:///usr/local/bin/cfg/rt_config.json",
    "isDirectConnect": 0,
    "isTickerPlantConnect": 0
}

For a direct connection to a kdb+ process:

{
    "isDirectConnect": 1,
    "directConnectHost": "db",
    "directConnectPort": 2345,
    "directConnectFunction": "insert",
    "directConnectMode": "sync",
    "directConnectTimeout": 5000,
    "directConnectPublishFormat": "table"
}

The feed handler calls directConnectFunction on the process with the table name and the data. The default function is insert.

The publish format sets what the feed handler passes as the data. Set directConnectPublishFormat to table to pass a q table, or list to pass a list of columns. The default is table for a direct connection and list for a ticker plant. You can set publishFormat to apply one format to both. The mode-specific setting overrides it. Accepted values are table or flip, and list, lists, or columnlist.

For a ticker plant, tickerPlantConnectHost, tickerPlantConnectPort, and tickerPlantFunction are required:

{
    "isTickerPlantConnect": 1,
    "tickerPlantConnectHost": "tickerplant",
    "tickerPlantConnectPort": 5010,
    "tickerPlantFunction": ".u.upd",
    "tickerPlantMode": "sync",
    "tickerPlantPublishFormat": "list"
}

The following optional fields apply to both direct and ticker plant connections. For the ticker plant, replace the directConnect prefix with tickerPlant.

Field Description Default
directConnectMode Publish mode: sync or async. sync
directConnectTimeout Connection timeout in milliseconds. 5000
directConnectUsername Username, if the target process requires one. empty
directConnectPassword Password, if the target process requires one. empty
directConnectMaxReconnectAttempts Number of reconnect attempts if the connection drops. 3
directConnectReconnectDelay Delay between reconnect attempts, in milliseconds. 1000

Schema considerations

The dbSchemaFile defines the structure of the tables published downstream by the feed handler.

The following example shows a table definition in the dbSchemaFile:

<table>
  <name>trade</name>
  <id>trade</id>
  <columns>
    <column><name>sym</name><id>sym</id><type>SymbolAtom</type></column>
    <column><name>eventType</name><id>eventtype</id><type>SymbolAtom</type></column>
    <column><name>updateType</name><id>updatetype</id><type>SymbolAtom</type></column>
    <column><name>exchTime</name><type>TimespanAtom</type><fid>1690</fid></column>
    <column><name>tradeDate</name><type>DateAtom</type><fid>16</fid></column>
    <column><name>tradeTime</name><type>TimespanAtom</type><fid>15</fid></column>
    <column><name>price</name><type>FloatAtom</type><fid>12</fid></column>
    <column><name>volume</name><type>LongAtom</type><fid>13</fid></column>
    <column><name>updateId</name><id>updateid</id><type>IntAtom</type></column>
    <column><name>exchDate</name><type>DateAtom</type><fid>1688</fid></column>
  </columns>
</table>

Each table is defined by a list of columns. Each column has the following attributes:

  • name: The name of the column.
  • type: The kdb+ type of the column. The feed handler supports the following types for columns populated from an ACTIV field:
    • SymbolAtom: A kdb+ symbol
    • FloatAtom: A kdb+ float
    • LongAtom: A kdb+ long
    • DateAtom: A kdb+ date
    • TimespanAtom: A kdb+ timespan
    • CharList: A kdb+ string
    • ByteList: A kdb+ byte list
  • fid: The ACTIV field ID (FID) that populates the column.
  • id: A feed handler built-in value instead of an ACTIV field. The built-in values are sym (the symbol), eventtype, and updatetype, which use SymbolAtom, and updateid and permissionid, which use IntAtom.

If a column has an id, the feed handler uses the built-in value and ignores any fid. IntAtom is supported only for the updateid and permissionid built-in columns. For a column with any other type, and for an ACTIV field that has no value in an update, the feed handler publishes a null.

Published tables

The schema in the kdb Insights Enterprise Helm chart and the standalone example define the following tables. The kdb Insights SDK example defines only trade, quote, and cancor.

Table Contains Columns
trade Trades. sym, eventType, updateType, exchTime, tradeDate, tradeTime, price, volume, updateId, exchDate
quote Best bid and offer quotes. sym, permissionid, eventType, updateType, exchTime, bidTime, bidPrice, bidSize, bidAtMarketSize, bidAtMarketTime, askTime, askPrice, askSize, askAtMarketSize, askAtMarketTime, updateId, quoteDate, exchDate
cancor Trade cancellations and corrections. sym, eventType, updateType, tradeCorDate, tradeCorTime, tradeDate, tradeTime, price, volume, execID, origExecID, tradeExecVenue, updateId, conditions
openClose Opening and closing prices. sym, eventType, updateType, openPrice, closePrice, closeDate, updateId
snapshot Updates that have no event type (NONE). This table is usually empty. sym, eventType, updateType, mic, isin, context, createDate, currency

Note

The feed handler publishes only ACTIV updates. It doesn't publish the initial snapshot that ACTIV sends when you subscribe to a symbol. The feed handler uses that snapshot to cache the symbol's field values.

You can add columns for any ACTIV field by adding a column entry with the field's fid. To change which events fill each table, see Mapping configuration.

Mapping configuration

The eventToTableFile maps each table to the ACTIV event types that fill it. The following example shows the mapping:

{
    "snapshot": "NONE",
    "trade": "TRADE;TRADE_NON_REGULAR;POST_TRADE;TRADE_ALTERNATIVE_CURRENCY",
    "quote": "BBO_QUOTE",
    "cancor": "TRADE_CORRECTION;TRADE_CANCEL;POST_TRADE_CORRECTION;POST_TRADE_CANCEL;TRADE_ALTERNATIVE_CURRENCY_CORRECTION;TRADE_ALTERNATIVE_CURRENCY_CANCEL",
    "openClose": "CLOSE;OPEN",
    "auction": "AUCTION"
}

In this file, each table is mapped to a list of ACTIV event types, separated by ; or ,, with no spaces. A table that is not listed receives no events. An event type that appears under more than one table is published to each of them.

The feed handler ignores the following without logging a warning:

  • An event type name it doesn't recognize.
  • A mapped table that isn't defined in the dbSchemaFile. For example, the Helm chart maps auction, but its schema doesn't define an auction table, so the feed handler doesn't publish AUCTION events.

If you don't set eventToTableFile, the feed handler maps no events and publishes no data.

Subscriptions

The subscriptionFile lists what to subscribe to. How you write the entries depends on enableQueries.

enableQueries Entries Separator Example
false (default) Plain symbols Whitespace, such as spaces or newlines SHEL.L POW.TO
true ACTIV queries, including wildcards ; or newlines symbol=SHEL.L;symbol=GOOGL*

With enableQueries set to false, the feed handler subscribes to each entry as a symbol, such as SHEL.L. In this mode ; isn't a separator and symbol= isn't recognized, so symbol=SHEL.L;POW.TO is read as one symbol with that name. To use queries or wildcards, set enableQueries to true.

With enableQueries set to true, the feed handler passes each entry to ACTIV unchanged as a query. Use queries to subscribe to many symbols with one entry:

Query Subscribes to
symbol=SHEL.L One symbol.
symbol=GOOGL* Every symbol that starts with GOOGL, for example the listings of GOOGL on each exchange.
symbol=SHEL.L and exchange=L A symbol on one exchange.

The feed handler splits the entries only on ; and newlines, and sends each entry to ACTIV as a separate query. Each entry must be a complete query, so repeat symbol= for every entry. symbol=SHEL.L;POW.TO sends POW.TO as a query of its own, not as symbol=POW.TO.

The following subscription files are equivalent:

symbol=SHEL.L;symbol=POW.TO;symbol=GOOGL*
symbol=SHEL.L
symbol=POW.TO
symbol=GOOGL*

The feed handler doesn't trim spaces. Write symbol=SHEL.L;symbol=GOOGL*, not symbol=SHEL.L; symbol=GOOGL*. A space-separated list with enableQueries set to true is read as a single query.

Tip

A wildcard query can match many symbols, and each matched symbol sends its own updates. The feed handler distributes the entries to the worker threads in round-robin order, one entry at a time, so all the symbols that one wildcard query matches are handled by the same worker thread. To spread a large load across threads, split it into several queries.

In the Helm chart you set the entries with the subscription value, using the same rules. The chart default for enableQueries is true, so change subscription to ;-separated queries, or set enableQueries to "false". Also set subscriptionFile as described in Helm chart settings, or the subscription value isn't used.

In the Docker Compose examples, enableQueries isn't set, so it is false and the symbols file lists plain symbols. To use queries or wildcards in an example:

  1. Add KXFEED_ACTIV_AOP_ENABLE_QUERIES to the environment list of the feed service in the compose file.
  2. Add export KXFEED_ACTIV_AOP_ENABLE_QUERIES=true to the example's env file.
  3. Rewrite the entries in the symbols file as queries, for example symbol=SHEL.L;symbol=GOOGL*.

Next steps