Architecture

The architectural processes communicate via TCP/IP. Any q process can communicate with any other q process that is accessible on the network and listening for connections. (All elements downstream of the feed handlers are q processes)

Two types of communication:

• Synchronous: wait for a result to be returned
• Asynchronous: no wait and no result

All communication at the backend of the system (‘behind’ the Gateway) is async. Client communication to the GW is sync or async when using the QR/QP

Real-time ingress

The feed handlers ensure that all incoming data is parsed to the Refinery schema format. The freshly parsed data is then published to the tickerplant allowing for onward distribution to the rest of the platform.

Feed handlers can be written and developed by KX or by you:

• handle data capture and message decoding from the transport
• create a kdb+ table
• publish it directly to the Refinery tickerplant

Batch ingress

An inbound file monitoring process checks for new data files on a timer. The default format is CSV files formatted to correspond to the internal schema formats. To publish data in your own format, write a data parser to transform it to the internal Refinery schema.

Tickerplants

A tickerplant is a TCP-based data ‘fan-out’ process, implemented in software. It feeds real-time data to multiple downstream kdb+ processes for further processing. The Tickerplant itself does little or no processing or data manipulation, but simply forwards the data to subscribing processes. Data in the Tickerplant is transient not stored in memory.

The Tickerplant writes a message log to disk. This can be used for intraday recovery and persistence of data to the HDB.

Real-time databases

RDB processes subscribe to the Tickerplant and hold the current day’s data in memory. Use of the RDB allows performance of analytics on live streaming data in real-time. RDBs can be sharded to increase the scale of allowable memory. This also improves performance via parallel processing.

If multiple RDBs are in place, the incoming data will be distributed across them by the Tickerplant. Queries are automatically routed to the correct RDB/s by the GW.

Real-time engines

The real-time engines (RTEs) are complex event processors; they are streaming calculation engines designed to perform a specific task and often used for pub/sub functionality. Use of RTEs for regular, complex tasks helps take the processing load from the RDB.

The default Refinery RTE tracks all defined market data analytics for all subscribed instruments in real-time.

Historical databases

HDBs maintain all historical data (T-1) on disk. A single Refinery installation often has multiple HDBs, each one dealing with a different data set. HDB data is stored in splayed format (column split) and then partitioned by date. Each table is stored as a directory therein, with a kdb+ binary file for each column stored on disk.

For performance scaling, HDBs can be clustered. In this scenario, multiple instances of the same HDB are started and the GW/QR routes queries to the least busy instance based on queue depth or a simple round-robin distribution.

User query processing

Two main processes handle user queries to the Refinery.

Gateway (GW)

The Gateway manages both synchronous and asynchronous connections to the platform and deals with query routing to the underlying RDB/RTE/HDB processes.

Query Manager (QM)

The QM deals with streaming subscription requests from client applications.

The Gateway’s primary function is to act as a single entry and connection point to the system for users of the platform. It accepts API calls from the client and deals with the routing of that query to the backend data nodes for execution. It collates and joins results from those downstream processes and returns the result to the client. All communication is async, allowing queries to execute in parallel.

The QM manages subscriptions for streaming data from Refinery. The QM’s role is to broker the conversation between the client making a subscription and the RTE/CTP calculating the results. The QM deals with all subscription initialization, routing, reference counting (to avoid duplicate subscriptions), subscription cancellation and result publication. Multiple instances of the QM can be run for performance scaling.

Metadata

The first use of metadata in Refinery is to enable data nodes (any kdb+ process in the query path) to start and connect themselves into the platform.

When it starts, a data node publishes its metadata, including such things as the data source, the class and subclass of data it contains, the date range it contains data for and so on. This data is used to ’bind’ the process into the platform and let the GW processes know what data lives where inside the platform.

When a GW receives a client call, it matches the parameters in the call against its internal routing table. Any processes that have published matching metadata are passed to the API call for execution. All results are returned to the GW and from there back to the client.

Horizontal and vertical scaling

Refinery supports both horizontal and vertical scaling. There is no theoretical limit to the number of data nodes in Refinery or the number or size of the servers it runs on.

Vertical

Ignoring non-technical constraints like licensing, Refinery can run on any size of hardware, small or large, and will employ all computing resources available to it.

Horizontal

The nature of kdb+’s underlying interprocess communication protocol makes it trivial to distribute data nodes across multiple machines. As long as the GW/QR have network visibility to the processes, queries and results can be routed.

Real-time and historical data are inherently different. One is ‘in flight’ and constantly being updated; the other is ’at rest’ and written to disk. These dynamics mean different (yet similar) techniques are required to deal with these datasets.

High availability and failover

A hot-hot architecture for high availability:

• hot-hot in a data capture context: both sides of the system are live and capture data at the same time
• hot-cold from a query perspective: in normal circumstances, only one side of the system responds to user queries

Each process in the primary instance has a corresponding ‘mirrored’ process in the secondary. The GW on both sides of the system maintain a primary and secondary connection for each data node. The system supports process-level failover at the data-node level. There is no client interruption in the event of a data-node failure. The GW automatically routes to the secondary process.

Each client application maintains a primary and secondary connection handle to the primary and secondary GW processes. If the entire primary system goes offline or the GW/ fails, queries are immediately routed to the secondary.

Process-level failure

Site-level failure

24/7 operation and data persistence

Real-time data is resident in an RDB during the current day. All data older than that resides in an HDB on disk. Data persistence is managed in Refinery by a set of processes known collectively as the PDB or ‘Persisting DataBase’.

As the diagram shows, the PDB is a custom RDB that is not in the query path; instead, it simply logs and sorts all data from the tickerplant to an on disk cache during the data.

At rollover (usually midnight), the system begins sorting this cache and creating an HDB partition for the now previous day's data. Whilst this process is ongoing, the RDBs continue capturing data for the new day and query processing continues uninterrupted.

Once the new HDB partition is ready, a signal is sent, the HDB processes load the new partition and the RDBs drop the old data. This all occurs in a matter of milliseconds and eliminates any downtime.