Build a tick-capture system

The following instructions will build a basic system to stream market data using a real-time processing engine (RTE) which captures and stores the data in a real-time database (RDB), transfers data to an on-disk historical database (HDB), and displays collected data on dashboard. To build this system, a set of entities must be built in Kx Control in the following order:

  • A package for the system to maintain modular organization (dhtPackage).
  • A schema group and two schemas to organize and define the datasources to be captured, stored and used throughout the system (dhtSchemaGroup, dhtTrades, dhtQuotes).
  • A messaging server process to allow all our processes to communicate with each other (dhtDMServer)
  • A config parameter override to associate the message server with all the other processes.
  • A tickerplant process to capture, log and distribute data to other processes in the system (dhtTP).
  • A real-time database process to store “today’s” data in memory (dhtRDB).
  • A historical database process to store and manage access to data prior to today (dhtHDB).
  • A workflow to tie everything together and start the system (dhtWorkflow). This will initial start without data to make sure everything is working properly.
  • An analytic group containing two analytics used to populate the previously created databases (dhtDummyData, .dhtDummyData.loadAnalytics, .dhtDummyData.initRTE).
  • A real-time engine process to populate the real-time database (dhtRTE). In this system, use the RTE to populate the database. In a real system, data would be streaming into the system via one or more feedhandlers and RTEs might be used to perform data manipulations on the data, so it is useful to become familiar with their operation.
  • A dashboard and connection to view the data (dhtDashboard, dhtConnection).
  • A user to administrate the system through entitlements (dhtAdmin). For most of the tutorial, use the default system administrator.
  • A user to view dashboards using entitlements (dhtEndUser).
  • A terminate process to stop the system on a schedule (dhtTerminate).

The schematic shows an example of the system. In this example, the dhtRTE will be used to populate the datasources instead of external datasources and feedhandlers as would be expected.

Screenshot

Design conventions

Use the following design conventions when building a tick-capture system.

Do

  • Decide on a naming format and apply it consistently (e.g. camelCase, underscore_separated, etc). In our system, we will prefix all entities with dht followed by the entity type, e.g. dhtRDB.
  • Try to relate the name of the entity to the entity group by using a shared prefix, e.g. dhtSchemaGroup.
  • Use the descriptive field to describe the specifics of the entity.
  • Add comments to functions.
  • Delete entities once they are no longer used.

Don’t

  • Don’t modify anything prefixed with DS, Dx, Dmx or Delta. These are standard. Copy them instead if required.
  • Don’t make names too short; it makes them hard to understand.