a vanilla tick setup has a tickerplant (TP) logging to disk and publishing to an in-memory realtime database (RDB) - and at day end the RDB data is saved to disk as another day in the history database (HDB). Users typically query the RDB or HDB directly.
it doesn’t have to be that way, there are a *lot* of other ways of assembling the kdb+tick “building blocks” to reduce or share the load.
starting at the beginning with the TP: if this is running in zero-latency mode (i.e. all updates are published immediately to subscribers) it is completely over the top to have a client task that is only plotting graphs subscribe for instantaneous update - an update every few seconds would be quite adequate.
one way of doing this is to have a chained TP, or even a chain of them. The first TP would be a zero-latency TP - and would only have clients who truly need immediate update - it in turn would have as one of its clients a TP publishing bulk updates every 100ms, that in turn would have a chained tickerplant as client that only publishes updates every second. Clients then subscribe to the TP with granularity that suits their needs.
next in the chain comes the RDB. An RDB is an in-memory database, and by day end that can be taking a lot of memory. If clients are querying that data intra-day then the memory cost is reasonable - but if the data’s only being collected to be able to add to the HDB at dayend the overhead is unreasonable. In such a case it would make sense to write the data to disk during the day so that it’s ready for dayend processing, but with only a small memory footprint to build bulk updates.
the other extreme is when one RDB isn’t enough - then the same approach can be used with multiple chained RDBs. Depending on the sort of clients it has it may be enough for one of the chained RDBs to subscribe to a bulk-update TP rather than the fastest zero-latency one.
a chained RDB doesn’t have to subscribe to the whole “firehose”, it might be useful to have a TP with only the stocks building a particular Index, or perhaps only trades and no quotes.
see (again) chained tickerplant and rdb for kdb+tick
working with the TP logfile
the TP logs the updates published to subscribers to a file, in the event of a serious crash this file can be rescued using the utility functions in source:contrib/simon/tickrecover/rescuelog.q
another overlooked problem is that often users are fetching vast amounts of raw data to calculate something that could much better be built once, incrementally updated, and then made available to all interested clients. A simple example would be keeping a running Open/High/Low/Latest - much simpler to incrementally update with data from the TP each time something changes than to build from scratch. A much more interesting example is keeping a table of the latest trade and the associated quote for every stock - trivial to do realtime with the incremental updates from the TP, but impossible build from scratch in a timely fashion with the raw data.
the default version of c.q linked to above connects to a TP and starts collecting data - sometimes that’s not enough and you want replay the log through the task first (to get the Open/High/Low for the day, not just since starting the task for example). For that use clog.q instead: source:contrib/simon/tick/clog.q
by default the endofday processing simply saves the intra-day RDB down to disk after a little re-organisation - and example of additional processing (updating a permanent HLOC table and building an NBBO table from scratch) can be found in taq/daily.q