IPDB and EPDB

What is the IPDB?

Intraday Persisting Database

IPDB is responsible for the intraday persisting of incoming data. As data is streamed in, the IPDB writes batches of this data to disk. This is a configurable interval/row count for case-by-case optimisation of CPU usage and memory overhead.

What is the EPDB?

End-of-day Persisting Database

The EPDB is responsible for end-of-day procedures. This involves sorting new partitions, applying attributes and moving it to the HDB directory.

Why are they separate?

IPDB and EPDB are split as the end-of-day tasks are memory intensive and can be time consuming. During the time in which the sort is happening on yesterday's partition, ticks can still be inbound for the next partition. In a single PDB model, this means the ticks are queued up while the process is busy with the sort. This can result in high memory consumption in the tickerplant's outbound buffer.

By splitting the processes, new ticks can continue to be written to disk in the IPDB while the EPDB is busy sorting the previous partition. The overall result is lower memory usage and a more stable system.