Recovery log (rlog) files¶
This page describes recovery log (rlog) files, their role in preserving in-flight data before RT persistence, and their location on disk.
Recovery logs are short-term logs local to a node that allow for recovery of data in the event of a failure in data processing before the data has been transferred to, and persisted by, the consumer. rlogs are in place for all pre-RT ingestion processes to maintain a record of what data is in flight before being persisted by RT. rlogs can be "replayed" in order for processes to regain knowledge of unpersisted data.
These are located within the opdata directory, at ${OPDATA}/[Process]/rlog. The opdata directory is specified by the opDataDir attribute in systemParams.yaml.
rlogs are used for data processed by SDL and VEE. See the tables below for a description of these uses within kxs-core. You may have additional service classes that employ rlogs within your custom packages.
kxsSDL¶
| Folder | Description |
|---|---|
| deferred | Data that the SDL has deferred from VAL |
| pen | Penned data awaiting a certain event (i.e. completion of a request or arrival of other data) |
| SDL* | Contains the data messages that are sent to VEE (i.e. kxsReadingVal) |
* Single-node only.
kxsVEE¶
| Folder | Description |
|---|---|
| VEE | If message splitting mode is activated, a dictionary containing a count of readings currently in-process grouped by chanID (.vee.inProc). If message splitting is not activated, a table keyed on msgID that contains the list of channels in the message (chanInMsg) as well as any channels that this message is currently waiting on VEE to finish processing (chanWait). |
| pen | Penned data awaiting a certain event (i.e. completion of a request or arrival of other data) |
kxsProf¶
VEE makes use of the Prof process for more complex forms of validation and estimation.
| Folder | Description |
|---|---|
| pen | Penned data awaiting a certain event (i.e. completion of a request or arrival of other data) |