MRU tables¶

This page explains how most recently used (MRU) master data tables are cached in memory, how to choose an MRU key column, how RDBs handle updates for keys that aren't cached, and how to defer fetching those keys until they're queried.

About MRU tables¶

MRU describes both a class of table schema and the framework that handles those tables.

An MRU table is a master data table that's too large to hold in memory in full. You declare one by setting its schema type to mru. Instead of loading the whole table, RDBs and MDLs cache only a recent working set of rows. Rows that aren't cached stay on disk in IDB and are fetched on demand, and cached rows that fall out of use are purged to make room for more recently used ones.

An MRU table has the following requirements:

The table must be keyed.
The table must include a soft-delete flag column, conventionally named deleted or isDeleted. MRU tables support soft deletes only; rows are removed from disk at EOD.
The table must define an updateTsCol. The framework uses this column to derive the initial reference time for each MRU key, which is how it tracks recency.
The deployment must include at least one active IDB process. Without one, MRU tables aren't supported.

One column in the table is the MRU key, flagged with isMisc: true. Rows that share an MRU key value are fetched and purged together as a group. For example, if the MRU key is a meter ID, a fetch retrieves every row for that meter. If you don't flag a column, the framework uses the first column in the table.

Each MRU table has a companion delta table of type deltaMRU, which KX Sensors generates for you. Its name is the base table name plus the delta suffix set by suffixTyp, for example meterLimitsDelta or meterLimits_delta. In IDB, the delta table holds the changes written down at each end of interval (EOI) until end of day (EOD) merges them into the base table. In RDB, it's an in-memory, virtually partitioned table that's used by deferred fetch.

Choose an MRU key column¶

The MRU key determines how much data moves in a single fetch and how often fetches happen, so its cardinality matters:

If the key is too unique, each key groups few rows and the process issues frequent fetch calls to IDB.
If the key is too general, each key groups many rows and a single fetch transfers a large volume of data.

Look for a column with a natural grouping — one where a group of records is used frequently for a period, goes unused for a period, then is used again. That access pattern is what the cache is designed to exploit.

How the MRU fetch works¶

Every operation on an MRU table — insert, update, delete, and read — goes through a check for the MRU keys it needs:

If the required keys are already in memory, the operation completes immediately.
If the required keys exist in the system but aren't in memory, the process fetches the missing rows asynchronously from IDB through the gateway, reading both the on-disk base table and the on-disk delta table. The operation waits for the fetch, then completes as a callback once the rows are appended to memory. mruTO sets the timeout for these calls.

Either way, the reference time for each MRU key involved is updated, which is what keeps an active key in the cache.

Fetch on receive depends on the rcv function that mdbTables.yaml assigns to MRU tables, which is .mru.updMRU in the KX-supplied configuration. Don't override rcv for MRU tables in your own package.

Handling updates for keys that aren't cached¶

When an RDB receives an update for an MRU key, one of three things is true: the key is already in memory, the key is new, or the key exists on disk but isn't cached. In the first two cases, the RDB applies the update directly. The third case is governed by the mruRcvFetchEnabled system parameter, which selects one of two modes.

Fetch on receive (default)Deferred fetch

With mruRcvFetchEnabled set to true, the RDB fetches the key's rows from IDB before it applies the update. The in-memory table therefore always holds a complete group of rows for every cached key.

The cost is a round trip to IDB in the receive chain for every update that introduces a key not already cached. On feeds that touch many distinct keys, this adds latency to ingestion.

With mruRcvFetchEnabled set to false, the RDB doesn't fetch in the receive chain. Instead, it splits each incoming update:

Rows for keys that are in memory, or that are new, are applied to the base MRU table as usual.
Rows for keys that exist on disk but aren't cached are held in the current virtual partition of the in-memory delta table, for example meterLimitsDelta. No fetch is issued.

The fetch happens later, when something on the RDB asks for the key, such as a query. At that point the RDB retrieves the key's rows from IDB, then applies the rows it has been holding in the delta table over them, so the result reflects every update received since the last write-down.

The delta table's virtual partitions are dropped once DBW has written the interval they belong to down to IDB. By then the same updates are on disk, and a fetch returns them from IDB directly.

Note

mruRcvFetchEnabled affects RDBs only. MDLs always fetch on receive, because they need the complete row group in memory to validate mutations. The parameter is static, so restart the RDBs after you change it.

Deferred fetch relies on the RDB entry for deltaMRU tables in the KX-supplied mdbTables.yaml, which marks them as virtually partitioned:

   - process: rdb
     table: deltaMRU
     isvp: true

Don't override isvp for deltaMRU tables on RDBs in your own package.

Runtime behavior¶

Process initialization¶

MRU tables are initialized empty. Each process that holds MRU tables in memory makes a synchronous call to IDB to retrieve:

every existing MRU key for each table, mapped to the most recent reference time for that key
the average row size for each MRU table, which the process uses to estimate memory

The process then replays its log, and fetches any keys it saw during replay that aren't already in memory. With deferred fetch enabled, those updates are held in the in-memory delta table instead.

End of interval¶

At each EOI, all changes made to an MRU table during the interval are persisted to the on-disk delta version of the table.

Once the EOI completes, an MRU purge runs. It drops cached keys that haven't been updated within mruHoldTime since they were persisted to IDB. If the MRU tables still occupy more than rdbMaxMruPct of free memory, it purges further keys, oldest first.

End of day¶

At each EOD, the changes accumulated in the on-disk delta tables during the day's EOIs are compiled into a single base table, written to the 0 IDB partition for the next day. The last record for each key group becomes the active record.

Soft-deleted records — those whose deleted or isDeleted column is 1b — are removed from disk at this point. Because partition 0 holds the MRU tables, don't delete that directory. See Delete historical partitions.

Parameter	Description	Default
`mruRcvFetchEnabled`	Whether RDBs fetch an MRU key's rows from IDB in the receive chain when they receive an update for a key that isn't cached. Set to `false` to hold such updates in the in-memory delta table and defer the fetch until the key is next requested. Static.	`true`
`mruHoldTime`	The minimum time to retain MRU records in memory after they have been persisted to IDB. Any records not updated in the given time span are purged from RDB.	`0D01:00:00`
`rdbMaxMruPct`	The maximum percentage of free memory to allocate to MRU tables. Should be at most 25% of `rdbMemPctThr`.	`5`
`mruTO`	Timeout in seconds for the SAPI calls that fetch MRU data from IDB.	`120`

See System parameters for how to set these.