Housekeeping
Instance tables
A framework is provided to perform housekeeping on in-memory tables to prevent them from growing unbounded and bloating the process memory.
Tables can be specified for clean-up by adding parameters to the enhanced instance parameter override. The supported values are in the table below.
parameter | type | default | description |
---|---|---|---|
.px.cln.startTime |
time | 00:00:00.0 | Time to run instance table clean-up (UTC) |
.px.cln.frequency |
int | 24 | Interval between clean-up runs (hours) |
.px.cln.table |
string | Specifies the tables, number of hours to keep and min number of rows before cleanup (format is table,hours,rows) |
The .px.cln.table
format specifies the table name, number of hours to keep and the row threshold for housekeeping (optional). If the number of rows is below the threshold, houskeeping will not run.
Warning
The target table must have a time
column of type timestamp or datettime
- Cleanup the
.pm.accesslog
table if it exceeds 10,000 rows and keep the last 24 hours -.pm.accesslog,24,10000
- Cleanup the
.pm.querylog
table and only keep the last 24 hours (no row limit) -.pm.accesslog,24,
The .px.cln.table
field can be repeating to perform housekeeping for multiple tables.
Control housekeeping
There are several housekeeping tasks that run on Control, performing various actions:
- dc_housekeeping_auto_export.1 - exports any packages listed as autoExportPackage in
DC_EXPORT:DEFAULT
config parameter - dc_housekeeping_dc_checkpoint.1 - runs a Control checkpoint
- dc_housekeeping_garbage_collection.1 - runs garbage collection on Control process to free-up memory
- dc_housekeeping_qsrc_archive.1 - makes a backup of the
qsrc
folder to theqsrcarchive
directory - dc_housekeeping_tdir_archive.1 - makes a backup of the
tdir
folder to thetdirarchive
directory - dc_housekeeping_tdir_checkpoint.1 - runs a
tdir
checkpoint and clears the transaction log - dc_housekeeping_tdir_historise.1 - archives old data for audit tables to disk. Controlled by the
housekeeping:<DEFAULT>
configuration
These tasks run on default schedules. This can be changed by editing the tasks in KX Control. The tasks can also be run manually from KX Control. The default schedules are shown below
housekeeping tasks | default schedule |
---|---|
dc_housekeeping_auto_export.1 | not scheduled |
dc_housekeeping_dc_checkpoint.1 | every Monday at 0000 |
dc_housekeeping_garbage_collection.1 | not scheduled |
dc_housekeeping_qsrc_archive.1 | every 12 hours |
dc_housekeeping_tdir_archive.1 | every 12 hours |
dc_housekeeping_tdir_checkpoint.1 | every hour |
dc_housekeeping_tdir_historise.1 | every day at 0000 |
The ${DELTADATA_HOME}/DeltaControlData
directory contains persisted Control data, files and backups:
tdir
- persistence of in-memory Control datatdirarchive
- contains archives of previous versions oftdir
qsrc
- contains process template filesqsrcarchive
- contains archives of previous versions ofqsrc
localRepo
- contains Control workspace data used for KX Controltdirhist
- location of historised data
See Persistence for more information.
tdir_historise task
The dc_housekeeping_tdir_historise.1
task will run every day by default. It archives data from Control tables to disk in the directory ${DELTADATA_HOME}/DeltaControlData/tdirhist
. The tables to archive and how many days data to keep in Control memory is configured using the housekeeping:<DEFAULT>
parameter.
For example, any data older than 5 days will be removed from the .pm.querylog
table in Control and archived to ${DELTADATA_HOME}/DeltaControlData/tdirhist
when the dc_housekeeping_tdir_historise.1
task runs.
AXSTATE and revision housekeeping
dc_housekeeping_tdir_historise.1
can also be used to clean out entity revisions older than a certain number of days from ${DELTADATA_HOME}/DeltaControlData/localRepo/AXSTATE
. To enable this, add AXSTATE
to the housekeeping:<DEFAULT>
param.
When dc_housekeeping_tdir_historise.1
runs on schedule, entity revisions older than the configured number of days will be dropped, excluding the active revision, which will be kept regardless of age.
AXSTATE compression
Setting the environment variable AXSTATE_COMPRESSION
to 'yes' will compress new AXSTATE tables.
By default, compression will use the settings 17 2 6, which are the blockSize, algorithm (gzip), and zip levels respectively.
Optionally setting AXSTATE_COMPRESSION_SETTINGS
to a list of 3 integers will set the compression options as supported by set
export AXSTATE_COMPRESSION=yes
export AXSTATE_COMPRESSION_SETTINGS="17 2 6"
To compress an existing table, use the internal function -19! to backup and write down each column as compressed.
See Also: Archiving files