System parameters systemParams.yaml

This page explains the systemParams.yaml configuration file, covering dynamic vs. static parameters and multi-node configuration practices.

You set up your system parameters in systemParams.yaml. System parameters can be either dynamic or static depending on whether the dynamic flag is set to true or false. All parameter changes, whether dynamic or static, require a dynamic upgrade via the deploy command to take effect. However, static parameters also require a process restart to take effect, while changes to your dynamic parameters take effect as soon as the dynamic upgrade is applied. RT/RTO configuration and general system-monitoring setup are both covered later in this page as specific applications of systemParams.yaml.

Note

In a multi-node environment, your system parameters need not be the same on all nodes. For example, your eodTime on node A need not be the same as your eodTime on node B if you wish to implement EOD staggering.

Static parameters

systemParams:
  values:
   - name: opDataDir
     type: hsym
     description: Process-specific operational data directory. Substituted in …
     value: "${KXS_ROOT}/opdata${_NODE}/${INST}"
     dynamic: false

   - name: miscDataDir
     type: hsym
     description: Miscellaneous data directory. Substituted in place of ${MISC}.
     value: "${KXS_ROOT}/misc"
     dynamic: false

   - name: partFn
     type: symb
     description: Temporal partitioning function
     value: datep
     dynamic: false

   - name: hdbDir
     type: hsym
     description: Root of on-disk HDB
     value: "${KXS_ROOT}/db/hdb"
     dynamic: false

   - name: idbDir
     type: hsym
     description: Root of on-disk IDB
     value: "${KXS_ROOT}/db/idb"
     dynamic: false

   - name: initLoadSourceDirs
     type: hsyms
     description: Initial load extract directories
     value:
     dynamic: false
…

Attribute	Description	Default
appName	Application name	KX Sensors
csvDel	Delimiter in replicated CSV files	,
csvNestDel	Delimiter inside nested columns in replicated CSV files	\|
defTZ	Default time zone applied when a local time zone is required but unavailable	UTC
deltaThreshold	Specifies the age threshold of sensor data (in days) to be considered "late". This value determines how many days' worth of data goes into the base table before it is filtered to the delta table. The value must be ≥1.	10
detectRML	Sets whether the HA control layer looks for replication message loss	true
enumGroups	List of configuration parameter groups holding enumeration values. For example, a predefined list of error message codes indicating the degree of severity (1 = Info, 2 = Error, 3 = Warn, 4 = Fatal).	enum
eodInterval	Interval of the expected EOD time for splitting base and delta tables	0D00:00:01
eodTime	UTC time at which EOD is processed	02:00:00
eoiFreqMins	Intraday interval length in minutes	10
eoiOffset	Specifies how long after the scheduled interval boundary SM waits before signaling EOI. With the default of zero, SM signals EOI at the boundary itself. A non-zero value delays the EOI signal by the specified amount of time. The offset is clamped to the interval length (eoiFreqMins), and negative values are treated as zero.	0D00:00:00
flagGroups	List of configuration parameter groups holding flag values	flags
hdbDir	The root directory of the on-disk HDB. The path can include environment variables defined in $KXS_ROOT/kxsenv.	${KXS_ROOT}/db/hdb
hdbExclDays	Days of recent sensor data to exclude from HDB's in-memory cache (rdbHistDays if null)	0Ni
idbDir	The root directory of the on-disk IDB. The path can include environment variables defined in $KXS_ROOT/kxsenv.	${KXS_ROOT}/db/idb
ilChunkSize	Size in bytes of each chunk during initial loading of historical sensor data. If you select the default (-1), no chunking will occur. Chunking will slow down the loading of sensor data by a significant factor and is only recommended if your system has limited RAM.	-1
ilSortParallel	Performs sort and table finalization during initial loading of historical sensor data	true
initLoadSourceDirs	The directories for the initial loading of historical sensor data	(empty)
intervalUOMs	Units of measure measuring interval consumption (instead of snapshot readings)
mcHaAddr	Multicast group address and port for HA broadcast	:224.1.1.1:1103
mcStatAddr	Multicast group address and port for state/status/statistics broadcast	:224.1.1.1:1102
mcLocalAddr	Multicast group address and port for local broadcast	:224.1.1.1:1101
mdSnapshotRet	Number of recent MD snapshots to retain after an EOI (only required if you are splitting master data in memory)	1
mruRcvFetchEnabled	Whether RDBs fetch an MRU key's rows from IDB in the receive chain when they receive an update for a key that isn't cached. Set to false to hold such updates in the MRU table's in-memory delta table and defer the fetch until the key is next requested. MDLs always fetch on receive. See MRU tables.	true
nfyPurgeAfter	Period in days after which buffered data should be purged	10
nfySnapFreq	Snapshot frequency in minutes of buffered data	15
nxDiagsDir	NX Diagnostic Tool output directory	${NX_DIAGSDIR}${NODE}
opDataDir	Process-specific operational data directory	${KXS_OPDATADIR}/${INST}
partFn	Temporal partitioning function	datep
rdbHistDays	Days of sensor data before EOD to keep in RDB	7
rdgValScale	Scale factor applied to reading values	1e6
restBridgeLog	Whether the REST http2ipc bridge writes its own log file — for debugging only. See Call APIs over REST.	false
restMode	SAPI-to-endpoint mapping mode for REST support: restful, faithful, or unset for both. See Call APIs over REST.	(unset)
restPoolSize	Maximum number of simultaneous REST requests accepted by the http2ipc bridge. See Call APIs over REST.	255
rlogSSDir	Directory for the recovery log snapshot	${OPDATA}/rlog
sdlDeferSnapFreq	Snapshot frequency in minutes for deferred workload queue	15
sdlDelayInterval	Logging interval in seconds for transmission latency on SDL	30
sdlGcFreq	SDL garbage collection frequency in minutes. If set to 0 or less, no periodic GC is performed.	0
sdlIngestFn	Application-specific SDL ingestion function	.sdl.transform
sdlIngestAbortFn	Application-specific SDL aborted ingestion clean-up function	.sdl.ingestAbortNoop
sdlPreIngestFn	Deprecated column
sdlMbTimerFreq	SDL micro-batch instance timer frequency in milliseconds	1000
suffixTyp	Specifies the rules for formatting table name suffixes for delta and other system-generated tables (0 = camel case; 1 = underscore; 2 = lower case; 3 = upper case).	0. For example, if your base table name were custTable, any system-generated table would be named as follows: custTableDelta (option 0), custTable_delta (option 1), custTabledelta (option 2), custTableDELTA (option 3).
svcInstName	Template for naming service class instances	{PFX}{ALIAS}{_}{NODE}{SEQ}. For example, {kxs}{RDB}{_}{A}{1}.
testDir	Directory for test files	${KXS_DATA}/test
wmDir	The watermark directory	${OPDATA}/wm

Utilities Package only

Parameter	Description	Default
intervalUOMs	Units of measure for interval consumption (instead of snapshot readings)	(empty)
intu	Number of nanoseconds in a channel interval unit	1000000000
maxVeeHold	Maximum time delay expected from VEE before an HA message is considered "lost"	0D00:00:00
sapiConvUOM	If set to Yes, the SAPI will list the new unit of measure specified in the SAPI call. If set to No, the reading will be assigned the channel's unit of measure.	N
sdlConvUOM	Converts UOM of incoming readings to the UOM of a compatible pre-existing channel within the same sensor when resolving channel IDs	Y
veeDeferSnapFreq	Snapshot frequency in minutes for deferred VEE workload queue	15
veeInPfx	The canonical publish channel prefix to VEE

Dynamic parameters

Attribute	Description	Default
cdlConnTimeout	Connection timeout in milliseconds used by CDL when connecting to SGW	300
chkClnFreq	Frequency (in hours) of checking which old replicated CSV files should be deleted	1
connOpenTO	Connection open timeout, in milliseconds	1000
connRate	Socket transmission rate in bytes/second. This value is used to calculate values for TCP send and receive buffer sizes. Default TCP buffer sizes that influence bandwidth management are designed for much lower volumes of data than KX Sensors is expected to accommodate.	10000000
connRetryInt	Connection retry interval, in milliseconds	2000
connRTT	Network round-trip time in milliseconds. This value should reflect the maximum latency expected on the network. It is used to calculate values for TCP socket options (keep-alive and timeout settings) set to provide fast failure detection. See Network conditioning and fast failure detection for the recommended procedure for choosing a value.	2000
csvComprFun	Compression function for CSV replication process	.repl.zip
csvEnclQuote	For non-comma delimiter, indicates whether fields containing this delimiter, line breaks, or "s should be enclosed in double quotes with existing "s duplicated	N
csvPeek	When loading historical sensor data in csv format from a legacy system via an initial load into DBW, specifies the minimum number of characters to "look ahead" in order to locate the end of a record. You use it to calculate the split points when creating parallel ingestion blocks for a csv file. Your value should match the length of a typical record. If set too high or too low, the import of sensor readings will take longer than necessary.	500
csvRetDays	Number of days to retain the replicated CSV files	5
dbmChunkSize	The size of chunks when migrating tables	1000000
dbmDisablePend	Disables DBM wait for pending EOI completions	false
dbmRecompress	If set to true, the DBM process will check the data on disk to see whether the appropriate compression parameters have been applied. If they have, no further action will be taken; if not, the data on disk will be compressed. If set to false, the DBM process will automatically re-compress the data on disk without first checking the compression status.	false
dbmRetention	Enables row-level retention pruning by DBM. When set to true, DBM applies the retentionDuration and retentionWC properties configured in your table schemas to aging HDB partitions after EOD. See Row-level retention.	false
dbwChunkSize	The data chunk size employed by DBW when writing tables to disk. This is the number of rows of a table that DBW will write at a time during the end-of-day or end-of-interval process.	500000
dbwConnInterval	Interval in milliseconds at which DB processes retry connection to DBW	300
dbwConnTimeout	Connection timeout in milliseconds used by DB processes when connecting to DBW	2
dbwEoiHeadroom	Minimum time interval in seconds before next EOI for non-scheduled DBW processing such as dynamic updates and PIT insertions. Any requests submitted within this headroom period will be rejected with a NOT_READY error code.	60
dbwFailTerm	Whether or not DBW automatically shuts down as soon as it encounters a kdb+ or DBW-specific error.	true
dbwMaxRows	The maximum number of rows to maintain in DBW for partitioned tables	1000
dbwProcessMin	Minimum time interval in minutes required in DBW for aperiodic processing to be permitted	5
deltaMergeMinSize	The minimum size that may be considered for delta merge (in rows)	0W
deltaMergeRatio	The ratio of base table size above which delta table is merged into base	0.2
discConnTO	Timeout of Discovery connection in milliseconds	700
dlAsyncConnTO	Timeout in milliseconds when establishing connection for asynchronous send to primary data loader	1000
dlSyncConnMaxAttempts	Maximum number of data loader connection attempts before abandonment	10
dlSyncConnWait	Wait between synchronous data loader connection attempts in milliseconds	200
enforcePositivePK	Enforce integer type primary keys to have positive value when publishing (with performance penalty)	false
enforcePubSchema	Enforce table schemas when publishing (with performance penalty)	false
eodMaxCollapseKeys	Maximum number of unique keys in partitioned collapsed table to process at one time	1000000
eoiCollapsePartition	Whether or not collapsed processing is enabled during EOI for partitioned IDB tables with keys. Collapsed processing is the process by which uniqueness is enforced for kdb tables with unique keys that lose these unique keys when spread across multiple disk locations. If you disable this parameter, the partitioned-collapsed category of tables will no longer be collapsed during an EOI, resulting in non-unique key columns in IDB. (Collapsed processing will still occur during EOD for HDB tables, but EOD may run slower as a result). Note: Disabling this parameter may be required if some custom data retrieval API relies on table records being saved to IDB partitions after rdbHoldTime. (Collapsing removes older versions of the record from the snapshot, which means that record updates can prevent writing it to IDB partitions for many intervals if rdbHoldTime is non-zero).	true
eoxAckTO	Timeout in milliseconds for receipt of EOXz acknowledgements from HDBs and IDBs. If not all EOXz acknowledgements received within timeout interval, DBW proceeds with cleanup phase nevertheless.	60000
freeSlop	Free disk space slop margin in bytes for space-intensive operations (e.g. database conversion)	2000000
freeSlopPct	Free disk space slop percentage for space-intensive operations (e.g. database conversion), in addition to freeSlop	1
gc	The garbage collection mode: Immediate, Deferred or off (indicated by empty).	Immediate
hdbDiskPctThr	The HDB disk usage threshold percentage beyond which older partitions will be deleted. These partitions will be deleted during the end-of-day process.	0W
hdbMap	Whether or not DB map is enabled when HDB is loaded	false
hdbMemPctThr	Percentage of physical memory used by HDB before it will request EOP	0W
hdbMinPrtns	The minimum retention period for partitioned data expressed as the number of partitions to retain. This value must be ≥1.	0W
hdbRetPrtns	The default retention period for partitioned data expressed as the number of partitions to retain. This value must be ≥ hdbMinPrtns. (You can override the retention period for specific tables by creating an override in schema.yaml with an override value defined in the partitions column.) Any partitions prior to the current date that exceed the retention period will be deleted during the end-of-day process. The default value of 0W (infinity) means no deletion ever and must be changed to a fixed period. If you do not change it, you will eventually run out of disk space.	0W
idAllocSize	Block size when reserving IDs for master data such as sensor ID, channel ID, import ID, etc.	50
idbMap	Enables the DB map when IDB is loaded. The DB map organizes the partitions allowing for faster lookups.	true
idbReadInterval	Millisecond interval between retry attempts when initializing or reading snapshots from IDB	100
idbReadRetries	Maximum number of retry attempts when initializing or reading snapshots from IDB	3
idbRetainSOD	Retains start-of-day snapshot in IDB	Y
isREST	Master switch enabling REST support on SGW. See Call APIs over REST.	false
mdlLogCallDetails	Whether or not input and output of mutation calls are logged in .mdl.dbgi	N
minEoxzDefer	Minimum deferral time in milliseconds to run post-EOX processing such as rdb purging, garbage collection, db remapping, etc.	20
mruHoldTime	The minimum time to retain MRU records after they have been persisted to IDB. Any records not updated in the given time span will be purged from RDB. See MRU tables.	0D01:00:00
mruTO	Timeout in seconds for the SAPI calls that fetch MRU data from IDB. See MRU tables.	120
pruneEmsFreq	Frequency milliseconds of each attempt to prune EMS subscriber logs	60000
pubSyncInterval	Interval in milliseconds at which SDL attempts to send responses to synchronous publishes	250
pubSyncRetries	Maximum number of times SDL attempts to send responses to synchronous publishes	3
rdbHoldTime	The amount of data to retain in RDB at an EOI (End of Interval) specified as a span of time. A non-zero value is recommended if you want to optimize your queries for recently ingested data by storing a larger portion of data in memory (RDB) and delaying the write-down to IDB partitions. Note: This parameter is ignored if EOI is triggered by RDB in response to memory pressure. In that case, all data is purged from the RDB and written to the IDB partition. The same is true for EODs.	0
rdbMaxMruPct	The maximum percentage of free memory to allocate to MRU tables. Should be at most 25% of rdbMemPctThr. See MRU tables.	5
rdbMemPctThr	The percentage of physical memory used by RDB before it will request a data purge to DBW.	0W
replFilePerm	Unix permissions (in octal notation) for replicated CSV files	644
reportDir	Report output directory	${KXS_DATA}/reports
reportMaxRows	The maximum number of records to include in Control report	1000000
restBridgeRestartInterval	Time SGW waits before restarting the REST http2ipc bridge after it stops responding, in milliseconds. See Call APIs over REST.	250
restBridgeStartInterval	Time SGW waits for the REST http2ipc bridge to connect back after starting it before considering the start a failure, in milliseconds. See Call APIs over REST.	2500
schedAsSecDefer	Interval in milliseconds at which a primary HA client re-checks whether it has become secondary before running a deferred operation such as garbage collection	5000
sdlConnMDL	Indicates if SDL requires direct connection to MDL (e.g. for synchronous allocation of ImportLog IDs)	Y
sdlLatencyFreq	Frequency of SDL latency signal in seconds	15
sdlMdAckCheckFreq	The check frequency in milliseconds for SDL's MD push acknowledgement	200
sdlPubMD	Whether or not SDL publishes master data (MD) on a sensor data feed. If set to No, a second feed is required for master data.	false
sdlWriteDiscards	Whether or not SDL writes discarded watermarks/messages to a log. This parameter is set to false by default because there is a significant performance impact with having to write discarded watermarks and messages to disk.	false
sortLimitGB	The memory limit in GB when sorting splayed tables or partitions on-disk. This is used when sorting on-disk data during an end-of-day or an end-of-interval process.	10

Utilities Package only (dynamic)

Parameter	Description	Default
holidayYearA	First year of holidays maintained in holiday table	2015
holidayYearZ	Last year of holidays maintained in holiday table	2025
maxGapDays	The maximum gap in days to fill in a getReadings API call. If the call encounters gaps in consumption, KX Sensors will attempt to estimate the missing values using the VEE.	90
sdlAutoReg	The default auto-registration level used to govern the creation or modification of master data in response to the arrival of unexpected sensor data (value must be a name from kxsEnumAutoReg). You can override this value for a particular organization by using the sdlAutoReg parameter in kxsOrganization.	NONE
sdlExtenAutoRegOnly	If set to Y, restricts auto-extension of existing master data relationships to those created through auto-registration. If set to N, master data relationships created through other means such as an API are also candidates for auto-extension.	Y
sdlIgnoreUnreg	Indicates if, by default, errors arising from unregistered sensors should be suppressed during data ingestion. You can override this value for a particular organization by using the sdlIgnoreUnreg parameter in kxsOrganization.	N
sdlChanSensorThr	The maximum number of days that the SMP-channel relationship will be auto-extended to accommodate out-of-sequence readings
sdlSmpSensorThr	The maximum number of days that the SMP-sensor relationship will be auto-extended to accommodate out-of-sequence readings	0
veeMdWait	Minutes to wait for late master data in VEE	5

SMP-sensor relationship auto-extension

With sdlSmpSensorThr set to 5, and an SMP-sensor relationship already validated on Jan. 15:

Scenario 1: sensor data with a new SMP is received with a timestamp of Jan. 12 (inside the five-day range). Result: the existing SMP-sensor relationship is extended back to Jan. 12.

Scenario 2: sensor data with a new SMP is received with a timestamp of Jan. 6 (outside the five-day range). Result: a new SMP-sensor relationship is created for the period Jan. 6 through Jan. 15.

RT/RTO parameters

You set up your RT/RTO attributes in systemParams.yaml.

Attribute	Description	Dynamic	Default
exitOnRtError	Flag indicating whether a process should fatally crash upon receipt of an EMS error.	Y	N
chkPubTopic	Indicates whether the EMS topic of every published message should be validated.	Y	T
emsLogDir	Directory for EMS logs, substituted in place of ${EMSLOGDIR}.	N	${KXS_ROOT}/ems/${NODE}
rtoMaxRetries	Maximum number of retries to perform operations on RTO.	Y	25
rtoRetryInterval	Offset time in milliseconds for RTO to wait before attempting to retry an operation.	N	2500
rtoConnTimeout	Connection timeout used by RTO when connecting to RT, in milliseconds.	Y	2500
rtoChkDiskFreq	Retry frequency used for checking disk usage on the RT sequencer log directory, in seconds. Can be set to null where RT/TP logs are isolated on their own disk, to disable the check.	N	10
rtoDiskBufPct	Safety buffer of disk usage below the archiving limit (beyond this buffer, RTO shuts itself down).	Y	5

Changed in 3.3

rtLogDir, rtSeqDir, and rtSesDir are removed. All RT log and state directories now derive from emsLogDir. rtoRetryFreq is renamed to rtoRetryInterval (default 1000 to 2500), and rtoRetryLimit to rtoMaxRetries (default 10 to 25). Remove or rename these entries in your systemParams.yaml when you upgrade.

System monitoring parameters

The following global system parameters in systemParams.yaml govern KX Sensors performance monitoring.

Attribute	Description	Default
monFreq	The monitor reporting frequency in milliseconds	5000
monHaAlert	Name of the alert profile to use for HA event notifications to Monitor
monHttpTO	Timeout interval in seconds for HTTP operations	2
monHwWindow	Number of samples in hardware monitor context windows	1000
monNrIntvs	Number of missed monitoring intervals before an active process is declared non-responsive	2.1
monNrIntvsBusy	Number of missed monitoring intervals before a busy process is declared non-responsive	10.1
monRdbWindow	Window of data held in MonRDB after a flush is completed	1D00:00:00
monRdbPurge	Frequency of purging of data buffered in MonRDB memory	1D00:00:00
monReportIntv	Aggregation interval for generating monitoring report for various processes	0D01:00:00
monStatusFreq	The normal broadcast frequency of node status messages in milliseconds (important status changes are broadcast immediately)	60000
monStatusPct	The percentage change in the load factor required to trigger a status notification message from Monitor to all Gateways regarding the health of a node or process	20
dashWindow	Window of data held in DASH after a flush is completed	1D00:00:00
dashPurge	Frequency of purging of data buffered in DASH	1D00:00:00

Next steps

• RT log files
• About RT
• Set up query processing
• Stagger your EOIs