Configure VEE

This page explains how to activate VEE in systemParams.yaml, then describes the two VEE configuration objects that require setup, veeDefaults.yaml and valDefaults.yaml.

Activate VEE

The following system parameters, defined in systemParams.yaml, are used to activate various VEE components. These are all static parameters and require a restart of the appropriate process (VEE, SDL or Prof) to take effect.

Note

If VEE is busy and a restart is not possible, then you can update the in-memory dictionary that caches this default configuration information on initiation; namely, VEE_DEFS.

Attribute	Description	Default
estConfigured	Only available if both veeConfigured and veeEnabled = Y. Enables the estimation portion of VEE. When enabled, sensor data will pass through the estimation portion of VEE after validation.	Y
profConfigured	Only available if both veeConfigured and veeEnabled = Y. If set to Yes, sensor data will be routed to the Profile Engine on ingestion.	Y
profEnabled	Only available if both veeConfigured and veeEnabled = Y. If set to Yes and if profConfigured is also set to Yes, the Profile Engine will run and generate data to support the estimation process.	Y
veeConfigured	Whether or not data is published from SDL to VEE or whether it bypasses VEE altogether.	N
veeEnabled	Whether or not VEE will do full validations of your sensor data or the very minimum validation of your sensor data (for example, data type must be valid).	Y

There are two VEE configuration objects that require setup once VEE is activated: veeDefaults.yaml and valDefaults.yaml.

veeDefaults.yaml

The columns for veeDefaults.yaml are as follows:

Column	Description
name	Specifies the name of the default parameter
value	The value for the parameter as a string
type	kdb data type for the value of the parameter (for example, byte, bool, symbols, etc.)
description	Describes the functionality of the parameter

name	value	type	description
isEnabled	1	bool	Specifies whether or not VEE is enabled
isTrace	0	bool	Specifies whether or not to turn on trace-level logging
aduMinDays	7	short	Minimum number of days that must be accumulated into the ADU (Average Daily Usage) before it can be used
profDays	4	short	Target number of day-of-week instances to include in profile
profMinDays	2	short	Minimum number of days before the profile can be used to provide data for that day
profMaxDays	60	short	Maximum number of days between the oldest reading to be estimated and the most recent contribution to the profile
profAllowEst	1	bool	Specifies whether estimated data can be incorporated into the profile
calcProfile	1	byte	Indicates whether a dynamic profile should be calculated from historical data
useSameAccount	1	bool	Indicates whether consumption data behind a sensor/account relationship change can be considered in the profile
holidayDOW	0	byte	Specifies the day of the week to use when a day for which profile info is requested is a holiday
useSimilarDays	1	bool	Indicates whether similar days can be used as a proxy for the specific day-of-the-week needing estimation
dayGroups	1111122	int	Specifies the day-of-the-week equivalencies to be used when useSimilarDays is true
allowClass	1	bool	Indicates whether class profiles can be used when insufficient information is available in the historical profile
classChanID	0	int	Specifies the class channel ID to use as reference data for the class
rollover	0.4	real	Specifies the acceptable difference between ending and starting register values at or below which the sensor may be considered to have rolled over (percentage of its capacity)
highValue	0W	long	Specifies the maximum allowable difference between starting and ending registers

valDefaults.yaml

This object contains default configuration parameters specific to the validation process. Each row of valDefaults.yaml belongs to a rule (the check being configured) and defines one name/value pair for that rule, along with the kdb type of the value (for example, byte, bool, symbols, etc.).

The seq parameter sets the order in which rules run:

seq: gap, flags, spike, dip, highlow, zeroes, inactive, sum

The following sections describe each rule and its parameters.

gap

Identifies missing readings.

Name	Value	Type	Description
isEnabled	1	bool	Specifies whether the gap check is enabled
intvMinReadings	12	short	Specifies the minimum number of readings required to determine that the configured interval length is less than the observed length
fillGaps	1	bool	Indicates whether placebo entries should be generated for missing readings
maxGapDays	90	short	Maximum number of days for which missing readings will be generated
maxEstDays	15	short	Maximum number of days for which missing readings will be estimated
intvMaxWait	0	short	Number of minutes to wait upon receipt of registers to start generating gaps for missing intervals
action	PEST	byte	Action to be taken if a reading fails the gap check

flags

Examines each reading for various sensor conditions and takes a configured action based on the reading's state.

Name	Value	Type	Description
isEnabled	1	bool	Specifies whether the flags check is enabled
actions	`NO_DATA`INV_VALUE`PARTIAL_DATA`MODIFIED`POWER_ON`POWER_OFF`SHORT_INT`LONG_INT`TIME_CHANGE`CLOCK`RESET`OVERFLOW`REVERSE`TEST`LOCKED`TAMPER`FLUX_SAG`FLUX_SWELL`REQUEST:PEST	string	Comma-delimited string specifying the flag checks to perform

spike

Identifies suspiciously high readings. The check is implemented using one of three methods: fixed (uses a preconfigured limit as the spike threshold), rank (uses the ratio of the highest reading and a specified rank reading to compute the threshold), or spread (uses the mean and a specified number of standard deviations to compute the spike threshold).

Name	Value	Type	Description
isEnabled	1	bool	Indicates whether the spike check is enabled
method	fixed	symbol	Specifies the spike method to perform (fixed, rank, or spread)
limit	20000000	long	Spike limit threshold (fixed method only)
minReadings	10	short	Minimum number of readings required to perform the spike check (rank and spread methods only)
allowHist	1	bool	Indicates whether surrounding readings are to be used in the event that there are fewer than .val.p.spike.minReadings readings available (rank and spread methods only)
readings	100	short	Maximum number of readings to consider at a time when performing the spike check (rank method only)
rank	3	short	The rank of the reading to be considered along with the highest reading in performing the spike check (rank method only)
minMax	1000000	long	Minimum value of highest reading below which the spike check is not to be performed (rank method only)
ratio	0.5	real	Ratio of the relative difference between the highest reading and the ranked reading; values at or below this ratio pass the spike check (rank method only)
spread	2	real	Number of standard deviations of the sample set to be used in computing the spike threshold (spread method only)
action	PEST	byte	Action to be taken if a reading fails the spike check

dip

Identifies suspiciously low readings, using the same fixed/rank/spread implementation methods as the spike check.

Name	Value	Type	Description
isEnabled	1	bool	Indicates whether the dip check is enabled
method	spread	symbol	Specifies the method to perform (fixed, rank, or spread)
limit	5000	long	Dip limit threshold (fixed method only)
minReadings	10	short	Minimum number of readings required to perform the dip check (rank and spread methods only)
allowHist	1	bool	Indicates whether surrounding readings are to be used in the event that there are fewer than .val.p.dip.minReadings readings available (rank and spread methods only)
readings	100	short	Maximum number of readings to consider at a time when performing the dip check (rank method only)
rank	3	short	The rank of the reading to be considered along with the highest reading in performing the dip check (rank method only)
maxMin	10000	long	Maximum value of lowest reading above which the dip check is not performed (rank method only)
ratio	0.5	real	Ratio of the relative difference between the highest reading and the ranked reading; values at or below this ratio passes the dip check (rank method only)
spread	2	real	Number of standard deviations of the sample set to be used in computing the dip threshold (spread method only)
action	PEST	byte	Action to be taken if a reading fails the dip check

highlow

Identifies cumulative usage values that appear off-nominal when compared with expected historical norms, using either recent readings or the historical usage profile to compute average daily usage.

Name	Value	Type	Description
isEnabled	1	bool	Indicates whether the high-low check is enabled
method	recent	symbol	Specifies the method to use (recent or profile)
minHours	20	short	Minimum number of hours before the high-low check should be performed
allowEst	1	bool	Indicates whether estimated usage values may be used
minRatio	0.95	real	Minimum ratio of current usage to reference value
maxRatio	1.05	real	Maximum ratio of current usage to reference value
allowClass	1	bool	Indicates whether the class profile can be used when insufficient data is available in the historical profile
action	PEST	byte	Action to be taken if a reading fails the high-low check

zeroes

Identifies active sensors having a suspicious number of consecutive zero readings (even if this sequence extends beyond the reading group).

Name	Value	Type	Description
isEnabled	1	bool	Indicates whether the zeroes check is enabled
maxCount	336	short	Maximum number of allowable consecutive zeroes
action	PEST	byte	Action to be taken if a reading fails the zeroes check

inactive

Identifies inactive sensors having non-zero readings.

Name	Value	Type	Description
isEnabled	1	bool	Indicates whether the inactive check is enabled
action	PEST	byte	Action to be taken if a reading fails the inactive check

sum

Identifies interval data that does not match its corresponding register data.

Name	Value	Type	Description
isEnabled	1	bool	Indicates whether the sum check is enabled
tolerance	0.099	real	Acceptable deviation between the cumulative register readings and the related interval readings (either absolute value or fractional ratio)
isRatio	1	bool	Indicates whether the threshold is expressed as a ratio or an absolute value
maxRange	120	short	Maximum number of minutes between an available register read and the starting/ending intervals
action	PEST	byte	Action to be taken if a reading fails the sum check

range

Identifies interval data that falls above or below a given threshold, or inside or outside a pair of configured thresholds.

Name	Value	Type	Description
isEnabled	1	bool	Indicates whether the range check is enabled
action	PEST	byte	Action to be taken if a reading fails the range check
lowerLimit	5000	long	Use a lower limit to determine validity
upperLimit	20000000	long	Use an upper limit to determine validity
rejectInside	0b	bool	Reject readings if they lie inside a specified lower or upper limit

timestamp

Identifies interval data whose timestamps fall outside of a preconfigured period of reporting time.

Name	Value	Type	Description
isEnabled	1	bool	Indicates whether the timestamp check is enabled
value	now	symbol	Indicates whether the timestamp is compared to the current time or the data collection time
timespan	7D00:00:00.000000000	timespan	The timestamp limit threshold; readings that exceed this threshold will fail the timestamp check
action	PEST	byte	Action to be taken if a reading fails the timestamp check

precision

Identifies interval data with decimal values.

Name	Value	Type	Description
isEnabled	1	bool	Indicates whether the precision check is enabled
granularity	0	long	Specifies the value that must evenly divide a reading value to pass the check
action	PEST	byte	Action to be taken if a reading fails the precision check

Configure estimation

There are two configuration objects for VEE estimation: estDefaults.yaml and scaleDefaults.yaml.

estDefaults.yaml

This object contains the default configuration parameters specific to the estimation process. KX Sensors supports the following estimation types:

Type	Description
Accept modified	Use readings that have been modified by an external data collection system
Fill	Use last actual or last estimated reading before the gap to fill the missing intervals
Linear	Use linear interpolation between the first and last valid intervals surrounding a gap to produce an even distribution of values
Historical	Use historical data from individual sensors to fill the missing intervals
Class Profile	Use a pre-configured class profile to fill the missing intervals

The columns are as follows:

Column	Description
rule	Specifies the scaling method
name	The name of the parameter
value	Specifies the value of the parameter
type	kdb data type for the value of the parameter (for example, byte, bool, symbols, etc.)
description	Description of the functionality

rule	name	value	type	description
	seq	acceptmod, hist, class, linear, fill	symbol	Estimation sequence.
acceptmod	isEnabled	1	bool	Indicates whether acceptmod estimation is enabled
fill	isEnabled	1	bool	Indicates whether fill estimation is enabled
	minGap	0	int	Minimum number of minutes requiring estimation for the fill method to be applied
	maxGap	21600	int	Maximum number of minutes requiring estimation for the fill method to be applied
	useLast	1	bool	Indicates whether the last value before the gap should be used as the fill element
	value	100000	long	Fill value to be used if useLast is false
	scaleSeq	classadu, cumsum	symbols	Scale sequence to apply to estimated data
linear	isEnabled	1	bool	Indicates whether linear estimation is enabled
	minGap	0	int	Minimum number of minutes requiring estimation for the linear method to be applied
	maxGap	21600	int	Maximum number of minutes requiring estimation for the linear method to be applied
	scaleSeq	cumsum	symbols	Scale sequence to apply to estimated data
hist	isEnabled	1	bool	Indicates whether historical estimation is enabled
	minGap	0	int	Minimum number of minutes requiring estimation for the historical method to be applied
	maxGap	21600	int	Maximum number of minutes requiring estimation for the historical method to be applied
	scaleSeq	cumsum	symbols	Scale sequence to apply to estimated data
class	isEnabled	0	bool	Indicates whether class estimation is enabled
	minGap	0	int	Minimum number of minutes requiring estimation for the class method to be applied
	maxGap	21600	int	Maximum number of minutes requiring estimation for the class method to be applied
	scaleSeq	classadu, cumsum	symbols	Scale sequence to apply to estimated data

scaleDefaults.yaml

This table contains the default scale parameters for estimation types. Scaling is carried out when the sum of estimated data differs from the sum of missing data; each estimate is then multiplied by some scale factor to align the interval data with the register read difference for that period.

The columns of scaleDefaults.yaml are:

Column	Description
rule	Specifies the scaling method
name	The name of the parameter
value	Specifies the value of the parameter
type	kdb data type for the value of the parameter (for example, byte, bool, symbols, etc.)
description	Description of the functionality

The current scaling methods are:

Method	Description
cumsum	Cumulative sum scaling method, which normalizes estimated values according to the differences in the cumulative register readings at the beginning and end of the reading period.
classadu	Class profile average daily usage, normalizes estimated values from a class profile in accordance with contemporaneous average daily usage for the sensor.

Result of the scaling method:

• Readings that failed scaling are flagged.
• Successful scaled readings are removed.

rule	name	value	type	description
cumsum	isEnabled	1	bool	Indicates whether cumsum scaling is enabled
	maxRange	120	short	Maximum number of minutes between an available register read and the starting/ending intervals
	minDelta	1000	long	Minimum difference between register readings and valid interval readings for cumsum scaling to be applied
classadu	isEnabled	1	bool	Indicates whether class ADU scaling is enabled
	maxAfter	120	short	Maximum number of minutes between an available register read and the ending gap interval for class ADU scaling to be applied
	maxBefore	120	short	Maximum number of minutes between an available register read and the starting gap interval for class ADU scaling to be applied
	minRange	60	short	Minimum number of minutes between starting and ending register reads for class ADU scaling to be applied
	minDelta	1000	long	Minimum difference between register readings and valid interval readings for class ADU scaling to be applied

Next steps

• Configure VEE by organization/channel
• Enable your VEE and prof service classes
• General considerations and maintenance