Validation, Estimation and Editing (VEE)¶
This page introduces the VEE engine's data pipeline and the validation and estimation rule sequences used to assess incoming sensor data quality.
The VEE engine allows you to validate, estimate and edit incoming data so that it can subsequently be assumed to be of an acceptable level of quality.
Overview¶
The following graphic shows the data pipeline for sensor validation and estimation.
Sensor data typically includes periodic usage data as well as cumulative sensor register readings and may arrive either in a continuous stream or in a series of batches. Data is first subjected to a preliminary validation process that checks the syntactical correctness of the received data and verifies that the transmitting organization is known to KX. Then, depending upon how you configure your system, it may be subjected to detailed validation rules that look for missing readings, usage pattern anomalies and overall consistency.
Validation rules¶
Validation rules are processed in a predefined validation sequence (.val.p.seq) that is fully configurable. Values that fail to meet the expected quality may have their reading state marked PEST (Pending Estimation) or PMAN (Pending Manual Intervention). If all readings are in the state PVAL (Pending Validation) at the end of the defined validation sequence, then they are marked as valid and data requiring estimation is run through the Estimation portion of VEE, if configured.
Estimation rules¶
Like validation rules, estimation rules are processed in a predefined estimation sequence (.est.p.seq) that is fully configurable. If the first rule in the sequence is successful, the status of the reading will be set to EST (Estimated) and no further processing is required. If, however, the first rule fails, KX Sensors will evaluate the second rule. If the second rule fails, KX Sensors will evaluate the third rule. And so on and so forth. If all attempts at estimation fail, the status of the reading will be set to PMAN (Pending Manual Intervention).