Skip to content

Disk requirements and layouts

This page covers disk type recommendations and how disk I/O affects ingestion and query performance in KX Sensors.

Exact disk requirements depend on your particular use case and workload. For most systems, use directly attached SSDs. Network-attached storage with slower I/O may be suitable for HDB tiers that you query infrequently.

Also consider RAID for throughput or redundancy.

Disk I/O directly affects data write-down performance during EOIs and EODs, as well as query performance. It also directly affects ingestion performance, because all ingestion is routed via Reliable Transport (RT) or Transit Portal (TP+), both of which persist all data to disk.

To avoid I/O contention during spikes of activity, provision separate devices for the following categories of data:

  • IDB and HDB files
  • RT data logs
  • Application logs
  • Reliable Transport application logs. As of KX Sensors 3.3, these are written to the KX Sensors application log directory along with every other process, so they share that device by default. See Application logs. In KX Sensors v3 releases before 3.3, RT application logs went to the Docker logs instead: you specified the location in the DOCKER_DATA_ROOT variable in install.profile during installation, and edited it directly in the systemd Docker unit file afterwards.

To size each disk, run the representative workload in a staging system and extrapolate the growth of data, taking into consideration the retention periods you want for all the categories listed above. Generally, retain several days of application and RT logs to support troubleshooting and analysis in production systems.

Once the system is in production, monitor the drives backing each of these locations closely to avoid disk space issues and potential data loss. See Set up volume/hardware monitoring.

Note

Log output can be sizeable. For example, 20 GB per day for application logs and RT logs is typical.

Next steps