RT Stream log files

When diagnosing issues with a data pipeline that uses an RT cluster (known in the UI as a KX Insights Stream) it is helpful to know where the stream logs reside. This is intended to assist you when the data does not flow to database, allowing you to find the files and identify how far the messages have travelled through the pipeline.

Stream log files

Each publisher and subscriber writes and reads from a set of RT log files in a directory representing a stream of messages.

Publishers and subscribers specify a 64 bit position in the stream when writing or reading from it. That position first identifies the log file which contains that message, with RT log files being named as log.x.y where:

• x is the top 20 bits of the position
• y is the next 14 bits of the position

The remaining 30 bits then locate the start of the message in the correct log file. Therefore, each log file has a maximum size of 1GB before automatically rolling to the next log file.

External publishers

component	file location
ODBC driver on Linux	/tmp/rt
Java SDK on Linux	KXI_LOG_PATH environment variable
ODBC driver or Java SDK on Windows	Path defined in HKEY_CURRENT_USER\SOFTWARE\kx\kodbcDriver\LogPath

RT Stream

component	file location
Inbound RT Stream files	/s/in/$TOPIC
Outbound RT Stream files	/s/out/OUT

Internal Publishers

component	file location
Stream Processor	/sp/checkpoints/rt_pub/\({workername}.\)-${streamname}

Internal Subscribers

component	file location
Stream Processor	/sp/checkpoints/rt_pub/\({ASSEMBLY_NAME}-\)
Storage Manager	/logs/rt/{ASSEMBLY_NAME}-dap-rdb-[012]/{ASSEMBLY_NAME}-${streamname}