Logging, recovery and replication¶
Software or hardware problems can cause a q server process to fail, possibly resulting in loss of data not saved to disk at the time of the failure. A q server can use logging of updates to avoid data loss when failures occur; note that the message is logged only if it changes the state of the process’ data.
Logging is enabled by using the
-L command-line arguments.
$ q logTest -l
q)\p 5001 q)\l trade.q
q)/ this is a client q)h: hopen `:localhost:5001 q)h "insert[`trade](10:30:01; `intel; 88.5; 1625)" q)h "last trade" time | 10:30:01.000 sym | `intel price| 88.5 size | 1625
$ q logTest -l
q)last trade time | 10:30:01.000 sym | `intel price| 88.5 size | 1625
Updates done locally in the server process are logged to disk only if they are sent as messages to self. The syntax for this uses
0 as the handle:
q) // in server q)0 ("insert";`trade; (10:30:01.000; `intel; 88.5; 1625))
A logging server uses a .log file and a .qdb data file. The command
However, the checkpoint is path-dependent. Consider the following:
/tmp/qtest$ q qtest -l q)
/tmp/qtest$ ls qtest.log
/tmp/qtest$ $ls qtest.log qtest.qdb
*.qdbcheckpoint file is placed in the latter directory. For instance:
/tmp/qtest$ q qtest -l
q)\cd ../newqdir q)\l
/tmp/qtest$ ls qtest.log /tmp/qtest$ cd ../newqdir /tmp/newqdir$ ls qtest.qdb
$/tmp/qtest$ q /tmp/testlog -l
q).z.f /tmp/testlog q)\cd ../newqdir q)\l
/tmp/qtest$ ls . testlog.log testlog.qdb /tmp/qtest$ ls ../newqdir .
When you type
$ q logTest -l
-l option is recommended if you trust (or duplicate) the machine where the server is running. The
-L option involves an actual disk write (assuming hardware write-cache is disabled).
Another option is to use no logging. This is used with test, read-only, read-mostly, trusted, duplicated or cache databases.
Errors and rollbacks¶
If either message handler (
.z.ps) throws any error, and the state was changed during that message processing, this will cause a rollback.
Given a logging q process listening on port 5000, e.g. started with
$ q test -l -p 5000
$ q -r :localhost:5000:username:password
$ q /mylogs/test -l -p 5000
On start-up, the replicating process connects to the logging process, gets the log filename and record count, opens the log file, plays back that count of records from the log file, and continues to receive updates via TCP/IP. Each record is executed via value.
If the replicating process loses its connection to the logging process, you can detect that with
.z.pc. To resubscribe to the logging process, restart the replicating process.
Currently, only a single replicating process can subscribe to the master process. If another q process attempts to replicate from the master, the previous replicating process will no longer receive updates. If you need multiple replicating processes, you might like to consider kdb+tick.
A concise implementation of logger for q applications.