Logging, recovery and replication¶
Software or hardware problems can cause a q server process to fail, possibly resulting in loss of data not saved to disk at the time of the failure. A q server can use logging of updates to avoid data loss when failures occur; note that the message is logged only if it changes the state of the process’ data.
Logging is enabled by using the
-L command-line arguments.
$ q logTest -l
Now update messages from clients are logged. For instance:
q)\p 5001 q)\l trade.q
Assume that the q server process dies. If we now restart it with logging on, the updates logged to disk are not lost:
q)/ this is a client q)h: hopen `:localhost:5001 q)h "insert[`trade](10:30:01; `intel; 88.5; 1625)" q)h "last trade" time | 10:30:01.000 sym | `intel price| 88.5 size | 1625
$ q logTest -l
q)last trade time | 10:30:01.000 sym | `intel price| 88.5 size | 1625
Updates done locally in the server process are logged to disk only if they are sent as messages to self. The syntax for this uses
0 as the handle:
q) // in server q)0 ("insert";`trade; (10:30:01.000; `intel; 88.5; 1625))
A logging server uses a .log file and a .qdb data file. The command
checkpoints the .qdb file and empties the log file.
However, the checkpoint is path-dependent. Consider the following:
A listing of the directory gives:
/tmp/qtest$ q qtest -l q)
Back in q:
/tmp/qtest$ ls qtest.log
However, if there is a change of directory within the q session then the
/tmp/qtest$ $ls qtest.log qtest.qdb
*.qdbcheckpoint file is placed in the latter directory. For instance:
/tmp/qtest$ q qtest -l
q)\cd ../newqdir q)\l
The simplest solution is to provide a full path to the log file at invocation.
/tmp/qtest$ ls qtest.log /tmp/qtest$ cd ../newqdir /tmp/newqdir$ ls qtest.qdb
$/tmp/qtest$ q /tmp/testlog -l
q).z.f /tmp/testlog q)\cd ../newqdir q)\l
/tmp/qtest$ ls . testlog.log testlog.qdb /tmp/qtest$ ls ../newqdir .
When you type
this reads the data file (.qdb), log file, and the q script file logTest.q, if present. If any of the three files exists (.q, .qdb, and .log), they should all be in the same directory.
$ q logTest -l
-l option is recommended if you trust (or duplicate) the machine where the server is running. The
-L option involves an actual disk write (assuming hardware write-cache is disabled).
Another option is to use no logging. This is used with test, read-only, read-mostly, trusted, duplicated or cache databases.
Errors and rollbacks¶
If either message handler (
.z.ps) throws any error, and the state was changed during that message processing, this will cause a rollback.
Given a logging q process listening on port 5000, e.g. started with
a single q process can replicate that logging process via
$ q test -l -p 5000
if starting these processes from different directories, be sure to specify the absolute path for the logging process, e.g.
$ q -r :localhost:5000:username:password
the replicating process will receive this information when it connects. On start-up, the replicating process will load the .qdb file and play back the log file, and continue to receive updates via IPC.
$ q /mylogs/test -l -p 5000
Currently, only a single replicating process can subscribe to the master process. If another q process attempts to replicate from the master, the previous replicating process will no longer receive updates. If you need multiple replicating processes, you might like to consider kdb+tick.