Logging, recovery and replication

Software or hardware problems can cause a q server process to fail, possibly resulting in loss of data not saved to disk at the time of the failure. A q server can use logging of updates to avoid data loss when failures occur; note that the message is logged only if it changes the state of the process’ data.

Logging is enabled by using the -l or -L command-line arguments.

$ q logTest -l
q)\p 5001
q)\l trade.q
Now update messages from clients are logged. For instance:
q)/ this is a client
q)h: hopen `:localhost:5001
q)h "insert[`trade](10:30:01; `intel; 88.5; 1625)"
q)h "last trade"
time | 10:30:01.000
sym  | `intel
price| 88.5
size | 1625
Assume that the q server process dies. If we now restart it with logging on, the updates logged to disk are not lost:
$ q logTest -l
q)last trade
time | 10:30:01.000
sym  | `intel
price| 88.5
size | 1625

Warning

Updates done locally in the server process are logged to disk only if they are sent as messages to self. The syntax for this uses 0 as the handle:

q) // in server
q)0 ("insert";`trade; (10:30:01.000; `intel; 88.5; 1625))

Checkpointing

A logging server uses a .log file and a .qdb data file. The command

q)\l
checkpoints the .qdb file and empties the log file.

However, the checkpoint is path-dependent. Consider the following:

/tmp/qtest$ q qtest -l
q)
A listing of the directory gives:
/tmp/qtest$ ls
qtest.log
Back in q:
q)\l
yields
/tmp/qtest$ $ls
qtest.log
qtest.qdb
However, if there is a change of directory within the q session then the *.qdb checkpoint file is placed in the latter directory. For instance:
/tmp/qtest$ q qtest -l
q)\cd ../newqdir
q)\l
results in
/tmp/qtest$ ls
qtest.log
/tmp/qtest$ cd ../newqdir
/tmp/newqdir$ ls
qtest.qdb
The simplest solution is to provide a full path to the log file at invocation.
$/tmp/qtest$ q /tmp/testlog -l
q).z.f
/tmp/testlog
q)\cd ../newqdir
q)\l
results in
/tmp/qtest$ ls
. testlog.log testlog.qdb
/tmp/qtest$ ls ../newqdir
.

Files

When you type

$ q logTest -l
this reads the data file (.qdb), log file, and the q script file logTest.q, if present. If any of the three files exists (.q, .qdb, and .log), they should all be in the same directory.

Logging options

The -l option is recommended if you trust (or duplicate) the machine where the server is running. The -L option involves an actual disk write (assuming hardware write-cache is disabled).

Another option is to use no logging. This is used with test, read-only, read-mostly, trusted, duplicated or cache databases.

Errors and rollbacks

If either message handler (.z.pg or .z.ps) throws any error, and the state was changed during that message processing, this will cause a rollback.

Replication

Given a logging q process listening on port 5000, e.g. started with

$ q test -l -p 5000
a single q process can replicate that logging process via
$ q -r :localhost:5000:username:password
if starting these processes from different directories, be sure to specify the absolute path for the logging process, e.g.
$ q /mylogs/test -l -p 5000
the replicating process will receive this information when it connects. On start-up, the replicating process will load the .qdb file and play back the log file, and continue to receive updates via IPC.

Currently, only a single replicating process can subscribe to the master process. If another q process attempts to replicate from the master, the previous replicating process will no longer receive updates. If you need multiple replicating processes, you might like to consider kdb+tick.