Cookbook/Logging

From Kx Wiki
Jump to: navigation, search

The wiki is moving to a new format and this page is no longer maintained. You can find the new page at code.kx.com/q/cookbook/logging.

The wiki will remain in place until the migration is complete. If you prefer the wiki to the new format, please tell the Librarian why.

Contents

Logging, recovery and replication

Software or hardware problems can cause a kdb+ server process to fail, possibly resulting in loss of data not saved to disk at the time of the failure. A kdb+ server can use logging of updates to avoid data loss when failures occur; note that the msg is logged only if it changes the state of the process' data.

Logging is enabled by using the '-l' or '-L' command-line arguments.

q logTest -l
q)\p 5001
q)\l trade.q

Now update messages from clients are logged. For instance:

q)/ this is a client
q)h: hopen `:localhost:5001
q)h "insert[`trade](10:30:01; `intel; 88.5; 1625)"
q)h "last trade"
time | 10:30:01.000
sym  | `intel
price| 88.5
size | 1625

Assume that the kdb+ server process dies. If we now restart it with logging on, the updates logged to disk are not lost:

q logTest -l
q)last trade
time | 10:30:01.000
sym  | `intel
price| 88.5
size | 1625
Warning.png Updates done locally in the server process are logged to disk only if they are sent as messages to self. The syntax for this uses '0' as the handle. Example:
q) // in server
q)0 ("insert";`trade; (10:30:01.000; `intel; 88.5; 1625))

Checkpointing

A logging server uses a .log file and a .qdb data file. The command

q)\l

checkpoints the .qdb file and empties the log file.

However, the checkpoint is path dependent. Consider the following:

/tmp/qtest$ q qtest -l
q)

A listing of the directory gives:

/tmp/qtest$ ls
qtest.log

Back in q:

q)\l

yields

/tmp/qtest$ $ls
qtest.log
qtest.qdb

However, if there is a change of directory within the q session then the *.qdb checkpoint file is placed in the latter directory. For instance:

/tmp/qtest$ q qtest -l
q)\cd ../newqdir
q)\l

results in

/tmp/qtest$ ls
qtest.log
/tmp/qtest$ cd ../newqdir
/tmp/newqdir$ ls
qtest.qdb

The simplest solution is to provide a full path to the log file at invocation.

$/tmp/qtest$ q /tmp/testlog -l
q).z.f
/tmp/testlog
q)\cd ../newqdir
q)\l

results in

/tmp/qtest$ ls
. testlog.log testlog.qdb
/tmp/qtest$ ls ../newqdir
.

Files

When you type

q logTest -l

this reads the data file (.qdb), log file, and the q script file logTest.q, if present. If any of the three files exists (.q, .qdb, and .log), they should all be in the same directory.

Logging options

The '-l' option is recommended if you trust (or duplicate) the machine where the server is running. The '-L' option involves an actual disk write (assuming hardware write cache is disabled).

Another option is to use no logging. This is used with test, readonly, readmostly, trusted, duplicated or cache databases.

Errors and rollbacks

If either message handler (.z.pg or .z.ps) throws any error, and the state was changed during that message processing, this will cause a rollback.

Replication

Given a logging kdb+ process listening on port 5000, e.g. started with

q test -l -p 5000

a single kdb+ process can replicate that logging process via

q -r :localhost:5000:username:password

if starting these processes from different directories, be sure to specify the absolute path for the logging process, e.g.

q /mylogs/test -l -p 5000

the replicating process will receive this information when it connects. On startup, the replicating process will load the .qdb file and play back the log file, and continue to receive updates via ipc.

Currently, only a single replicating process can subscribe to the master process. If another kdb+ process attempts to replicate from the master, the previous replicating process will no longer receive updates. If you need multiple replicating processes, you might like to consider kdb+tick

Personal tools
Namespaces
Variants
Actions
Navigation
Print/export
Toolbox