Skip to content

QLog

Logging for kdb+ applications

Quick start

QLog supports multiple endpoint types through the same simple interface and lets you write to them concurrently. It generates structured, formatted log messages tagged with a severity level and component name. Routing rules can also be configured to suppress or route based on these tags.

Existing q libraries that implement their own formatting can still use QLog via the base APIs. This lets them do their own formatting but still take advantage of the QLog-supported endpoints.

Integration with cloud-logging applications providers can easily be achieved using logging agents. These can be set up alongside running containers to capture their output and forward to logging endpoints applications.

Quick start

The Quick Start guide includes basic examples of the library itself and also how to set QLog up alongside a logging agent for each cloud integration.

Interface

The logging interface provides APIs to publish structured messages to log endpoints. Each message is timestamped and decorated with additional metadata. This allows log consumers to parse and act on the messages.

Each message is timestamped by the log API and tagged with a severity level and component name. The level generally indicates what priority to assign to the event. The component is usually used to identify the part of an application generating the logs, i.e. a single process may load multiple libraries using different components. These two tags are also used for routing.

An example message:

{"time":"2020-12-15T14:28:23.098432000z","component":"Monitor","level":"INFO","message":"Simple message"}

The .com_kx_log.init initializes the log endpoints and the default routing. The example below sets up a stdout endpoint with no specific routing rules. The ID returned identifies that endpoint and can be used to update routings or close it later.

q)0N!id:.com_kx_log.init[`:fd://stdout; ()]
,8c6b8b64-6815-6084-0a3e-178401251b68

Each log component is generated separately and has its own set of API handlers. A component is generated by calling the .com_kx_log.new API. The returned value is a dictionary of APIs with one per severity level. The component can specify a routing for this specific component otherwise it will inherit the default. The example shows the generation of a component and logging messages of different severities.

q).mon.log:.com_kx_log.new[`Monitor; ()]
q)key .mon.log
`trace`debug`info`warn`error`fatal

q).mon.log.debug "Monitor initialized"
{"time":"2020-12-15T14:39:47.884003000z","component":"Monitor","level":"DEBUG","message":"Monitor initialized"}
q).mon.log.fatal "Process state corrupted"
{"time":"2020-12-15T14:40:06.852199000z","component":"Monitor","level":"FATAL","message":"Process state corrupted"}

Formatting

As shown above, the log handlers generate structured JSON messages by default. The API argument is encoded in a message field, with the rest of the fields populated by the API. In the simplest form, this argument can be a string.

q).mon.log.debug "Monitor initialized"
{"time":"2020-12-15T14:39:47.884003000z","component":"Monitor","level":"DEBUG","message":"Monitor initialized"}

However it is common practice for applications to build a log string from text and arguments. The drawback of this is that it is onerous on the developer, error-prone and less performant as the string is built even if the log is not published. To support this the log API supports a list format, consisting of a string template and arguments.

q)list:("Initialized connection to uid=%1, name=%2"; rand 10; `monitor)
q).mon.log.debug list
{"time":"2020-12-15T15:24:21.993561000z","component":"Monitor","level":"DEBUG","message":"Initialized connection to uid=9, name=monitor"}

The input can also be a dictionary with a message key corresponding to the log message. The message value can be a string or list as above and the rest of the keys will be joined to the JSON message.

q).mon.log.fatal `message`version!("Process state corrupt"; "1.0.2")
{"time":"2020-12-15T15:25:17.289927000z","component":"Monitor","level":"FATAL","message":"Process state corrupt","version":"1.0.2"}

Service details

Commonly an application will register some service metadata in order to record it with each log message. This is supported by the application registering a dictionary of metadata, which gets appended to the log payload. The .com_kx_log.setServiceDetails API is used to set the metadata.

q).com_kx_log.setServiceDetails `service`version!(`rdb; "1.0.2")
q).mon.log.fatal "Process state corrupt"
{"time":"2020-12-15T16:04:11.049534000z","component":"Monitor","level":"FATAL","message":"Process state corrupt","service":"rdb","version":"1.0.2"}

The service details can also be set using the .com_kx_log.configure API.

Correlators

APIs are provided to manage logging correlators. This enables the application to generate or provide a correlator into QLog. This correlator will be added to the log payload. It can be used to group log entries together and potentially to a specific source event.

The .com_kx_log.setCorrelator and .com_kx_log.unsetCorrelator APIs set and unset the correlator. Below is an example of how this might work.

q)0N!id:.com_kx_log.setCorrelator[]
"f3f3a5aa-7ac7-374e-20f5-d264c99041a6"

q).mon.log.info[("API request received from %1"; "gw")]
{"time":"2020-12-15T16:13:51.270965000z","corr":"f3f3a5aa-7ac7-374e-20f5-d264c99041a6","component":"Monitor","level":"INFO","message":"API request received from gw"}

q).mon.log.debug "Request complete"
{"time":"2020-12-15T16:13:51.271137000z","corr":"f3f3a5aa-7ac7-374e-20f5-d264c99041a6","component":"Monitor","level":"DEBUG","message":"Request complete"}

q).com_kx_log.unsetCorrelator[]

This implementation could easily be injected into the .z.p[gs] handlers and provide log correlation on all incoming sync and async requests.

Routing

Routing allows different log levels and components to be sent to different endpoints. This is especially useful when using cloud applications where charges are based on the volume of messages being logged. In this scenario, an application might log all messages with level of ERROR and above to the cloud with everything else logged to files.

Each logging component can be configured with its own routing rules per endpoint and if not configured will inherit the default values. When a message is being logged, the routing is checked and if the severity level is equal or above the configured value for the component, then it will be logged for that endpoint.

The default levels (in order of severity) are; TRACE, DEBUG, INFO, WARN, ERROR, FATAL.

The example below illustrates this using two endpoints with different routings.

  • Default routing logs everything to STDOUT and only logs INFO and above to file
  • Monitor component uses the default routing
  • Discovery uses a custom routing where logs everything INFO and above to stdout, and only ERROR and above to file.
q).com_kx_log.i.levels
`TRACE`DEBUG`INFO`WARN`ERROR`FATAL

q)0N!ids:.com_kx_log.init[`:fd://stdout`:fd:///tmp/app.log; ``INFO]
8c6b8b64-6815-6084-0a3e-178401251b68 5ae7962d-49f2-404d-5aec-f7c8abbae288

q).mon.log:.com_kx_log.new[`Monitor; ()]
q).mon.log.trace "Started monitor"            // logs to stdout only
q).mon.log.fatal "Process corruption"         // logs to stdout & file

q).sd.log:.com_kx_log.new[`Discovery; ids!`INFO`ERROR]
q).sd.log.trace "Initialized discovery"       // no logs published
q).sd.log.error "Discovery server corrupt"    // logs to stdout & file

Routings can be checked or updated using the .com_kx_log.getRoutings and .com_kx_log.setRouting respectively.

Existing library integration

Applications may wish to use their own existing log library for handling log formatting, and levels or components. In this case, these APIs can use the base APIs for creating endpoints and logging messages.

An endpoint can be created using .com_kx_log.lopen and a message logged with .com_kx_log.msg. A basic example is provided below where the process opens a connection to two endpoints using .com_kx_lopen and writes messages to both using .com_kx_log.msg. The existing .app.log handler has been updated to call the QLog API.

.com_kx_log.lopen'[`:fd://stdout`:fd:///tmp/app.log]

// .app.log:{[x] -1 .j.j x}
.app.log:{[x] .com_kx_log.msg .j.j x; }

API reference

Full API reference

Endpoints

The logging endpoints in QLog are encoded as URLs with two main types: file descriptors and REST.

The file descriptor endpoints supported are;

:fd://stdout
:fd://stderr
:fd:///path/to/file.log

Note the leading slash for a fully-qualified directory.

REST endpoints are encoded as standard HTTP/S URLs. There are three types supported, corresponding to each of the major cloud providers.

:https://logging.googleapis.com
:https://logs.${region}.amazonaws.com
:https://${workspaceID}.ods.opinsights.azure.com

Endpoints are set up using .com_kx_log.lopen or .com_kx_log.init. An endpoint can be a symbol or a dictionary. The symbol is simply the URL but the dictionary format enables some further flexibility. It supports the following keys

url         symbol    endpoint URL
provider    symbol    REST provider: gcp, aws, azure
metadata    dict      dictionary of endpoint metadata
formatter   symbol    per-endpoint formatter API

The url field is required for file descriptors but is optional for REST endpoints; it will use a default URL if blank.

The provider field is required for REST endpoints, as it indicates which cloud URL and API format to use.

Metadata

Metadata is used to provide additional information about the endpoint. It is mainly used for REST endpoints to provide required parameters. When building requests to write to each endpoint, the metadata can be used in the payload or for authentication. It is also available in the endpoint formatters so customization to the log messages is possible.

Formatters

When supporting different endpoints concurrently, the message payload for each will differ, e.g. writing to a REST endpoint requires building a HTTP request, whereas STDOUT is a string. To assist with this, each endpoint can have its own formatter.

Default formatters are provided for the supported endpoint types but the application can specify a custom formatter with the endpoint argument. The default formatters are:

.com_kx_log.fd.fmt  [entry;metadata]
.com_kx_log.gcp.fmt [entry;metadata]
.com_kx_log.aws.fmt [entry;metadata]
.com_kx_log.az.fmt  [entry;metadata]

where

  • entry (string or dict) is the log entry
  • metadata (dict) is endpoint metadata

returns the formatted message.

The first of these, .com_kx_log.fd.fmt, uses entry as the entire message body (it ignores extra dictionary fields or endpoint metadata) and returns it as a string.

The others join message, metadata and log dictionary fields and return a dictionary.

REST

GCP

To log to Cloud Logging when not on a GCP instance, you need a bearer token.

Download the Gcloud command-line tool.

Go to the service account page and download a service account key file.

Export the path to that file: 

export GOOGLE_APPLICATION_CREDENTIALS=/path/to/file.json

Generate the access token:

gcloud auth application-default print-access-token

Include this as the bearerToken in metadata. (If you are on a GCP instance, the bearer token can be omitted.)

// Replace {projectID} and {logType} with the appropriate values. 
// LogType must be URL-encoded.
metadata:`logName`resource`bearerToken!
  ("projects/{projectID}/logs/{logType}"; 
    (enlist `type)!(enlist "gce_instance"); 
    "myToken")

// By default, logs are sent to https://logging.googleapis.com
// You can log to a different URL by including it 
// in the dictionary passed to lopen
// .com_kx_log.lopen`url`provider`metadata!(`:https://logging.googleapis.com;`gcp;md)
.com_kx_log.lopen `provider`metadata!(`gcp; metadata)
.log.msg "Hello World!"

AWS

// To authenticate against AWS, the credentials must be associated 
// with the URL receiving the log messages.
// Replace AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY with your credentials
.kurl.register (`aws_cred; "*.amazonaws.com"; "";
    `AccessKeyId`SecretAccessKey!(AWS_ACCESS_KEY_ID; AWS_SECRET_ACCESS_KEY))

// The group and stream must already exist in CloudWatch
// Replace <region> with the CloudWatch region you are using (e.g. us-east-2)
metadata:`logGroup`logStream!("myGroup"; "myStream")
.com_kx_log.lopen `url`provider`metadata!
  (`$":https://logs.<region>.amazonaws.com"; `aws; metadata)
.com_kx_log.msg "Hello world!"

Two calls on first log

Two calls are required the first time logs are sent to AWS, as the sequence token must be retrieved.

If you close the endpoint immediately after sending a message, it will fail with InvalidSequenceTokenException.

Azure

As Azure authentication is workspace-specific, .kurl.register and .kurl.deregister are called automatically when opening or closing endpoints.

// Replace AZURE_SHARED_KEY and WORKSPACE_ID with the appropriate values
metadata: (!) . flip (
  (`logType;      "loggingTest");
  // The shared key is either the primary or secondary key of the workspace
  (`sharedKey;    AZURE_SHARED_KEY);
  (`workspaceID;  WORKSPACE_ID) )

// By default, logs are sent to https://{workspaceID}.ods.opinsights.azure.com
// You can log to a different URL by including it in the dictionary passed to lopen
// .com_kx_log.lopen `provider`metadata`url!
//   (`azure; metadata; `$":https://{workspaceID}.ods.opinsights.azure.com")
id: .com_kx_log.lopen `provider`metadata!(`azure; metadata)
.com_kx_log.msg "Hello world!"
.com_kx_log.lclose id

Configuration

Elements of the log library are configurable. The best way to configure qlog is to set the KXI_LOG_CONFIG environment variable and point it to a JSON config file. This allows the system administrator to use a system-wide config file.

The file supports;

  • log endpoints
  • format mode
  • component-level routings
$ export KXI_LOG_CONFIG=/opt/kx/app/qlog.json
$ cat $KXI_LOG_CONFIG
{
  "endpoints": [ "fd://stdout", "fd://stderr" ],
  "formatMode": "json",
  "routings": {
    "DEFAULT": "INFO",
    "qlog": "DEBUG"
  }
}

This file will be loaded on startup and automatically applied. A sample file is provided here.

Other options

The library can also be configured using individual environment variables or by invoking the .com_kx_log.configure API.

The following environment variables are supported and applied on load in the same way as KXI_LOG_CONFIG.

Env Var Description Example
KXI_LOG_DEST Endpoints fd://stdout,fd://stderr
KXI_LOG_FORMAT Message format json
KXI_LOG_LEVELS Component routing DEFAULT:INFO, qlog:DEBUG

The .com_kx_log.configure API can be invoked by user code and provides more configuration options. It takes a dictionary and should be done on startup prior to initializing endpoints.

The default message format can be modified:

  • The JSON timestamp field is a kdb+ UTC datetime called time by default. The name and kdb+ times used can be set with the jsonTime key. Supported kdb+ types are one of "zptdZPTD".
  • The message format can be changed to text by setting formatMode to text.
  • The application can use its own formatter to generate log strings by setting the customFormatter field.

The default severity levels are TRACE, DEBUG, INFO, WARNING, ERROR, and FATAL. These can be changed to custom values using the logLevels field. The list should be ordered by severity from low to high.

Service details can also be specified using this API. The serviceDetails field should be set as a dictionary.

See the .com_kx_log.configure definition under the APIs section for examples.

This API should be called before initializing endpoints and performing any logging.