Skip to content

Logging

Logging is a standard tool for system operators to gain observability into their applications. kdb Insights Enterprise uses standard industry practices for logging. Log messages are generated in a structured format and emitted to STDOUT where they are consumed by standard logging stacks. Kubernetes collects the log messages into files, where log agents consume them and send them to a downstream log aggregator.

Kubernetes logging.

To configure the log level of your application at installation time, use the details provided on the Configure logging page.

Logging stack

kdb Insights Enterprise does not ship with a logging stack, you can choose one based on your use-case and preference. The three main cloud providers (GCP, AWS, and ACS) have managed Kubernetes services which provide the ability to automatically integrate with their logging stacks.

Alternatively, you can use a third-party agent such as fluent-bit.

You can configure fluent-bit to forward to any of its supported downstream targets.

Install

Supported targets

KX terraform deploys

If installing your environment with the KX Terraform scripts, the kdb Insights Enterprise Terraform modules are configured by default to deploy and utilize fluent-bit for each of the cloud providers.

Sending logs to other log aggregation systems

The fluent-bit configuration is stored on the following files and can be updated as required:

terraform/k8s_config_aws/logging.tf
terraform/k8s_config_azure/logging.tf
terraform/k8s_config_gcp/logging.tf

Audit logging

You can use audit logging in kdb Insights and kdb Insights Enterprise to identify user changes and state changing actions that may affect the running of the system.

Enable audit logging by adding the following setting in your global values.yaml file:

global:
  auditLog:
    enabled: true

Refer to Applying configuration changes for more details on how to make changes to the values.yaml file.

Audit logging is available for:

  • Logins
  • Queries using one of the following:
  • Package actions via the CLI or web interface including:
    • Creation and deletion of packages, databases, pipelines and views
    • Edit to packages, databases, pipelines and views
    • Deployment and teardown or packages, databases and pipelines
  • File uploads via the Upload reader in a stream processor pipeline

Audit log message format

All audit log messages have the "component": "Audit" tag to separate them from standard log messages.

They also contain the "level" and "message" tags common to all log messages, which identify severity and the action to be logged respectively.

The following is an indicative example of an audit log message:

{"time":"<time>","component":"Audit","level":"INFO","message":"<message>","corr":"<correlation-id>"}

Correlation ID

All audit logs have a correlation ID which persists throughout the lifetime of the event. Each audit log message can propagate through to multiple different components in kdb Insights or kdb Insights Enterprise. A single audit event can create multiple log messages. Messages relating to the same event all share the same correlation ID.

The correlation ID is automatically assigned by the API gateway when a request passes through it - this includes queries, login events, and package actions such as deployments and configuration changes. It appears in the log as "corr" and can be used to precisely identify which event a log message refers to.

Note

corr is distinct from logCorr and auditID, which are optional parameters a caller can set on a REST request to tag and trace their own queries through the logs. See Specifying logCorr and auditID for details.

Tracing a query to a user

To identify which user ran a specific query, use the corr field to trace the request across audit log entries.

  1. Find the correlation ID for the query of interest. It appears as the "corr" field in any log entry for that request.

  2. Search logs for that correlation ID using the CLI:

    kxi obs logs --corr "<correlation-id>"

    If your monitoring stack includes Grafana, you can also use the pre-built kdb Insights Grafana Dashboard to search by correlation ID.

    Alternatively, search directly with kubectl:

    kubectl logs <pod-name> | grep '<correlation-id>'

  3. Find the userID in the matching audit log entries. The corr value is generated for all audited actions, including logins, queries, package actions through the CLI or web interface, and file uploads via the Upload reader. The audit log entry for the correlation ID will include a userID that identifies who performed the action.

    For queries made through the Query Window, Views, or REST, audit log messages from the RC, DAP, and Agg components include the userID directly. For example:

    {"component":"Audit","level":"INFO","message":"Executing api=.kxi.getData userID=<user-id> auditID=<corr-id> ..."}

    The username can also be found directly in the SG (Service Gateway) logs, which include a user field in INFO-level audit entries.

  4. Look up the user in Keycloak using the userID: In the Keycloak admin UI, navigate to Events and search by user ID. You must first enable login/logout event tracking under Events > Config - without this, user lookup by ID will return no results. See the Keycloak documentation for further details.

Back to top