Skip to content

How to keep a q process running

This page explains how to keep a q process running without a person watching it: what each approach gives you, how to set it up, and which q behaviors matter when a process stops, restarts or fails.

Many q processes are short-lived. A script parses a file, writes the result, and exits. Others are meant to run for days or months:

  • A writer accepts data from upstream processes and is responsible for persisting it. If it stops, incoming data has nowhere to go, and anything it held in memory but had not written is at risk. A tickerplant, which logs every update to disk before passing it on, is the classic example.
  • An HDB (historical database) serves queries on a database to other users. It has to answer at all times, including after the machine reboots at 3 a.m.

Starting such a process at a terminal works until you close the terminal, the process crashes, or the machine restarts. The following table compares approaches that handle those events to different degrees. Choose the simplest one that covers the events you care about.

Approach Survives logout Restarts after a crash Starts at boot Spans several hosts
nohup, tmux or screen Yes No No No
Daemon Yes No No No
systemd service Yes Yes Yes No
Docker or Docker Compose Yes Yes Yes No
Kubernetes Yes Yes Yes Yes
Your own supervisor Depends Yes Depends Depends

The examples on this page run Linux commands and use an HDB started like this:

q hdb.q -db /data/hdb -p 5010 -q

where hdb.q loads the database named by -db:

/ hdb.q: serve the database given by -db
o:.Q.opt .z.x
.Q.lo[hsym `$first o`db;0b;0b]

.Q.lo loads the database like \l. With its second argument 0b, it does not change the working directory to the database root, so a file that the process writes with a relative path, such as a log or a PID file, stays out of the database. With its third argument 0b, it does not run any scripts that it finds in the database directory. A real HDB script also defines the functions that clients call, as on the HDB page.

Prerequisites

  • KDB-X is installed, with a valid license.
  • A Linux host. The systemd, Docker and Kubernetes sections need those tools installed and the permissions to use them.

How q behaves without a terminal

Every approach on this page runs q without an interactive terminal. Four q behaviors decide whether that works.

q needs a listening port to stay alive. When standard input is closed or redirected from /dev/null, q runs its startup script and then exits with code 0, unless it listens on a port with -p. A timer alone does not keep it running. A server process has a port anyway. A process that only does scheduled work needs one too.

An error during startup ends the process with code 1. At a terminal, q stops at the first error in a script and waits at a debug prompt. Without a terminal, there is no prompt to wait at, so q prints the error to stderr and exits with code 1. Supervisors read this as a failure.

SIGTERM is a clean exit. When q receives SIGTERM, it finishes the task in progress and then exits as if it had evaluated exit 0. Before it exits, it calls .z.exit, which is the place to flush or save state. A writer might define:

.z.exit:{[code] `:/data/writer/trade set trade; -1"saved trade before exit ",string code;}

SIGTERM is the signal that kill, systemctl stop, docker stop and Kubernetes send to stop a process. Each of them sends SIGKILL if the process has not exited within a grace period, so make that period longer than your .z.exit handler takes.

SIGKILL and crashes skip .z.exit. A process killed with kill -9, by the kernel out-of-memory killer, or by a crash, ends without running .z.exit, and its exit status reports the signal (137 for SIGKILL in a shell). State held only in memory is lost. Design a writer so that it can rebuild its state after a restart: a tickerplant's subscribers, for example, replay its log.

A client can also end the process by sending exit 0, or exit with any other code, over IPC. To prevent that, start q with -u 1, which also restricts file access, or control what clients may run with message handlers.

Run in the background

The quickest way to keep a process running after you log out is to detach it from your terminal with nohup:

nohup q hdb.q -db /data/hdb -p 5010 -q < /dev/null > hdb.log 2>&1 &
  • nohup makes the process ignore the hang-up signal that the terminal sends when it closes.
  • < /dev/null closes standard input, so q does not try to read from a terminal that has gone.
  • > hdb.log 2>&1 sends stdout and stderr to a file. Without it, nohup writes them to nohup.out.
  • & runs the command in the background and returns the prompt.

To stop the process later, you need its process ID (PID). Have q record it at startup by adding a line to hdb.q that writes .z.i to a file:

`:hdb.pid 0: enlist string .z.i;

Then stop the process with kill $(cat hdb.pid), which sends SIGTERM.

tmux and screen take a different approach: they keep a terminal session running on the server that you can detach from and reattach to later. q keeps its interactive console, which is useful during development:

tmux new-session -d -s hdb 'q hdb.q -db /data/hdb -p 5010'
tmux attach -t hdb      # see the q console; press Ctrl+B, then D, to detach again

The q prompt in that session is live. Entering \\ or pressing Ctrl+C in the wrong window stops or interrupts the HDB.

These methods are good enough for development and demonstrations. They are not enough for production, because nothing watches the process:

  • If q crashes or something stops it, it stays down until someone notices and starts it again.
  • After a reboot, nothing starts it.
  • The process depends on how one person started it: their environment variables, their working directory, and their shell. Some systems also stop all of a user's processes when that user logs out, whether or not they used nohup.
  • hdb.log grows without limit.

Run as a daemon

A daemon is a process that runs in the background, detached from any terminal. A program becomes a daemon by forking and starting a new session, so that no terminal controls it. It also closes or redirects standard input, output and error, and usually writes a PID file. The nohup command above does most of this. The setsid command from util-linux does the rest, by starting q in a new session:

setsid q hdb.q -db /data/hdb -p 5010 -q < /dev/null > hdb.log 2>&1 &

Use the .z.i PID file from the previous section rather than $!, because setsid can fork, and then $! is the PID of the wrong process.

A daemon survives logout, but it has the same gaps as nohup: nothing restarts it after a crash or starts it at boot. Service managers such as systemd close those gaps and also take care of detaching the process, so you rarely need to daemonize q yourself.

Run as a systemd service

systemd is the service manager on most Linux distributions. It starts services at boot, restarts them when they fail, captures their output, and stops them in order at shutdown.

Steps

  1. Create a user to run the service, so that the process does not run as root, and a directory for KDB-X that the user owns:

    sudo useradd --system --create-home kdb
sudo install -d -o kdb -g kdb /opt/kx

  2. As that user, install KDB-X. When the installer asks for the install location, choose a custom location and enter /opt/kx. The binary is then /opt/kx/bin/q. Put your scripts in a directory such as /home/kdb/app.

    Install outside home directories on SELinux systems

    By default, KDB-X installs into $HOME/.kx. On distributions with SELinux in enforcing mode, such as Red Hat Enterprise Linux and Rocky Linux, systemd cannot run a program from a home directory, and the service fails with status=203/EXEC and Permission denied. Check the mode with getenforce. Installing under /opt avoids the problem. To keep an existing installation in a home directory instead, give the binary the bin_t file type with semanage fcontext and restorecon.

  3. Create a unit file, /etc/systemd/system/kdbx-hdb.service:

    [Unit]
Description=KDB-X HDB on port 5010
After=network-online.target
Wants=network-online.target
RequiresMountsFor=/data/hdb
StartLimitIntervalSec=300
StartLimitBurst=5

[Service]
User=kdb
Group=kdb
WorkingDirectory=/home/kdb/app
ExecStart=/opt/kx/bin/q hdb.q -db /data/hdb -p 5010 -q
Restart=on-failure
RestartSec=5
TimeoutStopSec=60
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

  4. Load the unit, enable it to start at boot, and start it now:

    sudo systemctl daemon-reload
sudo systemctl enable --now kdbx-hdb

What the settings do

  • ExecStart runs q in the foreground. Do not add nohup or &: systemd tracks the process it started, and a process that detaches itself looks to systemd as if it has exited. Use an absolute path to q. q sets QHOME to the directory above the binary, /opt/kx, so it finds the license that the installer put there. If the license is elsewhere, add Environment=QLIC=/path/to/license/folder.
  • systemd connects standard input to /dev/null, so the -p option is required.
  • RequiresMountsFor waits for the file system that holds the database, and After=network-online.target waits for the network.
  • Restart=on-failure restarts q when it exits with a non-zero code or is killed by a signal, for example after a crash or kill -9. It does not restart q after a clean exit with code 0, such as systemctl stop, SIGTERM, or exit 0 from a client. For a process that must restart even then, use Restart=always. systemctl stop still stops it.
  • StartLimitIntervalSec and StartLimitBurst stop the restarts if q fails five times in five minutes. A process with a bad configuration then stays down and reports a failure, instead of restarting forever.
  • TimeoutStopSec is the grace period between SIGTERM and SIGKILL. Make it longer than your .z.exit handler takes.
  • LimitNOFILE raises the open-file limit. A partitioned database can need many file descriptors; see resource limits.

Operate the service

systemctl status kdbx-hdb          # state, PID and the latest log lines
sudo systemctl restart kdbx-hdb    # stop, then start
journalctl -u kdbx-hdb -f          # follow stdout and stderr

systemd sends the process's stdout and stderr to the journal, which rotates them. You do not need to redirect them to a file.

Run several instances from one unit

An HDB often runs as several processes on different ports. A template unit has @ in its name, and systemd replaces %i in it with the text after the @. Save the kdbx-hdb.service unit as /etc/systemd/system/kdbx-hdb@.service, with this ExecStart line:

ExecStart=/opt/kx/bin/q hdb.q -db /data/hdb -p %i -q

Then start one instance per port:

sudo systemctl daemon-reload
sudo systemctl enable --now kdbx-hdb@5010 kdbx-hdb@5011

Run in Docker or Docker Compose

A container packages q, its scripts and their dependencies into one image that runs the same way on every host. Docker's restart policies give you the same restart-after-crash behavior as systemd. Read how to run KDB-X in containers for the CPU, memory and storage settings that affect performance.

The examples below use the official kdbx-q image from the KX downloads portal. The image runs q by default, as a user named kx with user ID 10001, in the working directory /opt/app. It sets QLIC to /run/secrets/kx, so q looks for the license file in that directory. To try the image interactively, mount your license file there:

docker run --rm -it \
  -v "$HOME/.kx/kc.lic:/run/secrets/kx/kc.lic:ro" \
  portal.dl.kx.com/kdbx-q:5.0.20261015-rocky9-r1

Steps

  1. Keep the license out of the image. Mount the license file from the host into /run/secrets/kx, keeping its name: kc.lic for a Community Edition license, or k4.lic.
  2. Mount the directory that holds hdb.q and healthcheck.q at /opt/app, or build your own image from kdbx-q that copies them there.
  3. Keep the data out of the image too. Mount the database directory, or a directory that a writer writes to, as a volume, so that it outlives the container. The user ID 10001 must be able to read the scripts, the license and the data, and write to any directory the process writes to.

  4. Create compose.yaml:

    services:
  hdb:
    image: portal.dl.kx.com/kdbx-q:5.0.20261015-rocky9-r1
    command: ["q", "hdb.q", "-db", "/data/hdb", "-p", "5010", "-q"]
    restart: unless-stopped
    stop_grace_period: 60s
    volumes:
      - ./app:/opt/app:ro
      - $HOME/.kx/kc.lic:/run/secrets/kx/kc.lic:ro
      - /data/hdb:/data/hdb:ro
    ports:
      - "5010:5010"
    ulimits:
      nofile: 65536
    healthcheck:
      test: ["CMD", "q", "healthcheck.q", "-q"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 60s

  5. Start it in the background:

    docker compose up -d
docker compose logs -f hdb

The equivalent single-container command is:

docker run -d --name hdb --restart unless-stopped --stop-timeout 60 \
  -v "$PWD/app":/opt/app:ro -v "$HOME/.kx/kc.lic":/run/secrets/kx/kc.lic:ro \
  -v /data/hdb:/data/hdb:ro -p 5010:5010 \
  portal.dl.kx.com/kdbx-q:5.0.20261015-rocky9-r1 \
  q hdb.q -db /data/hdb -p 5010 -q

What the settings do

  • restart: unless-stopped restarts the container whenever q exits, and starts it again when the Docker daemon starts, for example after a reboot. It does not start a container that you stopped yourself. The other policies are no, the default; on-failure, which restarts only after a non-zero exit; and always.
  • Do not run the container with -it. A server container has no terminal, so the -p option is required.
  • q runs as process 1 in the container, so docker stop sends SIGTERM to q itself, and q exits cleanly. stop_grace_period, or --stop-timeout, sets the time before Docker sends SIGKILL. The default is 10 seconds, which can be too short for a writer that saves state on exit.
  • Docker captures stdout and stderr. Configure a logging driver with rotation, or the logs grow without limit.

Health checks

A process can be running and listening on its port but still unable to answer, for example while it is stuck in a long query. A health check runs a command inside the container at an interval. Here, healthcheck.q opens a connection to the HDB, runs a trivial query, and sets the exit code that Docker reads:

/ healthcheck.q: exit 0 if the q process on port 5010 answers a query, 1 otherwise
h:@[hopen;(`::5010;2000);0Ni]
exit $[null h;1;@[{2=x"1+1"};h;0b];0;1]

Docker marks the container unhealthy after retries failed checks in a row, and healthy again when a check passes. An HDB that is busy with one long query fails this check too, because it cannot answer until the query ends. Plain Docker and Docker Compose do not restart an unhealthy container. They only report the status. To act on it, use an orchestrator such as Kubernetes, or your own monitoring.

Several processes

Docker Compose can start a whole system from one file, for example a tickerplant, an RDB (real-time database) and several HDBs, each as a service with its own restart policy. Use depends_on to start them in order, and the service names as host names in hopen calls between them. Compose runs everything on one host.

Run in Kubernetes

Kubernetes runs containers across a cluster of machines. You describe the state you want, for example "two HDB processes, each with 8 GB of memory, reading this volume", and Kubernetes keeps the cluster in that state. If a process exits, its container restarts. If a machine fails, its workload moves to another machine.

A unit of work in Kubernetes is a pod, one or more containers that run together. You rarely create pods directly. Instead:

  • A Deployment runs a number of identical, interchangeable pods. It suits HDBs that read the same database, provided several pods can mount the storage at once.
  • A StatefulSet gives each pod a stable name and its own persistent volume, which it keeps across restarts. It suits a writer, which must come back with the data it wrote.

This Deployment runs two HDB pods:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: hdb
spec:
  replicas: 2
  selector:
    matchLabels:
      app: hdb
  template:
    metadata:
      labels:
        app: hdb
    spec:
      terminationGracePeriodSeconds: 60
      containers:
        - name: hdb
          image: portal.dl.kx.com/kdbx-q:5.0.20261015-rocky9-r1
          command: ["q"]
          args: ["hdb.q", "-db", "/data/hdb", "-p", "5010", "-q"]
          ports:
            - containerPort: 5010
          resources:
            requests:
              cpu: "4"
              memory: 8Gi
            limits:
              cpu: "4"
              memory: 8Gi
          startupProbe:
            tcpSocket:
              port: 5010
            periodSeconds: 5
            failureThreshold: 60
          readinessProbe:
            exec:
              command: ["q", "healthcheck.q", "-q"]
            periodSeconds: 10
            timeoutSeconds: 5
          livenessProbe:
            tcpSocket:
              port: 5010
            periodSeconds: 20
            failureThreshold: 3
          volumeMounts:
            - name: scripts
              mountPath: /opt/app
              readOnly: true
            - name: license
              mountPath: /run/secrets/kx
              readOnly: true
            - name: hdb-data
              mountPath: /data/hdb
              readOnly: true
      volumes:
        - name: scripts
          configMap:
            name: hdb-scripts
        - name: license
          secret:
            secretName: kdbx-license
        - name: hdb-data
          persistentVolumeClaim:
            claimName: hdb-data
            readOnly: true

The pods mount hdb.q and healthcheck.q from a ConfigMap at /opt/app, the working directory of the kdbx-q image, and the license from a Secret at /run/secrets/kx, where the image's QLIC points. The database comes from a PersistentVolumeClaim named hdb-data, which your cluster administrator provides. Create the ConfigMap and the Secret, using k4.lic or kc.lic as appropriate, then apply the manifest:

kubectl create configmap hdb-scripts --from-file=hdb.q --from-file=healthcheck.q
kubectl create secret generic kdbx-license --from-file=kc.lic
kubectl apply -f hdb.yaml
kubectl get pods -l app=hdb

The probes are how Kubernetes decides what to do with a pod:

  • The startup probe gives the HDB time to load before the other probes start. Here it allows up to 5 minutes (60 checks, 5 seconds apart).
  • The readiness probe decides whether the pod receives traffic. It runs the same healthcheck.q as the Docker example. A pod that fails it is removed from the Service that routes connections, but keeps running.
  • The liveness probe decides whether the container is restarted. Keep it simpler and more tolerant than the readiness probe: an HDB that is busy with one long query is still alive, and restarting it would end that query for nothing.

When a pod is stopped, Kubernetes sends SIGTERM and waits terminationGracePeriodSeconds, 30 by default, before it sends SIGKILL. If a container fails repeatedly, Kubernetes waits longer between each restart and reports the pod as CrashLoopBackOff.

Set memory requests and limits to the same value so that the pod's memory is reserved. Read containers for how CPU and memory limits interact with q.

Kubernetes has a lot to learn and run. It is worth the effort when you operate many processes across several machines, or when your organization already runs a cluster. For KDB-X, the DB Service provides a complete database that is deployed with Docker Compose on one node or with Kubernetes on a cluster, so you do not have to write these manifests yourself.

Write your own supervisor

Some teams start and watch q processes with their own tools: a shell script, a scheduler, or a q process that supervises other q processes. Because a q process can detect when a connection closes, through .z.pc, a short q script can restart a process when its connection drops:

/ watchdog.q: start a q process and restart it when its connection drops
cfg:.Q.def[([port:5010i])] .Q.opt .z.x
cmd:"q hdb.q -p ",string[cfg`port]," -q < /dev/null >> hdb.log 2>&1 &"
h:0Ni
.z.pc:{[w] if[w=h; h::0Ni; -2 string[.z.p]," lost port ",string[cfg`port],", restarting"; system cmd]}
.z.ts:{if[null h; h::@[hopen;(`$"::",string cfg`port;500);0Ni]]}
system cmd
\t 1000

The script starts the child process, tries every second to connect to it, and starts it again when the connection closes:

$ q watchdog.q -port 5010 -p 5020 -q
2026.11.06D16:19:47.097530000 lost port 5010, restarting

The watchdog listens on port 5020 only so that it keeps running without a terminal. It shows the idea, but it lacks what systemd and Kubernetes provide:

  • It cannot tell a crash from a deliberate stop. If you send kill to the child to stop it, the watchdog starts it again.
  • It has no limit on restarts, so a child that fails at startup is restarted every second.
  • The watchdog itself needs supervising. Run it as a systemd service.
  • It works on one host only.

A custom supervisor is worth building when you need what a generic one cannot know: the order in which a tickerplant, its subscribers and the HDBs must start, a restart that must first replay a log, or a central view of every process in a system. Run each supervisor under systemd, Docker or Kubernetes so that something restarts it too.

What restarting does not solve

A restart brings a process back. It does not bring back what the process knew, and it does not make the process available while it is down.

  • State. An RDB that restarts is empty until it replays the tickerplant log. A writer that crashes loses whatever it had not written. Plan the recovery as part of the startup script.
  • Clients. Every connection to the process breaks when it stops. Clients must reconnect, for example with a retry around hopen.
  • Downtime. A process is unavailable while it restarts and loads. To keep serving queries during that time, run more than one HDB with something in front to spread the queries. For machine and site failures, see disaster-recovery planning.
  • Crash loops. A process that fails because of bad configuration or a full disk fails again after the restart. Limit the restarts, as the systemd and Kubernetes examples do, and alert someone.
  • Being noticed. Monitor the process and alert on failures and restarts. A process that restarts every hour can look healthy whenever you check it. Use the Logging module to produce logs that your monitoring can read.

Next steps