# How to keep a q process running _This page explains how to keep a q process running without a person watching it: what each approach gives you, how to set it up, and which q behaviors matter when a process stops, restarts or fails._ Many q processes are short-lived. A script parses a file, writes the result, and [exits](use-scripts.md#exit-from-a-script). Others are meant to run for days or months: - A _writer_ accepts data from upstream processes and is responsible for persisting it. If it stops, incoming data has nowhere to go, and anything it held in memory but had not written is at risk. A [tickerplant](../manage_streaming_data/architecture.md#tickerplant-tp), which logs every update to disk before passing it on, is the classic example. - An [HDB](../interact_with_databases/hdb.md) (historical database) serves queries on a database to other users. It has to answer at all times, including after the machine reboots at 3 a.m. Starting such a process at a terminal works until you close the terminal, the process crashes, or the machine restarts. The following table compares approaches that handle those events to different degrees. Choose the simplest one that covers the events you care about. | Approach | Survives logout | Restarts after a crash | Starts at boot | Spans several hosts | | --- | --- | --- | --- | --- | | [`nohup`, `tmux` or `screen`](#run-in-the-background) | Yes | No | No | No | | [Daemon](#run-as-a-daemon) | Yes | No | No | No | | [systemd service](#run-as-a-systemd-service) | Yes | Yes | Yes | No | | [Docker or Docker Compose](#run-in-docker-or-docker-compose) | Yes | Yes | Yes | No | | [Kubernetes](#run-in-kubernetes) | Yes | Yes | Yes | Yes | | [Your own supervisor](#write-your-own-supervisor) | Depends | Yes | Depends | Depends | The examples on this page run Linux commands and use an HDB started like this: ```bash q hdb.q -db /data/hdb -p 5010 -q ``` where `hdb.q` loads the database named by `-db`: ```q / hdb.q: serve the database given by -db o:.Q.opt .z.x .Q.lo[hsym `$first o`db;0b;0b] ``` [`.Q.lo`](../../ref/dotq.md#lo-load-without) loads the database like [`\l`](../interact_with_databases/loading-db.md#loading-with-l). With its second argument `0b`, it does not change the working directory to the database root, so a file that the process writes with a relative path, such as a log or a PID file, stays out of the database. With its third argument `0b`, it does not run any scripts that it finds in the database directory. A real HDB script also defines the functions that clients call, as on the [HDB page](../interact_with_databases/hdb.md). ## Prerequisites - KDB-X is [installed](../../get_started/kdb-x-install.md), with a valid license. - A Linux host. The systemd, Docker and Kubernetes sections need those tools installed and the permissions to use them. ## How q behaves without a terminal Every approach on this page runs q without an interactive terminal. Four q behaviors decide whether that works. **q needs a listening port to stay alive.** When standard input is closed or redirected from `/dev/null`, q runs its startup script and then exits with code `0`, unless it listens on a port with [`-p`](../../ref/cmdline.md#-p-listening-port). A [timer](../../ref/syscmds.md#t-timer) alone does not keep it running. A server process has a port anyway. A process that only does scheduled work needs one too. **An error during startup ends the process with code `1`.** At a terminal, q stops at the [first error in a script](use-scripts.md#when-a-script-has-a-bad-line) and waits at a debug prompt. Without a terminal, there is no prompt to wait at, so q prints the error to stderr and exits with code `1`. Supervisors read this as a failure. **`SIGTERM` is a clean exit.** When q receives `SIGTERM`, it finishes the task in progress and then exits as if it had evaluated `exit 0`. Before it exits, it calls [`.z.exit`](../../ref/dotz.md#zexit-action-on-exit), which is the place to flush or save state. A writer might define: ```q .z.exit:{[code] `:/data/writer/trade set trade; -1"saved trade before exit ",string code;} ``` `SIGTERM` is the signal that `kill`, `systemctl stop`, `docker stop` and Kubernetes send to stop a process. Each of them sends `SIGKILL` if the process has not exited within a grace period, so make that period longer than your `.z.exit` handler takes. **`SIGKILL` and crashes skip `.z.exit`.** A process killed with `kill -9`, by the kernel out-of-memory killer, or by a crash, ends without running `.z.exit`, and its exit status reports the signal (`137` for `SIGKILL` in a shell). State held only in memory is lost. Design a writer so that it can rebuild its state after a restart: a tickerplant's subscribers, for example, [replay its log](../manage_streaming_data/data-recovery.md). A client can also end the process by sending `exit 0`, or `exit` with any other code, over [IPC](../io_and_communication/basics.md). To prevent that, start q with [`-u 1`](../../ref/cmdline.md#-u-usr-pwd-local), which also restricts file access, or control what clients may run with [message handlers](../io_and_communication/security.md#authorization). ## Run in the background The quickest way to keep a process running after you log out is to detach it from your terminal with `nohup`: ```bash nohup q hdb.q -db /data/hdb -p 5010 -q < /dev/null > hdb.log 2>&1 & ``` - `nohup` makes the process ignore the hang-up signal that the terminal sends when it closes. - `< /dev/null` closes standard input, so q does not try to read from a terminal that has gone. - `> hdb.log 2>&1` sends stdout and stderr to a file. Without it, `nohup` writes them to `nohup.out`. - `&` runs the command in the background and returns the prompt. To stop the process later, you need its process ID (PID). Have q record it at startup by adding a line to `hdb.q` that writes [`.z.i`](../../ref/dotz.md#zi-pid) to a file: ```q `:hdb.pid 0: enlist string .z.i; ``` Then stop the process with `kill $(cat hdb.pid)`, which sends `SIGTERM`. `tmux` and `screen` take a different approach: they keep a terminal session running on the server that you can detach from and reattach to later. q keeps its interactive console, which is useful during development: ```bash tmux new-session -d -s hdb 'q hdb.q -db /data/hdb -p 5010' tmux attach -t hdb # see the q console; press Ctrl+B, then D, to detach again ``` The q prompt in that session is live. Entering `\\` or pressing `Ctrl+C` in the wrong window stops or interrupts the HDB. These methods are good enough for development and demonstrations. They are not enough for production, because nothing watches the process: - If q crashes or something stops it, it stays down until someone notices and starts it again. - After a reboot, nothing starts it. - The process depends on how one person started it: their environment variables, their working directory, and their shell. Some systems also stop all of a user's processes when that user logs out, whether or not they used `nohup`. - `hdb.log` grows without limit. ## Run as a daemon A _daemon_ is a process that runs in the background, detached from any terminal. A program becomes a daemon by forking and starting a new session, so that no terminal controls it. It also closes or redirects standard input, output and error, and usually writes a PID file. The `nohup` command above does most of this. The `setsid` command from util-linux does the rest, by starting q in a new session: ```bash setsid q hdb.q -db /data/hdb -p 5010 -q < /dev/null > hdb.log 2>&1 & ``` Use the `.z.i` PID file from the [previous section](#run-in-the-background) rather than `$!`, because `setsid` can fork, and then `$!` is the PID of the wrong process. A daemon survives logout, but it has the same gaps as `nohup`: nothing restarts it after a crash or starts it at boot. Service managers such as systemd close those gaps and also take care of detaching the process, so you rarely need to daemonize q yourself. ## Run as a systemd service systemd is the service manager on most Linux distributions. It starts services at boot, restarts them when they fail, captures their output, and stops them in order at shutdown. ### Steps 1. Create a user to run the service, so that the process does not run as root, and a directory for KDB-X that the user owns: ```bash sudo useradd --system --create-home kdb sudo install -d -o kdb -g kdb /opt/kx ``` 1. As that user, [install KDB-X](../../get_started/kdb-x-install.md). When the installer asks for the install location, choose a custom location and enter `/opt/kx`. The binary is then `/opt/kx/bin/q`. Put your scripts in a directory such as `/home/kdb/app`. > **Warning — Install outside home directories on SELinux systems** > > > By default, KDB-X installs into `$HOME/.kx`. On distributions with SELinux in enforcing mode, such as Red Hat Enterprise Linux and Rocky Linux, systemd cannot run a program from a home directory, and the service fails with `status=203/EXEC` and `Permission denied`. Check the mode with `getenforce`. Installing under `/opt` avoids the problem. To keep an existing installation in a home directory instead, give the binary the `bin_t` file type with `semanage fcontext` and `restorecon`. > 1. Create a unit file, `/etc/systemd/system/kdbx-hdb.service`: ```ini [Unit] Description=KDB-X HDB on port 5010 After=network-online.target Wants=network-online.target RequiresMountsFor=/data/hdb StartLimitIntervalSec=300 StartLimitBurst=5 [Service] User=kdb Group=kdb WorkingDirectory=/home/kdb/app ExecStart=/opt/kx/bin/q hdb.q -db /data/hdb -p 5010 -q Restart=on-failure RestartSec=5 TimeoutStopSec=60 LimitNOFILE=65536 [Install] WantedBy=multi-user.target ``` 1. Load the unit, enable it to start at boot, and start it now: ```bash sudo systemctl daemon-reload sudo systemctl enable --now kdbx-hdb ``` ### What the settings do - `ExecStart` runs q in the foreground. Do not add `nohup` or `&`: systemd tracks the process it started, and a process that detaches itself looks to systemd as if it has exited. Use an absolute path to q. q sets `QHOME` to the directory above the binary, `/opt/kx`, so it finds the license that the installer put there. If the license is elsewhere, add `Environment=QLIC=/path/to/license/folder`. - systemd connects standard input to `/dev/null`, so the `-p` option [is required](#how-q-behaves-without-a-terminal). - `RequiresMountsFor` waits for the file system that holds the database, and `After=network-online.target` waits for the network. - `Restart=on-failure` restarts q when it exits with a non-zero code or is killed by a signal, for example after a crash or `kill -9`. It does not restart q after a clean exit with code `0`, such as `systemctl stop`, `SIGTERM`, or `exit 0` from a client. For a process that must restart even then, use `Restart=always`. `systemctl stop` still stops it. - `StartLimitIntervalSec` and `StartLimitBurst` stop the restarts if q fails five times in five minutes. A process with a bad configuration then stays down and reports a failure, instead of restarting forever. - `TimeoutStopSec` is the grace period between `SIGTERM` and `SIGKILL`. Make it longer than your `.z.exit` handler takes. - `LimitNOFILE` raises the open-file limit. A partitioned database can need many file descriptors; see [resource limits](../performance/os.md#resource-limits-and-swap). ### Operate the service ```bash systemctl status kdbx-hdb # state, PID and the latest log lines sudo systemctl restart kdbx-hdb # stop, then start journalctl -u kdbx-hdb -f # follow stdout and stderr ``` systemd sends the process's stdout and stderr to the journal, which rotates them. You do not need to redirect them to a file. ### Run several instances from one unit An HDB often runs as [several processes](../interact_with_databases/concurrency.md) on different ports. A _template unit_ has `@` in its name, and systemd replaces `%i` in it with the text after the `@`. Save the `kdbx-hdb.service` unit as `/etc/systemd/system/kdbx-hdb@.service`, with this `ExecStart` line: ```ini ExecStart=/opt/kx/bin/q hdb.q -db /data/hdb -p %i -q ``` Then start one instance per port: ```bash sudo systemctl daemon-reload sudo systemctl enable --now kdbx-hdb@5010 kdbx-hdb@5011 ``` ## Run in Docker or Docker Compose A container packages q, its scripts and their dependencies into one image that runs the same way on every host. Docker's restart policies give you the same restart-after-crash behavior as systemd. Read [how to run KDB-X in containers](../performance/containers.md) for the CPU, memory and storage settings that affect performance. The examples below use the official `kdbx-q` image from the [KX downloads portal](https://portal.dl.kx.com/). The image runs `q` by default, as a user named `kx` with user ID 10001, in the working directory `/opt/app`. It sets [`QLIC`](../../ref/files.md) to `/run/secrets/kx`, so q looks for the license file in that directory. To try the image interactively, mount your license file there: ```bash docker run --rm -it \ -v "$HOME/.kx/kc.lic:/run/secrets/kx/kc.lic:ro" \ portal.dl.kx.com/kdbx-q:5.0.20261015-rocky9-r1 ``` ### Steps 1. Keep the license out of the image. Mount the license file from the host into `/run/secrets/kx`, keeping its name: `kc.lic` for a Community Edition license, or `k4.lic`. 1. Mount the directory that holds `hdb.q` and `healthcheck.q` at `/opt/app`, or build your own image from `kdbx-q` that copies them there. 1. Keep the data out of the image too. Mount the database directory, or a directory that a writer writes to, as a volume, so that it outlives the container. The user ID 10001 must be able to read the scripts, the license and the data, and write to any directory the process writes to. 1. Create `compose.yaml`: ```yaml services: hdb: image: portal.dl.kx.com/kdbx-q:5.0.20261015-rocky9-r1 command: ["q", "hdb.q", "-db", "/data/hdb", "-p", "5010", "-q"] restart: unless-stopped stop_grace_period: 60s volumes: - ./app:/opt/app:ro - $HOME/.kx/kc.lic:/run/secrets/kx/kc.lic:ro - /data/hdb:/data/hdb:ro ports: - "5010:5010" ulimits: nofile: 65536 healthcheck: test: ["CMD", "q", "healthcheck.q", "-q"] interval: 30s timeout: 10s retries: 3 start_period: 60s ``` 1. Start it in the background: ```bash docker compose up -d docker compose logs -f hdb ``` The equivalent single-container command is: ```bash docker run -d --name hdb --restart unless-stopped --stop-timeout 60 \ -v "$PWD/app":/opt/app:ro -v "$HOME/.kx/kc.lic":/run/secrets/kx/kc.lic:ro \ -v /data/hdb:/data/hdb:ro -p 5010:5010 \ portal.dl.kx.com/kdbx-q:5.0.20261015-rocky9-r1 \ q hdb.q -db /data/hdb -p 5010 -q ``` ### What the settings do - `restart: unless-stopped` restarts the container whenever q exits, and starts it again when the Docker daemon starts, for example after a reboot. It does not start a container that you stopped yourself. The other policies are `no`, the default; `on-failure`, which restarts only after a non-zero exit; and `always`. - Do not run the container with `-it`. A server container has no terminal, so the `-p` option [is required](#how-q-behaves-without-a-terminal). - q runs as process 1 in the container, so `docker stop` sends `SIGTERM` to q itself, and q [exits cleanly](#how-q-behaves-without-a-terminal). `stop_grace_period`, or `--stop-timeout`, sets the time before Docker sends `SIGKILL`. The default is 10 seconds, which can be too short for a writer that saves state on exit. - Docker captures stdout and stderr. Configure a [logging driver](https://docs.docker.com/engine/logging/configure/) with rotation, or the logs grow without limit. ### Health checks A process can be running and listening on its port but still unable to answer, for example while it is stuck in a long query. A health check runs a command inside the container at an interval. Here, `healthcheck.q` opens a connection to the HDB, runs a trivial query, and sets the exit code that Docker reads: ```q / healthcheck.q: exit 0 if the q process on port 5010 answers a query, 1 otherwise h:@[hopen;(`::5010;2000);0Ni] exit $[null h;1;@[{2=x"1+1"};h;0b];0;1] ``` Docker marks the container `unhealthy` after `retries` failed checks in a row, and `healthy` again when a check passes. An HDB that is busy with one long query fails this check too, because it cannot answer until the query ends. Plain Docker and Docker Compose do not restart an unhealthy container. They only report the status. To act on it, use an orchestrator such as Kubernetes, or your own monitoring. ### Several processes Docker Compose can start a whole system from one file, for example a tickerplant, an RDB (real-time database) and several HDBs, each as a service with its own restart policy. Use `depends_on` to start them in order, and the service names as host names in `hopen` calls between them. Compose runs everything on one host. ## Run in Kubernetes Kubernetes runs containers across a cluster of machines. You describe the state you want, for example "two HDB processes, each with 8 GB of memory, reading this volume", and Kubernetes keeps the cluster in that state. If a process exits, its container restarts. If a machine fails, its workload moves to another machine. A unit of work in Kubernetes is a _pod_, one or more containers that run together. You rarely create pods directly. Instead: - A **Deployment** runs a number of identical, interchangeable pods. It suits HDBs that read the same database, provided several pods can mount the storage at once. - A **StatefulSet** gives each pod a stable name and its own persistent volume, which it keeps across restarts. It suits a writer, which must come back with the data it wrote. This Deployment runs two HDB pods: ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: hdb spec: replicas: 2 selector: matchLabels: app: hdb template: metadata: labels: app: hdb spec: terminationGracePeriodSeconds: 60 containers: - name: hdb image: portal.dl.kx.com/kdbx-q:5.0.20261015-rocky9-r1 command: ["q"] args: ["hdb.q", "-db", "/data/hdb", "-p", "5010", "-q"] ports: - containerPort: 5010 resources: requests: cpu: "4" memory: 8Gi limits: cpu: "4" memory: 8Gi startupProbe: tcpSocket: port: 5010 periodSeconds: 5 failureThreshold: 60 readinessProbe: exec: command: ["q", "healthcheck.q", "-q"] periodSeconds: 10 timeoutSeconds: 5 livenessProbe: tcpSocket: port: 5010 periodSeconds: 20 failureThreshold: 3 volumeMounts: - name: scripts mountPath: /opt/app readOnly: true - name: license mountPath: /run/secrets/kx readOnly: true - name: hdb-data mountPath: /data/hdb readOnly: true volumes: - name: scripts configMap: name: hdb-scripts - name: license secret: secretName: kdbx-license - name: hdb-data persistentVolumeClaim: claimName: hdb-data readOnly: true ``` The pods mount `hdb.q` and `healthcheck.q` from a ConfigMap at `/opt/app`, the working directory of the `kdbx-q` image, and the license from a Secret at `/run/secrets/kx`, where the image's `QLIC` points. The database comes from a PersistentVolumeClaim named `hdb-data`, which your cluster administrator provides. Create the ConfigMap and the Secret, using `k4.lic` or `kc.lic` as appropriate, then apply the manifest: ```bash kubectl create configmap hdb-scripts --from-file=hdb.q --from-file=healthcheck.q kubectl create secret generic kdbx-license --from-file=kc.lic kubectl apply -f hdb.yaml kubectl get pods -l app=hdb ``` The probes are how Kubernetes decides what to do with a pod: - The **startup probe** gives the HDB time to load before the other probes start. Here it allows up to 5 minutes (60 checks, 5 seconds apart). - The **readiness probe** decides whether the pod receives traffic. It runs the same `healthcheck.q` as the [Docker example](#health-checks). A pod that fails it is removed from the [Service](https://kubernetes.io/docs/concepts/services-networking/service/) that routes connections, but keeps running. - The **liveness probe** decides whether the container is restarted. Keep it simpler and more tolerant than the readiness probe: an HDB that is busy with one long query is still alive, and restarting it would end that query for nothing. When a pod is stopped, Kubernetes sends `SIGTERM` and waits `terminationGracePeriodSeconds`, 30 by default, before it sends `SIGKILL`. If a container fails repeatedly, Kubernetes waits longer between each restart and reports the pod as `CrashLoopBackOff`. Set memory requests and limits to the same value so that the pod's memory is reserved. Read [containers](../performance/containers.md) for how CPU and memory limits interact with q. Kubernetes has a lot to learn and run. It is worth the effort when you operate many processes across several machines, or when your organization already runs a cluster. For KDB-X, the [DB Service](../../services/db-service/introduction.md) provides a complete database that is deployed with Docker Compose on one node or with Kubernetes on a cluster, so you do not have to write these manifests yourself. ## Write your own supervisor Some teams start and watch q processes with their own tools: a shell script, a scheduler, or a q process that supervises other q processes. Because a q process can detect when a connection closes, through [`.z.pc`](../../ref/dotz.md#zpc-close), a short q script can restart a process when its connection drops: ```q / watchdog.q: start a q process and restart it when its connection drops cfg:.Q.def[([port:5010i])] .Q.opt .z.x cmd:"q hdb.q -p ",string[cfg`port]," -q < /dev/null >> hdb.log 2>&1 &" h:0Ni .z.pc:{[w] if[w=h; h::0Ni; -2 string[.z.p]," lost port ",string[cfg`port],", restarting"; system cmd]} .z.ts:{if[null h; h::@[hopen;(`$"::",string cfg`port;500);0Ni]]} system cmd \t 1000 ``` The script starts the child process, tries every second to connect to it, and starts it again when the connection closes: ```bash $ q watchdog.q -port 5010 -p 5020 -q 2026.11.06D16:19:47.097530000 lost port 5010, restarting ``` The watchdog listens on port 5020 only so that it keeps running without a terminal. It shows the idea, but it lacks what systemd and Kubernetes provide: - It cannot tell a crash from a deliberate stop. If you send `kill` to the child to stop it, the watchdog starts it again. - It has no limit on restarts, so a child that fails at startup is restarted every second. - The watchdog itself needs supervising. Run it as a systemd service. - It works on one host only. A custom supervisor is worth building when you need what a generic one cannot know: the order in which a tickerplant, its subscribers and the HDBs must start, a restart that must first replay a log, or a central view of every process in a system. Run each supervisor under systemd, Docker or Kubernetes so that something restarts it too. ## What restarting does not solve A restart brings a process back. It does not bring back what the process knew, and it does not make the process available while it is down. - **State.** An RDB that restarts is empty until it replays the tickerplant log. A writer that crashes loses whatever it had not written. Plan the [recovery](../manage_streaming_data/data-recovery.md) as part of the startup script. - **Clients.** Every connection to the process breaks when it stops. Clients must reconnect, for example with a retry around [`hopen`](../../ref/hopen.md). - **Downtime.** A process is unavailable while it restarts and loads. To keep serving queries during that time, run more than one HDB with something in front to [spread the queries](../interact_with_databases/concurrency.md). For machine and site failures, see [disaster-recovery planning](../manage_streaming_data/disaster-recovery/index.md). - **Crash loops.** A process that fails because of bad configuration or a full disk fails again after the restart. Limit the restarts, as the systemd and Kubernetes examples do, and alert someone. - **Being noticed.** Monitor the process and alert on failures and restarts. A process that restarts every hour can look healthy whenever you check it. Use the [Logging module](../../modules/logging/overview.md) to produce logs that your monitoring can read. ## Next steps - Write the startup script: [develop scripts](use-scripts.md), including [command-line parameters](use-scripts.md#handle-command-line-parameters) and [exit codes](use-scripts.md#exit-from-a-script). - Serve a database: [historical database (HDB)](../interact_with_databases/hdb.md) and [handling concurrent queries](../interact_with_databases/concurrency.md). - Tune the container or host: [containers and virtual machines](../performance/containers.md) and [operating system](../performance/os.md). - Protect the process from its clients: [security](../io_and_communication/security.md). - Read the references: [`.z.exit`](../../ref/dotz.md#zexit-action-on-exit), [`.z.pc`](../../ref/dotz.md#zpc-close), [`.z.i`](../../ref/dotz.md#zi-pid), [`exit`](../../ref/exit.md), and [command-line options](../../ref/cmdline.md).