Running the Stream Processor
Run and manage Stream Processor images individually, or as part of a group
For clarity in the examples below, environment variables are used to configure the pipelines; a
config.yaml file could be used as well.
Working from the following project directory:
$ ls spec.q
With the following
.qsp.run .qsp.read.fromCallback[`upd] .qsp.window.timer[00:00:05] .qsp.write.toConsole
Running in Kubernetes
To deploy and run in Kubernetes using the provided Coordinator service, follow the Kubernetes configuration and deployment instructions for launching the Coordinator within the cluster. The instructions also detail how to deploy and teardown a pipeline once the Coordinator service has started.
Running in Docker Compose
The above examples can be run in Docker Compose with an appropriate Docker Compose file (
Configuration to add Service Discovery or Monitoring
version: "3.3" services: controller: image: registry.dl.kx.com/kxi-sp-controller:1.2.0 ports: - 6000:6000 environment: - KDB_LICENSE_B64 command: ["-p", "6000"] deploy: restart_policy: condition: on-failure worker: image: registry.dl.kx.com/kxi-sp-worker:1.2.0 ports: - 5000 environment: - KDB_LICENSE_B64 - KXI_SP_SPEC=/app/spec.q - KXI_SP_PARENT_HOST=controller:6000 volumes: - .:/app command: ["-p", "5000"] deploy: restart_policy: condition: on-failure depends_on: - controller
With this Docker Compose file, the Controller and Worker can be created at once with:
Alternatively, to run with multiple Workers as before, change the Controller to expect more Workers:
controller: .. environment: .. - KXI_SP_MIN_WORKERS=3 ..
If the Worker needs to reach the host network, change the Worker to use an ephemeral host port:
worker: .. ports: - 5000 ..
Then scale the Docker Compose by running with the
docker-compose up --scale worker=3
or by setting a replica count:
worker: .. deploy: replicas: 3 ..
Running separate containers
Running with one Worker
First, create a
kx network and a Controller to orchestrate and manage the pipeline.
docker network create kx docker run -it -p 6000:6000 \ --network=kx \ -e "KDB_LICENSE_B64=$KDB_LICENSE_B64" \ # Set the kdb+ license to use --restart unless-stopped \ # Restart the Controller if it dies registry.dl.kx.com/kxi-sp-controller:1.2.0 -p 6000
A Controller then needs Workers to orchestrate. We need to know the hostname of the Controller.
CONTAINER ID IMAGE .. PORTS NAMES 0d05f4679db2 kxi-sp-controller:1.2.0 .. 0.0.0.0:6000->6000/tcp, :::6000->6000/tcp cranky_mclaren
Note the container ID of the Controller, and change the
KXI_SP_PARENT_HOST below to the container ID output from that command.
Bind in the project directory to make the spec available.
A Worker can be created with:
docker run -it -p 5000:5000 \ --network=kx \ -v "$(pwd)":/app \ # Bind in the project directory -e KXI_SP_SPEC="/app/spec.q" \ # Point to the bound spec file -e KXI_SP_PARENT_HOST="0d05f4679db2:6000" \ # Point Worker to its Controller -e "KDB_LICENSE_B64=$KDB_LICENSE_B64" \ # Set the kdb+ license to use --restart unless-stopped \ # Restart the Worker if it dies registry.dl.kx.com/kxi-sp-worker:1.2.0 -p 5000
Running with multiple Workers
Rather than running the pipeline with a single Worker, some pipelines (such as those reading from Kafka or callback functions) can be parallelized by orchestrating multiple Workers.
To do this, start a new Controller with a greater number of required Workers:
$ docker run -it -p 6000:6000 \ --network=kx \ -e "KDB_LICENSE_B64=$KDB_LICENSE_B64" \ -e KXI_SP_MIN_WORKERS=3 \ # Set this pipeline to use 3 Workers --restart unless-stopped \ registry.dl.kx.com/kxi-sp-controller:1.2.0 -p 6000
Then launch the required number of Workers. Here we use a loop to set each to a known port.
KXI_SP_PARENT_HOST to the new Controller’s container ID.
for port in 5001 5002 5003; do docker run -it -p $port:5000 \ --network=kx \ -v "$(pwd)":/app \ -e KXI_SP_SPEC="/app/spec.q" \ -e KXI_SP_PARENT_HOST="0d05f4679db2:6000" \ -e "KDB_LICENSE_B64=$KDB_LICENSE_B64" \ registry.dl.kx.com/kxi-sp-worker:1.2.0 -p 5000 done
There will now be three Workers up and running, on ports 5001, 5002, and 5003.