Securing Pipeline Credentials

This page outlines how to create and reference environment variables as node parameters and have them resolve at runtime. This can facilitate the management of sensitive data within pipelines by separating these values from the pipeline definition and mounting them into pipelines as environment variables at runtime.

This functionality can be used for any environment variables, not just credentials

The methods to secure credentials using environment variables will vary between Kubernetes and Docker

Use Environment Variables in pipelines

If an environment variable is defined in the worker, its value can be used as a parameter within your pipeline. In the following example, the Kafka broker being connected to requires SASL authentication, which in turn depends on configured username and password variables. These environment variables can be created by following the Kubernetes and Docker guides.

qPython

When creating a q pipeline:

Use the .qsp.useVar function to set an environment variable as a node parameter.
In the code snippet below, the $KAFKA_USERNAME and $KAFKA_PASSWORD environment variables are specified using this API.

The variables are resolved to their values at runtime.

username : .qsp.useVar "KAFKA_USERNAME";
password : .qsp.useVar "KAFKA_PASSWORD";
options : `sasl.username`sasl.password!(username; password);
.qsp.run
    .qsp.read.fromKafka[topic; broker; .qsp.use enlist[`options]!enlist options]

Snippet detail

username : .qsp.useVar "KAFKA_USERNAME" - Creates an object to resolve the $KAFKA_USERNAME environment variable.
password : .qsp.useVar "KAFKA_PASSWORD" - Creates an object to resolve the $KAFKA_PASSWORD environment variable.
options :sasl.usernamesasl.password!(username; password); - Builds a q dictionary with two keys; sasl.username and sasl.password, whose values come from the environment variable objects.
.qsp.run - Launches the pipeline.
.qsp.read.fromKafka[topic; broker; - Creates the Kafka reader with a specific topic and broker.
.qsp.use enlist[options]!enlist options]` - Pulls the username and password into the Kafka reader.

When creating a Python pipeline:

Use the kxi.sp.use_var function to set an environment variable as a node parameter.
In the code snippet below, the $KAFKA_USERNAME and $KAFKA_PASSWORD environment variables are specified using this API.

The variables are resolved to their values at runtime.

from kxi import sp
username = sp.use_var("KAFKA_USERNAME")
password = sp.use_var("KAFKA_PASSWORD")
options = {"sasl.username": username, "sasl.password": password};
sp.run(
    sp.read.from_kafka(topic; broker; {"options": options})

Snippet detail

from kxi import sp - Imports the SP API.
username = sp.use_var("KAFKA_USERNAME") - Creates an object to resolve the $KAFKA_USERNAME environment variable.
password = sp.use_var("KAFKA_PASSWORD") - Creates an object to resolve the $KAFKA_PASSWORD environment variable.
options = {"sasl.username": username, "sasl.password": password}; - Builds the Kafka authentication options.
sp.run - Starts the pipeline.
sp.read.from_kafka(topic; broker; {"options": options} - Defines the Kafka reader using the environment variables in the Kafka authentication options.

Kubernetes Guide

Secret Creation

To create a Kubernetes Secret, define it in a YAML manifest and apply it with kubectl. For example:

apiVersion: v1
kind: Secret
    metadata:
        name: kafka-secret
        type: Opaque
    data:
        username: <base64-encoded-username>
        password: <base64-encoded-password>

Kubernetes YAML manifest details

apiVersion: v1 - The API group and version to use.
kind: Secret - Tells Kubernetes that you are creating a Secret.
metadata: This section provides the information:
- name: kafka-secret - The name of the secret. In this example, we are creating a secret to be used by the Kafka reader so have named it kafka-secret. Any name can be used.
- type: Opaque - Tells Kubernetes this is a generic user-defined type.
data: This contains the actual data for the secret, and says that the values must be base64 encoded strings, not plain text.
- username: <base64-encoded-username> - Specify the actual username here.
- password: <base64-encoded-password> - Specify the actual password here.

When creating secrets, the data values must be base64 encoded.

If you don't have access to the Kubernetes cluster to create a Secret, contact your system administrator.

Mount Secrets as Environment Variables

Once a secret is created (either directly, as described above, or provisioned from a secrets manager), you can mount its keys as environment variables into your pipeline(s). This is the recommended way for pipelines to consume credentials.

Using the kafka-secret created in the previous section, mount the username and password keys into the pipeline as environment variables. For this example, call these variables KAFKA_USERNAME and KAFKA_PASSWORD. The names of these environment variables must match the values used in the node parameters.

When deploying a pipeline, using either q or python, add the following to your pipeline YAML definition to define the environment variables.

base: q
name: kafka
spec: file://src/kafka.q
env:
 - name: KAFKA_USERNAME
    valueFrom:
        secretKeyRef:
            name: kafka-secret
            key: username
 - name: KAFKA_PASSWORD
    valueFrom:
        secretKeyRef:
            name: kafka-secret
            key: password

YAML detail

This defines two environment variables:

KAFKA_USERNAME whose value is pulled securely from a Kubernetes Secret named kafka-secret, using the key username.
KAFKA_PASSWORD whose value is pulled from the same secret, using the key password.

Docker Guide

Because Docker does not support the concept of secrets, the values are used directly as variables:

CLIDocker Compose

Defining environment variables in a Docker container can be done using:

CLI arguments, or

--env KAFKA_USERNAME=user --env KAFKA_PASSWORD=password

an env file
```
--env-file=.env
```

If Docker Compose is being used, the variables can be defined in the compose file using:

the values directly, or

services:
    sp_worker:
        environment:
            KAFKA_USERNAME: "user"
            KAFKA_PASSWORD: "password"

an env file

services:
    sp_worker:
        env_file: ".env"

Once the variables are set, the methods to reference them are the same.