Skip to content

Template Packages

This page describes how to author packages that contain templated YAML files, render them into deployable packages, and deploy them with values supplied at deploy time.

A package can contain templated YAML files. You render, or resolve, these templates into a normal pushable and deployable package on demand, supplying values that customize the result. This lets you maintain a single package definition and produce many concrete variants from it, for example the same database and pipeline definitions deployed per environment or per table.

Manage from the CLI, view from the CLI or web interface

Template packages are designed to be administered from the command line interface (CLI). The web interface does not support setting template values; however, a template package renders with its default values in place, and shows as a normal deployment when it is running.

Overview

Two special file types are recognized inside a package:

File Role
Any *.yaml.j2 file Jinja2 template
values.yaml (package root) Default values used during rendering

Template files are regular package data files. The CLI treats them like any other file, so no special handling is required when you author or push a package that contains them.

There are three ways to supply values and produce a deployment:

Approach When to use
Render locally Inspect or diff the rendered output before pushing, then push the result as a regular package.
Render at push time Render as part of the push, in one step. The templates are retained and re-rendered at deploy.
Render at deploy time Push the package once with templates intact, then parameterize each deployment.

Every approach takes the same value flags and applies them in the same priority order. Refer to Render templates.

Author templates

A template file is a Jinja2-rendered YAML file. It must include a metadata.type field so that kdb Insights Enterprise knows which component type to create from the rendered output.

# tables/trade.yaml.j2
metadata:
  type: table
  apiVersion: v1
name: {{ table_name }}
type: partitioned
prtnCol: ts
sortColsDisk: [ts]
sortColsOrd: [ts]
columns:
  - name: ts
    type: timestamp

For details of the templating syntax, refer to the Jinja2 template documentation.

The supported metadata.type values are:

Value Component
table Table
pipeline Pipeline
view View
database Database
sequencer Sequencer
shard Shard
udf UDF
deployment_config DeploymentConfig
manifest Manifest, refer to Deploy a template by name
router Router
values None. Literal per-template defaults, refer to Embedded template defaults

For more information on these components, refer to the package object model.

Default values file

Place a values.yaml file at the package root to supply default values for all templates in the package:

# values.yaml
table_name: trade
pipeline_name: feed
env: prod

Only the root values.yaml is used

Nested files such as subdir/values.yaml are treated as ordinary package files, not as the values file.

Embedded template defaults

A template can also declare its own defaults inline, as the first YAML document in the file, with metadata: { type: values }. This lets a template ship sensible defaults for its own variables without requiring every caller to supply them.

# shards/query.yaml.j2
metadata: { type: values }
replicas: 1
resources:
  requests: { cpu: "1", memory: "1Gi" }
---
metadata: { type: shard, apiVersion: v1 }
name: {{ deployment_name }}
daps:
  instances:
    da:
      size: {{ replicas }}
      k8sPolicy:
        resources:
          requests: { cpu: "{{ resources.requests.cpu }}", memory: "{{ resources.requests.memory }}" }

These defaults are extracted before rendering, so they are not themselves templated. They sit at the lowest priority, which means values.yaml and any -f, --set-file, or --set overrides take precedence.

Package context variables

Every render exposes a package object in the Jinja2 context. It describes the package being rendered and is populated independently of values.yaml, -f, and --set:

Variable Description
package.name Package name.
package.version Package version.
package.entrypoints The entrypoints map from the package's manifest.yaml, for example {default: src/init.q}.
package.dependencies The package's declared dependencies, as strings.
package.database The package's database, or None if it does not have one. Its shard configuration is at package.database.shard.
package.databases The same database, as a list. A package can currently have at most one database, so this holds either nothing or a single entry, and package.database is shorthand for the first one.
deployment_name The name of the deployment being produced, supplied by the server at deploy time. Equals kxi pm deploy --name when given, otherwise package.name. Exposed at the top level, not under package.

For example, this template names its component after the package, and points the component at the package's own data source and volume, without any of those values being supplied by the caller:

# databases/replica/shard.yaml.j2
metadata:
  type: shard
  apiVersion: v1
name: {{ package.name }}-replica
daps:
  instances:
    da:
      mountList: [idb]
      source: {{ package.name }}
mounts:
  idb: { type: local, baseURI: file:///data/db/idb, partition: ordinal,
         volume: { claimName: {{ package.name }}-idb } }

The bundled read-replica template uses package.name the same way, to read the volumes of the deployment it is rendered against.

package.database reflects the package as it stands before the render

package.database and package.databases are a snapshot of the package taken before rendering begins. They hold the components it already has, whether authored locally or loaded from the server at deploy time. Nothing produced by the current render appears in them, including a database rendered by another template. Use them when a template needs to read an existing database's configuration, such as its shard settings, rather than defining its own.

Fail a render deliberately

A template can call {{ fail("message") }} to abort rendering with a validation error. Use this to assert preconditions, for example that a parent package is already deployed. A local render raises the error directly; a deploy-time render surfaces it as an HTTP 400 response.

Render templates

Templates are rendered in one of three places: on your machine, during a push, or on the server at deploy time. All three take the same value flags:

Flag Description
-f, --values <file> Load values from a YAML file. Repeatable, and merged left to right.
--set KEY=VALUE Override a single value. Repeatable, and takes the highest priority. Values are YAML-type-coerced, and dotted keys are supported, for example a.b.c=val.
--set-file KEY=FILEPATH Read a file and use its contents as the value for KEY. Supports dotted keys, as --set does. Not supported by kxi pm push.
-o, --output <dir> Write the rendered package to this directory instead of the default. Local renders only.

Values are merged in the same priority order everywhere, lowest to highest:

  1. Embedded template defaults (metadata: { type: values } documents inside a template).
  2. The package's root values.yaml.
  3. -f/--values files, applied left to right.
  4. --set-file values, where supported.
  5. --set values.

Render locally

Use the kxi package template command to render templates without deploying. The command writes a package directory in which every template is replaced by its rendered component file, which you can then push as a regular package.

kxi package template ./my-package

This creates {package-name}-rendered/ in the current working directory, where package-name comes from the package manifest rather than the directory name. The rendered package contains no *.yaml.j2 or values.yaml files, because both are consumed during rendering.

deployment_name is a deploy-time value

Only a deploy supplies deployment_name. Rendering a template that uses it locally, by supplying a value with --set, freezes that one name into the output and consumes the template, so every deployment of the rendered package reuses it and --name has no effect. Push packages that use deployment_name with their templates intact and let the server render them, once per deployment. Refer to Render at deploy time.

Overwriting an existing output directory

If the output directory already exists, you are prompted to confirm before it is replaced.

Examples

# Render with the defaults from values.yaml
kxi package template ./my-package

# Override values inline
kxi package template ./my-package --set table_name=quote --set pipeline_name=enricher

# Use a custom values file
kxi package template ./my-package -f custom-values.yaml

# Combine a base values file with inline overrides
kxi package template ./my-package -f base.yaml --set table_name=spot

# Take a value from a file's contents
kxi package template ./my-package --set-file table_schema=schema.txt --set pipeline_name=feed

# Write to a custom output directory
kxi package template ./my-package -o /tmp/rendered-pkg

The output directory is a loadable package. Inspect it, push it, or diff it against an earlier render to verify your template logic before deploying.

Render at push time

Instead of rendering to disk first, you can render templates as part of a push:

kxi pm push ./my-package --set table_name=quote -f prod-values.yaml

When you supply --set or -f, the CLI does three things before the package leaves your machine:

  1. Renders the package's templates with the merged values, using the same priority order as a local render, and adds the resulting components to the package.
  2. Keeps the *.yaml.j2 files in the pushed package, so the package can still be rendered later.
  3. Writes the merged values into the package's values.yaml, replacing whatever defaults were there.

Step 3 is the one that carries your values through to a deployment. Because the templates are retained, the server renders them again when the package is deployed, using the pushed values.yaml as its base values and merging any deploy-time values on top.

If you supply neither flag, a package that still contains *.yaml.j2 files is pushed as is, with its templates and original values.yaml intact. You can then parameterize it later, at deploy time. Refer to Render at deploy time.

What the second render does to your components

When the server renders the retained templates, each rendered component replaces the component of the same name already in the package. A component whose name is itself templated therefore renders under a new name at deploy time, and does not replace the one baked in at push time, so both end up in the deployment.

The examples below use a package holding one template that names a table from table_name, with table_name: default in the package's values.yaml.

kxi pm push ./my-package --set table_name=pushoverride
kxi pm deploy my-package
Stage Result
As pushed Table pushoverride. Templates retained, values.yaml now reads table_name: pushoverride.
Second render Base values supply pushoverride; nothing overrides it.
Deployed Table pushoverride.

The intended result. The second render produced the same table name, so it replaced the one baked in at push time. Your push values reached the deployment through values.yaml.

kxi pm push ./my-package --set table_name=pushoverride
kxi pm deploy my-package --set table_name=deployoverride
Stage Result
As pushed Table pushoverride. Templates retained, values.yaml now reads table_name: pushoverride.
Second render Deploy-time --set wins, so the template renders table deployoverride.
Deployed Tables deployoverride and pushoverride.

Not the intended result. deployoverride does not match the name baked in at push time, so it is added rather than substituted, and the stale pushoverride table is deployed alongside it.

kxi pm push ./my-package
kxi pm deploy my-package --set table_name=deployoverride
Stage Result
As pushed No table. Templates and the original values.yaml (table_name: default) intact.
Second render Deploy-time --set wins, so the template renders table deployoverride.
Deployed Table deployoverride.

The intended result, with one render and nothing to leave behind.

Do not override a templated component's name at deploy time

As the second example shows, pushing values and then overriding those same values at deploy time can deploy components from both renders. Choose one of the following instead:

  • Parameterize at deploy time. Push the package with its templates and defaults intact, without --set or -f, and supply values only on kxi pm deploy.
  • Make your push values final. Render locally with kxi package template and push the rendered output. It contains no templates, so it cannot be re-rendered and cannot pick up deploy-time values at all.
  • Repeat your values. If you must push with values, supply the same values again at deploy time so both renders agree, as in the first example.

Render at deploy time

A template package that was pushed with its templates intact can be parameterized at deploy time, without a separate render and push round trip:

kxi pm deploy my-package --set table_name=quote -f prod-values.yaml

The package's own values.yaml is merged on the server from the pushed package, not read from your local directory. Otherwise the priority order is the same as for a local render.

--name also applies to a whole-package deployment. It names the resulting deployment, and templates are rendered against that instance name, so a template that references {{ deployment_name }} sees the named instance rather than the package's own name. This lets you deploy the same templated package several times under different names without its templated resources, such as a templated database's shard and sequencer or a templated pipeline's name, colliding between instances.

Deploy a template by name

kdb Insights Enterprise can render a single named template against a package at deploy time and stand the result up as an independent deployment. The template package is pushed once, with its templates intact, and values are supplied per deployment. In the below example, the named template is read-replica, which is a template bundled with kdb Insights Enterprise.

# Deploy the read-replica template against insights-demo, as insights-demo-query
kxi pm deploy insights-demo --template read-replica --name insights-demo-query --set replicas=3

# Deploy two instances of the same package-owned template, side by side
kxi pm deploy my-package --template pipelines/feed.yaml --name feed-inst-a --set pipeline_name=feed-a
kxi pm deploy my-package --template pipelines/feed.yaml --name feed-inst-b --set pipeline_name=feed-b

Each call renders the template with the supplied values and creates a separate deployment, which appears as an assembly labelled as a template instance.

Deploying a template does not touch the package's own deployment

Omitting --template deploys the package itself, so existing package deployments are unaffected.

--template takes the template's name, which depends on where the template came from:

  • A template in a package, authored in the package directory tree. The name is its path relative to the package root with the trailing .j2 removed, so pipelines/feed.yaml.j2 becomes pipelines/feed.yaml.
  • A bundled template supplied with kdb Insights Enterprise. The name is just the template name, such as read-replica.

--name is optional. If you omit it, the name is resolved in the following order:

  1. The name declared in the template's rendered metadata: { type: manifest } document, if the template sets one.
  2. An auto-generated name of the form {package name}-{8 random hex characters}, sanitized to a valid Kubernetes resource name.

The equivalent REST call is POST /v2/deployments, documented in the Package Manager API reference.

Bundled templates

Bundled templates ship with kdb Insights Enterprise and are available to any package you can deploy against, so you author nothing and deploy them by name. A package-owned template of the same name takes precedence over the bundled one.

The reference bundled template is read-replica, which stands up additional data access processes over a deployed package's existing rdb, idb and hdb mounts. For its values, lifecycle, and a worked example, refer to deploy read replicas.

Troubleshooting

If the rendered output is missing components you expected, check the warnings printed to stderr. A template is skipped when it:

  • references an undefined variable (warning)
  • declares an unsupported metadata.type (warning)
  • renders to a YAML value that is not a dictionary (logged at debug level only, so run kxi package --debug template ... to see it)

A deploy-time render happens on the server, so its messages go to the package manager service logs rather than your terminal. To see them locally, reproduce the render with kxi package template and the same values.

Other behaviour worth knowing:

  • If a template renders to a component with the same name as an existing package component, the rendered version replaces it.
  • A templated Database or Shard, and any templated Table belonging to it, are folded into a single database, regardless of which template file is rendered first.
  • If the package contains no *.yaml.j2 files, a warning is emitted but the command still succeeds and writes the package as is.

Next steps

Back to top