Skip to content

Templating Tables Across Packages

This walkthrough shows how to keep one authoritative set of table schemas and render it into several packages, so that adding or changing a table is a single edit.

When several packages need the same tables, copying the schema files into each package means every change has to be repeated, and the copies drift. Instead, hold the schemas in one values file and give each package a small template that renders them into table components. Each package still owns its own database, defined once as an ordinary package component; the template only supplies the tables, and never modifies the database.

Prerequisites

  • The kxi CLI installed and authenticated against your kdb Insights Enterprise environment.
  • Familiarity with template packages.

Set up the files

You author all of the files below yourself; none of them ship with kdb Insights Enterprise. shared/ is not a package, just a directory you keep beside your packages to hold the single copy of each file.

Two packages share one set of table schemas, with one template file each:

shared/values.yaml            # All table schemas. The only file you edit.
shared/prod.yaml              # Production-only overrides. Optional.
shared/tables.yaml.j2         # Your master copy of the template.
pkga/tables/tables.yaml.j2    # Identical copy of shared/tables.yaml.j2
pkga/databases/pkga-db/       # The package's own database. Not templated.
pkgb/tables/tables.yaml.j2    # Identical copy of shared/tables.yaml.j2
pkgb/databases/pkgb-db/       # The package's own database. Not templated.

Create the shared values file, holding every table schema keyed by table name. Keying by name rather than using a list is what lets --set target an individual table later:

# shared/values.yaml
# All table schemas. Keyed by name so --set can target one table.
schemas:
  trade:
    type: partitioned
    prtnCol: ts
    sortColsDisk: [ts]
    columns:
      - { name: ts, type: timestamp }
      - { name: sym, type: symbol }
      - { name: price, type: float }
      - { name: size, type: long }

  quote:
    type: partitioned
    prtnCol: ts
    sortColsDisk: [ts]
    columns:
      - { name: ts, type: timestamp }
      - { name: sym, type: symbol }
      - { name: bid, type: float }
      - { name: ask, type: float }

  alx:
    type: splayed
    columns:
      - { name: sym, type: symbol }
      - { name: vwap, type: float }

Create the template. It emits one table document per schema, separated by ---, adding the key as each table's name:

# shared/tables.yaml.j2
# Keep identical in every package. Never needs editing.
# Do not put Jinja delimiters in comments here - they still evaluate.
{% for table_name, schema in schemas.items() %}
{% if not loop.first %}---{% endif %}
{{ dict(schema, name=table_name, metadata=dict(type="table", apiVersion="v1")) | tojson }}
{% endfor %}

Each document is emitted as JSON, which is valid YAML, so the schema's fields pass straight through. Only name and metadata are added, which means any field the table model accepts can be set in shared/values.yaml without touching the template. That is why trade and quote carry prtnCol and sortColsDisk while the splayed alx does not.

schemas is just a name this template uses

What matters is that the template reads the same name, and that every rendered document sets metadata.type. Naming the block tables or models in shared/values.yaml works just as well, and the values are then addressed as --set tables.mytable.prtnCol=date or --set models.mytable.prtnCol=date.

Because the rendered documents are tables rather than a database, the package's database is left exactly as you defined it. The rendered tables are attached to the package, and appear in the manifest under tables.

Create the two packages, copy the template into each at tables/tables.yaml.j2, and give each one a database to host the tables, added once as a normal package component:

# Create the packages
kxi package init pkga
kxi package init pkgb

# Copy in the shared template
mkdir -p pkga/tables pkgb/tables
cp shared/tables.yaml.j2 pkga/tables/tables.yaml.j2
cp shared/tables.yaml.j2 pkgb/tables/tables.yaml.j2

# Each package needs its own database
kxi package add --to ./pkga database --name pkga-db
kxi package add --to ./pkgb database --name pkgb-db

The template needs no per-package values, so the packages differ only in their own name and database.

Render

Render each package, passing the shared schemas with -f:

kxi package template ./pkga -f shared/values.yaml -o out/pkga
kxi package template ./pkgb -f shared/values.yaml -o out/pkgb

Confirm both packages received the same tables:

diff -r out/pkga/tables out/pkgb/tables

No output means the schema sets are identical. Each rendered package is a normal package: inspect it, push it, or deploy it.

Add or change a table

Edit shared/values.yaml and re-render. Nothing else changes, and every package picks up the edit on its next render.

Add a package

kxi package init ./pkgc
kxi package add --to ./pkgc database --name pkgc-db
mkdir -p pkgc/tables
cp shared/tables.yaml.j2 pkgc/tables/tables.yaml.j2
kxi package template ./pkgc -f shared/values.yaml -o out/pkgc

Override per environment

Values are merged lowest to highest: the package's own root values.yaml if it has one, then -f files left to right, then --set-file, then --set.

Hold each environment's differences in their own file, naming only the fields that differ:

# shared/prod.yaml
schemas:
  trade:
    sortColsDisk: [ts, sym]

Layer it over the shared schemas, or override a single field inline:

# Layer the production overrides over the shared schemas
kxi package template ./pkga -f shared/values.yaml -f shared/prod.yaml -o out/pkga

# Override a single field for one render
kxi package template ./pkga -f shared/values.yaml --set schemas.trade.prtnCol=date -o out/pkga

The merge is deep, so trade keeps its type, prtnCol and columns from shared/values.yaml and gains sortColsDisk from shared/prod.yaml.

Things to watch for

  • --set does not support list indexing. schemas[0].prtnCol=date is silently ignored, so address each schema by its key, as in --set schemas.trade.prtnCol=date.
  • Jinja2 renders YAML comments. Delimiters inside a # comment are still evaluated, and an undefined name there yields an empty package with exit code 0.
  • Always set metadata.type on every document. Without it the render is not recognised as a package component, and the whole rendered file is written into the package as a plain data file instead, so no tables are created.
  • -o cannot point inside the package directory; such a path is silently relocated to the parent. If the target exists you are prompted, and it is then deleted.
  • manifest.databases.<db>.tables: [] in the rendered output is expected, even though the tables rendered successfully. The package model holds tables at the top level, under tables, and the database references them at deploy time. Their order in the manifest follows the order the template emits them, which is the order of the keys in shared/values.yaml, so the template can control it.

Changing table order after a render

Because the order comes from the template's output, reordering the keys in shared/values.yaml, or sorting them in the template, changes the order in every package that uses it. To reorder the tables of one package only, render it with kxi package template and edit the tables block of the rendered manifest.yaml before pushing.

Next steps

  • Read the template packages reference for the full set of commands, context variables, and deploy-time options.
  • Learn how to upload and deploy the rendered packages.
Back to top