Skip to content

Patches and Overlays

This page provides an overview of overlays and patches and how they work with kxi packaging in kdb Insights Enterprise.

Use templates for new parameterized packages

Package templates are the replacement for overlays when you are building a new parameterized package. Use the kxi package template command for that work. Overlays and patches are retained for compatibility with existing packages.

Patches are partial configuration blocks that can be overlaid onto a given package.

You can store them and ship them with a package to provide a quick way to modify settings and generate different packages from a given base package, overlaying patches on top until you get your desired state.

Patches work by utilizing the Package object model and manipulating it using kustomize.

The name parameter of the object being overlaid (for example: a database or table) is used as a key and array types are merged according to the strategic merge mechanism.

There is no mechanism to patch code files currently, only configuration files, which are databases and pipelines.

Using patches

Adding a pipeline

  1. To create a patch, use the add command and specify patch as your new object:

    kxi package -q init mypkg --force
kxi package -q add --to mypkg patch --name my-first-patch
cat mypkg/patches/my-first-patch.yaml

    metadata:
  name: target
kind: Package
apiVersion: pakx/v1
spec:
  manifest: {}
  tables:
    schemas: []
  databases: []
  pipelines: []
  router: {}
  views: []
  deployment_config: {}

    This adds a patch file to the patches directory. The new file includes the structure required for a patch file, but no pipeline.

    Be careful when you modify the patch file

    • Do not modify the headers above the spec: field
    • It is good practice to remove everything inside spec: except what you want to patch

  2. You can see the patch object is now present in the manifest file:

    kxi package info mypkg

    ╭──────────────┬──────────────────────────────────────╮
│ Name         │ Value                                │
├──────────────┼──────────────────────────────────────┤
│ Package UUID │ 05cb7fbb-93d9-463a-ad50-4ffe043378ef │
│ Package Name │ mypkg (0.0.1)                        │
│ Owner        │ -                                    │
│ Components   │ -                                    │
│ Workloads    │ -                                    │
│ Dependencies │ -                                    │
╰──────────────┴──────────────────────────────────────╯

  3. Modify the patch to add a new pipeline. Keep the headers above spec: unchanged, and leave only the section you want to patch inside spec::

    cat > mypkg/patches/my-first-patch.yaml <<'EOF'
metadata:
  name: target
kind: Package
apiVersion: pakx/v1
spec:
  pipelines:
  - name: mynewpipeline
    spec: init.q
EOF
cat mypkg/patches/my-first-patch.yaml

metadata:
  name: target
kind: Package
apiVersion: pakx/v1
spec:
  pipelines:
  - name: mynewpipeline
    spec: init.q

  4. Apply or "overlay" this patch onto your current package using the overlay command:

    Without specifying any other information, the overlay command resolves All the patches found in the manifest:

    kxi package overlay mypkg
cat mypkg/manifest.yaml | grep pipelines: -A5

WARNING  Creating mypkg/init.q with default spec
pipelines:
  mynewpipeline:
    file: pipelines/mynewpipeline.yaml
views: {}
udf_namespaces:
- udf

    Overlaying a patch can create a missing entrypoint file

    A patch that references a source file which does not exist yet, such as the spec: init.q above, makes overlay create that file with a default spec and print a warning. This is the one case where init.q appears without you adding it; kxi package init never creates it.

    When you specify patches after the package name, you can select any valid patch file, they don't even need to be located inside a package.

    kxi package overlay mypkg mypkg/patches/my-first-patch.yaml
cat mypkg/manifest.yaml | grep pipelines: -A5

    RuntimeError Kustomize binaries not found
To install the required dependencies, please run:
`uv pip install pakxcli[kustomize]`
N.B. you may need to supply `--extra-index-url` with credentials the above 
command
pipelines: {}
views: {}
patches:
- metadata:
    type: ''
  name: my-first-patch

Modifying tables

To modify a table, you must patch the Database because tables are contained inside the Database construct :

kxi package -q init mypkg --force
kxi package -q add --to mypkg database --name mydb
# This automatically populates a "default" table

kxi package -q add --to mypkg patch --name my-db-patch

# Write the complete patch file. The headers above `spec:` are unchanged from the
# generated file; only the section being patched is kept inside `spec:`.
cat > mypkg/patches/my-db-patch.yaml <<'EOF'
metadata:
  name: target
kind: Package
apiVersion: pakx/v1
spec:
  databases:
  - name: mydb
    tables:
      schemas:
      - name: default
        description: We've changed the default table's description too!
        type: basic
        columns:
        - name: myfloat
          type: float
          description: we've added a new float column called myfloat
        - name: x2
          type: symbol
          description: The default x2 column has changed type
EOF

cat mypkg/patches/my-db-patch.yaml
kxi package overlay mypkg

echo ""
echo "----Resultant schema for 'default' table----"
echo "--------------------------------------------"
cat mypkg/databases/mydb/tables/default.yaml

WARNING  Path mypkg already exists.
WARNING  Overwriting existing package manifest at mypkg/manifest.yaml
metadata:
  name: target
kind: Package
apiVersion: pakx/v1
spec:
  databases:
  - name: mydb
    tables:
      schemas:
      - name: default
        description: We've changed the default table's description too!
        type: basic
        columns:
        - name: myfloat
          type: float
          description: we've added a new float column called myfloat
        - name: x2
          type: symbol
          description: The default x2 column has changed type

----Resultant schema for 'default' table----
--------------------------------------------
# yaml-language-server: $schema=https://code.kx.com/insights/enterprise/packaging/schemas/package.json#/$defs/Table
columns:
- description: we've added a new float column called myfloat
  name: myfloat
  type: float
- description: The default x2 column has changed type
  name: x2
  type: symbol
description: We've changed the default table's description too!
name: default
type: basic
metadata:
  apiVersion: v1
  type: table
  tags: []

Column ordering

After patching you can observe the columns have changed order. The new column has been added in at the top. If your application requires specific column ordering, then you need to do this manually.

Referencing dependency patches

You can also reference patches held in other packages to minimize the need to copy patch specs that you use regularly.

Dependency packages must be siblings of the target package

Dependency packages referenced by a patch must be available as sibling package directories in the same parent directory as the target package.

You do this in the following way:

# create mypkg and myotherpkg as siblings in the same parent directory
kxi package -q init mypkg --force
kxi package -q init myotherpkg --force

# add a patch to myotherpkg
kxi package -q add --to myotherpkg patch --name patch-from-afar
cat > myotherpkg/patches/patch-from-afar.yaml <<'EOF'
metadata:
  name: target
kind: Package
apiVersion: pakx/v1
spec:
  pipelines:
  - name: mynewpipeline
    spec: init.q
EOF

# add a dependency on myotherpkg to mypkg
kxi package add --to mypkg dep --name myotherpkg --version 0.0.1

# add a patch in mypkg referencing a patch in myotherpkg
kxi package add --to mypkg patch --name patch-from-afar --dep myotherpkg

kxi package overlay mypkg

# see that you have successfully used the patch adding a new pipeline from myotherpkg
cat mypkg/manifest.yaml | grep pipelines: -A5

WARNING  Path mypkg already exists.
WARNING  Overwriting existing package manifest at mypkg/manifest.yaml
pipelines:
  mynewpipeline:
    file: pipelines/mynewpipeline.yaml
views: {}
udf_namespaces:
- udf
Back to top