Patches and Overlays

This page provides an overview of overlays and patches and how they work with kxi packaging in kdb Insights Enterprise.

Patches are partial configuration blocks that can be overlaid onto a given package.

You can store them and ship them with a package to provide a quick way to modify settings and generate different packages from a given base package, overlaying patches on top until you get your desired state.

Patches work by utilizing the Package object model and manipulating it using kustomize.

The name parameter of the object being overlaid (for example: a database or table) is used as a key and array types are merged according to the strategic merge mechanism.

There is no mechanism to patch code files currently, only configuration files, which are databases and pipelines.

Using patches

Adding a pipeline

To create a patch, use the add command and specify patch as your new object:

kxi package -q init mypkg --force
kxi package -q add --to mypkg patch --name my-first-patch
cat mypkg/patches/my-first-patch.yaml

uuid: 863b093d-f727-480c-ac11-44fc01c48d9d
kind: Package
apiVersion: pakx/v1
metadata:
  name: target
spec:
  uuid: 277b1dd2-2ece-4fc4-a0ef-c2bd013fa1c3
  manifest: {}
  tables:
    uuid: fff8d782-6dfa-4091-b98f-f68e9771d3cc
    schemas: []
  databases: []
  pipelines: []
  router: {}
  views: []
  deployment_config: {}

This adds a patch file to the patches directory. The new file includes the structure required for a patch file, but no pipeline.

Be careful when you modify the patch file

Do not modify the headers above the spec: field
It is good practice to remove everything inside spec: except what you want to patch

You can see the patch object is now present in the manifest file:

kxi package info mypkg

╭──────────────┬──────────────────────────────────────╮
│ Name         │ Value                                │
├──────────────┼──────────────────────────────────────┤
│ Package UUID │ 3c509960-dd20-4b75-828f-6bcee8b376ff │
│ Package Name │ mypkg (0.0.1)                        │
│ Author(s)    │ root                                 │
│ Components   │ ╭────────────┬─────────┬────────╮    │
│              │ │ Type       │ Name    │ Path   │    │
│              │ ├────────────┼─────────┼────────┤    │
│              │ │ Entrypoint │ default │ init.q │    │
│              │ ╰────────────┴─────────┴────────╯    │
│ Dependencies │ -                                    │
╰──────────────┴──────────────────────────────────────╯

Modify the patch to add a new pipeline:

patchspec=$(cat mypkg/patches/my-first-patch.yaml | grep spec: -B10 | cat - <(echo "  pipelines: [{name: mynewpipeline, spec: init.q}]"))
echo "$patchspec" > mypkg/patches/my-first-patch.yaml
cat mypkg/patches/my-first-patch.yaml

uuid: 863b093d-f727-480c-ac11-44fc01c48d9d
kind: Package
apiVersion: pakx/v1
metadata:
  name: target
spec:
  pipelines: [{name: mynewpipeline, spec: init.q}]

Apply or "overlay" this patch onto your current package using the overlay command:

Implicit

Without specifying any other information, the overlay command resolves All the patches found in the manifest:

kxi package overlay mypkg
cat mypkg/manifest.yaml | grep pipelines: -A5

RuntimeError Kustomize binaries not found
To install the required dependencies, please run:
`uv pip install pakxcli[kustomize]`
N.B. you may need to supply `--extra-index-url` with credentials the above 
command
pipelines: {}
views: {}
udf_namespaces:
- udf
- policy
- uda

Explicit

When you specify patches after the package name, you can select any valid patch file, they dont even need to be located inside a package.

kxi package overlay mypkg mypkg/patches/my-first-patch.yaml
cat mypkg/manifest.yaml | grep pipelines: -A5

RuntimeError Kustomize binaries not found
To install the required dependencies, please run:
`uv pip install pakxcli[kustomize]`
N.B. you may need to supply `--extra-index-url` with credentials the above 
command
pipelines: {}
views: {}
udf_namespaces:
- udf
- policy
- uda

Modifying tables

To modify a table, you must patch the Database because tables are contained inside the Database construct :

WARNING  Path mypkg already exists.                                             
WARNING  Overwriting existing package manifest at mypkg/manifest.yaml           
uuid: 2a218b99-2128-4451-bf4b-ff6e049f504a
kind: Package
apiVersion: pakx/v1
metadata:
  name: target
spec:
  databases:
      - name: mydb
        tables:
          schemas: 
            - name: default
              description: We've changed the default table's description too!
              type: basic 
              columns:
                - name: myfloat
                  type: float
                  description: we've added a new float column called myfloat 
                - name: x2 
                  type: symbol
                  description: The default x2 column has changed type
RuntimeError Kustomize binaries not found
To install the required dependencies, please run:
`uv pip install pakxcli[kustomize]`
N.B. you may need to supply `--extra-index-url` with credentials the above 
command

----Resultant schema for 'default' table----
--------------------------------------------
cat: mypkg/databases/mydb/tables/default.yaml: No such file or directory

Column ordering

After patching you can observe the columns have changed order. The new column has been added in at the top. If your application requires specific column ordering, then you need to do this manually.

Referencing dependency patches

You can also reference patches held in other packages to minimize the need to copy patch specs that you use regularly.

Referenced dependencies must already be installed.

Referencing a patch from a dependency can only be done after that dependency is installed.

In the below example you explicitly run: kxi package install myotherdep. This allows the kxi package to resolve it properly when you then call overlay.

You do this in the following way:

kxi package -q init myotherpkg --force
# setup

kxi package -q add --to myotherpkg patch --name patch-from-afar
patchspec=$(cat myotherpkg/patches/patch-from-afar.yaml | grep spec: -B10 | cat - <(echo "  pipelines: [{name: mynewpipeline, spec: init.q}]"))
echo "$patchspec" > myotherpkg/patches/patch-from-afar.yaml
# add a patch to myother pkg

_=$(kxi package -q install myotherpkg)
# install myotherpkg so that you can reference it 

kxi package -q add --to mypkg dep --name myotherpkg --version 0.0.1 
# add a dependency on myotherpkg to mypkg

kxi package -q add --to mypkg patch --name patch-from-afar --dep myotherpkg
# add a patch in mypkg referencing a patch in myotherpkg

kxi package -q overlay mypkg

# see that you have successfully used the patch adding a new pipeline from myotherpkg
cat mypkg/manifest.yaml | grep pipelines: -A5

╭─ Error ──────────────────────────────────────────────────────────────────────╮
│ This command has been deleted and is no longer available.                    │
╰──────────────────────────────────────────────────────────────────────────────╯

RuntimeError Kustomize binaries not found
To install the required dependencies, please run:
`uv pip install pakxcli[kustomize]`
N.B. you may need to supply `--extra-index-url` with credentials the above 
command
pipelines: {}
views: {}
udf_namespaces:
- udf
- policy
- uda