Skip to content

Overlays & Patches

For some context

Take a look at:

Questions

What are Patches ?

  • Patches are partial configuration blocks that can be "overlaid" onto a given Package object.
  • We can store them and ship them with a package in order to provide us a quick way to modify settings in the package.
  • We can also use them to generate different packages from a given base package by entities defined in that base and "overlaying" patches on top of it until we get our desired state

How does it work ?

Patches work by utilizing the Package object model and manipulating it using kustomize. The functionality is a little bit pared back to remove scope for user error.

The name parameter of any given object is used as a key and array types will be merged according to the strategic merge mechanism.

There's no mechanism to patch code files currently, only config

Using Patches

E.g. Adding pipeline

In order to create a patch we can simply do the usual add command, in this case specifying patch as your new object:

kxi package -q init mypkg --force
kxi package -q add --to mypkg patch --name my-first-patch
cat mypkg/patches/my-first-patch.yaml

kind: Package
apiVersion: pakx/v1
metadata:
  name: target
spec:
  manifest: {}
  databases: []
  pipelines: []
  router: null
  deployment_config: null

This will add a patch file in the patches directory, the patches file will basically be empty but gives you an idea of how to structure your patch further.

What to do with this template patch

  • Do not modify anything above the spec: field
  • Good practice to remove everything inside spec: except for what you want to patch

Understanding the package object model

Check out the Package Spec Reference for the full object defintion

We can see the patch object is also present in the manifest file:

kxi package info mypkg

==PATH==
/builds/kxdev/kxinsights/documentation/code-kx-com/mypkg

==OBJECT==
Package

==Manifest==
name: mypkg
version: 0.0.1
metadata:
  description: ''
  authors:
  - name: root
    email: ''
entrypoints:
  default: init.q
patches:
- name: my-first-patch

Let's modify the patch to add a new pipeline.

patchspec=$(cat mypkg/patches/my-first-patch.yaml | grep spec: -B10 | cat - <(echo "  pipelines: [{name: mynewpipeline, spec: init.q}]"))
echo "$patchspec" > mypkg/patches/my-first-patch.yaml
cat mypkg/patches/my-first-patch.yaml

kind: Package
apiVersion: pakx/v1
metadata:
  name: target
spec:
  pipelines: [{name: mynewpipeline, spec: init.q}]

We can then apply or "overlay" this patch onto our current package using the overlay command:

Without specifying any other information, the overlay command resolves All the patches found in the manifest

kxi package overlay mypkg
kxi package info mypkg | grep pipelines: -A5

Found patch to overlay: mypkg/patches/my-first-patch.yaml
Writing /builds/kxdev/kxinsights/documentation/code-kx-com/mypkg/manifest.yaml
pipelines:
  mynewpipeline:
    file: pipelines/mynewpipeline.yaml
router: ''
deployment_config: ''
patches:

When we specify patches after the package name, we can select any valid patch file, they dont even need to be located inside a package!

kxi package overlay mypkg mypkg/patches/my-first-patch.yaml
kxi package info mypkg | grep pipelines: -A5

Writing /builds/kxdev/kxinsights/documentation/code-kx-com/mypkg/manifest.yaml
pipelines:
  mynewpipeline:
    file: pipelines/mynewpipeline.yaml
router: ''
deployment_config: ''
patches:

E.g. Modifying Tables

Because Tables are contained inside the Database construct we must patch the Database:

kxi package -q init mypkg --force
kxi package -q add --to mypkg database --name mydb 
# This automatically populates a "default" table

kxi package -q add --to mypkg patch --name my-db-patch

read -r -d '' dbspec <<'EOF'
    databases:
      - name: mydb
        tables:
          schemas: 
            - name: default
              columns:
                - name: myfloat
                  type: float
                  description: we've added a new float column called myfloat 
                - name: x2 
                  type: symbol
                  description: The default x2 column has changed type
              description: We've changed the default table's description too!
EOF

# The patch might look a little complex, but it's just a few nested structs 
# N.B We have indented it here to nest it under the `spec` field

patchspec=$(cat mypkg/patches/my-db-patch.yaml | grep spec: -B10 | cat - <(echo "  $dbspec"))
echo "$patchspec" > mypkg/patches/my-db-patch.yaml
# This is just to stitch the header with the dbspec we wrote

cat mypkg/patches/my-db-patch.yaml
kxi package overlay mypkg

echo""
echo "----Resultant schema for 'default' table----"
echo "--------------------------------------------"
cat mypkg/databases/mydb/schema.yaml

kind: Package
apiVersion: pakx/v1
metadata:
  name: target
spec:
  databases:
      - name: mydb
        tables:
          schemas: 
            - name: default
              columns:
                - name: myfloat
                  type: float
                  description: we've added a new float column called myfloat 
                - name: x2 
                  type: symbol
                  description: The default x2 column has changed type
              description: We've changed the default table's description too!
Found patch to overlay: mypkg/patches/my-db-patch.yaml
Writing /builds/kxdev/kxinsights/documentation/code-kx-com/mypkg/manifest.yaml

----Resultant schema for 'default' table----
--------------------------------------------
schemas:
- name: default
  columns:
  - description: we've added a new float column called myfloat
    name: myfloat
    type: float
  - description: The default x2 column has changed type
    name: x2
    type: symbol
  - description: test col t1
    name: t1
    type: timestamp
  description: We've changed the default table's description too!
  prtnCol: t1
  type: partitioned

Column Ordering

After patching we can observe the columns have changed order. The new column has been added in at the top. If your application requires a specific column ordering then you will need to do this manually.

Referencing Deps Patches

We are able to reference patches held in other packages too so that we don't need to continually copy patch specs that we like!

Referenced Dep must already be installed

Referencing a patch from a dependency can only be done after that dep is installed. In the below example we explicitly run: kxi package install myotherdep This allows the kxi package to resolve it properly when we then call overlay

We do this in the following way:

kxi package -q init myotherpkg --force
# setup

kxi package -q add --to myotherpkg patch --name patch-from-afar
patchspec=$(cat myotherpkg/patches/patch-from-afar.yaml | grep spec: -B10 | cat - <(echo "  pipelines: [{name: mynewpipeline, spec: init.q}]"))
echo "$patchspec" > myotherpkg/patches/patch-from-afar.yaml
# add a patch to myother pkg

_=$(kxi package -q install myotherpkg)
# install myotherpkg so that we can reference it 

kxi package -q add --to mypkg dep --name myotherpkg --version 0.0.1 
# add a dependency on myotherpkg to mypkg

kxi package -q add --to mypkg patch --name patch-from-afar --dep myotherpkg
# add a patch in mypkg referencing a patch in myotherpkg

kxi package -q overlay mypkg

# see that we've successfully used the patch adding a new pipeline from myotherpkg!
kxi package info mypkg | grep pipelines: -A5

pipelines:
  mynewpipeline:
    file: pipelines/mynewpipeline.yaml
router: ''
deployment_config: ''
patches: