Skip to content

Overlays & Patches

For some context

Take a look at:

Questions

What are Patches ?

  • Patches are partial configuration blocks that can be "overlaid" onto a given Package object.
  • We can store them and ship them with a package in order to provide us a quick way to modify settings in the package.
  • We can also use them to generate different packages from a given base package by entities defined in that base and "overlaying" patches on top of it until we get our desired state

How does it work ?

Patches work by utilizing the Package object model and manipulating it using kustomize. The functionality is a little bit pared back to remove scope for user error.

The name parameter of any given object is used as a key and array types will be merged according to the strategic merge mechanism.

There's no mechanism to patch code files currently, only config

Using Patches

E.g. Adding pipeline

In order to create a patch we can simply do the usual add command, in this case specifying patch as your new object:

kxi package -q init mypkg --force
kxi package -q add --to mypkg patch --name my-first-patch
cat mypkg/patches/my-first-patch.yaml

uuid: null
kind: Package
apiVersion: pakx/v1
metadata:
  name: target
spec:
  uuid: 5aa3bcf1-9d87-452d-b120-d5418125652a
  manifest: {}
  tables:
    uuid: null
    schemas: []
  databases: []
  pipelines: []
  router: null
  deployment_config: null

This will add a patch file in the patches directory, the patches file will basically be empty but gives you an idea of how to structure your patch further.

What to do with this template patch

  • Do not modify anything above the spec: field
  • Good practice to remove everything inside spec: except for what you want to patch

Understanding the package object model

Check out the Package Spec Reference for the full object defintion

We can see the patch object is also present in the manifest file:

kxi package info mypkg

==PATH==
/builds/kxdev/kxinsights/documentation/code-kx-com/mypkg

==OBJECT==
Package

==Manifest==
name: mypkg
version: 0.0.1
metadata:
  description: ''
  authors:
  - name: root
    email: ''
entrypoints:
  default: init.q
patches:
- name: my-first-patch

Let's modify the patch to add a new pipeline.

patchspec=$(cat mypkg/patches/my-first-patch.yaml | grep spec: -B10 | cat - <(echo "  pipelines: [{name: mynewpipeline, spec: init.q}]"))
echo "$patchspec" > mypkg/patches/my-first-patch.yaml
cat mypkg/patches/my-first-patch.yaml

uuid: null
kind: Package
apiVersion: pakx/v1
metadata:
  name: target
spec:
  pipelines: [{name: mynewpipeline, spec: init.q}]

We can then apply or "overlay" this patch onto our current package using the overlay command:

Without specifying any other information, the overlay command resolves All the patches found in the manifest

kxi package overlay mypkg
kxi package info mypkg | grep pipelines: -A5

Found patch to overlay: mypkg/patches/my-first-patch.yaml
Writing /builds/kxdev/kxinsights/documentation/code-kx-com/mypkg/manifest.yaml
pipelines:
  mynewpipeline:
    file: pipelines/mynewpipeline.yaml
router: ''
deployment_config: ''
patches:

When we specify patches after the package name, we can select any valid patch file, they dont even need to be located inside a package!

kxi package overlay mypkg mypkg/patches/my-first-patch.yaml
kxi package info mypkg | grep pipelines: -A5

Writing /builds/kxdev/kxinsights/documentation/code-kx-com/mypkg/manifest.yaml
pipelines:
  mynewpipeline:
    file: pipelines/mynewpipeline.yaml
router: ''
deployment_config: ''
patches:

E.g. Modifying Tables

Because Tables are contained inside the Database construct we must patch the Database:

uuid: null
kind: Package
apiVersion: pakx/v1
metadata:
  name: target
spec:
  databases:
      - name: mydb
        tables:
          schemas: 
            - name: default
              columns:
                - name: myfloat
                  type: float
                  description: we've added a new float column called myfloat 
                - name: x2 
                  type: symbol
                  description: The default x2 column has changed type
              description: We've changed the default table's description too!
Found patch to overlay: mypkg/patches/my-db-patch.yaml
Below 1 Errors have message:
field required
databases -> 0 -> tables -> schemas -> 0 -> type
Showing details of first error below:

ValidationError when building Table
1 validation error for Table
type
  field required (type=value_error.missing)
Fields for Table
=======================
field         required    type                 class                description
------------  ----------  -------------------  -------------------  ---------------------------------------------------------
name          False       ConstrainedStrValue  ConstrainedStrValue  Table name
uuid          False       UUID                 UUID                 None
blockSize     False       int                  int                  Number of rows to keep in-memory before write to disk
columns       True        Column               Column               A List of column schemas
description   False       str                  str                  A string describing the purpose of this table
isSharded     False       bool                 bool                 Specifies if a reference data table is distributed across
                                                                    multiple assemblies
oldName       False       ConstrainedStrValue  ConstrainedStrValue  Old table name, for automated renaming
primaryKeys   False       List                 ConstrainedStrValue  Names of primary key columns
prtnCol       False       ConstrainedStrValue  ConstrainedStrValue  A timestamp column name used for data partitioning
sortColsDisk  False       List                 ConstrainedStrValue  Names of columns to sort on when stored on disk
sortColsMem   False       List                 ConstrainedStrValue  Names of columns to sort on when stored in memory
sortColsOrd   False       List                 ConstrainedStrValue  Names of columns to sort on when stored on disk with an
                                                                    ordinal partition scheme
type          True        Type3                Type3                One of {splayed, partitioned, basic}. Required


----Resultant schema for 'default' table----
--------------------------------------------
cat: mypkg/databases/mydb/schema.yaml: No such file or directory

Column Ordering

After patching we can observe the columns have changed order. The new column has been added in at the top. If your application requires a specific column ordering then you will need to do this manually.

Referencing Deps Patches

We are able to reference patches held in other packages too so that we don't need to continually copy patch specs that we like!

Referenced Dep must already be installed

Referencing a patch from a dependency can only be done after that dep is installed. In the below example we explicitly run: kxi package install myotherdep This allows the kxi package to resolve it properly when we then call overlay

We do this in the following way:

kxi package -q init myotherpkg --force
# setup

kxi package -q add --to myotherpkg patch --name patch-from-afar
patchspec=$(cat myotherpkg/patches/patch-from-afar.yaml | grep spec: -B10 | cat - <(echo "  pipelines: [{name: mynewpipeline, spec: init.q}]"))
echo "$patchspec" > myotherpkg/patches/patch-from-afar.yaml
# add a patch to myother pkg

_=$(kxi package -q install myotherpkg)
# install myotherpkg so that we can reference it 

kxi package -q add --to mypkg dep --name myotherpkg --version 0.0.1 
# add a dependency on myotherpkg to mypkg

kxi package -q add --to mypkg patch --name patch-from-afar --dep myotherpkg
# add a patch in mypkg referencing a patch in myotherpkg

kxi package -q overlay mypkg

# see that we've successfully used the patch adding a new pipeline from myotherpkg!
kxi package info mypkg | grep pipelines: -A5

Below 1 Errors have message:
field required
databases -> 0 -> tables -> schemas -> 0 -> type
Showing details of first error below:

ValidationError when building Table
1 validation error for Table
type
  field required (type=value_error.missing)
Fields for Table
=======================
field         required    type                 class                description
------------  ----------  -------------------  -------------------  ---------------------------------------------------------
name          False       ConstrainedStrValue  ConstrainedStrValue  Table name
uuid          False       UUID                 UUID                 None
blockSize     False       int                  int                  Number of rows to keep in-memory before write to disk
columns       True        Column               Column               A List of column schemas
description   False       str                  str                  A string describing the purpose of this table
isSharded     False       bool                 bool                 Specifies if a reference data table is distributed across
                                                                    multiple assemblies
oldName       False       ConstrainedStrValue  ConstrainedStrValue  Old table name, for automated renaming
primaryKeys   False       List                 ConstrainedStrValue  Names of primary key columns
prtnCol       False       ConstrainedStrValue  ConstrainedStrValue  A timestamp column name used for data partitioning
sortColsDisk  False       List                 ConstrainedStrValue  Names of columns to sort on when stored on disk
sortColsMem   False       List                 ConstrainedStrValue  Names of columns to sort on when stored in memory
sortColsOrd   False       List                 ConstrainedStrValue  Names of columns to sort on when stored on disk with an
                                                                    ordinal partition scheme
type          True        Type3                Type3                One of {splayed, partitioned, basic}. Required

pipelines: {}
router: ''
deployment_config: ''
patches:
- name: my-db-patch
- name: patch-from-afar