Skip to content

Overlays and patches

For some context

Take a look at:

Questions

What are Patches ?

  • Patches are partial configuration blocks that can be "overlaid" onto a given Package object
  • You can store them and ship them with a package in order to provide us a quick way to modify settings in the package.
  • You can also use them to generate different packages from a given base package by entities defined in that base and "overlaying" patches on top of it until you get your desired state

How does it work ?

Patches work by utilizing the Package object model and manipulating it using kustomize. The functionality is a little bit pared back to remove scope for user error.

The name parameter of any given object is used as a key and array types will be merged according to the strategic merge mechanism.

There is no mechanism to patch code files currently, only config

Using patches

Adding a pipeline

  1. In order to create a patch you can simply use the usual add command, in this case specifying patch as your new object:

    kxi package -q init mypkg --force
    kxi package -q add --to mypkg patch --name my-first-patch
    cat mypkg/patches/my-first-patch.yaml
    
    uuid: null
    kind: Package
    apiVersion: pakx/v1
    metadata:
      name: target
    spec:
      uuid: 52b6bfcf-ff3e-414e-9111-9e8122a74f8b
      manifest: {}
      tables:
        uuid: null
        schemas: []
      databases: []
      pipelines: []
      router: null
      deployment_config: null
    

    This will add a patch file in the patches directory, the patches file will basically be empty but gives you an idea of how to structure your patch further.

    What to do with this template patch

    • Do not modify anything above the spec: field
    • It is good practice to remove everything inside spec: except what you want to patch

    Understanding the package object model

    Check out the package spec reference for the full object defintion.

  2. You can see the patch object is now present in the manifest file:

    kxi package info mypkg
    
    ==PATH==
    /builds/kxdev/kxinsights/documentation/code-kx-com/mypkg
    
    ==OBJECT==
    Package
    
    ==Manifest==
    name: mypkg
    version: 0.0.1
    metadata:
      description: ''
      authors:
      - name: root
        email: ''
    entrypoints:
      default: init.q
    patches:
    - name: my-first-patch
    

  3. Modify the patch to add a new pipeline:

    patchspec=$(cat mypkg/patches/my-first-patch.yaml | grep spec: -B10 | cat - <(echo "  pipelines: [{name: mynewpipeline, spec: init.q}]"))
    echo "$patchspec" > mypkg/patches/my-first-patch.yaml
    cat mypkg/patches/my-first-patch.yaml
    
    uuid: null
    kind: Package
    apiVersion: pakx/v1
    metadata:
      name: target
    spec:
      pipelines: [{name: mynewpipeline, spec: init.q}]
    

  4. Apply or "overlay" this patch onto your current package using the overlay command:

    Without specifying any other information, the overlay command resolves All the patches found in the manifest:

    kxi package overlay mypkg
    kxi package info mypkg | grep pipelines: -A5
    
    Found patch to overlay: mypkg/patches/my-first-patch.yaml
    Writing /builds/kxdev/kxinsights/documentation/code-kx-com/mypkg/manifest.yaml
    pipelines:
      mynewpipeline:
        file: pipelines/mynewpipeline.yaml
    router: ''
    deployment_config: ''
    patches:
    

    When you specify patches after the package name, you can select any valid patch file, they dont even need to be located inside a package.

    kxi package overlay mypkg mypkg/patches/my-first-patch.yaml
    kxi package info mypkg | grep pipelines: -A5
    
    Writing /builds/kxdev/kxinsights/documentation/code-kx-com/mypkg/manifest.yaml
    pipelines:
      mynewpipeline:
        file: pipelines/mynewpipeline.yaml
    router: ''
    deployment_config: ''
    patches:
    

Modifying tables

Because Tables are contained inside the Database construct you must patch the Database:

kxi package -q init mypkg --force
kxi package -q add --to mypkg database --name mydb 
# This automatically populates a "default" table

kxi package -q add --to mypkg patch --name my-db-patch

read -r -d '' dbspec <<'EOF'
    databases:
      - name: mydb
        tables:
          schemas: 
            - name: default
              description: We've changed the default table's description too!
              type: basic 
              columns:
                - name: myfloat
                  type: float
                  description: we've added a new float column called myfloat 
                - name: x2 
                  type: symbol
                  description: The default x2 column has changed type
EOF

# The patch might look a little complex, but it's just a few nested structs 
# N.B We have indented it here to nest it under the `spec` field

patchspec=$(cat mypkg/patches/my-db-patch.yaml | grep spec: -B10 | cat - <(echo "  $dbspec"))
echo "$patchspec" > mypkg/patches/my-db-patch.yaml
# This is just to stitch the header with the dbspec we wrote

cat mypkg/patches/my-db-patch.yaml
kxi package overlay mypkg

echo""
echo "----Resultant schema for 'default' table----"
echo "--------------------------------------------"
cat mypkg/databases/mydb/tables/default.yaml
uuid: null
kind: Package
apiVersion: pakx/v1
metadata:
  name: target
spec:
  databases:
      - name: mydb
        tables:
          schemas: 
            - name: default
              description: We've changed the default table's description too!
              type: basic 
              columns:
                - name: myfloat
                  type: float
                  description: we've added a new float column called myfloat 
                - name: x2 
                  type: symbol
                  description: The default x2 column has changed type
Found patch to overlay: mypkg/patches/my-db-patch.yaml
Writing /builds/kxdev/kxinsights/documentation/code-kx-com/mypkg/manifest.yaml

----Resultant schema for 'default' table----
--------------------------------------------
name: default
columns:
- description: we've added a new float column called myfloat
  name: myfloat
  type: float
- description: The default x2 column has changed type
  name: x2
  type: symbol
description: We've changed the default table's description too!
type: basic

Column ordering

After patching you can observe the columns have changed order. The new column has been added in at the top. If your application requires a specific column ordering then you will need to do this manually.

Referencing Deps patches

You are able to reference patches held in other packages too so that you don't need to continually copy patch specs that you often use.

Referenced Dep must already be installed

Referencing a patch from a dependency can only be done after that dep is installed. In the below example you explicitly run: kxi package install myotherdep. This allows the kxi package to resolve it properly when you then call overlay.

You do this in the following way:

kxi package -q init myotherpkg --force
# setup

kxi package -q add --to myotherpkg patch --name patch-from-afar
patchspec=$(cat myotherpkg/patches/patch-from-afar.yaml | grep spec: -B10 | cat - <(echo "  pipelines: [{name: mynewpipeline, spec: init.q}]"))
echo "$patchspec" > myotherpkg/patches/patch-from-afar.yaml
# add a patch to myother pkg

_=$(kxi package -q install myotherpkg)
# install myotherpkg so that you can reference it 

kxi package -q add --to mypkg dep --name myotherpkg --version 0.0.1 
# add a dependency on myotherpkg to mypkg

kxi package -q add --to mypkg patch --name patch-from-afar --dep myotherpkg
# add a patch in mypkg referencing a patch in myotherpkg

kxi package -q overlay mypkg

# see that you have successfully used the patch adding a new pipeline from myotherpkg
kxi package info mypkg | grep pipelines: -A5
pipelines:
  mynewpipeline:
    file: pipelines/mynewpipeline.yaml
router: ''
deployment_config: ''
patches: