Packaging quickstart

This section outlines how to create, upload, deploy, and manage a package containing a database and pipeline in kdb Insights Enterprise, including adding and using UDFs in the UI.

By the end of this quickstart section you will be able to:

1. Create a package locally that contains a pipeline and a database
2. Upload and deploy the package to kdb Insights Enterprise
3. Manage the package in the kdb Insights Enterprise UI

Create a Package

Create the package locally

To create a local package that contains a pipeline to inject 100 random floats between 0 and 100 and writes them to the database follow the steps below:

[Optional] Setup paths. This can be done via configuration instead. It's a non critical step.

export KX_PACKAGE_PATH=docs-packages/packages
export KX_ARTIFACT_PATH=docs-packages/artifacts
mkdir -p $KX_PACKAGE_PATH
mkdir -p $KX_ARTIFACT_PATH

1. Create a package that contains a database and a single table:

   export PKG=mypackage
   kxi package init ${PKG} --version 1.0.0
   kxi package add --to ${PKG} database --name mydb
   kxi package add --to ${PKG} table --name mytab
   

   Default table definition.

   By default when a table is added to a package it is defined as a partitioned table with 2 columns: t1 :timestamp and x1:float.

   This is all that is required for the quickstart, but if you wish to update the table you can edit the tables/mytab.yaml file.

2. Add query components:

   kxi package add --to $PKG router
   kxi package add --to $PKG agg
   kxi package add --to $PKG rc
   

3. Add a pipeline to inject 100 random floats between 0 and 100:

   kxi package add --to ${PKG} pipeline --name mypipeline
   cat << EOF > ${PKG}/src/mypipeline.q
   demodata: ([] t1: 100#2024.03.15D00:00:00; x1: 100?100.0)
   .qsp.run
           .qsp.read.fromExpr["demodata"]
           .qsp.write.toStream["mytab";"mypackage-mydb"]
   EOF
   

   Pipeline source file

   To change the source file of the pipeline you can modify the spec field in pipelines/mypipeline.yaml in your package.

4. Check your package contains all the expected components:

   kxi package info ${PKG}
   

   ==PATH==
   /builds/kxdev/documentation/insights/mypackage
   
   ==OBJECT==
   Package
   
   ==Manifest==
   uuid: 7966329b-963a-4681-8e1d-eb800934ad6e
   name: mypackage
   version: 1.0.0
   metadata:
     description: ''
     authors:
     - {}
   entrypoints:
     default: init.q
   tables:
     mytab:
       file: tables/mytab.yaml
   databases:
     mydb:
       dir: databases/mydb
       shards:
       - mydb-shard
       tables: []
   pipelines:
     mypipeline:
       file: pipelines/mypipeline.yaml
   router: router/router.yaml
   

Upload and deploy the package

[Optional] List the currently available packages on kdb Insights Enterprise

kxi pm list

Currently there are no packages available.

1. Upload the package to kdb Insights Enterprise:

   kxi pm push ${PKG}
   
   Push with Package(name=mypackage, version=1.0.0) and force=False           
   ╭───────────┬─────────┬────────────┬──────────────────────────────────────╮
   │ name      │ version │ owner.name │ id                                   │
   ├───────────┼─────────┼────────────┼──────────────────────────────────────┤
   │ mypackage │ 1.0.0   │ purple     │ 89e80c01-c660-4436-aae8-74f0276e4568 │
   ╰───────────┴─────────┴────────────┴──────────────────────────────────────╯
   

2. Re-run the kxi pm list command. Your new package now appears in this list:

   kxi pm list
   
   List with type=package                                                              
   ╭───────────┬─────────┬────────┬────────────┬──────────────────────────────────────╮
   │ name      │ version │ access │ owner.name │ id                                   │
   ├───────────┼─────────┼────────┼────────────┼──────────────────────────────────────┤
   │ mypackage │ 1.0.0   │ ARWX   │ purple     │ 89e80c01-c660-4436-aae8-74f0276e4568 │
   ╰───────────┴─────────┴────────┴────────────┴──────────────────────────────────────╯ 
   

   At this stage, your package is uploaded to kdb Insights Enterprise and the database and pipeline will be visible in the UI.

3. Deploy the package to ingest the data into the database:

   kxi pm deploy ${PKG} 
   
   Deploy with pkg_name='mypackage', version='1.0.0', env={}                         
   ╭───────────┬─────────┬───────────────────┬──────────────────────────────────────╮
   │ name      │ version │ deployment.status │ id                                   │
   ├───────────┼─────────┼───────────────────┼──────────────────────────────────────┤
   │ mypackage │ 1.0.0   │ UNKNOWN           │ 89e80c01-c660-4436-aae8-74f0276e4568 │
   ╰───────────┴─────────┴───────────────────┴──────────────────────────────────────╯
   

4. Later you can check the status:

   Package running OKPackage has errors

   kxi pm list 
   
   List with type=package                                                                                         
   ╭───────────┬─────────┬───────────────────┬──────┬────────┬────────────┬──────────────────────────────────────╮
   │ name      │ version │ deployment.status │ data │ access │ owner.name │ id                                   │
   ├───────────┼─────────┼───────────────────┼──────┼────────┼────────────┼──────────────────────────────────────┤
   │ mypackage │ 1.0.0   │  RUNNING          │ yes  │ ARWX   │ purple     │ 89e80c01-c660-4436-aae8-74f0276e4568 │
   ╰───────────┴─────────┴───────────────────┴──────┴────────┴────────────┴──────────────────────────────────────╯
   

   kxi pm list
   
   List with type=package                                                                                         
   ╭───────────┬─────────┬───────────────────┬──────┬────────┬────────────┬──────────────────────────────────────╮
   │ name      │ version │ deployment.status │ data │ access │ owner.name │ id                                   │
   ├───────────┼─────────┼───────────────────┼──────┼────────┼────────────┼──────────────────────────────────────┤
   │ mypackage │ 1.0.0   │ Deployment Error  │ yes  │ ARWX   │ purple     │ 89e80c01-c660-4436-aae8-74f0276e4568 │
   ╰───────────┴─────────┴───────────────────┴──────┴────────┴────────────┴──────────────────────────────────────╯
   

5. Open the web interface and view the package in the Packages list. The package will be listed and shown in a RUNNING state.

Manage the package in the UI

If you log into your kdb Insights Enterprise instance you can:

• Find mypackage and mypipeline displayed in the left-hand menu as shown below.

  

• View the database details, make modifications and redeploy via the UI.

  

• View the pipeline details, make modifications to the pipeline settings and redeploy via the UI.

  

  Managing pipelines that are defined using q or Python via the CLI.

  Currently if a pipeline specification is defined via the CLI using q or Python code, rather than using the Pipeline Editor in the UI, you cannot edit the code in the UI only the pipeline settings."

• Query the data in the database using the Query Window.

  

Remove a package

To remove the package created in this quickstart from kdb Insights Enterprise, run the following command:

    kxi pm rm --teardown mypackage

    Teardown with deployment_id='mypackage', cleanup=True                  
    ╭───────────┬─────────┬────────┬──────────────────────────────────────╮
    │ name      │ version │ access │ id                                   │
    ├───────────┼─────────┼────────┼──────────────────────────────────────┤
    │ mypackage │ 1.0.0   │ ARWX   │ 89e80c01-c660-4436-aae8-74f0276e4568 │
    ╰───────────┴─────────┴────────┴──────────────────────────────────────╯


    Rm with 'mypackage' and version=None                                       
    ╭───────────┬─────────┬────────────┬──────────────────────────────────────╮
    │ name      │ version │ owner.name │ id                                   │
    ├───────────┼─────────┼────────────┼──────────────────────────────────────┤
    │ mypackage │ 1.0.0   │ purple     │ 89e80c01-c660-4436-aae8-74f0276e4568 │
    ╰───────────┴─────────┴────────────┴──────────────────────────────────────╯

You can also remove your locally stored packages and .kxi files in a similar way

kxi package uninstall $PKG/1.0.0 --force && kxi package uninstall --artifacts $PKG/1.0.0 --force

{}
{}

Next Steps

• Creating a UDA: details on how to create a UDA, add it to a package, and deploy it to kdb Insights.
• UDF Example: details on how to upload and interact with a UDF within kdb Insights.