Skip to content

Packaging quickstart

kxi package help

The kxi package command offers a wide range of tools to build, manage and even deploy packages to kdb Insights Enterprise. It has a detailed help documentation which has been collapsed below for brevity.

kxi package --help

Usage: kxi package [OPTIONS] COMMAND [ARGS]...

  KX Package Import/Export CLI.

Options:
  --artifact-store DIRECTORY    Directory in which to store packed
                                artifacts;[default] $KX_ARTIFACT_PATH.[env
                                var: KX_ARTIFACT_PATH]
  --pkg-lib DIRECTORY           Directory in which packages will be
                                installed;[default] $KX_PACKAGE_PATH.[env var:
                                KX_PACKAGE_PATH]
  --deployment-store DIRECTORY  Directory in which deployment data is
                                stored;[default] $KX_DEPLOYMENT_PATH.[env var:
                                KX_DEPLOYMENT_PATH]
  --version                     Version information.
  --debug                       Enable stacktrace print statements for easier
                                debugging.
  -v, --verbose                 Increase verbosity level
  -q, --quiet                   Disable logs. Only print errors.
  --help                        Show this message and exit.

Commands:

  When pulling/pushing to a kdb Insights Enterprise package repository, use these commands:

    pull           Download an artifact from the running insights package
                   manager service to...
    push           Publish an artifact to the running insights package manager
                   service.
    remote-list    List all installed packages or artifacts on the running
                   insights package...
    remote-remove  Remove packages or artifacts from the running insights
                   package manager...

  For local development, use these commands:

    convert    Converts Assembly specs to packages or changes package format.
    info       show some info about the package
    init       Creates a bare package at the specified target path.
    install    Install a package given a requirement or path.
    list       List all installed packages or artifacts.
    lock       Lock the q files within a source directory.
    overlay    Overlay a package using a spec file containing a subset of
               fields.
    packit     Create a package artifact given a source code directory.
    refresh    Refresh a package, picking up any available changes from disk
    uninstall  Uninstall specified locally installed packages and artifacts.
    unpack     Unpack an artifact to a specified location.
    validate   Validate the contents of a source directory to ensure it is a
               valid package.

  For managing deployments & runtime of packages on kdb Insights Enterprise, use these commands:

    deploy    Deploy a package to an insights instance
    teardown  Teardown a deployed packaged running on an insights instance

  For managing components added to package, use these commands:

    add    Add an entity to the specified package.
    copy   Copy an entity from the specified package.
    field  Find, list or mutate fields within a package.
    rm     Remove an entity from the specified package.

Create a package

Before continuing...

Make sure that you have:

Interactions with packages can be local or remote. In this section we will deal with creating a package on your local system.

By the end of this step, you should be able to...

  1. Initialize a package locally
  2. Update this package to include user-defined code
  3. Package this entity to generate an artifact
  4. Install this package to your local system
  1. Set your package path:

    This can be done in config to avoid this step

    export KX_PACKAGE_PATH=docs-packages/packages
    export KX_ARTIFACT_PATH=docs-packages/artifacts
    mkdir -p $KX_PACKAGE_PATH
    mkdir -p $KX_ARTIFACT_PATH
    

  2. Create a package:

    kxi package init qpackage
    
    Creating package at location: qpackage
    Writing qpackage/manifest.yaml
    

    Getting package information

    You can see some info about the package by calling the info argument as follows:

    kxi package info qpackage
    
    ==PATH==
    /builds/kxdev/kxinsights/documentation/code-kx-com/qpackage
    
    ==OBJECT==
    Package
    
    ==Manifest==
    name: qpackage
    version: 0.0.1
    metadata:
      description: ''
      authors:
      - name: root
        email: ''
    entrypoints:
      default: init.q
    

  3. Create a file src/myudf.q within the qpackage directory.

    This file contains a User-Defined Function (UDF).

    cd qpackage
    mkdir -p src/
    
    cat << EOF > src/myudf.q
    .test.variable:1b
    
    // @udf.name("sp_map")
    // @udf.tag("sp")
    // @udf.category("map")
    .test.sp.map:{[table;params]
    select from table where x<10
    }
    EOF
    cd ..
    

  4. Modify the init.q file which is your default entrypoint in order to ensure that your new module gets loaded by default.

    cd qpackage
    cat << EOF > init.q
    // Load the src/myudf.q file relative to package root
    .kxi.packages.file.load["src/myudf.q"]
    EOF
    cd ..
    

  5. Package the artifact ensuring you define the version of the package on the command line.

    What does packit mean in this context?

    Packing here creates an archive (zip) of the package directory and writes it to KX_ARTIFACT_PATH. The extension of the artifact is .kxi but it behaves just like a zip.

    kxi package convert qpackage --fmt json
    kxi package packit qpackage --version 1.0.0 --tag
    
    Refreshing package before packing...
    Writing qpackage/manifest.json
    Creating package from qpackage
    Package created: /tmp/artifact_store/qpackage-1.0.0.kxi
    qpackage-1.0.0.kxi
    

    Yaml vs Json

    You may have noticed the convert call above. This is necessary because most local systems don't have support for kdbyaml which does the yaml parsing in q. This will be changed in an upcoming release but for now, when running locally, it is easier to stick with json.

  6. List the current artifacts available locally.

    kxi package list --artifacts
    
    {
        "qpackage": [
            {
                "fields": {},
                "version": "1.0.0"
            }
        ]
    }
    

  7. Install the package locally such that it can be loaded into Python or q processes locally.

    What does installing mean in this context?

    Installing here simply means "unpacking" the .kxi object into the KX_PACKAGE_PATH, so that later when you load the code, it can be located

    kxi package install qpackage-1.0.0.kxi --force
    
    qpackage-1.0.0-local
    {
        "qpackage": [
            {
                "version": "1.0.0",
                "_status": "InstallationStatus.SUCCESS"
            }
        ]
    }
    

    Do I need to packit before install?

    You can do an install directly on the package, without calling packit, if the package is present in the current directory:

    kxi package install qpackage
    

    However, as there is no artifact you cannot push it to kdb Insights Enterprise or share it with colleagues.

Congratulations, you have developed, packed and installed a package onto your kdb Insights Enterprise instance.

Run the package code locally

By the end of this step, you should be able to...

  1. Install the kxi.packages library
  2. Run the code locally
  1. Install the kxi import library.

    pip install --extra-index-url https://$KXI_PYPI_USER:$KXI_PYPI_PASS@nexus.dl.kx.com/repository/kxi/simple/ kxi.packages
    
  2. To see your available packages you can simply run:

    import kxi.packages as kxi
    kxi.setup_env.set_package_path("docs-packages/packages")
    kxi.init()
    print(kxi.packages.list())
    
    name versions
    0  qpackage    1.0.0
    

    q).kxi.packages.list.all[]
    name           versions
    -----------------------
    "qpackage"    "1.0.0"
    
  3. In order to load and run any functions you have defined you can use the below:

    import pykx as kx
    kxi.packages.load("qpackage","1.0.0")
    print(kx.q(".test"))
    
            | ::
    variable| 1b
    sp      | ``map!(::;{[table;params]
    select from table where x<10
    })
    

    q).kxi.packages.load["qpackage";"1.0.0"]
    q).test
    variable| 1b
    sp      | ``map!(::;{[table;params]
    select from table where x<10
    })
    

kxi.packages.load

The kxi.packages.load function behaves like a "global" import, assigning all variables relative to the current namespace. In python it is similar to doing from module import *.

Congratulations you have run your package code! Next you will push it to_ kdb Insights Enterprise_ and interact with it there.

Push a package

Before continuing...

Make sure that you have:

By the end of this step, you should be able to...

  1. Push the code to kdb Insights Enterprise
  2. Run the code in kdb Insights Enterprise
  1. List the currently available packages on the system

    kxi package remote-list
    
    {}
    

    Currently there are no packages available.

  2. Push your new package up to kdb Insights Enterprise:

    kxi package push qpackage/1.0.0
    
    {
        "qpackage": [
            {
                "version": "1.0.0",
                "_status": "InstallationStatus.SUCCESS"
            }
        ]
    }
    

  3. Re-running the remote-list command, you can now see your new package:

    kxi package remote-list
    
    {
        "qpackage": [
            "1.0.0"
        ]
    }
    

Your package is on the system and the code is available for you to use.

Using code in kdb Insights Enterprise

If you log into your kdb Insights Enterprise instance you can now leverage your code in the UI in the following ways:

  • List and load the content of a package within a Query session.

    ScratchpadListLoad

  • Using an available UDF within an Pipeline developed within a Query session.

    ScratchpadSPUDF

  • Define a map node within a Pipeline to use a UDF.

    PipelineUDF

Removing packages

Once you are finished you can delete the package from the kdb Insights Enterprise instance.

```bash
kxi package remote-remove qpackage/1.0.0 --force
```
```json
{
    "qpackage": [
        {
            "_status": "DELETED",
            "version": "1.0.0"
        }
    ]
}
```

remote-remove

remote-remove by default removes the packages but not the Artifacts. In order to clear the artifacts as well run:

kxi package remote-remove --artifacts qpackage/1.0.0 --force
{
    "qpackage": [
        {
            "_status": "DELETED",
            "version": "1.0.0"
        }
    ]
}

You can also remove your locally stored packages and artifacts in a similar way:

kxi package uninstall qpackage/1.0.0 --force && kxi package uninstall --artifacts qpackage/1.0.0 --force
{}
{
    "qpackage": [
        {
            "_status": "DELETED",
            "version": "1.0.0"
        }
    ]
}

Next steps