Skip to content

Using a package

Using the code stored within your packages can be done both locally within your personal development environment or within various specified locations within kdb Insights Enterprise, this document aims to outline how both of these use-cases can be satisfied,

Using a package locally

Local usage of a package is facilitated through download and usage of the Python and `q APIs respectively. These APIs allow you to prototype the loading of your package's content using various entrypoints, additionally allowing you to develop user-defined functions and create unit tests to verify that the functions operate as expected.

The following examples outline how you can locally list and load available packages and UDFs in addition to developing a unit test for a specified UDF. This example uses the package generated and installed following the command line interface API quickstart guide here.

>>> import pykx as kx
>>> import kxi.packages as pakx
>>> pakx.init()

# List all installed packages
>>> pakx.packages.list()
       name versions
0  qpackage    1.0.0

# List all available UDFs
>>> pakx.udfs.list()
     name      function language                              file_path udf_sym description category tag   package version
0  sp_map  .test.sp.map        q  /tmp/packages/qpackage/1.0.0/src/sp.q     udf                [map]  sp  qpackage   1.0.0

# Load and test a UDF
>>> sp_udf = pakx.udfs.load('sp_map', 'qpackage', '1.0.0')
sp_udf(kx.q('([]100?100;100?1f)'), None)
pykx.Table(pykx.q('
x x1        
------------
6 0.785033  
9 0.08123546
8 0.6108817 
8 0.4976492 
1 0.09059026
0 0.6203014 
4 0.1627662 
2 0.9149882 
5 0.3869818 
7 0.2430836 
0 0.07347808
'))

# Load the full package using the default entrypoint
>>> pakx.packages.load('qpackage', '1.0.0')
>>> kx.q('.kxi.packages.loaded')
pykx.Table(pykx.q('
package    location                       version entry     init
----------------------------------------------------------------
""         ""                             ""      ""        0   
"qpackage" "/tmp/packages/qpackage/1.0.0" "1.0.0" "default" 1   
'))

The following example uses version 1.4.0 of the kxi_packages API

q)\l kxi_packages/1.4.0/init.q

// List all installed packages
q).kxi.packages.list.all[]
name       versions
-------------------
"qpackage" "1.0.0" 

// List all installed UDFs
name     function       language file_path                               udf_..
-----------------------------------------------------------------------------..
"sp_map" ".test.sp.map" ,"q"     "/tmp/packages/qpackage/1.0.0/src/sp.q" "udf..

// Load and test a UDF
q)spUDF:.kxi.udfs.load["sp_map";"qpackage";"1.0.0"]
q)spUDF[([]100?100;100?1f);::]
x x1        
------------
6 0.785033  
9 0.08123546
8 0.6108817 
8 0.4976492 
1 0.09059026
0 0.6203014 
4 0.1627662 
2 0.9149882 
5 0.3869818 
7 0.2430836 
0 0.07347808

// Load the full package using the default entrypoint
q).kxi.packages.load["qpackage";"1.0.0"]
q).kxi.packages.loaded
package    location                       version entry     init
----------------------------------------------------------------
""         ""                             ""      ""        0   
"qpackage" "/tmp/packages/qpackage/1.0.0" "1.0.0" "default" 1   

Using a package within kdb Insights Enterprise

Once you have uploaded a package to kdb Insights Enterprise following steps within the "Using a package" page or a package has been made available for you, depending on your use-case there are three locations where it can be utilized.

  1. Scratchpads
  2. Stream Processor
  3. Database

Each of these provide you with functionality for the use of packaged code, user defined functions and custom query APIs as discussed separately in the "Creating a package" page. The following sections outline how packages can be used in each case and provides links to more detailed documentation for each of the locations that are relevant.

Scratchpads

As discussed within the "Analyzing your data" section of the documentation packages and UDFs can be used within scratchpads through use of the Python and q APIs. Usage of these within the context of scratchpad is outlined in general respectively for Python and for q.

The following examples show use of the scratchpad in a Python and q setting when interacting with a package test_pkg generated using the code and techniques described in "Creating a package" and Uploading a package.

This package can be downloaded here should you wish to follow along.

The following provides a screenshot of the package Python functionality operating within the scratchpad

python package scratchpad

Code snippet for package usage within Python Scratchpad
import kxi.packages as pakx
import pykx as kx

# List available packages
pakx.packages.list()

# List available UDFs
pakx.udfs.list()

# Load a package and test available variable has been loaded
pakx.packages.load('test_pkg', '1.0.0')
kx.q('.test.variable')

# Retrieve a defined UDF and output it's definition
udf = pakx.udfs.load("custom_map", "test_pkg")
udf

# Generate some test data to use the udf
test_data = kx.q('([]100?1f;100?1f)')
udf(test_data, {'column': 'x1', 'threshold': 0.5})

The following provides a screenshot of the package q functionality operating within the scratchpad q package scratchpad

Code snippet for package usage within q Scratchpad
// List available packages
.kxi.packages.list.all[]

// List available UDFs
.kxi.udfs.list.all[]

// Load a package and test available variable has been loaded
.kxi.packages.load["test_pkg";"1.0.0"]
.test.variable

// Retrieve a defined UDF and output it's definition
udf:.kxi.udfs.load["custom_map";"test_pkg"]
udf

// Generate some test data to use the udf
testData:([]100?1f;100?1f)
udf[testData;`column`threshold!(`x1;0.5)]

Stream Processor

Within the Stream Processor you can utilize your UDFs and packages when deploying pipelines via the UI or within deployment assemblies using the kdb Insights CLI. These packages and UDFs can be called directly within the code node definitions of your pipelines in Python or q. The following shows, under various scenarios, usage of the UDF custom_map within the test_pkg available and used above.

In all cases your interactions with packages and UDFs are facilitated through the use of the Python and q APIs for package/UDF interaction. More specifically, for the deployment of UDFs through interaction with the Stream Processor's API for retrieval and wrapping of these functions for use with the Stream Processor which can be found for Python and q

UI driven development

In both cases below the Stream Processor pipeline being deployed is as follows

SP package example nodes

In each case the expression node is defined as follows

Expression Node

([]100?1f;100?1f)

Once complete the map node can be configured as follows for each case

Python SP node

SP Python Node Code
import kxi.sp as sp

udf = sp.udf('custom_map',
    'test_pkg',
    '1.0.0',
    {'column': 'x1',
     'threshold': 0.5})

def func(data):
    return udf(data)

q SP node

SP q Node Code
{[data]
    config:`column`threshold!(`x1;0.5);
    UDF: .qsp.udf[
        "custom_map";
        "test_pkg";
        .qsp.use enlist[`params]!enlist config];
    UDF data
    }

API driven development

In addition to deployments via the UI, you can deploy Stream Processor pipelines to kdb Insights Enterprise through definition within assembly configuration and deployment using the kdb Insights CLI as outlined here.

The definition of Stream Processor entries to these files is described fully here. The examples presented below show definition of both q and Python definitions of pipelines, each example includes any additional important configuration that's needed in addition to the pipeline specification as necessary.

base: py-ml
spec: |-
    import pykx as kx
    import kxi.sp as sp
    import kxi.packages as pakx

    pakx.packages.load('test_pkg', '1.0.0')

    udf = sp.udf(
            'custom_map',
            'test_pkg',
            '1.0.0',
            {'columns': 'x', 'threshold': 0.5})

    sp.run(
        sp.read.from_expr('([]100?1f;100?1f)') |
        sp.map(udf) |
        sp.write.to_console()
        )
base: q-ml
spec: |-
    .kxi.packages.load["test_pkg";"1.0.0"]
    udf: .qsp.udf[
            "custom_map";
            "test_pkg";
            .qsp.use `version`params!("1.0.0";`columns`threshold!("x";0.5))
            ];
    .qsp.run .qsp.read.fromExpr["([]100?1f;100?1f)"] .qsp.map[udf] .qsp.write.toConsole[]

Database

As outlined here use of packages within the database is facilitated through the definition and loading of specific entrypoints namely data-access and aggregator to allow code to be loaded in the Data Access Processes and Aggregator separately. Further information on the definition of custom query APIs and the addition of custom code to the various database components can be found here.

API driven development

The following shows an example configuration for the loading of custom code to both the aggregator and data access processes which can be included within your assembly configuration. In both cases we are loading the content of two packages fin and ml and in each case loading version 1.0.0 of the code.

This will load the aggregator entrypoint for the ml and fin packages. Put these values into the insights values.yaml you install with.

service-gateway:
    aggregator:
      env:
        KXI_PACKAGES: "ml:1.0.0,fin:1.0.0"

qe-gateway:
    aggregator:
      env:
        KXI_PACKAGES: "ml:1.0.0,fin:1.0.0"

This will load the data-access entrypoint for the ml and fin packages. Put these values into an assembly yaml.

dap:
  instances:
    rdb:
      env:
        - name: KXI_PACKAGES
          value: "ml:1.0.0,fin:1.0.0"
    idb:
      env:
        - name: KXI_PACKAGES
          value: "ml:1.0.0,fin:1.0.0"
    hdb:
      env:
      - name: KXI_PACKAGES
        value: "ml:1.0.0,fin:1.0.0"