Skip to content

Deploying quickstart

Deploying a package

By the end of this step, you should be able to...

  1. Add a pipeline to your package
  2. Deploy that package with pipeline
  3. Understand and configure analytics for the database

Looking at the deploy command you can see there is quite a lot of information, but a lot of it is to do with configuring your kdb Insights Enterprise connection.

The main SOURCE variable is simply a package to deploy (or a reference to a package).

kxi package deploy --help

Usage: kxi package deploy [OPTIONS] SOURCE

  Deploy a package to an insights instance

Options:
  --with-version / --without-version
                                  Include the package's version in the
                                  deployment name. If --with-version is set
                                  the deployment will have a name like
                                  pkgname-100. If --without-version is set the
                                  deployment will have a name like pkgname.
                                  Note this is enabled by default to avoid
                                  ambiguity  [default: without-version]
  --remote / --local              Deploy a package from a remote kdb Insights
                                  Enterprise package repo (--remote)[default]
                                  or using a local package (--local) [WARN:
                                  'local' may be deprecated in future
                                  releases]
  --via [operator|controller]     Specify the deployment method. Available
                                  options: operator, controller
  --rm-existing-data              Remove the data associated with the old
                                  deployment
  --force                         Teardown running deployments if they share
                                  any properties with the package
  --db TEXT                       Deploy an existing package's database (must
                                  be defined in the package)
  --pipeline TEXT                 Deploy an existing package's pipeline (must
                                  be defined in the package)
  --url TEXT                      Insights URL[env var: INSIGHTS_URL; default:
                                  https://replace.me]
  --realm TEXT                    Realm[env var: INSIGHTS_REALM; default:
                                  insights]
  --client-id TEXT                Client id[env var: INSIGHTS_CLIENT_ID;
                                  default: test-publisher]
  --client-secret TEXT            Client secret[env var:
                                  INSIGHTS_CLIENT_SECRET; default: special-
                                  secret]
  --auth-enabled / --auth-disabled
                                  Will attempt to retrieve bearer token for
                                  request  [env var: KXI_AUTH_ENABLED]
  --help                          Show this message and exit.

Deploying analytics: pipeline

This section shows you how to create a package and add a pipeline.

  1. Create your package:

    kxi package init mynewpkg --force 
    kxi package add --to mynewpkg pipeline --name mypipeline
    cat << EOF > mynewpkg/init.q
    .z.ts:{publish[([]10?10)]};
    system"t 1000";
    .qsp.run 
        .qsp.read.fromCallback[\`publish] 
        .qsp.write.toConsole[]
    EOF
    
    Creating package at location: mynewpkg
    Writing mynewpkg/manifest.yaml
    Writing mynewpkg/manifest.yaml
    

    Pipeline entrypoint

    By default the pipeline above will read init.q so you are modifying that module, but you can load from any file in the package. To change its entrypoint you can modify the spec field in pipelines/mypipeline.yaml in your package.

  2. Look at your package:

    kxi package info mynewpkg
    
    ==PATH==
    /builds/kxdev/kxinsights/documentation/code-kx-com/mynewpkg
    
    ==OBJECT==
    Package
    
    ==Manifest==
    name: mynewpkg
    version: 0.0.1
    metadata:
      description: ''
      authors:
      - name: root
        email: ''
    entrypoints:
      default: init.q
    pipelines:
      mypipeline:
        file: pipelines/mypipeline.yaml
    

    You can see it is referencing a new pipeline spec.

  3. In order to run the package, you need to first packit and push it:

    kxi package packit mynewpkg --tag
    kxi package push mynewpkg/0.0.1 --force
    
    Refreshing package before packing...
    Writing mynewpkg/manifest.yaml
    Creating package from mynewpkg
    Package created: /var/folders/6b/_lv5sfrj0l96g_1tc36bh_xr0000gn/T/artifact_store/mynewpkg-0.0.1.kxi
    mynewpkg-0.0.1.kxi
    {
        "mynewpkg": [
            {
                "version": "0.0.1",
                "_status": "InstallationStatus.SUCCESS"
            }
        ]
    }
    
  4. You are now ready to deploy the package:

    kxi package deploy mynewpkg/0.0.1
    
    Deploying: name=mynewpkg, ver=0.0.1
    {
        "packageName": "mynewpkg",
        "packageVersion": "0.0.1",
        "uuid": "1375378f-349f-49a6-b0b4-bd3a6b3a8298",
        "status": "Deployed",
        "pipelines": [
            "123c398d-39b6-0c34-4f98-9668bc64e262"
        ],
        "databases": [
            "61386e60-d05d-888b-7c40-2df184282a8a"
        ],
        "assemblies": [
            "b2b9569c-0bde-9d8b-7098-95ebfc7fdebe"
        ],
        "streams": [
            "42fc7ade-0f05-21e1-f9f3-086433554706",
            "54778e5d-3686-d29c-a6d5-e8a6bc4c9e4d"
        ],
        "schemas": [
            "1afca3bf-6516-deaf-cee2-286e863a9d11"
        ],
        "update_time": "2023-07-03T13:30:59.946457",
        "instance": "https://example.at.kx.com",
        "name": "mynewpkg-001-8298",
        "deploy_name": "mynewpkg-001",
        "error": {
            "content": ""
        }
    }
    

In this example you have created a package with some analytics and a pipeline. You have then deployed the pipeline and run the analytic inside the pipeline.

Where did the database come from ?

You may notice that you get a database, streams (I/O busses) and schemas deploying as part of your package which only contained a pipeline. This is due to a current limitationwith the deployment mechanism, which expects a database to exist. In future versions this dependency will be removed and the pipeline will come up alone.

Deploying analytics: DAP

This quickstart section shows you how to load an analytic into a database for querying.

Databases and their accompanying components (daps & agg) have a slightly different path for loading analytics during a deployment.

Whereas pipelines have a dedicated field for specifying their spec (or analytic), the data access and aggregation processes do not. Instead, you need to define special Entrypoints in the manifest that the data access and aggregation processes are hook onto. The src files themselves have no specific naming constraints.

Currently the entrypoints must be added manually as follows:

entrypoints:
    default: init.q
    data-access: src/da.q
    aggregator: src/agg.q  
"entrypoints": {
   "default": "init.q",
   "data-access": "src/da.q",
   "aggregator": "src/agg.q"
}

To add a database to a package take the following steps:

  1. Add a database to the package:

    kxi package add --to mynewpkg database --name mydb
    
    Writing mynewpkg/manifest.yaml
    

    mkdir mynewpkg/src
    echo "show \"hello\"" > mynewpkg/src/da.q 
    echo "show \"hello\"" > mynewpkg/src/agg.q
    
    mkdir: cannot create directory ‘mynewpkg/src’: File exists
    

  2. Ensure that the Database configuration in the package is told which package to load by adding the package name to the dap spec:

    kxi package add --to mynewpkg patch --name update_db_env
    cat << EOF > mynewpkg/patches/update_db_env.yaml
    kind: Package
    apiVersion: pakx/v1
    metadata:
    name: target
    spec:
    databases:
        - name: mydb
        shards:
            - name: mydb-shard
            daps:
                instances:
                rdb:
                    env: 
                        - name: KXI_PACKAGES
                        value: mynewpkg:0.0.1
                idb:
                    env: 
                        - name: KXI_PACKAGES
                        value: mynewpkg:0.0.1
                hdb:
                    env: 
                        - name: KXI_PACKAGES
                        value: mynewpkg:0.0.1
    EOF
    
    Writing mynewpkg/manifest.yaml
    

  3. Apply our patch to our local config of the database to ensure our environment flags are added:

    Found patch to overlay: mynewpkg/patches/update_db_env.yaml
    ParserError while parsing a block collection
      in "<unicode string>", line 7, column 5:
            - name: mydb
            ^
    expected <block end>, but found '?'
      in "<unicode string>", line 8, column 5:
            shards:
            ^
    

    What source-ry is this?

    We have used a patch to programatically update the database config.

  4. View the resultant yaml:

    
    

  5. Deploy the package and see the "hello" log message turn up in the DAPs

    Reading logs

    In order to read logs you can:

    • See them from the kdb Insights Enterprise UI
    • Use the REST endpoint for logs
    • Log onto your cluster and find the processes
    kxi deploy mynewpkg/0.0.1
    
    {
        "packageName": "mynewpkg",
        "packageVersion": "0.0.1",
        "uuid": "3375378f-349f-49a6-b0b4-bd3a6b3a8298",
        "status": "Deployed",
        "pipelines": [
            "323c398d-39b6-0c34-4f98-9668bc64e262"
        ],
        "databases": [
            "31386e60-d05d-888b-7c40-2df184282a8a"
        ],
        "assemblies": [
            "a2b9569c-0bde-9d8b-7098-95ebfc7fdebe"
        ],
        "streams": [
            "72fc7ade-0f05-21e1-f9f3-086433554706",
            "94778e5d-3686-d29c-a6d5-e8a6bc4c9e4d"
        ],
        "schemas": [
            "0afca3bf-6516-deaf-cee2-286e863a9d11"
        ],
        "update_time": "2023-07-03T13:35:59.946457",
        "instance": "https://example.at.kx.com",
        "name": "mynewpkg-001-8498",
        "deploy_name": "mynewpkg-001",
        "error": {
            "content": ""
        }
    }
    

Deploying analytics: Aggregator

Though getting analytics into the aggregator is done in a similar method to the above, the aggregator is not currently handled by the packaging framework and so this must be added manually during the kdb Insights Enterprise installation (or afterwards if you have access to the cluster using kubectl).

Next steps