Skip to content

Quickstart

Use the ML-Registry as a standalone entity or integrate it with the Stream Processor

The quickstart introduces the use of this component in real-world applications. It is not fully descriptive: explore the API documentation for each section to get the most out of it.

Two workflows with the kxi-ml-registry package:

  1. Within a user-generated qpk Docker image, use the registry
    • as a standalone storage entity
    • in conjunction with the Stream Processor
  2. Run it standalone on premises

Download kxi-ml-registry.qpk

Package as a dependency

Set the environment variables required for generation of qpk Docker image.

kxi-ml-registry requires the base image centos:8

export QPDOCKER_BASE=centos:8

Write requirements.txt in the root of your application.

numpy
scipy
scikit-learn<=0.23.0
statsmodels
matplotlib
sobol-seq
pandas>=1.0

Generate a qp.json to make the package a dependency of your application. It must contain both

  • the kxi-ml-registry package
  • py to pick up the necessary Python requirements as dependencies
├── init.q
├── qp.json
├── requirements.txt
└── deps/
    └── kxi-ml-registry.qpk

qp.json:

{
  "myapp" : {
    "name"    : "my-application",
    "depends" : [ "py", "deps/kxi-ml-registry" ],
    "entry"   : [ "init.q" ]
  }
}

Create the Docker image for your application:

qp build myapp

Run your application using either QPacker or Docker. It can be used to interact either with

  • an ML Registry as a standalone entity
  • a Stream Processor

QPacker

$ qp run myapp
q).ml.registry.new
          | ::
registry  | locked[`modelName`version`vendor`bins`deployType`code!(`;::;`;10f;0b;`)]
experiment| locked[`modelName`version`vendor`bins`deployType`code!(`;::;`;10f;0b;`)]

QPacker does not yet support integration with Stream Processor.

Docker

Integrating with the Stream Processor: the example.q script located in the examples/ folder in the QPK download.

N:1000
data:([]x:asc N?1f;x1:desc N?10;x2:N?1f;y:asc N?5)
features:flip value flip delete y from data
clf:.p.import[`sklearn.tree]`:DecisionTreeClassifier
clf:clf[`max_depth pykw 3]
clf[`:fit][features;data`y]
// Save the model
.ml.registry.set.model["/tmp";clf;"dtc"; "sklearn";::]
.test.cache:()

.qsp.run
  .qsp.read.fromCallback[`publish]
  .qsp.ml.predict[{delete y from x};`yhat;.qsp.use `registry`model!("/tmp";"dtc")]
  .qsp.map[{.test.cache,: x}]

Run the Docker image without exposing Stream Processor functionality:

$ source qpbuild/.env
$ docker run -it -e "KDB_LICENSE_B64=$KDB_LICENSE_B64" $myapp
q).ml.registry.new
          | ::
registry  | locked[`modelName`version`vendor`bins`deployType`code!(`;::;`;10f;0b;`)]
experiment| locked[`modelName`version`vendor`bins`deployType`code!(`;::;`;10f;0b;`)]

Run the Docker image explicitly pointing to a Stream Processor spec:

$ docker run -it -p 5000:5000 \
  -v "$(pwd)"/examples:/app \
  -e "KDB_LICENSE_B64=$KDB_LICENSE_B64" \
  -e KXI_SP_SPEC="/app/example.q" \
  $myapp -p 5000
q)\p
5000i
q)@[publish;([]100?1f;100?1f;100?1f);{::}]

Standalone/on-prem

You can consume ML Registry as a standalone library without Docker, but integration with the Stream Processor is not then supported.

All QPKs are ZIPs and can be unpacked

unzip kxi-ml-registry.qpk

Check the requirements are installed. To use this application on-prem you need embedPy installed locally and the Python requirements:

pip install -r kxi-ml-registry/requirements.txt

Initialize the library from the unzipped directory using the startq.q script:

$ cd kxi-ml-registry
$ q startq.q
q).ml.registry.new
      | ::
registry  | locked[`modelName`version`vendor`bins`deployType`code!(`;::;`;10f;0b;`)]
experiment| locked[`modelName`version`vendor`bins`deployType`code!(`;::;`;10f;0b;`)]

Define and use a loading function in an initialization script.

├── init.q
└── deps/
    ├── kxi-ml-registry/
    └── kxi-ml-registry.qpk

init.q:

// Define the location from which your dependecies are to be loaded
PACKAGE_PATH:{[path]$[count path;path;"deps"]}[getenv`PACKAGE_PATH]
// Define the loading function
loadfunc:{[pkg]
  pwd:system"cd";
  system"cd ",PACKAGE_PATH;
  if[not(`$pkg)in key`:.;
    system"cd ",pwd;
    '"unable to locate package: ",pkg
    ];
  system"cd ",pkg;
  err:@[{system"l ",x;::};"startq.q";::];
  system "cd ",pwd;
  if[10h=type err;'"Failed to load package: ",err] }
// Load the unzipped qpk
@[loadfunc;"kxi-ml-registry";{'"Failed to load package with error: ",x}]
$ q init.q
q).ml.registry.new
          | ::
    registry  | locked[`modelName`version`vendor`bins`deployType`code!(`;::;`;10f;0b;`)]
    experiment| locked[`modelName`version`vendor`bins`deployType`code!(`;::;`;10f;0b;`)]