Skip to content

Loading

Once saved to the ML Registry following the instructions outlined here, entities that have been persisted should be accessible to any user permissioned with access to the registry save location. The .ml.registry.get namespace provides all the callable functions used for the retrieval of objects from a registry. All functionality within this namespace is described below.

.ml.registry.get.model

Retrieve a model from an ML Registry

.ml.registry.get.model[folderPath;experimentName;modelName;version]

Parameters:

name type description
folderPath dictionary | string | :: A folder path indicating the location of the registry. Can be one of 3 options: a dictionary containing the vendor and location as a string, e.g. enlist[`local]!enlist"myReg"; a string indicating the local path; or a generic null to use the current .ml.registry.location pulled from CLI/JSON.
experimentName string | :: The name of an experiment from which to retrieve a model, if no modelName is provided the newest model within this experiment will be used. If neither modelName or experimentName are defined the newest model within the "unnamedExperiments" section is chosen. This may contain details of a subexperiment Eg. EXP1/SUBEXP1.
modelName string | :: The name of the model to be retrieved in the case this is null, the newest model associated with the experiment is retrieved.
version long[] | :: The specific version of a named model to retrieve, a list of length 2 with (major;minor) version number, in the case that this is null the newest model is retrieved.

Returns:

type description
dictionary The model and information related to the generation of the model.

When using .ml.registry.set.model users can include code files to be loaded on model retrieval, these files can be q,p,py or k extensions. On invocation of this function these files are loaded prior to model retrieval.

Examples:

Example 1: Get the latest version of 'model'

// Set a number of models within a new registry
q).ml.registry.set.model[::;::;{x};"model";"q";::]
q).ml.registry.set.model[::;::;{x+1};"model";"q";::]
q).ml.registry.set.model[::;::;{x+2};"model1";"q";::]

// Get the latest addition to the Registry
q).ml.registry.get.model[::;::;::;::]
modelInfo| `registry`model`monitoring!(`description`modelInformation`experime..
model    | {x+2}

// Get the latest version of 'model'
q).ml.registry.get.model[::;::;"model";::]
modelInfo| `registry`model`monitoring!(`description`modelInformation`experime..
model    | {x+1}

Example 2: Get version 1.0 of 'model'

q).ml.registry.get.model[::;::;"model";1 0]
modelInfo| `registry`model`monitoring!(`description`modelInformation`experime..
model    | {x}

.ml.registry.get.modelStore

Retrieve the modelStore table associated with an ML Registry

.ml.registry.get.modelStore[folderPath;config]

Parameters:

name type description
folderPath dictionary | string | :: A folder path indicating the location of the registry. Can be one of 3 options: a dictionary containing the vendor and location as a string, e.g. enlist[`local]!enlist"myReg"; a string indicating the local path; or a generic null to use the current .ml.registry.location pulled from CLI/JSON.
config :: Currently unused, must be passed as ::.

Returns:

type description
::

Examples:

Example 1: Retrieve the modelStore table

q).ml.registry.get.modelStore[::;::]
q)modelStore
registrationTime              experimentName modelName uniqueID              ..
-----------------------------------------------------------------------------..
2021.06.01D08:51:28.593730000 "undefined"    "mymodel" 7a214d0a-d9d2-890e-014..

.ml.registry.get.metric

Retrieve metric information associated with a model

.ml.registry.get.metric[folderPath;experimentName;modelName;version;param]

Parameters:

name type description
folderPath dictionary | string | :: A folder path indicating the location of the registry. Can be one of 3 options: a dictionary containing the vendor and location as a string, e.g. enlist[`local]!enlist"myReg"; a string indicating the local path; or a generic null to use the current .ml.registry.location pulled from CLI/JSON.
experimentName string | :: The name of an experiment from which to retrieve metrics associated with a model, if no modelName is provided the newest model within this experiment will be used. If neither modelName or experimentName are defined the newest model within the "unnamedExperiments" section is chosen. This may contain details of a subexperiment Eg. EXP1/SUBEXP1.
modelName string | :: The name of the model to retrieve metrics from. In the case this is null, the newest model associated with the experiment is retrieved.
version long[] | :: The specific version of a named model to retrieve metrics from, a list of length 2 with (major;minor) version number, in the case that this is null the newest model is retrieved.
param :: | dictionary | symbol | string Search parameters for the retrieval of metrics. In the case when this is a string, it is converted to a symbol.

Returns:

type description
table The metric table for a specific model, which may potentially be filtered.

Examples:

Example 1: Retrieve all metrics named metric1

// Log a number of metrics associated with a model
q).ml.registry.set.model[::;::;{x};"mymodel";"q";::]
q).ml.registry.log.metric[::;::;::;::;`metric1;2.0]
q).ml.registry.log.metric[::;::;::;::;`metric1;2.1]
q).ml.registry.log.metric[::;::;::;::;`metric2;1.0]
q).ml.registry.log.metric[::;::;::;::;`metric2;1.0]
q).ml.registry.log.metric[::;::;::;::;`metric3;3.0]

// Retrieve all metrics associated with the model
q).ml.registry.get.metric[::;::;::;::;::]
timestamp                     metricName metricValue
----------------------------------------------------
2021.06.01D09:51:35.638489000 metric1    2
2021.06.01D09:51:35.652863000 metric1    2.1
2021.06.01D09:51:35.666593000 metric2    1
2021.06.01D09:51:35.679152000 metric2    1
2021.06.01D09:51:35.694630000 metric3    3

// Retrieve all metrics named `metric1
q).ml.registry.get.metric[::;::;::;::;`metric1]
timestamp                     metricName metricValue
----------------------------------------------------
2021.06.01D09:51:35.638489000 metric1    2
2021.06.01D09:51:35.652863000 metric1    2.1

Example 2: Retrieve multiple metrics

q).ml.registry.get.metric[::;::;::;::;`metric2`metric3]
timestamp                     metricName metricValue
----------------------------------------------------
2021.06.01D09:51:35.666593000 metric2    1
2021.06.01D09:51:35.679152000 metric2    1
2021.06.01D09:51:35.694630000 metric3    3

Example 3: Equivalently this can be done using a dictionary input

q).ml.registry.get.metric[::;::;::;::;enlist[`metricName]!enlist `metric2`metric3]
timestamp                     metricName metricValue
----------------------------------------------------
2021.06.01D09:51:35.666593000 metric2    1
2021.06.01D09:51:35.679152000 metric2    1
2021.06.01D09:51:35.694630000 metric3    3

.ml.registry.get.parameters

Retrieve parameter information associated with a model

.ml.registry.get.parameters[folderPath;experimentName;modelName;version;paramName]

Parameters:

name type description
folderPath dictionary | string | :: A folder path indicating the location of the registry. Can be one of 3 options: a dictionary containing the vendor and location as a string, e.g. enlist[`local]!enlist"myReg"; a string indicating the local path; or a generic null to use the current .ml.registry.location pulled from CLI/JSON.
experimentName string | :: The name of an experiment from which to retrieve parameters associated with a model. If no modelName is provided the newest model within this experiment will be used. If neither modelName or experimentName are defined the newest model within the "unnamedExperiments" section is chosen. This may contain details of a subexperiment Eg. EXP1/SUBEXP1.
modelName string | :: The name of the model from which parameters are to be retrieved. In the case this is null, the newest model associated with the experiment is retrieved.
version long[] | :: The specific version of a named model to retrieve, a list of length 2 with (major;minor) version number, in the case that this is null the newest model is retrieved.
paramName symbol | string The name of the parameter to retrieve.

Returns:

type description
string | dictionary | table | float The value of the parameter associated with a named parameter saved for the model.

Examples:

Example 1: Retrieve set parameters

// Set a number of parameters associated with a model
q).ml.registry.set.parameters[::;::;"mymodel";1 0;"paramFile1";`param1`param2!1 2]
q).ml.registry.set.parameters[::;::;"mymodel";1 0;"paramFile2";("value1";"value2")]

// Retrieve the set parameters
q).ml.registry.get.parameters[::;::;::;::;`paramFile1]
param1| 1
param2| 2

q).ml.registry.get.parameters[::;::;::;::;`paramFile2]
"value1"
"value2"

.ml.registry.get.predict

Retrieve a model from the ML Registry wrapping in a common interface

.ml.registry.get.predict[folderPath;experimentName;modelName;version]

Parameters:

name type description
folderPath dictionary | string | :: A folder path indicating the location of the registry. Can be one of 3 options: a dictionary containing the vendor and location as a string, e.g. enlist[`local]!enlist"myReg"; a string indicating the local path; or a generic null to use the current .ml.registry.location pulled from CLI/JSON.
experimentName string | :: The name of an experiment from which to retrieve a model, if no modelName is provided the newest model within this experiment will be used. If neither modelName or experimentName are defined the newest model within the "unnamedExperiments" section is chosen. This may contain details of a subexperiment Eg. EXP1/SUBEXP1.
modelName string | :: The name of the model to be retrieved in the case this is null, the newest model associated with the experiment is retrieved.
version long[] | :: The specific version of a named model to retrieve, a list of length 2 with (major;minor) version number, in the case that this is null the newest model is retrieved.

Returns:

type description
function A wrapped version of the model providing a common callable interface for all models within the ML Registry. This model can accept vector/matrix/table/dictionary input and will return predictions generated by the model.

Models within the ML Registry can be of many forms q/Python/sklearn/keras etc. As such this function provides a common entry point to allow the models to be retrieved such that they are all callable using the same function call.

When using .ml.registry.set.model users can include code files to be loaded on model retrieval, these files can be q,p,py or k extensions. On invocation of this function these files are loaded prior to model retrieval.

Examples:

Example 1: Get the latest addition to the Registry

// Set a number of models within a new registry
q).ml.registry.set.model[::;::;{x};"model";"q";::]
q).ml.registry.set.model[::;::;{x+1};"model";"q";::]
q).ml.registry.set.model[::;::;{x+2};"model1";"q";::]

// Get the latest addition to the Registry
q).ml.registry.get.predict[::;::;::;::]
{x+2}{[data;bool]
  dataType:type data;
  if[dataType<=20;:data];
  data:$[98h=dat..

Example 2: Get the latest version of 'model'

q).ml.registry.get.predict[::;::;"model";::]
{x+1}{[data;bool]
  dataType:type data;
  if[dataType<=20;:data];
  data:$[98h=dat..

Example 3: Get version 1.0 of 'model'

q).ml.registry.get.predict[::;::;"model";1 0]
{x}{[data;bool]
  dataType:type data;
  if[dataType<=20;:data];
  data:$[98h=dat..

.ml.registry.get.update

Retrieve the update method for models within the ML Registry wrapping in a common interface

.ml.registry.get.update[folderPath;experimentName;modelName;version;supervised]

Parameters:

Name Type Description
folderPath dictionary | string | :: A folder path indicating the location of the registry. Can be one of 3 options: a dictionary containing the vendor and location as a string, e.g. enlist[`local]!enlist"myReg"; a string indicating the local path; or a generic null to use the current .ml.registry.location pulled from CLI/JSON.
experimentName string | :: The name of an experiment from which to retrieve a model, if no modelName is provided the newest model within this experiment will be used. If neither modelName or experimentName are defined the newest model within the "unnamedExperiments" section is chosen. This may contain details of a subexperiment Eg. EXP1/SUBEXP1.
modelName string | :: The name of the model to be retrieved in the case this is null, the newest model associated with the experiment is retrieved.
version long[] | :: The specific version of a named model to retrieve, a list of length 2 with (major;minor) version number, in the case that this is null the newest model is retrieved.
supervised boolean Is the model being retrieved a supervised (1b) or unsupervised (0b) model. This changes the number of expected inputs to the returned function.

Returns:

type description
function A wrapped version of the model providing a common callable interface for all models within the ML Registry. This model can accept vector/matrix/table/dictionary input and will return an updated version of the originally persisted model.

Models stored within the ML Registry can be of many forms q/sklearn/keras etc. Many of these formats can have an 'update' capability to allow these models to be updated as new data becomes available. As such this function provides a common entry point to allow the models update functionality to be retrieved in a common format.

In order to be retrieved from the registry the model must contain the following characteristics

Model Type Supported Requirements
q Yes Model originally saved to registry must contain an update key.
sklearn Yes Model originally saved to registry must support the partial_fit method.
keras No
Pytorch No

When using .ml.registry.set.model users can include code files to be loaded on model retrieval, these files can be q,p,py or k extensions. On invocation of this function these files are loaded prior to model retrieval.

Examples:

Example 1: Get the latest sklearn updatable model from the Registry

// Fit models to be persisted to the registry
q)X:100 2#200?1f
q)yReg:100?1f
q)yClass:100?0b
q)online1:.ml.online.clust.sequentialKMeans.fit[flip X;`e2dist;3;::;::]
q)online2:.ml.online.sgd.linearRegression.fit[X;yReg;1b;::]
q)sgdClass:.p.import[`sklearn.linear_model][`:SGDClassifier]
q)sgdModel:sgdClass[pykwargs `max_iter`tol!(1000;0.003)][`:fit] . (X;yClass)

// Set a number of models within a new registry
q).ml.registry.set.model[::;:::online1;"onlineCluster";"q";::]
q).ml.registry.set.model[::;::;online2;"onlineRegression";"q";::]
q).ml.registry.set.model[::;::;sgdModel;"SklearnSGD";"sklearn";::]

// Get the latest sklearn updatable model from the Registry
q).ml.registry.get.update[::;::;"SklearnSGD";::;1b]
.[{[f;x]embedPy[f;x]}[foreign]enlist]{(x y;z)}[locked[;0b]]

Example 2: Get a q updatable supervised model from the Registry

q).ml.registry.get.update[::;::;"onlineRegression";::;1b]
.[{[config;secure;features;target]
  modelInfo:config`modelInfo;
  theta:mode..{(x y;z)}[locked[;0b]]

Example 3: Get a q updatable unsupervised model from the Registry

q).ml.registry.get.update[::;::;"onlineCluster";::;0b]
{[returnInfo;data]
  modelInfo:returnInfo`modelInfo;
  inputs:modelInfo`input..locked[;0b]