Storing

The ML Registry allows users to persist a variety of versioned entities to disk and cloud storage applications. The ML Registry provides this persistence functionality across a number of Python submodules, namely, kxi.ml.registry.[new/set/log]. All supported functionality within these submodules is described below.

`kxi.ml.registry.new.registry`

Create a new registry.

Parameters:

Name	Type	Description	Default
`folder_path`	`Union[str, dict]`	Either a string denoting the folder path location to create the registry, or a dictionary specifying the vendor (as key) and the path (as value), ex: dict(aws='s3://kx-ml-registry-bucket') or None to default to local current working directory	`None`
`config`	`dict`	Either a registry configuration as dictionary, or None	`None`

Returns:

Type	Description
`dict`	Registry configuration as a dictionary

Examples:

Create a registry in various locations depending on argument type.

Default arguments will create a registry in the current working directory:

>>> from kxi import ml
>>> ml.init()
>>> ml.registry.new.registry()
{'storage': 'local',
'folderPath': b'.',
'registryPath': b'./KX_ML_REGISTRY',
'modelStorePath': ':./KX_ML_REGISTRY/modelStore'}

Passing the folder path as a string will create a registry in the specified local folder:

>>> ml.registry.new.registry(folder_path="/tmp")
{'storage': 'local',
'folderPath': b'/tmp',
'registryPath': b'/tmp/KX_ML_REGISTRY',
'modelStorePath': ':/tmp/KX_ML_REGISTRY/modelStore'}

Calling with a dictionary will create a registry within the cloud:

>>> ml.registry.new.registry(folder_path=dict(aws="s3://my-bucket/"))

`kxi.ml.registry.new.experiment`

Create a new experiment within the specified registry.

Parameters:

Name	Type	Description	Default
`experiment_name`	`str`	The name of the experiment as a string	required
`folder_path`	`Union[str, dict]`	Either a string denoting the folder path location to create the registry, or a dictionary specifying the vendor (as key) and the path (as value), ex: dict(aws='s3://kx-ml-registry-bucket') or None to default to local current working directory	`None`
`config`	`dict`	Either an experiment configuration as dictionary, or None	`None`

Returns:

Type	Description
`dict`	Experiment configuration as a dictionary

Examples:

Create an experiment named "day0" in the registry located under /tmp:

>>> from kxi import ml
>>> ml.init()
>>> ml.registry.new.experiment(experiment_name="day0", folder_path="/tmp")

`kxi.ml.registry.log.metric`

Associate a timestamp and named metric value with a model stored in the Registry.

Parameters:

Name	Type	Description	Default
`metric_name`		Name of the metric to be associated with a model as an 'str'	required
`metric_value`	`Any`	Value of the metric to be stored with the model	required
`folder_path`	`Union[str, dict]`	Either a string denoting the folder path where the registry exists, or a dictionary specifying the vendor (as key) and the path (as value), ex: {'aws':'s3://kx-ml-registry-bucket'} or None to default to local current working directory	`None`
`experiment_name`	`str`	The name of the experiment the model is associated with as a string	`None`
`model_name`	`str`	The name of the model to which the parameter is to be associated, if None the latest model added to the registry will be used	`None`
`version`	`list`	A list of the major and minor versions of the model - [major, minor]. If None, the latest version of the model associated with model_name is used.	`None`

Examples:

Log a validation metric along with the corresponding model:

>>> from kxi import ml
>>> ml.init()
>>> from sklearn.linear_model import LinearRegression
>>> from sklearn.metrics import mean_squared_error
>>> import numpy as np
>>> regressor = LinearRegression().fit(np.random.randn(10, 1), np.random.rand(10))
>>> ml.registry.set.model(model=regressor,
                          model_name="linear_regression",
                          model_type="sklearn",
                          folder_path="/tmp",
                          experiment_name="day0")
>>> X = np.random.randn(5, 1)
>>> y_true = np.random.rand(5)
>>> y_pred = regressor.predict(X)
>>> mse = mean_squared_error(y_true, y_pred)
>>> ml.registry.log.metric(metric_name="mse",
                           metric_value=mse,
                           folder_path="/tmp",
                           experiment_name="day0",
                           model_name="linear_regression")

`kxi.ml.registry.set.model`

Persist a model to the Registry.

Parameters:

Name	Type	Description	Default
`model`		The model to be saved to the Registry, this can be one of: 1. A Python function which can be saved using 'joblib.dump' 2. A fit scikit-learn model which contains a 'predict' method 3. A Keras model containing a 'predict' method 4. A PyTorch model saved which can be saved using 'torch.save' 5. A Theano model which can be saved using 'joblib.dump', users must additionally include a path to the code needed for running this function via the 'code' key in the 'opts' argument 6. A q model 7. A path to any of the above models saved to disk	required
`model_name`	`str`	Name to be associated with the model when saved as a 'str'.	required
`model_type`	`str`	The type of model that is being saved, namely 'q', 'sklearn', 'keras', 'python', 'torch' or 'theano' as an 'str'.	required
`folder_path`	`Union[str, dict]`	Either a string denoting the folder path where the registry exists, or a dictionary specifying the vendor (as key) and the path (as value), ex: {'aws':'s3://kx-ml-registry-bucket'} or None to default to local current working directory.	`None`
`experiment_name`	`str`	The name of the experiment the model is associated with as a string.	`None`
`data`	`dict`	User supplied reference data for the parsing of statistical information about the data to be used for monitoring purposes 'kx.Table', 'np.records', 'pd.DataFrame' or 'pa.Table' objects supported	`None`
`requirements`	`Union[bool, str, List[str]]`	Are Python requirements to be associated with a model, there are 2 options 1. bool - Whether to run 'pip freeze' against your current environment 2. str - Specifying the path to a 'requirements.txt' file by default no requirements are saved with the model 3. List[str] - A list of package names we want to link to the model	`None`
`major`	`bool`	Boolean indicating if the addition of this model a 'major' update i.e. 1.1 -> 2.0 rather than 1.1 -> 1.2	`None`
`major_version`	`int`	What major version of the model is to be incremented? By default the function will increment based on the maximal version in the registry, users can overwrite this to increment a minor version of a previous model.	`None`
`code`	`str`	Reference to the location of any files '.py', '.p', '.k' or '.q' required by the model, these will be loaded prior to loading the model on retrieval as such this should be used if the model being retrieved has prerequisites.	`None`
`axis`	`bool`	Boolean indicating if data should be passed into the model - 'horizontally', column per feature indicated by 'True' - 'vertically', row per feature indicated by 'False'	`None`

Returns:

Type	Description
`UUID`	UUID indicating the unique model added to the Registry

Examples:

Save a trained model to a local registry:

>>> from kxi import ml
>>> ml.init()
>>> from sklearn.linear_model import LinearRegression
>>> import numpy as np
>>> regressor = LinearRegression().fit(np.random.randn(10, 1), np.random.rand(10))
>>> ml.registry.set.model(model=regressor,
                          model_name="linear_regression",
                          model_type="sklearn",
                          folder_path="/tmp",
                          experiment_name="day0")
UUID('1fd69913-2ea8-6436-2f5d-129ae1261070')

`kxi.ml.registry.set.parameters`

Associate parameters with a model stored in the Registry.

Parameters:

Name	Type	Description	Default
`param_name`	`str`	Name of the parameter to be associated with a model as a string	required
`params`	`str`	Value of the parameter to be stored as a json file	required
`folder_path`	`Union[str, dict]`	Either a string denoting the folder path where the registry exists, or a dictionary specifying the vendor (as key) and the path (as value), ex: {'aws':'s3://kx-ml-registry-bucket'} or None to default to local current working directory	`None`
`experiment_name`	`str`	The name of the experiment the model is associated with as a string	`None`
`model_name`	`str`	The name of the model to which the parameter is to be associated, if None the latest model added to the registry will be used	`None`
`version`	`list`	A list of the major and minor versions of the model - [major, minor]. If None, the latest version of the model associated with model_name is used.	`None`

Returns:

Type	Description
`str`	A string containing the path of the created parameter file.

Examples:

Save hyperparameters used during training along with the corresponding model:

>>> from kxi import ml
>>> ml.init()
>>> from sklearn.linear_model import QuantileRegressor
>>> import numpy as np
>>> alpha = 0.0
>>> regressor = QuantileRegressor(alpha=alpha).fit(np.random.randn(10, 1),
                                                   np.random.rand(10))
>>> ml.registry.set.model(model=regressor,
                          model_name="quantile_regression",
                          model_type="sklearn",
                          folder_path="/tmp",
                          experiment_name="day0")
>>> ml.registry.set.parameters(param_name="alpha",
                               params=alpha,
                               folder_path="/tmp",
                               experiment_name="day0",
                               model_name="quantile_regression")
':/tmp/KX_ML_REGISTRY/namedExperiments/day0/quantile_regression/1.0/params/alpha.json'