Clustering

The following outlines the variadic function definitions provided with the kdb Insights ML Analytics library for Clustering. Full breakdowns of the algorithms represented can be found here, this includes, via examples, the use of the function returns for prediction/update, this is not outlined below explicitly.

Note

All arguments marked with an asterisk are optional and can be input using the notation defined in the function calls section of the model monitoring documentation.

K-Means

.ml.kxi.clust.kmeans.fit

Fit a K Means model

.ml.kxi.clust.kmeans.fit[X]

Parameters:

name	type	description
X	any	Input/training data of N dimensions.

options:

name	type	description	default
df	symbol	Distance function used in clustering.	edist
k	long	The number of clusters.	8
centers	dictionary|null	Initial cluster centers. If null, initial centers are calculated using k++/random initialisation. If dictionary, must contain num and centroids which define the number of points in a cluster and the cluster location often calculated from a previous 'fit' phase.	::
config	dictionary	Any additional configuration required for application of clustering, supported options defined here.	::

Returns:

type	description
dictionary	All information collected during the fitting of a model, along with prediction and update functionality.

Examples:

Example 1: Fit a model in default configuration using only required arguments

// Generate feature data
q)data:([]100?1f;100?1f;100?1f)

// Fit model
q)show mdl1:.ml.kxi.clust.kmeans.fit data
modelInfo| `repPts`clust`data`inputs!((0.8139576 0.09132079 0.2219031;0.67699..
predict  | {[config;data]
  config:config[`modelInfo];
  data:clust.util.floatCo..
update   | {[config;data]
  modelConfig:config[`modelInfo];
  data:clust.util.fl..
q)mdl1[`modelInfo;`inputs]
df  | `e2dist
k   | 8
iter| 100
kpp | 1b

Example 2: Fit model modifying the default behavior using additional arguments

// Generate feature data
q)data:([]100?1f;100?1f;100?1f)

// Fit model
q)show mdl2:.ml.kxi.clust.kmeans.fit[data;.var.kwargs`df`k!(`edist;3)]
modelInfo| `repPts`clust`data`inputs!((0.8148896 0.3256995 0.5313307;0.236372..
predict  | {[config;data]
  config:config[`modelInfo];
  data:clust.util.floatCo..
update   | {[config;data]
  modelConfig:config[`modelInfo];
  data:clust.util.fl..
q)mdl2[`modelInfo;`inputs]
df  | `edist
k   | 3
iter| 100
kpp | 1b

Affinity Propagation

.ml.kxi.clust.ap.fit

Fit a Affinity Propagation model

.ml.kxi.clust.ap.fit[X]

Parameters:

name	type	description
X	any	Input/training data of N dimensions.

options:

name	type	description	default
df	symbol	Distance function used in clustering.	nege2dist
damp	float	Damping coefficient.	0.5
diag	function	Preference function for the diagonal of the similarity matrix.	med
iter	dictionary	Max allowed iterations and the max iterations without a change in clusters. When null is passed in `total`noChange!200 50 are used.	::

Returns:

type	description
dictionary	All information collected during the fitting of a model, along with prediction functionality.

Examples:

Example 1:

// Generate feature data
q)data:([]100?1f;100?1f;100?1f)

// Fit a model in default configuration using only required arguments
q)show mdl1:.ml.kxi.clust.ap.fit data
modelInfo| `data`inputs`clust`exemplars!((0.8599461 0.2452222 0.6070236 0.686..
predict  | {[config;data]
  config:config`modelInfo;
  data:clust.util.floatConv..
q)mdl1[`modelInfo;`inputs]
df  | `nege2dist
damp| 0.5
diag| k){avg x(<x)@_.5*-1 0+#x,:()}
iter| `run`total`noChange!0 200 15

Example 2:

// Generate feature data
q)data:([]100?1f;100?1f;100?1f)

// Fit a model modifying the default behavior using a mix of positional and keyword arguments
q)damp:.75
q)show mdl2:.ml.kxi.clust.ap.fit[data;damp;.var.kw[`diag;max]]
modelInfo| `data`inputs`clust`exemplars!((0.8599461 0.2452222 0.6070236 0.686..
predict  | {[config;data]
  config:config`modelInfo;
  data:clust.util.floatConv..
q)mdl2[`modelInfo;`inputs]
df  | `nege2dist
damp| 0.75
diag| max
iter| `run`total`noChange!0 200 15

DBSCAN

.ml.kxi.clust.dbscan.fit

Fit a DBSCAN model

.ml.kxi.clust.dbscan.fit[X]

Parameters:

name	type	description
X	any	Input/training data of N dimensions.

options:

name	type	description	default
df	symbol	Distance function used in clustering.	e2dist
minPts	long	Minimum number of points required in a given neighborhood to define a cluster.	5
eps	float	Epsilon radius.	0.5

Returns:

type	description
dictionary	All information collected during the fitting of a model, along with prediction and update functionality.

Examples:

Example 1:

// Generate feature data
q)data:([]100?1f;100?1f;100?1f)

// Fit a model in default configuration using only required arguments
q)show mdl1:.ml.kxi.clust.dbscan.fit data
modelInfo| `data`inputs`clust`tab!((0.8599461 0.2452222 0.6070236 0.6868635 0..
predict  | {[config;data]
  config:config[`modelInfo];
  data:clust.util.floatCo..
update   | {[config;data]
  modelConfig:config[`modelInfo];
  data:clust.util.fl..
q)mdl1[`modelInfo;`inputs]
df    | `e2dist
minPts| 5
eps   | 0.5

Example 2:

// Generate feature data
q)data:([]100?1f;100?1f;100?1f)

// Fit a model modifying the default behavior using positional arguments
q)df:`edist
q)eps:.75
q)show mdl2:.ml.kxi.clust.dbscan.fit[data;df;eps]
modelInfo| `data`inputs`clust`tab!((0.8599461 0.2452222 0.6070236 0.6868635 0..
predict  | {[config;data]
  config:config[`modelInfo];
  data:clust.util.floatCo..
update   | {[config;data]
  modelConfig:config[`modelInfo];
  data:clust.util.fl..
q)mdl2[`modelInfo;`inputs]
df    | `edist
minPts| 5
eps   | 0.75

CURE

.ml.kxi.clust.cure.fit

Fit a CURE model

.ml.kxi.clust.cure.fit[X]

Parameters:

name	type	description
X	any	Input/training data of N dimensions.

options:

name	type	description	default
df	symbol	Distance function used in clustering.	e2dist
n	long	Number of representative points.	5
c	float	Compression ratio.	0

Returns:

type	description
dictionary	All information collected during the fitting of a model, along with prediction functionality.

Examples:

Example 1:

// Generate feature data
q)data:([]100?1f;100?1f;100?1f)

// Fit a model in default configuration using only required arguments
q)show mdl1:.ml.kxi.clust.cure.fit data
modelInfo| `data`inputs`dgram!((0.8599461 0.2452222 0.6070236 0.6868635 0.837..
predict  | {[config;data;cutDict]
  data:clust.util.floatConversion util.tabConvert..
q)mdl1[`modelInfo;`inputs]
df| `e2dist
n | 5
c | 0

Example 2:

// Generate feature data
q)data:([]100?1f;100?1f;100?1f)

// Fit a model modifying the default behavior using additional arguments
q)show mdl2:.ml.kxi.clust.cure.fit[data;.var.kwargs`n`c!(4;.1)]
modelInfo| `data`inputs`dgram!((0.8599461 0.2452222 0.6070236 0.6868635 0.837..
predict  | {[config;data;cutDict]
  data:clust.util.floatConversion util.tabConvert..
q)mdl2[`modelInfo;`inputs]
df| `e2dist
n | 4
c | 0.1

.ml.kxi.clust.cure.fitPredict

Fit and predict on CURE model

.ml.kxi.clust.cure.fitPredict[X]

Parameters:

name	type	description
X	any	Input/training data of N dimensions.

options:

name	type	description	default
df	symbol	Distance function used in clustering.	e2dist
n	long	Number of representative points.	5
c	float	Compression ratio.	0
cutDict	dictionary	Cutting algo to use when splitting the data into clusters (`k/`dist) and a value defining the cutting threshold.	enlist[`k]!enlist 5

Returns:

type	description
dictionary	All information collected during the fitting of a model, along with predicted clusters and prediction functionality.

Examples:

Example 1:

// Generate feature data
q)show data:2 10#20?10.
1.473702 4.080537 3.03448  9.659883  7.874197 4.734442 8.423141 2.7..
0.72077  5.450964 4.625792 0.6486378 6.951865 9.674697 7.26315  2.4..

// Fit a CURE model and cut the dendrogram into 3 clusters
// Use a mix of positional and keyword arguments
q).ml.kxi.clust.cure.fitPredict[data;.var.kw[`df;`edist];.var.kw[`cutDict;enlist[`k]!enlist 3]]
modelInfo| `data`inputs`dgram!((1.473702 4.080537 3.03448 9.659883 ..
predict  | {[config;data;cutDict]
  updConfig:clust.i.prepPred[config;cutDict..
clust    | 0 0 0 1 1 2 1 0 1 0

Hierarchical Clustering

.ml.kxi.clust.hc.fit

Fit a Hierarchical clustering model

.ml.kxi.clust.hc.fit[X]

Parameters:

name	type	description
X	any	Input/training data of N dimensions.

options:

name	type	description	default
df	symbol	Distance function.	e2dist
lf	symbol	Linkage function.	ward

Returns:

type	description
dictionary	All information collected during the fitting of a model, along with prediction functionality.

Examples:

Example 1:

// Generate feature data
q)data:([]100?1f;100?1f;100?1f)

// Fit a model in default configuration using only required arguments
q)show mdl1:.ml.kxi.clust.hc.fit[data]
modelInfo| `data`inputs`dgram!((0.8599461 0.2452222 0.6070236 0.6868635 0.837..
predict  | {[config;data;cutDict]
  data:clust.util.floatConversion util.tabConvert..
q)mdl1[`modelInfo;`inputs]
df| e2dist
lf| ward

Example 2:

// Generate feature data
q)data:([]100?1f;100?1f;100?1f)

// Fit a model modifying the default behavior using only positional arguments
q)df:`mdist
q)show mdl2:.ml.kxi.clust.hc.fit[data;df]
modelInfo| `data`inputs`dgram!((0.8599461 0.2452222 0.6070236 0.6868635 0.837..
predict  | {[config;data;cutDict]
  data:clust.util.floatConversion util.tabConvert..
q)mdl2[`modelInfo;`inputs]
df| mdist
lf| complete

.ml.kxi.clust.hc.fitPredict

Fit and predict on a hierarchical clustering model

.ml.kxi.clust.hc.fit[X]

Parameters:

name	type	description
X	any	Input/training data of N dimensions.

options:

name	type	description	default
df	symbol	Distance function.	e2dist
lf	symbol	Linkage function.	ward
cutDict	dictionary	Cutting algo to use when splitting the data into clusters (`k/`dist) and a value defining the cutting threshold.	enlist[`k]!enlist 5

Returns:

type	description
dictionary	All information collected during the fitting of a model, along with prediction functionality.

Examples:

Example 1:

// Generate feature data
q)show data:2 10#20?10.
6.01551  9.775468 9.809354 4.237163 5.424916 1.994707 2.496307 2.599..
1.046143 7.154895 8.098937 2.546309 6.298331 0.249301 5.341463 4.106..

// Fit a HC model and cut the dendrogram into 4 clusters
// Use only keyword arguments
q).ml.clust.hc.fitPredict[data;.var.kwargs`lf`k!(`single;4)]
modelInfo| `data`inputs`dgram!((6.01551 9.775468 9.809354 4.237163 5..
predict  | {[config;data;cutDict]
  updConfig:clust.i.prepPred[config;cutDict..
clust    | 0 2 2 0 1 3 0 0 0 1