Metrics

.ml Statistical analysis metrics accuracy Accuracy of classification results classReport Statistical information about classification results confDict True/false positives and true/false negatives as dictionary confMatrix Confusion matrix corrMatrix Table-like correlation matrix for a simple table crossEntropy Categorical cross entropy covMatrix Covariance matrix describe Descriptive information about a table fBetaScore Fbeta-score on classification results logLoss Logarithmic loss mae Mean absolute error mape Mean absolute percentage error matthewCorr Matthews correlation coefficient mse Mean square error percentile Percentile calculation for an array precision Precision of a binary classifier r2Score R2-score rmse Root mean square error rmsle Root mean square logarithmic error roc X- and Y-axis values for an ROC curve rocAucScore Area under an ROC curve sensitivity Sensitivity of a binary classifier smape Symmetric mean absolute error specificity Specificity of a binary classifier sse Sum squared error tScore One-sample t-test score tScoreEqual T-test for independent samples with unequal variances

KxSystems/ml/util/metrics.q

The toolkit contains an extensive list of commonly used metrics for evaluating the performance of machine-learning algorithms. These cover the testing of both regression and classification results.

.ml.accuracy

Accuracy of classification results

.ml.accuracy[pred;true]

Where

• pred is a vector/matrix of predicted labels
• true is a vector/matrix of true labels

returns the accuracy of predictions made.

q).ml.accuracy[1000?0b;1000?0b] / binary classifier
0.482
q).ml.accuracy[1000?10;1000?10] / non-binary classifier
0.108
q).ml.accuracy[10 2#20?10;10 2#20?10] / support for matrices of predictions and true labels
0.3 0.1

.ml.classReport

Statistical information about classification results

.ml.classReport[pred;true]

Where

• pred is a vector of predicted labels.
• true is a vector of true labels.

returns the accuracy, precision, f1 scores and the support (number of occurrences) of each class.

q)n:1000
q)xr:n?5
q)yr:n?5
q).ml.classReport[xr;yr]
class     precision recall f1_score support
-------------------------------------------
0         0.2       0.23   0.22     179
1         0.22      0.22   0.22     193
2         0.21      0.21   0.21     192
3         0.19      0.19   0.19     218
4         0.21      0.17   0.19     218
avg/total 0.206     0.204  0.206    1000
q)xb:n?0b
q)yb:n?0b
q).ml.classReport[xb;yb]
class     precision recall f1_score support
-------------------------------------------
0         0.51      0.51   0.51     496
1         0.52      0.52   0.52     504
avg/total 0.515     0.515  0.515    1000
q)xc:n?`A`B`C
q)yc:n?`A`B`C
q).ml.classReport[xc;yc]
class     precision recall f1_score support
-------------------------------------------
A         0.34      0.33   0.33     336
B         0.33      0.36   0.35     331
C         0.32      0.3    0.31     333
avg/total 0.33      0.33   0.33     1000

Deprecated

This function was previously defined as `.ml.classreport. That is still callable but will be deprecated after version 3.0.

.ml.confDict

True/false positives and true/false negatives

.ml.confDict[pred;true;posClass]

Where

• pred is a vector of (binary or multi-class) predicted labels
• true is a vector of (binary or multi-class) true labels
• posClass is an atom denoting the positive class label

returns a dictionary giving the count of true positives (tp), true negatives (tn), false positives (fp) and false negatives (fn).

q).ml.confDict[100?"AB";100?"AB";"A"]  / non-numeric inputs
tn| 25
fp| 25
fn| 21
tp| 29
q).ml.confDict[100?0b;100?0b;1b]  / boolean input
tn| 25
fp| 27
fn| 26
tp| 22
q).ml.confDict[100?5;100?5;4]    / supports multiclass by converting to boolean representation
tn| 60
fp| 18
fn| 19
tp| 3

Deprecated

This function was previously defined as .ml.confdict. That is still callable but will be deprecated after version 3.0.

.ml.confMatrix

Confusion matrix

.ml.confMatrix[pred;true]

Where

• pred is a vector of (binary or multi-class) predicted labels
• true is a vector of (binary or multi-class) true labels

returns a confusion matrix.

q).ml.confMatrix[100?"AB";100?"AB"]  / non-numeric inputs
A| 22 30
B| 22 26
q).ml.confMatrix[100?0b;100?0b]      / boolean input
0| 20 26
1| 26 28
q).ml.confMatrix[100?5;100?5]        / supports multiclass by converting to boolean representation
0| 5 6 2 2 1
1| 1 6 5 4 3
2| 3 5 2 4 4
3| 6 5 9 2 8
4| 3 5 4 2 3

Deprecated

This function was previously defined as .ml.confmat. That is still callable but will be deprecated after version 3.0.

.ml.corrMatrix

Calculate the correlation of a matrix or table

.ml.corrMatrix data

Where

• data is a sample from a distribution as a matrix or simple table

returns the covariance of the data

q)tab:([]A:asc 100?1f;B:desc 100?1000f;C:100?100)
q).ml.corrMatrix tab
 | A          B          C
-| --------------------------------
A| 1          -0.9945903 -0.2273659
B| -0.9945903 1          0.2287606
C| -0.2273659 0.2287606  1

Deprecated

This function was previously defined as .ml.corrmat. That is still callable but will be deprecated after version 3.0.

.ml.crossEntropy

Categorical cross entropy

.ml.crossEntropy[pred;prob]

Where

• pred is a vector of indices representing class labels
• prob is a list of vectors representing the probability of belonging to each class

returns the categorical cross entropy for each class.

q)p%:sum each p:1000 5#5000?1f
q)g:(first idesc@)each p / good labels
q)b:(first iasc@)each p  / bad labels
q).ml.crossEntropy[g;p]
1.07453
q).ml.crossEntropy[b;p]
3.187829

Deprecated

This function was previously defined as .ml.crossentropy. That is still callable but will be deprecated after version 3.0.

.ml.covMatrix

Covariance of a matrix

.ml.covMatrix matrix

Where

• matrix is a sample from a distribution

returns the covariance matrix.

q)show mat:(5?10;5?10;5?10)
3 6 0 8 4
1 7 4 7 6
6 9 9 8 7
q).ml.covMatrix[mat]
7.36 4   0.04
4    5.2 1.6
0.04 1.6 1.36

Deprecated

This function was previously defined as .ml.cvm. That is still callable but will be deprecated after version 3.0.

.ml.f1Score

F-1 score for classification results

.ml.f1Score[pred;true;posClass]

Where

• pred is a vector of predicted labels
• true is a vector of true labels
• posClass is the positive class

returns the F-1 score between predicted and true values.

q)xr:1000?5
q)yr:1000?5
q).ml.f1Score[xr;yr;4]
0.1980676
q)xb:1000?0b
q)yb:1000?0b
q).ml.f1Score[xb;yb;0b]
0.4655532

Deprecated

This function was previously defined as .ml.f1score. That is still callable but will be deprecated after version 3.0.

.ml.fBetaScore

F-beta score for classification results

.ml.fBetaScore[pred;true;posClass;beta]

Where

• pred is a vector of predicted labels
• true is a vector of true labels
• posClass is the positive class
• beta is the value of beta

returns the F-beta score between predicted and true labels.

q)xr:1000?5
q)yr:1000?5
q).ml.fBetaScore[xr;yr;4;.5]
0.2254098
q)xb:1000?0b
q)yb:1000?0b
q).ml.fBetaScore[xb;yb;1b;.5]
0.5191595

Deprecated

This function was previously defined as .ml.fbscore. That is still callable but will be deprecated after version 3.0.

.ml.logLoss

Logarithmic loss

.ml.logLoss[pred;prob]

Where

• pred is a vector of class labels 0/1
• prob is a list of vectors representing the probability of belonging to each class

returns the total logarithmic loss.

q)v:1000?2
q)g:g,'1-g:1-(.5*v)+1000?.5 / good predictions
q)b:b,'1-b:1000?.5          / bad predictions
q).ml.logLoss[v;g]
0.3005881
q).ml.logLoss[v;b]
1.032062

Deprecated

This function was previously defined as .ml.logloss. That is still callable but will be deprecated after version 3.0.

.ml.mae

Mean absolute error

.ml.mae[pred;true]

Where

• pred is a vector of predicted values
• true is a vector of true values

returns the mean absolute error between predicted and true values.

q).ml.mae[100?0b;100?0b]
0.44
q).ml.mae[100?5;100?5]
1.73

.ml.mape

Mean absolute percentage error

.ml.mape[pred;true]

Where

• pred is a vector of predicted values
• true is a vector of true values

returns the mean absolute percentage error between predicted and true values. All values must be floats.

q).ml.mape[100?5.0;100?5.0]
660.9362

.ml.matthewCorr

Matthews-correlation coefficient

.ml.matthewCorr[pred;true]

Where

• pred is a vector of predicted labels
• true is a vector of the true labels

returns the Matthews-correlation coefficient between predicted and true values.

q).ml.matthewCorr[100?0b;100?0b]
0.1256775
q).ml.matthewCorr[100?5;100?5]
7.880334e-06

Deprecated

This function was previously defined as .ml.matcorr. That is still callable but will be deprecated after version 3.0.

.ml.mse

Mean square error

.ml.mse[pred;true]

Where

• pred is a vector of predicted values
• true is a vector of true values

returns the mean squared error between predicted values and the true values.

q).ml.mse[asc 100?1f;asc 100?1f]
0.0004801384
q).ml.mse[asc 100?1f;desc 100?1f]
0.3202164

.ml.precision

Precision of a binary classifier

.ml.precision[pred;true;posClass]

Where

• pred is a vector of (binary) predicted values
• true is a vector of (binary) true values
• posClass is the binary value defined to be 'true'

returns a measure of the precision.

q).ml.precision[1000?0b;1000?0b;1b]
0.5020161
q).ml.precision[1000?"AB";1000?"AB";"B"]
0.499002

.ml.r2Score

R2-score for regression model validation

.ml.r2Score[pred;true]

Where

• pred are (continuous) predicted values
• true are (continuous) true values

returns the R2-score between the true and predicted values. Values close to 1 indicate good prediction, while negative values indicate poor predictors of the system behavior.

q)xg:asc 1000?50f           / 'good values'
q)yg:asc 1000?50f
q).ml.r2Score[xg;yg]
0.9966209
q)xb:asc 1000?50f           / 'bad values'
q)yb:desc 1000?50f
q).ml.r2Score[xb;yb]
-2.981791

Deprecated

This function was previously defined as .ml.r2score. That is still callable but will be deprecated after version 3.0.

.ml.rmse

Root mean squared error for regression model validation

.ml.rmse[pred;true]

Where

• pred is a vector of predicted values
• true is a vector of true values

returns the root mean squared error for a regression model between predicted and true values.

q)n:10000
q)xg:asc n?100f
q)yg:asc n?100f
q).ml.rmse[xg;yg]
0.5321886
q)xb:n?100f
q).ml.rmse[xb;yg]
41.03232

.ml.rmsle

Root mean squared logarithmic error

.ml.rmsle[pred;true]

Where

• pred is a vector of predicted values
• true is a vector of true values

returns the root mean squared logarithmic error between predicted and true values.

q).ml.rmsle[100?0b;100?0b]
0.5187039
q).ml.rmsle[100?5;100?5]
0.8465022

.ml.roc

X and Y axis values for an ROC curve

.ml.roc[label;prob]

Where

• label is the label associated with a prediction
• prob is the probability that a prediction belongs to the positive class

returns the coordinates of the true-positive and false-positive values associated with the ROC curve.

q)v:raze reverse 50?'til[20+1]xprev\:20#1b
q)p:asc count[v]?1f
q).ml.roc[v;p]
0           0           0           0           0           ..
0.001937984 0.003875969 0.005813953 0.007751938 0.009689922 ..

.ml.rocAucScore

Area under an ROC curve

.ml.rocAucScore[x;y]

Where

• label is the label associated with a prediction
• prob is the probability that a prediction belongs to the positive class

returns the area under the ROC curve.

q)v:raze reverse 50?'til[20+1]xprev\:20#1b
q)p:asc count[v]?1f
q).ml.rocAucScore[v;p]
0.8696362

Deprecated

This function was previously defined as .ml.rocaucscore. That is still callable but will be deprecated after version 3.0.

.ml.sensitivity

Sensitivity of a binary classifier

.ml.sensitivity[pred;true;posClass]

Where

• pred is a vector of (binary) predicted values
• true is a vector of (binary) true values
• posClass is the binary value defined to be 'true'

returns a measure of the sensitivity.

q).ml.sensitivity[1000?0b;1000?0b;1b]
0.4876033
q).ml.sensitivity[1000?`class1`class2;1000?`class1`class2;`class1]
0.5326923

.ml.smape

Symmetric mean absolute percentage error

.ml.smape[pred;true]

Where

• pred is a vector of predicted values
• true is a vector of true values

returns the symmetric mean absolute percentage error between predicted and true values.

q).ml.smape[100?0b;100?0b]
92f
q).ml.smape[100?5;100?5]
105.781

.ml.specificity

Specificity of a binary classifier

.ml.specificity[pred;true;posClass]

Where

• pred is a vector of (binary) predicted values
• true is a vector of (binary) true values
• posClass is the binary value defined to be 'true'

returns a measure of the specificity.

q).ml.specificity[1000?0b;1000?0b;1b]
0.5426829
q).ml.specificity[1000?100 200;1000?100 200;200]
0.4676409

.ml.sse

Sum squared error

.ml.sse[pred;true]

Where

• pred is a vector of predicted values
• true is a vector of true values

returns the sum squared error between predicted and true values.

q).ml.sse[asc 100?1f;asc 100?1f]
0.4833875
q).ml.sse[asc 100?1f;desc 100?1f]
32.06403

.ml.tScore

One-sample t-test score

.ml.tScore[sample;mu]

Where

• sample is a set of samples from a distribution
• mu is the population mean

returns the one sample t-score for a distribution with less than 30 samples.

q)x:25?100f
q)y:15
q).ml.tScore[x;y]
7.634824

Deprecated

This function was previously defined as .ml.tscore. That is still callable but will be deprecated after version 3.0.

.ml.tScoreEqual

T-test for independent samples with equal variances and equal sample size

.ml.tScoreEqual[sample1;sample2]

Where

• sample1 and sample2 are independent sample sets with equal variance and sample size

returns their t-test score.

q)x:10+(-.5+avg each 0N 20#1000?1f)*sqrt 20*12 / ~N(10,1)
q)y:20+(-.5+avg each 0N 20#1000?1f)*sqrt 20*12 / ~N(20,1)
q)(avg;dev)@\:/:(x;y) / check dist
9.87473  1.010691
19.84484 0.9437789
q).ml.tScoreEqual[x;y]
50.46957

Deprecated

This function was previously defined as .ml.tscoreeq. That is still callable but will be deprecated after version 3.0.