Working with MATLAB¶
Installation¶
Versions
As MATLAB/datafeed toolbox evolves features or instruction below are subject to revisions. Please refer to toolbox documentation for latest version. Users have reported that this works with more recent versions (e.g. R2015b on RHEL 6.8/2016b and 2017a on macOS).
See also community-supported native connector dmarienko/kdbml
Download and unzip kx_kdbplus.zip. Add the resulting directory to your MATLAB path, for example in MATLAB
>> addpath('/Users/Developer/matlabkx')
Support for MATLAB is a part of Datafeed Toolbox for MATLAB: since R2007a edition.
The MATLAB integration depends on the two Java files c.jar
and jdbc.jar
.
KxSystems/kdb/c/c.jar
KxSystems/kdb/c/jdbc.jar
Add the JAR files to the classpath used by MATLAB. It can be added permanently by editing classpath.txt
(type edit classpath.txt
at the MATLAB prompt) or for the duration of a particular session using the javaaddpath
function, for example
>> javaaddpath /home/myusername/jdbc.jar
>> javaaddpath /home/myusername/c.jar
Installation directory
In these examples change /home/myusername
to the directory where jdbc.jar
and c.jar
are installed.
Alternatively, this can be achieved in a MATLAB source file (i.e., *.m file) adding the following two functions before calling kx
functions.
javaaddpath('/home/myusername/jdbc.jar')
javaaddpath('/home/myusername/c.jar')
Confirm they have been added successfully using the javaclasspath
function.
>> javaclasspath
STATIC JAVA PATH
...
/opt/matlab/2015b/java/jar/toolbox/stats.jar
/opt/matlab/2015b/java/jar/toolbox/symbol.jar
DYNAMIC JAVA PATH
/home/myusername/jdbc.jar
/home/myusername/c.jar
>>
Connecting to a q process¶
First, we start up a kdb+ process that we wish to communicate with from MATLAB and load some sample data into it.
Save following as tradedata.q
file
/ List of securities
seclist:([name:`ACME`ABC`DEF`XYZ]
market:`US`UK`JP`US)
/ Distinct list of securities
secs: distinct exec name from seclist
n:5000
/ Data table
trade:([]sec:`seclist$n?secs;price:n?100.0;volume:100*10+n?20;exchange:5+n?2.0;date:2004.01.01+n?499)
/ Intra day tick data table
intraday:([]sec:`seclist$n?secs;price:n?100.0;volume:100*10+n?20;exchange:5+n?2.0;time:08:00:00.0+n?43200000)
/ Function with one input parameter
/ Return total trading volume for given security
totalvolume:{[stock] select volume from trade where sec = stock}
/ Function with two input parameters
/ Return total trading volume for given security with volume greate than
/ given value
totalvolume2:{[stock;minvolume] select sum(volume) from trade where sec = stock, volume > minvolume}
Then
q tradedata.q -p 5001
q)show trade
sec price volume exchange date
----------------------------------------
ACME 89.5897 1300 6.58303 2005.04.26
ABC 4.346879 2000 5.957694 2004.03.08
DEF 2.486644 1000 5.304114 2004.03.18
ACME 42.26209 1600 5.31383 2004.03.14
DEF 67.79352 2500 5.954478 2004.04.21
DEF 85.56155 1300 6.462338 2004.03.15
ACME 52.65432 1800 5.240313 2005.02.05
ABC 22.43142 2700 5.088007 2005.03.13
ABC 58.26731 2100 5.220929 2004.09.10
XYZ 74.14568 2900 5.075229 2004.08.24
DEF 35.67741 1500 6.064387 2004.03.12
DEF 30.37496 1300 5.025874 2004.03.24
ABC 20.30781 1000 6.642873 2005.02.02
DEF 2.984627 1200 6.346634 2004.12.15
ACME 28.80098 2100 5.591732 2004.09.19
DEF 45.20084 2800 5.481197 2004.08.01
DEF 29.25037 1000 6.065474 2005.02.05
XYZ 50.68805 1700 6.901464 2004.11.02
DEF 41.79832 2300 6.016484 2005.05.04
XYZ 13.64856 2900 6.435824 2005.04.03
..
q)
We then start a new MATLAB session. From here on, >>
represents the MATLAB prompt.
We’re now ready to open a connection to the q process:
>> q = kx('localhost',5001)
q =
handle: [1x1 c]
ipaddress: 'localhost'
port: 5001
Credentials
We can also pass a username:password string as the third parameter to the kx
function if it is required to log in to the q process.
The q
value is a normal MATLAB object and we can inspect the listed properties. We’ll use this value in all our communications with the q process.
We close a connection using the close
function:
>> close(q)
Installation errors
If there is a problem with either the installation of the q integration, or the jar file is not found, we’ll get an error along the lines of:
??? Undefined function or method 'c' for input arguments of type 'char'.
Error in ==> kx.kx at 51
w.handle = c(ip,p);
Or if the socket is not currently connected then any future communications will result in an error like:
??? Java exception occurred:
java.net.SocketException: Socket closed
at java.net.SocketOutputStream.socketWrite(Unknown Source)
at java.net.SocketOutputStream.write(Unknown Source)
at c.w(c.java:99)
at c.k(c.java:107)
at c.k(c.java:108)
Error in ==> kx.fetch at 65
t = c.handle.k(varargin{1});
Using the kdb+ process¶
It is typical to perform basic interactions with a database using the fetch
function via a connected handle. For example in a legacy database we might perform this:
x = fetch(q,'select * from tablename')
We can use this function to perform basic interaction with kdb+, where we expect a value to be returned. This need not be a query but in fact can be general chunks of code. Using q as a calculator, we can compute the average of 0 to 999.
>> fetch(q,'avg til 1000')
ans =
499.5000
Fetching data from kdb+¶
The fetch
function can be used to get q data such as lists, as well as tables. Given the list c
:
q)c:((til 100);(til 100))
q)c
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 ..
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 ..
Then we can fetch it:
>> hundreds = fetch(q, 'c')
hundreds =
java.lang.Object[]:
[100×1 int64]
[100×1 int64]
We can use the cell
function to strip the Java array wrapper away:
>> hundreds_as_cell = cell(hundreds)
hundreds_as_cell =
2×1 cell array
{100×1 int64}
{100×1 int64}
Tables are returned as an object with an array property for each column. Taking the first 10 rows of the trade
table as an example:
q)10#trade
sec price volume exchange date
----------------------------------------
ACME 89.5897 1300 6.58303 2005.04.26
ABC 4.346879 2000 5.957694 2004.03.08
DEF 2.486644 1000 5.304114 2004.03.18
ACME 42.26209 1600 5.31383 2004.03.14
DEF 67.79352 2500 5.954478 2004.04.21
DEF 85.56155 1300 6.462338 2004.03.15
ACME 52.65432 1800 5.240313 2005.02.05
ABC 22.43142 2700 5.088007 2005.03.13
ABC 58.26731 2100 5.220929 2004.09.10
XYZ 74.14568 2900 5.075229 2004.08.24
Will be returned in MATLAB:
>> ten = fetch(q, '10#trade')
ten =
sec: {10×1 cell}
price: [10×1 double]
volume: [10×1 int64]
exchange: [10×1 double]
date: [10×1 double]
With suitable computation in q, we can return data suitable for immediate plotting. Here we compute a 10-item moving average over the `ACME
prices:
q)mavg[10;exec price from trade where sec=`ACME]
89.5897 65.9259 61.50204 53.32677 54.74408 57.39743 57.15958 62.33525 56.8732..
>> acme = fetch(q,'mavg[10;exec price from trade where sec=`ACME]')
Metadata¶
The q integration in MATLAB provides the tables
meta function.
>> tables(q)
ans =
'intraday'
'seclist'
'trade'
The experienced q user can use the \v
command to see all values in the directory:
>> fetch(q,'\v')
ans =
'a'
'b'
'c'
'intraday'
'n'
'seclist'
'secs'
'trade'
Sending data to q¶
We can use the fetch
function to cause side effects in the kdb+ process, such as inserting data into a table.
Given a table b
:
q)b:([] a:1 2; b:1 2)
q)b
a b
---
1 1
2 2
Then we can add a row like this:
>> fetch(q,'b,:(3;3)')
ans =
[]
and, sure enough, on the q side we see the new data:
q)show b
a b
---
1 1
2 2
3 3
The q integration also provides an insert
function: this takes an array of items in the row and may be more convenient for certain purposes.
>> insert(q,'b',{4,4})
shows on the q side as:
q)show b
a b
---
1 1
2 2
3 3
4 4
A more complicated row shows the potential advantage to better effect:
>> insert(q,'trade',{'`ACME',100.45,400,.0453,'2005.04.28'})
Be warned though, that errors will not be detected very well. For example the following expression silently fails!
>> insert(q,'b',{1,2,3})
whereas the equivalent fetch
call provokes an error:
>> fetch(q,'b,:(1;2;3)')
Error using fetch (line 64)
Java exception occurred:
kx.c$KException: length
at kx.c.k(c.java:110)
at kx.c.k(c.java:111)
at kx.c.k(c.java:112)
Async commands to q¶
The exec
function is used for sending asynchronous commands to q; ones we do not expect a response to, and which may be performed in the background while we continue interacting with the MATLAB process.
Here we establish a large-ish data structure in the kdb+ process:
>> exec(q,'big_data:10000000?100')
Then we take the average of the data, delete it from the namespace and close the connection:
>> fetch(q,'avg big_data')
ans =
49.4976
>> exec(q,'delete big_data from `.')
>> close(q)
Handling null¶
kdb+ has the ability to set values to null. MATLAB doesnt have a corresponding null type, so if your data contains nulls you may wish to filter or detect them.
MATLAB has the ability to call static methods within Java. The NULL
method can provide the null values for the different data types. For example
NullInt=kx.c.NULL('i')
NullLong=kx.c.NULL('j')
NullDouble=kx.c.NULL('f')
NullDate=kx.c.NULL('d')
With this, you can test values for null. The following shows that the comparison will return true when requesting null values from a kdb+ connection named conn:
fetch(conn,'0Ni')== NullInt
fetch(conn,'0N')== NullLong
fetch(conn,'0Nd')== NullDate
isequaln(fetch(conn,'0Ni'),NullInt)
isequaln(fetch(conn,'0N'), NullLong)
isequaln(fetch(conn,'0Nd'), NullDate)
isequaln(fetch(conn,'0Nf'), NullDouble)
An alternative is to have your query include a filter for nulls (if they are populated), so they arent retrieved by MATLAB.
Getting more help¶
Start with help kx
in your MATLAB session and also see help kx.fetch
and so on for further details of the integration.
MathWorks provides functions overview, usage instructions and some examples on the toolbox webpage.