How to use PyKX within q

This page provides details on how to run PyKX within a q session, including how to evaluate and execute Python code, how to interact with objects, and how to call a function.

Introduction

PyKX is a Python-first interface to the programming language q and its underlying database kdb+. To overcome a few limitations, PyKX allows you to run Python within q, similarly to embedPy. The ability to execute and manipulate Python objects within a q session helps two types of users in the following ways:

kdb+/q users can build applications which embed machine learning/data science libraries in production q infrastructures.
Users of Python plotting libraries can visualize and explore the outcomes of their analyses.

Getting started

Prerequisites

Before you run PyKX within q, make sure you:

Have access to a running q environment. [Follow the q installation guide.]
Have installed the licensed version of PyKX.

Install

Run the following command to install the pykx.q script into your $QHOME directory:

python -c "import pykx;pykx.install_into_QHOME()"

If you previously had embedPy installed, pass:

python -c "import pykx;pykx.install_into_QHOME(overwrite_embedpy=True)"

If you cannot edit the files in QHOME, copy them to your local folder and load pykx.q from there:

python -c "import pykx;pykx.install_into_QHOME(to_local_folder=True)"

Initialize

Initialize the library as follows:

q)\l pykx.q
q).pykx
console    | {pyexec"pykx.console.PyConsole().interact(banner='', exitmsg='')"}
getattr    | code
get        | {[f;x]r:wrap f x 0;$[count x:1_x;.[;x];]r}[code]enlist
setattr    | {i.load[(`set_attr;3)][unwrap x;y;i.convertArg[i.toDefault z]`.]}
set        | {i.load[(`set_global;2)][x; i.convertArg[i.toDefault y]`.]}
print      | {$[type[x]in 104 105 112h;i.repr[0b] unwrap x;show x];}
repr       | {$[type[x]in 104 105 112h;i.repr[1b] unwrap x;.Q.s x]}
import     | {[f;x]r:wrap f x 0;$[count x:1_x;.[;x];]r}[code]enlist
..

How to use the library

Use this library to complete a wide variety of tasks, from the simple execution of Python code through to the generation of streaming applications containing machine learning models. The next sections outline various use-case-agnostic scenarios that you can follow.

Evaluate and Execute Python

Differences between evaluation and execution

Python evaluation (unlike Python execution) does not allow side effects. Any attempt at variable assignment or class definition signals an error. To execute a string with side effects, use .pykx.pyexec or .p.e.

Difference between eval and exec in Python

What’s a Python side effect?

A Python function has side effects if it might do more than return a value, for example, modify the state or interact with external entities/systems in a noticeable way. Such effects could manifest as changes to input arguments, modifications to global variables, file operations, or network communications.

Evaluate Python code

To evaluate Python code with PyKX, pass a string of Python code to a variety of PyKX functions as shown below.

For example, if you want to evaluate and return the result to q, use the function .pykx.qeval:

q).pykx.qeval"1+2"
3

Similarly, to evaluate Python code and return the result as a foreign object denoting the underlying Python object, use:

q)show a:.pykx.pyeval"1+2"
foreign
q)print a
3

Finally, to return a hybrid representation that you can edit to return the q or Python representation, run the following:

q)show b:.pykx.eval"1+2"
{[f;x].pykx.util.pykx[f;x]}[foreign]enlist
q)b`       // Convert to a q object
3
q)b`.      // Convert to a Python foreign
foreign

Execute Python code

This interface allows you to execute Python code in a variety of ways:

a) Execute directly with the .pykx.pyexec function

This is incredibly useful if you need to script execution of Python code within a library:

q).pykx.pyexec"import numpy as np"
q).pykx.pyexec"array = np.array([0, 1, 2, 3])"
q).pykx.pyexec"print(array)"
[0 1 2 3]

b) Use the PyKX console functionality

This is useful when interacting within a q session and you need to prototype a functionality in Python:

q).pykx.console[]
>>> import numpy as np
>>> print(np.linspace(0, 10, 5))
[ 0.   2.5  5.   7.5 10. ]
>>> quit()
q)

c) Use a p) prompt

This way of embedding the execution of Python code within a q script also provides backwards compatibility with embedPy:

q)p)import numpy as np
q)p)print(np.arange(1, 10, 2))
[1 3 5 7 9]

d) Load a .p file

This is a method of executing the contents of a Python file in bulk:

$ cat test.p
def func(x, y):
        return(x+y)
$ q pykx.q
q)\l test.p
q).pykx.get[`func]
{[f;x].pykx.util.pykx[f;x]}[foreign]enlist

Interact with PyKX objects

Foreign objects

At the lowest level, Python objects are represented in q as foreign objects, which contain pointers to objects in the Python memory space.

You can store foreign objects in variables just like any other q datatype, or as part of lists, dictionaries or tables. They will show up as foreign when inspected in the q console or using the string (or .Q.s) representation.

Serialization and IPC

Kdb+ cannot serialize foreign objects, nor send them over IPC. Foreign objects live in the embedded Python memory space. To pass them over IPC, first you have to convert them to q.

Create PyKX objects

q doesn't allow you to operate directly with foreign objects. Instead, Python objects are represented as PyKX objects, which wrap the underlying foreign objects. This helps to get and set attributes, index, call or convert the underlying foreign object to a q object.

Use .pykx.wrap to create a PyKX object from a foreign object.

q)x
foreign
q)p:.pykx.wrap x
q)p           /how a PyKX object looks
{[f;x].pykx.util.pykx[f;x]}[foreign]enlist

To retrieve PyKX objects directly from Python, choose between the following functions:

Function	Argument	Example
`.pykx.import`	symbol: name of a Python module or package, optional second argument is the name of an object within the module or package	np:.pykx.import`numpy
`.pykx.get`	symbol: name of a Python variable in `__main__`	v:.pykx.get`varName
`.pykx.eval`	string: Python code to evaluate	`x:.pykx.eval"1+1"`

Side effects

As with other Python evaluation functions, .pykx.eval does not allow side effects.

Convert data

For obj, a PyKX object representing Python data, to obtain the underlying data (as foreign object or q) use:

obj`. / get data as foreign
obj`  / get data as q

For example:

q)x:.pykx.eval"(1,2,3)"
q)x
{[f;x].pykx.util.pykx[f;x]}[foreign]enlist
q)x`.
foreign
q)x`
1 2 3

`None` and identity

Python None maps to the q identity function :: when converting from Python to q (and vice versa).

Exception!

When calling Python functions, methods or classes with a single q data argument, passing :: results in the Python object being called with no arguments, rather than a single argument of None. See the Zero-argument calls section for how to call a Python object with a single None argument.

Get attributes and properties

Given obj, a PyKX object representing a Python object, you can get an attribute or property by using:

obj`:attr         / equivalent to obj.attr in Python
obj`:attr1.attr2  / equivalent to obj.attr1.attr2 in Python

These expressions return PyKX objects, allowing you to chain operations together:

obj[`:attr1]`:attr2  / equivalent to obj.attr1.attr2 in Python

For example:

$ cat class.p
class obj:
    def __init__(self,x=0,y=0):
        self.x = x
        self.y = y

q)\l class.p
q)obj:.pykx.eval"obj(2,3)"
q)obj[`:x]`
2
q)obj[`:y]`
3

Set attributes and properties

Given obj, a PyKX object representing a Python object, you can set an attribute or property by using:

obj[:;`:attr;val]  / equivalent to obj.attr=val in Python

For example:

q)obj[`:x]`
2
q)obj[`:y]`
3
q)obj[:;`:x;10]
q)obj[:;`:y;20]
q)obj[`:x]`
10
q)obj[`:y]`
20

How to Index

Given lst, a PyKX object representing an indexable container object in Python, you can access the element at index i by using:

lst[@;i]    / equivalent to lst[i] in Python

Set the element at index i (to object #!pythonx) with this command:

lst[=;i;x]  / equivalent to lst[i]=x in Python

These expressions return PyKX objects, for instance:

q)lst:.pykx.eval"[True,2,3.0,'four']"
q)lst[@;0]`
1b
q)lst[@;-1]`
`four
q)lst'[@;;`]2 1 0 3
3f
2
1b
`four
q)lst[=;0;0b];
q)lst[=;-1;`last];
q)lst`
0b
2
3f
`last

Get methods

Given obj, a PyKX object representing a Python object, you can access a method by using:

obj`:method  / equivalent to obj.method in Python

When calling PyKX objects representing Python methods, the return of evaluation is a PyKX object. For example:

q)np:.pykx.import`numpy
q)np`:arange
{[f;x].pykx.util.pykx[f;x]}[foreign]enlist
q)arange:np`:arange                   / callable returning PyKX object
q)arange 12
{[f;x].pykx.util.pykx[f;x]}[foreign]enlist
q)arange[12]`
0 1 2 3 4 5 6 7 8 9 10 11

PyKX function API

Use the function API to achieve the following:

Call PyKX objects (to get PyKX objects).
Declare PyKX objects callable (to get q or foreign data).

The default return is a PyKX object. For q or foreign return type, you need to specify it.

Given func, a PyKX object representing a callable Python function or method, you can carry out the following operations:

func                   / func is callable by default (returning PyKX)
func arg               / call func(arg) (returning PyKX)
func[<]                / declare func callable (returning q)
func[<]arg             / call func(arg) (returning q)
func[<;arg]            / equivalent
func[>]                / declare func callable (returning foreign)
func[>]arg             / call func(arg) (returning foreign)
func[>;arg]            / equivalent

How to chain operations?

To chain together sequences of operations, return another PyKX object from a function or method call. Alternatively, call .pykx.import, .pykx.get and .pykx.eval.

PyKX examples

Example #1Example #2Example #3Example #4

$ cat test.p # used for tests
class obj:
    def __init__(self,x=0,y=0):
        self.x = x # attribute
        self.y = y # property (incrementing on get)
    @property
    def y(self):
        a=self.__y
        self.__y+=1
        return a
    @y.setter
    def y(self, y):
        self.__y = y
    def total(self):
        return self.x + self.y

q)\l test.p
q)obj:.pykx.get`obj / obj is the *class* not an instance of the class
q)o:obj[]           / call obj with no arguments to get an instance
q)o[`:x]`
0
q)o[;`]each 5#`:x
0 0 0 0 0
q)o[:;`:x;10]
q)o[`:x]`
10
q)o[`:y]`
1
q)o[;`]each 5#`:y
3 5 7 9 11
q)o[:;`:y;10]
q)o[;`]each 5#`:y
10 13 15 17 19
q)tot:o[`:total;<]
q)tot[]
30
q)tot[]
31

q)np:.pykx.import`numpy
q)v:np[`:arange;12]
q)v`
0 1 2 3 4 5 6 7 8 9 10 11
q)v[`:mean;<][]
5.5
q)rs:v[`:reshape;<]
q)rs[3;4]
0 1 2  3
4 5 6  7
8 9 10 11
q)rs[2;6]
0 1 2 3 4  5
6 7 8 9 10 11
q)np[`:arange;12][`:reshape;3;4]`
0 1 2  3
4 5 6  7
8 9 10 11

q)stdout:.pykx.import[`sys]`:stdout.write
q)stdout `$"hello\n";
hello
q)stderr:.pykx.import[`sys;`:stderr.write]
q)stderr `$"goodbye\n";
goodbye

q)oarg:.pykx.eval"10"
q)oarg`
10
q)ofunc:.pykx.eval["lambda x:2+x";<]
q)ofunc[1]
3
q)ofunc oarg
12
q)p)def add2(x,y):return x+y
q)add2:.pykx.get[`add2;<]
q)add2[1;oarg]
11

Function argument types

PyKX supports data type conversions between q and Python for Python native objects, Numpy objects, Pandas objects, PyArrow objects, and PyKX objects.

By default, when passing a q object to a callable function, it's converted to the most "natural" analogous type, as detailed below:

PyKX/q generic list objects become Python lists.
PyKX/q table/keyed table objects become Pandas equivalent DataFrames.
All other PyKX/q objects become their analogous numpy equivalent types.

Warning

Prior to PyKX 2.1.0, all conversions from q objects to Python would convert to their Numpy equivalent. To achieve this now, set the environment variable PYKX_DEFAULT_CONVERSION="np"

For function/method calls, control the default behavior of the conversions by setting .pykx.util.defaultConv:

q).pykx.util.defaultConv
"default"

You can apply one of the following values:

Python type	Default	Python	Numpy	Pandas	PyArrow	PyKX
Value:	"default"	"py"	"np"	"pd"	"pa"	"k"

In the example below, we start with Numpy and update the default types across all function calls:

NumpyPythonPandasPyArrowPyKX

q)typeFunc:.pykx.eval"lambda x:print(type(x))"
q)typeFunc 1; 
<class 'numpy.int64'>
q)typeFunc til 10;
<class 'numpy.ndarray'>
q)typeFunc (10?1f;10?1f)
<class 'list'>
q)typeFunc ([]100?1f;100?1f);
<class 'pandas.core.frame.DataFrame'>

q)typeFunc:.pykx.eval"lambda x:print(type(x))"
q).pykx.util.defaultConv:"py"
q)typeFunc 1;
<class 'int'>
q)typeFunc til 10;
<class 'list'>
q)typeFunc ([]100?1f;100?1f);
<class 'dict'>

q).pykx.util.defaultConv:"pd"
q)typeFunc 1;
<class 'numpy.int64'>
q)typeFunc til 10;
<class 'pandas.core.series.Series'>
q)typeFunc ([]100?1f;100?1f);
<class 'pandas.core.frame.DataFrame'>

q).pykx.util.defaultConv:"pa"
q)typeFunc 1;
<class 'numpy.int64'>
q)typeFunc til 10;
<class 'pyarrow.lib.Int64Array'>
q)typeFunc ([]100?1f;100?1f);
<class 'pyarrow.lib.Table'>

q).pykx.util.defaultConv:"k"
q)typeFunc 1;
<class 'pykx.wrappers.LongAtom'>
q)typeFunc til 10;
<class 'pykx.wrappers.LongVector'>
q)typeFunc ([]100?1f;100?1f);
<class 'pykx.wrappers.Table'>

Alternatively, to modify individual arguments to functions, use the .pykx.to* functionality:

q)typeFunc:.pykx.eval"lambda x,y: [print(type(x)), print(type(y))]"
q)typeFunc[til 10;til 10];                          // Simulate passing both arguments with defaults
<class 'numpy.ndarray'>
<class 'numpy.ndarray'>
q)typeFunc[til 10].pykx.topd til 10;               // Pass in the second argument as Pandas series
<class 'numpy.ndarray'>
<class 'pandas.core.series.Series'>
q)typeFunc[.pykx.topa([]100?1f);til 10];           // Pass in first argument as PyArrow Table
<class 'pyarrow.lib.Table'>
<class 'numpy.ndarray'>
q)typeFunc[.pykx.tok til 10;.pykx.tok ([]100?1f)]; // Pass in two PyKX objects
<class 'pykx.wrappers.LongVector'>
<class 'pykx.wrappers.Table'>

Set Python variables

You can set variables in Python __main__ by using .pykx.set:

q).pykx.set[`var1;42]
q).pykx.qeval"var1"
42
q).pykx.set[`var2;{x*2}]
q)qfunc:.pykx.get[`var2;<]
{[f;x].pykx.util.pykx[f;x]}[foreign]enlist
q)qfunc[3]
6

Function calls

Python allows you to call functions with:

A variable number of arguments
A mixture of positional and keyword arguments
Implicit (default) arguments

This is available in the PyKX function-call interface, as detailed below:

Callable PyKX objects are variadic (they accept a variable number of arguments).
Default arguments are applied where no explicit arguments are given.
Individual keyword arguments are specified using the (infix) pykw operator.
A list of positional arguments can be passed using pyarglist (like Python *args).
A dictionary of keyword arguments can be passed using pykwargs (like Python **kwargs).

Keyword arguments last

You can combine positional arguments, lists of positional arguments, keyword arguments, and a dictionary of keyword arguments. However, all keyword arguments must always follow any positional arguments. The dictionary of keyword arguments (if given) must be specified last.

Examples

q)p)import numpy as np
q)p)def func(a=1,b=2,c=3,d=4):return np.array([a,b,c,d,a*b*c*d])
q)qfunc:.pykx.get[`func;<] / callable, returning q

Enter positional arguments directly. Function calling is variadic, so you can exclude later arguments:

q)qfunc[2;2;2;2]   / all positional args specified
2 2 2 2 16
q)qfunc[2;2]       / first 2 positional args specified
2 2 3 4 48
q)qfunc[]          / no args specified
1 2 3 4 24
q)qfunc[2;2;2;2;2] / error if too many args specified
'TypeError('func() takes from 0 to 4 positional arguments but 5 were given')
  [0]  qfunc[2;2;2;2;2] / error if too many args specified
       ^

Specify individual keyword arguments with the pykw operator (applied infix). The order of keyword arguments doesn't matter.

q)qfunc[`d pykw 1;`c pykw 2;`b pykw 3;`a pykw 4] / all keyword args specified
4 3 2 1 24
q)qfunc[1;2;`d pykw 3;`c pykw 4]   / mix of positional and keyword args
1 2 4 3 24
q)qfunc[`a pykw 2;`b pykw 2;2;2]   / error if positional args after keyword args
'TypeError("func() got multiple values for argument 'a'")
  [0]  qfunc[`a pykw 1;pyarglist 2 2 2] 
       ^
q)qfunc[`a pykw 2;`a pykw 2]       / error if duplicate keyword args
'Expected only unique key names for keyword arguments in function call
  [0]  qfunc[`a pykw 2;`a pykw 2]  
       ^

To specify a list of positional arguments, use pyarglist (similar to Python’s *args). Again, keyword arguments must follow positional arguments.

q)qfunc[pyarglist 1 1 1 1]          / full positional list specified
1 1 1 1 1
q)qfunc[pyarglist 1 1]              / partial positional list specified
1 1 3 4 12
q)qfunc[1;1;pyarglist 2 2]          / mix of positional args and positional list
1 1 2 2 4
q)qfunc[pyarglist 1 1;`d pykw 5]    / mix of positional list and keyword args
1 1 3 5 15
q)qfunc[pyarglist til 10]           / error if too many args specified
'TypeError('func() takes from 0 to 4 positional arguments but 10 were given')
  [0]  qfunc[pyarglist til 10]      / error if too many args specified
       ^
q)qfunc[`a pykw 1;pyarglist 2 2 2]  / error if positional list after keyword args
'TypeError("func() got multiple values for argument 'a'")
  [0]  qfunc[`a pykw 1;pyarglist 2 2 2] 
       ^

You can specify a dictionary of keyword arguments by using pykwargs (similar to Python’s **kwargs). If present, this argument must be the last argument.

q)qfunc[pykwargs`d`c`b`a!1 2 3 4]             / full keyword dict specified
4 3 2 1 24
q)qfunc[2;2;pykwargs`d`c!3 3]                 / mix of positional args and keyword dict
2 2 3 3 36
q)qfunc[`d pykw 1;`c pykw 2;pykwargs`a`b!3 4] / mix of keyword args and keyword dict
3 4 2 1 24
q)qfunc[pykwargs`d`c!3 3;2;2]                 / error if keyword dict not last
'pykwargs last
q)qfunc[pykwargs`a`a!1 2]                     / error if duplicate keyword names
'dupnames

You can combine all four methods in a single function call if the order follows the above rules.

q)qfunc[4;pyarglist enlist 3;`c pykw 2;pykwargs enlist[`d]!enlist 1]
4 3 2 1 24

pykw, pykwargs, and pyarglist

Before defining functions containing pykw, pykwargs, or pyarglist within a script, you must explicitly load the file p.q. Failure to do so results in errors.

Zero-argument calls

In Python these two calls are not equivalent:

func()       #call with no arguments
func(None)   #call with argument None

PyKX function called with :: calls Python with no arguments

Although :: in q corresponds to None in Python, if a PyKX function is called with :: as its only argument, the corresponding Python function will be called with no arguments.

To call a Python function with None as its sole argument, retrieve None as a foreign object in q and pass that as the argument:

q)pynone:.pykx.eval"None"
q)pyfunc:.pykx.eval["print"]
q)pyfunc pynone;
None

Python	Form	q
`func()`	call with no arguments	`func[]` or `func[::]`
`func(None)`	call with argument `None`	`func[.pykx.eval"None"]`

q functions applied to empty argument lists

The rank (number of arguments) of a q function is determined by its signature, an optional list of arguments at the beginning of its definition. If the signature is omitted, the default arguments are as many of x, y and z as appear, and its rank is 1, 2, or 3.

If it has no signature, and does not refer to x, y, or z, it has rank 1. It is implicitly unary. If it is then applied to an empty argument list, the value of x defaults to (::).

So func[::] is equivalent to func[] – and in Python to func(), not func[None].

Print or return

.pykx.repr returns the string representation of a Python object, either PyKX or foreign. You can print this representation to stdout by using .pykx.print. Here's how to use this function with a q object:

q)x:.pykx.eval"{'a':1,'b':2}"
q).pykx.repr x
"{'a': 1, 'b': 2}"
q).pykx.print x
{'a': 1, 'b': 2}

q).pykx.repr ([]5?1f;5?1f)
"x         x1       \n-------------------\n0.3017723 0.3927524\n0.785033  0.5..
q).pykx.print ([]5?1f;5?1f)
x         x1        
--------------------
0.6137452 0.4931835 
0.5294808 0.5785203 
0.6916099 0.08388858
0.2296615 0.1959907 
0.6919531 0.375638

Aliases in the root

For convenience, pykx.q defines print in the default namespace of q, as aliases for .pykx.print. To prevent the aliasing of this function, set either:

UNSET_PYKX_GLOBALS as an environment variable.
unsetPyKXGlobals as a command line argument when initializing your q session.

How to use PyKX within q

Introduction

Getting started

Prerequisites

Install

Initialize

How to use the library

Evaluate and Execute Python

Evaluate Python code

Execute Python code

Interact with PyKX objects

Foreign objects

Create PyKX objects

Convert data

None and identity

Get attributes and properties

Set attributes and properties

How to Index

Get methods

PyKX function API

PyKX examples

Function argument types

Set Python variables

Function calls

Examples

Zero-argument calls

Print or return

Aliases in the root

`None` and identity