embedPy¶

Fusion for kdb+

Allows the kdb+ interpreter to manipulate Python objects and call Python functions.

Requirements¶

kdb+ ≥3.5 64-bit
Python ≥3.5

Build and install¶

Download the code from KxSystems/embedPy and follow installation instructions there.

Back-incompatible changes¶

V0.2-beta -> V1.0¶

Attribute access from embedPy object


q)obj`ATTRNAME     / old
q)obj`:ATTRNAME    / new

embedPy objects can be called directly without explicitly specifying the call return type, the default return type is an embedPy object

V0.1-beta -> V0.2beta in V0.2-beta¶

V0.2-beta features a number of changes back-incompatible with the previous release, V0.1-beta.

Most notably, the default type used in many operations is now the embedPy type, rather than the foreign type. Differences between these types (and the associated APIs) are set out below.

Running Python code¶

Run the examples¶

To run the following examples, load p.q.

q)\l p.q

Execute Python code¶

The interface allows execution of Python code directly in a q console or from a script. In both console and scripts, prefix Python code with p).

q)p)print(1+2)
3

Q scripts (but not the console) can load and execute multiline Python code. Prefix the first line of the code with p) and indent subsequent lines of Python code according to the usual Python indentation rules.

$ cat embedPytest.q
a:1                   / q code
p)def add1(arg1):     # Python code
    return arg1+1     # still Python code

In a q session

q)\l embedPytest.q
q)p)print(add1(12))
13

Full scripts of Python code can be executed in q, using the .p file extension (not .py). The script is loaded as usual.

$ cat helloq.p 
print("Hello q!")

q)\l helloq.p
Hello q!

Evaluate Python code¶

To evaluate Python code (as a string) and return results to q, use .p.qeval.

q).p.qeval"1+2"
3

Side effects Python evaluation (unlike Python execution) does not allow side effects. Any attempt at variable assignment or class definition will signal an error. To execute a string performing side effects, use .p.e. A more detailed explanation of the difference between eval and exec in Python can be found here.

EmbedPy objects¶

Foreign objects¶

At the lowest level, Python objects are represented in q as foreign objects, which contain pointers to objects in the Python memory space.

Foreign objects can be stored in variables just like any other q datatype, or as part of lists, dictionaries or tables. They will display as foreign when inspected in the q console or using the string (or .Q.s) representation.

Serialization Kdb+ cannot serialize foreign objects, nor send them over IPC: they live in the embedded Python memory space. To pass these objects over IPC, first convert them to q.

EmbedPy objects¶

Foreign objects cannot be directly operated on in q. Instead, Python objects are typically represented as embedPy objects, which wrap the underlying foreign objects. This provides the ability to get and set attributes, index, call or convert the underlying foreign object to a q.

Use .p.wrap to create an embedPy object from a foreign object.

q)x
foreign
q)p:.p.wrap x
q)p           /how an embedPy object looks
{[f;x]embedPy[f;x]}[foreign]enlist

More commonly, embedPy objects are retrieved directly from Python using one of the following functions:

function	argument	example
`.p.import`	symbol: name of a Python module or package, optional second argument is the name of an object within the module or package	np:.p.import`numpy
`.p.get`	symbol: name of a Python variable in `__main__`	v:.p.get`varName
`.p.eval`	string: Python code to evaluate	`x:.p.eval"1+1"`

Side effects As with other Python evaluation functions, .p.eval does not permit side effects.

Converting data¶

Given obj, an embedPy object representing Python data, we can get the underlying data (as foreign or q) using

obj`. / get data as foreign
obj`  / get data as q

e.g.

q)x:.p.eval"(1,2,3)"
q)x
{[f;x]embedPy[f;x]}[foreign]enlist
q)x`.
foreign
q)x`
1 2 3

`None` and identity¶

Python None maps to the q identity function :: when converting from Python to q (and vice versa).

There is one important exception to this. When calling Python functions, methods or classes with a single q data argument, passing :: will result in the Python object being called with no arguments, rather than a single argument of None. See the section below on Zero-argument calls for how to explicitly call a Python callable with a single None argument.

Getting attributes and properties¶

Given obj, an embedPy object representing a Python object, we can get an attribute or property directly using

obj`:attr         / equivalent to obj.attr in Python
obj`:attr1.attr2  / equivalent to obj.attr1.attr2 in Python

These expressions return embedPy objects, allowing users to chain operations together.

obj[`:attr1]`:attr2  / equivalent to obj.attr1.attr2 in Python

e.g.

$ cat class.p 
class obj:
    def __init__(self,x=0,y=0):
        self.x = x
        self.y = y

q)\l class.p
q)obj:.p.eval"obj(2,3)"
q)obj[`:x]`
2
q)obj[`:y]`
3

Setting attributes and properties¶

Given obj, an embedPy object representing a Python object, we can set an attribute or property directly using

obj[:;`:attr;val]  / equivalent to obj.attr=val in Python

e.g.

q)obj[`:x]`
2
q)obj[`:y]`
3
q)obj[:;`:x;10]
q)obj[:;`:y;20]
q)obj[`:x]`
10
q)obj[`:y]`
20

Indexing¶

Given lst, an embedPy object representing an indexable container object in Python, we can access the element at index i using

lst[@;i]    / equivalent to lst[i] in Python

We can set the element at index i (to object x) using

lst[=;i;x]  / equivalent to lst[i]=x in Python

These expressions return embedPy objects, e.g.

q)lst:.p.eval"[True,2,3.0,'four']"
q)lst[@;0]`
1b
q)lst[@;-1]`
"four"
q)lst'[@;;`]2 1 0 3
3f
2
1b
"four"
q)lst[=;0;0b]
q)lst[=;-1;`last]
q)lst`
0b
2
3f
"last"

Getting methods¶

Given obj, an embedPy object representing a Python object, we can access a method directly using

obj`:method  / equivalent to obj.method in Python

This will return an embedPy object, calling this object is described below.

Function calls¶

EmbedPy objects representing callable Python functions or methods are callable by default with an embedPy return. They can be declared callable in q returning q or foreign using.

.p.qcallable (declare callable with q return)
.p.pycallable (declare callable with foreign return)

The result of each of these functions represents the same underlying Python function or method, but now callable in q, e.g.

q)np:.p.import`numpy
q)np`:arange
{[f;x]embedPy[f;x]}[foreign]enlist
q)arange:np`:arange                   / callable returning embedPy
q)arange 12
{[f;x]embedPy[f;x]}[foreign]enlist
q)arange[12]`
0 1 2 3 4 5 6 7 8 9 10 11
q)arange_py:.p.pycallable np`:arange / callable returning foreign
q)arange_py 12
foreign
q)arange_q:.p.qcallable np`:arange   / callable returning q
q)arange_q 12
0 1 2 3 4 5 6 7 8 9 10 11

embedPy function API¶

Using the function API, embedPy objects can be called directly (returning embedPy) or declared callable returning q or foreign data.

Users explicitly specify the return type as q or foreign, the default is embedPy.
Given func, an embedPy object representing a callable Python function or method, we can carry out the following operations:

func                   / func is callable by default (returning embedPy)
func arg               / call func(arg) (returning embedPy)

func[<]                / declare func callable (returning q)
func[<]arg             / call func(arg) (returning q)
func[<;arg]            / equivalent

func[>]                / declare func callable (returning foreign)
func[>]arg             / call func(arg) (returning foreign)
func[>;arg]            / equivalent

Chaining operations Returning another embedPy object from a function or method call, allows users to chain together sequences of operations. We can also chain these operations together with calls to .p.import, .p.get and .p.eval.

embedPy examples¶

Some examples

$ cat test.p # used for tests
class obj:
    def __init__(self,x=0,y=0):
        self.x = x # attribute
        self.y = y # property (incrementing on get)
    @property
    def y(self):
        a=self.__y
        self.__y+=1
        return a
    @y.setter
    def y(self, y):
        self.__y = y
    def total(self):
        return self.x + self.y

q)\l test.p
q)obj:.p.get`obj / obj is the *class* not an instance of the class
q)o:obj[]        / call obj with no arguments to get an instance
q)o[`:x]`
0
q)o[;`]each 5#`:x
0 0 0 0 0
q)o[:;`:x;10]
q)o[`:x]`
10
q)o[`:y]`
0
q)o[;`]each 5#`:y
1 2 3 4 5
q)o[:;`:y;10]
q)o[;`]each 5#`:y
10 11 12 13 14
q)tot:.p.qcallable o`:total
q)tot[]
25
q)tot[]
26

q)np:.p.import`numpy
q)v:np[`:arange;12]
q)v`
0 1 2 3 4 5 6 7 8 9 10 11
q)v[`:mean;<][]
5.5
q)rs:v[`:reshape;<]
q)rs[3;4]
0 1 2  3 
4 5 6  7 
8 9 10 11
q)rs[2;6]
0 1 2 3 4  5 
6 7 8 9 10 11
q)np[`:arange;12][`:reshape;3;4]`
0 1 2  3 
4 5 6  7 
8 9 10 11
q)np[`:arange;12][`:reshape;3;4][`:T]`
0 4 8 
1 5 9 
2 6 10
3 7 11

q)stdout:.p.import[`sys]`:stdout.write
q)stdout"hello\n";
hello
q)stderr:.p.import[`sys;`:stderr.write]
q)stderr"goodbye\n";
goodbye

q)oarg:.p.eval"10"
q)oarg`
10
q)ofunc:.p.eval["lambda x:2+x";<]
q)ofunc 1
3
q)ofunc oarg
12
q)p)def add2(x,y):return x+y
q)add2:.p.get[`add2;<]
q)add2[1;oarg]
11

Setting Python variables¶

Variables can be set in Python __main__ using .p.set

q).p.set[`var1;42]
q).p.qeval"var1"
42

Function calls¶

Python allows for calling functions with

A variable number of arguments
A mixture of positional and keyword arguments
Implicit (default) arguments

All of these features are available through the embedPy function-call interface.
Specifically:

Callable embedPy objects are variadic
Default arguments are applied where no explicit arguments are given
Individual keyword arguments are specified using the (infix) pykw operator
A list of positional arguments can be passed using pyarglist (like Python *args)
A dictionary of keyword arguments can be passed using pykwargs (like Python **kwargs)

Keyword arguments last We can combine positional arguments, lists of positional arguments, keyword arguments and a dictionary of keyword arguments. However, all keyword arguments must always follow any positional arguments. The dictionary of keyword arguments (if given) must be specified last.

Example function calls¶

q)p)def func(a=1,b=2,c=3,d=4):return (a,b,c,d,a*b*c*d)
q)qfunc:.p.get[`func;<] / callable, returning q

Positional arguments are entered directly.
Function calling is variadic, so later arguments can be excluded.

q)qfunc[2;2;2;2]   / all positional args specified
2 2 2 2 16
q)qfunc[2;2]       / first 2 positional args specified
2 2 3 4 48
q)qfunc[]          / no args specified
1 2 3 4 24
q)qfunc[2;2;2;2;2] / error if too many args specified
TypeError: func() takes from 0 to 4 positional arguments but 5 were given
'p.c:73 call pyerr

Individual keyword arguments can be specified using the pykw operator (applied infix).
Any keyword arguments must follow positional arguments, but the order of keyword arguments does not matter.

q)qfunc[`d pykw 1;`c pykw 2;`b pykw 3;`a pykw 4] / all keyword args specified
4 3 2 1 24
q)qfunc[1;2;`d pykw 3;`c pykw 4]   / mix of positional and keyword args
1 2 4 3 24
q)qfunc[`a pykw 2;`b pykw 2;2;2]   / error if positional args after keyword args
'keywords last
q)qfunc[`a pykw 2;`a pykw 2]       / error if duplicate keyword args
'dupnames

A list of positional arguments can be specified using pyarglist (similar to Python’s *args).
Again, keyword arguments must follow positional arguments.

q)qfunc[pyarglist 1 1 1 1]          / full positional list specified
1 1 1 1 1
q)qfunc[pyarglist 1 1]              / partial positional list specified
1 1 3 4 12
q)qfunc[1;1;pyarglist 2 2]          / mix of positional args and positional list
1 1 2 2 4
q)qfunc[pyarglist 1 1;`d pykw 5]    / mix of positional list and keyword args
1 1 3 5 15
q)qfunc[pyarglist til 10]           / error if too many args specified
TypeError: func() takes from 0 to 4 positional arguments but 10 were given
'p.c:73 call pyerr
q)qfunc[`a pykw 1;pyarglist 2 2 2]  / error if positional list after keyword args
'keywords last

A dictionary of keyword arguments can be specified using pykwargs (similar to Python’s **kwargs).
If present, this argument must be the last argument specified.

q)qfunc[pykwargs`d`c`b`a!1 2 3 4]             / full keyword dict specified
4 3 2 1 24
q)qfunc[2;2;pykwargs`d`c!3 3]                 / mix of positional args and keyword dict
2 2 3 3 36
q)qfunc[`d pykw 1;`c pykw 2;pykwargs`a`b!3 4] / mix of keyword args and keyword dict
3 4 2 1 24
q)qfunc[pykwargs`d`c!3 3;2;2]                 / error if keyword dict not last   
'pykwargs last
q)qfunc[pykwargs`a`a!1 2]                     / error if duplicate keyword names
'dupnames

All 4 methods can be combined in a single function call, as long as the order follows the above rules.

q)qfunc[4;pyarglist enlist 3;`c pykw 2;pykwargs enlist[`d]!enlist 1]    
4 3 2 1 24

Zero-argument calls¶

In Python these two calls are not equivalent:

func()       #call with no arguments
func(None)   #call with argument None

EmbedPy function called with :: calls Python with no arguments

Although :: in q corresponds to None in Python, if an embedPy function is called with :: as its only argument, the corresponding Python function will be called with no arguments.

To call a Python function with None as its sole argument, retrieve None as a foreign object in q and pass that as the argument.

q)pynone:.p.eval"None"
q).p.eval["print";pynone];
None

python	form	q
`func()`	call with no arguments	`func[]` or `func[::]`
`func(None)`	call with argument `None`	`func[.p.eval"None"]`

Q functions applied to empty argument lists

The rank (number of arguments) of a q function is determined by its signature, an optional list of arguments at the beginning of its definition. If the signature is omitted, the default arguments are as many of x, y and z as appear, and its rank is 1, 2, or 3.

If it has no signature, and does not refer to x, y, or z, it has rank 1. It is implicitly unary. If it is then applied to an empty argument list, the value of x defaults to (::).

So func[::] is equivalent to func[] – and in Python to func(), not func[None].

Dictionary keys and values¶

Python dictionaries convert to q dictionaries, and vice versa.

q)p)pyd={'one':1,'two':2,'three':3}
q)qd:.p.get`pyd
q)qd`
one  | 1
two  | 2
three| 3
q).p.eval["print"]qd;
{'one': 1, 'two': 2, 'three': 3}

Functions are also provided to retrieve the keys and values directly from an embedPy dictionary, without performing the conversion to a q dictionary.

.p.key returns the keys
.p.value returns the values

In each case, the result is an embedPy object.

q).p.key[qd]`
"one"
"two"
"three"
q).p.value[qd]`
1 2 3

Printing and help¶

.p.repr returns the string representation of a Python object, embedPy or foreign. This representation can be printed to stdout using .p.print.

q)x:.p.eval"{'a':1,'b':2}"
q).p.repr x
"{'a': 1, 'b': 2}"
q).p.print x
{'a': 1, 'b': 2}

.p.helpstr returns the string representation of Python’s help for a Python object, embedPy or foreign. Python interactive help for an object is accessed using .p.help.

q)n:.p.eval"42"
q).p.helpstr n
"int(x=0) -> integer\nint(x, base=10) -> integer\n\nConvert a number or strin..
q).p.help n / interactive help

Aliases in the root¶

For convenience, p.q defines print and help in the default namespace of q, as aliases for .p.print and .p.help. To prevent this, comment out the relevant code in p.q before loading.

{@[`.;x;:;get x]}each`help`print; / comment to remove from global namespace

Closures and generators¶

Closures¶

Closures allow us to define functions that retain state between successive calls, avoiding the need for global variables.

To create a closure in embedPy, we must:

Define a function in q with
- 2+ arguments: the current state and at least one other (possibly dummy) argument
- 2 return values: the new state and the return value
Wrap the function using .p.closure, which takes 2 arguments:
- the q function
- the initial state

Functions without arguments The dummy argument is needed if we want the resulting function to take no arguments.

Example 1: `til`¶

We can define a closure to return incrementing natural numbers, similar to the q til function.

The state x is the last value returned

q)xtil:{[x;dummy]x,x+:1}

Create the closure with initial state -1, so the first value returned will be 0

q)ftil:.p.closure[xtil;-1][<]
q)ftil[]
0
q)ftil[]
1
q)ftil[]
2
q)ftil[]
3

Example 2: Fibonacci¶

The Fibonacci sequence is a sequence in which each number is the sum of the two numbers preceding it.
Starting with 0 and 1, the sequence goes x(n) = x(n-1) + x(n-2)

i.e. 0, 1, 1, 2, 3, 5, 8, 13, 21, 34, …

The state x will be the last two values produced.

q)xfib:{[x;dummy](x[1],r;r:sum x)}

Create the closure with initial state 0 1, so the first value produced will be 1.

q)fib:.p.closure[xfib;0 1][<]
q)fib[]
1
q)fib[]
2
q)fib[]
3
q)fib[]
5
q)fib[]
8
q)fib[]
13

Example 3: Running sum¶

In this example, we will allow a numeric argument to be passed to the closure, removing the need for a dummy argument. The closure will keep track of all arguments passed so far, and return a running sum.

The state x will be the total so far.

q)xrunsum:{x,x+:y}

Create the closure with initial state 0, so the first value produced will be the first argument passed.

q)runsum:.p.closure[xrunsum;0][<]
q)runsum 2
2
q)runsum 3
5
q)runsum -2.5
2.5
q)runsum 0
2.5
q)runsum 10
12.5

Generators¶

Generators are objects that we can iterate over (e.g. in a for-loop) to produce sequences of values.
EmbedPy allows us to produce generators for use in Python functions and statements where they are required.

To create a generator in embedPy, we must

Define a function in q (as per closures) with:
- 2 arguments - the current state and a dummy argument
- 2 return values - the new state and the return value
Wrap the function using .p.generator, which takes 3 arguments:
- the q function
- the initial state
- the max number of iterations, or :: to run indefinitely

Example 1: Factorials¶

The factorial ( $n!$ ) of a non-negative integer $n$ , is the product of all positive integers less than or equal to $n$ .

We can create a sequence of factorials (starting with 1), with the sequence x(n) = x(n-1) * n

The state x will be a 2-item list comprising

the last value used in the product
the last value returned

q)xfact:{[x;dummy](x;last x:prds x+1 0)}

Create two generators, each with initial state 0 1.

q)fact4:.p.generator[xfact;0 1;4]     / generates first 4 factorial values
q)factinf:.p.generator[xfact;0 1;::]  / generates factorial values indefinitely

The resulting generators can be used as iterators in Python.

q).p.set[`fact4]fact4
q)p)for x in fact4:print(x)
1
2
6
24
q).p.set[`factinf]factinf
q).p.e"for x in factinf:\n print(x)\n if x>1000:break"  / force break to stop iteration
1
2
6
24
120
720
5040

Example 2: Look-and-say¶

The look-and-say sequence is the sequence of integers beginning as follows: 1, 11, 21, 1211, 111221, 312211, 13112221, 1113213211

1 is read off as “one 1” or 11.
11 is read off as “two 1s” or 21.
21 is read off as “one 2, then one 1” or 1211.
1211 is read off as “one 1, one 2, then two 1s” or 111221

The state x will be the last value produced.

q)xlook:{[x;dummy]r,r:"J"$raze string[count each s],'first each s:(where differ s)_s:string x}

Create a generator (to run for 7 iterations) with initial state 1, so the first value produced will be 11.

q)look:.p.generator[xlook;1;7]

This can be used as an iterator in Python.

q).p.set[`look]look
q)p)for x in look:print(x)
11
21
1211
111221
312211
13112221
1113213211

Example 3: Successive sublists¶

We can define a closure to extract successive sublists, of a given size, from a larger list.

The state x will be a 3-item list comprising

the list
the start index
the sublist size

q)xsub:{[x;d](@[x;1;+;x 2];sublist[x 1 2]x 0)}

To create a generator (to run for 6 iterations), extracting sublists of size 6 from .Q.A (list of 26 alphabetical chars)

 q)sub:.p.generator[xsub;(.Q.A;0;6);6]

This can be used as an iterator in Python.

q).p.set[`sub]sub
q)p)for x in sub:print(x)
ABCDEF
GHIJKL
MNOPQR
STUVWX
YZ

q)

embedPy¶

Requirements¶

Build and install¶

Back-incompatible changes¶

V0.2-beta -> V1.0¶

V0.1-beta -> V0.2beta in V0.2-beta¶

Running Python code¶

Run the examples¶

Execute Python code¶

Evaluate Python code¶

EmbedPy objects¶

Foreign objects¶

EmbedPy objects¶

Converting data¶

None and identity¶

Getting attributes and properties¶

Setting attributes and properties¶

Indexing¶

Getting methods¶

Function calls¶

embedPy function API¶

embedPy examples¶

Setting Python variables¶

Function calls¶

Example function calls¶

Zero-argument calls¶

Dictionary keys and values¶

Printing and help¶

Aliases in the root¶

Closures and generators¶

Closures¶

Example 1: til¶

Example 2: Fibonacci¶

Example 3: Running sum¶

Generators¶

Example 1: Factorials¶

Example 2: Look-and-say¶

Example 3: Successive sublists¶

`None` and identity¶

Example 1: `til`¶