Jupyter Notebooks¶

This notebook demonstrates how to use the q Magic command in a Jupyter notebook.

The Jupyter q magic command in PyKX allows you to execute q code within a Jupyter notebook. It provides seamless integration with the q programming language.

This example Notebook has the following sections:

1. Import PyKX
2. Create the external q process
3. Execute against Embedded q
4. SQL interface
5. q namespaces
6. (Advanced) q over IPC

1. Import PyKX¶

To run this example, first import the PyKX library:

In [2]:

import pykx as kx

import pykx as kx

2. Create the external q process¶

You can run an external q process by using the following Python code:

In [3]:

import subprocess
import time

try:
    with kx.PyKXReimport():
        proc = subprocess.Popen(
            ('q', '-p', '5000')
        )
    time.sleep(2)
except:
    raise kx.QError('Unable to create q process on port 5000')

import subprocess import time try: with kx.PyKXReimport(): proc = subprocess.Popen( ('q', '-p', '5000') ) time.sleep(2) except: raise kx.QError('Unable to create q process on port 5000')

Or execute this command in a terminal:

$ q -p 5000

3. Execute against Embedded q¶

To execute q code within PyKX's EmbeddedQ module, type %%q at the beginning of the cell:

In [4]:

%%q
til 10

%%q til 10

0 1 2 3 4 5 6 7 8 9

After %%q you can further add two execution options:

Execution option	Description
--debug	Prints the q backtrace before raising a QError if the cell gives an error.
--display	Calls display rather than the default print on returned objects.

In [5]:

%%q
([] a: 1 2 3)

%%q ([] a: 1 2 3)

a
-
1
2
3

In [6]:

%%q --display
([] a: 1 2 3)

%%q --display ([] a: 1 2 3)

	a
0	1
1	2
2	3

Executing against an external q process over IPC¶

Connection information can also be included after the %%q to connect to a remote q process over IPC.

Here is the list of currently supported connection parameters. If they specify a type a second value is expected to follow them to be used as the parameter. If no type follows them they can be used as a stand alone flag.

--host: A string object denoting the host to connect to
--port: An int object denoting the port to connect over
--user: A str object denoting the username to use when connecting
--password: A str object denoting the password to use when connecting
--timeout: A float object denoting the time in seconds before the query
      times out, defaults to no timeout
--nolarge: Disable messages over 2GB being sent / received
--tls: Use a tls connection
--unix: Use a unix connection
--reconnection_attempts: An int object denoting how many
      reconnection attempts to make
--noctx: Disable the context interface

Connect to a q server running on localhost at port 5001 as user using password password and disable the context interface.

In [7]:

%%q --host localhost --port 5000 --user user --pass password --noctx
til 10

%%q --host localhost --port 5000 --user user --pass password --noctx til 10

0 1 2 3 4 5 6 7 8 9

All connection arguments are optional with the exception of the --port argument. If --host is not provided localhost will be used as the default host.

In [8]:

%%q --port 5000
tab:([]a:1000?1000; b:1000?500.0; c:1000?`AAPL`MSFT`GOOG);

%%q --port 5000 tab:([]a:1000?1000; b:1000?500.0; c:1000?`AAPL`MSFT`GOOG);

It is possible to execute q code spanning multiple lines.

In [9]:

%%q --port 5000
afunc: {[x; y]
  x + y  
  };
afunc[0; 1]
afunc[2; 3]

%%q --port 5000 afunc: {[x; y] x + y }; afunc[0; 1] afunc[2; 3]

1
5

4. SQL interface¶

The s) syntax runs SQL queries against local tables within the q process.

Note: To use the SQL interface, first you need to load the s.k_ library.

In [10]:

%%q
\l s.k_
tab:([]a:1000?1000; b:1000?500.0; c:1000?`AAPL`MSFT`GOOG);
s) select * from tab where a>500 and b<250.0 limit 5

%%q \l s.k_ tab:([]a:1000?1000; b:1000?500.0; c:1000?`AAPL`MSFT`GOOG); s) select * from tab where a>500 and b<250.0 limit 5

a   b        c   
-----------------
916 149.4238 AAPL
898 97.74763 MSFT
900 45.35602 MSFT
677 150.6862 MSFT
741 151.1293 MSFT

5. q namespaces¶

You can use q namespaces, and switch between them with \d.

Note: The namespace is reset back to the base namespace . between cells.

In [11]:

%%q
\d .example
f: {[x] til x};

%%q \d .example f: {[x] til x};

In [12]:

%%q
\d
.example.f[10]

%%q \d .example.f[10]

.example
0 1 2 3 4 5 6 7 8 9

6. (Advanced) q over IPC¶

After %%q you can include connection information, if you wish to connect to a remote q process over IPC.

The list of supported connection parameters is below. The rule is:

• If they have a type, it must be followed by a second value/parameter.
• If there's no type after them, you can use them as a standalone flag.

Parameter	Object type and description
--host	(string) The host to connect to.
--port	(integer) The port to connect over.
--user	(string) The username to use when connecting.
--password	(string) The password to use when connecting.
--timeout	(float) The time in seconds before the query times out. Defaults to no timeout.
--nolarge	Disable messages over 2GB being sent / received.
--tls	Use a tls connection.
--unix	Use a unix connection.
--reconnection_attempts	(integer) How many reconnection attempts to make.
--noctx	Disable the context interface.

Connect to a q server running on localhost at port 5000 as user using password password and disable the context interface.

In [13]:

%%q --host localhost --port 5000 --user user --pass password --noctx
til 10

%%q --host localhost --port 5000 --user user --pass password --noctx til 10

0 1 2 3 4 5 6 7 8 9

All connection arguments are optional, except the --port argument. If --host is not provided localhost is the default host.

In [14]:

%%q --port 5000
tab:([]a:1000?1000; b:1000?500.0; c:1000?`AAPL`MSFT`GOOG);

%%q --port 5000 tab:([]a:1000?1000; b:1000?500.0; c:1000?`AAPL`MSFT`GOOG);

Note that it's possible to execute q code spanning multiple lines:

In [15]:

%%q --port 5000
afunc: {[x; y]
  x + y  
  };
afunc[0; 1]
afunc[2; 3]

%%q --port 5000 afunc: {[x; y] x + y }; afunc[0; 1] afunc[2; 3]

1
5

In [16]:

# Shutdown the q process we were connected to for the IPC demo
proc.kill()

# Shutdown the q process we were connected to for the IPC demo proc.kill()

q first mode¶

q first mode can be enabled by importing PyKX after setting the environment variable PYKX_JUPYTERQ to true, or at runtime use:

In [17]:

kx.util.jupyter_qfirst_enable()

kx.util.jupyter_qfirst_enable()

PyKX now running in 'jupyter_qfirst' mode. All cells by default will be run as q code. 
Include '%%py' at the beginning of each cell to run as python code. 

Once enabled, you can call q code without needing to include %%q at the beginning of a cell.

In [18]:

t:3?15t*3
t

t:3?15t*3 t

43:44:24.793 36:47:38.375 31:48:16.540

In this state, you can execute Python code as well, but those cells must include %%py.

In [19]:

%%py
for fruit in ['apple', 'orange', 'banana']:
    print(fruit)

%%py for fruit in ['apple', 'orange', 'banana']: print(fruit)

apple
orange
banana

If you wish to exit q first mode, simply run the following code and the notebook will revert back to default, Python first execution.

In [20]:

%%py
kx.util.jupyter_qfirst_disable()

%%py kx.util.jupyter_qfirst_disable()

PyKX now running in 'python' mode (default). All cells by default will be run as python code. 
Include '%%q' at the beginning of each cell to run as q code. 

In [21]:

for x in range(3):
    print(x * 1.5)

for x in range(3): print(x * 1.5)

0.0
1.5
3.0

To enable qfirst mode from q, run the following.

In [22]:

%%q
.pykx.enableJupyter()

%%q .pykx.enableJupyter()

PyKX now running in 'jupyter_qfirst' mode. All cells by default will be run as q code. 
Include '%%py' at the beginning of each cell to run as python code. 

And to return to Python first execution run the code below.

In [23]:

.pykx.disableJupyter()

.pykx.disableJupyter()

PyKX now running in 'python' mode (default). All cells by default will be run as python code. 
Include '%%q' at the beginning of each cell to run as q code. 

Saving code blocks¶

The --save feature allows user to save code in a cell as a q file.

To use this feature, include --save followed by the path of the file.

Note: If the q script errors the file will not be saved.

Note: Using --save on an IPC connection cell will save the file on the remote host.

In [24]:

%%q --save ../../new_file.q
vals:til 10
vals * 3

%%q --save ../../new_file.q vals:til 10 vals * 3

0 3 6 9 12 15 18 21 24 27
Cell contents saved to '../../new_file.q'.

If the user wants to save a code block without executing them first, they can include --execute False at beginning of a cell.

Note: Nothing is outputted when the code below is ran.

In [25]:

%%q --save ../../new_file.q --execute False
new_val:3 6 9
new_val

%%q --save ../../new_file.q --execute False new_val:3 6 9 new_val

Cell contents saved to '../../new_file.q' without cell logic being run. To run the cell remove '--execute False'.

File paths that end in .q_ will automatically be created as locked files without the need for any additional input.

In [26]:

%%q --save ../../new_secretfile.q_
pub_vals:til 10
secret_func:{x+7}
secret_func pub_vals

%%q --save ../../new_secretfile.q_ pub_vals:til 10 secret_func:{x+7} secret_func pub_vals

7 8 9 10 11 12 13 14 15 16
Cell contents saved to '../../new_secretfile.q_'.