Multi-Threaded Execution

Warning

This module is a Beta Feature and is subject to change. To enable this functionality for testing please follow the configuration instructions here setting PYKX_BETA_FEATURES='true' and PYKX_THREADING='true'.

Introduction

One major limitation of EmbeddedQ when using python with multi-threading is that only the main thread (the thread that imports PyKX and loads libq) is allowed to modify state within EmbeddedQ. However if you wanted to use one of Pythons multi-threading libraries whether that is the threading library or asyncio or any other library that allows Python to utilise multiple threads at once, and have those threads modify state in some way; whether that be to upsert a row to a global table, open QConnection instances or any other use case that requires the threads to modify state. You would not be able to do that by default in PyKX.

This beta feature allows these use cases to become possible by spawning a background thread that all calls into EmbeddedQ will be run on. This background thread is created at the C level using libpthread with lightweight future objects to ensure the lowest overhead possible for passing calls onto a secondary thread. This allows multi-threaded programs to modify state within the spawned threads safely, without losing out on performance.

Note

While using PyKX Threading it is not possible to also use the functionality within pykx.q, it is also not possible to have q call back into Python.

How to enable

This beta feature requires an extra opt-in step. While the overhead for offloading calls onto a secondary thread is low, there will always be a cost to forcing a thread context switch to process a call into EmbeddedQ. Therefore you will need to enable both the PYKX_BETA_FEATURES environment variable as well as the PYKX_THREADING environment variable.

Warning

Because using PyKX Threading spawns a background thread to run all queries to EmbeddedQ, you must ensure that you call kx.shutdown_thread() at the end of your script to ensure that this background thread is properly shutdown at the end. If you fail to do this the background thread will be left running after the script is finished. The best way to ensure this always happens is to start a main function for your script within a try - finally block.

import os
import asyncio
os.environ['PYKX_THREADING'] = '1'
os.environ['PYKX_BETA_FEATURES'] = '1'
import pykx as kx

def main(): # Your scripts entry point
    ...

if __name__ == '__main__':
    try:
        main()
    finally:
        kx.shutdown_thread() # This will be called if the script completes normally or errors early

More complete examples

More examples showing this functionality in use can be found here.