Skip to content

Parallel Processing in KDB.AI

This page explores two primary methods of parallel processing: worker processes and multithreading.

Parallel processing allows KDB.AI to execute multiple operations concurrently, improving throughput for supported workloads.

What is a worker?

A worker is an independent KDB.AI process that executes operations in parallel. Workers perform tasks such as data ingestion, similarity search, queries, and other system operations. Each worker can use multiple threads to perform work more efficiently.

KDB.AI supports two worker types:

  • General workers perform compute-intensive operations, including data ingestion, similarity search, queries, and other user workloads.
  • Internal workers perform lightweight internal system operations, such as administrative requests.

Multi-worker setup

A multi-worker configuration allows KDB.AI to process more operations concurrently. Configure the number of workers when starting KDB.AI Server. Configure the number of threads for each worker based on your workload. You can also add and remove workers in a running deployment to adjust processing capacity without recreating databases or restarting the server.

Key considerations

When configuring workers and threads, keep the following in mind:

  1. Thread contention

    • Base the number of workers and threads on user concurrency requirements.
    • As a general rule:
    number_of_workers * number_of_threads <= number_of_cores

  2. Memory utilization

    • Each worker performs tasks such as data ingestion and database loading, increasing overall memory utilization.
    • Each active worker can load its own copy of the resident index state, so fixed index memory scales with the number of workers actually handling load, not just NUM_WRK's configured value. Size RAM for every worker you expect to be simultaneously active, plus process and page-cache headroom.
    • A worker only counts as "active" once it receives a request. Send requests sequentially, one at a time, and they're typically routed to a single worker throughout, so the other configured workers never get exercised and their memory cost never shows up. Confirm with overlapping or concurrent requests, and inspect each worker process, before concluding a multi-worker setup has no effect.

  3. Scaling workers

    • Adding workers can improve throughput for concurrent workloads by allowing KDB.AI to process more operations in parallel.
    • Additional workers increase CPU and memory usage.
    • Choose the number of workers based on your workload and available system resources.

Configure the number of workers

Use the NUM_WRK environment variable to specify the number of workers when starting KDB.AI Server.

Syntax

NUM_WRK=<number_of_workers>

Example

Set the number of workers when starting KDB.AI Server:

# On the `docker run` command:
-e NUM_WRK=1

If not specified, NUM_WRK defaults to 1.

Managing workers

You can dynamically manage workers in a running deployment to adjust processing capacity.

  • Add general workers to increase capacity for ingestion, search, query, and other heavy operations.
  • Add internal workers for internal system operations.
  • Remove workers when additional processing capacity is no longer required.
  • Use getProcessInfo to view running workers and retrieve worker IDs.

License requirement

Adding workers requires a k4 license. Refer to Workers for details.

For step-by-step instructions, refer to Manage workers.

Common multi-worker use cases

A multi-worker configuration is beneficial for workloads that can be processed concurrently, such as:

  1. Parallel inserts to different tables

    • Insert data into different tables in parallel.

    Note

    Multi-worker setups do not support parallel inserts into the same table. If multiple insert requests target the same table, KDB.AI uses a single worker to process all inserts.

  2. Parallel searches and queries

    • Run search and query operations in parallel across different databases or within the same database.

Multithreading

Multithreading allows a worker to execute multiple tasks concurrently, improving performance for supported operations. In KDB.AI, use the THREADS environment variable to configure the number of threads available to each worker.

What is a multithreaded operation?

A multithreaded operation divides work into smaller tasks that execute concurrently across multiple threads. This can significantly reduce processing time for large datasets and computationally intensive operations.

Configure the number of threads

Use the THREADS environment variable to specify the number of threads each worker uses during multithreaded operations.

Syntax

THREADS=<number_of_threads>

Example

Set the number of threads when starting KDB.AI Server.

# On the `docker run` command:
-e THREADS="8"

We recommend setting THREADS to the number of CPU cores available on the machine running KDB.AI Server. If not specified, the variable is not passed to the q process, resulting in a single-threaded q process.

Benefits of multithreading

  • Improved performance: Configure THREADS to improve the performance of supported multithreaded operations.
  • Flexibility: Adjust the number of threads to match your server resources and workload requirements.

Common multithreading use cases

Multithreading can improve the performance of:

Core budget is a ceiling, not a target

Treat number_of_workers * number_of_threads <= number_of_cores as a limit on contention, not a target to hit. Pushing either value up to that limit doesn't guarantee the best throughput. The best split is workload-specific, and throughput can plateau or regress past a certain point as you add threads, for a given index and machine. Compare your intended split against the nearest smaller and larger split with a warmed, representative workload before committing to it.

Next steps

Now that you're familiar with parallel processing, you can: