Skip to content

Parallel Processing in KDB.AI

This page explores two primary methods of parallel processing: worker processes and multithreading.

Parallel processing allows KDB.AI to execute multiple operations concurrently, improving throughput for supported workloads.

What is a worker?

A worker is an independent KDB.AI process that executes operations in parallel. Workers perform tasks such as data ingestion, similarity search, queries, and other system operations. Each worker can use multiple threads to perform work more efficiently.

KDB.AI supports two worker types:

  • General workers perform compute-intensive operations, including data ingestion, similarity search, queries, and other user workloads.
  • Internal workers perform lightweight internal system operations, such as administrative requests.

Multi-worker setup

A multi-worker configuration allows KDB.AI to process more operations concurrently. Configure the number of workers when starting KDB.AI Server. Configure the number of threads for each worker based on your workload. You can also add and remove workers in a running deployment to adjust processing capacity without recreating databases or restarting the server.

Key considerations

When configuring workers and threads, keep the following in mind:

  1. Thread contention

    • Base the number of workers and threads on user concurrency requirements.
    • As a general rule:
    number_of_workers * number_of_threads <= number_of_cores

  2. Memory utilization

    • Each worker performs tasks such as data ingestion and database loading, increasing overall memory utilization.

  3. Scaling workers

    • Adding workers can improve throughput for concurrent workloads by allowing KDB.AI to process more operations in parallel.
    • Additional workers increase CPU and memory usage.
    • Choose the number of workers based on your workload and available system resources.

Configure the number of workers

Use the NUM_WRK environment variable to specify the number of workers when starting KDB.AI Server.

Syntax

NUM_WRK=<number_of_workers>

Example

Set the number of workers when starting KDB.AI Server:

# On the `docker run` command:
-e NUM_WRK=1

If not specified, NUM_WRK defaults to 1.

Managing workers

You can dynamically manage workers in a running deployment to adjust processing capacity.

  • Add general workers to increase capacity for ingestion, search, query, and other heavy operations.
  • Add internal workers for internal system operations.
  • Remove workers when additional processing capacity is no longer required.
  • Use getProcessInfo to view running workers and retrieve worker IDs.

License requirement

Adding workers requires a k4 license. Refer to Workers for details.

For step-by-step instructions, refer to Manage workers.

Common multi-worker use cases

A multi-worker configuration is beneficial for workloads that can be processed concurrently, such as:

  1. Parallel inserts to different tables

    • Insert data into different tables in parallel.

    Note

    Multi-worker setups do not support parallel inserts into the same table. If multiple insert requests target the same table, KDB.AI uses a single worker to process all inserts.

  2. Parallel searches and queries

    • Run search and query operations in parallel across different databases or within the same database.

Multithreading

Multithreading allows a worker to execute multiple tasks concurrently, improving performance for supported operations. In KDB.AI, use the THREADS environment variable to configure the number of threads available to each worker.

What is a multithreaded operation?

A multithreaded operation divides work into smaller tasks that execute concurrently across multiple threads. This can significantly reduce processing time for large datasets and computationally intensive operations.

Configure the number of threads

Use the THREADS environment variable to specify the number of threads each worker uses during multithreaded operations.

Syntax

THREADS=<number_of_threads>

Example

Set the number of threads when starting KDB.AI Server.

# On the `docker run` command:
-e THREADS="8"

We recommend setting THREADS to the number of CPU cores available on the machine running KDB.AI Server. If not specified, the variable is not passed to the q process, resulting in a single-threaded q process.

Benefits of multithreading

  • Improved performance: Configure THREADS to improve the performance of supported multithreaded operations.
  • Flexibility: Adjust the number of threads to match your server resources and workload requirements.

Common multithreading use cases

Multithreading can improve the performance of:

Next steps

Now that you're familiar with parallel processing, you can: