Skip to content

Troubleshooting Errors

This guide helps you diagnose and resolve errors returned by the KDB-X GPU module. Each error includes its likely causes and step-by-step remediation.

How to use this guide

  1. Locate the error code in the Error index.
  2. Review the summary to confirm it matches your issue.
  3. Check the likely causes to narrow down the problem.
  4. Follow the remediation steps in order.
  5. If unresolved, escalate with logs and a reproducible example.

Error index

Category Error code Summary
Internal GPU_INTERNAL_LOGIC_ERROR Unexpected internal control flow
GPU_SELECT_DEADLOCK Interpreter deadlock detected
GPU_LAUNCH_FAIL GPU kernel launch failed
GPU_CALCULATE_SCRATCH_ERR Scratch memory calculation failed
GPU_Q_ERROR A q C API call signaled an error
GPU_DEPENDENCY_ERR An internal library has thrown an error
Memory GPU_ARENA_MAP_FAILED Host allocator creation failed
GPU_HOST_ALLOC_FAILED Host memory allocation failed
GPU_CUDA_ALLOC_FAILED GPU VRAM allocation failed
GPU_CREATE_GLOBAL_SET Global set creation failed
GPU_CUDA_COPY_FAILED CUDA memory copy failed
Type/parse GPU_PARSE_FAIL Input parsing failed
GPU_PARSE_NOT_IMPLEMENTED Input parsing not implemented
GPU_INVALID_STATEMENT Statement rejected during parsing
GPU_TYPE_UNSUPPORTED Type not supported for operation
GPU_INVALID_DOMAIN Enumeration domain not found
Data and column GPU_SYMBOL_SORTING_NOT_IMPLEMENTED Sorting on symbols or character arrays is not supported
GPU_EXPECTED_FOREIGN Expected a GPU foreign object
GPU_WRONG_DEVICE Object belongs to a different GPU device
GPU_NOT_A_COLUMN Symbol not found in column list
GPU_INVALID_VALUE Input value is not valid for the operation
GPU_TENSOR_SHAPE Tensor shape or layout not supported
GPU_TENSOR_NONFINITE PCA input or intermediate is not finite
GPU_TENSOR_PCA_CONVERGENCE PCA solver did not converge
GPU_TENSOR_PCA_RANK Requested Gram PCA directions are numerically rank deficient
GPU_TENSOR_PCA_SPECTRUM PCA solver returned a spectrum that is not valid
GPU_TENSOR_NOT_POSITIVE_DEFINITE Matrix is not positive definite
GPU_AXIS_MISMATCH Tensor axis names do not align
Other GPU_UNKNOWN Unclassified error
GPU_NOT_IMPLEMENTED Feature not available
GPU_CUDA_CALL_FAILED A CUDA runtime call failed
GPU_CUDA_RUNTIME_MISMATCH CUDA runtime major version differs from build
GPU_COMPUTE_CAPABILITY_MISMATCH Device compute capability differs from build arch

Internal errors

GPU_INTERNAL_LOGIC_ERROR

Summary: Unexpected internal control flow

The GPU module reached a code path that should be unreachable under normal operation. This indicates a bug within the module itself rather than incorrect user input.

Remediation steps

  1. Note the exact query or operation that triggered the error.
  2. Try downgrading to the previous stable release to confirm it is a regression.
  3. Report the issue to KDB-X with a reproducible test case.

This indicates a system-level issue. Escalate to KDB-X support.

GPU_SELECT_DEADLOCK

Summary: Interpreter deadlock detected

The GPU interpreter has reached a deadlock state where two or more operations are waiting on each other and can no longer proceed. This is caused by a bug in the module's concurrency logic and should not occur under normal operation.

Remediation steps

  1. Stop the GPU process.
  2. Upgrade to the latest module version where the bug may be fixed.
  3. Report to KDB-X support - this error should not occur.

This indicates a system-level issue. Escalate to KDB-X support.

GPU_LAUNCH_FAIL

Summary: GPU kernel launch failed

An attempt to launch a CUDA kernel on the GPU device failed. The kernel was not dispatched for execution.

Likely causes

  • CUDA driver or device in an error state
  • Internal kernel configuration that is not valid

Remediation steps

  1. Check nvidia-smi for GPU health and error state.
  2. Run compute-sanitizer to detect device errors.
  3. Reboot the host to reset the CUDA driver and GPU state.
  4. Verify the CUDA toolkit and driver versions are compatible.

GPU_CALCULATE_SCRATCH_ERR

Summary: Scratch memory calculation failed

The module failed to determine how much scratch (temporary working) memory a GPU kernel requires before launching it. This is a pre-launch validation step.

Likely causes

  • A bug in the scratch-size calculation logic for a specific kernel
  • An unsupported parameter combination passed to the calculation routine

Remediation steps

  1. Note the specific operation that triggered this error.
  2. Report to KDB-X support with full context - this error should not occur.

This indicates a system-level issue. Escalate to KDB-X support.

GPU_Q_ERROR

Summary: A q C API call signaled an error

A call into the q C API (for example, building the result table) signaled an error. The GPU stages of the operation completed, but the result could not be handed back to q.

Likely causes

  • q workspace exhaustion (wsfull) while materializing the result
  • A result structure that q rejects, indicating a bug in the module

Remediation steps

  1. Check q's workspace usage against the -w limit and increase it if the result is large.
  2. Reduce the result size (fewer rows or columns) and retry.
  3. If memory is not the issue, report to KDB-X support with a reproducible example.

GPU_DEPENDENCY_ERR

Summary: An internal library has thrown an error

An internal dependency has thrown an unexpected error that the GPU module cannot handle.

Likely causes

  • CUDA or host memory is exhausted.
  • The dependency detects an error originating elsewhere in the codebase.
  • The dependency encounters an internal issue.

Remediation steps

  1. Run the failing workload under compute-sanitizer to collect more diagnostic information.
  2. Report the issue to KDB-X support with a reproducible test case.

Memory errors

GPU_ARENA_MAP_FAILED

Summary: Host allocator creation failed

The module failed to create the memory arena (allocator) used to manage intermediate data structures on the host (CPU-side memory). This is a prerequisite for any subsequent GPU operation.

Likely causes

  • Insufficient host RAM available at the time of the call
  • Memory fragmentation preventing a large contiguous allocation
  • OS-level memory limits (for example, ulimit) being exceeded

Remediation steps

  1. Check host memory usage with free -h or a system monitor.
  2. Stop other memory-heavy processes to free up RAM.
  3. Check ulimit -v and increase virtual memory limits if required.

GPU_HOST_ALLOC_FAILED

Summary: Host memory allocation failed

An attempt to allocate an intermediate data structure on the host (CPU RAM) failed. The allocator was created successfully, but the specific allocation request could not be fulfilled.

Likely causes

  • System RAM is fully exhausted
  • The requested allocation size exceeds available contiguous memory

Remediation steps

  1. Monitor memory usage during the operation with top or htop.
  2. Reduce the size of the input dataset or split it into smaller batches.
  3. Restart the process to clear any leaked allocations.
  4. Add more physical RAM or increase available swap space.

GPU_CUDA_ALLOC_FAILED

Summary: GPU VRAM allocation failed

A CUDA memory allocation call (cudaMalloc or cudaMallocAsync) returned cudaErrorMemoryAllocation, meaning the GPU does not have enough free VRAM to satisfy the request.

Likely causes

  • GPU VRAM is exhausted by the current workload
  • A CUDA module failed to load on a nearly-full device: with lazy module loading a module's device memory is allocated at the first launch of one of its kernels, and after a failed load the process needs a restart before any further kernel can load
  • Other processes are consuming VRAM (for example, display driver or other CUDA applications)
  • Memory fragmentation on the GPU leaving no contiguous free block of sufficient size
  • Requesting more memory than the GPU physically has

Remediation steps

  1. Run nvidia-smi to inspect current VRAM usage.
  2. Stop or pause other GPU-bound processes to free VRAM.
  3. Reduce dataset size.
  4. Switch to a GPU with more VRAM if the workload fundamentally requires it.
  5. Set CUDA_VISIBLE_DEVICES to isolate the process to a less-loaded GPU.

This is the most common error in production GPU workloads. Monitor VRAM headroom proactively.

GPU_CREATE_GLOBAL_SET

Summary: Global set creation failed

The module failed to create a global set, most likely due to a CUDA memory allocation failure on the device.

Likely causes

  • Insufficient GPU VRAM available at the time of the call
  • A preceding cudaMalloc failure that has not been handled
  • GPU device in an error state

Remediation steps

  1. Run nvidia-smi to inspect current VRAM usage.
  2. Stop or pause other GPU-bound processes to free VRAM.
  3. Check for a preceding GPU_CUDA_ALLOC_FAILED error in the same session - resolve that first.
  4. Reboot the host to reset the CUDA driver and GPU state if the device appears unhealthy.

GPU_CUDA_COPY_FAILED

Summary: CUDA memory copy failed

A CUDA memory copy operation failed while gpulib was transferring data between host and GPU memory.

Likely causes

  • The GPU operation failed while copying input data, result data, or internal metadata.
  • The GPU may be out of memory, or the requested operation may be too large for the available device memory.
  • An earlier GPU operation may have failed, but the GPU module reports the error only during a later copy.
  • The GPU context, device, or stream may be in a state that is not valid.

Remediation steps

  1. Retry the operation in a fresh q session.
  2. Reduce the size of the input data and try again.
  3. Check that the selected GPU is available and has enough free memory.
  4. If the problem is reproducible, report it with the full q expression, input sizes, gpulib version, CUDA error message, and GPU model.

Type/parse errors

GPU_PARSE_FAIL

Summary: Input parsing failed

The GPU module was unable to parse the provided input. The input did not conform to the expected format.

Likely causes

  • Malformed parse tree or expression
  • Unexpected parse tree structure

Remediation steps

  1. Validate the query syntax against the KDB-X GPU module documentation.
  2. Simplify the query to isolate which part is causing the parse failure.

GPU_PARSE_NOT_IMPLEMENTED

Summary: Input parsing not implemented

The GPU module recognized the input but parsing for this construct has not yet been implemented.

Likely causes

  • Using a query construct that is planned but not yet supported by the GPU parser
  • A syntax variant that the CPU parser accepts but the GPU parser does not yet handle

Remediation steps

  1. Rewrite the query to avoid the unimplemented construct.
  2. Use the CPU fallback path in the interim.
  3. Contact KDB-X to request prioritization or a workaround.

GPU_INVALID_STATEMENT

Summary: Statement rejected during parsing

The input was parseable but the resulting statement was determined to be semantically incorrect. The statement structure violates the module's rules.

Likely causes

  • Type mismatch between operands in an expression
  • Use of a keyword in an unsupported context
  • Missing required clauses or arguments in a statement

Remediation steps

  1. Break complex statements into smaller parts to locate the construct that is not valid.
  2. Check operand types - ensure all types are compatible with the operation.
  3. Check if the same query works on the CPU with functional selects.

GPU_TYPE_UNSUPPORTED

Summary: Type not supported for operation

The data type of an operand is not supported for the requested GPU operation. The operation exists but does not handle this specific type.

Likely causes

  • Using a data type that has not yet been implemented for this operation on the GPU
  • Mixing types in an operation that requires homogeneous inputs
  • A gather spec or spec item of a kind it does not accept, such as a bare boolean vector, a symbol, a float, a function, or an enumeration written for a one-key range as `step!-1 instead of (enlist `step)!enlist -1

Remediation steps

  1. Check the KDB-X docs for supported types for the operation.
  2. Cast the data to a supported type before passing it to the operation.

GPU_INVALID_DOMAIN

Summary: Enumeration domain not found

The named domain for an enumeration could not be located.

Likely causes

  • Referencing an enum domain that has not been defined
  • Corrupted or incomplete enum metadata

Remediation steps

  1. Verify that the enumeration domain is correctly defined.
  2. Re-register or re-initialize the enum domain if it has been removed.
  3. Validate enum metadata integrity in the underlying data files.

Data and column errors

GPU_SYMBOL_SORTING_NOT_IMPLEMENTED

Summary: Sorting on symbols or character arrays is not supported

The query attempted to sort on a symbol or character array column, which is not supported by the GPU module.

Likely causes

  • Using xasc, xdesc, or an order by clause on a symbol or character array column
  • An implicit sort triggered by a keyed table operation on a symbol column

Remediation steps

  1. Convert the symbol column to an enum before sorting - for example, update sym:`sym$sym from table.
  2. Perform the sort on the CPU and then move the result to the GPU.

GPU_EXPECTED_FOREIGN

Summary: Expected a GPU foreign object

The module expected the input object to be a GPU foreign (a table or array resident on the GPU) but received a non-foreign object.

Likely causes

  • Passing a CPU-resident table directly to a GPU operation without first moving it with .gpu.to
  • A prior .gpu.to call that failed silently, leaving a CPU object in place

Remediation steps

  1. Ensure the table or array has been moved to the GPU with .gpu.to before passing it to GPU operations.
  2. Confirm the return value of .gpu.to - it should display as a composite of foreign objects.

GPU_WRONG_DEVICE

Summary: Object belongs to a different GPU device

The object being accessed was allocated on a different GPU device than the one currently active.

Likely causes

  • Using a multi-GPU setup where data was moved to one device but the operation is running on another
  • The active device changed between the .gpu.to call and the operation

Remediation steps

  1. Check the active device with .gpu.getDev.
  2. Verify which device the object was allocated on - it must match the active device.
  3. Use .gpu.setDev to set the correct active device before running the operation.
  4. If using multiple devices, ensure data and operations are consistently routed to the same device.

GPU_NOT_A_COLUMN

Summary: Symbol not found in column list

A symbol used in the query could not be matched to any column in the table.

Likely causes

  • A column name is misspelled in the query
  • Referencing a column that does not exist in the GPU table
  • Case mismatch in the column name

Remediation steps

  1. Check the column names in the table with cols or key .gpu.to table.
  2. Verify the spelling and case of all column references in the query.
  3. Confirm the correct table is being queried.

GPU_INVALID_VALUE

Summary: Input value is not valid for the operation

An input value was rejected by the operation. The value's type may be supported, but the specific value could not be processed.

Likely causes

  • An argument outside the range or form the operation accepts, such as an empty list of tensors passed to concat
  • A gather position outside the extent of its axis (any position on an axis of extent zero), or a null position
  • A gather range with a step of zero, or with a key other than start, stop and step, or a key given twice
  • Applying the operation to a host-resident column failed in q

Remediation steps

  1. Review the arguments passed to the operation and confirm they match the documented form.
  2. Simplify the call to isolate which argument is being rejected.
  3. Check whether the same operation succeeds on the CPU - if it fails there too, correct the input data.

GPU_TENSOR_SHAPE

Summary: Tensor shape or layout not supported

A tensor operation rejected the rank, dimensions, or memory layout of an operand.

Likely causes

  • Operand ranks the operation does not accept, such as a non-matrix input to inv
  • Dimension mismatch, such as unequal inner dimensions in mmu or unequal observation counts in lsq
  • Broadcast operands whose trailing dimensions differ where neither is 1
  • concat inputs whose ranks differ, whose extents differ on an axis other than the concatenation axis, or that are rank zero
  • A reduction (sum, avg, prd, min, max, first, last) axis outside the tensor's rank or repeated after normalization; a scan (sums, prds, mins, maxs), moving statistic (mavg, msum, mdev), sort (asc, desc) or concat axis outside the tensor's rank; or an axis other than 0 or -1 for a rank-zero tensor
  • A PCA component count, ddof or Gram order outside the documented bounds
  • A gather spec with more items than the tensor has axes, or a boolean mask whose length differs from the extent of its axis
  • A zero-extent dimension where the operation requires data
  • An element count or byte size that overflows

Remediation steps

  1. Check .gpu.tensor.shape and .gpu.tensor.meta of each operand against the operation's documented contract.
  2. For broadcasting, align the trailing dimensions or add size-1 axes with .gpu.tensor.reshape; for the reductions, the scans, the sorts, and concat, pass axes within the tensor's rank, and give concat tensors whose other extents agree. For gather, give at most one spec item per axis and each mask the length of its axis.

GPU_TENSOR_NONFINITE

Summary: PCA input or intermediate is not finite

PCA requires finite input, centered values and total energy.

Likely causes

  • NaN or infinity in the input
  • Overflow during centering or energy calculation

Remediation steps

  1. Remove nonfinite observations and rescale extreme input values before PCA.

GPU_TENSOR_PCA_CONVERGENCE

Summary: PCA solver did not converge

The requested cuSOLVER factorization reported a convergence failure.

Likely causes

  • A numerically difficult input matrix

Remediation steps

  1. Check input scale and conditioning; report a reproducible failure if it persists.

GPU_TENSOR_PCA_RANK

Summary: Requested Gram PCA directions are numerically rank deficient

The explicit eigh backend cannot safely recover a requested component whose eigenvalue is near zero.

Likely causes

  • Requested components exceed numerical rank
  • Squaring the condition number in the Gram matrix loses small directions

Remediation steps

  1. Request fewer components or explicitly use the full SVD backend.

GPU_TENSOR_PCA_SPECTRUM

Summary: PCA solver returned a spectrum that is not valid

PCA found a nonfinite or materially negative value in the returned spectrum.

Likely causes

  • Numerical overflow or instability in the factorization

Remediation steps

  1. Check the input scale and conditioning; do not silently retry a different backend.

GPU_TENSOR_NOT_POSITIVE_DEFINITE

Summary: Matrix is not positive definite

A Cholesky factorization found a non-positive leading minor, so the matrix has no Cholesky factor. chol reads only the input's lower triangle, so the rejected matrix is the symmetric one that triangle implies, whatever the upper triangle holds.

Likely causes

  • A matrix that is not symmetric positive definite, such as one built from unstructured random values
  • A semidefinite or rank-deficient matrix, including a covariance matrix estimated from fewer observations than variables
  • A covariance or Gram matrix that lost definiteness to rounding, shrinkage, or interpolated inputs

Remediation steps

  1. Confirm the intended matrix is symmetric positive definite: every leading minor determinant must be positive.
  2. For a GPU tensor x, build a Gram matrix as .gpu.tensor.mmu[x;.gpu.tensor.transpose x], which is positive semidefinite, then add a positive ridge to the diagonal to make it definite.
  3. For a near-definite covariance matrix, add a small positive multiple of the identity, or clip its negative eigenvalues on the host before uploading.

GPU_AXIS_MISMATCH

Summary: Tensor axis names do not align

A tensor contraction, broadcast or concatenation found incompatible coordinate names on axes that must align.

Likely causes

  • Both contracted axes are named, but their symbols differ in value or order
  • Two named operands give the same axis of a broadcast result different names
  • concat inputs that name an axis other than the concatenation axis with different symbols

Remediation steps

  1. Select or reorder table columns so named contraction axes have the same symbols in the same order.
  2. Remove axis names only when positional contraction, broadcasting or concatenation is intentional.

Other errors

GPU_UNKNOWN

Summary: Unclassified error

An unexpected error occurred that does not match any known error category.

Likely causes

  • A bug in the error reporting code

Remediation steps

  1. Capture the full stack trace.
  2. Attempt to reproduce with a minimal input to isolate the trigger.
  3. File a bug report with KDB-X support, including the trace and reproduction steps.

GPU_NOT_IMPLEMENTED

Summary: Feature not available

The requested operation or feature has not yet been implemented in this version of the GPU module.

Likely causes

  • Using a query syntax or function that is planned but not yet released
  • Calling an operation on a data type that does not yet have GPU support

Remediation steps

  1. Rewrite the query to avoid the unimplemented construct.
  2. Contact KDB-X to request prioritization or a workaround.

GPU_CUDA_CALL_FAILED

Summary: A CUDA runtime call failed

A CUDA runtime call that gpulib does not classify more specifically (device selection, stream synchronization or stream callback scheduling) returned an error. The operation was abandoned and the session survives; the first error reported is preserved.

Likely causes

  • The CUDA context or stream is in an error state, often after an earlier asynchronous kernel failure.
  • The device was reset or lost.
  • A driver or runtime fault.

Remediation steps

  1. Check nvidia-smi and the preceding GPU error in the session.
  2. Retry the operation after clearing GPU memory.
  3. If the context is lost, restart the q process.

GPU_CUDA_RUNTIME_MISMATCH

Summary: CUDA runtime major version differs from build

The major version of the CUDA runtime detected at load time does not match the major version of the CUDA toolkit the GPU module was compiled against.

Likely causes

  • The installed CUDA runtime was upgraded or downgraded across a major version after the module was downloaded.
  • Several CUDA toolkits are installed and the wrong one is first on the library search path.

Remediation steps

  1. Check the installed runtime with nvidia-smi and nvcc --version.
  2. Install a CUDA runtime whose major version matches the one the module was built against.
  3. If several toolkits are installed, set LD_LIBRARY_PATH to point at the intended CUDA runtime.

GPU_COMPUTE_CAPABILITY_MISMATCH

Summary: Device compute capability differs from build arch

The compute capability (SM version) reported by the active GPU does not match the architecture the module's kernels were compiled for.

Likely causes

  • Running a module built for one architecture (for example sm_90/Hopper) on a device of another (for example sm_80/Ampere).
  • In a multi-GPU host, the active device is of a different architecture than the build target.

Remediation steps

  1. Check the device compute capability with nvidia-smi --query-gpu=compute_cap --format=csv.
  2. In a multi-GPU host, use .gpu.setDev or CUDA_VISIBLE_DEVICES to select a device matching the build arch.