# Troubleshooting Errors _This guide helps you diagnose and resolve errors returned by the KDB-X GPU module. Each error includes its likely causes and step-by-step remediation._ ## How to use this guide 1. Locate the error code in the [Error index](#error-index). 2. Review the **summary** to confirm it matches your issue. 3. Check the **likely causes** to narrow down the problem. 4. Follow the **remediation steps** in order. 5. If unresolved, escalate with logs and a reproducible example. --- ## Error index | **Category** | **Error code** | **Summary** | |---|---|---| | **Internal** | [GPU_INTERNAL_LOGIC_ERROR](#gpu_internal_logic_error) | Unexpected internal control flow | | | [GPU_SELECT_DEADLOCK](#gpu_select_deadlock) | Interpreter deadlock detected | | | [GPU_LAUNCH_FAIL](#gpu_launch_fail) | GPU kernel launch failed | | | [GPU_CALCULATE_SCRATCH_ERR](#gpu_calculate_scratch_err) | Scratch memory calculation failed | | | [GPU_Q_ERROR](#gpu_q_error) | A q C API call signaled an error | | | [GPU_DEPENDENCY_ERR](#gpu_dependency_err) | An internal library has thrown an error | | **Memory** | [GPU_ARENA_MAP_FAILED](#gpu_arena_map_failed) | Host allocator creation failed | | | [GPU_HOST_ALLOC_FAILED](#gpu_host_alloc_failed) | Host memory allocation failed | | | [GPU_CUDA_ALLOC_FAILED](#gpu_cuda_alloc_failed) | GPU VRAM allocation failed | | | [GPU_CREATE_GLOBAL_SET](#gpu_create_global_set) | Global set creation failed | | | [GPU_CUDA_COPY_FAILED](#gpu_cuda_copy_failed) | CUDA memory copy failed | | **Type/parse** | [GPU_PARSE_FAIL](#gpu_parse_fail) | Input parsing failed | | | [GPU_PARSE_NOT_IMPLEMENTED](#gpu_parse_not_implemented) | Input parsing not implemented | | | [GPU_INVALID_STATEMENT](#gpu_invalid_statement) | Statement rejected during parsing | | | [GPU_TYPE_UNSUPPORTED](#gpu_type_unsupported) | Type not supported for operation | | | [GPU_INVALID_DOMAIN](#gpu_invalid_domain) | Enumeration domain not found | | **Data and column** | [GPU_SYMBOL_SORTING_NOT_IMPLEMENTED](#gpu_symbol_sorting_not_implemented) | Sorting on symbols or character arrays is not supported | | | [GPU_EXPECTED_FOREIGN](#gpu_expected_foreign) | Expected a GPU foreign object | | | [GPU_WRONG_DEVICE](#gpu_wrong_device) | Object belongs to a different GPU device | | | [GPU_NOT_A_COLUMN](#gpu_not_a_column) | Symbol not found in column list | | | [GPU_INVALID_VALUE](#gpu_invalid_value) | Input value is not valid for the operation | | | [GPU_TENSOR_SHAPE](#gpu_tensor_shape) | Tensor shape or layout not supported | | | [GPU_TENSOR_NONFINITE](#gpu_tensor_nonfinite) | PCA input or intermediate is not finite | | | [GPU_TENSOR_PCA_CONVERGENCE](#gpu_tensor_pca_convergence) | PCA solver did not converge | | | [GPU_TENSOR_PCA_RANK](#gpu_tensor_pca_rank) | Requested Gram PCA directions are numerically rank deficient | | | [GPU_TENSOR_PCA_SPECTRUM](#gpu_tensor_pca_spectrum) | PCA solver returned a spectrum that is not valid | | | [GPU_TENSOR_NOT_POSITIVE_DEFINITE](#gpu_tensor_not_positive_definite) | Matrix is not positive definite | | | [GPU_AXIS_MISMATCH](#gpu_axis_mismatch) | Tensor axis names do not align | | **Other** | [GPU_UNKNOWN](#gpu_unknown) | Unclassified error | | | [GPU_NOT_IMPLEMENTED](#gpu_not_implemented) | Feature not available | | | [GPU_CUDA_CALL_FAILED](#gpu_cuda_call_failed) | A CUDA runtime call failed | | | [GPU_CUDA_RUNTIME_MISMATCH](#gpu_cuda_runtime_mismatch) | CUDA runtime major version differs from build | | | [GPU_COMPUTE_CAPABILITY_MISMATCH](#gpu_compute_capability_mismatch) | Device compute capability differs from build arch | --- ## Internal errors ### `GPU_INTERNAL_LOGIC_ERROR` **Summary:** Unexpected internal control flow The GPU module reached a code path that should be unreachable under normal operation. This indicates a bug within the module itself rather than incorrect user input. **Remediation steps** 1. Note the exact query or operation that triggered the error. 1. Try downgrading to the previous stable release to confirm it is a regression. 1. Report the issue to KDB-X with a reproducible test case. > **Warning — This indicates a system-level issue. Escalate to [KDB-X support](../../help-support/support.md).** > > --- ### `GPU_SELECT_DEADLOCK` **Summary:** Interpreter deadlock detected The GPU interpreter has reached a deadlock state where two or more operations are waiting on each other and can no longer proceed. This is caused by a bug in the module's concurrency logic and should not occur under normal operation. **Remediation steps** 1. Stop the GPU process. 1. Upgrade to the latest module version where the bug may be fixed. 1. Report to KDB-X support - this error should not occur. > **Warning — This indicates a system-level issue. Escalate to [KDB-X support](../../help-support/support.md).** > > --- ### `GPU_LAUNCH_FAIL` **Summary:** GPU kernel launch failed An attempt to launch a CUDA kernel on the GPU device failed. The kernel was not dispatched for execution. **Likely causes** - CUDA driver or device in an error state - Internal kernel configuration that is not valid **Remediation steps** 1. Check `nvidia-smi` for GPU health and error state. 1. Run `compute-sanitizer` to detect device errors. 1. Reboot the host to reset the CUDA driver and GPU state. 1. Verify the CUDA toolkit and driver versions are compatible. --- ### `GPU_CALCULATE_SCRATCH_ERR` **Summary:** Scratch memory calculation failed The module failed to determine how much scratch (temporary working) memory a GPU kernel requires before launching it. This is a pre-launch validation step. **Likely causes** - A bug in the scratch-size calculation logic for a specific kernel - An unsupported parameter combination passed to the calculation routine **Remediation steps** 1. Note the specific operation that triggered this error. 1. Report to KDB-X support with full context - this error should not occur. > **Warning — This indicates a system-level issue. Escalate to [KDB-X support](../../help-support/support.md).** > > --- ### `GPU_Q_ERROR` **Summary:** A q C API call signaled an error A call into the q C API (for example, building the result table) signaled an error. The GPU stages of the operation completed, but the result could not be handed back to q. **Likely causes** - q workspace exhaustion (`wsfull`) while materializing the result - A result structure that q rejects, indicating a bug in the module **Remediation steps** 1. Check q's workspace usage against the `-w` limit and increase it if the result is large. 1. Reduce the result size (fewer rows or columns) and retry. 1. If memory is not the issue, report to KDB-X support with a reproducible example. --- ### `GPU_DEPENDENCY_ERR` **Summary:** An internal library has thrown an error An internal dependency has thrown an unexpected error that the GPU module cannot handle. **Likely causes** - CUDA or host memory is exhausted. - The dependency detects an error originating elsewhere in the codebase. - The dependency encounters an internal issue. **Remediation steps** 1. Run the failing workload under `compute-sanitizer` to collect more diagnostic information. 1. Report the issue to KDB-X support with a reproducible test case. --- ## Memory errors ### `GPU_ARENA_MAP_FAILED` **Summary:** Host allocator creation failed The module failed to create the memory arena (allocator) used to manage intermediate data structures on the host (CPU-side memory). This is a prerequisite for any subsequent GPU operation. **Likely causes** - Insufficient host RAM available at the time of the call - Memory fragmentation preventing a large contiguous allocation - OS-level memory limits (for example, `ulimit`) being exceeded **Remediation steps** 1. Check host memory usage with `free -h` or a system monitor. 1. Stop other memory-heavy processes to free up RAM. 1. Check `ulimit -v` and increase virtual memory limits if required. --- ### `GPU_HOST_ALLOC_FAILED` **Summary:** Host memory allocation failed An attempt to allocate an intermediate data structure on the host (CPU RAM) failed. The allocator was created successfully, but the specific allocation request could not be fulfilled. **Likely causes** - System RAM is fully exhausted - The requested allocation size exceeds available contiguous memory **Remediation steps** 1. Monitor memory usage during the operation with `top` or `htop`. 1. Reduce the size of the input dataset or split it into smaller batches. 1. Restart the process to clear any leaked allocations. 1. Add more physical RAM or increase available swap space. --- ### `GPU_CUDA_ALLOC_FAILED` **Summary:** GPU VRAM allocation failed A CUDA memory allocation call (`cudaMalloc` or `cudaMallocAsync`) returned `cudaErrorMemoryAllocation`, meaning the GPU does not have enough free VRAM to satisfy the request. **Likely causes** - GPU VRAM is exhausted by the current workload - A CUDA module failed to load on a nearly-full device: with lazy module loading a module's device memory is allocated at the first launch of one of its kernels, and after a failed load the process needs a restart before any further kernel can load - Other processes are consuming VRAM (for example, display driver or other CUDA applications) - Memory fragmentation on the GPU leaving no contiguous free block of sufficient size - Requesting more memory than the GPU physically has **Remediation steps** 1. Run `nvidia-smi` to inspect current VRAM usage. 1. Stop or pause other GPU-bound processes to free VRAM. 1. Reduce dataset size. 1. Switch to a GPU with more VRAM if the workload fundamentally requires it. 1. Set `CUDA_VISIBLE_DEVICES` to isolate the process to a less-loaded GPU. > **Note — This is the most common error in production GPU workloads. Monitor VRAM headroom proactively.** > > --- ### `GPU_CREATE_GLOBAL_SET` **Summary:** Global set creation failed The module failed to create a global set, most likely due to a CUDA memory allocation failure on the device. **Likely causes** - Insufficient GPU VRAM available at the time of the call - A preceding `cudaMalloc` failure that has not been handled - GPU device in an error state **Remediation steps** 1. Run `nvidia-smi` to inspect current VRAM usage. 1. Stop or pause other GPU-bound processes to free VRAM. 1. Check for a preceding GPU_CUDA_ALLOC_FAILED error in the same session - resolve that first. 1. Reboot the host to reset the CUDA driver and GPU state if the device appears unhealthy. --- ### `GPU_CUDA_COPY_FAILED` **Summary:** CUDA memory copy failed A CUDA memory copy operation failed while gpulib was transferring data between host and GPU memory. **Likely causes** - The GPU operation failed while copying input data, result data, or internal metadata. - The GPU may be out of memory, or the requested operation may be too large for the available device memory. - An earlier GPU operation may have failed, but the GPU module reports the error only during a later copy. - The GPU context, device, or stream may be in a state that is not valid. **Remediation steps** 1. Retry the operation in a fresh q session. 1. Reduce the size of the input data and try again. 1. Check that the selected GPU is available and has enough free memory. 1. If the problem is reproducible, report it with the full q expression, input sizes, gpulib version, CUDA error message, and GPU model. --- ## Type/parse errors ### `GPU_PARSE_FAIL` **Summary:** Input parsing failed The GPU module was unable to parse the provided input. The input did not conform to the expected format. **Likely causes** - Malformed parse tree or expression - Unexpected parse tree structure **Remediation steps** 1. Validate the query syntax against the KDB-X GPU module documentation. 1. Simplify the query to isolate which part is causing the parse failure. --- ### `GPU_PARSE_NOT_IMPLEMENTED` **Summary:** Input parsing not implemented The GPU module recognized the input but parsing for this construct has not yet been implemented. **Likely causes** - Using a query construct that is planned but not yet supported by the GPU parser - A syntax variant that the CPU parser accepts but the GPU parser does not yet handle **Remediation steps** 1. Rewrite the query to avoid the unimplemented construct. 1. Use the CPU fallback path in the interim. 1. [Contact KDB-X](../../help-support/support.md) to request prioritization or a workaround. --- ### `GPU_INVALID_STATEMENT` **Summary:** Statement rejected during parsing The input was parseable but the resulting statement was determined to be semantically incorrect. The statement structure violates the module's rules. **Likely causes** - Type mismatch between operands in an expression - Use of a keyword in an unsupported context - Missing required clauses or arguments in a statement **Remediation steps** 1. Break complex statements into smaller parts to locate the construct that is not valid. 1. Check operand types - ensure all types are compatible with the operation. 1. Check if the same query works on the CPU with [functional selects](../../ref/funsql.md). --- ### `GPU_TYPE_UNSUPPORTED` **Summary:** Type not supported for operation The data type of an operand is not supported for the requested GPU operation. The operation exists but does not handle this specific type. **Likely causes** - Using a data type that has not yet been implemented for this operation on the GPU - Mixing types in an operation that requires homogeneous inputs - A `gather` spec or spec item of a kind it does not accept, such as a bare boolean vector, a symbol, a float, a function, or an enumeration written for a one-key range as `` `step!-1 `` instead of ``(enlist `step)!enlist -1`` **Remediation steps** 1. Check the KDB-X docs for supported types for the operation. 1. Cast the data to a supported type before passing it to the operation. --- ### `GPU_INVALID_DOMAIN` **Summary:** Enumeration domain not found The named domain for an enumeration could not be located. **Likely causes** - Referencing an enum domain that has not been defined - Corrupted or incomplete enum metadata **Remediation steps** 1. Verify that the enumeration domain is correctly defined. 1. Re-register or re-initialize the enum domain if it has been removed. 1. Validate enum metadata integrity in the underlying data files. --- ## Data and column errors ### `GPU_SYMBOL_SORTING_NOT_IMPLEMENTED` **Summary:** Sorting on symbols or character arrays is not supported The query attempted to sort on a symbol or character array column, which is not supported by the GPU module. **Likely causes** - Using `xasc`, `xdesc`, or an `order by` clause on a symbol or character array column - An implicit sort triggered by a keyed table operation on a symbol column **Remediation steps** 1. Convert the symbol column to an enum before sorting - for example, `` update sym:`sym$sym from table ``. 1. Perform the sort on the CPU and then move the result to the GPU. --- ### `GPU_EXPECTED_FOREIGN` **Summary:** Expected a GPU foreign object The module expected the input object to be a GPU foreign (a table or array resident on the GPU) but received a non-foreign object. **Likely causes** - Passing a CPU-resident table directly to a GPU operation without first moving it with `.gpu.to` - A prior `.gpu.to` call that failed silently, leaving a CPU object in place **Remediation steps** 1. Ensure the table or array has been moved to the GPU with `.gpu.to` before passing it to GPU operations. 1. Confirm the return value of `.gpu.to` - it should display as a composite of `foreign` objects. --- ### `GPU_WRONG_DEVICE` **Summary:** Object belongs to a different GPU device The object being accessed was allocated on a different GPU device than the one currently active. **Likely causes** - Using a multi-GPU setup where data was moved to one device but the operation is running on another - The active device changed between the `.gpu.to` call and the operation **Remediation steps** 1. Check the active device with `.gpu.getDev`. 1. Verify which device the object was allocated on - it must match the active device. 1. Use `.gpu.setDev` to set the correct active device before running the operation. 1. If using multiple devices, ensure data and operations are consistently routed to the same device. --- ### `GPU_NOT_A_COLUMN` **Summary:** Symbol not found in column list A symbol used in the query could not be matched to any column in the table. **Likely causes** - A column name is misspelled in the query - Referencing a column that does not exist in the GPU table - Case mismatch in the column name **Remediation steps** 1. Check the column names in the table with `cols` or `key .gpu.to table`. 1. Verify the spelling and case of all column references in the query. 1. Confirm the correct table is being queried. --- ### `GPU_INVALID_VALUE` **Summary:** Input value is not valid for the operation An input value was rejected by the operation. The value's type may be supported, but the specific value could not be processed. **Likely causes** - An argument outside the range or form the operation accepts, such as an empty list of tensors passed to `concat` - A `gather` position outside the extent of its axis (any position on an axis of extent zero), or a null position - A `gather` range with a `step` of zero, or with a key other than `start`, `stop` and `step`, or a key given twice - Applying the operation to a host-resident column failed in q **Remediation steps** 1. Review the arguments passed to the operation and confirm they match the documented form. 1. Simplify the call to isolate which argument is being rejected. 1. Check whether the same operation succeeds on the CPU - if it fails there too, correct the input data. --- ### `GPU_TENSOR_SHAPE` **Summary:** Tensor shape or layout not supported A tensor operation rejected the rank, dimensions, or memory layout of an operand. **Likely causes** - Operand ranks the operation does not accept, such as a non-matrix input to `inv` - Dimension mismatch, such as unequal inner dimensions in `mmu` or unequal observation counts in `lsq` - Broadcast operands whose trailing dimensions differ where neither is 1 - `concat` inputs whose ranks differ, whose extents differ on an axis other than the concatenation axis, or that are rank zero - A reduction (`sum`, `avg`, `prd`, `min`, `max`, `first`, `last`) axis outside the tensor's rank or repeated after normalization; a scan (`sums`, `prds`, `mins`, `maxs`), moving statistic (`mavg`, `msum`, `mdev`), sort (`asc`, `desc`) or `concat` axis outside the tensor's rank; or an axis other than `0` or `-1` for a rank-zero tensor - A PCA component count, `ddof` or Gram order outside the documented bounds - A `gather` spec with more items than the tensor has axes, or a boolean mask whose length differs from the extent of its axis - A zero-extent dimension where the operation requires data - An element count or byte size that overflows **Remediation steps** 1. Check `.gpu.tensor.shape` and `.gpu.tensor.meta` of each operand against the operation's documented contract. 1. For broadcasting, align the trailing dimensions or add size-1 axes with `.gpu.tensor.reshape`; for the reductions, the scans, the sorts, and `concat`, pass axes within the tensor's rank, and give `concat` tensors whose other extents agree. For `gather`, give at most one spec item per axis and each mask the length of its axis. --- ### `GPU_TENSOR_NONFINITE` **Summary:** PCA input or intermediate is not finite PCA requires finite input, centered values and total energy. **Likely causes** - NaN or infinity in the input - Overflow during centering or energy calculation **Remediation steps** 1. Remove nonfinite observations and rescale extreme input values before PCA. --- ### `GPU_TENSOR_PCA_CONVERGENCE` **Summary:** PCA solver did not converge The requested cuSOLVER factorization reported a convergence failure. **Likely causes** - A numerically difficult input matrix **Remediation steps** 1. Check input scale and conditioning; report a reproducible failure if it persists. --- ### `GPU_TENSOR_PCA_RANK` **Summary:** Requested Gram PCA directions are numerically rank deficient The explicit eigh backend cannot safely recover a requested component whose eigenvalue is near zero. **Likely causes** - Requested components exceed numerical rank - Squaring the condition number in the Gram matrix loses small directions **Remediation steps** 1. Request fewer components or explicitly use the full SVD backend. --- ### `GPU_TENSOR_PCA_SPECTRUM` **Summary:** PCA solver returned a spectrum that is not valid PCA found a nonfinite or materially negative value in the returned spectrum. **Likely causes** - Numerical overflow or instability in the factorization **Remediation steps** 1. Check the input scale and conditioning; do not silently retry a different backend. --- ### `GPU_TENSOR_NOT_POSITIVE_DEFINITE` **Summary:** Matrix is not positive definite A Cholesky factorization found a non-positive leading minor, so the matrix has no Cholesky factor. `chol` reads only the input's lower triangle, so the rejected matrix is the symmetric one that triangle implies, whatever the upper triangle holds. **Likely causes** - A matrix that is not symmetric positive definite, such as one built from unstructured random values - A semidefinite or rank-deficient matrix, including a covariance matrix estimated from fewer observations than variables - A covariance or Gram matrix that lost definiteness to rounding, shrinkage, or interpolated inputs **Remediation steps** 1. Confirm the intended matrix is symmetric positive definite: every leading minor determinant must be positive. 1. For a GPU tensor `x`, build a Gram matrix as `.gpu.tensor.mmu[x;.gpu.tensor.transpose x]`, which is positive semidefinite, then add a positive ridge to the diagonal to make it definite. 1. For a near-definite covariance matrix, add a small positive multiple of the identity, or clip its negative eigenvalues on the host before uploading. --- ### `GPU_AXIS_MISMATCH` **Summary:** Tensor axis names do not align A tensor contraction, broadcast or concatenation found incompatible coordinate names on axes that must align. **Likely causes** - Both contracted axes are named, but their symbols differ in value or order - Two named operands give the same axis of a broadcast result different names - `concat` inputs that name an axis other than the concatenation axis with different symbols **Remediation steps** 1. Select or reorder table columns so named contraction axes have the same symbols in the same order. 1. Remove axis names only when positional contraction, broadcasting or concatenation is intentional. --- ## Other errors ### `GPU_UNKNOWN` **Summary:** Unclassified error An unexpected error occurred that does not match any known error category. **Likely causes** - A bug in the error reporting code **Remediation steps** 1. Capture the full stack trace. 1. Attempt to reproduce with a minimal input to isolate the trigger. 1. File a bug report with [KDB-X support](../../help-support/support.md), including the trace and reproduction steps. --- ### `GPU_NOT_IMPLEMENTED` **Summary:** Feature not available The requested operation or feature has not yet been implemented in this version of the GPU module. **Likely causes** - Using a query syntax or function that is planned but not yet released - Calling an operation on a data type that does not yet have GPU support **Remediation steps** 1. Rewrite the query to avoid the unimplemented construct. 1. [Contact KDB-X](../../help-support/support.md) to request prioritization or a workaround. --- ### `GPU_CUDA_CALL_FAILED` **Summary:** A CUDA runtime call failed A CUDA runtime call that gpulib does not classify more specifically (device selection, stream synchronization or stream callback scheduling) returned an error. The operation was abandoned and the session survives; the first error reported is preserved. **Likely causes** - The CUDA context or stream is in an error state, often after an earlier asynchronous kernel failure. - The device was reset or lost. - A driver or runtime fault. **Remediation steps** 1. Check `nvidia-smi` and the preceding GPU error in the session. 1. Retry the operation after clearing GPU memory. 1. If the context is lost, restart the q process. --- ### `GPU_CUDA_RUNTIME_MISMATCH` **Summary:** CUDA runtime major version differs from build The major version of the CUDA runtime detected at load time does not match the major version of the CUDA toolkit the GPU module was compiled against. **Likely causes** - The installed CUDA runtime was upgraded or downgraded across a major version after the module was downloaded. - Several CUDA toolkits are installed and the wrong one is first on the library search path. **Remediation steps** 1. Check the installed runtime with `nvidia-smi` and `nvcc --version`. 1. Install a CUDA runtime whose major version matches the one the module was built against. 1. If several toolkits are installed, set `LD_LIBRARY_PATH` to point at the intended CUDA runtime. --- ### `GPU_COMPUTE_CAPABILITY_MISMATCH` **Summary:** Device compute capability differs from build arch The compute capability (SM version) reported by the active GPU does not match the architecture the module's kernels were compiled for. **Likely causes** - Running a module built for one architecture (for example sm_90/Hopper) on a device of another (for example sm_80/Ampere). - In a multi-GPU host, the active device is of a different architecture than the build target. **Remediation steps** 1. Check the device compute capability with `nvidia-smi --query-gpu=compute_cap --format=csv`. 1. In a multi-GPU host, use `.gpu.setDev` or `CUDA_VISIBLE_DEVICES` to select a device matching the build arch.