# Memory management _This page explains how a q process allocates memory for its data, when that memory is shared, copied, and freed, which memory sits outside its heap, and how to measure and limit it._ A q process holds the data it works on in memory, so it can hold a large amount of data in RAM. Knowing how q sizes, shares, and releases that memory lets you predict what a query costs. It also explains why a process can hold more memory than its variables seem to need, and how to give that memory back. ## The heap q manages its own memory. Each thread has a _heap_: memory that q has requested from the operating system and allocates objects from. When q needs more memory, it requests more from the operating system in large units. When an object is freed, its memory goes back to the heap for reuse rather than to the operating system. The [garbage collection](#garbage-collection) section covers when memory goes back to the operating system. [`.Q.w[]`](../../ref/dotq.md#w-memory-stats) shows the heap of the calling thread. `used` is the memory taken by live objects, `heap` is the size of the heap, and `peak` is the largest the heap has been: ```q q).Q.w[]`used`heap`peak 468432 67108864 67108864 ``` [Secondary threads](../../ref/syscmds.md#s-number-of-secondary-threads) each have their own heap. Several [`peach`](../performance/code-optimization.md#limits-of-peach) calls can therefore hold several working sets at once. To see every heap, call `.Q.w` once in each secondary thread and join the results to the main thread's statistics. In this session, started with `q -s 4`, each thread has just built and freed a list of a different size: ```q q){v:til 10000000*1+x; 0} peach til 4; q).Q.w[],.Q.w peach til system"s" used heap peak wmax mmap mphy syms symw ------------------------------------------------------------ 535824 67108864 67108864 0 0 539992428544 777 39930 336 201326592 201326592 0 0 539992428544 336 335544320 335544320 0 0 539992428544 336 335544320 335544320 0 0 539992428544 336 603979776 603979776 0 0 539992428544 ``` The first row is the main thread, and the other rows are the secondary threads. The rows do not identify which thread is which. `syms` and `symw` are reported only for the main thread. Not all of a process's memory is on the heap. [Memory outside the heap](#memory-outside-the-heap) describes mapped files and the symbol pool. ## Buddy allocation q's allocator is a _buddy allocator_. It hands out memory only in blocks whose size is a power of two. A request is rounded up to the smallest block that holds it. When a smaller block is needed, a larger free block is split in half, and the two halves are _buddies_. When both buddies are free again, they merge back into the larger block. This makes allocation and release fast and keeps the heap tidy, at the cost of some unused space in each block. The blocks range from 16 bytes up to 64 MB. An object larger than 64 MB gets its own allocation directly from the operating system, also rounded up to a power of two. ### Simple lists The most common data structure in q is the _simple list_, a list whose items are all atoms of one type. Table columns, the keys and values of most dictionaries, and strings are all simple lists. A simple list is a single block: a 16-byte header holding the type, attribute, reference count, and item count, followed by the items packed end to end. So a simple list of `n` items, each `w` bytes wide, takes the smallest power of two that is at least `16+n*w` bytes. An atom takes 16 bytes, or 32 bytes for a GUID. [`\ts`](../../ref/syscmds.md#ts-time-and-space) shows the effect. It reports the time an expression takes and the memory it uses, which here is one block plus a small fixed overhead. A 16,384-byte block holds exactly 2,046 longs (16+2046×8). One more long needs a 32,768-byte block: ```q q)\ts til 2046 0 16560 q)\ts til 2047 0 32944 ``` A list just past a power-of-two boundary uses almost twice the memory its items need. On average, expect about 1.5 times the size of the data. > **Tip — `-22!` is not the memory size** > > > [`-22!x`](../../ref/internal.md#-22x-uncompressed-length) returns the size of `x` when serialized, which is close to its size on disk, not the block it occupies in memory: > > ```q > q)-22!til 1000000 > 8000014 > q)\ts til 1000000 > 2 8388784 > ``` > #### Spare capacity The unused end of a block is room to grow. Appending to a list in place with [`,:`](../../ref/assign.md) uses that room and allocates nothing until the block is full. `v:v,1` makes a new list every time: ```q q)v:til 2000 / 16016 bytes of data in a 16384-byte block q)\ts v,:1 / append in place 0 752 q)\ts v:v,1 / build a new list and assign it 0 16752 q)\ts do[40;v,:1] / still fits 0 1232 q)\ts v,:til 100 / no room left: move to a 32768-byte block 0 34016 ``` ### General lists A _general list_ (type `0h`) holds items of mixed type, or items that are themselves lists. It is a block of 8-byte pointers, and each item is a separate object with its own header, allocated in its own power-of-two block. A string column, a list of strings, is a general list. Many short items are therefore much more expensive than a single simple list holding the same data. A column of one million 8-character strings takes about 40 MB: 8 MB of pointers plus a 32-byte block for each string. The same values as symbols are a simple list of 8 MB: ```q q)x:1000000?`8 q)u:.Q.w[]`used q)y:string x q).Q.w[][`used]-u 40384816 ``` Symbols are interned: the text of each distinct symbol is stored once, and a symbol list holds references to it. That makes symbols the natural choice for repeated, low-cardinality values. [Text data](data_structures/atoms.md#text-data-characters-strings-and-symbols) covers when to use each. Items of a general list can also be shared. `1000000#enlist "abcdefgh"` holds one million pointers to a single string, not one million copies. [Reference counting](#reference-counting) explains how. ### Attributes [Attributes](../../ref/set-attribute.md) other than `` `s# `` keep a lookup structure beside the list. In typical use that structure is small next to the data it speeds up, but its size depends on the attribute and on the data: - `` `p# `` stores one entry for each distinct value, so it costs little on the column it is meant for: few distinct values, each in a long run. On one million longs in 100 runs, it adds less than 1 KB to the 8 MB list. - `` `u# `` keeps a hash table of every item, because every item is distinct. It takes about three times the memory of the list. It is most often applied to short lists, such as the keys of a dictionary or a keyed table, where the absolute cost is small. - `` `g# `` keeps the row indices of each distinct value, so it takes at least as much memory again as the list itself. The attribute is usually worth its memory when it is on the column you search. Measure it on your own data with the method in [measure the memory an object holds](#measure-the-memory-an-object-holds). ## Reference counting Every q object carries a _reference count_: the number of names, list items, and function arguments that refer to it. Assignment does not copy an object. It adds a reference to it. [`-16!`](../../ref/internal.md#-16x-ref-count) returns the count: ```q q)a:til 1000000 q)-16!a 1 q)b:a / b refers to the same list: no memory is allocated q)-16!a 2 q)f:{-16!x} q)f a / so does the argument of a function 2 ``` When the count falls to zero, the object is freed and its block goes back to the heap. That happens when a variable is reassigned or deleted. It also happens when a function returns, which frees its local variables, unless the function returns them. Tables share in the same way. A table is a dictionary of columns, and each column is a list with its own reference count. Copying a table, or adding a column that is the same as an existing one, adds references to the existing columns instead of copying them: ```q q)t:([]a:til 1000000) q)t2:t q)update b:a from `t2 `t2 q)-16!t`a / column a is shared by t and t2, and is also column b of t2 3 ``` ## Copy-on-write Because objects are shared, q cannot change one in place while anything else refers to it. When you modify an object whose reference count is more than one, q copies it and modifies the copy. The other references keep the original. This is called _copy-on-write_: ```q q)a:til 1000000 q)b:a q)u:.Q.w[]`used q)b[0]:42 / b is shared, so q copies it first q).Q.w[][`used]-u 8388608 q)-16!a / a and b are now separate lists 1 q)-16!b 1 ``` When the reference count is one, q can modify the object in place. Whether it does depends on the form of the update. Amending by name updates the variable in place, but an expression that amends a value and then assigns the result builds a new list: ```q q)v:til 1000000 q)\ts v[0]:42 / in place 0 896 q)\ts @[`v;0;:;42] / in place: amend by name 0 1264 q)\ts v:@[v;0;:;42] / copy, then assign 3 8389024 ``` A function that modifies its argument works on a copy whenever the caller still holds a reference to the argument, which is usually the case. [Update in place where safe](../performance/code-optimization.md#update-in-place-where-safe) covers when an in-place form saves memory. ### Deep copy Copy-on-write means q code almost never needs an explicit copy: `b:a` is safe, because changing `b` later leaves `a` untouched. A physically separate copy is mainly useful in benchmarking, for example to time an update without the copy that copy-on-write would add. Production code needs one only in exceptional cases. To make one, serialize the object and deserialize it with [`-9!-8!`](../../ref/internal.md#-8x-to-bytes). This builds the object again from scratch, as sending it over [interprocess communication (IPC)](../io_and_communication/basics.md) would: ```q q)s:`a`b`c q)t:([]sym:`s$1000000?s;p:1000000?1f;n:1000000#enlist "abc") q)u:.Q.w[]`used q)t2:-9!-8!t q).Q.w[][`used]-u 57166048 q)-16!first t`n / in t, every item of n refers to one string 1000000 q)-16!first t2`n / in t2, each item is a string of its own 1 q)meta t2 / enumerations come back as symbols c | t f a ---| ----- sym| s p | f n | C ``` A deep copy is not an exact copy. Shared items are no longer shared, so it can take much more memory than the original: in `t`, column `n` points one million times to a single string, while in `t2` it holds one million separate strings, about 32 MB more. [Enumerations](../../ref/enumerations.md) become symbols, which also breaks foreign keys, so `t2~t` is `0b`. ## Garbage collection When an object is freed, its block goes back to the heap, not to the operating system. Later allocations reuse free blocks, which is faster than asking the operating system for memory again. The cost is that a process that once needed a large heap keeps it. The _garbage-collection mode_ decides when freed memory is returned to the operating system. Set it with [`-g`](../../ref/cmdline.md#-g-garbage-collection) on the command line or [`\g`](../../ref/syscmds.md#g-garbage-collection-mode) in a running process. Deferred mode (`-g 0`, the default) : Freed memory stays on the heap until [`.Q.gc[]`](../../ref/dotq.md#gc-garbage-collect) runs or an allocation needs it. Immediate mode (`-g 1`) : A freed block of 64 MB or more goes back to the operating system at once. Smaller blocks still stay on the heap. Each large allocation then costs a system call. `.Q.gc[]` merges free blocks back into their original allocation units, returns the free units of 64 MB or more to the operating system, and returns the number of bytes released. In deferred mode: ```q q)a:til 10000000 q).Q.w[]`used`heap 134686208 201326592 q)delete a from `. `. q).Q.w[]`used`heap / the list is freed, but the heap keeps its memory 468448 201326592 q).Q.gc[] 134217728 q).Q.w[]`used`heap 468448 67108864 ``` In immediate mode, the same `delete` returns the 128 MB block at once, and `.Q.gc[]` has nothing large to release. When main-thread code calls `.Q.gc[]`, it also runs in each secondary thread. In the session from [the heap](#the-heap), it returns 1207959552 bytes, and every secondary heap shrinks back to 64 MB. Called from a secondary thread, it collects only that thread's heap. Setting the mode with `\g` also runs `.Q.gc[]`. q also collects without being asked. When an allocation would exceed the [workspace limit](#memory-limits), or the operating system refuses a request for memory, q runs `.Q.gc[]` and tries again. ### Fragmentation A 64 MB unit can go back to the operating system only when every block in it is free. A few small objects that are still in use, scattered over many units, can stop the heap from shrinking even though almost all of it is free. In this example, keeping 1,000 small lists, one in every 1,000, keeps the whole 1 GB heap: ```q q)a:{til 100} each til 1000000 / one million lists of 100 longs: about 1 GB q)b:a 1000*til 1000 / keep one list in every thousand q)delete a from `. `. q).Q.gc[] / every 64 MB unit still holds a live list 0 q).Q.w[]`used`heap 1500672 1073741824 q)delete b from `. `. q).Q.gc[] 1006632960 ``` Long-lived objects that are built while a process is busy with large temporary data cause this pattern. For example, a cache filled during a large query can stay scattered across the heap. Collecting the heap is also slower when it is fragmented, and nothing else runs in the process while `.Q.gc[]` runs. [Historical database (HDB)](../interact_with_databases/hdb.md#garbage-collection) covers scheduling collections in a server that must keep answering queries. ## Memory limits [`-w`](../../ref/cmdline.md#-w-workspace) sets a workspace limit in MB for the heaps of all threads. The default is 0, which means no limit. [`\w`](../../ref/syscmds.md#w-workspace) can raise the limit in a running process if it was set at startup. When an allocation would take the heap past the limit, q first collects garbage. If the heap is still over the limit, q prints `-w abort`, and the process exits with exit code 1. [Protected evaluation](protected-evaluation.md) does not catch it: ```bash $ q -w 100 q)@[til;20000000;{x}] '2026.11.08T09:34:12.654 -w abort $ echo $? 1 ``` If the operating system refuses to provide more memory, q signals [`'wsfull`](../../ref/errors.md#wsfull) instead. `-w` limits only the q heap. Mapped files, the symbol pool, and the page cache are outside it. For a limit on the whole process, use operating-system controls such as cgroups. [Containers](../performance/containers.md) and [hardware](../performance/hardware.md#memory) cover sizing memory for a deployment. ## Memory outside the heap Some of a process's memory is not on the heap, so `used` and `heap` do not count it: - **Mapped files.** When q loads a [database](../interact_with_databases/loading-db.md#what-l-maps) or a file with [`get`](../../ref/get.md), it maps the columns into memory instead of copying them to the heap. The operating system reads pages from disk as they are touched. The `mmap` field of `.Q.w[]` reports the mapped bytes, and [`-23!`](../../ref/internal.md#-23x-memory-map) asks the operating system to read the pages of a mapped object in advance. What a query computes from mapped columns is on the heap. - **Symbols.** The `syms` and `symw` fields of `.Q.w[]` report the number of interned symbols and the memory they use. The symbol pool never shrinks during a session: deleting the variables that use a symbol, or running `.Q.gc[]`, does not remove the symbol. Avoid creating symbols from unbounded, high-cardinality data such as order IDs. ## Memory utilities | Utility | What it does | | --- | --- | | [`\ts`](../../ref/syscmds.md#ts-time-and-space), [`.Q.ts`](../../ref/dotq.md#ts-time-and-space) | Time and memory used by an expression, including temporary objects. | | [`\w`](../../ref/syscmds.md#w-workspace), [`.Q.w[]`](../../ref/dotq.md#w-memory-stats) | Memory statistics of the calling thread: `used`, `heap`, `peak`, `wmax` (the `-w` limit), `mmap`, `mphy` (physical memory), `syms`, and `symw`. `\w n` changes the workspace limit. | | [`.Q.gc[]`](../../ref/dotq.md#gc-garbage-collect) | Collect garbage and return the number of bytes released to the operating system. | | [`-g`](../../ref/cmdline.md#-g-garbage-collection), [`\g`](../../ref/syscmds.md#g-garbage-collection-mode) | Garbage-collection mode: 0 deferred, 1 immediate. | | [`-w`](../../ref/cmdline.md#-w-workspace) | Workspace limit in MB. | | [`-16!x`](../../ref/internal.md#-16x-ref-count) | Reference count of `x`. | | [`-22!x`](../../ref/internal.md#-22x-uncompressed-length) | Serialized size of `x`. This is not its size in memory. | | [`-23!x`](../../ref/internal.md#-23x-memory-map) | Read the pages of a mapped object into memory. | ### Measure the memory an object holds `\ts` reports all the memory an expression allocates, including temporary objects that are freed before it finishes. To find the memory a result keeps, compare `used` before and after you create it: ```q q)u:.Q.w[]`used q)t:([]a:til 1000000;b:1000000?1f) q).Q.w[][`used]-u 16777424 ``` The result is approximate: it includes the small allocations made by the statements themselves, and it leaves out anything the result shares with objects that already existed. Deleting an object releases its memory only when nothing else refers to it. ### Process memory `\ts` and `.Q.w[]` report only the heap of the calling thread. For the memory of the whole process, including secondary-thread heaps and resident mapped pages, use operating-system tools such as `top`, `htop`, and `ps` on Linux and macOS, Activity Monitor on macOS, or Task Manager on Windows. On Linux, a q process can read its own figures from `/proc`. `VmRSS` is the memory the process holds now, and `VmHWM` is the most it has held since it started: ```q q)system"grep -E 'VmHWM|VmRSS' /proc/",string[.z.i],"/status" "VmHWM:\t 88672 kB" "VmRSS:\t 88672 kB" ``` `VmHWM` is the figure to use when you size memory for a process, because it includes the peaks that `.Q.gc[]` has since returned to the operating system. ## Next steps - Reduce the memory a query needs with [code optimization](../performance/code-optimization.md). - Size memory for a deployment with [hardware](../performance/hardware.md#memory) and [containers](../performance/containers.md). - Schedule garbage collection for a server process in [historical database (HDB)](../interact_with_databases/hdb.md#garbage-collection).