DotQ/DotQDotgc

From Kx Wiki
Jump to: navigation, search

.Q.gc

(Added in v2.7 2010.08.05, enhanced with coalesce in v2.7 2011.09.15, and executes in slave threads since v2.7 2011.09.21)

Executive summary - invokes the garbage collector.
Returns the amount of memory that was returned to the OS.

The more detailed description of memory management in kdb+ is as follows

kdb+ uses reference counting, and http://en.wikipedia.org/wiki/Buddy_memory_allocation. The chosen buddy algorithm dices bucket sizes according to powers of 2, and the heap expands in powers of 64MB.
Reference counting means there is never any garbage (so .Q.gc is not accurate in its terminology) and memory is returned to the heap as soon as it is no longer referenced; if that memory is a vector using >=64MB it may be returned immediately to the OS depending on the cmd line param -g.

When .Q.gc[] is invoked, it attempts to coalesce diced blocks into their original 64MB block, and then returns blocks >=64MB to the OS.

Coalescing is always deferred, i.e. can only be triggered by a call to .Q.gc[].

When slave threads are used, .Q.gc[] in the main thread also executes gc in the slave threads.

.Q.gc[] can take several seconds to execute on large memory systems that have a fragmented heap, and hence is not recommended for frequent use in a time critical path of code. Consider running with the cmd line option -g 1 which will return larger blocks of memory to the os without trying to coalesce the smaller blocks.

q)a:til 10000000
q).Q.w[]
used| 67233056
heap| 134217728
peak| 134217728
wmax| 0
mmap| 0
syms| 534
symw| 23926
q).Q.gc[]
0j
q)delete a from `.
`.
q).Q.gc[]
67108864j
q).Q.w[]
used| 128768
heap| 67108864
peak| 134217728
wmax| 0
mmap| 0
syms| 535
symw| 23956

Note that memory can become fragmented and therefore difficult to release back to the OS. An example scenario

q)v:{(10#"a";10000#"b")}each til 10000000;
q).Q.w[]
used| 164614358256
heap| 164752261120
peak| 164752261120
wmax| 0
mmap| 0
mphy| 270538350592
syms| 569
symw| 24934
q).Q.gc[]
134217728
q).Q.w[]
used| 164614358256
heap| 164618043392
peak| 164752261120
wmax| 0
mmap| 0
mphy| 270538350592
syms| 570
symw| 24964
q)v:v[;0] / just retain refs to the small char vectors of "aaaaaaaa"
q).Q.gc[] / the vectors of "bbb.."s will come from the same memory chunks so can't be freed
134217728
q).Q.w[]
used| 454358256
heap| 164618043392
peak| 164752261120
wmax| 0
mmap| 0
mphy| 270538350592
syms| 570
symw| 24964
q)v:-8!v;0N!.Q.gc[];v:-9!v;.Q.w[] / serialize, release, deserialize
164483825664 / amount freed by gc
used| 454358848
heap| 738197504
peak| 164886478848
wmax| 0
mmap| 0
mphy| 270538350592
syms| 570
symw| 24964

so if you have many nested data, e.g. columns of char vectors, or an awful lot of grouping you may be fragmenting memory quite heavily.

Personal tools
Namespaces
Variants
Actions
Navigation
Print/export
Toolbox