Q Type Wrappers

pykx.wrappers

Wrappers for q data structures, with conversion functions to Python/Numpy/Pandas/Arrow.

Under PyKX, q has its own memory space in which it stores q data structures in the same way it is stored within a regular q process. PyKX provides Pythonic wrappers around these objects in q memory.

In general, these wrappers consist of a pointer to a location in q memory, and a collection of methods to operate on/with that data.

Memory in q is managed by reference counting, and so wrapper objects (i.e. pykx.K objects) hold a reference to the underlying object in q memory, and release it when the Python wrapper is deallocated.

A benefit of this approach to interacting with q data is that all conversions are deferred until explicitly performed via one of the conversion methods:

• .py for Python
• .np for Numpy
• .pd for Pandas
• .pa for PyArrow

Example:

>>> import pykx as kx
>>> t = kx.q('([] x: 1 2 3; y: abc)') # Create a table in q memory
>>> x = t['x'] # Index into it in q
>>> x
pykx.LongVector(pykx.q('1 2 3'))
>>> x.np() # Convert the vector to Numpy
array([1, 2, 3])
>>> t.pd() # Convert the table to Pandas
x  y
0  1  a
1  2  b
2  3  c
>>> t.pa() # Convert the table to PyArrow
pyarrow.Table
x: int64
y: string
----
x: [[1,2,3]]
y: [["a","b","c"]]
>>> t.py() # Convert the table to Python
{'x': [1, 2, 3], 'y': ['a', 'b', 'c']}

Conversions from q to Python avoid copying data where possible, but that is not always possible. Furthermore, conversions can result in loss of data about type information - not a loss of the data itself, but of the type information that informs how it will be interpreted.

For example, if we had a q second atom pykx.SecondAtom('04:08:16'), and we converted it to Python using its .py method, we would get datetime.timedelta(seconds=14896). If we convert that back to q using the pykx.K constructor, we get pykx.TimespanAtom(pykx.q('0D04:08:16.000000000')).

We started with a pykx.SecondAtom, but after converting to Python and then back to q, we get a pykx.TimestampAtom. This is because that is what datetime.timedelta converts to by default, but the net effect is that we lost type information about the q data.

This round-trip-lossiness can be prevented in 2 ways:

1. If possible, avoid converting the pykx.K object to a Python/Numpy/Pandas/PyArrow type in the first place. No information can be lost during a conversion if no conversion occurs. It is frequently the case that no conversion actually needs to occur. As an example, instead of converting an entire pykx.Table into a Pandas dataframe, only to use a few columns from it, one can index into the pykx.Table directly to get the desired columns, thereby avoiding the conversion of the entire table.
2. Conversions from Python to q can be controlled by specifying the desired type. Using pykx.K as a constructor forces it to chose what q type the data should be converted to (using the same mechanism as pykx.toq), but by using the class of the desired q type directly, e.g. pykx.SecondAtom, one can override the defaults.

So to avoid the loss of type information from the previous example, we could run pykx.SecondAtom(datetime.timedelta(seconds=14896)) instead of pykx.K(datetime.timedelta(seconds=14896)).

Wrapper type hierarchy

The classes in the diagram are all attributes of pykx. They should be accessed as pykx.K, pykx.LongVector, pykx.Table, etc.

graph LR
K --> Atom
K --> Collection
Atom --> GUIDAtom
Atom --> NumericAtom
NumericAtom --> IntegralNumericAtom
IntegralNumericAtom --> BooleanAtom
IntegralNumericAtom --> ByteAtom
IntegralNumericAtom --> ShortAtom
IntegralNumericAtom --> IntAtom
IntegralNumericAtom --> LongAtom
NumericAtom --> NonIntegralNumericAtom
NonIntegralNumericAtom --> RealAtom
NonIntegralNumericAtom --> FloatAtom
Atom --> CharAtom
Atom --> SymbolAtom
Atom --> TemporalAtom
TemporalAtom --> TemporalFixedAtom
TemporalFixedAtom --> TimestampAtom
TemporalFixedAtom --> MonthAtom
TemporalFixedAtom --> DateAtom
TemporalFixedAtom --> DatetimeAtom
TemporalAtom --> TemporalSpanAtom
TemporalSpanAtom --> TimespanAtom
TemporalSpanAtom --> MinuteAtom
TemporalSpanAtom --> SecondAtom
TemporalSpanAtom --> TimeAtom
Atom --> EnumAtom
Atom --> Function
Function --> Lambda
Function --> UnaryPrimitive
UnaryPrimitive --> Identity
Identity --> ProjectionNull
Function --> Operator
Function --> Iterator
Function --> Projection
Function --> Composition
Function --> AppliedIterator
AppliedIterator --> Each
AppliedIterator --> Over
AppliedIterator --> Scan
AppliedIterator --> EachPrior
AppliedIterator --> EachRight
AppliedIterator --> EachLeft
Function --> Foreign
Collection --> Vector
Vector --> List
Vector --> GUIDVector
Vector --> NumericVector
NumericVector --> IntegralNumericVector
IntegralNumericVector --> BooleanVector
IntegralNumericVector --> ByteVector
IntegralNumericVector --> ShortVector
IntegralNumericVector --> IntVector
IntegralNumericVector --> LongVector
NumericVector --> NonIntegralNumericVector
NonIntegralNumericVector --> RealVector
NonIntegralNumericVector --> FloatVector
Vector --> CharVector
Vector --> SymbolVector
Vector --> TemporalVector
TemporalVector --> TemporalFixedVector
TemporalFixedVector --> TimestampVector
TemporalFixedVector --> MonthVector
TemporalFixedVector --> DateVector
TemporalFixedVector --> DatetimeVector
TemporalVector --> TemporalSpanVector
TemporalSpanVector --> TimespanVector
TemporalSpanVector --> MinuteVector
TemporalSpanVector --> SecondVector
TemporalSpanVector --> TimeVector
Vector --> EnumVector
Collection --> Mapping
Mapping --> Dictionary
Dictionary --> KeyedTable
Mapping --> Table
Table --> SplayedTable
SplayedTable --> PartitionedTable

K

K(x)

Base type for all q objects.

Parameters:

Name Type Description Default
x Any

An object that will be converted into a pykx.K object via pykx.toq.

required

Atom

Bases: K

pykx.Collection

Base type for all q atoms, including singular basic values, and functions.

EnumAtom

Bases: Atom

Wrapper for q enum atoms.

TemporalAtom

Bases: Atom

Base type for all q temporal atoms.

TemporalSpanAtom

Bases: TemporalAtom

Base type for all q temporal atoms which represent a span of time.

TemporalFixedAtom

Bases: TemporalAtom

Base type for all q temporal atoms which represent a fixed date/time.

TimeAtom

Bases: TemporalSpanAtom

Wrapper for q time atoms.

SecondAtom

Bases: TemporalSpanAtom

Wrapper for q second atoms.

MinuteAtom

Bases: TemporalSpanAtom

Wrapper for q minute atoms.

TimespanAtom

Bases: TemporalSpanAtom

Wrapper for q timespan atoms.

DatetimeAtom

DatetimeAtom(*args, **kwargs)

Bases: TemporalFixedAtom

The q datetime type is deprecated.

PyKX does not provide a rich interface for the q datetime type, as it is depreceated. Avoid using it whenever possible.

Wrapper for q datetime atoms.

DateAtom

Bases: TemporalFixedAtom

Wrapper for q date atoms.

MonthAtom

Bases: TemporalFixedAtom

Wrapper for q month atoms.

TimestampAtom

Bases: TemporalFixedAtom

Wrapper for q timestamp atoms.

SymbolAtom

Bases: Atom

Unique symbols are never deallocated!

Reserve symbol data for values that are recurring. Avoid using symbols for data being generated over time (e.g. random symbols) as memory usage will continually increase.

pykx.CharVector

Wrapper for q symbol atoms.

CharAtom

Bases: Atom

Wrapper for q char (i.e. 8 bit ASCII value) atoms.

NumericAtom

Bases: Atom

Base type for all q numeric atoms.

NonIntegralNumericAtom

Bases: NumericAtom, Real

Base type for all q non-integral numeric atoms.

FloatAtom

Bases: NonIntegralNumericAtom

Wrapper for q float (i.e. 64 bit float) atoms.

RealAtom

Bases: NonIntegralNumericAtom

Wrapper for q real (i.e. 32 bit float) atoms.

IntegralNumericAtom

Bases: NumericAtom, Integral

Base type for all q integral numeric atoms.

LongAtom

Bases: IntegralNumericAtom

Wrapper for q long (i.e. 64 bit signed integer) atoms.

IntAtom

Bases: IntegralNumericAtom

Wrapper for q int (i.e. 32 bit signed integer) atoms.

ShortAtom

Bases: IntegralNumericAtom

Wrapper for q short (i.e. 16 bit signed integer) atoms.

ByteAtom

Bases: IntegralNumericAtom

Wrapper for q byte (i.e. 8 bit unsigned integer) atoms.

GUIDAtom

Bases: Atom

Wrapper for q GUID atoms.

BooleanAtom

Bases: IntegralNumericAtom

Wrapper for q boolean atoms.

Collection

Bases: K

pykx.Collection

Base type for all q collections (i.e. non-atoms), including vectors, and mappings.

Vector

Bases: Collection, abc.Sequence

Base type for all q vectors, which are ordered collections of a particular type.

List

Bases: Vector

The memory layout of a q list is special.

All other vector types (see: subclasses of pykx.Vector) are structured in-memory as a K object which contains metadata, followed immediately by the data in the vector. By contrast, q lists are a a vector of pointers to K objects, so they are structured in-memory as a K object containing metadata, followed immediately by pointers. As a result, the base data "contained" by the list is located elsewhere in memory. This has performance and ownership implications in q, which carry over to PyKX.

Wrapper for q lists, which are vectors of K objects of any type.

np

np(*, raw=False, has_nulls=None)

Provides a Numpy representation of the list.

NumericVector

Bases: Vector

Base type for all q numeric vectors.

IntegralNumericVector

Bases: NumericVector

Base type for all q integral numeric vectors.

BooleanVector

Bases: IntegralNumericVector

Wrapper for q boolean vectors.

GUIDVector

Bases: Vector

Wrapper for q GUID vectors.

np

np(*, raw=False, has_nulls=None)

Another docstring.

ByteVector

Bases: IntegralNumericVector

Wrapper for q byte (i.e. 8 bit unsigned integer) vectors.

ShortVector

Bases: IntegralNumericVector

Wrapper for q short (i.e. 16 bit signed integer) vectors.

IntVector

Bases: IntegralNumericVector

Wrapper for q int (i.e. 32 bit signed integer) vectors.

LongVector

Bases: IntegralNumericVector

Wrapper for q long (i.e. 64 bit signed integer) vectors.

NonIntegralNumericVector

Bases: NumericVector

Base type for all q non-integral numeric vectors.

RealVector

Bases: NonIntegralNumericVector

Wrapper for q real (i.e. 32 bit float) vectors.

FloatVector

Bases: NonIntegralNumericVector

Wrapper for q float (i.e. 64 bit float) vectors.

CharVector

Bases: Vector

pykx.SymbolAtom

Wrapper for q char (i.e. 8 bit ASCII value) vectors.

SymbolVector

Bases: Vector

Unique symbols are never deallocated!

Reserve symbol data for values that are recurring. Avoid using symbols for data being generated over time (e.g. random symbols) as memory usage will continually increase.

Wrapper for q symbol vectors.

TemporalVector

Bases: Vector

Base type for all q temporal vectors.

TemporalFixedVector

Bases: TemporalVector

Base type for all q temporal vectors which represent a fixed date/time.

TemporalSpanVector

Bases: TemporalVector

Base type for all q temporal vectors which represent a span of time.

TimestampVector

Bases: TemporalFixedVector

Wrapper for q timestamp vectors.

MonthVector

Bases: TemporalFixedVector

Wrapper for q month vectors.

DateVector

Bases: TemporalFixedVector

Wrapper for q date vectors.

DatetimeVector

DatetimeVector(*args, **kwargs)

Bases: TemporalFixedVector

The q datetime type is deprecated.

PyKX does not provide a rich interface for the q datetime type, as it is depreceated. Avoid using it whenever possible.

Wrapper for q datetime vectors.

TimespanVector

Bases: TemporalSpanVector

Wrapper for q timespan vectors.

MinuteVector

Bases: TemporalSpanVector

Wrapper for q minute vectors.

SecondVector

Bases: TemporalSpanVector

Wrapper for q second vectors.

TimeVector

Bases: TemporalSpanVector

Wrapper for q time vectors.

EnumVector

Bases: Vector

Wrapper for q enum vectors.

Anymap

Bases: List

Wrapper for q mapped lists, also known as "anymaps".

Mapping

Mapping(*args, **kwargs)

Bases: Collection, abc.Mapping

Base type for all q mappings, including tables, and dictionaries.

Table

Table(*args, **kwargs)

Bases: Mapping

Wrapper for q tables, including in-memory tables, splayed tables, and partitioned tables.

Note: Despite the name, keyed tables are actually dictionaries.

SplayedTable

SplayedTable(*args, **kwargs)

Bases: Table

Wrapper for q splayed tables.

PartitionedTable

PartitionedTable(*args, **kwargs)

Bases: SplayedTable

Wrapper for q partitioned tables.

Dictionary

Dictionary(*args, **kwargs)

Bases: Mapping

Wrapper for q dictionaries, including regular dictionaries, and keyed tables.

KeyedTable

Bases: Dictionary

Wrapper for q keyed tables, which are a kind of table-like dictionary.

Function

Bases: Atom

Base type for all q functions.

Function objects can be called as if they were Python functions. All provided arguments will be converted to q using pykx.toq, and the execution of the function will happen in q.

... can be used to omit an argument, resulting in a function projection.

Lambda

Bases: Function

Wrapper for q lambda functions.

Lambda's are the most basic kind of function in q. They can take between 0 and 8 parameters (inclusive), which all must be q objects themselves. If the provided parameters are not pykx.K objects, they will be converted into them using pykx.toq.

Unlike other pykx.Function subclasses, Lambda objects can be called with keyword arguments, using the names of the parameters from q.

UnaryPrimative

Bases: Function

pykx.Identity

Wrapper for q unary primative functions, including ::, and other built-ins.

Unary primitives are a class of built-in q functions which take exactly one parameter. New ones cannot be defined by a user through any normal means.

Identity

Bases: UnaryPrimative

Wrapper for the q identity function, also known as generic null (::).

Most types in q have their own null value, but :: is used as a generic/untyped null in contexts where a non-null q object would otherwise be, e.g. as a null value for a generic list, or as a null value in a dictionary.

:: is also the identity function. It takes a single q object as a parameter, which it returns unchanged.

ProjectionNull

Bases: Identity

Projection nulls are unwieldy.

There are very few scenarios in which a typical user needs to work with a projection null directly, and doing so can be very error-prone. Instead of using them directly, operate on / work with projections, which use them implicitly.

Wrapper for the q projection null.

Projection null is a special q object which may initially seem to be generic null (::), but is actually a magic value used to create projections.

When a projection null is provided as an argument to a q function, the result is a function projection.

Operator

Bases: Function

Wrapper for q operator functions.

Operators include @, *, +, !, :, ^, and many more. They are documented on the q reference page: https://code.kx.com/q/ref/#operators

Iterator

Bases: Function

Wrappers for q iterator functions.

Iterators include the mapping iterators (', ':, /:, and \:), and the accumulating iterators (/, and \). They are documented on the q reference page: https://code.kx.com/q/ref/#iterators

Projection

Bases: Function

Wrapper for q function projections.

Similar to functools.partial, q functions can have some of their parameters fixed in advance, resulting in a new function, which is a projection. When this projection is called, the fixed parameters are no longer required, and cannot be provided.

If the original function had n parameters, and it had m of them provided, the result would be a function (projection) that has m parameters.

In PyKX, the special Python singleton ... is used to represent [projection null][pykx.ProjectionNull]

params

params()

The param names from the base function that have not been set.

args

args()

The supplied arguments to the function being projected.

The arguments in the tuple are ordered as they were applied to the function, with projection nulls to fill the empty spaces before and in-between the supplied arguments. The tuple may either end with the last supplied argument, or have some trailing projection nulls depending on how the projection was created.

Examples:

>>> f = kx.q('{x+y+z}')
>>> f.args
()
>>> f(..., 2)
pykx.Projection(pykx.q('{x+y+z}[;2]'))
>>> f(..., 2).args
(pykx.ProjectionNull(pykx.q('::')), pykx.LongAtom(pykx.q('2')))
>>> f(..., 2, ...)
pykx.Projection(pykx.q('{x+y+z}[;2;]'))
>>> f(..., 2, ...).args
(pykx.ProjectionNull(pykx.q('::')), pykx.LongAtom(pykx.q('2')), pykx.ProjectionNull(pykx.q('::')))

func

func()

The original function with no fixed parameters.

With the original function, a new projection can be created, or it can simply be called with every parameter set.

Composition

Bases: Function

Wrapper for q function compositions.

Functions in q can be directly composed, as opposed to creating a new lambda function that applies a chain of functions. Direct composition of functions lends itself to a style which is referred to as "point-free" or "tacit" programming.

AppliedIterator

Bases: Function

Base type for all q functions that have had an iterator applied to them.

Iterators, also known as adverbs, are like Python decorators, in that they are functions which take a function as their argument, and return a modified version of it. The iterators themselves are of the type pykx.Iterator, but when applied to a function a new type (which is a subclass of AppliedIterator) is created depending on what iterator was used.

Each

Bases: AppliedIterator

Wrapper for functions with the 'each' iterator applied to them.

Over

Bases: AppliedIterator

Wrapper for functions with the 'over' iterator applied to them.

Scan

Bases: AppliedIterator

Wrapper for functions with the 'scan' iterator applied to them.

EachPrior

Bases: AppliedIterator

Wrapper for functions with the 'each-prior' iterator applied to them.

EachRight

Bases: AppliedIterator

Wrapper for functions with the 'each-right' iterator applied to them.

EachLeft

Bases: AppliedIterator

Wrapper for functions with the 'each-left' iterator applied to them.

Foreign

Bases: Function

Wrapper for foreign objects, i.e. wrapped pointers to regions outside of q memory.

SymbolicFunction

SymbolicFunction(*args, **kwargs)

Bases: Function, SymbolAtom

Special wrapper type representing a symbol atom that can be used as a function.

sym

sym()

The symbolic function as a plain symbol.

func

func()

The symbolic function as a regular function, obtained by dereferencing the symbol.

with_execution_ctx

with_execution_ctx(execution_ctx)`

Get a new symbolic function that will be evaluated within the provided q instance.