Q Type Wrappers

`pykx.wrappers`

Wrappers for q data structures, with conversion functions to Python/Numpy/Pandas/Arrow.

Under PyKX, q has its own memory space in which it stores q data structures in the same way it is stored within a regular q process. PyKX provides Pythonic wrappers around these objects in q memory.

In general, these wrappers consist of a pointer to a location in q memory, and a collection of methods to operate on/with that data.

Memory in q is managed by reference counting, and so wrapper objects (i.e. pykx.K objects) hold a reference to the underlying object in q memory, and release it when the Python wrapper is deallocated.

A benefit of this approach to interacting with q data is that all conversions are deferred until explicitly performed via one of the conversion methods:

.py for Python
.np for Numpy
.pd for Pandas
.pa for PyArrow

Example:

>>> import pykx as kx
>>> t = kx.q('([] x: 1 2 3; y: `a`b`c)') # Create a table in q memory
>>> x = t['x'] # Index into it in q
>>> x
pykx.LongVector(pykx.q('1 2 3'))
>>> x.np() # Convert the vector to Numpy
array([1, 2, 3])
>>> t.pd() # Convert the table to Pandas
   x  y
0  1  a
1  2  b
2  3  c
>>> t.pa() # Convert the table to PyArrow
pyarrow.Table
x: int64
y: string
----
x: [[1,2,3]]
y: [["a","b","c"]]
>>> t.py() # Convert the table to Python
{'x': [1, 2, 3], 'y': ['a', 'b', 'c']}

Conversions from q to Python avoid copying data where possible, but that is not always possible. Furthermore, conversions can result in loss of data about type information - not a loss of the data itself, but of the type information that informs how it will be interpreted.

For example, if we had a q second atom pykx.SecondAtom('04:08:16'), and we converted it to Python using its .py method, we would get datetime.timedelta(seconds=14896). If we convert that back to q using the pykx.K constructor, we get pykx.TimespanAtom(pykx.q('0D04:08:16.000000000')).

We started with a pykx.SecondAtom, but after converting to Python and then back to q, we get a pykx.TimestampAtom. This is because that is what datetime.timedelta converts to by default, but the net effect is that we lost type information about the q data.

This round-trip-lossiness can be prevented in 2 ways:

If possible, avoid converting the pykx.K object to a Python/Numpy/Pandas/PyArrow type in the first place. No information can be lost during a conversion if no conversion occurs. It is frequently the case that no conversion actually needs to occur. As an example, instead of converting an entire pykx.Table into a Pandas dataframe, only to use a few columns from it, one can index into the pykx.Table directly to get the desired columns, thereby avoiding the conversion of the entire table.
Conversions from Python to q can be controlled by specifying the desired type. Using pykx.K as a constructor forces it to chose what q type the data should be converted to (using the same mechanism as pykx.toq), but by using the class of the desired q type directly, e.g. pykx.SecondAtom, one can override the defaults.

So to avoid the loss of type information from the previous example, we could run pykx.SecondAtom(datetime.timedelta(seconds=14896)) instead of pykx.K(datetime.timedelta(seconds=14896)).

Wrapper type hierarchy

The classes in the diagram are all attributes of pykx. They should be accessed as pykx.K, pykx.LongVector, pykx.Table, etc.

graph LR
  K --> Atom
  K --> Collection
  Atom --> GUIDAtom
  Atom --> NumericAtom
  NumericAtom --> IntegralNumericAtom
  IntegralNumericAtom --> BooleanAtom
  IntegralNumericAtom --> ByteAtom
  IntegralNumericAtom --> ShortAtom
  IntegralNumericAtom --> IntAtom
  IntegralNumericAtom --> LongAtom
  NumericAtom --> NonIntegralNumericAtom
  NonIntegralNumericAtom --> RealAtom
  NonIntegralNumericAtom --> FloatAtom
  Atom --> CharAtom
  Atom --> SymbolAtom
  Atom --> TemporalAtom
  TemporalAtom --> TemporalFixedAtom
  TemporalFixedAtom --> TimestampAtom
  TemporalFixedAtom --> MonthAtom
  TemporalFixedAtom --> DateAtom
  TemporalFixedAtom --> DatetimeAtom
  TemporalAtom --> TemporalSpanAtom
  TemporalSpanAtom --> TimespanAtom
  TemporalSpanAtom --> MinuteAtom
  TemporalSpanAtom --> SecondAtom
  TemporalSpanAtom --> TimeAtom
  Atom --> EnumAtom
  Atom --> Function
  Function --> Lambda
  Function --> UnaryPrimitive
  UnaryPrimitive --> Identity
  Identity --> ProjectionNull
  Function --> Operator
  Function --> Iterator
  Function --> Projection
  Function --> Composition
  Function --> AppliedIterator
  AppliedIterator --> Each
  AppliedIterator --> Over
  AppliedIterator --> Scan
  AppliedIterator --> EachPrior
  AppliedIterator --> EachRight
  AppliedIterator --> EachLeft
  Function --> Foreign
  Collection --> Vector
  Vector --> List
  Vector --> GUIDVector
  Vector --> NumericVector
  NumericVector --> IntegralNumericVector
  IntegralNumericVector --> BooleanVector
  IntegralNumericVector --> ByteVector
  IntegralNumericVector --> ShortVector
  IntegralNumericVector --> IntVector
  IntegralNumericVector --> LongVector
  NumericVector --> NonIntegralNumericVector
  NonIntegralNumericVector --> RealVector
  NonIntegralNumericVector --> FloatVector
  Vector --> CharVector
  Vector --> SymbolVector
  Vector --> TemporalVector
  TemporalVector --> TemporalFixedVector
  TemporalFixedVector --> TimestampVector
  TemporalFixedVector --> MonthVector
  TemporalFixedVector --> DateVector
  TemporalFixedVector --> DatetimeVector
  TemporalVector --> TemporalSpanVector
  TemporalSpanVector --> TimespanVector
  TemporalSpanVector --> MinuteVector
  TemporalSpanVector --> SecondVector
  TemporalSpanVector --> TimeVector
  Vector --> EnumVector
  Collection --> Mapping
  Mapping --> Dictionary
  Dictionary --> KeyedTable
  Mapping --> Table
  Table --> SplayedTable
  SplayedTable --> PartitionedTable

`K`

K(x, *, cast=None)

Base type for all q objects.

Parameters:

Name	Type	Description	Default
`x`	`Any`	An object that will be converted into a `pykx.K` object via `pykx.toq`.	required

`Atom`

Bases: K

See Also

pykx.Collection

Base type for all q atoms, including singular basic values, and functions.

`EnumAtom`

Bases: Atom

Wrapper for q enum atoms.

`TemporalAtom`

Bases: Atom

Base type for all q temporal atoms.

`TemporalSpanAtom`

Bases: TemporalAtom

Base type for all q temporal atoms which represent a span of time.

`TemporalFixedAtom`

Bases: TemporalAtom

Base type for all q temporal atoms which represent a fixed date/time.

`TimeAtom`

Bases: TemporalSpanAtom

Wrapper for q time atoms.

`SecondAtom`

Bases: TemporalSpanAtom

Wrapper for q second atoms.

`MinuteAtom`

Bases: TemporalSpanAtom

Wrapper for q minute atoms.

`TimespanAtom`

Bases: TemporalSpanAtom

Wrapper for q timespan atoms.

`DatetimeAtom`

DatetimeAtom(*args, **kwargs)

Bases: TemporalFixedAtom

The q datetime type is deprecated.

PyKX does not provide a rich interface for the q datetime type, as it is depreceated. Avoid using it whenever possible.

Wrapper for q datetime atoms.

`DateAtom`

Bases: TemporalFixedAtom

Wrapper for q date atoms.

`MonthAtom`

Bases: TemporalFixedAtom

Wrapper for q month atoms.

`TimestampAtom`

Bases: TemporalFixedAtom

Wrapper for q timestamp atoms.

`SymbolAtom`

Bases: Atom

Unique symbols are never deallocated!

Reserve symbol data for values that are recurring. Avoid using symbols for data being generated over time (e.g. random symbols) as memory usage will continually increase.

See Also

pykx.CharVector

Wrapper for q symbol atoms.

`CharAtom`

Bases: Atom

Wrapper for q char (i.e. 8 bit ASCII value) atoms.

`NumericAtom`

Bases: Atom

Base type for all q numeric atoms.

`NonIntegralNumericAtom`

Bases: NumericAtom, Real

Base type for all q non-integral numeric atoms.

`FloatAtom`

Bases: NonIntegralNumericAtom

Wrapper for q float (i.e. 64 bit float) atoms.

`RealAtom`

Bases: NonIntegralNumericAtom

Wrapper for q real (i.e. 32 bit float) atoms.

`IntegralNumericAtom`

Bases: NumericAtom, Integral

Base type for all q integral numeric atoms.

`LongAtom`

Bases: IntegralNumericAtom

Wrapper for q long (i.e. 64 bit signed integer) atoms.

`IntAtom`

Bases: IntegralNumericAtom

Wrapper for q int (i.e. 32 bit signed integer) atoms.

`ShortAtom`

Bases: IntegralNumericAtom

Wrapper for q short (i.e. 16 bit signed integer) atoms.

`ByteAtom`

Bases: IntegralNumericAtom

Wrapper for q byte (i.e. 8 bit unsigned integer) atoms.

`GUIDAtom`

Bases: Atom

Wrapper for q GUID atoms.

`BooleanAtom`

Bases: IntegralNumericAtom

Wrapper for q boolean atoms.

`Collection`

Bases: K

See Also

pykx.Collection

Base type for all q collections (i.e. non-atoms), including vectors, and mappings.

`Vector`

Bases: Collection, abc.Sequence

Base type for all q vectors, which are ordered collections of a particular type.

`List`

Bases: Vector

The memory layout of a q list is special.

All other vector types (see: subclasses of pykx.Vector) are structured in-memory as a K object which contains metadata, followed immediately by the data in the vector. By contrast, q lists are a a vector of pointers to K objects, so they are structured in-memory as a K object containing metadata, followed immediately by pointers. As a result, the base data "contained" by the list is located elsewhere in memory. This has performance and ownership implications in q, which carry over to PyKX.

Wrapper for q lists, which are vectors of K objects of any type.

np

np(*, raw=False, has_nulls=None)

Provides a Numpy representation of the list.

`NumericVector`

Bases: Vector

Base type for all q numeric vectors.

`IntegralNumericVector`

Bases: NumericVector

Base type for all q integral numeric vectors.

`BooleanVector`

Bases: IntegralNumericVector

Wrapper for q boolean vectors.

`GUIDVector`

Bases: Vector

Wrapper for q GUID vectors.

`ByteVector`

Bases: IntegralNumericVector

Wrapper for q byte (i.e. 8 bit unsigned integer) vectors.

`ShortVector`

Bases: IntegralNumericVector

Wrapper for q short (i.e. 16 bit signed integer) vectors.

`IntVector`

Bases: IntegralNumericVector

Wrapper for q int (i.e. 32 bit signed integer) vectors.

`LongVector`

Bases: IntegralNumericVector

Wrapper for q long (i.e. 64 bit signed integer) vectors.

`NonIntegralNumericVector`

Bases: NumericVector

Base type for all q non-integral numeric vectors.

`RealVector`

Bases: NonIntegralNumericVector

Wrapper for q real (i.e. 32 bit float) vectors.

`FloatVector`

Bases: NonIntegralNumericVector

Wrapper for q float (i.e. 64 bit float) vectors.

`CharVector`

Bases: Vector

See Also

pykx.SymbolAtom

Wrapper for q char (i.e. 8 bit ASCII value) vectors.

`SymbolVector`

Bases: Vector

Unique symbols are never deallocated!

Reserve symbol data for values that are recurring. Avoid using symbols for data being generated over time (e.g. random symbols) as memory usage will continually increase.

Wrapper for q symbol vectors.

`TemporalVector`

Bases: Vector

Base type for all q temporal vectors.

`TemporalFixedVector`

Bases: TemporalVector

Base type for all q temporal vectors which represent a fixed date/time.

`TemporalSpanVector`

Bases: TemporalVector

Base type for all q temporal vectors which represent a span of time.

`TimestampVector`

Bases: TemporalFixedVector

Wrapper for q timestamp vectors.

`MonthVector`

Bases: TemporalFixedVector

Wrapper for q month vectors.

`DateVector`

Bases: TemporalFixedVector

Wrapper for q date vectors.

`DatetimeVector`

DatetimeVector(*args, **kwargs)

Bases: TemporalFixedVector

The q datetime type is deprecated.

PyKX does not provide a rich interface for the q datetime type, as it is depreceated. Avoid using it whenever possible.

Wrapper for q datetime vectors.

`TimespanVector`

Bases: TemporalSpanVector

Wrapper for q timespan vectors.

`MinuteVector`

Bases: TemporalSpanVector

Wrapper for q minute vectors.

`SecondVector`

Bases: TemporalSpanVector

Wrapper for q second vectors.

`TimeVector`

Bases: TemporalSpanVector

Wrapper for q time vectors.

`EnumVector`

Bases: Vector

Wrapper for q enum vectors.

`Anymap`

Bases: List

Wrapper for q mapped lists, also known as "anymaps".

`Mapping`

Mapping(*args, **kwargs)

Bases: Collection, abc.Mapping

Base type for all q mappings, including tables, and dictionaries.

`Table`

Table(*args, **kwargs)

Bases: Mapping

See Also

pykx.SplayedTable
pykx.PartitionedTable

Wrapper for q tables, including in-memory tables, splayed tables, and partitioned tables.

Note: Despite the name, keyed tables are actually dictionaries.

`SplayedTable`

SplayedTable(*args, **kwargs)

Bases: Table

Wrapper for q splayed tables.

`PartitionedTable`

PartitionedTable(*args, **kwargs)

Bases: SplayedTable

Wrapper for q partitioned tables.

`Dictionary`

Dictionary(*args, **kwargs)

Bases: Mapping

Wrapper for q dictionaries, including regular dictionaries, and keyed tables.

`KeyedTable`

Bases: Dictionary

Wrapper for q keyed tables, which are a kind of table-like dictionary.

Refer to chapter 8.4 of Q for Mortals for more information about q keyed tables.

`Function`

Bases: Atom

Base type for all q functions.

Function objects can be called as if they were Python functions. All provided arguments will be converted to q using pykx.toq, and the execution of the function will happen in q.

... can be used to omit an argument, resulting in a function projection.

Refer to chapter 6 of Q for Mortals for more information about q functions.

`Lambda`

Bases: Function

Wrapper for q lambda functions.

Lambda's are the most basic kind of function in q. They can take between 0 and 8 parameters (inclusive), which all must be q objects themselves. If the provided parameters are not pykx.K objects, they will be converted into them using pykx.toq.

Unlike other pykx.Function subclasses, Lambda objects can be called with keyword arguments, using the names of the parameters from q.

`UnaryPrimitive`

Bases: Function

See Also

pykx.Identity

Wrapper for q unary primitive functions, including ::, and other built-ins.

Unary primitives are a class of built-in q functions which take exactly one parameter. New ones cannot be defined by a user through any normal means.

`Identity`

Bases: UnaryPrimitive

Wrapper for the q identity function, also known as generic null (::).

Most types in q have their own null value, but :: is used as a generic/untyped null in contexts where a non-null q object would otherwise be, e.g. as a null value for a generic list, or as a null value in a dictionary.

:: is also the identity function. It takes a single q object as a parameter, which it returns unchanged.

`ProjectionNull`

Bases: Identity

Projection nulls are unwieldy.

There are very few scenarios in which a typical user needs to work with a projection null directly, and doing so can be very error-prone. Instead of using them directly, operate on / work with projections, which use them implicitly.

Wrapper for the q projection null.

Projection null is a special q object which may initially seem to be generic null (::), but is actually a magic value used to create projections.

When a projection null is provided as an argument to a q function, the result is a function projection.

`Operator`

Bases: Function

Wrapper for q operator functions.

Operators include @, *, +, !, :, ^, and many more. They are documented on the q reference page: https://code.kx.com/q/ref/#operators

`Iterator`

Bases: Function

Wrappers for q iterator functions.

Iterators include the mapping iterators (', ':, /:, and \:), and the accumulating iterators (/, and \). They are documented on the q reference page: https://code.kx.com/q/ref/#iterators

`Projection`

Bases: Function

Wrapper for q function projections.

Similar to functools.partial, q functions can have some of their parameters fixed in advance, resulting in a new function, which is a projection. When this projection is called, the fixed parameters are no longer required, and cannot be provided.

If the original function had n parameters, and it had m of them provided, the result would be a function (projection) that has m parameters.

In PyKX, the special Python singleton ... is used to represent [projection null][pykx.ProjectionNull]

params

params()

The param names from the base function that have not been set.

args

args()

The supplied arguments to the function being projected.

The arguments in the tuple are ordered as they were applied to the function, with projection nulls to fill the empty spaces before and in-between the supplied arguments. The tuple may either end with the last supplied argument, or have some trailing projection nulls depending on how the projection was created.

Examples:

>>> f = kx.q('{x+y+z}')
>>> f.args
()
>>> f(..., 2)
pykx.Projection(pykx.q('{x+y+z}[;2]'))
>>> f(..., 2).args
(pykx.ProjectionNull(pykx.q('::')), pykx.LongAtom(pykx.q('2')))
>>> f(..., 2, ...)
pykx.Projection(pykx.q('{x+y+z}[;2;]'))
>>> f(..., 2, ...).args
(pykx.ProjectionNull(pykx.q('::')), pykx.LongAtom(pykx.q('2')), pykx.ProjectionNull(pykx.q('::')))

func

func()

The original function with no fixed parameters.

With the original function, a new projection can be created, or it can simply be called with every parameter set.

`Composition`

Bases: Function

Wrapper for q function compositions.

Functions in q can be directly composed, as opposed to creating a new lambda function that applies a chain of functions. Direct composition of functions lends itself to a style which is referred to as "point-free" or "tacit" programming.

`AppliedIterator`

Bases: Function

Base type for all q functions that have had an iterator applied to them.

Iterators, also known as adverbs, are like Python decorators, in that they are functions which take a function as their argument, and return a modified version of it. The iterators themselves are of the type pykx.Iterator, but when applied to a function a new type (which is a subclass of AppliedIterator) is created depending on what iterator was used.