Features & Fixes
- Added ability to skip symlinking
$QHOMEby setting the environment variable
- The dependency on the system library
libcurlhas been made optional for Linux. If it is missing on Linux, a warning will be emitted instead of an error being raised, and the KX Insights Core library
kurlwill not be fully loaded. Windows and macOS are unaffected, as they don't support the KX Insights Core features to begin with.
Features & Fixes
- Splayed and partitioned tables no longer emit warnings when instantiated.
pykx.Q.sql, which is a wrapper around KXI Core SQL.
.pykx.pyevalno longer segfault when called with a character atom.
- Updated several
pykx.toqtests so that they would not randomly fail.
- Fixed error when pickling
pykx.util.BlockManagerin certain esoteric situations.
pandas.MultiIndexobjects created by PyKX having
pykx.SymbolAtomobjects within them - now they have
strobjects instead, as they normally would.
- Upgraded the included KX Insights Core libraries to version 3.0.0.
pykx.toq.from_datetime_date, which converts
datetime.dateobjects into any q temporal atom that can represent a date (defaulting to a date atom).
- Fixed error when user specifies
- Fixed recursion error when accessing a non-existent attribute of
pykx.qwhile in unlicensed mode. Now an attribute error is raised instead.
- Fixed build error introduced by new rules enforced by new versions of setuptools.
- Fixed support for
- The KXIC libraries are now loaded after q has been fully initialized, rather than during the initialization. This significantly reduces the time it takes to import PyKX.
- PyKX now uses a single location for
libdirectory within the installed package. The top-level contents of the
$QHOMEdirectory (prior to PyKX updating the env var when embedded q is initialized) will be symlinked into PyKX's
libdirectory, along with the content of any subdirectories under
w64). This enables loading scripts and libraries located in the original
$QHOMEdirectory during q initialization.
- Improved performance (both execution speed and memory usage) of calling
pykx.Vectorinstances. The best practice is still to use the
npmethod instead of calling
pykx.Vectoris now a subclass of
pykx.Mappingis not a subclass of
pykx.AsyncQConnectionand added support for asynchronous IPC with
await. Refer to the
pykx.AsyncQConnectiondocs for more details.
- Pandas dataframes containing Pandas extension arrays not originally created as Numpy arrays would result in errors when attempting to convert to q. For example a Dataframe with index of type
pandas.MultiIndex.from_arrayswould result in an error in conversion.
- Improved performance of converting
numpy.arrayof strings, and also the conversion back from a
- Improved performance of converting
timedelta64to the various
Deprecations & Removals
pykx.QConnection.__call__has been renamed to the less confusing name
syncparameter remains, but its usage will result in a
DeprecationWarningbeing emitted. The
syncparameter will be removed in a future version.
Features & Fixes
- Updated to stable classifier (
Development Status :: 5 - Production/Stable) in project metadata. Despite this update being done in version 1.0.1, version 1.0.0 is still the first stable release of PyKX.
- PyKX now provides source distributions (
sdist). It can be downloaded from PyPI using
pip download --no-binary=:all: --no-deps pykx. As noted in the installation docs, installations built from the source will only receive support on a best-effort basis.
- Fixed Pandas NaT conversion to q types. Now
pykx.toq(pandas.NaT, ktype=ktype)produces a null temporal atom for any given
- Added a doc page for limitations of embedded q.
- Added a test to ensure large vectors are correctly handled (5 GiB).
- Always use synchronous queries internally, i.e. fix
- Disabled the context interface over IPC. This is a temporary measure that will be reversed once q function objects are updated to run in the environment they were defined in by default.
- Reduced the time it takes to import PyKX. There are plans to reduce it further, as
import pykxremains fairly slow.
- Updated to KXI Core 2.1 & rename
- Misc test updates.
- Misc doc updates.
To switch from Pykdb to PyKX, you will need to update the name of the dependency from
pykx in your
setup.cfg/etc. When Pykdb was renamed to PyKX, its version number was reset. The first public release of PyKX has the version number 1.0.0, and will employ semantic versioning.
Pay close attention to the renames listed below, as well as the removals. Many things have been moved to the top-level, or otherwise reorganized. A common idiom with Pykdb was the following:
from pykdb import q, k
It is recommended that the following be used instead:
import pykx as kx
This way the many attributes at the top-level can be easily accessed without any loss of context, for example:
kx.q # Can be called to execute q code kx.K # Base type for objects in q; can be used to convert a Python object into a q type kx.SymbolAtom # Type for symbol atoms; can be used to convert a `str` or `bytes` into a symbol atom kx.QContext # Represents a q context via the PyKX context interface kx.QConnection # Can be called to connect to a q process via q IPC kx.PyKXException # Base exception type for exceptions specific to PyKX and q kx.QError # Exception type for errors that occur in q kx.LicenseException # Exception type raised when features that require a license are used without kx.QHOME # Path from which to load q files, set by $QHOME environment variable kx.QARGS # List of arguments provided to the embedded q instance at startup, set by $QARGS environment variable # etc.
You can no longer rely on the context being reset to the global context after each call into embedded q, however IPC calls are unaffected.
- Pykdb has been renamed to PyKX.
adaptmodule has been renamed to
toq, and it can be called directly. Instead of
pykdb.adapt.adapt(x)one should write
kmodule has been renamed to
wrappers. All wrapper classes can be accessed from the top-level, i.e.
- The "module interface" (
pykdb.module_interface) has been renamed to the "context interface" (
pykx.QConnectioninstances) have a
ctxattribute, which is the global
pykx.Qinstance. Usually, one need not directly access the global context. Instead, one can access its subcontexts directly e.g.
KdbError(and its subclasses) have been renamed to
pykdb.ctx.KdbContexthas been renamed to
pykx.ctx.QContext, and is available from the top-level, i.e.
Connectionclass in the IPC module has been renamed to
QConnection, and is now available at the top-level, i.e.
- The q type wrapper
DynamicLoadhas been renamed to
Deprecations & Removals
pykdb.q.ipcattribute has been removed. The IPC module can be accessed directly instead at
pykx.ipc, but generally one will only need to access the
QConnectionclass, which can be accessed at the top-level:
pykdb.q.Kattribute has been removed. Instead,
Ktypes can be used as constructors for that type by leveraging the
toqmodule. For example, instead of
pykdb.q.K(x)one should write
pykx.K(x). Instead of
pykx.q.K(x, k_type=pykx.k.SymbolAtom)one should write
QErrorsubclasses have been removed, as identifying them is error prone, and we are unable to provide helpful error messages for most of them.
pykx.kdbsingleton class has been removed.
- More Numpy, Pandas, and PyArrow versions are supported. Current
pyarrow>=3.0.0are supported. PyArrow remains an optional dependency.
- A dependency on
find-libpython~=0.2was added. This is only used when running PyKX under a q process (see details in the section below about new alpha features).
- A dependency on the system library
libcurlwas added for Linux. This dependency will be made optional in a future release.
Features & Fixes
pykx.Qclass has been added as the base class for
pykx.EmbeddedQ(the class for
pykx.EmbeddedQprocess now perisists its context between calls.
- The console now works over IPC.
- The query module now works over IPC. Because
Kobjects hold no reference to the
qinstance that created them (be it local or over IPC),
Ktables no longer have
deletemethods with themselves projected in as the first argument. That is to say, instead of writing
q.qsql.select(t, ...), where
pykx.qor an instance of
twas obtained from
- The context interface now works over IPC.
- Nulls and infinities are now handled as nulls and infinities, rather than as their underlying values.
pykx.Collection.has_infshave been added. Numpy, Pandas, and PyArrow handles integral nulls with masked arrays, and they handle temporal nulls with
NaNcontinues to be used for real/float nulls. The general Python representation (from
Kobjects for nulls and infinities.
pykx.Kobjects now either raises a
TypeError, or return the unambiguously correct result. For ambiguous cases such as
.all(), or a length check instead.
- Assignment to q reserved words or the q context now raises a
pykdb.adapt.adapt_list) now works in unlicensed mode.
q.sqlare now placeholders (set to
None). The query interface can be accessed from
QContextobjects are now context handlers, e.g.
with pykx.q.dbmaint: # operate in .dbmaint within this block. This context handler supports arbitrary nesting.
__getitem__now raises a
pykx.LicenseExceptionwhen used in unlicensed mode. Previously it worked for a few select types only. If running in unlicensed mode, one should perform all q indexing in the connected q process, and all Python indexing after converting the
Kobject to a Python/Numpy/Pandas/PyArrow object.
pykdb.ipc.Connection) objects now have an informative/idiomatic repr.
- Calls to
pykx.qnow support up to 8 arguments beyond the required query at position 0, similar to calling
pykx.QConnectioninstances. These arguments are applied to the result of the query.
- Embedded q is now used to count the number of rows a table has.
- All dynamic linking to
libehas been replaced by dynamic loading. As a result, the modules previously known as
adapt_unlicensedhave been unified under
- PyKX now attempts to initialize embedded q when
pykxis imported, rather than when
pykx.qis first accessed. As a result, the error-prone practice of supplying the
pykx.kdbsingleton class with arguments for embedded q is now impossible.
- Arguments for embedded q can now be supplied via the environment variable
$QARGSin the form of command-line arguments. For example,
QARGS='--unlicensed'causes PyKX to enter unlicensed mode when it is started, and
QARGS='-o 8'causes embedded q to use an offset from UTC of 8 hours. These could be combined as
QARGS='--unlicensed -o 8'.
- Added the
--licensedstartup flag (to be provided via the
$QARGSenvironment variable), which can be used to raise a
pykx.PyKXException(rather than emitting a warning) if PyKX fails to start in licensed mode (likely because of a missing/invalid q license).
- PyKX Linux wheels are now PEP 600 compliant, built to the
- Misc other bug fixes.
- Misc doc improvements.
- Converting nested lists from q to Python is much faster.
- Internally, PyKX now calls q functions with arguments directly instead of creating a
pykx.Functioninstance then calling it. This results in modest performance benefits in some cases.
- The context interface no longer loads every element of a context when the context is first accessed, thereby removing the computation spike, which could be particularly intense for large q contexts.
New Alpha Features
Alpha features are subject to change
Alpha features are not stable will be subject to changes without notice. Use at your own risk.
- q can now load PyKX by loading the q file
pykx.qcan be copied into
pykx.install_into_QHOME(). When loaded into q, it will define the
.pykxnamespace, which notably has
.pykx.pyeval. This allows for Python code to be run within q libraries and applications without some of the limitations of embedded q such as the lack of the q main loop, or the lack of timers. When q loads
pykx.q, it attempts to source the currently active Python environment by running
python, then fetching the environment details from it.