Convert temporal data types in PyKX
This page provides details on how to convert temporal data types in PyKX.
Converting temporal data types in PyKX involves handling timestamp/datetime types and duration types.
Timestamp/Datetime types
When converting temporal types, note that Python and q use different epoch values:
- q: Epoch starts at 2000.
- Python: Epoch starts at 1970.
Note
The following details focus on NumPy
but similar considerations should be taken into account when converting Python, Pandas, and PyArrow objects.
The Nulls and Infinities page should also be consulted for information.
The 30-year epoch offset means there are times which are unreachable in one or the other language:
TimestampVector | datetime64[ns] | |
---|---|---|
Minimum value | 1707.09.22D00:12:43.145224194 |
1677-09-21T00:12:43.145224194 |
Maximum value | 2292.04.10D23:47:16.854775806 |
2262-04-11T23:47:16.854775807 |
As such the range of times which can be directly converted should be considered:
- Minimum value:
1707-09-22T00:12:43.145224194
- Maximum value:
2262-04-11T23:47:16.854775807
As mentioned on the Nulls and infinitespage, most q data types have null, negative infinity, and infinity values.
q representation | datetime64[ns] | |
---|---|---|
Null | 0Np |
NaT |
Negative Infinity | -0Wp |
1707-09-22T00:12:43.145224193 |
Infinity | 0Wp |
Overflow cannot be represented |
Converting from q to NumPy using .np()
, 0Np
and -0Wp
results in meaningful values. However, using 0Wp
causes an overflow:
>>> kx.q('0N -0W 0Wp').np()
array(['NaT', '1707-09-22T00:12:43.145224193', '1707-09-22T00:12:43.145224191'], dtype='datetime64[ns]')
Converting to q using toq
by default, results in meaningful values only for the NumPy maximum values:
>>> arr = np.array(['NaT', '1677-09-21T00:12:43.145224194', '2262-04-11T23:47:16.854775807'], dtype='datetime64[ns]')
>>> kx.toq(arr)
pykx.TimestampVector(pykx.q('2262.04.11D23:47:16.854775808 2262.04.11D23:47:16.854775810 2262.04.11D23:47:16.854775807'))
To additionally convert NaT
, use the handle_nulls
keyword:
>>> arr = np.array(['NaT', '1677-09-21T00:12:43.145224194', '2262-04-11T23:47:16.854775807'], dtype='datetime64[ns]')
>>> kx.toq(arr, handle_nulls=True)
pykx.TimestampVector(pykx.q('0N 2262.04.11D23:47:16.854775810 2262.04.11D23:47:16.854775807'))
Use raw=True
to request that the epoch offset is not applied. This allows for the underlying numeric values to be accessed directly:
>>> kx.q('0N -0W 0Wp').np(raw=True)
array([-9223372036854775808, -9223372036854775807, 9223372036854775807])
Passing back to q with toq
these are presented as the long null, negative infinity, and infinity:
>>> kx.toq(kx.q('0N -0W 0Wp').np(raw=True))
pykx.LongVector(pykx.q('0N -0W 0W'))
Pass ktype
during toq
to specify desired types:
>>> kx.toq(pd.DataFrame(data= {'d':np.array(['2020-09-08T07:06:05'], dtype='datetime64[s]')}), ktype={'d':kx.DateVector})
pykx.Table(pykx.q('
d
----------
2020.09.08
'))
Note
- Dictionary based conversion is only supported when operating in licensed mode.
- Data is first converted to the default type and then cast to the desired type.
Info
-
In NumPy further data types exist
datetime64[us]
,datetime64[ms]
,datetime64[s]
which due to their lower precision have a wider range of dates they can represent. When converted using to q usingtoq
these all present as qTimestamp
type and as such only dates within the range this data type can represent should be converted. -
Pandas 2.* changes behavior and conversions should be reviewed as part of an upgrade of this package. PyKX to Pythonic data type mapping includes examples showing differences seen when calling
.pd()
.
Duration types
Duration types do not have the issue of epoch offsets, but some range limitations exist when converting between Python and PyKX.
kx.SecondVector
and kx.MinuteVector
convert to timedelta64[s]
:
q representation | timedelta64[s] | |
---|---|---|
kx.SecondVector Null |
0Nv |
NaT |
kx.SecondVector Negative Infinity |
-0Wv |
-24856 days +20:45:53 |
kx.SecondVector Infinity |
0Wv |
24855 days 03:14:07 |
kx.MinuteVector Null |
0Nu |
NaT |
kx.MinuteVector Negative Infinity |
-0Wu |
-1491309 days +21:53:00 |
kx.MinuteVector Infinity |
0Wu |
1491308 days 02:07:00 |
When converting Python to q using toq
, timedelta64[s]
is 64 bit and converts to kx.SecondVector
which is 32 bit:
SecondVector | timedelta64[s] | |
---|---|---|
Minimum value | **:14:06 |
106751991167300 days 15:30:07 |
Maximum value | -**:14:06 |
-106751991167301 days +08:29:53 |
As such, you should consider the range of times which can be directly converted:
- Minimum value:
-24856 days +20:45:54
- Maximum value:
24855 days 03:14:06
q does not display values of second type over 99:59:59
, beyond this **
is displayed in the hour field.
The data is still stored correctly and displays when converted:
>>> kx.q('99:59:59 +1')
pykx.SecondAtom(pykx.q('**:00:00'))
>>> kx.q('99:59:59 +1').pd()
Timedelta('4 days 04:00:00')