Reading and Writing files with PyKX
pykx.read
QReader
QReader(q)
Read data using q.
csv
csv(
path,
types=None,
delimiter=",",
as_table=True,
filter_type=None,
filter_columns=None,
custom=None,
)
Reads a CSV file as a table or dictionary.
Column types are guessed if not provided.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
path |
Union[str, Path, k.SymbolAtom]
|
The path to the CSV file. |
required |
types |
Optional[Union[bytes, k.CharAtom, k.CharVector]]
|
Can be a dictionary of columns and their types or a |
None
|
delimiter |
Union[str, bytes, k.CharAtom]
|
A single character representing the delimiter between values. |
','
|
as_table |
Union[bool, k.BooleanAtom]
|
|
True
|
filter_type |
Union[str, k.CharVector]
|
Can be |
None
|
filter_columns |
Union[str, list, k.CharVector, k.SymbolAtom, k.SymbolVector]
|
Used in tandem with |
None
|
custom |
dict
|
A dictionary used to change default values in csvutil.q. |
None
|
Returns:
Type | Description |
---|---|
Union[k.Table, k.Dictionary]
|
The CSV data as a |
See Also
CSV Type Guessing Table
Type Character | Type | Condition(s) |
---|---|---|
* | List | - Any type of width greater than 30. - Remaining unknown types. |
B | BooleanAtom | - Matching Byte or Char, maxwidth 1, no decimal points, at least 1 of [0fFnN] and 1 of [1tTyY] in columns.- Matching Byte or Char, maxwidth 1, no decimal points, all elements in [01tTfFyYnN] . |
G | GUIDAtom | - Matches GUID-like structure. - Matches structure wrapped in { } . |
X | ByteAtom | - Maxwidth of 2, comprised of [0-9] AND [abcdefABCDEF] . |
H | ShortAtom | - Matches Integer with maxwidth less than 7. |
I | IntAtom | - Numerical of size between 7 and 15 with exactly 3 decimal points (IP Address). - Matches Long with maxwidth less than 12. |
J | LongAtom | - Numerical, no decimal points, all elements +- or 0-9 . |
E | RealAtom | - Matches float with maxwidth less than 9. |
F | FloatAtom | - Numerical, maxwidth greater than 2, fewer than 2 decimal points, / present.- Numerical, fewer than 2 decimal points, maxwidth greater than 1. |
C | CharAtom | - Empty columns. Remaining unknown types of size 1. |
S | SymbolAtom | - Remaining unknown types of maxwidth 2-11 and granularity of less than 10. |
P | TimestampAtom | - Numerical, maxwidth 11-29, fewer than 4 decimals matching YYYY[./-]MM[./-]DD |
M | MonthAtom | - Matching either numerical, Int, Byte, Real or Float, fewer than 2 decimal points, maxwidth 4-7 |
D | DateAtom | - Matching Integer, maxwidth 6 or 8. - Numerical, 0 decimal points, maxwidth 8-10. - Numerical, 2 decimal points, maxwidth 8-10. - No decimal points maxwidth 5-9, matching date with 3 letter month code eg.(9nov1989). |
N | TimespanAtom | - Numerical, maxwidth 15, no decimal points, all values 0-9 .- Numerical, maxwidth 3-29, 1 decimal point, matching *[0-9]D[0-9]* .- Numerical, maxwidth 3-28, 1 decimal point. |
U | MinuteAtom | - Matching Byte, maxwidth 4, matching [012][0-9][0-5][0-9] .- Numerical, maxwidth 4 or 5, no decimal points, matching *[0-9]:[0-5][0-9] . |
V | SecondAtom | - Matching Integer, maxwidth 6, matching [012][0-9][0-5][0-9][0-5][0-9] .- Matching Time, maxwidth 7 or 8, no decimal points. |
T | TimeAtom | - Numerical, maxwidth 9, no decimal points, all values numeric. - Numerical, maxwidth 7 - 12, fewer than 2 decimal points, matching [0-9]:[0-5][0-9]:[0-5][0-9] .- Matching Real or Float, maxwidth 7-12, 1 decimal point, matching [0-9][0-5][0-9][0-5][0-9] . |
Examples:
Read a comma seperated CSV file into a pykx.Table
guessing the datatypes of each
column.
table = q.read.csv('example.csv')
Read a tab seperated CSV file into a pykx.Table
while specifying the columns
datatypes to be a pykx.SymbolVector
followed by two pykx.LongVector
columns.
table = q.read.csv('example.csv', 'SJJ', ' ')
Read a comma separated CSV file into a pykx.Dictionary
, guessing the datatypes of
each column.
table = q.read.csv('example.csv', None, None, False)
Read a comma separated CSV file specifying the type of the three columns
named x1
, x2
and x3
to be of type Integer
, GUID
and Timestamp
.
table = q.read.csv('example.csv', {'x1':kx.IntAtom,'x2':kx.GUIDAtom,'x3':kx.TimestampAtom})
Read a comma separated CSV file specifying only columns that include the word "name" in them.
table = q.read.csv('example.csv', filter_type = "like", filter_columns = '*name*')
Read a comma separated CSV file changing the guessing variables to change the number of lines read and used to guess the type of the column.
table = q.read.csv('example.csv', custom = {"READLINES":1000})
splayed
splayed(root, name)
Loads a splayed table.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
root |
Union[str, Path, k.SymbolAtom]
|
The path to the root directory of the splayed table. |
required |
name |
Union[str, k.SymbolAtom]
|
The name of the table to read. |
required |
Returns:
Type | Description |
---|---|
k.SplayedTable
|
The splayed table as a |
See Also
Examples:
Reads a splayed table named t
found within the current directory
table = q.read.splayed('.', 't')
Reads a splayed table named splayed
found within the /tmp
directory
table = q.read.splayed('/tmp', 'splayed')
fixed
fixed(path, types, widths)
Loads a file of typed data with fixed-width fields.
It is expected that there will either be a newline after every record, or none at all.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
path |
Union[str, Path, k.SymbolAtom]
|
The path to the file containing the fixed-width field data. |
required |
types |
Union[bytes, k.CharVector]
|
A string of uppercase characters representing the types. Space is used to drop a column. |
required |
widths |
Union[List[int], k.LongVector]
|
The width in bytes of each field. |
required |
Returns:
Type | Description |
---|---|
k.List
|
The data as a |
Examples:
Read a file of fixed width data into a pykx.List
of two pykx.LongVectors
the first
with a size of 1 character and the second with a size of 2 characters.
data = q.read.fixed('example_file', [b'J', b'J'], [1, 2])
json
json(path)
Reads a JSON file into a k.Table.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
path |
Union[str, Path, k.SymbolAtom]
|
The path to the JSON file. |
required |
Returns:
Type | Description |
---|---|
JSONKTypes
|
The JSON data as a |
See Also
Examples:
Read a JSON file.
data = q.read.json('example.json')
serialized
serialized(path)
Reads a binary file containing serialized q data.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
path |
Union[str, Path, k.SymbolAtom]
|
The path to the q data file. |
required |
Returns:
Type | Description |
---|---|
k.K
|
The q data file converted to a |
See Also
Examples:
Read a q data file containing a serialized table into a pykx.Table
object.
table = q.read.serialized('q_table_file')
pykx.write
QWriter
QWriter(q)
Write data using q.
splayed
splayed(root, name, table)
Splays and writes a q table to disk.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
root |
Union[str, Path, k.SymbolAtom]
|
The path to the root directory of the splayed table. |
required |
name |
Union[str, k.SymbolAtom]
|
The name of the table, which will be written to disk. |
required |
table |
Union[k.Table, pd.DataFrame]
|
A table-like object to be written as a splayed table. |
required |
Returns:
Type | Description |
---|---|
Path
|
The path to the splayed table on disk. |
See Also
Examples:
Write a pandas DataFrame
to disk as a splayed table in the current directory.
df = pd.DataFrame([[x, 2 * x] for x in range(5)])
q.write.splayed('.', 'splayed_table', df)
Write a pykx.Table
to disk as a splayed table at /tmp/splayed_table
.
table = q('([] a: 10 20 30 40; b: 114 113 98 121)')
q.write.splayed('/tmp', 'splayed_table', table)
serialized
serialized(path, data)
Writes a q object to a binary data file using q serialization.
This method is a wrapper around the q function set
, and as with any q function, arguments
which are not pykx.K
objects are automatically converted into them.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
path |
Union[str, Path, k.SymbolAtom]
|
The path to write the q object to. |
required |
data |
Any
|
An object that will be converted to q, then serialized to disk. |
required |
Returns:
Type | Description |
---|---|
Path
|
A |
See Also
Examples:
Serialize and write a pandas.DataFrame
to disk in the current directory.
df = q('([] a: til 5; b: 2 * til 5)').pd()
q.write.serialized('serialized_table', df)
Serialize and write a Python int
to disk in the current directory.
q.write.serialized('serialized_int', 145)
csv
csv(path, table, delimiter=',')
Writes a given table to a CSV file.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
path |
Union[str, Path, k.SymbolAtom]
|
The path to the CSV file. |
required |
delimiter |
Optional[Union[str, bytes, k.CharAtom]]
|
A single character representing the delimeter between values. |
','
|
table |
Union[k.Table, pd.DataFrame]
|
A table like object to be written as a csv file. |
required |
Returns:
Type | Description |
---|---|
Path
|
A |
See Also
Examples:
Write a pandas DataFrame
to disk as a csv file in the current directory using a
comma as a seperator between values.
df = q('([] a: til 5; b: 2 * til 5)').pd()
q.write.csv('example.csv', df)
Write a pykx.Table
to disk as a csv file in the current directory using a tab as a
seperator between values.
table = q('([] a: 10 20 30 40; b: 114 113 98 121)')
q.write.csv('example.csv', table, ' ')
json
json(path, data)
Writes a JSON representation of the given q object to a file.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
path |
Union[str, Path, k.SymbolAtom]
|
The path to the JSON file. |
required |
data |
Any
|
Any type to be serialized and written as a JSON file. |
required |
Returns:
Type | Description |
---|---|
Path
|
A |
See Also
Examples:
Convert a pandas Dataframe
to JSON and then write it to disk in the current
directory.
df = q('([] a: til 5; b: 2 * til 5)').pd()
q.write.json('example.json', df)
Convert a Python int
to JSON and then write it to disk in the current directory.
q.write.json('example.json', 143)
Convert a Python dictionary
to JSON and then write it to disk in the current
directory.
dictionary = {'a': 'hello', 'b':'pykx', 'c':2.71}
q.write.json('example.json', dictionary)