Function reference¶

The kdb+/HDF5 interface is a wrapper for kdb+ around the HDF Groups C API.

The following functions are exposed within the .hdf5 namespace, allowing users to convert data between the HDF5 data format and kdb+.

.hdf5 - HDF5 interface functionality

Create createFile Create a named HDF5 file createGroup Create a single or multiple group levels

Write writeAttr Write a named attribute associated to a group or dataset writeData Write data to a named dataset

Read readAttr Read data from named attribute readData Read data from a named dataset

Link createExternal Create a hard external link between locations in two files createHard Create a hard link between two locations in a file createSoft Create a soft link between two locations in a file delLink Delete an existing soft or hard link

Utility copyObject Copy a group or dataset to another file dataSize Size of a HDF5 dataset in MB errorOn Turn the HDF5 C API error signalling on errorOff Turn the HDF5 C API error signalling off (default) fileSize Size of a HDF5 in MB getAttrShape Dimensionality of an attribute getDataShape Dimensionality of a dataset gc Run garbage collect on free HDF5 lists of all types isAttr Whether this attribute exists ishdf5 Whether a file of HDF5 format isObject Whether object exists isThreadsafe Whether HDF5 lib is built with multithread support ls Representation of the structure of a HDF5 file version Version of the HDF5 C API being used

For simplicity in each of the examples below it should be assumed that unless otherwise specified a file/group/dataset/attribute that is being manipulated exists and is valid and the results displayed coincide with correct execution in that example.

`copyObject`¶

Copy a HDF5 dataset or group from one file to another

.hdf5.copyObject[fname;oname;fdest;odest]

Where

fname is the name of the file containing the object to be copied as a string
oname is the name of the object to be copied as a string
fdest is the name of the destination file for the copied object as a string
odest it the name of the path in the destination file to which the copied object is associated as a string

return null on successful execution, signals error if there are issues accessing files.

q).hdf5.writeData["test.h5";"dset";100?5]
q).hdf5.createFile["testing.h5"]
// Copy original object to a new file
q).hdf5.copyObject["test.h5";"dset";"testing.h5";"data"]
q).hdf5.readData["testing.h5";"data"]
4 0 2 1 2 1 2 3 2 4 1 0 2 4 1 2 0 1 1 2 1 0 0 1 ..

`createExternal`¶

Creates an external link to an object in a different file

.hdf5.createExternal[lfile;lpath;tfile;tpath]

Where

lfile is the name as a string of the HDF5 file containing the end point for the new external link
lpath is the name as a string of the object acting as the end point for the link
tfile is the name as a string of the target HDF5 file containing the target object
tpath is the name as a string of the target object which is linked to lpath

returns null on successful execution and creates the external link.

// Produce a new .h5 file and link a path within this to a dataset in another h5 file
// using the create external link functionality within the api
q).hdf5.createFile "test_external.h5"
q).hdf5.createExternal["test_external.h5";"/dset";"test.h5";"/dset"]
// Access the data from this new link
q).hdf5.readData["test_external.h5";"/dset"]
0.4707883 0.6346716
0.9672398 0.2306385
0.949975  0.439081
0.5759051 0.5919004
0.8481567 0.389056

`createFile`¶

Create a named HDF5 file

.hdf5.createFile fname

Where fname is the name of a HDF5 file as a string, returns null on successful creation of file. Attempting to create a file that already exists will result in an error

// Create named file
q).hdf5.createFile "test.h5"

// Attempt to create this file again
q).hdf5.createFile "test.h5"
'error creating file

`createGroup`¶

Create a group or set of groups within a file

.hdf5.createGroup[fname;gname]

Where

fname is the name of a HDF5 file as a string
gname is the structure of the groups to be created as a string

returns null on successful execution and creates the desired groups.

// Create a single group
q).hdf5.createGroup["test.h5";"SingleGroup"]
// Create a set of groups
q).hdf5.createGroup["test.h5";"Group1/SubGroup1/SubGroup2"]

`createHard`¶

Create a hard link to a physical address in a file

.hdf5.createHard[lfile;lpath;tpath]

Where

lfile is the name of a HDF5 file as a string
lpath is the name of the object acting as the end point for the link as a string
tpath is the name of the target path for the new hard link as a string

returns null on successful execution and creates the hard link.

// Create a Group 'G1' to house the new hard link to the dataset
q).hdf5.createGroup["test.h5";"G1"]
// Create a hard link from existing 'dset' to 'G1/dset'
q).hdf5.createHard["test.h5";"G1/dset";"dset"]
// Access the data using the new link
q)5#.hdf5.readData["test.h5";"/G1/dset"]
0.4707883 0.6346716
0.9672398 0.2306385
0.949975  0.439081
0.5759051 0.5919004
0.8481567 0.389056

`createSoft`¶

Create a soft/symbolic link between two locations in a HDF5 file

.hdf5.createSoft[lfile;lpath;tpath]

Where

lfile is the name as a string of a HDF5 file
lpath is the name as a string of the object acting as the end point for the link
tpath is the name as a string of of the target path for the new soft link

returns null on successful execution and creates the specified soft link.

File paths

Neither lpath nor tpath need exist before creating the soft link. They can be created afterwards.

But if the links are to be contained within a group, the group must exist.

// Create a group 'G1' within the file 'test.h5'
q).hdf5.createGroup["test.h5";"G1"]

// Create a soft link (symbolic link) between two unpopulated points in the file
q).hdf5.createSoft["test.h5";"/G1/dset1";"/G1/dset2"]

// Write data to 'G1/dset2'
.hdf5.writeData["test.h5";"G1/dset2";4 5#20?10]

// Read the data from the location 'G1/dset1'
q).hdf5.readData["test.h5";"/G1/dset1"]
8 1 9 5 4 6
6 1 8 5 4 9
2 7 0 1 9 2
1 8 8 1 7 2

`dataSize`¶

Size of an uncompressed HDF5 dataset in MB

.hdf5.dataSize[fname;dname]

Where

fname is the name of a HDF5 file as a string
dname is the name of the dataset as a string

returns the size of a HDF5 dataset in MB.

q).hdf5.dataSize["test.h5";"dset"]
8f

`delLink`¶

Delete an assigned external hard or soft link

.hdf5.delLink[lfile;lpath]

Where

lfile is the name of a HDF5 file as a string
lpath is the name of the link to be deleted as a string

returns a message indicating successful deletion of a link.

// Delete a soft link
q).hdf5.delLink["test.h5";"/G1/dset1"]
"Successfully deleted the link"
// Attempt to delete the link again
q).hdf5.delLink["test.h5";"/G1/dset1"]
'could not delete specified link

`errorOff`¶

Turn off printing of errors from HDF5 C API

.hdf5.errorOff[]

returns null on successful execution.

The interface is initialized with HDF5 erroring turned off.

// Failing function called with HDF5 error ons
q).hdf5.createFile "test.h5"
HDF5-DIAG: Error detected in HDF5 (1.10.5) thread 0:
  #000: H5F.c line 444 in H5Fcreate(): unable to create file
    major: File accessibilty
    minor: Unable to open file
  #001: H5Fint.c line 1558 in H5F_open(): unable to open file
    major: File accessibilty
    minor: Unable to open file
  #002: H5FD.c line 734 in H5FD_open(): open failed
    major: Virtual File Layer
    minor: Unable to initialize object
  #003: H5FDsec2.c line 346 in H5FD_sec2_open(): unable to open file:
    major: File accessibilty
    minor: Unable to open file
'error creating file
  [0]  .hdf5.createFile "test.h5"

// Turn off errors and repeat the above function call
q).hdf5.errorOff[]
q).hdf5.createFile "test.h5"
'error creating file

`errorOn`¶

Turns on printing of errors from HDF5 C API

.hdf5.errorOn[]

returns null on successful execution.

The interface is initialized with HDF5 erroring turned off.

// Execute an invalid command with error printing off
q).hdf5.createFile "test.h5"
'error creating file

// Turn error printing on and repeat command
q).hdf5.errorOn[]
q).hdf5.createFile "test.h5"
HDF5-DIAG: Error detected in HDF5 (1.10.5) thread 0:
  #000: H5F.c line 444 in H5Fcreate(): unable to create file
    major: File accessibilty
    minor: Unable to open file
  #001: H5Fint.c line 1558 in H5F_open(): unable to open file
    major: File accessibilty
    minor: Unable to open file
  #002: H5FD.c line 734 in H5FD_open(): open failed
    major: Virtual File Layer
    minor: Unable to initialize object
  #003: H5FDsec2.c line 346 in H5FD_sec2_open(): unable to open file:
    major: File accessibilty
    minor: Unable to open file
'error creating file

`fileSize`¶

Size of a HDF5 file in MB

.hdf5.fileSize fname

Where fname is the name of a HDF5 file as a string returns the size of a HDF5 in MB.

q).hdf5.fileSize "test.h5"
8.002048

`gc`¶

Garbage collect on free HDF5 lists of all types

.hdf5.gc[]

returns 0i on successful execution.

q).hdf5.gc[]
0i

`getAttrShape`¶

Get the shape of an attribute dataset

.hdf5.getAttrShape[fname;oname;aname]

Where

fname is the name as a string of a HDF5 file
oname is the name as a string of an object (group/dataset) to which an attribute is associated
aname is the name as a string of an attribute

returns an numerical list indicating the attribute shape.

q).hdf5.getAttrShape["test.h5";"dset";"Temperatures"]
1 10

`getDataShape`¶

Get the shape of a dataset

.hdf5.getDataShape[fname;dname]

Where

fname is the name of a HDF5 file as a string
dname is the name of the dataset of interest as a string

returns a numerical list indicating the shape of the dataset.

q).hdf5.getDataShape["test.h5";"dset"]
2 4

`isAttr`¶

Does the named attribute exist

.hdf5.isAttr[fname;oname;aname]

fname is the name as a string of a HDF5 file
oname is the name as a string of an object (group/dataset) to which the attribute is associated
aname is the name as a string of the attribute

returns a boolean indicating whether the attribute exists.

q).hdf5.isAttr["test.h5";"dset";"Temperatures"]
1b
q).hdf5.isAttr["test.h5";"dset";"Temps"]
0b

`ishdf5`¶

Is the specified file a HDF5 file

.hdf5.ishdf5 fname

Where fname is the name of a HDF5 file as a string, returns a boolean indicating if the file is a HDF5 file or not.

q).hdf5.ishdf5 "test.h5"
1b
q).hdf5.ishdf5 "test.txt"
0b

`isObject`¶

Whether an object is a group/dataset

.hdf5.isObject[fname;oname]

Where

fname is the name of a HDF5 file as a string
oname is the name of an object (group/dataset) as a string

returns a boolean indicating whether the object exists.

// Dataset exists
q).hdf5.isObject["test.h5";"dset"]
1b
// Group exists
q).hdf5.isObject["test.h5";"G1"]
1b
// Object doesn't exist
q).hdf5.isObject["test.h5";"not_obj"]
0b

`isThreadsafe`¶

Whether HDF5 lib is build with multi thread support

.hdf5.isThreadsafe[]

Returns a boolean indicating whether the HDF5 library was build with multithread support.

`ls`¶

Display the structure of a HDF5 file

.hdf5.ls fname

Where fname is the name of a HDF5 file as a string, returns a dict of the overall structure of the HDF5 file.

q).hdf5.ls "test.h5"
type | group
name | `Group1
value| `type`name`value!(`group`group;(`SG1;`SG2);((`symbol$())!();`type`name`value!(,`group;,`SG3;,(`symbol$())!())))

`readAttr`¶

Read the data contained in a HDF5 attribute to kdb+

.hdf5.readAttr[fname;oname;aname]

Where

fname is the name of a HDF5 file as a string
oname is the name of the object (group/dataset) with which the attribute being read from is associated as a string
aname is the name of the attribute from which data is to be read as a string

returns a kdb+ dataset containing the data associated with the named HDF5 attribute

// Read numeric data from attribute
q).hdf5.readAttr["test.h5";"dset";"temperature"]
0.08123546 0.9367503 
0.2782122  0.2392341 
0.1508133  0.1567317 
0.9785     0.7043314 
0.9441671  0.7833686 

// Read string data from attribute 
q).hdf5.readAttr["test.h5";"dset";"Description"]
"This data relates to experiment 1"
"Time difference between values = 1ms"

`readData`¶

Read the data contained in a HDF5 dataset to kdb+

.hdf5.readAttr[fname;dname]

Where

fname is the name of a HDF5 file as a string
dname is the name of the dataset which is to be read from as a string

returns a kdb+ dataset containing the data associated with a HDF5 dataset

// Read numeric data
q)5#.hdf5.readData["test.h5";"dset"]
7.263142  9.216436 
1.809536  6.434637 
2.907093  0.7347808
3.159526  3.410485 
8.617972  5.548864 

// Read string data
q).hdf5.readData["test.h5";"strdset"]
"Value 1 = 0.01"
"Value 2 = 0.02"
"Value 3 = 0.04"

// Read table
q).hdf5.readData["test.h5";"table"]
x          x1         x2 x3           x4  
------------------------------------------
0.2400771  0.7263287  1  08:12:43.345 apah
0.1039355  0.4886361  1  04:42:08.234 mlfb
0.2353326  0.6535973  0  17:54:05.495 fmbf
0.6423479  0.02810674 0  16:58:19.059 iibh
0.5778177  0.4449053  1  12:47:14.478 fpam

// Read dictionary
q).hdf5.readData["test.h5";"dict"]
x1| 0 1 2 3 4 5 6 7 8 9
x2| `nbmb`gbdn`ijca`khjj`gncf
x3| 2003.01.16D01:51:45.083828416 2002.11.25D05:50:34.964843088 2000.09.19D10.

Definitions of external, hard and soft links as they pertain to the HDF5 data format

`version`¶

Display C API major/minor/release versions

.hdf5.version[]

Returns a dictionary with the major/minor and release versions of the HDF group’s C API.

q).hdf5.version[]
Major  | 1
Minor  | 10
Release| 5

`writeAttr`¶

Write a kdb+ dataset to a HDF5 attribute

.hdf5.writeAttr[fname;oname;aname;dset]

Where

fname is the name of a HDF5 file as a string
oname is the name of an object (group/dataset) to which an attribute is to be written to as a string
aname is the name of the attribute to be written to as a string
dset dataset which is being written to HDF5 format

returns null on successful writing of data to attribute. Failure to write to the attribute will signal an appropriate error.

Watch out

To write an attribute to a dataset/group the file and dataset/group specified must already exist.

Attributes can only be written to with data types where the mapping q => HDF5 => q does not require a datatype_kdb attribute associated with the data.

// Writing attribute whe file does not exist
q).hdf5.writeAttr["test.h5";"dset";"temperature";10 2#20?1f]
'error opening file

// Writing attribute when dataset or group does not exist
q).hdf5.writeAttr["test.h5";"dset";"temperature";10 2#20?1f]
'error opening dataset/group

// Writing data with an inappropriate type
q).hdf5.writeAttr["test.h5";"dset";"times";10 2#20?0t]
'kdb+ type can not be mapped to an appropriate attribute

// Successful execution
q).hdf5.writeAttr["test.h5";"dset";"temperature";10 2#20?1f]

`writeData`¶

Write a kdb+ dataset to a HDF5 dataset

.hdf5.writeData[fname;dname;dset]

Where

fname is the name of a HDF5 file as a string
dname is the name as a string of the dataset which is to be written to
dset dataset which is being written to HDF5 format

returns null on successful writing of data to dataset. Failure to write to the dataset will signal an appropriate error.

To write a dataset to a file, the file must already exist.

// Writing dataset when file does not exist
q).hdf5.writeData["test.h5";"dset";10 2#20?10f]
'error opening file

// Write numeric dataset
q).hdf5.writeData["test.h5";"dset";10 2#20?10]

// Write string data
q).hdf5.writeData["test.h5";"strdset";enlist "test string"]

// Write tabular data
q).hdf5.writeData["test.h5";"table";([]100?1f;100?1f;100?0b;100?0t;100?`4)]

// Write dictionary data
q).hdf5.writeData["test.h5";"dict";`x1`x2`x3!(til 10;5?`4;100?0p)]

Function reference¶

copyObject¶

createExternal¶

createFile¶

createGroup¶

createHard¶

createSoft¶

dataSize¶

delLink¶

errorOff¶

errorOn¶

fileSize¶

gc¶

getAttrShape¶

getDataShape¶

isAttr¶

ishdf5¶

isObject¶

isThreadsafe¶

ls¶

readAttr¶

readData¶

version¶

writeAttr¶

writeData¶

`copyObject`¶

`createExternal`¶

`createFile`¶

`createGroup`¶

`createHard`¶

`createSoft`¶

`dataSize`¶

`delLink`¶

`errorOff`¶

`errorOn`¶

`fileSize`¶

`gc`¶

`getAttrShape`¶

`getDataShape`¶

`isAttr`¶

`ishdf5`¶

`isObject`¶

`isThreadsafe`¶

`ls`¶

`readAttr`¶

`readData`¶

`version`¶

`writeAttr`¶

`writeData`¶