Function reference
The kdb+/HDF5 interface is a wrapper for kdb+ around the HDF Groups C API.
The following functions are exposed within the .hdf5
namespace, allowing users to convert data between the HDF5 data format and kdb+.
.hdf5 - HDF5 interface functionality
Create createFile Create a named HDF5 file createGroup Create a single or multiple group levels
Write writeAttr Write a named attribute associated to a group or dataset writeData Write data to a named dataset
Read readAttr Read data from named attribute readData Read data from a named dataset
Link createExternal Create a hard external link between locations in two files createHard Create a hard link between two locations in a file createSoft Create a soft link between two locations in a file delLink Delete an existing soft or hard link
Utility copyObject Copy a group or dataset to another file dataSize Size of a HDF5 dataset in MB errorOn Turn the HDF5 C API error signalling on errorOff Turn the HDF5 C API error signalling off (default) fileSize Size of a HDF5 in MB getAttrShape Dimensionality of an attribute getDataShape Dimensionality of a dataset gc Run garbage collect on free HDF5 lists of all types isAttr Whether this attribute exists ishdf5 Whether a file of HDF5 format isObject Whether object exists ls ls-like representation of the structure of a HDF5 file version Version of the HDF5 C API being used
For simplicity in each of the examples below it should be assumed that unless otherwise specified a file/group/dataset/attribute that is being manipulated exists and is valid and the results displayed coincide with correct execution in that example.
.hdf5.copyObject
Copy a HDF5 dataset or group from one file to another
Syntax: .hdf5.copyObject[fname;oname;fdest;odest]
Where
fname
is the name of the file containing the object to be copied as a stringoname
is the name of the object to be copied as a stringfdest
is the name of the destination file for the copied object as a stringodest
it the name of the path in the destination file to which the copied object is associated as a string
return null on successful execution, signals error if there are issues accessing files.
q).hdf5.writeData["test.h5";"dset";100?5]
q).hdf5.createFile["testing.h5"]
// Copy original object to a new file
q).hdf5.copyObject["test.h5";"dset";"testing.h5";"data"]
q).hdf5.readData["testing.h5";"data"]
4 0 2 1 2 1 2 3 2 4 1 0 2 4 1 2 0 1 1 2 1 0 0 1 ..
.hdf5.createExternal
Creates an external link to an object in a different file
Syntax: .hdf5.createExternal[lfile;lpath;tfile;tpath]
Where
lfile
is the name as a string of the HDF5 file containing the end point for the new external linklpath
is the name as a string of the object acting as the end point for the linktfile
is the name as a string of the target HDF5 file containing the target objecttpath
is the name as a string of the target object which is linked to lpath
returns null on successful execution and creates the external link.
// Produce a new .h5 file and link a path within this to a dataset in another h5 file
// using the create external link functionality within the api
q).hdf5.createFile["test_external.h5"]
q).hdf5.createExternal["test_external.h5";"/dset";"test.h5";"/dset"]
// Access the data from this new link
q).hdf5.readData["test_external.h5";"/dset"]
0.4707883 0.6346716
0.9672398 0.2306385
0.949975 0.439081
0.5759051 0.5919004
0.8481567 0.389056
.hdf5.createFile
Create a named HDF5 file
Syntax: .hdf5.createFile[fname]
Where fname
is the name of a HDF5 file as a string, returns null on successful creation of file. Attempting to create a file that already exists will result in an error
// Create named file
q).hdf5.createFile["test.h5"]
// Attempt to create this file again
q).hdf5.createFile["test.h5"]
'error creating file
.hdf5.createGroup
Create a group or set of groups within a file
Syntax: .hdf5.createGroup[fname;gname]
Where
fname
is the name of a HDF5 file as a stringgname
is the structure of the groups to be created as a string
returns null on successful execution and creates the desired groups.
// Create a single group
q).hdf5.createGroup["test.h5";"SingleGroup"]
// Create a set of groups
q).hdf5.createGroup["test.h5";"Group1/SubGroup1/SubGroup2"]
The following functions relate to writing of kdb+ data to HDF5 files as either attributes or datasets.
.hdf5.createHard
Create a hard link to a physical address in a file
Syntax: .hdf5.createHard[lfile;lpath;tpath]
Where
lfile
is the name of a HDF5 file as a stringlpath
is the name of the object acting as the end point for the link as a stringtpath
is the name of the target path for the new hard link as a string
returns null on successful execution and creates the hard link.
// Create a Group 'G1' to house the new hard link to the dataset
q).hdf5.createGroup["test.h5";"G1"]
// Create a hard link from existing 'dset' to 'G1/dset'
q).hdf5.createHard["test.h5";"G1/dset";"dset"]
// Access the data using the new link
q)5#.hdf5.readData["test.h5";"/G1/dset"]
0.4707883 0.6346716
0.9672398 0.2306385
0.949975 0.439081
0.5759051 0.5919004
0.8481567 0.389056
.hdf5.createSoft
Create a soft/symbolic link between two locations in a HDF5 file
Syntax: .hdf5.createSoft[lfile;lpath;tpath]
Where
lfile
is the name as a string of a HDF5 filelpath
is the name as a string of the object acting as the end point for the linktpath
is the name as a string of of the target path for the new soft link
returns null on successful execution and creates the specified soft link.
File paths
Neither lpath
nor tpath
need exist before creating the soft link. They can be created afterwards.
But if the links are to be contained within a group, the group must exist.
// Create a group 'G1' within the file 'test.h5'
q).hdf5.createGroup["test.h5";"G1"]
// Create a soft link (symbolic link) between two unpopulated points in the file
q).hdf5.createSoft["test.h5";"/G1/dset1";"/G1/dset2"]
// Write data to 'G1/dset2'
.hdf5.writeData["test.h5";"G1/dset2";4 5#20?10]
// Read the data from the location 'G1/dset1'
q).hdf5.readData["tets.h5";"/G1/dset1"]
8 1 9 5 4 6
6 1 8 5 4 9
2 7 0 1 9 2
1 8 8 1 7 2
.hdf5.dataSize
Size of an uncompressed HDF5 dataset in MB
Syntax: .hdf5.dataSize[fname;dname]
Where
fname
is the name of a HDF5 file as a stringdname
is the name of the dataset as a string
returns the size of a HDF5 dataset in MB.
q).hdf5.dataSize["test.h5";"dset"]
8f
.hdf5.delLink
Delete an assigned external hard or soft link
Syntax: .hdf5.delLink[lfile;lpath]
Where
lfile
is the name of a HDF5 file as a stringlpath
is the name of the link to be deleted as a string
returns a message indicating successful deletion of a link.
// Create a delete a soft link
q).hdf5.delLink["test.h5";"/G1/dset1"]
"Successfully deleted the link"
// Attempt to delete the link again
q).hdf5.delLink["test.h5";"/G1/dset1"]
'could not delete specified link
.hdf5.errorOff
Turn off printing of errors from HDF5 C API
Syntax: .hdf5.errorOff[]
returns null on successful execution.
The interface is initialized with HDF5 erroring turned off.
// Failing function called with HDF5 error ons
q).hdf5.createFile["test.h5"]
HDF5-DIAG: Error detected in HDF5 (1.10.5) thread 0:
#000: H5F.c line 444 in H5Fcreate(): unable to create file
major: File accessibilty
minor: Unable to open file
#001: H5Fint.c line 1558 in H5F_open(): unable to open file
major: File accessibilty
minor: Unable to open file
#002: H5FD.c line 734 in H5FD_open(): open failed
major: Virtual File Layer
minor: Unable to initialize object
#003: H5FDsec2.c line 346 in H5FD_sec2_open(): unable to open file:
major: File accessibilty
minor: Unable to open file
'error creating file
[0] .hdf5.createFile["test.h5"]
// Turn off errors and repeat the above function call
q).hdf5.errorOff[]
q).hdf5.createFile["test.h5"]
'error creating file
.hdf5.errorOn
Turns on printing of errors from HDF5 C API
Syntax: .hdf5.errorOn[]
returns null on successful execution.
The interface is initialized with HDF5 erroring turned off.
// Execute an invalid command with error printing off
q).hdf5.createFile["test.h5"]
'error creating file
// Turn error printing on and repeat command
q).hdf5.errorOn[]
q).hdf5.createFile["test.h5"]
HDF5-DIAG: Error detected in HDF5 (1.10.5) thread 0:
#000: H5F.c line 444 in H5Fcreate(): unable to create file
major: File accessibilty
minor: Unable to open file
#001: H5Fint.c line 1558 in H5F_open(): unable to open file
major: File accessibilty
minor: Unable to open file
#002: H5FD.c line 734 in H5FD_open(): open failed
major: Virtual File Layer
minor: Unable to initialize object
#003: H5FDsec2.c line 346 in H5FD_sec2_open(): unable to open file:
major: File accessibilty
minor: Unable to open file
'error creating file
.hdf5.fileSize
Size of a HDF5 file in MB
Syntax: .hdf5.fileSize[fname]
Where fname
is the name of a HDF5 file as a string returns the size of a HDF5 in MB.
q).hdf5.fileSize["test.h5"]
8.002048
.hdf5.gc
Garbage collect on free HDF5 lists of all types
Syntax: .hdf5.gc[]
returns 0i
on successful execution.
q).hdf5.gc[]
0i
.hdf5.getAttrShape
Get the shape of an attribute dataset
Syntax: .hdf5.getAttrShape[fname;oname;aname]
Where
fname
is the name as a string of a HDF5 fileoname
is the name as a string of an object (group/dataset) to which an attribute is associatedaname
is the name as a string of an attribute
returns an numerical list indicating the attribute shape.
q).hdf5.getAttrShape["test.h5";"dset";"Temperatures"]
1 10
.hdf5.getDataShape
Get the shape of a dataset
Syntax: .hdf5.getDataShape[fname;dname]
Where
fname
is the name of a HDF5 file as a stringdname
is the name of the dataset of interest as a string
returns a numerical list indicating the shape of the dataset.
q).hdf5.getDataShape["test.h5";"dset"]
2 4
.hdf5.isAttr
Does the named attribute exist
Syntax: .hdf5.isAttr[fname;oname;aname]
fname
is the name as a string of a HDF5 fileoname
is the name as a string of an object (group/dataset) to which the attribute is associatedaname
is the name as a string of the attribute
returns a boolean indicating whether the attribute exists.
q).hdf5.isAttr["test.h5";"dset";"Temperatures"]
1b
q).hdf5.isAttr["test.h5";"dset";"Temps"]
0b
.hdf5.ishdf5
Is the specified file a HDF5 file
Syntax: .hdf5.ishdf5[fname]
Where fname
is the name of a HDF5 file as a string, returns a boolean indicating if the file is a HDF5 file or not.
q).hdf5.ishdf5["test.h5"]
1b
q).hdf5.ishdf5["test.txt"]
0b
.hdf5.isObject
Whether an object is a group/dataset
Syntax: .hdf5.isObject[fname;oname]
Where
fname
is the name of a HDF5 file as a stringoname
is the name of an object (group/dataset) as a string
returns a boolean indicating whether the object exists.
// Dataset exists
q).hdf5.isObject["test.h5";"dset"]
1b
// Group exists
q).hdf5.isObject["test.h5";"G1"]
1b
// Object doesn't exist
q).hdf5.isObject["test.h5";"not_obj"]
0b
.hdf5.ls
Display the structure of a HDF5 file
Syntax: .hdf5.ls[fname]
Where fname
is the name of a HDF5 file as a string, displays the overall structure of the HDF5 file and returns a null.
q).hdf5.ls["test.h5"]
{
[Group](#hdf5group): G1 {
Group: G2 {
Dataset: group_dset
}
Dataset: dset1
}
[Dataset](#hdf5dataset): dset
}
.hdf5.readAttr
Read the data contained in a HDF5 attribute to kdb+
Syntax: .hdf5.readAttr[fname;oname;aname]
Where
fname
is the name of a HDF5 file as a stringoname
is the name of the object (group/dataset) with which the attribute being read from is associated as a stringaname
is the name of the attribute from which data is to be read as a string
returns a kdb+ dataset containing the data associated with the named HDF5 attribute
// Read numeric data from attribute
q).hdf5.readAttr["test.h5";"dset";"temperature"]
0.08123546 0.9367503
0.2782122 0.2392341
0.1508133 0.1567317
0.9785 0.7043314
0.9441671 0.7833686
// Read string data from attribute
q).hdf5.readAttr["test.h5";"dset";"Description"]
"This data relates to experiment 1"
"Time difference between values = 1ms"
.hdf5.readData
Read the data contained in a HDF5 dataset to kdb+
Syntax: .hdf5.readAttr[fname;dname]
Where
fname
is the name of a HDF5 file as a stringdname
is the name of the dataset which is to be read from as a string
returns a kdb+ dataset containing the data associated with a HDF5 dataset
// Read numeric data
q)5#.hdf5.readData["test.h5";"dset"]
7.263142 9.216436
1.809536 6.434637
2.907093 0.7347808
3.159526 3.410485
8.617972 5.548864
// Read string data
q).hdf5.readData["test.h5";"strdset"]
"Value 1 = 0.01"
"Value 2 = 0.02"
"Value 3 = 0.04"
// Read table
q).hdf5.readData["test.h5";"table"]
x x1 x2 x3 x4
------------------------------------------
0.2400771 0.7263287 1 08:12:43.345 apah
0.1039355 0.4886361 1 04:42:08.234 mlfb
0.2353326 0.6535973 0 17:54:05.495 fmbf
0.6423479 0.02810674 0 16:58:19.059 iibh
0.5778177 0.4449053 1 12:47:14.478 fpam
// Read dictionary
q).hdf5.readData["test.h5";"dict"]
x1| 0 1 2 3 4 5 6 7 8 9
x2| `nbmb`gbdn`ijca`khjj`gncf
x3| 2003.01.16D01:51:45.083828416 2002.11.25D05:50:34.964843088 2000.09.19D10.
The following functions allow you to link datasets/groups internally or externally to a HDF5 file.
Definitions of external, hard and soft links as they pertain to the HDF5 data format
.hdf5.version
Display C API major/minor/release versions
Syntax: .hdf5.version[]
Returns a dictionary with the major/minor and release versions of the HDF group’s C API.
q).hdf5.version[]
Major | 1
Minor | 10
Release| 5
.hdf5.writeAttr
Write a kdb+ dataset to a HDF5 attribute
Syntax: .hdf5.writeAttr[fname;oname;aname;dset]
Where
fname
is the name of a HDF5 file as a stringoname
is the name of an object (group/dataset) to which an attribute is to be written to as a stringaname
is the name of the attribute to be written to as a stringdset
dataset which is being written to HDF5 format
returns null on successful writing of data to attribute. Failure to write to the attribute will signal an appropriate error.
Watch out
To write an attribute to a dataset/group the file and dataset/group specified must already exist.
Attributes can only be written to with data types where the mapping q => HDF5 => q does not require a datatype_kdb
attribute associated with the data.
// Writing attribute whe file does not exist
q).hdf5.writeAttr["test.h5";"dset";"temperature";10 2#20?1f]
'error opening file
// Writing attribute when dataset or group does not exist
q).hdf5.writeAttr["test.h5";"dset";"temperature";10 2#20?1f]
'error opening dataset/group
// Writing data with an inappropriate type
q).hdf5.writeAttr["test.h5";"dset";"times";10 2#20?0t]
'kdb+ type can not be mapped to an appropriate attribute
// Successful execution
q).hdf5.writeAttr["test.h5";"dset";"temperature";10 2#20?1f]
.hdf5.writeData
Write a kdb+ dataset to a HDF5 dataset
Syntax: .hdf5.writeData[fname;dname;dset]
Where
fname
is the name of a HDF5 file as a stringdname
is the name as a string of the dataset which is to be written todset
dataset which is being written to HDF5 format
returns null on successful writing of data to dataset. Failure to write to the dataset will signal an appropriate error.
To write a dataset to a file, the file must already exist."
// Writing dataset when file does not exist
q).hdf5.writeData["test.h5";"dset";10 2#20?10f]
'error opening file
// Write numeric dataset
q).hdf5.writeData["test.h5";"dset";10 2#20?10]
// Write string data
q).hdf5.writeData["test.h5";"strdset";enlist "test string"]
// Write tabular data
q).hdf5.writeData["test.h5";"table";([]100?1f;100?1f;100?0b;100?0t;100?`4)]
// Write dictionary data
q).hdf5.writeData["test.h5";"dict";`x1`x2`x3!(til 10;5?`4;100?0p)]
The following functions relate to the reading of HDF5 datasets/attributes to an equivalent kdb+ representation.