Sparse array support in HDF5
HDF5 provides indexed storage: http://www.hdfgroup.org/HDF5/doc/TechNotes/RawDStorage.html
One workaround is to create the dataset with a compression
option. For example, in Python using h5py:
import h5py
f = h5py.File('my.h5', 'w')
d = f.create_dataset('a', dtype='f', shape=(512, 512, 512), fillvalue=-999.,
compression='gzip', compression_opts=9)
d[3, 4, 5] = 6
f.close()
The resulting file is 4.5 KB. Without compression, this same file would be about 512 MB. That's a compression of 99.999%, because most of the data are -999.
(or whatever fillvalue
you want).
The equivalent can be achieved using the C++ HDF5 API by setting H5::DSetCreatPropList::setDeflate to 9, with an example shown in h5group.cpp.
Chunked datasets (H5D_CHUNKED) allow sparse storage but depending on your data, the overhead may be important.
Take a typical array and try both sparse and non-sparse and then compare the file sizes, then you will see if it is really worth.