HDF5 files as back-end

The Hierarchical Data Format version 5 (HDF5) is an open source file format that supports large, complex, heterogeneous data. This format has different advantages that make it very suitable to store large datasets together with their metadata in a way that allows to access quickly all the information. As digitalDLSorteR needs to simulate large amounts of pseudo-bulk samples to reach good trained models and uses as input single-cell RNA-Seq datasets whose size is getting much bigger over time, it implements a set of functionalities that offer the possibility to use HDF5 files as back-end for each step where large data are required:

To use this format, digitalDLSorteR mainly uses the HDF5Array and DelayedArray packages, although some functionalities have been implemented using directly rhdf5. For more information about these packages, we recommend their corresponding vignettes and this workshop by Peter Hickey: Effectively using the DelayedArray framework to support the analysis of large datasets.

General usage

In Building new deconvolution models, some examples of its usage are shown. On the whole, the important parameters that must be considered are:

The simplest way to use it is by setting just the file.backend parameter as in the examples provided in Building new deconvolution models.

Disclaimer

HDF5 files are a very useful tool which allows working with large datasets that would otherwise be impossible. However, it is important to note that runtimes may be longer when using them, as accessing data from RAM is always faster than from disk. Therefore, we recommend using this functionality only in case of very large datasets and limited computational resources. As the HDF5Array and DelayedArray authors point: If you can load your data into memory and still compute on it, then you’re always going to have a better time doing it that way.