Biobabel: A Unified Interface for Reading Diverse Physiology File Formats Including Cardiac, Respiration, and Electrodermal Data

Supported data formats

At the time of writing biobabel supports the data formats shown in Table 1.

The format of input files is guessed automatically at the time of reading, using clues such as file extension, but if these are insufficiently informative, guesses are made based on sniffing of the file. For more information about inferring formats and metadata, please see Section.

Within Python, the following code is sufficient to read a data file:

import biobabel as bb
bio = bb.load('tests/example.hdf5')

Then, we can view basic properties of the data file:

bio.print()

This will produce an overview of the dataset indicating sampling frequencies and durations:

Summary of Simulated data
- date 07/20/2023 10:48:32 EDT-0400
 
Participant 'a'
- channel a_ecg [ modality ecg ] 15000 samples @ 1000.0 Hz = 15.0 s
- channel a_ppg [ modality ppg ] 15000 samples @ 1000.0 Hz = 15.0 s
 
Participant 'b'
- channel b_ecg [ modality ecg ] 15000 samples @ 1000.0 Hz = 15.0 s

And easily inspect the data using a plot:

bio.plot()

which produces the output shown in Figure 2.

Figure 2

Biobabel internal data structure

Internally, biobabel stores physiological datasets in a Biodata object (bio in the above example). Under the hood, this object contains a number of data streams, each of which is a single dimension data array with some associated key-value metadata, such as sampling frequency, participant ID, etc. Each data stream is identified with a unique ID.

The channel metadata allows us to easily find channels by data type:

bio.find_channels({'modality':'ecg'}) # return which ECG data channels

This returns a set of channel IDs:

['a_ecg', 'b_ecg']

The channel IDs can then be used to query the channel metadata (in dictionary format) and extract its data:

hdr,dat = bio.get('a_ecg')
hdr # find the associated metadata for this channel

which returns the metadata in hdr:

{'id': 'a_ecg',
 'participant': 'a',
 'sampling_frequency': 1000,
 'modality': 'ecg'}

In biobabel, each data stream can have its own sampling frequency, but all data streams are assumed to start at the same time. In my experience analyzing physiological data, this common starting time assumption was sensible, since it holds true in most applications and making this assumption simplifies subsequent data handling. For data formats in which this assumption does not necessarily hold true (e.g. XDF), biobabel will be padded the data streams with missing data samples (NAN) so that all streams start at the same time.

biobabel also supports markers, which are points in time at which specific events are recorded to occur. This can be start/stop markers indicating separate recording segments (e.g. append-markers in BioPAC Acknowledge files). Markers are stored in the Biodata object and can be accessed using bio.get_markers() (to find the marker names) and bio.get_marker(<NAME>) (to extract the corresponding time points). In default plotting functions of biobabel, they are indicated with dashed vertical lines (Figure 2).

biobabel allows a number of typical data management steps that most packages do not straight-forwardly allow, such as cropping the data to a selected time range (bio.crop(t_start,t_end)) and dropping or selecting channels.

Finally, data can be saved in the biobabel native HDF5-based format (bio.save).

For labs engaging in hyperscanning, biobabel seamlessly accommodates support for data from multiple participants. Each data stream can be allocated to a specific participant, allowing the software to find all participants bio.get_participants() or get channels for a specific participant (bio.find_ channels({'participant':'b'})).

Inferring formats and meta data

biobabel makes two kinds of inferences when it reads files.

First, biobabel attempts to infer what is the file format of an input file. For example, is a given file in the EDF or AcqKnowledge format?

• Default strategy. The function biobabel.io. guess_dialect first checks the extension of the file name. For example, if the file ends with .edf, biobabel will guess it is an EDF file. Some extensions are ambiguous, such as .txt, which is used for OpenSignals files but also for Teensy-ECG formats. In this case, biobabel.io.guess_dialect reads the first line of the file, which for OpenSignals files should contain the string OpenSignals.

• What happens when the default strategy is wrong. In most cases, if the wrong file format is guessed, an error will be thrown. This is the desired behavior, since it hands control back to the user to override the file format choice, as exemplified in tests/tes_ wrong_extension.py.

• How to correct when the default strategy is wrong. The user can override the file format guess by specifying the file format manually. For example, if biobabel.load(filename) incorrectly guesses the file to be an EDF file, while it should be an AcqKnowledge file, the user can force the software to read the file as an AcqKnowledge by calling biobabel.load_acq(filename). This function returns exactly the same biodata object that would have been returned if biobabel.load() had guessed the file format correctly.

Second, biobabel attempts to infer metadata for a given input file when it is not explicitly indicated. Examples of such metadata are the type of physiological data contained in a given channel, e.g., PPG or ECG data.

• Default strategy. biobabel will read modality information from the input file if present. For example, the HDF5 file format contains this information, which is simply taken over by biobabel. In the case of file formats that do not explicitly encode the modality of data channels, biobabel will search if the channel name contains any of the major modalities (e.g., ECG, PPG) and, if found, enter this as the channel modality.

• What happens when the default strategy is wrong. The default strategy may not always provide the correct modality for a channel. The result of this will be that a channel will not be found when searching channels for a given modality, e.g., the channel will not show up in a search of all ECG channels via dat.find ({'modality':'ecg'}).

• How to correct when the default strategy is wrong. The user can manually override the channel modality after the file is read. For instance, biodata.update_ channel('b_ecg',{'modality':'eeg'}) will set the modality of the channel to eeg. This is illustrated in the test file tests/test_guess_ modality.py.

Easy previewing and some manipulation from the command line

biobabel provides simple accessible previewing of data files directly from the command line. This functionality is inspired by AFNI [12], a toolbox of shell scripts for neuroimaging analysis.

The following shell scripts are currently included and available automatically if the package is installed via pip:

• bioinfo <filename> which reads the data file and prints a summary (a wrapper around biodata. print())

• biobabel <filename> which reads the data file and produces a simple plot (a wrapper around biodata.view())

• tohdf5 <filename> which converts a data file in any of the supported formats into biobabel’s native HDF5 format.

• biosplit <filename> which splits the data along its integrated markers (which often correspond to different recording sessions) into multiple separate files (e.g. <filename_001>, <filename_002> etc.)

• bioview <filename> which launches a graphical user interface (GUI) reader allowing interactive inspection of data as shown below.

Accuracy

The software has been tested over the years during which it has been in development, which is from roughly 2021 until 2025, the time at which its basic functionality is considered stable. The software is part of standard lab pipelines at the BRAMS center in Montreal and is shared with colleagues around the world. Because of this, the software has undergone repeated cycles of error corrections based on field observations. Data read from specific files was compared against the same data read in other (commercial) interfaces. A variety of data file formats has been tested in this manner over the years to ensure the accuracy of the data read in biobabel.

Test suite

A standardized test suite based on pytest is developed and being expanded. By running the pytest command from the root of the package, a series of tests is run in which sample files (included in the subfolder tests/samples) are read and the output is compared against the known values. The included sample files have different file formats that the software is supposed to recognize automatically. Also included are tests where the package is supposed to throw an exception, such as when reading a file assumed to be of the wrong format (e.g., tests/test_wrong_extension.py). Finally, a test is included that verifies the data of a file that is converted from OpenSignals format to HDF5 (tests/test_convert_h5.py). These tests are not meant to be an exhaustive test of all functionalities of the package, but rather provide examples of imagined use cases. The user can also manually run these tests by launching the appropriate scripts in the tests subdirectory.

Computational overhead

What is the computational overhead of biobabel? Since biobabel is an added layer on top of existing packages, it is conceivable that it adds computation time to already data-intensive pipelines, which would impair the usefulness of the package.

To test this, the following performance stress test was performed. A large physiological data file was collected, involving eight data channels with data sampled at 1 kHz and a total duration of 12709.3 seconds (approximately 3.5 h). The data was written in the Biopac Acqknowledge format, amounting to a file size of 194 Mb. The time was measured for the data to be loaded into biobabel (using biobabel.load()), as well as the time taken to load it directly using the dedicated Python package bioread. This process was repeated 10 times to gain an estimate of variability.

Reading time in biobabel took 1965 ms (SD 37 ms), whereas using bioread took 1732 ms (SD 29 ms). As a result, the overhead imposed by biobabel is estimated to be around 233 ms for a file of this size, i.e., 11.9%. In summary, biobabel imposes some computational overhead.