Interoperable CSV for Environmental Data Archival and Exchange: iCSV

Comparison of environmental data formats. × = poor/none, ✓ = great/yes, () = partly. Human-readability refers to the ability to interpret the data without specialized software. Self-documenting indicates whether only the file itself is enough to understand the content of the data. Standard metadata refers to support for established metadata conventions including units (e.g., meters, degrees Celsius, kg/m³). File size evaluates storage efficiency. Software requirements considers the ease of implementing readers/writers for the format, and existing supported software.

Benchmark of two iCSV format variants (uncompressed and gzip-compressed) with half-hourly meteorological and snowpack measurements from Weissfluhjoch, Davos, Switzerland. This dataset (Wever, 2017) has been read with MeteoIO and written out in diverse formats on a 10th generation Intel i9-10900 computer with 32 GB RAM. For each benchmark, three warm-up runs have been conducted, followed by ten timed runs. The median of such runs is shown in the table. The SMET output is optimized for readability with added white-space padding, leading to a much larger file size.

Benchmark of compression algorithms for iCSV files. The Weissfluhjoch dataset (Wever, 2017) is used and compressed by command line tools on a 10th-generation Intel i9-10900 computer with 32 GB RAM. For each benchmark, three warm-up runs have been conducted, followed by ten timed runs. The median of such runs is shown in the table. The baseline no compression case reads the file content to the standard output and redirects it to a file in order to represent the data transfer speed. The MeteoIO internal compression with gzip forces the Z_FILTERED mode that is specially tailored to repetitive patterns in data.