Which Data Format To Store Scientific Data Should I Use? A Performance Analysis

Gecášek, Daniel; Solanik, Michal; Genči, Ján

doi:10.2478/aei-2022-0015

Abstract

A lot of scientific work is dedicated to the analysis of data. Most of the analyzed data, like data from space missions, are structured. The choice of data format can affect various characteristics - read/write speed of standard files, read/write speed of small files and read/write speed of compressed data formats. In this paper, we analyze binary data formats, proposed types of the tests and testing methods, and compare their performance with human-readable text format. We also discuss compressed and uncompressed modes available for data formats like HDF5 and netCDF. When disregarding precision, the best data format from the size perspective is lossy HDF5 without compression. Losless HDF5 without compression show the best speed performance. Lossy HDF5 without compression is the best balance between size reduction and speed. However, for specific criteria and types of files, there might be better candidates as detailed in the conclusion.

References

[1] Van den EYNDEN, V. – CORTI, L. – WOOLLARD, M. – BISHOP, L. – HORTON, L.: Managing and sharing data; a best practice guide for researchers, UK Data Archive, No. () 2011
Search in Google Scholar Back to article
[2] Space Physics Data Facility NASA / Goddard Space Flight Center: CDF User’s Guide,, No. 3.8.1 (2021)
Search in Google Scholar Back to article
[3] REW, R. – DAVIS, G.: NetCDF: an interface for scientific data access, IEEE computer graphics and applications 10, No. 4 (1990) 76–82
Search in Google Scholar Back to article
[4] KOZIOL, Q. – ROBINSON, D. et all.: Hdf5, Lawrence Berkeley National Lab.(LBNL), Berkeley, CA (United States), No. (2018)
Search in Google Scholar Back to article
[5] GECÁŠEK, D. – BOBIK, P. – GENCI, J.: Transparency of magnetosphere for cosmic rays in last two millennia, Proceedings of 37th International Cosmic Ray Conference — PoS(ICRC2021) 395, No. - (2021) 133510.22323/1.395.1335
Search in Google Scholar Back to article
[6] BLOMER, J.: A quantitative review of data formats for HEP analyses, Journal of Physics: Conference Series 1085, No. 3 (2018) 032020
Search in Google Scholar Back to article
[7] BALLE, T. – JOHNSEN, P.: Improving i/o performance of the weather research and forecast (WRF) model, Cray User Group 2016, No. (2016) 123
Search in Google Scholar Back to article
[8] ISENBURG, M. – SNOEYINK, J.: Binary compression rates for ASCII formats, Proceedings of the eighth international conference on 3D Web technology, No. (2003) 17310.1145/636593.636619
Search in Google Scholar Back to article
[9] LANEY, D. – LANGER, S. – WEBER, Ch. – LINDSTROM, P. – WEGENER, A.: Assessing the effects of data compression in simulations using physically motivated metrics, Scientific Programming 22, No. 2 (2014) 141
Search in Google Scholar Back to article
[10] AUBERT, P. – VUILLAUME, T. – MAURIN, G. – JACQUEMIER, J. – LAMANNA, G. – EMAD, N.: Polynomial data compression for large-scale physics experiments, Computing and Software for Big Science 2, No. 1 (2018) 1–9
Search in Google Scholar Back to article
[11] BERGER, J. – FRANKENFELD, U. – LINDEN-STRUTH, V. – PLAMPER, P. – RÖHRICH, D. – SCHÄFER, E. – SCHULZ, M. W. – STEINBECK, T. M. – STOCK, R. – SULIMMA, K. et all: TPC data compression, Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment 489, No. 1-3 (2002) 406–42110.1016/S0168-9002(02)00792-1
Search in Google Scholar Back to article
[12] KOHLHASE, C. E. – Penzo, P. A.: Voyager mission description, Space science reviews 21, No. 2 (1977) 77–101
Search in Google Scholar Back to article
[13] WHITTLESEY, P. L. et al.: The solar probe analyzers—electrons on the parker solar probe., The Astrophysical Journal Supplement Series 246, No. 2 (2020) 74
Search in Google Scholar Back to article
[14] BASILIO, R. R. – POLLOCK, H. R. – HUNYADILAY, S. L.: OCO-2 (Orbiting Carbon Observatory-2) mission operations planning and initial operations experiences., Sensors, Systems, and Next-Generation Satellites XVIII. 9241, No. (2014) 31–42
Search in Google Scholar Back to article
[15] ZHOU, L. – MURTY D. – XINGPIN, L.: An overview of the Joint Polar Satellite System (JPSS) science data product calibration and validation., Remote Sensing 8, No. 2 (2016) 139
Search in Google Scholar Back to article

Which Data Format To Store Scientific Data Should I Use? A Performance Analysis

Abstract

Paradigm

My account