Scientific data is often handled by highly specialized file formats, which all aim to solve the same fundamental problems; fast access times to data at the lowest latency achievable for some of the largest datasets currently in use. This is especially true for earth data analysis, where petabytes of data are processed daily, with three of the most prominent formats used being HDF5, NetCDF4 and more recently Zarr. All of them have their own respective benefits and shortcomings, offering a variety of features, some shared between, some unique to them. Inherently knowing which format to use best for certain use-cases is not straight forward. They need to be evaluated carefully to carve out the best suited contender, as the environments these file formats are used in heavily depend on the interactions between complex software and hardware stacks. This thesis presents a benchmark framework to emulate different scenarios for HDF5, NetCDF4 and Zarr leveraging select features to potentially increase performance like chunking, subfiling, parallel I/O and asynchronous I/O wherever possible. Throughput measures provide a broader picture for areas where formats still have potential to grow or might be limited already by external factors such as file systems like Lustre. The benchmark is written in both Python and C to evaluate the influence different programming languages and concepts can have on performance, to evaluate the importance for code maintenance and future extensibility. As all formats inherently try to cover the same important aspects of scientific computing needs, these metrics will help guide recommendations for future developments. The results showed that file formats in Python like NetCDF4 are superior to other formats and features for parallelized workloads, including their respective counterparts in C, while newer entries like Zarr can match or even outperform more longstanding formats. This supports the notion to switch away from more complex programming languages like C towards languages like Python.

@misc{AIEIZHANP25,
	author	 = {Lucas Philipp},
	title	 = {{Analyzing I/O Efficiency in Zarr, HDF5, and NetCDF4}},
	advisors	 = {Michael Kuhn and Jannek Squar},
	year	 = {2025},
	month	 = {05},
	school	 = {Universität Hamburg},
	howpublished	 = {{Online \url{https://wr.informatik.uni-hamburg.de/_media/research:theses:lucas_philipp_analyzing_i_o_efficiency_in_zarr_hdf5_and_netcdf4.pdf}}},
	type	 = {Bachelor's Thesis},
	abstract	 = {Scientific data is often handled by highly specialized file formats, which all aim to
			solve the same fundamental problems; fast access times to data at the lowest latency
			achievable for some of the largest datasets currently in use. This is especially true for
			earth data analysis, where petabytes of data are processed daily, with three of the most
			prominent formats used being HDF5, NetCDF4 and more recently Zarr. All of them have their own
			respective benefits and shortcomings, offering a variety of features, some shared between,
			some unique to them. Inherently knowing which format to use best for certain use-cases is not
			straight forward. They need to be evaluated carefully to carve out the best suited contender,
			as the environments these file formats are used in heavily depend on the interactions between
			complex software and hardware stacks. This thesis presents a benchmark framework to emulate
			different scenarios for HDF5, NetCDF4 and Zarr leveraging select features to potentially
			increase performance like chunking, subfiling, parallel I/O and asynchronous I/O wherever
			possible. Throughput measures provide a broader picture for areas where formats still have
			potential to grow or might be limited already by external factors such as file systems like
			Lustre. The benchmark is written in both Python and C to evaluate the influence different
			programming languages and concepts can have on performance, to evaluate the importance for
			code maintenance and future extensibility. As all formats inherently try to cover the same
			important aspects of scientific computing needs, these metrics will help guide recommendations
			for future developments. The results showed that file formats in Python like NetCDF4 are
			superior to other formats and features for parallelized workloads, including their respective
			counterparts in C, while newer entries like Zarr can match or even outperform more
			longstanding formats. This supports the notion to switch away from more complex programming
			languages like C towards languages like Python.},
}