In this work we design and introduce PyCdo. PyCdo is a prototype for a Python interface that extends the popular data processing tool CDO with in-situ capability. The interface is intended to be used with Python post processing workflows directly applied on output data at runtime of simulations. We introduce the surrounding HPC environment relevant for the design and implementation, including aspects that require the design to in incorporate the type of user base and through this the usability and maintainability. Further, we give an introduction into the architecture of for this work relevant parts of CDO and show the realization of the design with a heavy emphasis on the maintainability, the usability and a short on boarding process for the use and the development of PyCdo. We provide a working Python interface for CDO that is capable of reducing the normally required file I/O drastically as all data transfer between CDO execution within the same script can be done in-memory. Additionally the workflows for post-processing, including the use of Xarray and compatible plotting libraries, can be streamlined with PyCdo. Last, we benchmark PyCdo and show that it can outperform CDO, but also that the current version has serious flaws in terms of scalability. In the end we can provide a solid prototype and alternative to bash scripts.

@mastersthesis{EIPOCSDTCH24,
	author	 = {Oliver Heidmann},
	title	 = {{Enabling In-Situ Processing of Climate Simulation Data through CDO}},
	advisors	 = {Anna Fuchs and Jannek Squar and Ralf Müller},
	year	 = {2024},
	month	 = {11},
	school	 = {Universität Hamburg},
	howpublished	 = {{Online \url{https://wr.informatik.uni-hamburg.de/_media/research:theses:oliver_heidmann_enabling_in_situ_processing_of_climate_simulation_data_through_cdo.pdf}}},
	type	 = {Master's Thesis},
	abstract	 = {In this work we design and introduce PyCdo. PyCdo is a prototype for a Python
			interface that extends the popular data processing tool CDO with in-situ capability. The
			interface is intended to be used with Python post processing workflows directly applied on
			output data at runtime of simulations. We introduce the surrounding HPC environment relevant
			for the design and implementation, including aspects that require the design to in incorporate
			the type of user base and through this the usability and maintainability. Further, we give an
			introduction into the architecture of for this work relevant parts of CDO and show the
			realization of the design with a heavy emphasis on the maintainability, the usability and a
			short on boarding process for the use and the development of PyCdo. We provide a working
			Python interface for CDO that is capable of reducing the normally required file I/O
			drastically as all data transfer between CDO execution within the same script can be done
			in-memory. Additionally the workflows for post-processing, including the use of Xarray and
			compatible plotting libraries, can be streamlined with PyCdo. Last, we benchmark PyCdo and
			show that it can outperform CDO, but also that the current version has serious flaws in terms
			of scalability. In the end we can provide a solid prototype and alternative to bash scripts.},
}