Because of the slow improvements of storage hardware, compression has become very important for high performance computing. Efficient strategies that provide a good compromise between computational overhead and compression ratio have been developed in recent years. However, when data reduction is used, usually a single strategy is applied to the whole system. These solution generally do not take advantage of the structure within files, which is often known beforehand. This thesis explores several data transformation techniques that can take advantage of patterns within certain types of data to improve compression results. Specific examples are developed and their applications, strengths and weaknesses are discussed. With an array of transformations to choose from, users can make the best choice for each file type, leading to an overall reduction of space. To make this usable in a HPC environment, the transforms are implemented into an I/O library. ADIOS is chosen for this as it provides an easy way to configure I/O parameters and metadata, as well as an extensible framework for transparent on the fly data transformations. The prototyping and implementation process of the transformations is detailed and their effectiveness is tested and evaluated on scientific climate data. Results show that the transforms are quite powerful in theory, but do not have a great effect on real data. While not improving compression results, the discrete cosine transformation is worthwhile on its own, providing an option to sacrifice accuracy for size reduction.

@misc{ADTARTIAD15,
	author	 = {Tim Alexander Dobert},
	title	 = {{Advanced Data Transformation and Reduction Techniques in ADIOS}},
	advisors	 = {Michael Kuhn},
	year	 = {2015},
	month	 = {10},
	school	 = {Universität Hamburg},
	type	 = {Bachelor's Thesis},
	abstract	 = {Because of the slow improvements of storage hardware, compression has become very important for high performance computing. Efficient strategies that provide a good
compromise between computational overhead and compression ratio have been developed in recent years. However, when data reduction is used, usually a single strategy is applied to the whole system. These solution generally do not take advantage of the structure within files, which is often known beforehand. This thesis explores several data transformation techniques that can take advantage of patterns within certain types of data to improve compression results. Specific examples are developed and their applications, strengths and weaknesses are discussed. With an array of transformations to choose from, users can make the best choice for each file type, leading to an overall reduction of space. To make this usable in a HPC environment, the transforms are implemented into an I/O library. ADIOS is chosen for this as it provides an easy way to configure I/O parameters and metadata, as well as an extensible framework for transparent on the fly data transformations. The prototyping and implementation process of the transformations is detailed and their effectiveness is tested and evaluated on scientific climate data. Results show that the transforms are quite powerful in theory, but do not have a great effect on real data. While not improving compression results, the discrete cosine transformation is worthwhile on its own, providing an option to sacrifice accuracy for size reduction.},
}