Table of Contents
Vorlesung „Big Data Analytics“
This lecture introduces theory and techniques to analyze large volumes of data. Big data is usually created by experiments, observations or humans. Besides the sheer volume, data can be characterized by the following characteristics: the velocity it is produced, the variability of its structure, the suboptimal data quality and its inherent value.
We can gain knowledge by analyzing this data using techniques from statistics and machine learning. Global players like Google and Facebook use the introduced techniques for targeted advertising to optimize revenue. However, the techniques are also applicable in the scientific context.
In the exercises, selected open source tools such as Apache Pig, Hive, Spark or Neo4j are utilized to reveal interesting properties of publicly available data sets. The exercises teach the language R and Python and build upon them.
The lecture is a “Wahlpflichtmodul/Vertiefung” in the Master of Computer science; interested students of other degree programs are also welcome – please contact the organizer.
It is expected that attendees have experience in any programming language (e.g., Java). Knowledge about Python, SQL and machine learning is not necessary but helpful.
Information about the course
|Location||DKRZ, room 034|
|Time lecture||Friday 12:15 - 13:45|
|Time exercise||Friday 14:00 - 15:30|
|First meeting||Friday 2017-10-20 12:15|
Note that it is mandatory to subscribe to the mailing list.
Schedule and material
- Big Data Challenges and Characteristics, Analytical Workflows, Use Cases, Programming
- Exercise 1: Introduction to R and Python, data processing: CSV-files and raw text, basic data visualization
- 2017-10-27 - Room change: we are in the Bundesstraße 43 (Bioinformatik, ZBH) Room 16
- Exercise 2: Data exploration, data cleaning/extraction using the Wikipedia dataset
- Exercise 3: Data Exploration (Diamond), Data Ingestion Using PostgreSQL, Database Schema: Relational & OLAP Cube
- Exercise 4: Data exploration (Chicago-Crime), Map-Reduce, visualization of Wikipedia Data
- 2017-11-17 - Guest talk by Dr. Philipp Neumann: Exascale Computing and Big Data
Slides – Exercise
- Exercise 5: Data exploration (Wikipedia), Data Cleaning IMDB, JOIN using Map-Reduce
- 2017-11-24 - Topic 5. Machine Learning
Slides – Exercise
- Exercise 6: Data exploration (Weather data), spatial data in PostGIS, machine learning
- 2017-12-01 - Topic 6. Processing Relational Data with Hive
Slides – Exercise
- Exercise 7: Data exploration (IMDb Quotes), classification (titanic), word frequencies/external scripts (Hive)
- 2017-12-08 - Topic 7. Graph Processing with Neo4J and REST APIs
Slides (Neo4j) – Slides (Rest) – Exercise
- Exercise 8: Neo4j Cypher data model, queries, clustering Wikipedia, HDFS REST API
- 2017-12-15 - Topic 8. Columnar Access with HBase and Document Storage with MongoDB – Slides (HBase) – Slides (Mongo) – Exercise
- Exercise 9: HBase data model and import, classification of Wikipedia, document data model
- 2017-12-22 - We will have exercises only
- 2018-01-12 - Topic 9. Data Flow Languages & Pig Latin and Performance Aspects – Slides (Pig) – Slides (Performance) – Exercise
- Exercise 10: Data flow programming, Pig examples, performance analysis
- 2018-01-19 - Topic 10. In-Memory Computation with Spark – Slides – Exercise
- Exercise 11: Spark basics, distance metrics, clustering, performance analysis
- 2018-01-26 - Topic 11. Stream Processing with (with Storm, Spark, Flink) – Slides – Exercise
- Exercise 12: Find movie quotes, streaming data model for crime data
- 2018-02-02 - Topic 12. Overview of Tools in the Hadoop Ecosystem – Slides
- 2018-02-06 Examination 10:00 (60 minutes), ESA C
- 2018-02-27 Examination 10:00 (60 minutes), DKRZ, R034
- Diverse R-Topics: Veranstaltung Programmierung in R
- Book: Data Science for Dummies, Lillian Pierson, Wiley Verlag
- Book: Big Data - Priciples and best practices of scalable real-time data systems, Nathan Marz und James Warren, Manning Verlag
- Horton Works Platform: http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.2.4/index.html
- Introductions to programming languages
- R Books:
- Machine Learning with R, Second Edition, Brett Lantz, 2015
- Python Books:
- Interesting tools:
- http://ipython.org/notebook.html Python Notebook, vgl. Laborbuch mit Experimentalbeschreibung und Ergebnissen.
- Cheat cheats:
- Resource for data science: https://www.kaggle.com/