Introduction to Big Data
Distributed storage and computation — the ideas behind Hadoop, Spark and the modern data stack.
§ 1Calendar
- Quiz 1—
- Quiz 2—
- End-term examination—
§ 2Previous Year Questions
No papers uploaded yet.
§ 3Resources
Course material1
-
Drive folder
drive.google.comIITM login required
Notes2
-
Course notes
Notes and study material for this course.
iitmbsdegree.inby IITM BS Students' Corner -
Topic-wise notes
Topic-wise notes on ML Study Hub.
degreedost.comby ML Study Hub
Syllabus
The official course syllabus, transcribed from the IITM BS Data Science & Applications course booklet.
§ 1Course Description
This course introduces the practical aspects of large-scale analytics (big data), beginning with foundational concepts in big data and cloud computing across hardware, systems, and software, and progressing to algorithm design and execution at scale.
Lectures-Tutorials-Practicals-Credits: 4-0-0-4
§ 2Faculty
- Dr. Rangarajan Vasudevan, Co-Founder & Chief Data Officer, Lentre.ai
§ 3Week-by-week
Week 1: Big data fundamentals; Cloud and Spark; MapReduce concepts; Sorting and hashing.
Week 2: Cloud fundamentals; Cloud-native architecture; Cloud services; Big data in cloud.
Week 3: Data pipelines; Data formats; Relational vs non-relational data; Data lakes and warehouses; Data extraction.
Week 4: Hadoop and Spark; Spark architecture; Spark SQL and streaming; Hands-on data processing.
Week 5: ETL/ELT; Data processing patterns; Metadata and lifecycle; Scheduling.
Week 6: SQL and relational algebra; Query planning and optimization; Join algorithms; Spark SQL optimization; NoSQL databases.
Week 7: Streaming vs batch; Streaming fundamentals; Pub/Sub; Dataflow and Beam.
Week 8: Kafka architecture; Message semantics; Exactly-once processing; Spark streaming; Windowing and real-time processing.
Week 9: Big data ML; Distributed ML; Spark MLlib; MLOps; Best practices.
Week 10: Deep learning on big data; Distributed training; Streaming inference; Course recap.
Week 11: Prep week for final project; Summarizing key concepts.
OPPE
Practice material for the Introduction to Big Data online proctored programming exams. No practice material has been shared for it yet. If you have some, please send it in.