← Back to courses
Data Science and AI
Big Data with Apache Spark
Process large datasets with PySpark, Spark SQL and Structured Streaming
A classroom course in processing data too large for one machine. The modules follow the programming guides of the Apache Spark 4.3 documentation: PySpark, Spark SQL and DataFrames, the RDD programming guide, Structured Streaming, MLlib, Spark Connect and Declarative Pipelines. This course prepares for no exam.
10Modules
IntermediateLevel
3 monthsDuration
COURSE THUMBNAIL
What you will be able to do
Explain how Spark distributes work across a cluster
Transform large datasets with PySpark DataFrames and Spark SQL
Read from and write to common data sources
Process a stream of data with Structured Streaming
Train a model on a large dataset with MLlib
Course outline
01
How Spark works: clusters, jobs and a quick start
02
PySpark and DataFrames
03
Spark SQL
04
Reading and writing data sources
05
RDDs, accumulators and broadcast variables
06
Structured Streaming
07
Machine learning with MLlib
08
Spark Connect
09
Declarative Pipelines
10
Project: a batch and streaming pipeline
Tools you will use
Apache SparkPySparkSpark SQLStructured StreamingMLlibPython
What you need
Comfortable programming in Python
Working knowledge of SQL
