Data Career Hub
Menu
SQLPythonPySparkApache SparkKafkaAirflow
ProjectsProject 4 of 8

Intermediate project · Project 4 of 8

Large-Scale Batch Processing Pipeline

Process a large public dataset (several gigabytes or more) with PySpark into partitioned, query-ready tables, and document how you found and fixed the main performance bottleneck.

  • Intermediate
  • PySpark · Parquet or Delta Lake · Local Spark or a small cloud cluster
  • 2 min read
  • Updated Oct 2026

Requirements

  • Read a dataset of at least several gigabytes
  • Clean, join with a lookup table and aggregate
  • Write partitioned Parquet or Delta output by date
  • Rerun any date without duplicating output
  • Record a before-and-after performance analysis from the Spark UI

Technology stack

PySpark, Parquet or Delta Lake, Local Spark or a small cloud cluster

Dataset

Use a large public dataset with a clear licence, such as public trip records or open web-analytics samples. Record the source and licence in your README.

Business context

Interviewers for Spark roles want evidence that you have worked with data too big for a laptop’s memory and know how to find bottlenecks. This project produces exactly that evidence: a working job and a written performance analysis.

Architecture

  1. Raw files in a raw/ folder or bucket.
  2. PySpark job parameterised by processing date.
  3. Broadcast join with a small lookup table.
  4. Aggregation to the reporting grain.
  5. Partitioned output with overwrite for the processed date.
A single rerunnable batch job; each date is processed independently.

Write the performance analysis as you go: screenshots of the stage timeline, the task-duration spread, shuffle sizes and what changed after your fix. Report the timings you actually observed on your hardware.

Build steps

  1. Explore a sample and define an explicit schema
  2. Write the job with a processing-date parameter
  3. Join with a small lookup table using a broadcast
  4. Aggregate to the target grain
  5. Write with partition overwrite for the processed date
  6. Run on the full data and capture the Spark UI
  7. Identify the slowest stage and fix its cause (skew, shuffle size or small files)
  8. Rerun and compare

Testing

  • Unit tests for transformation functions on small DataFrames
  • Rerun a date and compare row counts and checksums

Data quality

  • Required columns non-null after cleaning
  • Row counts reconcile from input to output, accounting for filtered rows

Monitoring

  • Job duration, input and output rows per run
  • Number and size of output files

Cost considerations

  • Run locally where possible; on cloud, use small clusters and shut them down
  • Avoid many tiny output files

Interview questions this project prepares you for

  • What was the slowest stage and why?
  • How did you choose the number of partitions?
  • How does your write stay idempotent?
  • What would change at 100 times the data?

Resume bullets (add only what you measured)

Use these as structure. Replace the wording with results you actually produced and can show; never claim numbers you did not measure.

  • Built a PySpark batch pipeline over a multi-gigabyte public dataset with partitioned, idempotent outputs
  • Diagnosed the slowest stage in the Spark UI and resolved it by changing the join strategy and partitioning, documenting the analysis

Extensions

  • Add Delta Lake and MERGE for late-arriving corrections
  • Schedule daily runs and backfills with Airflow

By Data Career Hub Editorial · Last reviewed Oct 2026

Progress is saved in this browser only. No account needed.

Search
Filter by type