Welcome to

pySparkReference.com

Your technical destination for pySpark 3.5.

databricks infra

→  Why do we need smaller executors, and why databricks is fine with one executor per node

meta

→  Single node spark notebook setup using docker

performance

→  Dealing with data skew using salting - work in progress

io

→  Loading data from files

Tools

Spark Memory Calculator - WORK IN PROGRESS - Get the max partition size your executors can handle