Read on Twitter
Troubleshooting slow Spark jobs is a special type of data engineering torture!
What are the common culprits? 1/5
What are the common culprits? 1/5
- Inadequate initial file parallelism If your upstream data tables are written to too few files, you can't increase the parallelism much unless you work with the upstream pipeline to write more files. This can increase the speed of your job dramatically! 2/5
- Not enough memory/disk spillage
Disk spillage happens when Spark needs more RAM to process the data and uses ROM instead. Disk spillage hurts perf! You can solve bumping up executor memory or increasing parallelism. This can also be caused by skew 3/5
Disk spillage happens when Spark needs more RAM to process the data and uses ROM instead. Disk spillage hurts perf! You can solve bumping up executor memory or increasing parallelism. This can also be caused by skew 3/5
- Skew Skew happens when there's an uneven distribution of data based on the keys you're GROUPing or JOINing on. Salting the GROUP BY with a random number can be a solution. If you're using Spark 3, enable adaptive execution and that will solve the skew problem easily! 4/5
- UDF issues UDFs can cause all sorts of garbage collection issues if they are very memory intensive. Avoid using stateful UDFs because they'll crush your job's performance! If you liked this content, subscribe to my newsletter at: zachwilson.tech #dataengineering
• • •
Missing some Tweet in this thread? You can try to
force a refresh
