Data Engineer · Mid-level · Theory

A Spark job that joins two large tables runs for hours, and the Spark UI shows that almost all tasks finished within a minute, while a few hang for a very long time. What could be the cause, and how do you fix it?

~4 min by voice

How to approach it

The interviewer wants to see whether you recognize data skew from its telltale picture in the Spark UI, understand why it happens during a shuffle, and know a set of practical ways to fix it, rather than just "add more resources".

Have you seen this question in a real interview?

Answers0

Avg. score—

Strong—

Practiced0

Topics

spark data skew

Similar questions

A Spark job is processing a large join: almost all tasks in the stage finish in a minute, but the last two hang for an hour and sometimes fail with OOM. What's going on, and how would you fix it? Senior Your Spark job writes data to a data lake as Parquet, and over time queries against that table keep getting slower, while the folder now has hundreds of thousands of small files. Why did this happen, and what will you do? Mid-level Tell me what Apache Spark is and how it processes data. Junior Walk me through how Spark executes a query from the moment an action is called to the result: what are stages, shuffles and lazy evaluation, and where is performance usually lost? Senior

Answer by voice

The interviewer will ask this question and listen. Think out loud and ask for a hint if you need one. When you're done, pause or tap “That's my answer”. In about a minute, your score and the points you covered show up here.

Best with headphones · ~4 min