Data Engineer · Senior · Theory

A Spark job is processing a large join: almost all tasks in the stage finish in a minute, but the last two hang for an hour and sometimes fail with OOM. What's going on, and how would you fix it?

~4 min by voice

How to approach it

The interviewer checks whether the candidate recognizes data skew on the join key from the symptoms, can confirm it with facts rather than guesses, and knows the set of fixes along with their costs: from filtering junk keys to AQE, broadcast and salting.

Have you seen this question in a real interview?

Answers0

Avg. score—

Strong—

Practiced0

Topics

spark data skew

Similar questions

Your Spark job writes data to a data lake as Parquet, and over time queries against that table keep getting slower, while the folder now has hundreds of thousands of small files. Why did this happen, and what will you do? Mid-level Tell me what Apache Spark is and how it processes data. Junior A Spark job that joins two large tables runs for hours, and the Spark UI shows that almost all tasks finished within a minute, while a few hang for a very long time. What could be the cause, and how do you fix it? Mid-level Walk me through how Spark executes a query from the moment an action is called to the result: what are stages, shuffles and lazy evaluation, and where is performance usually lost? Senior

Answer by voice

The interviewer will ask this question and listen. Think out loud and ask for a hint if you need one. When you're done, pause or tap “That's my answer”. In about a minute, your score and the points you covered show up here.

Best with headphones · ~4 min