Data Engineer · Mid-level · Theory

Your Spark job writes data to a data lake as Parquet, and over time queries against that table keep getting slower, while the folder now has hundreds of thousands of small files. Why did this happen, and what will you do?

~4 min by voice

How to approach it

They're checking whether the candidate understands the nature of the small files problem in a data lake: where the files come from when Spark writes, why they slow down reads, and whether they can treat both the cause (the write) and the consequences (compaction).

Have you seen this question in a real interview?

Answers0

Avg. score—

Strong—

Practiced0

Topics

spark partitioning

Similar questions

A Spark job is processing a large join: almost all tasks in the stage finish in a minute, but the last two hang for an hour and sometimes fail with OOM. What's going on, and how would you fix it? Senior Tell me what Apache Spark is and how it processes data. Junior A Spark job that joins two large tables runs for hours, and the Spark UI shows that almost all tasks finished within a minute, while a few hang for a very long time. What could be the cause, and how do you fix it? Mid-level Walk me through how Spark executes a query from the moment an action is called to the result: what are stages, shuffles and lazy evaluation, and where is performance usually lost? Senior

Answer by voice

The interviewer will ask this question and listen. Think out loud and ask for a hint if you need one. When you're done, pause or tap “That's my answer”. In about a minute, your score and the points you covered show up here.

Best with headphones · ~4 min