Data Engineer · Junior · Theory

Tell me what Apache Spark is and how it processes data.

~4 min by voice

How to approach it

This checks whether you understand Spark's basic model: distributed processing on a cluster through partitions, the difference between transformations and actions, lazy evaluation, and why Spark is faster than MapReduce. They don't want the product's history but an understanding of what happens to the data when you write code.

Have you seen this question in a real interview?

Answers0

Avg. score—

Strong—

Practiced0

Topics

spark big data

Similar questions

A Spark job is processing a large join: almost all tasks in the stage finish in a minute, but the last two hang for an hour and sometimes fail with OOM. What's going on, and how would you fix it? Senior Your Spark job writes data to a data lake as Parquet, and over time queries against that table keep getting slower, while the folder now has hundreds of thousands of small files. Why did this happen, and what will you do? Mid-level A Spark job that joins two large tables runs for hours, and the Spark UI shows that almost all tasks finished within a minute, while a few hang for a very long time. What could be the cause, and how do you fix it? Mid-level Walk me through how Spark executes a query from the moment an action is called to the result: what are stages, shuffles and lazy evaluation, and where is performance usually lost? Senior

Answer by voice

The interviewer will ask this question and listen. Think out loud and ask for a hint if you need one. When you're done, pause or tap “That's my answer”. In about a minute, your score and the points you covered show up here.

Best with headphones · ~4 min