Ctrl K
Sign inStart free
Python · Senior · System design

Every day you need to process a CSV export of about 100 gigabytes in Python: clean it, aggregate it, and load it into a database. You have a machine with 16 cores and 32 gigabytes of RAM. How would you design the processing?

~10 min by voice

How to approach it

They're checking whether you can process data larger than memory: streaming, in parallel around the GIL, with cheap handoff of work between processes, a fast and safe load into the database, and the ability to restart after a failure.

Have you seen this question in a real interview?
Answers0
Avg. score—
Strong—
Practiced0
Topics
data processingmultiprocessing
Similar questionsDesign a service where users upload CSV files of up to several gigabytes, and the service validates them and loads them into PostgreSQL. How would you build it in Python? Mid-level

Answer by voice

The interviewer will ask this question and listen. Think out loud and ask for a hint if you need one. When you're done, pause or tap “That's my answer”. In about a minute, your score and the points you covered show up here.

Sign in to answer

Best with headphones · ~10 min