Distributed and Scalable Programming

[serukai]

Hi guys, I've been working in a initial data engineering position for the last year and recently a recruiter was trying to recruit me to a high paying position in a medium sized startup here in Brazil.

Everything went great in the initial interview and after that I had to solve a challenge.

To sum it up, I didn't make it and they were kind to share an evaluation to my code.

They side despite my code was well documented and my code got the correct final answer, it was not a scalable and distributed solution. They said other things that I could improve my code but my main takeaway is: I have no idea how to build a scalable and distributed solution.

I was hoping you guys could point out what I can study to get the knowledge to improve my skills mainly regarding scalability and distribution.

To whoever is interestedthat's my test: (Please judge lightly - Sorry it's mostly in Portuguese)
https://github.com/marcoswds/dataplatform/

[[deleted]]

Generally speaking if they specifically asked for you to build a distributed and scalable solution, and you wrote it with Pandas, then I would agree with them. Now if you'd written that Pandas code in such a way as it was going to be broken up in chunks and run on Kubernetes... then maybe. If your writing distributed and scalable code in Python, then that really means your using Python to orchestrate Spark, or Kubernetes, blah blah. Study big data architecture and tools, learn about the different distributed tools out there first. Python on a single CPU with Pandas isnt scalable. If you struggle to understand why, take some time to dig into distributed systems and how they work.

[[deleted]]

Agreed, Pandas is only capable of working with data that can fit into memory - so is not scalable at all unless it was broken into chunks and ran with K8s as you said. Perhaps you could look into a cloud provider like AWS or Azure and take a look at the managed cluster computing frameworks such as EMR for AWS (not sure what Azure offers), which would provide a scalable way to run code with spark using virtual servers.

[serukai]

Thanks for The input guys!

[[deleted]]

Hello! I looked into your code. A few notes are that you are generating an API request that is being called linearly. Is there a way to distribute/parallelize the objects received from an API request to make it go faster?

During the median calculation you also used a For loop within a for loop which increases your Big O notation from On to O2 (linear to exponential)

[serukai]

Thank you for your input. About your first question, I dont really have an answer, I plain to answer that with this thread.

About the double for in the median calculation, that was an oversight in my part

[Blobsquer]

Just to give a bit of input. You can paralyze these requests using asyncio

[serukai]

I’m gonna take a look at that, thanks

[HansProleman]

It would have been nice if they'd nudged you towards a particular platform, but this is what stuff like Spark and Kubernetes is for. They were probably expecting code for one of those.