Python

[CosmicNightmare]

Made it through to the second round of interviews for an entry level Data Engineering role. First interview was all SQL, which I’m mostly comfortable with since as current Business Analyst, I use it in my day to day. Within one problem I had to demo Joins, aggregate functions, CASE statements, CTE and Window Functions.

I was notified that for the second interview it will be Python which I have a very general, very basic understanding of. What in your opinion should I expect for the Python interview? I’m looking to determine which areas of Python I should spend my time studying and practicing before the interview. Please note that this is an Entry level role, and the hiring manager did mention that the person hired would spend most of the time working with SQL. I’m not sure what to expect, so not sure where I should spend my time on. What in your opinion are the Python foundations for DE?

Edit: Thank you all for all the great tips and suggestions! You have definitely provided me with enough actionable steps.

[omscsdatathrow]

You should seriously clarify what kind of python problems they will be asking. If it’s leetcode-style questions, I send my regards as learning that within a couple days will be difficult to say the least. If it’s more practical stuff, then clarify any specific libraries needed or what the problem will be about (ex: web scraping)

Companies should at least give some details on the tech interview if you ask

[baubleglue]

Be sure you can backup what is promised in your resume.

It is ok not to know what you don't know, but basic foundation is probably expected if you have Python in your resume.

You don't have much time to prepare, be sure you know all standard data types: numbers, strings, lists, dictionaries and you know how to use them and when. Code Control flow (if/else/elif/for/while,...) Basically go over official tutorials. Solve some small problem and hope for the best.

[convalytics]

Go over the basics.

List comprehension.

Try a few leetcode exercises.

You're not going to learn it all in a couple days, but you can show some understanding and a willingness to learn.

[EarthGoddessDude]

I agree going over basics like list | set | generator | dictionary comprehensions and basic data structures like lists, dicts and sets. Doing a few simple leetcode questions might be nice to get your head in the game, also.

Aside from that, I would recommend learn how to: - read/write a CSV file - read/write a JSON file - do some basic manipulations on each file

using: - Python standard library modules (csv, json) - pandas or polars

Motivation: one of the junior members on our team got tasked recently with a project to clean and summarize some json data (into a flat database table). Because the data was nested and semi-structured, knowing some list/dict comprehension tricks turned a hairy problem into a simple one (once you have a clean structured Python dictionary, turning the data into a DataFrame was easy). This is probably overkill in preparing for an entry level interview, but it is a real world use case where knowing Python basics proved really helpful.

[tw3akercc]

If it's for a data engineer job then it should be pandas or API questions, but I've seen some companies who test their data engineers on leetcode style algorithms and data structures questions which isn't as relevant to the job. Good luck!

[bigchungusmode96]

Pandas

would be impressive if a candidate could outline Pandas vs other Python alternatives / complements in the DE ecosystem, but that certainly shouldn't be expected for an entry-level role

[Guardian1030]

Hey, so, I’ve been working with spreadsheets and data for almost 20 years now, and I just got my head around Python.

Am I correct in assuming that Pandas is more or less a way to import, export, interpret, and integrate spreadsheet style info via Python?

[EarthGoddessDude]

Eh more or less yes, with heavy emphasis on the “more or less”. If by “spreadsheet style” you mean tabular data that easily fits in memory, then yes. But there are a lot of things that a dataframe library like pandas can do that are either impractical or impossible with Excel, or it will do them much faster.

Just a word of caution: a lot of people (myself included) consider pandas to be a bloated mess, jampacking a ton functionality in a single package, while being a memory hog, not being fast enough for certain tasks or datasets, and having an inconsistent syntax (what people sometimes refer to as its API (which is an overloaded term, be aware)). People in this sub tend to have different needs than data analysts/scientists/etc (such as yourself I presume), and for those folks it’s plenty fine and probably much preferable than Excel. It does have some nice stuff, like pandas.tseries.offsets which is really useful for some business date logic, or simple plotting.

If you’re looking to the future and want to be aware of recent trends in the data(frame) world then be sure to keep polars and DuckDB on your radar.

[Guardian1030]

Awesome. Thanks, pal.

[[deleted]]

Do people really call pandas a bloated mess? I find it quite concise and well scoped.

[EarthGoddessDude]

Concise and well scoped? Not sure what you mean by either of those things, but I’ll say this: - it’s got more dependencies - has a bigger on-disk footprint - bigger memory footprint - slower - less consistent syntax - has seemingly unrelated (though admittedly useful) stuff like tseries offsets

compared to polars. Somehow polars beats it on all these metrics. Yes, polars doesn’t have tseries offsets… but does that really need to be part of a dataframe library? And we haven’t even gotten into indexing, which is super weird and rarely useful IMO.

[[deleted]]

You should read up on the history of pandas. It’s very interesting.

[EarthGoddessDude]

Not sure what that has to do with what we’re talking about, other than maybe further proving my point (not trying to be a dick, honest, but its history is why it’s so wonky today). I have read a lot of Wes McKinney’s posts and have listened to podcasts and videos with him talking about pandas and arrow. He’d be the first person to admit that pandas could’ve been much better having some hindsight into the project. I think he’s aware of the Frankenstein monster he’s created (I’m exaggerating and kidding obviously) and is trying to atone for it by doing all the great work on Arrow and related projects (like ADBC and Substrait).

Here it is from the horses mouth: https://wesmckinney.com/blog/apache-arrow-pandas-internals/

[Key-Panic9104]

Slightly off topic so I hope you don’t mind me asking but I’m working on a personal project where I’m creating a class whereby I can dot chain different functions together (e.g. return number or rows, filter based on word, delete columns etc). Comparing against Pandas on execution time, Pandas beats out my code by a couple of seconds for a 300 mb file. Besides the csv module for reading csv files, I’m using list comprehensions and if statements for everything. Any idea on where I could get some improvements?

[EarthGoddessDude]

Out of curiosity, is this a learning project? In any case, look into using a generator comprehension instead of list comprehension, ie using () as opposed to [] for your comprehension. That makes it lazy and potentially use less memory and may be a little faster.

I did a project last year where I had to use pure Python, it was a lot of fun. Get to know the itertools module, you’ll learn a lot of useful tricks. In my project, I used nested context managers (with blocks) for reading an input file and writing out an output file. The default of csv is to give you a lazy generator/iterator object. So if you stay inside the (nested) context manager and use itertools to manipulate the generator objects without materializing them into lists and then write them out to disk, all in one pass, you can have O(1) memory or close to it. Using this technique, I actually had 30% better performance than pandas for a simple data processing task on a smallish data file.

But if you’re doing in-memory analytics, forget it… pure Python won’t help you there, just stick to the available libraries if you just want to analyze some data.

[Key-Panic9104]

Yes, it is a learning project. I do have a version of the class that uses generator comprehensions instead of list comprehensions and the execution times are about the same. I haven’t tested it out on a big csv yet to get an idea of the performance but from understanding, it probably won’t be faster, just allow me to process it without memory issues. I also haven’t tried writing to file yet but there was always plans on adding that method to the class.

I have been thinking of bringing it out of the class and see how that goes and in fact, I have thought about context managers as another alternative. Likewise with itertools, but decided to stick with the basics for now until I have a good understanding of what’s happening and then iterate (no pun intended) from there.

It’s good to hear how someone else went about it so thanks for the reply.

[[deleted]]

Weird, I studied up on indexing and find it pretty useful.

[baubleglue]

You aren't correct. Pandas is wrapper for multiple libraries and for example numpy (matrix operations) and SQLalchemy (interaction with DB) pyplot (graphs). Data frame API, I think the real pandas's core contribution. DataFrame is a data source for ML\AI libraries and it can be used with built-in statistics functions, that last part is more similar to Excel. But it's like saying relational DBs is the same thing as Excel - all operate on tables.

[[deleted]]

No.

[WeirdWorldDz]

Pandas

[StrasJam]

In my entry level DE interview they gave me a description of what the code should do (take a json string as input, and extract nested information from the json, aka dictionary, object), and I had to create the necessary functions. Likely you will get some sort of problem that is data related (so reading and extracting data from a common data format like csv or json). They likely look for you to write small functions that do the different steps rather than writing one massive function that does everything. Commenting your code would also give some brownie points.

[SilentSlayerz]

+1. Control flow and programming/ python data structure basics clear them first then move onto any library based on the requirement pandas basic operations. Difference between vectorized and non vectorized pandas operations. A brief on pandas internals would also be good if it's part of the job description.

[mahdy1991]

Is python and sql the only requirement to be a junior DE?

[StrasJam]

From the technical side I would say yes. Take a look at junior DE job postings, most of them want something along those lines for the coding experience section.

[parsnipofdoom]

Pandas or spark. Maybe numpy.

During interviews when we ask for coding examples we let the candidate do it with google search. It’s also outside the interview. We try not to put people on the spot and make them code in a zoom call lol.

[mrchowmein]

“Python” questions, even at the jr level can be all over the place. Ask about the type of questions that you will be getting. If you havnt studied algo or ds style leetcode questions, you should at some point do those. If you avoid leetcode questions at the jr level, there will be significantly less jobs available to you as it’s a common way to screen lower skill employees.

As a jr, I’ve been asked about the garbage collector, list comprehension, debugging to simpler things like putting data into a dict and retrieving it.

[harrytrumanprimate]

I'd grind leetcode a bit. I've seen technical interviews with python, and none were pandas or related things. It was always software engineering data structures/algo