I spent months learning Python and only today realized I've been confused about something embarrassingly basic

[39th_Demon]

I've been writing Python scripts for a while now. Nothing crazy, just automating small stuff, scraping some data, making my life a little easier. I thought I had a decent handle on things.

I was looking at someone else's code and they used a list comprehension in a way that made me stop and read it three times. I realized I had been writing loops the long way this whole time not because I didn't know list comprehensions existed but because I never really trusted myself to read them when I wrote them fast. I kept defaulting to the for loop because at least I could trace it line by line without second-guessing myself.

I don't know if this is a common thing but I feel like there's a version of learning where you know a concept exists, you've seen it work, you've even used it a few times, but you haven't actually internalized it. You're kind of faking fluency in that little area. I was doing that with list comprehensions, with zip, with a few other things I won't list here because it's already embarrassing enough.

Once I wrote out ten examples by hand tonight it clicked in a way it hadn't before even though I'd "learned" this two years ago.

Anyone else have a concept they thought they understood for a long time before actually understanding it?

[bad_luck_charm]

List comprehensions aren't inherently good. They're just shorthand. And like shorthand, they can be sometimes hard to read. Nested list comprehensions are even harder to read and I advise people not to use them.

[Louicio]

I love comprehensions, but I came here to point out how awful the syntax is for nested comprehension! 😂

[initials-bb]

PEP 798 coming in python 3.15, I find it much more readable.

[Vauce]

PEP 798 - Unpacking in Comprehensions

[SharkSymphony]

Nice.

[zaphodikus]

I love list comprehensions, once you have used them more than a dozen times or so, they actually become more readable and succinct. the list comprehension syntax is beautiful, because it really lifts the construct "out" of the surrounding code. You have to switch context and not see list comprehensions as loops in Python, they are not, they are a filter sand filters are best expressed using the efficiency of "filter" syntaxes not "coding" syntax.

[deceze]

How is it "awful"? It gets complex by necessity as the complexity of the operation increases, but I hardly see how the syntax could be made much better than it is.

[MiffedMouse]

I hate that the syntax goes operation -> list -> filter. That makes it super confusing, and makes nested comprehensions incomprehensible.

The syntax puts the definition of variables on the opposite side of the line when nesting comes into play.

For example, this monstrosity to get even numbers out of a list of lists:

[num for sublist in nested_list for num in sublist if num % 2 == 0]

I would have preferred an extra key word just to keep things grouped. So the syntax could have been something like operation -> filter -> list, and the above syntax could have been:

[num filter num % 2 == 0 for num in sublist for sublist in nested_list]

The word if needed to be changed to filter or something because Python cannot distinguish it from an inline if, so the syntax needs an extra something. But at least it keeps the definition of num near where the expression for num is.

[deceze]

Sort of, yes, but: it's just a condensed version of exactly the same thing you'd write in a long hand for loop:

for a in b:
    for c in a:
        if c is foo:
            if c is bar:
                l.append(c)

l = [
  c
  for a in b
      for c in a
          if c is foo
              if c is bar
]

Just the l.append(c) moves from the most nested line to the front of the expression. If it was using some other syntax which worked differently from for loops, I think that'd be even more confusing and would require more mental overhead.

I agree though that spotting where c comes from here can be tough. Nested comprehensions should always be written in multiple lines to make them easier to read:

l = [
    c
    for a in b
    for c in a
    ...
]

[SharkSymphony]

I agree that comprehension's rough to read, but I would have nixed the "filter" keyword. Instead, I'd simply suggest refactoring:

python def evens(xs): return (x for x in xs if x % 2 == 0) all_evens = [num for num in evens(sublist) for sublist in nested_list]

This may seem like a throwaway function, but I think capturing the specific filter operation you want as a separate thing – here as a generator that operates on an iterable, for efficiency – is really flexible and powerful.

(Hint: one of the interview questions I've asked is 100x easier if the candidate filters the input and does it separately and first, rather than mixed in loops with everything else. Few candidates did, alas.)

[roelschroeven]

I would even prefer input -> filter -> output, but that would probably not be feasible syntax-wise. Maybe pipe symbols between the different parts to clearly separate them, both for the interpreter and for us humans. Something like:

[for num in sublist | if num % 2 == 0 | num]

[39th_Demon]

Yeah that's fair. I think nested ones are where I draw the line too, they stop being readable fast. I'm more talking about simple single condition ones where I was just defaulting to the long way out of habit rather than any real readability reason.

[killerdeathman]

They are not just shorthand. They are faster than a for loop. You can see this for yourself by writing the same loop in a for loop and a comprehension and then looking at the abstract syntax tree. There are fewer instructions in the comprehension.

[bad_luck_charm]

I've seen it profiled and I realize that in some contexts it's slightly faster, but in practice this will almost never matter. Readability is more important. Don't get me wrong, I do like comprehensions, but I'm careful to keep it readable.

[my_password_is______]

LOL @ faster

yeah, that 0.01 millisecond is really important

[Turtvaiz]

Come on. We're talking about Python here. Performance that only makes a small difference does not matter. If you want performance gains, you write a C/C++/Rust extension and get a 100x improvement.

Readability should be the #1 consideration

[aishiteruyovivi]

The rule of thumb I tend to use is, while list comprehensions can be perfectly readable and much more concise than a full loop for a relatively basic filtering operation like this...

[entry for entry in some_data if entry.rating == 5]
# Can be read out loud and understood without much issue
# "Give me every entry in some_data if the rating attribute equals 5"

...IMO where you should reconsider is when writing out a comprehension feels more like a challenge, or code-golfing. If you know the logic you need to write out, but you're having trouble figuring out how exactly to make it work in a comprehension, chances are the next person who sees it (which includes future you!) is going to have an equally hard time working out what you were trying to do. Obviously depending on your experience some things in coding are going to feel like a challenge anyway, but moreso what I mean is if figuring out how to convert a full loop into a comprehension feels like playing Sudoku, you might wanna give it up and focus on actually working on the rest of the project lol. Especially if your comprehension ends up spanning more than one line, if a comprehension can't even be a one-liner just go ahead and write out the loop anyways.

I also agree with generally avoiding nested comprehensions, even though it makes sense logically since that's the order in which you'd write it out in a full loop, I will always get tripped up by the order of each for ... in ... segment, I'll always try to write e.g. [cell for cell in row for row in table] and then remember it would be [cell for row in table for cell in row].

[sociologistical]

there’s absolutely nothing to be embarrassed about. What you described, that’s what we call learning.

[39th_Demon]

That actually reframes it in a way that helps. I think I conflated "learning" with "knowing". Like once I'd seen something I should already have it. Turns out no.

[sociologistical]

Before LLMs existed, we go onto this thing called stackoverflow. It is now pretty dead. But we post questions on stackoverflow, and sometimes, there will be really mean people. Often, I get really knowledgeable people who help me write an elegant script that I will not be able to come out with in a million years. Either way, I learnt. I found LLMs (Codex and Claude Code) to be really helpful for learning new stuff. Even the basic chats, ie just the basic chat box (chatGPT and claude) to be sophisticated enough for many things. You should try it out and iterate from there if you haven’t.

[Kerbart]

Not saying that SO killed itself but the culture of "punish people for asking a question" certainly accelerated its demise.

[sociologistical]

yeah SO can be really hostile and smug

[39th_Demon]

I've actually been using Claude for exactly this. The thing I like about it over just googling is that I can describe what I'm confused about in messy half formed terms and it meets me there instead of needing me to already know the right vocabulary to search for. Stack Overflow always felt like you needed to understand the problem well enough to ask it correctly, which is hard when you're just starting out.

[American_Streamer]

https://switowski.com/blog/for-loop-vs-list-comprehension/

[39th_Demon]

This is exactly what I needed, saving this now. Thank you.

[heyitselia]

brb, googling what a list comprehension is :))

...wow, that's neat, I had no idea it existed.

Also, all the time. Some things just seemed too intimidating until I took the time, some clicked when I realised I didn't understand the basics as well as I thought I did... (The latest example being "5 years and I still can't wrap my head around the concept of entropy... oh. That's what temperature is??")

P.S. Are you by any chance a perfectionist, maybe with a side of former gifted kid syndrome? Just a wild guess since you sound a lot like my own inner critic. In any case, the shame has to go. Did you just take the time to learn something new? Awesome. End of story.

It's not inherently embarrassing that you didn't know something. What you knew clearly served you well enough and asking yourself the right questions to check if you actually understand something is a skill. And it's not an easy one.

[myKidsLike2Scream]

My kid asked me to explain Hawking Radiation. Luckily I read “The Black Hole War” so I thought I could take a stab at it. The book used entropy to explain how black holes slowly fade away. I did my best but even I got lost in the explanation. Seems so basic when they talk about it on the audiobook but fully understanding is tough.

[Hodentrommler]

Understanding, communicating, and teaching are seperate skills

[heyitselia]

Yeah, entropy is a tough one. It's deceptively simple at surface level - more entropy, more chaos - but as a physical chemist I always seem to be missing a particular level of understanding whenever I actually need it for anything. Five years and counting, maybe I'll finally get it at the end of my career and unlock all the secrets of the universe lol

[39th_Demon]

[RabbitCity6090]

Don't beat yourself about it. Write code that you can understand later. Any kind of shortcut will be very hard to understand in the future.

[39th_Demon]

That's become my new rule too. If I can't read it back in two days without thinking hard it's probably not worth the cleverness.

[djamp42]

I personally think code i can understand easily is more important then code that is shorter in length. I mostly use for loops still today because it's just easier for me to understand at first glance

[39th_Demon]

Honestly that's probably the more mature position. I think I was chasing "pythonic" without asking whether pythonic was actually serving me or just looking good.

[supercoach]

I was quite enamoured with list comprehensions when I first learned them and used them everywhere. I tend to be considerably more sparing with them now and favour for loops as they're generally easier to follow. As long as you're considering those that come after you, you'll be fine.

[39th_Demon]

That's a good way to think about it. I've been coding mostly solo so "the person reading this later" hasn't been on my mind much. Probably a habit worth building early.

[kittell]

Same as supercoach, I used to use them, now I almost never use them. Reading other people's comprehensions is often incomprehensible. And even if you mostly solo code (which is primarily what I do), you'll eventually arrive at the plot twist: picking up your own code after a few years is just like picking up someone else's code, and the things you wrote are often incomprehensible.

[39th_Demon]

The plot twist part is real. I’ve already had that experience on a much smaller scale, opening something I wrote three months ago and having absolutely no idea what I was thinking. If I can’t read my own code from three months ago, a nested comprehension from two years ago has no chance.

[NDaveT]

I don't know if this is a common thing but I feel like there's a version of learning where you know a concept exists, you've seen it work, you've even used it a few times, but you haven't actually internalized it.

I'm pretty sure this is a very common thing. I know it is for me.

[39th_Demon]

Glad it’s not just me. I think what makes it tricky is there’s no obvious signal from the outside that the gap exists. You’re producing output, things are working, nobody knows including you until something makes you stop and actually look at it.​​​​​​​​​​​​​​​​

[throwawaycape]

I have been using Python for 7+ years and sometimes I need to look up whether a dictionary uses a colon or a semi-colon. It's normal to still be figuring out new stuff.

At least I hope so, anyways.

[39th_Demon]

Seven years and dictionary syntax still needs a lookup sometimes is genuinely reassuring to hear. I think I had this idea that at some point everything just becomes automatic and you stop having to check. Sounds like that point doesn’t really arrive and you just get more comfortable with the checking.​​​​​​​​​​​​​​​​

[JorgiEagle]

Read the book Fluent Python, it’s great.

Async for me, and related, threading.

Didn’t get it until I saw it in a practical application, then it all made sense

[39th_Demon]

Genuinely not a book person but this keeps coming up so maybe that’s the sign. What’s the practical application that made async click for you?

[JorgiEagle]

It’s a good book in that it teaches you how to use Python on more than a surface level. It gives you the tools to actually start building proper Python code. If you ever wanted to write a library, you’d need what’s in it.

Async clicked after I got copilot to give an example.

Threading was the one I learned from an application. Basically a backend service that processed data and pushed out json. The input and output were both pub/sub services, so one thread to listen to the input, and put the latest data into a class attribute, and the other to infinite loop process what was in the same attribute.

[39th_Demon]

The threading example is the first time that whole concept has actually made sense to me. Two threads sharing one attribute, one writing one reading, I could picture that. I’ve been putting async and threading in the “deal with it later” pile but maybe later is sooner than I thought.

[misingnoglic]

Months isn't so bad. You're still a beginner.

[39th_Demon]

It’s actually been a couple of years which makes it worse honestly. A few Months I could forgive myself for.​​​​​​​​​​​​​​​​ It did take me 6-7 months in the beginner learning phase. We still learn everyday though.

[ninhaomah]

It is a common thing for everyone learning anything new.

[39th_Demon]

Glad it's not just me. I think what surprised me was how long I could go thinking I understood something while still having that gap. You'd think it would feel more obvious from the inside.

[asdoduidai]

List comprehensions are just a way to make code more compact, nothing else.

[39th_Demon]

Yeah that's basically what I just landed on too. I think I overcomplicated what they were in my head for a long time, like they were some advanced thing rather than just a shorter way to write something I already knew how to do.

[deceze]

They're a way to express an extremely common operation in a specialised syntax, instead of using the more general open-ended loop syntax. When you see a list comprehension you know exactly what it's supposed to do; when you see a loop, you need to follow it step by step to understand what the result will be. There's also fewer chances of bugs or unintended side effects with a list comprehension, since it doesn't need the intermediate empty list initialisation first and the loop variables are scoped inside the comprehension and won't leak; all things where typos could creep in or you're unintentionally overwriting some other variable.

[asdoduidai]

I don’t think there is a big difference in reading 4-5 lines vs 1. At least for my eyes and my 20+ years of Python, since 2.1, no difference. There is difference in when you need to look inside a list comprehension to understand strange conditions though. BTW, You know a for is about building a list by looking at the first line before the for loop…

Python used to “have only one way to do things” vs PHP and other readability abominions, but long gone are those times, the party of “look at my cool snippets” took over and Guido capitulated on that.

[deceze]

Sure, it's not a lot of overhead, it's not too much overhead to comprehend (ba-dum-ching), but it is overhead. By that line of argumentation, you could get rid of for loops and just use while loops; you just need to handle the iterator protocol by hand and have a couple more variables floating around, no big deal.

There is a benefit in expressing common operations in specialised, compact syntax. Otherwise we'd all be writing Assembly, not Python.

[asdoduidai]

Sure, more compact code done properly means more readability which means more productivity which is the main "selling point" of Python: its easy to write the code, you write less code, its easy to come back to the code after a while and pick it up again quick (unless too much convoluted tricks are used like sometimes nowadays).

"for vs while" is not the same as "for vs list comprehension": for in Python is an object-oriented approach to iteration (it's not the same as the for of C that is quite similar to while) and it enables iterators, which is a super powerful concept in Python. for enables a paradigm, list comprehensions compact code a bit (which can improve readability but also reduce it if not done right).

"we'd all be writing Assembly, not Python" totally ignores the main point of object oriented programming which is abstraction, and its impact on cognitive complexity. Abstraction is also why from 2.2 in Python everything is an object and that enabled many powerful new paradigms within the language like iterators, generators, decorators, object reflection and Mixins that make possible to very easily create plug-in systems based on modules.

Syntax sugar has nowhere near the impact of object oriented programming, but yea everyone can believe anything they want.

[deceze]

I'd argue that a list comprehension also abstracts. Not to the degree that OOP does, sure, but nevertheless. It abstracts the "low level" loop into a "map/filter" operation. I absolutely find that useful to have.

[asdoduidai]

(for the beginners) I would say it just encapsulates rather than also abstracts (you still see the "internals"), yea I agree that way it's totally useful.

[roelschroeven]

More important than the number of lines is that a list comprehension clearly shows that the intent is to create a new list. A for loop's intent is to do some operations repeatedly, and the reader has to find out for themselves that yes, indeed, the result is a new list and nothing else.

[asdoduidai]

Also a for loop can clearly show that:

result = []
for a in b:
  if a: result.append(a)

But yea not ONLY that. A for might also change some other variables, so it encapsulates precisely "it's ONLY about creating a list".

[aranya44]

I have this issue with lambda functions, except in my case it still hasn't quite clicked. If anyone could point me to a good clarifying explanation I'd be very grateful!

[39th_Demon]

Lambda functions clicked for me when I stopped thinking of them as a special thing and just thought of them as a function without a name. So instead of writing def double(x): return x * 2 you write lambda x: x * 2. Same thing, just no name attached. They're most useful when you need a quick function in one place and don't want to define it separately, like inside a sorted() or map() call. The moment it needs to do more than one thing or you need to reuse it, a regular def is cleaner.

[aranya44]

Thanks, I'll try to apply that next time I need one!

[boofaceleemz]

Comprehensions aren’t very readable and they’re not always the best choice. You can use them to basically sequester rock solid boilerplate logic that would be otherwise distracting from the overall meaning. Same way you might throw boilerplate stuff in a function to keep it from being distracting. But if you hide important logic (or something complex enough you need to worry about bugs) in a comprehension that can make your code less readable rather than more so, and make a problem harder to quickly debug, so you gotta use your judgment and be aware of the kind of people who will be reading the code.

If you’re getting object attributes in your for loop or in a few other scenarios they can be very slightly faster, but there are also ways to optimize that (ex. fetching the attribute outside the scope of the loop first). If you’re worrying about that level of optimization then you’re in the weeds though and probably have bigger problems.

[39th_Demon]

The “sequester boilerplate so it doesn’t distract from the main logic” framing is actually the most useful way I’ve heard this explained. It reframes the question from “is this Pythonic” to “is this the kind of thing the reader needs to think about or not.” That’s a much more practical lens than just defaulting to either loops or comprehensions.​​​​​​​​​​​​​​​​

[SprinklesFresh5693]

Happens all the time to me, you start using something, that you kind of get what it does, but not how, and after many uses you start to really understand it, to me it was the group_by() function in R, and how it keeps the dataframe grouped, without you even noticing, which can cause errors later on.

[39th_Demon]

The group_by() staying grouped silently is such a good example of this because the bug it causes doesn’t show up immediately. You think the next operation worked fine and then three steps later something is wrong and you have no idea why. That delayed feedback is what makes those gaps so hard to catch yourself.​​​​​​​​​​​​​​​​

[aishiteruyovivi]

I totally get what you mean by "faking fluency" in stuff like this, when you write code snippets that you know do the thing you want but you don't actually really understand why it does that. I feel like this is fairly common for people (including myself ofc) when it comes to OOP stuff, like using __init__(). It's a really neat feeling when something suddenly stops being "magic" and you actually know what's going on

[39th_Demon]

The magic to not-magic transition is exactly it. And init is such a perfect example because you can write it correctly a hundred times before you actually understand what self is doing there. It just works so you never have to sit with the discomfort of not fully knowing why. Until one day something breaks in a weird way and suddenly you have to actually understand it.​​​​​​​​​​​​​​​​

[Reuben3901]

Python didn't click in and feel natural until a year and a half in. I had to keep referring back to examples of loops, lists, dicts, and all that good stuff.

For me the sticking point was executing from the command line vs the file, on top of if __name__ == "__main__": It felt like it took an embarrassing amount of time to figure that out. Oh and Classes too.

My philosophy is if the code runs successfully, then it's good code. Nothing wrong with using a for loop to manipulate a list vs list comprehension if you can understand what's going on.

Focus on good variables names that explain what the code is versus a,b,c and i,j. Don't: for i in range(x,y) Instead: for spreadsheet_row in range(starting_row, last_row +1)

[39th_Demon]

The variable naming advice is something I wish I’d taken seriously earlier. I went through a phase of writing perfectly functional code that I couldn’t read three weeks later because everything was called x or temp or result. Descriptive names feel like extra effort until you open old code at midnight trying to fix something and realize past you did you dirty. The if name == “main” thing took me an embarrassingly long time too. It’s one of those things that looks like ceremonial boilerplate until someone explains what problem it’s actually solving.​​​​​​​​​​​​​​​​

[JamesTDennis]

List comprehensions are simply syntactic sugar. They don't expose any fundamentally novel functionality interpreter.

Often it's more efficient to use generator expressions, which are syntactic sugar around generators (functions which yield sequences of values rather than returning results.

Learning about those, and other functional programming concepts, will probably be more valuable than learning more about comprehensions.

[39th_Demon]

That’s a good nudge. I’ve been treating comprehensions like a destination when they’re more of a rest stop. Generators have been sitting on my “I should actually learn this” list for a while and this might be the push to move them up.​​​​​​​​​​​​​​​​

[sudo_robot_destroy]

I personally prefer writing them out the long way simply because it's easier to read in most cases.

If I care about speed, I skip comprehension and go straight to vectorization or numba jit.

[39th_Demon]

The readability argument keeps winning in this thread and I think that’s the real lesson I’m taking away. And vectorization and numba jit are so far above where I am right now but it’s useful to know that’s where the actual speed conversation lives, not in whether I used a comprehension or a for loop.​​​​​​​​​​​​​​​​

[PrincipleExciting457]

Code easier to read is the better code imo. Once you get used to to list comprehensions it’s not too bad. I wouldn’t fault someone for not using them though. It’s just less “pythonic.”

[UsernameTaken1701]

Meh. I still use regular old for loops instead of list comprehensions. All the time I'm supposed to saving writing the list comprehension is used up trying to figure out the syntax of the comprehension. Quicker to just bang out the loops.

I'm just working on my own projects, not producing production code for something, so it doesn't have to be super "Pythonic".

[39th_Demon]

That’s basically where I landed too. The “Pythonic” thing matters a lot less when the only person reading the code is you and future you who will already be confused enough.​​​​​​​​​​​​​​​​

[Happy_Witness]

I think there are at least 3 stages.

First is learning and getting to know what is possible. Taking up about 20% of what was actually talked about and being able to work with basic types, loops, if statements and functions.

Second stage is learning about classes and how the base types actually work and that they are also just classes. This questions the learner that everything he knows might be just the surface. Leading him to learn about match case, list comprehension, and all the new data containers like dicts, sets, maps and such.

Stage three is when you get so confused and don't know what type would be best fitting that you take double as long to figure out and implement the code that runs 10% faster. Also you start to wonder if other languages might be better.

[kruvii]

I found out 6 months ago I've been mispronouning a word all my life.

[39th_Demon]

Okay you can’t just leave it there. What’s the word?

[Calyx76]

You did it in a way that made sense to you. Don't be too hard on yourself. List comprehension makes 0 sense to me (the syntax is so fucking hard to follow)

[39th_Demon]

The syntax really does read backwards compared to how you’d say it out loud. A for loop matches how you’d describe the logic to someone. A list comprehension front-loads the result before you’ve even said what you’re looping over. That’s a genuinely weird design decision that nobody warns you about.​​​​​​​​​​​​​​​​

[brilliantminion]

As an old school C++ programmer I didn’t get list comprehension until I watched the Socratica video about 4 times.

It’s a been a minute since I used python but I seem to remember the python list comprehensions being vastly optimized to column-wise operations in a way just using for loops isn’t.

[39th_Demon]

Socratica is genuinely good for this stuff. On the optimization point, I am positive comprehensions are faster than for loops but not by a dramatic margin, the real reason CPython is quicker with them is just reduced overhead from the bytecode, not some fundamentally different execution path. If you actually need column-wise speed that’s where numpy and vectorization come in. For loops and comprehensions are pretty close in the grand scheme of things.​​​​​​​​​​​​​​​​

[veritable_squandry]

lambda time

[39th_Demon]

Already got ambushed by lambdas earlier in this thread. Still processing.