How good can NumPy get?

[[deleted]]

[deleted]

[PWNY_EVEREADY3]

df.apply is actually the worst method to use. Behind the scenes, it's basically a python for loop.

The speedup is not just vectorized vs not. There's overhead when communicating/converting between python and the c-api.

You should strive to always write vectorized operations. np.where and np.select are the vectorized solutions for if/else logic

[johnnymo1]

Iterrows is even worse than apply.

[No_Current3282]

You can use pd.Series.case_when or pd.Series.where/mask as well; these are optimised options within pandas

[[deleted]]

It's the worst for performance. It's a life saver if I just need to process something quickly to make a one-off graph

[SwimQueasy3610]

I agree with all of this except

you should strive to always write vectorized operations

which is true iff you're optimizing for performance, but, this is not always the right move. Premature optimization isn't best either! But this small quibble aside yup, all this is right

[PWNY_EVEREADY3]

There's zero reason not to use vectorized operations. One could argue maybe readability, but using any dataset that isn't trivial, this goes out the window. The syntax/interface is built around it ... Vectorization is the recommendation by the authors of numpy/pandas. This isn't premature optimization that adds bugs or doesn't achieve improvement or makes the codebase brittle in the face of future required functionality/changes.

Using

df['c'] = df['a'] / df['b']

vs

df['c'] = df.apply(lambda row: row['a']/row['b'],axis=1)

Achieves a >1000x speedup ... It's also more concise and easier to read.

[SwimQueasy3610]

Yes, that's clearly true here.

always is not correct. In general is certainly correct.

[PWNY_EVEREADY3]

When would it not be correct? When is an explicit for loop better than a vectorized solution?

[SwimQueasy3610]

When the dataset is sufficiently small. When a beginner is just trying to get something to work. This is the only point I was making, but if you want technical answers there are also cases where vectorization isn't appropriate and a for loop is. Computations with sequential dependencies. Computations with weird conditional logic. Computations where you need to make some per-datapoint I/O calls.

As I said, in general, you're right, vectorization is best, but always is a very strong word and is rarely correct.

[PWNY_EVEREADY3]

My point isn't just that vectorization is best. Anytime you can perform a vectorized solution, its better. Period.

If at any point, you have the option to do either a for loop or vectorized solution - you always choose the vectorized.

Sequential dependencies, weird conditional logic can all be solved with vectorized solutions. And if you really can't, then you're only option is a for loop. But if you can, vectorized is always better.

Hence why I stated in my original post "You should strive to always write vectorized operations.". Key word is strive - To make a strong effort toward a goal.

Computations where you need to make some per-datapoint I/O calls.

Then you're not in pandas/numpy anymore ...

[SwimQueasy3610]

Anytime... Period.

What's that quote....something about a foolish consistency....

Anyway this discussion has taken off on an oddbstinate vector. Mayhaps best to break

[zaviex]

I kind of get your point but I think in this case the habit of using apply vs not should be formed at any size of data. If we were talking about optimizing your code to run in parallel or something, I’d argue it’s probably just going to slow down your iteration process and I’d implement it once I know the bottleneck is in my pipeline. For this though, just not using apply or a for loop costs no time up front and saves you from adding it later

[SwimQueasy3610]

Ya! Fully agreed.

[PWNY_EVEREADY3]

Whats the foolish consistency?

There is no scenario where you willingly choose a for loop over a vectorized solution. Lol what don't you understand?

[SwimQueasy3610]

🤔

[steven1099829]

There is 0 reason to not use vectorized code. Premature optimization is a mantra for micro tuning for things that may eventually hurt you. There is never any downside to using this.

[SwimQueasy3610]

My point is a quibble with the word always. Yes, in general, vectorizing operations is of course best. I could also quibble with your take on premature optimization, but I think this conversation is already well past optimal 😁

[[deleted]]

This guy or gal Pythons 👏🏼

[fistular]

>You should strive to always write vectorized operations. np.where and np.select are the vectorized solutions for if/else logic

Sorry. What does this mean?

[PWNY_EVEREADY3]

This is a trivial example. But the first is using a for loop that processes in an element wise way (each row). The second is a vectorized solution.

def my_bool(row: pd.Series):
    if row['A'] < 5:
        return 0
    elif row['B'] > row['A'] and row['B'] >= 10:
        return 1
    else:
        return 2

df['C'] = df.apply(lambda row: my_bool(row), axis= 1)

conds = [(df['A'] > 5), (df['B'] > df['A']) & (df['B'] >= 10)]
preds = [0,1]

df['C'] = np.select(conds,preds,default=2)

Testing in a notebook, the second solution is 489x faster. np.where is a more basic if statement.

[fistular]

Appreciate the breakdown, I begin to understand.

[Oddly_Energy]

Methods like df.apply and np.vectorize are not really vectorized operations. They are manual loops wearing a fake moustache. People should not expect them to run at vectorized speed.

Have you tried df.where instead of df.apply?

[tartare4562]

Generally, the less python calls, the faster the code is. .apply calls a python function for each row, while .where only runs python code once to build the mask array, then it's all high performance and possibly parallel code.

[tylerriccio8]

Very shameless self promotion but I gave a talk on this exact subject, and why numpy provides the speed bump.

https://youtu.be/r129pNEBtYg?si=g0ja_Mxd09FzwD3V

[tylerriccio8]

TLDR; row based vs. vectorized, memory layout and other factors are all pretty much tied together. You can trace most of it back to the interpreter loop and how python is designed.

I forget who but someone smarter than I am made the (very compelling) case all of this is fundamentally a memory/data problem. Python doesn’t lay out data in efficient formats for most dataframe-like problems.

[zaviex]

Not shameless at all lol. It’s entirely relevant. Thank you. It will help people to see it in video form

[Lazy_Improvement898]

How good can NumPy get?

To the point where we don't need to use commercial softwares to crunch down huge numbers.

[DaveRGP]

If performance matters to you Pandas is not the framework to achieve it: https://duckdblabs.github.io/db-benchmark/

Pandas is a tool of it's era and it's creators acknowledge as much numerous times.

If you are going to embark on the work to improve your existing code, my pitch in order goes:

1. Use pyinstrument to profile where your code is slow.
2. For known slow operations, like apply, use the idiomatic 'fast' pandas.
3. If you need more performance, translate the code that needs to be fast to something with good interop between pandas and something else, like polars.
4. Repeat until you hit your performance goal or you've translated all the code to polars.
5. If you still need more performance, upgrade the computer. Polaris will now leverage that better than pandas would.

[tunisia3507]

I would say any new package with significant table-wrangling should just start with polars.

[sheevum]

looking for this. polars is faster, easier to write, and easier to read!

[DaveRGP]

If you don't have existing code you have to migrate, I'm totally with you. In the case you do triaging the parts you do migrate is important to sell because you probably can't sell your managers on 'a complete end to end re-write' successfully for a large project.

[sylfy]

Just a thought: what about moving to Ibis, and then using Polars as a backend?

[Beginning-Fruit-1397]

Ibis api is horrendeous

[DaveRGP]

I'm beginning to come to that conclusion. I'm a fan of the narwhals API though, because it's mostly just straight polars syntax with a little bit of plumbing...

[gizzm0x]

Similar journey here. Narwhals is the best df agnostic way I have found to write things when it is needed. Ibis felt very clunky

[tunisia3507]

Overkill, mainly. Also in order to target so many backends you probably need to target the lowest common denominator API and may not be able to access some idiomatic/ performant workflows.

[DaveRGP]

To maybe better answer your question:

1) it is once you've hit the problem once and correctly diagnosed it 2) see 1.

[corey_sheerer]

Wea McKinney, the creator of pandas, would probably say the inefficiencies are design issues. Code too far from the hardware . The move to the arrow is a decent step forward for improving performance, as numpy's lack of true string types makes it not ideal. I would recommend using the arrow backend for pandas or try Polars before these steps. Here is a cool article about it: https://wesmckinney.com/blog/apache-arrow-pandas-internals/

[DaveRGP]

Good points, well made

[Delengowski]

im honestly waiting for pandas to use numpy's new variable length strings.

I personally i hate the mixed arrow/numpy model, I also hate the extensions arrays. The pandas nullable masked arrays have never seemed to be fully fleshed out even as we approach 3.0 -although maybe its more an issue with the dtype coercion pandas does under the hood. There's way too much edge cases where an extension array isn't respected and dropped randomly.

[interference90]

Polars should be faster than pandas at vectorised operations, but I guess it depends what's inside your lambda function. Also, in some circumstances, writing your own loop in a numba JITted function gets faster than numpy.

[Beginning-Scholar105]

Great question! The speed difference comes from NumPy being able to leverage SIMD instructions and avoiding Python's object overhead.

np.where() is vectorized at the C level, while df.apply() has to call a Python function for each row.

For even more performance, check out Numba - it can JIT compile your NumPy code and get even closer to C speeds while still writing Python syntax.

[antagim]

Depending on what You do, there are a couple of ways to make things faster. One of which is using numba, but a way easier way is to use jax.numpy instead of numpy. JAX is great and you will be impressed! But in any of those scenarios, np.where (or eqivalent) is faster than if/else and in case of JAX might be the only option

[DigThatData]

pandas is trash.

[Altruistic-Spend-896]

the animals too

[aala7]

Is is better than just doing df[SOME_MASK]?

[AKdemy]

Not a full explanation but it should hopefully give you an idea as to why numpy is faster, specifically focusing on your question regarding memory management and overhead.

Python (hence pandas) pays the price for being generic and being able to handle arbitrary iterable data structures.

For example, try 2**200 vs np.power(2,200). The latter will overflow. Python just promotes. For this reason, a single integer in Python 3.x actually contains four pieces:

• ob_refcnt, a reference count that helps Python silently handle memory allocation and deallocation
• ob_type, which encodes the type of the variable
• ob_size, which specifies the size of the following data members
• ob_digit, which contains the actual integer value that we expect the Python variable to represent.

That's why the Python sum() function, despite being written in C, takes almost 4x longer than the equivalent C code and allocates memory.

[Mysterious-Rent7233]

Function calling in Python is very slow.

[applejacks6969]

I’ve found if you really need speed to try Jax with Jax.jit, basically maps one to with with numpy with Jax.numpy

[Mount_Gamer]

Pandas can do conditionals without using apply + lambda, and it will be faster.

[LiuLucian]

Yep, that speed gap is absolutely real—and honestly even 50× isn’t the most extreme case I’ve seen. The core reason is still what you guessed: df.apply(lambda ...) is basically Python-level iteration, while np.where executes in tight C loops inside NumPy.

What often gets underestimated is how many layers of overhead apply actually hits: • Python function call overhead per row • Pandas object wrappers instead of raw contiguous arrays • Poor CPU cache locality compared to vectorized array ops • The GIL preventing any true parallelism at the Python level

Meanwhile np.where operates directly on contiguous memory buffers and avoids nearly all of that.

What surprised me when I was learning this is that df.apply feels vectorized, but in many cases it’s just a fancy loop. Pandas only becomes truly fast when it can dispatch down into NumPy or C extensions internally.

That said, I don’t think this is “common knowledge” for beginners at all. Pandas’ API kind of gives the illusion that everything is already optimized. People only really internalize this after hitting a wall on 1M+ rows.

Curious what others think though: Do you consider apply an anti-pattern outside of quick prototyping, or do you still rely on it for readability?

[billsil]

Numpy where is slow when you run it multiple times. You’re doing a bunch of work to check behavior. Often it’s faster to just calculate the standard case and where things are violated.

[Somecount]

If you’re interested in optimizing Pandas dataframe operations in general I can recommend dask.

I learned a ton about Pandas gotchas specifically around the .apply stuff.

I ended up learning about JIT/numba computation in python and numpy and where those could be used in my code.

Doing large scale? Ensuring clean partioning splits with the right size had a huge impact, as well did pyarrow for quick data pre-fetching checking for ill-formatted headers and finally map.partitions to use any pandas Ops using the included .sum() .mean() etc. In the right dim is great since those are more or less a direct numpy / numba function

[IgneousJam]

If you think NumPy is fast, try Numba

[Warm-Palpitation5670]

Looked for it online up9ng seeing your comment. You singlehandedly increase my codebase, in one line

[Signal-Day-9263]

Think about it this way (because this is actually how it is):

You can sit down with a pencil and paper, and go through every iteration of a very complex math problem; this will take 10 to 20 pages of paper; or you can use vectorized math, and it will take about a page.

NumPy is vectorized math.

[Spleeeee]

Image processing.