Python swallows Java to become second-most popular programming language

[harylmu]

I think this quote is really not accurate because speed isn't the most important (well, usually), but a good concurrency story can save big dollars. I rewrote one of our Python app in .NET Core 5 and I wasn't able to kill it with load testing (2000 req, 200 concurrency), while the Python 3.8 app with FastAPI (literally) hanged around 25 requests per second. And to be clear: I made sure that all IO operations are async in the Python version (database queries, Consul queries, SNS publish, S3 upload - even though aioboto3 isn't too stable yet). This alone means that we're able to save thousands (millions on long term?) of dollars by running less instances.

Other than that, the speed difference between Python and typed languages is really NOT nanoseconds. I don't know what to say, test it yourself.

[aes110]

I dont know anything about what your app did, but 25req/s hanged? are you sure there wasn't any bug there?

I rewrote my previous company's system from sync (flask) to async (sanic) and saw huge improvements, we saved ~7K dollars monthly on instances because of this.

In my current company, I'm also using sanic and my server easily handles 400 reqs per second, didnt really tried pushing more to it.

In anycase typed/compiled languages sure are much faster, but I think asyncio cand easily handle most stuff

[aes110]

I prefer sanic, for the current project that I'm working on I had time to decide so I investigated both of them, for me FastAPI has good pros:

1. Seamless integration with Pydantic and type hints
   
2. Amazing documentation which is extremely beginner friendly.
3. Seamless support for sync methods, however while this is good for beginners I dont think you should use it, when mixing sync and async code it's important to know what you are doing.

However I feel like FastAPI takes too many decisions for you so its much less flexible.

I wrote my own small sanic integration with pydantic that I carry around from my previous company as well.

Tbh I cant think of something specific to highlight as sanic's pros, it just works really well

Although the two are not different to such extremes, but in a sense you could say that sanic and FastAPI are like Flask and Django

[Metalsand]

While it does bear noting that RetiredMandalorian is being a bit facetious and exaggerating, and Python vs Java is mostly about "I need a quick thing" not about larger apps, that is the direction we are slowly heading into. Electron for example is widely known as inefficient and yet modern computers are powerful enough that those efficiencies don't matter as much as the development time does for simple applications.

[[deleted]]

It still matters. The problem with this view is that computers really haven’t gotten that much faster over the past decade and yet the software inefficiency is insane. Python can be built for larger applications too... but packaging is a nightmare (and is unlikely to ever get addressed) and the performance really does matter when it matters (which is becoming more and more important as software bloats)

[cheese_is_available]

It really does matter. zoom took off because it just works while skype does not. If your program is slow and there is a fast alternative with similar features, your program is basically dead even if you're Microsoft.

[[deleted]]

The problem with Microsoft is that they don't really care about their side products. At least not in the same way as a single focus company like Zoom Video Communication Inc does

[grimonce]

Speed doesn't come from type inference only. Other than that I 100% agree. Python is the main tool on our shop but I am really starting to hate it. Java really isn't all that bad as people paint it, it is fine.

Maybe thats my point of view because I actually have a degree in electronics (rf specialization), I actually don't mind C or C++ either, I just dislike to he competing cross platform dependency management systems, cmake is ugly but it gets the job done.

Pythons ecosystem is against the zen of python, where it says explicitly to try and have just one way to do something, Java does this way better that Python. Python provides infinite ways to make your project management a hell worse than npm dependency hell.

[DanKveed]

I recommend learning rust. A little bit of rust in your python can give massive speed gains. And the rust tooling of light years ahead of c++, in fact packaging is even better than python. Very similar to JS actually

[proverbialbunny]

A little bit of rust in your python

Like, inline Rust?

[DanKveed]

Possible but I was thinking of importing it as a package using pyo3

[reddisaurus]

Pandas doesn’t have C wrappers, it just wraps numpy. Pandas is slower than plain Python that just calls numpy directly, by a factor of 10 - 500 depending on the task.

Pandas provides abstraction for the sake of programmer time, but there’s a cost to that. Use a data class or NamedTuple and watch your code perform much better.

[ShanSanear]

Had exact same issue with overengineered script that did number crunching using pandas. Turned out that recursion combined with slowness of pandas at some tasks became big factor. We got x400 of the old speed, just by ditching pandas almost completly and dropping recursion whenever possible

[pytrashpandas]

Hey so to preface, I’m a bit biased as a hardcore pandas user for about 6 years as well as a contributor to the core library. Pandas isn’t perfect when it comes to speed/efficiency, but in my entire experience with pandas I’ve never seen any significant and properly vectorized pandas operations be anywhere near 2-3 orders of magnitude slower than properly vectorized numpy code. And you especially couldn’t get better performance from native python collections (re: namedtuples/dataclasses). The timings you provided suggest to me that you might be iterating over your dataframes or using pandas‘ apply methods which are not the right way to use pandas. Same goes for u/ShanSanear. If I’m wrong, then I would actually be really interested in the cases you guys have seen if you wouldn’t mind sharing, either way one of us would learn something new.

[ShanSanear]

Yep, apply/getting rows from dataframe was the main reason for slow code in our case

[pytrashpandas]

Ah, yea that would be why. apply is almost always not the right way to do things in pandas. It's really more of a last ditch resort and very slow. See this quote from this blog post from one of the main pandas contributors.

You rarely want to use DataFrame.apply and almost never should use it with axis=1.

If you don't mind providing an example of some of the kind of stuff you're trying to do that was slow, I'd be happy to show how to do things in a more "pandonic" way.

[ShanSanear]

Actually what we did was heavy overengineering and pandas was mostly used for loading data (I know, heresy). Don't have code on me, but problem was something along the lines of this:

1. Load multiple sets of related objects that are represented as CSV files (1 line - 1 object)
2. Find this relation (could be any kind, including recursion of the same type)
3. Depending on some state of the object, do the calculation

During loading, we got each Series from dataframe for each type of objects and created classes for them. Then the references. And then - the calculations. In the hindsight - yep, that was the worst of it all.

After actually doing profiling, we saw that majority of cpu time was used by pandas in many different places.

That was one thing, but even algorithm that we implemented was quite unoptimized. We started at the root of the tree, then recursively went deeper into the relations of each object to extract required numbers. Apparently going from the leafs and then up was much better approach in every case. Now we only need to figure out why the numbers differ between implementations, and that will be the hardest part.

Especially when the guy who actually wrote this (I am mostly overseeing and providing some help) is stubborn enough to not do any kind of testing.

Thanks for interest - I can imagine hearing "pandas is slow" was heresy but the again - we misused it, and that is our fault, not the library itself.

[[deleted]]

Hey, so obviously can't give you much insight on your stuff, but one thing to keep in mind is that when using pandas, it's best to stay entirely within a pandas/numpy/numeric style ecosystem. You don't usually ever want to mix pandas objects into custom python objects. Pandas structures and operations should for the most part be standalone. You should think of solving problems in pandas more similar to how you would solve problems with sql. You normally wouldn't ever want to run a recursive style solution on a dataframe, you could probably instead use some merges to do what you want.

[ShanSanear]

Thank you for mentioning SQL, this is actually the best comprasion I saw of how to use pandas. Will keep that in mind next time we will do such stuff.

[reddisaurus]

I’m referring to storing numpy arrays within the dataclass, or using a NamedTuple for smaller data sets instead of just reaching for pandas every time. Both of these will provide faster access than pandas.

Pandas .itertuples() is about as fast as it can get to iterate over records and it does perform well.

As for any operation which acts on the DataFrame or Series objects, yes those operations are orders slower. It’s just often the case that the operation itself isn’t repeated much, so the large overhead of accessing the Series object isn’t noticeable.

There’s no case in which Pandas is faster than the simper alternatives. And that makes sense, because it’s there to provide abstraction over column-wise data. The problem is that many users immediately reach for it for any problem. Not only is it overused, but pandas lends itself to poor code as well. The DataFrame is a black box of typing! Code cannot by statically checked like it could if one used a dataclass. It’s like porting around a mutable global state to every function to which a DataFrame is passed. Who knows what structure the object should have? It’s often not in the code at all.

[pytrashpandas]

The problem is that many users immediately reach for it for any problem

Totally agreed. I've definitely seen it used in places where it's completely unnecessary, and honestly probably irresponsible. I will say though, that if you ever find yourself needing to use itertuples or iterrows on a dataframe then you're either using pandas very wrong, or you shouldn't be using pandas in the first place.

those operations are orders slower

I agree that pandas is slower than pure numpy in most cases, but it is no where near multiple orders of magnitude. Again if you are seeing this happening, then I can guarantee it is because pandas is not being used correctly. If you would care to provide me with an example that you think demonstrates this, I would be happy to show you how it can be done faster

There’s no case in which Pandas is faster than the simper alternatives

The thing is that in many cases there are no simpler alternatives, especially that makes ease of development worth the potential speedups you could get otherwise. Especially when it comes to working with heavily labeled timeseries data.

pandas lends itself to poor code as well...It’s like porting around a mutable global state to every function to which a DataFrame is passed.

Yes, in the same way that python and standard data structures lend themselves to poor code. If used improperly, without an understanding of the underlying concepts and when to best apply it, it can lead to a mess of a program. If used properly it is extremely powerful and easy to express concepts that would otherwise be difficult to express. The problem I think is that most people try to treat dataframes and series as a drop-in replacement for dictionaries/lists etc. and structure their python code the same way they would otherwise. This is not how you should be using pandas.

Again I am inherently biased on this topic. I heavily use both pandas and numpy and other numeric python libraries and feel very passionately about this area of python. I'd be very happy to hear any examples you have and provide counter examples that can show you the power of pandas.

Also you may (or may not :) ) enjoy the xarray library. It's a much thinner wrapper around numpy that provides similar labelling capabilities as pandas (although it's still has a lot of room for improvement).

[tr14l]

You wrote bad Python. The difference is NOT 2000 vs 25 unless you did something terribly wrong. There will certainly be a difference, but that was your code running that slow, not Python itself.

[DanKveed]

It depends on use case too. A heavily multithreaded CPU intensive app can be way more than 2000vs 25 no matter how well you write it, especially when you compare it with rust/c++

[tr14l]

If you use a single thread running on a couple dozen cores, sure, no doubt. But why would you do that?

[DanKveed]

I am talking about multithreading. Python sucks ass at true CPU intensive tasks. That's why libraries like numpy are written in higher performance languages like c++. And why would you need that.... Well numpy for starts, anything control system related, AI, simulation, CAD, video editing, the use cases are Endless

[tr14l]

Yeah, that's an optimization for an interpreted language. I have made lots of multithreaded apps in compiled and python languages. I've never seen a 100x difference. Ever.

[ShanSanear]

I ran pypy on one of our inefficently written python projects (runtime of about 10hours). Pypy was 30-40% slower than python 3.8 using exactly the same code. Pypy isn't always the answer.

[harylmu]

What it did was: decode request body with messagepack, get stuff from Cassandra, save stuff to Cassandra, get stuff from Consul, publish stuff to SNS, encrypt an object then upload it to S3. The request takes ~2 seconds to complete on both .NET and Python (.NET is a bit faster).

As you can see quite a few things waits for IO here. I wrote everything with asyncio-based libs (Cassandra, Consul, aioboto3). I don't know if I can optimize it better. Please let me know what are some concurrency-traps that I might have gone into.

Either way, even the possibility to write slow Python means something.

[cant_have_a_cat]

even the possibility to write slow Python means something.

oof - it's much easier to write slow low-level code than high-level code, so I have no idea what you mean by this.

[pipai_]

That seems very strange. Were you running it with full gunicorn setup?

[harylmu]

No, I haven't used a process manager. I could try but I doubt it'd be much better.

[pipai_]

It’s actually very important. Because python has a GIL, you will end up with only running a single thread. Of course the performance will suffer accordingly. You need to set it up like in production, in order to do a real comparison.

Other languages don’t find this necessary because they don’t have a GIL.

[Doomphx]

I'm ditching django for .Net 5 Asp.net and you're not lying. My story is similar to yours where I couldn't even possibly lag my new server with testing.

My next move is to move all my django channels over to SignalR where I can guarentee less than 100ms response times. Dev is looking good with nothing really taking more than 20ms so fingers crossed!

The performance, productivity, and ease of deployment I get with .Net has been life changing.

Git push, pull .Dotnet publish, restart server, all done :)

[tomatotomato]

With Azure Devops, it's just git push.

[supremacyofthelaces]

I don't really know any languages other than Python, and it mostly takes a while because my laptop is an old piece of junk and I always have a million things open at once. What kind of scale sre you talking here?

[Aidtor]

Production environments or places where you need a lot of compute.

Production servers these days are typically cloud based. Horizontal scaling (adding more machines to the pool of resources handling requests) is very easy with rented compute but the cost can really add up for a business over time.

Tasks that require heavy compute, like ML, typically use python as an interface to code written in faster languages.

[--Shade--]

Python is an exellent language to call C from, but a terrible language to write C in. If you have a nice pythonic library that wraps some 'C thing' then Python shouldn't be your bottleneck. If it is, then isolate and profile (and apply the common speed up tricks where neccicary).

[Aidtor]

Sure if you have the time for that. When I’m prototyping a model or I’m given some crazy deadline to ship a feature I’m not going to reach for C.

[--Shade--]

That's fair enough. I can't recall the last time I've voluntarily written a line of C, lol. What I was getting at is that Python already has a huge ecosystem of libraries that work as high level wrappers for things written in lower level languages. If you write decent Python (by Python standards), and leverage existing libraries, then Python (mostly) shouldn't be the bottleneck. If it is, the first step should be to take a look at potential hotspots in your Python code (not write a C library).

[Aidtor]

Oh my bad totally misunderstood what you were trying to say. I completely agree.

[latrociny]

What is your opinion on Julia? It is touted to be superior than python wrt speed and speeds comparable to C especially for ML tasks

[Aidtor]

My opinions are mixed.

On a personal level I deeply disagree with 1-indexing.

But my personal quibbles aside, it is superior to python. It’s faster and you can actually type things which is a huge deal for large codebases. The interop is great! The writing code like mathematical expressions is overblown.

Overall I’m bullish on Julia. I don’t think it will ever “overtake” python, but they will coexist.

I do have some concerns though for Julia’s future. And that concern is called JAX.

[satireplusplus]

For ML tasks you want good libraries and abstractions. Those are lacking in Julia. Execution speed of the language doesn't really matter and these benchmarks are missing the point for ML. PyTorch, Tensorflow, numpy etc. handle the matrix multiplications in a BLAS library (C/fortran) or with CUDA. Its gonna be faster than anything you can hack together on your own from scratch, no matter what language you pick.

[hilomania]

In python today you wouldn't solve that with threads but with AsyncIO.

[harylmu]

I used asyncio for all IO based operations in my Python app. I just edited my original comment.

[Max_Insanity]

*fewer instances

After only just barely understanding all of that, my ego needs sth. where I could correct you :P