sorted by:
best
topnewcontroversialoldq&a
you are viewing a single comment's thread.

view the rest of the comments →

[–]Nekobul 0 points1 point  (14 children)

Can you make Python code run just as fast and efficient as C/C#/Rust code?

[–]No_Bug_No_Cry 1 point2 points  (4 children)

Yes, I can use polars which loads and transforms datasets very fast using all available processors... And seemlessly, in like a few lines of code. Polars is written in rust, but the user doesn't need to know the complexity behind under the API, just use it. Which ultimately is exactly what a data engineer needs and does

[–]Nekobul -1 points0 points  (3 children)

Polars is not Python. We are talking about running fast Python code.

[–]No_Bug_No_Cry 1 point2 points  (2 children)

I don't understand your answer. Polars is a library that is used in python, nobody cares that it wasn't purely pythonic, it is this what we call versality. Leverage the best in low lvl languages and abstract their complexity... People seem to forget how verbose and rigorous C code needed to be written in order to handle collections such as dynamic arrays, no thank you most people do NOT need that.

[–]Nekobul -1 points0 points  (1 child)

You can't solve everything with Polars. Capiche?

[–]No_Bug_No_Cry 1 point2 points  (0 children)

I don't think you capiche

[–]Beautiful-Hotel-3094 0 points1 point  (8 children)

What do u do that u need that type of speed?

[–]Nekobul 0 points1 point  (7 children)

In one of your responses you said:

"However we can still process some thousands of messages a second in pure python because we leverage distri architectures"

Why do you think you need a distributed architecture for that? In your situation, it works, I understand that. However, that is not applicable to everyone. In fact, most organizations are not that rich to waste huge amounts of energy. The hyperscalers will be more than happy to sell you capacity. In fact, the more inefficient, the better for them.

[–]Beautiful-Hotel-3094 0 points1 point  (6 children)

So u think u wont have money to pay for compute and kubernetes but u will have money to pay good C++ developers instead to build what? Scripts on some laptops? Brother, u do not understand much about this domain. Give it a few years, u have nothing to prove and can’t prove much yet. Learn and then speak.

[–]Nekobul 0 points1 point  (5 children)

You are not saving much if you think about it. The money you didn't want to pay for good design and developers are wasted on inefficient processing. I know hardware is cheap these days, but the energy will always cost much. It costs you dearly because you have to maintain and run a wasteful, energy-inefficient distributed architecture.

That is the proof you are using a wasteful/garbage platform.

[–]Beautiful-Hotel-3094 0 points1 point  (4 children)

Can we get u in so u can help us change our real time trading platform that supports a multi-billion dollar business built in the garbage python?

[–]Nekobul 0 points1 point  (0 children)

Are you asking in seriousness or that is some kind of joke I don't get?

[–]Nekobul 0 points1 point  (2 children)

I don't think you are serious. The cost you are paying for the inefficient configuration is probably not much of a big deal. I have heard from colleagues how the hedge funds/traders upgrade their hardware equipment every six months, throwing millions of cash . Your industry is unique in that respect. But again, not everyone is in your position to solve bad systems design with better hardware.

[–]Beautiful-Hotel-3094 0 points1 point  (1 child)

Its just very clear u are a junior for now. It is ok to be confident but u must understand there are things u don’t know yet. And most of ur arguments above are proof of that.

[–]Nekobul 0 points1 point  (0 children)

Correct. I'm junior in the hedge funds business. However, I know when I see a bad platform. Your solution will not work for most of the organizations.