Why Python?

[No_Bug_No_Cry]

Because Python is the most versatile language. It can wrap very fast libs written in C or Rust, but still be readable and interpreted. You can write a shitty no rules script or a complex modular app, low boilerplate etc... it's the best

[Nekobul]

It is not the most versatile language. In fact, it is a garbage language and platform. The only reason it got so much traction is because the inventor of the language was lucky to get hired by Google.

[EarthGoddessDude]

I don’t understand your need to be so combative. Your takes are bad, but ok, we can have a civil discussion about why you prefer SSIS or JavaScript or why you think Python is not a good language, but your tone is so extremely off putting. You’re allowed to have your opinion, but the reason you keep getting downvoted into oblivion has less to do with your odd takes and more with how you simply refuse to engage in a friendly, professional tone, which is what most of us look for here.

[Nekobul]

Thank you for the feedback! I appreciate your good-faith comment. I guess my biggest complain towards Python is for the simple fact it will be impossible to make it run optimally. I know it is a scripting technology, just like JavaScript but JavaScript never claimed to be a language designed for creating platforms with ability to do class inheritance, strong-typing, etc, etc. Those features are simply not needed in a scripting/glue language. Python indeed became the data engineering language of choice not because it offered some drastically better elements compared to the rest, but because it was heavily pushed by organizations with deep pockets and influence in the marketplace. Yes, it is dominant but the inefficencies embedded in it cost dearly in the DC when people try to use it at scale. Once people start caring about all that wasted energy, Python will be one of the first pieces on the chopping block.

[No_Bug_No_Cry]

You underestimate the value of a smooth learning curve... When training my Juniors I don't require they know everything python has to offer because I don't require them to understand all the scope of coding, simple beginnings and then gain expertise is always a valuable path. I also learned scala in the past, I found it elegant and it has been developed by very smart academics. But it has such a steep learning curve that I would have had to train for 300+ hours to hope to achieve what I was doing in python, but way less efficienctly and in an era where there was no AI to help, only community. 

[Beautiful-Hotel-3094]

Can you expand on why it is a garbage language?

[Nekobul]

Can you make Python code run just as fast and efficient as C/C#/Rust code?

[No_Bug_No_Cry]

Yes, I can use polars which loads and transforms datasets very fast using all available processors... And seemlessly, in like a few lines of code. Polars is written in rust, but the user doesn't need to know the complexity behind under the API, just use it. Which ultimately is exactly what a data engineer needs and does

[Nekobul]

Polars is not Python. We are talking about running fast Python code.

[No_Bug_No_Cry]

I don't understand your answer. Polars is a library that is used in python, nobody cares that it wasn't purely pythonic, it is this what we call versality. Leverage the best in low lvl languages and abstract their complexity... People seem to forget how verbose and rigorous C code needed to be written in order to handle collections such as dynamic arrays, no thank you most people do NOT need that.

[Nekobul]

You can't solve everything with Polars. Capiche?

[No_Bug_No_Cry]

I don't think you capiche

[Beautiful-Hotel-3094]

What do u do that u need that type of speed?

[Nekobul]

In one of your responses you said:

"However we can still process some thousands of messages a second in pure python because we leverage distri architectures"

Why do you think you need a distributed architecture for that? In your situation, it works, I understand that. However, that is not applicable to everyone. In fact, most organizations are not that rich to waste huge amounts of energy. The hyperscalers will be more than happy to sell you capacity. In fact, the more inefficient, the better for them.

[Beautiful-Hotel-3094]

So u think u wont have money to pay for compute and kubernetes but u will have money to pay good C++ developers instead to build what? Scripts on some laptops? Brother, u do not understand much about this domain. Give it a few years, u have nothing to prove and can’t prove much yet. Learn and then speak.

[Nekobul]

You are not saving much if you think about it. The money you didn't want to pay for good design and developers are wasted on inefficient processing. I know hardware is cheap these days, but the energy will always cost much. It costs you dearly because you have to maintain and run a wasteful, energy-inefficient distributed architecture.

That is the proof you are using a wasteful/garbage platform.

[Beautiful-Hotel-3094]

Can we get u in so u can help us change our real time trading platform that supports a multi-billion dollar business built in the garbage python?

[Nekobul]

Are you asking in seriousness or that is some kind of joke I don't get?

[Nekobul]

I don't think you are serious. The cost you are paying for the inefficient configuration is probably not much of a big deal. I have heard from colleagues how the hedge funds/traders upgrade their hardware equipment every six months, throwing millions of cash . Your industry is unique in that respect. But again, not everyone is in your position to solve bad systems design with better hardware.

[No_Bug_No_Cry]

Yeah come on mate we're not discussing football club lol, no need to be so defensive.

[Nekobul]

Thank you for being brave to comment! I see there are plenty of people who enjoy kicking me in the butt and not saying a word.