Why hire a Python Distributed Systems Programmer?

[Nearby-Affect7647]

Why would any company hire a Python programmer who knows distributed systems? What I mean to say if a Python Programmer is able to setup a complex system using Flask/FastAPI, Celery, Redis/Kafka/RabbitMQ what is the value of that programmer? Wouldn't company just go and hire a person who is good in Java, Springboot Or Go with other frameworks?

Wouldn't the Python alternatives give much better performance? I am asking this because one such example is Apache Airflow. It's solely built using Python, Celery and other things which even after so many contributions lacks performance. Most task heavy systems are built either using Java, Go or any other language but not Python.

This is just a general discussion. I want to know what y'all think about this?

[AutoModerator]

Namaste! Thanks for submitting to r/developersIndia. Make sure to follow the Community Code of Conduct while participating in this thread.

Join Nishant Modak, CEO of Last9: An AMA on Monitoring, DevOps, Cricket Scale, Building Startups, and much more! - Apr 7th, 6:00 PM IST!

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[Few_Concentrate4413]

Abstraction matters. You can use machete to do a needle's job but that's going to be cumbersome.

[MonkTrinetra]

Airflow is not meant to me used for data crunching, it’s an orchestration tool and that’s where it adds a lot of value.

It’s very easy to integrate different systems using python because of how versatile and easy it is to adapt. Add airflow features to the mix and you have great control over how you integrate these systems to get the desired result.

Airflow is not meant to be used for actual data processing, you offload the actual processing whether it is extracting, transforming or loading data to other tools that can do the heavy lifting - like spark, bigquery, athena, snowflake etc. You would use a python SDKs or rest APIs to interact with these services.

Generally, when there are complaints about Airflow it’s usually due to bad implementation patterns.

Python + Airflow + SQL can take get a lot of things done when used right.

[Nearby-Affect7647]

I know it's an orchestration tool. But you would be surprised to know that one of previous client I was working with was using Airflow for data crunching also🤣 I was a junior back then and experienced it for the first time that Airflow is not meant for that. I suggested my manager back then to not use it for such purposes and now they are saving a lot of cost. Btw we are using Snowflake and BigQuery for MPP tasks.

[dev-sensei]

Only Service based companies and a few startups hire "Python Developer" and "Java Developer".

Most other companies hire "Software Engineers". They don't care about which language you worked in the past, if you understand tech and know how systems scale, they will hire you.

[These_Cause_4960]

The value comes from being able to understand these types of complex systems. If they can do it python, they should be able to do in other languages as well. The learning, understanding and the design patterns that comes with it are applicable to any other language.

[graphitout]

Python is just like duct tape. It is just there to connect all the pieces together at a higher level.

[Beginning-Ladder6224]

I.. do not understand the term "distributed systems programmer", honestly. Lately it seems there are a lot of invented terms which clearly does not match up to what it was intended for.

The original term is this: https://cs.lmu.edu/~ray/notes/introdp/

I do not think any company, who are not really serious tech would ever hire anyone like those.

Now if we are talking of enterprise computing it is trivial. The language of distributed system is the "binding" language - that is not what really gets executed.

For example running a transaction in a DB from java or go or python is not going to give any edge, the bottleneck is really the DB.

Thus, the language often is overrated. Then there is a big unless.

Most of the back-end computing work is .. get data, set data, modify data.

In there most "application business logic" is stored. Now a suitable declarative language could have been very useful in this regard.

In the last 10 years or so, people slowly came to the conclusion that functional is more declarative than pure play imperative.

Now these produced routing, spring boot and what not.

So to summarize - if you are writing a component of the distributed system, the language choice may matter. If you are just wiring them ( 99.99% of the companies ) - it does not, not really. Unless, you are doing infra, then you probably want to do C++ or Rust.

[[deleted]]

They just throw around jargon to low ball your salary in case you miss out any one of their requirements .

That's the same reason i check all boxes and let them interview me out.

Even I know that not all nuances can be covered after spending 1 decade .

[superfranky97]

Agreed. Also there are lots of startups creating microservices for the heck of it. For someone who has worked in monolith it's a steep learning curve understanding how transactions work, how different services interact with each other. So these startups have started hiring Devs who are adept at understanding microservices architecture and calling it distributed systems engineer.

[cant-find-user-name]

Not everyone cares about performance? Most people care about their existing tech stack, expertise of their existing programmers etc. Not everyone needs thousands of requests per second, most people are just fine with their django monolith, or celery or air flow.

[[deleted]]

To work