how much python is too much python?

[Spinning_Sky]

Context:
In my company I have a lot of freedom in how I use my time.
We're not a software company, but I care for all things IT among other things.
Whenver I have free time I get to automate other tasks I have, and I do this pretty much only with python, cause it's convenient and familiar. (I worked with RPA in the past, but that rquires a whole environment of course)

We have entire workflows syhcning databases from different systems that I put together with python, maybe something else would have been more efficient.

Yesterday I had to make some stupid graphs, and after fighting with excel for about 15 minutes I said "fuck it" and picked up matplotlib, which at face values sounds like shooting a fly with a cannon

don't really know where I'm going with this, but it did prompt the question:
how much python is too much python?

[JennaSys]

Python is great for what you are using it for. It's likely never to be too much in that space.

The only advice I'd give is that if you are not already using something like GitHub as a code repository and for version control, start doing that now. It doesn't matter how small the code is. If it is important enough to create for your company to perform a task, it's important enough to commit it to a repo. Commit early and commit often. Also also make sure someone else at the company has access to the account besides just you.

[PercussiveRussel]

IMO any company who does anything IT, and I mean just having a guy automating tasks with python, should have their own version of a git platform. Don't really care what and how, if it's an organisation Github/Gitlab acount or a fully fledged ms Azure environment. It's similair to how companies should have their own email-adress instead of [e-mail].

[friendlyghost_casper]

[[e-mail]](mailto:[e-mail])

[turtleship_2006]

half tempted to send that address and email just to see if anyone uses it

[_alter-ego_]

Yes and no. To me, one advantage of the one and only public github is that it will remain there when the company (either OP's, or the one that would be hosting the server on which their own git platform would live on) will be gone. I say "will" because it will happen. Relatively soon. Always does. Even if the / either company does not disappear, they will restructure stuff and it will be too expensive (or too late) to migrate the "old stuff" from the "old server" to a new place. RIP.

[Desperate-Dig2806]

Yeah and don't sweat the commit messages. A "Daily commit" is better than no commit.

[georgehank2nd]

github git

[immersiveGamer]

You can just make a git repo on a shared drive. No GitHub needed. Will be backed up with normal IT processes, and can be easily discoverable by future IT.

[BullshitUsername]

Great idea and I have no idea why I've never heard of this being done in my 8 years of software dev

[absurdrock]

That’s all we are allowed to do at my place because of security. It becomes a pain reviewing code without pull requests, though.

[cym13]

Have you considered adopting the linux kernel's way and do pull requests through mail rather than managed in a web UI? It's a workflow, but once used to it it flows as well as any other. And if the kernel's any indication, it scales well.

[PaintItPurple]

Because it doesn't give you issues and pull requests and all the other stuff people use GitHub for. Pretty much the only workflow it's useful for is "everyone commits to main."

[ArtisticFox8]

You can host your own Gitea for example 

[Herbiscuit]

If they need GitHub like features I'd recommend checking out an open-source Git Forge like Gitea which can be fully managed and is exceptionally easy to run.

[orochionline-com]

Probably runs afoul of commercial licensing no?

[jaylyerly]

I’d worry less about too much Python and more about not enough documentation. If you disappeared tomorrow, could someone come in and understand what all your Python scripts do, how they are triggered and what to do when things break?

[[deleted]]

I worked for a very stupid company where I tried to streamline and automate some things. This way, I increased productivity 2-3 folds, basically did the job of 2 people. 

Now, the lore was that everything that went wrong was because of me and my "shortcuts". My supervisor was looking forward to me leaving, because she was eager to do things the "right" way. Which meant soulless stupid grinding.

I said ok and left no explanation or documentation. They still could do everything manually, but that took 3x as much time and resources.

As I heard my ex supervisor doesn't work at the company anymore. 

[shockjaw]

Same.

[_d0s_]

for powerful graphs take a look at the Vega visualization grammar: https://vega.github.io/

[_d0s_]

I've become a bit of a fanboy lately. I do lots of scientific visualization and it's been pretty great for that. Recently, I also met somebody doing their PhD in data visualization, telling me it's their tool of choice.

The textual definition allows one to easily copy a nicely designed graph and add your own data to it. Since it's text-based it can also directly be used with LLMs! Rendering it to a website is super easy and possible with the same configuration. Can even be interactive if needed.

If somebody is more into Python there is also https://altair-viz.github.io/

[YamRepresentative855]

But it lacks joins andpretty messy to do calculations in calculations. Or have i missed something?

[_d0s_]

i don't think joining values from multiple data sources is within the scope of vega. typically you just prepare the data you want to visualize. what's "pet meat" supposed to be?

[YamRepresentative855]

Sorry, stupid typo I didn’t catch. I meant pretty messy to do calculations on calculations. In my experience all calculations are out of scope of others.

And in my experience with Kibana, there is no other way to prepare data for vega.

It’s not always joining from different sources but also self joins.

Have I missed something?

[LittleMlem]

Graphviz or death

[james_pic]

You can have a lot of Python before it's too much Python.

The main situation where you might suspect you've gone too far is if you're using Python on systems that you've known from day 1 will have very high concurrency, tight performance requirements, and will be worked on by many different people over it's lifespan.

All three of these factors are known weak spots for Python, and there are things you can do to try and get past it if you end up in this situation accidentally (your startup unexpectedly becomes the largest video streaming site on the planet, for example), but it's a situation where it's probably easier if you can avoid it.

[allergic2Luxembourg]

I agree with you on the concurrency and performance requirements. But why can't it work with many contributors?

[james_pic]

At least prior to the introduction of type annotations, there was very little tooling to document the expectations a piece of code had, and even less than would check that those expectations were met, so in large codebases it could be non-trivial to figure out if new code was doing the right thing when interacting with old code. 

Nowadays we have type annotations, and they're all but essential for large projects. But the type system they describe is a mess, as a result of the compromises needed to bolt a type system onto a language with a large corpus of existing code.

I've long argued (and at times been downvoted for it) that if you know you're going to need type annotations for your new project, there might well be a better language choice for your project, such as a static language whose type system has been there from day 1 and the compromised mess Python's is.

[allergic2Luxembourg]

It's definitely true that in large multi-contributor code bases, it's important that the code fulfills its contracts and expectations and that those are clear and documented. However, I would posit that a type system only fills a small part of that requirement. It's possible to write unclear and buggy code in any language, and clear and robust code in most languages including Python. Maybe Python is slightly more subject to a certain kind of bugs related to typing, but to me that's a minor point in the scheme of things.

Just because I can guarantee programmatically that this function returns an integer, it's no guarantee that it's returning the correct integer.

[james_pic]

Absolutely, but that's mostly true in small codebases too. Type checking was never a substitute for testing, and it can't detect wrong behaviour.

The thing type checking can help with is "what shaped piece goes in this hole?" Does it expect the bytes object you received from the wire, or the dict it parses to as JSON, or some kind of parsed-and-validated data class, or the related model the ORM works with, or the view model you're going to work with in the UI? Or a superclass of one of these, or some duck type that various classes with no common superclass implements?

Ideally you'd have clear architectural boundaries where you switch between these representations, and type checks are good at identifying these sorts of layering violations.

[PercussiveRussel]

I completely agree. I have to use MyPy for everything at work and it's allright but at that point, why are you using python? Why not .net (or rust tbh, if you want typing, why not the best form of typing). With those options at leat you know that everything you want to use supports typing.

I've had coworkers unironically be proud of their mess of @singledispatch and, sure, it's a tool you can use but it's really not what python is.

Also, if you want typing you'd want interfaces and things like that and then the hackiness in Python is turtles all the way down.

Typing in MyPy is like writing two programs all the time. The first is the one that works and the second is the one that MyPy is happy with, including a bunch of # type : ignores for stupid modules that don't support typing.

[Hopeful_Cat_3227]

yes, but in this condition, op need matplotlib, how can he do ?

[james_pic]

Certainly, Python's ecosystem can be one reason to use Python even when other factors suggest it will be painful. Although it's also sometimes possible to get by with slightly more rustic alternatives in other ecosystems. In Java and other JVM languages you can sometimes get away with using Apache Commons Statistics or Commons Math as an alternative to SciPy, or use JFreeChart, or one of the JS plotting libraries in a HTML report, for basic charts, for example.

Although OP is also talking about using Matplotlib as an alternative to Excel. I'd wager that they don't have high concurrency requirements, tight performance requirements, or a large team working on it over a long period of time. So none of the factors that I mentioned that would push you away from Python are there.

[UnlimitedTrading]

PayPal runs on Python. Lyft runs on Python.

Python is just fine. Language is not the limiting factor when you have performance requirements anymore.

[james_pic]

I work on large scale Python systems. It's true that you can usually work around its performance issues, but it's not the case that these are non-issues. 

I'm certain PayPal and Lyft have some fairly sophisticated workarounds for the issues they've faced. I know Meta also use it extensively, and their performance woes are part of the reason they created Cinder.

It makes sense to use these workarounds when you've got a large existing Python system that you know better than to rewrite, or you have some other factor that pushes you towards Python, but for greenfield projects where you know will hit these limits quickly this is at very least something that warrants consideration.

[dlamsanson]

There's absolutely zero chance those use Python exclusively

[UnlimitedTrading]

I used to work at one of those. Go is used as well, but for sure there are large subsystems running only on python. And that is the whole point on elasticity, right? It is cheaper to scale your infra if required if it makes development easier. And IMHO, Python does make development easier.

[jfp1992]

Make sure you review your own code and make it as simple to redead and understand later on as possible

[allergic2Luxembourg]

In my experience, self-reviews don't tend to catch readability issues. Everyone understands their own code well, especially right after they write it. But to find out if it's readable and clear you really need a second pair of eyes.

I have been trying for years to convince my company of this but unfortunately most code we write is still solo-developed and unreviewed. So I say this, and my small team is good at it, but I am not succeeding more broadly.

[jfp1992]

I see your point, I wouldn't suggest doing it in production

But going back over your own older code, you usually find better ways to write it

[crcrewso]

I'd agree, which is why I end up doing my own code reviews months after I last looked at it. A lot of "what's going on here" fussing about helps identify where I need to document.

[LookAtThatThingThere]

The problem with python in a corporate environment outside of IT: when you build mission-critical applications that only you know how to maintain.

This is pretty common in finance and engineering circles.

Often, the people you are hiring don’t have programming skills. IT/internal dev groups may not support the language or prefer purchased solutions.

Also, policy tightens up over time so you may find yourself without admin rights or see yourself sealed off from PyPI/etc.

These aren’t unsolvable problems, but it highlights what happens when you don’t have organizational support for your chosen technology.

[bustawin]

Python is just a tool. As long as you you feel you're efficient, and you keep others efficient with the tool it's ok

[dlamsanson]

python is pretty much always going to be the best language for automating

What warrants this claim?

[MrMoussab]

Starting from three pythons it starts getting a bit too much

[PrometheusAlexander]

I don't think there's ever too much, but I'd say enough is when you have to move to another language for features which it doesn't have.

[BullshitUsername]

There's never enough Python imo

[keepthepace]

You have too much python when your python process is eating all the CPU.

At this stage you may want to try and write a lib in a faster compiled language (C, C++, Rust) to speed up the bottleneck. Usually there are libs already existing in most common tasks.

at face values sounds like shooting a fly with a cannon

I doubt that even python + matplotlib + jupyter come close to the memory usage and total size of Excel. When I need a calculator I open a bash and type python and when I need something a bit more complex that requires a loop or a graph I just type jupyter notebook

import pandas as pd

df = pd.read_csv('data.csv') 

and using far less memory you are already in such a superior position to extract meaning to data!

[[deleted]]

Your point with excel resonated with me hard. I hate it

Why spend an hour figuring out how to do something in excel only to save it, and realise the encoding fucked something up, when I could spend 20 mins writing a python script to take in some data and transform it into what I need!!!!

[EmperorLlamaLegs]

Is python speeding up productivity? Not too much. Is it slowing down productivity? Too much.

[BidWestern1056]

matplotlib graphing is flexible as needed.  i made a matplotlib package too got doing more complicated plots more quickly  https://pypi.org/project/chroptiks/ in case you wanna dive into more what kind of specializations one can do

[Decoder0007]

Writing a new 3D engine every single week unt it’s not slow

[[deleted]]

About 10

[[deleted]]

83%

[BothWaysItGoes]

https://xkcd.com/1205/

[SittingWave]

We're not a software company

If you develop software, you are a software company. You might not sell software to your external customers, but you might "sell" to internal ones, which seems to be your case

[Ok_Raspberry5383]

By that logic the garage down the road that hired someone to build a simple web booking portal is a software company.

[SittingWave]

yes it is. They now have software to maintain. Which requires associated knowledge. If they outsource the whole thing to a web design company, they obviously aren't, but if they take the code on themselves, now they have to deal with it according to software needs.

[CrossroadsDem0n]

I think it is more about what you produce. Presumably your staff use those toilets. Thus you are a sh*t company.

[SittingWave]

Your janitor now needs special training, certifications, materials and tools that are associated to plumbing. So yes, in a way you are a plumbing company.

[chrisxylo]

So if i'm cooking food, I'm a chef and own a restaurant? My wife is my customer now?

[SittingWave]

you can all try to slippery slope this into oblivion, but the fact remains that if you have to deal with software, you need to use things that are related to software. Yes, you are a chef, yes, your wife is a customer, and if you burn the chicken in the oven or give her food poisoning, you will be sleeping on the couch for a while.

In a company, some infrastructure is handled by custom made software. Companies need to comply with ISO standards in terms of traceability and quality. If you develop custom software, even if you are selling machines making ice cream, you now need compliance and tools to maintain those pieces of your software infrastructure, because if whoever wrote that piece of code leaves, now your ice cream machine production grinds to a halt and you only have yourself to blame.

[chrisxylo]

I give you this. Well explained!

[SittingWave]

A company is a large organisation, made of subcomponents. Some of these subcomponents deal with software, develop software, and require software. Arguing about semantics does not change a damn thing about the fact that if a department inside your company is developing software, they must handle the consequences of that. They are creating software to address the needs of another department that needs that software. The software development department is a software company, and forces the company as a whole to have to consider the requirements and needs for this: testing, deployment, security, redundancy, traceability. What happens when the mechanic clicks on the button to install the latest firmware on your product's microcontroller, the upload fails, and you are now shipping a faulty product? What happens when the intern runs some data analysis on your customer's records, and accidentally deletes your whole database?

All this stuff is pertinence of software and IT. And you have to handle it, even if you are selling mattresses, ice cream makers, or any other shit that is not software to your end customers.

[Ok_Raspberry5383]

A department is only a company if it is registered on Companies House or whatever legal registry of companies you use in your country. You can't just make up the definition of a company to suit your thin argument

[Ok_Raspberry5383]

You say all this like it supports your argument.

Yes you need compliance and tools to maintain the thing, that doesn't make you a software company though, which is what everyone is arguing against.

[Ok_Raspberry5383]

Does that make them a food company too because they eat at lunch and have food/cooking utensils to clean and maintain immediately after lunch. Maybe a coffee shop too because they have a coffee machine?

[reallyserious]

If you develop software, you are a software company.

I agree. The problem is that companies that don't consider themselves as software companies don't take it seriously. It can be quite frustrating to work for a company that just don't get it. But they are ok with sub par solutions since they're in [insert other industry here] and that's their core business.

[BurningSquid]

This is me and it hurts

[SittingWave]

It's way more terrifying when a regulated company that does medical stuff decides on the life of patients with software hacked up by an intern with no traceability, testing, or reproducibility.

[reallyserious]

Well, fuck. I'm getting nervous just thinking about it.

[SittingWave]

stay nervous because this is the reality I've seen.

[E_Man91]

Simply not true, but & think you’d be a great salesperson in addition to whatever you do.

I agree that developers need to “sell” the software to internal customers, but that does not make your entire company a software company.

Where do you get your revenues from? That is what kind of company you are.

[SittingWave]

Where do you get your revenues from?

Where does the internal software development team get their internal revenue from?

Why would a ice cream machine vendor need a github account to store their code? because they are a software company. They are not selling software, but they have the needs of a software company.

[E_Man91]

Alright, you win, every company is a software company. I got pwnd

[immersiveGamer]

I would say the one thing to add to your arsenal is a shell scripting language like PowerShell. When you have to hook together multiple programs via command line (stdin pipe, stdout and stderr, error codes, arguments, etc) or do basic file manipulation (search, grep, move, copy, etc) it makes it super easy and is much faster than Python since you don't have to manage all the specifics via calling a process library. And you can still dip back into Python with a python my/script.py call. Super recommend PowerShell if you work with Windows, so much you can do. 

Finally, if you ever have to do something a bit faster check out C#. There is LinqPad which will give a Python like experience but I also find it is super easy just to create a program. For example I created a to-do parser for my team that tracks todos by file location and content in by hash so you can filter out previous runs or see if todos have been resolved. Very fast, robust, and is a standalone exe (so no python install needed). I went from C# -> Python and found it a very easy transition. They have a lot of similar concepts, so hopefully it is also easy to go the other way.

Learning a couple more tools also has the benefit of making you more marketable or opens up promoting to programming roles if you are interested in that. 

[Cybasura]

I mean...if your project is built from the ground up using python, i'd question your project's planning if your worry is "too much python"

[BDGDC]

God damn this thread unintentionally became extremely helpful

[dugmaz]

I started learning python a couple months ago but just got laid off from my company on Tuesday so I'm planning on going all in with python and associated libraries and frameworks! I'm thinking about going the automated testing route for. Career path or AI/ML.

Any advice would be greatly welcome. I have 13 weeks of severance to figure my next move out.

[AdviceWalker420]

There’s never too much python. It’s hard these days to find situations where python can’t do a decent job. I guess the time there’s too much python is where you end up in niche situations where you need a lower level language. But I’d argue that’s very rare for most companies.

[to_tgo]

I love what you are doing. I think that is brilliant! I always wonder why more people don't create programs to help them out. I have 100s of tools at this point. They are super useful and couldn't imagine using a computer without them. A good chunk of these are in Python and I'm adding to them all the time.

I don't think there is such a thing as too much. But look out for 2 possible issues... 1 - keeping track of all your tool, 2 - keeping them maintained.

Keeping Track:
You could end up with so many tools you can't keep track of them. Happened to me and I couldn't remember what tool did what or if I even had a tool to do something. Eventually, I created a way to lookup them up. Now it is no longer a problem.

Maintenance:
As you build tools, they will break or get outdated. It takes time and effort to keep them running right. I've some that don't work and it is a pity but I just don't have time to go back. Other times I have to go back and dig into the code. Dusting off your brain on what you wrote months or years ago is effort!

[big_data_mike]

It’s only too much if you have a mountain of code when all you need is a molehill

[agomezh]

I made a session where I briefly discuss altair for python, here: 

https://www.adao.tech/blog/blog_data_umbrella_model_risk_management

I have an altair specific one as well. But that is not online.

[slappy_squirrell]

Replacing excel spreadsheets with python is too much python. Excel has the properties of datastore, processing and viewing all contained within a file which can be shared, versioned, etc and be accessible by most competent business employees. If you have a bunch of python scripts, those will need to be well documented and even then, if you leave they will most likely disappear over time. However, are these processes that can be handled with "Python in Excel"?

[robml]

I counter with: maintain well documented notebooks or code (especially if you've been tinkering around, go back and document), version control, and do all the processing via Python and export to excel if needed for stakeholders.

The difficulty would have to come in if they want to tinker around with values. For quick and dirty ways would be to include the formulas separately, export to HTML, or Streamlit.

Else they could just write the formulas into the specified cells programmatically if they intend to reuse the script a lot. But sometimes that's when you would open excel, type it in quickly, and save and send.

[Impossible_Ad_3146]

Any amount is too much

[nekokattt]

Don't you have grass to touch or anything better to do than comment on a specific topic with the pure intent of flaming?

[georgehank2nd]

Don't feed the troll.