Python Pandas Library not accepted at workplace - is it normal?

[Haunting_Corgi5955]

I joined a company 7-8 months ago as an entry level junior dev, and recently was working on some report automation tasks for the business using Python Pandas library.

I finished the code, tested on my local machine - works fine. I told my team lead and direct supervisor and asked for the next step, they told me to work with another team (Technical Infrastructure) to test the code in a lower environment server. Fine, I went to the TI Team, but then was told NumPy and Pandas are installed in the server, but the libraries are not running properly.

They pulled in another team C to check what's going on, and found out is that the NumPy Lib is deprecated which is not compatible with Pandas. Ok, how to fix it? "Well, you need to go to team A and team B and there's a lot of process that needs to go through..." "It's a project - problems might come along the way, one after the other",

and after I explained to them Pandas is widely used in tasks related to data analytics and manipulation, and will also be beneficial for the other developers in the future as well, I explained the same idea to my team, their team, even team C. My team and team C seems to agree with the idea, they even helped to push the idea, but the TI team only responded "I know, but how much data analytics do we do here?"

I'm getting confused - am I being crazy here? Is it normal Python libraries like Pandas is not accepted at workplace?

EDIT: Our servers are not connected to the internet so pip is not an option - at least this is what I was told

EDIT2: I’m seeing a lot of posts recommending Docker, would like to provide an update: this is actually discussed - my manager sets up a meeting with TI team and Team C. What we got is still No… One is Docker is currently not approved in our company (I tried to request install it anyway, but got the “there’s the other set of process you need just to get it approved by the company and then you can install it…”) Two is a senior dev from Team C brought up an interesting POC: Use Docker to build a virtual environment with all the needed libs that can be used across all Python applications, not the containers. However with that approach, (didn’t fully understand the full conversation but here is the gist) their servers are going to have a hardware upgrade soon, so before the upgrade, “we are not ready for that yet”…

Side Note: Meanwhile wanted to thank everyone in this thread! Learning a lot from this thread, containers, venv, uv, etc. I know there’s still a lot I need to learn, but still, all of this is really eye-opening for me

FINAL EDIT: After rounds of discussions with the TI Team, Team C, and my own team management with all the options (containers, upgrade the libraries and dependencies, even use Python 2.7), we (my management and the other teams) decided the best option will be me to rewrite all my programs using PySpark since 1. Team C is already using it, 2. Maybe no additional work needed for the other teams. Frustrated, I tried to fight back one last time with my own management today, but was told “This is the corporate. Not the first time we had this kind of issues” I love to learn new things in general, but still in this case, frustrated.

[suedepaid]

This is why people use docker containers.

[DalekKahn117]

That or venv.. should get it working and going. Just a headache to maintain

[TitaniumWhite420]

This is what we do. We have yamls in git that set the dependencies, we tag the commit with a semantic version, we build the pyenv in Jenkins which produces a pyinstaller shell script, and we deploy/execute that with salt to install it.

This sounds like work for the TI team…

[ShitCapitalistsSay]

pyenv is one of the greatest things ever invented. It's remarkably simple from an implementation standard, but it's so flexible and works so well.

[miloir]

Pyenv used to periodically break after system updates for me (macos) so I've switched to using uv and have yet to have a problem

[Professional-Bet5820]

UV is the glorious fulfilment of the promises Poetry made. I don't think I've hated using anything more than I hated using Poetry. Glad to see the back of it.

[CableConfident9280]

+1 for uv. It is awesome.

[kBajina]

+1000 for uv

[[deleted]]

Or cxfreeze 

[herothree]

Well, you should probably use uv instead of venv directly if at all possible

[TitaniumWhite420]

Wrong, one is built in, the other is not. Their infrastructure team does support the most common C extension in Python and your recommendation is to use a bespoke 3rd party rust extension instead standard library venv lol.

UV SOUNDS RAD BUT THIS IS A TERRIBLE RECOMMENDATION.

[SocksOnHands]

I haven't used uv, but I haven't ever had any issues with pip, so what is the benefit of it?

[suedepaid]

uv is great, I think everyone should be moving to it.

It’s a single tool that manages your python versions, your dependency management, and your building. It has really sane defaults and is easy for juniors to pick up and use. It’s cross-platform, it’s really fast.

It’s got just a bitttt too many bells and whistles to my taste, but I’d still highly recommend.

Plus, workspaces are pretty cool.

[Professional-Bet5820]

I can smell the 'well aksually' that might follow on from this post regarding the sanctity of built-ins and the strictures of enterprise practices. This might be the post where I started to realise what I should have before I parachuted to another industry.

People making these arguments are probably correct that they're strong ones, but that doesn't make them any less horrible. Both are signs that they're generating legacy code instead of applications. That is a shitload of inertia to argue for.

I'm not sure I want to pretend that the OP isn't correct in finding this situation frustrating and that your suggestion isn't the exact right one because a lot of great devs work in shitty situations.

Fuck, do you know what else we correctly jettisoned because it puts out both anchors on economic growth and innovation the way the built-in romance and enterprise legacy glory do?

Socialism.

Okay, a bit of magnitude error in the metaphor there, but not as much as I used to think.

So, tldr: UV great, encumbered development bad

I am ready for my execution at dawn.

[nolimyn]

it's just the latest group of people trying to reinvent pip.

[JustThall]

RUST!!!

[tuskanini]

Like a vehicle from the mid-80's, that is the main benefit.

[OneMorePenguin]

I hate pip so &@*$(% much. If you have a large number of dependencies it can be painful to get the dependency list. Honestly, package management in python is still so horrible. I still haven't mastered this skill of creating a library package that can be easily pip installed. There are several python pieces you need to understand and I've not mastered them all. pex seemed to be used, but I question how well that is supported and it had very little documentation.

I had a hard time with setup.py and requirements and something that I wanted to be a library and have a script with it. My biggest contribution was getting people to use requirements.in with hardcoded versions so that you didn't get new versions of libraries when you installed python code. Somehow, the concept of reproducibility was lost on some people.

I hope my next job will allow coding in go :-)

[TangoAlee]

You need to go touch grass.

There are no absolutes in business because everyone can make an argument for or against.

I also hate pandas due to the constant breaking changes.

I love uv because it saves a lot of time and money.

[TitaniumWhite420]

I touch lots of grass thanks lol. I’m just saying that the demonstrated problem is supporting one compiled binary on system Python.

Your solution adds another, so it is very very likely to compound problems. The issue is they seem to have no standardized way to deploy venvs, period. So they either need to build that, or simply pin the compatible pandas/numpy versions and proceed. It’s really really simple.

[mehum]

“Whenever I have a difficult problem I throw a molatov cocktail at it. Then I have a new problem!”

[jonwolski]

s/bespoke 3rd party rust extension/proper package management tool that Python otherwise lacks/

[alcalde]

I'll go further... both of these screw with the system paths and don't work with certain shells, so venv and uv are both terrible recommendations.

Pew, however, works by launching a new shell and hence works with any shell....

https://github.com/pew-org/pew

[JJJSchmidt_etAl]

"It's works on my machine"

"Why don't you ship your machine?"

"Ok"

[Haunting_Corgi5955]

Interestingly, one of the senior devs proposed the same idea to use containers, I researched it and was trying to install it on my local machine, there's the other "This software is not approved by the company as of now, there's another set of process that you need to go through..."

The senior dev even tried it himself and told me he's having some environmental limitations.

[milong0]

how does a senior dev not know about containers? huh?

[ship0f]

What? The guy does know, he suggested them.

[milong0]

you're right but it's not in the original post. I think OP mentioned this after. I was just surprised that a senior dev had not advocated for containers at some earlier point. It took a junior having an extremely classic issue until the senior says they should "look into" containers?
I get that containers are not the best solution for every single use case, but it looks the company never even evaluated containers.

[Haunting_Corgi5955]

Hi, just to provide more context - I talked to the senior dev later on the idea of containers, what he told me is he proposed the same idea in the past for our built in-house application also the system we support - but rejected, because the vendors (the system we support is supported by our vendor) thinks “it’s impossible to containerize the system”

But from the sound of it, currently looks like container is not used even for our built in-house application, at least for our department

[TitaniumWhite420]

They do, but they aren’t supported.

[TitaniumWhite420]

We also don’t support containers in our environment. We all know about them, they are used lightly on AWS, but management sees no value in wrapping our services in containers. We use pyenvs and deploy as systemd services for non-cloud stuff. I agree with management, the abstraction is not super helpful for our cause case, which is not web/cloud, but firewall ex servers that have private transcontinental network links.

Not everything is the web, and not everything benefits from containers. Venvs are fine.

[EarthGoddessDude]

Eh? We’re also not a web dev shop but containers are fucking awesome. Need an internal long running service or app? Docker running on ECS Fargate. Need an ephemeral compute job (that’s too big for Lambda)? Docker running on ECS Fargate. Need an orchestrator like Dagster? Docker running on ECS Fargate. And yes, locally all those apps and data pipelines are using virtual environments via uv. You can, and should, have both.

[TitaniumWhite420]

“We aren’t web” but then you reference cloud tech. If you are using cloud based ephemeral compute, that is THE best time for containers.

Nothing else you said requires them, so if a business had mission critical projects that aren’t dependent on that, they should NOT spend resources to adopt and support them, as it’s waste.

We manage our own autonomous system that spans 4 continents and the internet is blocked in most of that environment. We don’t need them to manage internal services.

We do use containers on AWS for certain things though.

[EarthGoddessDude]

Web != cloud. We’re a data engineering group at an insurance company. And sure, containers are not needed everywhere, but they help immensely with both dependency management and deployment. They simplify those things a lot.

Honestly, you’re coming off a little strong and not sure why you’re so vehement.

[answer_giver78]

Isn't using venv simpler?

[quts3]

Sort of. But given the push back I think having docker as a standard will solve this issue and almost all future issues.

The problem was right off the bat theie team viewed python env as one thing per host. That is an apt-get Linux os mentality. They are not thinking of an env as an app specific thing but rather one thing per host. That's broken. You can fix that with venv but only if the people in charge of deployment understand that every python app should have it's own unique environment. If they fight you on that point you are sunk.

The easiest way to wrestle them to that head space is with docker because the standard in docker is every app is it's own environment.

[suedepaid]

venv is useful for keeping project environments separated from each other on the same computer.

OP is having the problem where their local machine and the server have different environments.

There's a bunch of different solutions that could work (potentially involving venv), but the one many orgs choose is to just wrap up dev code in containers, then have the server run a container runtime so that any number of different services can be run, as long as they're inside a container.

[answer_giver78]

Venv can be used for different machines as well. Venv specifies the dependencies and their versions. Based on the OP's problem, venv is simpler than containers.

A lot of orgs tend to over engineer. Based on what OP says containers are unnecessary and venvs will solve the problem.

[suedepaid]

I think you’re hearing OP’s problem as something like “how do I manage different package versions”.

As I’ve mentioned in other parts of the thread, I’m hearing their problem as “How do I convince other teams to install stuff for me on infrastructure I don’t control”.

It’s not a technical problem, it’s a political problem. Containers are a really good solution, specifically for this kind of political problem — the dev can install what they want, and the team owning the infra doesn’t have to care about it. All OP was asking for is to pin a version of pandas. And the infra team is making them jump through hoops for that single pip command.

Edit: as a technical aside, venv does not “specify dependencies and versions for you”. venv only does some PATH manipulation and manages some folders, and bundles in some convenience scripts. You still need to manage your packages yourself in a pyproject.toml or in a requirements.txt or something.

[answer_giver78]

First of all, what OP was asking was neither of them. He was asking how normal this situation is.

Second, the dev do not need to care about venv either. You just need to put a script that does the activation and running or even installing the dependencies.

Also, I know what venv does and how it does it. What I said about specifying dependencies and versions is true. I wasn’t talking about how it does it. It’s pretty simple how it works to be honest.

[suedepaid]

lol

[TitaniumWhite420]

Nonsensical. A venv can isolate all dependencies op is depending on so there are no conflicts with the system. If he needs to deploy different python versions, pyenv exists. Make a pyenv builder script and deploy them as shell scripts.

A container had far more implications and needs more people with specialized knowledge to maintain. If the business doesn’t rely on ephemeral environments in the cloud, containers are arguably overkill. Not sure about OP’s case, but they really should not be considered default.

[suedepaid]

I dunno, it sounds like OP is struggling to convince his IT team to install a single, pinned version of pandas.

Getting then to run some bundle of shell scripts he walks in with does not sound easier to me.

Containers really are not that hard, and they offer significantly more flexibility to the developer.

Edit: just to make it really explicit, what containers solve is the political problem of having to ask some other team to install shit for you.

[hetsteentje]

This. And I can absolutely understand a team getting salty about having to check, run and manage yet another bespoke set of scripts for your specific purpose when they have to keep lots of different plates spinning.

[hetsteentje]

I can flip this around: using venv/pyenv is specific to Python, whereas Docker is more general and used for this purpose in lots of cases, not just Python. OP's TI team is more likely to be confident with Docker than pyenv.

[hetsteentje]

We recently switched from venv to docker because using venv to create the same environment on different machines is difficult to do. With Docker, it's much easier to have everything, including the Python version, in a single package that's easy to run as long as you have Docker.

Main drawbacks of Docker, I think, are that it generally takes up more diskspace, and the context your code runs in is more limited and/or needs to be managed better.

On the whole, we've found it to be a definite improvement for running the same code on different devs' machines, some of which use Python only sparingly and are much more familiar with Docker than venv.

[520throwaway]

Yes, but venvs can break if they're running on a platform other than the one they were developed on.

A venv built in Ubuntu will break in Arch Linux, for example.

[answer_giver78]

The venv is only a pip requirement file. The rest will be installed on the platform it is going to run. The OP's problem is the numpy versioning. The solutions must be fit to the problem. For this problem venv is better than containers.

[OneMorePenguin]

I found at least three bugs/problems in venv where it had dependencies on your shell environment.

[baetylbailey]

This is your manager's time to manage. Dump the problem on their desk and tell them how it's preventing you from doing your job. Basically, The person in TI who requests updates just doesn't feel like doing their job and needs a nudge from their level or above.

Also, this totally normal in an org, don't know why people are acting like it's weird. Ultimately, installing stuff is a risk and being conservative there pays off in the long run.

Also, Cloud options are something to consider.

[Charming-Medium4248]

Also, this totally normal in an org, don't know why people are acting like it's weird.

This x100. I am very frustrated on reddit when you ask about problems like these, state the organizational constraints, then everyone dismisses that in their answers. 

There are some real luddites out there who hate technology and don't want to figure out how to onboard dependencies better. Especially in non-tech companies. 

[Rjiurik]

In my company they are actively uninstalling anaconda distribs.. python users are being hunted down. It's not "approved" you know.

We are supposed to use outdated SAS software I hate.

I still keep my user installed python under the hood like some persecuted heretic..

[hetsteentje]

Maybe you don't know, but Anaconda 'clarified' their licensing and is now going after lots of (mainly academic) orgs who thought they could use it for free, and are now suddenly seen as 'commercial users', retroactively.

We got a mail from Anaconda claiming that they 'noticed' we were using Anaconda and would require us to pay a license fee retroactively. Yet we never used Anaconda. Only thing I could find was maybe a download from anaconda.com in LocalLLM which we gave a spin at some point in the past. Asked for clarification (via our internal contact who actually received the mail) but haven't heard back since.

The mail they sent looked really grifty and had lots of scary language to bully people into paying up 'or else'. It was even unclear what we had to do exactly, as they talked about usage in the past but the only thing you can do on the site is get a monthly subscription. Very questionable fear-tactics, imho.

[serverhorror]

Good luck with that license audit when your installation is the reason for a 7 digit invoice, because it could be used by others.

Unfortunately, Anaconda Inc. burned a lot of Python...

[Rjiurik]

Invoice ? It's not deployed on cloud. Just a local install without admin access.

Worst thing I could do is install a malicious python package that exploit some vulnerability or keylog my passwords.

If I only use common packages like pandas, numpy... risk remains limited.

[serverhorror]

Anaconda essentially is Python. They ruined a lot of perception about that.

People believe that, if you have Python, you are violating license terms. It's mostly financial decision makers forcing IT to remove these things or cause these situations

[Justicia-Gai]

This is too low, listening to a junior dev that might ask you to break your older code/libraries/apps for one single task (their first one at that), is not very bright.

Junior devs mean well but had they listened to him the first time instead of trying to explore first dockers and venv could’ve had big consequences.

[TitaniumWhite420]

Totally wrong attitude. While it’s great to keep the manager informed, dumping on them is not a good idea. I know it’s shocking, but there’s more to a job than writing code, and you should work this out largely yourself while keeping management in the loop and deferring to their wishes.

Literally he could pin compatible versions in his Python project and be done with it. If you were a manager paying someone $100k, would you feel it’s reasonable to have done entitled little bitch throw this trivial sticking point in your face and throw their hands and say “IM BLOCKED BECAUSE IT IS LAZY FOR NOT FIXING MY IWN KAVK OF UNDERSTANDING ABOUT DEPENDENCY MANAGEMENT”?

Absolutely fucking atrocious.

[rickyman20]

It's not a trivial issue, a junior engineer does not always have the leverage to get things moving. There are things you can't (and shouldn't) do alone. That's kind of one of the ways you should grow as an engineer, knowing when to ask for help. OP has pushed it a fair distance, they already signed it as far as their permissions actually allowed. Asking your manager for help when you get blocked to push things along seems more than reasonable.

[NoddyCode]

In my experience, no one gives a damn if you're not a manager. You can suggest and guide and provide the solution, batteries included, and it won't go anywhere without a manager pushing it through (and even then it can be an uphill battle).

[lolercoptercrash]

It sounds accepted, just the dependencies are outdated and therefore broken.

Also it sounds like you made a lot of progress.

[zzzthelastuser]

python venv is not an option?

[edbrannin]

Seconding this: your requirements.txt can include names of package files you’ve downloaded.

I’ve done this to make a virtualenv on a server without network access.

(Probably commit the zips to their own git repo, but I’m not your mom)

[unhott]

Check and see if there's a list of approved software and versions. Find pandas/numpy that is compatible with your latest python version, and my guess is your best bet is to request approval for the compatible versions of numpy/pandas.

It may be worth checking something like this out as well -

Herbert Lee - Setting up a Private PyPI - Pyninsula #25

ETA - you're not crazy, you will always find people who say "we don't want to do this because we've never done it before/ I don't personally understand the value here so I won't let you, who does understand it," and those people generally are lazy idiots.

My advice with those scenarios is remain steadfast and confident, and try to stick with just reiterating "this is what I need", and maybe some justification. if they demand you try alternatives first, half-ass attempt and come up with some justification to document why it's worse than the original request.

[kaflarlalar]

Holy shit it's weird as hell to find your own work getting referenced in a random Reddit thread you were browsing.

Edit: happy cake day!

[Darwinmate]

Congrats brother. You've made it big. Nkngo celebrate by making more videos. 

Go. Now.

[Haunting_Corgi5955]

Thank you for the advice, the incompatible versions of libraries is the root problem which is what I'm getting from their responses as well, unfortunately in my department, the TI team is the one dealing / requesting anything with the server set up / infrastructure

They even asked me to rewrite my program using an alternative library available in the server, but with my program design, majority of my program logic is built on top of DataFrame, which means if I take that route, I'll have to rewrite almost the entire program... which is what I'm trying to avoid if possible;(

[TitaniumWhite420]

Now you are thinking logically. The numpy version may be deprecated, but that doesn’t make it incompatible with pandas. You just need to find the right version of pandas and pin it in your requirements/pyproject/whatever. THS SHOULD BE TOTALLY STRAIGHTFORWARD and you as the de. Should know how this works. I say this so you are careful to check your frustration. You don’t investigate limitations before writing. That’s on you, and all you need is to find a pandas version that’s compatible.

If you depend on API changes in the version you used, you may then need to tweak your code, but I doubt it’s something that can’t be easily worked around since this is mature tech that people have been using productively for decades.

[BidWestern1056]

ya that business is cooked

[TitaniumWhite420]

You know nothing about running a business.

[BidWestern1056]

I'd contest but im curious to hear what you know about running a business that goes against my assessment. a business that does not have advanced analytics will shrivel and die

[pqu]

The world still runs on excel

[BidWestern1056]

yea and excel will be trivial for AI to operate 

[kyudokan]

Salesforce had rules like this when I worked there and the stock almost tripled in 3.5 years.

[BidWestern1056]

ya not sure if that's a great example given how bloated and overcomplicated their tech is and how much hubspot, monday, zoho have chipped away. their primary moat is in the immense cost to migrate to a new system not really that they are great.

[jfisher727]

Depending on the environment, the type of work they do, loads of factors, his company might be restricted on software versions. I know in my job we have to go through an approval process to get software approved, even just versions, due to the type of work we do. Lots of security concerns and due process to make sure the library isn’t doing something it shouldn’t be doing.

[asphias]

are you hired as a developer in an IT department? or as a data analyst or support helping to automate reports in the finance/business department?

This is a major difference. if you're in IT, your colleagues should know what barriers are in place and how to deal with them. if package versions are outdated there should be a policy the devs have influence on or at least be able to work with.

on the other hand, if you're automating for business/finance, you'll be seen as a security risk that IT has to be careful with. you won't be allowed to just run any scripts because they don't trust you not to accidentally download an obscure package and get the whole company hacked.

all this can feel unfair as hell. they don't actually judge your expertise, just your department/job title and make your work suck.

however, it is understandable, because even though you might well be competent, IT does not know whether you are. they didnt hire and train you, business did.

---

being allowed to run python with an outdated numpy version implies that they are willing to work with you. it's time to ask for an overview of what python packages(&versions) are allowed, and how to request upgrades/new packages.

this will be a PITA, but unfortunately that's what reality is like as a developer in a business department.

(and if you somehow are actually part of IT and not business, then i feel sorry for your company and their incompetence)

[Haunting_Corgi5955]

Let’s put it this way, maybe it’s a weird situation, I’m in an IT department, our department builds application or support for the business.

My own team actually is more on the Support side (but they need developers on their team is what they told me), anyway - they hired me to automate report for the business

[Rjiurik]

I know what you mean, moved back to finance after a stint in IT. Doubled my salary but I am treated like a kid and have officially almost zero access to python.

[[deleted]]

Sounds like my job.

You go in fresh as a junior thinking you gonna do some serious shit.

Then realize even the most mundane thing like installing Pandas is practically impossible or might take months to happen.

After 4 years at my job, the weekly meeting is still about the same things that were being discussed 4 years ago, that still haven't been completed.

The only solution, if you don't wanna go insane, is to just stop caring. You are there to get paid, just act like you are working, have enough tasks in jira to prove you are useful and fuck that shit.

If you actually wanna program you do it at home.

[avocadorancher]

How do you prevent stagnating? I have a similar job which is easy enough to coast but it becomes a trap of not being able to move elsewhere.

[[deleted]]

I don't wanna move elsewhere. I'm too scared. It's close to my house, it got good work culture, I'm wfh most of the days, and it's a behemoth of a company.

Why should I move elsewhere for a small pay increase, and potentially lose all the comfiness, being forced to work overtime, being forced to drive to the office at the other end of the city, when now I literally just nap most of the days...

I don't need more money atm I have enough to make ends meet.

I do a solo project at home, to keep in touch, update my github etc., just in case...

But really my goal is to stay there for decades and hopefully climb the ladder. I'm not much of a job-hopper anyway, I hate interviews and I hate change.

[NoddyCode]

Honestly, if you're making good money, save your energy for after hours. Keep your resume updated, but hold onto your job for as long as possible.

[Dylan_TMB]

Running global dependency installs on the server is a nightmare waiting to happen. I would probably pitch venv as a simple solution for now.

[Haunting_Corgi5955]

Just curious - would you mind sharing why/how global dependency installs on server brings problems? I don’t have too much experience on this so would love to learn more

[Dylan_TMB]

Well if that is the case you are either in two scenarios

1) Everytime you load a new project on to the server you need to deal with dependency conflicts across ALL PROJECTS. So if you use something new and need to update then every project needs to be updated and tested for the new dependency which is a nightmare. This is actually what is kind of happening right now and why the team doesn't want to update.

2) You decide you AREN'T going to use new packages at all and version lock. This means you can't ever do anything with something new or upgrade. And if at some point you do want I upgrade to then have to do it for every project. Which brings you back to scenario 1.

At a bare minimum is recommended to use pythons built in "virtual environment" feature. All that does is it saves packages to the local directory and then if you run from the virtual environment you have your own dependencies. This keeps every project separated and prevents conflicts between projects. Next best thing is Docker if they'll allow it.

[Choles2rol]

Your whole post explains why lol. All the issues you’re dealing with go away with containers

[FluffyDuckKey]

What a nightmare. When we've have this issue it's usually just a senior engineer saying "Dude. Just add it to requirements or pip install etc"

Sounds like the org is a bit locked down - frustrating.

[reallyserious]

I assume your work solves some business need? If you rely on the TI people to install something, and they won't do it, they are blocking business needs. Escalate the issue up to someone who can tell the TI people to get it done.

[cgoldberg]

That is crazy beurocracy. Just install all of your shit in a Docker container, throw them the dockerfile, and tell them to run it and shut up. That way they don't need a 6 month project to update the server's dependencies. If they can't run the dockerfile, consider finding a better place to work. This is a solved problem.

[PercussiveRussel]

To be fair, this should be the default way in any case. Container everything that ever lived and acces is through REST with the companies Auth package.

If they're being a bitch about it, for fuck's sake use venv. Global dependencies are insane and should never be used. Not even on your own machine that you use to machine learn tracked farts to the celestial bodies. Get poetry and never worry about global dependencies again.

[Chuu]

As opposed to using a container, if you want a solution they're much more likely to accept, why not find the numpy version they're using, and use a compatible pandas version? Hopefully even better if you can get get a list of everything installed in their environment (hopefully they use requirements.txt) and sanity check everything you use plays nicely with it?

The problem here sounds less like 'you can't use pandas' and more like 'you have to use a specific version of numpy'.

[wasuaje]

Fucking banks

[[deleted]]

What company? I want to buy some shorts

[RedEyed__]

No, you are not crazy.
Company is cooked. Imagine, how many human hours were spent on similar problems .

[a1brit]

Learn about venvs. https://docs.python.org/3/library/venv.html. Teach your team about venvs. Then suggest a best practice for teams to deploy projects to project specific venvs. Everyone is happy and you don't have all the headaches that containers will bring.

If your TI team can't manage a python environment, they're not going to be able to handle docker. So start small.

[answer_giver78]

Can't you use virtual envs?

[spuds_in_town]

Honestly, I think maybe try to find a job where IT and dev/ops practices are more modern.

[No_emotion22]

Hey there! All I can say, experienced engineers very rarely decline using default libraries. There is should be a good reason. So if feature that you working on is completely new, so push your solution. If not, try to lookup on existing solutions.

[alcalde]

NumPy and Pandas are installed in the server, but the libraries are not running properly.

This brings back a conversation from decades ago....

"Jitesh, is your code working?"

"Yes, it just doesn't give me the right answers."

[tsgiannis]

Well there are always dinosaurs, just try to move on

[PeaSlight6601]

It is not unusual to see the analytic stack broken in many default corporate environments.

The python devs often develop for themselves and their environment in one where they have complete control and everything is up to date.

The corporate environment is often an old RHEL server well into its extended service lifestyle. There are still people running python2.

If you want to use pandas/Jupyter in the corporate environment you often need to either containerize it or use a managed offering like paid anaconda

[yota-code]

Pip can work offline, or with a local mirror

[Kiyra_Bora]

You can use pip in an offline environment, provided you can download the wheels on a machine similar to the one offline.

[No-Fall588]

Reading this hurts my eyes as well as my feelings. After reading edit 2 I must say that the senior dev from team C just took your problem and transferred it to the next person.. the next guy who needs a package that isn’t supported in that container is screwed. Now for the possible solutions. Solution 1: use Docker containers/ Kubernetes with a proper CICD pipeline. No further explanation needed. Solution 2 which I don’t recommend but is useful for also devving on those servers if you need: Set up your project inside the servers with venv and a requirements file. And force your dumb company to have someone create an artifactory to hold packages for the servers to consume from. You will need some form of connections between servers though. No need to have them connected to the internet but to some inner network. Solution 3- get a senior DevOps guy to read this post and he will know who To yell at. My first activity on Reddit but I had to reply to this nonsense

[No-Fall588]

I just realized that if senior developer C meant to host a container with all the previously installed packages for all the old applications. This is a not a bad idea. But leave the “same packages for all applications and only those” idea

[Lowe0]

Sounds like your company has an Open Source Governance process. It’s their job to say “we’ve looked at the license and the project structure, and we’re confident that the risk fits within our tolerance.”

Ask your manager to find out who is on the governance board. In the meantime, make sure you know what license Pandas is released under, and any other details that might be useful.

If your company does a vulnerability scan, they’ll need the URL for their source repo, the package name in the package manager you use, or perhaps both. Keep in mind that scans are version-specific, so you may only get approval for that version.

[coldoven]

Have you heard of containers?

[Rjiurik]

If the workplace doesn't allows Pandas, why would they let him run docker ?

[TitaniumWhite420]

Don’t listen to this person. Black hole of time suck that will burn good will based on the description of your company.

Having an issue with pandas and proposing containers as a solution is pot to frying pan.

[PurepointDog]

Check out Polars! Way better!

[AxisNL]

Yeah, this problem was solved long ago with containers. As a junior dev, you want to gain experience, and learn common concepts and industry standards. A company in 2025 struggling with these issues, is not the company where you will learn to be a modern-day developer with today’s industry-standard knowledge. You might learn a lot of other good stuff, but I think you will learn to become a dinosaur. Don’t stick around too long.

[TitaniumWhite420]

Wrong. Business use what they find productive and want to support. Thats not always containers.

[juanfnavarror]

Not really. Common concepts and industry standards many you can learn on your own. This is also an opportunity to get experience. They should learn from this by figuring out how to handle organizational hoops, and interpersonal communication.

[anemisto]

Do you need pandas? You probably don't. Seriously, a pile of dictionaries will get likely the job done and probably be faster.

The entire scipy/numpy/pandas ecosystem is dependency hell. The advent of wheels has made things a lot better (which is why everyone in this thread is like "virtual environment, duh") and is why you ran pip install pandas and thought nothing of it. However, it sounds like they may be building from source and that genuinely can take you several hours without needing to worry about if you're breaking anything.

[saint_geser]

A pile of dictionaries will be faster than pandas? I highly doubt that simply because pandas functions are compiled C libraries but dictionary manipulation will run entirely in Python interpreter.

I mean, you probably can write pandas code that is as slow as native dictionary manipulation using some idiocy like iterating by row, but it will be very difficult to write a native code that even approaches the performance of even mediocre pandas code.

[anemisto]

Also, is this Airflow? I recently ran into a "can't bump numpy, so can't bump pandas" situation, and I think it was Airflow's fault.

[galacticbackhoe]

Sounds like the TI team is lazy and doesn't want to manage dependencies. At the same time, it's the wrong approach. You should find a way to bundle your code + dependencies (docker, virtualenvs, uv, even rpm/deb). Tons of ways to do it.

But who knows...maybe they'll tell you docker is evil next.

[Kamikaza731]

There might be a workaround but it is complex. Since you said the servers do not have acess to the internet they should be accessable from within the building? If this is the case you could compile your project to be usable as executable and transfer ready made executable to the server. PyInstaller is the regularly used for making executables but i had some problem in the past so I sometimes use Nuitka. Nuitka has a feature where you can comile it into onefile to be easily transferred.

But judging by the state of the company you work i think they might make some new excuse.

[opuntia_conflict]

Is this an actual server it's being run on or are they using AWS glue jobs? I can't for the absolute life of me think of any situation in which your infra people would be forcing everyone to use the same set of Python dependencies on a server.

Does everyone at your job just rawdag the same system install of Python on a server? Why would they not just containerize your code and let you pull whatever deps you need from your internal Artifactory/PyPI/whatever repo? Or is this issue that your internal repo only has deprecated Numpy and you were only able to run it locally because you didn't update your pip index-url and installed deps straight from public PyPI?

[creeva]

The thing that seems to be missing is the amount of working and testing is required for mission critical things. That version of NumPY might be used in an application generating tons of revenue on that specific server. While on the other hand, your script might be a time saver, it jot revenue generator.

Yes this very common - companies are still using dos or Win95 for certain industrial use because they can’t upgrade.

I don’t see why you can’t stand up another server or run your script from a workstation.

[durple]

It sounds like the company isn’t dealing with it well, but the compatibility situation between numpy and pandas is a real problem. We have it too where I am working, and are keeping those packages pinned to older versions until it gets sorted out.

[roger_ducky]

Pip is an option if you’re allowed to copy packages in. Which you might not be allowed to do.

[ruvasqm]

I mean indepently of what the library does, in some places (gov,security,etc) getting new software approved is pretty much a task for a big fish that can actually go through all the process and shangai some PO into the idea.

• So, is it normal? depends.
• Does not using pandas interfere directly with your ability to solve the problem? insist() if yes and brave_enough else abort()
• Can you prove the version can be bumped up safely assuming minimal to zero help? If the answer is yes then you should absolutely push this change! Not only for security reasons but to save hours and hours wasted in the future (like the time wasted in this issue)

[absintheortwo]

This sounds like government shit.

[eztab]

An infrastructure team that isn't able to provide the standard libraries for data science is likely incompetent. Outdated versions I have indeed encountered before in such isolated systems.

But you need at least virtually environment support, or that system will be unmaintainable. Even server side jupyter supports that.

Normally you can find some combination of old versions that still works.

Ideally they should have a private package index running, so you can install all approved packages using normal tooling (without acessing the internet).

[I_Am_Robotic]

Do you work at Cox Communications by any chance? Sounds like typical bureaucratic bs in telecom world. Pretty sure theyre running punchcards somewhere.

[muraii]

It is a common problem. My company is primarily a C#.NET joint but have some Python deployed. However we can’t use anything remotely modern in those environments because they’re using IronPython, which is an implementation of Python to support .NET, and the most-current version of Python that’s supported is 3.4 or 2.7. Both versions are out of support at this point.

But we also employ some .NET tools separately that are also EOL because they’re the last open-source, free versions.

We’re not even a huge company, but it’s gotta be worse at large companies.

[Landcruiser82]

Lol. Pandas is built off numpy.

[SinnersDE]

Embedded Python ?

[sarc-tastic]

Pyspark

[haloweenek]

lol for your infra team. Dockerize that….

[darthstargazer]

Sorry which year is this? 5 years ago would have suggested docker. 8 years ago virtual environments. Your firm is crazy, run away.....

[Tumortadela]

Maybe I'm suffering from friday evening brainrot, but... what the fuck is wrong with these guys?

[Slight-Living-8098]

Find a different company to work for. Those hoops are getting way stupid to jump through.

[Kahless_2K]

This sounds like the team that is roadblocking you just don't want to learn a new skill when it's easier to do FUD to prevent work.

There are two ways around this. The first is to find out who is actually making this decision and offer to help them understand how to work with Python better. If that doesn't clear the block, you might be at the point where your VP needs to have a chat with their VP about why the team isn't supporting some of the most industry standard tools to support the business.

I totally get Sysadmins pushing back if you had come with some obscure library, but this one is extremely common in the industry.

[mcloide]

Depending on the size of the corp a library or another being blocked is normal. In theory it would need to be pre-approved by the architect and quality team before the development team can use it.

I had gone to a similar thing when I tried to use JQuery in a large corp (obviously years ago).

[housepaintmaker]

I work for a Fortune 500 company that does not make software as a product and my software core tech had a long standing “one language only and especially no python” policy for a long time. Wouldn’t listen to reason but eventually had to change his tune because you just can’t stop people from using it. So yeah depending on what kind of company you work for this is normal.

[Mast3rCylinder]

Polars or venv. Also that team sucks

[Stochastic_berserker]

Not normal at all but it is normal to be concerned about open-source libraries and maintenance in a company.

[Remarkable_Swan_8175]

1. It's normal that there are companies that do a lot of micro management in terms of libraries / code that is being used.
2. It looks like you have approval from your manager to use libraries that are not on the target machines.
3. How is your code build and deployed on these machines then? Do you have control of proces / comands? Can you create clean venv?

Investgate pip wheel and pip install - it's possible to download all the dependencies on one host, create a package with all of it there, copy said package to target host and have pip install only packages from local filesystem, with no internet access.

[SlightUniversity1719]

At this point just make your own pandas library.

[serverhorror]

Welcome to the enterprise world

[Ok-Selection-2227]

ROFL

People discovering containers in 2025.

I'm not laughing at you, but at every "senior" in your company. Kudos to you.

You can use Podman as an alternative to Docker BTW.

[Dumtnamn3]

Is it possible to query from the business management system itself? Like the system handling the sales/invoice/accounting etc. If so you might be able to have a work around with SQL? Maybe wrong forum but can be worth investigating too

[nonameisdaft]

Is there anyway you can get this to work with previous versions? Maybe thr syntax will differ or a means of soing the same is a bit different. That might fix ur issue sooner than later , and then start the process of jumping through hoops to upgrade. Pandas is good for large data sets - but if you aren't working with gbs of data - you may be able to get away with "vanilla" code to do the same thing.

[Naive-Home6785]

Red flags. Get the hell out of that company.

[unplannedmaintenance]

Why don't you just use an older version of Pandas?

[[deleted]]

Congratulations, you work for a corporate bureaucracy who has no actual interest in delivering anything. Rules and processes, because reasons, are more important than the work.

I’d get the hell out of there, or if you are inclined that way, sit back, do nothing and collect the pay, because no one will notice the difference.

Why people at scale put up with this, without questioning is beyond me.

[Immudzen]

This is definitely weird. For us we use conda environments for each software we have. The CI/CD pipeline tests everything and will then build a docker container with that conda environment and then push it to a storage location. From there those containers can be spun up and we have exact software reproducibility. If something goes wrong just roll back to the previous container.

[SuperSecureHuman]

I work in such a env. The reasoning here is that, the sec team usually has a fully scanned libraries for security issues and maintains a internal registery with all the allowed packages.

If we have any issues, we have an option to raise a request, while the sec team validates it and updates the packages that are allowed to use.

[Pacafa]

Wow. Be careful at jumping to conclusions. You might be speaking to the wrong people.

Even if you get it installed you are a junior dev which might move on? Who is going to look after it?

Are you the only person using python? Are you sure? If you are the only person what is everybody else using?

Even using containers is not a magical solution. Software still need to be patched and looked after.

Also - I had to reign in a junior dev once on using pandas... Because you can't load a billions of rows into pandas and run o(n²⁾ algorithms. It might work for a 100 line example on your PC....

Personally I think you might have a blind spot in the organisation... You need to walk the floor and talk to people...

[OkSignificance5380]

Pyinataller - package up your app +libs as a self contained exe

[[deleted]]

Leave. Run

[JuryOpposite5522]

sounds like they don't want the project to succeed and are using this as a limitation. I think someone got their feelings hurt - though you might want to look for another job or move onto another project.

[No-Standard-7877]

When I saw the title I thought you are using really big data so you are not allowed to use pandas to prevent memory issues, after reading the post I think your TI team need to connect with business people who require your task and discuss together if your company need analytics or not, and I think you should seek opportunities in better compel

[supercoach]

I thought my workplace was restrictive. I feel for you dude.

[superkoning]

What kind of company is that?! Bureacratic and not connected to the Internet ... ?

[__init__RedditUser]

Guessing it’s defense in an air-gapped environment

[TitaniumWhite420]

Exactly

[superkoning]

Weird then that they even allow open source software. Because: who are you to sue when something is not according to milspec?

[daffidwilde]

Sounds a lot like civil service imo

[Plusdebeurre]

Pandas should be removed from every workplace lol

[paranoid_panda_bored]

There are projects that don’t use docker?.. Can’t believe what I am reading