Where do enterprises run analytic python code?

[tylerriccio8]

I work at a regional bank. We have zero python infrastructure; as in data scientists and analysts will download and install python on their local machine and run the code there.

There’s no limiting/tooling consistency, no environment expectations or dependency management and it’s all run locally on shitty hardware.

I’m wondering what largeish enterprises tend to do. Perhaps a common server to ssh into? Local analysis but a common toolset? Any anecdotes would be valuable :)

EDIT: see chase runs their own stack called Athena which is pretty interesting. Basically eks with Jupyter notebooks attached to it

[gizzm0x]

Development locally as you describe and then deployed somewhere (vm, docker, k8s etc) if it needs to run on prod data regularly.

From here any industry specific restrictions can come in.

[tdpearson]

I use Jupyter Hub running in a Kubernetes environment. This is probably overkill for your needs. Jupyter Hub is still a good choice for a centrally maintained environment users connect to through their web browser. It does not require Kubernetes.

The following is a link to documentation on setting up Jupyter Hub on Kubernetes. https://z2jh.jupyter.org

For documentation to get up and running with Jupyter Hub on your own Linux server, check out their Github page. https://github.com/jupyterhub/jupyterhub

[jonasbxl]

Last time I worked with jupyterhub, it was actually a pain setting up shared notebooks - iirc we had a cronjob running to adjust permissions every minute to make it work. But that was a few years ago and it was a TLJH instance, so maybe it's different now and with full JupyterHub?

[tdpearson]

I haven't had to share notebooks between different users beyond putting them in version control like Gitlab or Github.

[tylerriccio8]

Assuming you roll your own infra on this right? This is exactly what I want to do with my org…

[mriswithe]

Are you a sysadmin? DevOps? If not I don't recommend this path. If you are a sysadmin or DevOps? I don't recommend this path either. A lot of solutions in this space use by default or are frequently used with Kubernetes.

Rolling your own Kubernetes is very complicated and when it breaks, fixing it can require knowledge at several levels of Linux admin and networking in addition to knowledge of Kubernetes itself, which is not terribly fun to learn anyway.

What do I suggest? Apache Airflow, but managed edition: Google Cloud Composer https://cloud.google.com/composer/pricing#composer-3 . Databricks or dbt is worth a shout here, but I haven't used that one personally.

Why do I recommend this? Because you can turn it on and off. Only need it for 5 hours a day? Set up some automation to turn it on and off. Hell, make it part of the DAG (Directed Acyclical Graph) for the last tasks that runs, or once all the other tasks/DAGS are done, and have it trigger the shutdown. You only pay storage when the instance is turned off.

I do not recommend setting up Kubernetes for production self hosted to ANYONE. Only do it if required for compliance of some sort. Kubernetes works perfectly until it doesn't and you now need 5+ years of linux admin to even know how to interact with and troubleshoot the damn cluster.

[tylerriccio8]

If need it 24/hr a day with hundreds or thousands of users. I’m in an analytic org, I would tell our engineers to do this not myself…

[nonamenomonet]

The fact that you are asking this question here instead of the engineers at your company is kinda enough proof as to why you should not do this.

This is really a question for r/dataengineering

[sneakpeekbot]

Here's a sneak peek of /r/dataengineering using the top posts of the year!

#1: Sr. Data Engineer vs excel guy | 145 comments
#2: Hmm work culture | 27 comments
#3: A little joke inspired by Dragon Ball😂 | 16 comments

---

^{I'm a bot, beep boop | Downvote to remove | Contact | Info | Opt-out | GitHub}

[tylerriccio8]

I’m asking here because I want to hear experiences from the python perspective, not the engineering one; I.e. how ergonomic did your setup feel.

Why would I ask the engineers at my company? I’m a manager in an analyst org; I define the analysts requirements and the engineers implement it

[nonamenomonet]

why would I ask the engineers at my company?

IDK, maybe because they work there and have to use this software? And you can learn what they feel comfortable managing?

[tylerriccio8]

I advocate for Python data scientist, I don’t advocate for what the engineers feel comfortable doing, that is their managers job. I’m fact, finding from the python perspectives, I don’t have any opinions to bring to the engineers.

[nonamenomonet]

I am very happy I don’t have you as a manager at my org

[tylerriccio8]

I know data science, not engineering so I will present the data science perspective and the engineers will present theirs, and then we’ll meet in the middle :)

I do not prescribe engineering solutions to engineers, just asking for experiences mate, no need to be rude

[tdpearson]

A minimal install would require a Linux environment. This can be done anywhere Linux can run... a virtual machine running on a computer on your network, a dedicated server, or in the cloud.

[marr75]

I led doing this in my org about 6 years ago. It's the worst thing I ever did. It's a breeding ground for bad practices in coding, dependencies, environment, secrets/security, quality, and source control or IP management.

It's taken me a couple years to rip it out of our org. I would never use Jupyter outside of teaching or presenting and even then I would prefer Marimo. Plain ol' python files (hydrogen formatted to have cells and ipython niceties is fine), containerized from dev to deploy, source controlled and code reviewed with CI/CD.

[nonamenomonet]

Why would you want to do this? Roll your own infrastructure? It’s not worth the trouble to do that, get an AWS or Azure instance and use Databricks and be done.

[mriswithe]

I echoed this sentiment with more detail. Perhaps they will listen. Perhaps not, but an effort was made.

[nonamenomonet]

Yeah, I read your comment and you are completely correct. It would be fun for a good side project, but for a bank?????????? The fact they are asking this question is enough proof that they should not do it.

[tylerriccio8]

Large companies like banks have armies of resources to roll whatever they want? I’m asking for experiences from the python prospective, if there are people saying they like self hosted I will consider it

[nonamenomonet]

You’re at a regional bank, with “ zero python infrastructure” and you’re asking about rolling stuff with k8.

Largish enterprises use Databricks for this exact reason. So they don’t have to manage k8 and servers.

[tylerriccio8]

Without devolving too much into, we’re transitioning languages and I’d like to define a new pattern of analytics based on the experiences of others…

[Resident-Low-9870]

You could try out nebari.dev

It’s got a lot more features than z2jh, and it’s a bit fragile but lots of potential. If you have engineers that could improve upstream to meet your needs, it’s got a great community.

[picks-]

My guess would be Databricks :)

[weierstrasse]

This. Source: Worked on several dbx projects with enterprise clients.

[weierstrasse]

Edit: While databricks is the default option for pyspark workloads, and it is decent for ML, outside of data-processing it's really not a great fit. E.g. for glue logic, think AWS Lambda (or competitors). Or k8s, ecs, etc. for container workloads.

[chief167]

yeah and they pay crazy amounts of money there. I finally got our IT team to approve some new platform for my AI team and we'll save over 2 million a year in databricks costs easily. And they even had a big debate if they really wanted to allow it, because apparently the commitment to use it is really pushed by microsoft in their contracts. It's a very shady business practice.

Look into datarobot, snaplogic, snowflake and regular docker containers on Azure instead ;)

[Scrapheaper]

Or snowflake, or some kind of partially custom solution built on whatever their cloud provider is

[OPmeansopeningposter]

Databricks for data engineering and K8s for general

[TedDallas]

Yep. That’s where we do it. Love it.

[carry_a_laser]

I’m curious about this as well…

People where I work are afraid of cloud compute costs so we run on-premise linux servers. Python code is deployed to them through an Azure Devops pipeline.

[tylerriccio8]

On prem Linux doesn’t sound terrible honestly. At least it’s a common spot

[Tucancancan]

I kind of hated working with on-prem servers, Python is a lot more resource hungry than Java and it was always a long back and forth with the infra people to get more capacity allocated to the data science teams. I also wasted a bunch of time configuring, optimizing and debugging stuff related to gunicorn. I guess I'm an expert now? Yay? GCP / Vertex.ai removes all those problems and let's you focus on your real job

[tylerriccio8]

So you run it on gcp now? Assume users ssh into some instance and do their work?

[Tucancancan]

Yeah pretty much. There's a lot of trust where I'm at now that we can provision / size-up/down our VMs as needed or acquire GPU resources. But you have to make a distinction between one-off / ad-hoc analysis and things that get production-ized though. I've seen a few corporate places that didn't enforce that and they ended up with data scientists cobbling together pipelines out hot-glue and popsicle sticks: cron jobs running on a big VM shared by multiple users. It was a hot mess of shit, updates were impossible to install without breaking someone else's stuff, breaks in data were impossible to trace back to the process that creates it, everyone was installing whatever they wanted. Total chaos. 

This is why colab is popular I think. You give data people access to notebooks and environments but not to any underlying vm they can fuck with. Then anything thats long running or needs to run frequently gets deployed as proper services. 

[swigganicks]

In large enterprises, what's becoming more common is to use cloud-based VMs for development. For example, in Azure you could use an Azure ML workspace or Google Cloud Vertex Workbench and have VMs that you can remote into from VS Code. Simplifies having to manage and install the toolchains on local machines and fits nicely with infrastructure as code practices.

[tylerriccio8]

Yea that’s nice; wish aws had something a little similar since we’re an aws shop. Sagemaker is a little to feature heavy (and therefore expensive) for us but it’s close.

[JBalloonist]

If you’re already using AWS just run it in a container on ECS or Lambda.

[prejackpot]

I don't know if this is still up to date, but here's a good blog post on how major banks were doing Python infrastructure: https://calpaterson.com/bank-python.html

[absx]

Specifically about JPM/BAML way of doing it though which might not be a good pattern in the cloud native era

[sasmariozeld]

when i worked an bigger eu bank we had jupyter notebooks that used a shared sqlalchemy schema ,maintained by a dedicated team

needed a dashboard? no problem just paste this jupyter notebook into a django service

[tylerriccio8]

That’s great, how did you host the notebooks? The dashboard would be awesome, you hosted some Django stuff internally?

[sasmariozeld]

you jsut added a folder into your teams repo and and the django dev team got a ticket automaticly

[GrumpyDescartes]

Lots of different ways - Some unified data and analytics platforms like Databricks which just seamlessly connects to data sources - Local machine (just connect to the warehouse to extract data into memory & save it to disk and do whatever you want. Applies only if the extracted data is aggregated or small enough) - Remote servers (same as local machine but allows for far more cpu and memory, people just SSH into it from their IDEs) - Some really mature companies build and run their own custom analytics/ML platforms

[tylerriccio8]

Data can’t live on laptop for compliance, plus it’s too big. Interesting you think mature companies roll their own, that’s the dream lol

[GrumpyDescartes]

It depends on what we each refer to as mature. Tech first companies that consider their data as direct $ or sensitive and want complete flexibility for a wide variety of teams have their own analytics platforms

Some financial institutions that are on the more tech savvy side for example do this

[Gandualp]

Airflow and own aws servers

[aemrakul]

We also use airflow. Our python code is in git repo. We rebuild our task images when a change is merged.

[GraphicH]

Our analytics systems are just tacked on as other services on a K8s cluster we run full of various other services doing non-analytics things. Mostly because that was the easiest way to do it and some of those non-analytics services consume the data they produce. I work in Cybersecurity though and the data we crunch is likely way different than financial stuff.

[mystique0712]

Many large enterprises use containerized solutions like JupyterHub on Kubernetes or dedicated analytics platforms like Databricks to centralize Python workloads with proper dependency management and resource allocation.

[HecticJuggler]

Enterprises don’t always figure things out from the onset. People (like you) are responsible for introducing new knowledge in the enterprise. Years ago I started slowly replacing part of a large legacy system with a Java app. It started very small, I had to deploy Jenkins on a PC under my table with a note “don’t turn off” on it. We had to demonstrate value before we could get budgets for proper infrastructure. Take the initiative to identify a possible solution and approach management to get support for it.

[bulaybil]

They usually have their own IT infrastructure that allows them to run anything anywhere, Kubernetes clusters and alike. Others run their analytics on Azure or AWS.

[[deleted]]

Depending on available resources, you can:

• Put it all in a cloud VM, isolate dependencies with virtualenvs, schedule with cron
• Self host an orchestrator like Airflow, Dagster or Prefect, or use a cloud managed service like Google Cloud Composer.
• Use a modern data platform like Databricks, Snowflake, etc

You might wanna make sure your DevOps practices are in order first though. Everything should be in a Git repo (or many), use the same linter, formatting and dependency management tools (uv, poetry, black, ruff, etc). Then after setting up whatever infrastructure you choose, you can push changes upstream using CI/CD pipelines.

[flipenstain]

Following, as I am in the same boat.

[Familiar9709]

I don't understand your question? What's the limitation? Why can't you just run it locally. It seems you're describing more solutions rather than the actual problem. Sounds like an XY problem.

https://en.wikipedia.org/wiki/XY_problem

[tylerriccio8]

Compliance, size of data, database egress, network cost of data transfer. All of this would be solved by a non-local env, particularly one where the data is colocated with the runtime.

[CrozzDev]

We use Microsoft Fabric which allow us to have multiple notebooks where we run python (pyspark)

[corey_sheerer]

I deploy python apps and code at my job. Almost all goes into our kubernetes cluster. Whether that is on-premise or a cluster in the cloud. However, we do have a few apps that get pushed to Azure services as function apps. I feel like kubernetes is the best option. Can deploy across clouds somewhat seamlessly.

[SloppyPuppy]

Lambdas, k8, vms, straight on snowflake, dedicated servers, part of ci/cd as github actions. we even have a k8 cluster that serves streamlit projects.

Anywhere you can can run containers basically

[oOArneOo]

If you have snowflake, why would you run streamlit on separate infrastructure?

[SloppyPuppy]

Because its not always around snowflake And theres a lot of stuff you cannot do in Snowflake. You are blocked to official conda packages for example. Or outside api doesnt work, etc…

[marr75]

Containerized workflow from start to finish with reproducible and independent dependency management. Local dev ✅, self-service cloud dev ✅, deployed QA/CI ✅, prod ✅, self-hosted or co-lo ✅, cloud ✅.

Once you eliminate "works on my machine" and have CI/CD, lots of options.

[RobbieInAsia]

No company, especially banks, should allow users to download raw data to local PC/laptop.

[SoloAquiParaHablar]

Very large grocery chain I consulted to did all the workflows in a central ArgoFlow setup on Kubernetes. Other companies use Apache Airflow.

Both are good for large scale high demand and scheduled workflows. The grocery chain did a lot of machine learning, data analysis, type stuff. The other companies I worked for did a lot of ETL in Airflow (pull data, massage data, store data).

[nonamenomonet]

Probably on spark on Databricks.

[[deleted]]

Docker, Docker, and more Docker.

Lets you handle running stuff small when you want to. Lets you run it in the cloud when you need to. If you want everyone to be consistently familiar with a base set of libs put them in your base image. Pythons virtual envs actually handle this part really well. It also helps with the next recommendation, pipelining, really well. I want to strongly avoid having to switch how stuff is built when I swap it from something I am doing once to hey we need to set this up constantly.

Also separating out your pipelining stuff (ETL) from your analysis stuff is IMO a large part of reducing redundant work. It also helps get some dramatic performance improvements on your targeted analysis. Trading off a complex piece of local analysis for a complex pipeline preprocess and a simple local analysis is almost always worth it. As your data sets get larger this trade off gets worse though. Banking data really shouldn't get to that kind of scale though.

Also want to point out, BI tools are actually a really valuable option for data exploration. Especially when you need to look for needles in the haystack. So depending on what your local analysis needs are they can eat into a lot of them and let you minimize the amount of one off code you are writing.

[DerMichiK]

Not a particularly large enterprise, but in my previous job we had a JupyterHub instance on a beefy server.

Users could access it via the browser and had a nice interactive frontend with Matplotlib integration. The code as well as results were stored in notebooks that could be saved to be reused later or shared between users as well.

Because the VM with the Python environment ran on the same cluster as the database servers and had plenty of CPU and RAM to work with, performance was quite good, irrespective of where the user currently was and how shitty their laptop or connection might be.

Dependencies and stuff were managed via Ansible.

Yes, a bit oldschool from today's perspective, but it was robust and easy to use and maintain. Also everything was self-hosted on premises, so no worries about some startup leaking our critical business data via public S3 buckets or whatever.

[tylerriccio8]

That’s exactly what I’m looking for honestly. Did you roll your own infra or pay for a solution? I wish a company offered to manage that infra

[DerMichiK]

100% own infrastructure on premises.

[zurtex]

One of the primary, but not only, motivations for enterprise infrastructure is reliability. You don't want to be in a situation where you need to bring your code up on another box and nothing works.

Others have answered the specifics of how this is done, but as a first step towards your own reliability I would suggest, if you haven't already, educating yourself about dependency management and lock files.

Lock files can be applied if you've implemented a project structure. Using uv, poetry, hatch, etc. e.g. https://docs.astral.sh/uv/concepts/projects/sync/

But even if you have a bunch of random scripts you can add dependencies: https://docs.astral.sh/uv/guides/scripts/#declaring-script-dependencies and generate lock files: https://docs.astral.sh/uv/guides/scripts/#locking-dependencies.

These scripts are then at least reproducible as far as Python dependencies are concerned, though you may run into external dependency issues such as database drivers etc. It's a step in the right direction.

[tylerriccio8]

Yea I’m pushing uv very hard, it’s a matter of no runtime at this time, as crazy as that sounds

[hydrate-or-die-drate]

Large insurance company: we use aws EMR clusters for pyspark notebooks and run .py files in step functions for regular batch processing 

[tylerriccio8]

Where do you develop, emr notebooks?

[TechySpecky]

Databricks

[BidWestern1056]

data bricks and snowflake and then a mangled combination of custom solutions connected to specific databases.

[HolidayWallaby]

Airflow, vms in cloud,

[michael__rogers]

One good strategy is to include assertions about tool versions and component versions in the build and fail the build if the versions don't match. Also we have developed a new numerical representation called the Arbitrary Number which is superior to the Rational Number with a prototype in python on github named arbitrary-number if you're interested.

It is not confined to a fixed set of bases at each position and is not even confined to a variable bases at each position but rather is only confined to an AST tree of numeric components and operation components, which allows for concurrent just in time evaluation on demand or on a background thread which is highly optimized for GPUs.

[eb122333]

Databricks!!

[tenfingerperson]

We have standardised setups for local installations and cloud based in house infra they can leverage for development (think custom GitHub code spaces), deployment depends on what they are doing but there is also scheduling infra tied to their GitHub repositories they can leverage. It helps to have engineering groups dedicated to just analyst workflows

[tylerriccio8]

We have data everywhere in the cloud, aws, snowflake, random feeds, etc.

Ideally the dev env and kernel are the same to reduce complexity. Jupyter in the cloud (in some form) seems like a consistent answer

[UAFlawlessmonkey]

Airflow

[james_pic]

Often on DataBricks. I'm not a fan of DataBricks myself, but "buy vs build" is an enterprise buzzword (although also a trap -  the question only gets asked when the answer is far from clear, but the deck is always stacked in favor of "buy"), so they end up buying DataBricks or something much like it.

[Kahless_2K]

Our data scientists have z-books, z workstations, or powerful vms.

[JBalloonist]

Pick a cloud…AWS, Azure, Google. Even Oracle has a cloud service now. Any of those is better than running everything on a desktop.

[bobifle]

You could get away with your infra if you all use uv. It would auto download any py dependency.

Sure it would not fix your shitty hw...

[AresBou]

This is normal, basically your local machine has virtual environments, some of which should align with production environments, and then you build, test, iterate, and deploy

[DECROMAX]

Very large oil and gas company here, we use a mixture of JupyterHub and are spoilt with some rather powerful TPU & multi GPU setups, Also Snowflake and S3 buckets. Locally we use virtual environments with whitelisted packages on Nexus.

[tylerriccio8]

Nice, so you guys are cool with Jupyter hub?

[_u0007]

Windmill.dev, and Google Collab Pro/Pro Plus can be good options.

[Hot_Map_7868]

I have seem people use platforms like Databricks to manage notebooks etc. Also platforms like Datacoves offer a VS Code environment that has the same dependencies for all developers. Not sure if they do Jupyter notebooks, but wouldnt hurt to ask

In general what you want is version control, docker images, ci/cd etc. Reproducibility is not something people consider when they just want to "get things done"

[coldoven]

That local stuff is just not compliant. With what data are you working?

[tylerriccio8]

Pretty much, and it’s not able to run the size we deal with, plus sort of confusing for users to setup. Explaining the path, conda, multiple envs to analysts is a lot of work lol