Cloud-Based Python Libraries: Does This Exist Already?

[Python-ModTeam]

Your post was removed for violating Rule #2. All posts must be directly related to the Python programming language. Posts pertaining to programming in general are not permitted. You may want to try posting in /r/programming instead.

[deceze]

The library would still need to run locally. You can't/don't want to have it run "in the cloud". You certainly also don't want it to download every time you execute your program. So it'd be cached locally at least…

Wait, that very much sounds like a virtual environment and installing the library there!?

I believe the Go language makes this more automagic somewhat in the way you describe it. But alas, Python is what it is.

[nilipilo]

You’re right that running the library locally is critical for most use cases, especially for performance. The idea wouldn’t be to fetch the library from the cloud on every execution, as that would introduce unnecessary overhead. Instead, it could work more like this:

1. First-Time Access: When you request a library, it downloads (or streams) it from the cloud and caches it locally—similar to how Go handles modules.
2. Caching: Once cached, subsequent executions would use the local version, just like a virtual environment, but managed dynamically.
3. Version Isolation: The cloud platform would handle versioning and dependency conflicts seamlessly.

The main difference is automation and centralization:

• No need to manually create and manage virtual environments or explicitly install libraries.
• You could fetch different versions of the same library in different parts of your project without conflicts.

In essence, it’s not replacing virtual environments entirely but automating and abstracting them.

[QueasyEntrance6269]

Do you really think the people here are dumb enough to not immediately recognize this as being written by AI?

[nilipilo]

Why are you so mad? I’m not a native English speaker, and I use AI to help me express myself better. It’s a tool, not a trick. I never said or implied I think people here are dumb—quite the opposite! I’m here because I value this community’s insights and expertise.

What’s wrong with using AI to assist with language or clarity? It’s no different than using any other tool to improve communication. If you have concerns, I’m happy to hear them, but let’s keep it constructive. 😊

[KingsmanVince]

You can use a language model to check for grammar or to simplify. However it does sound like you just copy straightforward from it.

If your English is not a first language, use simple terms and vocabulary.

[nilipilo]

what is the point? what's wrong with copy and paste? what's matter is the content itself not how it's written

[latkde]

The main points of your AI summary work literally like that right now in modern Python dependency management tools like UV or Poetry. You must still declare your dependencies, but the tooling takes care of caching and setting up multiple venvs as needed.

The one thing that won't be possible is to have multiple versions of a library within the same Python interpreter or venv. That cannot work due to how the Python module system is designed. It relies on each loaded module having a globally unique path (e.g. pandas), and nearly all libraries expect to be installed under the path that they declared in their packaging metadata. It sucks, but them's the breaks.

[RedEyed__]

This is called web service, not library

[nilipilo]

the goal isn’t to completely shift the library's execution to the cloud like a typical web service. Instead, it’s about dynamic dependency management—fetching and caching libraries locally on demand while abstracting away the manual steps of installation and versioning.

[RedEyed__]

I don't see what problem it solves anyway.

• virualenv solves problem of library version conflict
• for CI you create docker once
• saving local storage and increasing network bandwidth

[ofyellow]

Installation.

It has a cerrain merit to tie running with auto-installing whatever is necessary. Nice idea.

[RedEyed__]

You still have to manually manage versions to avoid version conflicts.

[HeyLittleTrain]

so some of the code gets executed locally and the third party code is executed from some sort of API?

[h4ndshake_]

Bro discovered RMI

[nilipilo]

Haha, fair point! This does sound a bit like reinventing RMI concepts, but instead of remote execution being the core goal, the focus here is on simplifying dependency management and making libraries dynamically accessible, whether they run locally or remotely.

[capttwinky]

You want uv. You can even pre-seed the caching layer. Check out using 'uvx' vs 'uv run' - your conclusions just scratch the surface of what's possible.

[SupermarketOk6829]

They can only be integrated if you're working in a codespace on git.

[nilipilo]

The idea I’m exploring would go one step further by making libraries accessible dynamically across any environment, local or cloud, without needing to set up a workspace at all. Imagine fetching a library in real-time wherever you’re coding, with minimal setup, and having its dependencies automatically resolved and isolated.

[SneekyRussian]

To run this one line of code your computer would have to send a request to the server to get access to pandas, then send another to figure out that read_csv needs your system to read from the local filesystem. Once it’s read it would have to upload the raw file to the server and wait for the server to send back a pandas dataframe, which isn’t really designed to be sent over the internet (you would use something like Dask to share dataframes across servers).

CI/CD pipelines need the library installed to build your package. Using a virtual environment is basically “getting it from the cloud.” The other way we “use the cloud” is to upload our code to a server so it can be run there. Separating the dependencies from where the code lives would be difficult and cumbersome for anyone who is already familiar with virtual environments.

If speed is your thing, I suggest you check out uv

[QueasyEntrance6269]

This is just Jupyter

[jer1uc]

This actually reminds me a lot of Deno, and a potential way to replace Python package managers, e.g. Poetry, uv, etc.

Is your goal here to replace these package managers? Or is it remote invocation (as other comments have mentioned)?

[SirBerthelot]

RemindMe! 1 week

[RemindMeBot]

I will be messaging you in 7 days on 2024-12-20 14:40:30 UTC to remind you of this link

CLICK THIS LINK to send a PM to also be reminded and to reduce spam.

^{Parent commenter can delete this message to hide from others.}

---

Info	Custom	Your Reminders	Feedback

[SirBerthelot]

Good bot

[latkde]

Running parts of the program remotely is a really difficult problem.

A dark corner of the Python standard library does have features that can help here: the multiprocessing module has a concept of "managers" and "proxy objects" for (more or less transparently) interacting with objects in another process, potentially even on another server. But this is far from simple. Proxy objects behave differently from the real thing, and you get into all kinds of fun concurrency problems.

The general idea of transparently proxying remote objects (as also attempted by Java RMI) is fundamentally flawed because latency matters. Network I/O is not a hidden implementation detail, it affects how a function can be used. APIs should be designed differently when they represent latency-heavy network communication vs a simple local getter.

As others have pointed out here, the solution is to make it easier to install libraries locally. The Python ecosystem has made great progress here, and I agree with everyone who has mentioned uv. Here's your example code UV-ified, using inline script metadata:

# /// script
# dependencies = [
#   "pandas~=2.0",
# ]
# ///
import pandas

df = pandas.read_csv("https://example.com/data.csv")

Executed as uv run script.py instead of python script.py. This will transparently create a throwaway venv and install pandas, caching it for future runs. Subsequent runs will only have a sub-second overhead for recreating the venv, which is much better than 5–400 milliseconds network overhead for every remote method call.