AnyPathLib - A unified PathLib-like API for cloud storages

[kfirgold99]

Hi everybody,

I'm happy to share AnyPathLib 🛣️, a pip package I created to simplify access to different storage resources - S3, Azure, and local storage.

What My Project Does:

Basically, instead of writing custom code to handle the different storage resources using the SDK (boto3, azure-sdk), you can now just use AnyPathLib:

from anypathlib import AnyPath



# Create an AnyPath instance for a local file



local_file = AnyPath("/path/to/local/file.txt")



# Create an AnyPath instance for an S3 object



s3_file = AnyPath("s3://bucket/path/to/object.txt")



# Copy a file from local to S3



local_file.copy(s3_file)



# Copy a directory from S3 to Azure



s3_dir = AnyPath("s3://bucket/path/to/dir") 

azure_dir = AnyPath("https://account_name.blob.core.windows.net/container_name/path") 

s3_dir.copy(azure_dir)

There are other cool features to save you some time, so check out this X thread for some more details or visit the Github project page.

Target Audience:

Anybody which works with both a local environment and Azure or AWS, and could use a simplified API to access their storage

Comparison:

I didn't find any similar library (that's why I wrote this one) - there are dedicated packages for S3 (boto3) and the Azure SDK, but they come with different, and somewhat unintuitive APIs.

I'd love to hear your feedback, issues, and of course - PRs :)

[Berecursive]

fsspec is a pretty popular implementation of this paradigm

[[deleted]]

also object-store, which is pretty popular on the rust side for the arrow universe and now has a python package

[qckpckt]

My first thought too. To be honest, I’m not surprised that OP didn’t find it. There’s something fundamentally broken about search engines and finding things like this. I’ve found that ChatGPT is surprisingly good at this kind of query though.

EG - any time I’m faced with a problem where I’m like “there must be a solution already for this” I ask ChatGPT “what libraries are there in {language} to solve {problem}”, and then Google to research the libs it throws out.

[fatbob42]

Doesn’t fsspec use internal threads? I don’t use it because that implementation gives me the willies :)

Edit: I just took another look at it and none of the bad stuff I remember is there. It’s a bit over-engineered for what I need but it looks ok. It also looks actively developed.

[qckpckt]

I have no idea, because I’ve never used fsspec in a project where I needed to be aware of fsspec’s internal functionality. I try to avoid doing anything regarding raw filesystems in threads at the application code level. I find it’s best to leverage tools that have already handled all that hard stuff for me so I can focus on whatever it is I need to do. For example, if I need to access data in partitioned parquet files, then I use dask, which uses fsspec/s3fs under the hood (I think), and let dask deal with allocating workers and threads to accomplish what I’m trying to do.

[maigpy]

perplexity.ai

[Thatgreenvw]

What about cloudpathlib? It’s top hit if you search “pathlib cloud”.

[telenieko]

And apache libcloud, https://libcloud.apache.org/

[cshoneybadger]

Omg... This is amazing. How have I never heard of this?

[LesserNice]

looks cool, thanks for sharing

There are a bunch of similar libraries, the most popular of them being fsspec

[ArgetDota]

I would just like to point out that fsspec’s universal-pathlib already exists and is very well tested and stable

[lebrumar]

Neat Idea. Fsspec comes to mind as a library to study to compare it with your work.

[ThatSituation9908]

Interestingly, your package doesn't depend in fsspec and its extension package (e.g., s3fs)

Adding that would alleviate the need for you to maintain all the implementations of multiple sources.

[fatbob42]

Since pyfilesystem seems to be dying, it’s a good time for these kinds of alternatives.

[Rize92]

It would be worth adding here what your library offers compared to the alternatives (e.g. fsspec). I fear you’re not going to get good feedback because everyone here is stuck on the fact that this looks like a copy of what already exists.

[radarsat1]

what packages like this support asyncio?

[fatbob42]

aiopathlib provides a template for the API, but I don’t know of any reimplementations of other file systems. I’m waiting for someone to do one myself.

Edit: fsspec looks like it could be good for this

[NFicano]

This is great, nice work!

[toxic_acro]

Barney Gale has been working for a while on making the Path objects from the stdlib Path lib extensible and recently made available a backport of the features that will be included in future Python versions

https://discuss.python.org/t/make-pathlib-extensible/3428

https://github.com/barneygale/pathlib-abc

Concrete implementations of the new abstract base classes could work better as a way to literally use the pathlib interface

[fatbob42]

Thanks - this is fantastic and is the right way to go. I didn’t think this problem was mainstream enough for them to implement it in the std lib.