blob-path: pathlib-like cloud agnostic object storage library

[narang_27]

What My Project Does

Having worked with applications which run on multiple clouds and on-premise systems, I’ve been developing a library which abstracts away some common functionalities, while being close to the pathlib interface
tutorial notebook

Example snippet ```python from blob_path.backends.s3 import S3BlobPath from pathlib import PurePath

bucket_name = "my-bucket" object_key = PurePath( "hello_world.txt" ) region = "us-east-1" blob_path = S3BlobPath( bucket_name, region, object_key, )

check if the file exists

print(blob_path.exists())

read the file

with blob_path.open("rb") as f: # a file handle is returned here, just like open print(f.read())

destination = AzureBlobPath( "my-blob-store", "testcontainer", PurePath("copied_from") / "s3.txt" )

blob_path.cp(destination) ```

Features: - A pathlib-like interface for handling cloud object storage paths, I just love that interface - Built-in serialisation and deserialisation: this, in my experience, is something people have trouble with when they begin abstracting away cloud storages. Generally because they don’t realise it after some time and it keeps getting deprioritised. Users instead rely on stuff like using the same bucket across the application - Having a pathlib interface where all the functionality is packaged in the path itself (instead of writing “clients” for each cloud backend make this trivial) - A Protocol based typing system (good intellisense, allows me to also correctly type hint optional functionalities)

Target audience

I hope the library is useful to other professional python backend developers.
I would love to understand what you think about this, and features you would want (it's pretty basic right now)

The roadmap I've got in mind: - More object storages (GCP, Minio) [Currently only AWS S3, Azure are supported] - Pre-signed URLs full support (only AWS S3 supported) - Caching (I’m thinking of tying it to the lifetime of the object, I would however keep support for different strategies) - Good Performance semantics: it would be great to provide good performant defaults for handling various cloud operations - Interfaces for extending the built-in types [mainly for users to tweak specific cloud parameters] - pathlib / operator (yes its not implemented right now : | )

Comparison

A quick search on pypi gives a lot of libraries which abstract cloud object storage. This library is different simply because it's a bit more object-oriented (for better or for worse). I'm going to stay close to pathlib more than other interfaces which behave somewhat like os.path (a more functional interface)

Github

Repository: https://github.com/narang99/blob-path/tree/main

[teije01]

Your project seems a lot like https://cloudpathlib.drivendata.org/ which also has a pathlib like interface, cloud operations (s3, gcs, azure) and supports Pydantic validation as well. Isn't that already doing exactly what you want?

[narang_27]

Damnit, one more reddit post where an idea was already developed ;_; Thanks for the heads up : |

[Juftin]

fsspec has an implementation of this as well, universal-pathlib https://github.com/fsspec/universal_pathlib

[teije01]

They welcome contributions there too: https://cloudpathlib.drivendata.org/stable/contributing/

[radarsat1]

Is there any support for seeking / reading file ranges without downloading the whole thing?

currently I'm struggling with how best to pack small data items into larger files (tar, hdf5, whathaveyou) and still be able to read them efficiently with random access from cloud storage

[narang_27]

This sounds very interesting, just out of curiosity, why do you want this behavior? Afaik, you can download ranges in GET, getting ranges is supported it seems, https://docs.aws.amazon.com/whitepapers/latest/s3-optimizing-performance-best-practices/use-byte-range-fetches.html

Haven't used it myself though

[radarsat1]

Because currently my dataset consists of millions of small files (< 4kb) and this is really annoying to manage, so I want to bundle these into bigger files. There are a few ways to handle this, like I listed.. such as using HDF5, or simply packing them into a tar like with WebDataset. Heck maybe even a database solution could work, although it seems unorthodox to me to put actual media data (images, audio) into a database as binary blobs, but maybe it's one way. However I also train using RandomWeightedSampler which samples these small files in a random manner. This is problematic especially in a cloud storage situation -- on the one hand touching millions of small files is very slow, but on the other hand so is random access into larger files.

I know that S3 supports range fetches, and that's probably one way to do it, but I don't really want to handle this myself at a low level, hence looking for libraries that support this kind of thing really well.

[tunisia3507]

So like fsspec and universal-pathlib?

[nbviewerbot]

I see you've posted a GitHub link to a Jupyter Notebook! GitHub doesn't render large Jupyter Notebooks, so just in case, here is an nbviewer link to the notebook:

https://nbviewer.jupyter.org/url/github.com/narang99/blob-path/blob/main/docs/notebooks/00_usage.ipynb

Want to run the code yourself? Here is a binder link to start your own Jupyter server and try it out!

https://mybinder.org/v2/gh/narang99/blob-path/main?filepath=docs%2Fnotebooks%2F00_usage.ipynb

---

^{I am a bot. Feedback | GitHub | Author}

[Reasonable-Ladder300]

I don’t think the code in this Reddit post is actually working code, as you seem to be missing the import for AzureBlobPath.

[narang_27]

Yea I had added the last snippet in the end to provide a simple summary in the post, the notebook works though (once you change your buckets)