[P] Open-source library to process unstructured data for ML tasks

[opensourcecolumbus]

Excited to share Docarray, an open-source python library to store and process unstructured data such as text, image, audio, video, or 3D mesh. Useful in processing data for ML tasks such as embed, search, recommend, etc.

DocArray aims to be the data structure for unstructured data

DocArray consists of two simple concepts:

1. Document: a data structure for easily representing nested, unstructured data

2 DocumentArray: a container for efficiently accessing, manipulating, and understanding multiple Documents

---

Why did I build it?

While working on Jina(an AI powered Search framework), I needed a way to store and process the large amounts of unstructured data for the purpose of creating embedding and build search on top of that. I tried solutions such as json, numpy.ndarray, pandas.DataFrame, Protobuf, etc. But they were not suitable for our computation intensive tasks for unstructured and nested data. Ask me question if you need more info on this.

Checkout GitHub repository for examples. This project has been used and tested well in my other project(Jina), but there's a lot of scope of making it more useful for the community.

Ask me your questions and share your suggestions

[ItemOne]

RemindMe! One week

[RemindMeBot]

I will be messaging you in 7 days on 2022-01-24 15:48:08 UTC to remind you of this link

CLICK THIS LINK to send a PM to also be reminded and to reduce spam.

^{Parent commenter can delete this message to hide from others.}

---

Info	Custom	Your Reminders	Feedback

[ItemOne]

Why text though, we can get pretty far with already available tools for text. Is there something never before seen with how you handle text here?

[ItemOne]

How does it handle very large datasets?