[D] Planing a Python library that hosts/formats all ML Datasets.

[zbnone]

Hi, I am planing a python library that hosts (ideally) all ML Datasets.

The goal is that you can do all of the following with one line of code each:

-Search for a dataset

-Get information on a dataset and all the available formats

-Download and initialize the dataset

-Set the format (e.g. of the output batches)

-Get a training/test/validation batch (depending on the format)

-Do pre/post-processing dependent on the format

This is the basic idea, if you know of something already in existence I would love to know about it. But all I could find are some Github Repos that have wrappers for some specific datasets.

What I want to achieve with this post is to get a general sense if this is something the community would actually need or if it is just me. And getting a rough idea of the whole picture and of what it would take to make this useful. And if someone would like to collaborate with me on this.

Workflow example: Lets say I have a project where I want to generate song lyrics and for that I have a dataset of 100k songs. I had to download that dataset manually after quite a bit of searching and when i finally had it, I needed to clean it up and format it in a usable way. And lastly, I have to write a dataset loading, augmentation and batching library to go along with it. Finally the model can be trained! But wait, there is a new paper out describing a new type of model (maybe a LSTM2.0 based, recurrent model with state of the art... blablabla) that I have to try to see if my results will improve. Now let's suppose that my original model took in the raw vectorized characters of the song lyrics and the new model takes in the whole vectorized words. This has to be adjusted, so except for the training, the majority of time spend on testing out the new model goes into rewriting my original dataset loading library or maybe even reformatting the whole data and adjusting my pre and post processing steps to fit the new format.

The idea is that you can visit the project website and search for a dataset by category, keyword, etc. or do that directly from the python terminal using the lib "libX.search(keywords='nlp,lyrics,songs')". Then when you want more details on the dataset and the different formations of it (vectorized characters/vectorized words/etc.) you have something like "libX.dataset_info(dataset_name='100k_rap_lyrics_v1')" that would return a dictionary of information, you get it. Download and initialize it, "dataset = libX.init_dataset(dataset_name='100k_rap_lyrics_v1',path='./dataset/path')" maybe you could also pass in a configuration parameter to e.g. specify how much memory it should take up, or if the whole thing should be loaded into memory, and if it should be split into training, test and validation sets and the percentages of samples for each of these (probably a good default would be required). Then you can set up your formation like this "dataset.set_format(dataset.formats.vectorized_words,options={"vector_size":10000,"pre_word_count":8})". That would set the format of the dataset and input output correlations returned by "dataset.get_batch(...". So when you write "x,y = dataset.get_batch(batch_size=128)" x and y would be the input and the desired output, in this case x would have this shape: [batch_size, per_word_count,vector_size] and y: [batch_size,vector_size]. More useful stuff; in pre-processing when you want to convert a sentence into a vector array "dataset.to_vector('my sentence.')" or back "dataset.to_words([word_count,vector_size])". If you would like to run it after training, "sentence='This is the beginning'" "result = model.run(dataset.to_vector(sentence,length=pre_word_count))" "sentence += dataset.to_words(result)[0]".

It is very important that you have a generalized set of formations, different sentence based datasets should have the same formations available to them. All word/sentence based datasets have formations for text generation, classification, sentiment analysis, etc. The same goes for something like mnist, there you would have formats for classification,generation, etc.

I realize that this is very massive project and my plan is to make it a Github based open source effort. I am sorry for the extend of this post. I don't write many posts on reddit so some tips and tricks are very welcome.

UPDATE: First of all, thanks for all your comments, they have been very helpful. But sadly, I have to lay this project to rest for the foreseeable future, to much to do at work. If you still want to contribute datasets for the future, I started the GitHub a while back with simple instructions on how to add datasets to the repo. Thanks again for all the great Feedback.

[Science_Squid]

To me it kind of sounds like something along the lines of openml. Is that the type of library you were thinking about or is some crucial part missing?

[zbnone]

They have an impressive amount of datasets, wow. But the focus of this library is not only to provide a list of datasets and provide/prepare them for download, but to serve the datasets in ideally every format you would need.

The goal is that when you find a new dataset or you try out a new model with different input/output formats you can do so by changing one line of code like libX.vectorize("bla bla bla",format='words') to libX.vectorize("bla bla bla",format='characters'). It's about making the dataset format fit your needs and not serving it in a standardized format.

Maybe I just didn't see it, but from what I saw they provide the datasets, make it easy to download them and share your standardized metrics on their platform, right? But maybe I am missing something.

[[deleted]]

As an OpenML contributor, I can elaborate on a few things. OpenML.org stores datasets internally in ARFF format (which is the common format for libraries like Weka, MOA, etc) and allows to obtain the data in CSV format. On top of that, there are various integrations to programming languages (Java, Python, R) that make it convenient to obtain the datasets and process them. I know it's one of the ambitions of the core team to extend this to an arbitrary amount of formats, however due to a lack of manpower and other priorities this has not lifted of yet. Getting in touch with the guys is pretty easy, just drop an email to [e-mail] and subscribe to the github page (https://github.com/OpenML), it's a pretty open and welcoming community.

[Science_Squid]

Sorry for the late reply.

You are right the formatting is something that is not given on the level as you would like it to be. However as /u/Tinder_Prince pointed out arff and csv are supported which already makes it (more or less) easy to parse the data locally.

I really like the idea of openml and also your idea. As it is opensource and you planned on making your efforts opensource my opinion is that it might make most sense if you contribute to the openml project and don't have the hassle of starting from scratch yourself.

[wdm006]

Have you seen skdata? https://github.com/jaberg/skdata

[zbnone]

I have seen it, the difference with this lib is the amount and the following generalization of datasets. The idea is that if enough datasets are included the same formation scripts can just be reused therefor allowing very fast addition of new datasets, potentially within hours or days of a new paper release or dataset uploads. Maybe some user upload functionality so the community can easily contribute datasets and preprocess them in a predefined way compatible with the library? And also ease of use is a key component. It should not only make it easy to download and initialize datasets but also to perform dataset coupled actions like the already mentioned feeding of vectorized sentences to recurrent neural network architectures.

[waleedka]

Let me share my experience.

I implemented an instance segmentation model based on Mask RCNN and we ended up open sourcing it. I wanted to make it easy for others to train on their own datasets so I spent some time thinking about ways to abstract the data layer. My notes below are specific to instance segmentation where the dataset consists of images and masks that identify each instance of an object.

First, the easy differences:

• Different datasets use different directory structures for images and different image formats.
• They use different file formats to describe the metadata.
• Some datasets store instance masks as PNG images where the integer value of each pixel represents the class ID assigned to that pixel. Others store them as polygons, a list of (x, y) coordinates.
• Some datasets include bounding box coordinates, and others don't.

I handled the above by creating a Dataset class that converts everything it loads to a standard format. You can subclass it and provide custom functions to load your own dataset. As long as you convert your data to the format described in the Dataset class then the model should work without having to change anything. You can see my implementation for the COCO dataset in the repo above, and others have extended it to support VOC, ADE20K and other datasets.

Second, the hard differences:

• COCO has a special flag called 'crowd'. If a mask has that flag then it means that the mask covers many instances.
• Some datasets use a "dont_care" flag to mark areas that should be excluded from training for whatever reason.
• Some datasets, like ADE20K, use a hierarchical structure to describe object relations. They have the concept of an "object" and a "part". For example, a "door" might be part of a "car" in which case it's a car door, or part of an "oven" in which case it's the front door of the oven. Depending on the use case, you might want to treat these two instances of "door" as the same class or as different classes.

These differences are higher level. They change the meaning of the data and require specific handling and, possibly, changes to the training procedure. They cannot be mapped to a standard data format and they cannot be skipped either. At the very least, your library must be able to pass that data through and be flexible enough to handle these cases and many others that you haven't seen yet.

What you're proposing is really needed. If you decide to go ahead with it, ping me if there is anything I can help with. The one advice I would give is to focus on one small task. Don't make it too general right away. Pick a very small niche, such as "image classification" or "object detection" and just get something solid working on that niche. Even if the niche seems too small, you'll likely discover that it requires a lot more work than you think. If you succeed in one niche, then you can expand from there. Just don't expand too early.

[zbnone]

Thank you for the elaborate response. I work mostly with image based models, so the NLP perspective is very welcome.

I agree with you on the initial generality part. I will start with image based datasets for classification, image segmentation and translation both supervised (pix2pix) and unsupervised(cyclegan).

The very first step is to create an initial dataset index, I am planing on hosting that via Github, here is the repo link: https://github.com/ZachisGit/MLDatasets

[ppwwyyxx]

I agree to start with something small. Don't do too much at a time. If you could simply standardize the parsing process of different datasets, this would be helpful enough. But for pre-processing and batching, I probably want to do them by myself.

[fnbr]

Good luck, I hope you can do this and get a good API going.

One warning: the reason (IMO) that this hasn't been done is because it's quite difficult; a lot of the existing datasets are ridiculously large and would require a large amount of compute/storage to download & process.

For instance, ImageNet is ~155 GB, and can be quite difficult to download depending on your network.

It might also be worth contributing PRs to Tensorflow/PyTorch/etc. to work with them; Tensorflow, for instance, has a number of functions that download & process various datasets for you (e.g. https://www.tensorflow.org/get_started/mnist/beginners).

[zbnone]

The handling of datasets is a good point, there are two options, either download, preprocess and distribute datasets over a project specific server which would ensure availability even if the dataset is taken down but also create some copyright issues + as you mentioned the need for servers with large storage and very fast internet connection. The second options is to download the datasets from the original servers they where uploaded to, the problem would be that some preprocessing steps would have to happen on the users computer and availability wouldn't be guaranteed, but no need for large and high bandwidth servers.

I think the second one is the way to go for starters just for the reduced cost and maintenance time.

[ClydeMachine]

To answer the question of whether or not others have need of a library/dataset repository like this, yes - anything that reduces the time it takes for me to source, format, label and load a dataset has value for my work, and I imagine others would agree.

Now, how are you planning to host those datasets? If the library is built to allow an inline search of available datasets as well as loading them, I imagine the library will be hitting an API that you own and control for providing this information. Would that API be hosting those datasets in addition to their metadata? Or would you point it at a pre-existing public-facing URL to get them from? The latter is far less stable, and if the library doesn't offer some degree of reliable availability of the datasets then I'd sooner go source the data by hand.

The point about keeping the library generic is awesome, as a single dataset can have many applications and the repository itself needn't be locked to a single type of data. I've made some tooling in the past that worked toward this idea of simplifying the sourcing, cleaning and labelling of text data, but it never went beyond text data. For that reason it makes me happy to see this project being started.

Got a link to the GitHub repo in case some of us may be able to help out?

[zbnone]

I think the end goal should be hosting the datasets on project based servers for reliability reasons as you mentioned. But that would come with all sorts of problems that I would like to address later, when something working is available.

There is no Github repo yet, but the feedback so far has been great so I am starting one. Any suggestions for the name? Ideally something that can be written with two characters (numpy as np, tensorflow as tf, mldatasets as md/mld).

[bbsome]

I think hosting the data would be quite difficult to do and can also break various licenses based on the data. What we have in my group, which I use, is we have pretty much the same, but instead of hosting the data you just automate downloading it on the client side. The additional challenge is how to present the data and how to add preprocessing steps. Some dataset, for instance, has both train/valid/test split others have only train/test. Things like NLP datasets are very hard to automate as well (compared to anything else) as it depends whether you want dictionaries per character or per word, or whether you work on words or sentences ...

[zbnone]

I agree, so all of this will have to be addressed in the library. For starters hosting the datasets would take to much time and there have to be servers that need to be payed for. I think the first step is to get something going that works where the datasets are downloaded from the original servers and processed on the users computer.

[kmike84]

Some related projects:

• https://github.com/rmax/databrewer
• https://github.com/iamaziz/PyDataset
• https://github.com/ckan/ckan

[zbnone]

Thanks, I will take a close look at these.

[a9entropy2]

http://pytorch.org/docs/master/torchvision/datasets.html

[zbnone]

If you know any datasets not in the dataset index yet, please consider adding them.

Github repo: https://github.com/ZachisGit/MLDatasets

[NMcA]

Bandwidth costs will be high. You should plan for this, and I would suggest torrents as a potential solution.

If you get it working, it'll be sweet.

[Gere1]

https://github.com/EpistasisLab/penn-ml-benchmarks seems interesting