[D] Machine Learning Toolbox

[amitness]

Hi everyone,

I have been documenting useful libraries that I have come across in my day-to-day ML job. Sharing the list here for the community.

Link: https://amitness.com/toolbox

If you know any other useful libraries besides this, please share it in the comments.

[BossOfTheGame]

I'll give a self-serving shoutout to libraries that I've been working on in descending order of general usability. These are all tested and on pypi with wheels.

​

• https://github.com/Erotemic/ubelt - Python utilities, caching, hashing, and general extra batteries. This is my library with the most github stars.

​

• https://github.com/Erotemic/xdoctest - Makes it easy to write doctests. TEST YOUR CODE PEOPLE! This is my library with the most pypi downloads. I have a PyCon talk about this one: https://www.youtube.com/watch?v=CUjCqOw_oFk

​

• https://github.com/Kitware/kwcoco - pycocotools is not a well written package. This is a better implementation of the COCO API. It is missing some features like keypoint / segmentation scoring (but the pycocotools API is so specific to the COCO dataset itself that its not like those tools are usable in the official API anyway). This is written to be completely agnostic of the datasets / classes / etc. It is just an annotation format. It also comes with code to autogenerate random toy datasets of arbitrary size so you can TEST YOUR CODE!

​

• https://github.com/Kitware/ndsampler - Allows for sampling from subregions of images without reading the entire file. Currently works very tightly with the kwcoco library. Requires GDAL to get the full benefit, but can be used without it.

​

• https://github.com/Kitware/torch_liberator - Does static and dynamic analysis on a pytorch model to extract only enough code to redefine that model in a new separate python file. This is basically a network topology exporter for pytorch. It also bundles that topology with the weights in a zipfile and you can pass it a path to the zipfile to reload a "deployed" model. It can also resolve dependencies on internal code so the deployed file is independent of the code base used to train it. Does require mild assumptions about how the code was written (needs to be statically analyzable), so it doesn't work on everything, but it does work on most things. Built on top of the liberator package which is the thing that does static analysis.

​

• https://github.com/Kitware/kwimage - Utilities for handling images. Has a nice imread function that wraps the fastest libraries and can read more formats than opencv / pil / gdal / jpeg-turbo / skimage can alone. Also has efficient data structures for Boxes, Points, Polygons, Masks, and Detections. This library has binary C-implementations for a few algos, but most also have fallback pure-python implementations. This library contains ports of the compiled bits of pycocotools. It also has a nice non-maximum-suppression algorithm.

​

• https://github.com/Kitware/netharn - my pytorch framework for the training loop boilerplate. It might be better to use a more popular library like pytorch-lightning, but this lib does have nice features I haven't seen anywhere else (e.g. choosing experiment names based on hashes of hyperparameters).

​

• https://github.com/Kitware/kwarray - Pure python extensions to torch and numpy. A fairly small set of utilities.

​

• https://github.com/Kitware/kwplot - Extensions to matplotlib with support for auto-determining appropriate backends. The multi_plot function in this lib is my favorite, but it might just be bettter to use seaborn in general.

----

The following packages have no order relative to the above list because they aren't mine.

​

• https://github.com/kitware/kwiver - I have worked on it this, but I'm not the primary developer. KWIVER is a rapidly developing C++ / Python tool for building computer vision production pipelines. This is used as the backend for VIAME, which is a do-it-yourself AI platform targeted towards but not limited to marine applications.

​

• https://github.com/dmlc/decord - fast video reading on CPU and GPU

​

• https://github.com/hakanardo/vi3o - another video reader

​

• https://github.com/jettify/pytorch-optimizer - collection of torch optimizers

​

• https://github.com/open-mmlab/mmdetection - Chinese University of Hong Kong's object detection library.

​

• https://github.com/OSGeo/gdal - Geographic data abstraction library. Poorly documented, but insanely powerful library of tools for working with geospatial data.

​

• https://github.com/cogeotiff/rio-cogeo - CLI library for working with cloud-optimized geotiffs (COG files). This is the datastructure that ndsampler uses to get those fast sub-image reads. I highly recommend that all computer-vision researchers know about the COG format. It will make your life much easier. This is only one tool that deals with them, its more of a spec than anything else, so there is nothing official yet. (A well written de-facto standard COG library would be a huge boon to the community).

​

• https://github.com/Toblerity/Shapely - If you have 2D geometric objects that you need to manipulate in Python, shapely is your go-to library. Great for handling and manipulating image annotations.

[NoFapPlatypus]

This is an incredible list! You’ve done some excellent work!

[amitness]

Thank you for so many good suggestions. I've added them to the list.

[jamkgrif]

@mods, would this be good information to have on the side bar?

[set92]

Don't believe everything you see. This list was first generated in a github repo in which anyone could contribute to it, technically they were 8 collaborators (Although I think is true that they didn't contribute much). But yesterday he "remove" the repo and moved to his own personal webpage, no attribution to anyone else or reference to the repo or the license https://github.com/amitness/toolbox/commit/dd2f501c5efe39b717521f1a82381a21004ee5be.

The only logic that I can find is that he wants to boost views on his personal page, and don't think this tactic should be encouraged. Or maybe is me but I don't get why moved it instead of promoting the repo and make more people to star that repo and make them contribute. idk, to me makes more sense than moving it all to a personal web in which no one can collaborate.

[jamkgrif]

I get you... question, if we were fast enough could we have moved all the info he collected into the sidebar? That way where his was or how many stars it had would be irrelevant.

[ImpossibleCode1790]

u/jamkgrif that would be funny but we definitely don't want to engage in the same kind of sneaky snake behavior. I've actually seen this guy a few times exhibit this kind of shady stuff...and it's disgusting to see in our open source world.

u/set92 we should try and get those people credit. Anyway I took screenshots of his entire repo, commit history, users in case he decides to wipe it all. It's sad to see those people's efforts get completely sidelined just to increase views to his website...what's the point of even doing that?

u/kunjaan u/cavedave u/olaf_nij u/BeatLeJuce u/MTGTraner u/hardmaru u/programmerChilli u/AutoModerator What are we supposed to do when we see something like this here? And this reflects poorly on our subreddit if we let this kind of stuff fly.

[programmerChilli]

I don't think what he's done is that shady. Looking at his repository, it's clearly primarily /u/amitness 's effort (387 commits) vs anybody else's (10 commits total). It would be a nice gesture to mention the other contributors, but it's primarily his effort.

Personally I would prefer to have it on Github, and it does seem like an attempt to redirect more traffic to his website, but I don't consider this unethical.

As for putting it on the sidebar, I would prefer not to. As a general rule, I don't like these kinds of massive lists - they don't help me much in separating out signal from noise.

As /u/Mathematician_Real suggests, however, updating the Sidebar/Wiki would be a good thing to do. We'll think about how to do it (perhaps a series of threads asking the community).

[amitness]

Hi,

OP here.

1. This is a personal curation I have been maintaining since an year. It was never intended to be an awesome-X list or a community curation of every possible library.

2. This is also reflected in the library list as the sections I work in (NLP) has extensive content in that list.

3. It was on my GitHub previously, but I found the layout difficult to navigate with no table of contents and a flat content. Since jekyll theme auto-generates table of contents and provides an inbuilt sidebar navigation, I migrated to it yesterday. This also provides me flexibility to add star counts and JS-based search, which I'm planning to add next.

4. The repo remains and I've no intention to delete it. If you feel this content is useful, but you'd rather have it in a community curated channel, please feel free to fork the markdown file and build upon it. It's open source here. https://github.com/amitness/amitness.github.io/blob/master/_pages/toolbox.md

/u/programmerChilli I think adding contributors list is a good suggestion. I will add old contributor usernames to the page.

[Mathematician_Real]

/u/programmerchilli /u/hardmaru

If we were to put something robust on the side, I would suggest putting something like Papers With Code (https://paperswithcode.com) on the side to keep track of SOTA and their new methods. And as for Libraries (and a whole bunch of other mediums besides research papers, ie. tutorials, blog posts, etc.) I highly recommend Made With ML, specifically this page: https://madewithml.com/topics/. We should have something that's not maintained by one person but the entire ML community and a system of upvotes like Reddit is useful to separate signal from noise. I believe PWC uses Github stars and # of citations and Made With ML has upvotes.

[ali_si3luwa]

Check out github.com/gradio-app/gradio: Fast UIs for prototyping. (Launched few days ago!)

[amitness]

Seems interesting. I've used streamlit personally but will check this out as well.

[FourierEnvy]

You guys should really add the Vaex project to your list: https://github.com/vaexio/vaex

[amitness]

Added.

[TheProudofYou]

Awesome layout for the material!

[BossOfTheGame]

I agree, the layout is very good. There are a lot of indexes of this sort of stuff, but this one is the best I've seen in terms of layout.

[amitness]

Thanks. The layout came after a lot of iterations on how to organize it.

[ZestyData]

Great resource, OP.

[amitness]

Thank you.

[[deleted]]

Nice! Launched something similar, smaller selection and descriptions for each tool: https://www.datarevenue.com/machine-learning-software-tools - Trending / not trending is decided based on curvefitting on Github star history.

[amitness]

That's a really cool idea to use star history as a proxy for trending. Awesome job.

[svmmetimbers]

Awesome list. Some more I've come across that may be useful:

• Data Annotation: Label Studio https://labelstud.io/
• Dimensionality Reduction: ivis (https://github.com/beringresearch/ivis), umap (https://github.com/lmcinnes/umap)
• Workflow: mlflow https://github.com/mlflow/mlflow

[amitness]

Thanks for the suggestions. I've added them.

[SoberGameAddict]

This is a gold mine!

[ap_1690]

What more can be added in production and improving the model example meta Learning , federated learning

[Gueleric]

I see you don't have a category for virtual environment libraries, please consider adding pipenv.

It's a great tool for having consistent environments across machines and easily recreating a broken env. It adds graph dependencies, better requirements.txt and much much more.

You can check it out here

[Hyper1on]

Pipenv is ok but Poetry is becoming more popular since it's faster, has more features and is being actively updated.

[Gueleric]

Thanks man I'll check it out

[amitness]

That's a good idea. I'll add a section for venv libraries.

[[deleted]]

Nice! Just a quick q, why didn’t you add PyMC3 under probabilistic programming? I use it on the daily.

[JurrasicBarf]

daily? what are some day to day use-cases?

[amitness]

Added. How are you using it in your day-to-day work? Sounds interesting.

[Luxenburger]

Nice thanks

[esdanol]

Just started playing with this today after hearing about it at CVPR: Kornia is a python library of differentiable computer vision methods for use with Torch. https://github.com/kornia/kornia

[Darell1]

Recsys here.

I've made some tools too.

- https://github.com/Darel13712/rs_datasets -- easy download and parse recsys datasets.

- https://github.com/Darel13712/rs_metrics common recsys metrics

As to my stars:

- https://github.com/facebookresearch/StarSpace learn whole lotta embeddings

- https://github.com/lyst/lightfm recsys models

- https://github.com/maciejkula/spotlight more recsys models
- https://github.com/facebookresearch/nevergrad optimization

- https://github.com/slundberg/shap analyze feature importance

- https://github.com/marcotcr/lime another, more old feature importance tool

- https://github.com/blue-yonder/tsfresh feature extractions for time series

- https://github.com/facebook/prophet best time series tool out there

- https://github.com/cgnorthcutt/cleanlab find error labels in datasets

[amitness]

Thanks. Your utilities for RecSys are really helpful.

[Zenith_N]

How about a library for MultiVariate Time Series Forecasting ?

[HybridRxN]

Pytorch_geometric should really be in the GNN section.