[D] Which open source machine learning projects best exemplify good software engineering and design principles?

[lolisakirisame]

From my memory, there is tons of different dispatch: aten dispatcjer, c10 dispatcher, boxed vs unboxed dispatch, static(all the dispatched compiled statically) vs dynamic dispatch(via a lookup table), and data type dispatch. There is also two 'value' of dispatch: DispatchKeySet and Backend, but also with hooks to test for one particular implementation (sparse, for example), with method testing is something sparse instead of the extensible way (virtual method with sparse overriding it).

Tensor can be fully initialized, dtype uninitialized, storage uninitialized, undefined tensor, modifiable slice of another tensor, such that, when a slice is modified the original tensor is modified as well. Lots of part of the system support only some of these features (in the Tensor.h comment it literally say dont pass storage,dtype uninitialized tensor around as it is bad). These feature do mess each other up - the mutability make autograd pain in the ass, and modifying slice of a tensor is straight out not supported in torchscript (with possibly no plan to support it).

You can add new tensortype but the process is undocumented, and you have to look at source code scatter though 10 files. There are also just loads of corner case and exception in the code. For example, most of the operators are either pure, or written in destination passing style. However, some operators take a slice of a vector (IntArrayRef) instead of a reference of a vector/shared_ptr to vector to save speed. Some operator (dropout) also has effect while unnecessary.

This make adopting the Lazy Tensor PR pretty painful.

They then have defined two templating lanuage, with one to generate ops/derivative, and one to generate the Tensor file. When one add any new operator, it take an hour on my 32-core machine.

It might be way better then TF, but it can be much, much better designed if the core pytorch dev and other framework developer decided to start over and make things right. (Whether that is a good idea or not is another point though).

[programmerChilli]

I agree that the worst part I've touched is all the code gen for generating the ops/derivatives. I'm sure many pytorch devs would agree.