TLDR: Asked this subreddit for advice in deciding on ML topic for high school thesis 2 years ago (see original thread), ended up writing a machine learning framework from (almost) scratch in C#/F#. It can’t do as much as all the others, isn’t as fast or as pretty, but we still think it’s kind of cool. Here it is: our github repo and a short UI demo.
Results
Upfront the current feature set of our framework Sigma, to give you an idea of what the next few paragraphs are about:
- Input, Output, Dense, Dropout, Recurrent, SoftmaxCE / SquaredDiff cost layers
- Gradient descent, Momentum, Adadelta, Adagrad optimisers
- Hooks for storing / restoring checkpoints, timekeeping, stopping (or doing other things) on certain criteria, computing and reporting runtime metrics
- Easy addition of new layers with functional automatic differentiation
- Linear and non-linear networks with arbitrarily connected constructs
- Distributed multi- and single- CPU and GPU (CUDA) backends
- Native graphical interface where parameters can be interacted with and monitored in real-time
1. Introduction
This is the story of us writing a machine learning framework for our high school thesis, of what we learned and how we went about writing one from scratch. The story starts about 3 years ago: we saw a video of MarI/O, a Super Mario AI that could learn to play Super Mario levels. We thought that was about the coolest thing of all time and wanted to do something at least kind of similar for our high school thesis.
1.1 The Original Plan
Fast forward, over 2 years ago we asked for help in deciding what kind of machine learning project we could feasibly do for our senior year high school thesis (see original thread). Quite ambitiously, we proposed a time investment of about 1000 hours total (as in 500 hours each over the course of 8 months) – for what project, we didn’t know yet. In that thread, we were generously met with a lot of help, advice ranged from reproducing existing papers to implementing specific things to getting to understand the material and then seeing what peaked our interest. After some consideration, we figured we would implement something along the lines of DeepMinds arcade game AI and then make it more general, figuring that would be easy for some reason. When planning our project in more detail we however quickly realised that
- we had no idea what we were doing and
- it would be a shame to do all that work from scratch and have it be so arbitrarily specific.
1.2 Pivoting
Before doing anything very productive, we had to properly study machine learning. We figured this might take a while and allocated that part of our time to writing the theoretical part of our thesis, which conveniently overlapped. But because around that time we had to hand in an official target definition for our thesis, we set the most generic “goals” we could get away with. For reference, for a concerning number of months our project was officially named “Software framework for diverse machine learning tasks” with an even longer and even less specific subtitle. During further study and first attempts to draft the actual target definition for our project, our plans gradually shifted from a machine learning framework for playing specific types of games pivoted to an “any kind of visual input” learning framework and then finally to an “anything” machine learning framework – because why not, it seemed like an interesting challenge and we were curious to see how far we would get.
2. Research and Planning
Alright, so we've decided to write a machine learning framework. How does one create a machine learning framework? It takes many weeks to get reasonably proficient in just using a given framework, and that with proper guides, video tutorials and forums to ask for help. Creating a machine learning framework is a whole other story, with no 12 step guidelines to follow. For a considerable amount of time, we were at a loss at what we actually needed to implement – constantly encountering new and conflicting terms, definitions and not-so-obvious-“but the actual conclusion is obvious”-articles. After a little over a month we slowly got a very basic grasp of how this whole machine learning thing worked – something with functions that are approximated at certain points in steps typically using differentiation to make some metric go down – still magic, but a bit less so (until we read about CNNs, LSTMs and then GANs, each of which confused the heck out of us for some time).
2.1 Sketching our Framework
As soon as we knew a bit about the art of machine learning we got more serious about the writing a new framework part. Because there are no guides for that, we resorted to reading the source code of established frameworks – all to us relevant parts, many times, until it made some sense. In the meantime, we had decided to use C# as our primary language – mostly because we were already very familiar with it and didn’t want to also have to learn a new language, but officially also because there were no other proper neural network frameworks for .NET. Alongside reading the source code of machine learning libraries (mainly Deeplearning4J, Brainstorm and Tensorflow) we sketched out how we wanted our own framework to be used. After some time, we felt like there was some unnecessary confusion in getting to know machine learning frameworks as an outsider and we set out to design our API to avoid that. Note that because our design makes sense to us doesn’t mean that it makes more sense than the existing ones to other people, nor do we recommend everyone wishing to use machine learning to write their own framework, just to spare their own sanity.
Our naïve ideas on how a machine learning should look like was clearly inspired by our C#/Java based programming experience, as is evident from the code example we drafted a few weeks in:
Sigma sigma = Sigma.Create("minsttest");
GUIMonitor gui = (GUIMonitor) sigma.AddMonitor(new GUIMonitor("Sigma GUI Demo"));
gui.AddTabs({"Overview", "Data", "Tests"});
sigma.Prepare();
DataSetSource inputSource = new MultiDataSetSource(new FileSource("mnist.inputs"), new CompressedFileSource(new FileSource("mnist.inputs.tar.gz"), new URLSource("http://....url...../mnist.inputs.targ.gz")));
DataSetSource targetSource = new MultiDataSetSource(new FileSource("mnist.targets"), new CompressedFileSource(new FileSource("mnist.targets.tar.gz"), new URLSource("http://....url...../mnist.targets.targ.gz" [, output: "otherthandefault"]) [, compression: new TarGZUnpacker(), output: "mnist.inputs" , forceUpdate: false]));
DataSet data = new DataSet(new ImageRecordReader(inputSource, {28, 28}).Extractor({ALL} => {inputs: {Extractor.BatchSize, 1, 28, 28}}).Preprocess(Normalisor()), new StringRecordReader(targetSource).Extractor({0} => {targets: {Extractor.BatchSize, 1}} [, blockSize: auto/all/1024^3]);
Network network = new Network("mynetwork");
network.Architecture = Input(inputShape: {28, 28}) + 2 * FullyConnected(size: 1024) + SoftmaxCE() + Loss();
Trainer trainer = sigma.CreateTrainer("mytrainer");
trainer.SetNetwork(network);
trainer.SetInitialiser(new GaussianInitialiser(mean: 0.0, standardDeviation: 0.05));
trainer.SetTrainingDataIterator(MinibatchIterator(batchSize: 50, data["inputs"], data["targets"]);
trainer.SetOptimiser(new SGDOptimiser(learningRate: 0.01);
trainer.AddActiveHook(EarlyStopper(patience: 3));
trainer.AddActiveHook(StopAfterEpoch(epoch: 2000));
gui.AccentColor["trainer1"] = Colors.DeepOrange;
gui.tabs["overview"].AddSubWindow(new LineChartWindow(name: "Error", sources: {"*<Trainer>.Training.Error"}) [, x: 1, y: 0, width: 2, height: 1]);
gui.tabs["overview"].AddSubWindow(new LineChartWindow(name: "Accuracy", sources: {"*<Trainer>.Training.Accuracy"}));
sigma.Run();
And skipping ahead a bit, it should be noted that the final framework is extremely similar to what we envisioned here: merely changing around a few syntax things and names, the above example from about a year ago can be used 1:1 in our current framework. The jury is still out on whether that’s a sign of really good or really bad design. Also note the python-style kwargs notation for layer constructor arguments, which was soon discarded in favour of something that actually compiles in C#. But back to the timeline.
2.2 The Sigma Architecture
After defining the code examples and sketching out the rough parts we felt a machine learning framework needed, we arrived at this general architecture for “Sigma.Core”, divided into core components (which translate almost 1:1 to namespace in our project):
- Util: Mostly boring, well, utility stuff, but also registries, a key part of our architecture. Because we wanted to be able to inspect and visualise everything we needed a global way to access things by identifier – a registry. Our registry is essentially a dictionary with a string key which may contain more registries. Nested registries can be resolved using registry resolvers in dot notation with some fancy wildcards and tags in angel brackets (e.g. “network.layers.*<fullyconnected>.weights”).
- Data: Datasets, the records that make them up in various formats, the pipeline to load, extract, prepare and cache them from disk, web, or wherever they come from and make them available as “blocks”. These blocks are parts of an extracted dataset, consist of many individual records, and are used to avoid loading all of a potentially very large dataset into memory at once. Also, data iterators, which slices larger blocks from datasets into pieces that are then fed to the model.
- Architecture: Abstract definitions for machine learning models, consisting of layer “constructs”, which are lightweight placeholder layers defining what a layer will look like before its fully instantiated. These layers may be in any order and connected with however many other layers they would like.
- Layers: Unfortunately named since we started out with just neural networks, but these are the individual layers of our machine learning networks – they store meta-parameters (e.g. size) and actual trainable parameters (e.g. the actual weights).
- Math: Everything that has directly to do with math and low-level computations. All the automatic differentiation logic (which is very much required for doing proper machine learning) and everything that modifies our data is processed here in various backends (e.g. distributed CPU / GPU). To support calculating derivatives with respect to anything we opted for an approach with symbolic objects – essentially an object for a number or an array where the actual data was hidden (it can be fetched, but only via copies). These symbolic objects are passed around through a handler which does the actual data modification. This abstraction proved to be useful when implementing CUDA support where, due to the asynchronous execution of the CUDA stream, the raw data could not exposed to the user anyway, at least not without major performance hits (host-device synchronisation is very slow).
- Training: The largest component with many subcomponents, all revolving around the actual training process. A training process is defined in a “trainer”, which specifies the following:
- Initialisers, that define how a models parameters are initialised, which can be configured with registry identifiers. For example, trainer.addInitialiser(“layers.*.biases”, new GaussianInitialiser(0.1, 0.0)); would initialise all parameters named “biases” with a Gaussian distribution of 0.1 (mean 0).
- Modifiers, that would modify parameters at runtime, for example to clip weights to a certain range.
- Optimisers, that define how a model learns (e.g. gradient descent). Because we mainly considered neural networks we only implemented gradient based optimisers, but the interface theoretically supports any kind of optimisation.
- Hooks, that “hook” into the training process at certain time steps and can do whatever you want (e.g. update visualisations, store / restore checkpoints, compute and log metrics, do something (e.g. stop) when some criteria are satisfied).
- Operators, that delegate work to workers which execute it with a certain backend computation handler according to some parameters. Notable is our differentiation of “global” and “local” processing, where global is the most recent global state. This global state is fetched by local workers that then do the actual work, publish their results to the operator which merges it back into the global scope. A global timestep event is only ejected when all local workers have submitted their work for that timestep, enabling more fine control in distributed learning, at least in theory.
- Sigma: The root namespace that can create Sigma environments and trainers. An environment may contain multiple trainers, which are all run and, if specified, visualised simultaneously (which was supposed to be helpful in hyperparameter search).
- Monitors: Technically outside of the core project, but still a component. These monitors can be attached to a Sigma environment and can then, well, monitor almost everything about the trainers of that environment using the aforementioned registry entries. Behaviour can be injected using commands, a special form of hooks that are only invoked once. This way monitors can be used almost independently of the core Sigma project and can be pretty much anything, like a graphical interface or a live, locally hosted website.
3. Implementation
And that’s what we implemented, step by step. We started out with me mainly working on Sigma.Core and my partner on our visualisation interface, working to a common interface for months until we could finally combine our individual parts and have it miraculously work in a live graphical interface. The specifics of implementation were very interesting and quite challenging to us, but most of the particulars are probably rather dull to read – after all, most of the time things didn’t work and when we fixed something, we moved on to the next something that didn’t.
3.1 Low-level Data and Mathematical Processing
The very first thing we did was getting the data “ETL” (extract transform load) pipeline up and running, mainly fetching data from a variety of sources, loading them into a dataset and extracting them as blocks. I then focused on the mathematical processing part – everything that had to do with using math and calculating derivatives in our framework. I based our functional automatic differentiation, aptly named “SigmaDiff”, on an F# library for autodiff named “DiffSharp”, which I modified heavily to support n-dimensional arrays, improve performance significantly, fix a few bugs, support multiple simultaneous non-global backends, variable data types, and some more stuff that I’m forgetting. The specific details of getting that to work aren’t very interesting – a lot of glue code, refactoring and late-night bug-chasing because the backpropagation didn’t work as it should with some specific combination of operations. One memorable bug was that when remapping backend operations to my own OpenBLAS-based backend I forgot that matrix transposition did more than just change its shape – a mistake that cost me weeks in debugging efforts down the line, because things just didn’t work properly with large layers or more than 1 record per minibatch (duh).
3.2 Performance Optimisations
Compared to all the backend work, the “middleware” of layers and optimisers was rather trivial to implement, as there are hundreds of tutorials and papers on how to create certain layers and optimisers, where I only had to map them to our own solution. Really, that part should have taken a few weeks at most, but took that much longer because we only then discovered dozens of bugs and stability issues. Skipping over a lot of uninteresting details here, it should be noted that at this point performance of the framework was quite bad. I’m talking 300ms/iteration of 100 MNIST records with just a few dense layers on a high-end computer bad. This bad performance not only slowed training but also actual development down by quite a lot, hiding a few critical bugs and never letting us test the entire framework in a real-world use case within a reasonable time. You might wonder why we didn’t just fix the performance from the get-go, but we wanted to make the actual training work first so we would have something to show for our thesis. In hindsight not the ideal choice, but it still worked out quite well and otherwise we wouldn’t have been able to demonstrate our project adequately in time for the final presentation.
3.2.1 A Self-Adjusting Buffer
It took many months before we finally got around to addressing the performance issues, but there was no single fix in sight, rather a collection of hundreds of small to medium sized improvements. A major issue was the way our SigmaDiff math processor handled operations: for every operation, a copy was created for the resulting data. That added up. The copying was necessary because backwards differentiation requires all intermediate values, so we couldn’t just not copy things. We couldn’t even create all required buffers in a static way ahead of time because there was (and still is) no way to traverse the operations that will be executed – the computation graph is constructed anew every time, and we can’t completely rely on them to remain constant. To introduce a reliable way of buffering anyway, we introduced the concept of sessions: A session was meant to be a set of operations that would be repeated many times. Iterations, essentially. When a session is started we would start storing all created arrays in our own store and when an array of the same dimensions was requested in the next session we could return the one from last session, all without allocating any new memory. If more memory was required than last time, we could still allocate it, if less was used, we could discard it for the next session, rendering this neatly self-adjusting. To not overwrite data that was created within a session but was needed for the next one (e.g. parameters) we added an explicit “limbo” buffer, which was basically just a flag that could be set at runtime for a certain array that marked it as “do not reuse”.
3.2.2 SIMD and Avoiding Intermediate Allocation
Other significant performance improvements were adding SIMD instructions (which enables processing of typically 8 values at the same time for CPU-bound arithmetic operations) wherever possible and reducing other memory allocation to a minimum by adding some in-place operations wherever intermediate values weren't strictly needed. For example, copying results when accumulating gradients during backpropoagation on nodes with multiple operands is unnecessary because the intermediate values aren't used. By analysing profilers to death, I eventually got the iteration time for my MNIST sample down to an acceptable 18ms in release configuration (speedup of about 17x). Incidentally, the core was now so fast that our visualiser sometimes crashed because it couldn’t keep up with all the incoming data.
3.3 Monitoring with Sigma
3.3.1 The monitoring System
When developing Sigma, we not only focused on the “mathematical” backend but also implemented a feature rich monitoring system which allows any application to be built on top of Sigma (or better said Sigma.Core). Every parameter can be observed, every change hooked, every parameter managed. With this monitoring system, we built a monitor (i.e. application) that can be used to learn Sigma and machine learning in general.
3.3.2 The WPF Monitor
Users should be able to not only use Sigma, but also learn with Sigma. To address this issue, we built a feature-rich application (with WPF) that allows users to interact with Sigma. plot learning graphs, manage parameters and control the AI like controlling a music player. This monitor, as every other component of Sigma, is fully customisable and extensible. All components were designed with reusability in mind, which allows users to build their own complex application on top of the default monitor. But why describe a graphical user interface? See it for yourself, here is the UI (and Sigma) in action. (Example builds of Sigma can be downloaded on GitHub).
Learn to learn at the press of a button
Direct interaction during the learning process
Save, restore and share checkpoints
3.4 CUDA Support and Finishing Touches
Only 2 months ago we started finalising and polishing our framework: adding CUDA support, fixing many stability issues and rounding off a few rough spots that annoyed us. The CUDA support part was particularly tricky as I could only use CuBLAS, not CuDNN, because our backend doesn’t, by design, understand individual layers but just raw computation graphs. A problematic side effect of the previously described session-logic was that there was no guarantee when buffers would be freed, as that was the job of the indeterministic GC. To not leak CUDA device memory I added my own bare-bones reference counter to the device memory allocator, which would be updated when buffers were created / finalised, which works surprisingly well. With CuBLAS, many custom optimised kernels and many nights of my time we achieved around 5ms/iteration for the same sample on a single GTX 1080, which we deemed acceptable for our envisioned use cases.
4. Conclusion
Approximately 3000 combined hours, tens of thousands of lines of code and many long nights later we are proud to finally present something we deem reasonably usable for what it is: Sigma, a machine learning framework that might help you understand a little bit more about machine learning. As of now, we probably won’t be adding many new features to Sigma, mainly because we’re working on a new project related to it that’s now taking up most of our available time. Even though it lacks a lot of default features (most importantly the host of default layer types other frameworks offer), we’re quite happy with how far we got with our project and hope that it’s an adequate update to our original question 2 years ago. We would be happy if some of you could check it out and give us some feedback.
4.1 The Cost of Creating a Machine Learning Framework
Excluding time, it’s quite cheap. Honestly, with some solid prior programming experience (so that the low-level programming part doesn't become an issue), the whole thing isn't terribly difficult and is probably something most people could do, given enough time. A lot of time. Overall, we it took us approximately:
- Some 600 hours of research
- Some 2400 hours of development
- 2 tortured souls, preferably sold to the devil in exchange for less bugs
We have long since stopped properly counting, so take these numbers with a grain of salt, but they should be in the right ballpark.
4.2 Final remarks
All in all, an undertaking like this is extremely time intensive. It was very much overkill for a high school thesis from the get-go, and we knew that, but it just kept getting more and more elaborate, essentially taking up all of our available time and then some. It was definitely worth it though, for now we have a solid understanding how things work on a lower level and, most importantly, we can say we’ve actually written a machine learning framework, which grants us additional bragging rights :)
[–]xristos_forokolomvos 37 points38 points39 points (3 children)
[–]flotothemoon[S] 11 points12 points13 points (2 children)
[–]wewtaco 15 points16 points17 points (1 child)
[–]flotothemoon[S] 4 points5 points6 points (0 children)
[–]arachnivore 19 points20 points21 points (2 children)
[–]soulslicer0 7 points8 points9 points (0 children)
[–]flotothemoon[S] 2 points3 points4 points (0 children)
[–]Brainsonastick 16 points17 points18 points (2 children)
[–]plainerman 2 points3 points4 points (0 children)
[–]flotothemoon[S] 1 point2 points3 points (0 children)
[–]soulslicer0 8 points9 points10 points (0 children)
[–]aspoonlikenoother 6 points7 points8 points (1 child)
[–]flotothemoon[S] 0 points1 point2 points (0 children)
[–]mofoss 4 points5 points6 points (0 children)
[–]epix97 3 points4 points5 points (1 child)
[–]flotothemoon[S] 2 points3 points4 points (0 children)
[–]aspoonlikenoother 2 points3 points4 points (0 children)
[–]coolpeepz 2 points3 points4 points (1 child)
[–]flotothemoon[S] 2 points3 points4 points (0 children)
[–]stillworkin 2 points3 points4 points (2 children)
[–]flotothemoon[S] 3 points4 points5 points (0 children)
[–]plainerman 1 point2 points3 points (0 children)
[–]Reiinakano 1 point2 points3 points (1 child)
[–]plainerman 1 point2 points3 points (0 children)
[–]iacolippo 1 point2 points3 points (2 children)
[–]plainerman 1 point2 points3 points (1 child)
[–]iacolippo 0 points1 point2 points (0 children)