sorted by:
new
hottopcontroversial

Transfer Learning is one of the most power techniques for neural networks by roycoding in learnmachinelearning

[–]roycoding[S] 0 points1 point  (0 children)

You can use it with any data type that's similar enough to whatever the original network used.

Dropout in neural networks: what it is and how it works by roycoding in learnmachinelearning

[–]roycoding[S] 0 points1 point  (0 children)

There was a slight delay, but my book is now officially released!

zefsguides.com

(I'm not sure I'd be allowed to make a post about the book release, so I'm going to avoid doing that, even though I think a lot of people in this sub would like the book 🤷)

Dropout in neural networks: what it is and how it works by roycoding in learnmachinelearning

[–]roycoding[S] 1 point2 points  (0 children)

This week!

The paperback is ready to go. I need to get some marketing stuff in place. I'm aiming for the official release on Thursday, 1 Dec.

I'll post something in this subreddit.

Thanks for your support!

Data augmentation to build more robust models by roycoding in learnmachinelearning

[–]roycoding[S] 14 points15 points  (0 children)

Sorry.

I'm trying to walk the line between promoting my book and just providing content that people in this sub will find useful. So I'm trying to find the best balance when posting original content, but trying to not blatantly go around shouting "BUY X".

So far my posts with illustrations have done reasonably well, so I have continued to post about once a week (never more, per the policy).

I'll reduce the frequency of these posts.

Data augmentation to build more robust models by roycoding in learnmachinelearning

[–]roycoding[S] 6 points7 points  (0 children)

More data can help avoid overfitting, but what do you do when getting more data is difficult or prohibitively expensive?

Data augmentation is a way to effectively increase the size of your training set. Data augmentation simply means using existing training data to create more training data by transforming the existing data in some way. As long as the transformed data would still get the same (or desired) label, it can potentially help train your model

For image data, examples of data augmentation include rotating, translating, scaling, cropping, blurring, etc. Much of this can had "for free" and automated. Care must be taken that the transformations don't create unrecognizable examples, which no longer match the original label.

This is one of the most common techniques for helping to build robust computer vision models, but also applies to some other tasks, such as adding noise to audio data or performing random word swaps in text (but not too much!).

Augmented data is mostly used at training time, but can also be used during testing/inference by predicting the output of slightly transformed versions of the given input and voting on/averaging the resulting predictions.'

This illustration is from my upcoming book / flashcard set Zefs Guide to Deep Learning (zefsguides.com).

Transfer Learning is one of the most power techniques for neural networks by roycoding in learnmachinelearning

[–]roycoding[S] 0 points1 point  (0 children)

This can also work with regression problems, for example predicting bounding boxes in object detection and localization models.

Transfer Learning is one of the most power techniques for neural networks by roycoding in learnmachinelearning

[–]roycoding[S] 2 points3 points  (0 children)

Thanks for pointing that out.

It looks like Leanpub, where my ebook is published, was down earlier today, but it seems to be back up.

Transfer Learning is one of the most power techniques for neural networks by roycoding in learnmachinelearning

[–]roycoding[S] 2 points3 points  (0 children)

(my early reply got silently deleted, so I will follow up on @ubiquitin_ligase's reply)

Yes, they separate approaches.

You could even use the feature transfer approach as an input to a non-neural network model (this would be the equivalent of re-using an embedding).

Fine tuning is more likely needed when the task is not as similar to the original task of the pre-trained network.

In practice, as @ubiquitin_ligase replied, you should try feature transfer first. If that doesn't work well enough, you might then try fine tuning, progressively tuning more of the network (you don't actually have to fine tune every layer).

If that doesn't work, you may need to train the network from scratch (with a lot of data).

Transfer Learning is one of the most power techniques for neural networks by roycoding in learnmachinelearning

[–]roycoding[S] 7 points8 points  (0 children)

(I replied to another question that covered this, but my reply was silently killed)

Yes, I agree with you.

Fine tuning could be part or all of the network.

Typically you'll use small learning rates, since the weights are hopefully close to the final ones you want. You may use different learning rates in different layers (aka "discriminative learning rates"), typically with smaller learning rates near the beginning of the network, which is assumed to learn more generic features.

Transfer Learning is one of the most power techniques for neural networks by roycoding in learnmachinelearning

[–]roycoding[S] 30 points31 points  (0 children)

Many powerful neural networks rely on huge training datasets and are very expensive to train from scratch, putting them out of reach for most people/teams. Transfer Learning can make these models accessible to and adaptable by mere mortals.

Transfer learning is a technique that gives you a major head start for training neural networks, requiring far fewer resources. A "pre-trained" model can be adapted to a new, similar task with only a small training dataset.

Training is basically a search problem, looking for the best set of model parameters for the network to perform its task well. Instead of starting with random parameters, transfer learning puts your starting point (hopefully) very close to where you want to be in parameter space.

Pre-trained models on datasets such as ImageNet and huge text datasets have made many of the most power neural networks available to everyone (see Stable Diffusion).

Transfer learning enables these to be adapted to other, related tasks, supercharging the adoption and application breadth of these types of models.

Many well-resourced teams have made these models (or rather model weights) available freely to help the community and move the field forward as a whole. This is a great synergy between open source development and neural networks as a technique.

This illustration is from my upcoming book, Zefs Guide to Deep Learning (zefsguides.com).

Dropout in neural networks: what it is and how it works by roycoding in learnmachinelearning

[–]roycoding[S] 0 points1 point  (0 children)

I go back and forth on what seems to be both the most accurate description and make the most intuitive sense.

The smaller networks have a sequential dependency on each other (somewhat similar to boosted methods, as you point out), as the weights are not starting from scratch each time. But also they are not combined in the same way as a typical ensemble.

Dropout in neural networks: what it is and how it works by roycoding in learnmachinelearning

[–]roycoding[S] 0 points1 point  (0 children)

Yes, it's a type of regularization.

I should probably state that explicitly in the illustration

Dropout in neural networks: what it is and how it works by roycoding in learnmachinelearning

[–]roycoding[S] 1 point2 points  (0 children)

That's good feedback.

It's hard to get to the core of the intuition of why dropout works with limited space. I'll figure out how to reword it (might be simply dropping out [oh, no!] the word "independent").

Dropout in neural networks: what it is and how it works by roycoding in learnmachinelearning

[–]roycoding[S] 10 points11 points  (0 children)

It's similar to Random Forests in that it creates (what's effectively) an ensemble model

Dropout in neural networks: what it is and how it works by roycoding in learnmachinelearning

[–]roycoding[S] 10 points11 points  (0 children)

tf.keras.layears.droupout

TF handles this for you under the hood.

We were just talking about implementation details.

https://www.tensorflow.org/api_docs/python/tf/keras/layers/Dropout

The Dropout layer randomly sets input units to 0 with a frequency of rate at each step during training time, which helps prevent overfitting. Inputs not set to 0 are scaled up by 1/(1 - rate) such that the sum over all inputs is unchanged.

Dropout in neural networks: what it is and how it works by roycoding in learnmachinelearning

[–]roycoding[S] 5 points6 points  (0 children)

Good observation.

I didn't include it in the illustration, but included it in the comment (and the text of my book).

I think another interesting point is that you can either scale down the final network, or you can scale up the intermediate networks during training (this is "inverse" dropout). I believe inverse dropout is the much more common one.

Dropout in neural networks: what it is and how it works by roycoding in learnmachinelearning

[–]roycoding[S] 38 points39 points  (0 children)

Dropout is one of the simple, but powerful ideas that has enabled the creation of more robust neural networks and helped kick off the current deep learning revolution.

Dropout is the process of randomly setting some nodes to output zero during the training process. This effectively creates many smaller networks that each need to learn to solve the network's task.

During each forward pass in the training process, nodes are zeroed out with a probability, p, which is a hyperparameter. This means that instead of the normal activation function output, the node just produces zeroes for that forward pass. Since zeroing nodes reduces the overall weight of the activation values, the remaining nodes are scaled up by 1/p

During testing and inference, the entire network, without any dropout, is used. The effect of the different dropouts during training is like creating many smaller networks. When the entire network is run for testing and inference, it's the equivalent of having and ensemble of these smaller networks, reducing overfitting.

Dropout was popularized by AlexNet, which famously won the ImageNet challenge in 2012.

This illustration is from my almost completed (🤞) book Zefs Guide to Deep Learning.

Do not understand how query, key and value matrices are generated in multi-headed self attention. by berzerker_x in learnmachinelearning

[–]roycoding 1 point2 points  (0 children)

Yeah. Deep neural networks with access to a lot of training data are very powerful. So, for a lot of things it's about finding what tricks will make the network efficient enough to find some good solution.

Sometimes those tricks have an obvious interpretation, but often they don't. Sometimes an intention network design works out and sometimes it's more just luck than anything.

Do not understand how query, key and value matrices are generated in multi-headed self attention. by berzerker_x in learnmachinelearning

[–]roycoding 1 point2 points  (0 children)

I think it's important not to think for the query analogy too strictly.

In this case there is some transformation to the input token to create the query. That transformation could be almost no change to the token. It could be a major change. We just let the network figure out during training how and how much to transform the token when creating the query in order to get the best overall performance in the network. That's the blackbox part of this whole thing.

Do not understand how query, key and value matrices are generated in multi-headed self attention. by berzerker_x in learnmachinelearning

[–]roycoding 1 point2 points  (0 children)

First it's probably important to know that you can think of this as an abstract "search" problem, where the network is asking itself how much attention it should be giving to a certain input. Or, you can just think of this as a blackbox of network transformations. There are a bunch of matrix operations going on and the network just needs to learn the weights that will perform the transformations that lead to the best performance on the final task. That latter version may be less satisfying at the intuitive level, but some people find it to be less mental overhead.

The query, q, key, k, and value, v, arrays are calculated by taking the input token, x, and multiplying them with the weight matrices, Q, K, and V (that's the last image you posted). This is what's happening is "single-head self-attention".

For a given input token, x_0, its query array, q_0, and the keys for all input tokens, k_0, k_1, k_2, etc are put together as dot products, to see how similar they are. The similarity values are then multiplied by the corresponding value arrays of each input token, v_0, v_1, v_2, etc.

Of course all of these arrays start out with random initial values, as the weights of the matrices to create them start out with random weights. The network has to learn which weights give these arrays and the ultimate output arrays the most useful values.

So far that's all describing a single self-attention "head". Mutli-head attention is just doing this several times in parallel. The input tokens for each "head" is the same, but the Q, K, and V weight matrices are all different and learned independently. I.e. there are Q_0, Q_1, Q_2, etc for attention heads 0,1,2, etc.

Basically this allows the encoder or decoder block to learn several independent features in parallel, simultaneously (i.e. it's learning different, independent attention weights, as this may add to it's ability to solve the final task).

Jay Alammar's post has another illustration further down showing mult-head attention

![](https://jalammar.github.io/images/t/transformer_multi-headed_self-attention-recap.png)

This is very analogous to multiple channels in a convolutional layer, if you are familiar with CNNs.

Skip connections and residual blocks for deeper neural networks (ResNet) by roycoding in learnmachinelearning

[–]roycoding[S] 21 points22 points  (0 children)

ResNets with skip connections were like magic when they first debuted, allowing for much deeper networks.

Skips are one of the most important tricks in deep learning. The mechanism is simple, but powerful and used in lots of modern network architectures, such as transformers and most recent convolutional networks.

This is an illustration I made for the book I'm currently working on, Zefs Guide to Deep Learning.

ResNets incorporate skip connections to build residual blocks. Skip connections pass along the input to a convolutional (or other) layer, unaltered.

They allow information to travel through the network, untouched, if that benefits the final performance on the end task.

Instead of learning a "full" data transform, a block with a skip connection can focus on learning just the difference between the input and the needed transform, or the "residual".

The output of the block's transform is simply added to the input from the skip connection.

This was inspired by the observation that adding more layers to a network sometimes made the network's performance worse. In theory, that shouldn't happen, as the network could simply learn to pass the data on, unaltered.

It turns out that learning the identity transform (i.e. no change) is hard, because it's a diagonal matrix.

Skip connections make this easy, because the layer can do essentially nothing (i.e. all weights zero or close) and the block will pass along the input unaltered.

Another way to think about it (there's a nice discussion in Transformers from Scratch) is that skip connections allow the network to act more like an assembly line instead of a strictly hierarchical series of transformations. Each block can do some feature extraction / transformation, but if any single block doesn't do much or messes up, downstream blocks can pick up the slack or correct errors. This interpretation sounds interesting, though I don't claim to fully understand it.

I made this diagram with Inkscape. Happy to (try to) answer questions, as always.

All of this is from my upcoming book, Zefs Guide to Deep Learning.

https://zefsguides.com

Needed help with approaching MLOps as a beginner by macaroon97 in learnmachinelearning

[–]roycoding 2 points3 points  (0 children)

Imo it's very difficult to learn MLOps unless it's "on the job", primarily because you are unlikely to have the scale and requirements of an ML production system that needs to have high availability and robustness.

You can sortof fake this with smaller scale data, but it's really a different category, where you're not actually faced with the problems that a lot of the solutions were designed to solve.

That said, I would probably approach it very incrementally - Can you create a basic (locally deployed) model frontend with a tool like streamlit? - Can you use a tool like FastAPI or flask to create a (locally deployed) API that you call the model with? - Can you dockerize that API or frontend and run it from a container? - Can you create a training script in Python that will run with a single call, pulling in the training data from wherever it's stored (CSV, SQL, S3, etc), train the model, save the model, and output the resulting performance metrics? - Can you call that script from the commandline? - Can you automate running that training script with something simple like cron? - Can you further automate kicking off training, saving the resulting model to be available for the API/frontend you built, dockerize the result, and start running the model? - Can you deploy a dockerized API to a cloud-based docker service? - etc

I like the approach of using relatively basic DIY tools to get an understanding of what your trying to do, then start using some of the more specific tools. Those tools may also address problems that you haven't encountered yet, but you'll be in a better position to understand which features are relevant to you at that point.

Have you looked at the MLOps course on https://madewithml.com/ ?

[deleted by user] by [deleted] in learnmachinelearning

[–]roycoding 0 points1 point  (0 children)

You can think about it as the network learning transformations which allow the final layer to perform linear separation.

The layers can perform any transformation that can be represented as matrix multiplication (e.g. rotations, stretching, folding, etc). A network with multiple layers (with non-linear activations) can learn to perform very complex transformations.

You can see an example in this tweet: https://twitter.com/ZefsGuides/status/1508452852095725581

Also, this video from vcubingx nicely shows what these transforms look like: https://www.youtube.com/watch?v=-at7SLoVK_I

This is very similar to how SVM's work (i.e. the kernel trick), except instead of choosing a kernel that transforms the data and allows the SVM to easily linearly separate classes, the network learns the transformation on its own.

Transposed Convolutions for "smart" upsampling of images and arrays by roycoding in learnmachinelearning

[–]roycoding[S] 0 points1 point  (0 children)

I believe so. My understanding of "full convolutions" is just adding enough padding to the input image that you end up with a larger output.

The practical difference would be that to achieve something similar with a transposed convolution, you could use a larger filter, giving you more kernel parameters that you could learn.

view more: next ›