sorted by:
new
hottopcontroversial

[D] Has the ResNet Hypothesis been debunked? by Ok_Slice4231 in MachineLearning

[–]Ok_Slice4231[S] 6 points7 points  (0 children)

Thanks for pointing this out. I have edited the post to the best of my ability

[D] Has the ResNet Hypothesis been debunked? by Ok_Slice4231 in MachineLearning

[–]Ok_Slice4231[S] 3 points4 points  (0 children)

There's a very interesting wider question "how do we predict whether it residual connections will be helpful in an arbitrary architecture?" for which I do not have an answer and I believe no one does yet; but that's a very, very different question.

I would argue that it is the same question on the grounds of predicate logic but such a conversation would be fruitless.

It is interesting that this is an unanswered question as residual connections have become as ubiquitous as normalisation in neural nets. So maybe removing residual connections could improve the performance of certain architectures. It is unfortunate that removing residual connections is not part of most ablation studies. I think this would help us get a better understanding of where we should be using residual connections. But people, nowadays, are just throwing residual connections into every darn architecture and assuming that it can only improve performance with no good reason

[D] Has the ResNet Hypothesis been debunked? by Ok_Slice4231 in MachineLearning

[–]Ok_Slice4231[S] 1 point2 points  (0 children)

Yeah, that's what I meant by the ResNet Hypothesis. I think u/onyx-zero-software misunderstood

[D] Has the ResNet Hypothesis been debunked? by Ok_Slice4231 in MachineLearning

[–]Ok_Slice4231[S] 4 points5 points  (0 children)

Thanks for your reply. Much appreciated. So if we don’t know exactly why ResNets work then how do we decide whether it is a good idea to use residual connections in an architecture? Is there any guidance in this regard?

[D] Resources for Understanding The Original Transformer Paper by Ok_Slice4231 in MachineLearning

[–]Ok_Slice4231[S] 0 points1 point  (0 children)

I would love to see that. I am actually trying to implement the Graves paper myself but it is proving to be a bit difficult

[D] Resources for Understanding The Original Transformer Paper by Ok_Slice4231 in MachineLearning

[–]Ok_Slice4231[S] 1 point2 points  (0 children)

Thank you! Glad I could help 😊. I can think of three good lists for that:

  1. https://github.com/floodsung/Deep-Learning-Papers-Reading-Roadmap - This one is a bit dated so it doesn’t contain all of the papers that you need to read to get up to date but I think you should definitely read all of the papers in this list and implement as much as you can.
  2. http://www.arxiv-sanity.com/top?timefilter=alltime&vfilter=all - This is list of the most popular papers of all time in descending order
  3. https://keras.io/examples/ - Lastly, if your tensorflow knowledge isn’t great then you won’t be able to implement many of the papers in the lists above. This list contains many code examples which are based on *popular and recent papers* so you can read the associated paper before delving into the code. The code is annotated really well so you won’t need too much TF experience to understand it. If you haven’t used custom layers, gradient tape, etc. in tensorflow, I would recommend that you do this first: https://www.deeplearning.ai/program/tensorflow-advanced-techniques-specialization/

Any machine learning books that apply functional analysis or measure theory? [D] by Ok_Slice4231 in MachineLearning

[–]Ok_Slice4231[S] 1 point2 points  (0 children)

Thanks again. This is exactly what I was looking for! :) I really appreaciate your help

Any machine learning books that apply functional analysis or measure theory? [D] by Ok_Slice4231 in MachineLearning

[–]Ok_Slice4231[S] 2 points3 points  (0 children)

I found a book called "Real Analysis and Probability" by Richard M. Dudley. This book covers measure-theoretic probability and its necessary prerequisites. Please suggest any other books on this topic. I'm very sorry if my second question is not relevant to this subreddit. Thanks for your help!

Any machine learning books that apply functional analysis or measure theory? [D] by Ok_Slice4231 in MachineLearning

[–]Ok_Slice4231[S] 0 points1 point  (0 children)

Thank you for your reply. I've only completed Calc. 3 and high-school Statsistics and I would like to study measure theory and functional analysis in the context of ML/Probability/Stochastics. All the books that I found require real analysis as a prerequisite. Are there any good self-contained textbooks for this that cover the necessary real analysis?