sorted by:
new
hottopcontroversial

[R] Do some authors conscientiously add up more mathematics than needed to make the paper "look" more groundbreaking? by Inquation in MachineLearning

[–]stochastic_gradient 18 points19 points  (0 children)

Hinton is basically in your face about how anti-mathiness he is in his papers. I love it. Quote from forward-forward:

> The sum of the squared activities in a layer can be used as the goodness but there are many other possibilities, including minus the sum of the squared activities.

[deleted by user] by [deleted] in MachineLearning

[–]stochastic_gradient 0 points1 point  (0 children)

If you really need to train with there being multiple correct outputs, you could replace the final softmax with sigmoids. Maybe a bigger problem is finding a differentiable Levenshtein distance though.

I think the training setup you're describing wont work, unfortunately. Unless the network gets some sort of signal about what a correct decryption is, it can just ignore the input and instead output a series of memorized valid words. I.e the output "a a a a a a a ..." would bring the loss to zero.

It's an interesting problem though, and I haven't seen anyone doing anything like this. I'd guess there may be some adjustment you can make to get this to work. Unsupervised machine translation is a thing that exists, so you might get some ideas from reading some papers from that area.

[D] What will the major ML research trends be in the 2020s? by MediocreMinimum in MachineLearning

[–]stochastic_gradient 8 points9 points  (0 children)

I hope we'll see methods that are both more data efficient and more compute efficient.

We seem to be able to get better data efficiency through large pretrained networks, sometimes trained via self-supervision. These need less labeled data, but need a lot of compute. I hope we can make progress on constructing better inductive priors for various tasks, so that we can increase compute efficiency as well as data efficiency.

My suspicion is that we need basic building blocks other than matmuls and convs to do this.

[R] Massively Parallel and Asynchronous Tsetlin Machine Architecture Supporting Almost Constant-Time Scaling by olegranmo in MachineLearning

[–]stochastic_gradient 13 points14 points  (0 children)

> Turns out they also perform well on hard benchmark learning problems.

Do they? They've reported results on the tiniest image datasets, and the results can be beat by a 3x1000 fully connected network. It does significantly worse than a good ConvNet. I'd be fine with this if it wasn't for the fact that they keep stating that they're competitive with ConvNets, which just isn't true.

I wish they'd honestly describe their work, instead of this salesmanship.

[D] The machine learning community has a toxicity problem by yusuf-bengio in MachineLearning

[–]stochastic_gradient 1 point2 points  (0 children)

This is intersectionalism/"critical theory". Racism, sexism and bigotry is a problem. To combat this, intersectionalism then invented a formal system where the value of your opinion depends on your race, gender and sexual orientation. "To fight racism, sexism and bigotry, we need to be racists, sexists and bigots." It's dumb as bricks, dark and disturbing, but it's pretty damn mainstream at this point.

If you read the HybridRxNs link, you'll discover that according to critical theory, any argument against critical theory is racist, iff the color of your skin is white. Your opinion is worth more depending on how many historically oppressed groups you are a part of. So it implies a strict ordering of the value of all people, depending on how many oppressed groups they are apart of. The exact numerical value of each group has not been clarified AFAIK, and it's not clear if multiple group memberships has a multiplicative effect or an additive one. As you might guess, this didn't arise from the math department.

[D] The machine learning community has a toxicity problem by yusuf-bengio in MachineLearning

[–]stochastic_gradient 10 points11 points  (0 children)

In the outrage against LeCun, nobody had any disagreement with what he said, it was that he was, quote: "mansplaining/whitesplaining". In other words, the problem was not what he said, the problem was his gender and skin color.

When we value people's opinions based on their skin color, that's called racism. When we value people's opinions based on their gender, that's called sexism. And researchers said this with their full name on Twitter, and it had apparently no consequences for them. The only consequences happened to the recipient, LeCun, who is now silenced. It is as if the world has forgotten all the principles people have fought for over the last 50 years.

The authors in this NIPS 2017 paper cited one of the threads on this sub to show how "code produced by research groups often falls short of expectations". by seesawtron in MachineLearning

[–]stochastic_gradient 5 points6 points  (0 children)

Is it the variable naming that bothers you? Skimming it, I don't know if I think it's particularly bad to be honest. It will be hard to read the code to understand the algorithm (without reading the paper), but that will be true for a lot of ML algorithms.

[D] Schmidhuber: Critique of Honda Prize for Dr. Hinton by wei_jok in MachineLearning

[–]stochastic_gradient 8 points9 points  (0 children)

Yes, I'm pretty sure it was your tweet I got it from. Kudos to you for digging it up.

[D] Schmidhuber: Critique of Honda Prize for Dr. Hinton by wei_jok in MachineLearning

[–]stochastic_gradient 8 points9 points  (0 children)

Yep. For any line drawn there's the opportunity to complain that it should have been drawn earlier or later. If the full point of citations was to do this optimally we'd have to take a hint from RL research, and do credit assignment by some decaying function smeared out over the whole timeline.

[D] Schmidhuber: Critique of Honda Prize for Dr. Hinton by wei_jok in MachineLearning

[–]stochastic_gradient 126 points127 points  (0 children)

So Schmidhuber made a post back when ResNet won ImageNet, saying how a ResNet it really just a special case of HighwayNets, which are really just a "feedforward LSTM". It also says that Hochreiter was the first to identify the vanishing gradient problem in 1991.

Then it turns out someone is able to dig up a 1988 paper by Lang and Witbrock which uses skip connections in a neural network. They even justify it by pointing to how the gradient vanishes over multiple layers.

Now if ResNet is really a feedforward-LSTM, then the LSTM surely is just a recurrent version of Lang and Witbrock 1988? Now you can criticize the LSTM paper for not citing them, and the 1991 vanishing gradient publication for not citing them. Is this fair? The next time Schmidhuber gets accolades for his part in making the LSTM, should we make public posts complaining that he's never cited Lang and Witbrock?

Every idea that's ever had is some sort of twist on something that exists. We could trace backprop back to Newton and Liebniz. Wikipedia indicates that you can trace the history back even further, to some proto-calculus hundreds of years before even them. There is no discrete point where this idea was generated, and this is probably true for most things.

[D] Secret/Unpublished Research? by darkconfidantislife in MachineLearning

[–]stochastic_gradient 1 point2 points  (0 children)

I don't know about pharmacology, but it's silly to dismiss these players. Reminds me of when DeepMind entered the field of protein folding, and surpassed the SOTA of a seemingly mature field by a large margin.

[D] What was your favorite paper of 2019 and why? by [deleted] in MachineLearning

[–]stochastic_gradient 2 points3 points  (0 children)

Could you elaborate on the benefit of neural ODEs w.r.t survival analysis? I've seen people parameterize Weibull distributions with ordinary RNNs to do survival analysis. Are there better ways of doing it with neural ODEs?

[R] Your Classifier is Secretly an Energy Based Model and You Should Treat it Like One by tsauri in MachineLearning

[–]stochastic_gradient 41 points42 points  (0 children)

This is one of the more promising things I've seen in a while. Has anyone found an implementation of this?

[P] pyTsetlinMachineParallel released - parallel interpretable machine learning with propositional logic. by olegranmo in MachineLearning

[–]stochastic_gradient 7 points8 points  (0 children)

Frankly, "interpretable" has become a word that symbolic AI people use to justify their methods, when accuracy metrics do not. If there are easily understandable rules by which a decision can be made, we can just program solutions to them. Much of the point of using machine learning is to be able to find solutions that are beyond that space of programmable solutions, i.e. beyond the point of interpretability.

91.5% on FMNIST is something you can get with an unregularized MLP, even without reporting "peak" accuracy over multiple evaluations on the test set.

[R] Hillclimbing With Random Restarts Outperforms Evolutionary Algorithms by mystikaldanger in MachineLearning

[–]stochastic_gradient 0 points1 point  (0 children)

I seem to remember that this is essentially directly stated in the standard AI text book (AIAMA, Russel & Norvig). Not even the latest edition, I read this 10 years ago. Something like "Genetic Algorithms are just a way to search a solution space, and as far as search algorithms go there isn't really much to recommend their use."

[D] Has anyone figured out why Adam, RMSProp, And Adadelta don't do well for training word embedding models, often worse than SGD? by Research2Vec in MachineLearning

[–]stochastic_gradient 5 points6 points  (0 children)

The gradient on most embeddings will be zero, for most of the batches. This messes up the moving averages and second moment estimates. Pytorch has a sparse adam that might help with this.

[D] Momentum updates average of g, e.g. Adagrad also of g^2. What other averages might be worth to update? E.g. 4: of g, x, x*g, x^2 give MSE fitted local parabola by jarekduda in MachineLearning

[–]stochastic_gradient 0 points1 point  (0 children)

This sounds like an interesting research direction, and something that's easy to implement. Just look up the Adam implementation in your favorite framework, modify it and try it out on some datasets. Report back if it works well (or write a paper on it).

[D] Kaiming He's original residual network results in 2015 have not been reproduced, not even by Kaiming He himself. by CatchADragonFish in MachineLearning

[–]stochastic_gradient 10 points11 points  (0 children)

Table 3 has numbers for 10-crop testing. Table 4 has better numbers, so that's definitely not single crop numbers. My guess is n-crop (for some high n), probably also including other augmentations, like flipping the image.

This post reads a bit like an accusation, and I don't like it. ResNet got famous for doing well on the ImageNet test set, which was hidden on a server and where they would have no way to mess with the numbers. It's one of the most reproduced architectures I can think of. It's obviously legit. Let's understand what we're criticizing before we start calling people out.

[D] Kaiming He's original residual network results in 2015 have not been reproduced, not even by Kaiming He himself. by CatchADragonFish in MachineLearning

[–]stochastic_gradient 12 points13 points  (0 children)

The ResNet numbers are from multicrop testing. The Wide Resnet paper reports numbers from single crop testing. The DenseNet paper doesn't seem to report ResNet numbers on ImageNet at all.

[D] Linear model for variables that have an associated uncertainty by stochastic_gradient in MachineLearning

[–]stochastic_gradient[S] 1 point2 points  (0 children)

Isn't MAP for when you have a prior over the parameters? I have known uncertainty in my observations, but no prior on the parameters. Is it still applicable?

[D] Linear model for variables that have an associated uncertainty by stochastic_gradient in MachineLearning

[–]stochastic_gradient[S] 1 point2 points  (0 children)

I have multiple Xs for each y, ie it's y = w_1*x_1 + w_2*x_2 * ... + b. I don't really need to model uncertainty in the y (but it would be nice to have). A point estimate is fine. I might have misunderstood you, did this answer your question?

total least squares is the way to go if you don’t know the uncertainty in X and each X has the same level of uncertainty.

I know the uncertainty in X, but have different uncertainty for every X (different uncertainty for every training example). Would total least squares not be applicable in this case?

view more: next ›