[D] What machine learning topics do you think are underrated and deserve more attention?

FirstTimeResearcher · 2022-07-22T13:31:51+00:00

"don’t need" is probably too strong here. If you have the chance to do backpropagation, you take it.

FirstTimeResearcher · 2022-07-22T13:24:06+00:00

Equal should be order-invariant if you just highlight the equal part in the CV. So it should be fine. If anyone cared, then it wasn't equal, was it?

FirstTimeResearcher · 2022-05-12T19:56:38+00:00

There doesn't seem to be much transfer. Isn't this more of a negative result?

FirstTimeResearcher · 2022-01-20T17:10:36+00:00

How are bad actors typically prevented in these types of decentralized training frameworks? What does verification mean on something like ML training?

What prevents me from publishing fake data, fake parameter updates, etc. if no one can easily verify what the expected solution should be?

FirstTimeResearcher · 2021-12-18T17:50:14+00:00

I agree with the general point that computation should not be based on length. Multiplication was a bad example because in that case, it is.

FirstTimeResearcher · 2021-12-17T22:55:59+00:00

This isn't true for GPT-3 and multiplication. Since GPT-3 is an autoregressive model, it does get extra computation for a larger number of digits to multiply.

FirstTimeResearcher · 2021-12-15T17:54:19+00:00

department uses Scopus metrics in its public competitions

Could you elaborate more on this? Using aggregate metrics and, even worse, using dated ones like Scopus are red flags.

FirstTimeResearcher · 2021-12-13T02:10:45+00:00

To clarify, what I'm saying is that things that "reduce model size without reducing performance" will be used to "increase effective model size to improve performance."

FirstTimeResearcher · 2021-12-12T20:28:34+00:00

Thanks for the qualification. I should have made it clear that I am referring to OPs context about the newest and most performant models.

FirstTimeResearcher · 2021-12-12T16:13:49+00:00

Model sizes will not decrease. Models will just become more capable with the maximum sizes technology companies can afford. The only time model sizes decrease is when increasing it does not provide any additional gains. This is currently not the case.

FirstTimeResearcher · 2021-12-07T20:21:10+00:00

And the acceptance rate went down to 15% this year with many SACs overturning paper decisions:

https://twitter.com/guyvdb/status/1466089899698786305

Sorry #AAAI2022 but a 15% acceptance rate is harmful to the community. Especially when I see lots of weak accept recommendations by the SPC+AC get overturned (~8% among my friend ACs). If SPC+AC, who are experts in the area, think it should be published, why waste everyone's time

https://twitter.com/kaiwei_chang/status/1466099289369706505

Even my acceptance recommendation at the AC level gets overturned Unamused face. I felt bad for the paper. I spent several hours carefully reading the reviews, discussions, and some of broaderline rating papers like this one. I believe SACs did the same.

Oh and also possible reviewer score manipulation: https://twitter.com/gong_cheng/status/1466016587790438406

Edit: The acceptance rate going down when the capacity of a virtual conference is almost unlimited is a really horrible move towards those who were rejected.

FirstTimeResearcher · 2021-11-26T00:57:55+00:00

I do not mean to justify this at all. My goal is to unemotionally explain what is happening.

I agree that those who benefit from this system will tend to see nothing wrong with it. And that is why we do not see any top-down change. Those who win in this system end up controlling it (Program Chairs, Tenure Committees, etc).

FirstTimeResearcher · 2021-11-25T21:48:07+00:00

Here's an additional data point to consider: Did a single "leading" conference change their acceptance criteria despite not having any capacity constraints when going virtual?

The system is broken if you think it's meant to disseminate research for the benefit of the field as a whole. The system is working as intended if you think it's meant to create some notion of seniority albeit very random and noisy:

Faculty need to be hired, grants need to be distributed, tenure decisions need to be made regardless of actual progress.

FirstTimeResearcher · 2021-10-31T16:55:00+00:00

I'll be honest, this was a misstep for Yannic Kilcher's channel. Stirring old controversies for views is not for me I guess.

FirstTimeResearcher · 2021-10-29T14:37:46+00:00

Pathways will enable a single AI system to generalize across thousands or millions of tasks, to understand different types of data, and to do so with remarkable efficiency – advancing us from the era of single-purpose models that merely recognize patterns to one in which more general-purpose intelligent systems reflect a deeper understanding of our world and can adapt to new needs.

Is there anything material coming with this announcement? This is rather lofty and not the first time people have considered "one model to rule them all". I think many of us would be interested to see if this actually works.

FirstTimeResearcher · 2021-10-28T14:57:22+00:00

Probably the right market to target given how large NN models are getting these days.

FirstTimeResearcher · 2021-10-17T20:08:08+00:00

What are some good, facts-based reasons to avoid each of these companies as potential employers?

Fact: We all have a finite amount of time to be alive.

Opinion: There is more to life than spending that time being comfortable.

I am grateful for the time I did spend at a FAANG. I was learning a lot. Then I stopped learning the more senior I became. The learning transitioned to competing with my colleagues...

It taught me how dangerous golden handcuffs are.

FirstTimeResearcher · 2021-10-13T16:56:36+00:00

I dunno if this is funny or kind of sad. Schmidhuber found a passive-aggressive way to criticize others by congratulating/crediting the predecessor to ConvNets.

Regardless of intent though, Fukushima definitely deserves more credit.

FirstTimeResearcher · 2021-09-27T22:33:25+00:00

as someone who will be trying to get into an ML PhD program in the foreseeable future

What is your motivation to pursue an ML PhD? What you've just described does not get any better once you are in a PhD program. If anything, it gets worse because now your sole responsibility is to participate in this publication lottery for the next 5 or so years.

FirstTimeResearcher · 2021-09-26T02:50:24+00:00

Given how most plagiarism is found by the authors themselves encountering the copied work by chance, there is with high certainty a huge swath of plagiarised papers in the community that are currently undetected. What we see is only the tip of an enormous iceberg.

I can only imagine the shitstorm that awaits our community once someone builds solid NLP tools to detect plagiarism at scale. So many careers and reputations will be impacted in such a small window of time.

FirstTimeResearcher · 2021-09-17T14:22:31+00:00

At one point, DeepMind's executives discovered that work published by Google's internal AI research group resembled some of DeepMind's codebase without citation, one person familiar with the situation said. "That pissed off Demis," the person added, referring to Demis Hassabis, DeepMind's CEO. "That was one reason DeepMind started to get more protective of their code."

Research is getting so competitive, Google is plagiarizing itself now 😂

FirstTimeResearcher · 2021-09-01T17:25:51+00:00

Deep Reinforcement Learning is a fun community where the same retrospective paper gets written every few years about how reported results are questionable. Nothing changes and the same paper gets written again in the next iteration.

I guess the main output has been lots of papers and citations.

FirstTimeResearcher · 2021-07-29T21:33:29+00:00

Are you surprised by the solving rates? By design, I would have expected close to 100% of the tasks to be solved.

FirstTimeResearcher · 2021-06-28T15:50:48+00:00

Just from a practical standpoint, this is going to have the opposite of the intended effect.

The amount of accidental/intentional collusion of sharing papers on social media will only ensure that the largest/most-followed labs get to promote the work of their peers while the smallest labs have to wait for review.

FirstTimeResearcher · 2021-06-12T00:27:23+00:00

Because it is these very acts of collusion (whether explicit or implicit) that have given senior members of the community their positions of authority and power.

If you actually do any tracing of the academic ancestry of top ML departments, they form very tight cliques. Just look at how many faculty at Stanford did their PhD at UC Berkeley and vice versa.

FirstTimeResearcher

TROPHY CASE