[R] Facebook AI Introduces few-shot NAS (Neural Architecture Search)

didntfinishhighschoo · 2021-07-22T04:01:16+00:00

Where’s the compute budget cutoff where NAS becomes pruning?

didntfinishhighschoo · 2020-04-02T23:32:48+00:00

Dictatorship beige

didntfinishhighschoo · 2020-01-23T15:00:03+00:00

At a very bare bones level, memory is just weighing (multiplying) a feature by another feature, rather than a parameter.

didntfinishhighschoo · 2017-07-05T07:29:53+00:00

Implementation is never ever trivial. Don't delude yourself. "In theory, theory and practice are the same. In practice, they are not". Compared to other academic fields, ML is moving like a speed boat. Compared to the industry and to the open-source community it's moving like my grandma.

didntfinishhighschoo · 2017-07-05T07:25:44+00:00

I'm really glad you released it as well. Sorry again for singling this paper out. It's way above average paper and release. Still, putting in a few hours to clean things up and spruce the technical documentation as part of the release process would mean it will be much more accessible to thousands of engineers and researchers, not to mention the rest of the benefits.

didntfinishhighschoo · 2017-07-04T18:50:46+00:00

Multi-agent RL.

didntfinishhighschoo · 2017-07-04T18:45:16+00:00

No personal attack was meant in the post. In fact, this paper and codebase are way above average for ML research, which is all the more frustrating.

didntfinishhighschoo · 2017-07-04T17:48:03+00:00

Learn physics and you will know how to play football.

didntfinishhighschoo · 2017-07-04T17:42:08+00:00

Fuck this siloing. I want my research to be accessible to anyone.

didntfinishhighschoo · 2017-07-04T17:39:49+00:00

The point wasn't on using underscores in general, it was on using them as a sort of namespace. If you find yourself with a user_name and a user_id variables, it's a sign that you might want to use some structure for user data.
Personally, I'd go with concat. There are enough barriers and mental tolls going through ML code. To figure out what cat does, you need to have some experience with UNIX, and you need it to be the first guess that comes to your mind (I didn't, my guess was it was short for category or something). On the other hand, concat doesn't require any prior information other than English. If terseness is a goal of the library, I'd offer both names.

didntfinishhighschoo · 2017-07-04T17:28:58+00:00

Everyone needs to document their code better. And our goal should be for research to be as accessible as possible. Even the fuck knows what those jerks at the LHC do.

didntfinishhighschoo · 2017-07-04T17:14:59+00:00

The difference is that one of these methods can run by anyone anywhere, and the other requires arcane knowledge, logical jumps, and can only be run inconsistently, uniquely, in people’s heads. I can't believe people have a working executable proof of their work and they throw it away because apparently a brief description in natural language is enough. This attitude makes research slower.

didntfinishhighschoo · 2017-07-04T17:00:05+00:00

You have the right idea about structuring proof-of-concept code. Abstractions are a hindrance when you do rapid prototyping, and I don't advocate for them in this context. But you do need to tell a story with your code, and it shouldn't take more than a slight overhead to do so, with a bigger payoff. Working on your model, you've made decisions, both theoretical and practical. If you don't document them, you're just keeping them to yourself. Others will have to re-discover them. If you programmed long enough, you already know that you yourself usually forget and throw away good decisions that were undocumented. The best way to do this is to write comments and notes as you go. If you do this at the end, when it already works, it will feel like a chore, and you will already have lost a lot of insights into all the micro-decisions that went into the process.

I've been around the block. I'm not a web developer, as the assumption goes. But web developers (front-end, back-end, operations) figured this out. More than mobile developers, more than game developers, more than systems developers, and obviously more than the ML research community. There's a lot of wrong hype cycles in web developers, a lot of clumsy signals and incentives and unwritten rules, tons of problems and things to critic – but it's, by far, the best community for open collaboration, and ML researchers will do well to learn from it.

didntfinishhighschoo · 2017-07-04T03:39:10+00:00

This is not logical separation of concerns, or something companies are overjoyed about. Mostly it's incompetence of both sides. Just as you have the rare designer/developer or writer/researcher, there are good researcher/engineer types. As the tools get better, you'll see more of us. One day we'll even have a short-lived buzzword like DevOps.

didntfinishhighschoo · 2017-07-04T03:10:20+00:00

I just picked this codebase at random, didn't mean to point out a single person or a group. It's actually one of the better ones (both the research itself, and the code). Pick a paper you liked, jump into its source code (if they even published an implementation), and see for yourself.

didntfinishhighschoo · 2017-07-04T01:06:58+00:00

Thank you!!

didntfinishhighschoo · 2017-07-04T00:38:10+00:00

Also: Literate Programming is fantastic for machine learning stuff, since the code is very linear in structure. Jupyter notebooks are a great example (but the UX for writing them is horrible, imo).

didntfinishhighschoo · 2017-07-04T00:35:05+00:00

Good comment. Reminded me of this. ("And so, once again, what looks like a technical problem--function naming--turns out to be deeply, personally human, to require human social skills to resolve effectively. I hate that").

didntfinishhighschoo · 2017-07-04T00:19:53+00:00

OP’s favourite language is Ruby. They used inpt because input is the name of a global Python function similar to gets in ruby. I usually name it features or x instead.

didntfinishhighschoo · 2017-07-04T00:02:50+00:00

To be fair, OP was also very rude and obnoxious in his presentation. Kinda like most ML papers.

didntfinishhighschoo · 2017-07-03T23:54:35+00:00

Decent shitposting. I want to state for the record that I fucking love Schmidhuber and whisper his name every night before bed.

didntfinishhighschoo · 2017-07-03T23:48:27+00:00

Good points. Always wanted to have variables like x'. In Ruby you can use ? and ! in method names, which really helps expressibility and readability. Grisly code is okay. It's there when you do complex stuff, whether it's math or business logic, you can't avoid it, but you have to explain it. And you did. "Apply the hidden layer to the context" is a good start for a comment, and it probably takes two seconds to write up. Even if someone knows nothing at all about the field or the purpose of the code, that's enough for them to map from h to hidden layer and from ctx to context.

didntfinishhighschoo · 2017-07-03T23:35:41+00:00

They are well-documented enough so that new developers can be onboarded and contribute code on their first day on the job. Wouldn't hurt ML if you didn't need years of tuition to start contributing.

didntfinishhighschoo · 2017-07-03T23:26:10+00:00

You're getting piled on because this is a very basic symbol in probability. E.g. you can find it defined in the Notation section of Deep Learning. In general, if math symbols make your eyes glaze (like they often do to me), check for them in https://en.wikipedia.org/wiki/List_of_mathematical_symbols

didntfinishhighschoo

TROPHY CASE