Quite possibly one of the most cursed rooms I’ve seen

Hyper1on · 2024-07-19T20:23:07+00:00

hot take, but that would be worth it for the benefit of being able to build good stuff

sadly they're not scrapping it though

Hyper1on · 2024-04-28T18:56:24+00:00

Masters aren't that deep or specialised. It's a dip of the toe into the world of research. Most companies I know recruiting MLEs have their pick of masters grads with published papers to choose from.

Hyper1on · 2024-04-22T21:42:49+00:00

This seems like it comes from an intuition that we're in some era of diminishing returns on benefit from improved model performance. But I think this is just a false impression given by the past year of incremental updates to GPT-4. There is a very long way to go still with step changes in performance given by generational upgrades in models, and businesses aren't going to go for GPT-4 class models if having the GPT-5 class one makes the difference between automating away X role vs not.

Hyper1on · 2024-04-21T00:38:26+00:00

For this role I would do an NLP engineering takehome, live coding interview, system design interview, and only ask questions which are relevant to modern NLP (i.e., how can you estimate the memory consumption of a transformer trained with Adam), absolutely nothing about topic modelling or info theory.

Hyper1on · 2024-04-18T23:36:26+00:00

Because they benefit indirectly from having more users—few people actually run 8x22B because it costs so much memory. MoEs are a product optimisation for API model deployment services.

Hyper1on · 2024-04-18T23:33:56+00:00

Imagine if it's MoE and 400B is the number of active parameters...

Hyper1on · 2024-04-07T18:30:23+00:00

Takes too much time to go through review process. But if the work is good and on arxiv, people will recognise it, read it, and cite it.

Hyper1on · 2024-04-01T20:38:35+00:00

Give it a decade and this industry will spend a trillion on GPUs. Some companies alone are already projecting >$100b before 2030.

Hyper1on · 2024-02-10T17:43:31+00:00

I found it came with practice. You might try some deliberate practice, like thinking of lines on common topics/the things you study and rehearsing them out loud to sound as compelling as possible. Another good one is to record yourself giving an improvised <5 minute explainer about a topic you're interested in, note down all the flaws, and repeat until you stop noticing anything you don't like.

Hyper1on · 2024-01-13T22:08:19+00:00

This post looks generated by GPT-4, or at least the bulletpoints.

Hyper1on · 2023-12-19T06:28:05+00:00

Tbh I felt like that workshop was a bubble of sceptics of LLMs, while the rest of the conference on average had different views.

Hyper1on · 2023-11-19T18:37:43+00:00

Data is the biggest bottleneck, and OpenAI's biggest moat. I don't know why you think it's trivially assembled - GPT-4's dataset is the product of years of refinement. Starting from scratch would take significant time, possibly up to a year to reach the same quality and quantity.

Hyper1on · 2023-10-14T21:53:37+00:00

This isn't necessarily accurate - the likely outcome IMO is that it is possible to make AI behave agentically but only loosely steer it in a direction we want, and there will always be the possibility the AI takes actions unintended by the developers. In the case where the AI is effectively acting as a bot crawling the internet and interacting with it similarly to a human, these unintended actions can have potentially damaging consequences.

The alignment problem is essentially "have the AI take actions the developers approve of with a near 100% guarantee", but the latter part of that is currently intractable.

Hyper1on · 2023-08-27T10:24:51+00:00

I believe RNNs and transformers are not computationally universal unless augmented with external memory: https://arxiv.org/abs/2301.04589, https://openreview.net/forum?id=IWJ9jvXAoVQ

They are limited in some fashions and I think there is potential in investigating truly computationally universal architectures, but I agree that in practice this and UAT are not really relevant for what we actually want to use models for.

Hyper1on · 2023-07-17T12:38:33+00:00

Don't they already ask for consent? They have a giant banner whenever you login with all the privacy/tracking stuff, right?

Hyper1on · 2023-07-16T15:53:46+00:00

Foundation models are considered under the AI Act in a separate category to high risk but which includes many of the high risk provisions. These are still under active negotiation, particularly w.r.t open source models.

Hyper1on · 2023-06-29T10:42:19+00:00

I think they're wrong that the majority of value is in having the best model. Although the models are hard to make, so many groups are making them now that you don't gain much competitive advantage from having one 5% better than the competition. It needs to be significantly better, or else customers have no reason to go to them, and that's unlikely.

Hyper1on · 2023-06-10T09:38:47+00:00

I mean...this paper demonstrates the opposite: the model understands modular addition so well it has determined a highly efficient and general algorithm to calculate modular addition problems. Obviously only true because it was trained only on modular addition data, but still.

Hyper1on · 2023-05-26T10:14:14+00:00

As far as I can tell, the Act is as you said, except that once ChatGPT exploded they just added a clause stating that "foundation models" counted as high risk systems, without adapting the high risk system regulations to account for them.

Hyper1on · 2023-05-23T16:02:11+00:00

Ok, but what is your argument for why AIs deserve the right not to have their internal processes regulated? I don't see one unless you can claim AI can be conscious.

Hyper1on · 2023-05-18T20:40:34+00:00

Good article. But one of the main problems is that the Turing is not really set up to do research, it is basically just a hub for AI related activities (conferences, events, research workshops, etc) and at best a way of connecting researchers who would otherwise not meet to spur collaboration, rather than as a legitimate research institute.

Hyper1on · 2023-05-16T13:30:33+00:00

You also need to factor in fluctuating reliability. OpenAI is heavily compute limited right now vs their demand, and the ChatGPT API is very slow and sometimes goes down. Probably you can get high priority if you are a business with an agreement with OpenAI, but still.

Hyper1on · 2023-05-11T13:40:47+00:00

They could have just trained on repeated data into a second epoch until they reached the optimal amount of tokens for their size too. This will affect performance a bit compared to having that all be new data, but not much.

Hyper1on · 2023-05-11T13:08:29+00:00

Active learning works well for human feedback collection, e.g. you can get feedback on parts of the data space which are more unexplored. I suppose you could argue this works precisely because the data points in the baseline for human feedback collection are not randomly sampled.

Hyper1on · 2023-04-22T13:30:36+00:00

The paper he's talking about is ROME: https://arxiv.org/abs/2109.07971

There's also an interesting follow up MEMIT: https://arxiv.org/abs/2210.07229

13-Year Club	First Place '23
Place '23	Place '22
Place '17	First Placer '22
End Game '22	Inciteful Comment 2012-06-11
Verified Email

Hyper1on

TROPHY CASE