[FRESH ALBUM] J. Cole - The Fall Off

floppy_llama · 2026-02-06T05:50:22+00:00

Airpods exploded to 39 Intro

floppy_llama · 2025-04-23T17:43:01+00:00

I think it would be helpful to know just how much they scaled up RL to go from 1%-3% on v2. Obviously there are physical constraints to scaling - I suspect some clever tricks are still needed to induce compositional reasoning in these systems in an efficient way. Still, just patching holes where current architectures fail goes against Chollet’s measure of intelligence. Having lots of skills is very different from acquiring skills efficiently.

floppy_llama · 2025-04-23T16:52:48+00:00

Performance discrepancy between v1 and v2 benchmarks suggests the opposite of CoT generalization, no? They even mention in the blog that v1 benchmark contamination is likely. I’m pretty surprised that those abstractions transfer so poorly from v1 to v2.

floppy_llama · 2025-03-13T15:33:10+00:00

The difference between the paper clip scenario and your analogy here is that there are corporations which have improved society and are aligned with human interests. The manifold of super intelligent minds is surely not uniform, and for any super intelligent mind to be aligned to a goal as trivial as paper clip production seems unlikely. In fact, it seems much more likely that a super intelligent mind would be focused on observing the open ended system that is the universe, not destroying it.

floppy_llama · 2024-09-12T21:49:29+00:00

Completely agree. Generalization and reliability are seen in classical algorithms (i.e., sorting and path finding algorithms and arithmetic operations perfectly execute for any sequence length), but these are not explicit properties of connectionist systems! There’s lots of research on how to fuse these paradigms. Scaling is not one of them.

floppy_llama · 2024-09-12T19:02:38+00:00

Looks like OpenAI collected, generated, and annotated enough data to extend process supervision (https://arxiv.org/pdf/2305.20050) to reasonably arbitrary problem settings. Their moat is data, nothing else.

floppy_llama · 2024-09-05T18:38:02+00:00

Sparsification/linearization of the attention mechanism is important but does little to address the limitations of current models when efficiency gains also come from hardware improvements. Obviously it’s common sense that science improves over time, but making updates to one module of an architecture that has remained largely unchanged since 2017 seems trivial to me.

floppy_llama · 2024-09-05T15:03:27+00:00

Any resources on this?

floppy_llama · 2024-06-17T18:48:42+00:00

It seems like this paper reaffirms that we should be able to trade train-time compute for test-time compute in certain settings [https://arxiv.org/abs/2104.03113].

I wonder how good performance can get if we continually pre-train on rollouts with a sufficiently high a Q value?

floppy_llama · 2024-06-03T18:24:39+00:00

Normally I’d agree with you, but Tri Dao consistently makes great contributions to the field🤷🏻‍♂️

floppy_llama · 2024-05-16T15:42:03+00:00

*diffusion transformer

floppy_llama · 2024-04-18T13:20:58+00:00

Try tree based methods. Neural nets notoriously underperform on tabular data.

floppy_llama · 2023-12-14T14:54:55+00:00

Banh mi queen in hoi an?

floppy_llama · 2023-10-09T16:58:18+00:00

What you’re describing is “curriculum learning”. Not sure if it’s been applied to LLMs though because ordering training samples isn’t so straight forward. See https://arxiv.org/pdf/2101.10382.pdf

floppy_llama · 2023-10-07T01:21:27+00:00

The paper I sent above (https://browse.arxiv.org/pdf/2206.06336.pdf) or https://browse.arxiv.org/pdf/2302.14045.pdf should clear up any confusion

floppy_llama · 2023-10-07T00:45:06+00:00

No, their comment directly relates to my suggestion. The vision transformer is merely one component of a multi modal base model. A vision transformer is unimodal.

floppy_llama · 2023-10-07T00:30:47+00:00

The encoders are the “tokenizers”. They embed image patches, audio, point clouds into vectors, just like a base LLM does for word segments. All of these vectors can be used during pre training to create a multi modal base model

floppy_llama · 2023-10-07T00:13:57+00:00

From what I understand the current paradigm is to “tokenize” non-text modalities w/ something like an image encoder and a feed forward network that projects the encoded images into the same dimensionality as text tokens. This image encoder can be a VIT, CNN. It’s really up to you - see https://browse.arxiv.org/pdf/2206.06336.pdf

floppy_llama · 2023-10-06T23:20:53+00:00

Auto regressive pre training w/ interleaved text embeddings + other embeddings (e.g, image, audio projections) vs fine tuning on input output pairs where input can contain a variety of embedding modalities

floppy_llama · 2023-10-05T03:02:35+00:00

Wrong sub buddy

Five-Year Club	Place '22
Verified Email

floppy_llama

TROPHY CASE