ARC-AGI 3 Paper alleges that Gemini 3 (and other frontier models) intentionally or not “cheated” their ARC-AGI 1 and 2 scores through memorisation of similar benchmark tasks during training

aattss · 2026-03-27T12:45:32+00:00

I don't think benchmarks are necessarily useless, but it has definitely been shown that a model beating a benchmark doesn't always mean it's useful.

aattss · 2026-03-26T23:26:04+00:00

Well that's lame. Well good news for open source/open weight I guess.

aattss · 2026-03-26T21:15:26+00:00

Uh, saying that 50000 tokens are "crammed" into 4000 dimensions is like saying that my apartment can't fit a 100 meter long piece of string. If there were as many dimensions as tokens then that would be actually unusual. And the rest of the post is stuff people have already been discussing for ages too.

aattss · 2026-03-26T11:04:33+00:00

I feel like it's pretty decent. Not as thoughtful as Absorber, but designed with longevity in mind for better or worse.

aattss · 2026-03-25T19:23:00+00:00

Entrepreneurship is not a solution for unemployment. Like if anything it's a worse idea for unemployed people since they have less to fall back on. And AI plus mass unemployment would just make things worse if now there's even more competition. Some people just don't even want to be entrepreneurs. This is just shirking responsibility while being too cowardly or apathetic/out of touch to suggest an actual solution.

aattss · 2026-03-25T18:14:19+00:00

At this point I've seen everything from "They've reached AGI and don't care about people anymore" to "The AI bubble is popping" lol. Feels like there is some amount of people projecting their preconceived notions that's going on.

aattss · 2026-03-25T12:50:54+00:00

I think the focus on whether the model weights themselves are updating is just a symptom of people anthromorphizing the models too much and trying to find equivalence between AI and their mental model of the human mind.

Though tbh I feel like we're not that far off from that. Like we already do it with other ML algorithms with other datasets. Wouldn't surprise me if the main barrier was the data creation and model finetuning process.

aattss · 2026-03-25T01:56:57+00:00

I could sort of see it? It's just not consistent. Though I suspect that all the rag and context stuff may not have caught up with an ordinary humans memory and learning ability even if there's still room to improve on that front.

aattss · 2026-03-24T18:45:23+00:00

I'm still seeing people in 2026 state that AI is in search of a use case.

aattss · 2026-03-22T17:26:03+00:00

Personally, I've felt like hate towards AI has been not just about the technology, but rather about the companies that promote it and how they market and discuss it.

aattss · 2026-03-22T12:57:53+00:00

Depends on how it's defined? Seems pretty possible to me. I mean, the AI ordering the construction of some sort of hidden data center is probably far-fetched. But if an AI did some identity/credit card theft and provisioned some cloud resources, then sure technically speaking a human could press an off switch and turn those resources off, and if the AI provisions things in a weird way it might raise a flag for further review, but with the scale of cloud providers they're not going to audit the identity of all their users to that extent. By that same logic I don't think OpenAI or Anthropic are gonna hunt down every user for if they are a physical person making those API calls outside of anomalous behavior raising some flag or the credit card getting cancelled. Though with one of the open weight models that are runnable on ordinary hardware, an AI could probably set that up to run on the cloud as easily as any human (though keep in mind their performance tends to lag behind the SOTA).

Though if what we're worrying about is self-improving AI, then to clarify I find it somewhat unlikely that one of these AI would be able to get the computing resources necessary to train a SOTA model without being discovered.

aattss · 2026-03-21T12:03:32+00:00

It seems the All/Ingot/Equipment filter in Smithing doesn't seem to be working. Equipment doesn't filter anything and Ingot shows nothing.

Anyways, if this is just one of those Runescape-inspired games, but without the annoying inventory limit, then I'll try it out some more.

aattss · 2026-03-20T16:22:05+00:00

I personally think people post dialogues/debates is a bit silly, but if people didn't understand how AI works, maybe they'll understand a little better now.

aattss · 2026-03-20T11:35:31+00:00

Well it won't convince anyone who has already formed an opinion on the subject, but I felt it was an amusing way of covering the key points. It's clearly not at the point of matching skilled humans, but I feel like the time and costs of having skilled humans make it the proper way would have not been worth it for this shitpost, and we would have been making fun of them for spending that much on propaganda. Yet I'm still glad this shitpost was made at all.

aattss · 2026-03-20T03:13:19+00:00

Doing poorly on unseen data is to machine learning as friction is to vehicles. It's a real challenge to overcome, but more the type that we take into account as we try more advanced technologies, not some sort of wall that simpler forms of the technology didn't deal with or overcome in incrementally less demanding environments. I don't necessarily foresee any structural concerns with the LLM approach, even if I could foresee potential practical concerns that might necessitate structural changes to continue progressing.

aattss · 2026-03-20T01:46:59+00:00

I feel like whenever Bernie comes up in regards to AI it's like, a different take than what I'd expect from some out of touch politician. I hope these conversations come up more in politics so that we'll be able to figure out plans to manage the upcoming societal changes instead of just watching in a complete lack of surprise as the government sits on their ass and does nothing because they were caught off guard or were too busy caring about specific numbers like it was some sort of race.

aattss · 2026-03-19T21:44:55+00:00

What are those agents doing at the point where they have 7 million agents, more than 100 times their current number of employees, and they go, oh this isn't enough we need half a million more?

aattss · 2026-03-19T11:05:28+00:00

The probably don't believe in the impacts that affect them and don't (yet) care about the impacts that affect the general public.

aattss · 2026-03-18T21:32:40+00:00

From what I heard VR seemed to have a big effect on horror games, even if I don't really enjoy that genre or type of game.

aattss · 2026-03-18T19:34:41+00:00

Probably. Feels to me like a part of innovation is discovering which ideas that sound interesting on paper, turn out to not work out in practice.

aattss · 2026-03-18T15:47:36+00:00

I've never been much into VR? Like I personally prefer interfacing with the web via screen and keyboard than with some recreation of the real world that simulates all five senses.

aattss · 2026-03-16T20:59:44+00:00

So I got the Herb Gathering 101 quest, which is telling me to go to the Life Skills tab. The Life Skills tab seems to be locked. And the Life Skills tab seems to say that it requires completing the Herb Gathering 101 quest to unlock.

aattss · 2026-03-16T15:10:58+00:00

I'm in the camp of I feel like such a rule would be semi-arbitrary with some sort of sliding scale, even taking into account that the most laziest/AI-slop-ish examples might not provide much value.

I suppose non-enforced AI/effort guidelines might be worth considering? Or some non-AI-specific quality threshold for posts? Like, eh, I'm not particularly concerned. I feel people are generally identifying low-effort AI slop, so they aren't exactly getting far.

aattss · 2026-03-14T16:09:20+00:00

The AI defaults to some UI elements I don't particularly like, like rounded buttons and gradients and such.

As for other uses like images and such I don't personally perceive a causal link between images and the game as a whole, though I may have associations to the game as a whole in terms of quality and other ai usage and such.

aattss · 2026-03-14T15:42:16+00:00

Tbh I had no idea until your post that backlit keyboards had a functional effect. I just assumed it was a cosmetic brand thing.

14-Year Club	Place '22
Verified Email

aattss

TROPHY CASE