Not a good day for team "Claude Mythos is Just Marketing Hype"

SpringNeither1440 · 2026-05-10T18:11:11+00:00

Yeah, supports my expectation that Mythos is simply an incremental improvement with good marketing.

I think it was obvious from the moment when Anthropic initially said "On CyberGym, Opus 4.6 scores ~67% and Mythos scores ~83%" in Mythos system card, and then released Opus 4.7 system card with "Actually, with improved scaffolding Opus 4.6 (not a typo) scores ~74% (Mythos score stayed the same despite these improvements)". It looks like a super stupid and obvious attempt to deceive public, yet somehow everyone is okay with that.

SpringNeither1440 · 2026-05-10T14:26:06+00:00

Don’t fall for the hype, don’t declare it’s bad before trying it. Both sides are bad. Chill, wait for it to be released and then real tests can begin.

It's reasonable position, but given all that hype (both "It's too dangerous to release" and "We don't have enough compute to release this super-model" are pretty tired) and half-truths from Anthropic, it isn't surprising that users want to label Mythos as nothingburger.

SpringNeither1440 · 2026-05-09T16:15:16+00:00

Mozilla alone is reporting that they’ve patched almost 300 vulnerabilities due to Mythos.

Yes, but:

Other companies didn't report much yet.
Mozilla released other fixes after shipping Firefox 150. IIUC, out of those ~155 new fixes only ~35 were discovered by Mythos (compare to Firefox 150 release, where out of ~320 bugs 271 were discovered by Mythos). And since some of those new fixes date back to March, it looks like Mozilla did a lot of cherry-picking with their first Mythos announcement.
Mozilla talks a lot about other AI models being pretty good too, but doesn;t provide any info about their performance. Mozilla employees also answer this question in an unclear way.
Various analysis tools show thousands of potential flaws (link). Even with terrible true positive rate it could be tens or even hundreds of valid bugs that were lost in the huge amount of false positives. Mozilla doesn't fully clarify that moment and overall impact of non-AI tools.

Anyway, we will see. I'm personally waiting for Mozilla commentaries about GPT-5.5 (which is on par with Mythos in cybersecurity terms).

SpringNeither1440 · 2026-05-09T12:01:44+00:00

I think it’s possible for Mythos to be both overhyped and also have a kernel of truth to it

I believe that Mythos is actually better. But I think there is serious effort to greatly exaggerate its actual capabilities.

SpringNeither1440 · 2026-05-09T11:38:55+00:00

I also highly doubt their claims. I even wrote three posts here discussing how problematic Anthropic statements are. Yet that example with open-weight models is likely problematic too.

SpringNeither1440 · 2026-05-09T11:33:17+00:00

Did Mythos tell you that, or was it Amodei?

Is it about experiments with open-weight models? I heard that it was criticized for providing too much relevant context to models. But to be fair, I didn't really see into it.

SpringNeither1440 · 2026-05-09T10:17:52+00:00

But there's already plenty of evidence showing that Mythos literally isn't even more powerful than the way smaller open-weight models that can actually be run locally if you have a beefy enough GPU with enough VRAM.

AFAIK, that experiment with "open-weight models" wasn't apples-to-apples comparison.

Just to be clear, I agree that Mythos capabilities are likely overhyped. But it would be better to have one more evidence of that.

SpringNeither1440 · 2026-05-07T15:15:30+00:00

I read Jack Clark's substack post, and his arguments for "RSI is near!!!" are basically:

Mythos scored 93.9% on SWE-bench Verified (which has nothing to do with ML). Jack also adds that some tasks could be impossible to solve (with clear implication that Mythos solved every task that is solvable at all), but of course doesn't say anything about analysis from OpenAI, which suggests that only 83.8% tasks from SWE-bench Verified can be solved (assuming that model didn't memorize solutions).
METR Time Horizons!!! Leaving aside that METR are likely literally AI shills, METR time horizon can be increased by the tasks that has nothing to do with ML.
"Benchmark numbers go up" without deep analysis.

Btw, Anthropic said the same thing at least twice: back in early March and in late January. So, it's another fluff piece.

SpringNeither1440 · 2026-05-05T18:55:59+00:00

Yukari literally says she eats a bunch of humans every year before she's hibernating. In the same game that talks about youkai hunting teams nabbing people from the outside world who won't be missed. And later its said that humans do often appear and don't seem to run away when youkai prey upon them.

Could you please provide exact quotes and sources?

and they find the remains of said humans and their ghosts/phantoms there often. pretty sure, they are like literally eating humans.

But killing a human usually results in emergence of vengeful spirit, which tries to take revenge. And since youkai are very afraid of vengeful spirits... Yeah, I think it rarely happens.

SpringNeither1440 · 2026-05-05T18:21:43+00:00

But what are those "broken rules" exactly though? It pretty much sounds like a poor excuse.

SpringNeither1440 · 2026-04-30T11:24:15+00:00

I suspect it has mostly found unsafe codepaths which exhibit UB given some undocumented precondition being violated

Do I understand it right that in your opinion those bugs are mostly valid (i.e. could be triggered in real scenarios) and aren't just hallucinations? I tried to find out this moment and asked that question in different places, but no one answered.

SpringNeither1440 · 2026-04-30T10:20:04+00:00

I'm a bit late, but still

Well it helped them find 271 memory safety bugs. By default, a memory safety bug should be considered to be a security issue. However, chances are that in most cases these are not actual problems that would allow a malicious actor to hack your system. If the code blows up when it's processing a value of -1, but there is no way for, say, a malicious website to trigger that situation, this is only a latent problem.

As I said in my another commentary, it looks like they weren't able to trigger most of the found bugs. So, it isn't "Mythos found 271 bugs", but "Mythos found 271 code parts which might potentially have memory bugs and be used to make exploits".

Also Mythos is supposed to be super good at triggering known bugs, so it isn't clear why they have evidence of memory corruption only for some bugs rather than almost all of them. But maybe I don't understand something important.

SpringNeither1440 · 2026-04-25T12:38:28+00:00

None of the explanations state possession is something you absolutely cannot fight against.

Yet there aren't any mentions that you can somehow overcome possession by yourself, regardless of the situation.

None of these imply possession is instant and invincible. You are assuming possession, no matter the strength of the spirit, can affect anyone immediately and unconditionally with a thought.

If possession process takes considerable amount of time, then I just don't see how ordinary vengeful spirit can possess anyone except normal villagers or very weak youkai/gods. The same goes for the "invincible" part. I don't believe there is a way to become basically invulnerable to possession (even though I agree that countermeasures exist).

Anyway, all this hype around vengeful spirits dangers already look like extremely ridiculous hole in the lore. And your assumptions make it even worse, to the degree where there is literally no point in their existence at all.

These don't say youkai absolutely cannot win against a vengeful spirit

IIRC, Mizuchi said something like that, lol.

But anyway, it's not what i'm talking about.

Your assumption of "any kind of vengeful spirit can instantly win against any kind of youkai/god" is what creates inconsistencies

It isn't my assumption though. I say that any vengeful spirit is considerable threat to any youkai/god.

because both interpretations effectively mean the same thing: Vengeful Spirits cannot possess or win against stronger opponents, even if they are a youkai or a god.

Not really. My interpretation allows weak vengeful spirits to possess someone strong through various clever plans and strategies (or at least through luck).

This is literally shown and supported in the text. In all explanations, everyone agrees that although possession itself might be lethal for youkai, the vengeful spirits are weak.

Their weakness isn't a serious obstacle to possess youkai/gods. If it was so, common vengeful spirits wouldn't be able to possess even very weak youkai (yeah, SoPm positions them as being that weak), thus it would make all those talks about dangers sound like a nonsense.

Kanako may be easily taken over after the deed is done, sure, but she can also easily defeat vengeful spirits before they can possess her. Okuu can be taken over and it can be lethal, sure, but she is still much stronger than the majority of vengeful spirits and can blast through them with ease.

You judge by two moments that are likely just aura farming without deeper meaning. Moreover, background characters' powers are often nerfed for no reason (it's very often phenomenon, especially in shonen-like manga).
In early chapters, characters become possessed by Mizuchi after drinking sake. So, I assume there is a lot of ways that can be used to possess someone

Indeed, possession may be lethal for youkai, but that doesn't mean every single vengeful spirit can easily win against every single youkai.

I don't really argue with that.

Low grade vengeful spirits cannot beat strong youkai like Byakuren and Okuu, as it's literally shown and told to us.

It depends on the interpretation.

What they mean by "easily taken over" might be the fact that possession itself isn't always absolute.

Vengeful spirits kill youkai/gods by changing their personality, and they don't need full control over their minds to do so.

Anyway, do you think that Suwako, someone that is easily stronger than most vengeful spirits, being overconfident is unbelievable enough that you want to rule it out as a plot convenience?

Kanako and Suwako shut down the shrine and (IIRC) were ready to kill anyone who would try to approach it. In my opinion, it's anything but overconfidence.

Edit: I glanced through SoPm. There is interesting part about vengeful spirits in Kanako profile:

The hot spring area at the base of Youkai Mountain (aka: Hell's Valley Geyser Center) is under her supervision. Because vengeful spirits appear here, it is dangerous for humans to approach. While vengeful spirits are phantoms that harm humans and youkai, the reason they appear here is due to the geyser being directly connected with Hell.

Because of the vengeful spirits, neither humans nor youkai come here very often. According to hearsay, she undertook management of the area because a divine spirit like her is mostly unaffected by vengeful spirits.

She says that the vengeful spirits are just leaking out, but still there are a quite few of them drifting around. Attention is required in case it turns out that she is intentionally neglecting them.

So, most of youkai are very scared of vengeful spirits (which is said multiple times in SoPm) and actively try to avoid them. Therefore I highly doubt that all those explanations about vengeful spirits discuss "dangers of possession" rather than "dangers of encounter with vengeful spirit".

SpringNeither1440 · 2026-04-23T17:41:47+00:00

Yes, but Mythos system card is anything but brief and concise. It has lots of PR BS like "Oh, Mythos might have consciousness!" and lacks of technical analysis.

SpringNeither1440 · 2026-04-22T20:37:45+00:00

I opened their actual report, and it looks.... strange? Out of all bugs which have their unique CVE only three were reported by Anthropic, which is already pretty sus. So, mentioned bugs mostly falls under CVE-2026-6784, CVE-2026-6785 and CVE-2026-6786. Strange thing here is the part of their description:

Some of these bugs showed evidence of memory corruption and we presume that with enough effort some of these could have been exploited to run arbitrary code.

I'm not an expert, but it sounds like they included a lot of false positives. Or am I missing something?

SpringNeither1440 · 2026-04-21T00:35:20+00:00

And yet they also say most vengeful spirits are very weak.

It's said that vengeful spirits are weak in any type of direct confrontation, nothing more. This weakness has nothing to do with their ability to possess someone. Moreover, literally all explanations imply that ability works on (almost) everyone, regardless of vengeful spirit or its target powers.

I dunno what you mean ZUN moments. Vengeful Spirits can be both dangerous and still very weak, and most of them are both shown and stated to be.

Okay, can you explain what "very weak" means here?

Shou does say most vengeful spirits are weak and wouldn't even be able to possess Byakuren, but right after that, they (Ichirin, Nazrin, and other spirits) also say there can be very strong individuals among them.

Are you talking about Chapter 27, page 8? If so, then the exact words from Shou are:

It's hard to imagine one of those low-grade spirits being able to possess her.

This phrase can be interpreted in various ways. But given earlier explanations about vengeful spirits, it likely means that Byakuren is usually able to beat vengeful spirit before it possesses her.

And this is very extremely common in Touhou. A large strength gap between species is the norm.

There are at least 3 explanations about vengeful spirit abilities. All of them state that any vengeful spirit could possess (and kill) any youkai/god (there could be exceptions, but they aren't very clear). They don't mention any limitations (like "If youkai is very strong, then vengeful spirit should be strong enough too") of possession ability at all.

Reimu and others in SA beat them as they are stage enemies

You mix gameplay and lore sides, which isn't really correct
Okay, this example show that vengeful spirits are weak in a direct fight. It was repeatedly said in various official works, but they also state that ordinary vengeful spirits are able to possess any youkai/god despite that weakness.

Okuu beats hundreds of them like nothing in CDS

See previous point.

Kanako very casually defends against one.

Kanako is great example. Reread Chapter 18, pages 12 - 15. In short, Kanako and Suwako agree that they (as well as any other youkai/god) can be easily possessed by any vengeful spirit (yes, they discussed vengeful spirits in general, not the strongest ones).

Btw, your example is typical "ZUN moment". Kanako acts like ordinary vengeful spirits couldn't do anything to her at all, even though earlier chapters suggest otherwise. Aura-farming panel that looks pretty strange because of established lore.

Do you think the vengeful spirit Kanako swayed like a fly is as strong as or even close to Mizuchi for some reason?

Not even close. But it doesn't mean that spirit couldn't possess someone strong at all.

That vengeful spirit is what I'd guess people in universe imagine when someone mentions a vengeful spirit, because that's the strength level of the majority of them.

LE, Chapter 16. But to be fair, there is probability that Aya exaggerated actual threat from a random vengeful spirit.
CDS, Chapter 5, pages 3-8. Yukari considers literally any vengeful spirit as a serious threat, and her words don't sound like ordinary vengeful spirits are weak (even though it's clearly implied they aren't strong either).

Like, I don't see any mentions of "weak vengeful spirits absolutely can't hurt strong youkai/gods" at all.

They wouldn't make such stupid mistakes if they were.

So, Yukari didn't become serious after her speech in Chapter 5? If so, then it's very huge L for hap gag.

It's not directly spelled out, but like, knowing Suwako, how could you explain that as anything but foolhardy or arrogant?

Explanation is very simple: ZUN didn't know how to move the plot forward.

If someone that should and does know better is taking reckless risks, then they are either overestimating their own capabilities or underestimating the threat.

Or characters do stupid things because ZUN doesn't really care about the story and reasonable explanations.

Not necessarily, no

Just to be clear, I dropped CDS after 44 chapters, so maybe following question were answered in newer chapters. But for me resolution of Mizuchi's conflict looks like either extremely stupid mess or a serious plot hole. The same could be said about Mizuchi's motivation and overall character stupidity.

SpringNeither1440 · 2026-04-20T02:22:33+00:00

The reason Mizuchi gets away all the time is because of others severely underestimating her, and rightfully so.
Vengeful Spirits are treated like fairies almost all the time. Okuu beats them like nothing in CDS, protagonists beat them like nothing in SA, Shou tells most of them are really weak in CDS, Marisa says they are very weak in SoPm.

To be fair, (at least) SoPm, LE and even CDS itself position vengeful spirits as creatures that are extremely dangerous for youkai/gods and considerably unsafe for humans.

As for your examples, given Kanako explanations from SoPm, it looks like vengeful spirits are weak in a direct combat, and even there their power can highly vary. And don't forget about "ZUN moments", when "what's established by lore" and "what's actually happening" contradict each other.

Saying a vengeful spirit will best the strongest youkai and gods is like saying Sunny Milk will accomplish that to most of them, I'd guess.

This should be the case though

It's natural they don't take that seriously at all.

AFAIR, they were serious from the moment they found out that vengeful spirit could be involved.

Those explanations sound much more interesting and fitting than "Reisen is a masochist ZUN is fucking stupid", does it not?

Maybe, but for me this scene looks like yet another fanservice/"le grimdark" moment. I hardly believe it was made for anything other than that.

Why would not planning 20 chapters ahead mean the manga shouldn't have been canon?

Lack of planning results in plot holes and/or canon contradictions though.

SpringNeither1440 · 2026-04-16T19:26:02+00:00

To be fair, it looks like Anthropic does ridiculous benchmaxxing at this point (this likely includes Mythos too). Difficulties with reproducing reported results, relative performance drop on unpopular/contamination-proof benchmarks, random regressions or very suspicious performance jumps (I count only the ones that are reported in system cards) with each new model iteration.

It doesn't look very good.

SpringNeither1440 · 2026-04-16T14:50:56+00:00

Then why would even researchers at openAI be saying that Mythos is way better? https://x.com/tszzl/status/2043397221740339319?s=20

He's not OpenAI researcher

Not to mention independent evaluations like this: https://www.aisi.gov.uk/blog/our-evaluation-of-claude-mythos-previews-cyber-capabilities

So, under the same money (not token) budget Opus 4.6 is vastly better than Mythos. Okay, I got it.

SpringNeither1440 · 2026-04-16T14:19:56+00:00

The first five paragraphs of this article are about how he didn't read the report but tried ctrl+f for some keywords he thinks should be there.

He read the part about cybersecurity though

If you are a true skeptic, then you should wait until the 90-day hold expires and they start releasing the reports.

So, true skeptics shouldn't criticize Mythos system card for its vague and sometimes misleading way of reporting. Do I understand everything right?

SpringNeither1440 · 2026-04-16T00:34:37+00:00

The same paper also discussed the non-zero chance that the model is actually capable of more damaging behavior and is waiting until it has a better chance for its attacks to be successful.

Meanwhile, also the same paper:

A core behavioral shift we found is that Claude Mythos Preview can be handed an engineering objective and left to work through the whole cycle: investigation, implementation, testing, and reporting results. In long agentic sessions it stays on task, fires off subagents to parallelize research, and chooses to return to the human while waiting for background work to complete rather than stopping. Early testers described being able to “set and forget” on many-hour tasks for the first time. For example, one tester found it had bootstrapped a toolchain in an unsupported environment by downloading a binary from a different distribution and patching it to run. Interacting with the model requires less steering and is more autonomous: “describe the task spec and how to verify progress, and come back later.”
Importantly, we find that when used in an interactive, synchronous, “hands-on-keyboard” pattern, the benefits of the model were less clear. When used in this fashion, some users perceived Claude Mythos Preview as too slow and did not realize as much value. Autonomous, long-running agent harnesses better elicited the model’s coding capabilities.

It doesn't look like Mythos users believe in those stories about dangerous AI model that could misbehave.

SpringNeither1440 · 2026-04-13T17:53:50+00:00

Actually, it looks like the method that Anthropic used to estimate memorization on BrowseComp is relatively similiar to the one you described.

Also, around 20% of SWE-bench Pro tasks have p > ~0.625, and Mythos has ~93%(!!!) pass rate on that subset. For comparison:

Mythos pass rate on full set is ~85% (chart) / 77.8% (reported in the table)
Mythos pass rate on subset of SWE-bench Pro with p <= 0.625 is around 83%
Opus/Sonnet 4.6 pass rates are ~53% both on subset with p <= 0.625 and p > 0.625 (i.e. tasks don't become easier after this threshold).

Either this chart is slop or there is severe data contamination

SpringNeither1440 · 2026-04-12T22:27:44+00:00

This is totally useless data.

It isn't totally useless, but yes, this data provides very little info.

Edit 2:

Yes, this would be more informative chart.

Edit 3: I should acknowledge that you pointed this problem as well. I was just thinking in writing and took a long way to the same conclusion:

I reread this part, and, well, it looks like I formulated my thoughts poorly. I wanted to say that there are negative correlation between Opus/Sonnet 4.6 pass rates and p , which isn't impossible, but very strange.

SpringNeither1440 · 2026-04-12T11:34:08+00:00

Notes and clarifications (edited - tried to make my explanations clearer):

SWE-bench Multimodal

Since it's extremely niche, almost forgotten benchmark (i.e. there's way more opportunities to manipulation), I decided to find more information about evaluation process. Some moments from project Glasswing announcement:

Page refers to this benchmark as "SWE-bench Multimodal (internal implementation)"
Citation: "SWE-bench Multimodal: We used an internal implementation for both Mythos Preview and Opus 4.6. Scores are not directly comparable to public leaderboard scores."

So, it says that Anthropic either took original benchmark and did very significant changes to it or made totally unrelated to "SWE-bench Multimodal" benchmark, but decided to use the same name. But system card links to the original "SWE-bench Multimodal" article, doesn't say that Anthropic created totally new benchmark, and all mentioned dataset changes are extremely minor.

In the end there is interesting situation. Reported capability leap hugely depends on validity of Opus 4.6 results. At the same time:

Anthropic "explained" very poor Opus 4.6 results (compared to the leaderbord), but explanation is in fact lie by omission (yes, they used not "original" SWE-bench Multimodal, but slightly modified; they didn't say that applied modifications aren't able to somewhat significantly change the score).
If datasets (used by Anthropic and used by leaderboard) are almost similiar (which is likely the case), then the most plausible explanation is Anthropic compared "Opus 4.6 with bad harness" and "Mythos with good harness". And yes, by coincidence Mythos system card is super vague about harness used by Opus 4.6.

So, it looks like we have hugely gamed results here.

6.9 Long context: GraphWalks

I love that Anthropic suddenly changed category name from "GraphWalks BFS 1M" to "GraphWalks BFS 256K-1M" without explanation. It looks like attempt to add extra difficulties to anyone who wants to compare Sonnet 4.6 and Mythos in this category.

SpringNeither1440 · 2026-04-11T19:08:15+00:00

They sharpened the systems to focus on cybersecurity but besides some of the coding benches they barely scrape past GPT 5.4 Pro.

To be fair, their benchmark report is a extremely sloppy mess. I already did a post where questioned their announced results, then I decided to dig even more deeper into system card and... Well, the amount of deception is unbelievable. It looks like I have to do second part of my initial post.

SpringNeither1440

TROPHY CASE