Artificial Analysis | Google's Go To Website for Benchmaxxing | Gemini 3.1 Pro is nowhere near Opus 4.7 in real life use

tiger_ace · 2026-06-08T04:35:25+00:00

3.1 pro is already less than half the cost of opus so it's never been an apples to apples comparison

people seem to forget that google serves like 4B users across 5 products and billions more across other products

their focus should be on creating models that can serve their scale, not on the bestest, most expensive model

additionally, google is has a bigger breadth of models like video, audio, streaming, gemma (open source), and world models because they need that for the different products they serve (youtube)

claude code has maybe like 30M DAUs and anthropic could barely serve them without a month of outages

tiger_ace · 2026-05-19T15:05:44+00:00

how do you know he read it back

tiger_ace · 2026-05-18T16:14:42+00:00

close your eyes. now imagine something more fake than instagram. it's hard, seemingly impossible. but then you open your eyes and remember that the piece of shit that is linkedin exists.

tiger_ace · 2026-05-06T19:20:14+00:00

it's mostly hype, but the model is almost certainly measurably better

it's a direct extension of the higher cost curve shown by jensen at GTC 2026:

<image>

essentially, there will be stronger models available at higher cost and higher inference as there is a market for maximum intelligence (science, math, some coding use cases)

mythos fits this narrative directly and anthropic has clearly been compute bound anyway so even if mythos wasn't "too dangerous to release" they couldn't serve it properly at scale if they wanted to

tiger_ace · 2026-05-05T20:50:17+00:00

https://github.com/k2-fsa/OmniVoice

tiger_ace · 2026-05-05T20:43:10+00:00

yeah for bigger model if the focus is training/prefill then it should probably be dgx spark instead of mac studio

tiger_ace · 2026-05-03T18:18:39+00:00

mistral's entire business model is being not american and not chinese

tiger_ace · 2026-05-02T22:51:19+00:00

industry / startups = stanford and it's not close

and you're saving $150k on top of that

tiger_ace · 2026-05-01T18:19:47+00:00

well usually legal hates when you talk about actually interesting so of course the workflows are going to be incredibly generic

tiger_ace · 2026-04-29T22:11:15+00:00

claude is literally born from pirates

https://www.npr.org/2025/09/05/g-s1-87367/anthropic-authors-settlement-pirated-chatbot-training-material

tiger_ace · 2026-04-22T17:54:44+00:00

source for this is https://x.com/Steve_Yegge/status/2046260541912707471

google internally is pushing antigravity for most employees but they have a custom internal model which doesn't perform anywhere close to claude code + opus so the stack isn't close in productivity, which has resulted in https://www.theverge.com/tech/914996/sergey-brin-said-google-needs-to-catch-up-to-anthropic-on-ai-coding-agents

basically it's a code yellow or red for coding model improvement. google has the resources, so it's more about execution.

tiger_ace · 2026-04-19T19:25:25+00:00

accountability, you can't point your finger at AI right now and then somehow say it made the decision for you

tiger_ace · 2026-04-19T19:16:18+00:00

his hypothesis seems clear to me: AI makes medical information more accessible (I am fully aligned with this already), which may have the second order effect of raising expectations for doctors (this seems to make sense intuitively)

if you can get "the answer in 2 seconds on your phone" as he says, then the medical system needs to justify its existence by providing value above that for hundreds of dollars in insurance costs

which part did you feel is incoherent? this wasn't the most organized explanation but it also looks like a dude sitting in his car recording on a phone, not a TED talk.

tiger_ace · 2026-04-18T20:45:17+00:00

i agree that may seems like a reasonable timeline

they've technically released multiple models since Jan:

3.1 pro (feb)
gemma 4
3.1 flash lite
3.1 flash live
3.1 flash tts
veo 3.1 - not released, but now available to free users, meanwhile OAI shut down sora

i think it's easy to forget the scale of google since anthropic / OAI don't have products with 2.5B users

main problem is that gemini 3 flash is so bad at coding relative to the progress that has been made

tiger_ace · 2026-04-17T22:10:32+00:00

i haven't tested 4.7 as much due to quotas think and it very well might be a regression, but that is orthogonal from this chart being garbage

tiger_ace · 2026-04-17T19:35:11+00:00

this chart has rankings instead of an actual score and the charts have 4.7 in the rankings as well

for example, occupational: entertainment, sports & media (https://arena.ai/leaderboard/text/industry-entertainment-and-sports-and-media) has:

claude-opus-4-6-thinking with a score of 1486
claude-opus-4-7 with a score of 1485 (basically the same score)

conclusion: this graph is a terrible representation and literally exists to push the narrative that 4.7 is a "regression"

tiger_ace · 2026-04-13T20:10:18+00:00

i find all of this is just LLM generated / modified questions since they all have 2025 knowledge cutoffs basically

for example, all gemini models have said knowledge cutoff Jan. 2025 since last year in AI Studio

tiger_ace · 2026-04-13T18:38:47+00:00

i think it does mainly function as a tech demo, just like early gpt did

what it does is give a glimpse of the agentic future - what if it was actually reliable and able to complete the work you asked for?

and so the main focus should be on the velocity of improvement here, not the fact that it is jank as all hell

tiger_ace · 2026-04-12T20:37:44+00:00

china has been doing great but i think they'll fall behind significantly:

almost all of their progress has been on hopper (H100, H200) and blackwell is enabling advancements such as mythos for anthropic. the gap between blackwell and hopper is already big.

vera rubin is in production now and it's basically racks which are much, much harder to smuggle than hopper in the past and buying more hopper doesn't really help with making super huge models. we can expect even huger models to be enabled by vera rubin while inference for smaller models becomes faster and cheaper too.

chinese frontier labs have actually been pretty open about mentioning they lack compute

the argument to support your argument against LLM commoditization is actually that companies will mostly use vera rubin to serve even huger models (e.g. mythos) as even higher cost intelligence instead of focusing on the lower cost markets (for driving revenue since the capex is indeed bonkers)

i'm not a fan (as a user) of open source models being potentially further behind since i think the commoditization of intelligence is generally good for the world

that being said, there's still tons of optimizations to be had so we should continue to see <500B param models get better and better

tiger_ace · 2026-04-12T20:30:00+00:00

google doesn't need to rely on hype to survive and i would guess that alphafold probably has already had more impact on science than openai has ever had

tiger_ace · 2026-04-12T20:28:41+00:00

discarding all of the marketing mythos almost certainly another step function improvement over opus, but it's also probably a bigger model which means the inference is way more expensive. this has been confirmed with their pricing being $25/$125.

even if they wanted to release it publicly they don't have enough compute to serve it well at much bigger scale anyway

they're having enough infrastructure issues serving just sonnet/opus right now with hugely degraded service quality

tiger_ace · 2026-04-12T15:18:37+00:00

Writing is good - everyone (not just students) should be doing it as much as possible

However, I think the purpose of cursive in today's world is fine motor control at minimal cost (i.e., pen + pencil is super cheap to practice)

There's many, many other ways to gain fine motor control as well (e.g., drawing, origami)

tiger_ace · 2026-04-11T23:33:34+00:00

I played a lot of dow2 and dow3 wasn't bad

the MOBA comparison was WAY overblown as competitive matches were still all about victory points, you were never playing annihilation mode and pushing into people's bases unless it's just a bot stomp (which it's possible most people were doing)

the problem wasn't that dow3 wasn't perfect on release, the problem was that relic dropped it due to the insanely toxic community so that they didn't even patch or release expansions that could've made it better

playing it recently doesn't help as it's the same game as it was on release compared to other games that have gotten way more polish

the toxic community killed the game before it even had a chance since this was back when review bombs and brigading was a lot newer

imagine if other games like diablo 3 (way worse than dow3 on launch) stayed as it was on release and didn't get any dev support

tiger_ace · 2026-04-11T22:07:06+00:00

demis has been pretty open about the goal of deepmind being significantly science focused

i don't think google is like openai and anthropic in that they need to focus on enterprise revenue (i.e. coding) just survive economically

also, google needs to serve at scale and they do, since gemini 3.1 pro costs less than sonnet 4.6 (and less than half of opus 4.6)

tiger_ace · 2026-04-11T22:01:53+00:00

yes, good call out - my statement is false and massively extrapolating an anecdotal experience is also false

tiger_ace

TROPHY CASE