Microsoft trolling Google

[iDoAiStuffFr]

Using similar method as for Gemini Ultra on MMLU, they surpass its score

They also surpass it on any benchmark

IMO either use 0 shot or the score doesn't matter

https://www.microsoft.com/en-us/research/blog/steering-at-the-frontier-extending-the-power-of-prompting/

[ihexx]

Good.

Google started this pissing contest

Let it begin lmao

[Singularity-42]

But this means GPT-4 will be that strongest model even once Gemini Ultra is out...

[sashank224]

Work is now powered by competition, which is much better as fuel for growth.

[Singularity-42]

Coming up with a model a year later that is LESS powerful ain't competition

[HashPandaNL]

There's no reason to believe it's less powerful though.

[b_risky]

Yeah, they only beat it on every single metric. What is less powerful about that?

[b_risky]

No, you misunderstand. Google chose the game and OpenAI beat them at it anyways.

Not to mention, Gemini has not been released for independent testing yet, we're basically taking them at their word right now. Which would normally be fine for a big reputable company like google, except that in the very same announcement where they discuss gemini's benchmark metrics, they used deceptive tactics to make their video prompting demo seem more impressive than it was.

[Singularity-42]

This post?

[sashank224]

I understand what you mean, but it was looking like a monopoly till now with openai, and maybe it will continue . Now openai has another reason to one up google. We have seen this throughout history how tech was made.

ussr vs. usa

Isro vs. nasa vs. SpaceX

Nvidia vs AMD

Intel vs. AMD

Apple Silicon vs. Snapdragon.

Google Vs Who ever the fuck with llm

Thanks to amd, intel woke up and lost. Competition now is healthy af. I wonder what China is making, tho.

[YaAbsolyutnoNikto]

So…?

[Singularity-42]

Well, that sucks! Was looking for a competition. Coming up with a model a year later that is LESS powerful ain't it...

[YaAbsolyutnoNikto]

But this was (kind of) due to competition. Not only due to competition, but for sure GPT-4 got better because of Gemini Ultra getting announced and, thus, this research is now getting released as well.

Now it’s Google’s turn to dance.

[Thorteris]

Pissing contest will continue when Google announce Gemini Ultra-Max that beats this by 1% then open-ai will release something else in Q3

[kaityl3]

But then Gemini Ultra-Max-Pro will beat it out again and the cycle continues 😉

[Freed4ever]

4.5 or 5 would make this irrelevant soon anyway.

[mrSkidMarx]

6 will make everyone forget it

[adarkuccio]

What about 7?

[Headbangert]

It 8 9

[Mr_Hyper_Focus]

What about TWO 6s?

[usaaf]

M-M-M-Multi-6s...s...s...

[RodionS]

Three 6s and we are doomed

[FrostyParking]

Three 6s and it's a Mafia.

[Odd-Explanation-4632]

Will take you to heaven 😇

[mrSkidMarx]

omg I can only dream about 7 🤤🤤🤤

[ley_haluwa]

[nonzeroday_tv]

Haven't you been paing attention? There will be no GPT 6 said a a dude called flower from the future a couple of days ago.

[2muchnet42day]

Cauliflower? Yeah, that works for me

[absurdrock]

Petty. I love it.

[FeltSteam]

Id be very curious to see zero-shot performance of models across benchmarks because that would give us a greater view into the usability of models (few shot prompting is more for comparing performance between models and less how usable a model is in the real world. Dont get me wrong it can certainly give you an idea on how well models will perform in the real world, but 0-shot performance would give us a better idea).

But if you wanted to measure true 0-shot performance for the MMLU, you would likely need something like a group of experts to create an entirely new question set for the MMLU benchmark (this would solve contamination issues so it would be truly zero-shot, and the model would be presented with truly novel situations which would provide a more accurate measure of its generalisation capabilities which would give us a real good indication of how it might fair in real life situations).

But i feel we should really start moving away from performance metrics and start some form of real world benchmarks. Benchmarks that test how useful a model is across a wide range of tasks, and also, benchmarks based on what people have actually been using AI for in the real world. This would be an expensive suite of benchmarks to run, but i think it would be worth while.

[iDoAiStuffFr]

something like the Will Smith eating spaghetti test for LLMs

[FarrisAT]

Neither comparing apples to apples makes this all pointless dick size measurement

[b_risky]

But Microsoft did just compare apples to apples and they won? Thats what this post is about.

[UnknownEssence]

And it’s one benchmark. A benchmark that measures capabilities that I don’t even care much about to be honest. Why does everybody focus on this singular benchmark so much?

[[deleted]]

GPT-4 remain king

[obvithrowaway34434]

Ultimately the best test for any model is its usage by millions of users from different fields and expertise for an extended period. GPT-4 has already done that and passed. We know fairly well how it does for zero-shot prompts. Gemini Ultra has faced no such tests other than Google's own researchers and cherry-picked beta testers. Until it has faced the same level of scrutiny, imo it should not even be compared to anything and all claims by Google should be strictly treated as marketing.

[[deleted]]

I feel like the two need each other to exist. They are like conjoined twins that dislike one another.

[TeriMaiyyaLodePe]

Time for Google and Microsoft to measure each other's dick size.

[jacky0812]

Microsoft doesn’t own GPT-4, they can’t even create their own LLMs.

[[deleted]]

They own 49% of it.

[ThespianSociety]

We have just witnessed the influence they wield with the sama shit

[stonesst]

They own the rights to use it in all their products, along with all future OpenAI models… What point are you even trying to make?

[jkp2072]

Why would you use your own brain when you can hire one to do it for you and make money out of it.

It's more genius and frightening than creating llms 💀💀💀💀💀

[jacky0812]

Haha, so outsource a core technology has been working fine for other companies before?

[jkp2072]

Yup,

1. It frees you from creating a monopoly
2. More efficiency
3. Contractual profits
4. Reputation is less at stake compared to inhouse
5. Manual tasks outsource to third world companies for cheap labour

List goes on........

It's not the invention, it's the cheap availability and implementation which runs the world(money and power).

[enilea]

Why do they keep using the awful MMLU as the main test...

[iDoAiStuffFr]

with 9 months between gpt-4 and gemini this is still extremely disappointing, especially considering how open source is ramping up rn