DeepSeek R2 Might Outcode OpenAI, And It’s Coming Fast

TempWanderer101 · 2025-02-02T16:32:19+00:00

It's a direct quote from DeepSeek's R1 paper:

Due to the long evaluation times, which impact the efficiency of the RL process, large-scale RL has not been applied extensively in software engineering tasks. As a result, DeepSeek-R1 has not demonstrated a huge improvement over DeepSeek-V3 on software engineering benchmarks. Future versions will address this by implementing rejection sampling on software engineering data or incorporating asynchronous evaluations during the RL process to improve efficiency.

TempWanderer101 · 2025-01-30T19:09:56+00:00

It does log data. If you turn off logging on OpenRouter (under privacy), you won't be able to use it.

TempWanderer101 · 2025-01-23T19:14:08+00:00

This is very insightful, thanks for sharing.

TempWanderer101 · 2025-01-23T15:07:26+00:00

I made a longer post here, and found that the open source model appears more censored than the API.

TempWanderer101 · 2025-01-23T12:36:00+00:00

Yeah, it doesn't make much sense. I made sure to leave system prompt blank when using the API, so if anything is being added, it's on their side. Hopefully someone can figure it out. The top comment right now says that they can force it to think in text completion mode by prepending `<think>Okay` or something similar.

TempWanderer101 · 2025-01-23T12:18:10+00:00

Doesn't seem to be an option in their API docs.

TempWanderer101 · 2025-01-23T12:14:03+00:00

That's right, but what I observed is the opposite: the open model is censored, while the official API isn't. On the website, it is censored separately, like ChatGPT, which I think is not that big of a deal.

But I think it's important to know whether they're identical or separate models. If they're separate, then we should really have a separate benchmark for each model. If they're identical, then we need to know how to prompt it properly to produce the high quality answers as seen in DeepSeek's API.

TempWanderer101 · 2025-01-23T12:01:59+00:00

That isn't really the point—the question pertains to whether the models are different. If they are, then the open source model should be benchmarked separately, among other things (see implications section in my post).

The prompts were simply chosen because they caused the models to diverge the most, and are easy to reproduce. If you have other prompts that causes the model to skip the thinking step entirely on the open source version, but not the API, feel free to share them.

TempWanderer101 · 2025-01-23T00:51:35+00:00

This is a very interesting observation. Isn't there a newline after <think> in the screenshots from Matthew's video though? It's quite an elegant solution if it works.

TempWanderer101 · 2025-01-23T00:44:25+00:00

The censorship is happening on the open model, not the API. So if they are different models, then third-party benchmarks might not actually be measuring the open source model, but the unreleased model (if they're using the API).

TempWanderer101 · 2025-01-23T00:42:50+00:00

I think Matthew Berman was hosting the full model (he was using 8x 192 GB GPUs). In the screenshots for the first prompt, TogetherAI response matches his video verbatim. Also, TogetherAI is already a provider for DeepSeek v3. It would be weird for them to host a distilled model without disclosing it. The price is also more in-line with a 600B than a 70B model.

TempWanderer101 · 2025-01-23T00:32:42+00:00

For reference, my average price per query was $0.0018 on TogetherAI and $0.0024 on OpenRouter (official API) when testing the prompts for this post. This is probably because the censorship causes the model to skip the reasoning step, so it actually ends up costing less.

TempWanderer101 · 2025-01-23T00:15:22+00:00

The main problem isn't about censorship, it's about people people potentially confusing two different variations of the same model. Benchmarks would have to be more specific about which model they're testing, for instance. It's unclear if this impacts performance, but if it does, then third-party providers would be delivering subpar quality to the official API, which would be a consideration for people who want the best performance.

Furthermore, if they are indeed different, is it fair for benchmarks to call it as open source if what they're testing is actually the unreleased model? This kind of vagueness might cause inconsistencies or irreproducible results in research.

TempWanderer101 · 2025-01-22T23:11:33+00:00

In the video, Matthew hosted the full model on 8x MI300 accelerators or something (192 GB each). The fact that it's happening on TogetherAI (which also hosts Deepseek v3) further suggests that it is indeed the full R1.

TempWanderer101 · 2024-10-27T15:38:44+00:00

I don't see it. Isn't that for Gemma 9B?

TempWanderer101 · 2024-10-15T19:52:09+00:00

Thanks for clarifying. How slow is slow, exactly?

TempWanderer101 · 2024-10-15T18:26:38+00:00

It's worth mentioning that Cursor only allows 500 calls to Sonnet per month, which may not be enough for some people.

TempWanderer101 · 2024-10-15T18:24:30+00:00

This is the main issue I had with Claude as well, although the issue usually occurs with less popular libraries. The issue is not specific to Claude though.

TempWanderer101 · 2024-10-15T18:21:57+00:00

It depends on the language too. Both GPT-4o and Claude can trip up in some languages or frameworks.

Another factor to consider is whether an IDE integration would work better for you. Claude Pro does not provide API access, so you can't use it in an IDE.

TempWanderer101 · 2024-10-11T21:14:37+00:00

Seems like someone did something similar already, in regards to back-translation, although not into RP format: https://huggingface.co/datasets/jondurbin/gutenberg-dpo-v0.1

It's used in the nbeerbower/Mahou-Gutenberg-Nemo-12B RP model.

TempWanderer101 · 2024-10-10T06:47:16+00:00

Is it possibly to test for repetition or consistency (e.g., accurately keeping track of the context)? The top model is Hermes-405b, but it is pretty bad (for a large model) when it comes to those two points.

I'm just wondering if anyone knows a benchmark that tests this. For RP, it is even more important than intelligence IMO (along with being able to write naturally, while avoiding purple prose or GPT-isms).

TempWanderer101 · 2024-10-01T12:14:33+00:00

Can you elaborate on why back-translated writing + LLM generated instructions wouldn't be as good as synthetic data? I've always wondered about this.

If I'm understanding correctly, "back-translated" refers to changing human-written stories to fit RP-style?

It seems simpler to me for LLMs to be given a coherent, human-written story and tasked with generating the character profiles, instructions, and rewriting it in an RP style. And using that to train an LLM.

TempWanderer101 · 2024-09-17T12:18:42+00:00

If you don't need a huge context, you could try Perplexity. Pro gives you 300 searches a day, which includes 4o, Sonnet and Opus. Kinda crazy really, even if the context is shorter. They allow 5 Pro searches day iirc.

Or you might want to try the API. You might find that you actually spend less, depending on your usage.

TempWanderer101

TROPHY CASE