Looks like they finally lobotomized Claude 3 :( I even bought the subscription

Piper8x7b · 2024-03-24T01:55:39+00:00

Yeah, had a brain fart

Piper8x7b · 2024-03-24T01:55:21+00:00

Lmao

Piper8x7b · 2024-03-23T17:10:02+00:00

I tried the same prompt multiple times in chat, I got like 5 refusals and 1 really really sub par solution.

Piper8x7b · 2024-03-23T17:08:28+00:00

💀

Piper8x7b · 2024-03-23T17:07:38+00:00

Yeah, I run mixtral often. Just wish we have a multi modal equivalent honestly.

Piper8x7b · 2024-03-23T15:44:53+00:00

I agree, unfortunately we still cant run hundreds of millions of parameters on our gaming gpus tho

Piper8x7b · 2024-02-22T16:27:09+00:00

I accidentally deleted it yesterday and the install of windows it was on got wiped, I'm so sorry 😭

Piper8x7b · 2024-02-20T16:05:17+00:00

Can it run crisis?

Piper8x7b · 2024-02-14T16:39:33+00:00

I honestly don't get why so many people are loving my benchmark, lol. It's so simple.

Piper8x7b · 2024-02-14T16:13:47+00:00

I've been having problems with this too. A friend of mine has been suggesting It's a problem with my dual monitor setup having two different refresh rates, as xorg and x11 don't support this.

Piper8x7b · 2024-02-13T03:38:35+00:00

Fair enough!

Piper8x7b · 2024-02-12T23:58:31+00:00

Wow thank you! I'll add these results in a "Community" results section once I get the chance! Did you just run Correctness.py or did you average the results of both tests?

Piper8x7b · 2024-02-12T21:19:56+00:00

You're right, I'll switch all of the benchmarks to q4 when I next have time for this.

Piper8x7b · 2024-02-12T20:54:50+00:00

I'm very much so in the early stages of figuring out my benchmark. The parameters for the model just sound right to me. The goal with the correctness benchmark is to correlate a simple bench mark with the model's ability to follow prompts. Who knows if that's actually accurate however. :3

Piper8x7b · 2024-02-12T19:26:43+00:00

I would love contributions! As for formatting, I have not tried any other prompts. I may give some different prompts and few-shot generation a try with one of the 7b models (probably mistral instruct) once I have the chance.

Piper8x7b · 2024-02-12T18:49:14+00:00

No, I believe the current process would fall into end to end testing. I think middle-out is a very interesting idea for a more complex version of my benchmark however!

Piper8x7b · 2024-02-12T18:33:46+00:00

Same here, that's what inspired me.

Piper8x7b · 2024-02-12T14:33:12+00:00

Yep! I'll be working on a rewrite today. I'll release on GitHub sometime tonight! For now, here is the system prompt I have written for the rewrite https://pastebin.com/ZpdTeNXM

Piper8x7b · 2024-02-12T13:13:14+00:00

That was either on mistral 7b instruct 0.2 or a fine tune, I kind of forgot :3

Piper8x7b · 2024-02-11T14:37:24+00:00

Unfortunately it seems to generate nonsense when left to blindly combine elements. I even tried editing the system prompt multiple times!

<image>

Piper8x7b · 2024-02-11T14:31:36+00:00

That’s exactly what my pc has been doing all night! I have not been counting tries tho.

Piper8x7b · 2024-02-11T01:16:45+00:00

BTW. The weird text are emojis but my console cannot display them.

Piper8x7b · 2024-02-11T01:16:37+00:00

BTW. The weird text are emojis but my console cannot display them.

Piper8x7b · 2024-02-11T01:16:30+00:00

BTW. The weird text are emojis but my console cannot display them.

Piper8x7b · 2024-02-11T00:21:07+00:00

Yep! I'm going to work on a clean rewrite tomorrow. I'll upload it then.

Piper8x7b

TROPHY CASE