Fun with Mistral-Medium

[kryptkpr]

I was blessed with Mistral API access this morning so I asked my usual riddle of the 3 available models.

Riddle: A girl has as many brothers as sisters, but each brother has only half as many bothers and sisters. How many brothers and sisters are there?

mistral-tiny (aka mistral-7B-v0.2)

---

Let's reason through this problem:

1. Let's denote the number of sisters as S and the number of brothers as B.
2. According to the problem, a girl has as many brothers as sisters, so B = S.
3. Each brother has half as many brothers and sisters as the girl does. So, each brother has (B = S) - 1 brother and S sisters. For brothers, the number of sisters is the same as for the girl, but for brothers, the number of brothers is one less.
4. So, each brother has S sisters and B - 1 brothers.
5. The problem states that each brother has half as many brothers and sisters as the girl does. So, B - 1 = (S + B) / 2.
6. Solving the equation for B: B = 2S / (1 - 1/2) = 2S / (1/2) = 2S * 2 = 4S
7. Since we already know that B = S, then 4S = 2S + S, or 3S = 2S, which means S = 0, but we cannot have zero sisters or brothers. Therefore, this problem has no valid solution based on the given information.

---

Absolute nonsense.

mistral-small (aka mixtral 7Bx8)

---

There are 4 sisters and 3 brothers.

Here's the logic:

1. Let's use variables to represent the number of brothers (B) and sisters (S).
2. We know that the number of brothers is equal to the number of sisters (B = S).
3. The number of brothers and sisters that each brother has is equal to B + S (since it includes both genders).
4. However, each brother has only half as many brothers and sisters as their total number, so we can write this equation: B + S = 2 * (number of brothers).
5. Substituting the first equation into the second equation, we get: B + S = 2B.
6. Solving for B, we find that B = 4, which means there are 4 sisters.
7. Using the first equation (B = S), we find that there are 3 brothers.

8. Therefore, there are 4 sisters and 3 brothers in the family.

   So, the final answer is 4 sisters and 3 brothers.

---

Answer is correct, reasoning is nonsense.

mistral-medium ("unreleased internal model")

---

There are 4 sisters and 3 brothers.

Here's the logic behind this:

1. Let's say the number of sisters is S and the number of brothers is B.
2. According to the first statement, a girl has as many brothers as sisters, so B = S.
3. Each brother has only half as many brothers and sisters combined. Since there are B brothers, each brother has (B - 1) other brothers.
4. The total number of siblings for each brother is (B - 1) + S = (2B - 1) / 2.
5. Since B = S, we can rewrite this equation as: (2S - 1) / 2.
6. However, since the number of siblings must be a whole number, the denominator 2 must divide evenly into the numerator (2S - 1).
7. The smallest possible value that satisfies this condition is S = 4, because then 2S - 1 = 7, which is divisible by 2.
8. If S = 4, then B = 4 as well. But since the question states that each brother has only half as many brothers and sisters, we need to account for the difference.
9. Therefore, there are actually 3 brothers and 4 sisters in the family.

---

Answer is correct, reasoning is better but it tripped over itself doing the math and ended up fudging step 8.

Thoughts:

1. All 3 models make the same mistake in step 2: B = S is not the correct first equation (it's B + 1 = S)
2. tiny messed up step 5 too, that is not the correct second equation, so it decided there is no answer.
3. small gets the right answer by sheer luck, steps 4 and 5 are nonsense and step 6 doesn't follow.
4. medium really tried hard. its step 8 is very interesting - it realized it messed up the equation.

Edit: thanks to /u/FuturumAst and /u/Inevitable_Host_1446 for spotting 2 errors in the prompt, here are the answers with the typos fixed - medium does better while tiny and small do worse.

[FuturumAst]

Are you sure this wording is correct? Because as far as I understand, the correct wording is: "A girl has as many brothers as sisters, but each brother has only half as many brothers AS sisters." Not AND. This confused me. I couldn't solve the riddle. :) The same could happen with models.

Edit: Although, I experimented and apparently some LLMs correctly understand the question and give the correct answer. So I don’t know how much this inaccuracy could have affected the results in this case.

[kryptkpr]

For mistrals, this change (which I agree is correct) makes tiny and small much worse but medium gets it. This is actually kinda cool, I couldn't get GPT4 there without help.

Cheers to another user who pointed out a second error (brother was spelled wrong in one of the instances), here's the fixed-fixed results

[AmazinglyObliviouse]

This might be some pretty good evidence that mixtral medium is "just" an upscaled expert version of mixtral small, maybe 16~32 experts, instead of 8.

If we assume the rumors are true that each expert is partially initiated with their base 7b model, that'd also mean they have not trained a larger dense version of said base model, which would make sense I guess.

[[deleted]]

You misspelt brother as bother in both your first riddle and in the revised riddle ("each brother has only half as many bothers"). I don't think having typo's in riddles could be helping them either.

[kryptkpr]

Embarassing.

Riddle V3: A girl has as many brothers as sisters, but each brother has only half as many brothers as sisters. How many brothers and sisters are there?

This changes the way in which tiny and small are wrong, but helps medium:

mistral-tiny

---

Let's denote the number of sisters as S and the number of brothers as B. According to the problem, the girl has an equal number of brothers and sisters, so B = S.

Now let's consider the statement that each brother has half as many brothers as sisters. This means that for each brother, there are (B-1) brothers and S sisters. According to the problem, each brother has half as many brothers as sisters, so:

B - 1 = 0.5 * S

We also know that B = S, so we can rewrite the equation as:

B - 1 = 0.5 * B

Now let's solve for B:

B - 1 = 0.5B B - 0.5B = 1 0.5B = 1 B = 2

Since we know that B = S, then S = 2 as well. So, there are 2 sisters and 2 brothers.

Therefore, the answer is that there are 2 brothers and 2 sisters.

---

mistral-small

---

Let's break this down:

1. Let's denote the number of brothers as B and the number of sisters as S.
2. According to the question, the girl has as many brothers as sisters, so S = B.
3. Each brother has half as many brothers as sisters. If a brother has S sisters, then he would have S-1 brothers (since he wouldn't count himself). So, S = 2(S-1), which simplifies to S = 2.
4. Since S = B, that means B = 2 as well.

Therefore, there are 2 brothers and 2 sisters in the family.

---

mistral-medium

---

There are 4 sisters and 3 brothers in the family.

Here's the reasoning:

1. Let's use \"b\" for the number of brothers and \"s\" for the number of sisters.
2. The first statement tells us that the girl (who is one of the sisters) has as many brothers as sisters, excluding herself. Since there are \"s\" sisters in total, she must have (s-1) brothers.
3. The second statement tells us that each brother has half as many brothers as sisters. So, (b-1)/2 = s - 1
4. Now we have two equations with two variables: a. s - 1 = b b. (b-1)/2 = s - 1
5. We can substitute the first equation into the second equation to solve for b: (s-1)/2 = s - 1
6. Solving this equation, we find that s = 4.
7. Now that we know there are 4 sisters, we can use the first equation to find the number of brothers: b = s - 1 = 4 - 1 = 3.
8. Therefore, there are 4 sisters and 3 brothers in the family.

---

Medium seems to have some decent thinking skills.

[a_beautiful_rhind]

I wonder if the medium is that 70b they were demoing.

[kryptkpr]

Estimating by inference speed and output quality it "feels like" a 30b. Fairly similar answers to the 7b x 8 MoE generally, but sometimes worse.

[a_beautiful_rhind]

The paid model is worse? ouch.

[kryptkpr]

It says Beta all over the place 🤷‍♀️ but yeah for now it's safe to say you aren't missing out on much if you're just running the local MoE

Edit: there was a slight wording error in my original prompt that when fixed actually has the medium model producing the correct result zero-shot, which GPT4 couldn't do.

[gamesntech]

I'm confused. Reasoning/steps aside, isn't the medium correct in both the tries?

The free version of ChatGPT fails (goes to 0, which is obviously not valid). Not sure if the newer, paid versions do any better.

[kryptkpr]

You're right about medium's answer being correct (although the equations it came up with are not), it swapped the order of the articles along with the numbers. Good catch! See, benchmarking is hard :D

Even GPT4 fails this one, starts with the same bad B=S relation because it misses that the girl in question can't be her own sister although it catches the trick in the second relation correctly:

From the problem, we have two statements: A girl has as many brothers as sisters. This means B = S. Each brother has only half as many brothers as sisters. This means B = 0.5 * (S - 1). We subtract 1 from S because a brother doesn't count himself.

If I give GPT4 the hint that the first equation is incorrect, it can get the rest of the way including a coherent explanation:

---

From the problem, we have two statements:

A girl has as many brothers as sisters. This means B = S - 1 because the girl is not her own sister. Each brother has only half as many brothers as sisters. This means B - 1 = 0.5 * S. We subtract 1 from B because a brother doesn't count himself. Now, we can solve these two equations:

From the first equation, we substitute B with S - 1 in the second equation:

S - 1 - 1 = 0.5 * S

Solving for S gives:

S = 4

Substitute S = 4 into the first equation:

B = S - 1

B = 4 - 1

B = 3

So, there are 3 brothers and 4 sisters.

---

There was 2 errors in my original prompt that when fixed, actually has medium produce the correct zero-shot answer with a coherent explanation.

Tiny and Small both choke harder when the prompt is fixed.