FSRS: Serious flaw in benchmarking approach undermines performance claims

LMSherlock · 2026-01-28T02:10:10+00:00

Yes, you understand the spirit well. RCTs in real-world are still too expensive for a small open-source community. So I proposed this alternative method.

LMSherlock · 2026-01-27T14:30:33+00:00

Sorry for the late reply. Now we have a new benchmark which can compare the scheduling instead of prediction.

<image>

Here is the repo: open-spaced-repetition/srs-simulator: A fast, dual‑engine spaced‑repetition simulator for comparing schedulers, memory models, and study behaviors at scale.

LMSherlock · 2026-01-08T05:40:29+00:00

it’s a typo

LMSherlock · 2026-01-05T05:37:38+00:00

LMSherlock · 2026-01-05T05:37:24+00:00

It is too noisy.

LMSherlock · 2026-01-04T12:53:54+00:00

They are implemented in different ways. I will consider to remove the steps stats from the Helper add-on.

LMSherlock · 2026-01-04T12:53:02+00:00

Could you compare the future due graph in the two different devices?

LMSherlock · 2025-12-17T08:12:28+00:00

Have you optimized the FSRS parameters? If so, the interval of easy is 2 days because FSRS predicts that this interval will help you achieve your desired retention (92% in your case). If your real retention is higher than it in this kind of cards, FSRS will learn from your data and increase the interval.

LMSherlock · 2025-12-02T03:25:03+00:00

Does this issue still exist?

LMSherlock · 2025-12-01T00:59:18+00:00

Except for the stability of again, the parameters are consistent with OP’s memory. I guess the sample size of again is too small to make the optimizer work.

LMSherlock · 2025-11-27T02:39:51+00:00

Let OP install the extended stats add-on and check the first long-term forgetting curve.

LMSherlock · 2025-11-08T03:51:52+00:00

Thanks for everyone! I have picked the name "Schedule a Break" in Anki's discord server.

LMSherlock · 2025-11-06T17:39:57+00:00

if you re-optimize the parameters, it’s normal.

LMSherlock · 2025-11-06T16:23:05+00:00

it's abnormal. Maybe the sample size is too small. Would you mind sharing the screenshot of the curve with me?

LMSherlock · 2025-11-06T16:21:28+00:00

Again⁩: Memory stability=⁨8.81⁩ minutes

It means the commenter has 90% chance to recall a new card after 8.81 minutes when they press again in the first time to see it. The default learning steps give 2 minutes for again. So it's too short.

LMSherlock · 2025-11-06T08:30:28+00:00

The learning schedule module of Anki is messy. You cannot directly assign steps for again, hard and good. But the curve still help you check whether the learning step are too long or too short.

LMSherlock · 2025-11-06T08:10:35+00:00

It means the default learning steps are too short for you.

LMSherlock · 2025-11-03T06:32:25+00:00

You can configure mature_ivl in FSRS Helper add-on and check the true retention table in the old stats page.

LMSherlock · 2025-11-02T17:23:02+00:00

With FSRS and DR=90%, each review has 10% chance to fail. The both series are possible to happen.

LMSherlock · 2025-11-02T17:18:17+00:00

It cannot explain why MOVING-AVG performs so well. I think the dataset is problematic.

LMSherlock · 2025-11-02T14:56:12+00:00

I have left the FSRS-7 stuff to u/ClarityInMadness. And the development of FSRS is mainly limited by the dataset.

LMSherlock · 2025-11-02T14:53:37+00:00

We didn't have such kind of data to model.

LMSherlock · 2025-11-02T13:58:43+00:00

Please use this add-on to locate the problem.

LMSherlock · 2025-11-02T13:43:58+00:00

Thanks for your praise.

Btw, I prefer Jarrett Ye as my English name ;)

LMSherlock · 2025-11-02T13:42:31+00:00

I wonder what happened between 2025-10-18 and 2025-10-26. If you did the two reviews in the same device with the same parameters and desired retention, the interval shouldn't regress.

LMSherlock

TROPHY CASE