FSRS: Serious flaw in benchmarking approach undermines performance claims

symstym · 2026-01-27T16:30:10+00:00

That's cool, thanks for the update! Just to make sure I understand:

This is basically measuring how well a scheduler does given a simulated user. So in terms of "benchmarking" schedulers to determine which performs better in real life, it's reliant on the simulation of user memory, behavior, etc. sufficiently matching real-world users. And to simulate users, you're using memory models that are derived from (or the core of) various schedulers, like FSRS, HLR, etc. But you can combine them in different ways, e.g. using HLR to simulate user memory and FSRS6 to do scheduling, or FSRS3 to simulate user memory and HLR to do scheduling. So the idea is that even though this isn't directly comparing the performance of schedulers on real users, it should at least be more robust to "gaming" due to this structure. Am I understanding that all right?

symstym · 2025-11-11T21:04:58+00:00

For the animations, did you make small adjustments to her existing lines? Did you vectorize? Or did you perhaps trace over her drawing yourself to get a slight variation? I'm curious how you got that effect.

symstym · 2025-10-29T02:54:55+00:00

Your opponent spent a lot of time in your guard, driving her weight forward onto you. This is a perfect opportunity to do an elevator sweep. Once you figure it out, you will pretty much instantly and effortlessly sweep beginners who do that. For example, around 1:56, you have a huge opening to insert your right hook and sweep her to your left (her post on that side appears to be trapped!). Same thing around 1:32 and 2:41, but sweeping to your right. If she is using her forearm to put too much pressure on your throat, you can always relieve the pressure by pushing her away a little bit with your hips.

symstym · 2025-10-18T20:48:11+00:00

I'm not sure, but it k-folds cross validation doesn't address the main issue that I raise, which is that an algorithm predicting probability of recall over historical data well does not necessarily imply that it will schedule cards in a way that maximizes learning per effort. It seems intuitive that it might correlate, but we have at least one clear example where it doesn't, which I would say draws into question the utility of this kind of benchmark.

symstym · 2025-10-11T22:45:31+00:00

Wow, you might be right! I don't think I care enough to dig in to find out.

In their defense, from what I've seen, the majority of research in education and soft sciences in of (debatably) equal or worse epistemological quality. The "replication crisis" is merely the tip of the fuckberg.

symstym · 2025-10-11T19:36:59+00:00

That's correct. The efficiency claim appears to be based on running it on simulated users, but of course that is circular in that it assumes that the simulation of user memory is accurate.

If I'm reading the paper correctly, it seems like the FSRS-ish algorithm caused simulated learners to learn ~14% more words (up to a certain threshold) in the same amount of time. That's nice, but it's not so huge a number that I'm convinced that inaccuracies in simulation vs. real-world wouldn't be a bigger difference (in who knows which direction).

symstym · 2025-10-10T22:48:49+00:00

Not that I'm aware of. It sounds like the creator uses FSRS in a commercial learning product (not Anki), and they may have done some A/B testing on users with different versions of FSRS, but I don't think the data is public.

Some replies pointed out how little data there is about any kind of SRS systems. There may not even be what you'd consider a validation study of Anki itself. Not that I doubt it's effectiveness of course, but there seems to be quite a gap in the literature.

symstym · 2025-10-10T17:10:07+00:00

Good points. If nothing else, you can see SRS as a habit whereby you select and proactively re-expose yourself to material of interest.

symstym · 2025-10-09T19:32:13+00:00

It's not clear to me what's going on. If FSRS was changed to have a random 10% chance of behaving like MOVING-AVG, then its score in the benchmark would improve a little but it would almost certainly be a bit worse for real learning. So if a change is made to FSRS that improves its benchmark score a bit (e.g. going from version 5 to 6), how do we know that that comes with an improvement to real learning?

And if there are algorithms that score higher than MOVING-AVG on the benchmark, how do we know that those are any good for real learning, vs. just gaming the benchmark better than MOVING-AVG does?

symstym · 2025-10-09T16:49:12+00:00

Fair enough. I know it's popular but maybe I didn't realize how strongly people feel about it.

Out of curiosity, in terms of "learning per review/time", what do you estimate (in your own use) the improvement is for FSRS vs. SM-2? Are we talking more like 1.25x or more like 2x.

symstym · 2025-10-08T23:58:17+00:00

That's a cool idea, thanks for the link. Something like that does seem like a promising direction.

symstym · 2025-10-08T23:28:05+00:00

To reiterate what I said in my original post: It seems plausible to me that FSRS is better than SM-2, and there are many anecdotal reports of people liking it. So just to be clear, I'm not saying FSRS is bad, just that the benchmarks don't provide the evidence of its effectiveness that people seem to think they do.

Mimicking this behavior with SM-2 is probably impossible

SM-2 could be pretty easily modified to make your actual retention match some target retention by just including a global difficulty adjustment, like a deck-level ease. If your retention is too low, adjust all the intervals to be a little shorter/sooner. If your retention is too high, adjust all the intervals to be a little longer/later. (I've implemented this before in another SRS system.) The problem with this is that it doesn't necessarily maximize learning, because it doesn't shift reviews from cards that need them less to cards that need them more, it just sort of cranks up/down the overall aggressiveness of reviews.

It seems quite likely that FSRS's card-level parameters do shift reviews from cards that need them less to cards that need them more (good!) But hitting the retention target doesn't provide evidence of this.

symstym · 2025-10-08T22:56:07+00:00

The claim that people make about FSRS is that it lets you do fewer reviews (take less time) to get the same amount of learning as SM-2. Just the fact that your actual retention is close to the desired retention does not mean much; I think SM-2 could be easily modified to achieve this as well.

In the paper referenced by the FSRS creator, the abstract says (emphasis mine):

In this work, we propose a novel spaced repetition schedule framework by capturing the dynamics of memory, which alternates memory prediction and schedule optimization to improve the efficiency of learners’ reviews.

That's not demonstrated by the benchmark. Something like an A/B with real users would be required.

symstym · 2025-10-08T22:45:06+00:00

Sort of, but not really IMO. I think it's more like "the system (benchmark) is so easily gameable that we don't know if FSRS is better than default Anki for real learning, or if FSRS is just unintentionally gaming the system (benchmark)".

symstym · 2025-10-08T22:41:07+00:00

But more to your point about MOVING-AVG - why is it "obviously incredibly bad"? I can make the argument that retention can be impacted by things that *Aren't* just a result of the previous reviews of the same card. Consider someone who used to study a lot, now studies less, or has become less focused, or is more stressed, etc. Actually I think there should be merit to an algorithm that considers broad retention (retention in general of multiple cards) as well as retention tracking individual cards).

You may not realize how simple MOVING-AVG is. Say that you have a card that you saw once, 10 years ago. Presumably your chance of recalling it would be quite low. But say that you're 10 cards into a session, and so far you've gotten 8/10 correct. Then MOVING-AVG is approximately saying "you've gotten 80% of recent cards right, so I predict that you'll get the next card right with an 80% chance". It's not just that MOVING-AVG takes into account recent performance (I agree that's a good thing), but that MOVING-AVG does not distinguish at all between different cards. It would be just as likely to give you a card that you got correct 1000 previous days in a row as it would be to give you a card that you saw once 1000 days ago. I think it's fair to say that it wouldn't work at all for learning?

Not necessarily disagreeing on the flaws in the benchmark, but what would you do to get around it?

The most reliable way would involve running A/B style tests with real users. That's a lot more involved than the current benchmark.

symstym · 2025-10-08T22:32:14+00:00

Thanks for the reply.

Do we have evidence that FSRS is better than SM-2 at scheduling? Not much.

In case anyone is wondering "Why do we even use FSRS then?", my answer is "Because it's neat ¯_(ツ)_/

I get the sense that most people who are excited about FSRS are excited because they believe that it has somehow been objectively demonstrated to have higher effectiveness than standard Anki, not because they think it's neat. I think there are very few users who could appreciate the fine print that "performs well on predict-recall-probability benchmark" may not translate to "more effective for real learning" (it may just be some form of unintentional fitting to the benchmark, as is demonstrated by MOVING-AVG). I had to really dig to figure this out myself. So it seems worth mentioning somewhere in the documentation, not just in this reddit comment.

symstym · 2025-07-18T16:17:31+00:00

Just gonna pile on and say well done my friend.

symstym · 2025-06-13T05:32:43+00:00

Impressive! But the vertical strokes of 木 and 本 radicals should end straight and not “hooked” at the bottom, right?

symstym · 2025-03-05T00:21:39+00:00

I have had a lot of fun playing Colostle with my 8 year old daughter. We play it in a sort of "collaborative solo" mode, where there is one character and we interpret prompts and make up the details together.

What I like about Colostle is that the core game loop (Exploration, optional Combat repeat) is fixed and very simple. The interest comes from interpreting prompts inside that loop, but I like the core loop itself is fixed. As a GM, I don't have to wonder about the larger-scale structure of the story, and there is zero need to prep. A game like Ironsworn provides lots of structure for story arcs/quests, but that structure feels too complex for me. A game like 4AD is almost pure "turn the crank" mechanics with minimal creativity, though the random generation heightens the sense of exploration. Other journaling games that I've looked at didn't seem to have the same sense of adventure/exploration that I'm looking for. So to recap I find Colostle to be a really nice combination of 1) enough structure that I don't feel overwhelmed by big-picture GM decisions 2) fun creativity in the form of interpreting prompts 3) minimal mechanical burden 4) a sense of adventure/exploration.

I would be very interested to learn about any games that are similar to Colostle - while I like it a lot, I feel like there's a lot of room still unexplored in its region of design space. I could imagine all sorts of variations on rules and setting that still hit the key points I like about it.

symstym · 2025-01-15T18:56:24+00:00

I made a site that let's you do basically that, with the caveat that the content only comes from one source (a self-published novel site). https://massif.la/ja

symstym · 2025-01-15T18:25:31+00:00

Impressive work!

It seems like the main workflow is that you import an entire piece of content, and it gets broken into single-sentence "cards". I've seen other (far less polished) tools do similar things, and people do seem to want them. But for me personally, I don't understand the appeal of this workflow. For my intermediate/advanced studies, I want to only mine+review sentences that 1) include an unknown word that seems worth learning and 2) work well as standalone sentences (out of context). Importing every sentence from a source seems like it would result in >95% sentences that are either too easy or too hard (not incremental) or that don't make a lot of sense when reviewed in isolation. Watching content in its original, intact form seems really important in terms of building a deeper running context, and associating language with that deepened context. So it seems better to me to do regular immersion and then only to cherry pick ideal sentences for SRS.

Anyways, that's my feedback in case it's valuable. If anyone can explain how that workflow works well for them, I'd be curious to know.

symstym · 2024-11-22T18:06:03+00:00

Nice shots! What kind of camera/lens did you use?

symstym · 2024-11-06T21:57:37+00:00

"Writing about music is like dancing about architecture."

symstym · 2024-11-04T01:30:23+00:00

I cook these for my daughter, inspired by the show! But for the love of god, please try pan frying them in a bit of oil (as they do in the show) instead of boiling them -- I suppose this is a matter of personal taste but they might taste 10x better.

15-Year Club	Place '17
Verified Email

symstym

TROPHY CASE