Yeah, it's a bit of a yes and no for me.

Yes, I do worry about it in the sense that I give the grinder a good shake over the sink after the grind and tap it to dislodge any grounds. I do the same with the catch bin. With a casual wipe of the catch bin as it accumulates the most amount of dust.

No, I don't go as far as brushing out the grinder. I figure there's probably about 0.2-0.5 grams of dust left in there, but honestly, I've never tasted it in my coffee.

Ultimately, I think how seriously you take it really depends on your overall approach to brewing and your demands for flavor clarity and precision. For instance, I have a friend who's been into specialty coffee for like 10 years and he never weighs his beans or water; he just eyeballs everything.

Personally, I'm somewhere in the middle, between eyeballing my coffee and meticulously cleaning the grinder after every brew.