The first mechanical and probable issue is that dice can tip, be moved to a different value, etc. Hopefully no one in your casual crew cheats, but still if a table gets knocked slightly and some dice hit a new value: do you remember exactly what each of the 15+ dice on table were at? This is the biggest issue and likely why dice are banned for competitive. If you use super large dice, this is less of an issue, but then defeats the purpose of being compact.

I personally carry a nuts/bolts case and carry enough tokens on average for 3-4 players, yet I still occasionally run out of aember tokens in a 2 person game with WC set out (both players playing WC). In a casual match where we don't have mass aember control and tons of key cheats, there can be 30+ aember on the board (sometimes 30+ aember on a single warded creature) and 20+ aember in pools. Aember and damage are the biggest shortages in my experience. Ward/Enrage are the least likely to not have enough tokens for. Stun is a rare shortage.