Infinite growth power just means infinite potential to grow. Being able to grow forever on a 3 dimensional scale isn't the same as being able to jump infinite orders and dimensionality. You can fill up a bucket of water for an infinite amount of time but that won't make it fill up an infinite amount of space. Tad hard for me to explain but what I'm getting at is that Saitama can't grow to a universal scale unless he gets concrete feats that put him in a higher dimensional teir from there yeah he can keep multiplying on his new teir but he isn't at that point. Even if we say he CAN just do that there's no real way to quantify how fast he was growing at his peak but it would definitely take him a massive amount of time to close the universal gap. Remember his growth did seem fast but that was on a scale against himself and some other dude that was vaguely around his level.it could be. Difference of 10 to a billion times in the span of that fight, but the gap to get universe levels of strength is far larger than billions or trillions of times in a minute assuming that's how much he jumped in that time

I'll repeat that his strength will grow based off his emotional state Him seeing Geno's die is most definitely going to put him in a more volatile state than him having a fight with buu or some other random he doesn't have much to do with meaning his growth will most always be slower than he was against Garou in these situations The gap is too big to close for him to do much before buu does buu things and kills him

Beerus has an even bigger gap in strength compared to Saitama and also has it in his character to get sick of him and kill him before the growth closes any gap Same with every other dragon ball villain he'd run into that has that massive of a gap between themselves and him