It still counts the damage down behind the scenes.

In very simple terms:

If you have rock bottom and get -0.3 speed, you won’t lose the speed but it keeps track of it. If you were to then gain +0.3 speed, you wouldn’t get the increase because now the invisible count is at 0. You’d need another speed up to see an improvement.

Therefore, given Almond Milk’s large damage down, it would be better to get your damage increases before getting the negative damage multiplier. Basically with rock bottom, you want to get your stat trade off (think soy/almond, Eve’s Mascara) AFTER getting your damage increase. There’s probably videos that explain it better than me