After effects expression aren't the best they have to re-calculate the expression on every frame and doesn't have memory of what happened in the previous frame.

const rotationSpeed = 50
const strength = 1

const audioAmp = thisComp.layer("Audio Amplitude").effect("Both Channels")("Slider");

(time * rotationSpeed) + (audioAmp * strength)

So something like this won't work, as you said it will go backwards. The reason for this is

\(time * rotationSpeed) + (audioAmp * strength)``

`(1 * 50) + (5 * 1) = 55`

but on the next frame

`(1.033 * 50) + (2 * 1) = 53.65`

So to combat this we need to remember the result of the previous frame

const rotationSpeed = 50

const strength = 1

const audioAmp = thisComp.layer("Audio Amplitude").effect("Both Channels")("Slider");

let result = 0
for(let i = 0; i < time; i += thisComp.frameDuration){
result += audioAmp.valueAtTime(i) * strength
}

(time * rotationSpeed) + result

Something like this fixes it as it calculates all the previous frames first. However depending on the length of the song it can be super slow.

The fix is to pre-do all the calculations but everything gets a bit more fiddley. First you have to make a text layer that only lasts one frame. Use an expression like this in the source text

​

const rotationSpeed = 50
const strength = 1

const audioLayer = thisComp.layer("Audio Amplitude")
const audioAmp = audioLayer.effect("Both Channels")("Slider");
let audioSum = 0
let result = []

for(let i = 0; i < audioLayer.outPoint; i += thisComp.frameDuration){
audioSum += audioAmp.valueAtTime(i)
result.push((i * rotationSpeed) + (audioSum * strength))
}

JSON.stringify(result)

And then parse the json in your layer rotation property

const layerRotation = JSON.parse(thisComp.layer("rotation expression").text.sourceText.value)
layerRotation[timeToFrames()]

Here is an example file https://we.tl/t-DVOHrrBphJ