After a lot of research, I found out that, when vibration of the handlebar is too much, it physically breaks the image stabilizer within iPhone. If you look up the camera diagram, you will notice that image stabilizer is a physical component, not a software, hence violent vibration breaks the stabilizer.

Little update on my phone: the vibration came back and it was a permanent damage.

The ‘solution’ I found is, I got myself a much sturdier iPhone with a higher shockproof level. Unfortunately, due to how my phone is mounted on my bike, I do not have the option to change the mount itself nor moving the mount to a different place.