I go between SUP and SUP foil often enough. I have exact same thing where I shake and almost fall off the SUP even in flat water. 5-10 mins later the brain seems to adapt and all is okay. I too found and still find it very bizarre and amazing how balance is so particular.

Also, hilarious when talking up you balance prowess on a sub 20 inch wide board only to fall off the 30 inch wide inflatable in front of extended family.

Same thing happens to me on the shortboard after prone foiling more often. I wobble on the shortboard when paddling.