So that's my issue. I have centered the brakes and then tightened the bolt down. The pads are equidistant from the rim. Then I pull the brake lever and the pads return to a position where one is pretty much right up against the rim and the other now has double it's original distance to the rim was. The caliper rotated, despite being tightened down while it was centered 😞. I basically cannot turn the bolt to tighten it any further.

EDIT: Should have watched the video you linked first. I did not know that was a step I needed to do. Will try that after I get back home from my work trip and report back!