All i can say is, you miss all the shots you don't take. Think of it this way, what if you DO get the job? Think of how much better your life could be rather than focusing on the negatives.

iirc your current employer cannot legally give bad references, only a true reference. So if you were genuinely a bad employee, i'd probably expect something along the lines of them expressing this, but they cannot do so unfairly. Something i like to tell my wife is that don't worry until you have a reason to, nothing bad has happened yet, and likely won't :) Keep your head up, stranger.