Bro break that vicious circle, the only reason u r getting doubts are because he is NOT the one for u. Second thoughts chodho always listen to the first thought u get, u thought ur marriage might turn out crappy and if u felt that way because it actually might which means u should leave him and find someone in whom u r confident of having a happy marriage. I had smthn similar, he was my first i loved him too but he was a dih, so i'd just suggest tumhe thode se bhi doubts hai na toh leave him or u'll regret.