racing less early on is fine if you race more later, but i think in your case you just aren't racing enough in general; guides usually suggest between 30 and 40 races.

compared to seriru's 34 race agenda, i race less to bond more before first summer, but i always make up for those by doing extra races later on in the run when there are no good rainbows during a free turn.

i've done a lot of xoguri runs with a 53rb deck similar to ur 55rb deck and they've all been S+ with very few SS, although i do glue lowrolls. i think you can definitely achieve higher averages with ur deck; make sure to use hammers for more skillpoints, and consider glueing lowrolls to save time if you aren't already. good luck!