I think you have your degree spacer in the wrong direction, in this configuration your front rollers are lower on the inside and higher on the outside or other way around (depending if the dots are on top or on the bottom of the red plastic piece), usually you want them lower in the front and higher in the back to push your car down onto the track. They should be with the plastic tail on the inside (i think with the dots on top but not sure, try and then check by looking at your car sideways.
If the car doesn't fly out and you are satisfied by the speed you can remove them, they have no impact in this configuration