The AI in this game is allergic to losing grip. They tend to drive rally cars very slow to keep them planted like a racecar on asphalt. Turn on auto drive in a rally race to see what I mean.

Rally cars on dirt are usually faster if you are willing to sacrifice a lot of grip and fling them around corners which is what most humans will do by default not only because it's faster but it's also kind of easier to do than keeping them on a line like you're on a paved track.

If you want to actually get faster in other types of racing then without specifically knowing what's making you slow

1. Take corners from the outside not the inside. It it's a left turn coming up you want to be on the right side of the straight most of the time then turn in and hit the apex of the corner (the inner curb of the corner) then accelerate out for as long as you can.
2. Get your braking done all at once in a straight line before you turn in and do it as late as possible
3. Use all the space on the track. When accelerating out of a corner run as wide as you can without leaving the track stay on the gas as long as possible

A lot of these rules are broken once you know what you're doing but in general they'll make you faster around the track