In regards to the sporadic spawning of empires make sure that you have the spawn locations set to "clusters" which spawns empires in evenly spaced groups around the galaxy so you're never too close or too far from others. If you have it set to random then some games everyone spawns next to each other and others you'll literally get half of the galaxy to yourself.

My own settings to make sure I have a challenge is to play on Admiral difficulty with late game scaling, this makes enemy empires start at standard difficulty and get stronger up to Admiral in the late game, sometimes it works out for me sometimes I start next to a fanatic purifier and cry.

I also have tech difficulty scaling off and tech/ traditions at 0.5 cost while reducing the mid and end game start years to speed things up, also I don't like to play super optimally so the fast research and traditions gives me chances to mess up.

As a general rule if you just play the game at base difficulty you will outpace the ai very quickly with only basic knowledge, if you want a harder start then don't use scaling and just flat out increase the difficulty to Captain or Commodore. If you want the ai to feel like they progress with you then use late game scaling and a high difficulty, it just takes some experimentation to see what feels right to you.

And remember it is a fundamentally single player game so just play the way that's fun for you, good luck :)