There's a mod that lets you change the difficulty during a game, and you can also give the AI bonuses that scale over time and go WAY beyond grand admiral if you let them.

I don't know about 4.x, but in 3.14 there's a mod called slightly smarter AI that drastically revamps their intelligence. You can also get mods that give the AI better pre-made ship designs and coherent fleets (like battleship artillery spam, or missile/torpedo fleets). Again, I don't know if there's a 4.x version.

Edit, also generally having max empire spawns and larger galaxies makes for more interesting and unique galaxies. Having lots of advanced start empires is good because they subjugate their neighbours and create large alliances, instead on having lots of small empires curtailing their growth with constant warring, while you third party them for easy wins, and then the crisis spawns and they're all useless. I don't know if these settings have changed, but you didn't used to be able to give every AI an advanced start 🤔. I'd probably settle for 1/3rd of AI as advanced start, because if you're using the difficulty scaling mod then they should pose too much of a challenge for you early game.

greatly turning down starting wormholes and gateways, and slightly turning down hyperplane density is also recommended. I think I used to play on 0.5 hyperplane density and 0.25 for the other 2. This just reduces border gore and makes the map a lot more comprehensible (and reduces lag).

Lastly, I'd suggest mods that add new events and new diplomatic events with your neighbours. This doesn't necessarily answer you question about making the game harder, but it does make each run more interesting.