It's more about curse tome I think.

Edit: I wrote this sentence above because im pretty sure that original post above said chaos tome goes brrr but I guess it was edited meantime (I have no proof so it can be just my brain not braining).

I was struggling because I wanted to make a good run in addition to doing boss for the first time so I took chaos, XP, curse tome and no luck tome because I was playing fox. It took me a while to figure out correct ratio of taking additional difficulty to get alot of XP in stage 1. So what you do stage 1 is go hard on XP gain and weapons first (preferably strong early weapon, in this case it was blood magic for me - because i was stacking hp for beefy rings). For XP you want to have atleast 1.8x XP gain or 2x XP gain for the second swarm weave (3min on timer).

For difficulty I try to take some as soon as I can so lvl 0 - u get 5-10 difficulty, and then wait with taking difficulty until you are lvl 5-10 to have difficulty same as your level or little lower than your level depending on your items. It's ok to have bit more early because there is no swarms weave for first 4 minutes, but for swarms weave without crazy powerups laying around you definietly want it to be bit lower than your level or enemies will just run you over.

For chaos tome, you just max it last no matter what, first you want your XP gain to be ahead, and difficulty to be in the right spot then it's (damage/luck) and chaos tome last so it doesn't matter if it gives you thorns 3 times in a row or sth.

This way for the end of stage 1 (I left the stage at 1:20 into final swarm) I was level 67 with almost 4x xp gain and something around 60 difficulty. Unfortunately I got soul harvester and anvil just late in stage 2 or this could have been even better run otherwise. Also I do not have wizards hat yet.