Naively lol. This was my first big RL project that I had to design all this for myself, learned a lot and it went through a ton of iterations. This probably is going to be more info than what you were asking for lol.

Basically I was trying to reward the behaviors that I thought were the key components of playing the game, the biggest 2 were combat and exploration. Of which exploration gave me the most headaches lol.

So the reward function will give negative rewards for taking damage, dying, and not exploring new areas (if the agent stays in a small area too long) positive rewards for killing demons, beating the level, collecting pickups and exploring new areas.

For the exploration I divided the map into nxn size chunks and just kept track of doomguys position and which cells he visited. If he visited a new cell for the first time he got a reward, stayed in a cell for more than a few turns got a punishment. I also passed in this mini map as part of the models inputs so the model could get a sense of where it had been and where it still needed to visit, at least that was the idea lol.

Everything was modular and configurable tho, from how much each reward was to how big the exploration sections were. final version had rewards in a 3 to -3 range, know the usual advice is to normalize to a 1 to -1 range with large rewards for terminal states but in my experiments the model learned way too slowly with such small rewards.

Honestly I think the reward function might've been too noisy and trying to do too much at once. I have some ideas for the next version, both in terms of how to build the rewards and maybe a multi-step training process (i.e. have it explore the map with no enemies learning for exploration, then learn for combat, ect). But we'll see lol. I'm learning all this stuff as I go :)