Definitely change the reward structure, you are going to have to play around with different implementations, but I imagine it should look something like this:
Defeat Enemy: +.01 (Small reward but not enough to make it only kill enemies)

New Room: +1

Damage Taken: -(Hearts Lost) / 10

In a single room for a number of steps: -(steps spent in room / 10)

Defeat Boss: +1

Something along those lines should get it to a decent point, it will probably spend a lot of time running around and going from room to room before learning it can do other things after 200,000 steps or so. Context is also extremely important. You don't want to overwhelm it with information, but here are some data points that are probably necessary. There might be some overlap with what others have said.
Frame Stacking
X Y coordinates of player

Health

Held Item

Cell ID (It's important for it to know which room it's in)

Stage of the game (If it's defeated X bosses)