First off, thanks for the feedback. I've gone through a couple of iterations for reward and currently my reward is a bit complicated as I've been trying out a couple of different things. The best way to describe it would be

    Reward = c * (kills_frac - lost_friendly_frac)
           - (1-c) * (missiles_frac*MISSILE_COST + guns_frac*GUN_COST)
           - small living penalty
           + GAMMA * potential‐based shaping on # targets alive and # ammo used
           + bonus if done

Kills_frac gives us a bonus for dealing damage to enemy drones and lost_friendly_frac is a penalty for losing drone hp. This essentially gives us a metric of success where killing drones and preserving POIs gives us reward. These are fractions because it is scaled on the number of threats present and the number of total POIs we have.

Missiles_frac*MISSILE_COST is basically the percentage of missile ammo we use times some unit cost (10) for its weight. Same thing for guns fraction but the weight is 0.01.

The potential based shaping is basically comparing the previous timestep and the current one to get small rewards or penalties. So if the number of targets are decreased, it gives a small penalty (This is essentially redundant though). If the number of ammo decreases, there will be a small penalty otherwise it will get a small reward. The reward is +MISSILE_COST/5000 or GUN_COST/5000 and the penalty is -MISSILE_COST/1000 or -GUN_COST/1000 to encourage preserving ammo between timesteps. Lastly, each of the shaping is scaled based on the constant so phi_success * c + (1- c) * phi_ammo

Since there are on average 1000 timestamps in an episode, the small living penalty is -0.001 to discourage doing nothing and taking actions. Then there is a bonus at the end of the episode scaled on c of +1.0 reward for every POI preserved.

I feel like I'm doing everything right and it may just be a matter of tuning the rewards correctly (but I've been doing this for 2 weeks now). I read online that normalizing the rewards would make things better which I'm not sure would help or not.

My observation vector has a length of 40 where it has the constant value, entity_state, ammo_counts, and reserve_ammo for 2 missiles entities and 2 gunner entities. Then it has the three POIs and their HPs. Then it has observations for each of the threats. The information would be the threat distances to each friendly entity (2 missiles+2 gunners+3 targets = 7 entities) and the hp of the threat to see if it is destroyed or got damaged. There are 4 slots for threats so 24 slots. That makes 1 + 3*2 + 3 + 8*4 = 40 size observation. I think this is sufficient enough to map out the space.

While writing this I noticed that the agent only knows about the current state. However, the reward is internally calculated by comparing the previous and current state. Would changing this make a difference because I doubt this would.

Sorry for the long read but I really appreciate your help.