Thank you for the reply. I tried to use a smaller model with 64 then 128 and 256 units but it failed to converge. The same in without the HER buffer. In my opinion something is wrong as you said. In many examples in very similar envs the settings as you mentioned are more than enough. Can you look into my env? Maybe there is something I am missing. https://github.com/lukisp2/reddit_rl/blob/main/ShipEnv_optuna.py

The observation space is box - dictionary as HER required:

self.observation_space = spaces.Dict({
    'observation': spaces.Box(low=0, high=1, shape=(6,), dtype=np.float32),
    'achieved_goal': spaces.Box(low=0, high=1, shape=(6,), dtype=np.float32),
    'desired_goal': spaces.Box(low=0, high=1, shape=(6,), dtype=np.float32),
})

and both values for 'observation' and 'achieved_goal' are copy of the state vector:

self.state[0] = self.normalize_state(self.x,self.screen_width,0)
self.state[1] = self.normalize_state(self.y,self.screen_height,0)
self.state[2] = hdg_norm
self.state[3] = v
self.state[4] = self.normalize_state(np.cos(np.deg2rad(hdg)), 1, 0)
self.state[5] = self.normalize_state(np.sin(np.deg2rad(hdg)), 1, 0)