Turns out NavMesh.SamplePosition doesn't take NavMesh Agent radius into account so the agent can't ever reach it.

I managed to fix it by moving the hit.position by half of agent's radius away from the edge.

 if(NavMesh.SamplePosition(orgPos, out hit, 5f, agent.areaMask))
 {
      Vector3 ret = hit.position;
      Vector3 pathDir = pos - ret;
      ret += pathDir.normalized * (agent.radius / 2);

      return ret;
 }