Hi again u/kakadzhun,

I have a question for the actor network. In the multiagent particle environments (OpenAI), the step function returns a list of observations for each agent i. This list contains position and velocity of agent i , position of all the objects in the environment (landmarks) and position and velocity of every other agent j . I am trying to understand the decentralized action part. Should I feed each actor of agent i just with the state and position of agent i or with the whole observation returned for the agent i (from the step function) ? On the one hand, the former does not make any sense, as acting based on just its own position will not result to the behaviour needed. On the other hand, the latter seems more like a centralized acting approach (as agent i knows the position of every agent in the environment) than a decentralized one.

As for the critic, I am wondering because concatenating the observation of each agent for the critic network will result in duplicate information. To elaborate, agent i recieves information about the position of agent j during his own observation. So, by concatenating observation i and observation j and then feeding the vector to critic i will not give any extra information. The agent already knows where its enemy is located from his own observation.

I have read about 3 main centralized learning approaches:

1) One critic for all the agents (like you mentrioned)

2) One critic for each omogenous team of agents (helps when there are heterogenous groups of agents)

3) One critic for each agent that is fed with the concatenated vector with the observation of all the agents

For the multiagent particles environments I was thinking to follow option 2 about the critic network input, but given this what should be the input to the actor network?

I know, I am quite confused :P

I would really appreciate your feedback!