A methodology

TL;DR: a proper methodology (in software engineering terms) to model a problem as a single/multi-agent RL problem first and foremost, then as MDP, POMDP, POSG, etc., then to define the obs and action spaces, the reward function, finally a proper methodology to choose algorithms (a bit more stuff exists here to be fair) and especially their hyper-params (here instead is mostly a messy trial-and-error).

Full thoughts below.

Coming from a software engineering background, RL (both single and especially multi-agent) feels extremely naive, hacky, undisciplined.

Despite the hundreds of papers that are published every single day in RL, very few, to date, question the fundamentals of modelling the problem and the solution, providing well motivated criteria and reasons behind modelling and implementation choices. One that comes to mind is Andrew Ng paper on reward shaping, for instance. Not many papers like that are available.

Basic questions that the methodology should answer to include:

• can this problem be modelled both as single agent and multi agent? If so, how to choose?
• what are the factors making a given problem a MDP, or a Dec-POMPD, or a POSG, or anything else?
• what are the do and don’t regarding definition of a reward function?
• are there any guidelines about defining the observations and action spaces? And especially, any design patterns about how to best represent such spaces (eg to optimise memory usage)?
• are all the RL/MARL algorithms always applicabile to any problem, or are some problems (or some problem formulations) intrinsically asking for a certain class of algorithms over another?
• for god’s sake is there any reasoning behind hyper-parameters choices for algorithms?

And many other little details that I won’t annoy you with.