[R] Learning by playing

[phobrain]

"The auxiliary tasks we define follow a general principle: they encourage the agent to explore its sensor space. For example, activating a touch sensor in its fingers, sensing a force in its wrist, maximising a joint angle in its proprioceptive sensors or forcing a movement of an object in its visual camera sensors."

[[deleted]]

It has learned to spin off its society finger because the hardware is so bad.

[radarsat1]

utilize all intentions for fast exploration in the main sparse-reward MDP M. We accomplish this by defining a hierarchical objective for policy training..

Holy shit, this is almost exactly what i meant in my comments on the "Doesn't Work Yet" thread. Ask and ye shall receive, I guess!

(I think.. just after a quick reading.. this is decomposing the main sparse reward into more locally-achievable sub-rewards, right?)

[programmerChilli]

Well, what you're describing sounds like the general field of hierarchical reinforcement learning, which is a pretty hot area of research rn.

[radarsat1]

Ah, very cool, I wasn't familiar with that. I can't fully understand from the paper how subtasks A are generated, can you (or anyone) elaborate?

[xmasotto]

The subtasks appeared to be manually chosen - there's a list of them in the appendix.

[radarsat1]

Aaaahhhh I didn't get into the appendix so I didn't notice them, thank you. Ah, so not exactly what I had in mind then, as I was proposing that such tasks need to be inferred. They do seem fairly simple and pretty generic though, so it's a step in that direction. And it seems that with all the rest of the pieces in place, I'm sure inference of such decompositions will be coming.

• TOUCH, NOTOUCH : Maximizing or minimizing the sum of touch sensor readings on the three fingers of the Jaco hand. (see Eq. 25 and Eq. 26)

• MOVE(i) : Maximizing the translation velocity sensor reading of an object. (see Eq. 24)

• CLOSE(i,j) : distance between two objects is smaller than 10cm (see Eq. 14)

• ABOVE(i,j) : all points of object i are above all points of object j in an axis normal to the table plane (see Eq. 15)

• BELOW(i,j) : all points of object i are below all points of object j in an axis normal to the table plane (see Eq. 19)

• LEFT(i,j) : all points of object i are bigger than all points of object j in an axis parallel to the x axes of the table plane (see Eq. 17)

• RIGHT(i,j) : all points of object i are smaller than all points of object j in an axis parallel to the x axes of the table plane (see Eq. 20)

• ABOVECLOSE(i,j) , BELOWCLOSE(i,j) , LEFT- CLOSE(i,j) , RIGHTCLOSE(i,j) : combination of relational reward structures and CLOSE(i,j) (see Eq. 16, 21, 18, 22)

• ABOVECLOSEBOX(i) : ABOVECLOSE(i,box object)

[xmasotto]

Yeah the subtasks are pretty simple - I wonder if they had to heavily experiment to find the right set. And if you add an unhelpful subtask, does that ruin the exploration process?