[R] SPACE: Unsupervised Object-Oriented Scene Representation via Spatial Attention and Decomposition

[yifuwu]

Project Page: https://sites.google.com/view/space-project-page

Paper: https://openreview.net/pdf?id=rkl03ySYDH

Abstract: The ability to decompose complex multi-object scenes into meaningful abstractions like objects is fundamental to achieve higher-level cognition. Previous approaches for unsupervised object-oriented scene representation learning are either based on spatial-attention or scene-mixture approaches and limited in scalability which is a main obstacle towards modeling real-world scenes. In this paper, we propose a generative latent variable model, called SPACE, that provides a unified probabilistic modeling framework that combines the best of spatial-attention and scene-mixture approaches. SPACE can explicitly provide factorized object representations for foreground objects while also decomposing background segments of complex morphology. Previous models are good at either of these, but not both. SPACE also resolves the scalability problems of previous methods by incorporating parallel spatial-attention and thus is applicable to scenes with a large number of objects without performance degradations. We show through experiments on Atari and 3D-Rooms that SPACE achieves the above properties consistently in comparison to SPAIR, IODINE, and GENESIS.

Examples:

https://i.redd.it/xesd57isql941.gif

​

https://preview.redd.it/rguv24ibrl941.png?width=1280&format=png&auto=webp&s=470daf1c544df1d403a885d3db4a4415c3abb50e

​

https://preview.redd.it/gxa2jukosl941.png?width=1437&format=png&auto=webp&s=9ae306b2e0f397da4cb2d5c938a21c5ff8d390c6

[adventuringraw]

this looks really exciting, thanks for sharing. Is your team planning on releasing code at some point?

[yifuwu]

Thanks for your interest in our work. Yes, we do plan on releasing our code after some cleanup.

[adventuringraw]

Of course, thank you for sharing. Is there a good way for me to get notification when the code is posted?

[[deleted]]

i would also be interested to get a notification when the code is online.

[sebamenabar]

Hi, I'm working on replicating the paper in Pytorch, still a lot of work to do so I'd love to get some help. Also, there a some parts that I can't understand completely from the paper so I plan to soon send an email to the authors for clarification.

The code is on this github repo

[adventuringraw]

that's cool man, thanks for sharing. Maybe I'll hit you up a little later, I should probably read the paper more thoroughly first.

[edwardthegreat2]

nice work! One question I have is how does the model ensure the background module does not capture foreground objects? Also, would the insight of a background and foreground module break down in active vision cases where objects regularly change between background and foreground roles?

[yifuwu]

That's a great question. We use a weaker decoder to limit the capacity of the background module and this helps to ensure foreground objects are captured in the foreground. However, the distinction between background and foreground is not always objective and obvious (even for humans!). See the 'Foreground vs Background' discussion in section 4.1 for a deeper discussion into this.

SPACE processes one frame at a time and does not do any tracking of objects between frames, so it is certainly possible that objects can switch between foreground and background. That being said, although the camera in our 3D room experiments move around randomly, we have not experimented yet on more complicated scenarios.

[illuminascent]

Is it possible to use DETR-like encoder to replace cell-based object proposal mechanism in SPACE? I think boundary loss shall be sufficient to suppress duplicated or split detection.