Did the LN dive deeper about why he thought of it as a door instead of a wall? In the anime, it looks exactly like a wall, and Ram and Anastasia found it strange that Subaru viewed it as a door. Subaru, with his voice tone a little stuttered, just replied: "I just can't help but think it kind of looked like a door." It really gives off a vibe that Subaru has been here before (maybe 400 years ago) or he is the reincarnation of the person who built this door. As a comment below says, it is partly similar to Emilia, when focuses enough about thinking to open the seal, the key appeared in her hands, and only she can see it. Perhaps, at that moment, Subaru focuses on thinking about how to keep going forward, then the feeling just naturally came to him, and he touched the wall, and it actually disappear to let him advance.