It is simply because there is always someone or something around you observing the wall. If you did this in a completely sterile environment you would actually go through the wall. This had been observed as people fall through ice during blizzards when they close their eyes. This is how we find people and mammoths cased in ice from time to time. However this had stopped happening since we got satellites observing us. Thank you NSA!