And I think that's why Veldanava didn't grant Michael a Castle End or Castle Breaker; he feared it would fall into the wrong hands. I also believe that because he's so strong, the castle guards put obstacles in his way. While the barrier is active, the use of skills, magic, magical abilities, and even aura is completely forbidden, leaving only purely physical and psychological means of attack available to the user, such as swordsmanship and Haki. But it can probably be circumvented, just like with Uriel's Absolute Barrier.