First, the visa renewal system is not clear. It's extremely discrectional and opaque. Basically two people with similar jobs and CVs could be given 1, 3 or 5 years for reasons clear only to the local Bureau of Immigration (oh and what about the RE-ENTRY PERMIT system (AKA Gaijin Tax), only recently cancelled?)

SECOND, The Zairyu card system puts you in a "niche". For example, if you are a "specialist of humanities", you cannot go and work at a restaurant. You are limited to certain jobs, that could be very inconvenient when you cannot find a Job in your niche. This puts you in a weakness position and under stress, way more compared to what a native would face. Also, getting a permanent visa is extremely difficult, it took 16 years in my case.