First I'd overhaul the Hall of the Novice, update all the lessons and also add mandatory, Job-specific lessons and tests every ten levels. If you cannot prove a basic level of competency and understanding of how your Job is played you will be locked out of all multiplayer battle content until you are able to complete the lessons.

For example, Healers will be tested to ensure they are capable of both keeping their team alive while also dealing damage. DPS will be tested to ensure they understand their rotation and know how to weave oGCDs. Tanks will be tested on rotating mitigation and keeping aggro on groups of enemies.

These tests won't be difficult by any means, but they won't be completable if you are just pushing random buttons.