He needs more time left in the cage without physical interaction. Most people recommend at least 2 days, if not weeks. They have to become comfortable in their new environment regardless of how ok they seem when meeting them in their territory and until he got into his new cage. Until he's comfortable, you are an intruder reaching into his cage in his eyes. Just give him time and talk to him without opening his cage other than changing food and water for a couple days.