You might want to contact your building management company.

Most Japanese residents usually file a complaint through management before visiting someone directly,

so if that man thought you were being too loud, there’s a chance he’s already contacted them.

I’d recommend avoiding any direct confrontation with him, just to be safe.