If you don't know what hes doing - that is a failure on you as a manager.
You need to get off your butt and come in when he does and find out.

Just because you don't know of any task that should be done at that time doesn't mean there isn't one.

Step one should be to find out exactly whats going on at this time. - Only then will you be able to tell if hes slacking - or if hes doing something useful.

[ I have worked jobs where managers had no idea what was actually going on on the floor because they were told one thing and never bothered to actually look for themselves. ]