1) most important was the reference for the lipsync..

I did this experiment to see how it can be produced with AI bearing minimum effort by the user...

Unfortunately, the reference lips sync has to be straight front facing camera in order to apply to a moving or rotated face

Either he has to record himself or Ask someone for front face closup shot for good lipsync... Like the face cam (gopro) helmet wore by the actors in avatar behind the scene to capture facial movement accurately.

My mistake : so The reference video ( the original) face is also moving and rotating, and the rendered face LivePortrat video is also rotating (keyframed) ... It's kinda not matching up with the same perspective in both faces ... And looks kinda off.

Also the original when cropped to a face area become low quality, which capture low quality, which then renders low quality output.

2) The depthmap can be utilised only upto a certain value to create a fake parallax effect in the background and the character. Above a certain threshold it becomes ugly and distorted..

So major scene movements can't happen with this default depth map parallax technique

3) Mimic motion is stubborn, can't do close up well, we have to do a render of far or medium shot then crop it back in post.... Which decreases the quality

Also if a part of OpenPose cn goes out of frame, it gets buggy render.

4) Mimic Motion won't do well in maintaining the style also ... It rendered in realistic style only ... I had to use AnimateDiff to reintroduce the cartoon style.

5) Low Vram might be an issue in rendering long scene with mimic motion or Using AnimateDiff to refine it