Audio2Face

This was done before Nvidia added Audio2Emotion to Omniverse.

This is a lip sync test I did with Nvidia's Audio2Face software (so turn audio on to hear it). This is using Unreal's new Mesh to Metahuman workflow with an exaggerated face.

If I just used Audio2Face, I got the 'dead eye' look, so I mixed in some iPhone Live Link facial capture for the eyes and brows in an additive animation track. The iPhone capture was much more 'twitchier' than I remember, going from 0-100 really quickly (some crazy brow movements in there).

I might do a test with Faceware to see if that looks more natural.

Just for fun I rendered it with Unreal's Pathtracer. On my 3080 this took about a minute a frame, so about 17.5 hours to render. I did a regular Lumen/Raytraced render and it took about 20 minutes. I think Pathtracing looks way better except for the eyes which look strangely too bright.

I was pleased with the results of Audio2Face, but I think I'd want to do a manual pass to make sure it hits the M, P and B sounds a bit harder. It would also take a good bit of work to add some emotion to it.