Jeong hyo Kim
AVATAR
ANIMOJI R&D
Here is some character project kicked off with 2 Machine Learning developers. We were interested by Apple's AR Kit and try to make facial capture app on mobile. It started with facial R&D including making blendshapes for 2 different parts. First one is to make the facial shape by user customization system. And second is to create realtime facial recognization tool.
Shape Customization Test
So we started with facial customization test. I created sample face with neutral expression. And made total 23 blendshapes. Each shape contains single characteristics that can be used as customization setting. Then we cobined them into the original model and export all to glb. format.
Sample screenshot of Shape Blendshapes
Shape Test with Built in Blendshapes in glTF Viewer
You can see it just acts like avatar customization tool
Facial Expression Test
This time we tested expression test. I created 52 blendshapes that is for the facial expression and also 15 trigger face which is set of expression like mad, frown, puke etc. You can see the test video. We were planning to use them to achieve smoother facial expression motion capture than original face rig.
Expression Blendshapes combined
Facial Motion Capture with Mobile
While developers now test those blendshapes through their logic, I found out app called iFacialMotion that can be used with the facerig. I decided to test them to see if this model is compatible with all the facial expression. I succeeded making it work and tried to make my face as extreme as possible and enjoy my model following my facial expression realtime.
iFacialMocap
Screen Recorded using iFacialMocap on mobile
We could see its following my facial expession through camera
Character Design R&D
After test is done and list up all the needed assets, we started defining the style of the avatars that will be used in the app. I searched few styles and did few sketches. And created this girl character. At that time, I was hugely inspired by the character styles from series of works "Kung-fu Piggy" by Koudai Waanabe.
Modeling of the Character
Character Concepts
We brought out some ideas to create few videos for the marketing resources and decided to the walking motion. Rigging and motion was done by Mixamo but later skinned again in maya. And later did test in Unity Too.
Character Concepts
Walking Test done in Unity with Clay Shader test
