SenseMaltose

In the field of live streaming, product sales are flourishing nowadays. Many top live hosts and celebrities have joined in this prosperous industry. At the same time, a new model of virtual avatar-led product sales has emerged that can guarantee effective promotion even during off-peak periods. However, existing virtual selling avatars mostly appear mechanical and clumsy. There is a lack of high-quality virtual avatars in the market.

We develop SenseMaltose, an AIGC-Avatar system for live-streaming product promotion. The system supports continuous hours of live streaming and can promote products in various categories with unrepeated sentences. The avatar could also have different but characteristic styles of voice and language habits. Also, this system can allow fully autonomous interaction between the host avatar and the audience in the live rooms.

The core features include:

• Body motion is driven automatically without human intervention or actions but matches the context.
• Facial expressions are driven by voice simultaneously, matching corresponding expression expectations.
• The ability to interact with the audience (comments in liveroom). The context should be logical and meet the avatar’s personality and background.
• All these functionalities operate automatically. The system is able to finish the whole round of live shows without users’ intervention.

My contributions to this project are:

• Build up the first-generation pipeline to validate its feasibility.
• Unreal Engine development in :
  • Animation Graph: for motion, facial motion, and audio integrated AIGC-Avatar pipeline.
  • Websocket Servers: for transferring data and control settings between Unreal Engine end and Back end.
  • Avatar Related: to improve avatar performance, including auto eye blink, auto eye focus, simple bone solver, and character-dependent facial data calibration.
  • Composite Layers: a light-weight implementation of composite layer manager that can stack picture and video sources in Unreal Engine viewports, providing smoothly transitions and accurate color management.