Do you remember those stunning fight scenes in Journey To Immortality?
And also these subtle facial expressions.
Behind these smooth and vivid animations, motion capture technology plays an indispensable role.
Since 1915, when animation master Max Fleischer developed the rotoscope technique, the embryonic form of motion capture technology emerged. It wasn't until 1990, in the movie Total Recall, that motion capture technology made its official debut on the big screen. Entering the new century, motion capture technology experienced a golden period of development. In 2008, the film Avatar made motion capture technology widely known to the public, allowing Original Force to fully engage with top international motion capture technology and begin its own journey.
Consequently, in 2020, the animated series A Record of a Journey To Immortality's Journey to ImJourney To Immortalityity, created using a complete real-time animation production process involving live-action and facial expression capture + real-time UE engine, made a stunning debut, opening up a new genre in Chinese animation. Original Force has continued to advance on the path of motion capture technology, including real-time online motion capture technology, OF_DeepFace: a neural network-based markerless real-time facial animation capture system, and a neural network-based fully automated facial animation generation system for data processing.
The self-developed industry technology of Original Force is continuously upgrading.
Compared to traditional animation production, motion capture technology can more quickly and conveniently digitize human movements by recording human motion data. This allows real performances to be swiftly turned into virtual data and basic animations, greatly enhancing animation production efficiency. Therefore, in domestic animation and games, the usage rate of motion capture technology has been increasing. Power as a partner of internationally renowned companies such as Microsoft, Sony, EA, Netflix, and Tencent, captures thousands of minutes of motion capture data almost every year. This vast accumulated database is not only one of Power's core competitive advantages but also provides a solid data foundation for Power to continue optimizing and upgrading its motion capture technology.
Today, Original Force is once again launching powerful tools that enhance the quality and efficiency of motion capture in both hardware and software: the markerless dual-camera single-image facial capture system DeepFace 2.0 and the neural network-based motion capture solving system DeepMocap!
Let's take a look at our latest achievements!
Technology Upgrade
Markerless binocular single-image facial capture DeepFace 2.0
In 2021, the proprietary markerless facial capture technology, Deepface, developed by Original Force, was officially put into production for the company's animated films and series. This technology was also made available to the industry at large, significantly enhancing the productivity of the animation production process. It liberated artists from a substantial amount of tedious manual correction work, allowing them to focus more on improving the artistic quality of their creations.
Over the past three years, artists and production management departments have provided extensive feedback to Deepface based on their workflow needs. In response to this feedback, the R&D department has undertaken targeted efforts and has finally completed an overhaul of the entire facial capture system. Recently, they officially launched a markerless facial capture solution based on a self-developed binocular helmet!
So, the first question is—what is a binocular helmet?
In the animated image above, the motion capture actor is wearing a monocular helmet, which is also the most common tool for facial capture today. However, the quality of facial motion capture data collected by monocular helmets still has deficiencies. In terms of details, monocular helmets cannot perfectly capture the actor's micro-expressions, and the solved facial capture data contains many parts that require manual adjustments by artists. Therefore, the Original Force Technology team had a sudden idea: if one eye cannot capture well, what about two eyes?
Original Force self-developed binocular motion capture helmet.
Original Force self-developed binocular motion capture helmet is a lightweight binocular facial expression video capture device.
The design of the helmet's shape takes into account lightweight, stability, and ergonomics. By utilizing head scans from anthropometric data, it can be custom-fitted to the shape of the head, providing a personalized design for each user. Additionally, the helmet employs nylon sintering printing technology, which significantly reduces the helmet's weight and enhances its stability, thereby minimizing discomfort for motion capture actors. Users can also control the helmet size through a ratchet mechanism, allowing for tightening and securing for different head sizes. The camera angle is adjustable as well, maximizing the reduction of resistance for actors during immersive performances.
If a helmet is small and lightweight, will it affect its functionality?
The answer is, no.
The self-developed Force dual-eye motion capture helmet can simultaneously capture two 1080p, 60-frame images, while the camera's cross-sectional diameter is only 21mm, minimizing obstruction of the actor's line of sight. Additionally, the helmet is equipped with 12 built-in 940nm infrared fill lights, which, when used with the infrared lens, can eliminate facial shadows, making the image output more stable. It also supports switching between black-and-white and color modes.
To achieve this level of technical presentation, the core lies in a card built into the helmet—a single image transmission board card. This card is a single image transmission capture device that can capture data signals from two cameras, one above and one below. Through the proprietary hardware programming of Original Force, the data is integrated into a three-way signal and output. The image transmission uses the YUV422 encoding method, and with Original Force's proprietary software, the signals can be decomposed through an algorithm to form two image signals.
As a result, the dual-lens helmet breaks the limitation that dual-channel signals must use two image transmission cards, reducing the data acquisition traffic load and significantly alleviating the issue of excessive hardware weight that could be caused by dual acquisition cards.
Single image showing the cross-sectional structure of a circuit board
With so many technological upgrades, how effective are the improvements?
Comparison of Monocular and Binocular Facial Motion Capture Effects
In terms of facial capture systems, Original Force Technology team has modified the internal algorithm of DeepFace from single image input to dual image input, and improved the structure based on the original network architecture by deepening the network layers and adding residual modules. This allows the network to more accurately learn the correspondence between character animations and actor performances. The most intuitive manifestation is that dual-camera facial capture significantly outperforms single-camera capture networks in capturing fine movements, such as whether the eyes are fully closed and the subtle opening and closing of the mouth.
System update
DeepMocap: A neural network-based motion capture solving system
Based on different acquisition methods, motion capture technology can be divided into inertial motion capture, optical motion capture, and facial expression capture, among others. Currently, the most common type on the market is optical motion capture. It uses traditional algorithms, and its working principle is to calculate the skeletal motion data in three-dimensional space from the captured marker points using an algorithmic system.
However, there is a significant issue with the traditional process—signal instability. In the optical capture process, the positions of the captured light markers are often obscured or incorrectly identified, resulting in errors in skeletal information resolution. Therefore, in the traditional capture process, a considerable amount of manpower is often required to manually correct these erroneous data, leading to low production efficiency.
After three years of intensive research and development, Original Force has finally developed a deep learning algorithm and corresponding application tools that can directly solve skeletal data from captured light spot markers using neural networks. Let's first take a look at the comparison of the results.
To more clearly showcase the comparison effect, the playback speed is approximately 70% of the original speed.
The color of the normal motion capture dots on the actor corresponds to the color of the actor's skeleton. Red dots indicate unrecognized capture markers, and red skeletons indicate incorrectly recognized movements.
It can be seen that in traditional motion capture solving software, when multiple actors have entangled or interacting actions that result in capture marker points being obscured, or when special actions by actors cause capture marker points to be obscured and lost and unable to be recovered, traditional motion capture solving software cannot identify the correct skeletal positions. However, Original Force self-developed neural network-based motion capture solving system DeepMocap can solve this problem very well.
How does it achieve such an effect?
DeepMocap algorithm is based on a vast amount of actual production motion capture data and has augmented the motion capture data to conform to prior distributions as a training dataset. The algorithm consists of two main networks: the preceding network uses a transformer structure to calibrate and process the position data of the captured marker points; the subsequent network utilizes graph convolution to extract information, employing an Encoder-Decoder structure to associate the captured marker data with skeletal data, thereby solving for correct skeletal animation data even under imperfect or incorrect captured marker point information. After the network training is completed, DeepMocap can still deduce reasonable and correct skeletal animation data even when the input includes inaccurate captured marker point position data.
Moreover, this system includes two sets of solving modules: a real-time solving module and a post-processing solving module, integrated into the motion capture software MotionBuilder as plugins. In the future, we also plan to add more optimization modules to enhance the final animation data, such as using neural networks to adjust motion data that is not natural enough into smooth and natural skeletal animation data, and tools for automatically generating natural transition animations between different key poses.
Compared to traditional processes, this system has two main advantages:
1. Compared to traditional workflows, this system has a higher tolerance for the loss or mislabeling of marker points when capturing light dots. The real-time solving module can still provide relatively reasonable skeletal positions even when marker points are lost or mislabeled, making it easier for shooting technicians to view real-time effects. Additionally, as the training dataset continues to expand, the network can continuously iterate and optimize, steadily improving the final solving results.
2. The post-processing solver module utilizes information from previous and subsequent frames to further improve the accuracy of the solving results. It significantly reduces the workload of motion capture personnel in manually correcting capture point markers frame by frame to generate reasonable skeletal animation data, thereby saving a substantial amount of production time and labor costs.
By leveraging a combination of hardware + software, Original Force has successfully optimized two common challenges in existing motion capture technology: inadequate capture of fine movements and suboptimal skeletal position solving. This technological breakthrough not only injects powerful technical momentum into Original Force's own project production but also revitalizes the motion capture field within the industry. It significantly enhances the quality and efficiency of motion capture, thereby freeing up human labor.
In the future, Original Force will continue to optimize and innovate more key industry technologies to provide clients with more efficient and professional 3D digital content services. Currently, Original Force is fully opening up the aforementioned technologies for collaboration within the industry, with the aim of further expanding the application and development of motion capture technology and digital technology in more industry fields through exploration and feedback.
If you are interested in the above technology and willing to engage in deeper communication and collaboration with Original Force R&D department, please send an email to dev@of3d.com to contact Original Force.
Message
Top