DDT:一种基于扩散驱动变压器的视频人体网格恢复框架 DDT: A Diffusion-Driven Transformer-based Framework for Human Mesh Recovery from a Video

作者:Ce Zheng Guo-Jun Qi Chen Chen

人体网格恢复(HMR)为游戏、人机交互和虚拟现实等各种现实世界应用提供了丰富的人体信息。与基于单个图像的方法相比,基于视频的方法可以利用时间信息,通过引入人体运动先验来进一步提高性能。然而,像VIBE这样的多对多方法存在运动平滑和时间不一致的问题。而像TCMR和MPS-Net这样的多对一方法依赖于未来的框架,这是非因果的,并且在推理过程中时间效率低下。为了应对这些挑战,提出了一种新颖的基于扩散驱动变压器的视频HMR框架(DDT)。DDT被设计用于从输入序列中解码特定的运动模式,增强运动平滑度和时间一致性。作为一种多对多的方法,我们的DDT解码器输出所有帧的人体网格,使DDT在时间效率至关重要的现实世界应用中更可行

Human mesh recovery (HMR) provides rich human body information for variousreal-world applications such as gaming, human-computer interaction, and virtualreality. Compared to single image-based methods, video-based methods canutilize temporal information to further improve performance by incorporatinghuman body motion priors. However, many-to-many approaches such as VIBE sufferfrom motion smoothness and temporal inconsistency. While many-to-one approachessuch as TCMR and MPS-Net rely on the future frames, which is non-causal andtime inefficient during inference. To address these challenges, a novelDiffusion-Driven Transformer-based framework (DDT) for video-based HMR ispresented. DDT is designed to decode specific motion patterns from the inputsequence, enhancing motion smoothness and temporal consistency. As amany-to-many approach, the decoder of our DDT outputs the human mesh of all theframes, making DDT more viable for real-world applications where timeefficiency is crucial and a causal model is desired. Extensive experiments areconducted on the widely used datasets (Human3.6M, MPI-INF-3DHP, and 3DPW),which demonstrated the effectiveness and efficiency of our DDT.

论文链接:http://arxiv.org/pdf/2303.13397v1

更多计算机论文:http://cspaper.cn/

Related posts