FireRedTeam 开源人像动画框架 DynamicPose 并发布预训练模型

FireRedTeam; DynamicPose

Repository navigation

NotificationsYou must be signed in to change notification settings; Fork6 (6); Star84 (84)You must be signed in to star a repository

DynamicPose, a simple and robust framework for animating human images.

Repository files navigation

README

We introduce DynamicPose, a simple and robust framework for animating human images, specifically designed for portrait animation driven by human pose sequences. In summary, our key contributions are as follows:

Large-Scale Motion: Our model supports large-scale motion in diverse environments and generalizes well to non-realistic scenes, such as cartoons; High-Quality Video Generation: The model can generate high-quality video dance sequences from a single photo, outperforming most open-source models in the same domain; Accurate Pose Alignment: We employed a high-accuracy pose detection algorithm and a pose alignment algorithm, which enables us to maintain pose accuracy while preserving the consistency of human body limbs to the greatest extent possible; Comprehensive Code Release: We will gradually release the code for data filtering, data preprocessing, data augmentation, model training (DeepSpeed Zero2), as well as optimized inference scripts

We are committed to providing the complete source code for free and regularly updating DynamicPose. By open-sourcing this technology, we aim to drive advancements in the digital human field and promote the widespread adoption of virtual human technology across various industries. If you are interested in any of the modules, please feel free to email us to discuss further. Additionally, if our work can benefit you, we would greatly appreciate it if you could give us a star ⭐!

[08/28/2024] 🔥 Release DynamicPose project and pretrained models; [08/28/2024] 🔥 Release pose server and pose align algorithm; In the coming two weeks, we will release Comfyui and Gradio of DynamicPose

Inference codes and pretrained weights; Release pose server based FaseAPI and pose align algorithm; Comfyui of DynamicPose; Huggingface Gradio demo; Data cleaning and preprocessing pipeline; Training scripts

Build Environtment

We Recommend a python version >=3.10 and cuda version =11.7. Then build environment as follows:

Download weights

You can download weights manually as follows.

Manually downloading: You can also download weights manually, which has some steps:

Download our trained weights, which include four parts: denoising_unet.pth, reference_unet.pth, pose_guider.pth and motion_module.pth; Download pretrained weight of based models and other components:

StableDiffusion V1.5; sd-vae-ft-mse; image_encoder; Download rtmpose weights (rtmw-x_simcc-cocktail14_pt-ucoco_270e-384x288-f840f204_20231122.pth, tmdet_m_8xb32-100e_coco-obj365-person-235e8209.pth) and the corresponding scripts from mmpose repository

Finally, these weights should be orgnized as follows:

stage 1 image inference:

stage 2 video inference:

You can refer the format of configs to add your own reference images or pose videos. First, extract the keypoints from the input reference images and target pose for alignment:

ComfyUI

-DynamicPose-Comfyui

This work also has some limitations, which are outlined below:

When the input image features a profile face, the model is prone to generating distorted faces; When the background is complex, the model struggles to accurately distinguish between the human body region and the background region; When the input image features a person with objects attached to their hands, such as bags or phones, the model has difficulty deciding whether to include these objects in the generated output

We thank AnimateAnyone for their technical report, and have refer much to Moore-AnimateAnyone and diffusers; We thank open-source components like dwpose, Stable Diffusion, rtmpose, etc

code: The code of DynamicPose is released under the MIT License; other models: Other open-source models used must comply with their license, such as stable-diffusion-v1-5, dwpose, rtmpose, etc

DynamicPose, a simple and robust framework for animating human images.