FireRedTeam 开源 PhotoPoster：姿态驱动的肖像动画图像生成项目

FireRedTeam; PhotoPoster

Repository navigation

NotificationsYou must be signed in to change notification settings; Fork13 (13); Star207 (207)You must be signed in to star a repository

To support and further the research in the field of portrait animation , we are excited to launch PhotoPoster, an open project for pose-driven image generation.

Repository files navigation

README

Introduction

The field of portrait animation has seen significant advancements in recent years, largely due to progress in diffusion models. To support and further this research, we are excited to launch PhotoPoster, an open project for pose-driven image generation.In the realm of image-to-image (i2i) translation, PhotoPoster stands out by incorporating a refinement stage to significantly enhance overall image quality. For more detailed information and visual demonstrations, please visit our project page. We are committed to opening complete source code for free and regularly updating PhotoPoster with several planned enhancements, including improvements in algorithm module engineering, data processing pipelines, and training code optimization.Additionally, we are gearing up to release DynamicPose in two weeks, which is a complementary image-to-video (i2v) project.

Our lab is a collective of passionate AIGC enthusiasts dedicated to the development of highly controllable video generation techniques. We pledge to openly share all of our findings with the community at no cost, and we warmly invite you to join us in discussion and collaborative research. If you are interested in exploring or contributing to any of these modules, please feel free to email us for project discussions. Btw, if our open source work can benefit you, we would greatly appreciate it if you could give us a star ⭐!

[08/21/2024] 🔥 Release PhotoPoster project and pretrained models; [08/21/2024] 🔥 Release pose server based FastAPI; In the coming two weeks, we will release comfyui of PhotoPoster and DynamicPose

Demos

Todo:

Release inference codes and checkpoints of pre generation stage; Release inference codes and checkpoints of hand restoration net; Release pose server based FaseAPI; Release Comfyui of the PhotoPoster; Release project of DynamicPose; Release inference codes and checkpoints of face restoration net; Release training codes; Release the data cleaning and preprocessing pipeline code; Huggingface Gradio demo

Contents

Getting Started; Inference; Acknowledgement; License

Getting Started

1. Installation

We recommend a python version >=3.10 and cuda version =11.7. Then build environment as follows:

You can download weights manually as follows.

pose server

DWPose; RTMPose; pre generation

Download the weights of these components:

SD1.5; sd-vae-ft-mse; image_encoder; Download our trained weights, which include three parts: denoising_unet.pth, reference_unet.pth and pose_guider.pth; hand restoration

Download the weights of sd-2-inpaint; Download our trained weights, which include two parts: config.json and diffusion_pytorch_model.safetensors

Finally, these weights should be organized in pretrained_weights as follows:

Inference

1. pose server

Firstly, you need to prepare the pose images and human keypoint coordinates extracted from the original images. Considering that the pose models are frequently queried, to avoid loading the same model multiple times and improve the usability of the pose module, we deployed the pose models' inference service using a simple FastAPI script. And we also provides a concise client demo. Specifically, you only need to run the following script to deploy the pose script's inference service, and keep it running in the program background without terminating it.

The following is a simple client script. And you will get two new files: assets/stage1/ref_img1.png.pose.png and assets/stage1/ref_img1.png.pose.pkl, which represent the pose image and keypoint information, respectively.

2. pre generation

Here is the cli command for running stage1 inference scripts:

You can modify the configuration file to change the image you want to test. Additionally, we have implemented a simple yet useful distributed inference computation framework in the inference code. If you have multiple GPUs and want to enable multiple computing processes for large-scale computation, you can modify the CUDA_VISIBLE_DEVICES, world_size, and global_id parameters and run multiple inference scripts simultaneously. The inference speed will exhibit linear growth.

3. hand restoration

The hand restoration network requires the output of stage1, including the result image and the pose image (aligned), as input. Here is the cli command for running hand restoration inference scripts:

Acknowledgement

We thank AnimateAnyone for their technical report, and have refer much to Moore-AnimateAnyone and diffusers; We thank open-source components like dwpose, Stable Diffusion, rtmpose, etc

License

code: The code of PhotoPoster is released under the MIT License; other models: Other open-source models used must comply with their license, such as sd2-inpainting, dwpose, rtmpose, etc

Citation

To support and further the research in the field of portrait animation , we are excited to launch PhotoPoster, an open project for pose-driven image generation.