安装

# 下载源码
git clone https://github.com/ali-vilab/dreamtalk
cd dreamtalk

conda create -n dreamtalk python=3.10
conda activate dreamtalk

conda install -c conda-forge yacs==0.1.8
conda install -c conda-forge numpy==1.21.5
conda install -c conda-forge av==10.0.0
conda install ffmpeg

# 修改requirements.txt中opencv-python的版本: opencv-python==4.9.0.80
pip install -r requirements.txt

# CPU版
# conda install pytorch==2.1.2 torchvision==0.16.2 torchaudio==2.1.2 cpuonly -c pytorch
# GPU版(https://pytorch.org/get-started/locally/)
conda install pytorch==2.1.2 torchvision==0.16.2 torchaudio==2.1.2 pytorch-cuda=11.8 -c pytorch -c nvidia

pip install urllib3==1.26.6
pip install transformers==4.28.1
conda install -c conda-forge dlib
pip install chardet
conda install -c conda-forge blas

模型下载

下载到dreamtalk/checkpoints目录

• https://modelscope.cn/api/v1/models/iic/dreamtalk/repo?Revision=master&FilePath=checkpoints/denoising_network.pth
• https://modelscope.cn/api/v1/models/iic/dreamtalk/repo?Revision=master&FilePath=checkpoints/renderer.pt

下载到dreamtalk/AI-ModelScope

cd dreamtalk/AI-ModelScope
git clone https://www.modelscope.cn/AI-ModelScope/wav2vec2-large-xlsr-53-english.git

修改inference_for_demo_video.py

WAV2VEC_MODEL_PATH = "/xxxx/dreamtalk/AI-ModelScope/wav2vec2-large-xlsr-53-english"

...

# get wav2vec feat from audio
wav2vec_processor = Wav2Vec2Processor.from_pretrained(WAV2VEC_MODEL_PATH)

wav2vec_model = (
    Wav2Vec2Model.from_pretrained(WAV2VEC_MODEL_PATH)
    .eval()
    .to(device)
)

测试验证

GPU运行

python inference_for_demo_video.py \
--wav_path /vxiao/funasr-runtime-resources/models/output.wav \
--style_clip_path data/style_clip/3DMM/M030_front_neutral_level1_001.mat \
--pose_path data/pose/RichardShelby_front_neutral_level1_001.mat \
--image_path /vxiao/SadTalker/examples/source_image/art_5.png \
--cfg_scale 1.0 \
--max_gen_len 30 \
--output_name test01

运行结果

• 耗时29秒

CPU运行

python inference_for_demo_video.py \
--wav_path /vxiao/funasr-runtime-resources/models/output.wav \
--style_clip_path data/style_clip/3DMM/M030_front_neutral_level1_001.mat \
--pose_path data/pose/RichardShelby_front_neutral_level1_001.mat \
--image_path /vxiao/SadTalker/examples/source_image/art_5.png \
--cfg_scale 1.0 \
--max_gen_len 30 \
--device cpu \
--output_name test01

运行结果

• 耗时176s

部署过程遇到的问题&处理

No CMAKE_CXX_COMPILER could be found

yum install gcc gcc-c++

ImportError: liblapack.so.3: cannot open shared object file: No such file or directory

yum install lapack-devel

参考资料

• 源码
• RuntimeError: Input type (torch.cuda.FloatTensor) and weight type (torch.FloatTensor) should be the
• 问题解决：Input type (torch.cuda.FloatTensor) and weight type (torch.FloatTensor) should be the