【项目实现】ResShift：通过残差位移实现图像超分辨率的高效扩散模型（NeurIPS 2023，聚焦）

基于扩散的图像超分辨率（SR）方法主要受到数百甚至数千个采样步骤要求导致推理速度低的限制。现有的加速度采样技术不可避免地在一定程度上牺牲了性能，导致SR结果过于模糊。为了解决这个问题，我们提出了一种新颖而高效的SR扩散模型，该模型显着减少了扩散步骤的数量，从而消除了推理过程中对后加速的需要及其相关的性能下降。该方法通过移动高分辨率图像和低分辨率图像之间的残差来构建一个马尔可夫链，大大提高了转换效率。此外，还制定了详细的噪声计划，以灵活控制扩散过程中的变速和噪声强度。大量实验表明，即使只有 15 个采样步骤，所提出的方法在合成和真实世界数据集上都获得了优于或至少与当前最先进的方法相当的性能。

论文地址：https://arxiv.org/abs/2307.12348

项目地址：https://github.com/zsyOAOA/ResShift

主机和显卡：【Intel® Xeon(R) CPU E5-2623 v3 @ 3.00GHz × 16】【NVIDIA Corporation GP102 [TITAN Xp] 】

项目效果图：

可能遇见的问题：

AttributeError: module ‘gradio‘ has no attribute ‘outputs‘

AttributeError: module ‘gradio‘ has no attribute ‘image’

RuntimeError: CUDA out of memory.

解决办法

AttributeError: module ‘gradio‘ has no attribute ‘outputs‘

AttributeError: module ‘gradio‘ has no attribute ‘image’

这两个问题属于一类，安装正确的gradio版本就可以了

pip install gradio==3.50.2

RuntimeError: CUDA out of memory.

Traceback (most recent call last):
File "/media/visionx/monica/project/ResShift/inference_resshift.py", line 107, in <module>
main()
File "/media/visionx/monica/project/ResShift/inference_resshift.py", line 104, in main
resshift_sampler.inference(args.in_path, args.out_path, bs=1, noise_repeat=False)
File "/media/visionx/monica/project/ResShift/sampler.py", line 234, in inference
im_sr_tensor = _process_per_image(im_lq_tensor)
File "/media/visionx/monica/project/ResShift/sampler.py", line 186, in _process_per_image
im_sr_tensor = self.sample_func(
File "/media/visionx/monica/project/ResShift/sampler.py", line 136, in sample_func
results = self.base_diffusion.p_sample_loop(
File "/media/visionx/monica/project/ResShift/models/gaussian_diffusion.py", line 410, in p_sample_loop
return self.decode_first_stage(final, first_stage_model)
File "/media/visionx/monica/project/ResShift/models/gaussian_diffusion.py", line 473, in decode_first_stage
out = first_stage_model.decode(z_sample)
File "/media/visionx/monica/project/ResShift/ldm/models/autoencoder.py", line 35, in decode
quant, emb_loss, info = self.quantize(h)
File "/home/visionx/anaconda3/envs/ResShift/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/media/visionx/monica/project/ResShift/ldm/modules/vqvae/quantize.py", line 281, in forward
torch.sum(self.embedding.weight**2, dim=1) - 2 * \
RuntimeError: CUDA out of memory. Tried to allocate 4.00 GiB (GPU 0; 11.90 GiB total capacity; 8.35 GiB already allocated; 2.66 GiB free; 8.50 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

解决办法：

CUDA_VISIBLE_DEVICES=0 python inference_resshift.py -i ./image/1.jpg -o ./result --task realsrx4 --chop_size 512

改为

CUDA_VISIBLE_DEVICES=0 python inference_resshift.py -i ./image/1.jpg -o ./result --task realsrx4 --chop_size 256