ControlNet核心基础知识

---

一、环境配置和安装需要使用的库

1.创建并激活环境

conda create -n controlnet python=3.8
conda activate controlnet

2.安装需要的库

pip install opencv-python transformers accelerate
pip install diffusers
pip install xformers

二、准备数据及相关模型

1. 下载图像数据：知名画作《戴珍珠耳环的少女》
2. 下载controlnet模型：lllyasviel/sd-controlnet-canny
3. 下载stable-diffusion-v1-5模型runwayml/stable-diffusion-v1-5（可以换成别的）

模型下载的方式：采用git远程clone下来，具体方式可以参考之前的内容：huggingface学习 | 云服务器使用git-lfs下载huggingface上的模型文件；

三、参照样例编写代码

（一）导入相关库

from diffusers import StableDiffusionControlNetPipeline, ControlNetModel, UniPCMultistepScheduler
from diffusers.utils import load_image
import numpy as np
import torch

import cv2
from PIL import Image

（二）准备数据（以知名画作《戴珍珠耳环的少女》为例）

1. 可以直接从huggingface官网加载图片：

# download an image
image = load_image(
    "https://hf.co/datasets/huggingface/documentation-images/resolve/main/diffusers/input_image_vermeer.png"
)
image = np.array(image)

2. 也可以从本地读取图片：

# download an image
image_path="./input_image_vermeer.png"
image=Image.open(image_path)
image = np.array(image)

（三）将这张图片交给Canny Edge边缘提取器进行预处理

# get canny image
image = cv2.Canny(image, 100, 200)
image = image[:, :, None]
image = np.concatenate([image, image, image], axis=2)
canny_image = Image.fromarray(image)

canny_image.save("./controlnet_result0.png")

边缘提取预处理之后的效果如下图所示：

（四）图像生成

1. 导入runwaylml/stable-diffusion-v1-5模型以及能够处理Canny Edge的ControlNet模型

controlnet = ControlNetModel.from_pretrained("./sd-controlnet-canny", torch_dtype=torch.float16)
pipe = StableDiffusionControlNetPipeline.from_pretrained(
    "./stable-diffusion-v1-5", controlnet=controlnet, torch_dtype=torch.float16
)

2. 使用当前速度最快的扩散模型调度器-UniPCMultistepScheduler，迭代20次就可以达到之前默认调度器50次迭代的效果

# speed up diffusion process with faster scheduler and memory optimization
pipe.scheduler = UniPCMultistepScheduler.from_config(pipe.scheduler.config)
# remove following line if xformers is not installed
pipe.enable_xformers_memory_efficient_attention()

pipe.enable_model_cpu_offload()

3. 生成最终图像

generator = torch.manual_seed(0)
image = pipe(
    "futuristic-looking woman", num_inference_steps=20, generator=generator, image=canny_image
).images[0]
image.save("./controlnet_result1.png")

最终生成的效果图如下所示：

四、完整代码

from diffusers import StableDiffusionControlNetPipeline, ControlNetModel, UniPCMultistepScheduler
from diffusers.utils import load_image
import numpy as np
import torch

import cv2
from PIL import Image

# download an image
image_path="./input_image_vermeer.png"
image=Image.open(image_path)
image = np.array(image)

# get canny image
image = cv2.Canny(image, 100, 200)
image = image[:, :, None]
image = np.concatenate([image, image, image], axis=2)
canny_image = Image.fromarray(image)

canny_image.save("./controlnet_result0.png")

# load control net and stable diffusion v1-5
controlnet = ControlNetModel.from_pretrained("./sd-controlnet-canny", torch_dtype=torch.float16)
pipe = StableDiffusionControlNetPipeline.from_pretrained(
    "./stable-diffusion-v1-5", controlnet=controlnet, torch_dtype=torch.float16
)

# speed up diffusion process with faster scheduler and memory optimization
pipe.scheduler = UniPCMultistepScheduler.from_config(pipe.scheduler.config)
# remove following line if xformers is not installed
pipe.enable_xformers_memory_efficient_attention()

pipe.enable_model_cpu_offload()

# generate image
generator = torch.manual_seed(0)
image = pipe(
    "futuristic-looking woman", num_inference_steps=20, generator=generator, image=canny_image
).images[0]
image.save("./controlnet_result1.png")

参考：
Huggingface中ControlNet pipeline介绍
扩散模型实战（十三）：ControlNet结构以及训练过程