How to transfer controlnet to your own model: Update readme

[specblades]

Please, add more detailed tutorial how to transfer controlnet to your own model

[2blackbar]

I think each model for use with this code it has built in SD1.5 + its own weights for features detection, but it ignores its own SD1.5 and uses your modl from get go when you use this repo code...
What i would like instead is separating feature detection weights from SD1.5 so the file is smaller loads faster and then it would work the same as its working now ( isuppose) It kinda loads SD1.5 twice when you use it on plain 1.5 model from what i read

[Mikubill]

separating feature detection weights from SD1.5

use https://github.com/Mikubill/sd-webui-controlnet/blob/main/extract_controlnet.py

[2blackbar]

thats grat ill use it and report if it works

[nasenbaer43]

Unless I don' tunderstand it: the script extracts the controlnet from the 5.7 GB files (see https://huggingface.co/lllyasviel/ControlNet/tree/main/models). I guess the resulting files will be quite smaller?

Is there any chance that someone who has already downloaded all these models to please upload the extracted results somewhere?

Seems like a bit of "waste" for everyone to download 7 * 5.7 GB.

[2blackbar]

WORKED !!! Thanks a lot !!!
For anyone else , make bat file and place in models directory, then run it, bat looks like this inside , just change path to yours and change names accordingly
python extract_controlnet.py --src F:\sd\extensions\sd-webui-controlnet\models\control_sd15_canny.pth --dst F:\sd\extensions\sd-webui-controlnet\models\control_canny.pth

[2blackbar]

Unless I don' tunderstand it: the script extracts the controlnet from the 5.7 GB files (see https://huggingface.co/lllyasviel/ControlNet/tree/main/models). I guess the resulting files will be quite smaller?

Is there any chance that someone who has already downloaded all these models to please upload the extracted results somewhere?

Seems like a bit of "waste" for everyone to download 7 * 5.7 GB.

i posted how to do it in post above , make bat file and just run it, its easy, im going to extract more, you can do it on your own and it gives you 1.4gb model not 5.6gb modl, so it saves VRAM and loads fast , tested and works.
Imo they should upload these to huggingface repo...

[nasenbaer43]

Sorry you misunderstood me. I was talking about the extracted models, aka your 1.4 GB files.
Of course I can download the 5.7 GB files and extract them myself :)

Since you already did it: any chance you can please upload them anywhere?

[2blackbar]

Sorry you misunderstood me. I was talking about the extracted models, aka your 1.4 GB files. Of course I can download the 5.7 GB files and extract them myself :)

Since you already did it: any chance you can please upload them anywhere?

Here you have 3, you want more try to convert with the way i posted , its really very easy
https://drive.google.com/file/d/1iVlm_LDOiCirGnxrGrzZtVyJf2PeMvoR/view?usp=share_link
https://drive.google.com/file/d/13BerWa0M_jpvWOiLvxJd5zqadkZ8hZwd/view?usp=share_link
https://drive.google.com/file/d/1sfyZH8N_7lWbFBb8Pb_9J_6o9g3q3yJt/view?usp=share_link

[MadLightTheDoggo]

WORKED !!! Thanks a lot !!! For anyone else , make bat file and place in models directory, then run it, bat looks like this inside , just change path to yours and change names accordingly python extract_controlnet.py --src F:\sd\extensions\sd-webui-controlnet\models\control_sd15_canny.pth --dst F:\sd\extensions\sd-webui-controlnet\models\control_canny.pth

You have to place the .bat into the directory with extract_controlnet.py and NOT INTO \models\ folder.

[Hetaneko]

here i uploaded the trimmed models with safetensors format
https://huggingface.co/Hetaneko/Controlnet-models/tree/main/controlnet_safetensors

[Echolink50]

here i uploaded the trimmed models with safetensors format https://huggingface.co/Hetaneko/Controlnet-models/tree/main/controlnet_safetensors

Sweet. Thanks a lot

[ClashSAN]

here's pruned f16 safetensors set - https://huggingface.co/webui/ControlNet-modules-safetensors/tree/main

[cerarslan]

Excuse me guys, what is use case of above the models? Just merge our custom models with those?

[Echolink50]

Excuse me guys, what is use case of above the models? Just merge our custom models with those?

No. Just leave your chosen model selected in it's usual location. Load your chosen preprocessor and controlnet model in the drop-down menu

[GrennKren]

here's pruned f16 safetensors set - https://huggingface.co/webui/ControlNet-modules-safetensors/tree/main

take off sunglasses
Oh my..

[paulo-coronado]

Excuse me guys, but I think the initial question was how to transfer ControlNet to a custom model (such as Anythingv3, RealisticVision, Deliberate etc.). I found amazing how you extract the weights and reduce the file size, but I would like to know how @Mikubill transferred ControlNet to a model, did he merged SD15Model + ControlNet?... I used this guide to create a control_any3_openpose.pth file. However, I compared the images created by this model and by ControlNet extension (by @Mikubill), using same seed and configs, and the results are a bit different... I don't know why... any thoughts?

These are the results I got (same prompt, seed etc.):

[deifos]

@paulo-coronado I'm looking for the same thing how can I merge a controlnet model to my custom one, where you able to get that going?

[paulo-coronado]

@deifos Hey brother! You can merge by following this guide. Although, the author is not recommending merging that way anymore, because the results are a little bit different. The recommendation is to use A1111 and its API. If you really need to merge them using the tool_transfer will make the job nicely!

[deifos]

@deifos Hey brother! You can merge by following this guide. Although, the author is not recommending merging that way anymore, because the results are a little bit different. The recommendation is to use A1111 and its API. If you really need to merge them using the tool_transfer will make the job nicely!

Thank man, I was able to merge it but how did you use it, did you load it inside the controlnet/models folder or the stable diffusion models folder because it does not seem to have any effect on my generations.

[paulo-coronado]

I am not sure if you can use it in A1111 (maybe replacing control_canny-fp16.safetensors with your merged model). The way lllyasviel created was to use it in gradio_pose2image.py - just replace the following line as the guide suggests:

model.load_state_dict(load_state_dict('./models/control_any3_openpose.pth', location='cpu'))

[alelordelo]

WORKED !!! Thanks a lot !!!
For anyone else , make bat file and place in models directory, then run it, bat looks like this inside , just change path to yours and change names accordingly
python extract_controlnet.py --src F:\sd\extensions\sd-webui-controlnet\models\control_sd15_canny.pth --dst F:\sd\extensions\sd-webui-controlnet\models\control_canny.pth

HI @2blackbar and @paulo-coronado ,

I tried now to extract but the converted.ckpt model has the same size ocustomtrainedCN.ckpt, which is 8.5GB.

Shouldn't this reduce the file size?

reported issue here:
#895