HomeTutorials
[SD Advanced Tutorial]What is hypernetwork
31

[SD Advanced Tutorial]What is hypernetwork

AI  Learning Assistant No 1
AI Learning Assistant No 1
August 22nd, 2023

Are you ready to take your Stable Diffusion skills to the next level? If so, let’s talk about hypernetwork models. In this post, you’ll learn everything you need to know about hypernetworks and how to use them to achieve the results you’re looking for. I’ll cover everything from the basics of what a hypernetwork model is to how to use it with AUTOMATIC1111 Stable Diffusion GUI. Plus, I’ll even show you some of my favorite hypernetworks that I use in my own work. So, buckle up and let’s dive into the world of hypernetwork models.

What is a hypernetwork in Stable Diffusion?

Hypernetwork is a fine-tuning technique initially developed by Novel AI, an early adopter of Stable Diffusion. It is a small neural network attached to a Stable Diffusion model to modify its style.

Where is the small hypernetwork inserted? It is, of course, the most critical part of the Stable Diffusion model: the cross-attention module of the noise predictor UNet. LoRA models similarly modify this part of Stable Diffusion models but in a different way.

The hypernetwork is usually a straightforward neural network: A fully connected linear network with dropout and activation. Just like the ones you would learn in the introductory course on neural networks. They hijack the cross-attention module by inserting two networks to transform the key and query vectors. Compare the original and the hijacked model architecture below.

During training, the Stable Diffusion model is locked but the attached hypernetwork is allowed to change. Since the hypernetwork is small, training is fast and demands limited resources. Training can be done on a run-of-the-mill computer. Fast training and small file sizes are the main appeals of hypernetworks.

You should be aware that it is NOT the same as the hypernetwork commonly known in machine learning. That’s a network that generates weights for another network. So no, the hypernetwork of Stable Diffusion was not invented in 2016.

Difference from other model types

I will explain the difference between hypernetworks and other model types to help you understand and decide which one to use.

This section is for curious minds or model trainers. You can skip to the next section if you don’t care how they work.

Checkpoint models

Checkpoint models contain all the necessary information to generate images. You can recognize them by their large file size. They range from 2 to 7 GB. Hypernetwork is typically below 200 MB.

Hypernetwork cannot function alone. It needs to work with a checkpoint model to generate images.

The checkpoint model is more powerful than a hypernetwork. It can store styles a lot better than a hypernetwork. When training a checkpoint model, the whole model is fine-tuned. When training a hypernetwork, only the hypernetwork is fine-tuned.

LoRA models

LoRA models are most similar to hypernetworks. They are both small and only modify the cross-attention module. The difference lies in how they modify it. A LoRA model modifies the cross-attention by changing its weight. Hypernetwork does it by inserting additional networks.

Users generally find LoRA models produce better results. Their file sizes are similar, typically below 200MB, and way smaller than checkpoint models.

LoRA is a data storage method. It does not define the training process, which can either be dreambooth or additional training. Hypernetwork defines the training.

Embeddings

Embeddings are the result of a fine-tuning method called textual inversion. Like hypernetwork, textual inversion does not change the model. It simply defines new keywords to achieve certain styles.

Textual inversion and hypernetwork work on different parts of a Stable Diffusion model. Textual inversion creates new embeddings in the text encoder. Hypernetwork inserts a small network into the cross-attention module of the noise predictor.

In my experience, embedding is slightly more powerful than hypernetworks.

Where to find hypernetworks

The best place is civitai.com. Filter the models type with Hypernetwork.

How to use hypernetworks

I will show you how to use hypernetworks in AUTOMATIC1111 Stable Diffusion GUI. You can use this GUI on WindowsMac, or Google Colab.

Step 1: Install a hypernetwork model

To install hypernetwork models in AUTOMATIC1111 webui, put the model files in the following folder.

stable-diffusion-webui/models/hypernetworks Step 2: Use a hypernetwork model

To use a hypernetwork, put the following phrase in the prompt.

hypernet:filename:multiplier filename is the file name of the hypernetwork, excluding the extension (.pt.bin, etc).

multiplier is the weight applied to the hypernetwork model. The default is 1. Setting it to 0 disables the model.

How can you be sure the filename is correct? Instead of writing this phrase, you should click on the model button under the big “Generate” button.

Click on the Hypernetworks tab. You should see a list of hypernetworks installed. Click on the one you want to use.

The hypernet phrase will be inserted in the prompt.

Be aware that the hypernet phrase is not treated as part of the prompt. It merely directs which hypernetworks to use. It will be removed after the hypernetwork is applied. So you cannot use any prompt syntax like [keyword1:keyword2:0.5] with them.

Step 3: Testing and creating art with the model

To give yourself the greatest chance of success in unlocking the intended style, start by using it with the model it was trained with. But don’t stop there. Some hypernetworks require specific prompts or only work with certain subjects, so be sure to check out the prompt examples on the model page to see what works best.

And here’s a pro tip: if you notice your image is looking a bit too saturated, it may be a sign that you need to adjust the multiplier. It’s an easy fix. Stable Diffusion can sometimes interpret color saturation as the perfect way to hit the mark, but reducing the multiplier can help bring things back into balance.

Once you’ve confirmed that your hypernetwork is working its magic, why not experiment with using it on other models? You never know what interesting and unexpected effects might come up, and let’s be real, it’s just plain fun to play around with. So go ahead, let your creativity run wild.

Some hypernetworks I use

Here are my biased selection of hypernetworks.

Water Elemental

Water Elemental is a unique hypernetwork that can turn anything into water! Use the phrase “water elemental” before the subject. Make sure to describe the background. You can use this hypernetwork with the Stable Diffusion v1.5. Change the hypernetwork weight to adjust the water effect.

Water Elemental Hypernetwork Model Page

Prompt:

water elemental woman walking across a busy street hypernet:waterElemental_10:0.7

Prompt:

water elemental a boy running on water hypernet:waterElemental_10:1

InCase Style

InCase Style is used with the Anything v3 model. It modifies the Anything v3 model to produce a more mature anime style.

InCase Hybernetwork Model Page

Anything v3 Model Page

Prompt:

detailed face, a beautiful woman, explorer in forest, white top, short brown pants, hat, sky background, realism, small breast hypernet:incaseStyle_incaseAnythingV3:1 Negative prompt: moon, ugly, tiling, poorly drawn hands, poorly drawn feet, poorly drawn face, out of frame, extra limbs, disfigured, deformed, body out of frame, bad anatomy, watermark, signature, cut off, low contrast, underexposed, overexposed, bad art, beginner, amateur, distorted face, blurry, draft, grainy, large breast

Gothic RPG Artstyle

Gothic RPG Artstyle produces a stylish monochronic illustration style. Use with Protogen model.

Gothic RPG Artstyle Hypernetwork Model Page

Protogen v2.2 Model Page

Prompt:

drawing male leather jacket cyberpunk 2077 on a city street by WoD1 hypernet:gothicRPGArtstyle_v1:1

Reprinted from View Original
Collection Navigation
Collection
# 1 [SD Advanced Tutoria]How to fix parameters?
# 2 [SD Advanced Tutorial]How to automatically fix faces and hands
# 3 [SD Advanced Tutorial]How to run SDXL models
# 4 [SD Advanced Tutorial]Speed up Stable Diffusion
# 5 [SD Advanced Tutorial]3 ways to control lighting in Stable Diffusion
# 6 [SD Advanced Tutorial]Video to video with Stable Diffusion
# 7 [SD Advanced Tutorial]3 methods to upscale images in SD
# 8 [SD Advanced Tutorial]Control image composition in SD
# 9 [SD Advanced Tutorial]How to generate animals in SD
# 10 [SD Advanced Tutorial]How to make a video with SD
# 11 [SD Advanced Tutorial]How to generate realistic people in SD
# 12 [SD Advanced Tutorial]Stable Diffusion SDXL Beta Model
# 13 [SD Advanced Tutorial]SD Samplers: A Comprehensive Guide
# 14 [SD Advanced Tutorial]Midjourney vs SD: Which one should you pick?
# 15 [SD Advanced Tutorial]What is hypernetwork
# 16 [SD Advanced Tutorial]AUTOMATIC1111: A Beginner’s Guide
# 17 [SD Advanced Tutoria]What are LoRA models and how to use them in AUTOMATIC1111
# 18 [SD Advanced Tutoria]ControlNet v1.1: A complete guide
# 19 [SD Advanced Tutoria]How to remove extra limbs with Stable Diffusion inpainting
# 20 [SD Advanced Tutoria]Stable Diffusion prompt: a definitive guide
# 21 [SD Advanced Tutoria]Instruct Pix2Pix: Edit and stylize photos with text
# 22 [SD Advanced Tutoria]How to use outpainting to extend images
# 23 [SD Advanced Tutoria]ChatGPT: How to generate prompts for Stable Diffusion
# 24 [SD Advanced Tutoria]How to run Stable Diffusion on Google Colab (AUTOMATIC1111)
# 25 [SD Advanced Tutoria]Stable Diffusion Installation Tutorial (Mac M1/M2)
# 26 [SD Advanced Tutoria]How to use negative prompts?
# 27 [SD Advanced Tutoria]How does negative prompt work?
# 28 [SD Advanced Tutoria]How does Stable Diffusion work?
# 29 [SD Advanced Tutoria]Stable Diffusion Workflow
# 30 [SD Advanced Tutoria]Depth-to-image in SD 2
# 31 [SD Advanced Tutoria]How to install SD on Windows
# 32 [SD Advanced Tutoria]How to use embeddings in Stable Diffusion
# 33 [SD Advanced Tutoria]How to install Stable Diffusion 2.1 in AUTOMATIC1111 GUI
# 34 [SD Advanced Tutoria]How to cartoonize photo with Stable Diffusion
# 35 [SD Advanced Tutoria]How to use Dreambooth to put anything in Stable Diffusion (Colab notebook)
# 36 [SD Advanced Tutoria]How to use VAE to improve eyes and faces
# 37 [SD Advanced Tutoria]Turn amateur into professional with img2img
# 38 [SD Advanced Tutoria]How to use AI image upscaler to improve details
# 39 [SD Advanced Tutoria]How to run Stable Diffusion 2.0 and a first look
# 40 [SD Advanced Tutoria]A Beginner's Guide to SD Models
# 41 [SD Advanced Tutoria]How to stylize images using Stable Diffusion AI
# 42 [SD Advanced Tutoria]Basic information of Stable Diffusion
# 43 [SD Advanced Tutoria]Top Free Stable Diffusion AI Image Generator Sites
# 44 [SD Advanced Tutoria]Image AI generates fashion ideas
# 45 [SD Advanced Tutoria]Make an Animated GIF Using SD
# 46 [SD Advanced Tutoria]Common Problems in AI
# 47 [SD Advanced Tutoria]Change prompt word parameters
# 48 [SD Advanced Tutoria]AI removes unwanted objects
# 49 [SD Advanced Tutoria]Fine-tune AI images with tips
# 50 [SD Advanced Tutoria]What is Stable Diffusion?
 

Comments

no dataCoffee time! Feel free to comment