ControlNet big update: just rely on the prompt words to be able to accurately P-map, keep the style of painting unchanged, netizens: the effect is comparable to a customized large model

Hayo News

May 15th, 2023

View OriginalTranslated by Google

Stable Diffusion plug-in, "AI painting detail control master" ControlNet ushered in a major update:

Just use the text prompts to modify the details of the image arbitrarily while maintaining the main characteristics of the image .

For example, change the look of the beauty from hair to clothes , and make the expression more friendly:

Or let the model switch from the sweet girl next door to the high-cold Yujie , the orientation of the body and head, and the background are all changed:

——No matter how the details are modified, the "soul" of the original picture is still there.

In addition to this style, it can also handle the animation type just right:

AI design blogger @sundyme from Twitter said:

The effect is better than expected! Only one reference picture is needed to complete the above transformation, and some pictures can almost achieve the effect of customizing large models .

Ahem, friends in the AI painting circle, cheer up, it’s fun again.

(ps. The first three renderings are from YouTube blogger @Olivio Sarikas, and the second one is from Twitter blogger @sundyme)

New on ControlNet: Image retouching function that preserves the original image style

The above update actually refers to a preprocessor called "reference-only" .

It does not require any control model, and directly uses the reference image to guide the diffusion.

According to the author, this function is actually similar to the "inpaint" function, but it will not cause the image to collapse.

(Inpaint is a partial redrawing function in the Stable Diffusion web UI, which can redraw unsatisfactory, that is, manually masked places.)

Some experienced players may know a trick, which is to use inpaint for image diffusion.

Say you have a 512x512 image of a dog, and want to generate another 512x512 image of the same dog.

At this point you can concatenate the 512x512 dog image and the 512x512 blank image into a 1024x512 image, then use the inpaint function to mask out the blank 512x512 part and diffuse the dog image with a similar appearance.

In this process, because the images are simply and roughly stitched together, and there will be distortion, the effect is generally not satisfactory.

With "reference-only" it's different:

It can directly link the attention layer of SD ("Stable Diffusion") to any independent image, so that SD can directly read these images as a reference.

That is to say, if you want to make changes while maintaining the style of the original image, you can use the prompt words to operate directly on the original image.

As shown in the official example picture, change a standing puppy into a running action:

You just need to upgrade your ControlNet to version 1.1.153 or above, then select “reference-only” as the preprocessor, upload the picture of the dog, and input the prompt word “a dog running on grassland, best quality……”, SD will be I will only use this picture of yours as a reference for revisions.

Netizens: ControlNet's best feature yet

As soon as the "reference-only" function came out, many netizens started to experience it.

Some call this one of ControlNet's best features yet:

Pass an anime picture with a character pose, and then write a hint that seems to have nothing to do with the original picture. Suddenly, the effect you want is based on the original image. Really strong, even to the point of changing the rules of the game.

Others claim:

It's time to pick up all the waste pictures that were discarded before and restore them.

Of course, there are some who think it is not so perfect (for example, the earrings of the beautiful woman in the first rendering are wrong, and the hair in the second picture is also incomplete), but netizens still said that "the direction is right after all".