[SD Advanced Tutorial]How to generate animals in SD

There are not many tutorials for generating animals. In this article, we will go through some techniques to generate

• Realistic animals
• Cute animal images
• Animal vector arts
• Fantasy animals with human bodies
• Controlling composition with ControlNet

Software

We will use AUTOMATIC1111 Stable Diffusion GUI to generate animal images. You can use this GUI on Windows, Mac, or Google Colab.

Realistic wildlife animals

Since the goal is to generate realistic photographic images, you will need to include the keyword “photo”.

The prompt should start like

photo of … Subject

First, you will need to pick your subject(s). For example:

• Lion
• Pack of wolf
• Red panda
• Peacock
• teacup kitty
• etc…

Scene

The scene controls the background and surroundings. Because of the association effect, if you don’t add scene keywords, you will usually get the natural habitats of the wild animals.

• snow
• river
• tree
• forest
• grassland, grass field
• on a couch

Lighting

Lighting has a large effect on how the images look. Good lighting makes an image interesting.

• dark studio
• rim lighting
• sunset
• dramatic lighting

Others

Use realistic keywords similar to those for generating realistic people. For example:

• dslr
• ultra quality
• film grain
• 8K UHD

In my experience, more of these keywords are not always better. Using a few of them would already do the trick. Using too many may result in poor anatomy. I suspect many of these keywords are associated with human photos and could impair animal photos.

The following phrases can enhance the aesthetic of the wildlife images

• National Geographic Wildlife photo of the year
• The American Landscape Contest
• Wildlife photography contest

You can find more keywords in our prompt generator.

Models

You should use a model with a realistic style. For example:

• Realistic Vision
• Dreamlike Photoreal

Examples of realistic animal images

Here are some example prompts for generating realistic images. Feel free to use or remix.

Model: Realistic Vision v2.0

Prompt:

National Geographic Wildlife photo of the year, elephant trunk pointing up in new york city, night, dark studio, depth of field, trunk pointing up Negative prompt:

deformed, disfigured, underexposed, overexposed

Model: Realistic Vision v2.0

Prompt:

National Geographic Wildlife photo of the year, red panda, evening light, sunset, rim lighting Negative prompt:

deformed, disfigured

Model: Realistic Vision v2.0

Prompt:

Photo of (Lion:1.2) on a couch, flower in vase, dof, film grain, Fujifilm XT3, crystal clear, 8K UHD, dark studio

Model: Realistic Vision v2.0

Prompt:

National Geographic Wildlife photo of the year, siberian cat on river, evening light, sunset, rim lighting, depth of field Negative prompt:

deformed, disfigured

Model: Realistic Vision v2.0

Prompt:

National Geographic Wildlife photo of the year, peacock flying , evening light, sunset, rim lighting, depth of field Negative Prompt:

deformed, disfigured, underexposed, overexposed

Cute animals

Using Models

If you simply want to generate some cute animal pictures, a very simple prompt that includes the word “cute” will do the job. Pick a model to achieve a certain style.

Model: DreamShaper

Prompt:

A cute kitten

Modifying style

You can also add keywords to modify the style further with the same model.

Model: DreamShaper

Prompt:

a cute kitty, (extremely detailed CG unity 8k wallpaper), professional majestic impressionism oil painting Negative prompt: cartoon, 3d, disfigured, deformed easynegative

Chinese Zodiac LoRA

The Chinese Zodiac LoRA generates cute animals in a cartoon style. Use the LoRA with the sunshinemix_sunlightmixPruned model.

You can modify the prompt below to generate other animals. The suggested animals of this model are pig, bear, chook, monkey, sheep, horse, snake, dragon, bunny, tiger, cow, and rat.

Prompt:

pig, Exquisite City, (sky:1.3), (Miniature tree:1.3),Miniature object, many flowers, glowing mushrooms, (creek:1.3), lots of fruits, cute colorful animal protagonist, Firefly,meteor, Colorful cloud,Complicated background, rainbow, studio lighting, auora, rim light <lora:Chinese zodiac:1> Negative prompt:

Void background,black background

Animal vector art

You can generate animals in different vector art styles.

Anime style

The example below uses an anime model with a simple prompt. Many anime models are fine-tuned with people, especially girls. So use the keywords people and girl in the negative prompt to get only the animal.

Model: MeinaMix

Prompt:

vector art of a horse, white background Negative prompt：

bad art, amateur, girl, people, riding

Animal Stickers

This technique generates vector art by drawing a sticker on a white background. It does not always work, but you should get some images that can be easily cut out using Photoshop’s Select Subject function.

Model: Stable Diffusion v2.1 (768)

Prompt:

vector art of a tiger illustration stickers, ((vivid colors, colorful, pastel cute colors)), white background low poly, tetric, mosaic, disfigured, kitsch, ugly, oversaturated, grain, low-res, Deformed, blurry, bad anatomy, disfigured, poorly drawn face, mutation, mutated, extra limb, ugly, poorly drawn hands, missing limb, blurry, floating limbs, disconnected limbs, malformed hands, blur, out of focus, long neck, long body, ugly, disgusting, poorly drawn, mutilated, mangled, old, surreal, pixel-art, black and white, childish

Animals with human clothing

You can generate animals with human bodies… it works with the Realism Engine v1. (Note that this is a v2 fine-tuned model, you will need to download the accompanying config file to use it in AUTOMATIC1111.)

Prompt:

a goat wearing a suit, dark studio Negative prompt:

3d render, cgi, painting, drawing, cartoon, anime

Controlling poses

You can control the composition to some extent using ControlNet. You can even transfer human poses, although the animals can look strange because animal and human bodies are so different.

OpenPose

An exception is a close-up of the face. You will need to use a reference image that is a close-up human face. For example, the one below.

ControlNet: OpenPose

Model: Realistic Vision v2

Prompt:

National Geographic Wildlife photo of the year, a siberian cat, evening light, sunset, rim lighting Negative prompt:

deformed, disfigured Here are some close-up images of animals generated.

Canny Edge

You can transfer the composition of a wildlife photo to your image using Canny Edge.

ControlNet Setting:

• Preprocessor: Canny
• Model: Canny-fp16
• Control Weight: 0.65
• Starting control step: 0
• Ending control Step: 0.5

Prompt:

National Geographic Wildlife photo of the year, a deer, evening light, sunset, rim lighting Negative prompt:

deformed, disfigured, woman, man, people Reference image for ControlNet:

Generated images (various animals):

Final notes

I hope you are now familiar with some techniques that you can use to generate the animal images you want with Stable Diffusion.

It is pretty normal to get an imperfect image using any of these techniques. All you need to do is to fix some spots here or there with inpainting. So don’t give up a good image with minor defects!