Remember our previous article on using GAN, Clip, DALL-E to generate Pokémon, now it's time to use Stable Diffusion
In this article, I'll show how to take Pokedex descriptions from Pokedex entries from different games in the Pokémon series, and use Stable Diffusion to generate images from those defiances, so you can see how the AI interprets those descriptions. In this article, I only generated the first 150 Pokémon, if you need others, you can try it yourself.
Step 1: Get a Pokedex entry
The first thing is to get the description of Pokedex. These Pokedex descriptions will serve as text prompts for generated images. My initial thought was to write a webscraper for pokemon.com, where Pokedex Number (NPN) performs search queries. It's not that hard to do, but there's a really nice little Python library called PyPokedex that I recommend checking out if you're interested in programming and Pokémon!
PyPokedex uses PokeAPI to obtain Pokémon data, so you can use PyPokedex to obtain various information, such as NPN, name, height, weight, type, basic statistics, etc. In addition, PyPokedex also has a method called "get_description", which returns the Pokedex description of each different game in the form of a dictionary.
A basic PyPokedex query looks like this:
pokemon=pypokedex.get(dex=poke_id)
This query will return an instantiated object that will contain all the information for a given Pokémon based on the NPN value.
poke_name=pokemon.name
The next step is to get the Pokedex description with the get_descriptions method. Here we use the description in the Pokedex of "Pokemon Yellow", because everyone in this generation should know
yellow_description=pokemon.get_descriptions()[ver]
These two results are then combined so that the hint to the AI looks like this:
prompt = poke_name + " " + yellow_description
Here we use the Pokémon's name as part of the hint, as there are some Pokedex entries that are a bit vague and don't always yield good results. So the Pokémon's name, being able to create "true variants" that are more like the original. You can remove it if you don't like it, but I'm happy with the result.
Step Two: Set Up Stable AI
We already have the text, let's prepare the Stable Diffusion model. Install the necessary packages with the following line of code:
%pipinstall—quiet—upgradediffuserstransformersacceleratemediapyscipyftfyspacy
The xformers package also needs to be included to help us create images. This needs to train the corresponding wheels package according to your python version, so be sure not to make a mistake
github_url="https://github.com/brian6091/xformers-wheels"
xformer_id="0.0.15.dev0+4c06c79"
xformers_wheels=f"xformers-{xformer_id}.d20221205-cp38-cp38-linux_x86_64.whl
%pipinstall-q {github_url}/releases/download/{xformer_id}/{xformers_wheels}
Next, specify the model you want to use. Here choose dreamlike-photoreal-2.0.
model_id=“dreamlike-art/dreamlike-photoreal-2.0”
We use a simple and fast method here: directly use StableDiffusionPipeline to generate pictures
importmediapyasmedia
importtorch
fromdiffusersimportStableDiffusionPipeline
device="cuda"
ifmodel_id.startswith("stabilityai/"):
model_revision="fp16"
else:
model_revision=None
ifschedulerisNone:
pipe=StableDiffusionPipeline.from_pretrained(
model_id,
torch_dtype=torch.float16,
revision=model_revision,
)
else:
pipe=StableDiffusionPipeline.from_pretrained(
model_id,
scheduler=scheduler,
torch_dtype=torch.float16,
revision=model_revision,
)
pipe=pipe.to(device)
pipe.enable_xformers_memory_efficient_attention()
ifmodel_id.endswith('-base'):
image_length=512
else:
image_length=768
Now that all the steps are ready, the official generation begins:
remove_safety=False
num_images=4
ifremove_safety:
negative_prompt=None
else:
negative_prompt="nude, naked"
images=pipe(
prompt,
height=image_length,
width=image_length,
num_inference_steps=25,
guidance_scale=9,
num_images_per_prompt=num_images,
negative_prompt=negative_prompt,
).images
media.show_images(images)
Step 3: Integrate these codes into functions
The above code can already generate a single image based on a single query. But we need to process 150 of them. So I integrated them into a function and called it in a loop.
defmakePokemonFromPokedex(ver,nPokemon):
#Loop over nPokemons to get the descritptions and generate images for each
#poke_id = 1
forpoke_idinrange(1, nPokemon+1, 1):
#print(poke_id)
#Specify which Pokemon we want to query using its ID number
pokemon=pypokedex.get(dex=poke_id)
#print(pokemon)
#This is the name of the Pokemon we are querying
poke_name=pokemon.name
#This is the PokeDex desciption for the current Pokemon
yellow_description=pokemon.get_descriptions()[ver]
#This is the prompt I'll feed to the AI
prompt=poke_name+" "+yellow_description
#print(prompt)
remove_safety=False
num_images=4
ifremove_safety:
negative_prompt=None
else:
negative_prompt="nude, naked"
images=pipe(
prompt,
height=image_length,
width=image_length,
num_inference_steps=25,
guidance_scale=9,
num_images_per_prompt=num_images,
negative_prompt=negative_prompt,
).images
fname='poke_'+str(poke_id)
get_concat_h_multi_blank(images).save(fname+'.jpg')
Look at the results below
Result display
Iron Chrysalis
Metapod — The prompt for this is “It is waiting for the moment to evolve. At this stage, it can only harden, so it remains motionless to avoid attack”.
Unicorn
Weedle —T he prompt for this is “Beware of the sharp stinger on its head. It hides in grass and bushes where it eats leaves”.
Big Needle Bee
Beedrill— The prompt for this is “It has three poisonous stingers on its forelegs and its tail. They are used to jab its enemy repeatedly”.
Arbor snake
Ekans— The prompt for this is “The older it gets, the longer it grows. At night, it wraps its long body around tree branches to rest.”.
Mountain rat
Sandshrew— The prompt for this is “It loves to bathe in the grit of dry, sandy areas. By sand bathing, the Pokémon rids itself of dirt and moisture clinging to its body.”. Image by author.
Chuanshanwang
Sandslash— The prompt for this is “The drier the area Sandslash lives in, the harder and smoother the Pokémon’s spikes will feel when touched.”. Image by author.
Nitorino
Nidorino— The prompt for this is “With a horn that’s harder than diamond, this Pokémon goes around shattering boulders as it searches for a moon stone.”. Image by author.
nine tails
Ninetales— The prompt for this is “It is said to live 1,000 years, and each of its tails is loaded with supernatural powers”. Image by author.
walking grass
Oddish— The prompt for this is “If exposed to moonlight, it starts to move. It roams far and wide at night to scatter its seeds.”. Image by author.
Gophers
Diglett— The prompt for this is “If a Diglett digs through a field, it leaves the soil perfectly tilled and ideal for planting crops.”. Image by author.
mosquito coil tadpole
Poliwag— The prompt for this is “For Poliwag, swimming is easier than walking. The swirl pattern on its belly is actually part of the Pokémon’s innards showing through the skin.”. Image by author.
Stinger jellyfish
Tentacruel— The prompt for this is “When the red orbs on Tentacruel’s head glow brightly, watch out. The Pokémon is about to fire off a burst of ultrasonic waves.”. Image by author.
https://avoid.overfit.cn/post/7f072f5e107145d0b6502a956d4e3ede
By Victor Murcia