Running a Startup on AWS? Get Funding With AWS JumpStart. Click Here to Learn More

2021 Fillmore Street #1128

}

24/7 solutions

A grid of images using Stable-Diffusion and Dreambooth

Using Stable Diffusion to create personalized AI art – Part 1 of 3

Using Stable Diffusion to create personalized AI art is possible; However, it takes a little bit of work. Stable Diffusion is an open-source text-to-image AI model that can generate amazing images from given texts in seconds. This first article will help you get the default Stable Diffusion installed on your local machine and check that it can run.

Introduction

Stable Diffusion is an open-source text-to-image AI model that can generate amazing images from given texts in seconds. However, getting this program to run can take a bit of work. This will be part of a series of 3 articles from installation to personalized prompts to the future that AI image generation implies.

1. Installing Stable-Diffusion WebUI locally

Getting Stable Diffusion running locally on your machine so you have complete control over all the elements, also access to a wealth of extensions and options.

2. Training Personalized Models with Dreambooth

You may need to rent a server for a GPU powerful enough to train your personalized model ($0.50 per hour), although if you have a powerful machine locally you might avoid this.

3. The Future Applications and Pitfalls

This third part of the series will chart the potential future of the text-to-image AI, the copyright issues, and the backlash that this technology is already facing.

“Boston university in the 1920s, ultra-realistic, colour, concept art, intricate details, night, thunder, raining, eerie, Arkham Horror, Call of Cthulhu, elder sign, highly detailed, dark fantasy, photorealistic, octane render, 8k, unreal engine 5. art by artgerm and Greg Rutkowski and Alphonse Mucha”

Limitations

While Stable-Diffusion will be running locally on your machine, we will be using a paid-for server online to run Dreambooth. The reason for this is the need for a very high-power GPU to run the training model, This algorithm requires a GPU with a minimum of 24GB of VRAM. While the EVGA GeForce RTX 3090 FTW3 is a great option, if you can get your hands on one, the price for a new one at time of writing is in excess of $1,500 USD.

Secondly, a lot of this tutorial has been tested mostly on windows with an NVIDIA-based GPU, although there are links to pages that explain the process for Macs/Linux and AMD-based GPU machines.

After this first how to is complete you’ll be able to generate landscapes, scenes, famous people and iconic scenes etc on your local machine, if you want to generate images with your face, you’ll probably need to run a remote $/h server to do so. The cost is very low and you can generate many training models for less than $5, more on that later.

“((path in the forest)), leading to a dark cave!!! entrance, exquisite masterpiece painting by Vermeer, trending on artstation”

( ( path in the forest ) ), leading to a dark cave!!! entrance, exquisite masterpiece painting by vermeer, trending on artstation

Local System Requirements

First, install all the various programs and programming systems we will need to run Stable-Diffusion locally:

Create a folder on your local machine where you want to store the models and generate your images (make sure the disk drive has enough space to do all this!

  • I will use (D:\AI) but the name and location are arbitrary, I merely have a very full C:\ drive so I used my secondary one.

Files to download

Install Stable Diffusion WebUI

Stable diffusion WebUI is not 100% necessary; you can run everything from the command line; however, the local web version running on your browser makes the process 1000 times easier.

What does this tutorial cover?

There are lots of forks (versions) of the Stable Diffusion WebUI, many tutorials work with the NVIDIA GPU and CUDA, and many Macs don’t have Nvidia, but don’t despair we’ve tried to create a tutorial to help you install it on Mac or Windows.

If you have troubles with your particular system or installation please visit the Stable Diffusion Reddit subreddit and see if someone has created a Stable Diffusion system that meets your local machine requirements.

One-click installer for Windows and a macOS alternative

While this article is based on a more detailed installation process of the Stable Diffusion WebUI project the community is rapidly evolving and an easier installation method is one of those things desired by most.

There is  A1111’s Stable Diffusion WebUI Easy Installer, which automates most of the download/install steps below. If it works for you, great. If not, the manual process is not so bad.

For MacOS users, there’s also an easy install option and app called DiffusionBee which works great with Apple processors (works a tad slower with Intel chips).

Installing on a machine with an Nvidia GPU

Search and launch “Anaconda Prompt” from the Start Menu.

  • Go into your new folder [cd AI] (D:\AI).
    • If you’re trying to change drive in the anaconda prompt, just type the drive name “D:\
    • If you’re trying to change directory use the cd command
      cd D:\AI
  • Clone Stable-Diffusion-webui
  • Download “Stable Diffusion Model Weights 1.4”
    • You will need to create a free Hugging Face account sd-v1-4.ckpt
    • Rename the downloaded file from “sd-v1-4.ckpt” to “model.ckpt”.
  • Move the newly renamed “model.ckpt” file to the Stable Diffusion “Models” folder (D:\AI\stable-diffusion-webui\models).
  • Download the GFPGAN File.
    • GFPGAN Face Correction 1.3
    • Move the “GFPGANv1.3.pth” file to the root Stable Diffusion WebUI folder (D:\AI\stable-diffusion-webui).

Installing with an AMD graphics GPU (Mac)

You’ll need to run these extra steps if you’re using an AMD GPU which is very common on Mac devices. Based on the following article https://github.com/AUTOMATIC1111/stable-diffusion-webui/wiki/Install-and-Run-on-AMD-GPUs

Launch “Anaconda Prompt” from the Start Menu.

  • Go into your new folder [cd AI] (D:\AI).
    • If you’re trying to change drive in anaconda prompt, just type the drive name “D:\”
    • If you’re trying to change directory, use the cd command
      cd D:\AI
  • Clone Stable-Diffusion-webui
  • Download “Stable Diffusion Model Weights 1.4”
    • You will need to create a free Hugging Face account
      sd-v1-4.ckpt
    • Rename the downloaded file from “sd-v1-4.ckpt” to “model.ckpt”.
    • Move the newly renamed “model.ckpt” file to the Stable Diffusion “Models” folder (D:\AI\stable-diffusion-webui\models).
  • Download the GFPGAN File.
    GFPGAN Face Correction 1.3
    • Move the “GFPGANv1.3.pth” file to the root Stable Diffusion WebUI folder (D:\AI\stable-diffusion-webui).
    • In the root folder (D:\AI\stable-diffusion-webui) do the following terminal commands

For the Mac

We needed to do the following from the command line.

python -m venv venv
source venv/bin/activate
python -m pip install --upgrade pip wheel

# It's possible that you don't need "--precision full"
TORCH_COMMAND='pip install torch torchvision --extra-index-url https://download.pytorch.org/whl/rocm5.1.1' 
python launch.py --precision full --no-half --skip-torch-cuda-test

Issues and problems

There are an abundance of systems, graphics cards, and issues that exist in installing something this complex so again If you have troubles with your particular system or installation please visit the Stable Diffusion Reddit subreddit and see if someone has created a Stable Diffusion system that meets your local machine requirements. Or linked to a ‘how to’ on the steps required.

As stated before if you’re not using an NVIDIA GPU or you are on a Mac, it can be more challenging (but it is possible).

Step 2 - Run Webui

If you’re on windows

Run/Double-click on the “webui-user.bat” inside the “stable-diffusion-webui” folder (this will take quite a while the first time you run the .bat).

If you’re on a Mac

Run the “./webui.sh” inside the “stable-diffusion-webui” folder (this will take quite a while the first time you run the file) if you don’t want to run from the terminal every time do the following.

Copy the webui.sh and edit the copy to say webui.command
Enter the terminal and navigate to the root of stable diffusion
D:\AI\stable-diffusion-webui
Run the command chmod +x webui.command

You can now open WebUI by clicking on the command and not use the terminal.

Open a web browser and go to: http://127.0.0.1:7860

Step 3 - Play

Now, you should have Stable Diffusion running on your machine! On the Txt2Image prompt, you can type a series of suggestions and you’ll generate a single response. We found these descriptions on the website www.lexica.art 

There are a few settings you’ll need to know, but we’ll cover those in future. The Stable-Diffusion WebUI covers over eight tabs, extensions, training and various features.

The prompt

This is where you type the description you want to create, there is a negative

Negative prompt

These are things which you don’t want to see in your generated image. We found a list for reducing some of the worst issues.

 3d, ugly face, (disfigured), (bad art), (deformed), (poorly drawn), (extra limbs), strange colours, blurry, boring, sketch, lacklustre, repetitive, cropped, hands

Other settings

The Batch Count and Batch Size  are ones to watch and say the number of batches and images you wish to produce, on a low GPU computer, we found that only a Batch size of 2 worked, while on a gaming machine, 8 images take a few minutes.

Local Machine - Prompt 1

Baroque oil painting of key visual portrait concept art of anime maids entrenched in the great war, brutalist, dark fantasy, rule of thirds golden ratio, fake detail, trending pixiv fanbox, acrylic palette knife, style of Makoto Shinkai studio Ghibli Genshin impact Jamie Wyeth James Gilleard, Greg Rutkowski, Chiho Aoshima

( ( path in the forest ) ), leading to a dark cave!!! entrance, exquisite masterpiece painting by vermeer, trending on artstation

Local Machine - Prompt 2

“Gandalf, D&D, fantasy, intricate, elegant, highly detailed, digital painting, artstation, concept art, matte, sharp focus, illustration, hearthstone, art by artgerm and Greg Rutkowski and Alphonse Mucha

Finishing Points

This tutorial was about Stable Diffusion 1.4 and recently things have changed with the newer version of 2. We’ll cover more about the evolution of Stable Diffusion tech and AI art in the 3rd article.

 

Useful Websites

James
James

After a long history of software and web development (both front and back end). I left the financial programming world to try my hand at other things and other experiences.

Excellent English skills and presentation abilities from my teaching combined with a high level of programming knowledge which I have pursued consistently at the same time.

I am especially interested in how technology can be combined with education and collaboration in projects, businesses and learning.

Competent in: AWS, Java; JSTL, Hibernate, Lucene and Swing. PHP, Zend. Web development in HTML, CSS and CMS’ WordPress, Drupal, and Joomla!

Read more at http://jamesrtyrrell.com where I’m looking at the future of AI art and other topics of interest.

Related Articles

Here’s Why You Should Work with an AWS Partner

Here’s Why You Should Work with an AWS Partner

Amazon Web Services is understandably a difficult platform to adapt to and utilize fully upon first getting started. Some organizations can be selected to become certified partners to indirectly extend services to help build on the Amazon Cloud. Finding a certified company to help build out is undoubtedly the best way to significantly simplify, streamline, and reduce the cost of utilizing AWS.

Amazon Web Services – CodeCatalyst

Amazon Web Services – CodeCatalyst

When a development team is building out an application, it helps to have access to the same resources, have the tools for planning and testing, and to have access to the application all in one place. CodeCatalyst comes with a slew of continuous integration/continuous development (CI/CD) tools and can leverage other AWS services and be connected to other AWS projects on an account. As a collaborative tool, it is easy to introduce new members into the project and to log all activity or all tests from a single dashboard. It’s a complete package of all the tools needed to securely work on every step of an application’s lifecycle.

The Definitive Guide to AWS Pricing

The Definitive Guide to AWS Pricing

Perhaps the biggest issue with AWS that its competitors edge out on is the confusing pricing model. It does promise the capacity to help users save significantly on funds that otherwise by avoiding spending on unnecessary resources, but getting to that point isn’t always clear. We will be covering in greater detail how this works.

Download our 10-Step Cloud Migration ChecklistYou'll get direct access to our full-length guide on Google Docs. From here, you will be able to make a copy, download the content, and share it with your team.