I’ve been trying to do txt2img generative AI on my Windows gaming PC that has an RTX 3090.
Setup
There are two well-know UI options for running Stable Diffusion locally:
A1111 is arguably the de-facto standard but requires installing a system Python to get working.
ComfyUI has a truly portable self-contained installer, so I went with that. In the end, I actually like ComfyUI more because it truly makes you think about the distinct tools you are using.
To install you just download a the zipped package from Github then run the correct bat file after downloading a checkpoint model and putting that into ComfyUI\models\checkpoints.
For a general purpose checkpoint you can start with Stable Diffusion 2.1 or SDXL Turbo. You’ll want the SafeTensors files. I’ve downloaded my models from Huggingface but apparently CivitAI is also a good source. That requires an account to download though.
Generating Images
ComfyUI should open up with a simple pipeline you can use. It is preconfigured with a positive and negative prompt. ComfyUI has a lot of good examples of simple workflows. There is also a community managed manual.
In general, your workflow will follow one of two paths.
Add new models or extensions of existing models
There’s a handful of different ways to extend existing checkpoints.
- Textual Inversions - These are extra embeddings for your checkpoint.
- Hypernetworks - A new set of weights for the checkpoint that were trained using another model (from neural networks)
- LoRA - Fine tuning part of the model using a technique from Microsoft (cheaper than finetuning the whole thing)
You can download these from Huggingface and adds cards for them into your workflow.
More information about what these are can be found here.
Add ComfyUI plugins
Sometimes you want to include a tool that isn’t supported out of the box by ComfyUI. For example, InstantID. You can install plugins to add that functionality (e.g. ComfyUI_InstantID).
Here’s a partial list of known plugins . ComfyUI-Manager will allow management and installation of these plugins. ComfyUI examples also have some plugin links.
Prompting Gotchas
ComfyUI uses a different formula for weighting in prompts than A1111.
A1111 goes into detail on their implementation here. At a high level though:
()and[]used to add or decrease 1.1x emphasis on a given part of a prompt{}and[]used to add or decrease 1.05x emphasis in NovelAI- Now only
()is recommended and used with a specific weight like(word:0.25) - All weights are normalized before processing.
ComfyUI uses the same () and specific weight style that A1111 now recommends. However, the weights are not normalized, as covered in this FAQ.
Stuff I Still Don’t Understand
- Inpainting
- ControlNet