sd-webui-decadetw-auto-prompt-llm

KO | EN

gitlite — search

#javascript #hacktoberfest #react #ai #golang #rust #docker #nodejs #java #awesome #awesome-list #vue #security #open-source

sd-webui-decadetw-auto-prompt-llm

(★ 71)

sd-webui-auto-prompt-llm

sd-webui-decadetw-auto-prompt-llm Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

### Quick Links

* Auto prompt by LLM and LLM-Vision (Trigger more details out inside model) 
 * SD-WEB-UI: https://github.com/xlinx/sd-webui-decadetw-auto-prompt-llm
 * ComfyUI: https://github.com/xlinx/ComfyUI-decadetw-auto-prompt-llm
* Auto msg to ur mobile (LINE | Telegram | Discord)
 * SD-WEB-UI :https://github.com/xlinx/sd-webui-decadetw-auto-messaging-realtime
 * ComfyUI: https://github.com/xlinx/ComfyUI-decadetw-auto-messaging-realtime
* I'm SD-VJ. (share SD-generating-process in realtime by gpu) 
 * SD-WEB-UI: https://github.com/xlinx/sd-webui-decadetw-spout-syphon-im-vj
 * ComfyUI: https://github.com/xlinx/ComfyUI-decadetw-spout-syphon-im-vj
* CivitAI Info|discuss:
 * https://civitai.com/articles/6988/extornode-using-llm-trigger-more-detail-that-u-never-thought
 * https://civitai.com/articles/6989/extornode-sd-image-auto-msg-to-u-mobile-realtime
 * https://civitai.com/articles/7090/share-sd-img-to-3rd-software-gpu-share-memory-realtime-spout-or-syphon
<hr/>

# SD-WEB-UI | ComfyUI | decadetw-Auto-Prompt-LLM-Vision

<hr/>

## 💡Update Log
  * [add|20240730] | 🟢 LLM Recursive Prompt
  * [add|20240730] | 🟢 Keep ur prompt ahead each request
  * [add|20240731] | 🟢 LLM Vision
  * [add|20240803] | 🟢 translateFunction 
    * When LLM answered, use LLM translate result to your favorite language. 
      * ex: Chinese. It's just for your reference, which won't affect SD.
  * [add|20240808] | 🟢 Before and After script | exe-command
  * [add|20240808] | 🟠 release LLM VRAM everytimes
  * [add|20250120] | 🟢 add chatBot for DeepSeek.R1 & openAI.DeepResearch model. Lets chat&train ur LLM first then use 2.3 keyword list get prompt

* ![LLM-Text-Chat.png](https://raw.githubusercontent.com/xlinx/sd-webui-decadetw-auto-prompt-llm/main/images/LLM-Text-Chat.png)

## 💡Motivation
* Call LLM       : auto prompt for batch generate images 
* Call LLM-Vision: auto prompt for batch generate images
* Image will get more details that u never though before.
* prompt detail is important

## 💡Usage

#### LLM-Text
* batch image generate with LLM
    * a story

* Using Recursive prompt say a story with image generate
* Using LLM
  * when generate forever mode
      * example as follows figure Red-box.
      * just tell LLM who, when or what
      * LLM will take care details.
  * when a story-board mode (You can generate serial image follow a story by LLM context.)
      * its like comic book
      * a superstar on stage
      * she is singing
      * people give her flower
      * a fashion men is walking.
#### LLM-Vision  👀
* batch image generate with LLM-Vision
  * let LLM-Vision see a magazine
  * see series of image 
  * see last-one-img for next-image
  * make a serious of image like comic

#### Before and After script
* support load script or exe-command Before-LLM and After-LLM
* javascript fetch POST method (install Yourself )
    * security issue, but u can consider as follows
    * https://github.com/pmcculler/sd-dynamic-javascript
    * https://github.com/ThereforeGames/unprompted
    * https://github.com/adieyal/sd-dynamic-prompts
    * https://en.wikipedia.org/wiki/Server-side_request_forgery
    * and Command Line Arg --allow-code

---

<table>
<tr>
 <td colspan="2">[🟢] stable-diffusion-webui-AUTOMATIC1111</td>
</tr>
<tr>
 <td colspan="2">[🟢] stable-diffusion-webui-forge</td>
</tr>
<tr>
 <td colspan="2">[🟢] ComfyUI</td>
</tr>
 <tr>
 <td colspan="2">1. SD-Prompt ✦ </td>
 
 </tr>
 <tr>
 <td colspan="2">1girl</td>
 </tr>
<tr>
 <td>2.1 LLM-Text ✦</td>
 <td>2.2 LLM-Vision ✦</td>
 </tr>
 <tr>
 <td>a super star on stage.</td>
 <td>Who is she in image?</td>
 </tr>
<tr>
 <td>2.3 LLM-Text-sys-prompt ✦</td>
 <td>2.4 LLM-Vision-sys-prompt ✦</td>
 </tr>
 <tr>
 <td>You are an AI prompt word engineer. Use the provided keywords to create a beautiful composition. Only the prompt words are needed, not your feelings. Customize the style, scene, decoration, etc., and be as detailed as possible without endings.</td>
 <td>You are an AI prompt word engineer. Use the provided image to create a beautiful composition. Only the prompt words are needed, not your feelings. Customize the style, scene, decoration, etc., and be as detailed as possible without endings.</td>
 </tr>
<tr>
 <td colspan="2">3. LLM will answer other detail ✦</td>
 </tr>
<tr >
 <td colspan="2"> The superstar, with their hair flowing in the wind, stands on the stage. The lights dance around them, creating a magical moment that fills everyone present with awe. Their eyes shine bright, as if they are ready to take on the world.</td>
 </tr>
<tr >
 <td colspan="2">The superstar stands tall in their sparkling costume, surrounded by fans who chant and cheer their name. The lights shine down on them, making their hair shine like silver. The crowd is electric, every muscle tense, waiting for the superstar to perform</td>
 </tr>
</table>
<table>
 <tr>
 <td colspan="2">
 4. Main Interface | sd-web-ui | ComfyUI 
 </td>
 </tr>
 <tr>
 <td><img src="images/rm_mainui_sdwebui.png"></td>
 <td><img src="images/rm_mainui_comfyui.png"></td>
 </tr>
 <tr>
 <td colspan="2"><img src="images/rm_use1.png"></img></td>
 </tr>
 <tr>
 <td colspan="2"><img src="images/rm_use2.png"></img></td>
 </tr>
 <tr>
 <td colspan="2"><img src="images/rm_use3.png"></img></td>
 </tr>
 <tr>
 <td colspan="2">
 ComfyUI Manager | search keyword: auto
 </td>
 </tr>
 <tr>
 <td colspan="2"><img src="images/comfyui_manageer_search_keyword_auto.png" ></td>
 </tr>

</table>

### Usage

<table style="border-width:0px" >
<tr>
 <td>
 Input
 </td>
 <td>
 Output
 </td>
 </tr>
 <tr>
 <td> 
 LLM-Text: a superstar on stage. 
 <hr/>
 LLM-Vision: What's pose in this image?. 
 <hr/>
 (okay, its cool.)
 </td>
 <td>
 <img width="500" src="images/u1.png" alt=""/>
 </td>
 </tr>
<tr>
 <td>
 LLM-Text: a superstar on stage. 
 <hr/>
 LLM-Vision: with a zebra image 
 <hr/>
 (okie, cooool show dress. At least we don't have half zebra half human.)
 </td>
 <td colspan="1"><img width="500" src="images/u2.png" alt=""></img></td>
 </tr>
 <tr>
 <td>LLM-Text: a superstar on stage. 
 <hr/>(okay, its cool.)</td>
 <td><img width="500" src="images/u3.png"></img></td>
 </tr>
 <tr>
 <td>LLM: a superstar on stage.<hr/> (Wow... the describe of light is great.)</td>
 <td><img width="500" src="images/u4.png"></img></td>
 </tr>
 <tr>
 <td>LLM: a superstar on stage.<hr/> (hnn... funny, it does make sense.)</td>
 <td><img width="500" src="images/u5.png"></img></td>
 </tr>
 <tr>
 <td>
 CHALLENGE<hr/> LLM-vision:A Snow White girl walk in forest.<hr/> (detect ur LLM-Vision Model IQ; if u didnt get white dress and lot of snow.... plz let me know model name)
 <hr>
 <li>SD model: 
 <a href="https://civitai.com/models/618692/flux?modelVersionId=691639">
 Flux.1 D
 </a>
 </li>
 <li>LLM model: 
 <a href="https://huggingface.co/FiditeNemini/Llama-3.1-Unhinged-Vision-8B-GGUF">
 llava-llama-3.1-8b
 </a>
 </li>
 <li>LLM model: 
 <a href="https://huggingface.co/Lewdiculous/Eris_PrimeV4-Vision-32k-7B-GGUF-IQ-Imatrix#quantization-information">
 Eris_PrimeV4-Vision-32k-7B-IQ3_XXS
 </a>
 </li>
 </td>
 <td>
 <img width="500" src="images/u6.png"/>
 <img width="500" src="images/u7.png"></img>
 </td>
 </tr>
<tr>
 <td>
 FLUX model<hr/> hnn...NSFW show. I'm not mean that, but not a wrong answer. <hr/>(Trigger more details; that u never thought about it.)
 <hr>
 <li>SD model: 
 <a href="https://civitai.com/models/618692/flux?modelVersionId=691639">
 Flux.1 D
 </a>
 </li>
 <li>LLM model: 
 <a href="https://huggingface.co/FiditeNemini/Llama-3.1-Unhinged-Vision-8B-GGUF">
 llava-llama-3.1-8b
 </a>
 </li>
 <li>LLM model: 
 <a href="https://huggingface.co/Lewdiculous/Eris_PrimeV4-Vision-32k-7B-GGUF-IQ-Imatrix#quantization-information">
 Eris_PrimeV4-Vision-32k-7B-IQ3_XXS
 </a>
 </li>
 </td>
 <td><img width="500" src="images/u8.png"></img></td>
 </tr>
<tr>
 <td>
 advanced use | before-after-action 
 <hr/> in fact, u can run any u want script | (storyboard) | random read line from txt send into LLM
 </td>
 <td><img width="500" src="images/readme_before_action.png"></img></td>
 </tr>
<tr>
 <td>
 Special LLM Loop 
 <hr/> Connect 1st LLM-Text output to 2nd LLM-Text Input
 </td>
 <td><img width="500" src="images/llm-text-loop.png"></img>
 <img width="500" src="images/webui-llm-loop-ui.png"></img>
 </td>
 </tr>
<tr>
 <td style="width: 40%">
 Special LLM Loop - keep each feature assign to different obj not mix it on one.
 <hr/> LLM-Text output ask looply : 
 <a href="https://github.com/xlinx/sd-webui-decadetw-auto-prompt-llm/discussions/14"> here</a>
 </td>
 <td>
 <img width="250" src="images/llm-loop-ex-5.png"/>
 <img width="250" src="images/llm-loop-ex-6.png"/>
 <img width="250" src="images/llm-loop-ex-1.png"/>
 <img width="250" src="images/llm-loop-ex-2.png"/>
 <img width="250" src="images/llm-loop-ex-3.png"/>
 <img width="250" src="images/llm-loop-ex-4.png"/>
 </td>
 </tr>
<tr>
 <td style="width: 40%">
 [new tool 20240915] Civitai Prompt Grabber
 <hr/> 
<ul>
<li>quick prompt from civitai. u can pick some prompt from another area model(ex indoor design or building model) with ur 1girl, ex: 1girl(up figure)) + in-door design model-prompt. then u will get full detail in background(bottom figure) : <a href="https://civitai.com/models/85691"> https://civitai.com/models/85691</a></li>
<li>this is good present in FLUX model. trigger more detail in background. Make the photo getting more realistic feeling</li>
<li>option1. just quick append prompt from other model from civitai or</li>
<li>option2. of course u can send it into LLM too.</li>
</ul>
</td>
 <td>
 <img width="350" src="images/ui_grabber.png"/>
 <img width="350" src="images/grabber3.png"/>
 </td>
 </tr>
<tr>
 <td style="width: 40%">
 [update] LLM-ask-LLM🌀
 <hr/> 
 [support] Cloud Service: Gemini Pro
 <hr/> 
<ul>
<li>cloud service: https://generativelanguage.googleapis.com/v1</li>
<li>model: gemini-1.5-flash (vision)</li>
<li>support text and vision</li>
<li>it will get more🌀 and more🌀 and more🌀 like....(bottom to top)</li>
</ul>
</td>
 <td>
 <img width="350" src="images/llm_ask_llm_.png"/>
 </td>
 </tr>

</table>

## Usage Tips

* tips1:
 * leave only 1 or fewer keyword(deep inside CLIP encode) for SD-Prompt, others just fitting into LLM
 * SD-Prompt: 1girl, [xxx,]<--(the keyword u use usually, u got usually image)
 * LLM-Prompt: xxx, yyy, zzz, <--(move it to here; trigger more detail that u never though.)
 * tips2:
 * leave only 1 or fewer keyword(deep inside CLIP encode) for SD-Prompt, others just fit into LLM
 * SD-Prompt: 1girl,
 * LLM-Prompt: a superstar on stage. <--(say a story)
 * tips3:
 * action script - Before
 * random/series pick prompt txt file random line fit into LLM-Text [[read_random_line.bat](https://github.com/xlinx/sd-webui-decadetw-auto-prompt-llm/blob/main/before-after-actions/read_random_line.bat)]
 * random/series pick image path file fit into LLM-Vision
 * action script - After
 * u can call what u want command
 * ex: release LLM VRAM each call: "curl http://localhost:11434/api/generate -d '{"model": "llama2", "keep_alive": 0}'" @Pdonor
 * ex: bra bra. Interactive anything.
 * tipsX: Enjoy it, inspire ur idea, and tell everybody how u use this.

## Installtion

* You need install LM Studio or ollama first.
    * [LM Studio](https://lmstudio.ai/): Start the LLM service on port 1234. (suggest use this one)
    * [ollama](https://ollama.com/): Start service on port 11434 .
* Pick one language model from under list
    * text base(small ~2G)
    * text&vision base(a little big ~8G)
* Start web-ui or ComfyUI install extensions or node
    * stable-diffusion-webui | stable-diffusion-webui-forge:  
      * go Extensions->Available [official] or Install from URL
        * https://github.com/xlinx/sd-webui-decadetw-auto-prompt-llm
    * ComfyUI: using Manager install node
      * Manager -> Customer Node Manager -> Search keyword: auto
      * https://github.com/ltdrdata/ComfyUI-Manager
      * https://registry.comfy.org/
      * https://ltdrdata.github.io/
* Open ur favorite UI
  * Lets inactive with LLM. go~
  * trigger more detail by LLM

## Suggestion software info list

* https://lmstudio.ai/ (win, mac, linux)
* https://ollama.com/ (win[beta], mac, linux)
* https://github.com/openai/openai-python
* https://github.com/LostRuins/koboldcpp (all os)

### Suggestion LLM Model

* LLM-text (normal, chat, assistant)
 * 4B VRAM<2G
 * CHE-72/Qwen1.5-4B-Chat-Q2_K-GGUF/qwen1.5-4b-chat-q2_k.gguf
 * https://huggingface.co/CHE-72/Qwen1.5-4B-Chat-Q2_K-GGUF
 * 7B VRAM<8G
 * ccpl17/Llama-3-Taiwan-8B-Instruct-GGUF/Llama-3-Taiwan-8B-Instruct.Q2_K.gguf
 * Lewdiculous/L3-8B-Stheno-v3.2-GGUF-IQ-Imatrix/L3-8B-Stheno-v3.2-IQ3_XXS-imat.gguf
 * Google-Gemma
 * https://huggingface.co/bartowski/gemma-2-9b-it-GGUF
 * bartowski/gemma-2-9b-it-GGUF/gemma-2-9b-it-IQ2_M.gguf
 * small and good for SD-Prompt

* LLM-vision 👀 (work with SDXL, VRAM >=8G is better )
 * https://huggingface.co/xtuner/llava-phi-3-mini-gguf
 * llava-phi-3-mini-mmproj-f16.gguf (600MB,vision adapter)
 * ⭐⭐⭐llava-phi-3-mini-f16.gguf (7G, main model)
 * https://huggingface.co/FiditeNemini/Llama-3.1-Unhinged-Vision-8B-GGUF
 * llava-llama-3.1-8b-mmproj-f16.gguf
 * ⭐⭐⭐Llama-3.1-Unhinged-Vision-8B-Q8.0.gguf
 * https://huggingface.co/Lewdiculous/Eris_PrimeV4-Vision-32k-7B-GGUF-IQ-Imatrix#quantization-information
 * quantization_options = ["Q4_K_M", "Q4_K_S", "IQ4_XS", "Q5_K_M", "Q5_K_S","Q6_K", "Q8_0", "IQ3_M", "IQ3_S", "IQ3_XXS"]
 * ⭐⭐⭐⭐⭐for low VRAM **super small**: IQ3_XXS (2.83G)
 * in fact, it's enough uses.

## Using Online LLM Service Setup example

### OpenAI ChatGPT

- In Auto-LLM Setup tab
  - LLM-URL=https://api.openai.com/v1
- get ur api key from openAI : https://platform.openai.com/api-keys
  - LLM-API-KEY = xxxxxxxxxxxxxxxxxxxxxxx
  - LLM-Model-Name = gpt-3.5-turbo
  
### Google Gemini
- LLM-URL= https://generativelanguage.googleapis.com/v1
- get the api key: https://ai.google.dev/gemini-api/docs/api-key?hl=zh-tw

### X Grok
- register here: https://x.ai/geo-block
- Its not open for my region. i cant test for u guys.

### claude.ai
- seems not for api call: https://claude.ai/upgrade

### Hugging face space
- https://huggingface.co/spaces

### Javascript!
security issue, but u can consider as follows.

* https://github.com/pmcculler/sd-dynamic-javascript
* https://github.com/ThereforeGames/unprompted
* https://github.com/adieyal/sd-dynamic-prompts
* https://en.wikipedia.org/wiki/Server-side_request_forgery
* and Command Line Arg --allow-code

## Buy me a Coca cola ☕

https://buymeacoffee.com/xxoooxx

## Colophon

Made for fun. I hope if brings you great joy, and perfect hair forever. Contact me with questions and comments, but not
threats, please. And feel free to contribute! Pull requests and ideas in Discussions or Issues will be taken quite
seriously!
--- https://decade.tw

sd-webui-decadetw-auto-prompt-llm

sd-webui-decadetw-auto-prompt-llm Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

sd-webui-decadetw-auto-prompt-llm

sd-webui-decadetw-auto-prompt-llm Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README