godot-ai-rpg

KO | EN

gitlite — search

#typescript #ai-agents #ai #dsh-plugin #deepseek-harness #open-source #cli #claude-code #codex #developer-tools #react #windows

★ 23 Open GitHub ↗

RPG prototype with full Speech To Speech Interaction using LLMs. Supports Metal and Vulkan.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

Update 13.08.2024: The project now uses the Instruct format for LLMs.

# Godot Voiced AI NPCs
A prototype using meta's LLaMA 3 / 3.1 as NPC controller / diaglogue manager with microphone input and text to speech output in realtime.

### Conversation and Interaction Workflow
The general workflow and tools for interaction with NPCS are intended to be the following:
- Player speaks to NPC (whisper.cpp) or types into chat
- Input is processed to AI prompt 
- AI generates response (llama.cpp Server)
- Speech is generated (piper-without-espeak with LibriTTS)

### Supported Platforms
Currently MacOS ARM64 and Windows X64. \
For GPU acelleration LLaMA.cpp was compiled with Vulkan for win-x64 and Metal for macos-arm64.

### LLaMA.cpp compatible Models
LLaMA 3 and 3.1 models that follow the Instruct format should work out of the box.

### Installation
Everything should work out of the box, you only need a fitting LLaMA 3 or 3.1 Instruct model and place it in model and rename it to model.gguf. \
Please read error messages carefully.

### Issues
If you have an issue, please post as much info as possible. \
Do not post screenshots of error message, copy them. \
All backend output is written to the godot console. \
Please copy this output to your issue as well unless you are 100% sure it is unrelated.

### Contribution
Feel free to make a PR to add to or improve the project. \
Keep it at a reasonable size, though, please. \
In case you are doing heavy lifting in your code, do it on a thread or async.

#### Possible, needed contributons
- A TTS model trained on different emotions would be greatly appreciated
- Someone needs to take a look at all speakers and determine the best ones

### Licensing

#### Code

This project's code - MIT \
llama.cpp - MIT \
vits - MIT \
piper-without-espeak - MIT \
whisper AI - MIT \
whisper.cpp - MIT \
whisper-godot - MIT \
LLaMA - Llama 3 Community License Agreement \
Libri TTS Medium - CC BY 4.0 - "LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech", Heiga Zen, Viet Dang, Rob Clark, Yu Zhang, Ron J. Weiss, Ye Jia, Zhifeng Chen, and Yonghui Wu, arXiv, 2019

#### Assets

hair.obj, elvs_grump_hair by Elvaerwyn, CC-BY, taken from MakeHuman Asset Pack Hair 02 https://static.makehumancommunity.org/assets/assetpacks/hair02.html \
skin_normal.png, Normalmap from Kamden Skin by Mindfront, CC-BY 4.0, taken from http://www.makehumancommunity.org/skin/kamden_skin.html \
Other NPC-related assets are public domain, big thanks for the makehuman project and contributors for existing and being generally very awesome.

godot-ai-rpg

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README