LLMUnity

KO | EN

gitlite — search

#javascript #python #hacktoberfest #react #ai #typescript #llm #go #golang #android #machine-learning #rust #deep-learning #linux

(★ 1,700)

Create characters in Unity with LLMs!

LLMUnity Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

<h3 align="center">Create characters in Unity with LLMs!</h3>

[![License: Apache](https://img.shields.io/badge/License-Apache_2.0-yellow.svg)](https://opensource.org/license/apache-2-0)
<a href="https://discord.gg/RwXKQb6zdv"><img src="https://discordapp.com/api/guilds/1194779009284841552/widget.png?style=shield"/></a>
[![Reddit](https://img.shields.io/badge/Reddit-%23FF4500.svg?style=flat&logo=Reddit&logoColor=white)](https://www.reddit.com/user/UndreamAI)
[![LinkedIn](https://img.shields.io/badge/LinkedIn-blue?style=flat&logo=linkedin&labelColor=blue)](https://www.linkedin.com/company/undreamai)
[![Asset Store](https://img.shields.io/badge/Asset%20Store-black.svg?style=flat&logo=unity)](https://assetstore.unity.com/packages/slug/273604)
[![GitHub Repo stars](https://img.shields.io/github/stars/undreamai/LLMUnity?style=flat&logo=github&color=f5f5f5)](https://github.com/undreamai/LLMUnity)
[![Documentation](https://img.shields.io/badge/Docs-white.svg?logo=data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAADAAAAAwEAYAAAAHkiXEAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAABl0RVh0U29mdHdhcmUAd3d3Lmlua3NjYXBlLm9yZ5vuPBoAAATqSURBVHic7ZtbiE1RGMc349K4M5EwklwjzUhJCMmTJPJAYjQXJJcH8+Blkry4lPJA8aAoJbekDLmUS6E8SHJL5AW5JPf77eHv93C22Wfttc/ee+0zc/4vv+bMXvusvfZa3/q+b33H80oqqaSSSmqrKnPdgXjUvbvYq5f4+7f486eb/rRajRsn7t4tPngg/vol/vkj/vghXr0q7tghzpyZ//79+on79omXLombNondukXrd9GoSxdx8mSxqUm8eVNkgAvl0aPioEFip07i6dP52z15Ig4fbvVY2VVFhbhokXjrlogJiWvAg/jwoXjqVO73+leUny9eiFVV5mfMlLDRBw+KX76ISQ+0LZ8/F00v4uJFsWPHFh83O+rdWzx3TnQ9wCZ+/Sqyl5iux1RmTu3aiYcPi64H1pasALypoOv4/8SJXraEbXc9kLbECxo2TKyuFj9/zt9u+XIvG8LWv3wpuh5QW86f3/JznT+fv93s2S23C1Z72wbhtH692LdvMvdPSgzkhAkiJhT16ZO/PRPOmcr+Rda4aa5nclTeuZP7PDgRpr1g40bPrQYOFF0PYKHEC+raVVy8OFy7R49EArvURU4mrUAqaTY0iB8/2rXD+XCm5mbR9QAWylevorV7/VpkL0ld06eLpkiyWPj9u93179+LpFZwZ1PXtGnitWui64GMStPmG7SH1NSIJBNHjvTSFZvRvHlise0N9JcBtW1/44Y4dqx45IjnU0JxAGLpklPx+9VZFwPp/9v/eZDGjxcZh7dv4+mXtch+up7Rca+MsJvxiRNi6nvBhg25HWprZMaPGeOlqxEjxGKz+XGRTAAmyJnq6sR370TXA2NLW+8HNjZ62dLOnaLrAQ1r2zmqPH482n0mTfJCKmEvCJHUooNZE/369Elct06kqiKsONRfulTEFDsX8QDlIa5nup9374pE8IiZHPY+ly+LZE/37/cM6mC6IB6Vl4urV6fzfUG6d0/csyf37wsXRFInaM4ckTjGdPg+apTYs6dI3RIWwH//1DV1qkiuxNY2FzrTd+2y6y8z2HQU6efZs+KBAyJZ4v+V0h6ArlwROaQP0uPH4ooV4sqV8Xz/4MF211M2wwoOq1mzRAq5Pnywa5+4KDHE9mI7ly0TO3fOvZ6/eZCoKwB32HS0SMFV1DNtImBKHYstBROoQ4fEQk2RaS+qrxejmj5M7NatIhWARS82xUJfAKahzFcdPnq0GLYgy7Rnbd8e6rGKRyzpuNzPBQty709RcNSZf/KkuHCh2GpMDyKbGNcLYE+YMkVks336NFx7XhTZ3szXiBaqtWvFuAOxM2dEZiyH8UErgc8JLNun7E0aFffSI7RP6owZmz9kSO73HjsmXr8ukppYsybSYyQvBp5QfOjQ3M9tRR496pGgLf1JtLlzRZJzlFzGp4SWDnUxFCrdvy+uWiWa3DJe3N69oj8uSEq8CER88uaNOGBAOv2ILGY69TBBJoM8O0t72zaRoztXBzlLlrT8XARW/IQq82JTMv3mKmv0/9CC4mJMYPwrMSETxAyurRUxQVmXP1fEid7mzeK3b+n2Jzb16CFu2SIWmtNJiriVxANsyq0uoCJfTk4G9y4t24/bSQ0rTkP6gVTG3mz//uKMGSK/ucId5Xe9lZUi5eMMLGUgz56J5Hxu3xZ50Xg3RMIltVn9BRja26PYsBHgAAAAAElFTkSuQmCC)](https://undream.ai/LLMUnity)

LLM for Unity enables seamless integration of Large Language Models (LLMs) within the Unity engine. 
It allows to create intelligent AI characters that players can interact with for an immersive experience. 
The package includes a Retrieval-Augmented Generation (RAG) system for semantic search across your data, which can be used to enhance the character's knowledge.

The LLM backend, [LlamaLib](https://github.com/undreamai/LlamaLib), is built on top of the awesome [llama.cpp](https://github.com/ggerganov/llama.cpp) library and  provided as a standalone C++/C# library.

<a href="#at-a-glance" style="color: black">At a glance</a>&nbsp;&nbsp;•&nbsp;
<a href="#how-to-help" style=color: black>How to help</a>&nbsp;&nbsp;•&nbsp;
<a href="#games--projects-using-llm-for-unity" style=color: black>Games / Projects using LLM for Unity</a>&nbsp;&nbsp;•&nbsp;
<a href="#setup" style=color: black>Setup</a>&nbsp;&nbsp;•&nbsp;
<a href="#quick-start" style=color: black>Quick start</a>&nbsp;&nbsp;•&nbsp;
<a href="#advanced-usage" style=color: black>Advanced usage</a>&nbsp;&nbsp;•&nbsp;
<a href="#semantic-search-with-a-rag-system" style=color: black>RAG</a>&nbsp;&nbsp;•&nbsp;
<a href="#llm-model-management" style=color: black>LLM model management</a>&nbsp;&nbsp;•&nbsp;
<a href="#examples" style=color: black>Examples</a>&nbsp;&nbsp;•&nbsp;
<a href="#license" style=color: black>License</a>

## At a glance
- 💻 Runs anywhere: PC, mobile or VR!
- ⚡ Blazing fast inference on CPU and GPU (Nvidia, AMD, Apple Metal)
- 🏠 Runs locally without internet access. No data ever leave your game!
- 📡 Supports remote server setup
- 🤗 Supports all major LLM models
- 🔍 Advanced RAG System (ANN search)
- 🔧 Easy to setup, call with a single line of code
- 💰 Free to use for both personal and commercial purposes

🧪 Tested on Unity: 2021 LTS, 2022 LTS, 2023, Unity 6 
🚦 [Upcoming Releases](https://github.com/orgs/undreamai/projects/2/views/10)

### Business inquiries
For business inquiries you can reach out at hello@undream.ai.

## How to help
- [⭐ Star](https://github.com/undreamai/LLMUnity) the repo, leave a [review](https://assetstore.unity.com/packages/slug/273604) and spread the word about the project!
- Join us at [Discord](https://discord.gg/RwXKQb6zdv) and say hi.
- [Contribute](CONTRIBUTING.md) by submitting feature requests, bugs or even your own PR.
- [![](https://img.shields.io/static/v1?label=Sponsor&message=%E2%9D%A4&logo=GitHub&color=%23fe8e86)](https://github.com/sponsors/amakropoulos) this work or buy me a [![Ko-fi](https://img.shields.io/badge/Ko--fi-FF5E5B?logo=ko-fi&logoColor=white)](https://ko-fi.com/amakropoulos) to allow even cooler features!

## Games / Projects using LLM for Unity
- [Verbal Verdict](https://store.steampowered.com/app/2778780/Verbal_Verdict/)
- [I, Chatbot: AISYLUM](https://store.epicgames.com/de/p/i-chatbot-aisylum-83b2b5)
- [Nameless Souls of the Void](https://unicorninteractive.itch.io/nameless-souls-of-the-void)
- [Murder in Aisle 4](https://roadedlich.itch.io/murder-in-aisle-4)
- [Finicky Food Delivery AI](https://helixngc7293.itch.io/finicky-food-delivery-ai)
- [AI Emotional Girlfriend](https://whynames.itch.io/aiemotionalgirlfriend)
- [Case Closed](https://store.steampowered.com/app/2532160/Case_Closed)
- [MaiMai AI Agent System](https://github.com/IhateCreatingUserNames2/MaiMai)
- [Claria Chat](https://play.google.com/store/apps/details?id=com.ProLink.ClariaChat)
- [Endless Casual Drive](https://store.steampowered.com/app/2928500/Endless_Casual_Drive/)
- [Dating App Simulator](https://store.steampowered.com/app/3415680/Dating_App_Simulator/)
- [Virtual Reality Mock Interview](https://vrmi.vercel.app/)
- [Velesio AI server](https://github.com/Velesio/Velesio-AIServer)
- [Dungeon Chat](https://www.meta.com/experiences/dungeonchat/8527310950709276/)
- [Tomonaka Desk](https://joycatdev.itch.io/tomonaka-desk)
- [Digital Humans](https://store.steampowered.com/app/3089280/Digital_Humans/)
- [CakeMix](https://ripenedpeach.itch.io/cakemix)
- [HeyWaifu](https://squirclegames.itch.io/hey-waifu-ai)
- [Love and Lie](https://store.steampowered.com/app/3886140/Love_and_Lie/)
- [Psycho Simulator](https://store.steampowered.com/app/1244620/Psycho_Simulator/)
- [False Flame](https://store.steampowered.com/app/4427250/False_Flame/)
- [CielChan: Anime Desktop AI Companion](https://store.steampowered.com/app/4529510/CielChan_Anime_Desktop_AI_Companion/)

Contact hello@undream.ai to add your project!

## Setup
_Method 1: Install using the asset store_
- Open the [LLM for Unity](https://assetstore.unity.com/packages/slug/273604) asset page and click `Add to My Assets`
- Open the Package Manager in Unity: `Window > Package Manager`
- Select the `Packages: My Assets` option from the drop-down
- Select the `LLM for Unity` package, click `Download` and then `Import`

_Method 2: Install using the GitHub repo:_
- Open the Package Manager in Unity: `Window > Package Manager`
- Click the `+` button and select `Add package from git URL`
- Use the repository URL `https://github.com/undreamai/LLMUnity.git` and click `Add`

## Quick start
<img height="300" src=".github/character.png"/>

#### 1. Setup the LLM

First you will setup the LLM for your game:
- Create an empty GameObject. In the GameObject Inspector click `Add Component` and select the `LLM` script.
- Download one of the default models with the `Download Model` button (~GBs). Or load your own .gguf model with the `Load model` button (see [LLM model management](#llm-model-management)).

#### 2. Create an AI Character
Then you can setup each of your characters as follows:
- Create an empty GameObject for the character. In the GameObject Inspector click `Add Component` and select the `LLMAgent` script.
- Define the role of your AI in the `System Prompt`.
- (Optional) Select the LLM constructed above in the `LLM` field if you have more than one LLM GameObjects.

#### 3. Use in Your Script
In your script you can then use it as follows:
``` c#
using LLMUnity;

public class MyScript {
  public LLMAgent llmAgent;
  
  void HandleReply(string replySoFar){
    // do something with the reply from the model as it is being produced
    Debug.Log(replySoFar);
  }
  
  void Game(){
    // handle the response as it is being produced
    ...
    _ = llmAgent.Chat("Hello bot!", HandleReply);
    ...
  }
  
  async void GameAsync(){
    // or handle the entire response in one go
    ...
    string reply = await llmAgent.Chat("Hello bot!");
    Debug.Log(reply);
    ...
  }
}
```
You can also specify a function to call when the model reply has been completed:
``` c#
  void ReplyCompleted(){
    // do something when the reply from the model is complete
    Debug.Log("The AI has finished replying");
  }
  
  void Game(){
    // your game function
    ...
    _ = llmAgent.Chat("Hello bot!", HandleReply, ReplyCompleted);
    ...
  }
```

To stop the chat without waiting for its completion you can use:
``` c#
    llmAgent.CancelRequests();
```

- Finally, in the Inspector of the GameObject of your script, select the LLMAgent GameObject created above as the llmAgent property.

That's it! Your AI character is ready to chat! ✨

## Advanced usage

<details>
<summary>Build a mobile app</summary>

For mobile apps you can use models with up to 1-2 billion parameters ("Tiny models" in the LLM model manager). 
Larger models will typically not work due to the limited mobile hardware.

**iOS**
iOS can be built with the default player settings.

**Android**
On Android you need to specify the `IL2CPP` scripting backend and the `ARM64` as the target architecture in the player settings. 
These settings can be accessed from the `Edit > Project Settings` menu within the `Player > Other Settings` section. 
<img width="400" src=".github/android.png">

Since mobile app sizes are typically small, you can download the LLM model the first time the app launches.
This functionality is enabled with the `Download on Build` option.
In your project you can wait until the model download is complete with:
``` c#
await LLM.WaitUntilModelSetup();
```
You can also receive calls the download progress during the model download:
``` c#
await LLM.WaitUntilModelSetup(SetProgress);

void SetProgress(float progress){
  string progressPercent = ((int)(progress * 100)).ToString() + "%";
  Debug.Log($"Download progress: {progressPercent}");
}
```
This is useful to present e.g. a progress bar.
The [MobileDemo](Samples~/MobileDemo) demonstrates an example application for Android / iOS.

</details>
<details>
<summary>Restrict the output of the LLM / Function calling / Grammar</summary>

To restrict the output of the LLM you can use a grammar, read more [here](https://github.com/ggerganov/llama.cpp/tree/master/grammars). 
The grammar can edited directly in the `Grammar` field of the LLMAgent or saved in a gbnf / json schema file and loaded with the `Load Grammar` button (Advanced options). 
For instance to receive replies in json format you can use the [json.gbnf](https://github.com/ggerganov/llama.cpp/blob/b4218/grammars/json.gbnf) grammar.

Alternatively you can set the grammar directly in your script:
``` c#
llmAgent.grammar = "your grammar here";
```

For function calling you can define similarly a grammar that allows only the function names as output, and then call the respective function. 
You can look into the [FunctionCalling](Samples~/FunctionCalling) sample for an example implementation.

</details>
<details>
<summary>Access / Save / Load your chat history</summary>
The chat history of a `LLMAgent` is retained in the `chat` variable that is a list of `ChatMessage` objects. 
The ChatMessage is a class that defines the `role` of the message and the `content`. 
The list contains alternating messages with the player prompt and the AI reply. 
You can modify the chat history and then set it to your LLMAgent GameObject:
``` c#
List<ChatMessage> newChat = new List<ChatMessage>();
...
llmAgent.chat = newChat;
```

To add new messages you can do:
``` c#
_ = llmAgent.AddUserMessage("your user message");
_ = llmAgent.AddAssistantMessage("your assistant reply");
```

To automatically save / load your chat history, you can specify the `Save` parameter of the LLMAgent to the filename (or relative path) of your choice.
The chat history is saved in the [persistentDataPath folder of Unity](https://docs.unity3d.com/ScriptReference/Application-persistentDataPath.html) as a json object.

To manually save your chat history, you can use:
``` c#
    _ = llmAgent.SaveHistory();
```
and to load the history:
``` c#
    _ = llmAgent.Loadistory();
```

</details>
<details>
<summary>Process the prompt at the beginning of your app for faster initial processing time</summary>

``` c#
  void WarmupCompleted(){
    // do something when the warmup is complete
    Debug.Log("The AI is nice and ready");
  }

void Game(){
    // your game function
    ...
    _ = llmAgent.Warmup(WarmupCompleted);
    ...
  }
```

</details>
<details>
<summary>Decide whether or not to add the message to the chat/prompt history</summary>

The last argument of the `Chat` function is a boolean that specifies whether to add the message to the history (default: true):
``` c#
  void Game(){
    // your game function
    ...
    string message = "Hello bot!";
    _ = llmAgent.Chat(message, HandleReply, ReplyCompleted, false);
    ...
  }
```

</details>
<details>
<summary>Use pure text completion</summary>

``` c#
  void Game(){
    // your game function
    ...
    string message = "The cat is away";
    _ = llmAgent.Completion(message, HandleReply, ReplyCompleted);
    ...
  }
```

</details>
<details>
<summary>Add a LLM / LLMAgent component programmatically</summary>

``` c#
using UnityEngine;
using LLMUnity;

public class MyScript : MonoBehaviour
{
    LLM llm;
    LLMAgent llmAgent;

async void Start()
    {
        // disable gameObject so that theAwake is not called immediately
        gameObject.SetActive(false);

// Add an LLM object
 llm = gameObject.AddComponent<LLM>();
 // set the model using the filename of the model.
 // The model needs to be added to the LLM model manager (see LLM model management) by loading or downloading it.
 // Otherwise the model file can be copied directly inside the StreamingAssets folder.
 llm.model = "Qwen3-4B-Q4_K_M.gguf";
 // optional: you can also set loras in a similar fashion and set their weights (if needed)
 llm.AddLora("my-lora.gguf");
 llm.AddLora("my-lora-2.gguf", 0.5f);
 // optional: set number of threads
 llm.numThreads = -1;
 // optional: enable GPU by setting the number of model layers to offload to it
 llm.numGPULayers = 10;

// Add an LLMAgent object
 llmAgent = gameObject.AddComponent<LLMAgent>();
 // set the LLM object that handles the model
 llmAgent.llm = llm;
 // set the character prompt
 llmAgent.systemPrompt = "A chat between a curious human and an artificial intelligence assistant.";
 // set the AI and player name
 llmAgent.assistantRole = "AI";
 llmAgent.userRole = "Human";
 // optional: set a save path
 llmAgent.save = "AICharacter1.json";
 // optional: set a grammar
 llmAgent.grammar = "your grammar here";

// re-enable gameObject
        gameObject.SetActive(true);
    }
}
```

</details>
<details>
<summary>Use a remote server</summary>

You can use a remote server to carry out the processing and implement characters that interact with it.

**Create the server** 
To create the server:
- Create a project with a GameObject using the `LLM` script as described above
- Enable the `Remote` option of the `LLM` and optionally configure the server port and API key
- Enable 'Allow Downloads Over HTTP' in the project settings
- Build and run to start the server

Alternatively you can use a server binary for easier deployment:
- Run the above scene from the Editor and copy the command from the Debug messages (starting with "Deploy server command:")
- Download and extract the [LlamaLib binaries](https://github.com/undreamai/LlamaLib/releases/download/v2.0.0/LlamaLib-v2.0.0.zip)
- From command line change directory to the `servers` folder selected and start the server by running the command copied from above.

**Create the characters** 
Create a second project with the game characters using the `LLMAgent` script as described above.
Enable the `Remote` option and configure the host with the IP address (starting with "http://") and port / API key of the server.

</details>
<details>
<summary>Compute embeddings using a LLM</summary>

The `Embeddings` function can be used to obtain the emdeddings of a phrase:
``` c#
 List<float> embeddings = await llmAgent.Embeddings("hi, how are you?");
```

</details>

A detailed documentation on function level can be found here:
<a href="https://undream.ai/LLMUnity"><img src="https://img.shields.io/badge/Documentation-white.svg?logo=data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAADAAAAAwEAYAAAAHkiXEAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAABl0RVh0U29mdHdhcmUAd3d3Lmlua3NjYXBlLm9yZ5vuPBoAAATqSURBVHic7ZtbiE1RGMc349K4M5EwklwjzUhJCMmTJPJAYjQXJJcH8+Blkry4lPJA8aAoJbekDLmUS6E8SHJL5AW5JPf77eHv93C22Wfttc/ee+0zc/4vv+bMXvusvfZa3/q+b33H80oqqaSSSmqrKnPdgXjUvbvYq5f4+7f486eb/rRajRsn7t4tPngg/vol/vkj/vghXr0q7tghzpyZ//79+on79omXLombNondukXrd9GoSxdx8mSxqUm8eVNkgAvl0aPioEFip07i6dP52z15Ig4fbvVY2VVFhbhokXjrlogJiWvAg/jwoXjqVO73+leUny9eiFVV5mfMlLDRBw+KX76ISQ+0LZ8/F00v4uJFsWPHFh83O+rdWzx3TnQ9wCZ+/Sqyl5iux1RmTu3aiYcPi64H1pasALypoOv4/8SJXraEbXc9kLbECxo2TKyuFj9/zt9u+XIvG8LWv3wpuh5QW86f3/JznT+fv93s2S23C1Z72wbhtH692LdvMvdPSgzkhAkiJhT16ZO/PRPOmcr+Rda4aa5nclTeuZP7PDgRpr1g40bPrQYOFF0PYKHEC+raVVy8OFy7R49EArvURU4mrUAqaTY0iB8/2rXD+XCm5mbR9QAWylevorV7/VpkL0ld06eLpkiyWPj9u93179+LpFZwZ1PXtGnitWui64GMStPmG7SH1NSIJBNHjvTSFZvRvHlise0N9JcBtW1/44Y4dqx45IjnU0JxAGLpklPx+9VZFwPp/9v/eZDGjxcZh7dv4+mXtch+up7Rca+MsJvxiRNi6nvBhg25HWprZMaPGeOlqxEjxGKz+XGRTAAmyJnq6sR370TXA2NLW+8HNjZ62dLOnaLrAQ1r2zmqPH482n0mTfJCKmEvCJHUooNZE/369Elct06kqiKsONRfulTEFDsX8QDlIa5nup9374pE8IiZHPY+ly+LZE/37/cM6mC6IB6Vl4urV6fzfUG6d0/csyf37wsXRFInaM4ckTjGdPg+apTYs6dI3RIWwH//1DV1qkiuxNY2FzrTd+2y6y8z2HQU6efZs+KBAyJZ4v+V0h6ArlwROaQP0uPH4ooV4sqV8Xz/4MF211M2wwoOq1mzRAq5Pnywa5+4KDHE9mI7ly0TO3fOvZ6/eZCoKwB32HS0SMFV1DNtImBKHYstBROoQ4fEQk2RaS+qrxejmj5M7NatIhWARS82xUJfAKahzFcdPnq0GLYgy7Rnbd8e6rGKRyzpuNzPBQty709RcNSZf/KkuHCh2GpMDyKbGNcLYE+YMkVks336NFx7XhTZ3szXiBaqtWvFuAOxM2dEZiyH8UErgc8JLNun7E0aFffSI7RP6owZmz9kSO73HjsmXr8ukppYsybSYyQvBp5QfOjQ3M9tRR496pGgLf1JtLlzRZJzlFzGp4SWDnUxFCrdvy+uWiWa3DJe3N69oj8uSEq8CER88uaNOGBAOv2ILGY69TBBJoM8O0t72zaRoztXBzlLlrT8XARW/IQq82JTMv3mKmv0/9CC4mJMYPwrMSETxAyurRUxQVmXP1fEid7mzeK3b+n2Jzb16CFu2SIWmtNJiriVxANsyq0uoCJfTk4G9y4t24/bSQ0rTkP6gVTG3mz//uKMGSK/ucId5Xe9lZUi5eMMLGUgz56J5Hxu3xZ50Xg3RMIltVn9BRja26PYsBHgAAAAAElFTkSuQmCC"/></a>

## Semantic search with a RAG system
LLM for Unity implements a super-fast similarity search functionality with a Retrieval-Augmented Generation (RAG) system. 
It is based on the LLM embeddings, and the Approximate Nearest Neighbors (ANN) search from the [usearch](https://github.com/unum-cloud/usearch) library. 
Semantic search works as follows.

**Building the data** You provide text inputs (a phrase, paragraph, document) to add to the data. 
Each input is split into chunks (optional) and encoded into embeddings with a LLM.

**Searching** You can then search for a query text input. 
The input is again encoded and the most similar text inputs or chunks in the data are retrieved.

To use semantic serch:
- create a GameObject for the LLM as described above. Download one of the provided RAG models or load your own (good options can be found at the [MTEB leaderboard](https://huggingface.co/spaces/mteb/leaderboard)).
- create an empty GameObject. In the GameObject Inspector click `Add Component` and select the `RAG` script.
- In the Search Type dropdown of the RAG select your preferred search method. `SimpleSearch` is a simple brute-force search, while`DBSearch` is a fast ANN method that should be preferred in most cases.
- In the Chunking Type dropdown of the RAG you can select a method for splitting the inputs into chunks. This is useful to have a more consistent meaning within each data part. Chunking methods for splitting according to tokens, words and sentences are provided.

Alternatively, you can create the RAG from code (where llm is your LLM):
``` c#
 RAG rag = gameObject.AddComponent<RAG>();
 rag.Init(SearchMethods.DBSearch, ChunkingMethods.SentenceSplitter, llm);
```

In your script you can then use it as follows :unicorn::
``` c#
using LLMUnity;

public class MyScript : MonoBehaviour
{
  RAG rag;

async void Game(){
    ...
    string[] inputs = new string[]{
      "Hi! I'm a search system.",
      "the weather is nice. I like it.",
      "I'm a RAG system"
    };
    // add the inputs to the RAG
    foreach (string input in inputs) await rag.Add(input);
    // get the 2 most similar inputs and their distance (dissimilarity) to the search query
    (string[] results, float[] distances) = await rag.Search("hello!", 2);
    // to get the most similar text parts (chunks), instead of full input, you can enable the returnChunks option
    rag.ReturnChunks(true);
    (results, distances) = await rag.Search("hello!", 2);
    ...
  }
}
```

You can also add / search text inputs for groups of data e.g. for a specific character or scene:
``` c#
    // add the inputs to the RAG for a group of data e.g. an orc character
    foreach (string input in inputs) await rag.Add(input, "orc");
    // get the 2 most similar inputs for the group of data e.g. the orc character
    (string[] results, float[] distances) = await rag.Search("how do you feel?", 2, "orc");
...

You can save the RAG state (stored in the `Assets/StreamingAssets` folder):
``` c#
rag.Save("rag.zip");
```
and load it from disk:
``` c#
await rag.Load("rag.zip");
```

You can use the RAG to feed relevant data to the LLM based on a user message:
``` c#
  string message = "How is the weather?";
  (string[] similarPhrases, float[] distances) = await rag.Search(message, 3);

string prompt = "Answer the user query based on the provided data.\n\n";
  prompt += $"User query: {message}\n\n";
  prompt += $"Data:\n";
  foreach (string similarPhrase in similarPhrases) prompt += $"\n- {similarPhrase}";

_ = llmAgent.Chat(prompt, HandleReply, ReplyCompleted);
```

The `RAG` sample includes an example RAG implementation as well as an example RAG-LLM integration.

That's all :sparkles:!

## LLM model management
LLM for Unity includes a built-in model manager for easy model handling. 
The model manager allows to load or download LLMs and can be found as part of the LLM GameObject: 
<img width="360" src=".github/LLM_manager.png">

You can download models with the `Download model` button. 
LLM for Unity includes different state of the art models built-in for different model sizes, quantised with the Q4_K_M method. 
Alternative models can be downloaded from [HuggingFace](https://huggingface.co/models?library=gguf&sort=downloads) in .gguf format. 
You can download a model locally and load it with the `Load model` button, or copy the URL in the `Download model > Custom URL` field to directly download it. 
If a HuggingFace model does not provide a gguf file, it can be converted to gguf with this [online converter](https://huggingface.co/spaces/ggml-org/gguf-my-repo).

Models added in the model manager are copied to the game during the building process. 
You can omit a model by deselecting the "Build" checkbox. 
To remove the model (but not delete it from disk) you can click the bin button. 
The path and URL of models can be diplayed in the expanded view of the model manager with the `>>` button: 
<img width="600" src=".github/LLM_manager_expanded.png">

You can create lighter builds by selecting the `Download on Build` option. 
The models will be downloaded the first time the game starts instead of bundled in the build. 
If you have loaded a model locally you need to set its URL through the expanded view, otherwise it will be copied in the build.

❕ Before using any model make sure you **check their license** ❕

## Examples
The [Samples~](Samples~) folder contains several examples of interaction 🤖:
- [SimpleInteraction](Samples~/SimpleInteraction): Simple interaction with an AI character
- [MultipleCharacters](Samples~/MultipleCharacters): Simple interaction using multiple AI characters
- [FunctionCalling](Samples~/FunctionCalling): Function calling sample with structured output from the LLM
- [RAG](Samples~/RAG): Semantic search using a Retrieval Augmented Generation (RAG) system. Includes example using a RAG to feed information to a LLM
- [MobileDemo](Samples~/MobileDemo): Example mobile app for Android / iOS with an initial screen displaying the model download progress
- [ChatBot](Samples~/ChatBot): Interaction between a player and a AI with a UI similar to a messaging app (see image below)
- [KnowledgeBaseGame](Samples~/KnowledgeBaseGame): Simple detective game using a knowledge base to provide information to the LLM based on [google/mysteryofthreebots](https://github.com/google/mysteryofthreebots)
 
<img width="400" src=".github/demo.gif">

To install a sample:
- Open the Package Manager: `Window > Package Manager`
- Select the `LLM for Unity` Package. From the `Samples` Tab, click `Import` next to the sample you want to install.

The samples can be run with the `Scene.unity` scene they contain inside their folder. 
In the scene, select the `LLM` GameObject and specify the LLM of your choice (see [LLM model management](#llm-model-management)). 
Save the scene, run and enjoy!

## License
The license of LLM for Unity is Apache 2.0 ([LICENSE.md](LICENSE.md)) and uses third-party software with MIT and Apache licenses.
Some models included in the asset define their own license terms, please review them before using each model.
Third-party licenses can be found in the ([Third Party Notices.md](<Third Party Notices.md>)).

LLMUnity

LLMUnity Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

LLMUnity

LLMUnity Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README