hass-agent-llm

KO | EN

gitlite — search

#python #deep-learning #nextjs #python3 #api #typescript #backend #cpp #css #html #llm #multimodal-learning

★ 46 Open GitHub ↗

Home Assistant custom component extending the conversation platform with OpenAI-compatible LLM integration, ChromaDB vector search, automatic memory extraction, and extensible tool system.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Home Agent

[![Version](https://img.shields.io/badge/version-0.9.5-blue.svg)](https://github.com/aradlein/hass-agent-llm/releases)
[![Build Status](https://github.com/aradlein/hass-agent-llm/workflows/CI/badge.svg)](https://github.com/aradlein/hass-agent-llm/actions)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
[![Home Assistant](https://img.shields.io/badge/Home%20Assistant-2026.3.1+-blue.svg)](https://www.home-assistant.io/)
[![HACS](https://img.shields.io/badge/HACS-Custom-orange.svg)](https://hacs.xyz/)

A highly customizable Home Assistant custom component that provides intelligent conversational AI capabilities with advanced tool calling, context injection, and conversation history management.

## What's New in v0.9.5

- **Vector Reindex Optimization** - Skip reindexing entities when state and attributes are unchanged, reducing unnecessary re-embeddings
- **Datetime Serialization Fix** - Properly handle datetime objects in tool results and vector DB context provider
- **Area Lookup Fix** - Use entity/device registry for area resolution instead of state.attributes for more reliable room detection

[View Full Changelog](https://github.com/aradlein/hass-agent-llm/releases)

---

### Previous Release: v0.9.4

- **Embedding Cache Memory Leak Fix** - Fixed a memory leak where the embedding cache accumulated stale entries for frequently-changing entities
- **CI Fixes** - Fixed version validation workflow, flake8, and black formatting issues

## Overview

Home Agent extends Home Assistant's native conversation platform to enable natural language control and monitoring of your smart home. It works with any OpenAI-compatible LLM provider, giving you flexibility to use cloud services or run models locally.

**Key Capabilities:**
- Natural language home control through any OpenAI-compatible LLM
- Automatic context injection - LLM knows your home's current state
- Persistent conversation memory across interactions
- Extensible tool system for custom integrations
- Streaming responses for voice assistants
- Long-term memory system for personalized experiences

## Features

### Core Features

- **LLM Integration** - Works with OpenAI, Azure OpenAI, Ollama, LocalAI, LM Studio, or any OpenAI-compatible endpoint
- **Entity Context** - Automatically provides relevant entity states to the LLM
- **Conversation History** - Maintains context across multiple interactions with persistent storage
- **Native Tools** - Built-in `ha_control` and `ha_query` tools for home automation
- **Custom Tools** - Define REST API and Home Assistant service tools in configuration
- **Event System** - Rich events for automation triggers and monitoring
- **Streaming Responses** - Low-latency streaming for voice assistant integration (~10x faster)

### Advanced Features

- **Vector Database Integration** - Semantic entity search using ChromaDB for efficient context management
- **Multi-LLM Support** - Use a fast local model for control + powerful cloud model for analysis
- **Memory System** - Automatic extraction and recall of facts, preferences, and context
- **Context Optimization** - Intelligent compression to stay within token limits
- **Tool Progress Indicators** - Real-time feedback during tool execution

## Requirements

### Required
- **Home Assistant** - Version 2026.3.1 or later (for Python 3.14 support)
- **Python Dependencies** - `aiohttp >= 3.9.0` (included with Home Assistant)

### Optional (Enable Advanced Features)
- **ChromaDB** - For vector database context mode
  - `chromadb-client == 1.5.3`
  - Required for: Vector DB context injection and memory system
- **OpenAI** - For embeddings in vector DB mode
  - `openai >= 1.3.8`
  - Required for: Vector DB entity indexing (alternative: use Ollama)
- **Wyoming Protocol TTS** - For streaming responses
  - Required for: Low-latency voice assistant integration

## Installation

### HACS (Recommended)

1. In HACS, go to **Integrations** → **⋮** → **Custom repositories**
2. Add repository: `https://github.com/aradlein/hass-agent-llm`
3. Category: **Integration**
4. Click **Add**
5. Search for "Home Agent" in HACS
6. Click **Install**
7. Restart Home Assistant
8. Go to Settings > Devices & Services > Add Integration
9. Search for "Home Agent" and follow the setup wizard

### Manual Installation

1. Download the latest release from GitHub
2. Copy the `custom_components/home_agent` directory to your Home Assistant `config/custom_components` folder
3. Restart Home Assistant
4. Go to Settings > Devices & Services > Add Integration
5. Search for "Home Agent" and complete the configuration

**For detailed installation instructions, see [Installation Guide](docs/INSTALLATION.md)**

## Quick Start

### 1. Add the Integration

Navigate to Settings > Devices & Services > Add Integration, search for "Home Agent", and configure:

- **Name**: Friendly name (e.g., "Home Agent")
- **LLM Base URL**: Your OpenAI-compatible endpoint
  - OpenAI: `https://api.openai.com/v1`
  - Azure OpenAI: `https://<resource>.openai.azure.com/openai/deployments/<deployment>`
  - Ollama (local): `http://localhost:11434/v1`
  - LocalAI: Your LocalAI URL
- **API Key**: Your API key (if required)
- **Model**: Model name (e.g., `gpt-4o-mini`, `llama3.2`, etc.)
- **Temperature**: 0.7 (recommended for most use cases)
- **Max Tokens**: 500 (adjust based on your needs)

### 2. Test Basic Functionality

Call the conversation service:

```yaml
service: home_agent.process
data:
  text: "Turn on the living room lights"
```

### 3. Explore Advanced Configuration

Access Settings > Devices & Services > Home Agent > Configure to:
- Configure context injection mode (direct or vector DB)
- Enable conversation history
- Set up custom tools
- Configure external LLM for complex queries
- Enable memory system
- Enable streaming for voice assistants

**For detailed configuration options, see [Configuration Reference](docs/CONFIGURATION.md)**

## Documentation

### Quick Start Guides
- [Installation](docs/INSTALLATION.md) - Get up and running in minutes
- [Configuration](docs/CONFIGURATION.md) - Essential settings explained
- [Example Configurations](docs/EXAMPLE_CONFIGS.md) - Ready-to-use configs for common providers
- [FAQ](docs/FAQ.md) - Top 20 questions answered

### Feature Guides
- [Memory System](docs/MEMORY_SYSTEM.md) - Enable long-term memory
- [Vector DB Setup](docs/VECTOR_DB_SETUP.md) - Semantic entity search
- [Custom Tools](docs/CUSTOM_TOOLS.md) - Extend with REST APIs and services
- [External LLM](docs/EXTERNAL_LLM.md) - Multi-LLM workflows

### Reference
- [Architecture Overview](docs/ARCHITECTURE.md) - Component diagrams and system design
- [API Reference](docs/API_REFERENCE.md) - Services, events, and tools
- [Troubleshooting](docs/TROUBLESHOOTING.md) - Quick fixes for common issues
- [Examples](docs/EXAMPLES.md) - 10 ready-to-use examples
- [Migration Guide](docs/MIGRATION.md) - Moving from extended_openai_conversation

## Usage Examples

### Voice Control

```yaml
# Use with Home Assistant voice assistant
# Just speak naturally to your voice assistant
"Turn on the kitchen lights to 50%"
"What's the temperature in the living room?"
"Is the front door locked?"
```

### Automation Integration

```yaml
automation:
  - alias: "Morning Briefing"
    trigger:
      - platform: time
        at: "07:00:00"
    action:
      - service: home_agent.process
        data:
          text: "Good morning! Please prepare for the day."
          conversation_id: "morning_routine"
```

### Custom Tool Example

```yaml
# configuration.yaml
home_agent:
  tools_custom:
    - name: check_weather
      description: "Get weather forecast"
      handler:
        type: rest
        url: "https://api.open-meteo.com/v1/forecast"
        method: GET
        query_params:
          latitude: "47.6062"
          longitude: "-122.3321"
          current: "temperature_2m,precipitation"
```

**For more examples, see [Examples Documentation](docs/EXAMPLES.md)**

## Voice Conversation Persistence

Home Agent automatically maintains conversation context across multiple voice interactions, enabling natural multi-turn conversations with your voice assistant.

### How It Works

- Each user/device combination maintains a persistent conversation session
- Sessions automatically expire after 1 hour of inactivity (configurable)
- Conversation history is preserved within each session
- Different devices maintain independent conversation contexts

### Example

**Before (without persistence):**
```
User: "What's the temperature in the living room?"
Agent: "The living room is 72°F"

[Later]
User: "What about the bedroom?"
Agent: "I don't have context about what you're asking about"
```

**After (with persistence):**
```
User: "What's the temperature in the living room?"
Agent: "The living room is 72°F"

[Later]
User: "What about the bedroom?"
Agent: "The bedroom temperature is 68°F"
```

### Configuration

Configure the session timeout in integration settings:
- Minimum: 60 seconds (1 minute)
- Maximum: 86400 seconds (24 hours)
- Default: 3600 seconds (1 hour)

### Clearing Conversations

Reset your conversation context using the `clear_conversation` service:

```yaml
# Clear conversation for current user/device
service: home_agent.clear_conversation

# Clear conversation for specific device
service: home_agent.clear_conversation
data:
  device_id: "kitchen_satellite"

# Clear all conversations
service: home_agent.clear_conversation
```

### Multi-Device Behavior

Each device maintains its own conversation context:
- Kitchen satellite remembers kitchen-related conversations
- Bedroom satellite has independent context
- Same user on different devices = different conversations

## Contributing

Contributions are welcome! Please:

1. Fork the repository
2. Create a feature branch (`git checkout -b feature/amazing-feature`)
3. Write tests for new functionality
4. Ensure all tests pass (`pytest tests/`)
5. Follow the coding standards in [Development Guide](docs/DEVELOPMENT.md)
6. Submit a pull request

## Testing

Home Agent maintains >80% code coverage with comprehensive unit and integration tests:

```bash
# Set up environment
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt
pip install -r requirements_dev.txt

# Run tests
pytest tests/unit/ -v

# Run with coverage
pytest tests/ --cov=custom_components.home_agent --cov-report=html
```

**Test Status**: 400+ passing tests across core functionality, vector DB, memory system, custom tools, and streaming.

## Support

- **Issues**: [GitHub Issues](https://github.com/aradlein/hass-agent-llm/issues)
- **Discussions**: [GitHub Discussions](https://github.com/aradlein/hass-agent-llm/discussions)
- **Documentation**: See [docs/](docs/) directory

## License

MIT License

## Credits

Built with inspiration from the extended_openai_conversation integration. Special thanks to the Home Assistant community.

## Changelog

### v0.9.5 (Latest)
- **Fix**: Skip vector reindex when entity state and attributes unchanged — reduces unnecessary re-embeddings
- **Fix**: Handle datetime serialization in tool results and vector DB context
- **Fix**: Use entity/device registry for area lookup instead of state.attributes

### v0.9.4
- **Fix**: Evict stale embedding cache entries on entity state change — prevents memory leak from frequently-changing entities (#111)
- **Fix**: Version validation CI workflow grep pattern anchored to avoid false mismatches
- **Fix**: Black formatting and flake8 compliance

### v0.9.3
> **Compatibility Notice:** This release targets **Home Assistant 2026.3.1** and **Python 3.14**. It may have backwards compatibility issues with older HA versions or Python 3.12/3.13. If you experience problems, please try a previous release.
- **Fix**: Upgraded `chromadb-client` to `1.5.3` to resolve Python 3.14 incompatibility (removal of `imghdr` standard library module)
- **Fix**: Updated dev requirements and test mocks to align with chromadb-client 1.5.3 API
- **Changed**: Minimum supported Home Assistant version updated to 2026.3.1

### v0.9.2
- **Fix**: Move initial entity indexing to background task to prevent setup timeout on large installations

### v0.9.1
- **Fix**: Resolve multiple memory leaks causing ~6GB growth over time — LRU eviction for embedding caches, debounced batch reindexing, HTTP session reuse, conversation history enforcement, and proper resource cleanup on shutdown

### v0.9.0
- **Feature**: Azure OpenAI support - native integration with Azure OpenAI deployments including API versioning and endpoint handling (#9)
- **Feature**: Universal language support - works with any Home Assistant language setting via MATCH_ALL (#15)
- **Feature**: Jinja template support for API key fields - use Home Assistant templates for dynamic secrets (#14)
- **Fix**: Improved compatibility with proxy gateways like Cloudflare AI Gateway (#17)

### v0.8.8
- **Feature**: Include entity/device labels in system prompt for better LLM context (contributed by @zopanix)
- **Testing**: Comprehensive test coverage for labels feature

### v0.8.7
- **Feature**: Custom service tools now support `return_response: true` for services that return data (like `calendar.get_events`)

### v0.8.6
- **Feature**: TTFT (Time To First Token) and voice pipeline metrics for performance monitoring
- **Feature**: Configurable max context tokens option (fixes #65)
- **Fix**: Memory retrieval no longer creates transient memories or duplicates
- **Enhancement**: Comprehensive testing improvements and GitHub issue fixes

### v0.8.5
- **Feature**: OpenAI-compatible embedding endpoints - Vector DB now respects configured `embedding_base_url` for OpenAI provider (fixes #6)
- **Enhancement**: Switched to async OpenAI client using Home Assistant's native HTTP client
- **Enhancement**: Improved retry logic with proper exponential backoff for embedding requests

### v0.8.4
- **Feature**: Reasoning model support - Filter `<think>...</think>` blocks from LLM output, enabling support for reasoning models like Qwen3, DeepSeek R1, and o1/o3
- **Fix**: Only send `keep_alive` parameter to Ollama backends; prevents 400 errors with OpenAI and other cloud APIs

### v0.8.3
- **Enhancement**: Updated minimum Home Assistant version to 2025.11.0 for improved compatibility
- **Fix**: CI test environment now uses pinned Home Assistant dependencies
- **Docs**: Successfully submitted to HACS for official community store listing

### v0.8.2
- **Feature**: Feature-based service filtering with intelligent parameter hints
- **Enhancement**: Brightness parameter standardization (brightness_pct 0-100 instead of brightness 0-255)
- **Fix**: Move play_media to base services for all media_player entities
- **Enhancement**: Comprehensive entity services reference documentation
- **Enhancement**: Improved system prompts with standardized parameter rules

hass-agent-llm

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README