smolagents-lmstudio-examples

KO | EN

gitlite — search

#hacktoberfest #ai #typescript #go #android #machine-learning #rust #docker #java #kubernetes #vue #awesome-list #ios #open-source

smolagents-lmstudio-examples

(★ 15)

No description available.

smolagents-lmstudio-examples Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Using smolagents with Local LLMs via LM Studio

This repository contains examples of how to use [smolagents](https://github.com/smol-ai/smol-agent) with local LLMs running on [LM Studio](https://lmstudio.ai/).

## Summary

This repository demonstrates how to use the smolagents library with local LLMs running on LM Studio. It includes:

1. A basic example of connecting to LM Studio and creating a simple agent
2. A coding assistant that can help with Python programming tasks
3. A research assistant that can search for information and summarize text
4. A test script to verify the connection to LM Studio

The examples show how to:
- Connect to LM Studio's OpenAI-compatible API
- Create tools using the `@tool` decorator
- Use the `ToolCallingAgent` to create agents that can use tools
- Handle conversation state in a conversational interface

## Project Structure

```
smolagents-lmstudio-examples/
├── README.md                     # Project documentation
├── requirements.txt              # Python dependencies
├── setup.py                      # Package setup script
├── install.sh                    # Unix/Mac installation script
├── install.bat                   # Windows installation script
├── .gitignore                    # Git ignore file
├── smolagents_rules.mdc          # Cursor rules for smolagents
├── test_lm_studio_connection.py  # Script to test LM Studio connection
├── local_llm_example.py          # Basic example of using a local LLM
├── coding_assistant_agent.py     # Example of a coding assistant agent
└── research_assistant_agent.py   # Example of a research assistant agent
```

## Prerequisites

- Python 3.8+
- smolagents with OpenAI support (`pip install 'smolagents[openai]'`)
- requests (`pip install requests`)
- LM Studio installed on your computer

## Installation

### Using Installation Scripts

#### On macOS/Linux:
```bash
# Make the script executable
chmod +x install.sh

# Run the installation script
./install.sh
```

#### On Windows:
```
# Run the installation script
install.bat
```

### Manual Installation

1. Create a virtual environment (optional but recommended):
   ```bash
   python -m venv venv
   
   # On macOS/Linux
   source venv/bin/activate
   
   # On Windows
   venv\Scripts\activate
   ```

2. Install dependencies:
   ```bash
   pip install -r requirements.txt
   ```

### Using setup.py

```bash
pip install -e .
# Or with OpenAI support
pip install -e ".[openai]"
```

## Setup Instructions

1. **Install LM Studio**:
   - Download and install LM Studio from [lmstudio.ai](https://lmstudio.ai/)
   - Launch LM Studio and download a model of your choice

2. **Start the Local Server in LM Studio**:
   - Load a model in LM Studio
   - Click on the "Local Server" tab
   - Click "Start Server"
   - The server should be running at `http://localhost:1234`

3. **Test the Connection**:
   - Run the test script to verify that you can connect to LM Studio:
     ```
     python test_lm_studio_connection.py
     ```
   - If successful, you'll see a confirmation message
   - If not, the script will provide troubleshooting instructions

4. **Run the Example Scripts**:
   - Make sure the LM Studio server is running
   - Run one of the example scripts:
     ```
     python local_llm_example.py
     ```
     or
     ```
     python coding_assistant_agent.py
     ```
     or
     ```
     python research_assistant_agent.py
     ```

## Example Files

- `test_lm_studio_connection.py`: A script to test if the connection to LM Studio is working properly
- `local_llm_example.py`: A simple example showing how to connect to LM Studio and create a basic agent
- `coding_assistant_agent.py`: A more advanced example demonstrating how to create a coding assistant with tools
- `research_assistant_agent.py`: An example of a research assistant that can search for information and summarize text

## Configuration

The key configuration for connecting to LM Studio is:

```python
from smolagents import OpenAIServerModel

model = OpenAIServerModel(
    model_id="local-model",  # This can be any name, LM Studio will use whatever model you have loaded
    api_base="http://localhost:1234/v1",  # Default LM Studio API endpoint
    api_key="not-needed",  # LM Studio doesn't require an API key by default
)
```

You can adjust the `api_base` URL if your LM Studio server is running on a different port or host.

## Using Tools with Local LLMs

The examples demonstrate how to create and use tools with your local LLMs:

1. **Import the necessary classes**:
   ```python
   from smolagents import OpenAIServerModel, ToolCallingAgent, tool
   ```

2. **Define a tool function with the @tool decorator**:
   ```python
   @tool
   def search_web(query: str) -> str:
       """
       Search the web for information.
       
       Args:
           query: The search query
           
       Returns:
           str: Search results
       """
       # Tool implementation
       return result
   ```

3. **Add the tool to your agent**:
   ```python
   agent = ToolCallingAgent(
       name="AssistantName",
       model=model,
       tools=[search_web],
   )
   ```

4. **Run the agent**:
   ```python
   response = agent.run("Your query here")
   print(response)
   ```

## Handling Conversation State

`ToolCallingAgent` doesn't maintain conversation state between runs. To create a conversational interface, you need to:

1. **Store conversation history manually**:
   ```python
   conversation_history = []
   ```

2. **Create a new agent for each interaction**:
   ```python
   agent = ToolCallingAgent(
       name="AssistantName",
       model=model,
       tools=[tools_list],
   )
   ```

3. **Include conversation history in the prompt**:
   ```python
   # Add user input to history
   conversation_history.append(f"User: {user_input}")
   
   # Create prompt with history
   full_prompt = "\n".join(conversation_history) + "\n\nPlease respond to the latest message."
   
   # Run the agent
   response = agent.run(full_prompt)
   
   # Add response to history
   conversation_history.append(f"Assistant: {response}")
   ```

## Available Agent Types

smolagents provides several types of agents:

- **ToolCallingAgent**: A general-purpose agent that can use tools via function calling
- **CodeAgent**: An agent that writes and executes Python code to solve tasks
- **MultiStepAgent**: A base class for agents that solve tasks in multiple steps

For most simple use cases with local LLMs, `ToolCallingAgent` is recommended.

## Notes on Model Performance

- The quality of responses depends on the model you have loaded in LM Studio
- Larger models (7B+ parameters) will generally provide better results
- Some models may not support all features (like function calling for tools)
- If you encounter issues with tools, try using a model that supports OpenAI function calling format
- For best results with tools, use models that have been fine-tuned for function calling

## Recommended Models for LM Studio

For the best experience with these examples, consider using these models in LM Studio:

- Gemma 3 12B Instruct (gemma-3-12b-it)
- Gemma 3 27B Instruct (gemma-3-27b-it)

The Gemma 3 models are highly recommended as they provide excellent function calling capabilities and overall performance. The 27B model is comparable to Gemini 1.5 Pro in quality, while the 12B model offers a good balance between performance and resource requirements.

## Troubleshooting

- **Connection Error**: Make sure LM Studio server is running and the port matches your configuration
- **Poor Responses**: Try loading a different/larger model in LM Studio
- **Tool Execution Issues**: Some local models may not support function calling properly. Try a different model or simplify your prompts.
- **API Errors**: If you get API errors, check the LM Studio logs for more information
- **Import Errors**: Make sure you're using the correct import paths:
  - `from smolagents import OpenAIServerModel, ToolCallingAgent, tool`
  - There is no `Agent` class in smolagents, use `ToolCallingAgent` or `CodeAgent` instead
- **Method Errors**: `ToolCallingAgent` uses `run()` method, not `chat()`:
  - Use `agent.run("Your query")` instead of `agent.chat("Your query")`
- **Parameter Errors**: `ToolCallingAgent` doesn't accept a `system_prompt` parameter. The required parameters are:
  - `name`: A name for the agent
  - `model`: The model to use
  - `tools`: A list of tools (can be empty `[]`)
- **ModuleNotFoundError**: If you get an error about missing 'openai', install the OpenAI dependency:
  - `pip install 'smolagents[openai]'`
- **Tool Creation Errors**: Use the `@tool` decorator pattern to create tools:
  ```python
  @tool
  def my_tool(param: str) -> str:
      """Tool description with Args: section"""
      return result
  ```

smolagents-lmstudio-examples

smolagents-lmstudio-examples Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

smolagents-lmstudio-examples

smolagents-lmstudio-examples Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README