py-deepeval-behave-bdd-testing-example

KO | EN

gitlite — search

#javascript #python #hacktoberfest #react #ai #typescript #llm #go #golang #android #machine-learning #rust #deep-learning #linux

py-deepeval-behave-bdd-testing-example

★ 11 Open GitHub ↗

An example that combines Behave (BDD testing) with DeepEval (LLM evaluation) to create human-readable, stakeholder-friendly tests for AI Agents / chatbots.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# BDD Testing for AI Agents / Chatbots with Behave and DeepEval

[![BDD Chatbot Tests](https://github.com/messeb/py-deepeval-behave-bdd-testing-example/actions/workflows/bdd-tests.yml/badge.svg)](https://github.com/messeb/py-deepeval-behave-bdd-testing-example/actions/workflows/bdd-tests.yml)

> 🎤 Presented at **WeAreDevelopers 2026** on 10.07.2026 — [View the slides on Speakerdeck](https://speakerdeck.com/messeb/testing-ai-agents-automated-evaluation-for-chatbots-and-rag-systems)

<a href="https://to.messeb.com/contact"><img src="http://github.messeb.com.s3.amazonaws.com/assets/messeb-linkedin-btn.png" width="120" height="34" style="max-width:100%;"></a> <br>
Contact me on LinkedIn: [@messingfeld](https://to.messeb.com/contact)

An example that combines [Behave](https://behave.readthedocs.io/) (BDD testing) with [DeepEval](https://docs.confident-ai.com/) (LLM evaluation) to create human-readable, stakeholder-friendly tests for AI Agents / chatbots.

## The Challenge

AI Agents and Chatbot validation remains an open challenge in many projects, especially when dealing with self-developed solutions. How do you ensure your chatbot responds correctly? How do you catch regressions when updating prompts or models?

Validation frameworks like [DeepEval](https://docs.confident-ai.com/) do an excellent job evaluating LLM outputs with metrics like correctness, relevancy, and hallucination detection. But there's still a gap: **the bridge between developers and stakeholders**.

This is where **BDD (Behavior-Driven Development)** shines. By describing conversations and expected outputs in plain, human-readable language, both developers and stakeholders can:
- Understand what the chatbot should do
- Define acceptance criteria together
- Review test results without reading code

## What is BDD with Gherkin?

**BDD (Behavior-Driven Development)** is a collaborative approach where tests are written in natural language that everyone can understand. **Gherkin** is the syntax used to write these tests:

```gherkin
Feature: Movie Recommendations

Scenario: User asks for a sci-fi movie
    Given the movie critic chatbot is initialized
    When I say "I'm in the mood for a sci-fi movie"
    Then the response should contain a question
    When I continue with "Something thought-provoking"
    Then the response should mention one of "Arrival, Ex Machina, Blade Runner"
```

This format uses **Given-When-Then** steps:
- **Given**: Set up the initial context
- **When**: Perform an action (send a message)
- **Then**: Verify the expected outcome

Stakeholders can read, review, and even write these scenarios without any programming knowledge.

## What is DeepEval?

**DeepEval** is an evaluation framework for LLM applications. Instead of simple string matching, it uses AI to evaluate responses with metrics like:

| Metric | What it measures |
|--------|------------------|
| **Correctness** | Is the response factually accurate given the context? |
| **Answer Relevancy** | Does the response actually address the question? |
| **Hallucination** | Did the model make things up? |
| **Coherence** | Does the response logically follow the conversation? |

This project combines Gherkin's readability with DeepEval's intelligent validation:

```gherkin
Then the response should be evaluated with correctness score of at least 0.7
And the response should be evaluated with answer relevancy score of at least 0.8
```

## Features

- **BDD Testing**: Write tests in Gherkin syntax (Given-When-Then)
- **Single-shot Tests**: Test individual prompt/response interactions
- **Conversation Tests**: Test multi-turn dialogue flows
- **DeepEval Metrics**: Evaluate responses with Correctness, Answer Relevancy, Coherence, and more
- **Stakeholder-friendly**: Test expectations defined in Gherkin files, not code
- **JUnit Reports**: Full Q&A and evaluation results in test reports

## Quick Start

### How It Works

This example uses a simple OpenAI Python chat client with a customizable [instruction prompt](src/chatbot/prompts/movie_critic.md) (a movie critic persona called "Cinemax"). The client supports both single-shot queries and multi-turn conversations.

**The integration works like this:**

1. **Behave** reads the Gherkin feature files and executes the test steps
2. **Step definitions** call the OpenAI chat client to send messages and receive responses
3. **DeepEval** evaluates the responses using LLM-based metrics
4. **Results** are captured in JUnit XML reports with full Q&A transcripts

```
┌─────────────────┐     ┌─────────────────┐     ┌─────────────────┐
│  Gherkin Files  │────▶│  Behave Steps   │────▶│  OpenAI Client  │
│  (.feature)     │     │  (Python)       │     │  (Chat API)     │
└─────────────────┘     └────────┬────────┘     └─────────────────┘
                                 │
                                 ▼
                        ┌─────────────────┐
                        │    DeepEval     │
                        │   (Metrics)     │
                        └─────────────────┘
```

### Continuous Validation with GitHub Actions

[![BDD Chatbot Tests](https://github.com/messeb/py-deepeval-behave-bdd-testing-example/actions/workflows/bdd-tests.yml/badge.svg)](https://github.com/messeb/py-deepeval-behave-bdd-testing-example/actions/workflows/bdd-tests.yml)

The included [GitHub Actions workflow](.github/workflows/bdd-tests.yml) enables continuous validation of your chatbot:

- **Automated testing** on every push and pull request
- **Scheduled runs** to catch model drift or API changes
- **JUnit reports** with detailed Q&A transcripts and evaluation scores
- **Manual triggers** for on-demand validation

This ensures your chatbot behavior is validated regularly, not just during development.

### Prerequisites

- Python 3.12+
- [uv](https://docs.astral.sh/uv/) package manager
- OpenAI API key

### Installation

```bash
# Install dependencies
make install

# Copy environment template and add your API key
cp .env.example .env
# Edit .env and add your OPENAI_API_KEY
```

### Running Tests

```bash
# Run all tests
make test

# Run with verbose output
make test-verbose

# Run specific feature
make test-single        # Single-shot tests only
make test-conversation  # Conversation tests only
```

### Interactive Chat

```bash
# Start interactive chat with the movie critic bot
make chat

# Single-shot query
uv run python -m chatbot --single "What's a good sci-fi movie?"
```

## Project Structure

```
.
├── features/
│   ├── single_shot.feature      # Single-shot test scenarios
│   ├── conversation.feature     # Multi-turn conversation tests
│   ├── environment.py           # Behave hooks
│   └── steps/
│       ├── chatbot_steps.py     # Chatbot interaction steps
│       └── validation_steps.py  # DeepEval validation steps
├── src/chatbot/
│   ├── client.py                # OpenAI client wrapper
│   ├── config.py                # Configuration management
│   ├── cli.py                   # CLI interface
│   └── prompts/
│       └── movie_critic.md      # Chatbot system prompt
├── pyproject.toml               # Project dependencies
├── Makefile                     # Task automation
└── .github/workflows/
    └── bdd-tests.yml            # CI/CD pipeline
```

## Writing Tests

### Gherkin Feature Files

```gherkin
Feature: Movie Recommendations

Scenario: Recommend a film noir
    Given the movie critic chatbot is initialized
    Given the chatbot has the following context
      """
      Film Noir Classics:
      - "Double Indemnity" (1944)
      - "The Maltese Falcon" (1941)
      """
    When I ask "What's a good film noir?"
    Then the response should mention one of "Double Indemnity, Maltese Falcon"
    And the response should be evaluated with correctness score of at least 0.7
```

### Available Steps

**Given:**
- `the movie critic chatbot is initialized`
- `a new conversation is started`
- `the chatbot has the following context` (with docstring)

**When:**
- `I ask "{prompt}"` - Single-shot query
- `I say "{message}"` - Start conversation
- `I continue with "{message}"` - Continue conversation

**Then (Assertions):**
- `the response should mention one of "{keywords}"`
- `the response should mention all of "{keywords}"`
- `the response should contain a question`
- `the response should be at least {n} characters`

**Then (DeepEval):**
- `the response should be evaluated with correctness score of at least {threshold}`
- `the response should be evaluated with answer relevancy score of at least {threshold}`
- `the response should be evaluated with coherence score of at least {threshold}`

## DeepEval Metrics

| Metric | Description |
|--------|-------------|
| Correctness (GEval) | Evaluates factual accuracy against expected output and context |
| Answer Relevancy | Measures if the response addresses the question |
| Coherence | Evaluates logical flow in conversations |
| Faithfulness | Checks consistency with provided context |

## Configuration

### Environment Variables

| Variable | Description | Default |
|----------|-------------|---------|
| `OPENAI_API_KEY` | OpenAI API key | Required |
| `OPENAI_MODEL` | Model to use | `gpt-4o-mini` |
| `LOG_LEVEL` | Logging level | `INFO` |

## License

MIT

py-deepeval-behave-bdd-testing-example

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README