prompt-brewery

KO | EN

gitlite — search

#python #docker #machine-learning #android #swift #nodejs #tensorflow #data-science #ios #boilerplate #numpy #gitlab-ci

★ 54 Open GitHub ↗

Prompt Brewery

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Prompt Brewery

[![Python 3.10](https://img.shields.io/badge/python-3.10+-blue.svg)](https://www.python.org/downloads/)
[![Jinja2 3.0.0](https://img.shields.io/badge/jinja-3.0.0-white.svg?logo=jinja&logoColor=black)](https://libraries.io/pypi/jinja2)
[![Gradio 4.0.0](https://img.shields.io/badge/GRADIO-4.0.0-blue)](https://libraries.io/pypi/gradio)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)

Try Prompt Brewery: [promptbrewery.inworld.ai](https://promptbrewery.inworld.ai)

## Disclaimer

[Prompt Brewery](https://promptbrewery.inworld.ai) is a static analysis tool for prompt templates, specifically designed for chat and prompt templates used with Large Language Models (LLMs).

With **50+** analysis [rules](https://github.com/inworld-ai/prompt-brewery/blob/main/RULES.md) tested across **800+** real-world and synthetic templates, Prompt Brewery helps identify potential issues, structural problems, and optimization opportunities in your templates.

## Prompt Engineering Challenges

Building successful AI applications hinges on meticulous [prompt engineering](https://x.com/karpathy/status/1937909397180796982).

Jinja has emerged as the de facto standard for templating these prompts, used in many prominent models on Hugging Face from leading providers.

However, Jinja's flexibility comes at a cost: it's notoriously **difficult to debug, test, and maintain**.

- **Unpredictable failures**: Without explicit typing, proper exception handling, or unit testing tools—combined with [multiple rendering engines without cross-compatibility](https://jbmoelker.github.io/jinja-compat-tests/)—you never know on which input your template will crash. 
- **Production debugging**: A simple syntax error might only surface with specific, unpredictable user inputs, leaving you debugging broken templates in production.

## Approach

Drawing from our experience powering 100M+ daily sessions in gaming, media, and apps like social platforms and learning tools, we built this static analysis tool to bring reliability and predictability to prompt engineering.

We use **static analysis and [basic fuzzing techniques](https://www.fuzzingbook.org/) to create comprehensive test suites for LLM templates**.

Static analysis cannot guarantee all cases, so we tested this tool on:
- **400+** real templates we collected (different use-cases, LLM evaluation templates, open-sourced templates)
- **Another 400+** generated synthetic tests to cover edge cases

However, if your template doesn't work, please [report](https://github.com/inworld-ai/prompt-brewery/issues) it as an issue so that we can add support for such cases.

## Installation

### Local Installation

```bash
pip install -r requirements.txt
```

### Docker Installation (Recommended)

For a containerized setup that doesn't require local Python installation:

```bash
# Quick start with web interface
docker-compose up --build

# Then open http://localhost:7860 in your browser
```

## Usage

### Command Line Interface

For quick analysis of template files, use the CLI tool:

```bash
# Basic usage
python pbrewery_cli.py template.jinja

# Output in JSON format
python pbrewery_cli.py template.jinja --json

# Save report to file
python pbrewery_cli.py template.jinja --output report.json
```

The CLI provides the same analysis as the Python API but with convenient command-line options for file processing and output formatting.

### Web Interface

Run the Gradio-based tool on your local machine:

```bash
python pbrewery_ui.py
```

Then open http://127.0.0.1:7860 in your browser.

### Python API

If you're familiar with Python, you can run it in your tests or anywhere else:

```python
import prompt_brewery as pb

# Assuming you have your template code in a tpl variable
report = pb.static_template_analysis(tpl)

# Now you have report['errors'], report['warnings'], report['info']
for error in report['errors']:
    print(f"ERROR: {error}")

for warning in report['warnings']:
    print(f"WARNING: {warning}")

for info in report['info']:
    print(f"INFO: {info}")
```

## Analysis Categories

🔴 **Errors**: Critical issues that prevent template rendering or indicate serious problems

🟡 **Warnings**: Potential issues that should be reviewed, such as:
- Caching inefficiencies
- Redundant content
- Unbalanced tags
- Formatting inconsistencies

📘 **Info**: General insights about template structure:
- Template statistics
- Chat format detection
- Tool usage patterns
- Required functions and methods

## Example Templates

The `demo_templates/` folder contains example templates including:
- Consumer applications (workout recommendations, travel planning, recipe creation)
- Chat message formats (Qwen and Gemma)
- Tool usage and function calling
- Character prompts and roleplay scenarios
- Various prompt engineering patterns

## Contributing

This tool is designed to help prompt engineers and LLM developers create better, more maintainable templates. Contributions and feedback are welcome!

Please see our contributing guidelines:

1. **Fork the repository**
2. **Create a feature branch**: `git checkout -b feature/amazing-feature`
3. **Commit changes**: `git commit -m 'Add amazing feature'`
4. **Push to branch**: `git push origin feature/amazing-feature`
5. **Open a Pull Request**

## Support

- **Bug Reports**: [GitHub Issues](https://github.com/inworld-ai/prompt-brewery/issues)
- **General Questions**: For general inquiries and support, please [email us](mailto:opensource@inworld.ai)

prompt-brewery

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README