llm-council-ranking-members

KO | EN

gitlite — search

#javascript #python #hacktoberfest #react #ai #typescript #llm #go #golang #android #machine-learning #rust #deep-learning #linux

llm-council-ranking-members

★ 9 Open GitHub ↗

The LLM Council with focus on Ranking Members using Elo-rating like models + user able to participate in responses and rankings (LLM Council based on Karpaty's repo)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# LLM Council Ranking Members

> **IMPORTANT**: This repo is based on [Andrej Karpathy's llm-council](https://github.com/karpathy/llm-council) -- see original README below (see [discussion here](https://github.com/orgs/community/discussions/11729) why I opted against a proper fork). The original repository does not yet have a LICENSE file. A license has been requested in [issue #58](https://github.com/karpathy/llm-council/issues/58). Once a license is added to the original repo, it will be added here as well.

![llmcouncilrankingmembers](https://raw.githubusercontent.com/gmgeorg/llm-council-ranking-members/main/images/llm_council_ranking_members_logo_gemini.png)

<sub>*Header image generated with Google Gemini*</sub>

**Why "Ranking Members"?**

This adaptation of LLM council focuses specifically on **principled ranking algorithms** for aggregating model evaluations and optional human-in-the-loop council members. Instead of simple vote counting or average positions, this repo uses the **Plackett-Luce model** (a probabilistic ranking algorithm from choice theory, akin Elo ratings) to properly weight responses based on peer evaluations. The "Ranking Members" suffix distinguishes this implementation's emphasis on ranking methodology and you -- the user -- becoming a ranking member as well and add your rankings as well -- pairwise rankings only or partial rankings are supported as well!

> **Note:** This is a modified and extended version of [the original LLM council repo](https://github.com/karpathy/llm-council). Added capabilities include improved UX / UI including responsive visualizations, Plackett-Luce rankings (like Elo ratings for **ranking members**), human-in-the-loop responses & rankings, and more.

## 🤝 Human-in-the-Loop Council Participation

**You're not just an observer—you're a council member!** This implementation allows humans to fully participate in the deliberation process:

- **Join as a Responder (Stage 1)**: Add your own answer alongside the AI models
- **Join as an Evaluator (Stage 2)**: Rank and evaluate all responses (including your own, blindly)
- **Equal Treatment**: Your responses are anonymized and evaluated by AI models just like any other council member
- **Test Your Reasoning**: See how your answers and evaluations compare to frontier AI models

A 3-stage deliberation system where multiple LLMs (and humans!) collaboratively answer questions through anonymous peer review.

## Quick Start

1. **Install dependencies:**

```bash
   uv sync
   cd frontend && npm install && cd ..
   ```

2. **Configure OpenRouter API key** in `.env`:

```bash
   OPENROUTER_API_KEY=sk-or-v1-...
   ```

3. **Run the app:**

```bash
   ./start.sh
   ```

Or manually: `uv run python -m backend.main` (backend) + `cd frontend && npm run dev` (frontend)

4. **Open** [http://localhost:5173](http://localhost:5173)

## How It Works

**Stage 1 (Collect):** Multiple LLMs answer your question in parallel. Responses shown in tabs.

**Stage 2 (Review):** Each LLM ranks all responses **anonymously** (prevents bias). Human users can also submit rankings via pairwise comparisons or full rankings. All rankings aggregated using Plackett-Luce model (similar to Elo ratings).

**Stage 3 (Synthesize):** A chair LLM creates the final answer **weighted by PL scores**—higher-ranked responses contribute proportionally more to the synthesis.

## Key Features

### 🎯 Anonymous Peer Review

The core innovation: LLMs evaluate each other's responses **without knowing whose response is whose**, preventing model bias and favoritism.

### 📊 Advanced Visualizations

![response-similarity](https://raw.githubusercontent.com/gmgeorg/llm-council-ranking-members/main/images/response_similarity.png)

- **Response Similarity Heatmap:** Auto-calculated embeddings (BAAI/bge-small-en-v1.5) show which models think alike
- **Chord Diagram:** Circular visualization of pairwise response similarities
- **Network Graph:** Force-directed layout revealing response clustering
- **Normalization Toggle:** Switch between raw and projected embeddings to highlight true differences vs. shared boilerplate

### 🏆 Learned Rankings for all participating council members (akin to Elo ratings)

Using a principled ranking algorithm using the **Plackett-Luce model**, closely related to Elo ratings (`choix` library).

![llm-rankings](https://raw.githubusercontent.com/gmgeorg/llm-council-ranking-members/main/images/llm_ranking.png)

- Calculates strength parameters and win probabilities for each model based on the pairwise or partial rankings from the council (and you the user if you want to participate)
- More robust than simple vote counting or average positions, and allows for partial rankings / pairwise comparisons
- Scores directly influence Stage 3 synthesis—chair weights responses proportionally by their PL scores

### 🎨 Modern UI/UX

- **Light/Dark Mode:** Clean interface with theme toggle (🌙/☀️)
- **Responsive Design:** Works seamlessly across screen sizes
- **Markdown Rendering:**  Improved markdown formatting for LLM responses
- **Tab Interface:** Easily inspect individual model outputs
- **Delete Conversations:** Ability to delete conversations from UI.
- **Configure Council Members**: Ability to configure / select which models are part of council and council chair.  Remembers choices over multiple sessions and also allows users to participate as a council member.

### 🤝 Human-in-the-Loop

- **Add Your Response (Stage 1):** Contribute your own answer alongside the LLMs
- **Submit Your Ranking (Stage 2):** Rank responses via pairwise comparisons or full ranking with optional rationale
- Human input included in Plackett-Luce calculations and shown to chair
- Test how your reasoning compares to frontier AI models

### ⚡ Dynamic & Flexible

- **Add Council Members On-the-Fly:** After seeing initial responses, add more models to the discussion
- **Model Selection UI:** Choose any combination from OpenRouter's catalog (free and paid models)
- **Preserved Preferences:** Your model selections saved to localStorage
- **Conversation History:** All sessions stored locally with full context

## Tech Stack

**Backend:**

- **FastAPI** - Modern async Python web framework
- **OpenRouter API** - Multi-model LLM gateway
- **FastEmbed** - Fast embeddings using various embedding options for response similarity analysis
- **Choix** - Probabilistic ranking with Plackett-Luce model (like Elo ratings)
- **JSON files** - Conversation storage in `data/conversations/`

**Frontend:**

- **React + Vite** - Fast, modern web development
- **Nivo** - Interactive charts (heatmaps, chord diagrams, etc.)
- **react-markdown** - Markdown rendering for LLM responses
- **remarkGfm** - GitHub Flavored Markdown support

**Package Management:** uv for Python, npm for JavaScript

---

## Vibe Code Alert 2.0

This extended version was coded up on a rainy weekend—exploring ranking algorithms, UX quality of life improvements, embedding visualizations, and human-in-the-loop deliberation. Like the original, it's provided **as-is with no guarantees or warranties** (akin to MIT LICENSE -- see `LICENSE` disclaimer above though!).

Use it for inspiration until the `LICENSE` disclaimer above is resolved.

---

## Original README (by [Karpathy](https://github.com/karpathy/llm-council))

![llmcouncil](https://raw.githubusercontent.com/gmgeorg/llm-council-ranking-members/main/header.jpg)

The idea of this repo is that instead of asking a question to your favorite LLM provider (e.g. OpenAI GPT 5.1, Google Gemini 3.0 Pro, Anthropic Claude Sonnet 4.5, xAI Grok 4, eg.c), you can group them into your "LLM Council Ranking Members". This repo is a simple, local web app that essentially looks like ChatGPT except it uses OpenRouter to send your query to multiple LLMs, it then asks them to review and rank each other's work, and finally a chair LLM produces the final response.

In a bit more detail, here is what happens when you submit a query:

1. **Stage 1: First opinions**. The user query is given to all LLMs individually, and the responses are collected. The individual responses are shown in a "tab view", so that the user can inspect them all one by one.
2. **Stage 2: Review**. Each individual LLM is given the responses of the other LLMs. Under the hood, the LLM identities are anonymized so that the LLM can't play favorites when judging their outputs. The LLM is asked to rank them in accuracy and insight.
3. **Stage 3: Final response**. The designated chair of the LLM Council Ranking Members takes all of the model's responses and compiles them into a single final answer that is presented to the user.

## Vibe Code Alert

This project was 99% vibe coded as a fun Saturday hack because I wanted to explore and evaluate a number of LLMs side by side in the process of [reading books together with LLMs](https://x.com/karpathy/status/1990577951671509438). It's nice and useful to see multiple responses side by side, and also the cross-opinions of all LLMs on each other's outputs. I'm not going to support it in any way, it's provided here as is for other people's inspiration and I don't intend to improve it. Code is ephemeral now and libraries are over, ask your LLM to change it in whatever way you like.

## Setup

### 1. Install Dependencies

The project uses [uv](https://docs.astral.sh/uv/) for project management.

**Backend:**

```bash
uv sync
```

**Frontend:**

```bash
cd frontend
npm install
cd ..
```

### 2. Configure API Key

Create a `.env` file in the project root:

```bash
OPENROUTER_API_KEY=sk-or-v1-...
```

Get your API key at [openrouter.ai](https://openrouter.ai/). Make sure to purchase the credits you need, or sign up for automatic top up.

### 3. Configure Models (Optional)

Edit `backend/config.py` to customize the council:

```python
COUNCIL_MODELS = [
    "openai/gpt-5.1",
    "google/gemini-3-pro-preview",
    "anthropic/claude-sonnet-4.5",
    "x-ai/grok-4",
]

CHAIR_MODEL = "google/gemini-3-pro-preview"
```

## Running the Application

**Option 1: Use the start script**

```bash
./start.sh
```

**Option 2: Run manually**

Terminal 1 (Backend):

```bash
uv run python -m backend.main
```

Terminal 2 (Frontend):

```bash
cd frontend
npm run dev
```

Then open <http://localhost:5173> in your browser.

## Tech Stack (v0)

- **Backend:** FastAPI (Python 3.10+), async httpx, OpenRouter API
- **Frontend:** React + Vite, react-markdown for rendering
- **Storage:** JSON files in `data/conversations/`
- **Package Management:** uv for Python, npm for JavaScript

llm-council-ranking-members

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README