copilot-harness-compare

KO | EN

gitlite — search

#javascript #ai #typescript #go #machine-learning #rust #deep-learning #macos #kubernetes #vue #pytorch #ios #mcp #open-source

copilot-harness-compare

(★ 9)

No description available.

copilot-harness-compare Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# 🧪 Copilot Harness Compare — Same Model, Different Harness

> *한국어 버전 → [README.ko.md](README.ko.md)*

A small reproducible experiment that asks one question:

**When the *exact same* LLM (Claude Sonnet 4.6) is used through three different agent harnesses, how much does the *harness* itself change the result?**

A "harness" here means everything around the model: system prompt, tool set, context-collection strategy, output streaming, approval policy, prompt caching, etc. This repo gives you a tiny but pathological Snake game (intentionally bugged) and a controlled procedure to make all three environments solve the same task.

---

## TL;DR

1. Open the `_original/` folder. It's a working-but-buggy modular Snake game (~11 small ES modules, JSDoc-annotated).
2. Three identical copies live in `run-1-copilot-default/`, `run-2-copilot-claude/`, `run-3-claude-extension/`.
3. Paste the prompt from [`PROMPT.md`](PROMPT.md) into each environment **verbatim**, with model fixed to **Claude Sonnet 4.6**.
4. Record token usage, total wall-clock time, files read, tools called, approvals requested, and whether the agent actually launched a server to verify.
5. Fill in `run-N-*/RESULT.md`. Compare.

You will almost certainly find that the *same model* behaves very differently across the three environments — sometimes in ways that flip your preference. That's the point.

---

## Why this exists

Mid-2026 the question *"is Copilot or Claude Code better?"* keeps getting muddled because almost everyone is comparing **different models in different harnesses**. This repo isolates the harness variable:

| | Run 1: Copilot (default) | Run 2: Copilot + Claude session | Run 3: Claude VS Code extension |
|---|---|---|---|
| Model | Claude Sonnet 4.6 | Claude Sonnet 4.6 | Claude Sonnet 4.6 |
| System prompt | GitHub/Microsoft | Anthropic (via Copilot) | Anthropic (direct) |
| Tool set | VS Code built-in tools (workspace index, partial edits) | VS Code tools + some Anthropic glue | Anthropic's own (Glob, Grep, Read, Edit, Bash) |
| Context strategy | Pre-built workspace index | Mostly index-based | Live shell exploration |
| Output style | Inline diffs via partial-edit tools | Mixed | Often full-file rewrites |
| Approval policy | Default approvals | Default approvals | Default approvals |

Same engine, three different chassis.

---

## The task

The Snake game is split across **11 small modules** so that:

- Bug *symptoms* (HUD never updates; arrow keys feel wrong) appear in obvious places (`hud.js`, `input.js`)
- Bug *root causes* are in completely different files (`config.js`, `direction.js`)

A shallow agent will patch the symptom. A thorough agent will trace back to the root cause. That gap is what we want to measure.

Number of bugs and their nature: see [`BUGS_ANSWER_KEY.md`](BUGS_ANSWER_KEY.md) (⚠️ spoilers — don't open until after you run the experiment).

---

## Repository layout

```
copilot-harness-compare/
├── README.md                    ← this file (English)
├── README.ko.md                 ← Korean version
├── LICENSE                      ← MIT
├── PROMPT.md                    ← the verbatim prompt to paste into each agent
├── RESULTS_TEMPLATE.md          ← copy this into each run-*/RESULT.md
├── BUGS_ANSWER_KEY.md           ← 🔒 spoiler-wrapped answer key
├── _original/                   ← reference copy (do not edit)
│   ├── index.html · style.css
│   └── js/  (config, direction, engine, food, events, hud, input, loop, main, renderer, state)
├── run-1-copilot-default/       ← identical copy + RESULT.md
├── run-2-copilot-claude/        ← identical copy + RESULT.md
└── run-3-claude-extension/      ← identical copy + RESULT.md
```

---

## How to run the experiment

### Prerequisites

- VS Code (latest stable, 2026-05+ recommended for v1.121 Claude-session support)
- A GitHub Copilot plan that includes Claude models (Pro or higher)
- Anthropic's *Claude for VS Code* extension installed and authenticated (for Run 3)
- `python3` for the local static server

### Step 1 — Hide the spoilers

Before opening any `run-N-*/` folder in VS Code, make sure the agent **cannot read `BUGS_ANSWER_KEY.md` or `_original/`**:

- **Recommended**: open each `run-N-*/` as its **own VS Code window** (File → New Window → open just that folder). The agent's workspace context is then scoped to that folder only.
- Or add a workspace-level `.copilotignore` that excludes the sibling paths.
- Or move `BUGS_ANSWER_KEY.md` out of the repo while running.

### Step 2 — Verify the starting state

```bash
cd copilot-harness-compare
for d in run-1-copilot-default run-2-copilot-claude run-3-claude-extension; do
  diff -rq _original "$d" | grep -v RESULT.md
done
# (should produce no output → source identical to _original)
```

### Step 3 — Run

For each of the three environments, in order:

1. Open `run-N-*/` in VS Code (own window).
2. Open the chat panel. Switch to **Agent mode**, **Default approvals** (no Autopilot, no Bypass).
3. Select **Claude Sonnet 4.6** as the model. For Run 2, also pick *Session Type → Claude*.
4. Note the start time (`date +%T` or a stopwatch).
5. Paste the **entire fenced block from `PROMPT.md`** into the chat. Send.
6. Do not type anything else until the agent declares it's done. If it asks you a question, answer only "계속해" / "continue" and note the stoppage in `RESULT.md`.
7. When the agent reports completion, note the end time and copy its token-usage / time block.
8. Open `http://localhost:8765/` (or whatever port the agent chose) and verify:
   - Eating food → Score increments.
   - `←` arrow → snake moves left. `→` arrow → snake moves right.
9. Fill in `run-N-*/RESULT.md` completely.

### Step 4 — Reset between runs (optional)

If you want to re-run a single environment from scratch:

```bash
cd copilot-harness-compare
rm -rf run-1-copilot-default
mkdir run-1-copilot-default
cp -R _original/. run-1-copilot-default/
cp RESULTS_TEMPLATE.md run-1-copilot-default/RESULT.md
```

### Step 5 — Compare

Once all three `RESULT.md` files are filled, write a short `COMPARISON.md` of your own. Suggested headings:

- **Speed**: which environment finished fastest? Wall clock vs net agent time.
- **Tokens / Cost**: at Sonnet 4.6 pricing ($3/M input, $15/M output), what did each run cost you?
- **Files read**: out of 11 modules, how many did each agent actually open?
- **Root cause vs symptom patch**: did the agent edit `config.js` / `direction.js` (correct) or hack around it in `engine.js` / `input.js` (symptom patch)?
- **Verification habit**: did the agent actually launch the server and try the game?
- **Approvals**: how many times did it stop to ask you?
- **Surprise findings**: anything unexpected (over-eager refactors, added test files, style changes, etc.)?

---

## Sharing results

If you run this, please file an issue or PR with your three filled `RESULT.md` files (or your own `COMPARISON.md` summary). Patterns across many machines / setups are interesting data.

Tag in commit message or issue title:
- VS Code version, OS
- Copilot plan tier
- Claude extension version
- Approximate region (latency matters)

---

## Limitations & caveats

- **Single task, single domain.** A small JS bug-hunt is biased toward harnesses with strong workspace indexing. Different tasks (build/test loops, large refactors, greenfield scaffolding) would reveal different strengths.
- **Token counts are approximate.** Different harnesses report tokens differently; do your best, document the method in `RESULT.md`.
- **Network and time-of-day affect latency** more than people realize. Run all three within the same hour if possible.
- **The "answer key" is a particular author's idea of root cause.** If an agent fixes the symptom in a way that's actually cleaner, document it; don't auto-deduct.

---

## Acknowledgements

Inspired by community discussions during 2026's VS Code multi-agent transition. The "harness vs model" framing is well-trodden; this is just a small reproducible bench.

## License

MIT — see [`LICENSE`](LICENSE).

copilot-harness-compare

copilot-harness-compare Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

copilot-harness-compare

copilot-harness-compare Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README