# GPT vs Gemini Test Results

Updated: `2026-05-27T20:32:30.092272+00:00`

## Experiment

- Goal: compare prompt tone on difficult but solvable coding tasks.
- Task set: 6 Exercism Python editing tasks.
- Tones: gentle, neutral, harsh.
- GPT is measured through the `pi` wrapper; `pi` itself is not treated as an independent model family.
- Success criterion: the task counts as solved only if its pytest suite passes after the run.

## GPT via pi

- Note: pi is used only as a wrapper/agent interface around GPT.
- Completed cells: 18
- Passed cells: 18
- Pass rate: 100%
- Mean runtime: 36.4s
- Median runtime: 34.6s

### By Tone

- `gentle`: 6/6 (mean 42.8s, median 41.5s)
- `harsh`: 6/6 (mean 34.2s, median 34.0s)
- `neutral`: 6/6 (mean 32.3s, median 33.5s)

### By Task

- `book-store`: 3/3 (gentle=pass, harsh=pass, neutral=pass)
- `dominoes`: 3/3 (gentle=pass, harsh=pass, neutral=pass)
- `poker`: 3/3 (gentle=pass, harsh=pass, neutral=pass)
- `rational-numbers`: 3/3 (gentle=pass, harsh=pass, neutral=pass)
- `variable-length-quantity`: 3/3 (gentle=pass, harsh=pass, neutral=pass)
- `word-search`: 3/3 (gentle=pass, harsh=pass, neutral=pass)

## Gemini

- Note: Gemini CLI results with the same tasks and tone variants.
- Completed cells: 18
- Passed cells: 18
- Pass rate: 100%
- Mean runtime: 71.6s
- Median runtime: 61.7s

### By Tone

- `gentle`: 6/6 (mean 98.3s, median 69.2s)
- `harsh`: 6/6 (mean 54.3s, median 55.6s)
- `neutral`: 6/6 (mean 62.1s, median 59.3s)

### By Task

- `book-store`: 3/3 (gentle=pass, harsh=pass, neutral=pass)
- `dominoes`: 3/3 (gentle=pass, harsh=pass, neutral=pass)
- `poker`: 3/3 (gentle=pass, harsh=pass, neutral=pass)
- `rational-numbers`: 3/3 (gentle=pass, harsh=pass, neutral=pass)
- `variable-length-quantity`: 3/3 (gentle=pass, harsh=pass, neutral=pass)
- `word-search`: 3/3 (gentle=pass, harsh=pass, neutral=pass)