When Bigger AI Models and High Thinking Modes Score Worse

When you start building an AI feature, the most common advice is to pick the largest flagship model available and turn its reasoning setting to High.

That works well for open-ended math, coding puzzles, or deep research. Most everyday software, though, asks AI to do a quieter job: fill in a structured template accurately, follow clear rules, suggest genuinely different options, and sound like a normal person.

To see which models handle that best, I ran over 500 controlled tests across Google Gemini (3.1 Pro, 3.7 Flash, 3.8 Flash, 4 Argon), OpenAI (GPT-6.1 Sol, GPT-6 Astra), Anthropic (Claude 5 Sonnet, Claude 5 Opus), and a quantized 8B open-weights model running locally.

On these structured tasks, larger models and High thinking modes frequently scored lower than smaller models on Default settings, while taking 1.7x to 3.4x longer to respond and using 5.2x to 6.5x more internal thinking tokens.

1. How I Evaluated 500+ Runs

When teams evaluate AI outputs in tools like Langfuse or Promptfoo, a common pattern is LLM-as-a-Judge: asking one AI to score another AI’s answer from 1 to 5. In my tests, that approach ran into two practical problems:

1. Shared blind spots for stiff phrasing: Because the judge is also an AI, it rewards formal-sounding language. When a model turns “spend less when bored” into “Discretionary capital deployment is entirely intentional”, an AI judge gives it a 5 out of 5, even though no person talks that way.
2. Slow, expensive re-grading: Every time you refine a check, re-grading all 12 model configurations means running thousands of new paid judge calls across old outputs.

Instead, I cached every model’s structured response locally so automated checks can re-score all 500+ runs in seconds with zero API calls, followed by a blind human read:

• Step 1: Automated Rule Checks: Deterministic code checks verify whether the model classified the request accurately, filled every required field in the schema, and respected negative and formatting constraints.
• Step 2: Option Variety Check: An automated semantic similarity check measures whether multiple suggested options are genuinely distinct behaviors rather than the same idea reworded multiple ways.
• Step 3: Blind “Read It Out Loud” Check: Once top models pass 95% to 100% of mechanical checks, automated scores stop separating them. I hide the model names at the bottom of a page, shuffle the outputs, and score each from 1 to 5 on a simple question: “Would a real person actually say this out loud?“

2. What the Blind Voice Check Caught

Hiding the model names showed why several flagship models and High thinking runs felt off despite high mechanical scores. They often wrote about everyday topics in a clinical or management-consultant tone:

• Everyday request: “Get fit without burning out”
  • Flagship / High thinking output: “Can drive systemic hypertrophy using only spatial resistance” and “Possess the movement durability to absorb harder physical demands”
• Everyday request: “Stop feeling wrecked every afternoon”
  • Flagship / High thinking output: “Can sustain optimal cellular energy output through the entire afternoon”
• Everyday request: “Stop buying things I do not need when I am bored”
  • High thinking output: “Discretionary capital deployment is entirely intentional”
  • 3.7 Flash Default output: “Spend only on what was planned in advance”

Comparing Default and High thinking side-by-side with the model labels hidden flipped the leaderboard:

• Pairwise blind A/B read: Default thinking won every decided comparison without a single loss, while using 35% fewer total tokens and 47% fewer thinking tokens.
• 3-way blind read (3.7 Flash Default vs. 3.7 Flash High vs. 3.8 Flash Default, with labels A, B, and C reshuffled on every prompt): 3.7 Flash Default ranked first three times as often as High thinking, despite High using 2.1x the thinking tokens and taking 15% longer.

3. Consolidated Benchmark Results

Across OpenAI, Google, and Anthropic, adding extra chain-of-thought budget or moving to the largest flagship tier on a structured task consistently increased wait time and token use without improving task accuracy:

Looking at median response time across the same 12 configurations shows the latency trade-off of turning reasoning to High:

(Note: Because Claude 5 Sonnet, Claude 5 Opus, and Gemini 3.1 Pro already trailed Gemini 3.7 Flash on both accuracy and response time in standard mode, I did not run separate High extended-thinking sweeps on those three arms.)

Three Patterns Behind the Numbers

1. Over-analysis on simple rules: On straightforward requests, smaller Default models followed the instructions directly. Given a large thinking budget, High thinking modes across both Gemini and GPT-6 frequently reasoned themselves into over-complicating simple categories or repeating the same option in slightly different words.
2. Where High thinking tokens go: Across High thinking runs, prompt inputs accounted for 12% of token spend, visible JSON answers accounted for 11%, and hidden thinking tokens accounted for 77% of total spend (87% of billed output tokens). Users wait up to two and a half minutes (162.6s) for an internal monologue they never see.
3. Latency on interactive prompts: Even when two models both pass 100% of mechanical checks on shorter interactive prompts, a lighter Default model can respond in half the time (~10s vs. ~21s) while producing wider variety across its suggestions.

4. Practical Takeaways

1. Let your schema handle structure: When a strict JSON schema and code checks already enforce the output format, you rarely need a flagship model or High reasoning mode for structural reliability.
2. Combine fast code checks with a small blind human read: Caching raw outputs locally lets you re-run automated checks across hundreds of past runs in seconds for free, keeping human review focused on tone.
3. Ask “Would a real person say this out loud?”: A quick blind voice check catches stiff consultant phrasing that automated checks and AI judges miss.
4. Measure thinking tokens separately from visible output: When hidden thinking accounts for 77% of spend (87% of billed output tokens), using Default reasoning on constrained tasks cuts token usage by 35% to 80% and wait times by 1.7x to 3.4x.
5. Benchmark before turning reasoning to High: Across OpenAI, Google, and Anthropic, extra reasoning budget on structured tasks often led to second-guessing simple rules, narrower option variety, or stiffer phrasing.