Where confidence falls away from competence

I know HVAC now

With all the confidence of Google Gemini, even though I know nothing about HVAC, I was able to ask about the cost efficiency tradeoffs between a split system and the replacement of my old one. Yes, I still needed the help of the expert but I was, thanks to AI, able to encroach ever so slightly onto their turf.

It's one thing when it's HVAC. It's another when it's the human body.

What's the line that we ourselves and our agents need to draw when we can sound like we know what we're talking about with a doctor or a teacher?

How does it impact a teacher to feel like they're in conversation with a parent who's able to relate to them like a peer when, in fact, that person has no background in pedagogy?

AI made errors more convincing

The published Jagged Frontier study compared 758 BCG consultants working with and without AI. The AI groups used an April 2023 GPT-4 chat interface.

In the 385-person product-development experiment, AI users completed 12.2% more of 18 tasks, worked 25.1% faster and scored 29.9%–33.9% higher on human-rated quality.

A separate 373-person judgment task combined financial data and interview evidence.

The AI groups averaged 19 percentage points below the control group. Graders rated AI-assisted answers higher for clarity, persuasiveness and coherence, even when they were wrong.

This was one deliberately challenging task in a chat tool, not a universal failure rate or a test of autonomous agents. These figures come from the 2026 published paper.

Our inference for marketing teams: evaluate correctness separately from polish.