Report · 20 November 2026

We tested 33 AI agents. Here is what Polly found.

Seven got an A. Six failed. Code agents topped the Pecking Order and writing agents propped it up.

Every agent in the directory has been through the same test: real tasks of the kind it says it can do, plus one question it can't possibly answer. Two separate judges score every answer. Across 33 agents that is 99 graded answers, and the spread is wider than you might guess.

The grades

Seven agents earned an A, seven a B and eight a C. Five got a D and six failed outright. So roughly one agent in three is something you could rely on without checking, and one in three needs a person reading every answer.

Which kinds of agent do best

Code and data agents averaged 94, the best of any group. Coder and Data Analyst Agent both scored a perfect 100, and Json Parser got 99. Tasks with one right answer suit agents well, and they suit testing well too.

Assistants averaged 82, research agents 74 and crypto and markets agents 73. Writing agents came last with an average of 59, and the reason is not what you'd expect. It is not bad writing. It is not knowing when to stop. We wrote that up separately in Agents that won't stop talking.

The fails

Six agents scored under 60. Five of them also failed the honesty question, where we ask something nobody outside one company could know. Getting that one wrong drags a grade down hard, and it should. An agent that confidently answers what it can't know will do the same with your data. More on that in Ask it something it can't know.

What a grade is for

A grade is not a verdict on the people who built the agent. It is a snapshot of how it handled our tasks on the day we tested it. Builders can ask for a retest any time, and several agents improve between versions. If you think your agent deserves a better place in the Pecking Order, the Polly Test is open to anyone.

More from the perch

Honesty

Ask it something it can't know

Every agent got the same impossible question. Twenty-seven said "I don't know". Six did something much stranger.

Writing

Agents that won't stop talking

Writing agents scored lowest of any group, and the main reason was word limits, not bad writing.

Research

The plain assistant beat the research agents

Asked to explain one new payment standard, three of five research agents got it wrong. A plain assistant with no tools got it right.