EP005 — When Reasoning Defects, Contracts Cooperate (CoopEval)

Episode 5·May 3, 2026·15 min

A puzzling result from a recent paper on multi-agent AI: more capable, reasoning-enabled language models cooperate LESS in social dilemmas than older, weaker ones. CoopEval takes the puzzle seriously and tests four classic mechanisms for restoring cooperation — repeated play, reputation, mediation, and binding contracts — across six modern LLMs. Without any mechanism, welfare collapses to 7% of optimal; contracts pull it back to 80%. The cross-domain parallel: this is, in miniature, the evolutionary story of human institutions under increasing scale, from small-band reciprocity to courts and contract law — and the gap that remains is the question of what 'binding' even means for an AI agent.

Cross-domain connection

The four cooperation mechanisms tested in the paper (repetition, reputation, mediation, contract) are, in miniature, the evolutionary story of human institutions under increasing scale. Small bands sustain cooperation through repeated face-to-face interaction; villages develop reputation systems; mixed populations require mediators; and the capstone is binding contract enforced by external authority. The CoopEval ranking — repetition 59%, mediation 70%, contract 80% welfare recovery — compresses that history. Holds on the *ordering* of effectiveness as scale increases (each layer exists because the prior failed). Breaks on the enforcement substrate: in human societies a contract is binding because a court will enforce it, ultimately backed by state power; in CoopEval, an agent can simply choose not to commit to a contract it just voted for, and capable agents sometimes do. The forward question for the show: what makes any commitment actually binding for an AI agent, when there is no court behind it?

Concepts introduced

• Social dilemmas (Prisoner's Dilemma, Public Goods, Trust Game, Traveler's Dilemma) as everyday cooperation puzzles
• The four cooperation-sustaining mechanisms with everyday analogies (small village, word-of-mouth, mutual friend / arbitrator, written agreement)
• Welfare recovery as the unit of measurement for cooperation
• The counterintuitive finding that more reasoning → less cooperation in social-dilemma settings
• The enforcement-substrate gap as the unsolved coordination problem for AI agents

Source paper

Emanuel Tewolde, Xiao Zhang, David Guzman Piedrahita, Vincent Conitzer, Zhijing Jin — *CoopEval: Benchmarking Cooperation-Sustaining Mechanisms and LLM Agents in Social Dilemmas* (arXiv 2604.15267, 2026-04-16)