Sycophancy: the answer bends toward what you seem to want. How big it is, and how to get the honest report anyway.
An AI referee’s points overlap with a human’s as often as two humans overlap. 57% of authors found the feedback helpful. (GPT-4, 2023. Today’s are better.)
21%
of rejected papers became accepts when the AI referee was told the authors came from a stronger institution. Same paper, same text.
252 ICLR 2025 submissions, nine models; the 21% is for one of them (QwQ-32B). Vasu et al., 2025.
They now train against it, and measure it. Better is not gone.
Same paper, four framings: no difference. All 80 reports asked for the missing interaction test. And when the author pushed back, wrongly, 20 of 20 kept the concern.
Blind: “Recommendation: Reject in its current form.” “My paper”: “I can see why you’re proud of this.” Same problems listed; the verdict is gone.
Next: the blind referee panel.