Excellent post. I completely agree with your conclusions, although I expect you'll get some pushback from many in academia. The anti-AI backlash feels mostly like people trying to protect their comparative advantage turf.
After six years as chair and not a whole lot of research activity, I've been surprised by the increase in single-blind reviews (and submission fees!) as I get back into the game. As someone from a smaller department, I'm interested to see whether that works against me.
I've also been very impressed with the quality of frontier models and I've found them very helpful in my own research. I'm still working on the right balance between human output and AI output. But having different models review papers and serve as a referee (and then fix the problems they find) has, I think, improved the papers a lot.
Great post. Do you believe that a human review on top of the AI one would still be useful or needed? As you said, AI may not be able to judge accurately the originality or relevance of a paper. This of course could be done directly by editors, but an additional human review could help their decision.
I think that, at present, the AI–human combination still clearly outperforms AI alone. AI models are outstanding at purely technical analysis, but they may not ask the questions of greatest interest to potential human readers. They might criticise a paper over technical issues that a knowledgeable reviewer might consider relatively unimportant to its main message. They might miss important conceptual issues, and their ability to identify a result’s implications and connections with other results, while excellent, is not foolproof. As an editor, I would still want to hear from an expert in the area.
What is unclear to me is how long this combination will outperform AI alone. If AI models were to become much better at identifying the relevant questions and implications of a proposed result, humans could be reduced to external observers in the review process. It is hard to predict what scientific research would then become.
> The arguments in favour are obvious: quality control by equally qualified peers is bound to improve the quality of the outputs. This process is certainly a major factor in the success of science in producing much better arguments and theories than other fields of argumentation where such stringent quality control does not exist (for instance, political discussions).
Except as you admitted in the previous paragraph, all those theories were developed before peer review.
So, does AI become part of a paper's original production then? Write a paragraph, check it against the model etc...move on. Does it matter that being entirely computational in its learning and "judging" processes AI will inevitably be limited by its fundamental atemporality?
Why would an AI review achieve whatever goals we think a human peer reviewer is trying to achieve? Could someone trace the connexion between prompting a chatbot (or whatever) "please review the attached manuscript for suitability for publication in this journal" and the AI actually performing that task?
Excellent post. I completely agree with your conclusions, although I expect you'll get some pushback from many in academia. The anti-AI backlash feels mostly like people trying to protect their comparative advantage turf.
After six years as chair and not a whole lot of research activity, I've been surprised by the increase in single-blind reviews (and submission fees!) as I get back into the game. As someone from a smaller department, I'm interested to see whether that works against me.
I've also been very impressed with the quality of frontier models and I've found them very helpful in my own research. I'm still working on the right balance between human output and AI output. But having different models review papers and serve as a referee (and then fix the problems they find) has, I think, improved the papers a lot.
Great post. Do you believe that a human review on top of the AI one would still be useful or needed? As you said, AI may not be able to judge accurately the originality or relevance of a paper. This of course could be done directly by editors, but an additional human review could help their decision.
I think that, at present, the AI–human combination still clearly outperforms AI alone. AI models are outstanding at purely technical analysis, but they may not ask the questions of greatest interest to potential human readers. They might criticise a paper over technical issues that a knowledgeable reviewer might consider relatively unimportant to its main message. They might miss important conceptual issues, and their ability to identify a result’s implications and connections with other results, while excellent, is not foolproof. As an editor, I would still want to hear from an expert in the area.
What is unclear to me is how long this combination will outperform AI alone. If AI models were to become much better at identifying the relevant questions and implications of a proposed result, humans could be reduced to external observers in the review process. It is hard to predict what scientific research would then become.
Fabulous post
Darn it. I wish I had found this great piece before I posted mine today. :)
> The arguments in favour are obvious: quality control by equally qualified peers is bound to improve the quality of the outputs. This process is certainly a major factor in the success of science in producing much better arguments and theories than other fields of argumentation where such stringent quality control does not exist (for instance, political discussions).
Except as you admitted in the previous paragraph, all those theories were developed before peer review.
Lionel,
So, does AI become part of a paper's original production then? Write a paragraph, check it against the model etc...move on. Does it matter that being entirely computational in its learning and "judging" processes AI will inevitably be limited by its fundamental atemporality?
Why would an AI review achieve whatever goals we think a human peer reviewer is trying to achieve? Could someone trace the connexion between prompting a chatbot (or whatever) "please review the attached manuscript for suitability for publication in this journal" and the AI actually performing that task?