If something really is rotten in the state of peer review, the latest experiment by the Shakespeare Quarterly suggests that it may be some time before academics embrace a possible remedy.
The Commons Science and Technology Committee recently launched an inquiry into peer review and was scheduled to have its first hearing yesterday. It comes amid concerns - voiced in many of the 93 written submissions the inquiry has received - about the current system's tendency to favour conservative research and the dangers of it being distorted by nepotism, exploited by unethical reviewers or undermined by a lack of appropriate expertise.
Among the inquiry's terms of reference are "the impact of IT and greater use of online resources". Many believe that online open peer review could improve the effectiveness of the process by exposing it to a wider range of publicly identified reviewers.
Last year, the Shakespeare Quarterly became the first humanities journal to experiment with open peer review when it invited comments on seven papers it was considering for inclusion in a special edition on Shakespeare and new media.
David Schalkwyk, editor of the journal and director of research at the Folger Shakespeare Library in Washington DC, described as "terrible" the imbalance of power in traditional "single-blind" peer review, in which the identity of the author is known to the reviewer but not vice versa.
He has adopted double-blind reviewing at the Quarterly (where neither party is known to the other), whereas the open peer-review experiment required both authors' and reviewers' names to be revealed.
Dr Schalkwyk said the experiment had been prompted primarily by a desire to harness the web's potential to support greater scholarly discussion. It also reflected a feeling that "if we were going to talk about Shakespeare and new media, we should practise new media ways of doing things".
However, he was so "pleased and excited" by what transpired - 41 people made more than 350 comments, many provoking responses from the authors - that he decided to repeat the experiment for this year's special edition on Shakespeare and performance.
"The special editions are the best format for open review because there is a particular focus. You can target groups of experts to be your committed reviewers, and you can assume there will be a general body that will also be interested," he said.
Seven articles were posted - although Dr Schalkwyk admitted that several authors had opted for traditional review instead. The final decisions on acceptance will be taken after the authors have had the chance to revise their essays in light of the comments.
Despite "buttonholing" around 25 committed reviewers, Dr Schalkwyk admitted that further cajoling - and a seven-day extension to the original deadline for comments - had been necessary to elicit the final total of 235 comments from 30 people (including the authors).
"We were really worried at one point, but we felt we got good enough commentary by the end," he said, adding that the quality did not differ greatly from that returned by traditional reviewers.
Heart on sleeve proves unpopular
Part of the difficulty of attracting open reviews, he said, related to pressures on academics' time. But he also thought that the "more traditional" subject matter in the second experiment had attracted a different "constituency" of readers who, unlike the new media enthusiasts, did not see "open review as the most normal thing in the world".
He said a lot of academics were reluctant to participate in open peer review because it put "a bit of pressure" on them to "take public responsibility for what they say".
"Our readers aren't untrustworthy or irresponsible, but, structurally, it changes the relationship with the author: it engenders a discussion and many people don't want to get into that," he said.
Dr Schalkwyk is considering allowing online reviewers to be anonymous in a future experiment to see whether it boosts the number of comments.
But decisions about whether a particular special edition should adopt open peer review will now be taken on the basis of whether the theme seems likely to appeal to a web-friendly constituency.
"We are back-pedalling a little," he confessed. He also conceded that open peer review might not turn out to be the best way to promote greater scholarly discussion.
"It may be that it is one thing to create a forum for open discussion and another to ensure work is properly peer reviewed," he said.
The future of open peer review will also be heavily dependent on the survival or otherwise of paper journals, he added. "If everything goes electronic then traditional modes of doing things will seem more old-fashioned. But the fact that we continue to be the lonely scout (in open peer review) is an indication that it isn't catching on."
?
?