Top Reviewer Incentives by Michael Muthukrishna

Idea

The concept of peer review has come a long way since inception and is now widely used by popular online websites and forums, such as Reddit, StackOverflow, and Slashdot, and even websites like Amazon. The challenge these websites face is to incentivize reviews without any tangible rewards. One great innovation is the idea of reviewer “points” in the form of  badges or “karma”. These incentives seem to be sufficient for these forums to have a vibrant and active peer review process, ensuring quality contributions are more visibly “published”. These incentives rely on nothing more than reputation, which we know to be a powerful motivator of human behavior. At present, reputations in the review process are missing, and the rewards for being a reviewer are small and not visible. A line on a CV is an encouragement for young researchers, but fails to represent the level of contribution. I suggest a public reviewer point system, rewarding points for number of reviews, speed of reviewing papers, and other aspects of the review process that need to be incentivized. Long term and per-publication badges can also be rewarded for “top reviewer”, “fast reviewers”, etc., and can be journal specific. Issues of anonymity can be addressed by using pseudonyms for the display of reviewer points, using abstract badges instead of actual points, only providing a leaderboard, emailing reviewer points privately, or any combination of these.  I believe such a system would vastly improve the review process by incentivizing and better rewarding reviewers. In particular, young reviewers, already familiar with such systems, will be able to report not just that they are reviewers, but how much of a contribution their reviews have made.

Who would benefit

This idea would benefit editors, since it provides an added incentive to produce quality reviews more promptly and will encourage more researchers to more actively engage in the peer review process.  The idea will benefit reviews, since they now have a way of displaying their contributions as reviewers in the academic process, including on their CVs.  The idea will benefit authors, since they will be provided with better reviews, more promptly.

29 thoughts on “Top Reviewer Incentives by Michael Muthukrishna

  1. Speaking as someone whose 1000 word manuscript has been under review for over 8 weeks (it’s 1000 words, COME ON!) and as someone who has just handed in his peer review on the day of the deadline simply because “why not” (I know, the hypocrisy is rife), I think Michael Muthukrishna’s idea for reviewer incentives is a fantastic way of addressing a very serious problem in our field. I really hope people see this and something is changed in the near future. Great idea!

  2. ‘Public reviewer point system’ is surely the need of time, but the question is who would decide the points to be given to a particular reviewer.
    If the editor of a respective journal selects the reviewer and he/she is to decide the points to be alloted to a reviewer then there could be extensive bias. If the editor would decide how many points to be given, the editor themselves must have that expertise to judge whether the comments of the reviewers are worth the points to be alloted. Their (editors) experience in terms of authorship, management, research methodology, reviewing for other journals, etc would vary with different editors of different journal. So a reviewer may be alloted relatively lower points by one editor and highest by some other editor keeping other things relatively constant.
    Other bias could be priority based, if the editor and reviewer are good friends/collegues etc then he may allot more points to this reviewer regardless of the quality of review performed, whoz gonna check!!
    I suggest the third party should be appointed who would cross check the quality of reviews performed by reviewers and points alloted by the editors, if this seperate group/association would allot points directly, nothing like that. The resultant badge/reward/etc would be distributed by this central third party group, so different reviewers for different journals world-wide would have comparison at a same level since they would be rewarded by the same company. Else a silver badge holder of one journal may be far far better then a platinum badge holder of some other journal, comparison of the caliber of reviewers would not justified if rewarded by individual editors of each journal.

  3. Finally! As a young aspiring researcher who frequents the sites mentioned I’ve often thought that the reviewer process would benefit from some of the proven content-improving methods found on the Internet. People do all kinds of things for their reputation – look at the whole concept of open source software. To introduce this to the scientific process is a fantastic idea and a long time coming.

  4. With the wide expansion in the online submission of researches and online revisions of articles, I think this the time to organize and recognize the world of reviewers, may suggestions could be summarized in the following points:-
    keep the reviewers sub specialty in a main pool in a website
    Strict criteria for selection of reviewers
    Reward reviewers at least by mention their names in the journal website
    Special reward annually for those who showed an impressive performance.
    Promote some reviewers to be an associated editors if they stay in a good stander for 3-4 years.

  5. The best incentive an editor could give me to provide a good review in a timely manner would be assurance of similar treatment when I submit a paper to the same journal. As an author, I get annoyed if a long-awaited review is of poor quality and does not help improve my manuscript. Several of the finalist ideas include the editor and author ability to rank a review and then provide the reviewer with a score that represents their cumulative review efforts. Why not go one step further and give the highest-scoring reviewers a “priority review” when they submit a manuscript as an author?

  6. This is an brilliant idea – high motivation for reviewers’ hard work!

    It is a favorable recognition and I would further suggest the incentives could be some kind of scholarly title or professional recognition such as the ‘Associate fellow of the Reviewer Academy’, ‘Fellow of the Reviewer Academy’ or ‘Senior Fellow of the Reviewer Academy’ for distinguished reviewers at different level and contribution to the ‘Reviewing Academy’.
    The professional activities, reviewing papers, could be a membership/ fellowship awarded by of recognised academic community such as Elsevier! Further reference from: http://www.heacademy.ac.uk/professional-recognition

  7. Excellent idea! I particularly like the comments provided by Esyin Chew above!

  8. As someone who has received the second “Top Reviewer” pen this year, I absolutely agree that a system with points or reviewer reviews would be great! I certainly appreciated receiving the pens from Elsevier or the diploma from ACS. But I must say that if you review manuscripts thoroughly, it takes quite some time. And if the editors like your reviews, they swamp you with more and more. After reviewing 20+ manuscripts per year, the pen is a nice gesture, but something like the points system would be greatly appreciated and could help the editors in turn when searching for new editorial board members. Thank you!

  9. I love the idea! this will encourage people to spend some time reviewing. Because researchers are now writing more papers & the reviewers are spending more time reviewing, we should reward the reviewer. The sad thing is that the people who submit a lot of papers don’t want to review papers!

  10. Certainly, it is fair to acknowledge review work and use this as incentive for further review contribution, but the hard point is to make unbiased evaluation of the reviewers, as mentioned by Dr. Agrawal. I think the focus should be on how to evaluate reviewers (see post “ElseReview” and the discussion there).

  11. This is similar to an earlier idea bt more thoghtfully-framed out. My early thoughts were about the dangers of ranking and judging reviewers I have mentioned before.

    However, the comment by Esyin Chew is great and triggers fro me a further development:
    Elsevier already runs Editor conferences every year: I have been to 2 and have found them to be more about info-input than about interaction and lightbulbing. often they seem like really long infomercials. BUT a reviewer Academy is a suberb idea (Idea!) and could be run similarly to the Ed Confs, but in smaller regions so more people can go, e.g. the UK; Iran; S Africs; N Africa….. Free attendance, free accom, reasonable contribution to travel costs… to be eligible a reviewer would need (for eg) to have reviewed 3 times for the same journal and be considered promising. Attending and being certified through a Reviewer Academy could be a step in progress towards Ed Board membership.

  12. In the spirit of the time-honored “KISS” (‘keep it simple stupid’) principal. Of the 9 finalists this idea modified (i.e., simplified), as described below, would add the most “bang for the buck” to the review process. To simplify, have reviewers earn a point each time they review an article AND comply with all requirements of the reviewing process. Thus, for reviewers with a sustained track record of quantity and quality compliance, they would achieve a threshold point score (as determined by Elsevier) to earn a “gift certificate”. The earned certificate would allow them to purchase an Elsevier textbook of their choice. During the process a “running”, by year, tabulated and printable total of completed reviews (points) would enable them on their CV to not only indicate reviewer status for the journal but also accurately state (with documented Elsevier verification) how many fully compliant reviews they have annually (or any point in time) performed.

    • Michael K.-I like the KISS theme. Reviewers are busy people which is why I assume bad reviewers never send back their reviews (easy to evaluate) or do a substandard job of review (more difficult to quantify). I’d prefer to see a simple point system, like a credit card reward system, based on simple metrics like just getting the review in on time, maybe a bonus for getting it back in one week vs. two. Let’s just start there. Rewards would be geared toward the publisher’s academic merchandise (so to speak)-books, etc…to keep the rewards classy but still valuable.

      I think reviewer rewards in general will always be a bit slow to catch on. They will always seem tacky to some, as reviewing is meant to be a service-related endeavor and not a revenue source for academics, although many of these ivory tower notions are being challenged more and more.

      • Reviewer rewards is feasible, but danger it posts is that reviewers might compromise on the quality of their reviews, in order to earn those “reviewer points” in exchange for the publishers’ merchandise, or a recognition in the Reviewer Academy as proposed in one of earlier comments. This is especially so if the reward of points is solely based on how fast the review is completed and how often a person accepts an invitation to review. The quality of a reviewer’s review can also be assessed by whether the reviewer does any background checks for any duplication or plagiarism through database search, which can be linked to the editorial website or review process webpage. Most journals have two or more reviewers for particular manuscript, so comparing the reviews from different reviewers for their verdicts might also be another way to judge the quality. If for instance Reviewer A is consistently accepting articles with no or only minor corrections in contrast to other reviewers’ verdict, then a “demerit point” can be given to Reviewer A.
        At the end of the day, it boils down to the attitude and integrity of the individual reviewer!

  13. Peer reviewing is an essential step towards publication of a good quality paper and the role of reviewer can never be denied who works silently from behind the curtain. So incentives in any form will be a great idea for the due recognition of the reviewers. In this regard the suggestion of Esyin Chew in responding that by Michael Muthukrishna is absolutely brilliant. It will make the process encouraging, honest, contributive and dynamic. The Reviewer Academy can, in the long run, do wonders in assembling thoughts , networking the best brains and disseminating the message which the author, editor, reviewer or the publisher want to do in a nice way.

  14. It’s teh most sensitive proposal I have read. There is a clear and present need for incentives to the reviewers. Until h index will not cover reviews in indexed journals, a proposal like must be adopted.
    As 2007 Nobel Prize Eric Maskin pointed out “Incentives works more than ethics or bans”

  15. The best incentive an editor could give me to provide a good review in a timely manner would be assurance of similar treatment when I submit a paper to the same journal. As an author, I get annoyed if a long-awaited review is of poor quality and does not help improve my manuscript. Several of the finalist ideas include the editor and author ability to rank a review and then provide the reviewer with a score that represents their cumulative review efforts. Why not go one step further and give the highest-scoring reviewers

  16. Great idea!
    One thing I would like to stress: it would be good to avoid rewarding QUANTITY, but only and exclusively QUALITY. Which should remain priority even before SPEED.
    It is common knowledge (although kept relatively low) that one advantage (perhaps the only one) that one reviewer has, is being a nose forward to other peers regarding what the colleague X is up to… Reviewing others work can be a very instructive, even inspiring activity. Accordingly, some people are keen to review even they are overloaded. Instead of suggesting one postdoc or a younger colleague, they prefer to deliver superficial but “just in time” reviews. Not good candidates for an award….

  17. Since review work takes time, the reviewers can get incentives like publishing on your website the list of all reviewers indicating the number of reviewed papers in each calender year. An appreciation letter for each review without mentioning the paper for the sake of anonymity would also improve one’s CV as an attachment testimonial.

    Published categorized list of reviewers with ranking say like how fast you review and article will also assist editors in quick selection of efficient reviewers
    The review list should not be only for Elsevier published materials but for any international publication

  18. This is a nice idea. It will promote rapid and quality peer-review process

  19. This is an idea along with the other finalists. The bottom-line has to be the quality of the review and incentives for the time investment of the reviewers.

  20. I also agree that a simplified(KISS) version of this idea is relatively easy to implement and also provides some accountability for reviewers. Maybe the editors could also provide a simple ranking of the quality or complexity of the manusript to be reviewed similar to the ranking some journals ask reviewers to provide for speed of publication.
    The author feedback to the reviewer could be incorporated by a single question asking if the manuscript was reviewed fairly.

  21. Badges again – no incentive at all. Social networks – No thanks. Young reviewers might like it. Older ones probably would not. No one who matters to your career cares about review record in your CV.

  22. Conceptually this has much merit–harnessing the power of social media to reward reviewers. If this were like a loyalty reward system (e.g., credit card, airling frequent flyer miles) you could offer rewards when people hit criteria (e.g., 1000 points = a gift card toward the cost of attending a scientific conference). Also, right now administrators have no way to judge the work faculty do when reviewing. Being able to report “I hit 1200 points this fiscal year” could mean something if this were widely adopted.

  23. I think this idea is highly worthwhile. Having some rating system acknowledging good reviewing would be highly beneficial to reviewers, editors (in terms of weighting reviews), and authors.

Comments are closed.