Tag: peer review

How to address reviewer comments

Posted on by tatyana

You got a revise-and-resubmit request from a journal – congratulations! What now? Here are step by step suggestions to maximize the probability of converting that R&R into a publication and minimize the number of revision rounds.

Preparing to write the reply

  • Copy-paste the reviewer comments into Word, Latex, or whatever you write in. Format the comments to distinguish them from the reply you will write (e.g. make the comments italic or bold). If the editor provided specific comments, do this with his/her comments as well.
  • Write what you’re going to do in response to each comment following the comment and make a to-do list based on that. Now you’ve got a super-rough draft of the reply!

Deciding how/if to address a comment

  • Start with the assumption that each comment is valid: the reviewer is innocent until proven guilty! In my experience as both author and reviewer, it’s not uncommon for authors to glance over a comment and dismissively conclude that a reviewer has misunderstood something about the paper. Consider the possibility that it is you who has misunderstood the comment and re-read it carefully. In general, carefully re-read each comment in the beginning, middle, and end of the revision process to make sure you haven’t misunderstood the point. If, after carefully considering a comment, you continue to think that it reflects a misunderstanding of the paper, try to figure out why the reviewer misunderstood your paper. Sure, reviewers can be careless, but just as often authors might think something in the paper is clear when it is not. In this case, carefully edit the part of the paper that may have confused the reviewer and make your reply to the comment something along the lines of “We apologize for the confusion. In fact, [EXPLAIN]. We have now revised lines/sections/pages X-Y to make this clearer.” Sometimes this is as simple as moving something from a footnote in the back of the paper to earlier in the paper or adding a footnote about something in the supplementary materials.
  • Remember that the editor is the one ultimately in charge. If an editor tells you not to address a particular comment, don’t address it. If an editor highlights a comment as specifically important, pay particular attention to it. If an editor has not said anything about a particular comment, assume that they want you to address it.
  • Address every comment unless it is impossible or the editor told you not to do it. Assume that the reviewers are acting in good faith and giving you feedback to improve your paper. Note that “addressing” a comment does not always mean you do exactly what the comment says. For example, if a reviewer says that “The analysis sample should be limited to X” and you think there are good objective reasons to keep your current sample, you can address the comment by showing results with sample X in the reply to the reviewer and clearly explaining why you believe it’s not the best sample to focus on.
  • Err on the side of comprehensiveness. There are no page limits when it comes to reviewer replies (this is not an invitation to overwhelm the reviewers by making the reply unnecessarily long though!), and if you decide that some exercise suggested by the reviewers isn’t important enough for the manuscript, go ahead and include the results of the exercise in the reply. A common phrase in my replies has been “To keep the length of the manuscript manageable, we have decided to not include this exercise in the paper.” but it always follows a reply where the results are shown to the reviewers!
  • Sometimes reviewer and editor requests can be burdensome, e.g., if you’re asked to run another experiment or collect more data. Ultimately, it’s your paper and your career, so you decide where the limits are, but keep in mind that by choosing to not address a particular comment, you weakly increase the risk of rejection.
  • If you’re in doubt about what a comment is asking you to do even after reading it carefully, ask a senior colleague to take a look.

You should not view the editor as someone you can email back and forth with whenever you want (they’re busy!), but there are times when it’s appropriate to send the editor an email before completing your revision

  • When reasonable reviewer suggestions contradict each other, but the editor did not clarify which direction you should take.
  • When a comment was highlighted as particularly important to address by the editor, but you don’t view it as feasible. Better to explain to the editor why you can’t do it and ask him or her up front if it’s a deal-breaker so you don’t spend time on all the other revisions only to be rejected.
  • When the required revisions are substantial, the suggestions are vague, and you want to run your revision plan by the editor before executing it.

Finally, some specific suggestions on how to address reviewers

  • Thank the reviewer at the beginning of your reply. They read your paper and provided comments!
  • Start the reply to each reviewer by outlining the key changes you have made in the response to the editor and other reviewers. Also note any major changes you made during the revision that didn’t stem from reviewer comments (e.g., because you thought of other beneficial changes yourself). Don’t expect reviewers to read the other reports and your replies to them (though they often may do that). Outlining changes made in response to the editor and other reviewers provides insurance, among other things: if a reviewer dislikes a change, they are much less likely to go after you if the change was made in response to another reviewer’s suggestion.
  • Make it easy for reviewers and the editor to see exactly what was changed. Aim to minimize the number of times the reviewer has to flip back and forth between your reply and the paper. Put copies of new tables/figures into the reply. Always note the page/line numbers that have changed. If the change is short (e.g., you added or revised a couple of sentences or added a new paragraph), paste the new language into the response document. (Don’t paste entire sections or multiple paragraphs that have been edited though.)
  • If you decided that a comment is not feasible to address, provide an objective explanation as to why. Don’t just write something along the lines of “We decided it would be better to not implement this suggestion.” without an explanation.
  • Avoid sounding defensive. For example, instead of writing “Although this issue was essentially addressed in Table 1, we have now added additional analysis to our supplementary materials”, write simply “We have now added additional analysis to our supplementary materials.”
  • Be professional no matter what. In many cases, the reviewers know who you are, and you may be interacting with them for years to come (without knowing it!). The editor definitely knows who you are, and unprofessional behavior can cost you. Even if the reviewers are rude, do not stoop to their level.
  • Try to make the responses as short as possible (but not shorter). This means editing them like you might a manuscript.
  • Remind the editor that you’re open to alternative ways of implementing the suggestions. If something you did as part of the revision is only in the replies, note that you chose not to put it in the paper but also that you would be open to doing so should the editor think it desirable. If you cut something to stay within the page/word limit, note that you’d be open to bringing it back if the editor prefers you to cut something else. You don’t need to state this for literally every single change, but a broad statement to that extent in the editor reply can only help you.

I know this sounds like a lot, but once you’ve used this approach a few times in R&Rs, it gets easier. Good luck!

How to write a good referee report

Posted on by tatyana

Given the centrality of peer review in academic publishing, it might astonish some to learn that peer review training is not a formal component of any PhD program. Academics largely learn how to do peer review by osmosis: through seeing reports written by their advisors and colleagues, through being on the receiving end of them, and through experience. The result is perhaps predictable: lots of disgruntled researchers and the formation of such groups as “Reviewer 2 must be stopped” on Facebook.

This post is my attempt to make the world a better place by giving some advice on peer review. I have written over 100 reports, and I would like to think I do a good and efficient job (then again, I also mostly learned through osmosis, so you be the judge). Some of my advice is based on a great paper by Berk, Harvey, and Hirshleifer: “How to Write an Effective Referee Report and Improve the Scientific Review Process” (Journal of Economic Perspectives, 2017).

  1. As a reviewer, your job is to decide whether the paper is publishable in its current form and what would make it publishable if it is not. This is a distinct role from that of a copyeditor, whose job is to scrutinize every word and sentence, or a coauthor, whose job is to improve the contribution and substance of the paper. A reviewer’s goal is not to improve the paper, but to evaluate it, even though in the process of evaluating it, he may make suggestions that improve it. Of course, it is difficult for people to completely separate their own opinions from objective facts, but the harder we strive to play the right role, the fairer and smoother the review process will be.
  2. Your explanation of the paper’s strengths and weaknesses is more important than your recommendation. Many of us agonize over whether to recommend rejection or revise-and-resubmit. But reviewers do not know how many other submissions the journal receives or what their quality is. Even if you think the paper is great, it may be rejected because there are many papers that are even better. And a mediocre paper may make the cut if the other submissions are inferior to it. So the biggest service you can do for the editor is to help her rank the paper against the other submissions she is handling. Thus, you should aim to explain to the editor of what’s most impressive about the paper and what is lacking. The recommendation itself is secondary. When I recommend a rejection, I use the letter to the editor to outline the issues that make the paper unpublishable (there are usually 1-3), and why I don’t think they can be fixed by the authors.
  3. In case of rejection, make it clear to the authors what the deal-breakers are. The most frustrating and confusing reports to get are ones that raise seemingly addressable issues but are accompanied by a rejection recommendation. It may seem easier to save the “worst” for the letter to the editor, but it will leave the authors trying to guess why exactly the paper was rejected. Anecdotally, the most likely conclusion they will come to is “The reviewer just didn’t like the paper and then looked for reasons to reject it”, which is how Reviewer 2 groups get formed. Of course, you should use professional and courteous language in your reports. But don’t hide your ultimate opinion about the paper from the authors.
  4. In case of a revise-and-resubmit, make it clear to the authors what the must-dos and nice-to-dos are. Point 1 does not mean you should avoid suggestions that wouldn’t make or break publication. Many of my papers were improved by suggestions that weren’t central to the revision (for example, a reviewer suggested a great title change once). So if you have a good idea for improving the paper, by all means share it with the authors. But keep in mind that they will have at least one or maybe two-three other reviewers to satisfy, and the “to do” list can quickly spiral out of control. Sometimes the editor will tell the authors which reviewer comments to address and which to ignore. But sometimes the editor will pass on the comments to the authors as is. By separating your comments into those you think are indispensable and those that are optional, you’ll be doing the authors a big favor.
  5. Don’t spend a lot of time on a paper that you’re sure you’re going to reject. This is perhaps the most controversial piece of advice (see this Tweet & subsequent discussions) because some authors view the review process as a “peer feedback” system. But it is not (see point 1). And, at least in economics, many of us are overwhelmed with review requests and editors sometimes have a hard time finding available reviewers. Treating the review process as “peer feedback” exacerbates this problem. If you think the authors’ basic premise is fundamentally flawed or the data are so problematic that no answer obtained from them would be credible, you should not feel obligated to give comments on other parts of the paper. This does not mean that you should not be thorough – there are few things more frustrating than a reviewer complaining about something that was explicitly addressed by the authors. But in such cases you do not need to give feedback on parts of the paper that did not affect your decision.

Finally, I’d like to wrap up with an outline of how I actually do the review. First, I print out a physical copy of the paper and read it, highlighting/underlining and making notes in the margins or on a piece of paper. Second, I write a summary of the paper in my own words (it is useful for the editor to get an objective summary of the paper, and the authors can make sure I was on the same page as them). Third, I go through my handwritten comments and type the most relevant ones up, elaborating as needed. Fourth, I number my comments (helpful for referencing them in later stages, if applicable), order them from most to least important, and separate the deal-breakers or must-dos from the nice-to-dos. Fifth, I highlight the deal breakers (if rejecting) or must-dos (if suggesting revisions) in the letter to the editor. Finally, regardless of my recommendation, I try to say something nice about the paper both in the editor letter and in the report. Regardless of its quality, most papers have something good about them, and authors might be just a tad happier if their hard work was acknowledged more often.

What to do after a rejection

Posted on by tatyana

Benjamin Franklin wrote, “in this world nothing can be said to be certain, except death and taxes” (though the earliest origin of that idea dates to Christopher Bullock in 1716, apparently). Most academics would agree that paper rejections also belong on that list. My 10 published papers have been rejected a total of 29 times. I also have two “archived” papers that were collectively rejected eight times before I gave up on them and four working papers that so far have been rejected seven times (two are now revise-and-resubmit, so the rate of rejection is decreasing). So I have a total of 44 rejections. I have ZERO papers that got a revise-and-resubmit at the first journal I submitted them to (= each of my papers has been rejected at least once). I’m not even counting conference and grant rejections.

Paper rejections come in many shapes and sizes: your run-of-the-mill “Nice paper, but not enough of a contribution for this journal” or “Too many little things wrong” rejections; a reviewer finding something genuinely wrong with your manuscript; boilerplate desk rejections; a half-page report from a lazy reviewer who clearly hasn’t read your paper; and the frustrating “I just don’t believe your results” rejection. Rejections don’t feel good, but given that they are inevitable, it’s important to learn how to deal with them and move past them as quickly as possible. Below, I provide some suggestions that have worked for me.

First, allow yourself to take a few days to “mourn” the decision. A few days of inaction after a rejection won’t make much of a difference. I typically don’t even read the referee reports closely until it’s been a few days because I’m not confident in my ability to take in the feedback objectively. By all means, trash-talk the referees to your colleagues (people at your institution almost surely won’t be asked to review your papers), join the “Reviewer 2 must be stopped” group on Facebook (especially if you don’t know what “Reviewer 2” refers to), have a drink or two (please drink responsibly), do some exercise, work on another paper, or binge-watch that show you’ve been waiting to see. Do be careful how you discuss your reports online or at conferences because you never know who your reviewers were or who might know who your reviewers were.

It is hard not to take rejections personally, but in the vast majority of cases, they are not. The reviewers rejected your paper, they did not reject you as a person or a researcher. Even the comments about your paper may not have anything to do with the quality of your paper. Some reviewers might strongly dislike a particular methodology or research area, others may have had a bad day or week, and some may operate in toxic environments where unnecessary harshness is disguised as “honesty”. Your reviewer may have been a graduate student doing a referee report for the first time or a senior professor drowning in service work. Almost everyone has a “Reviewer 2” story, including some of the best researchers, and you are not alone. If a reviewer seemed particularly unfair, talk to a senior colleague about appealing the decision. However, appeals are definitely not the standard way to deal with rejections.

Next comes the time for actual work. Unless the journal rejecting your paper was your last stop before you were going to abandon efforts to publish it, try to return to the reports within a week of the rejection and look at them objectively. It can be tempting to either (1) ignore the reports completely and send the paper back out as soon as possible or (2) treat the reports as a revise-and-resubmit and try to address all the reviewer’s comments. Neither approach is generally a good idea, for two reasons.

First, you may get the same reviewer again. In some fields, reviewing the same paper twice is not acceptable, so you may get a different draw in that case. But in economics and surely some other fields, it’s not uncommon for the same person to review the paper two or more times at different journals. In such a case, the best you can hope for if you didn’t change anything in your paper is that the reviewer will return the same report to the editor. But it’s also possible that the reviewer will be annoyed that you did not take into account any of the comments they worked hard to give you and treat your paper more harshly than the first time around. In short, you want to avoid giving the impression that you thought the comments so worthless that you did not address even one.

Second, even if you’re 100% sure you’re not going to get the same reviewer, it’s highly unlikely that the reviewers’ comments were completely idiosyncratic or idiotic. If you ignore a comment that you could have addressed and a subsequent reviewer has the same concern, your paper could end up rejected again for avoidable reasons. Despite all the “Reviewer 2” stories out there, I think the overall peer review process is far from completely broken, so it’s also very unlikely that all the comments are useless and wrong. In short, the best way to treat the reviewer reports following a rejection is as an opportunity to make your paper better.

When deciding whether to address a particular comment, I ask myself two things: (1) How likely is this comment to come up again? and (2) How easy is this for me to address? The higher the comment is on this two-dimensional likelihood-ease scale, the more you should jump at the chance to address it. Whether something is likely to come up again or not is the hardest question to answer. Here, thinking about comments you’ve gotten at conferences or asking colleagues for their feedback on a particular comment can be really helpful. Rigorous self-honesty helps too: with some introspection, most of us will be able to identify comments where the reviewer really does have a point. Once you’ve identified all the relevant comments, start addressing them one by one. Where to stop can be difficult to tell, but if you start with the comments that rank high on ease and/or likelihood, you can stop at any point with the knowledge that you’ve addressed the most important ones. For me, a good rule of thumb is that the paper should be ready to go back out within 1-3 months or less of not-full-time work (this is probably equivalent to about 1-3 weeks of full-time for me). Anything more than that is likely to be excessive in most circumstances.

I’ll wrap up with two specific suggestions. If a reviewer comment makes it seem like she or he misunderstood something about what you’re doing, try to see if you can make that part of the paper clearer. You have the privilege of knowing your paper better than anyone else, so what seems clear to you may not be to the average reader. If there is a comment that seems likely to come up again but would be really difficult to address, you have a few options. You can add a brief explanation as to why doing X would be difficult, possibly as a footnote, possibly as a suggested avenue for future research. This signals to reviewers that you are aware of X. Relatedly, you can hint that you could do X but it’s outside of the scope of the current paper. That allows a persistent reviewer to insist on seeing X in a revision but reduces the likelihood that they reject the paper because you didn’t already do X.

In the end, these steps don’t necessarily make rejections more pleasant, but they do move your paper closer to published!