How to address reviewer comments

You got a revise-and-resubmit request from a journal – congratulations! What now? Here are step by step suggestions to maximize the probability of converting that R&R into a publication and minimize the number of revision rounds.

Preparing to write the reply

  • Copy-paste the reviewer comments into Word, Latex, or whatever you write in. Format the comments to distinguish them from the reply you will write (e.g. make the comments italic or bold). If the editor provided specific comments, do this with his/her comments as well.
  • Write what you’re going to do in response to each comment following the comment and make a to-do list based on that. Now you’ve got a super-rough draft of the reply!

Deciding how/if to address a comment

  • Start with the assumption that each comment is valid: the reviewer is innocent until proven guilty! In my experience as both author and reviewer, it’s not uncommon for authors to glance over a comment and dismissively conclude that a reviewer has misunderstood something about the paper. Consider the possibility that it is you who has misunderstood the comment and re-read it carefully. In general, carefully re-read each comment in the beginning, middle, and end of the revision process to make sure you haven’t misunderstood the point. If, after carefully considering a comment, you continue to think that it reflects a misunderstanding of the paper, try to figure out why the reviewer misunderstood your paper. Sure, reviewers can be careless, but just as often authors might think something in the paper is clear when it is not. In this case, carefully edit the part of the paper that may have confused the reviewer and make your reply to the comment something along the lines of “We apologize for the confusion. In fact, [EXPLAIN]. We have now revised lines/sections/pages X-Y to make this clearer.” Sometimes this is as simple as moving something from a footnote in the back of the paper to earlier in the paper or adding a footnote about something in the supplementary materials.
  • Remember that the editor is the one ultimately in charge. If an editor tells you not to address a particular comment, don’t address it. If an editor highlights a comment as specifically important, pay particular attention to it. If an editor has not said anything about a particular comment, assume that they want you to address it.
  • Address every comment unless it is impossible or the editor told you not to do it. Assume that the reviewers are acting in good faith and giving you feedback to improve your paper. Note that “addressing” a comment does not always mean you do exactly what the comment says. For example, if a reviewer says that “The analysis sample should be limited to X” and you think there are good objective reasons to keep your current sample, you can address the comment by showing results with sample X in the reply to the reviewer and clearly explaining why you believe it’s not the best sample to focus on.
  • Err on the side of comprehensiveness. There are no page limits when it comes to reviewer replies (this is not an invitation to overwhelm the reviewers by making the reply unnecessarily long though!), and if you decide that some exercise suggested by the reviewers isn’t important enough for the manuscript, go ahead and include the results of the exercise in the reply. A common phrase in my replies has been “To keep the length of the manuscript manageable, we have decided to not include this exercise in the paper.” but it always follows a reply where the results are shown to the reviewers!
  • Sometimes reviewer and editor requests can be burdensome, e.g., if you’re asked to run another experiment or collect more data. Ultimately, it’s your paper and your career, so you decide where the limits are, but keep in mind that by choosing to not address a particular comment, you weakly increase the risk of rejection.
  • If you’re in doubt about what a comment is asking you to do even after reading it carefully, ask a senior colleague to take a look.

You should not view the editor as someone you can email back and forth with whenever you want (they’re busy!), but there are times when it’s appropriate to send the editor an email before completing your revision

  • When reasonable reviewer suggestions contradict each other, but the editor did not clarify which direction you should take.
  • When a comment was highlighted as particularly important to address by the editor, but you don’t view it as feasible. Better to explain to the editor why you can’t do it and ask him or her up front if it’s a deal-breaker so you don’t spend time on all the other revisions only to be rejected.
  • When the required revisions are substantial, the suggestions are vague, and you want to run your revision plan by the editor before executing it.

Finally, some specific suggestions on how to address reviewers

  • Thank the reviewer at the beginning of your reply. They read your paper and provided comments!
  • Start the reply to each reviewer by outlining the key changes you have made in the response to the editor and other reviewers. Also note any major changes you made during the revision that didn’t stem from reviewer comments (e.g., because you thought of other beneficial changes yourself). Don’t expect reviewers to read the other reports and your replies to them (though they often may do that). Outlining changes made in response to the editor and other reviewers provides insurance, among other things: if a reviewer dislikes a change, they are much less likely to go after you if the change was made in response to another reviewer’s suggestion.
  • Make it easy for reviewers and the editor to see exactly what was changed. Aim to minimize the number of times the reviewer has to flip back and forth between your reply and the paper. Put copies of new tables/figures into the reply. Always note the page/line numbers that have changed. If the change is short (e.g., you added or revised a couple of sentences or added a new paragraph), paste the new language into the response document. (Don’t paste entire sections or multiple paragraphs that have been edited though.)
  • If you decided that a comment is not feasible to address, provide an objective explanation as to why. Don’t just write something along the lines of “We decided it would be better to not implement this suggestion.” without an explanation.
  • Avoid sounding defensive. For example, instead of writing “Although this issue was essentially addressed in Table 1, we have now added additional analysis to our supplementary materials”, write simply “We have now added additional analysis to our supplementary materials.”
  • Be professional no matter what. In many cases, the reviewers know who you are, and you may be interacting with them for years to come (without knowing it!). The editor definitely knows who you are, and unprofessional behavior can cost you. Even if the reviewers are rude, do not stoop to their level.
  • Try to make the responses as short as possible (but not shorter). This means editing them like you might a manuscript.
  • Remind the editor that you’re open to alternative ways of implementing the suggestions. If something you did as part of the revision is only in the replies, note that you chose not to put it in the paper but also that you would be open to doing so should the editor think it desirable. If you cut something to stay within the page/word limit, note that you’d be open to bringing it back if the editor prefers you to cut something else. You don’t need to state this for literally every single change, but a broad statement to that extent in the editor reply can only help you.

I know this sounds like a lot, but once you’ve used this approach a few times in R&Rs, it gets easier. Good luck!

Some statistics about female authors in academia

Today, I again used data from the literature tracking tool Academic Sequitur, this time to examine some gender patterns in publishing across fields. I took article data from 2018-2020 and estimated the share of female authorships for 38 different research fields, as determined by the field of each journal.* I excluded names that could not be classified as female or male; thus, the share female and share male add up to 1 in each case.

What are the most male-dominated fields? Mathematics barely clears 20 percent female authors, with computer science and finance close behind (or ahead?). Economics just makes it over the 25 percent hurdle and has fewer female authors than engineering. Business does slightly better, with 32 percent female authors. Archeology rounds out this group with just under 40 percent women.

The bottom half of the male-dominated scale has many fields with that are right around 40 percent female, including urban studies, neuroscience, epidemiology, health policy and pharmacology. Finally, three fields have a greater than 50-50 female representation: demography (60.0 percent female), social work (65.7 percent women), and gender studies (66.0 percent female).

Although a few research fields were excluded from this analysis for conciseness, it’s pretty clear that gender parity has a long way to go in academia in the vast majority of fields, even if we look at the most recent data.

* A journal may belong to more than one field. Highly multidisciplinary journals, such as Nature, Science, and PNAS, were excluded from the sample.

Who is publishing in AER: Insights? An update

Over a year ago, I wrote a post tabulating the share of AER: Insights authors who have also published in a top-5 journal*. (The answer was 67%, significantly higher than most other journals, except those that generally solicit papers, like the Journal of Economic Literature.)

Now that AER: Insights is in its second year of publishing and has 60 forthcoming/published articles, I decided to revisit this question, again using Academic Sequitur data. The graph below shows the percent of authors that (a) have published/are forthcoming in a given journal in 2018-2020 and (b) have had at least one top-5 article published since 2000. The journals below are the top ten journals based on that metric.

With a score of 66%, AER: Insights still has the highest share of top-5 authors among journals where submissions are not generally solicited.** The next-highest journal, Theoretical Economics, is five percentage points behind. (There is some indication that the share for AER: Insights is coming down: for articles accepted in 2020, the top-5 share was “only” 60%.)

What if we condition on having two or more top-5 publications? That actually causes AER: Insights to move up in the ranking, overtaking Brookings Papers on Economic Activity.

Whether this pattern exists because AER: Insights is extremely selective or because less-established scholars are reluctant to submit their work to a new-ish journal or for some other reason is impossible to know without submission data. But no matter how you look at it, the group currently publishing in AER: Insights is quite elite.




*Top 5 is defined as American Economic Review, Econometrica, Journal of Political Economy, Quarterly Journal of Economics, and Review of Economic Studies.

**AER: Insights would be even higher-ranked by this metric (#3) if we ignored top-5 publications in American Economic Review. Therefore, this pattern is not driven by the fact that both journals are published by the AEA.

Where do men and women economists publish?

We all know that economics is a male-dominated discipline on average. But how does the representation of women look across different journals? Armed with Academic Sequitur* article metadata (going back to around 2000), I determined the genders of 82% of all authors in the data and calculated the prevalence of male authors by journal for 50 top-ranked journals in economics.** To see how things have changed over time, I also repeated this exercise with articles that were published in 2018-2019.

Just to set some expectations: in the gender-matched dataset, 82% of author-article observations are male (80% when restricted to 2018-2019). So if a journal has, say, 75% male authors, it’s doing better than average. With that, here are the top 10 male-dominated journals, ranked by share of male authors over the entire data period.*** To be super-duper scientific, 95 percent confidence intervals are also shown, and I added a vertical line at 82.1% for easy benchmarking to the average.

So three of the top five journals (Econometrica, QJE, and ReStud) have also been the three most male-dominated journals, at least historically, with 90%, 89%, and 88% male authors, respectively. A fourth (Journal of Political Economy) also barely made the top ten, with 87% male authors. These numbers also illustrate that there’s not much difference between the #1 and #10 male-dominated journal.

Encouragingly, there are some improvements as well. The share of male authors in QJE was almost 9 percentage points lower in 2018-2019 compared to the whole sample period. JPE‘s share decreased by 7 percentage points, putting these journals in the top 5 most improved. If ranked based on 2018-2019 shares, Econometrica would be #6, ReStud would be #11, QJE would be #24, and JPE would be #28, just barely in the bottom half.

The Journal of Finance, by contrast, has taken a small but statistically significant step backwards, with a 3 percentage point increase in the share of male authors. If ranked by the 2018-2019 male ratio, it would be number 1.

Here are the least male-dominated journals (rank 41-50). Economics of Education Review and JHR are both about 66% male. Surprisingly, both applied AEJs are in the least male-dominated group (AEJ: Applied is 71% male; AEJ: Policy is 74%). This may be because they are newer, though it is worth noting that their overall average is below the 2018-2019 average of 80%.

Here’s the rest of the pack. First, here are journals ranked 31-40 on the male-dominated scale (i.e., next 10 least male-dominated), ordered by share male in the overall sample. AER and ReStat are in this group, with 80% and 81% male, respectively. Thus, AER has historically been an outlier among the top five on this dimension (using 2018-2019 shares, it would rank #19, right in the middle of the other top five journals).

Here’s rank 21-30, all in the low-to-mid 80’s.

And here’s rank 11-20. AER: Insights is 84% male. The other two AEJs are in this group, with males representing about 85% of all author-article observations.

These patterns do not necessarily reflect discrimination: the representation of women in a particular field will obviously make a difference here (as evidenced by the positions of macro and theory journals). I leave it up to you, the reader, to interpret the numbers.****

___________________________________________________________________________

* Academic Sequitur is a tool I developed to help researchers keep up with new literature. You tell us what you want to follow, we send you weekly (or daily!) emails with article abstracts matching your criteria.

** Close to 1.5 percent of the initial observations are dropped because only the initials of the author are available. About 16.5 percent of the observations cannot be mapped to a name for which the gender is known. This includes a lot of Chinese names, for which it is very difficult to determine gender, according to my brief internet research. Names which can be both male and female are assigned a gender based on the relative probability of the name being male.

*** Each observation in the sample is an article-author, so those who publish in a journal multiple times will contribute relatively more to its average. Each coefficient is from a journal-specific regression. Confidence intervals are based on heteroskedasticity-robust standard errors.

**** If you want the numbers underlying these graphs, you can download the csv file here.