Tag: academic research

How to address reviewer comments

Posted on by tatyana

You got a revise-and-resubmit request from a journal – congratulations! What now? Here are step by step suggestions to maximize the probability of converting that R&R into a publication and minimize the number of revision rounds.

Preparing to write the reply

  • Copy-paste the reviewer comments into Word, Latex, or whatever you write in. Format the comments to distinguish them from the reply you will write (e.g. make the comments italic or bold). If the editor provided specific comments, do this with his/her comments as well.
  • Write what you’re going to do in response to each comment following the comment and make a to-do list based on that. Now you’ve got a super-rough draft of the reply!

Deciding how/if to address a comment

  • Start with the assumption that each comment is valid: the reviewer is innocent until proven guilty! In my experience as both author and reviewer, it’s not uncommon for authors to glance over a comment and dismissively conclude that a reviewer has misunderstood something about the paper. Consider the possibility that it is you who has misunderstood the comment and re-read it carefully. In general, carefully re-read each comment in the beginning, middle, and end of the revision process to make sure you haven’t misunderstood the point. If, after carefully considering a comment, you continue to think that it reflects a misunderstanding of the paper, try to figure out why the reviewer misunderstood your paper. Sure, reviewers can be careless, but just as often authors might think something in the paper is clear when it is not. In this case, carefully edit the part of the paper that may have confused the reviewer and make your reply to the comment something along the lines of “We apologize for the confusion. In fact, [EXPLAIN]. We have now revised lines/sections/pages X-Y to make this clearer.” Sometimes this is as simple as moving something from a footnote in the back of the paper to earlier in the paper or adding a footnote about something in the supplementary materials.
  • Remember that the editor is the one ultimately in charge. If an editor tells you not to address a particular comment, don’t address it. If an editor highlights a comment as specifically important, pay particular attention to it. If an editor has not said anything about a particular comment, assume that they want you to address it.
  • Address every comment unless it is impossible or the editor told you not to do it. Assume that the reviewers are acting in good faith and giving you feedback to improve your paper. Note that “addressing” a comment does not always mean you do exactly what the comment says. For example, if a reviewer says that “The analysis sample should be limited to X” and you think there are good objective reasons to keep your current sample, you can address the comment by showing results with sample X in the reply to the reviewer and clearly explaining why you believe it’s not the best sample to focus on.
  • Err on the side of comprehensiveness. There are no page limits when it comes to reviewer replies (this is not an invitation to overwhelm the reviewers by making the reply unnecessarily long though!), and if you decide that some exercise suggested by the reviewers isn’t important enough for the manuscript, go ahead and include the results of the exercise in the reply. A common phrase in my replies has been “To keep the length of the manuscript manageable, we have decided to not include this exercise in the paper.” but it always follows a reply where the results are shown to the reviewers!
  • Sometimes reviewer and editor requests can be burdensome, e.g., if you’re asked to run another experiment or collect more data. Ultimately, it’s your paper and your career, so you decide where the limits are, but keep in mind that by choosing to not address a particular comment, you weakly increase the risk of rejection.
  • If you’re in doubt about what a comment is asking you to do even after reading it carefully, ask a senior colleague to take a look.

You should not view the editor as someone you can email back and forth with whenever you want (they’re busy!), but there are times when it’s appropriate to send the editor an email before completing your revision

  • When reasonable reviewer suggestions contradict each other, but the editor did not clarify which direction you should take.
  • When a comment was highlighted as particularly important to address by the editor, but you don’t view it as feasible. Better to explain to the editor why you can’t do it and ask him or her up front if it’s a deal-breaker so you don’t spend time on all the other revisions only to be rejected.
  • When the required revisions are substantial, the suggestions are vague, and you want to run your revision plan by the editor before executing it.

Finally, some specific suggestions on how to address reviewers

  • Thank the reviewer at the beginning of your reply. They read your paper and provided comments!
  • Start the reply to each reviewer by outlining the key changes you have made in the response to the editor and other reviewers. Also note any major changes you made during the revision that didn’t stem from reviewer comments (e.g., because you thought of other beneficial changes yourself). Don’t expect reviewers to read the other reports and your replies to them (though they often may do that). Outlining changes made in response to the editor and other reviewers provides insurance, among other things: if a reviewer dislikes a change, they are much less likely to go after you if the change was made in response to another reviewer’s suggestion.
  • Make it easy for reviewers and the editor to see exactly what was changed. Aim to minimize the number of times the reviewer has to flip back and forth between your reply and the paper. Put copies of new tables/figures into the reply. Always note the page/line numbers that have changed. If the change is short (e.g., you added or revised a couple of sentences or added a new paragraph), paste the new language into the response document. (Don’t paste entire sections or multiple paragraphs that have been edited though.)
  • If you decided that a comment is not feasible to address, provide an objective explanation as to why. Don’t just write something along the lines of “We decided it would be better to not implement this suggestion.” without an explanation.
  • Avoid sounding defensive. For example, instead of writing “Although this issue was essentially addressed in Table 1, we have now added additional analysis to our supplementary materials”, write simply “We have now added additional analysis to our supplementary materials.”
  • Be professional no matter what. In many cases, the reviewers know who you are, and you may be interacting with them for years to come (without knowing it!). The editor definitely knows who you are, and unprofessional behavior can cost you. Even if the reviewers are rude, do not stoop to their level.
  • Try to make the responses as short as possible (but not shorter). This means editing them like you might a manuscript.
  • Remind the editor that you’re open to alternative ways of implementing the suggestions. If something you did as part of the revision is only in the replies, note that you chose not to put it in the paper but also that you would be open to doing so should the editor think it desirable. If you cut something to stay within the page/word limit, note that you’d be open to bringing it back if the editor prefers you to cut something else. You don’t need to state this for literally every single change, but a broad statement to that extent in the editor reply can only help you.

I know this sounds like a lot, but once you’ve used this approach a few times in R&Rs, it gets easier. Good luck!

Representation of countries in economics articles

Posted on by tatyana

Today, we’re using Academic Sequitur data to examine the representation of countries in economics articles. The exercise is simple*: if a country name appears in the title or abstract, we count that article as representing that country. An article can represent more than one country. Rather than looking at the total count of articles, we scale the counts by the country’s 2020 population, as reported by the World Bank, and exclude countries with fewer than 5 million people. The final sample includes 115 countries. (By the way, if you’re interested in a particular country, you can use Academic Sequitur to keep track of new articles matching that country!)

The graph below shows the top 20 countries, using 2019-2021 article data. It is perhaps unsurprising that Denmark and Sweden are almost at the very top of this chart, as these countries have notoriously rich administrative data used by a large number of empirical papers. The list consists of only highly-developed high-income countries. What is more surprising, perhaps, is that the United States is not in the top 10.

But, you say, the economics literature is not accused of being US-centric in general. It’s the top journals that overemphasize the US at the expense of other countries. The next graph shows the ratio of articles corresponding to a given country in the top 5 economics journals versus overall (so the scale of the x-axis is not necessarily informative). For clarity, we stick with the countries in the graph above. Indeed, the US now ranks #1. The bottom 6 countries–the highest of which was ranked #5 in the graph above–have zero articles in the top 5.

The last graph shows the bottom 20 countries, as defined by articles per million inhabitants. No high-income country makes this list, and the reasons for why some countries are near the bottom are clear (data from Somalia and Yemen are not readily available, to put it mildly). Ultimately, it’s not clear how much attention from economists any given country should get, so we leave it up to you to judge these patterns for yourself.

*Nothing is ever simple, of course. We also attribute articles to the United States/Canada if they mention a US state/Canadian province. We also consider country capitals and attribute an article to its respective country if a country capital is mentioned. Finally, abbreviations such as “US” and “UK” are also counted.

A new way of ranking journals*

Posted on by tatyana

Journal rankings can be controversial. At the same time, the quality of a journal in which one’s research is published is generally thought to be very important for a researcher’s career, and many researchers are thus rightly concerned about it.

Here at Academic Sequitur, we came up with a new way (to the best of our knowledge) to think about how a journal is perceived by the profession. This blog post focuses on the field of economics. We start with the premise that the top 5 economic journals (AER, Econometrica, JPE, QJE, and ReStud) are really the best in the profession, on average. Then we calculate what percent of authors who published in another journal have at least one “top 5” publication. The higher than number is, the more likely it is to be a high-quality journal. In non-top-5 journals, we considered all articles published since 2018 (results are similar if we start in 2009, when the AEJs were started). For calculating whether an author has a top-5 publication, we used top 5 articles since the year 2000.

Before I show you the results, it’s important to note that one thing which can affect this ranking is the topic of the journal under consideration. If a journal’s focus is not “sexy” enough for a top 5 journal, that’s likely going to lower its ranking. Whether this is a feature or a bug I will let you decide.

So with that, here are journals that come out on top, based on the overall proportion of authors publishing at least one article in any top 5 (the rest of the columns show the journal-specific proportions).

Four interesting things about these rankings: First, the shares are really high for eight out of ten of these journals, with about half the authors having at least one top 5. To me, this pattern suggests that we definitely shouldn’t overlook the non-top-5-journals when looking for quality articles. Second, these eight journals are pretty close to each other according to this metric, suggesting that quality differences between them are not large. Third, this metric largely aligns with what I think are the general perceptions of applied microeconomists in North America, with one exception: we seem to be giving the Journal of the European Economic Association less credit than it deserves. Fourth, the rankings would definitely change if we used specific journals for comparison rather than the overall top-5 metric.

Here’s the next set of journals. Keep in mind that we didn’t perform these calculations for all journals in our database (this is just a blog post, after all). So if you don’t see your favorite journal, that doesn’t mean it’s ranked lower than these. It just means we didn’t calculate a ranking for it. But if you’d like, leave us a comment and we’ll tell you how your favorite journal ranks!

While this metric is unlikely to be perfect, it is also unlikely to be worse than citation impact measures. And its benefit for economics specifically is that it isn’t as affected by publication lags as a citation-based measure. What do you think?

When to give up on a paper

Posted on by tatyana

Following the publication of the post on where to submit your paper, someone asked, “How do you know when it’s time to give up on a paper?”

This is a really hard question. We put a lot of work into our papers (I’m assuming in this post that it is a completed paper) and, despite the theoretical wisdom of “Ignore sunk costs”, it’s difficult to let go of months or years of hard work no matter how bleak things look. But there’s also no magic number of rejections beyond which it’s clear that you should just give up. Here are my two cents on how to make the decision.

First, here’s a clever trick I use to make “giving up” on a paper easier psychologically – I have never permanently given up on a paper. But I do have four papers and a lot of never-made-it-to-paper-stage-projects “on the back burner”. I haven’t worked on them for years and don’t plan on doing so unless I have nothing better to do. In other words, instead of asking the hard question of “Should I never try to publish this paper again?”, ask the easier question of “Should I prioritize other projects over this paper for now?” I always have the option to pull papers out of the “back burner” folder, but lo and behold, I keep having better projects to work on and don’t think much about the archived ones.

Of course, that still leaves the question of “Should I prioritize other projects over this paper?” open. I’ll discuss three related cases where this question becomes relevant and offer some general guidance for how to decide.

#1 Your paper has gotten rejected multiple (let’s say at least five) times for roughly the same reason, you don’t think you can do anything to address that shortcoming, and you have other, more promising, projects/ideas. If that reason is “this paper isn’t making enough of a contribution” AND you’ve revised your introduction substantially in between submissions to make the best possible case for your contribution, this may be a sign that it’s time to drop down a tier (though see some discussion below on when this is a good idea). At the same time, the contribution of a paper is hugely subjective. If the only thing reviewers find wrong with your paper is the contribution, then trying another journal within the same tier is fairly low-cost, assuming your contribution is actually within the realm of what gets published by the tier of journals you’ve been submitting to. Here, talking senior colleagues is especially helpful.

If the reason your paper keeps getting rejected is something related to the paper’s data/methodology – for example, no one believes your instrument, no matter how many robustness or placebo tests you’ve added – then dropping down a tier is also an option, but is less likely to be a successful strategy. I came close to giving up on a paper because no one seemed to like the IV. I ultimately decided to keep trying though because (a) a lot of the rejections were desk rejections, allowing me to re-submit without revising (since there was no real feedback given) and (b) I believed in the instrument myself and thought we made a good case for it. After six rejections, the paper was published.

By contrast, if your paper is getting rejected for diverse reasons, it is probably good to keep trying (though in that case I would recommend taking a close look at the writing to make sure your exposition is clear).

#2 You feel that your paper would only be publishable if you dropped to a tier of journals where your current colleagues generally don’t publish, you have other, more promising, projects/ideas. (Presumably, you think you need to drop down a tier because of numerous rejections. Otherwise, perhaps you are underestimating your paper!) For better or worse, publishing in a journal that your department really looks down on is sometimes viewed as a negative. So, if you otherwise have a good chance of getting tenure at your department (and want to get tenure at your department), you may want to put the project down and move on to something else. Two of my archived papers were archived for this reason.

#3 It looks like the path to publication in an acceptable-tier journal would be painful and you have other, more promising, projects/ideas. Maybe your case is not as extreme as the two cases above: you’ve had 3-4 rejections, you feel like you may have a shot at an acceptable but not stellar journal tier but, given the feedback you’ve gotten so far, you have a gut feeling that it would be painful for various reasons. Maybe a ref said the paper is not well-written and after taking a close look, you realize that the ref is right and that the whole paper needs an overhaul (I speak from experience). Maybe you have your own misgivings about the methodology/data and feel like an overhaul there is warranted. If you have other great projects in the pipeline with a lower cost-benefit ratio, by all means feel free to prioritize them. No one said you have to publish every paper you write.

Yes, I put “you have other, more promising, projects/ideas” in every entry on purpose. If you don’t have any other projects or ideas that have a reasonable shot at publishing at the same tier or higher than what you’ve been submitting to, then keep working on publishing the paper, even if it means a major overhaul. Use the suggestions I wrote about in a previous post on what to do after a rejection. While you wait for reviews, work on new projects and ideas and if a better one comes along and your submission gets rejected, by all means abandon the project.

A final word of caution is in order. According to my scientifically constructed chart below, our level of excitement about a project is always highest at the idea stage, when the promise seems unlimited and the pitfalls and barriers to getting there are not salient. So, if you find yourself constantly putting completed papers on the back burner and picking up new shiny ideas, stop! Go back to your best completed paper and publish it (and work on the shiny new ideas while you wait for reviews). Then repeat until you have a few publications.