Category: Academic Journals

More journal comparisons

Posted on by tatyana

A few weeks ago, I wrote a blog post on comparing journals by using the full distribution of their articles’ citations. I’m working on putting together a concise summary of these distributions, but in the meantime I thought I’d show two more interesting comparisons.

The first compares the 2016-2021 distribution of top-5 econ journals’ annual citations to those of the top-3 finance journals. Many finance departments view a top-3 finance publication as similar to a top-5 econ publication, even though one would think the latter is harder to achieve. But how much worse-cited are the top 3 finance journals?

The distribution is below. Turns out, the Journal of Finance looks better than the top-5 index for most of the distribution! The Journal of Financial Economics and the Review of Financial Studies are noticeably worse, but RFS packs a strong punch at the top of the distribution. Overall, publishing in JF looks like a great goal to aim for.

Next up, let’s check out some other good journals (American Economic Journal: Economic Policy, Review of Economics and Statistics, Journal of Public Economics, and the Journal of the European Economic Associations) and see how they compare to each other. (For a comparison of AEJs to each other, see previous blog post.)

Each of these journals is clearly dominated by the top 5 index (and I had to shorten the x-axis to make the lines easier to see). And within the set of journals, there are some clear winners. AEJ: Policy leads the pack. ReStat looks a bit better than JEEA but overall the two distributions are similar. JPubE comes in last.

Even though these graphs show clear stochastic dominance in some cases, it’s worth noting that if your article is being cited more than 10 times per year, it’s doing better than about 50% of top-5 articles! About 35% of AEJ: Policy articles, 25% of ReStat and JEEA articles, and 20% of JPubE articles are cited more than 10 times per year. Of course, parsing that many journals’ tables of contents can be overwhelming. Making it much easier to learn about relevant articles published in less prestigious journals is one reason I created Academic Sequitur. Check it out to make sure you’re not missing great articles you should know about!

A new way to compare journals

Posted on by tatyana

Journal rankings are often based on average citation metrics. But a journal can have high citation counts because it published a few great articles and many mediocre ones or because it published many good articles. A single number per journal also cannot be used to figure out how much overlap there is in the distribution of citations across journals. Of course, citations are far from a perfect quality metric, but they are arguably the best metric we currently have, and authors who are more highly cited are certainly more influential in the profession, on average.

I investigated the full distribution of journals’ citations, combining Academic Sequitur data with citation data from Semantic Scholar.* The exercise is simple: take all the papers that a journal has published in the past 5 years (2016-2021) and calculate the share of papers with fewer than C citations per year for various values of C. Plotting the shares against C gives a cumulative distribution function (CDF) of citations for each journal. The closer a journal’s CDF is to the x-axis, the more highly cited papers it has.

The CDFs for the top 5 economics journals is shown below. It’s very clear that the Quarterly Journal of Economics leads the pack, with many more highly cited papers than the other four journals across the distribution of C. For “moderate” citation numbers, the Journal of Political Economy is next, but it has fewer highly cited articles than the other top 5s. American Economic Review and Review of Economic Studies are pretty close together, but AER has more papers with very few citations and more papers with large numbers of citations. Econometrica has substantially more papers that are cited infrequently than the other four journals, but catches up to three of them at the top of the distribution.

The next obvious exercise is to compare the distribution of citations in the top 5 journals to other journals. If people like these graphs, I will follow up with more journals, but let’s start with the AEJs. To keep things simple, I combined the top 5 journal data into one CDF (which is overall pretty similar to AER‘s CDF).

Couple of things are worth highlighting. Even though AEJ: Macro has a relatively high number of low-cited articles, it’s a powerhouse when it comes to highly cited articles, outperforming the top 5 index. The best articles in AEJ: Applied are as good as those in the top 5 (at least up to the 100 cites/year cutoff). AEJ: Applied also has a notably better distribution of citations than AEJ: Policy. Finally, AEJ: Micro is clearly the worst of the bunch, although perhaps theory papers are just cited less frequently?

And, of course, there’s a good amount of overlap across journals. If your paper is being cited 15 times per year, it’s doing better than 60 percent of the papers in the top 5! About 31 percent of AEJ: Macro papers, 28 percent of AEJ: Applied papers, 24 percent of AEJ: Policy papers, and 9 percent of AEJ: Micro papers have 15 or more citations per year, so we’re not talking peanuts here. Such overlaps are why I think it’s so important to follow new relevant papers across a variety of journals.

What do you think about this way of visualizing journal quality? Which journal(s) do you want to see the distribution for?

*Due to the large number of articles, the matching process was automated, so the data aren’t perfect (e.g., some articles could not be matched). But any mis-matches are unlikely to substantively affect conclusions.

How to address reviewer comments

Posted on by tatyana

You got a revise-and-resubmit request from a journal – congratulations! What now? Here are step by step suggestions to maximize the probability of converting that R&R into a publication and minimize the number of revision rounds.

Preparing to write the reply

  • Copy-paste the reviewer comments into Word, Latex, or whatever you write in. Format the comments to distinguish them from the reply you will write (e.g. make the comments italic or bold). If the editor provided specific comments, do this with his/her comments as well.
  • Write what you’re going to do in response to each comment following the comment and make a to-do list based on that. Now you’ve got a super-rough draft of the reply!

Deciding how/if to address a comment

  • Start with the assumption that each comment is valid: the reviewer is innocent until proven guilty! In my experience as both author and reviewer, it’s not uncommon for authors to glance over a comment and dismissively conclude that a reviewer has misunderstood something about the paper. Consider the possibility that it is you who has misunderstood the comment and re-read it carefully. In general, carefully re-read each comment in the beginning, middle, and end of the revision process to make sure you haven’t misunderstood the point. If, after carefully considering a comment, you continue to think that it reflects a misunderstanding of the paper, try to figure out why the reviewer misunderstood your paper. Sure, reviewers can be careless, but just as often authors might think something in the paper is clear when it is not. In this case, carefully edit the part of the paper that may have confused the reviewer and make your reply to the comment something along the lines of “We apologize for the confusion. In fact, [EXPLAIN]. We have now revised lines/sections/pages X-Y to make this clearer.” Sometimes this is as simple as moving something from a footnote in the back of the paper to earlier in the paper or adding a footnote about something in the supplementary materials.
  • Remember that the editor is the one ultimately in charge. If an editor tells you not to address a particular comment, don’t address it. If an editor highlights a comment as specifically important, pay particular attention to it. If an editor has not said anything about a particular comment, assume that they want you to address it.
  • Address every comment unless it is impossible or the editor told you not to do it. Assume that the reviewers are acting in good faith and giving you feedback to improve your paper. Note that “addressing” a comment does not always mean you do exactly what the comment says. For example, if a reviewer says that “The analysis sample should be limited to X” and you think there are good objective reasons to keep your current sample, you can address the comment by showing results with sample X in the reply to the reviewer and clearly explaining why you believe it’s not the best sample to focus on.
  • Err on the side of comprehensiveness. There are no page limits when it comes to reviewer replies (this is not an invitation to overwhelm the reviewers by making the reply unnecessarily long though!), and if you decide that some exercise suggested by the reviewers isn’t important enough for the manuscript, go ahead and include the results of the exercise in the reply. A common phrase in my replies has been “To keep the length of the manuscript manageable, we have decided to not include this exercise in the paper.” but it always follows a reply where the results are shown to the reviewers!
  • Sometimes reviewer and editor requests can be burdensome, e.g., if you’re asked to run another experiment or collect more data. Ultimately, it’s your paper and your career, so you decide where the limits are, but keep in mind that by choosing to not address a particular comment, you weakly increase the risk of rejection.
  • If you’re in doubt about what a comment is asking you to do even after reading it carefully, ask a senior colleague to take a look.

You should not view the editor as someone you can email back and forth with whenever you want (they’re busy!), but there are times when it’s appropriate to send the editor an email before completing your revision

  • When reasonable reviewer suggestions contradict each other, but the editor did not clarify which direction you should take.
  • When a comment was highlighted as particularly important to address by the editor, but you don’t view it as feasible. Better to explain to the editor why you can’t do it and ask him or her up front if it’s a deal-breaker so you don’t spend time on all the other revisions only to be rejected.
  • When the required revisions are substantial, the suggestions are vague, and you want to run your revision plan by the editor before executing it.

Finally, some specific suggestions on how to address reviewers

  • Thank the reviewer at the beginning of your reply. They read your paper and provided comments!
  • Start the reply to each reviewer by outlining the key changes you have made in the response to the editor and other reviewers. Also note any major changes you made during the revision that didn’t stem from reviewer comments (e.g., because you thought of other beneficial changes yourself). Don’t expect reviewers to read the other reports and your replies to them (though they often may do that). Outlining changes made in response to the editor and other reviewers provides insurance, among other things: if a reviewer dislikes a change, they are much less likely to go after you if the change was made in response to another reviewer’s suggestion.
  • Make it easy for reviewers and the editor to see exactly what was changed. Aim to minimize the number of times the reviewer has to flip back and forth between your reply and the paper. Put copies of new tables/figures into the reply. Always note the page/line numbers that have changed. If the change is short (e.g., you added or revised a couple of sentences or added a new paragraph), paste the new language into the response document. (Don’t paste entire sections or multiple paragraphs that have been edited though.)
  • If you decided that a comment is not feasible to address, provide an objective explanation as to why. Don’t just write something along the lines of “We decided it would be better to not implement this suggestion.” without an explanation.
  • Avoid sounding defensive. For example, instead of writing “Although this issue was essentially addressed in Table 1, we have now added additional analysis to our supplementary materials”, write simply “We have now added additional analysis to our supplementary materials.”
  • Be professional no matter what. In many cases, the reviewers know who you are, and you may be interacting with them for years to come (without knowing it!). The editor definitely knows who you are, and unprofessional behavior can cost you. Even if the reviewers are rude, do not stoop to their level.
  • Try to make the responses as short as possible (but not shorter). This means editing them like you might a manuscript.
  • Remind the editor that you’re open to alternative ways of implementing the suggestions. If something you did as part of the revision is only in the replies, note that you chose not to put it in the paper but also that you would be open to doing so should the editor think it desirable. If you cut something to stay within the page/word limit, note that you’d be open to bringing it back if the editor prefers you to cut something else. You don’t need to state this for literally every single change, but a broad statement to that extent in the editor reply can only help you.

I know this sounds like a lot, but once you’ve used this approach a few times in R&Rs, it gets easier. Good luck!

Representation of countries in economics articles

Posted on by tatyana

Today, we’re using Academic Sequitur data to examine the representation of countries in economics articles. The exercise is simple*: if a country name appears in the title or abstract, we count that article as representing that country. An article can represent more than one country. Rather than looking at the total count of articles, we scale the counts by the country’s 2020 population, as reported by the World Bank, and exclude countries with fewer than 5 million people. The final sample includes 115 countries. (By the way, if you’re interested in a particular country, you can use Academic Sequitur to keep track of new articles matching that country!)

The graph below shows the top 20 countries, using 2019-2021 article data. It is perhaps unsurprising that Denmark and Sweden are almost at the very top of this chart, as these countries have notoriously rich administrative data used by a large number of empirical papers. The list consists of only highly-developed high-income countries. What is more surprising, perhaps, is that the United States is not in the top 10.

But, you say, the economics literature is not accused of being US-centric in general. It’s the top journals that overemphasize the US at the expense of other countries. The next graph shows the ratio of articles corresponding to a given country in the top 5 economics journals versus overall (so the scale of the x-axis is not necessarily informative). For clarity, we stick with the countries in the graph above. Indeed, the US now ranks #1. The bottom 6 countries–the highest of which was ranked #5 in the graph above–have zero articles in the top 5.

The last graph shows the bottom 20 countries, as defined by articles per million inhabitants. No high-income country makes this list, and the reasons for why some countries are near the bottom are clear (data from Somalia and Yemen are not readily available, to put it mildly). Ultimately, it’s not clear how much attention from economists any given country should get, so we leave it up to you to judge these patterns for yourself.

*Nothing is ever simple, of course. We also attribute articles to the United States/Canada if they mention a US state/Canadian province. We also consider country capitals and attribute an article to its respective country if a country capital is mentioned. Finally, abbreviations such as “US” and “UK” are also counted.