Tag: academic journals

A new way to compare journals

Posted on by tatyana

Journal rankings are often based on average citation metrics. But a journal can have high citation counts because it published a few great articles and many mediocre ones or because it published many good articles. A single number per journal also cannot be used to figure out how much overlap there is in the distribution of citations across journals. Of course, citations are far from a perfect quality metric, but they are arguably the best metric we currently have, and authors who are more highly cited are certainly more influential in the profession, on average.

I investigated the full distribution of journals’ citations, combining Academic Sequitur data with citation data from Semantic Scholar.* The exercise is simple: take all the papers that a journal has published in the past 5 years (2016-2021) and calculate the share of papers with fewer than C citations per year for various values of C. Plotting the shares against C gives a cumulative distribution function (CDF) of citations for each journal. The closer a journal’s CDF is to the x-axis, the more highly cited papers it has.

The CDFs for the top 5 economics journals is shown below. It’s very clear that the Quarterly Journal of Economics leads the pack, with many more highly cited papers than the other four journals across the distribution of C. For “moderate” citation numbers, the Journal of Political Economy is next, but it has fewer highly cited articles than the other top 5s. American Economic Review and Review of Economic Studies are pretty close together, but AER has more papers with very few citations and more papers with large numbers of citations. Econometrica has substantially more papers that are cited infrequently than the other four journals, but catches up to three of them at the top of the distribution.

The next obvious exercise is to compare the distribution of citations in the top 5 journals to other journals. If people like these graphs, I will follow up with more journals, but let’s start with the AEJs. To keep things simple, I combined the top 5 journal data into one CDF (which is overall pretty similar to AER‘s CDF).

Couple of things are worth highlighting. Even though AEJ: Macro has a relatively high number of low-cited articles, it’s a powerhouse when it comes to highly cited articles, outperforming the top 5 index. The best articles in AEJ: Applied are as good as those in the top 5 (at least up to the 100 cites/year cutoff). AEJ: Applied also has a notably better distribution of citations than AEJ: Policy. Finally, AEJ: Micro is clearly the worst of the bunch, although perhaps theory papers are just cited less frequently?

And, of course, there’s a good amount of overlap across journals. If your paper is being cited 15 times per year, it’s doing better than 60 percent of the papers in the top 5! About 31 percent of AEJ: Macro papers, 28 percent of AEJ: Applied papers, 24 percent of AEJ: Policy papers, and 9 percent of AEJ: Micro papers have 15 or more citations per year, so we’re not talking peanuts here. Such overlaps are why I think it’s so important to follow new relevant papers across a variety of journals.

What do you think about this way of visualizing journal quality? Which journal(s) do you want to see the distribution for?

*Due to the large number of articles, the matching process was automated, so the data aren’t perfect (e.g., some articles could not be matched). But any mis-matches are unlikely to substantively affect conclusions.

Representation of countries in economics articles

Posted on by tatyana

Today, we’re using Academic Sequitur data to examine the representation of countries in economics articles. The exercise is simple*: if a country name appears in the title or abstract, we count that article as representing that country. An article can represent more than one country. Rather than looking at the total count of articles, we scale the counts by the country’s 2020 population, as reported by the World Bank, and exclude countries with fewer than 5 million people. The final sample includes 115 countries. (By the way, if you’re interested in a particular country, you can use Academic Sequitur to keep track of new articles matching that country!)

The graph below shows the top 20 countries, using 2019-2021 article data. It is perhaps unsurprising that Denmark and Sweden are almost at the very top of this chart, as these countries have notoriously rich administrative data used by a large number of empirical papers. The list consists of only highly-developed high-income countries. What is more surprising, perhaps, is that the United States is not in the top 10.

But, you say, the economics literature is not accused of being US-centric in general. It’s the top journals that overemphasize the US at the expense of other countries. The next graph shows the ratio of articles corresponding to a given country in the top 5 economics journals versus overall (so the scale of the x-axis is not necessarily informative). For clarity, we stick with the countries in the graph above. Indeed, the US now ranks #1. The bottom 6 countries–the highest of which was ranked #5 in the graph above–have zero articles in the top 5.

The last graph shows the bottom 20 countries, as defined by articles per million inhabitants. No high-income country makes this list, and the reasons for why some countries are near the bottom are clear (data from Somalia and Yemen are not readily available, to put it mildly). Ultimately, it’s not clear how much attention from economists any given country should get, so we leave it up to you to judge these patterns for yourself.

*Nothing is ever simple, of course. We also attribute articles to the United States/Canada if they mention a US state/Canadian province. We also consider country capitals and attribute an article to its respective country if a country capital is mentioned. Finally, abbreviations such as “US” and “UK” are also counted.

Who is publishing in AER: Insights? An update

Posted on by tatyana

Over a year ago, I wrote a post tabulating the share of AER: Insights authors who have also published in a top-5 journal*. (The answer was 67%, significantly higher than most other journals, except those that generally solicit papers, like the Journal of Economic Literature.)

Now that AER: Insights is in its second year of publishing and has 60 forthcoming/published articles, I decided to revisit this question, again using Academic Sequitur data. The graph below shows the percent of authors that (a) have published/are forthcoming in a given journal in 2018-2020 and (b) have had at least one top-5 article published since 2000. The journals below are the top ten journals based on that metric.

With a score of 66%, AER: Insights still has the highest share of top-5 authors among journals where submissions are not generally solicited.** The next-highest journal, Theoretical Economics, is five percentage points behind. (There is some indication that the share for AER: Insights is coming down: for articles accepted in 2020, the top-5 share was “only” 60%.)

What if we condition on having two or more top-5 publications? That actually causes AER: Insights to move up in the ranking, overtaking Brookings Papers on Economic Activity.

Whether this pattern exists because AER: Insights is extremely selective or because less-established scholars are reluctant to submit their work to a new-ish journal or for some other reason is impossible to know without submission data. But no matter how you look at it, the group currently publishing in AER: Insights is quite elite.

*Top 5 is defined as American Economic Review, Econometrica, Journal of Political Economy, Quarterly Journal of Economics, and Review of Economic Studies.

**AER: Insights would be even higher-ranked by this metric (#3) if we ignored top-5 publications in American Economic Review. Therefore, this pattern is not driven by the fact that both journals are published by the AEA.

Political Science Journal Rankings

Posted on by tatyana

How do we judge how good a journal is? Ideally by the quality of articles it publishes. But the best systematic way of quantifying quality we’ve come up with so far are citation-based rankings. And these are far from perfect, as a simple Google Search will reveal (here’s one such article).

I’ve been using Academic Sequitur data to experiment with an alternative way of ranking journals. The basic idea is to calculate what percent of authors who published in journal X have also published in a top journal for that discipline (journals can also be ranked relative to every other journal, but the result is more difficult to understand). As you might imagine, this ranking is also not perfect, but it has yielded very reasonable results in economics (see here).

Now it’s time to try this ranking out in a field outside my own: Political Science. As a reference point, I took 3 top political science journals: American Political Science Review (APSR), American Journal of Political Science (AJPS), and Journal of Politics (JOP). I then calculated what percent of authors who published in each of 20 other journals since 2018 have also published a top-3 article at any point since 2000.

Here are the top 10 journals, according to this ranking (the above-mentioned stat is in the first column).


Quarterly Journal of Political Science and International Organization come out as the top 2. This is noteworthy because alternative lists of top political science journals suggested to me included these two journals! Political Analysis is a close second, followed by a group of 5 journals with very similar percentages overall (suggesting similar quality).

Below is the next set of ten. Since this is not my research area, I’m hoping you can tell me in the comments whether these rankings are reasonable or not! Happy publishing.

Finally, here’s an excel version of the full table, in case you want to re-sort by another column. Note that if a journal is not listed, that means I did not rank it. Feel free to ask about other journals in the comments.