A new way to compare journals

Journal rankings are often based on average citation metrics. But a journal can have high citation counts because it published a few great articles and many mediocre ones or because it published many good articles. A single number per journal also cannot be used to figure out how much overlap there is in the distribution of citations across journals. Of course, citations are far from a perfect quality metric, but they are arguably the best metric we currently have, and authors who are more highly cited are certainly more influential in the profession, on average.

I investigated the full distribution of journals’ citations, combining Academic Sequitur data with citation data from Semantic Scholar.* The exercise is simple: take all the papers that a journal has published in the past 5 years (2016-2021) and calculate the share of papers with fewer than C citations per year for various values of C. Plotting the shares against C gives a cumulative distribution function (CDF) of citations for each journal. The closer a journal’s CDF is to the x-axis, the more highly cited papers it has.

The CDFs for the top 5 economics journals is shown below. It’s very clear that the Quarterly Journal of Economics leads the pack, with many more highly cited papers than the other four journals across the distribution of C. For “moderate” citation numbers, the Journal of Political Economy is next, but it has fewer highly cited articles than the other top 5s. American Economic Review and Review of Economic Studies are pretty close together, but AER has more papers with very few citations and more papers with large numbers of citations. Econometrica has substantially more papers that are cited infrequently than the other four journals, but catches up to three of them at the top of the distribution.

The next obvious exercise is to compare the distribution of citations in the top 5 journals to other journals. If people like these graphs, I will follow up with more journals, but let’s start with the AEJs. To keep things simple, I combined the top 5 journal data into one CDF (which is overall pretty similar to AER‘s CDF).

Couple of things are worth highlighting. Even though AEJ: Macro has a relatively high number of low-cited articles, it’s a powerhouse when it comes to highly cited articles, outperforming the top 5 index. The best articles in AEJ: Applied are as good as those in the top 5 (at least up to the 100 cites/year cutoff). AEJ: Applied also has a notably better distribution of citations than AEJ: Policy. Finally, AEJ: Micro is clearly the worst of the bunch, although perhaps theory papers are just cited less frequently?

And, of course, there’s a good amount of overlap across journals. If your paper is being cited 15 times per year, it’s doing better than 60 percent of the papers in the top 5! About 31 percent of AEJ: Macro papers, 28 percent of AEJ: Applied papers, 24 percent of AEJ: Policy papers, and 9 percent of AEJ: Micro papers have 15 or more citations per year, so we’re not talking peanuts here. Such overlaps are why I think it’s so important to follow new relevant papers across a variety of journals.

What do you think about this way of visualizing journal quality? Which journal(s) do you want to see the distribution for?

*Due to the large number of articles, the matching process was automated, so the data aren’t perfect (e.g., some articles could not be matched). But any mis-matches are unlikely to substantively affect conclusions.

Just how significant are publication lags?

Generally, Academic Sequitur finds papers as soon as they are posted online. Increasingly, journals are posting papers as soon as they are accepted and correctly formatted (some even before then!), which means that when the “official” new issue is announced, the papers it consists of could have been hanging out on the web for months without being publicized. This month, I checked just how large this lag can be by seeing when Academic Sequitur found papers that were included in journals’ most recent issues.

I checked the top 5 economics journals: American Economic Review, Quarterly Journal of Economics, Journal of Political Economy, Econometrica, and Review of Economic Studies. For the January 2019 issue of AER, the included articles were all added to our database between July 10 and August 27, 2018 (corresponding to the dates they were posted). For the February 2019 issue of QJE, we found all the articles between August 20 and October 25, 2018. For the December 2018 issue of Econometrica, articles were found between June 4and December 12, 2018. For the December issue of the Journal of Political Economy, we located all articles between August 2 and November 21, 2018. Finally, for Review of Economic Studies, articles from the January 2019 issue were located between January 28, 2018 (yes, almost a year early!) and November 28, 2018.

Our users find out about articles at the time they are posted, not when they get grouped into an issue after languishing online for months. And I think that’s a huge plus!

Why I made Academic Sequitur

Mitch Kapor wrote “Getting information off the Internet is like taking a drink from a fire hydrant.” My own experience with this when it comes to staying up to date on research couldn’t be more accurate.

I started as an Assistant Professor at the University of Illinois in 2011. Somewhere around 2015, I decided I should be more systematic about keeping up to date with recent literature in my field. Up until then, I relied on conferences, a few mailing lists, and colleagues forwarding me papers to learn about what was happening in environmental economics and in the profession as a whole. But I still felt like I was missing some important papers (indeed, I would periodically learn of a relevant paper that was published several years ago).

My initial solution to this problem was to spend a few hours signing up for various journals’ tables of contents. I figured I should track some of the top general-interest economics journals as well as the top journals in my field. So I signed up for: Quarterly Journal of Economics, American Economic Review, Journal of Political Economy, Review of Economic Studies, American Economic Journal: Applied Microeconomics, American Economic Journal: Economic Policy, Journal of the European Economic Association, Journal of Environmental Economics and Management, Journal of the Association of Environmental and Resource Economists, Journal of Labor Economics, Journal of Risk and Uncertainty, NBER Weekly Paper Digest, and SSRN’s “environmental economics” list, which notified me about new pre-prints classified as environmental economics. I knew there might be some other papers I would miss, but I was confident that I would see most of them.

What happened next was equally disappointing to hearing about too few papers: I was overwhelmed by the number of irrelevant papers that came my way. The digests from various journals would arrive throughout the month and then sit there waiting for me to find the time to sort through them. Often, I would spend half an hour or more just skimming the titles and abstracts to find papers I was interested in. My brain hurt. Sometimes I would just delete the digest, overwhelmed by the task.

At some point in 2016, I decided to stop being a passive sufferer and do something about this problem, not just for myself but for others. Two years later, Academic Sequitur was born. I love our solution because it is very straightforward and transparent. Our users specify authors, journals, and/or keywords that they want to follow, and we then notify them of newly published relevant papers that meet their criteria in a daily or weekly digest.

One awesome aspect of this solution is that one can vastly expand the set of journals from which personalized updates are pulled without worrying too much about information overload (you can of course still create information overload by selecting too many journals). We don’t do any fancy machine learning, and I think that’s a feature not a bug: people can be sure that papers meeting their criteria will be delivered to their inbox and they won’t miss anything they care about.

With the amount of information out there, you would think it would be easy to stay informed. But we are not supercomputers who can process terabytes of information and distill it to what’s important ourselves. Science cannot progress as quickly if the right information is not being delivered to and absorbed by the right people. Both researchers and society need better tools for disseminating research results, and I’m proud that Academic Sequitur is a part of that.

Tatyana Deryugina, founder of Academic Sequitur