Part I: agricultural economics, lab experiments, field experiments & economics of education
Most of us have a sense that it is more difficult to get certain topics published in the top 5 economics journals (American Economic Review, Econometrica, Journal of Political Economy, Quarterly Journal of Economics, and Review of Economic Studies), but there is not much hard data on this. And if a particular topic appears infrequently in top journals, it may simply be because it’s a relatively rare topic overall.
To get more evidence on this issue, I used Academic Sequitur data, which covers the majority of widely-read journals in economics. The dataset I used contains articles from 139 economics journals and spans the years 2000-2019. On average, 6 percent of the papers in the dataset were published in a top 5 journal.
I classified papers into topics based on the presence of certain keywords in the abstract and title.* I chose the keywords carefully, aiming to both minimize the share of irrelevant articles and to minimize the omission of relevant ones. While there is certainly some measurement error, it should not bias the results. (Though readers should think of this as a “fun-level” analysis rather than a “rigorously peer-reviewed” analysis.)
I chose topics based on suggestions in response to an earlier Tweet of mine. To keep things manageable, I’m going to focus on a few topics at a time. To start off, I looked at agricultural economics (5.3% of articles in the dataset), field experiments (1.0% of articles), lab experiments (1.9% of articles), and education (1.8% of articles). I chose these to have some topic diversity and also because these topics were relatively easy to identify.** I then ran a simple OLS regression of a “top 5” indicator on each topic indicator (separately).***
The results are plotted in a graph below. Field experiments are much more likely to publish in a top 5 journal than in the other 134 journals (about 5 percentage points more likely!), while lab experiments are much less likely. Education doesn’t seem to be favored one way or the other, while agriculture is penalized about as much as field experiments are rewarded. Moral of the story: if you want to publish an ag paper in a top 5, make it a field experiment!
Now you might be saying, “I can’t even name 139 economics journals, so maybe this isn’t the relevant sample on which to run this regression.” Fair point (though see here for a way way longer list of econ journals). To address this, I restricted the set of journals to the 20 best-known general-interest journals—including the top 5—and re-generated the results.**** With the exception of lab experiments, the picture now looks quite different: both field experiments and education research are penalized by the top 5 journals, but agriculture is not.
Combining the two sets of results together, we can conclude that the top 5 penalize agricultural economics research but so do the other good general-interest journals. The top 5 journals also penalize field experiments relative to other good general-interest journals, but top general-interest journals as a whole rewards field experiments relative to other journals. Finally, top 5 journals penalize education relative to other good general-interest journals, but not relative to the field as a whole.
The second set of results is obviously sensitive to the set of journals considered. If I were to add field journals like the American Journal of Agricultural Economics, things would again look much worse for ag. And how much worse they look for a particular topic depends on how many articles the field journal publishes. So I prefer the most inclusive set of journals, but I welcome suggestions about which set of journals to use in future analyses! Would also love to hear everyone’s thoughts on this exercise in general, so please leave a comment.
——————————————————————————————————————–
Endnotes
*I did not use JEL codes because many journals do not require or publish these and we therefore do not collect them. JEL codes are also easier to select strategically than the words in the title and abstract.
** An article falls into the category of agricultural economics if it contains any of the following words/phrases in the abstract or title (not case-sensitive, partial word matches count): “farm”, “crop insurance”, “crop yield”, “cash crop”, “crop production”, “crops production”, “meat processing”, “dairy processing”, “grain market”, “crop management”, “agribusiness”, “beef”, “poultry”, “hog price”, “cattle industry”, “rice cultivation”, “wheat cultivation”, “grain cultivation”, “grain yield”, “crop diversity”, “soil conditions”, “dairy sector”, “hectare”, “sugar mill”, “corn seed”, “soybean seed”, “maize production”, “soil quality” “agricultural chemical use”, “forest”. Field experiment: “field experiment”, “experiment in the field”. Lab experiment: “lab experiment”, “laboratory experiment”, “experimental data”, “randomized subject”, “online experiment”. Education: “return to education”, “returns to education”, “college graduate”, “schooling complet”, “teacher”, “kindergarten”, “preschool”, “community college”, “academic achievement”, “academic performance”, “postsecondary”, “educational spending”, “student performance”, “student achievement”, “student outcome”, “student learning”, “higher education” “educational choice”, “student academic progress”, “public education”, “school facilit”, “education system”, “school voucher” “private school”, “school district”, “education intervention”. Articles may fall into multiple categories.
*** Standard errors are heteroskedasticity-robust
**** The 15 additional journals are (in alphabetical order): American Economic Journal: Applied Economics, American Economic Journal: Economic Policy, American Economic Journal: Macroeconomics, American Economic Journal: Microeconomics, American Economic Review: Insights, Economic Journal, Economic Policy, Economica, European Economic Review, Journal of the European Economic Association, Oxford Economic Papers, Quantitative Economics, RAND Journal of Economics, Review of Economics and Statistics, Scandinavian Journal of Economics.