Category: Research

A new way to compare journals

Posted on by tatyana

Journal rankings are often based on average citation metrics. But a journal can have high citation counts because it published a few great articles and many mediocre ones or because it published many good articles. A single number per journal also cannot be used to figure out how much overlap there is in the distribution of citations across journals. Of course, citations are far from a perfect quality metric, but they are arguably the best metric we currently have, and authors who are more highly cited are certainly more influential in the profession, on average.

I investigated the full distribution of journals’ citations, combining Academic Sequitur data with citation data from Semantic Scholar.* The exercise is simple: take all the papers that a journal has published in the past 5 years (2016-2021) and calculate the share of papers with fewer than C citations per year for various values of C. Plotting the shares against C gives a cumulative distribution function (CDF) of citations for each journal. The closer a journal’s CDF is to the x-axis, the more highly cited papers it has.

The CDFs for the top 5 economics journals is shown below. It’s very clear that the Quarterly Journal of Economics leads the pack, with many more highly cited papers than the other four journals across the distribution of C. For “moderate” citation numbers, the Journal of Political Economy is next, but it has fewer highly cited articles than the other top 5s. American Economic Review and Review of Economic Studies are pretty close together, but AER has more papers with very few citations and more papers with large numbers of citations. Econometrica has substantially more papers that are cited infrequently than the other four journals, but catches up to three of them at the top of the distribution.

The next obvious exercise is to compare the distribution of citations in the top 5 journals to other journals. If people like these graphs, I will follow up with more journals, but let’s start with the AEJs. To keep things simple, I combined the top 5 journal data into one CDF (which is overall pretty similar to AER‘s CDF).

Couple of things are worth highlighting. Even though AEJ: Macro has a relatively high number of low-cited articles, it’s a powerhouse when it comes to highly cited articles, outperforming the top 5 index. The best articles in AEJ: Applied are as good as those in the top 5 (at least up to the 100 cites/year cutoff). AEJ: Applied also has a notably better distribution of citations than AEJ: Policy. Finally, AEJ: Micro is clearly the worst of the bunch, although perhaps theory papers are just cited less frequently?

And, of course, there’s a good amount of overlap across journals. If your paper is being cited 15 times per year, it’s doing better than 60 percent of the papers in the top 5! About 31 percent of AEJ: Macro papers, 28 percent of AEJ: Applied papers, 24 percent of AEJ: Policy papers, and 9 percent of AEJ: Micro papers have 15 or more citations per year, so we’re not talking peanuts here. Such overlaps are why I think it’s so important to follow new relevant papers across a variety of journals.

What do you think about this way of visualizing journal quality? Which journal(s) do you want to see the distribution for?

*Due to the large number of articles, the matching process was automated, so the data aren’t perfect (e.g., some articles could not be matched). But any mis-matches are unlikely to substantively affect conclusions.

Who is publishing in AER: Insights? An update

Posted on by tatyana

Over a year ago, I wrote a post tabulating the share of AER: Insights authors who have also published in a top-5 journal*. (The answer was 67%, significantly higher than most other journals, except those that generally solicit papers, like the Journal of Economic Literature.)

Now that AER: Insights is in its second year of publishing and has 60 forthcoming/published articles, I decided to revisit this question, again using Academic Sequitur data. The graph below shows the percent of authors that (a) have published/are forthcoming in a given journal in 2018-2020 and (b) have had at least one top-5 article published since 2000. The journals below are the top ten journals based on that metric.

With a score of 66%, AER: Insights still has the highest share of top-5 authors among journals where submissions are not generally solicited.** The next-highest journal, Theoretical Economics, is five percentage points behind. (There is some indication that the share for AER: Insights is coming down: for articles accepted in 2020, the top-5 share was “only” 60%.)

What if we condition on having two or more top-5 publications? That actually causes AER: Insights to move up in the ranking, overtaking Brookings Papers on Economic Activity.

Whether this pattern exists because AER: Insights is extremely selective or because less-established scholars are reluctant to submit their work to a new-ish journal or for some other reason is impossible to know without submission data. But no matter how you look at it, the group currently publishing in AER: Insights is quite elite.

*Top 5 is defined as American Economic Review, Econometrica, Journal of Political Economy, Quarterly Journal of Economics, and Review of Economic Studies.

**AER: Insights would be even higher-ranked by this metric (#3) if we ignored top-5 publications in American Economic Review. Therefore, this pattern is not driven by the fact that both journals are published by the AEA.

What publishes in top-5 economics journals?

Posted on by tatyana

Part I: agricultural economics, lab experiments, field experiments & economics of education

Most of us have a sense that it is more difficult to get certain topics published in the top 5 economics journals (American Economic Review, Econometrica, Journal of Political Economy, Quarterly Journal of Economics, and Review of Economic Studies), but there is not much hard data on this. And if a particular topic appears infrequently in top journals, it may simply be because it’s a relatively rare topic overall.

To get more evidence on this issue, I used Academic Sequitur data, which covers the majority of widely-read journals in economics. The dataset I used contains articles from 139 economics journals and spans the years 2000-2019. On average, 6 percent of the papers in the dataset were published in a top 5 journal.

I classified papers into topics based on the presence of certain keywords in the abstract and title.* I chose the keywords carefully, aiming to both minimize the share of irrelevant articles and to minimize the omission of relevant ones. While there is certainly some measurement error, it should not bias the results. (Though readers should think of this as a “fun-level” analysis rather than a “rigorously peer-reviewed” analysis.)

I chose topics based on suggestions in response to an earlier Tweet of mine. To keep things manageable, I’m going to focus on a few topics at a time. To start off, I looked at agricultural economics (5.3% of articles in the dataset), field experiments (1.0% of articles), lab experiments (1.9% of articles), and education (1.8% of articles). I chose these to have some topic diversity and also because these topics were relatively easy to identify.** I then ran a simple OLS regression of a “top 5” indicator on each topic indicator (separately).***

The results are plotted in a graph below. Field experiments are much more likely to publish in a top 5 journal than in the other 134 journals (about 5 percentage points more likely!), while lab experiments are much less likely. Education doesn’t seem to be favored one way or the other, while agriculture is penalized about as much as field experiments are rewarded. Moral of the story: if you want to publish an ag paper in a top 5, make it a field experiment!

Now you might be saying, “I can’t even name 139 economics journals, so maybe this isn’t the relevant sample on which to run this regression.” Fair point (though see here for a way way longer list of econ journals). To address this, I restricted the set of journals to the 20 best-known general-interest journals—including the top 5—and re-generated the results.**** With the exception of lab experiments, the picture now looks quite different: both field experiments and education research are penalized by the top 5 journals, but agriculture is not.

Combining the two sets of results together, we can conclude that the top 5 penalize agricultural economics research but so do the other good general-interest journals. The top 5 journals also penalize field experiments relative to other good general-interest journals, but top general-interest journals as a whole rewards field experiments relative to other journals. Finally, top 5 journals penalize education relative to other good general-interest journals, but not relative to the field as a whole.

The second set of results is obviously sensitive to the set of journals considered. If I were to add field journals like the American Journal of Agricultural Economics, things would again look much worse for ag. And how much worse they look for a particular topic depends on how many articles the field journal publishes. So I prefer the most inclusive set of journals, but I welcome suggestions about which set of journals to use in future analyses! Would also love to hear everyone’s thoughts on this exercise in general, so please leave a comment.

——————————————————————————————————————–

Endnotes

*I did not use JEL codes because many journals do not require or publish these and we therefore do not collect them. JEL codes are also easier to select strategically than the words in the title and abstract.

** An article falls into the category of agricultural economics if it contains any of the following words/phrases in the abstract or title (not case-sensitive, partial word matches count): “farm”, “crop insurance”, “crop yield”, “cash crop”, “crop production”, “crops production”, “meat processing”, “dairy processing”, “grain market”, “crop management”, “agribusiness”, “beef”, “poultry”, “hog price”, “cattle industry”, “rice cultivation”, “wheat cultivation”, “grain cultivation”, “grain yield”, “crop diversity”, “soil conditions”, “dairy sector”, “hectare”, “sugar mill”, “corn seed”, “soybean seed”, “maize production”, “soil quality” “agricultural chemical use”, “forest”. Field experiment: “field experiment”, “experiment in the field”. Lab experiment: “lab experiment”, “laboratory experiment”, “experimental data”, “randomized subject”, “online experiment”. Education: “return to education”, “returns to education”, “college graduate”, “schooling complet”, “teacher”, “kindergarten”, “preschool”, “community college”, “academic achievement”, “academic performance”, “postsecondary”, “educational spending”, “student performance”, “student achievement”, “student outcome”, “student learning”, “higher education” “educational choice”, “student academic progress”, “public education”, “school facilit”, “education system”, “school voucher” “private school”, “school district”, “education intervention”. Articles may fall into multiple categories.

*** Standard errors are heteroskedasticity-robust

**** The 15 additional journals are (in alphabetical order): American Economic Journal: Applied Economics, American Economic Journal: Economic Policy, American Economic Journal: Macroeconomics, American Economic Journal: Microeconomics, American Economic Review: Insights, Economic Journal, Economic Policy, Economica, European Economic Review, Journal of the European Economic Association, Oxford Economic Papers, Quantitative Economics, RAND Journal of Economics, Review of Economics and Statistics, Scandinavian Journal of Economics.

Should you get a PhD?

Posted on by tatyana

When I asked my undergraduate advisor for a recommendation letter to PhD programs, he replied, with genuine surprise, “Why do you want to get a PhD?” I was too stunned by his question to ask him to elaborate. In my mind, why wouldn’t you get a PhD? You get to learn more about what you’re interested in, you usually get enough money to support yourself, and aren’t more educated people more employable in general?  

Since then, I have come to appreciate my advisor’s reluctance to unequivocally endorse PhDs. (For the record, I don’t think his reaction had anything to do with his opinion of me – he did write me a letter. Also for the record, I do not regret getting a PhD!). “A PhD is an expensive degree” is something I cannot say often enough. Yes, usually you do not pay for the degree directly, but the earnings and quality of life you give up for 4-6+ years are similarly important costs to consider. With that, here are important questions to ask yourself before enrolling in a PhD program.

1) Do you like doing research in the field you are considering? Most PhD programs are geared toward training researchers. And academic research is very different from research you may have done in a class. Class research projects are doable and have an easily identifiable end (aka the due date). Academic research can be incredibly fulfilling because you get to be at the frontier of knowledge. But it is also unpredictable, uncertain, and (usually) involves unforeseen and frustrating setbacks. In the course of a research project, you may discover something incredible or you may end up discarding months of hard work and starting over.

An important emphasis here is on “doing research”. I love reading about new discoveries in genetics and wanted to be a researcher in genetics when I was in high school. But then I learned more about how research in genetics is conducted and realized that being a consumer of research and a producer of research are two very different things. That example also highlights that just because you do not enjoy research in one discipline does not mean that there isn’t another discipline out there for you.

The best way to figure out if you like academic research is to work as a research/lab assistant for a professor or scientist. You will probably end up doing bottom-of-the-barrel work, but you will observe and experience how the process works, which will give you a pretty good idea of whether research is for you.

How do you find research opportunities? Your undergraduate institution may have formal programs. But it’s also perfectly fine to email professors directly and ask if they have research opportunities. You may not get a ton of responses, but you only need one. Looking for a full-time research assistant position is another good option. Finally, you can try research yourself by writing a senior thesis or independent study under the supervision of a professor.

2) How much will your career depend on successfully getting grants? I was happily oblivious to the fact that many academics’ careers live and die by whether they successfully raise money to support their research and their students. Luckily for me, in economics fundraising is optional. But in many other disciplines, applying for grants is an incredibly important part of a researcher’s career. Constantly trying to get new grants to keep your research agenda and students funded can be very stressful. Knowing what kind of funding pressures you might face is important information to incorporate into your decision.

3) What would you do if you didn’t get a PhD and where would that get you in 4-6 years? I always ask students who want to apply to a PhD program why they want a PhD (not with the surprised tone my undergrad advisor used though). About half the time, the answer makes it clear that the biggest reason is that they aren’t sure what to do next, so getting more education seems like a safe fallback option. If that describes you, spend some time researching other career options. And don’t just consider what entry-level jobs you could get instead of a PhD. Remember, a PhD is a big time commitment, so you should be comparing getting a PhD to spending 4-6 years working. Often, those years of work experience can get you far, both financially and in terms of doing interesting work.

4) How difficult is it to get a faculty/researcher position after the PhD program? If you have tried research and loved it and are satisfied with the grant funding situation in your chosen field, there’s still the harsh reality that, in many disciplines, only a small fraction of PhDs end up getting academic/researcher positions. Quite a few end up working in positions that are only tangentially related (Data Science and Finance are popular destination for math and physics PhDs).

The answer to this question obviously varies by institution, and there are no programs that can guarantee a research-based job after. If whether or not you get a PhD hinges on the answer to this question, I suggest applying, seeing where you get in, and then asking those programs about their placement records. If you don’t get into a program with a placement record that satisfies you, working for a year or two, beefing up your credentials, and trying again may be a good option.

I don’t mean to sound too negative about PhD programs. For many, including myself, the intellectual satisfaction of research and the ability to set your own course more than offsets the costs of a PhD program. But I think the world would be a better place if most prospective PhDs knew what exactly they were getting themselves into!