The life cycle of scholarly articles across fields of research

Year: 
2017
Topic: 
Education - Health
Methodology
Theory
tags: 

As many influential metrics used to assess researchers’ performance are based in citation counts, shedding light on citation behavior contributes to a better understanding of scientific outputs (see Bornmann and Daniel 2008). Nevertheless, little effort has been devoted to understanding and quantifying how article citations evolve over the years following an article’s publication and how these trends vary across fields of research. Given that several influential metrics restrict their input to a certain range in terms of the number of years since publication, this behavior is by no means neutral and should be taken into account when evaluating researchers across fields of research.

In a recently published article (Anauati et al. 2016) we showed that yearly citations to research articles published in the top-five economics journals have a clear-cut lyfe-cycle, which  varies greatly in shape across fields of economics research (i.e., applied, applied theory, theory, econometric methods). In that paper we proposed a methodology for identifying these cycles, allowing to take into consideration citation inflation (the fact that citations have become more common nowadays) and skewness in the distribution of citations. Based on these findings, in Galiani and Gálvez (2017) we delved further into the life-cycle of scholarly articles, this time focusing on differences across broader fields of research.  With the aim of covering a broad range of patterns in citation trends, we chose to study fields of research in the social sciences, life sciences, physical sciences, mathematical studies, and health studies. Specifically, we analyzed articles from the following disciplines: astronomy & astrophysics, biochemistry, biology, economics, finance, mathematics, medicine, physics, political science, psychology, sociology, and statistics. For each of these twelve fields of research, we chose five prestigious within-the-field general research journals and sampled articles following a strategy that ensured that our final sample would be representative in terms of periods-of-time and success as measured by total citation counts. Additionally, we made sure that our sampling strategy would not be affected by citation inflation. We ended up with a final sample of 59,707 research articles published between 1985 and 2000. Having constructed a representative sample of research articles, we gathered Google citations of each of them for every year starting from two years before publication up to late 2015, totaling more than five million individual citations. Here we present and place in context our main findings.

Raw trends in citation counts

As a first attempt to shed light on the trends in citation counts, Fig. 1 plots for each field of research the share of total citations received up to 2 and 5 years after publication relative to all citations received up to 15 years after publication. (Note that 2 and 5 years are the periods usually considered when calculating journal impact factors.)

Fig. 1 already points to the conclusion that citation dynamics vary greatly between fields of research. Notably, disciplines regarded as successful in terms of citation counts (e.g., economics and finance) tend to receive a small proportion of their citations in the first years after publication. This suggests that an important factor in driving the success of articles in those fields is that they are cited for longer periods rather than being the subject of a large surge in citations soon after their publication.

To provide a better way of visualizing differences in the evolution of citations across fields of research, Fig. 2 plots the number of citations per year since publication for the mean paper (solid line) and the median one (dashed line) for each field of research. To illustrate the effect of citation inflation, this figure differentiates estimates for the group of articles published during the period from 1985 to 1989 (1985-1989, in blue) and for the group published in the period from 1995 to 1999 (1995-1999, in red).

     

It is clear from Fig. 2 that estimated trends differ greatly across fields of research. In disciplines such as physics, astronomy & astrophysics, biochemistry, and biology, a clear-cut decline in the number of citations per year is observed after a period of time; in other disciplines, such as economics, finance, mathematics, political science, sociology, and statistics, non-descending curves are observed for mean citations per year. Fig. 2 also makes explicit two features that should be taken into account when analyzing the evolution of citation behavior across time and across fields of research. First, the fact that curves for the period 1995-1999 always lie at higher values than the ones estimated for 1985-1989 signals the presence of citation inflation, and that it has not occurred at the same rate in all fields of research. Take the case of the increment in mean citations for medicine compared to biology or physics. For medicine, an article published in the period 1985-1999 received little more than five citations per year, on average, in its peak year, while, for the period 1995-1999, in its peak year it received almost three times more citations. On the other hand, in the fields of biology and physics, the peak in citations per year did not vary as much, but the newer articles tended to age better (i.e., the decay in annual citations was less pronounced). Second, the fact that mean citation curves for any given period lie at higher values than the curves for median citations signals the existence of a strong positive skewness in the distribution of citations per year. Note that this asymmetry in the distribution of citations is common to all fields of research and persists as the papers age. Nevertheless, the strength of this trend also seems to differ across fields.

Life-cycle of scholarly articles across fields of research

As some curves plotted in Fig. 2 may be constantly ascending due to citation inflation, we used quantile regression (QR) for identifying the life cycle of papers across fields of research. The proposed specification makes it possible to control for secular trends in citations (taking into account citation inflation) and allows the analysis across different (conditional) levels of success of papers. Fig. 3 shows the results of estimating the life-cycle of scholarly articles across fields of research. The curves shown in purple correspond to life cycles as estimated on the basis of QR; the curves shown in green correspond to life cycles as estimated on the basis of OLS regressions.

        

When controlling for citation inflation, it is evident that, across all disciplines, annual citations exhibit a life-cycle pattern. Nevertheless, sharp differences are also observed between different fields of research. First, it is clear that the peak of annual citations is much higher for some disciplines than for others. Second, the peak in citations is not reached at the same time across fields and, after this peak is reached, annual citation values differ in the way that they decline. Third, differences between QR and OLS estimates also vary greatly across disciplines.

A similar analysis focusing on higher conditional quantiles of the response variable distribution makes it possible to examine shows that, even for highly successful research articles, a life-cycle pattern is observed in all the disciplines covered by this study. Moreover, this analysis again points to differences across disciplines.

Discussion and Conclusions

We find evidence that points to the presence of life-cycle patterns in annual citations across a broad range of disciplines. These patterns tend to vary greatly across different fields of research. Social sciences such as economics, political science, and sociology exhibit longer life cycles, with annual citations of articles tending to reach a peak later on than in other disciplines and then declining after this peak quite gradually. Mathematics and statistics articles display a somewhat similar pattern, although the number of citations per year is much lower than in the case of the social sciences. In addition, in a number of fields, article citations are much more concentrated in the first years after publication. This is the case for astronomy & astrophysics, biochemistry, biology, medicine and physics. As filtering articles by their age is a common practice when calculating evaluative metrics, these differences are by no mean neutral and should be taken into account when evaluating researchers or research institutions.


References:

Anauati, V, Galiani, S, and Gálvez, RH. "Quantifying the life cycle of scholarly articles across fields of economic research." Economic Inquiry 54.2 (2016): 1339-1355. VoxEU’s column based on this article available at http://voxeu.org/article/lifecycle-scholarly-articles-across-fields-economic-research.

Bornmann, L, and Daniel, H D (2008), “What do citation counts measure? A review of studies on citing behaviour.” Journal of Documentation 64(1): 45-80.

Galiani, S, and Gálvez, RH. "The life cycle of scholarly articles across fields of research." electronic preprint version available at SSRN.

Share this