Author: Lokker, Cynthia
Date published: January 1, 2012
Journal impact factors QIFs) are calculated as the number of times articles in a journal published over a two-year period (e.g., 2005-2006) are cited in all journals indexed by Thompson Reuters Institute of Scientific Information (ISI) during the following year (e.g., 2007), divided by the total number of substantive articles and reviews published in that journal in the first two years (e.g., 2005-2006) [1, 2]. This overall measure of mean "citedness" of articles in a journal is frequently regarded as a measure of journal quality, and, rightly or wrongly, the measure is often used to evaluate research performance, promote candidates, and make decisions about funding [I, 3-6]. The annual fall publication of JIFs by ISI is a much anticipated event as JIFs and article citation rates are important to journal authors, editors, and publishers. New journals must generally wait at least three years, and often much longer, before they obtain their first official JIF, and being added to the ISI list of publications requires meeting ISFs stringent criteria . Some journals have predicted and published their own JIFs for both internal use and solicitation of articles [8-10]. Though JIFs are often much criticized [3, 11-13], they are still the most widely used metric for measuring the overall quality of a biomédical journal [I, 3-5, 14, 15].
A recent study showed that the 2-year citation count of a high-quality clinical article can be predicted with about 60% accuracy using data readily available in the first month after publication . Variables relating to article quality (e.g., number of authors, abstraction in synoptic journals, multicentered study) were significant factors in the prediction model. Such early predictions of citation frequency can be used to direct articles to clinicians and researchers; select important material for systematic reviews, clinical practice guidelines, and health technology assessment projects; choose educational material; and use as promotion and tenure data for researchers . The current study sought to determine if attributes of articles and journals are associated with JIFs. If the JIF is a measure of quality, then attributes other than citation counts relating to the quality and relevance of an article should have positive associations with and contribute to explaining variation in JIFs. Such attributes would provide measures available at time of publication to predict the subsequent JIF. Association of initial quality features of articles would also provide independent support for the use of JIFs as a measure of quality.
A retrospective cohort study was conducted of all articles added to the McMaster University Premium Literature Service (PLUS) for the first 5 months of 2005 compared with JIFs for 2007. McMaster PLUS supports the production of a number of evidencebased journals and information services and has been described in detail elsewhere . The service involves trained research staff who read more than 150 clinical journals <http://hiru.mcmaster.ca/hiru/ journalslist.asp> and identify studies that meet explicit criteria for scientific merit for research about diagnosis, prevention and treatment, prognosis, etiology, quality improvement, and cost effectiveness <http://hiru.mcmaster.ca/hiru/InclusionCriteria.html>.
Articles that pass these category-specific criteria are then rated by physicians selected from a panel of more than 4,000 for relevance to clinical practice (the extent to which an article is pertinent to the rater's practice; 7point scale, with 7 highest) and newsworthiness (the extent to which the article's content represents something that clinicians in the rater's discipline are unlikely to know; similar 7-point scale). This rating is done using an online rating system called McMaster Online Rating of Evidence (MORE). Appendix A, online only, has the rating questions. At least 3 raters per pertinent clinical discipline assess each article. For example, an article on cardiac complications of diabetes mellitus would be rated by at least 3 general practitioners, 3 general internists, and 3 cardiologists. The MORE database only retains articles with average ratings of at least 4 on each scale. The ratings are publicly available through EvidenceUpdates <http:// plus.mcmaster.ca/EvidenceUpdates>. Although this service is restricted to relatively high-quality studies in higher-quality clinical journals, it includes unique markers of quality and relevance, available at the time of publication, with sufficient variation to predict citation counts of individual articles .
The data set for this study included 1,267 articles from 103 journals, the same dataset used in a previous study , excluding 7 articles from journals that do not have a published JIF. In 2009, after the JIFs including articles published in 2005 were released, each article in the dataset was allocated using random number generation (60:40 split) into a derivation set (760 articles in 99 journals) or a validation set (507 articles in 88 journals). Article- and journal-level variables that potentially influence JIF were gathered (Table 1). The variables included the proportion of articles for a given journal passing methods criteria in 2005, the proportion of articles abstracted in 1 or more of the evidence-based summary journals supported by McMaster PLUS (ACP Journal Club, Evidence-Based Medicine, Evidence-Based Nursing) in 2006, the mean number of disciplines that rated that journal's articles (a measure of audience breadth), and the mean newsworthiness and relevancy scores given by raters. Other variables known to be correlated with citation counts and article success were also considered (Table 1). Due to the journal sample size of 103, the number of variables was limited to 10, based on standard statistical practice . The study team used consensus to include variables based on their influence in predicting citation counts in the previous study , whether they could be influenced by authors or editors and /or whether they indicated scientific quality for studies of relevance to clinical practice. JEFs for 2007 for the journals were collected from ISI Web of Knowledge.
A multiple linear regression was performed on journal-level means from the derivation set and applied to the validation set using STATA Intercooled 9.0 software. The dependent variable was the natural log transformed JIF for 2007. The variable was transformed because the residual variance in models using non-transformed data was not constant, and this violated one assumption of conventional multiple regression models. The 10 independent variables each represented a summary score (i.e., the average of quantitative variables or the proportion for binary variables) across included articles within a journal for the first 5 months of 2005. A weighting factor was used to account for varying numbers of articles in the journals. The initial regression indicated that a number of variables were not significantly contributing to the model; these were removed, keeping only variables with P<0.1. The derivation model was tested against the validation dataset to evaluate the predictive accuracy of the model in new data.
The 103 included journals had 2007 JIFs ranging from 0.725 to 52.6. The mean JIF in the derivation set was 5.51 (95% CI 4.17 to 6.86), median 3.72 (95% CI 3.10 to 4.34), and in the validation set the mean was 5.86 (95% CI 4.38 to 7.34), median 3.75 (95% CI 3.43 to 4.58). From ISI Web of Knowledge, the median 2007 JIF for journals in the category "medicine, general & internal journals" was 0.835 and 0.712 for the category "medicine, research & experimental journals."
Only 5 of the 10 variables had a P<0.10 in the initial regression and were kept in the final prediction model (Table 2). Those removed were the proportion of articles that were selected for abstracting in a summary journal, mean relevance, mean sample size of studies, proportion of multicentered studies, and proportion of original studies. The reduced model was highly significant (R^sup 2^=0.607, CI 0.444 to 0.706) and was generalizable to the validation set with only a 1% reduction in the R^sup 2^. Four of the variables (proportion of articles published in a journal that passed McMaster PLUS quality criteria, mean newsworthiness score, number of authors, and the number of databases indexing the journal) were statistically significant and positively associated with the JEF. The number of disciplines rating the article (reflecting the range of clinical disciplines interested in the article) remained in the model because it had a P<0.10; it was of borderline statistical significance (P =0.075). The mean predicted JIF in the derivation set was 4.05 (95% CI 1.20 to 13.71), median 3.91 (95% CI 3.45 to 4.39), minrmum 1.13, maximum 56.15, and in the validation set, the mean was 3.95 (95% CI 1.00 to 8.05), median 4.24 (95% CI 3.98 to 5.40), minimum 0.82, maximum 52.59.
Each regression coefficient estimates the expected change in the value of the loge transformed JIF associated with a 1-unit increase in the corresponding independent variable (i.e., 1 more author, 1 more database indexed, 100% increase of proportion of articles passing criteria in 2005, 1-unit increase of mean newsworthiness score, 1 more discipline rating the article), with all the other predictors held constant. The magnitude of the regression coefficient reflects the units of its independent variable.
Because the dependent variable was log transformed and it was difficult to reach consensus on defining a meaningful change of each independent variable, a direct comparison of the effect of each independent variable on the dependent variable is not possible. The univariate R^sup 2^ value for each variable provides some indication of the relative importance of each, though these univariate values do not take into account collinearity with other independent variables (Table 2). The number of authors, number of indexing databases, and mean newsworthiness all had significant univariate R^sup 2^ values over 0.2. The proportion passing methods criteria had a significant R^sup 2^ of 0.08. The number of disciplines rating the article was not significant with an R^sup 2^ of 0.0005.
The results demonstrate that the subsequent JIF of a clinical journal is associated with quality and newsworthiness measures that can be assessed soon after publication of a journal's articles. Of the five measures in the final model, three are unique to McMaster PLUS (mean newsworthiness score, proportion of articles passing quality criteria, and number of disciplines rating the article). These three variables are in addition to two previously validated predictors of citation counts : the number of authors and the number of indexing services that include the journal title. The univariate R^sup 2^ values (Table 2) suggest that the number of authors and indexing databases have the most influence on JIF. Mean newsworthiness score and the proportion of articles passing quality criteria are also significant predictors. The JTF is often criticized as not truly reflecting journal quality. However, from the model, there is a strong association of the methodological quality of articles published in a journal and the JIF in this select set of journals.
Other studies of articles in general medicine have shown an association of JTF with study quality. Lee and colleagues  found that higher study quality scores corresponded to increased citations, JIF, journal circulation, low manuscript acceptance rates (i.e., more selectivity), and indexing in the Brandon/Hill list of journals and books recommended for a hospital library. Gluud and colleagues  found that JIF was a significant predictor of adequate randomization and increasing sample size, but not of study outcome, in hepatobiliary randomized controlled trials.
Few practicing clinicians have a need to cite articles, but they do need to know where they are most likely to read sound studies that apply to their practice. The JIF can guide them to high-quality and newsworthy research. A study by Saha and colleagues  showed that for nine general medicine journals, clinicians' (subjective) ratings of quality were strongly correlated with a journal's JIF.
General medical journals such as New England Journal of Medicine, Lancet, JAMA, and BMJ tend to have higher JEFs and broader audiences than more specialized journals such as Stroke and Worldviews on Nursing. Not surprisingly, this is reflected in the model presented here by the significant positive association with JEF of both the number of indexing databases and the number of disciplines selected to rate the articles. Journals publishing articles rated higher for newsworthiness have correspondingly higher JEFs. The implication of these findings is that clinical journals could increase their JEF by changing their peer-review procedures to include clinical ratings of submitted manuscripts as a supplement to detailed peer review. It is also important for journals to nominate their publications to as many databases for indexing as possible.
The two main theories for explaining citation behaviors are normative and social constructivist. In the first, science is viewed as rewarding quality and effort, essentially "what one says," by citing work based on article traits [31-33]. Social constructivist theory purports that citations are given based on "who one is" in the social hierarchy of one's field; citing articles is based on author traits [31-33]. Both contribute to citation decisions, but studies of citation patterns can often tease apart the relative importance of each domain, by including variables that measure both article quality (e.g., empirical and theoretical content, article length, number of references, tables or graphs, publication lag) and social factors (e.g., author nationality or gender, social ties among authors, institutional prestige) . The model presented here supports the normative theory, with article quality and newsworthiness characteristics contributing significantly to the JIF. It also supports the social constructivist theory with the significance of the number of authors, which is a possible surrogate for social recognition and an increased network of associations among people.
In a previous publication, the two-year citation counts for individual articles were predicted using nineteen variables and similar mulitivariate analyses . Eleven of these nineteen were statistically significant in the final model. Four overlap with the five variables in the current JIF study, all of them relating to measures of study quality . Given that citation counts are used in the calculation of JIFs, it is not surprising that overlap is present among the variables that predict them both.
Recently, measures of frequency of online access and citations of open access journals have been studied (Craig provides a review ). Some have found no open access advantage in terms of increased citations [34-36], while others have found a positive open access advantage [37-39]. Currently, there are no reliable ways to determine frequency of access of all open access journals, but in the future this measure could prove to be a significant factor in citation behavior.
Strengths and limitations
The study model predicts approximately 61% of the variation in journals' JIFs with measures that are available within days to months of publication, demonstrating a link between a journal's JIF and clinical newsworthiness and broad applicability (importance to multiple disciplines). The results are reliable, as shown by the minimal differences in model performance between the development and validation databases.
The study also provides an independent validation of the article selection process for evidence-based journals, including both methods assessment and second-order peer review (the rating by clinicians on the relevance and newsworthiness of the study to their practice) , illustrating that articles so rated at the time of publication are more likely to garner scientific interest. Unfortunately, it is not possible to determine in this observational study whether the article selection process and broadcasting of the selected articles in evidence-based alerting services actually stimulates citations, leading to an increased association. However, the prime target audiences for evidence-based services are practicing clinicians, who are unlikely to be citing studies in published articles.
The lack of complete predictability of JIFs with the included variables may be due to deficiencies in the predictors, impact factors, or, more importantly, measures of complementary (correlated) properties of articles. Citations of articles could be for context, notoriety, or criticism, rather than praise. Rater's assessments of newsworthiness, on the other hand, reflect whether a study provides new information that practitioners should know about for care of their patients. Further, the proportion of articles in a journal that meet methodological criteria for clinical quality reflects the journal's ability to attract and select studies that are more likely to be actionable from a clinical perspective. Thus, the identified measures are at least partly complementary to JIFs. Using a multilevel analysis, which would allow for teasing apart article- and journal-level effects, was investigated. However, since the main outcome was at the higher level (JIF) rather than at the lower level (article), such an analysis was not possible. Also, the JIF is based on citations of all articles in the journal.
A major limitation of this study of journal JIFs is that only a small subset of journals was used (n=103), with JIFs ranging from 0.725 to 52.6, which were heavily weighted in favor of highly regarded and highly used clinical titles. Appendix B, online only, lists the included journals. Also, the included articles were only those that met PLUS rigorous methodological criteria and minimum relevance and newsworthiness scores of 4 on scales of 1 to 7 (Appendix A, online only). MEDLINE indexes more than 5,000 journals and that number represents only a portion of the total biomédical journals currently published. The findings are not applicable to the full complement of biomédical journals but are most likely to be pertinent to clinical journals across a broad range of disciplines. Current research shows that online journals are more often cited. The 2005 sample does not include online journals, so the importance of this factor could not be tested.
The strength of predicting JIFs might be enhanced by including lower-quality studies in the models. As there are many more low-quality than high-quality studies, a random sample would be adequate to test the prediction model. Some of the variables excluded from analysis (Table 1) might add predictive value and should be considered in future research. Ultimately, it would be desirable to use a more robust and less controversial reference standard than JIF, which is based solely on citation counts of a somewhat arbitrary set of articles within journals over a fixed period.
Supplemental Appendixes A and B are available with the online version of this journal.
1. Garfield E. The history and meaning of the journal impact factor. JAMA. 2006 Jan 4;295(l):90-3.
2. ISI Web of Knowledge. Journal citation reports 2010 [Internet]. ISI [cited 6 Jun 2009]. <http://isiknowledge .com>. (subscription required).
3. Favaloro EJ. Measuring the quality of journals and journal articles: the impact factor tells but a portion of the story. Semin Thromb Hemost. 2008 Feb;34(l):7-25.
4. Glanzel W, Moed HF. Journal impact measures in bibliometric research. Scientometrics. 2002 Feb;53(2):171-93.
5. Moed H. Citation analysis in research evaluation. Dordrecht, The Netherlands: Springer; 2005.
6. Tobin MJ. Thirty years of impact factor and the journal. Am J Respir Crit Care Med. 2004 Aug 15;170(4):351-2.
7. Brown H. How impact factors changed medical publishing-and science. BMJ. 2007 Mar 17;334(7593):561-4.
8. Craig ID. The Journal of Sexual Medicine - impact factor predictions and analysis. J Sex Med. 2007 Jul;4(4 pt l):855-8.
9. Ketcham CM. Predicting impact factor one year in advance. Lab Invest. 2007 Jun;87(6):520-6.
10. Wu XF, Fu Q, Rousseau R. On indexing in the Web of Science and predicting journal impact factor. J Zhejiang Univ Sci B. 2008 Jul;9(7):582-90.
11. Brumback RA. Impact factor wars: episode V - the empire strikes back. J Child Neurol. 2009 Mar;24(3):260-2.
12. Pendlebury DA. The use and misuse of journal metrics and other citation indicators. Arch Immunol Ther Exp (Warsz.). 2009 Jan-Feb;57(l):1-11.
13. The PLoS Medicine Editors. The impact factor game. PLoS Med. 2006 Sep;3(6):e291.
14. Andersen J, Belmont J, Cho CT. Journal impact factor in the era of expanding literature. J Microbiol Immunol Infect. 2006 Dec;36(6):436-43.
15. Ketcham CM. The proper use of citation data in journal management. Arch Imrnunol Ther Exp (Warsz.). 2008 NovDec; 56(6):357-62.
16. Lokker C, McKibbon KA, McKinlay RJ, Wilczynski NL, Haynes RB. Prediction of citation counts for clinical articles at two years using data available within three weeks of publication: retrospective cohort study. BMJ. 2008 Mar 22;336(7645):655-7.
17. Haynes RB, Cotoi C, Holland J, Walters L, Wilczynski N, Jedraszewski D, McKinlay J, Parrish R, McKibbon KA. Second-order peer review of the medical literature for clinical practitioners. JAMA. 2006 Apr 19;295(15):1801-8.
18. Norman GR, Streiner DL. Biostatistics the bare essentials. Hamilton, ON, Canada: B.C. Decker; 2000.
19. Bhandari M, Busse JW, Devereaux PJ, Montori VM, Swiontkowski M, Tornetta P. Factors associated with citation rates in the orthopedic literature. Can J Surg. 2007 Apr;50:119-23.
20. Bornmann L, Daniel HD. Multiple publication on a single research study: does it pay? the influence of number of research articles on total citation counts in biomedicine. JASIST. 2007 Jun;58(8): 1100-7.
21. Figg WD, Dunn L, Liewehr DJ, Steinberg SM, Thurman PW, Barrett JC, Birkinshaw J. Scientific collaboration results in higher citation rates of published articles. Pharmacotherapy. 2006 Jun;26(6):759-67.
22. Kulkarni AV, Busse JW, Shams I. Characteristics associated with citation rate of the medical literature. PLoS ONE. 2007 May;2(5):e403.
23. Campbell FM. National bias: a comparison of citation practices by health professionals. Bull Med Lib Assoc. 1990 Oct;78(4):376-82.
24. West R, McHwaine A. What do citation counts count for in the field of addiction? an empirical evaluation of citation counts and their link with peer ratings of quality. Addiction. 2002 May;87(5):501-4.
25. Bhandari M, Busse JW, Devereaux PJ, Montori VM, Swiontkowski M, Tornetta P, Einhorn TA, Khera V, Schemitsch EH. Factors associated with citation rates in the orthopedic literature. Can J Surg. 2007 Apr;50(2):l 19-23.
26. González-Pereira B, Guerrero-Bote VP, Moya-Anegón F. The SJR indicator: a new indicator of journals' scientific prestige [Internet]. 2009 [cited 28 Jun 2011]. <http://www .arxiv.org/abs/0912.4141 > .
27. Bergstrom C. Eigenfactor: measuring the value and prestige of scholarly journals. C&RL News. 2007 May;68(5)
28. Lee KP, Schotland M, Bacchetti P, Bero LA. Association of journal quality indicators with methodological quality of clinical research articles. JAMA. 2002 Jun 5;287(21):2805-8.
29. Gluud LL, Sorensen TI, Gotzsche PC, Gluud C. The journal impact factor as a predictor of trial quality and outcomes: cohort study of hepatobiliary randomized clinical trials. Am J Gastroenterol. 2005 Nov;100(ll):2431-5.
30. Saha S, Saint S, Christakis DA. Impact factor: a valid measure of journal quality? J Med Libr Assoc. 2003 Jan;91(l):42-6.
31. Baldi S. Normative versus social constructivist processes in the allocation of citations: a network-analytic model. Am Sociol Rev. 1998 Dec;63(6):829-46.
32. Nicolaisen J. Citation analysis. ARIST. 2007;41(1):609-41.
33. Stewart JA. Achievement and ascriptive processes in recognition of scientific articles. Social Forces. 1983 Sep;62(l):166-89.
34. Davis PM, Lewenstein BV, Simon DH, Booth JG, Connolly MJL. Open access publishing, article downloads, and citations: randomised controlled trial. BMJ. 2008 JuI 31;337:a568.
35. Kurtz MJ, Henneken EA. Open access does not increase citations for research articles from the Astrophysical Journal [Internet]. 2007 [cited 14 Sep 2011]. <http://www.arxiv .org/abs/0709.0896>.
36. Davis PM. Open access, readership, citations: a randomized controlled trial of scientific journal publishing. FASEB J. 2011 Mar 30 [Epub ahead of print].
37. Brody T, Harnad S, Carr L. Earlier web usage statistics as predictors of later citation impact [Internet]. 2006 [cited 26 Jun 2006]. <http://eprints.ecs.soton.ac.uk/10713/>.
38. Davis PM. Author-choice open access publishing in the biological and medical literature: a citation analysis. JASIST. 2009 Jan;60(l):3-8.
39. Eysenbach G. Citation advantage of open access articles. PLoS Biol. 2006 May;4(5):el57.
Cynthia Lokker, PhD; R. Brian Haynes, MD, PhD; Rong Chu, MSc; K. Ann McKibbon, PhD; Nancy L. Wilczynski, PhD; Stephen D. Walter, PhD
Cynthia Lokker, PhD, firstname.lastname@example.org, Research Coordinator, Health Information Research Unit; R. Brian Haynes, MD, PhD, email@example.com, Chief, Health Information Research Unit, and Professor, Department of Medicine; Rong Chu, MSc, chur@ mcmaster.ca, PhD Candidate; K. Ann McKibbon, PhD, firstname.lastname@example.org, Associate Professor, Health Information Research Unit; Nancy L. Wilczynski, PhD, email@example.com, Research Manager, Health Information Research Unit; Stephen D. Walter, PhD, firstname.lastname@example.org, Professor; Department of Clinical Epidemiology and Biostatistics, Faculty of Health Sciences, McMaster University, CRL 125, 1280 Main Street West, Hamilton, ON, Canada, L8S 4K1
Received April 2011; accepted July 2011