|
Journal of Bone and Joint Surgery - British Volume, Vol 90-B, Issue 2,
125-126.
doi: 10.1302/0301-620X.90B2.20258 Copyright © 2008 by British Editorial Society of Bone and Joint Surgery
Beware the tyranny of impact factorsR. Smith, CBE, FMedSci, FRCS(E), Former Editor11 British Medical Journal, London SW4 0LD, UK. Correspondence should be sent to Mr R. Smith; e-mail: richardswsmith{at}yahoo.co.uk Abstract The application and interpretation of the impact factor of a journal is fraught with problems, some of which are explored in this editorial. The impact factor is a magic number. It can turn lecturers into professors, cause editors to whoop with delight, bring riches to the impoverished, and resuscitate moribund university departments. But it has a dark side too. Unlike the Michelin star, it has not yet led to suicide, but it may only be a matter of time. What is this magic number and why should any reader of the Journal of Bone and Joint Surgery be interested in it? Invented in 1955 by the information scientist Eugene Garfield,1 the impact factor has come to have prime importance in evaluating academics, university departments and scientific journals. When applied to journals, it in essence measures the number of times scholarly articles in a particular journal are cited in a larger group of journals. The notion is that an article must have some value if it has been cited. For example, the annual impact factor of a journal for 2006 would be as follows:
The impact factor is calculated by Thomson Scientific (Philadelphia, Pennsylvania) using a database of journals. The Annual Review of Immunology has an impact factor of 54.445, the New England Journal of Medicine around 35, the Lancet around 23, and the British Medical Journal around 9. The impact factor itself is calculated to three decimal places, giving it a spurious level of accuracy. Many journals do not have an impact factor at all because they are not included in Thomson Scientific’s database of 6164 journals, and of those that do 2728 (44%) have an impact factor of less than 1. The British volume of the Journal of Bone and Joint Surgery (JBJS Br) is not performing too badly with an impact factor of 1.79. But is the New England Journal of Medicine nearly 20 times ‘better’ than the JBJS Br, and the Lancet more than 10 times ‘better’? "Certainly not to me," I suspect will be the answer of most readers of the JBJS Br. This is where I must begin to unpick the many disadvantages of the impact factor, and some readers may, by the time I have finished, conclude, that this magic number has more problems than benefits. That the impact factor varies greatly between disciplines.2 Generally, basic scientific disciplines, such as molecular or cell biology or immunology, tend to have a very high impact factor and more applied sciences, including surgical science, a much lower one. This difference makes comparison between journals and university departments in different fields very difficult. Another deficiency of the impact factor is that it is strongly biased in favour of American journals. This is because the database contains predominantly American journals, and Americans tend to cite other Americans just as the British tend to cite the British.3 It is more than likely that the American Volume of the JBJS has a higher impact factor (2.444) than the British Volume, simply because it is American rather than because it is a better journal. But there is a much more fundamental problem when it comes to comparing the performance of departments or individuals based on the impact factors of the journals in which they publish; something that happens routinely and explains the growing obsession of academics with impact factors. There is little or no correlation between the impact or citations of an individual article and the impact factor of the journal in which the article is published.1,4 This happens because the impact factor of the journal is driven by a few articles that are highly cited. The journal Nature found that 89% of its citations in one year were accounted for by 25% of its articles. One article was cited more than a 1000 times, whereas most others were cited less than 20 times.5 Garfield6 looked at 38 million articles published between 1900 and 2005 and found that only 0.5% were cited more than 200 times and that half were never cited at all. So it is wholly unscientific and unmathematical to use the impact factor of a journal as a surrogate for the impact of a particular article, and yet this practice is routine. Another important problem with the impact factor lies in the definition of a ‘citable’ article. As various editors have discovered, the easiest and quickest way to increase a journal’s impact factor is to persuade Thomson Scientific to reduce the number of articles in the journal that it defines as citable, thereby reducing the size of the denominator in the calculation of the impact factor. There is no operational definition of what is citable, and there never could be because journals publish such a diverse range of material, not only research articles but letters, reviews, editorials, observations, news, and the list goes on. The editors of the Public Library of Science Medicine have described how their journal’s impact factor varied between 3 and 11 as they debated with Thomson Scientific what was citable. "It became clear," they wrote, "that the process of determining a journal’s impact factor is unscientific and arbitrary. We conclude that science is currently rated by a process that is itself unscientific, subjective, and secretive".7 Despite the many problems of impact factors, which have been described a number of times,1,7–10 they seem to have a growing importance in evaluating science, and the main concern of the editors of many journals is to increase their own impact factor. They ‘play’ the system in many ways; by studying the articles that are most highly cited, publishing more basic science even if it is incomprehensible to their readers, dispensing with articles that are not well cited even when popular with readers, increasing their rejection rates, publishing more items such as abstracts that can count in the numerator but not denominator calculation of the impact factor, encouraging or even obliging authors to increase citations to the journal in their articles,11,12 and flying to Philadelphia to browbeat Thomson Scientific into counting fewer articles as citable.9 The result, unfortunately, may be that which Malcolm Chiswick, previous editor of Archives of Disease in Childhood, described as an ‘impacted journal’, constipated with articles that are highly cited but of little interest to most readers because impact factors are, by definition, driven by researchers and not readers. So if editors have a journal like the JBJS Br, or most clinical journals that are read primarily by clinicians rather than researchers, they may ultimately reduce their readership as they attempt to increase their impact factor. As an increasing number of people have recognised the many problems with impact factors, attempts have been made to find an alternative, one of which includes measuring influences other than citations. But none has yet achieved the dominance of the impact factor, and its influence is likely to persist. The best strategy now for academics, editors, and readers of journals seems to be to recognise the importance of the impact factor but to be aware also of its flaws and to not become obsessed with it. References
|
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
