Subscribe to our newsletter

← See all posts

What’s so Bad About the Impact Factor – Part 3. What Did You All Say?

21st October 2015
 | Phill Jones
We asked you to join the conversation on #IFwhatswrong. Fortunately, some of you did.
We asked you to join the conversation on #IFwhatswrong. Fortunately, some of you did.

Over the past couple of weeks, I’ve taken a brief look at Impact Factor (IF) and some of the criticisms that are levelled at it. As promised, I’m going to report on some of the feedback that we’ve received, particularly on the Twitter hashtag #IFwhatswrong.

The week before last, I focused on one of the statistical objections to IF, put simply – it’s an arithmetic mean, but should be a median. The mathematical argument appealed to Riccardo Sapienza, from King’s College, who tweeted about it.

As a physicist, Sapienza is familiar with statistics and how they work. I was interested when he implied that the correct metric ought to be the most frequent citation number (mode), rather than the median. I imagine that we could debate the best type of average to use, but as I argued in my first post in this series, I’m not convinced that it makes much difference. I don’t think that changing the way IF is calculated would do much good for research or researchers.

Marie McVeigh, who has had a long association with both JCR and Thomson Reuters, including working as Director of Content Selection for Web of Science, as well as being former Director of Product Research at Research Square,  expressed some frustration with the question.

McVeigh makes an interesting point here. It’s remarkable that we’re still using IF in inappropriate ways despite decades of formal research and years of debate in academic and library circles about its overuse. It seems to me that the metric has become far too deeply embedded in the way that academics themselves think about the quality of research and that’s why we’re still discussing it.

Tibor Tscheke of Science Open tweeted

Tscheke’s point is key here, I believe. In fact, it was the issue that I settled on as the central issue with IF in my second post. The problem isn’t really how the evaluation is done but the way it is interpreted – as a proxy for the quality of the research contained within a journal when it’s really only a measure of the skill of the editorial team in selecting highly citable articles.

Vicky Williams contributed a number of tweets to the debate in which she set out a different objection, one which is fair to level at IF, but is also true of all citation based metrics including H-index and article based citation counting generally.

Williams raises a great question and a controversial point. If we talk about the types of metrics that might augment the IF and count for other forms of impact, a common objection is that alternative metrics don’t capture impact they count attention. Are academic citations more worthy than other uses of research output?

Julia Shaw, senior lecturer and researcher in the Department of Law and Social Sciences at London South Bank University answered that question very directly.

I have to stress that I’m speaking entirely for myself when I say that I broadly agree with Shaw here. It seems odd to me to elevate a citation in a review article that does not contribute to the argument as being impact, while at the same time, calling a news article that sways public opinion merely attention. Whether a mention is impactful is obviously strongly related to the medium, but context is very important.

Williams went onto cite the Effects of participation in EU frameworks programmes for research and technological development – for researchers, institutions and private companies in Denmark report published by the Ministry of Higher Education and Science in Denmark. The primary conclusion of the report being that participation by researchers, institutions, and companies in Horizon 2020 was measurably beneficial for traditional academic impact and that companies believed it to be economically beneficial. The primary benefit for companies was to allow the funding of activities that would not otherwise have been implemented. Unfortunately, the authors are unable to show statistically significant quantitative economic benefit. The report points out many of the challenges in measuring economic impact of research including the length of time it takes for it to be observable. The take home message for me is that there is still a need to develop metrics that are both fast and predictive, but that’s a whole other blog post.

So, what do I take away from this exercise? 

Some people are sick of talking about IF and about metrics but I think part of the issue is that we’ve not solved the underlying problem of the evaluation gap. We’re still using the IF as a blunt proxy measure and have not succeeded in agreeing to change, despite the fact that most people seem to think that we need to.

IF lacks the resolution to measure what many people rely on it to measure – the quality of research. It’s a metric designed to evaluate the quality of subscription access journals for selection in a library collection. Back when that was all it was used for, it wasn’t a problem, but now that academics are under increasing pressure to write high impact articles, it has introduced a level of gamesmanship to academic authorship that I think doesn’t benefit science.

At the same time, IF is too narrow to measure impact. Research affects society in a breadth of ways. By limiting the type of impact that we measure to citations, we fail to get the full picture.

My last observation is about the use of the word impact. I don’t like the way it’s used and I’m not the only one. At the Frankfurt STM conference last week, Richard Gedye, STM’s Director of Outreach Programmes, spoke about Reseach4Life, which is the collective name for Hinari, Agora, OARE, and ARDI, which provide access to academic literature in the developing world. During the Q+A, Richard Padley of Semantico took to the roaming mic to thank Gedye for saying impact without saying factor afterwards.

The name Impact Factor was an excellent piece of branding. It seems to have changed the way we think about research. Many see academic citations as the only form of impact. If I could, I’d rename the IF to journal citation factor. Maybe then, we can think more freely about the benefits of research to society and the reasons why we communicate it.

So maybe that’s what’s so bad about the Impact Factor: the name.