In today’s Wall Street Journal (online subscription required) Sharon Begley provides a rare look into the world of academic journal rankings. She describes some of the ways that scientific journals manipulate their “impact factors”.
The article shows that once you have a quantitative scheme to measure things, people will game them. It must matter on the margin, but perhaps the equilibrium effects are small—all this gaming may is easy to see through. And you can pick where to send your papers. You do want to send them to places with higher impact factors. Which means that there is a bit of a viscous circle here. On the other hand, it is not really costly to add extraneous cites.
How permanent are the effects? There must be some path dependence. But I have seen journals wax and wane over time.
All that said, I would think that people in the field see through much of this. It is when you need to talk to outsiders—the dean, university-wide promotion committees, or outside funding agencies—that this is going to matter. How much, I am not sure. It does, however, show that quantitative data is not obviously more objective than more qualitative data.
Here is an example from outside academics. There seems to be a new ipod killer every month or so—quantitatively a better mp3 player, more memory, higher bandwidth or whatever. They never catch on. Why? Probably because the ipod killers’ designs are worse. How would you completely quantify that other than saying people like the Ipod experience better? (Surely everything in not quantifiable by ease of use metrics, but also the coolness factor.)
Back to academics. Good ideas cause people to think differently. It’s not clear that depends on where the paper appears, as long as it is a credible place. And on the margin, I bet the impact factors matter less than the WSJ article implies.
UPDATE: I wonder what the ‘impact factor’ of Asian Journal of Mathematics is: Poincare conjecture solved. And what will it be if the proof is correct? (I know nothing about the journal or its impact factor. I am not a mathematician.)