Thursday, March 09, 2006

Fiddling the Numbers

There is some truth in the old saw that you either publish or perish. But that’s not the whole of it, because not all publications, you see, are created equal. Publications are considered ‘better’ or ‘worse’ in loosely defined ways. Scholars have tried repeatedly to generate an unequivocal ‘quality’ statistic for journal papers. But only one ever had legs – the ‘Impact Factor.’

The Impact Factor is the Holy Grail of academic publishing. High Impact Factors promise more subscriptions for your journal, promotion and tenure for your authors, and prestige and power for your editors. The Impact Factor beat out its rivals by the sheer force of money. You see, it is marketed by ISI, the journal citation subsidiary of publishing powerhouse Thomson Scientific. Since ISI was sold by its founder, Eugene Garfield, in the early 1990s, Thomson has responded to the increasing demand for academic accountability by producing software packages to help users probe its journal citation database, which in turn helps ISI by generating more revenue for access. ISI spends part of that revenue to preemptively field experts to defend their metrics wherever debate over their validity arises, much as the tobacco companies send lobbyists into every debate over smoking. The money involved is significant; the cost of buying certain ISI data rose almost fourfold between 1995 and 2003. The ‘Impact Factor’ is not only a Holy Relic – it is big business for Thomson.

The term ‘Impact Factor’ itself is a triumph of cagy marketing over candor. Thomson doesn’t actually claim to measure quality, usefulness, value, or anything else about an author's contribution that might actually mean anything. They have simply implied that they are measuring some vaguely articulated ‘impact.’ And in a way they are – their marketing department makes sure that this is the only measure in town, and that gives immense impact to their Impact Factor. Take that to the bank.

Over the past two decades I’ve been primarily publishing in the information systems area, where according to the Financial Times the two most influential journals are MIS Quarterly (MISQ) and Information Systems Research (ISR). Now a curious thing happened recently. Both journals underwent subtle but important transformations that boosted their Impact Factor from around 1, where both had hovered for nearly a decade to 2.884 for MISQ and an astonishing 3.512 for ISR – one of the highest impact factors of any journal tracked by Thomson!

How did these guys manage to do that? (Wouldn’t the other academic departments like to know!)

Well, as it turns out, the information systems folks weren’t the first to figure out how to manipulate the Impact Factor. Nearly a decade ago, the savviest citers who resided in the Organizational Behavior departments of universities figured out the game. Their main journals saw their Impact Factor jump – to 2.647 for Academy of Management Journal; 3.405 for Administrative Science Quarterly; and 3.717 for the Academy of Management Review – while nearly every other journal in business languished below 1. Contrast the performance of these three rather stodgy and abstract journals with the widely read (and highly influential) Harvard Business Review which is able to muster only an unimpressive 1.148 Impact Factor. Indeed, that’s pretty good. Most business journals are content to hover around 0.500 or so; an Impact Factor of 1 is doing pretty well, thank you.

So, just exactly what is this Impact Factor – this most Righteous of Right Stuff, Holiest of Holy Grails of academic success – and how can you fiddle the numbers to generate these astronomical values. I’ll tell you how MISQ and ISR did it in my next blog entry.


Blogger DJ said...

Dr. Westland,

I had a great time reading and barely trying to comprehend your interesting blog. I don't know if you remember me, but my name is Deepak and I took a couple of classes based on your two books at Nanyang Business School, Singapore in 2003. Your classes were among the few edifying courses in the entire MBA program - it was like getting a crash course on business history, financial valuation, and technology. You certainly were among the best professors I had in the MBA program and I am sure you the students at UST feel the same.

I am currently in the US at Oklahoma State University, Stillwater pursuing dual masters in telecom management and international studies. I plan to pursue a PhD in public policy with a focus on science and technology policy down the line. I must say the Southern US is a great contrast to the structured and sterile environment of Singapore :-) I see that you are coming out with a new book about Strategy. Could you please blog about it sometime down the road. Thank you for your patience in reading my ramblings and hope to read more of your blogging. Btw, I got here through a link from your homepage (hope I don't sound like a online stalker ;-) - as you can see it's Spring Break here and I have way too much time on my hands.


7:13 PM  

Post a Comment

<< Home