Tuesday, March 07, 2006

Picking Winners (Fiddling the Numbers, Part 2)

Impact Factor is calculated each year by the Institute for Scientific Information for those journals which it tracks, and are published in the Journal Citation Report, and is defined as is a measure of the frequency with which the "average article" in a journal has been cited in a particular year or period.

The Impact Factor is calculated on a three year period. For example, the 2006 Impact factor for a selected journal would be calculated as follows:
  • A = Number of times articles published in 2004-5 were cited in all of ISI’s tracked journals during 2006
  • B = Number of articles published in 2004-5 in the selected journal
  • A/B = 2006 Impact Factor
ISI excludes. news items, correspondence, errata, and other ‘non-scholarly’ publications from the denominator. So, to fiddle the Impact Factor, one needs to either increase a, or decrease b or the optimal strategy will do both. The number of articles cited in ISI’s tracked journals is going to be a function of
  • W: the rate at which papers are published in the journal (remember that this analysis is taking place at the individual journal level),
  • X: the number of journals that could possibly cite it,
  • Y: the number of citation references per citing paper, and
  • Z: the number of topic areas into which authors tend to group themselves.
  • A is proportional to W*X*Y/Z
A look at the past four years shows ISR publishing a steady 20 articles a year, while MISQ has jumped from 10 articles per year in 2002 to 30 in 2005. So if there is any manipulation, it must be in A rather than B. The field of IS has been reasonably stable at perhaps 500 active researchers, much of the publication effort coming from untenured assistant professors. From a journal editor’s standpoint, an increased Impact Factor can be accomplished by reducing W, or forcing more self-citations in Y, or by reducing Z. Y is difficult to manipulate, since an editor typically won’t create a new journal to compete with your own (though it has been done). The fastest and easiest way for the editor to increase A is by reducing the number of topic areas Z allowed to be published in their journal. Let’s investigate this further.

Because citations provide clarification and support for arguments presented in a paper, a paper seldom includes citations for papers outside its own topic area. Now consider what happens when we reduce the number of topic areas from three (see figures) to two, without changing anything else.



There is an increase in A because there are fewer choices for topics for publishing authors, and there are fewer choices for what to cite to support their own arguments. Even if all topics were created equal, this would be a powerful effect, but other factors make it even more powerful. The amount of increase in A is nonlinear and accelerating, because the number of papers published by topic area tends to follow a rank-frequency distribution dictated by Zipf’s Law (which is called Bradford’s Law in Library Science, after Samuel C. Bradford, the former librarian of the Science Museum in London). Thus pruning back the number of topic areas acceptable to a journal has an enormous influence on its Impact Factor. Let’s see the implications this has for ISR and MISQ.

The decision to manipulate a was made cautiously; initially it was made in suggestions presented by several current and former journal editors in several articles published in their journals. The articles suggested that the field of IS needed a central governing authority that determines – a priori – which research topics may or may not be considered IS research. The idea was to restrict the number areas in which academics could conduct research. The rationale for this at the time seemed a bit artificial, but I have a better appreciation for what they were doing now that I've seen its effect on Impact Factor.

This trend towards long reference lists in IS is accentuated by its copying – per the example set by these leading journals – in numerous second tier journals. Peffers and Tang, 2003 found that the IS research community – consisting of around 500 active researchers and around 20 active research topics – tended to publish in around 120 pure IS journals and around 200 related journals. IS is a discipline that has a small number of topics, a moderate number of researchers, and a large number of journals. When each article tends to cite a lot of references – this holds the potential for huge Impact Factors. The question up to this point was how to get everyone ‘citing in the same direction.’

So the journal editors got together and concocted arguments and selected methodologies for the correct picking of ‘core’ topics a priori, rather than allowing ideas to stand or fail based on their merits. The net result was that an essentially small group of people get to pick ‘the winners’ – the topics that are likely to be published in the future (then again, that is one of the prerogatives of an editor I suppose). Not surprisingly this small group tended to choose ‘allowed’ topics from their own research. These editorial suggestions have since been reinforced by editorial statements by the current editors of ISR, MISQ and the IS department of Management Science, and have become de facto editorial policy at the journals.

Many people find this approach to scholarship intellectually problematic, since in real science, we normally assume core topics are determined ex post facto through a system of checks and balances involving experiments and other evidence. In contrast to other research areas IS gets its very own theory of ‘Intelligent Design’ – a faith-based initiative for picking ‘allowable’ research in terms of the ‘IT artifacts’ with additional mechanics provided by a previously discredited concept – ‘nomological net.’ So ‘IT artifacts’ and ‘nomological nets’ are now central to any engagement in valid IS research.

The truly bizarre suggestion in all of this is that allowable research should only deal with information technology artifacts, dusty intellectual relics described as “bundles of material and cultural properties packaged in a socially recognizable form such as hardware and software.” There seems to be no rational justification offered for this idiosyncratic perspective, an odd one given that hardware (for example) is usually presented in discussion, research, and the press in terms of its physical, mechanical and electrical characteristics.

So, IS gets its own quasi-anthropological theory of primitive ‘found objects.’ Good luck investigating the latest technology. But as weak and silly as are the philosophical justifications for ‘IT artifacts’ and ‘nomological nets,’ there are clever and fully intended consequences. Now the editors can force everyone to ‘cite in the same direction.’ Of course this was the central and unstated subtext of this original proposal anyway. No one really cares about IT artifacts and nomological nets; in fact, no one even seems to know what they are. It is the Impact Factor that is the real IT artifact – the Holy Relic that validates the research being published. IT artifacts and nomological nets are just scripture that keep the flock from straying.

Proof of effectiveness of the new policy crops up in various statistics. Both ISR and MISQ papers are distinguished by the large sizes of their reference lists; a size that has steadily gotten larger over the years. ISR’s articles typically finish with 30 to 50 references, while MISQ’s recent articles have an astonishing 50 to 90 referenced articles. A closer inspection reveals that the same references appear regularly and repeatedly in multiple articles in the journal. Anecdotally, we understand that many of these references only appear after ‘suggestion’ by reviewers and editors to include them.

My quick tally of articles published in MISQ and ISR pre-2003 and post-2003 indicates a reduction of the ‘allowable’ topics from around 20 to about 6 – a 70% reduction in allowable topics. If citations in those topics were evenly distributed, this would imply a three-fold increase in Impact Factor, which is consistent with the actual increases in Impact Factors from around 1 to 2.884 for MISQ and 3.512 for ISR. The increase in publication rate at MISQ is probably responsible for the lower impact factor there.

The results by 2006 were nothing short of miraculous (Hallelujah!!). In just two years, ISR and MISQ have gone from ‘also rans’ to ‘winners’ – proof that a policy of ‘picking winners’ with a vengeance does indeed do the job.

Next, the consequences of ‘picking winners’

0 Comments:

Post a Comment

<< Home