Wrong Impact!
A. Heck
Observatoire
Astronomique
11, rue de
l'Université
F-67000
Strasbourg, France
aheck at cluster.u-strasbg.fr
http://vizier.u-strasbg.fr/~heck
[Paper
published in the European Astron. Soc. Newsl. 26
(Dec. 2003) 4-5]
Abstract
Unreliable impact factors have been published over a few years by ISI
for at least the Astrophysical Journal and Astronomy &
Astrophysics. Why did this happen? How serious is this? Read on for a few
details.
The bibliometric
context
Counting
publications is nowadays one of the ways to evaluate research. It is the foundation
stone of bibliometrics, itself part of the larger concept of scientometrics.
To be said in short terms, scientometrics is supposed to be an evaluation of
science through `objective’ criteria, while bibliometrics is centred on
publications. In practice, scientometrics is often reduced to bibliometrics (see
e.g. Schubert 2001). Most of (if not all) the information used today in
bibliometric studies is produced by the Institute of Scientific Information
(ISI). See for instance Abt (2003) for a presentation of ISI and of its
products.
Bibliometrics
triggers many criticisms (refer for instance to a brief synthesis in Heck 2002,
as well as to the references quoted therein). Opponents claim that
bibliometrics does not deal with the substance of research, i.e. the
progress of knowledge, but that it is mainly concerned with the productivity of
`papers’ and with their citations in subsequent publications (often referred to
as impact –see hereafter).
But
bibliometrics has also many proponents. It has the advantage of being simple
and easy. It readily quantifies the productivity of people and organizations.
It has become an activity per se, a kind of pan-disciplinary audimat
of research, keeping busy many people world-wide and being taken very seriously
in a number of circles, especially by sociologists and politicians of science,
as well as by decision makers and takers.
It is
however true that bibliometrics is a limited and partial tool, hence an
unsatisfactory one, that must be convolved with other indicators if one wishes
to perform efficient comparative evaluations and especially if one has to deal
with persons with similar qualifications, but different activity profiles,
within the same organization.
When
speaking of astronomy-related institutions, people involved in service
activities (resident astronomers operating instruments, maintainers of
resources and databases, …) and in other tasks (developers of instrumentation,
data/information handling specialists, …) would be largely disadvantaged by the
only consideration of bibliometrics since their primary activity is not aimed
at publishing. The same remark would be of application for staff members
spending a significant amount of their time in teaching, supervising theses,
and so on.
All such
activities belong however to the research context, even if one agrees with
Moravcsik (1973) that it is necessary to distinguish between scientific
activity, scientific production and scientific progress. Additionally, quality,
importance and impact of a specific work must be distinguished.
Why wrong
impact factors for some astronomical journals?
ISI’s
impact factors (IFs) are defined as the average ratio (for each journal) of the
number of citations to papers in it to the number of papers published, averaged
over the previous two years (Abt 2003). They can be used for weighting counts of
publications when evaluating individuals, institutions, proposals, projects,
etc.
Here
again opponents have a long list of criticisms (refer again to Heck 2002) among
which the most serious one is perhaps that IFs fail to address the variation of
quality within a journal.
But
let's face it: scientometrics and bibliometrics are flourishing disciplines
(see again Schubert 2001). They have led to a number of important papers for
astronomy and astrophysics. See for instance the review by Abt (2000) and the
list of publications[1]
compiled by the author.
It is
difficult to assess exactly how extensive is the usage of IFs by evaluation
bodies. It would seem, however, they are more frequently used in
In the
early 1990s, the editors of the major astronomical journals agreed on some
common requirements from authors[2]
and to simplify reference lists. For instance, the abbreviations of some
journals seemed unnecessarily long. Thus for the “Monthly Notices of the
Royal Astronomical Society” and the “Publications of the Astronomical
Society of the Pacific”, usually referred to as “Mon. Not. R. Astron.
Soc.” and “Publ. Astron. Soc. Pacific” respectively, it was decided to
use the compact acronyms MNRAS and PASP. Similar measures were recommended for
the dozen major journals[3].
It is estimated that those short acronyms saved 60 pages per year in the Astrophys.
J. (ApJ) each year.
ISI’s IFs
for two journals -- Astrophys. J. and Astron. Astrophys. -- dropped
however drastically in 1998. Inquiries to ISI showed that, for that period and
as result of an algorithm modification, the references to Astrophys. J.
had been appropriately credited to that journal but those to ApJ had not,
and similarly for the other journal (Abt 2003, Sandqvist 2003). The problem being
corrected in 2001 for ApJ and in 2002 for A&A, their IFs recovered to their
normal levels –-putting an end, as pleasantly commented by McNally (2003), to “MNRAS’
surge of glory”.
It is
still a mystery why ApJ and A&A were affected, and why, for instance, MNRAS
and AJ (Astronomical Journal) were not.
And
now?
As
information received from ISI does not hint at a publication of corrected data
for the years concerned, the usage of the corresponding numbers should be
avoided (ApJ for the years 1998-2000 and A&A for the years 1998-2001).
On a
more sociological stand, it may never be possible to assess the possible damage
caused by the usage of wrong IFs, especially the human damage at the level of
individuals, projects, etc., that underwent selection/competition where weights
involving those incorrect ratings might have been used.
Bibliometric
indices are questionable in various respects and one should never rely blindly
or entirely on them. This is why they are often used together with other
indicators, such as peer evaluation, discoveries and recognition (awards,
honours, invited lectures, etc.). These other indicators have also their own
share of shortcomings and ultimately it is always advisable to use all of them only
as a support to a frank discussion between wise men and women, possibly behind
closed doors.
References
Abt,
H.A. 2000, What can we Learn from Publications Studies, in Organizations and
Strategies in Astronomy -- Vol. 1, Ed. A. Heck, Kluwer Acad. Publ.,
Abt,
H.A. 2003, The Institute for Scientific Information and the Science Citation
Index, in Organizations and Strategies in Astronomy -- Vol. 4,
Ed. A.
Heck, Kluwer Acad. Publ.,
Heck, A.
2002, Editorial, in Organizations and Strategies in Astronomy -- Vol. 3,
Ed. A. Heck, Kluwer Acad. Publ.,
McNally,
D. 2003, Foreword, in Organizations and Strategies in Astronomy -- Vol. 4,
Ed. A. Heck, Kluwer Acad. Publ.,
Moravcsik,
M.J. 1973, Measures of Scientific Growth, J. Res. Pol.
2, 266-275.
Sandqvist,
Å. 2003, Astron. Astrophys. 402, E1.
Schubert,
A. 2001, Scientometrics: The Research Field and its Journal, in Organizations
and Strategies in Astronomy -- Vol. 2, Ed. A. Heck, Kluwer Acad. Publ.,
Dordrecht, 179-185.
[1] Updated in each volume of the
Organizations and Strategies in Astronomy (OSA) series and online at http://vizier.u-strasbg.fr/~heck/osabib.htm.
[2] See, e.g., Astron.
Astrophys. 235 (1990) E1-E2, Astron. J. 100 (1990) 1 and Astrophys. J. 357
(1990) 1.
[3] The exercise was sometimes
pushed too far by some managers of lesser astronomical journals.