Kurtz & Henneken Win Best JASIST Paper Award
Dr. Michael J. Kurtz
The Association for Information Science & Technology (ASIS&T) is delighted to announce that “Measuring Metrics – A 40-Year Longitudinal Cross-Validation of Citations, Downloads, and Peer Review in Astrophysics,” written by Michael J. Kurtz and Edwin A. Henneken and published in Volume 68 of the Journal of the Association for Information Science & Technology (JASIST) is the recipient of the ASIS&T Best JASIST Paper Award for 2018. The award’s purpose is to recognize the best refereed paper published in the volume year of JASIST preceding the ASIS&T annual meeting. JASIST is published by John Wiley & Sons, Inc.
The article was selected as the winner from among a pool of outstanding submissions that were judged based on these criteria: contribution; professional merit; and presentation quality.
Edwin A. Henneken
The award jury provided the following citation: “The paper was selected for its detailed examination of academic success, its longitudinal analysis based on the records of 922 scholars in astrophysics, and its well-reasoned discussion and analysis. The paper is highly readable and adds to the literature on sociometrics and altmetrics, as well as to the understanding of academic success measures.
Hiring an academic is, as the authors describe, a case of making a prediction about an individual’s future achievements. How then in hiring can we know that the evidence we are using actually predict future academic success? This is the question that Michael Kurtz and Edwin Henneken take on.
In a fascinating evaluation of academic metrics and long-term success, the authors examined the record of 922 individuals who received PhDs in astrophysics in 1972-76 in terms citation counts (Tori Statistic), downloads (Read10), and peer evaluations as exhibited by election to and prizes awarded from the American Astronomical Society and the National Academy of Science. The question posed is: Can one actually make a judgement from such quantitative measures about individual’s past, present and potential future scholarly production or impact? To make such a judgement reliably about one individual over another, the authors find it would require a citation or download count at least twice as large.
The authors also note the difference between publication in a high-ranking journal, a peer judgement of the work, and its impact through citation. Overall, they find that the three measures of citation, downloads, and peer evaluation are equally predictive of future performance, but identify different individuals. In considering all measures, the authors conclude that any application of these ideas should use a real-world rule-of-thumb of at least three times the citation and downloads counts to reliably distinguish the most productive future scholars.”
Upon learning of their article’s selection as the 2018 Best JASIST Paper Award winner, Dr. Kurtz, Astronomer at the Harvard-Smithsonian Center for Astrophysics, said, “Bottom line: when comparing similar people, citation differences less than a factor of two are not important.”
Henneken, an IT Specialist at the Harvard-Smithsonian Center for Astrophysics, said, “It’s in the interest of researchers individually and science in general to never forget that quality of research has many aspects that cannot be quantified. With our paper we hope to have sufficiently emphasized that using bibliometrics for the evaluation of researchers comes with significant caveats which should be taken very seriously. Numbers are just a small patch in a large, complex landscape.”
Henneken and Kurtz will accept the award at the 2018 meeting of the Association for Information Science & Technology (ASIS&T) which will be held in Vancouver, Canada, November 10-14, 2018.
###
Founded at the beginning of the information age in 1937, ASIS&T has promoted the development and understanding of information science and documentation. Through its 80 decades it has reflected a rapidly changing landscape, embracing and promoting new advances in technology to improve all aspects of information collection, management, storage, use and communications. ASIS&T members are at the forefront of research and practice within Information Science. Leading discoveries on how we organize, relate, learn, and access information, our members are changing the face of information utilization. From fine arts to health informatics, ASIS&T is a synergistic organization that allows the exchange of ideas to flourish for the benefit of the whole international field.