Bibliometrics-based heuristics: What is their definition and how can they be studied? - Research note
DOI:
https://doi.org/10.3145/epi.2020.jul.20Keywords:
Bibliometrics, Heuristics, Bibliometrics-based heuristics, Research evaluation, Decision strategiesAbstract
When scientists study the phenomena they are interested in, they apply sound methods and base their work on theoretical considerations. In contrast, when the fruits of their research are being evaluated, basic scientific standards do not seem to matter. Instead, simplistic bibliometric indicators (i.e., publication and citation counts) are, paradoxically, both widely used and criticized without any methodological and theoretical framework that would serve to ground both use and critique. Recently, however Bornmann and Marewski (2019) proposed such a framework. They developed bibliometrics-based heuristics (BBHs) based on the fast-and-frugal heuristics approach (Gigerenzer; Todd; ABC Research Group, 1999) to decision making, in order to conceptually understand and empirically investigate the quantitative evaluation of research as well as to effectively train end-users of bibliometrics (e.g., science managers, scientists). Heuristics are decision strategies that use part of the available information and ignore the rest. By exploiting the statistical structure of task environments, they can aid to make accurate, fast, effortless, and cost-efficient decisions without that trade-offs are incurred. Because of their simplicity, heuristics are easy to understand and communicate, enhancing the transparency of decision processes. In this commentary, we explain several BBHs and discuss how such heuristics can be employed in practice (using the evaluation of applicants for funding programs as one example). Furthermore, we outline why heuristics can perform well, and how they and their fit to task environments can be studied. In pointing to the potential of research on BBHs and to the risks that come with an under-researched, mindless usage of bibliometrics, this commentary contributes to make research evaluation more scientific.
Downloads
References
Binmore, Ken (2009). Rational decisions. Princeton, NJ, USA: Princeton University Press. ISBN: 978 0 691149899
Bishop, Doroty (2018). "Luck of the draw. Funders should assign research grants via a lottery system to reduce human bias". Nature index, May 7. https://www.natureindex.com/news-blog/luck-of-the-draw
Bornmann, Lutz (2019). "Bibliometric indicators". In: Atkinson, P.; Delamont, S.; Cernat, A.; Sakshaug, J. W.; Williams, R. SAGE Research methods foundations. Thousand Oaks, CA, USA: Sage. ISBN: 978 1 526421036
Bornmann, Lutz; Ganser, Christian; Tekles, Alexander; Leydesdorff, Loet (2020). "Does the hα index reinforce the Matthew effect in science? The introduction of agent-based simulations into scientometrics". Quantitative science studies, v. 1, n. 1, pp. 331-346. https://doi.org/10.1162/qss_a_00008
Bornmann, Lutz; Marewski, Julian N. (2019). "Heuristics as conceptual lens for understanding and studying the usage of bibliometrics in research evaluation". Scientometrics, n. 120, pp. 419-459. https://doi.org/10.1007/s11192-019-03018-x
Bornmann, Lutz; Mutz, Rí¼diger; Daniel, Hans Dieter (2008). "Are there better indices for evaluation purposes than the h index? A comparison of nine different variants of the h index using data from biomedicine". Journal of the American Society for Information Science and Technology, v. 59, n. 5, pp. 830-837. https://doi.org/10.1002/asi.20806
Brighton, Henry (2006). "Robust inference with simple cognitive models". In: Lebiere, C.; Wray, B. Between a rock and a hard place: Cognitive science principles meet Al-hard problems. Papers from the AAAI spring symposium (AAAI Tech Rep No SS-06-03). Menlo Park, CA, USA: AAAI Press, pp. 17-22. http://www.aaai.org/Papers/Symposia/Spring/2006/SS-06-02/SS06-02-006.pdf
Czerlinski, Jean; Gigerenzer, Gerd; Goldstein, Daniel G. (1999). "How good are simple heuristics?". In: Gigerenzer, G.; Todd, P. M.; ABC Research Group. Simple heuristics that make us smart. Oxford, UK: Oxford University Press, pp. 97-118. ISBN: 978 0 195143812
De-Rijcke, Sarah; Wouters, Paul F.; Rushforth, Alex D.; Franssen, Thomas P.; Hammarfelt, Bjí¶rn (2016). "Evaluation practices and effects of indicator use - a literature review". Research evaluation, v. 25, n. 2, pp. 161-169. https://doi.org/10.1093/reseval/rvv038
EMBO (2018). EMBO Young Investigator Programme. Application guidelines. Heidelberg, Germany: EMBO. https://www.embo.org/documents/YIP/application_guidelines.pdf
Evans, Jonathan St. B. T. (2008). "Dual-processing accounts of reasoning, judgment, and social cognition". Annual review of psychology, v. 59, pp. 255-278. https://doi.org/10.1146/annurev.psych.59.103006.093629
Gigerenzer, Gerd (2008). Rationality for mortals: How people cope with uncertainty. Oxford, UK: Oxford University Press. ISBN: 978 0 195328981
Gigerenzer, Gerd; Gaissmaier, Wolfgang (2011). "Heuristic decision making". Annual review of psychology, v. 62, pp. 451-482. https://doi.org/10.1146/annurev-psych-120709-145346
Gigerenzer, Gerd; Goldstein, Daniel G. (1996). "Reasoning the fast and frugal way: Models of bounded rationality". Psychological review, v. 103, n. 4, pp. 650-669. https://doi.org/10.1037/0033-295X.103.4.650
Gigerenzer, Gerd; Goldstein, Daniel G. (1999). "Betting on one good reason: The take the best heuristic". In: Gigerenzer, G.; Todd, P. M.; ABC Research Group. Simple heuristics that make us smart. Oxford, UK: Oxford University Press, pp. 75-95. ISBN: 978 0 195143812
Gigerenzer, Gerd; Hertwig, Ralph; Pachur, Thorsten (eds.) (2011). Heuristics: The foundations of adaptive behavior. Oxford, UK: Oxford University Press. ISBN: 978 0 199744282
Gigerenzer, Gerd; Marewski, Julian N. (2015). "Surrogate science: The idol of a universal method for scientific inference". Journal of management, v. 41, n. 2, pp. 421-440. https://doi.org/10.1177/0149206314547522
Gigerenzer, Gerd; Todd, Peter M.; ABC Research Group (1999). Simple heuristics that make us smart. Oxford, UK: Oxford University Press. ISBN: 978 0 195143812 https://www.researchgate.net/profile/Ulrich_Hoffrage/publication/227466812_Simple_Heuristics_That_Make_Us_Smart/links/00b7d517f86ec6844d000000/Simple-Heuristics-That-Make-Us-Smart.pdf
Goldstein, Daniel G.; Gigerenzer, Gerd (2009). "Fast and frugal forecasting". International journal of forecasting, v. 25, pp. 760-772. https://pure.mpg.de/rest/items/item_2099944_4/component/file_2099943/content
Hafenbrí¤dl, Sebastian; Waeger, Daniel; Marewski, Julian N.; Gigerenzer, Gerd (2016). "Applied decision making with fast-and-frugal heuristics". Journal of applied research in memory and cognition, v. 5, n. 2, pp. 215-231. https://doi.org/10.1016/j.jarmac.2016.04.011
Hauser, John R.; Wernerfelt, Birger (1990). "An evaluation cost model of consideration sets". Journal of consumer research, v. 16, pp. 393-408. https://doi.org/10.1086/209225
Hertwig, Ralph; Hoffrage, Ulrich; ABC Research Group (2013). Simple heuristics in a social world. New York, NY, USA: Oxford University Press. ISBN: 978 0 195388435 https://doi.org/10.1093/acprof:oso/9780195388435.001.0001
Hertwig, Ralph; Todd, Peter M. (2003). "More is not always better: The benefits of cognitive limits". In: Hardman, D.; Macchi, L. Thinking: Psychological perspectives on reasoning, judgment and decision making. Hoboken, NY; USA: Wiley, pp. 213-231. ISBN: 0 471 49457 7 http://www.al-edu.com/wp-content/uploads/2014/05/HarmanMacchi-eds-Thinking-Psychological-Perspectives-on-Reasoning-Judgement-and-Decision-Making.pdf
Hicks, Diana; Wouters, Paul; Waltman, Ludo; De-Rijcke, Sarah; Ràfols, Ismael (2015). "Bibliometrics: The Leiden Manifesto for research metrics". Nature, v. 520, n. 7548, pp. 429-431. https://doi.org/10.1038/520429a
Hirsch, Jorge E. (2019). "hα: An index to quantify an individual´s scientific leadership". Scientometrics, n. 118, pp. 673-686. https://doi.org/10.1007/s11192-018-2994-1
Kahnemann, Daniel; Slovic, Paul; Tversky, Amos (eds.) (1982). Judgment under uncertainty: Heuristics and biases. New York, NY, USA: Cambridge University Press. ISBN: 978 0 521 28414 1
Katsikopoulos, Konstantinos V. (2011). "Psychological heuristics for making inferences: Definition, performance, and the emerging theory and practice". Decision analysis, v. 8, n. 1, pp. 10-29. https://doi.org/10.1287/deca.1100.0191
Kelman, Mark (2011). The heuristics debate. Oxford, UK: Oxford University Press. ISBN: 978 0 199755608
Knight, Frank H. (1921). Risk, uncertainty and profit. New York, NY, USA: Houghton Mifflin.
MacRoberts, Michael H.; MacRoberts, Barbara R. (2017). "The mismeasure of science: Citation analysis". Journal of the Association for Information Science and Technology, v. 69, n. 3, pp. 474-482. https://doi.org/10.1002/asi.23970
Marewski, Julian N.; Gaissmaier, Wolfgang; Gigerenzer, Gerd (2010). "Good judgments do not require complex cognition". Cognitive processing, v. 11, pp. 103-121. https://doi.org/10.1007/s10339-009-0337-0
Marewski, Julian N.; Gaissmaier, Wolfgang; Schooler, Lael J.; Goldstein, Daniel G.; Gigerenzer, Gerd (2010). "From recognition to decisions: Extending and testing recognition-based models for multialternative inference". Psychonomic bulletin & review, v. 17, pp. 287-309. https://doi.org/10.3758/PBR.17.3.287
Marewski, Julian N.; Schooler, Lael J. (2011). "Cognitive niches: An ecological model of strategy selection". Psychological review, v. 118, n. 3, pp. 393-437. https://doi.org/10.1037/a0024143
Marx, Werner (2014). "The Shockley-Queisser paper - A notable example of a scientific sleeping beauty". Annalen der Physik, v. 526, n. 5-6, pp. A41-A45. https://doi.org/10.1002/andp.201400806
Merton, Robert K. (1968). "The Matthew effect in science". Science, v. 159, n. 3810, pp. 56-63. https://doi.org/10.1126/science.159.3810.56
Raab, Markus; Gigerenzer, Gerd (2015). "The power of simplicity: A fast-and-frugal heuristics approach to performance science". Frontiers in psychology, 6, 1672. https://doi.org/10.3389/fpsyg.2015.01672
Research Excellence Framework (2014). Research Excellence Framework. http://www.ref.ac.uk/2014
Rothenfluh, Tomas; Daniel, Hans-Dieter (2009). "Qualití¤tssicherung in Forschung, Lehre und Studium am Beispiel der Universití¤t Zí¼rich [Quality assurance in research, teaching, and studies exemplified by the University of Zurich]". In: Webler, W-D. Universití¤ten am Scheideweg?! - Chancen und Gefahren des gegenwí¤rtigen historischen Wandels in Verfassung, Selbstverstí¤ndnis und Aufgabenwahrnehmung [Universities at the crossroads?! Opportunities and dangers of current historical change in the constitution, self-conception and task performance]. Bielefeld, Germany: Universití¤tsVerlagWebler, pp. 259-278. ISBN: 978 3 937026 64 0
Savage, Leonard J. (1954). The foundations of statistics. New York, NY, USA: Wiley. ISBN: 978 0 486623498
Simon, Herbert A. (1956). "Rational choice and the structure of the environment". Psychological review, v. 63, n. 2, pp. 129-138. https://doi.org/10.1037/h0042769
Simon, Herbert A. (1990). "Invariants of human-behavior." Annual review of psychology, v. 41, pp. 1-19. https://doi.org/10.1146/annurev.ps.41.020190.000245
Todd, Peter M.; Gigerenzer, Gerd; ABC Research Group (2012). Ecological rationality: Intelligence in the world. New York, NY, USA: Oxford University Press. ISBN: 978 0 195315448 https://doi.org/10.1093/acprof:oso/9780195315448.001.0001
Waltman, Ludo (2018). "Responsible metrics: One size doesn´t fit all". In: Wouters P. Proceedings of the Science and Technology Indicators Conference 2018, Leiden. Science, Technology and Innovation indicators in transition. Leiden, The Netherlands: University of Leiden, pp. 526-531. http://sti2018.cwts.nl/proceedings
Downloads
Published
How to Cite
Issue
Section
License
Dissemination conditions of the articles once they are published
Authors can freely disseminate their articles on websites, social networks and repositories
However, the following conditions must be respected:
- Only the editorial version should be made public. Please do not publish preprints, postprints or proofs.
- Along with this copy, a specific mention of the publication in which the text has appeared must be included, also adding a clickable link to the URL: http://www.profesionaldelainformacion.com
- Only the final editorial version should be made public. Please do not publish preprints, postprints or proofs.
- Along with that copy, a specific mention of the publication in which the text has appeared must be included, also adding a clickable link to the URL: http://revista.profesionaldelainformacion.com
Profesional de la información journal offers the articles in open access with a Creative Commons BY license.
