Toward the consolidation of a multi-metric-based journal ranking and categorization system for computer science subject areas
DOI:
https://doi.org/10.3145/epi.2023.dic.03Keywords:
Journal rankings, Research evaluation, Indicators, Scientific journals, Metrics, Algorithms, Machine learning, Cluster analysis, Principal component analysis (PCA), t-distributed stochastic neighbor embedding (t-SNE), Cross tabulation, ESA indexAbstract
The evaluation of scientific journals poses challenges owing to the existence of various impact measures. This is because journal ranking is a multidimensional construct that may not be assessed effectively using a single metric such as an impact factor. A few studies have proposed an ensemble of metrics to prevent the bias induced by an individual metric. In this study, a multi-metric journal ranking method based on the standardized average index (SA index) was adopted to develop an extended standardized average index (ESA index). The ESA index utilizes six metrics: the CiteScore, Source Normalized Impact per Paper (SNIP), SCImago Journal Rank (SJR), Hirsh index (H-index), Eigenfactor Score, and Journal Impact Factor from three well-known databases (Scopus, SCImago Journal & Country Rank, and Web of Science). Experiments were conducted in two computer science subject areas: (1) artificial intelligence and (2) computer vision and pattern recognition. Comparing the results of the multi-metric-based journal ranking system with the SA index, it was demonstrated that the multi-metric ESA index exhibited high correlation with all other indicators and significantly outperformed the SA index. To further evaluate the performance of the model and determine the aggregate impact of bibliometric indices with the ESA index, we employed unsupervised machine learning techniques such as clustering coupled with principal component analysis (PCA) and t-distributed stochastic neighbor embedding (t-SNE). These techniques were utilized to measure the clustering impact of various bibliometric indicators on both the complete set of bibliometric features and the reduced set of features. Furthermore, the results of the ESA index were compared with those of other ranking systems, including the internationally recognized Scopus, SJR, and HEC Journal Recognition System (HJRS) used in Pakistan. These comparisons demonstrated that the multi-metric-based ESA index can serve as a valuable reference for publishers, journal editors, researchers, policymakers, librarians, and practitioners in journal selection, decision making, and professional assessment.Downloads
References
Abbas, Naveed-Neem; Ahmed, Tanveer; Shah, Syed-Habib-Ullah; Omar, Muhammad; Park, Han Woo (2019). “Investigating the applications of artificial intelligence in cyber security”. Scientometrics, v. 121, n. 2, pp. 1189-1211. https://doi.org/10.1007/s11192-019-03222-9
Allen, Liz; Jones, Ceri; Dolby, Kevin; Lynn, David; Walport, Mark (2009). “Looking for landmarks: The role of expert review and bibliometric analysis in evaluating scientific publication outputs”. PLoS one, v. 4, n. 6. https://doi.org/10.1371/journal.pone.0005910
Beliakov, Gleb; James, Simon (2011). “Citation-based journal ranks: The use of fuzzy measures”. Fuzzy sets and systems, v. 167, n. 1, pp. 101-119. https://doi.org/10.1016/j.fss.2010.08.011
Bergstrom, Carl (2007). “Eigenfactor: Measuring the value and prestige of scholarly journals”. College & research libraries news, v. 68, n. 5, pp. 314-316. https://doi.org/10.5860/crln.68.5.7804
Bollen, Johan; Van-de-Sompel, Herbert; Hagberg, Aric; Chute, Ryan (2009). “A principal component analysis of 39 scientific impact measures”. PLoS one, v. 4, n. 6. https://doi.org/10.1371/journal.pone.0006022
Dellavalle, Robert P.; Schilling, Lisa M.; Rodriguez, Marko A.; Van-de-Sompel, Herbert; Bollen, Johan (2007). “Refining dermatology journal impact factors using PageRank”. Journal of the American Academy of Dermatology, v. 57, n. 1, pp. 116-119. https://doi.org/10.1016/j.jaad.2007.03.005
Duan, Lian; Nasri, Farrokh; Paknejad, Javad (2018). “A data-driven method for rating management information systems journals in the same scale of the Association of Business Schools Journal Guide”. Expert systems, v. 35, n. 6. https://doi.org/10.1111/exsy.12309
Ennas, Gianfranco; Biggio, Battista; Di-Guardo, Maria-Chiara (2015). “Data-driven journal meta-ranking in business and management”. Scientometrics, v. 105, n. 3, pp. 1911-1929. https://doi.org/10.1007/s11192-015-1751-y
Feng, Lin; Zhou, Jian; Liu, Sheng-Lan; Cai, Ning; Yang, Jie (2020). “Analysis of journal evaluation indicators : an experimental study based on unsupervised Laplacian score”. Scientometrics, v. 124, pp. 233-254. https://doi.org/10.1007/s11192-020-03422-8
Fernández-Cano, Antonio; Fernández-Guerrero, Inés M. (2017). “A multivariate model for evaluating emergency medicine journals”. Scientometrics, v. 110, n. 2, pp. 991-1003. https://doi.org/10.1007/s11192-016-2197-6
Ferrer-Sapena, Antonia; Sánchez-Pérez, Enrique A.; Peset, Fernanda; González, Luis-Millán; Aleixandre-Benavent, Rafael (2016). “The impact factor as a measuring tool of the prestige of the journals in research assessment in mathematics”. Research evaluation, v. 25, n. 3, pp. 306-314. https://doi.org/10.1093/reseval/rvv041
Fersht, Alan (2009). “The most influential journals: Impact factor and eigenfactor”. Proceedings of the National Academy of Sciences of the United States of America, v. 106, n. 17, pp. 6883-6884. https://doi.org/10.1073/pnas.0903307106
González-Pereira, Borja; Guerrero-Bote, Vicente P.; De-Moya-Anegón, Félix (2010). “A new approach to the metric of journals scientific prestige: The SJR indicator”. Journal of informetrics, v. 4, n. 3, pp. 379-391. https://doi.org/10.1016/j.joi.2010.03.002
Halim, Zahid; Khan, Shafaq (2019). “A data science based framework to categorize academic journals”. Scientometrics, v. 119, n. 1. https://doi.org/10.1007/s11192-019-03035-w
Holmberg, Kim; Park, Han Woo (2018). “An altmetric investigation of the online visibility of South Korea-based scientific journals”. Scientometrics, v. 117, n. 1, pp. 603-613. https://doi.org/10.1007/s11192-018-2874-8
Hsu, Wen-Chin; Tsai, Chih-Fong; Li, Jia-Huan (2015). “A hybrid indicator for journal ranking: An example from the field of Health Care Sciences and Services”. Online information review, v. 39, n. 7, pp. 858-869. https://doi.org/10.1108/OIR-11-2014-0277
James, Chris; Colledge, Lisa; Meester, Wim; Azoulay, Norman; Plume, Andrew (2018). “CiteScore metrics: Creating journal metrics from the Scopus citation index”. ArXiv, 16 pp. https://onlinelibrary.wiley.com/doi/epdf/10.1002/leap.1246
Lacasse, Jeffrey R.; Hodge, David R.; Bean, Kristen F. (2011). “Evaluating the productivity of social work scholars using the h-index”. Research on social work practice, v. 21, n. 5, pp. 599-607. https://doi.org/10.1177/1049731511405069
Leydesdorff, Loet; Park, Han Woo (2017). Full and fractional counting in bibliometric networks. Journal of informetrics, 11, n. 1, pp. 117-120. https://doi.org/10.1016/j.joi.2016.11.007
Mañana-Rodríguez, Jorge (2015). “A critical review of SCImago Journal & Country Rank”. Research evaluation, v. 24, n. 4, pp. 343-354. https://doi.org/10.1093/reseval/rvu008
Moed, Henk F. (2010). “Measuring contextual citation impact of scientific journals”. Journal of informetrics, v. 4, n. 3, pp. 265-277. https://doi.org/10.1016/j.joi.2010.01.002
Morris, Huw; Harvey, Charles; Kelly, Aidan (2009). “Journal rankings and the ABS journal quality guide”. Management decision, v. 47, n. 9, pp. 1441-1451. https://doi.org/10.1108/00251740910995648
Mubarak, Muhammed; Seemee, Shabana (2021). “Journal prestige index: Expanding the horizons of assessment of research impact”. Journal of the College of Physicians and Surgeons Pakistan, v. 31, n. 11, pp. 1261-1262. https://doi.org/10.29271/jcpsp.2021.11.1261
Perera, Upeksha; Wijewickrema, Manjula (2018). “Relationship between journal-ranking metrics for a multidisciplinary set of journals”. Portal: Libraries and the academy, v. 18, n. 1, pp. 35-58. https://doi.org/10.1353/pla.2018.0003
Saarela, Mirka; Kärkkäinen, Tommi (2020). “Can we automate expert-based journal rankings? Analysis of the Finnish publication indicator”. Journal of informetrics, v. 14, n. 2. https://doi.org/10.1016/j.joi.2020.101008
Saarela, Mirka; Kärkkäinen, Tommi; Lahtonen, Tommi; Rossi, Tuomo (2016). “Expert-based versus citation-based ranking of scholarly and scientific publication channels”. Journal of informetrics, 10, n. 3, pp. 693-718. https://doi.org/10.1016/j.joi.2016.03.004
Setti, Gianluca (2013). “Bibliometric indicators: Why do we need more than one?”. IEEE Access, v. 1, pp. 232-246. https://doi.org/10.1109/ACCESS.2013.2261115
Tsai, Chih-Fong (2014). “Citation impact analysis of top ranked computer science journals and their rankings”. Journal of informetrics, v. 8, n. 2, pp. 318-328. https://doi.org/10.1016/j.joi.2014.01.002
Tüselmann, Heinz; Sinkovics, Rudolf R.; Pishchulov, Grigory (2015). “Towards a consolidation of worldwide journal rankings - A classification using random forests and aggregate rating via data envelopment analysis”. Omega (United Kingdom), v. 51, pp. 11-23. https://doi.org/10.1016/j.omega.2014.08.002
Walters, William H. (2017). “Do subjective journal ratings represent whole journals or typical articles? Unweighted or weighted citation impact?”. Journal of informetrics, v. 11, n. 3, pp. 730-744. https://doi.org/10.1016/j.joi.2017.05.001
Wical, Stephanie H.; Kocken, Gregory J. (2017). “Open access and promotion and tenure evaluation plans at the University of Wisconsin-Eau Claire”. Serials review, v. 43, n. 2, pp. 111-119. https://doi.org/10.1080/00987913.2017.1313024
Wyatt, Sally; Milojevič, Stasa; Park, Han Woo; Leydesdorff, Loet (2017). Quantitative and qualitative STS: The intellectual and practical contributions of Scientometrics. Boston, MA: MIT Press. https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2588336
Yuen, Jason (2018). “Comparison of Impact Factor, Eigenfactor Metrics, and SCImago Journal Rank Indicator and h-index for neurosurgical and spinal surgical journals”. World neurosurgery, v. 119, pp. e328-e337. https://doi.org/10.1016/j.wneu.2018.07.144
Zhu, Yu-Peng; Park, Han Woo (2022). “Profiling the most highly cited scholars from China: Who they are. To what extent they are interdisciplinary”. Profesional de la informacion, v. 31, n. 4, e310408. https://doi.org/10.3145/epi.2022.jul.08
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2023 Profesional de la información / Information Professional
This work is licensed under a Creative Commons Attribution 4.0 International License.
Dissemination conditions of the articles once they are published
Authors can freely disseminate their articles on websites, social networks and repositories
However, the following conditions must be respected:
- Only the editorial version should be made public. Please do not publish preprints, postprints or proofs.
- Along with this copy, a specific mention of the publication in which the text has appeared must be included, also adding a clickable link to the URL: http://www.profesionaldelainformacion.com
- Only the final editorial version should be made public. Please do not publish preprints, postprints or proofs.
- Along with that copy, a specific mention of the publication in which the text has appeared must be included, also adding a clickable link to the URL: http://revista.profesionaldelainformacion.com
Profesional de la información journal offers the articles in open access with a Creative Commons BY license.
