Toward the consolidation of a multi-metric-based journal ranking and categorization system for computer science subject areas

Authors

DOI:

https://doi.org/10.3145/epi.2023.dic.03

Keywords:

Journal rankings, Research evaluation, Indicators, Scientific journals, Metrics, Algorithms, Machine learning, Cluster analysis, Principal component analysis (PCA), t-distributed stochastic neighbor embedding (t-SNE), Cross tabulation, ESA index

Abstract

The evaluation of scientific journals poses challenges owing to the existence of various impact measures. This is because journal ranking is a multidimensional construct that may not be assessed effectively using a single metric such as an impact factor. A few studies have proposed an ensemble of metrics to prevent the bias induced by an individual metric. In this study, a multi-metric journal ranking method based on the standardized average index (SA index) was adopted to develop an extended standardized average index (ESA index). The ESA index utilizes six metrics: the CiteScore, Source Normalized Impact per Paper (SNIP), SCImago Journal Rank (SJR), Hirsh index (H-index), Eigenfactor Score, and Journal Impact Factor from three well-known databases (Scopus, SCImago Journal & Country Rank, and Web of Science). Experiments were conducted in two computer science subject areas: (1) artificial intelligence and (2) computer vision and pattern recognition. Comparing the results of the multi-metric-based journal ranking system with the SA index, it was demonstrated that the multi-metric ESA index exhibited high correlation with all other indicators and significantly outperformed the SA index. To further evaluate the performance of the model and determine the aggregate impact of bibliometric indices with the ESA index, we employed unsupervised machine learning techniques such as clustering coupled with principal component analysis (PCA) and t-distributed stochastic neighbor embedding (t-SNE). These techniques were utilized to measure the clustering impact of various bibliometric indicators on both the complete set of bibliometric features and the reduced set of features. Furthermore, the results of the ESA index were compared with those of other ranking systems, including the internationally recognized Scopus, SJR, and HEC Journal Recognition System (HJRS) used in Pakistan. These comparisons demonstrated that the multi-metric-based ESA index can serve as a valuable reference for publishers, journal editors, researchers, policymakers, librarians, and practitioners in journal selection, decision making, and professional assessment.

Downloads

Download data is not yet available.

References

Abbas, Naveed-Neem; Ahmed, Tanveer; Shah, Syed-Habib-Ullah; Omar, Muhammad; Park, Han Woo (2019). “Investigating the applications of artificial intelligence in cyber security”. Scientometrics, v. 121, n. 2, pp. 1189-1211. https://doi.org/10.1007/s11192-019-03222-9

Allen, Liz; Jones, Ceri; Dolby, Kevin; Lynn, David; Walport, Mark (2009). “Looking for landmarks: The role of expert review and bibliometric analysis in evaluating scientific publication outputs”. PLoS one, v. 4, n. 6. https://doi.org/10.1371/journal.pone.0005910

Beliakov, Gleb; James, Simon (2011). “Citation-based journal ranks: The use of fuzzy measures”. Fuzzy sets and systems, v. 167, n. 1, pp. 101-119. https://doi.org/10.1016/j.fss.2010.08.011

Bergstrom, Carl (2007). “Eigenfactor: Measuring the value and prestige of scholarly journals”. College & research libraries news, v. 68, n. 5, pp. 314-316. https://doi.org/10.5860/crln.68.5.7804

Bollen, Johan; Van-de-Sompel, Herbert; Hagberg, Aric; Chute, Ryan (2009). “A principal component analysis of 39 scientific impact measures”. PLoS one, v. 4, n. 6. https://doi.org/10.1371/journal.pone.0006022

Dellavalle, Robert P.; Schilling, Lisa M.; Rodriguez, Marko A.; Van-de-Sompel, Herbert; Bollen, Johan (2007). “Refining dermatology journal impact factors using PageRank”. Journal of the American Academy of Dermatology, v. 57, n. 1, pp. 116-119. https://doi.org/10.1016/j.jaad.2007.03.005

Duan, Lian; Nasri, Farrokh; Paknejad, Javad (2018). “A data-driven method for rating management information systems journals in the same scale of the Association of Business Schools Journal Guide”. Expert systems, v. 35, n. 6. https://doi.org/10.1111/exsy.12309

Ennas, Gianfranco; Biggio, Battista; Di-Guardo, Maria-Chiara (2015). “Data-driven journal meta-ranking in business and management”. Scientometrics, v. 105, n. 3, pp. 1911-1929. https://doi.org/10.1007/s11192-015-1751-y

Feng, Lin; Zhou, Jian; Liu, Sheng-Lan; Cai, Ning; Yang, Jie (2020). “Analysis of journal evaluation indicators : an experimental study based on unsupervised Laplacian score”. Scientometrics, v. 124, pp. 233-254. https://doi.org/10.1007/s11192-020-03422-8

Fernández-Cano, Antonio; Fernández-Guerrero, Inés M. (2017). “A multivariate model for evaluating emergency medicine journals”. Scientometrics, v. 110, n. 2, pp. 991-1003. https://doi.org/10.1007/s11192-016-2197-6

Ferrer-Sapena, Antonia; Sánchez-Pérez, Enrique A.; Peset, Fernanda; González, Luis-Millán; Aleixandre-Benavent, Rafael (2016). “The impact factor as a measuring tool of the prestige of the journals in research assessment in mathematics”. Research evaluation, v. 25, n. 3, pp. 306-314. https://doi.org/10.1093/reseval/rvv041

Fersht, Alan (2009). “The most influential journals: Impact factor and eigenfactor”. Proceedings of the National Academy of Sciences of the United States of America, v. 106, n. 17, pp. 6883-6884. https://doi.org/10.1073/pnas.0903307106

González-Pereira, Borja; Guerrero-Bote, Vicente P.; De-Moya-Anegón, Félix (2010). “A new approach to the metric of journals scientific prestige: The SJR indicator”. Journal of informetrics, v. 4, n. 3, pp. 379-391. https://doi.org/10.1016/j.joi.2010.03.002

Halim, Zahid; Khan, Shafaq (2019). “A data science based framework to categorize academic journals”. Scientometrics, v. 119, n. 1. https://doi.org/10.1007/s11192-019-03035-w

Holmberg, Kim; Park, Han Woo (2018). “An altmetric investigation of the online visibility of South Korea-based scientific journals”. Scientometrics, v. 117, n. 1, pp. 603-613. https://doi.org/10.1007/s11192-018-2874-8

Hsu, Wen-Chin; Tsai, Chih-Fong; Li, Jia-Huan (2015). “A hybrid indicator for journal ranking: An example from the field of Health Care Sciences and Services”. Online information review, v. 39, n. 7, pp. 858-869. https://doi.org/10.1108/OIR-11-2014-0277

James, Chris; Colledge, Lisa; Meester, Wim; Azoulay, Norman; Plume, Andrew (2018). “CiteScore metrics: Creating journal metrics from the Scopus citation index”. ArXiv, 16 pp. https://onlinelibrary.wiley.com/doi/epdf/10.1002/leap.1246

Lacasse, Jeffrey R.; Hodge, David R.; Bean, Kristen F. (2011). “Evaluating the productivity of social work scholars using the h-index”. Research on social work practice, v. 21, n. 5, pp. 599-607. https://doi.org/10.1177/1049731511405069

Leydesdorff, Loet; Park, Han Woo (2017). Full and fractional counting in bibliometric networks. Journal of informetrics, 11, n. 1, pp. 117-120. https://doi.org/10.1016/j.joi.2016.11.007

Mañana-Rodríguez, Jorge (2015). “A critical review of SCImago Journal & Country Rank”. Research evaluation, v. 24, n. 4, pp. 343-354. https://doi.org/10.1093/reseval/rvu008

Moed, Henk F. (2010). “Measuring contextual citation impact of scientific journals”. Journal of informetrics, v. 4, n. 3, pp. 265-277. https://doi.org/10.1016/j.joi.2010.01.002

Morris, Huw; Harvey, Charles; Kelly, Aidan (2009). “Journal rankings and the ABS journal quality guide”. Management decision, v. 47, n. 9, pp. 1441-1451. https://doi.org/10.1108/00251740910995648

Mubarak, Muhammed; Seemee, Shabana (2021). “Journal prestige index: Expanding the horizons of assessment of research impact”. Journal of the College of Physicians and Surgeons Pakistan, v. 31, n. 11, pp. 1261-1262. https://doi.org/10.29271/jcpsp.2021.11.1261

Perera, Upeksha; Wijewickrema, Manjula (2018). “Relationship between journal-ranking metrics for a multidisciplinary set of journals”. Portal: Libraries and the academy, v. 18, n. 1, pp. 35-58. https://doi.org/10.1353/pla.2018.0003

Saarela, Mirka; Kärkkäinen, Tommi (2020). “Can we automate expert-based journal rankings? Analysis of the Finnish publication indicator”. Journal of informetrics, v. 14, n. 2. https://doi.org/10.1016/j.joi.2020.101008

Saarela, Mirka; Kärkkäinen, Tommi; Lahtonen, Tommi; Rossi, Tuomo (2016). “Expert-based versus citation-based ranking of scholarly and scientific publication channels”. Journal of informetrics, 10, n. 3, pp. 693-718. https://doi.org/10.1016/j.joi.2016.03.004

Setti, Gianluca (2013). “Bibliometric indicators: Why do we need more than one?”. IEEE Access, v. 1, pp. 232-246. https://doi.org/10.1109/ACCESS.2013.2261115

Tsai, Chih-Fong (2014). “Citation impact analysis of top ranked computer science journals and their rankings”. Journal of informetrics, v. 8, n. 2, pp. 318-328. https://doi.org/10.1016/j.joi.2014.01.002

Tüselmann, Heinz; Sinkovics, Rudolf R.; Pishchulov, Grigory (2015). “Towards a consolidation of worldwide journal rankings - A classification using random forests and aggregate rating via data envelopment analysis”. Omega (United Kingdom), v. 51, pp. 11-23. https://doi.org/10.1016/j.omega.2014.08.002

Walters, William H. (2017). “Do subjective journal ratings represent whole journals or typical articles? Unweighted or weighted citation impact?”. Journal of informetrics, v. 11, n. 3, pp. 730-744. https://doi.org/10.1016/j.joi.2017.05.001

Wical, Stephanie H.; Kocken, Gregory J. (2017). “Open access and promotion and tenure evaluation plans at the University of Wisconsin-Eau Claire”. Serials review, v. 43, n. 2, pp. 111-119. https://doi.org/10.1080/00987913.2017.1313024

Wyatt, Sally; Milojevič, Stasa; Park, Han Woo; Leydesdorff, Loet (2017). Quantitative and qualitative STS: The intellectual and practical contributions of Scientometrics. Boston, MA: MIT Press. https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2588336

Yuen, Jason (2018). “Comparison of Impact Factor, Eigenfactor Metrics, and SCImago Journal Rank Indicator and h-index for neurosurgical and spinal surgical journals”. World neurosurgery, v. 119, pp. e328-e337. https://doi.org/10.1016/j.wneu.2018.07.144

Zhu, Yu-Peng; Park, Han Woo (2022). “Profiling the most highly cited scholars from China: Who they are. To what extent they are interdisciplinary”. Profesional de la informacion, v. 31, n. 4, e310408. https://doi.org/10.3145/epi.2022.jul.08

Downloads

Published

2023-12-29

How to Cite

Hameed, A., Omar, M., Bilal, M., & Park, H. W. (2023). Toward the consolidation of a multi-metric-based journal ranking and categorization system for computer science subject areas. Profesional De La información, 32(7). https://doi.org/10.3145/epi.2023.dic.03