gms | German Medical Science

65th Annual Meeting of the German Association for Medical Informatics, Biometry and Epidemiology (GMDS), Meeting of the Central European Network (CEN: German Region, Austro-Swiss Region and Polish Region) of the International Biometric Society (IBS)

06.09. - 09.09.2020, Berlin (online conference)

Article

Send article

Responsible metrics for evaluating research quality

Meeting Abstract

Search Medline for

  • Maia Salholz-Hillel - QUEST Center, Berlin Institute of Health (BIH) Center for Transforming Biomedical Research, Charité – Universitätsmedizin Berlin, Berlin, Germany
  • Delwen Franzen - QUEST Center, Berlin Institute of Health (BIH) Center for Transforming Biomedical Research, Charité – Universitätsmedizin Berlin, Berlin, Germany
  • Peter Grabitz - QUEST Center, Berlin Institute of Health (BIH) Center for Transforming Biomedical Research, Charité – Universitätsmedizin Berlin, Berlin, Germany
  • Martin Holst - QUEST Center, Berlin Institute of Health (BIH) Center for Transforming Biomedical Research, Charité – Universitätsmedizin Berlin, Berlin, Germany
  • Benjamin Carlisle - QUEST Center, Berlin Institute of Health (BIH) Center for Transforming Biomedical Research, Charité – Universitätsmedizin Berlin, Berlin, Germany
  • Lana Saksone - QUEST Center, Berlin Institute of Health (BIH) Center for Transforming Biomedical Research, Charité – Universitätsmedizin Berlin, Berlin, Germany
  • Daniel Strech - QUEST Center, Berlin Institute of Health (BIH) Center for Transforming Biomedical Research, Charité – Universitätsmedizin Berlin, Berlin, Germany

Deutsche Gesellschaft für Medizinische Informatik, Biometrie und Epidemiologie. 65th Annual Meeting of the German Association for Medical Informatics, Biometry and Epidemiology (GMDS), Meeting of the Central European Network (CEN: German Region, Austro-Swiss Region and Polish Region) of the International Biometric Society (IBS). Berlin, 06.-09.09.2020. Düsseldorf: German Medical Science GMS Publishing House; 2021. DocAbstr. 493

doi: 10.3205/20gmds102, urn:nbn:de:0183-20gmds1028

Published: February 26, 2021

© 2021 Salholz-Hillel et al.
This is an Open Access article distributed under the terms of the Creative Commons Attribution 4.0 License. See license information at http://creativecommons.org/licenses/by/4.0/.

Outline

Text

The quality of research, researchers, and research institutions is traditionally assessed with metrics such as journal impact factor, H-index, and number of publications. These metrics affect research choices (e.g., whether and how a researcher pursues a question), career success of individual researchers (e.g., hiring, tenure), and the status of research institutions (e.g., receipt of large-scale grants, rankings). However, these metrics fail to capture the full range of research value and often provoke research waste by incentivizing poor practices, for example, promoting the number rather than the quality of research publications.??????

Researchers, funders, and regulators have proposed alternate metrics to serve as more holistic indicators of good research practices and hence research value. Such practices include, for example, open access publication, pre-registration of research design and analysis, and timely publication of all research findings. To enable the adoption of new metrics at the institutional level, we must develop tools to allow for efficient generation of the metrics, create strategies for communicating these metrics with the institutions, and evaluate whether each metric serves as a successful indicator.

In this talk, I will present our work to develop automated pipelines to determine the proportion of research projects pursuing responsible practices; for example, are clinical trials pre-registered, and is the registration properly reported in the publication, abstract, and meta-data? I will also present a web tool for disseminating these metrics to research institutions. Finally, I will present a planned evaluation study to assess the accessibility and usefulness of these new metrics. This work is conducted within the context of Germany's 38 University Medical Centers, with the applied goal of improving research practices at these institutions as well as developing tools to support responsible knowledge generation in a broader context.

The authors declare that they have no competing interests.

The authors declare that an ethics committee vote is not required.

Outline

References

1.
Begley CG, Buchan AM, Dirnagl U. Robust research: Institutions must do their part for reproducibility. Nature. 2015 Sep 3;525(7567):25-7. DOI: 10.1038/525025a External link
2.
Hicks D, Wouters P, Waltman L, de Rijcke S, Rafols I. Bibliometrics: The Leiden Manifesto for research metrics. Nature. 2015 Apr 23;520(7548):429-31. DOI: 10.1038/520429a External link
3.
Ioannidis JP, Khoury MJ. Assessing value in biomedical research: the PQRST of appraisal and reward. JAMA. 2014 Aug 6;312(5):483-4. DOI: 10.1001/jama.2014.6932 External link
4.
Macleod MR, Michie S, Roberts I, Dirnagl U, Chalmers I, Ioannidis JP, Al-Shahi Salman R, Chan AW, Glasziou P. Biomedical research: increasing value, reducing waste. Lancet. 2014 Jan 11;383(9912):101-4. DOI: 10.1016/S0140-6736(13)62329-6 External link
5.
Moher D, Naudet F, Cristea IA, Miedema F, Ioannidis JPA, Goodman SN. Assessing scientists for hiring, promotion, and tenure. PLoS Biol. 2018 Mar 29;16(3):e2004089. DOI: 10.1371/journal.pbio.2004089 External link
6.
Nosek BA, Alter G, Banks GC, Borsboom D, Bowman SD, Breckler SJ, Buck S, Chambers CD, Chin G, Christensen G, Contestabile M, Dafoe A, Eich E, Freese J, Glennerster R, Goroff D, Green DP, Hesse B, Humphreys M, Ishiyama J, Karlan D, Kraut A, Lupia A, Mabry P, Madon TA, Malhotra N, Mayo-Wilson E, McNutt M, Miguel E, Paluck EL, Simonsohn U, Soderberg C, Spellman BA, Turitto J, VandenBos G, Vazire S, Wagenmakers EJ, Wilson R, Yarkoni T. SCIENTIFIC STANDARDS. Promoting an open research culture. Science. 2015 Jun 26;348(6242):1422-5. DOI: 10.1126/science.aab2374 External link
7.
San Francisco Declaration on Research Assessment (DORA). 2012 Dec 16. Available from: https://sfdora.org/ External link
8.
Wieschowski S, Riedel N, Wollmann K, Kahrass H, Müller-Ohlraun S, Schürmann C, Kelley S, Kszuk U, Siegerink B, Dirnagl U, Meerpohl J, Strech D. Result dissemination from clinical trials conducted at German university medical centers was delayed and incomplete. J Clin Epidemiol. 2019 Nov;115:37-45. DOI: 10.1016/j.jclinepi.2019.06.002 External link