Evaluation & Continuous Improvement
The Evaluation & Continuous Improvement (ECI) program is the data and evaluation arm of Georgia CTSA charting the impact of our hub's activities, helping to identify challenges in progress and develop recommendations for informed decision-making through evaluation data. We aim to set strategic directions for our alliance through assessing the advancement and impact of the Georgia CTSA on local, regional, and national clinical and translational science.
Members of the ECI team include:
- Eric Nehl, Ph.D. (Emory University)
- Nikki Llewellyn, Ph.D. (Emory University)
- Gaurav Rajgopal, MPH (Emory University)
- Kourtnii Farley, Ph.D., MPH (Morehouse School of Medicine)
- Amanda Abraham, Ph.D. (University of Georgia)
ECI regularly employs innovative approaches, including advanced bibliometric techniques, network analysis, geospatial analysis, web data extraction, and web applications, to perform evaluation activities.
Additional key functions of ECI include annual strategic planning for each core and function, dissemination of evaluation reports to Georgia CTSA leadership and oversight committees, and the development and adoption of innovative evaluation metrics and techniques (including the Common Metrics) to produce a comprehensive, data-rich assessment of progress and impact throughout Georgia.
Examples of our current projects involve quantitative and qualitative analysis of the scope and research impact of our Georgia CTSA Pediatrics program and of our larger alliance as a whole. Additionally, we are assessing Georgia CTSA's role in supporting investigators through the technology transfer process and reaching successful patenting of an invention.
Past projects have focused on evaluating the outcomes of Georgia CTSA-supported pilot program projects, as well as evaluating the needs of investigators engaging in the Georgia CTSA-supported K-Club program, aimed to provide educational support to fellows and faculty who seek additional career development award funding. The results of these evaluation projects can be found below.
Bibliometrics Research: Tracking Georgia CTSA Publications
View Supported Publications in PubMed
View Publication Portfolio Infographic
View Publication Highlights in Altmetric
Publications
Bibliometric publications authored by the Evaluation & Continuous Improvement program:
- Llewellyn, N.M, Nehl, E. J., Dave, G., DiazGranados, D., Flynn, D., Fournier, D., Hoyo, V., Pelfrey, C., & Casey, S. (2024) Translation in action: Influence, collaboration, and evolution of COVID-19 research with Clinical and Translational Science Awards consortium support. Clinical & Translational Science, 17(1):e13700. doi: 10.1111/cts.13700
- Llewellyn, N.M, Weber, A., Pelfrey, C., DiazGranados, D. & Nehl, E. J. (2023) Translating Scientific Discovery into Health Impact: Innovative Bibliometrics Bridge CTSA-Supported Publications to Policy. Academic Medicine, 98(8):896-903. doi: 10.1097/ACM.0000000000005225
- Llewellyn, N.M. & Nehl, E. J. (2022) Predicting Citation Impact from Altmetric Attention in Clinical and Translational Research: Do Big Splashes Lead to Ripple Effects? Clinical and Translational Science, 15(6), 1387-1392. doi: 10.1111/cts.13251
- Llewellyn, N. M., Weber, A. A., Fitzpatrick, A. M., & Nehl, E. J. (2022). Big splashes & ripple effects: a narrative review of the short- & long-term impact of publications supported by an NIH CTSA pediatrics program. Translational Pediatrics, 11(3), 411-422.
- Kim, Y.H., Levine, A.D., Nehl, E.J., & Walsh, J.P. (2020). A bibliometric measure of translational science. Scientometrics, 1-34.
- Llewellyn, N., Carter, D. R., DiazGranados, D., Pelfrey, C., Rollins, L. & Nehl, E.J. (2020). Scope, Influence & Inter-disciplinary Collaboration: the Publication Portfolio of the NIH Clinical and Translational Science Awards (CTSA) Program from 2006 through 2017. Evaluation & the Health Professions, 43(3), 169-179.
- Llewellyn, N.M., Carter, D. R., Rollins, L. & Nehl, E.J. (2018). Charting the Publication and Citation Impact of the NIH Clinical & Translational Science Award (CTSA) Program from 2006-2016. Academic Medicine, 93(8), 1162-1170.
Internal Reports
Bibliometric reports produced by the Evaluation & Continuous Improvement program:
- Making Waves: The Impact of the Georgia CTSA Publication Portfolio from 2007-2021
- Georgia CTSA Pediatrics Product Evaluation
- Georgia CTSA 2017 Publication Content Evaluation
- Georgia CTSA 2016 Publication & Citation Impact Evaluation
Infographics
Poster Presentations
ECI team presenting Enhancing Health Equity within the Georgia CTSA: Identifying Programmatic Opportunities for Increasing URM Faculty Support research at the 2023 Southeast Regional Conference.
Bibliometric posters presented by the Evaluation & Continuous Improvement program:
- Influence and Translation in Action: COVID-19 Research Supported by the CTSA Consortium from 2020-2022
- Altmetric Impact of the Georgia CTSA Publication Portfolio from 2007-2020: Do Big Splashes lead to Ripple Effects on Translation?
- Expanding the View of Research Impact - Altmetrics of the Georgia CTSA Pediatrics Program
- Big Splashes & Ripple Effects: The Short- and Long-term Impact of Georgia CTSA-supported Pediatric Publications
- A Qualitative and Quantitative Summary of the Content of Publications Supported by the National CTSA Program from 2006-2017: Diversity, Scope, and Inter-disciplinarity
- Charting the Publication and Citation Impact of the NIH CTSA Program from 2006 Through 2016
- Evolution of the Interdisciplinary Co-Citation Network Supported by the Georgia Clinical and Translational Science Alliance Program from 2006 Through 2016
Other Evaluation Research Projects
Publications
Publications authored by the Evaluation & Continuous Improvement program:
- Greteman, B.B., Rollins, L., Penn, A., Berg, A., Nehl, E., Llewellyn, N., Weber, A., George, M., Sabbs, D., Mubasher, M., & Akintobi, T. H. (2022). Identifying the Community-Engaged Translational Research Collaboration Experience and Health Interests of Community-Based Organizations Outside of Metropolitan Atlanta. Journal of the Georgia Public Health Association, 8(3), Article 18. doi:10.21203/rs.3.rs-52357/v1
- Roback, J.D., Tyburski, E.A., Alter, D., Asakrah, S., Chahroudi, A., Esper, A., Farmer, S., Figueroa, J., Frediani, J.K., Gonzalez, M.D., Gottfried, D.S., Guarner, J., Gupta, N.A., Heilman, S.S., Hill, C.E., Jerris, R., Kempker, R.R., Ingersoll, J., Levy, J.M.,... Lam W.A. (2021). The need for new test verification and regulatory support for innovative diagnostics. Nature Biotechnology. https://doi.org/10.1038/s41587-021-01047-7.
- Frediani, J.K., Levy, J.M., Rao, A., Bassit, L., Figueroa, J., Vos, M.B., Wood, A., Jerris, R., Leung-Pineda, V., Gonzalez, M.D., Rogers, B.B., Mavigner, M., Schinazi, R.F., Schoof, N., Waggoner, J.J., Kempker, R.R., Rebolledo, P.A., O’Neal, J.W., Stone, C.,... Lam, W.A. (2021). Multidisciplinary assessment of the Abbott BinaxNOW SARS-CoV-2 Point-of-Care Antigen Test in the context of emerging viral variants and self-administration. Scientific Reports. 11(1), 1-9.
- Nehl, E.J., Heilman, S.S., Ku, D., Gottfried, D.S., Farmer, S., Mannino, R., Tyburski, E., Sullivan, J., Suessmith, A., Bassit, L., Figueroa, J., Wood, A., Leong, T., Rao, A., Rogers, B., Jerris, R., Park, S., Gonzalez, M.D., Frediani, J.K.,... Lam W.A. (2021). The RADx Tech Test Verification Core and the ACME POCT in the evaluation of COVID-19 testing devices: A model for progress and change. IEEE Open Journal of Engineering in Medicine and Biology, 2, 142-151.
- Llewellyn, N.M., Adachi, J.J., Nehl, E. J., & Heilman, S.S. (2021). Participant perspectives on a seminar-based research career development program and its role in career independence. Journal of Investigative Medicine 2021; 69:775-780.
- Rollins, L., Llewellyn, N., Ngaiza, M., Nehl, E., Carter, D. R., & Sands, J. (2020). Using the payback framework to evaluate the outcomes of pilot projects supported by the Georgia Clinical and Translational Science Alliance. Journal of Clinical and Translational Science, 5(1):e48. doi: 10.1017/cts.2020.542
Internal Reports
Internal reports produced by the Evaluation & Continuous Improvement program:
- 2024 Strategic Planning Report
- 2023 Strategic Planning Report
- A Mixed Methods Evaluation of Georgia CTSA Support of the Technology Transfer Process
- 2020 Strategic Planning Report
- 2019 Strategic Planning Report
- 2018 Strategic Planning Report
- An Evaluation of Emory University School of Medicine & Children's Healthcare of Atlanta's K-Club Program 2017
Poster Presentations
Posters presented by the Evaluation & Continuous Improvement program:
- Translating Research to Policy: From Research Influencers to Substance Use Policymakers.
- Enhancing Health Equity within the Georgia CTSA: Identifying Programmatic Opportunities for Increasing URM Faculty Support
- Statewide Reach of Pediatrics Research Supported by the Georgia CTSA
- 2019 Mixed-Method Evaluation of Georgia CTSA’s Technology Transfer Process
- Leveraging Longitudinal Network Analysis to Evaluate Interdisciplinary Collaboration
- A Participatory Approach to Strategic Planning within a Clinical and Translational Science Hub: The Georgia CTSA
- An Application of the Payback Framework to Evaluate the Outcomes of Pilot Projects Supported by the Georgia Clinical and Translational Science Alliance from 2007-2014: Preliminary Results