Making Waves: Evaluating the Georgia CTSA Publication Portfolio from 2007-2020
In a new study, the Evaluation and Continuous Improvement (ECI) team of the Georgia CTSA evaluated publications characterized as Big Splashes with immediate impact and publications with Ripple Effects over time, to understand how Georgia CTSA-supported research is ‘making waves’ that can accelerate translation. Previous systematic evaluations of the Georgia CTSA’s publication portfolio revealed a robust, diverse, and highly impactful portfolio compared to similar CTSA hubs across the country. In 2020, ECI researchers examined the progress made in publications since the 2017 transition from the Atlanta Clinical and Translational Science Institute (ACTSI) to the Georgia CTSA, with the important addition of a novel kind of publication analysis, altmetrics. Traditionally, publication analyses involved studying publication productivity and influence through academic citations; altmetrics describe the influence of published research in non-academic spheres, such as media and community attention, and references in patents and public policy. Papers with early altmetric attention reflect Big Splashes with immediate public impact, whereas papers that accumulate many academic citations reflect Ripple Effects of influence over time.
Searching public databases for all publications acknowledging support from Georgia CTSA grants, the researchers found 3,351 articles that were supported by the Georgia CTSA from its inception in 2007 through 2020. So far, these articles have been cited over 120K times, for a mean citation impact score (NIH Relative Citation Ratio) of 2.3, meaning that these articles have been cited an average of 2.3 times as often as comparable articles from the same year and field. These publications have been referenced over 56K times in the Altmetric Explorer database, including 4,800+ times in news stories, 46,000+ times in Tweets, 856 times in blog posts, 195 times in Wikipedia, 897 times in patents and 414 times in policy documents (view a summary page of Georgia CTSA altmetric attention highlights).
"This evaluation provides a new angle on how Georgia CTSA-supported research is being used in both academic and non-academic circles. It allows us to understand how publications are making waves, both the big splashes in early public and community attention, as well as the ripple effects on academia, technological advancement, and health policy over time," says Nikki Llewellyn, PhD, ECI Research Projects Manager.
Big Splashes Lead to Ripple Effects
Prior research in numerous fields shows little relationship between altmetrics and academic citations count, but no previous research has examined this relationship in clinical and translational science, nor the relationship between altmetrics and more sophisticated citation impact metrics like the NIH’s Relative Citation Ratio. It is also unclear whether any relationship between altmetrics and citation is simply due to publication in journals with higher journal impact factors, which by their nature are more likely to be read and cited. Therefore, the researchers tested the hypothesis that early altmetric indicators of Big Splash publications predict citation Ripple Effects, independent of the journal’s impact.
The researchers found that within the Georgia CTSA publication portfolio, Altmetric Attention Scores, Relative Citation Ratio scores, and journal impact factors were all positively correlated with one another. Consistent with their hypothesis, analysis showed that altmetric attention predicts citation impact, even after controlling for the journal’s impact factor. Llewellyn explains, “Not surprisingly, publishing in high impact journals is associated with more academic citations over time. But, even after accounting for that effect, papers with more altmetric attention accrue significantly more citations than their peers.”
Findings indicated that Big Splash articles with more altmetric attention have Ripple Effects through increased citation influence, which is not entirely driven by publication in higher impact journals. Among the publications supported by the Georgia CTSA, public/community attention, such as references in news stories, tweets, and blogs, in addition to publication in high impact journals, heightened articles’ ability to achieve wider use and citation influence, opening the door to translational advancement. The ECI research team will present these findings at the 2021 Southeast Regional Clinical & Translational Conference before submitting for publication.
The Georgia CTSA is a statewide partnership between Emory, MSM, Georgia Tech, and UGA and is one of over 50 in a national consortium striving to improve the way biomedical research is conducted across the country. The consortium, funded through the National Center for Advancing Translational Sciences (NCATS) and the National Institutes of Health's Clinical and Translational Science Awards, shares a common vision to translate laboratory discoveries into treatments for patients, engage communities in clinical research efforts, and train the next generation of clinical investigators.