Impact of Low Back Pain Clinical Trials Measured by the Altmetric Score: Cross-Sectional Study

Introduction

There is growing interest from both authors and publishers in sharing the results of their studies over the Internet in order to increase their readership [1,2]. Similarly, consumers of research (including clinicians and patients) also share articles that they found interesting, useful, or controversial with their peers over the Internet. One of the ways that articles can be disseminated on a large scale is by sharing them on social media, such as Facebook, Twitter, Instagram, and others. Another way involves the use of reference manager websites such as Mendeley or Connotea [2]. Both methods offer new approaches to access, read, and discuss research articles; consequently, the dissemination of these articles increases [2-4].

The most traditional way of quantifying the scientific impact of an article or a journal is through the number of citations in peer-reviewed journals [5]. However, indices related to the number of journal citations do not necessarily reflect a greater dissemination of the content of articles to clinicians and patients. Until very recently, the impact of scientific articles on social media and reference managers was not quantified. In order to measure this type of impact a new score (named Altmetric) was created [2,6].

Altmetric is a tool developed by a group of British researchers [7]. Altmetric aims to understand the individual impact of each article through the attention attracted online (eg, on social media and reference managers) [6]. The Altmetric attention score is composed by two independent scoring systems: the Altmetric mentioned score and the Altmetric reader score [7]. The Altmetric mentioned score for an article reflects how widely an article is mentioned in a range of media, including social media (eg, Facebook, Twitter), newspapers, encyclopedias (eg, Wikipedia), online platforms (eg, Faculty1000 and Publication Peer-Reviews), videos on YouTube, sites on questions and answers (eg, Q&A stack overflow), and policy documents or PDF documents available over the Internet . Each of these mentions receives different weights to reflect the relative reach of each source, and contributes to the total score. For example, each mention on Facebook counts as 0.25 points while a mention on Twitter counts as 1.0 point. The Altmetric reader score can be visualized by clicking the Altmetric “donut” symbol (ie, a visual representation of the Altmetric score) and summing the number of readers. This score is hidden in the donut. The second score is the Altmetric reader score which measures the impact on online reference managers such as Mendeley, CiteULike and Connotea [7] . This score has identical weights for all reference managers (ie, 1.0 point for each mention). Readers can easily identify the Altmetric mentioned in the websites of most journals by clicking on the Altmetric “donut.”

Most journal articles about Altmetric published to date are only introductory tutorials or editorials [2,4-6,8]. Smith et al [9] published a discussion paper about the importance of Altmetric in the field of health sciences that aimed to quantify the social impact of these articles. Patthi et al [10] published a systematic review that retrieved seven articles published between 2010 to 2016 in the dental area that aimed to analyze the correlations between journal citations measured by the Web of Science website and Altmetric mentioned scores. This review concluded that journal citations and Altmetric scores are positively correlated (between r=0.30 and 0.61) [10]. Finally, Rosenkrantz et al [11] also observed positive correlations between citations and Altmetric scores in radiology journals.

There are three articles [12-14] that have measured the correlation among Altmetric, Tweets, blogs, and Mendeley (a reference management software). These studies observed a large increase in the number of Tweets and blogs related to scientific journals, and these variables were correlated with the Altmetric score . Previously published articles [13,15,16] showed that the number of Tweets can predict citations within the first three days of article publication. These findings indicate that Altmetric scores are likely to be correlated with the journal’s impact factor [8]. Rinald [17] published a tutorial about the benefits of open access journals with regards to visibility on social media. However, more research is needed to identify potential variables that might be associated with Altmetric scores, such as the journal impact factor, number of years since publication, study quality, and open access articles.

To our knowledge, there is no study describing the characteristics of randomized controlled trials (RCTs) and their Altmetric scores or predictive factors of Altmetric score. In this study, trials of nonpharmacological interventions for low back pain were chosen by the authors because back pain has the largest amount of evidence in the field of musculoskeletal health [18]. Additionally, back pain is extremely prevalent [19-21] and involves high costs [19,20,22]. According to a study that ranks the most disabling diseases in the world [21], low back pain has been one of the highest ranking musculoskeletal diseases since 1990 [21].

Therefore, the primary objective was to analyze potential factors related to the publishing journal (eg, online access) and the publishing trial (eg, trial quality) that could be associated with Altmetric scores in a random sample of low back pain RCTs. The secondary objective of this study was to describe the characteristics of these trials and their Altmetric scores.

Methods

Study Design

This is a cross-sectional study.

Search Strategies

We selected a random sample of 200 low back pain RCTs from the Physiotherapy Evidence Database (PEDro) [23]. We have chosen PEDro because this is the most comprehensive database of physiotherapy trials [24,25], and also because the PEDro scale has acceptably high reliability and validity [18]. In addition, all trials indexed on PEDro are rated for methodological quality using the 0-to-10-point PEDro scale [24-27]. The items are described below:

1. Eligibility criteria
2. Random allocation
3. Concealed allocation
4. Baseline comparability
5. Blinding of subjects
6. Blinding of therapists
7. Blinding of outcome assessors
8. Completeness of follow up
9. Intention to treat analysis
10. Between-group statistical comparisons
11. Presentation of point measures and measures of variability

The total PEDro score is computed by summing yes responses to items 2-11. The first item does not count in the final score because this is related to external validity. All trials on PEDro are rated by at least two trained raters and, in cases of disagreement, a final arbitration is performed by a senior rater.

On February 1, 2016 we identified all low back pain trials indexed on PEDro that were published in the time period of 2010-2015 and selected a random sample of 40%. We excluded trial protocols, preliminary analyses of trials, and secondary analyses. The search strategy is described as follows:

Strategy search: “2010 until 2015” [year of publication] and “low back pain” [part of body] and “pain” [problem] and “clinical trial” [method].

Data Extraction

Several pieces of data were extracted, as detailed in Textbox 1.

Data related to Altmetric scores and number of citations divided by years since publication were collected on May 10, 2016 for all articles because these scores are extremely dynamic.

Statistical Analyses

The number of years since publication of the article and the number of citations were determined as of May 10, 2016. The number of citations was normalized by the number of years since publication (number of citations divided by years since publication), as it is expected that older manuscripts are more likely to have a larger number of citations compared to newer ones. Descriptive statistics were used to present most of the data.

Separate multivariate regression models were built to predict (1) Altmetric mentioned score and (2) Altmetric reader score. The independent variables in both models were: impact factor, paper was published as open access (yes/no), total PEDro score, number of years since publication, normalized citation count, and type of title. These variables were chosen because it seems plausible that they would be associated with Altmetric scores. For example, we choose the variable number of years since publication and total PEDro score because they are related to the number of accesses on PEDro [18].

Initially, univariate regression analyses were performed and all variables that reached a P value of <0.20 were retained for inclusion in the multivariate model. Multivariate regression models were then built and the final model contained only variables that reached a statistical significance of P<0.05. The results were expressed as R² indexes (explained variability of the model) and the individual contribution of each variable was expressed through the presentation of β-coefficients and their respective 95% CIs. We used the Statistical Package for Social Sciences (SPSS) version 19 for the analyses.

Results

Selection of Eligible Articles

A total of 537 clinical trials were retrieved using the search strategy. Sixty-seven articles were excluded because they were related to conditions other than low back pain or were related to studies in progress. From the remaining 470 articles, 200 were randomly selected for analyses (Figure 1).

Descriptive Characteristics of Articles

Table 1 presents the characteristics of the trials. From the 200 articles, 186 had an Altmetric score with a mean mentioned score of 18.2 (SD 41.3) and a mean reader score of 34.9 (SD 41.6). Most of the articles were published in English, had a descriptive title (title describing the aim, but does not reveal the main conclusions) and were published as open access in journals that endorse the CONSORT statement. In addition, the mean impact factor of the journals publishing these trials was 2.1 (SD 2.6) with a mean total PEDro score of 5.8 points (SD 1.6; Table 1).

Predictive Factors

The univariate analysis for Altmetric mentioned score showed that being published in an open access journal was not independently associated with Altmetric mentioned score (Table 2). The final multivariate model is presented in Table 3. Four variables were associated with Altmetric mentioned score: impact factor (β-coefficient=3.4 points), number of years since publication (β coefficient=–4.9), number of citations divided by years since publication (β-coefficient=5.2 points), and descriptive title (β-coefficient=–29.4 points). This model accounts for 28% of the explained variance. The interpretation of this model is that older articles and those with descriptive titles were associated with a lower Altmetric mentioned score, whereas articles from journals with a higher impact factor and with greater citations were associated with a higher Altmetric mentioned score.

The univariate analysis for Altmetric reader score showed that being published in an open access journal was not associated with Altmetric reader score (Table 4). The multivariate analysis showed that one independent variable was associated with Altmetric reader score: number of citations divided by years since publication (β-coefficient=10.1 points, 95% CI 7.74-12.46). This single variable accounted for 31% of the explained variance.

‎View this figure

Discussion

Principal Findings

The primary objective of this study was to analyze potential factors that could be associated with Altmetric score. The secondary objective was to describe the characteristics of low back pain RCTs and their Altmetric scores. We found that trials with interrogative/declarative titles, those published in higher impact factor journals, those published more recently, and those with a larger number of citations were associated with a higher Altmetric mentioned score. We observed that the number of citations was also associated with a higher Altmetric reader score. Finally, we found that the Altmetric reader score was higher than Altmetric mentioned score. Most of the articles were published in English, had descriptive titles, and were published as open access in journals that endorse the CONSORT statement.

There are three previous articles that have measured correlations between the number of citations and Altmetric scores in medical journals [10,11,29], and numerous others studying the relationship between tweets (the main Altmetric score component) and citations ([15] being the first). The conclusions of these articles are very similar to our study: there is an association between citations and Altmetric scores. This information confirms that conventional measures of scientific impact (based on citations) are associated with social impact (based on social media). The difference between our study and these three previous studies is that we used multivariate regression analyses rather than simple correlations. We believe that this approach allowed us to focus on the key independent variables, and the beta coefficients we provide are more interpretable than correlation coefficients. For example, we can predict that for every citation received, 5.2 and 10.1 points will be added to the Altmetric mentioned and reader scores, respectively. Scientific impact appears to follow the social impact in back pain trials.

We observed an association between Altmetric mentioned score and the journal’s impact factor. The journal’s impact factor is a measure that reflects the number of citations of scientific articles published in the journal divided for the two previous years [5]. We might infer that journals with higher impact factors have more credibility to a wider range of readers, health care providers, and media, which may reflect a large number of posts in social media. Many of these journals may have well-developed media strategies, such as preparation and distribution of releases to the media. This action encourages the promotion of their papers in newspapers, blogs, and social media more rapidly and efficiently than journals that do not do this. These journals also format the online versions of articles so that a reader can easily click an icon to post details about the study on social media, usually by embedding a key figure from the article in the post.

We also observed that trials with declarative/interrogative titles were associated with higher Altmetric mentioned scores than those with descriptive titles. Our study is, to our knowledge, the first that has investigated the effect of the format of the articles title on Altmetric scores. There is evidence that articles with shorter titles are more likely to be highly cited [30-32]. Jamali et al [33] concluded that articles with interrogative titles are also associated with a larger number of citations and downloads [33]. Therefore, authors should be aware that shorter and interrogative titles should be considered in order to attract a wider audience for their manuscripts. Finally, we observed that papers published more recently also have a higher Altmetric score; it seems that recent studies are more likely to be shared. This finding should be investigated further in future studies.

The strength of this study is the use of a representative sample of trials (N=200, or 40% of all trial reports indexed on PEDro and categorized as “low back pain”). A possible limitation of this study is related to external validity, as our dataset contains only articles about low back pain. It would be important to replicate our study in other health disciplines.

Conclusion

Our study brings new insights for authors on how to increase the visibility of their articles. First, researchers should preferably select high impact factor journals for submission and use declarative or interrogative titles, as these factors are likely to increase the visibility of their studies in social media. Furthermore, we suggest new studies that use different research designs (eg, systematic reviews and guidelines) in order to externally validate our findings.