Published on in Vol 23, No 12 (2021): December

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/34051, first published .
Citation Advantage of Promoted Articles in a Cross-Publisher Distribution Platform: 36-Month Follow-up to a Randomized Controlled Trial

Citation Advantage of Promoted Articles in a Cross-Publisher Distribution Platform: 36-Month Follow-up to a Randomized Controlled Trial

Citation Advantage of Promoted Articles in a Cross-Publisher Distribution Platform: 36-Month Follow-up to a Randomized Controlled Trial

Original Paper

1Department of Psychiatry, University of Toronto, Toronto, ON, Canada

2TrendMD Inc, Toronto, ON, Canada

3University of Victoria, Victoria, BC, Canada

*all authors contributed equally

Corresponding Author:

Paul Kudlow, MD, PhD

Department of Psychiatry

University of Toronto

250 College Street

Toronto, ON, M5T 1R8

Canada

Phone: 1 416 480 6100

Email: paul.kudlow@gmail.com


Background: There are limited evidence-based strategies that have been shown to increase the rate at which peer-reviewed articles are cited. In a previously reported randomized controlled trial, we demonstrated that promotion of article links in an online cross-publisher distribution platform (TrendMD) persistently augments citation rates after 12 months, leading to a statistically significant 50% increase in citations relative to the control.

Objective: This study aims to investigate if the citation advantage of promoted articles upholds after 36 months.

Methods: A total of 3200 published articles in 64 peer-reviewed journals across 8 subject areas were block randomized at the subject level to either the TrendMD group (n=1600) or the control group (n=1600) of the study. Articles were promoted in the TrendMD Network for 6 months. We compared the citation rates in both groups after 36 months.

Results: At 36 months, we found the citation advantage endured; articles randomized to TrendMD showed a 28% increase in mean citations relative to the control. The difference in mean citations at 36 months for articles randomized to TrendMD versus the control was 10.52 (95% CI 3.79-17.25) and was statistically significant (P=.001).

Conclusions: To our knowledge, this is the first randomized controlled trial to demonstrate how a postpublication article promotion intervention can be used to persistently augment citations of peer-reviewed articles. TrendMD is an efficient digital tool for knowledge translation and dissemination to targeted audiences to facilitate the uptake of research.

J Med Internet Res 2021;23(12):e34051

doi:10.2196/34051

Keywords



Citations are a leading indicator of scholarly impact. They measure the spread of new knowledge; acknowledge the contribution of colleagues; and, in many fields, form the basis of tenure and promotion or even direct compensation [1,2]. Citations accrue from research getting noticed and used by authors when creating their own scholarly work. The problem is, there is a growing mismatch between the number of newly published articles and the number of papers an academic can discover and ultimately cite in the creation of their own scholarly work. There are over 8000 new peer-reviewed articles published daily, a figure that is growing exponentially, making it challenging for scholars to sift through potentially relevant literature [3,4]. As a result, many relevant papers that could be cited are missed by scholars [3]. Accordingly, data suggests that roughly 35% of articles published between 1990 and 2015 remain uncited; a figure that may be increasing [5]. A study published in 2021 found that the increasing enormity of academic literature may be impeding on the rise, progress, and adoption of new ideas; the authors suggest that changes are needed in scholarship dissemination to increase the discoverability of new concepts that researchers may not be searching for [6]. The growing mismatch between the number of articles published and the ability for readers to come across articles that they were not already seeking out has created a pressing need for strategies that augment the discoverability of scholarly content and to target content to potential knowledge users. Postpublication strategies such as promotion of articles in social media and other content distribution networks promise to enhance discoverability, which may increase the chances that the most relevant paper is found and ultimately cited by scholars in the creation of their own work. However, how to target promotional campaigns to the right audience remains an open question, and there is currently limited evidence showing which strategies yield a positive effect on scholarly article impact, such as that reflected by citation counts.

One of the most widely studied strategies examined to increase discoverability, accessibility, and consequential citation counts of scholarly work is publishing content in open access (OA) journals. OA increases visibility and hence can have a positive effect on usage metrics such as the number of downloads [7], but the effect on citations is less clear and likely discipline dependent [8-12]. Since the first carefully designed multivariate observational study in 2006 showed a clear OA citation advantage even when adjusted for possible confounders [8,9,13], several large observational studies have found OA articles accrue between 50% to 200% more citations compared to closed access articles; this is referred to as the “open access citation advantage” (OACA) [8,14]. The OACA, however, has not been universally accepted [7,15,16]. For example, in 2011, Davis [16] completed a randomized controlled trial study showing that making articles free and open did not yield an increase in citation counts over a 3-year period relative to closed access articles. Part of the reason why the OACA has been found in observational data and not experimental data is likely due to residual confounders. Even though the original study suggesting an OA citation advantage was carefully adjusted for multiple confounders [8], selection bias cannot be fully adjusted for and is one of the biggest confounders identified in observational research examining the OACA [17]; more prominent authors are more likely to pay to publish OA articles, and if authors are more likely to provide access to their “highest quality” articles, then OA articles will have more citations than closed access articles [11,18]. Furthermore, as OA content becomes increasingly ubiquitous in the scholarly ecosystem, the OACA, if any, is likely to extinguish, as all articles share the same OA characteristic.

Aside from publishing research in an OA journal, other studies have suggested that the promotion of articles in social media platforms may be used to augment the page views and citations of articles, but this too remains highly controversial. For example, two recent studies [19,20] found that intensive social media promotion significantly increased page views of scholarly articles. A more recent study [21] found that an intensive social media promotion strategy yielded a citation advantage to promoted articles. However, the findings of the Luc et al [21] study have come under considerable scrutiny due to significant methodological flaws that suggests that social media promotion did not yield a citation benefit at all [22-24]. The most significant methodological flaw was the fact that the papers listed in the Luc et al [21] paper may have not matched those described by their methods. Similarly, two rigorous randomized controlled trials [25,26] found that the promotion of articles in social media did not yield any increase in article page views. These results refer to organic tweeting, and an unexamined question remains if promoted tweets, which measurably lead to more visits, also lead to more citations. Taken together, though there are mixed data as to whether social media promotion increases article page views, there are currently no replicated data to suggest that the strategy yields a citation advantage to promoted articles.

Our group previously published studies examining the extent to which the promotion of articles in a novel cross-publisher distribution platform (TrendMD) increases article page views, usage, and citation counts. In a 2014 study, TrendMD article promotion led to an 87% increase in page views relative to the control in a 4-week randomized controlled trial [27]. These data were replicated in in a 3-week crossover trial that found that promotion of articles in the TrendMD Network yielded a 30% and 49% weekly increase in page views relative to the control [28]. In 2017, we completed a 4-week randomized controlled trial that found articles randomized to TrendMD had a 77% increase in article saves on Mendeley relative to the control [29]. These findings were particularly significant because Mendeley saves are not only a robust measure of article usage, but the metric is also strongly correlated to future citations [30-34]. Building on these data, in 2019, we conducted a randomized controlled trial including 3200 articles published across 8 subject areas (eg, medicine, physics, business, and humanities) and found that promotion of articles over 6 months conferred an overall statistically significant 50% citation advantage to promoted articles relative to the control at 12 months [35]. TrendMD promotion increased citations for 3 of 8 disciplines tested, yielding the largest citation advantage to articles published in the subject areas of health, medical, and life sciences. Taken together, results of our studies suggest that TrendMD confers a short-term page view, usage, and citation advantage; however, we do not know whether the measured advantages persist over time or becomes diluted by other factors [36].

In this study, we conducted a follow-up analysis to our 12-month randomized controlled trial to determine whether the citation advantage persists at 36 months. We also sought to determine whether TrendMD’s effect on citation counts were specific to particular disciplines or consistent across all disciplines. We hypothesized that the promotion of articles in TrendMD would yield a persistent citation advantage at 36 months. We further hypothesized that this effect would be seen across more disciplines than initially measured at 12 months.


Summary

The majority of the Methods section included herein were copied verbatim from the Methods section we describe in our previously published 12-month randomized controlled trial [35].

We conducted a 36-month randomized controlled trial that included 3200 articles published in 64 peer-reviewed journals across 8 subject areas. The length of the study was 6 months for the intervention and an additional 30 months of observation for a total of 36 months. We published our initial findings at 12 months [35]. We measured citations at 6, 12, and 36 months. The subject areas/categories were selected based on the 8 categories listed in Google Scholar [37]. The categories selected were business economics and management; chemical and materials sciences; engineering and computer science; health and medical sciences; humanities, literature, and arts; life sciences and earth sciences; physics and mathematics; and social sciences. For each subject area, the top 20 journals ranked by Google Scholar’s h-5 index were selected (Google Scholar only displays the top 20 journals in each subject area). Please see the supplementary material published in our 2019 study [35] for our rationale for using Google Scholar and the h-5 index in our selection criteria. Eight journals were then randomly selected with a random number generator from the top 20 in each subject area to be included in the study; we did this as opposed to selecting the top journals in each subject area so that our sample would include a randomized mixture of journals of high and lower impact in each subject area. Including both high and lower impact journals was important to our study because we wanted to mitigate the potential confounder that TrendMD promotion is only effective in high impact journals. Journals that were not indexed in Scopus or Web of Science were excluded from the study. Preprint servers such as ArXiv were also excluded from the study.

Starting from April 2018, 50 of the most recently published original articles or review articles in each journal were selected for inclusion in the study. Articles selected for inclusion were published online between January 2017 and April 2018; this includes early view articles. Articles were excluded if they did not contain an abstract or DOI. Block randomization using a random number generator at the subject level was used to randomize articles to either the control or the intervention arm of the study. For each subject area, 200 articles were randomized to the control, and 200 articles were randomized to the intervention. In total, 1600 articles were randomized to the control, and 1600 articles were randomized to the intervention. The overall study design is presented in Figure 1.

Figure 1. Overall study design.
View this figure

Intervention

TrendMD [38] is a cross-publisher article recommendation and distribution platform that, as of May 2019, was embedded on over 4700 journals and websites from 300 publishers and seen by approximately 125 million readers per month. Roughly two-thirds of the TrendMD Network is related to scientific, technical, and medical (STM) publications; the other one-third is a relatively even split between social sciences, humanities, and business publications. Participating publishers use TrendMD to distribute their published article links within the article recommendations displayed on articles within their journals (nonsponsored recommendations) or third-party journals within the TrendMD Network (sponsored recommendations; please see Figures 2 and 3). TrendMD’s content distribution model is benchmarked to similar services in the consumer web, where the leading networks Outbrain [39] and Taboola [40] generate the “From the web” and “You may like” recommendations seen alongside the content on many popular websites like CNN or BBC [29,35] (please see Figure 4 for reference).

The intervention consisted of the promotion of 1600 articles in the TrendMD Network for 6 months, between May 1, 2018, and November 2, 2018. Articles included in the TrendMD Network are displayed as recommended article links. Links to articles randomized to TrendMD were displayed as sponsored recommended links on publications participating in the TrendMD Network. There was an average of 4300 participating journals and 121 million readers per month during the course of the study. The frequency of sponsored article link placements is determined by a relevancy score based on the following: relatedness (ie, keyword overlap), collaborative filtering (similar to Amazon’s “people who bought this item also bought that item”), and user clickstream analysis (the Netflix approach, basing recommendations on the users’ interests expressed through their online history) [27-29,35]. As a result of the relevancy scoring system, some articles randomized to TrendMD were both seen more often (ie, accrued more link impressions) and clicked on more frequently than others in the TrendMD Network.

Figure 2. TrendMD: sponsored versus nonsponsored links.
View this figure
Figure 3. How the TrendMD Network works.
View this figure
Figure 4. TrendMD widget display.
View this figure

The sponsored links are displayed in the TrendMD Network as long as they are relevant and the advertiser account balance is greater than US $0. An account in TrendMD was created for this study. The 1600 articles randomized to TrendMD received a maximum total budget of US $9600 at a cost-per-click of US $0.10 for 96,000 sponsored TrendMD clicks. The account was allowed to spend up to a maximum for US $1600 per month, or 16,000 clicks per month, over the 6-month study period for a total of 96,000 clicks. The actual amount spent by the account was US $1600 per month; all clicks were delivered each month throughout the 6-month study. A summary of how TrendMD works and the outcomes measured can be seen in Figure 3.

Control

Articles randomized to the control (n=1600) received no promotion in the TrendMD Network. Articles randomized to the control received traffic by organic means (eg, Google, Google Scholar, or PubMed) and other means implemented by publishers or authors of content outside the context of this study.

Primary Outcome

The primary outcome of our study is the mean citation counts for articles randomized to TrendMD compared to the control at 36 months. Article citation counts at 36 months were extracted through the Scopus application programming interface (API) on March 30, 2021.

Secondary Outcomes

Thirty-six–Month Analysis

Mean citation counts after 36 months were compared for articles randomized to TrendMD versus control for each of the 8 subject areas. This analysis was completed to assess whether the effects of TrendMD promotion were discipline specific. Article citation counts were extracted from the Scopus API on March 30, 2021.

Twelve- and Six-Month Analysis

Mean citation counts after 12 and 6 months were compared for articles randomized to TrendMD versus the control. Article citation counts were extracted from the Scopus API on May 2, 2019, for the 12-month data and on November 2, 2018, for the 6-month data. These data were also published in our 2019 paper and included here for convenience to readers [35].

Statistical Analysis

We performed an a priori power calculation to determine the necessary sample size (n=1600 in each arm of the study) to detect differences in our primary outcome of mean citation counts between groups at 12 months. Please see our previously published paper for how we determined the required sample size in each arm of the study [35]. The study was not powered to detect differences in citation counts at 6 or 36 months nor was it powered to detect differences across the subject area–level comparisons at any of the time intervals. To power a study to detect differences at the subject level, each subject would have required between 1000 to 3000 articles in each arm of the study, which was not feasible for us to conduct for budgetary reasons.

Baseline characteristics of articles at the start of the study were tabulated and compared across randomized arms of the study. We categorized articles by subject area, access type (closed vs OA), Journal Impact Factor, and citations and Mendeley saves. Both the primary and secondary outcomes were analyzed with the two-sample t test on log-transformed data (1 + x). In the event that mean differences were statistically significant, we calculated effect sizes using Cohen d [41] on log-transformed data. Cohen d is defined as the difference between two means divided by a SD for the data. Lastly, we performed a stepwise and backward multivariate ordinary least squares (linear) regression analysis to determine the predictors of citations at 6, 12, and 36 months; we used log-transformed dependent variables in the model. A significance value of 0.1 was used for the removal of variables in our stepwise regression model. Journal Impact Factor, access type (ie, OA vs closed), baseline Mendeley saves and citations, and TrendMD clicks (ie, clicks on promoted article links) and impressions (ie, display of promoted article links) were covariates in the regression model; they were selected as covariates because each of them has known independent effects on citations (eg, Journal Impact Factor is a predictor of citation counts) and do not have issues of multicollinearity. We did not collect data on all possible predictors of citations (eg, number of authors or international collaboration) in our regression model, as this was out of scope to our analysis; the primary goal of our regression model was to determine if TrendMD clicks and impressions were independent predictors of our primary outcome of citations at 36 months. All regressions adjusted the SEs for clustering of citations using Huber-White SEs; this corrected for heteroscedasticity [42]. Residuals were analyzed for normality using the Kolmogorov-Smirnov test; we also analyzed the residuals using skewness and kurtosis statistical values and compared to SE values.

A 2-tailed P<.05 was considered statistically significant. To mitigate the type I error rate, the Bonferroni correction method was used to control for multiple comparisons made for the 8 disciplinary differences in mean citation counts [43]. A 2-tailed P<.006 was considered to be statistically significant for mean differences in Mendeley reader and citation counts across subject areas. Arithmetic mean values for Mendeley saves and citation counts are shown with 95% CIs on non–log-transformed data unless otherwise specified. Tests for normality were included in the model. SPSS version 25 (IBM Corp) was used to complete the statistical analyses.


Baseline Characteristics

The following was published verbatim in our previously published randomized controlled trial [35]; these data and discussion are included here for convenience of the reader. Overall, 3200 articles were randomized: 1600 to the TrendMD arm and 1600 to the control arm. The Kolmogorov-Smirnov test of the 6-month Mendeley saves (P=.61) and citation count (P=.13) data retained the null hypothesis that the distributions were log-normal within the control and TrendMD arms. Tables 1 and 2 show the baseline characteristics of articles randomized to TrendMD versus the control; no statistical tests were used to compare any metrics between groups at baseline according to the CONSORT (Consolidated Standards of Reporting Trials) 2010 guidelines [44].

Table 1. Overall baseline characteristics [35].
InterventionArticles, nOpen access, nJournal Impact Fact, mean (SD)Mendeley saves, mean (SD)Citation count, mean (SD)
Control16008814.77 (15.87)23.37 (36.33)0.86 (2.98)
TrendMD16009215.41 (16.56)23.79 (39.34)1.04 (3.37)
Table 2. Baseline characteristics by subject area [35].
Category and interventionOpen access, nJournal Impact Factor, mean (SD)Mendeley saves, mean (SD)Citation count, mean (SD)
Business, economics, and management

Control05.28 (1.59)33.99 (43.69)1.02 (1.60)

TrendMD05.53 (1.55)35.38 (35.60)1.24 (2.26)
Chemical and materials sciences

Control126.53 (12.20)23.92 (26.95)1.53 (7.27)

TrendMD131.37 (15.89)25.97 (43.57)1.58 (6.31)
Engineering and computer science

Control314.48 (9.74)10.44 (16.27)0.29 (0.65)

TrendMD214.82 (9.93)10.27 (16.83)0.35 (0.97)
Health and medical sciences

Control1135.98 (22.48)37.98 (52.55)1.93 (2.29)

TrendMD734.42 (21.87)38.04 (62.8)3.1 (5.55)
Humanities, literature, and arts

Control22.15 (0.86)13.39 (17.36)1.06 (2.08)

TrendMD22.15 (0.87)13.13 (15.97)0.95 (2.09)
Life sciences and earth sciences

Control5221.57 (13.8)43.33 (53.73)0.37 (0.74)

TrendMD5122.76 (14.42)44.13 (52.87)0.48 (0.85)
Physics and mathematics

Control107.67 (5.86)8.90 (14.33)0.33 (1.09)

TrendMD117.74 (5.85)9.04 (18.12)0.31 (1.29)
Social sciences

Control94.52 (1.08)15.01 (16.56)0.36 (1.16)

TrendMD184.51 (1.06)14.39 (17.27)0.35 (1.17)

Primary Outcome

Articles randomized to TrendMD (n=1600) showed a 28% increase in mean citations relative to the control at 36 months (Figure 5). The mean citations for articles randomized to TrendMD was 48.05 (SD 113.51), compared to 37.53 (SD 77.36) for articles randomized to the control. The difference in mean citations for TrendMD articles versus the control was 10.52 (95% CI 3.79-17.25) and was statistically significant (P=.001). The effect size of TrendMD on citations at 36 months was small (Cohen d 0.11; Table 3). The cumulative distribution of article citations at 36 months is shown in Figure 6. The fact that the TrendMD cumulative distribution curve is shifted to the right relative to the control indicates the promotion of increased citation rates across the entire distribution of articles; the effect was not limited to just a few outlying articles.

Figure 5. Mean citation count over 36 months: TrendMD versus control.
View this figure
Table 3. Citation counts at 36, 12, and 6 months.

36 months12 months6 months

ControlTrendMDControlTrendMDControlTrendMD
Citations, mean (SD)37.53 (77.36)48.05 (113.51)10.10 (19.09)15.16 (40.37)5.12 (10.65)6.18 (16.10)
Mean difference in citations (95% CI)N/Aa10.52 (3.79-17.25)N/A5.06 (2.87-7.25)N/A1.06 (0.12-2.01)
P valueN/A.001N/A<.001N/A.005
Cohen dN/A0.11N/A0.16N/A0.10

aN/A: not applicable.

Figure 6. Cumulative distribution of citations over 36 months: TrendMD versus control.
View this figure

Secondary Outcomes

Subject Area Differences in Mean Citations at 36 Months

At 36 months, TrendMD was not found to yield statistically significant increases in citation counts relative to the control in any of the individual subject areas. The largest relative difference in citation counts compared to the control was found to be in the subject area of health and medical sciences (41%); however, it was not found to be statistically significant (see Table 4 and Figure 7 for a breakdown of mean differences for each subject area).

Table 4. Citations at 36 months by subject area.
Category and interventionArticles, nCitations, mean (SD)Mean difference (95% CI)P value
Business, economics, and management3.95 (–0.66 to 8.56).24

Control20018.75 (20.45)


TrendMD20022.70 (26.14)

Chemical and materials sciences12.68 (–1.84 to 27.20).18

Control20060.46 (64.48)


TrendMD20073.14 (82.17)

Engineering and computer science8.61 (–3.07 to 20.29).40

Control20036.80 (46.04)


TrendMD20045.41 (70.31)

Health and medical sciences37.49 (–6.91 to 81.89).08

Control20092.43 (173.73)


TrendMD200129.92 (267.99)

Humanities, literature, and arts–0.35 (–3.73 to 4.43).89

Control20012.98 (23.63)


TrendMD20012.63 (17.40)

Life sciences and earth sciences13.51 (–0.91 to 27.93).03

Control20047.97 (66.14)


TrendMD20061.48 (79.95)

Physics and mathematics6.43 (–1.53 to 11.33).12

Control20014.46 (16.00)


TrendMD20020.89 (31.38)

Social sciences1.90 (–2.4 to 6.20).05

Control20016.39 (20.13)


TrendMD20018.29 (23.52)

Figure 7. Mean citation counts by subject area over 36 months: TrendMD versus control.
View this figure
Multivariate Regression Analysis

Overall, our multivariate regression models were a good fit to predict citations in all articles at 6, 12, and 36 months. The Kolmogorov-Smirnov test of the residuals (P=.21) were normal; values for skewness and kurtosis in the residuals were both less than 1 SE, which suggests that the residual values were not significantly different from the expected value of zero for a normal distribution. Our models predicted 41%, 43%, and 44% of the variation in citation counts at 6, 12, and 36 months, respectively. Clicks driven by TrendMD were found to be an independent predictor of citations; articles that received the greatest number of clicks from TrendMD had the largest citation advantage at 36 months (see Table 5 for the standardized beta coefficients for each covariate in the model).

Table 5. Multivariate regression model: standardized beta coefficients at 36, 12, and 6 months.a
Standardized beta coefficients36 months12 months6 months
Baseline citations0.2860.3290.323
Baseline Mendeley saves0.2090.2160.163
Journal Impact Factor0.2070.1750.222
Access typeNSb (0.664)cNS (0.847)NS (0.721)
TrendMD clicks0.2660.1820.153
TrendMD impressionsNS (0.342)0.0630.073

aAll other variables included in the regression model were significant at P<.001.

bNS: nonsignificant.

cSignificance values for nonsignificant variables excluded in the model are included in brackets.


To the best of our knowledge, this was the first randomized controlled trial to demonstrate how relatively brief postpublication promotion of peer-reviewed articles over a 6-month period can be used to persistently increase citation rates of those articles after 36 months. The overall citation advantage conferred by article promotion in the TrendMD Network was 28% relative to the control at 36 months, which dissipated from a 50% citation advantage at 12 months. Despite the overall citation advantage observed at 36 months, we did not find a statistically significant increase to citation counts within individual subject areas included in this study. There were, however, larger, albeit nonstatistically significant differences in citation counts measured in certain subjects versus others. For example, health and medical sciences articles saw larger citation increases than articles published in humanities, literature, and arts journals. One possible explanation for this could be because two-thirds of the TrendMD Network are STM journals; therefore, the effect size of promotion is likely to be larger for STM articles rather than humanities-related articles. The other possible explanation for the smaller citation advantages in some subject areas is because of differing publication cycle lengths, which have a direct effect on the speed in which citations accrue overtime. Articles in the fields of medicine, life sciences, and physics, for example, are typically published faster in comparison to articles in the humanities and social sciences [2].

Notwithstanding, our study was not powered to detect differences across the subject area–level comparisons; therefore, the negative findings across individual subject areas are likely due to the small sample sizes and resulting type II errors [45]. Given that we found an overall citation advantage at 36 months and found a Mendeley save advantage in 7 of the 8 subject areas at 6 months, it is reasonable to conclude that studies using larger sample sizes within subject areas over periods longer than 36 months may have shown an increase in citations. Future studies using larger sample sizes within subject areas over longer periods of time are needed to determine whether, and to what degree, promotion of articles in the TrendMD Network confers a citation advantage to individual subject areas.

We initially hypothesized that the overall 50% citation advantage observed at 12 months would increase over time due to the time it takes citations to accrue from manuscripts passing through peer review and onto publication [2]. However, our data suggests the citation advantage conferred to promoted articles dissipated between 12 and 36 months, which could be due to multiple factors affecting article citation rates, such as author credentials, subject matter, and cited references [36]. We do not know the precise timing of the peak of the citation advantage because we only measured citation counts at 6, 12, and 36 months. Future studies are warranted that measure citation rates at more frequent intervals to discern when the citation advantage conferred to promoted articles are at its maximum.

Based on these data, we can speculate on the mechanism in which TrendMD conferred a sustained citation advantage to promoted articles at 36 months. Readers clicking on promoted article links displayed in the widget that they may have not otherwise discovered, saved these articles to their Mendeley reference libraries, and cited these articles while creating their own scholarly work. This pathway is evidenced by the fact that page views (ie, clicks) driven by TrendMD were an independent predictor of both Mendeley saves at 6 months and citations at 6, 12, and 36 months (Table 5).

The findings of our study significantly add to the limited corpus of literature examining the efficacy of strategies to augment citation counts of peer-reviewed research. Though publishing research in OA journals is the most widely studied strategy to augment citation counts, the extent to which the OACA exists or is confounded by selection bias remains unknown [7,11,12,15,16]. In addition, even if the OACA did exist at some point in time in the past, the more that OA content becomes the standard, the less likely the strategy will yield citation benefits as the effect will be ubiquitous across all articles. Other strategies such as the promotion of peer-reviewed literature in social media platforms for the purpose of enhancing citation counts have similarly yielded inconclusive and, at times, conflicting results [21-23,25,26]. These data presented in our study address a pressing unmet need of authors, publishers, and funders for evidence-based strategies that can be used to enhance discoverability by the right targeted audience, which in turn augments the usage and citations counts of peer-reviewed content.

The research presented here has several strengths. First, the outcome of citation counts is unbiased and objective, increasing the reproducibility of the results. Second, we used a rigorous randomized controlled trial design, which minimizes the likelihood of bias and confounding. Third, our sample size was large, and our study was adequately powered for outcomes of differences in mean citations.

There are, however, several limitations to this research. First, all authors have a conflict of interest with the results presented as creators (PK and GE) or employees (TB) of TrendMD. Risk of bias, however, was mitigated by the randomized controlled trial design. Second, although our study was adequately powered to detect mean differences in citation counts across all articles at 12 months, our study was not adequately powered to detect mean differences in citation counts between disciplines. Third, our inclusion criteria of randomly selecting articles published in 8 out of the top 20 journals with the highest h-5 index in Google Scholar categories may make our results less generalizable to articles published in journals with lower Impact Factors. We attempted to mitigate this possible limitation by randomly selecting 8 of the top 20 journals in each subject. Future studies are needed to determine whether TrendMD still confers an enduring citation advantage to articles published in lower Impact Factor journals. Another limitation and question that stems from these data is whether longer periods of promotion beyond 6 months would yield larger and more enduring benefits to citation counts. Future studies using promotion periods longer than 6 months are needed to determine whether there is a dose-dependent relationship between the length of the article promotion period and the magnitude of citation advantage conferred. Our group has previously completed a study showing that there is indeed a dose-dependent relationship between the intensity of article promotion and page views [28]; however, the degree to which the increase in page views leads to a citation advantage is not known. It is also possible that a longer promotion period of articles may saturate over time as readers are presented with the same article links; future studies are needed to determine whether prolonged promotion of articles lead to diminishing rates of return for citations. Lastly, the number and type of publishers participating in the TrendMD Network may change over time, which may affect the reproducibility of our findings [35]. Past studies indicate that the efficacy of TrendMD promotion is dependent on the number and type of publishers participating in the Network [28,29]. In general, replicated data indicates that the more publishers across subject areas using TrendMD the greater the efficacy of the channel to confer benefits to article visibility and usage [35]. If publishers stopped using TrendMD, the channel is unlikely to be as effective at augmenting citation counts as described in this study. More generally, there are, of course, limitations to citation analyses; citations only reflect activity in academia, and the usefulness of citations as indicators varies greatly between fields [46].

Despite the limitations, this study demonstrates that the promotion of articles in a cross-publisher online distribution channel (TrendMD) over a 6-month period can be used to persistently increase citations of articles after 36 months. Though we did not find a statistically significant increase in citation counts within individual subject areas at 36 months, this was likely due to small sample sizes and insufficient power, which resulted in type II errors. The overall citation advantage conferred to promoted articles was observed at 6 months, appeared to peak at 12 months, and endured, albeit to a lower level relative to the control at 36 months.

Acknowledgments

This research was supported in part by two investment grants made by the Ontario Centres of Excellence (OCE), a Canadian government-funding program. The grants were OCE Market Readiness CC (Grant 22292) and OCE Market Readiness CB (Grant 23811). We also want to thank members of the TrendMD team for help with data acquisition.

Conflicts of Interest

PK is a cofounder and former Chief Executive Officer (CEO) of TrendMD Inc. TB is a full-time employee of TrendMD Inc. GE is a cofounder and former Chief Science Officer of TrendMD Inc. GE and PK currently hold no equity in TrendMD. GE is also founder and CEO of JMIR Publications, where TrendMD was initially tested and developed. The reviewers and editors of this paper were not aware of GE’s coauthorship during the peer-review process. JMIR Publications continues to offer TrendMD services to its authors and receives a commission.

This randomized study was not registered, explained by authors with the reason that the study does not require registration due to its nature. The editor granted an exception from ICMJE rules mandating prospective registration of randomized trials, because the study is a follow-up on an initial study involving peer-reviewed articles. It does not involve the randomization of people into groups and does not target any behavioural outcomes.

Multimedia Appendix 1

CONSORT non-eHealth checklist.

PDF File (Adobe PDF File), 65 KB

  1. Aksnes DW, Langfeldt L, Wouters P. Citations, Citation Indicators, and Research Quality: An Overview of Basic Concepts and Theories. SAGE Open 2019 Feb 07;9(1):215824401982957. [CrossRef]
  2. Johnson R, Watkinson A, Mabe M. The STM Report. White Paper Sep 24. Report No 1. STM. 2018 Jan 01.   URL: https://www.stm-assoc.org/2018_10_04_STM_Report_2018.pdf [accessed 2021-12-02]
  3. Tenopir C, Christian L, Kaufman J. Seeking, reading, and use of scholarly articles: an international study of perceptions and behavior of researchers. Publications 2019 Mar 06;7(1):18-25. [CrossRef]
  4. Van Noorden R. Scientists may be reaching a peak in reading habits. Nature 2014 Feb 3:15-17. [CrossRef]
  5. Van Noorden R. The science that's never been cited. Nature 2017 Dec 14;552(7684):162-164. [CrossRef] [Medline]
  6. Chu JSG, Evans JA. Slowed canonical progress in large fields of science. Proc Natl Acad Sci U S A 2021 Oct 12;118(41):e2021636118 [FREE Full text] [CrossRef] [Medline]
  7. Davis PM, Lewenstein BV, Simon DH, Booth JG, Connolly MJL. Open access publishing, article downloads, and citations: randomised controlled trial. BMJ 2008 Jul 31;337:a568 [FREE Full text] [CrossRef] [Medline]
  8. Eysenbach G. Citation advantage of open access articles. PLoS Biol 2006 May;4(5):e157 [FREE Full text] [CrossRef] [Medline]
  9. Eysenbach G. The open access advantage. J Med Internet Res 2006 May 15;8(2):e8 [FREE Full text] [CrossRef] [Medline]
  10. Björk BC, Solomon D. Open access versus subscription journals: a comparison of scientific impact. BMC Med 2012 Jul 17;10:73 [FREE Full text] [CrossRef] [Medline]
  11. Copiello S. The open access citation premium may depend on the openness and inclusiveness of the indexing database, but the relationship is controversial because it is ambiguous where the open access boundary lies. Scientometrics 2019 Oct 10;121(2):995-1018. [CrossRef]
  12. Basson I, Blanckenberg JP, Prozesky H. Do open access journal articles experience a citation advantage? Results and methodological reflections of an application of multiple measures to an analysis by WoS subject areas. Scientometrics 2020 Oct 17;126(1):459-484. [CrossRef]
  13. MacCallum CJ, Parthasarathy H. Open access increases citation rate. PLoS Biol 2006 May;4(5):e176 [FREE Full text] [CrossRef] [Medline]
  14. Ottaviani J. The post-embargo open access citation advantage: it exists (probably), its modest (usually), and the rich get richer (of course). PLoS One 2016;11(8):e0159614 [FREE Full text] [CrossRef] [Medline]
  15. Anderson R. Is rational discussion of open access possible? Insights 2014 Jul 7;27(2):171-180. [CrossRef]
  16. Davis PM. Open access, readership, citations: a randomized controlled trial of scientific journal publishing. FASEB J 2011 Jul;25(7):2129-2134. [CrossRef] [Medline]
  17. Craig I, Plume A, McVeigh M, Pringle J, Amin M. Do open access articles have greater citation impact? A critical review of the literature. J Informetrics 2007 Jul;1(3):239-248. [CrossRef]
  18. Kurtz MJ, Eichhorn G, Accomazzi A, Grant C, Demleitner M, Henneken E, et al. The effect of use and access on citations. Inf Processing Manage 2005 Dec;41(6):1395-1402. [CrossRef]
  19. Hawkins CM, Hunter M, Kolenic GE, Carlos RC. Social media and peer-reviewed medical journal readership: a randomized prospective controlled trial. J Am Coll Radiol 2017 May;14(5):596-602. [CrossRef] [Medline]
  20. Trueger NS, Bokarius AV, Carroll S, April MD, Thoma B. Impact of a physician-led social media sharing program on a medical journal's web traffic. J Am Coll Radiol 2018 Jan;15(1 Pt B):184-189. [CrossRef] [Medline]
  21. Luc JGY, Archer MA, Arora RC, Bender EM, Blitz A, Cooke DT, et al. Does tweeting improve citations? One-year results from the TSSMN prospective randomized trial. Ann Thorac Surg 2021 Jan;111(1):296-300. [CrossRef] [Medline]
  22. Davis P. Reanalysis of tweeting study yields no citation benefit. The Scholarly Kitchen.   URL: https://scholarlykitchen.sspnet.org/2020/07/13/tweeting-study-yields-no-benefit/ [accessed 2021-12-02]
  23. Davis P. Tweeting-citations authors speak, finally. The Scholarly Kitchen.   URL: https://scholarlykitchen.sspnet.org/2020/08/03/tweeting-citations-authors-speak-finally/ [accessed 2021-12-02]
  24. Luc JGY, Varghese TK, Antonoff MB. We stand by our data: a call for professional scholarly discourse. Ann Thorac Surg 2021 Mar;111(3):1095-1097. [CrossRef] [Medline]
  25. Fox CS, Bonaca MA, Ryan JJ, Massaro JM, Barry K, Loscalzo J. A randomized trial of social media from Circulation. Circulation 2015 Jan 06;131(1):28-33. [CrossRef] [Medline]
  26. Fox CS, Gurary EB, Ryan J, Bonaca M, Barry K, Loscalzo J, et al. Randomized controlled trial of social media: effect of increased intensity of the intervention. J Am Heart Assoc 2016 Apr 27;5(5):e003088 [FREE Full text] [CrossRef] [Medline]
  27. Kudlow P, Rutledge A, Eysenbach G. TrendMD: reaching larger, more targeted audiences by distributing scholarly content online. Editors Bull 2015 Jun 08;10(1):11-15. [CrossRef]
  28. Kudlow P, Rutledge A, Shachak A, McIntyre RS, Eysenbach G. TrendMD: helping scholarly content providers reach larger and more targeted audiences. Learned Publishing 2016 Feb 16;29(2):125-129. [CrossRef]
  29. Kudlow P, Cockerill M, Toccalino D, Dziadyk DB, Rutledge A, Shachak A, et al. Online distribution channel increases article usage on Mendeley: a randomized controlled trial. Scientometrics 2017;112(3):1537-1556 [FREE Full text] [CrossRef] [Medline]
  30. Aduku KJ, Thelwall M, Kousha K. Do Mendeley reader counts reflect the scholarly impact of conference papers? An investigation of computer science and engineering. Scientometrics 2017 Apr 13;112(1):573-581. [CrossRef]
  31. Li X, Thelwall M, Giustini D. Validating online reference managers for scholarly impact measurement. Scientometrics 2011 Dec 21;91(2):461-471. [CrossRef]
  32. Li X, Thelwall M. F1000, Mendeley and traditional bibliometric indicators. 2012 Presented at: Proceedings of the 17th International Conference of Bibliometrics; 2012; Valencia, Spain.
  33. Thelwall M, Nevill T. Could scientists use Altmetric.com scores to predict longer term citation counts? J Informetrics 2018 Feb 15;12(1):237-248. [CrossRef]
  34. Thelwall M, Sud P. Mendeley readership counts: an investigation of temporal and disciplinary differences. J Assoc Inf Sci Technol 2015 Jun 05;67(12):3036-3050. [CrossRef]
  35. Kudlow P, Bissky Dziadyk D, Rutledge A, Shachak A, Eysenbach G. The citation advantage of promoted articles in a cross‐publisher distribution platform: a 12‐month randomized controlled trial. J Assoc Inf Sci Technol 2019 Dec 23;71(10):1257-1274. [CrossRef]
  36. Onodera N, Yoshikane F. Factors affecting citation rates of research articles. J Assoc Inf Sci Technol 2014 May 21;66(4):739-764. [CrossRef]
  37. Top publications 2021. Google.   URL: https://scholar.google.ca/citations?view_op=top_venues&hl=en [accessed 2021-12-02]
  38. TrendMD.   URL: https://trendmd.com/ [accessed 2021-12-02]
  39. Outbrain.   URL: https://www.outbrain.com/ [accessed 2021-12-02]
  40. Taboola.   URL: https://www.taboola.com/ [accessed 2021-12-02]
  41. Cohen J. Statistical Power Analysis for the Behavioral Sciences. New York: Routledge; 1988.
  42. White H. A heteroskedasticity-consistent covariance matrix estimator and a direct test for heteroskedasticity. Econometrica 1980 May;48(4):817. [CrossRef]
  43. Chen S, Feng Z, Yi X. A general introduction to adjustment for multiple comparisons. J Thorac Dis 2017 Jun;9(6):1725-1729. [CrossRef] [Medline]
  44. de Boer MR, Waterlander WE, Kuijper LD, Steenhuis IH, Twisk JW. Testing for baseline differences in randomized controlled trials: an unhealthy research behavior that is hard to eradicate. Int J Behav Nutr Phys Act 2015 Jan 24;12:4 [FREE Full text] [CrossRef] [Medline]
  45. Freiman JA, Chalmers TC, Smith H, Kuebler RR. The importance of beta, the type II error and sample size in the design and interpretation of the randomized control trial. Survey of 71 "negative" trials. N Engl J Med 1978 Sep 28;299(13):690-694. [CrossRef] [Medline]
  46. MacRoberts MH, MacRoberts BR. Problems of citation analysis. Scientometrics 1996 Jul;36(3):435-444. [CrossRef]


API: application programming interface
CEO: Chief Executive Officer
CONSORT: Consolidated Standards of Reporting Trials
OA: open access
OACA: open access citation advantage
OCE: Ontario Centres of Excellence
STM: scientific, technical, and medical


Edited by R Kukafka; submitted 05.10.21; peer-reviewed by M Thelwall; comments to author 27.10.21; accepted 21.11.21; published 10.12.21

Copyright

©Paul Kudlow, Tashauna Brown, Gunther Eysenbach. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 10.12.2021.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.