Comparisons of Quality, Correctness, and Similarity Between ChatGPT-Generated and Human-Written Abstracts for Basic Research: Cross-Sectional Study

Introduction

ChatGPT is an advanced language model that marks a significant milestone in the development of conversational artificial intelligence (AI) systems, highlighting its contribution to the field of natural language processing. In scientific writing, ChatGPT may provide valuable support for drafting manuscripts, summarizing articles, translating languages, and refining text structures or wording [1]. Although the use of ChatGPT as a research assistant for scientific paper composition and publication has gained significant attention, several concerns have emerged. First, AI models trained on vast amounts of text data may inadvertently generate content that closely resembles existing scientific work, leading to instances of plagiarism [1]. Second, AI models can generate seemingly plausible content that lacks accuracy or may unintentionally introduce errors. Moreover, as these models are trained on data sets that may contain biases, they can inadvertently amplify such biases without proper supervision by domain experts. This raises concerns about the trustworthiness and reliability of the generated scientific papers. Third, as AI capabilities progress, there is concern that AI-generated scientific papers could deceive reviewers and educators, potentially resulting in unwarranted acceptance or recognition [2].

A recent study investigated the capabilities of AI language models in generating scientific abstracts. In this study, we provided the “title” and “journal style” to ChatGPT and prompted it to generate a scientific abstract [3]. We found that the abstracts generated by ChatGPT exhibited a high level of fluency and successfully deceived human reviewers. However, these abstracts contained fabricated data and high percentages of plagiarism and AI-generated content. Another study provided the full text of psychiatric papers (excluding the original abstract) to an AI model and prompted it to generate an abstract with the same style (structured or unstructured) as the original paper [4]. Notably, the structured abstracts generated by ChatGPT exhibited comparable quality to real abstracts, whereas the unstructured abstracts generated by ChatGPT displayed lower quality than the real abstracts [4]. Plagiarism was low in the AI-generated abstracts (7.55%). However, 30% (6/20) of the conclusions drafted by AI were incorrect [4]. Collectively, the observed disparities in the plagiarism percentages between these two studies can be attributed to the distinct prompt instructions used. Additionally, both studies used clinical medical research articles as the basis for generating scientific abstracts using AI models.

In this study, we aimed to assess the applicability of an AI model in generating abstracts for basic preclinical research. Drafting abstracts of basic studies may be more challenging than those of clinical studies because basic research delves into fundamental scientific principles, mechanisms, and theoretical concepts that can be highly complex and specialized. Basic research papers primarily target experts, scientists, and researchers with a high level of domain-specific knowledge. To date, no study has investigated the quality of abstracts generated by ChatGPT using full-text prompts. We hypothesized that the quality of abstracts generated by ChatGPT would differ from the quality observed in real abstracts.

Methods

Overview

We evaluated the quality and accuracy of the abstracts of basic research papers generated by ChatGPT. Basic research abstracts can be classified as unstructured or structured, and structured abstracts may consist of 3 or 4 subheadings. We selected 10 papers published in Nature, 10 in Genome Biology, and 10 in Biological Psychiatry, each. The abstract formats in Nature, Genome Biology, and Biological Psychiatry were unstructured, structured with 3 subheadings, and structured with 4 subheadings.

Generation of Abstract

We used ChatPDF [5], which is based on ChatGPT 3.5, to efficiently analyze the PDF file content. ChatPDF offers users a summary and answers questions regarding PDF files without nonexistent or self-created content, which is a major concern in ChatGPT [6]. We provided ChatPDF with full texts of the selected papers after excluding abstracts. The prompts provided to the ChatPDF were the following: “Please summarize a 200-word abstract” for papers published in Nature; “Please summarize a 250-word abstract including 3 paragraphs, namely ‘Background,’ ‘Results,’ and ‘Conclusions,’ for papers published in Genome Biology, and “Please summarize a 250-word abstract including 4 paragraphs, namely ‘Background,’ ‘Methods,’ ‘Results,’ and ‘Conclusions,’ for papers published in Biological Psychiatry.

Similarity, AI Content, and Subheadings

We evaluated similarity using a plagiarism comparison platform [7], which compares the generated and original abstracts for duplicate content. We also examined the AI output detectors for both generated and original abstracts using GPTZero [8]. GPTZero determined the content either as “entirely by AI,” “parts by AI,” or “entirely by a human” in each original and generated abstract. GPTZero also uses 2 measures to determine whether a text has been written by a human: perplexity and burstiness. Perplexity is a measure that assesses how well a language model predicts a text sample. A lower perplexity suggests that the text is easy to predict, which may indicate that it sounds more like machine-generated text. Burstiness is the occurrence of uncommon items appearing in random clusters over time (ie, creative variability), which is based on the idea that humans tend to mix long and short sentences, whereas AI sentences are uniform. Therefore, low burstiness suggests machine-generated text. We also compared the word counts in each subheading.

Experts and Quality Evaluation

A total of 8 experts in scientific writing and publishing evaluated the quality of the abstracts after reading the full texts of the research papers. The quality score was assigned using a Likert scale ranging from 0 to 10 (worst=0, not bad=5, extremely good=10) while considering the concise, precise, functional, unbiased, comprehensive, and self-sufficient aspects of the abstracts. In addition, experts were asked to identify abstracts written by human authors. Finally, the experts validated the conclusions generated by ChatPDF during the unblinding phase.

Statistical Analyses

Statistical analyses were performed using the SPSS (version 26; SPSS Inc). Scatter plots were generated using GraphPad Prism (version 8; GraphPad Software Inc). The pie charts were generated using Microsoft Excel 365. Box and distribution plots were generated in R using the ggplot2 package.

Group differences were analyzed using 2-sample t tests (for continuous variables) and chi-square tests (for categorical variables). If more than 20% of the cells expected cell counts of less than 5, the Fisher exact test was conducted for categorical variables. Correlations between 2 variables were examined using the Pearson correlation coefficient. Linear regression analysis was performed to determine the predictors of the scores. We considered the following predictors: the h-index and research year of the expert (continuous variables), the format of the abstract (categorical variable), the publication date of the paper (continuous variable), and whether the paper was open access (categorical variable). All t test analyses were 2-tailed, and P<.05 was considered significant.

Results

Characteristics of the Experts and Included Articles

The h-indices of the 8 experts ranged from 25 to 56, and their research experience ranged from 16 to 29 years (Table S1 in Multimedia Appendix 1). The 30 selected articles were published between the years 2022 and 2023. The average word count of the abstracts (Table S2 in Multimedia Appendix 1) was 240.30 (SD 21.37) for Biological Psychiatry, 212.30 (SD 20.66) for Genome Biology, and 142.90 (SD 11.43) for Nature. The full texts of the generated abstracts are listed in Table S3 in Multimedia Appendix 1. The visual representation of the operational process is presented in Figure 1. The details of the operational processes are presented in Multimedia Appendix 1.

‎

Similarity

The mean similarity (duplicate content) between the original and the generated abstracts was 3.20% (SD 1.83%) for unstructured abstracts, 2.10% (SD 2.17%) for structured abstracts with 3 subheadings, and 4.40% (SD 4.27%) for structured abstracts with 4 subheadings (Figure 2).

‎

AI Content

In the AI content analysis, the original abstracts had significantly higher perplexity scores than generated abstracts (structured abstracts with 3 subheadings: mean 110.72, SD 29.97 vs mean 59.49, SD 10.01; P<.001; structured abstracts with 4 subheadings: mean 105.17, SD 36.45 vs mean 60.58, SD 19.58; P=.002; unstructured abstracts: mean 165.57, SD 55.89 vs 47.70, SD 18.15; P=.002; Figure3A and Tables S4 and S5 in Multimedia Appendix 1). The original abstracts also had significantly higher burstiness scores compared to the generated abstracts (structured abstracts with 3 subheadings: mean 83.14, SD 17.15 vs mean 64.31, SD 8.24; P=.009; structured abstracts with 4 subheadings: mean 94.95, SD 38.38 vs mean 70.36, SD 31.48; P=.015; unstructured abstracts: mean 271.13, SD 459.07 vs mean 24.90, SD 15.49; P<.001; Figure 3B and Table S4 and S5 in Multimedia Appendix 1). Among the 30 AI-generated abstracts, 20 were judged as entirely human-generated content, 4 as part of the AI content, and 6 as entirely AI-generated content (Figure 3C and Table S4).

‎

Quality, Correlation, and Predictors

The original abstracts were assessed to have significantly higher quality than AI-generated abstracts (mean 8.09, SD 1.03 vs mean 4.72, SD 2.09; Figure 4D and Table 1). Comparison of various formats is as follows: unstructured abstracts (8.44, SD 0.69 vs mean 4.11, SD 1.98; Figure 4C), structured abstracts with 3 subheadings (mean 8.04, SD 1.06 vs mean 4.60, SD 2.33; Figure 4B), and structured abstracts with 4 subheadings (mean 7.79, SD 1.18 vs mean 5.46, SD 1.72; Figure 4A). The quality score was negatively correlated with the original and AI-generated abstracts in structured abstracts with 3 subheadings (coefficient –0.37; P<.01) and structured abstracts with 4 subheadings (coefficient –0.32; P<.01) but not in unstructured abstracts (P=.08; Table S6 in Multimedia Appendix 1). Table S7 in Multimedia Appendix 1 lists the predictors of abstract quality scores. A higher h-index of blinded raters (P<.001), more senior raters (P=.04), and structures with 4 subheadings (P=.03) were positively correlated with the quality score of AI-generated abstracts, while unstructured abstracts were negatively correlated (P=.003). Regarding human-written abstracts, a higher h-index of blinded raters (P<.001) and unstructured abstracts (P=.002) were positive predictors, whereas senior raters and those abstracts structured with 4 subheadings were negative predictors.

‎

Word Counts in Each Subheading

In structured abstracts with 4 subheadings, AI-generated abstracts had more words in the “conclusions” section compared to abstracts written by humans (71.7 vs 40.9 words; P=.002), but fewer words were used in the “results” section (58.8 vs 85.2 words; P=.001; Table 2). In structured abstracts with 3 subheadings, AI generated more words in the “conclusions” section compared to humans (92.8 vs 40.9 words; P<.001), but it generated fewer words in the “results” section (66.2 vs 135.6 words; P<.001; Table 2).

Accuracy of Authors’ Judgement and Correctness of Conclusions

Table 3 shows the accuracy of the judgments of the 8 experts. The judgment accuracies were 83.75% (67/80) for structured abstracts with 4 subheadings, 96.28% (77/80) for structured abstracts with 3 subheadings, and 100% (80/80) for unstructured abstracts. Finally, among the 30 ChatGPT-generated abstracts, 3 showed wrong conclusions.

Discussion

Principal Findings

In this study, we prompted ChatGPT with full texts of basic research papers and requested that abstracts be generated according to the style guidelines of the journals. The main findings are as follows: first, the similarity levels of the AI-generated abstracts were not particularly high (less than 20%) when using full texts as prompts. Second, in AI content analysis, the AI content detector tool can only identify one-third of the AI-generated abstracts. However, when compared to the original abstracts, AI-generated abstracts were rated as having lower scores for perplexity and burstiness, indicating that the difference between AI and human writing could still be detected. Third, in terms of quality rating, the AI-generated abstracts scored significantly lower than the original abstracts, and the difference was substantial (3.36 points on a 10-point rating scale). Fourth, the AI and human writers demonstrated different tendencies in the allocation of word counts to subheadings. Human writers tended to use more words in the results section, whereas AI writers tended to allocate a larger proportion of words to the conclusion section. Fifth, the identification accuracy between AI-generated and human-written abstracts was high (93%). In particular, 100% of the accuracy was achieved using the unstructured style. Finally, 3 of the 30 AI-generated abstracts yielded incorrect conclusions.

AI-generated content may exhibit high perplexity and burstiness scores. Perplexity and burstiness are metrics related to randomness and chaos, respectively. Perplexity measures the level of randomness or complexity in word usage, whereas burstiness quantifies variability in sentence length, structure, and tempos [9]. In natural language processing, AI models tend to write with a consistent tempo, resulting in low perplexity and burstiness, whereas human writers often exhibit bursts and lulls in their writing styles [9]. Although the AI content detector we used did not demonstrate sufficient accuracy in detecting AI-generated content, we still observed a significant difference in perplexity and burstiness between the human-written and AI-generated abstracts. It has been suggested that an AI output detector is better at distinguishing between original and ChatGPT-generated articles than a plagiarism detector [10].

In terms of quality assessment, our findings revealed that ChatGPT performed best in generating structured abstracts with 4 subheadings, followed by structured abstracts with 3 subheadings and unstructured abstracts. It appears that AI could generate higher-quality content when provided with more instructions in the form of subheadings, particularly when the word count per subheading is limited, as per our prompts. Notably, in unstructured abstracts, AI tended to focus more on introducing the study and its conclusion, while allocating less attention to the methods and results sections, which are crucial components presenting the core knowledge of the study. This observation is consistent with the distribution of words in structured abstracts. Human writers tended to allocate more words to the results section, whereas AI emphasized the conclusion section. This phenomenon contributes to the shallowness of the content generated by AI. The finding of higher-quality structured abstracts than unstructured abstracts generated by AI aligns with that of our previous study [4]. However, the difference in quality was more pronounced in this study involving preclinical research papers compared to clinical papers used in our previous study. ChatGPT appears to face greater challenges in interpreting professional preclinical research papers, resulting in lower quality and shallower abstracts for this specific domain. Nonetheless, a previous study [11] successfully used ChatGPT to generate highly convincing fraudulent scientific papers on neurosurgery, accomplished by fabricated data and tables using careful and step-by-step prompts over a duration of approximately 1 hour. Although some mistakes were identified in the generated articles, this finding suggests that AI-generated content can be concise and intricately crafted using multiple instructions (prompts) or training.

In contrast to previous studies [3,4], the accuracy of identifying AI-generated abstracts was only 68% in Gao et al [3] and 78% in our previous study [4]. However, the accuracy of identification by the 8 experts was notably high, reaching 93% for all abstracts and 100% for unstructured abstracts. The challenges associated with comprehending the intricate knowledge of preclinical research, which often leads to lower-quality outputs, contribute to experts’ ability to differentiate AI-generated content. The accuracy of identification demonstrated a correlation with the quality score—a higher identification rate was associated with lower quality. Notably, experts were able to identify 100% of the AI-generated unstructured abstracts, which scored lower in quality compared to structured abstracts. Finally, not all abstracts generated by AI showed correct conclusions.

Study Limitations

This study has several limitations. First, we used ChatPDF, a tool based on the ChatGPT 3.5. Therefore, the generalizability of our findings to other AI language models may be limited. Our results should be interpreted cautiously in the context of the current version of ChatPDF. Second, we provided only a single prompt for the AI model without any training. The results may differ when multiple prompts are used or when training procedures are incorporated. Third, we used a limited selection of AI content detectors. The applicability of our AI content detection results could vary if other tools were used. Finally, our study focused on generating scientific abstracts by prompting an AI model with scientific articles. However, there are numerous other applications in scientific research. Further investigation is necessary to evaluate the potential of AI in scientific writing.

Conclusions

In conclusion, AI-generated content can be identified by both human experts and computer-based analyses. AI may be capable of deceiving human experts in common or less specialized subjects. However, the limitations of AI have become more apparent in professional domains. AI-driven output detectors based on linguistic analyses are promising in computer science. Nevertheless, we anticipate ongoing competition between AI output detectors and writers.