According to ASCO Research AI-Generated Text in Scientific Abstracts is Increasing, and Originality.ai’s High Accuracy is Key to Preserving the Integrity of Scientific Literature

AI-generated content continues to become increasingly common in every sector, and scientific abstracts are no exception. In 2024, the American Society of Clinical Oncology (ASCO) noted a significant increase in the use of large language models (LLMs) for writing scientific abstracts in their study, “Characterizing the Increase in Artificial Intelligence Content Detection in Oncology Scientific Abstracts From 2021 to 2023.” 

The ASCO study evaluated the performance of three AI content detectors (Originality.ai, GPTZero, and Sapling) in identifying AI-generated content in scientific abstracts. The scientific abstracts were submitted to the ASCO Annual Meetings from 2021 to 2023.

Key Findings (TL;DR)

• Originality.ai accurately identified 96% of AI-generated abstracts by GPT-3.5 and GPT-4, with a sensitivity of over 95%.
• Originality.ai showed a moderate correlation (Spearman) with GPTZero (ρ = 0.284) and a lower correlation with Sapling (ρ = 0.143), indicating consistent performance.

Study Details

This study analyzed 15,553 oncology scientific abstracts from ASCO Annual Meetings between 2021 and 2023. AI-generated content in the abstracts increased significantly from 2021 to 2023. Logistic regression models were used to evaluate the association of predicted AI content with submission years and abstract characteristics. 

AI Text Detection Tools

• Three AI-Content Detection Tools: GPTZero, Originality.ai, and Sapling.

The table below clarifies why the researchers chose GPTZero, Originality.ai, and Sapling for this study and why they excluded other AI-generated text detection tools.

Dataset

• The dataset comprised 15,553 abstracts from ASCO Annual Meetings from 2021 to 2023, accessed through ASCO’s Data Library. 
• Key characteristics were tabulated for each abstract. The characteristics included abstract track, venue of presentation, inclusion of clinical trial number, as well as the countries and regions of the institutions the first author was affiliated with.

The table below shows characteristics of ASCO Annual Meeting abstracts and authors, from 2021 to 2023.

Evaluation Criteria

• AUROC (Area Under Receiver Operating Curve), AUPRC (Area Under Precision-Recall Curve) — for accuracy.
• Brier Score — for evaluating prediction error.
• Logistic Regression — for analyzing the association of AI content with abstract characteristics.
• Spearman Correlation — for comparing predictions between different detectors.

Originality.ai’s AI Detector Results

Finding 1: Perfect AUROC score of 1.00 for GPT-3.5 and nearly perfect for GPT-4

• AUROC for GPT-3.5 vs. Human: Perfect scores of 1.000 for all years

• AUROC for GPT-4 vs Human: Slight improvement over the years, reaching up to 0.997.

• AUROC for Mixed GPT-3.5 vs Human: 0.782 in 2021, improving slightly to 0.788 in 2023.

• AUROC for Mixed GPT-4 vs Human: 0.706 in 2021, improving to 0.715 in 2023.

Finding 2: High AUPRC to differentiate between AI-generated and human-written abstracts

• AUPRC for GPT-3.5 vs. Human: High performance, showing slight improvement over the years.

• AUPRC for GPT-4 vs Human: Strong performance with a slight improvement over the years.

• AUPRC for Mixed GPT-3.5 vs. Human: High performance, improving steadily.

• AUPRC for Mixed GPT-4 vs Human: Strong performance, improving over the years.

Finding 3: Low to Moderate Spearman Correlation with other detectors ensuring consistent performance

• Moderate correlation with GPTZero (ρ = 0.284) 
• Lower correlation with Sapling (ρ = 0.143)

Finding 4: Ranked second with a lower Brier Score, indicating accurate predictions with minimal error, suggesting a low rate of false positives

• Brier Score for GPT-3.5 vs. Human: Low scores, around 0.013 to 0.015.

• Brier Score for GPT-4 vs. Human: Improvement from 0.027 in 2021 to 0.025 in 2023.

• Brier Score for Mixed GPT-3.5 vs. Human: Around 0.400, showing high prediction error.

• Brier Score for Mixed GPT-4 vs. Human: Around 0.426, showing high prediction error.

Final Thoughts

Originality.ai excels in detecting AI-generated content in scientific abstracts. Its high accuracy, low false-positive rate, and adaptability to different abstract characteristics make it a critical tool for researchers, publishers, and academic institutions committed to preserving the integrity of scientific literature. AI-generated text detection tools are particularly important for maintaining trust in scientific research and publications.