Et Machina: Exploring the Use of Conversational Agents Such as ChatGPT in Scientific Writing

Abstract

Background: Advancements in natural language processing have led to the development of powerful language models like ChatGPT, capable of generating coherent text with minimal human input. While the potential for using ChatGPT in scientific paper writing is of growing interest, its full potential and usefulness remain unexplored. Method: We investigated the use of ChatGPT to write scientific papers on the topic of conversational agents in scientific writing. ChatGPT served as a tool to facilitate scientific writing, and we evaluated the quality, factuality, and coherence of the generated text. In this paper, we include information about the generation, processing, or adjustments made to the text in differently-colored blocks.∗ Position: We argue that large-scale trained conversational agents like ChatGPT can generate concise, grammatically correct, and well-structured text. However, occasional errors in factual accuracy, clarity, and scientific rigor were observed. We acknowledge the potential of ChatGPT and similar models as tools for generating initial drafts of scientific work. Conclusion: ChatGPT demonstrates promise as a scientific writing tool, but unsupervised use in this context is not yet recommended. Further research is necessary to enhance the accuracy and scientific validity of the generated text. Researchers should exercise caution and view ChatGPT as a complement to human writing rather than a replacement. We provide key recommendations for using conversational agents in scientific writing, including human oversight and training the model on scientific literature and unbiased data. By utilizing AI in the authorship process, we present an example of how AI involvement can be reported.