Performance of chatbots in queries concerning fundamental concepts in photochemistry

Masahiko Taniguchi,Jonathan S. Lindsey

DOI: https://doi.org/10.1111/php.14037

2024-11-06

Photochemistry and Photobiology

Abstract:The advent of chatbots raises the possibility of a paradigm shift in many disciplines of scientific research. Here, 13 photochemically relevant queries were posed to five chatbots. The queries include fundamental concepts, practical or philosophical matters, and properties of dyes. The chatbot responses ranged from moderately effective to glaringly deficient, with instances of a correct response embedded in scientific nonsense or even entirely meaningless responses. Chatbots are complementary to search engines and may comprise a useful tool for scientific research in the hands of a domain expert. The unreliable accuracy makes present chatbots unsuited for unguided educational purposes in photochemistry. The advent of chatbots raises the possibility of a paradigm shift across society including the most technical of fields with regard to access to information, generation of knowledge, and dissemination of education and training. Photochemistry is a scientific endeavor with roots in chemistry and physics and branches that encompass diverse disciplines ranging from astronomy to zoology. Here, five chatbots have each been challenged with 13 photochemically relevant queries. The chatbots included ChatGPT 3.5, ChatGPT 4.0, Copilot, Gemini Advanced, and Meta AI. The queries encompassed fundamental concepts (e.g., "Why is the fluorescence spectrum typically the mirror image of the absorption spectrum?"), practical matters (e.g., "What is the inner filter effect and how to avoid it?"), philosophical matters ("Please create the most important photochemistry questions."), and specific molecular features (e.g., "Why are azo dyes non‐fluorescent?"). The chatbots were moderately effective in answering queries concerning fundamental concepts in photochemistry but were glaringly deficient in specialized queries for dyes and fluorophores. In some instances, a correct response was embedded in verbose scientific nonsense whereas in others the entire response, while grammatically correct, was utterly meaningless. The unreliable accuracy makes present chatbots poorly suited for unaided educational purposes and highlights the importance of domain experts.

biochemistry & molecular biology,biophysics