Quality of Chatbot Responses to the Most Popular Questions Regarding Erectile Dysfunction

Objective: Erectile dysfunction (ED) is a common cause of male sexual dysfunction. We aimed to evaluate the quality of ChatGPT and Gemini’s responses to the most frequently asked questions about ED.

Methods: This study was conducted as a cross-sectional, observational study. Google Trends was used to determine the most frequently asked questions on the internet. ChatGPT-3.5 and Gemini were compared for these chatbots’ answers to the questions about ED. Two urologists with board certificates assessed the quality of responses using the Global Quality Score (GQS).

Results: Fifteen questions about ED were included according to the Google Trends. ChatGPT was able to answer all the questions systematically, whereas Gemini could not answer two questions. Upon assessing the quality of the responses provided by both researchers with the GQS, it was observed that the frequency of low-quality responses from Gemini exceeded that of ChatGPT. The agreement between researchers was 92% for ChatGPT and 95% for Gemini.

Conclusion: Despite the expeditious and comprehensive answers provided by chatbots, we identified inadequacies in their responses related to ED. In their current state, they cannot replace the patient-centered approach of healthcare professionals and require further development.

Cite this article as: Barlas İŞ, Tunç L. Quality of chatbot responses to the most popular questions regarding erectile dysfunction. Urol Res Pract. 2024;50(4):253-260.