Comparing Physician and Artificial Intelligence Chatbot Responses to Patient Questions Posted to a Public Social Media Forum

John W Ayers, Adam Poliak, Mark Dredze et al.

A recent study published in JAMA Internal Medicine compared the responses of an AI chatbot to those of verified physicians in addressing patient questions on Reddit's r/AskDocs forum. Evaluators, consisting of licensed healthcare professionals, found that chatbot responses were preferred 78.6% of the time and were rated significantly higher for quality and empathy than physician responses. The study suggests AI assistants may help healthcare professionals draft patient responses, potentially improving patient outcomes, reducing clinician burnout, and increasing access to high-quality healthcare.

AI Chatbot's Potential in Managing Patient Inquiries During COVID-19 

The AI chatbot's potential in managing the high volume of patient inquiries during the COVID-19 pandemic is noteworthy, and further exploration of this technology in clinical settings is warranted. Randomized trials could assess if AI assistants improve responses and patient outcomes while lowering clinician burnout.

Limitations of the Study on AI Chatbot's Effectiveness 

The study's limitations include its focus on a specific set of online forum questions rather than complex or nuanced health concerns. Further research is needed to evaluate chatbot effectiveness in different contexts and patient populations. While chatbots cannot replace healthcare professionals, they may provide a useful supplement in certain situations.

Conclusion on the Promise and Ethical Concerns of AI Assistants in Healthcare 

In conclusion, AI assistants show promise for improving clinician and patient outcomes in messaging workflows, although ethical concerns regarding accuracy must be addressed. Human review of AI-generated content remains necessary, and additional research is required to draw definitive conclusions about the benefits of AI assistants in healthcare.

Link to Article: Comparing Physician and Artificial Intelligence Chatbot Responses to Patient Questions Posted to a Public Social Media Forum | Health Informatics | JAMA Internal Medicine | JAMA Network