Comparative benchmarking of three artificial intelligence chatbots (ChatGPT-5.1, Qwen- 3 Max, and Perplexity AI) on faculty validated undergraduate respiratory physiology multiple-choice examinations
DOI:
https://doi.org/10.62438/tunismed.v104i05.6620Keywords:
Machine Learning, Medical Education, Multiple-Choice Questions, PhysiologyAbstract
Background: Artificial intelligence (AI) chatbots are increasingly used by medical students for learning and examination preparation. However, their reliability in mechanistically demanding disciplines such as respiratory physiology remains insufficiently evaluated using authentic examination
material. This study aimed to assess and compare the performance of three AI chatbots on validated undergraduate respiratory physiology examination questions.
Methods: We conducted a cross-sectional analytical study including all respiratory physiology multiple-choice questions (MCQs), each comprising five propositions, used in official undergraduate examinations at the Faculty of Medicine of Tunis during the academic years 2023–2024 and 2024–2025
(101 questions). Three AI chatbots (ChatGPT-5.1, Qwen-3 Max, and Perplexity AI) were evaluated using standardized French-language prompts. Exact question-level concordance with the faculty-validated answer key, established by the teaching staff responsible for respiratory physiology examinations,
was compared using Cochran’s Q test and pairwise McNemar tests. At the proposition level, responses were analyzed as binary outcomes (true/false) to compute accuracy, sensitivity, and specificity.
Results: Question-level exact concordance differed significantly across models (Cochran’s Q = 10.18, p = 0.006). ChatGPT showed the highest concordance rate (79.2%), followed by Qwen (74.3%) and Perplexity (62.4%). At the proposition level (505 propositions), ChatGPT achieved the highest overall accuracy (94.1%) and sensitivity for true propositions (96.5%), whereas Qwen demonstrated the highest specificity for false propositions (94.2%).
Conclusion: Although all evaluated AI chatbots performed well on respiratory physiology MCQs, substantial variability was observed across models. These findings ..( abstract truncated at 250 words).
Downloads
References
Vrdoljak J, Boban Z, Vilović M, Kumrić M, Božić J. A Review of Large Language Models in Medical Education, Clinical Decision Support,
and Healthcare Administration. Healthcare. 2025;13(6):603. doi:10.3390/healthcare13060603
Shool S, Adimi S, Saboori Amleshi R, Bitaraf E, Golpira R, Tara M. A systematic review of large language model (LLM) evaluations in
clinical medicine. BMC Med Inform Decis Mak. 2025;25(1):117. doi:10.1186/s12911-025-02954-4
Lu Z, Peng Y, Cohen T, Ghassemi M, Weng C, Tian S. Large language models in biomedicine and health: current research landscape and
future directions. J Am Med Inform Assoc. 2024;31(9):1801-1811. doi:10.1093/jamia/ocae202
Mondal H, Karri JKK, Ramasubramanian S, Mondal S, Juhi A, Gupta P. A qualitative survey on perception of medical students on the
use of large language models for educational purposes. Adv Physiol Educ. 2025;49(1):27-36. doi:10.1152/advan.00088.2024
Elmoselhi A. Clinically oriented laboratory illuminates challenging respiratory concepts to the pre-clinical medical students. Medical
Teacher. 2012;34(3):258-258. doi:10.3109/01421591003719012
Crozier T. Respiratory Physiology and Terminology. In: Lapinsky SE, Plante LA, eds. Respiratory Disease in Pregnancy. 1st ed. Cambridge
University Press; 2020:1-8. doi:10.1017/9781108163705.001
Cheng HM, Hoe SZ. Students’ responses under “negative pressure” to respiratory questions at the 15th physiology quiz international
event: 100 medical school teams from 22 countries. BLDE Univ J Health Sci. 2017;2(2):105. doi:10.4103/bjhs.bjhs_27_17
Badenhorst E, Mamede S, Abrahams A, et al. First-year medical students’ naïve beliefs about respiratory physiology. Adv Physiol
Educ. 2016;40(3):342-348. doi:10.1152/advan.00193.2015
Bharatha A, Ojeh N, Rabbi AMF, et al. Comparing the Performance of ChatGPT-4 and Medical Students on MCQs at Varied Levels of
Bloom's Taxonomy. AMEP. 2024;15:393-400. doi:10.2147/AMEP. S457408
Sadeq MA, Ghorab RMF, Ashry MH, et al. AI chatbots show promise but limitations on UK medical exam questions: a comparative
performance study. Sci Rep. 2024;14(1):18859. doi:10.1038/s41598-024-68996-2
Zhang R, Cai Q, Sartori A, Gayed N, Collette H. Comparing Artificial Intelligence Large Language Models in Medical Training: A
Performance Analysis of ChatGPT and DeepSeek on United States Medical Licensing Examination (USMLE) Style Questions. Cureus.
;17(8):e90212. doi:10.7759/cureus.90212
Uppalapati VK, Nag DS. A Comparative Analysis of AI Models in Complex Medical Decision-Making Scenarios: Evaluating ChatGPT,
Claude AI, Bard, and Perplexity. Cureus. 2024;16(1):e52485. doi:10.7759/cureus.52485
Agarwal M, Goswami A, Sharma P. Evaluating ChatGPT-3.5 and Claude-2 in Answering and Explaining Conceptual Medical
Physiology Multiple-Choice Questions. Cureus. 2023;15(9):e46222. doi:10.7759/cureus.46222
Alshammari AF, Madfa AA, Anazi BA, Alenezi YE, Alkurdi KA. Comparison of accuracy and consistency of AI Language models
when answering standardised dental MCQs. BMC Med Educ. 2025;25(1):1507. doi:10.1186/s12909-025-07624-7
Bhayana R, Fawzy A, Deng Y, Bleakney RR, Krishna S. Retrieval- Augmented Generation for Large Language Models in Radiology:
Another Leap Forward in Board Examination Performance. Radiology. 2024;313(1):e241489. doi:10.1148/radiol.241489
Masanneck L, Epping PZ, Meuth SG, Pawlitzki M. Evaluating Web Retrieval–Assisted Large Language Models With and Without
Whitelisting for Evidence-Based Neurology: Comparative Study. J Med Internet Res. 2025;27:e79379. doi:10.2196/79379
Kim Y, Shin S, Yoo SH. Performance of large language models in non-English medical ethics-related multiple choice questions:
comparison of ChatGPT performance across versions and languages. BMC Med Ethics. 2025;26:168. doi:10.1186/s12910-
-01316-z
Perez-Toro PA, Dineley J, Iniesta R, et al. Exploring biases related to the use of large language models in a multilingual depression
corpus. Sci Rep. 2025;15(1):36197. doi:10.1038/s41598-025- 19980-x
Qin L, Chen Q, Zhou Y, et al. A survey of multilingual large language models. Patterns (N Y). 2025;6(1):101118. doi:10.1016/j.
patter.2024.101118
Soulage CO, Van Coppenolle F, Guebre-Egziabher F. The conversational AI “ChatGPT” outperforms medical students
on a physiology university examination. Adv Physiol Educ. 2024;48(4):677-684. doi:10.1152/advan.00181.2023
Chytas D, Noussios G, Salmas M, Vasiliadis AV, Troupis T. Investigation of Studies on ChatGPT’s Ability to Answer Anatomy
Questions: A Self-Evaluation by ChatGPT and Comparison with an Evaluation by Gemini. Cureus. 2025;17(8):e90572. doi:10.7759/cureus.90572
Blank IA. What are large language models supposed to model? Trends Cogn Sci. 2023;27(11):987-989. doi:10.1016/j.
tics.2023.08.006
Banerjee A, Ahmad A, Bhalla P, Goyal K. Assessing the Efficacy of ChatGPT in Solving Questions Based on the Core Concepts in
Physiology. Cureus. 2023;15(8):e43314. doi:10.7759/cureus.43314
Ullah E, Parwani A, Baig MM, Singh R. Challenges and barriers of using large language models (LLM) such as ChatGPT for diagnostic
medicine with a focus on digital pathology – a recent scoping review. Diagn Pathol. 2024;19:43. doi:10.1186/s13000-024-01464-7
Omar M, Sorin V, Collins JD, et al. Multi-model assurance analysis showing large language models are highly vulnerable to adversarial
hallucination attacks during clinical decision support. Commun Med (Lond). 2025;5:330. doi:10.1038/s43856-025-01021-3
Roustan D, Bastardot F. The Clinicians’ Guide to Large Language Models: A General Perspective With a Focus on Hallucinations.
Interact J Med Res. 2025;14:e59823. doi:10.2196/59823
Herrera-Poyatos D, Peláez-González C, Zuheros C, et al. An overview of model uncertainty and variability in LLM-based
sentiment analysis: challenges, mitigation strategies, and the role of explainability. Front Artif Intell. 2025;8:1609097. doi:10.3389/
frai.2025.1609097
Waldo J, Boussard S. GPTs and Hallucination: Why do large language models hallucinate? Queue. 2024;22(4):19-33.
doi:10.1145/3688007
Sallam M, Alasfoor IM, Khalid SW, et al. Chinese generative AI models (DeepSeek and Qwen) rival ChatGPT-4 in ophthalmology
queries with excellent performance in Arabic and English. Narra J. 2025;5(1):e2371. doi:10.52225/narra.v5i1.2371
Kalam KA, Masoud FD, Muntaser A, Ranga R, Geng X, Goyal M. ChatGPT as a Learning Tool for Medical Students: Results From a
Randomized Controlled Trial. Cureus. 17(6):e85767. doi:10.7759/ cureus.85767
Gravina AG, Pellegrino R, Palladino G, Imperio G, Ventura A, Federico A. Charting new AI education in gastroenterology: Cross-sectional
evaluation of ChatGPT and perplexity AI in medical residency exam. Dig Liver Dis. 2024;56(8):1304-1311. doi:10.1016/j.dld.2024.02.019
Chimenea A, García-Díaz L, Antiñolo G. Exploring the Potential of Artificial Intelligence Language Models in Obstetrics with a Focus
on Fetal Medicine: An Evaluation of the Perplexity AI Model. Fetal Diagn Ther. 2024;51(2):125-132. doi:10.1159/000535345
Zhu S, Hu W, Yang Z, Yan J, Zhang F. Qwen-2.5 Outperforms Other Large Language Models in the Chinese National Nursing Licensing
Examination: Retrospective Cross-Sectional Comparative Study. JMIR Med Inform. 2025;13:e63731. doi:10.2196/63731
Additional Files
Published
Issue
Section
Categories
License
Copyright (c) 2026 La Tunisie Médicale
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
