Évaluation comparative de trois chatbots d’intelligence artificielle (ChatGPT-5.1, Qwen- 3 Max et Perplexity AI) sur des questions à choix multiples validées par la faculté en physiologie respiratoire de premier cycle
DOI :
https://doi.org/10.62438/tunismed.v104i05.6620Mots-clés :
Apprentissage automatique, Enseignement médical, Physiologie, Questions à choix multiplesRésumé
Introduction : Les chatbots d’intelligence artificielle (IA) sont de plus en plus utilisés par les étudiants en médecine. Cependant, leur fiabilité dans la physiologie respiratoire reste insuffisamment évaluée. Notre objectif a été d’évaluer et de comparer les performances de trois chatbots d’IA sur des
questions d'examen en physiologie respiratoire.
Méthodes : Nous avons mené une étude transversale incluant les questions à choix multiples (QCM) de physiologie respiratoire, comportant chacune cinq propositions, utilisées lors des examens de premier cycle à la Faculté de Médecine de Tunis au cours des années universitaires 2023–2024 et
2024–2025 (101 questions). Trois chatbots d’IA (ChatGPT-5.1, Qwen-3 Max et Perplexity AI) ont été évalués. La concordance exacte, au niveau de chaque question, avec la correction validée par l’équipe enseignante responsable des examens de physiologie respiratoire a été comparée à l’aide du
test Q de Cochran et de tests de McNemar. Au niveau des propositions, les réponses ont été analysées comme des variables binaires (vrai/faux) afin de calculer l’exactitude, la sensibilité et la spécificité.
Résultats : La concordance au niveau des questions a différé significativement entre les modèles (Q=10,18; p=0,006). ChatGPT a présenté le taux de concordance le plus élevé (79,2%), suivi de Qwen (74,3%) et de Perplexity (62,4%). Au niveau des propositions (505 propositions), ChatGPT a obtenu la
meilleure exactitude globale (94,1%) ainsi que la sensibilité la plus élevée pour les propositions vraies (96,5%), tandis que Qwen a présenté la meilleure spécificité pour les propositions fausses (94,2%).
Conclusion : Une variabilité significative entre les modèles a été observée. ..( résumé tronqué à 250 mots).
Téléchargements
Références
Vrdoljak J, Boban Z, Vilović M, Kumrić M, Božić J. A Review of Large Language Models in Medical Education, Clinical Decision Support,
and Healthcare Administration. Healthcare. 2025;13(6):603. doi:10.3390/healthcare13060603
Shool S, Adimi S, Saboori Amleshi R, Bitaraf E, Golpira R, Tara M. A systematic review of large language model (LLM) evaluations in
clinical medicine. BMC Med Inform Decis Mak. 2025;25(1):117. doi:10.1186/s12911-025-02954-4
Lu Z, Peng Y, Cohen T, Ghassemi M, Weng C, Tian S. Large language models in biomedicine and health: current research landscape and
future directions. J Am Med Inform Assoc. 2024;31(9):1801-1811. doi:10.1093/jamia/ocae202
Mondal H, Karri JKK, Ramasubramanian S, Mondal S, Juhi A, Gupta P. A qualitative survey on perception of medical students on the
use of large language models for educational purposes. Adv Physiol Educ. 2025;49(1):27-36. doi:10.1152/advan.00088.2024
Elmoselhi A. Clinically oriented laboratory illuminates challenging respiratory concepts to the pre-clinical medical students. Medical
Teacher. 2012;34(3):258-258. doi:10.3109/01421591003719012
Crozier T. Respiratory Physiology and Terminology. In: Lapinsky SE, Plante LA, eds. Respiratory Disease in Pregnancy. 1st ed. Cambridge
University Press; 2020:1-8. doi:10.1017/9781108163705.001
Cheng HM, Hoe SZ. Students’ responses under “negative pressure” to respiratory questions at the 15th physiology quiz international
event: 100 medical school teams from 22 countries. BLDE Univ J Health Sci. 2017;2(2):105. doi:10.4103/bjhs.bjhs_27_17
Badenhorst E, Mamede S, Abrahams A, et al. First-year medical students’ naïve beliefs about respiratory physiology. Adv Physiol
Educ. 2016;40(3):342-348. doi:10.1152/advan.00193.2015
Bharatha A, Ojeh N, Rabbi AMF, et al. Comparing the Performance of ChatGPT-4 and Medical Students on MCQs at Varied Levels of
Bloom's Taxonomy. AMEP. 2024;15:393-400. doi:10.2147/AMEP. S457408
Sadeq MA, Ghorab RMF, Ashry MH, et al. AI chatbots show promise but limitations on UK medical exam questions: a comparative
performance study. Sci Rep. 2024;14(1):18859. doi:10.1038/s41598-024-68996-2
Zhang R, Cai Q, Sartori A, Gayed N, Collette H. Comparing Artificial Intelligence Large Language Models in Medical Training: A
Performance Analysis of ChatGPT and DeepSeek on United States Medical Licensing Examination (USMLE) Style Questions. Cureus.
;17(8):e90212. doi:10.7759/cureus.90212
Uppalapati VK, Nag DS. A Comparative Analysis of AI Models in Complex Medical Decision-Making Scenarios: Evaluating ChatGPT,
Claude AI, Bard, and Perplexity. Cureus. 2024;16(1):e52485. doi:10.7759/cureus.52485
Agarwal M, Goswami A, Sharma P. Evaluating ChatGPT-3.5 and Claude-2 in Answering and Explaining Conceptual Medical
Physiology Multiple-Choice Questions. Cureus. 2023;15(9):e46222. doi:10.7759/cureus.46222
Alshammari AF, Madfa AA, Anazi BA, Alenezi YE, Alkurdi KA. Comparison of accuracy and consistency of AI Language models
when answering standardised dental MCQs. BMC Med Educ. 2025;25(1):1507. doi:10.1186/s12909-025-07624-7
Bhayana R, Fawzy A, Deng Y, Bleakney RR, Krishna S. Retrieval- Augmented Generation for Large Language Models in Radiology:
Another Leap Forward in Board Examination Performance. Radiology. 2024;313(1):e241489. doi:10.1148/radiol.241489
Masanneck L, Epping PZ, Meuth SG, Pawlitzki M. Evaluating Web Retrieval–Assisted Large Language Models With and Without
Whitelisting for Evidence-Based Neurology: Comparative Study. J Med Internet Res. 2025;27:e79379. doi:10.2196/79379
Kim Y, Shin S, Yoo SH. Performance of large language models in non-English medical ethics-related multiple choice questions:
comparison of ChatGPT performance across versions and languages. BMC Med Ethics. 2025;26:168. doi:10.1186/s12910-
-01316-z
Perez-Toro PA, Dineley J, Iniesta R, et al. Exploring biases related to the use of large language models in a multilingual depression
corpus. Sci Rep. 2025;15(1):36197. doi:10.1038/s41598-025- 19980-x
Qin L, Chen Q, Zhou Y, et al. A survey of multilingual large language models. Patterns (N Y). 2025;6(1):101118. doi:10.1016/j.
patter.2024.101118
Soulage CO, Van Coppenolle F, Guebre-Egziabher F. The conversational AI “ChatGPT” outperforms medical students
on a physiology university examination. Adv Physiol Educ. 2024;48(4):677-684. doi:10.1152/advan.00181.2023
Chytas D, Noussios G, Salmas M, Vasiliadis AV, Troupis T. Investigation of Studies on ChatGPT’s Ability to Answer Anatomy
Questions: A Self-Evaluation by ChatGPT and Comparison with an Evaluation by Gemini. Cureus. 2025;17(8):e90572. doi:10.7759/cureus.90572
Blank IA. What are large language models supposed to model? Trends Cogn Sci. 2023;27(11):987-989. doi:10.1016/j.
tics.2023.08.006
Banerjee A, Ahmad A, Bhalla P, Goyal K. Assessing the Efficacy of ChatGPT in Solving Questions Based on the Core Concepts in
Physiology. Cureus. 2023;15(8):e43314. doi:10.7759/cureus.43314
Ullah E, Parwani A, Baig MM, Singh R. Challenges and barriers of using large language models (LLM) such as ChatGPT for diagnostic
medicine with a focus on digital pathology – a recent scoping review. Diagn Pathol. 2024;19:43. doi:10.1186/s13000-024-01464-7
Omar M, Sorin V, Collins JD, et al. Multi-model assurance analysis showing large language models are highly vulnerable to adversarial
hallucination attacks during clinical decision support. Commun Med (Lond). 2025;5:330. doi:10.1038/s43856-025-01021-3
Roustan D, Bastardot F. The Clinicians’ Guide to Large Language Models: A General Perspective With a Focus on Hallucinations.
Interact J Med Res. 2025;14:e59823. doi:10.2196/59823
Herrera-Poyatos D, Peláez-González C, Zuheros C, et al. An overview of model uncertainty and variability in LLM-based
sentiment analysis: challenges, mitigation strategies, and the role of explainability. Front Artif Intell. 2025;8:1609097. doi:10.3389/
frai.2025.1609097
Waldo J, Boussard S. GPTs and Hallucination: Why do large language models hallucinate? Queue. 2024;22(4):19-33.
doi:10.1145/3688007
Sallam M, Alasfoor IM, Khalid SW, et al. Chinese generative AI models (DeepSeek and Qwen) rival ChatGPT-4 in ophthalmology
queries with excellent performance in Arabic and English. Narra J. 2025;5(1):e2371. doi:10.52225/narra.v5i1.2371
Kalam KA, Masoud FD, Muntaser A, Ranga R, Geng X, Goyal M. ChatGPT as a Learning Tool for Medical Students: Results From a
Randomized Controlled Trial. Cureus. 17(6):e85767. doi:10.7759/ cureus.85767
Gravina AG, Pellegrino R, Palladino G, Imperio G, Ventura A, Federico A. Charting new AI education in gastroenterology: Cross-sectional
evaluation of ChatGPT and perplexity AI in medical residency exam. Dig Liver Dis. 2024;56(8):1304-1311. doi:10.1016/j.dld.2024.02.019
Chimenea A, García-Díaz L, Antiñolo G. Exploring the Potential of Artificial Intelligence Language Models in Obstetrics with a Focus
on Fetal Medicine: An Evaluation of the Perplexity AI Model. Fetal Diagn Ther. 2024;51(2):125-132. doi:10.1159/000535345
Zhu S, Hu W, Yang Z, Yan J, Zhang F. Qwen-2.5 Outperforms Other Large Language Models in the Chinese National Nursing Licensing
Examination: Retrospective Cross-Sectional Comparative Study. JMIR Med Inform. 2025;13:e63731. doi:10.2196/63731
Fichiers supplémentaires
Publiée
Numéro
Rubrique
Catégories
Licence
Merci de créditer les auteurs lors de toute citation : La Tunisie Médicale (2026)
Ce travail est disponible sous licence Creative Commons Attribution - Pas d'Utilisation Commerciale - Pas de Modification 4.0 International.
