Evaluation of LLM-Generated Distractors of Multiple-Choice Questions for the Japanese National Nursing Examination

Abstract
This paper reports the evaluation results in the usefulness of distractors generated by large language models (LLMs) in creating multiple-choice questions for the Japanese National Nursing Examination. Our research questions are: “(RQ1) Do question writers adopt LLM-generated distractor candidates in question writing?” and “(RQ2) Does providing LLM-generated distractor candidates reduce the time for writing questions?”. We selected ten questions from the proprietary mockup examinations of the National Nursing Examination administered by a prep school, considering the analysis of the last ten-year questions of the National Nursing Examination. Distractors are generated by seven different LLMs, given a stem and a key for each question of the above ten, and they are compiled into the distractor candidate sets. Given a stem and a key for each question, 15 domain experts completed questions by filling in three distractors. Eight experts are provided with the LLM-generated distractor candidates; the other seven are not. The results of comparing the two groups provided us with affirmative answers to both RQs. The current evaluation remains subjective from the viewpoint of the question writers; it is necessary to evaluate whether questions generated with the assistance of LLM work in a real examination setting. Our future plan includes administering a large-scale mockup examination using both human-made and LLM-assisted questions and analysing the differences in the responses to both types of questions.

Recommended citation:
Yusei Kido, Hiroaki Yamada, Takenobu Tokunaga, Rika Kimura, Yuriko Miura, Yumi Sakyo, Naoko Hayashi. 2025. Evaluation of LLM-Generated Distractors of Multiple-Choice Questions for the Japanese National Nursing Examination. In the Proceedings of the 17th International Conference on Computer Supported Education (CSEDU 2025), pages 754-764.

Download paper here