The simple way
- Log in to GPT UiO.
- Upload your materials and ask: “Can you provide 10 high-quality MCQs based on these materials?”
- You’re done.
Caveat: Sometimes asks about the examples in the materials such as details of datasets or names of references. (E.g.: "What is the required structure of the 'desired dataset' for Component I?")
My workflow for crafting exams
The links lead to GPT UiO AI assistants with pre-prompts. Click on it - and try!
STEP 1: Extract main concepts from course materials (Concept extractor assistant)
This tries to extract main concepts from the course materials. GPT UiO cannot read PowerPoint directly, so the best way is to convert them to PDFs first. More material is typically better. STEP 1 produces a list of themes and topics. This output is then further processed in STEP 2.
STEP 2: Use these main concepts to generate MCQs (MCQ builder assistant)
You will get asked some questions, number of exam items, if all materials are complete. Then it will generate a blueprint (number of questions per topic area) and prodice items. In this step, I found it useful to generate more response options than needed, these will be trimmed in STEP 3. Again, be ready to copy the final output of this interaction.
Some variations:
- VARIANT: Questions that focus on definition of concepts only
- VARIANT: A MED focus version (USMLE reference)
STEP 3: Quality check MCQ options (Option-checker assistant)
Here the options get "trimmed", but it is still important to check.
VARIANT: Re-builds items to include more applied contexts
Choice of models
GPT UiO lets you choose between OpenAI's ChatGPT and a series of open source models. Mistral is a French AI company, for example. Different models will give different outputs, have different strengths and weaknesses. So worth to try what works best for your materials!
Recommended materials
- Determining the Quality of Assessment Items in Collaborations
- Haladyna, T. M., Downing, S. M., & Rodriguez, M. C. (2002). A Review of Multiple-Choice Item-Writing Guidelines for Classroom Assessment. Applied Measurement in Education, 15(3), 309–333. https://doi.org/10.1207/S15324818AME1503_5
The bare minimum (again)
Check the options - the most common issue is that the correct answer is always the longest. Check this carefully!
Other considerations
- Consider whom you are assessing and what level they are at. Does the exam assess recall of facts and definitions, or application, reasoning, and transfer?
- For the latter, questions should include a vignette, case, or applied scenario.
- Place complexity in the stem or vignette; keep the options generally short.
- Avoid negation (terms such as “NOT”; “Which is least likely…?” can work if carefully written).
Post-exam analyses and passing standard
- In ATLAS we develop an platform for simple psychometric analyses
- ATLAS will also support individualized feedback to students after the exam
- What is the passing score, do you make pass/fail decisions or do you give grades?