Every spring, we use student usage data to comb through our library of questions, looking for those that students struggle with the most. While some questions are simply difficult—even deliberately so—others might need to be reworded for clarity, or to make the feedback more helpful.
One example is a money creation question using balance sheets. The data shows that 33% of the students who worked this question in 2015 gave up before arriving at the correct answer, which flagged it as needing review. We found that the feedback addressed the changes to reserves, loans, and deposits, but then included only one response for incorrect changes to all other accounts: “Not every account is affected by the deposit and loan.”
This is too vague to be helpful, so we added five new feedback responses, including:
“The entry for the bank's property is incorrect. The bank did not buy or sell any equipment or buildings.”
Questions with lots of answers, like this one with eight balance sheet entries, need to be very specific about the error in order for the student to learn from the mistake. Real student data is a powerful tool we use to improve our questions as well as students' learning experience.