Skip navigation
All Places > Macmillan Learning Digital > Blog > 2016 > July

Keeping Students Engaged:

A Tale from Introductory Chemistry

(Part II)


By Kevin Revell, Ph.D.

Associate Professor of Chemistry and Consultant · Sapling Learning



I’ve been experimenting with differentiated learning; that is, using different techniques to connect with students’ unique learning styles. As I mentioned in Part I, I’ve found using Sapling Learning’s online homework to be one of the most effective tools for student engagement, and it is fairly common for me to see >90% of my students doing their homework assignments. In this post, I would like to mention another technique which I’ve found helpful:  lecture recording.


I first tried lecture recording with my organic chemistry class in the fall of 2012. Our school has a subscription to Tegrity (other products like Echo360 are also available), which enables the professor to record a voice-over of the screen projection. I use a tablet PC in class and provide skeletal slides beforehand. During class, I would work through the slides, and record the conversation. Because it was recorded in a live class and unedited, the audio was rough in parts, but if the students missed a topic, they could go back and listen later.  


A few weeks into testing this out, our instructional technology coordinator let me know that my recordings had been watched 110 times! This was encouraging, so I continued it the following semester, in my large, introductory chemistry class. What I found was really interesting: while only 40% of my introductory chemistry students watched the lecture replays even once, about 10% of my students watched them voraciously. By the end of the semester, I had multiple students who had watched over twenty hours of class recordings. These results were included in paper published in J. Chem. Educ., available here.


I found that many of the heaviest users were international/ESL students. Through the semester, several of these students told me how helpful this was for them, since they sometimes struggled to catch the subtleties of what I said in class. In fact, by mid-semester, if I said anything in class without turning the recording on, anxious hands would go up, reminding me to hit “start”.


In the two semesters since, I’ve transitioned to recording content outside of class, and using it in either a flipped format, or simply having the lecture material available online for review. I increasingly find that students love the video format - it seems to be the preferred learning style for many in this generation.


What about the correlation between viewing time and class performance? From what I’ve seen, the highest performing students don’t watch the recorded content as much, perhaps because they get it in class the first time. I think that completing homework correlates more closely with performance. Still, I believe that recorded lecture content can go a long ways toward supporting struggling students by helping them catch up on things they may have missed. And the more ways we can help our students learn, the better.


Do you use online homework or lecture capture in your flipped or hybrid classroom? Tell us how these techniques have impacted engagement among your students in the comments below.

Suppose your textbook asks the following question.

What happens in the market for oranges when there is a freeze in Florida?

This question could, and most likely does, exist in question banks written for every principles of economics text in North America. However, not all questions work with all textbooks. Set two texts side by side, and you'll quickly see that the differences run deeper than just the order in which topics are covered. Those differences are exactly what we look for when we create our templates.


So, how do we go about creating a template for Sapling Learning courses? Very carefully, going chapter-by-chapter. It can take between 20 and 80 hours to create a template, depending on how different the textbook is from its previous edition or market leaders.


Step 1: Read the Chapter

As we read textbooks, we pay close attention to what makes that particular text different from other texts for the same course.


What topics are covered?

Authors pick and choose from an incredible number of potential topics when writing a text, and it is important that every facet of every question is actually covered in the text at hand. Questions covering information the student cannot find in the text are frustrating for both the student and the professor who must deal with the complaints.


What order are the topics covered in?

Some topic sequences are can be very standardized, such as the developmental stages of an insect. But others can be presented in any number of sequences or groupings. Therefore, all the ideas or data in each question must have been presented by the time the student arrives at that particular question. Asking about a concept using background knowledge from chapter 15 in the homework for chapter 8 can be frustrating for the student.


What vocabulary is used?

The names and symbols in the periodic table of elements are the same in every chemistry text, but other fields are not so highly standardized. Those who know the synonyms and are comfortable using terms interchangeably might scarcely notice when an author uses one or the other, but less well-informed students will be confused by the use of different terms for the same concepts.


Step 2: Search the Library

Next, we scour our library for questions which are appropriate in both topic and language. Every distractor, every example, every term must be vetted as appropriate for the text, not just the question stem. Even the solution should be compatible to the text.


We also make note of the questions that we cannot find. The text might include an unusual topic or use unconventional vocabulary. Existing questions might include some detail this text omits, and so are unusable.


Just because we have questions about the terms a text uses does not mean we have questions suitable for that text. For example, in one case the term “binary fission” was presented in bold type in a biology text. Our biology team had authored several questions on binary fission, and so were not anticipating having any trouble covering this vocabulary term.


However, this particular text gave only a cursory definition of the binary fission and said nothing about the actual process. Now, the process questions are much more engaging than simple definition questions, so that is what our team had available. However, this meant that not a single existing binary fission question was usable for this text’s depth of knowledge.


Step 3: Edit Existing Questions

Often, we can fill content gaps by duplicating and then editing an existing question. This works well for small changes in vocabulary, or when only one example or distractor is inappropriate in what would otherwise be a good question for the text.


Here is an example from our general chemistry courses. These two questions are almost identical. The only difference is the vocabulary used. One asks for the “change in energy” while the other says “heat of reaction.” The data and the calculation are the same. Since textbooks vary on which nomenclature they use, both questions need to be available.




We often include two sets of terms in our questions so that one question can be used by different styles of textbooks. For example, many Sapling economics questions refer to “perfect, or pure, competition”. In other cases, we might have two, or even three, versions of the same basic question, each using a different set of terms. In this view of a section from our economics library, most of the externality graphs found under the different topic headings are the all same basic graphs with different labels.

Screen Shot 2016-07-20 at 9.45.37 AM.pngStep 4: Author New Questions

If a question cannot be found to edit, we author one specifically for the text. The template reviewer writes a detailed description of what is needed, sometimes even specifying which question type should be used.


This description is then taken by an author, who could be a member of our content team, a tech TA, or a contractor. While seeing to the needs of their instructors always comes first, most tech TAs are also highly experienced authors who enjoy helping the content team create questions when time allows.


Each question is reviewed by members of our content team for accuracy, clarity, appropriateness for the text, style, and grammar. Questions are typically reviewed by at least two people before being cleared for use. You can find more details on the authoring process in the article on the Authoring Process.


These new questions are then added to our library for use with other texts or by professors who want to customize their courses. We try to make sure all of our questions are written in a way which makes them broadly usable by avoiding references to specific statistics or examples, as these differ widely from text to text.


Once the newly authored questions are added to the appropriate assignments, the template is ready to go, and will be duplicated as the first step in creating a customized course according to the professor’s instructions.

Sapling Learning’s content is created by a team of educators and subject-matter experts. A typical question authored by Sapling Learning has passed through the hands of five people before it is placed in assignments and made available to professors and students. At least three of those individuals have Masters’ degrees or PhDs in the question’s subject area. We take care to ensure that questions placed in assignments and taken by students are of the highest quality, but the process of quality assurance does not end once an item has made it into active student assignments.


Periodic Review Process

Each year, the Content Team at Sapling Learning goes through a process we call periodic review. During periodic review, we gather, assess, and improve our most used questions, as well as questions not performing how we expected.


Every time a student makes an attempt on a question, that data is stored. For example, if a student makes three attempts on a homework question, gets generic (default) feedback for one attempt, specific feedback on the second attempt, and then gets the question correct on the third attempt, we don’t just record that the student got the answer correct; all of the tries are preserved.


During periodic review, we collect and organize all student data for questions we’ve authored. First, the Director of Innovation, Jon Harmon, compiles the summative question data, such as the number of students who got the question correct, the number of attempts needed to get the question correct, and the number of attempts students made before giving up on the question. The process of compiling the data takes multiple days on a very large machine. Five separate processors (the equivalent of five computers) are necessary. Jon monitors the process regularly and the machine produces a report every 100 items to ensure the compilation goes smoothly.


Once Jon has compiled the data, he passes it to the Director of Content, Clairissa Simmons, in a large Excel file. Clairissa makes sure the data is well defined, consistent with previous years, includes any data we may need in the future, and available for all disciplines within the Content Team. Next, Clairissa puts all the data into graphing and data organization software called Zoho Reports, which generates a standard set of graphs to compare across disciplines. From there, the subject-matter experts in the Content Team further refine and analyze the data to determine which questions need evaluation and improvement.


Screen Shot 2016-07-20 at 9.07.06 AM.png

This graph plots the questions with the highest average attempts made by students so our content experts can identify which questions students struggle to answer, even with our feedback. In this example, there is one question that has a much higher chance of a student giving up.


Organizing and Prioritizing Data

Each of the subjects have a slightly different approach to analyzing the data because of the disciplines’ specific needs.


The members of the Chemistry team divided their questions into those most taken from the subject’s taxonomy, the questions from the subject’s taxonomy with the lowest scores overall, and the most used questions from other taxonomies. A minimum of 25 questions were updated for each subject based on that division: 10 of the top used, 10 of the lowest scores, and 5 from the most used from other disciplines.


The two Biology subjects, Genetics and Introductory Biology, are newer to the market and thus organized their reviews slightly differently. Because this was the first year of data to analyze, it was most important to fix outlier questions. They organized their questions into those most given up on, most taken, and those with the most attempts. From there, any questions that were in more than one category were prioritized for review.


For economics, the team wanted to know if their problem questions were in need of update or in need of removal from one or more templates. For a given question, they compared the data for the question in cases where it was written for the text to cases where the question was not written for the text (that is, the question was originally written for one textbook but was added to a template for another textbook). This comparison was done for average attempts, average score, the average number of attempts when the answer is correct, and the average score when the answer is correct. Questions with poor ratios were then reviewed to determine if they were in need of improvement, or if they just needed to be taken out of some templates.


For physics (including conceptual, algebra-based, and calculus-based) and astronomy questions, the team chose 1–2 questions with the lowest scores, highest number of attempts, and the highest percentage of giving up; 1–2 of the most-used questions; and 1–2 of the least-used questions to evaluate for suitability. Chemical engineering did not receive periodic review this year.


For all disciplines, much of the updating was done through deprecate and replace, that is, authoring a fresh question and replacing the old item with the new one in assignments. To keep questions between students fair, the change will be made prior to the start of the next semester if some students in that assignment had already seen the older question. This separation of questions helps confirm that the updates we make are helpful and improve the student experience because we can compare the data for the old question with the data for the new one next year.


Additionally, all teams must consider that other disciplines use their questions, which is why most of the teams looked at the top used questions from other disciplines. For example, an introductory chemistry course likely uses many questions from our general chemistry taxonomy. The student experience for those general chemistry questions could differ very much for students in the introductory chemistry course compared to students in a general chemistry course, so it’s important to check that we’ve appropriately placed questions from outside the discipline.



Findings and Results

Data obtained in periodic review can indicate if we need to fix an item, and if so, it can sometimes indicate what we need to fix. For example, in economics periodic review, several questions had good scores for students using the textbook for which the questions were written, whereas students using other textbooks did poorly. The fix for an issue like that is to remove the question from templates for texts we determine are a bad fit with the question.


For other questions, periodic review identifies an issue, but we need to look into the question to determine potential causes.


Biology Example

The following intro bio question had a high number of attempts per student and a high number of students who gave up on the question.


Biology Before.png

To improve the question, the biology team rewrote the instructions in the question stem to give more explicit instructions. They also changed the difficulty of the item from medium to hard to better reflect how challenging the question is. Finally, they revised the solution to be more approachable and easier to read by removing excess content and bolding key terms.


Biology After.png

Introductory Chemistry Example

Sometimes the best fix for an item is to change the module type. In an intro chem item, the original question was multiple select, in which all the choices were correct.


Screen Shot 2016-07-20 at 9.17.00 AM.png

The chemistry team authored the question as a multiple choice question in which one of the answers is “all of the above”-style. This new setup is less tricky for students because it allows them to focus on content of the question. We also think it is less likely that students will second guess the “everything” response for multiple choice, since it is a specific choice, as opposed to an action of checking all checkboxes, which is not commonly correct in multiple select questions.


Screen Shot 2016-07-20 at 9.17.35 AM.png

Again in this case, the solution was improved. This time, however, the solution was expanded.

Screen Shot 2016-07-20 at 9.18.29 AM.pngScreen Shot 2016-07-20 at 9.18.54 AM.png

Physics Example

Finally, questions that are the top used are looked over even if their stats indicate it is a good question. These items often receive improved feedback, revised solutions, and updated art to increase the experience of the tens of thousands of students seeing it each semester. For example, the following physics question was updated throughout.


The question stem was simplified and the labels in the figure were made more clear.

Screen Shot 2016-07-20 at 9.20.40 AM.png

Screen Shot 2016-07-20 at 9.21.28 AM.pngThe introduction of art in the solution helps convey the concept of vector components.

Screen Shot 2016-07-20 at 9.23.10 AM.png

Screen Shot 2016-07-20 at 9.23.34 AM.pngImproving Periodic Review

Improvement does not end with just question content. We have always been careful to store more data than we knew what to do with in the moment. We can use previous years’ data as a benchmark when we find new ways to compile and analyze data.


This year, we refined the process for pulling the data so it can be more efficient. We also have more characterization of feedback this year, such as how often the default tab gets triggers per question is new (this helps us to identify items that need more specific feedback).


Eventually, we plan to allow the script to run continuously so we’ll have the most up-to-date information possible about the quality of our question libraries. For example, we might run periodic review on each new question after enough students to be statistically significant have tried it, instead of waiting until the end of the academic year. As we improve the periodic review process, we have even better tools to keep improving the content we provide for professors and students.

Our team of educators work in concert to craft original questions and feedback to provide you with quality content for your course. To create the exceptional questions instructors expect from Sapling Learning, each one goes through the hands of at least four people. The entire process takes an average of seven hours. A Sapling Learning question starts with a concept to teach a student, goes to an author, and then progresses through four different phases of review before being released to the ultimate reviewers, engaged students.


Screen Shot 2016-07-20 at 8.54.47 AM.png

This diagram summarizes the general flow of Sapling Learning questions among contributors. Experienced authors may skip the functional review stage (although functionality is still tested by the content reviewer) and succinct questions often skip copy editing.



A question is born with a “spec”, the specification for an idea of how to cover a topic, from a content team lead or a professor. Each team lead is a subject-matter expert with a master’s or Ph.D. in their field of expertise. Using their extensive teaching experience, a team lead will typically design a spec to encompass a topic covered in multiple textbooks. The general idea for a question is carefully paired with a module type, such as multiple choice, labeling, or ranking, to present the optimal way to identify and address common misconceptions. Generating each spec takes about 10 minutes to find and match a question with an answer module to best address a concept and its misconceptions.


15.2 The Eye and Vision, Structure of the Eyeball, Order of Retinal Cells
Basic anatomy of the ganglion cell, amacrine cell, horizontal cell, and photoreceptor. Have students place them in order from the innermost part of the eyeball to the outermost. Ranking module.
-Team Lead



Every author at Sapling Learning is a subject-matter expert with either a master’s or Ph.D. in his or her discipline. Each author is assigned a set of specs each week. Just like any experimental science, the authoring process is both thoughtful and creative, and requires at least 5 hours on average. An author will check multiple sources to find common themes and language consistent with most textbooks. The author uses their own teaching and mentoring experience to identify common misconceptions while creating each question, answer choices, solution, and feedback.


I changed the question stem to be "outer to inner" vs "inner to outer" so it followed the pathway of light. I thought that made more sense, especially when explaining things in the solution.

-Contract Author


Sapling Learning questions are unique because the authors craft targeted feedback to drive the student from misconception to the solution. As the hallmark feature of a Sapling Learning question, we pride ourselves on our high quality feedback. Students love our homework because our goal is to guide students past their misconceptions. If they’re confused, the targeted feedback is like a digital tutor helping them identify how to get back on track. Every one of our questions also has a full solution. Each solution addresses the specific topic in the question. It is meant to explain and instill the method or reasoning behind correct answer and outline any common misconceptions. Students often find the solution a particularly useful tool when preparing for exams.


Screen Shot 2016-07-20 at 8.58.41 AM.png

In this example of specific feedback added by the author, students will be given this feedback if they place “amacrine cell” over “horizontal cell.”


Authors are encouraged to use algorithmic variables, or “algos” while building questions. For each student that views a question, algos provide diverse answer choices. This reduces cheating, and pushes collaborating students to consider the concepts instead of the answer choices.  Algos are incorporated into many Sapling Learning questions, and each is carefully selected to ensure the variation is centered around a particular concept or misconception, while providing a distinct question that is pedagogically fair to every student. Our algos can also be linked together to provide targeted feedback and a solution that is always relevant for the student. In Biology, many of our algos are conceptual.


Screen Shot 2016-07-20 at 8.59.44 AM.png

In this question, the system will pick one of three terms to display to the student to rank. Based on that term, a different pluralization of the term is also generated to make the solution more readable to students.



Once the author has finished the first draft, the question undergoes functional and content reviews. These are iterative editorial and revision processes between each reviewer and the author. These review cycles allow us to improve each and every question for the benefit of your students. The functional reviewer thoroughly examines the question, taking on average an hour, to ensure Sapling Learning’s rigorous standards are maintained. A functional reviewer will check to ensure that the question works correctly, the question and answer choices are clear, the solution thoroughly explains the correct answer, and any misconceptions raised by incorrect choices are addressed. Strict attention is also paid to the layout, format, wording and style of a Sapling Learning question.


After the question has passed the functional review stage, a content reviewer checks each question for content-specific errors. For about an hour, the content reviewer dissects a question to ensures it is clear, concise, unambiguous, and factually correct. Each content reviewer is a subject-matter expert with a Masters or PhD in the relevant discipline, as well as teaching experience. This depth of knowledge and experience with students ensures all common misconceptions are addressed in our feedback. The content reviewer also assesses pedagogy for each part of a question making sure the feedback, guidance, and solution are not open to misinterpretation by a confused student. Each piece of scientific art used in a Sapling Learning question also undergoes the scrutiny of a content editor. Each image is checked for scientific accuracy, notation, and style, as well as for accessibility, such as to visually impaired students.


Screen Shot 2016-07-20 at 9.00.58 AM.pngEvery question has a detailed solution for the student. In this example, art was added to the solution to enhance student understanding.


Next, the question goes to a copy editor, who has a degree in English, experience editing STEM projects, and a strong technical background. A copy editor will spend approximately half an hour looking for spelling errors, grammatical issues, and ambiguous phrasing in the question. Typically, most syntax errors and typos that are not obviously content related are changed, whereas other queries about language clarity are noted for our content experts to review. Sapling Learning questions use specific verbiage and the copy editor highlights problems such as inappropriate use of anthropomorphisms, passive voice, and the appropriate use of parentheses, hyphens, and other punctuation. Each algorithmic variable is also checked to confirm it matches the context in which it is found.



The content team lead completes a final review of every question. The final review is the last check to ensure a question is functional, scientifically accurate, and that the language is appropriate to the topic. Only after their review, generally about 30 minutes, is it approved. Approval culminates in making the question live, which adds the question to the more than 20,000 already available in the Sapling Learning question bank. The live question is instantly accessible to any instructor or Technology TA, who can access the question by creating or editing an assignment in a course homework site.


Screen Shot 2016-07-20 at 9.02.20 AM.png

In this example, the student is triggering the feedback conditions shown in the screenshot earlier.


Ultimately, the effort and energy we put into SL content creation totals seven hours or more per question. This means that the 2,200+ questions that are current live for our Intro Biology Homework solution equates to over 385 full-time weeks of work, or over seven years of content creation and development in man hours for a single discipline. Our efforts in passionately creating content for your course helps save you time so you can spend more of your day educating your students.

Originally posted by Andrew S. Campbell, Ph.D.

Academics like you are rarely satisfied with the status quo, and that extends to the classroom. Thanks to modern advancements, a lecture course can be adjusted, and even transformed, with the goal of increasing student understanding and retention.

At Haverford College, Professor Joshua Schrier is innovating his classroom to emphasize problem solving, creating a more efficient and lively learning experience. Hear how he uses tools like Sapling Learning to facilitate this general chemistry course.

IITS Technology, Innovation & Pedagogy series: Joshua Schrier, Assistant Professor of Chemistry at Haverford College from Instructional technology on Vimeo.



Which techniques or resources do you use to engage your students? Share your thoughts in the comments below.

Keeping Students Engaged:

A Tale from Introductory Chemistry

(Part I)


by Kevin Revell, Ph.D.

Associate Professor of Chemistry and Consultant  · Sapling Learning


Just before the spring 2012 semester, I learned that I would be teaching the introductory chemistry class at Murray State University. I confess that I was less than thrilled with the assignment. It was two nights a week during my son’s basketball season, it was a large class that I had not taught in a long time, and it was largely freshman, with plenty of non-science majors meeting gen-ed requirements.   After teaching organic chemistry for so long, this was going to be an adventure.


To help things out, I decided to try a couple of new tools. For years, I had used a tablet PC in my lecture – posting skeletal notes before class, then fleshing out the concepts as we went through the class period. This semester, I decided to take this one step further and record the screen and audio for each lecture. And I wanted to use online homework. I had tried Sapling Learning in my organic course the semester before with terrific results. I was curious how this would translate to less-motivated students.


About four weeks into class, I began to notice a few unusual positives. First, in a room with a capacity of 144 students, there were almost no empty seats. Second, the students were doing WELL. Even in the second unit, I found that over 90% of the students were attempting the homework. This was not the high-attrition course I had expected.


Based on this, I decided to do a more systematic study: what was helping them succeed? How did the tablet-based lectures, the recorded lectures, and the online homework really contribute to student success? In order to investigate this, I began to correlate student performance with usage of the lecture-replay and online homework, and I administered a year-end survey to assess the student impressions of each tool. The full study was recently published in the Journal of Chemical Education, available here.


One of the biggest gains I saw was in the area of retention. In the previous five full-semester classes, the pass rate for this course was 71%. In my spring 2013 course, it jumped to 90%, with no significant change in the standardized test scores. Interestingly, the number of A’s didn’t change much. It turns out there are students who will work every problem, study aggressively, and get the A regardless of how the course is presented. The big jump was in the number of B’s and C’s earned. Based on these results, I believe that the combination of tools helped students stay more engaged, practice more, and earn a B instead of a C, or a C instead of a D.


And perhaps the most striking thing? Teaching introductory chemistry was an absolute blast. This semester, I actually asked to teach it again. I look forward to writing more on that in the weeks to come.

Originally posted by Kelli Mayes-Denker.

Have you had a student stop by during office hours only to mention that they "just don’t get chapter 5"? For a student, this specific chapter may be especially daunting. In reality, there are probably a few distinct concepts the student is struggling with, thus causing problems throughout the entire chapter.

Being an economist, it is important to find a way to more efficiently utilize office hours and maximize student understanding. For instance, I started asking students to email me to schedule a meeting during office hours. This allows me to be more prepared for them and target their needs.

In my preparation for the meeting, I could look up the homework scores in Sapling Learning and click on the score in the grade book to see detail of the student's work. Many times, the reality was that we could focus our meeting on specific questions to address the problematic areas.

With Sapling Learning, I can get the overall homework score and view each attempt that the student made on each question. It became increasingly helpful to walk through several of these with individual students, asking them specifics to gauge understanding and further remedy their knowledge of the content.

The outcome is a more efficient use of my time during office hours with the ability to focus the student in areas of importance as opposed to a general high-level review. With the student present, we could address the content, improve understanding and then spend some time getting to understand their interests.

Once I know a bit more about the student, these interests can be incorporated into lectures, class discussions and projects to further engage the student in the overall economics subject material. This created a winning situation for both of us.

Viewing student attempts in Sapling Learning: do it yourself or email your Tech TA to learn how

1. Go to the grades icon on the main page, left side

2. The grader report will open with all student showing, then click on a student score

3. Now you will see one student’s assignment and each question in the assignment

4. Click on a quick number at the far left, this will allow you to see all attempts for that question

The example below shows Linus Pauling’s assignment. Question 2 is showing with each of his 5 attempts. Click the incorrect or correct tabs at the top of the question to view each attempt.

I have recently sent out a number of emails to large groups of instructors--and gotten a lot of bounceback emails in return. I read all of these in case there are questions (which happens) or people who want to be removed from our email lists or who have retired (which also happens). Most of these automated out of office messages are fairly standard, but a few stand out because they show what you can do in the summer. Included in my list of great ones are: 

  • I'm off doing field work in Africa.
  • I am taking a group of students to the Gallapagos Islands.
  • I am doing a cross-country trip to visit family and some national parks.
  • I will be doing teaching and learning in Chile this summer.
  • I will be in Cuba and off the grid.

I hope all of you are having a great summer, doing something fabulous, and getting reinvigorated in your discipline! Fall will be here all too soon.