Clairissa Simmons

Sapling Learning Questions - Authoring Process

Blog Post created by Clairissa Simmons on Jul 20, 2016

Our team of educators work in concert to craft original questions and feedback to provide you with quality content for your course. To create the exceptional questions instructors expect from Sapling Learning, each one goes through the hands of at least four people. The entire process takes an average of seven hours. A Sapling Learning question starts with a concept to teach a student, goes to an author, and then progresses through four different phases of review before being released to the ultimate reviewers, engaged students.

 

Screen Shot 2016-07-20 at 8.54.47 AM.png

This diagram summarizes the general flow of Sapling Learning questions among contributors. Experienced authors may skip the functional review stage (although functionality is still tested by the content reviewer) and succinct questions often skip copy editing.

 

Conception

A question is born with a “spec”, the specification for an idea of how to cover a topic, from a content team lead or a professor. Each team lead is a subject-matter expert with a master’s or Ph.D. in their field of expertise. Using their extensive teaching experience, a team lead will typically design a spec to encompass a topic covered in multiple textbooks. The general idea for a question is carefully paired with a module type, such as multiple choice, labeling, or ranking, to present the optimal way to identify and address common misconceptions. Generating each spec takes about 10 minutes to find and match a question with an answer module to best address a concept and its misconceptions.

 

15.2 The Eye and Vision, Structure of the Eyeball, Order of Retinal Cells
Basic anatomy of the ganglion cell, amacrine cell, horizontal cell, and photoreceptor. Have students place them in order from the innermost part of the eyeball to the outermost. Ranking module.
-Team Lead

 

Development

Every author at Sapling Learning is a subject-matter expert with either a master’s or Ph.D. in his or her discipline. Each author is assigned a set of specs each week. Just like any experimental science, the authoring process is both thoughtful and creative, and requires at least 5 hours on average. An author will check multiple sources to find common themes and language consistent with most textbooks. The author uses their own teaching and mentoring experience to identify common misconceptions while creating each question, answer choices, solution, and feedback.

 

I changed the question stem to be "outer to inner" vs "inner to outer" so it followed the pathway of light. I thought that made more sense, especially when explaining things in the solution.

-Contract Author

 

Sapling Learning questions are unique because the authors craft targeted feedback to drive the student from misconception to the solution. As the hallmark feature of a Sapling Learning question, we pride ourselves on our high quality feedback. Students love our homework because our goal is to guide students past their misconceptions. If they’re confused, the targeted feedback is like a digital tutor helping them identify how to get back on track. Every one of our questions also has a full solution. Each solution addresses the specific topic in the question. It is meant to explain and instill the method or reasoning behind correct answer and outline any common misconceptions. Students often find the solution a particularly useful tool when preparing for exams.

 

Screen Shot 2016-07-20 at 8.58.41 AM.png

In this example of specific feedback added by the author, students will be given this feedback if they place “amacrine cell” over “horizontal cell.”

 

Authors are encouraged to use algorithmic variables, or “algos” while building questions. For each student that views a question, algos provide diverse answer choices. This reduces cheating, and pushes collaborating students to consider the concepts instead of the answer choices.  Algos are incorporated into many Sapling Learning questions, and each is carefully selected to ensure the variation is centered around a particular concept or misconception, while providing a distinct question that is pedagogically fair to every student. Our algos can also be linked together to provide targeted feedback and a solution that is always relevant for the student. In Biology, many of our algos are conceptual.

 

Screen Shot 2016-07-20 at 8.59.44 AM.png

In this question, the system will pick one of three terms to display to the student to rank. Based on that term, a different pluralization of the term is also generated to make the solution more readable to students.

 

Review

Once the author has finished the first draft, the question undergoes functional and content reviews. These are iterative editorial and revision processes between each reviewer and the author. These review cycles allow us to improve each and every question for the benefit of your students. The functional reviewer thoroughly examines the question, taking on average an hour, to ensure Sapling Learning’s rigorous standards are maintained. A functional reviewer will check to ensure that the question works correctly, the question and answer choices are clear, the solution thoroughly explains the correct answer, and any misconceptions raised by incorrect choices are addressed. Strict attention is also paid to the layout, format, wording and style of a Sapling Learning question.

 

After the question has passed the functional review stage, a content reviewer checks each question for content-specific errors. For about an hour, the content reviewer dissects a question to ensures it is clear, concise, unambiguous, and factually correct. Each content reviewer is a subject-matter expert with a Masters or PhD in the relevant discipline, as well as teaching experience. This depth of knowledge and experience with students ensures all common misconceptions are addressed in our feedback. The content reviewer also assesses pedagogy for each part of a question making sure the feedback, guidance, and solution are not open to misinterpretation by a confused student. Each piece of scientific art used in a Sapling Learning question also undergoes the scrutiny of a content editor. Each image is checked for scientific accuracy, notation, and style, as well as for accessibility, such as to visually impaired students.

 

Screen Shot 2016-07-20 at 9.00.58 AM.pngEvery question has a detailed solution for the student. In this example, art was added to the solution to enhance student understanding.

 

Next, the question goes to a copy editor, who has a degree in English, experience editing STEM projects, and a strong technical background. A copy editor will spend approximately half an hour looking for spelling errors, grammatical issues, and ambiguous phrasing in the question. Typically, most syntax errors and typos that are not obviously content related are changed, whereas other queries about language clarity are noted for our content experts to review. Sapling Learning questions use specific verbiage and the copy editor highlights problems such as inappropriate use of anthropomorphisms, passive voice, and the appropriate use of parentheses, hyphens, and other punctuation. Each algorithmic variable is also checked to confirm it matches the context in which it is found.

 

Release

The content team lead completes a final review of every question. The final review is the last check to ensure a question is functional, scientifically accurate, and that the language is appropriate to the topic. Only after their review, generally about 30 minutes, is it approved. Approval culminates in making the question live, which adds the question to the more than 20,000 already available in the Sapling Learning question bank. The live question is instantly accessible to any instructor or Technology TA, who can access the question by creating or editing an assignment in a course homework site.

 

Screen Shot 2016-07-20 at 9.02.20 AM.png

In this example, the student is triggering the feedback conditions shown in the screenshot earlier.

 

Ultimately, the effort and energy we put into SL content creation totals seven hours or more per question. This means that the 2,200+ questions that are current live for our Intro Biology Homework solution equates to over 385 full-time weeks of work, or over seven years of content creation and development in man hours for a single discipline. Our efforts in passionately creating content for your course helps save you time so you can spend more of your day educating your students.

Outcomes