april17_b.indd C&RL News April 2017 212 Graham Sherriff Interactive tutorials—the platform matters Learning from a comparative evaluation of tutorial platforms Graham Sherriff is instructional design librarian/liaison to the College of Engineering and Mathematical Sciences at the University of Vermont’s Baily/Howe Library, email: graham.sherriff@uvm.edu © 2017 Graham Sherriff There is a growing body of theory and research on the characteristics of effec- tive learning objects and library tutorials, but understanding the platforms used to create them presents its own challenges. New products continue to emerge, while established products continue to develop and upgrade. This article describes what one library accomplished by conducting a comparative evaluation of two platforms used to create the same tutorial model: the frame-based, live-web tutorial. Homegrown tutorials for local needs For several years, the Bailey/Howe Library at the University of Vermont (UVM) has flipped library instruction with online tutorials. In particular, UVM librarians use them to sup- port one-shot sessions integrated into the university’s first-year Foundational Writing and Information Literacy (FWIL) initiative. We aim to give the FWIL library sessions maximum impact by timing them to take place soon after the students have begun the research component of their course and by focusing the sessions on the challenges they are beginning to encounter. These are often difficult questions involving the evaluation of search results and sources, for example how to find diverse perspectives or how to find sources with the right level of technical detail. Addressing these questions in the class set- ting gives students the tools and techniques to move their research forward. The timing and focus of these sessions depend on flipped instruction in foundational concepts that enable students to begin their research in advance of the library session. We provide this instruction through a suite of five interactive tutorials covering the evaluation of information, periodicals and scholarly communication, databases, and search techniques.1 This flipped instruction ensures students have a common baseline of knowledge and allows librarians to focus their one-shots on students’ application of what they have learned. Each tutorial contains several questions and two have additional quizzes, but assess- ment is not the priority. They are primarily designed for formative learning: a learn- ing experience made effective through the learner’s self-guided discovery that brings each student to the same point of knowledge and understanding. Frame-based, live-web tutorials UVM created these five tutorials in Guide on the Side, the University of Arizona’s open source platform for creating frame-based, live-web tutorials.2 The learner accesses the ACRL TechConnect April 2017 213 C&RL News tutorial through their web browser, where a narrow left-side frame presents direc- tions and poses questions. This frame is also where the learner submits responses. The rest of the browser window is a larger right-side frame that allows the learner to engage with live web content, navigating, scanning, and scrolling in order to follow directions, complete tasks, and find an- swers. This model of tutorial strengthens learn- ing, formative learning in particular, in several ways. The use of live web content lends the experience more authenticity than captured content like screencasts. Learn- ers have the autonomy to make their own decisions about how best to complete tasks or respond to what they are seeing, in ad- dition to having control over their pacing and the time taken to complete the tutorial. For example, a student can be prompted to navigate to a library database and submit a search query that is relevant to their topic and answer questions that require critical reflection on the results. Frame-based tutorials also offer advan- tages to the librarian who produces them. They may have a significant production time —one study has estimated that a Guide on the Side tutorial requires three times as long to develop as a screencast with equivalent content3—but they may provide time sav- ings in the long run by being more scalable. Frame-based tutorials can be reproduced and easily updated in response to changes in web content or customized to meet the needs of different courses, learning outcomes, or student groups. UVM librarians have used templates and adapted the five FWIL tutorials to tailor tutorials to the needs of other courses and disciplines. Evaluating products UVM adopted Guide on the Side principally because of this scalability—each FWIL tuto- rial may be taken by up to 2,500 freshmen every year—and its suitability for formative learning. Another important consideration was the fact that Guide on the Side is free to install and maintain. As an open source program, it may be “free like kittens” but, once installed, the requirement for back- end maintenance is low. Yet Guide on the Side also has char- acteristics that, for UVM, are limitations. Most frustratingly, it has no functions for aggregating or reporting data generated by students’ responses to tutorial ques- tions and quizzes. Instead, each tutorial can route an HTML email “certificate” to the librarian or platform administrator. Analyzing the data in these certificates is then a time-consuming matter of manual data processing, which for UVM has been prohibitive beyond small samples. In 2016, Springshare launched LibWizard Tutorials, a new module within its LibWiz- ard product and an alternative platform for frame-based tutorials. The two platforms are ostensibly very similar: they create tutorials with the same two-frame structure, they both integrate interactions with live-web content, and they have similar quizzing features. But on closer inspection, LibWizard offered several useful functions that were not available in Guide on the Side, while lacking others that were integral features of our existing tutorials. What might be gained from a switch to LibWizard, and what might be lost or compromised? Developing an evaluative approach We needed a detailed review, evaluation, and comparison of the two products. To do this, our instructional design librarian (also our Guide on the Side administrator) and our coordinator of library instruction developed a set of criteria that embod- ies our needs and priorities. We collec- tively reviewed our existing tutorials, pay- ing closest attention to the ones with the highest usage and the deepest integration into the curriculum, and documented the characteristics that would be necessary in any platform. Our drafting was enhanced, and to a certain degree validated, by con- sidering the research literature on learning behavior and the characteristics of effective C&RL News April 2017 214 learning objects.4 We then sought input from our library’s director of instructional services, our ad- ministrator of Springshare licenses, and our lead users of tutorials across UVM’s two libraries. Facilitating this collective review of our tutorial needs ensured nothing sig- nificant had been missed. We now had our evaluative criteria, which we placed into a simple rubric, or- ganizing them into four categories: • Formative learning. Features that support the student’s ability to achieve a tutorial’s learning outcomes completely and in a manner that confers learning and confidence. • Summative assessment. Features that support quizzing and grading. • Data management. Features that support the aggregation and reporting of performance data. • Ease of use. Features that simplify the platform’s adoption, use, and administration. Each criterion was assigned a level of priority: • Essential. Characteristics that are inte- gral to how our tutorials work and the attain- ment of learn- ing outcomes. • I m p o r - tant. Elements that are broad- ly desirable but not essential. • Optional. Elements that o f f e r m i n o r enhancements a n d w o u l d not determine a decision on platform adop- tion. W e w e r e t h e n a b l e t o apply this ru- b r i c t o e a c h platform, identify their respective strengths and weaknesses, and consider the signifi- cance of each strength and weakness, and make a final decision.5 Advantages of an evaluation rubric This method of evaluating tutorial plat- forms offers several benefits for any li- brary that needs to select from a choice of comparable products. 1. Defining your needs. What does your library need from a tutorials platform? The process of creating an evaluation rubric helped us to define and articulate our tuto- rial needs, based on local circumstances: our students’ levels of information literacy, the points in the curriculum where we have instructional contact with them, the objec- tives of the library instruction program, the objectives of the FWIL initiative, and the practices of our instructional librarians. It can be a way to define tutorial needs on a program level. For example, our rubric shows very clearly that our library—espe- cially our instruction in the FWIL initia- tive—needs a tool supporting formative assessment more than a tool for summative assessment. One of UVM’s Guide on the Side tutorials, created by Erica DeFrain. View this article online for detailed images. April 2017 215 C&RL News 4. Creating a tool for the evaluation of future products. We developed our rubric to support the immediate need to select a platform, but we now have a tool for re-eval- uating a product in case of future upgrades or for evaluating any new frame-based tutorial platform that might come to market. Given the value of data on student performance, it seems likely that more platforms will emerge in the near future. 5. Creating a tool for identifying unsatisfied needs and advocating for product development. In one sense, our rubric is a wish list of all the things we would like to see in a tutorials platform and, unsurprisingly, neither of the two platforms offered every feature on the list. But iden- tifying “missing” features is important for thinking about a platform’s prospects for further development. It can be the basis for communicating with vendors. By providing detailed feedback on LibWizard, UVM has had some very positive dialogue with Spring- share about platform design and develop- ment. Finally, identifying missing features is a basis for continuing the conversation within instructional librarianship about what we need from a tutorial platform. It can also be a way to define the features that make a platform suited to differ- ent objectives. Creating the ru- bric drew our attention to the value of custom- ized feedback for different in- correct options in a multiple- choice question. If a student has selected a cer- tain incorrect option, it may reflect a certain m i s t a k e a n d custom feedback can explain what the learner needs to do differently to get the right answer. The capacity to present this kind of feedback is a feature of Guide on the Side that we were using but had previously undervalued. 2. Evaluating platforms, relative to needs. A rubric makes it possible to evalu- ate a tutorials platform, relative to the needs that have been defined. To what extent does a platform contain the features you need? If some features are lacking, are they essential, important, or optional? A rubric is also an ef- fective means for guiding a group’s collective evaluation and creating a uniform structure for everyone’s input. 3. Comparing platforms. The compara- tive aspect of the evaluation helps to identify the areas where one platform is better suited to local needs than the other—and the signifi- cance (or otherwise) of each advantage. In our situation, our evaluation showed us that we were considering similar platforms with small functional differences that, as we see it, embodied significant differences in tutorial design. Our evaluation led to the conclusion that Guide on the Side is oriented towards formative learning, while LibWizard is more oriented towards summative assessment. The same tutorial, recreated in LibWizard. View this article online for detailed images. C&RL News April 2017 216 Learning from the process UVM’s comparative evaluation of these two tutorial platforms has not lead to a final decision. We have adopted LibWizard for some tutorials, but continue to use Guide on the Side when the objective is forma- tive learning. Those decisions aside, the process itself has been instructive. The evaluation showed us clearly, and with much specificity, that the two plat- forms offer qualitatively different learn- ing experiences that relate to our local instructional needs in different ways. The possibility of switching from one to another requires us to think deeply about how our design practices would need to be rewired in a new platform. In the meantime, our evaluation is a useful resource as we con- tinue to monitor product developments and engage in dialogue with platform users and providers. Notes 1. Originally created by Erica DeFrain, UVM’s then-instructional design librarian. 2. Leslie Sult, Yvonne Mery, Rebecca Blakis- ton, and Elizabeth Kline, “A New Approach to Online Database Instruction: Developing the Guide on the Side,” Reference Services Review 41 (2013): 125–33, doi:10.1108/00907321311300947. 3. Susan Mikkelsen and Elizabeth Mc- Munn-Tetangco, “Guide on the Side: Testing the Tool and the Tutorials,” Internet Refer- ence Services Quarterly 19 (2014): 271–82, doi:10.1080/10875301.2014.948252. 4. A good overview of the literature is provided in Katherine Stiwinter’s “Using an Interactive Online Tutorial to Expand Library Instruction,” Internet Reference Services Quarterly 18 (2013): 15–41, doi:10.1080/108 75301.2013.777010. 5. Our rubric is posted at http://go.uvm. edu/libtutrubric. The award-winning source for jobs in Library and Information Science and Technology. JOB SEEKERS Filter and sort hundreds of job ads by position type, employer, location, job title, and more. EMPLOYERS Strengthen your candidate pool—ALA reaches the engaged professionals and students you want to hire. HRDR joblist.ala.org