Swann, W., & Cripe, M. K. (2018). Balancing resources and value in distance education course reviews: A case study at a mid-sized public university. International Online Journal of Education and Teaching (IOJET), 5(4),753-766. http://iojet.org/index.php/IOJET/article/view/402/292 Received: 09.03.2018 Received in revised form: 29.07.2018 Accepted: 13.09.2018 BALANCING RESOURCES AND VALUE IN DISTANCE EDUCATION COURSE REVIEWS: A CASE STUDY AT A MID-SIZED PUBLIC UNIVERSITY Research Article William Swann Youngstown State University wswann@ysu.edu *Corresponding author M. Kathleen Cripe Youngstown State University mlcripe@ysu.edu Biodata: William Swann has worked in the fields of e-learning and distance education for two decades. He has developed courses and managed development projects in the corporate environment and provided instructional design assistance in the academic environment. He is the author of studies on applied pedagogy and courseware design. Dr. Kathleen Cripe is Associate Professor of Teacher Education in the Beeghly College of Education at Youngstown State University, where she coordinates the Adolescent Young Adult Education program. She is the author of multiple studies on science and math education and received the 2017 Distinguished Professor Award for Excellence in Teaching. Copyright by Informascope. Material published and so copyrighted may not be published elsewhere without the written permission of IOJET. mailto:wswann@ysu.edu mailto:mlcripe@ysu.edu https://orcid.org/0000-0002-7997-3790 https://orcid.org/0000-0003-4676-6805 International Online Journal of Education and Teaching (IOJET) 2018, 5(4), 753-766. 753 BALANCING RESOURCES AND VALUE IN DISTANCE EDUCATION COURSE REVIEWS: A CASE STUDY AT A MID-SIZED PUBLIC UNIVERSITY William Swann wswann@ysu.edu M. Kathleen Cripe mlcripe@ysu.edu Abstract This study examines the development and implementation of a distance education course review process at a mid-sized public university. Four primary goals were set for the process: to provide substantive feedback, to cultivate engagement between DE faculty and staff, to provide support to course developers and reviewers, and to establish an effective balance between faculty resources and the value of feedback generated through the process. Feedback was collected through a survey of participating developers and reviewers (n=52). Responses broadly supported achievement of the four primary goals. Those who participated in multiple roles gave stronger ratings on all survey questions than those who participated through a single role. Based on qualitative and quantitative feedback, strengths and weaknesses of the process are discussed along with recommendations for institutions developing or refining a DE course review process. Keywords: review, standards, distance education, quality, efficiency 1. Introduction A process of reviewing distance education courses, whether formal or informal, is often part of the array of supports provided to faculty in distance education (DE) programs. It can be one of the more practical components of DE programs, prompting faculty and staff to work together in the assessment, improvement, and refinement of DE courses. Sets of standards for assessing the quality of courses have been developed at a number of colleges, universities, and associated accrediting bodies. Recent scholarly reviews of such processes point to a primary focus on the design of courses, with few standards addressing the areas of course delivery or faculty support (Martin et al, 2017; Southland & Mooney, 2015). Ramdass and Nemavhola (2018) point to faculty perceptions of a "compliance" or "check box" mentality in the application of some processes along with concerns over academic freedom, and several studies emphasize the importance of customizing quality review processes to meet the needs and conditions at particular institutions (Bari & Djouab, 2014; Moore, 2010; Pawlowski, 2006). This case study examines the development and implementation of a course review process at a mid-sized public university. Relevant conditions include a student body of approximately 13,000, a distance education program experiencing rapid growth, and an intensive focus on DE in certain departments where fully online degree programs are under development. By 2018, 12 fully online degrees were offered at the university. The growth of the program and number mailto:wswann@ysu.edu mailto:mlcripe@ysu.edu Swann & Cripe 754 of faculty involved were substantial enough to prompt concerns over excessive bureaucratization and the "compliance mentality" noted by Ramdass and Nemavhola. At the same time, the program is not so large as to preclude a personal approach to engagement with faculty and collaboration in the assessment and improvement processes. Building the university's process involved the identification of primary goals, development of a review rubric, design of the process, and implementation. Among other goals, there was a focus on developing supportive and substantive relationships between DE staff and faculty. With this in mind, faculty support was considered vital, as was the establishment of a balance between the time commitment from faculty and value of the resulting feedback. As of this writing, 96 courses have been reviewed and approved over a period of four years. Feedback was collected through a survey of participating developers and reviewers. Responses were broadly consistent with achievement of the primary goals, though average responses were stronger for some goals than others. Differences were noted based on the roles of participants. About a third were developers of courses, but never participated as a reviewer; another third reviewed, but did not develop; and the final third did both. Those who performed both roles gave more positive ratings on all 11 survey questions than those who participated through one role. Based on these findings, strengths and weaknesses are assessed along with recommendations likely to have relevance for similarly situated institutions (and which may have some relevance for smaller or larger institutions). 2. Goals The broad purpose of the YSU distance education course review process is to encourage the development and enhancement of the university's fully online courses. It provides a point of engagement between faculty who develop courses and others in the university community who may be of assistance, including subject-matter experts in the relevant academic fields and distance education support staff. The latter assist faculty in developing skills with the Learning Management System and other technologies as well as evaluating and applying appropriate pedagogies. To incentivize engagement with the review process, the university offers development compensation upon successful completion of a formal review. For purposes of developing the process, the broad goal of enhancing distance courses was broken down into four components: 1. Feedback: To provide substantive feedback instructors can use to improve their courses, including the identification of any significant deficiencies as well as suggestions for improvement with the greatest potential to enhance student learning. 2. Constructiveness: To cultivate an environment of positive engagement between instructors, course reviewers, and support staff. Substantive assistance should be provided to the developers of online courseware in an atmosphere of collegiality, encouragement, and shared purpose. 3. Support: To provide effective guidance to peer reviewers and course developers in understanding the process, completing it, and making the required or recommended revisions. A course developer should experience the process as an opportunity to partner with experts in the relevant academic fields and with distance learning professionals. 4. Efficiency: To make judicious use of faculty and staff resources. A balance should be established between the effort involved for faculty and the value of the resulting feedback. In the challenging fiscal environments that prevail in higher education, faculty are continually asked to do more with limited resources. Both the peer review International Online Journal of Education and Teaching (IOJET) 2018, 5(4), 753-766. 755 and instructor roles in a DE course review process require work that was not previously budgeted in faculty schedules. Note the relationship between the second, third, and fourth goals. In the absence of sufficient support, faculty may not experience the review process constructively. The same is true if they perceive the work as excessive or as focused more on process and documentation than substance. For this reason, efficiency and faculty supports were considered critical factors in the success of DE reviews. 3. Developing a Course Review Rubric A rubric for assessing distance education courses was developed by an instructional designer in the DE office in consultation with the university's Distance Education Advisory Committee (DEAC), which includes faculty who teach DE courses as well as staff and administrators. The design of the rubric focused primarily on the first and fourth goals identified above—to provide feedback to instructors on items that are most likely to improve student learning experiences and to establish a substantive and manageable set of review standards. The rubric was designed to support a review and revision process focused on student success in a manner that could be completed with reasonable efficiency. Rubric criteria were selected through consideration of three primary sources: • The literature on adult learning and pedagogy—in particular, elements that have been found to have the strongest impact on student success. • Distance education guidelines developed by the Council of Regional Accrediting Commissions (C-RAC) and recommended by YSU's accrediting body, the Higher Learning Commission (HLC). C-RAC guideline 6(b) calls for training of online faculty that "incorporates tested good practices in on-line learning pedagogy" (Higher Learning Commission, 2009). • Accessibility standards established in Section 508 of the Rehabilitation Act of 1973 (29 U.S.C. Section 794) and Title II of the Americans with Disabilities Act of 1990 (ADA, 42 U.S.C. Section 12131 et seq.), which require providing disabled persons with "access to and use of information and data that is comparable to" access provided to others (Section 508 (a) (1) (A) (ii)). It should also be noted that most staff and faculty involved in the rubric development process received training on the Quality Matters DE course review process, in some cases having developed a course and submitted it to QM review, and in other cases assisting faculty in navigating the QM process. Due to the grounding of QM criteria in the literature on adult learning, it would be difficult to exclude certain fundamental concepts applied through the QM process in any meaningful review of DE courses. For example, the alignment concept is foundational and well-attested in the literature (Dick, Carey, & Carey, 2001; Smith & Ragan, 2005; Reeves, 2006; Blumberg, 2009; Kauffman, 2015). The YSU rubric was developed through an examination of pedagogical and process literature. It has 16 standards, some of which are similar to the 42 standards in the Quality Matters rubric. 3.1. Selecting Rubric Criteria In developing a rubric, efforts were made to identify the criteria with greatest impact on student success while establishing a manageable and reasonably focused set of standards. 3.1.1. Learning objectives and alignment The first three of the selected 16 standards relate to learning objectives and the fourth to alignment of content, assignments, and assessments with those objectives. Objectives are the Swann & Cripe 756 foundation of the design process, establishing goals to guide the developer when building a course. With those goals in mind, an instructor selects appropriate pedagogies and assembles content, assignments, and assessments to support achievement of the objectives. Course content is aligned with the established objectives through this process (Dick, Carey, & Carey, 2001; Smith & Ragan, 2005; Reeves, 2006; Blumberg, 2009; Kauffman, 2015). Developers were asked to apply a taxonomy of levels of knowledge developed by Bloom (1956) to the selection of action verbs for course-level learning objectives. 3.1.2. Segmenting and sequencing Maintaining a student's sense of active engagement with course materials is a primary challenge in the online environment. This is facilitated through division of any lengthy content, such as a full lecture, into segments the student can digest easily before moving to another content item or interaction. Studies on active learning (Bonwell & Eison, 1991) and cognitive research (Mayer & Chandler, 2001; Mayer, Dow, & Mayer, 2003) suggest the importance of segmenting content, assignments, and assessments. Two additional studies found that segmenting of written and oral examples improved learning efficiency (Spanjers, Wouters, & Merriënboer, 2011; Spanjers et al, 2012). Cognitive and constructivist pedagogies also emphasize the building of lower-level skills (simple schema) to prepare the learner to tackle higher-level skills (complex schema). Initial learning should be supported through simple learning activities with low levels of cognitive load (Sweller & Cooper, 1985; Paas, Tuovinen, Tabbers, & Gerven, 2003). Higher-order skills are learned through engagement with problem-solving and instructional activities of the kind that require writing, discussion, analysis of case studies, presentations, and peer teaching (Bonwell & Eison, 1991). Dresner et al found sequencing of content to improve students' "higher-order thinking skills" (Dresner et al, 2014, 47). 3.1.3. Clarity Many instructors have discovered the increased importance in the online environment of providing clear instructions for each item as well as a consistent course structure that is easy to navigate. In a face-to-face class, the instructor is there to address questions about assignments and other course requirements. Equivalent communications in the online environment are typically asynchronous, which means the student's progress may be temporarily halted until an answer can be provided. Cognitive studies also demonstrate that clear, straightforward, and unambiguous content requires fewer mental resources to process than more complex content. In 13 of 14 studies by Mayer et al, a concise module led to better subsequent performance on problem-solving tasks than a more complex module (Mayer, 2008). The interaction of the learner with course content has been measured as the strongest contributor to student satisfaction in fully online courses. A recent study by Kuo et al noted the importance of "[o]rganization of content, document layout, and ease of accessing online content" in successful student engagement with online materials (Kuo, Walker, & Schroder, 2013, 30). C-RAC guideline 4(i) calls for "[c]ourse and program structures [that] provide schedule and support known to be effective in helping online learning students succeed" (Higher Learning Commission, 2009). 3.1.4. Multimedia This is both a cognitive learning principle and a universal design principle that applies broadly to content developed for the web. Learners use their limited short-term memory stores while digesting information, building mental models, and storing them in long-term memory. There are separate short-term memory stores for audio and visual information. Utilizing both International Online Journal of Education and Teaching (IOJET) 2018, 5(4), 753-766. 757 provides access to greater overall capacity than either does by itself (Baddeley, 1986; Mayer, 2008). An appropriate combination of media should be used to convey content, assignments, assessments, and interactive course elements. It may include written text, images, quizzes, discussions, audio, video, animations, interactive tutorials, or other formats. It was determined that a minimum of three media types may be reasonable for some courses, while most should apply a variety that includes more than three types. 3.1.5. Interaction Social learning theory (Bandura, 1969) suggests that a key component to learning is participation in communities of practice. People learn from seeing others perform tasks. The constructivist model (Bruner, 1966) offers similar perspectives. Learning begins within a certain social and cultural context, and students engage with one another and the instructor as they work to solve problems. Kuo et al measured learner-instructor interaction as the second most important factor influencing student satisfaction (Kuo, Walker, & Schroder, 2013, 30). C-RAC guideline 4(g) calls for "[c]ourse design and delivery [that] supports student-student and faculty-student interaction." Guideline 4(h) calls for "[c]urriculum design" and a "course management system [that] enable[s] active faculty contribution to the learning environment" (Higher Learning Commission, 2009). For courses with material that calls for relatively few interaction types, it was determined that a minimum of two might be reasonable. Most courses will apply more types, working towards the goal of establishing an active and engaging learning community. Examples of interaction types include discussions, quizzes, written assignments, email, student presentations, blogs, journals, group assignments, video conferences, and interactive tutorials. 3.1.6. Learner support standards A number learner support standards were also included in the rubric, including the prominent inclusion of contact information for technical help and other student services, a course schedule, clear explanations of grading standards and procedures, a list of required prerequisites, links to any required software or plug-ins, and instructions for accessing disability accommodations. 4. Process Developing a review process was particularly challenging, implicating all four of the primary goals identified here. On the one hand, reviews need to provide substantive and constructive feedback, while, on the other, the process should be reasonable in scope, with administrative requirements kept to practical limits. Both the developers and reviewers of courses can be encouraged to constructively engage with a process that focuses on valuable inputs and revisions, while undue administrative weight can draw their focus away from meaningful contributions. 4.1. Review Committees The DEAC discussed the question of how many participants to include on a review committee. On the one hand, a three reviewer committee could be expected to provide robust feedback, while also supporting a convenient tie-breaking mechanism should there be a difference of opinion over whether a standard is met. On the other hand, a two reviewer committee may also provide a reasonable amount of feedback, particularly if one reviewer is chosen from the developer's academic field and another from the course design professionals on the DE staff. Swann & Cripe 758 The two reviewer option was ultimately selected by the committee. The tie-breaking issue was resolved through a requirement that both reviewers agree that a standard is met in order for it to be considered met through the process. Of course, a two person committee can raise practical issues, if, for example, one reviewer does not participate at expected levels, or if the reviewer does not fully understand the process. Enabling one reviewer's judgment to result in a course not initially being approved may also raise issues. Through a consideration of the balance between resources and value, it was decided that a reduction in required faculty resources by one-third made it worthwhile to manage whatever issues arise from two person committees. 4.2. Supports A number of tools and resources were developed to support the process, including:  A web accessible form for reviewers to fill out and submit to complete their reviews  A set of guidelines for review commentary  A one-page summary of the review rubric  A more detailed document with a definition for each standard, guidance on applying it, sourcing of the standard in learning research, and links to tutorials and online resources that may be helpful  A summary sheet and tutorial on writing learning objectives  A summary sheet and tutorial on how to use recording software to make effective instructional videos  A tutorial on segmenting and integrating rich media content  A course developed on the university's LMS to collect and maintain these resources and to provide a step-by-step explanation of the review process The web-accessible review form consists of the sixteen review standards and their definitions, a selector to indicate whether they were met, and a text box for commentary to accompany each standard. Guidelines for commentary are provided as a bullet-point list at the top of the form. As the guidelines indicate, an explanatory comment is required for any standard judged by the reviewer not to be met. Comments are not required if a standard is met, but they can be provided if the reviewer has suggestions for improvement. Reviewers are encouraged to focus on the most significant potential areas of improvement and to keep their commentary at levels the developer is likely to find manageable. Packets of materials, including some of the above, were assembled and provided to developers and reviewers prior to their participation on a review committee. An initial meeting was scheduled to review the materials, explain the process, and answer any questions. A checklist was also developed for use by DE staff as a meeting agenda to ensure that key items were discussed. 5. Implementation Rolling out the process involved developing the supports listed above, creating a spreadsheet to track the status of reviews, soliciting volunteers to peer review, and arranging for initial meetings with reviewers and developers. The quantity of courses was initially high, requiring establishment of a review queue. Several adjustments were made as experience was gained during implementation. Initially, faculty determined when to turn in their courses for review, and peer and staff reviewers were then assigned, with a window of 4-5 weeks to complete their reviews. With staff time allotted International Online Journal of Education and Teaching (IOJET) 2018, 5(4), 753-766. 759 for the task, DE staff were often able to begin a review before the peer reviewer. In a portion of reviews, it was then discovered that the course was not likely to meet several standards and might need significant additional development. Examples include a course that has little structure or where content or assessment items are provided without instructions. In the determination of the review coordinator, the best course of action, in these cases, was to ask the faculty peer reviewer to pause the review, to provide informal feedback to the course developer, and to work with the developer to revise the course to include these fundamental elements. The process could then be started again and formal reviews completed, with the expectation that feedback through the formal reviews would further enhance the course. On the one hand, this strategy seems to have avoided unnecessary work on the part of reviewers, who would have otherwise completed formal reviews that included extensive documentation of course deficiencies, followed by revisions to the course by the developer, and then revisions to the reviews to reflect those substantive changes. It may have also streamlined the process for the developer, who had a single set of initial feedback to work with and assistance from an instructional designer to assess the feedback and make adjustments. On the other hand, it places a lot of emphasis on informal feedback from a single reviewer on the staff, without input from a faculty reviewer. On the whole, the advantages of this strategy may outweigh the disadvantages, but it needs to be implemented carefully to ensure the feedback is reasonable and constructive. Most courses submitted for review did not have fundamental issues that warranted a pause in the process, and nearly all courses successfully completed the process. As of this writing, 96 courses have been approved, with another 8 currently in review. From the perspective of DE staff, most developers and reviewers seemed to engage with the process constructively and were eager to share advice and apply suggested revisions. A few other anecdotal observations about implementation were shared by DE staff. For example, developers and reviewers seemed to benefit from personal support to a greater extent than from the range of documentary and online resources available to them. At the beginning of the process, a member of the DE staff visited their office to review the process with them and provided a packet of supporting materials. The availability of a staff member to answer questions and to assist with review steps at any point in the process was emphasized in these meetings. Documentary resources developed by DE staff seem to have been accessed through these meetings and subsequently utilized. By comparison, the online course about the process provided through the university LMS does not appear to have been widely used. It may have provided logistical support to DE staff, as a convenient repository of related materials, more so than direct support to faculty developers or reviewers. 6. Participant Surveys All course reviewers and developers were asked to provide feedback on the review process via a survey consisting of eleven statements and three open-ended questions. Respondents were asked to assess their agreement with the statements on a six-point scale ranging from strongly disagree to strongly agree. Their responses were converted to numeric values ranging from one (strongly disagree) to six (strongly agree). At least two of the eleven statements corresponded to each of the four primary goals of the process, and two additional statements reflected the overall purpose of the process. In the three open-ended questions, respondents were asked what they considered the best aspects of the process, the worst aspects, and any ideas for improvement they might share. Respondents were divided into three groups: 1. Those who developed one or more courses, but did not review another person's course (n=14) Swann & Cripe 760 2. Those who reviewed a course, but did not develop a course that went through the process (n=13) 3. Those who both developed and reviewed one or more courses (n=25) Of the 80 participants, 52 completed the survey, a response rate of 65%. Averages for each question, divided by group, are provided in Table 1. Table 1. Averages for quantitative questions Questions Groups Asked Relevant to Goal Group 1 (dev only) Group 2 (rev only) Group 3 (dev & rev) All 1. Quantity of feedback sufficient 1, 3 1 5.08 .. 5.52 5.37 2. Would like more thorough critique 1, 3 1 2.50 .. 2.20 2.31 3. Suggestions likely to improve course 1, 3 2 5.15 .. 5.48 5.37 4. Feedback overly critical 1, 3 2 2.57 .. 1.56 1.92 5. As reviewer, guidance sufficient 2, 3 3 .. 5.54 5.68 5.63 6. As developer, assistance with feedback & revisions sufficient 1, 3 3 5.14 .. 5.68 5.49 7. Process efficient (time spent on reviews & revisions) 1, 2, 3 4 4.79 5.00 5.72 5.29 8. More time consuming than anticipated 1, 2, 3 4 3.07 2.77 2.20 2.58 9. Process balanced (effort involved & value of feedback) 1, 2, 3 4 4.64 4.69 5.16 4.90 10. Feels student learning experience was improved 1, 3 .. 5.00 .. 5.12 5.08 11. Course improved as a result of process 1, 3 .. 5.14 .. 5.28 5.23 6.1. Goal 1: Feedback The first two statements were relevant to groups 1 and 3, but not group 2, since those who had only reviewed courses would not have received course feedback. The average for statement 1 is higher than most ratings provided through the survey, suggesting that developers found the quantity of feedback adequate. Statement 2 reflects a somewhat higher aspiration of providing International Online Journal of Education and Teaching (IOJET) 2018, 5(4), 753-766. 761 a sufficiently thorough critique. Lower values reflect a more favorable response, in this case. The average of 2.31 is below the mid-point, but not as close to the low end on the scale as the average for question 1 was to the high end, suggesting that respondents broadly felt the feedback was adequate, but in some cases felt they would have benefitted from additional feedback. 6.2. Goal 2: Constructiveness Statements 3 and 4 produced some of the strongest average values in the survey. Respondents tended to feel that suggestions provided through the process were likely to improve their courses (average of 5.37), and relatively few felt the feedback was overly critical (average of 1.92). The average for statement 3 is closer to the high end of the scale than the answers for 4 were to the low end, suggesting that some respondents did find the feedback overly critical. 6.3. Goal 3: Support The highest average values in the survey were given for support statements. Respondents had a strong sense of being given sufficient guidance with the process and assistance with course revisions. The average for reviewers (5.63) was highest in the survey and the average for developers (5.49) the second highest. 6.4. Goal 4: Efficiency Three statements were used to gauge the critical issue of process efficiency. Averages for all three were towards the desired end of the scale, but those trends were not as pronounced as those for the other three goals. Of the three efficiency statements, the strongest responses (average of 5.29) were found for question 7 about whether the process is efficient as a whole. Asked whether the process was more time consuming than expected, the average (2.58) was toward the desired (low) end of the scale, while reflecting substantial sentiment regarding a lengthy process. The process was judged as balanced between the effort involved and value of feedback, but convictions were not as strong (4.90) as they were for statements related to other goals in the process. 6.5. Broad Purpose Two statements focused on the broad goals of the process: the degree to which the respondent felt their course improved through the process and whether they thought student learning experiences were enhanced. Average responses were reasonably high—somewhat higher for perceived course improvement (5.23) than perceived student learning enhancement (5.08). These values are not as strong as those for most of the specific goals of the process. It may be more challenging to make gains in student learning than it is to achieve the individual goals identified for this process. Perhaps these results remind us to maintain some focus on our broad purpose, even as we work on specific aspects of the course development process that seem important to us. 6.6. Differences by Group Modest variations in overall averages for the 11 quantitative statements are discussed above, but there seem to be greater differences when responses are broken down by group. Ratings given by those who had both developed a course and reviewed at least one course were stronger on all questions than ratings by those who had only developed a course or only reviewed. This may reflect some selection bias, given that faculty were more likely to be asked to serve in both roles if they were already involved in some aspect of distance education. They may also have been more likely to agree to participate if they were already comfortable working in the distance environment. Swann & Cripe 762 The largest rating difference occurred for one of the two statements relating to the second goal, constructiveness. Those who only developed a course were more likely to view the feedback they received as overly critical than those who both developed and reviewed a course (a 1.01 point difference). Other differences were measured for two of the efficiency statements. Those who only developed courses were less likely to view the process as efficient in terms of the time spent on reviews and revisions (an average of .93 points lower) and more likely to find it more time consuming than they anticipated (an average of .87 points higher). Group comparisons are shown in Table 2. Statistical significance levels for the differences were calculated for all statements. The only statement where the differences were significant at the .05 level in two-tailed t-tests was statement 7 on process efficiency. Table 2. Group comparisons Questions Relevant to Goal Dev Only (avg) Rev Only (avg) Both (avg) Group 3 - Group 1 Group 3 - Group 2 t-Test (G1 & 3) t-Test (G2 & 3) 1. Quantity of feedback sufficient 1 5.08 .. 5.52 0.44 .. 0.23 .. 2. Would like more thorough critique 1 2.50 .. 2.20 -0.30 .. 0.42 .. 3. Suggestions likely to improve course 2 5.15 5.48 0.33 .. 0.38 .. 4. Feedback overly critical 2 2.57 1.56 -1.01 .. 0.07 .. 5. As reviewer, guidance sufficient 3 .. 5.54 5.68 .. 0.14 .. 0.44 6. As developer, assistance with feedback & revisions sufficient 3 5.14 .. 5.68 0.54 .. 0.18 .. 7. Process efficient (time spent on reviews & revisions) 4 4.79 5.00 5.72 0.93 0.72 0.04 0.02 8. More time consuming than anticipated 4 3.07 2.77 2.20 -0.87 -0.57 0.10 0.22 9. Process balanced (effort involved & value of feedback) 4 4.64 4.69 5.16 0.52 0.47 0.18 0.21 10. Feels student learning experience was improved 0 5.00 .. 5.12 0.12 .. 0.71 .. 11. Course improved as a result of process 0 5.14 .. 5.28 0.14 .. 0.66 .. The stronger values for Group 3 as compared to Groups 1 and 2 are mirrored in relatively dramatic differences in response rates between the groups. Those who both developed and reviewed a course had an 89% response rate, while those who only reviewed had a 57% rate and those who only developed had a 48% rate. Some combination of selection bias and differences in experience may be reflected in these results. Those with limited involvement in International Online Journal of Education and Teaching (IOJET) 2018, 5(4), 753-766. 763 the process seem to experience it as less balanced between the effort involved and value of feedback than those involved in both roles. Table 3. Response rates by group Group 1 (dev only) Group 2 (rev only) Group 3 (dev & rev) All Number of respondents 14 13 25 52 Total number surveyed 29 23 28 80 Response rate 48% 57% 89% 65% 6.7. Qualitative Responses Responses to the three open-ended questions were examined for common themes and inductively coded. Asked to identify the best thing about the process, the most commonly mentioned item was the personal support provided by DE staff. Respondents also cited the quantity and relevance of feedback received through the process and the course improvements it allowed them to make. Some felt the reviews gave credibility to their course by verifying that it meets quality standards. To a lesser extent, those in Group 2 (reviewers only) indicated that they liked being able to see the layout of other courses and the items their colleagues were including or emphasizing. The availability of clear instructions in process documentation was also mentioned. When asked about the worst thing in the process, the most frequent response was 'nothing.' The second most common response came from Groups 1 and 2 (reviewers and developers only), several of whom mentioned that the process was overly time consuming. All three groups indicated that making all their online materials ADA compliant was among the most difficult parts of the process, though some indicated they did not see this as a process issue. When asked for ideas for improvement, the most common reply suggested that the process was fine and did not need changing. Some suggestions were offered, but they did not coalesce around common themes. Someone in Group 2 (reviewers only) thought it would be helpful to receive reminders during the review process. Someone else made the helpful (and likely more universal) suggestion that reviewers receive payment for participation. One respondent in Group 3 indicated a desire for more time to develop a course, while another suggested providing faculty with a course in online teaching techniques supported by the university's LMS. 7. Analysis Responses to the three qualitative questions broadly reflect the quantitative responses, while providing further insight into them. Participants gauged the process as effective in quantitative measures related to the four primary goals, and qualitative responses largely support those assessments. Some goals appear to be more fully met than others. The feedback provided supports adjustments to the specific processes at the university, and it may provide valuable guidance for similarly situated universities, with some relevance for the variety of smaller and larger institutions. Questions about support received both the strongest scores on quantitative questions and the most positive comments in answers to qualitative questions. The greatest number of positive comments related to individualized support by the DE staff, with some additional comments on the clarity of instructions in the review rubric and other documentation. Individualized support offered through this process was broadly endorsed by DE developers and reviewers. An emphasis on faculty supports, particularly in-person guidance and assistance, are among the more valuable and impactful components of a DE review process. Swann & Cripe 764 The quantity and constructiveness of feedback were also supported, though there were some who found the feedback overly critical and others who felt they would benefit from a more thorough critique. The latter sentiment was somewhat stronger than the former, suggesting a significant desire by some for additional feedback. Developers who already have strong course fundamentals may find discussion of possible design improvements to be valuable. A course review process may be improved through a consideration of the needs of more advanced DE faculty, taking into account the drive for improvement in that population and the more detailed feedback and engagement with professionals on the DE staff they may find useful. Measures of process efficiency were also reasonably strong, but not as strong as average values for the other three goals. In particular, some found the process more time consuming than anticipated. On the one hand, it may not be advisable to adjust the process to reduce this perception dramatically. A robust process is likely to strike participants as requiring significant time and effort. On the other hand, comments from some participants report delays that would not be explained solely by the time and effort involved. Some may implicate the initial backlog in reviews, while others may reflect the addition of a review pause to the process. These findings support increased attention to the allocation of resources early in the implementation phase, but it is difficult to recommend any significant narrowing of the scope of DE reviews due to the need to maintain a robust process. While it would be possible to forego the initial course check, this would likely increase work for reviewers, who would need to review the course a second time after substantial revisions following their first review. It may be more advantageous to formalize the initial course check, with the understanding that full reviews do not begin until a course passes the initial check. It might also help to add a deadline, so all participants understand that the process closes by a certain date. Those involved in only one aspect of the process—development or reviewing, but not both—gave the lowest ratings on efficiency questions, suggesting they experienced the process as less balanced between the time involved and value of the feedback. Their average responses were weaker on all 11 quantitative questions. With this in mind, it may be worth contemplating the broad goals of the process and its role within the DE program. If one were to set a goal of introducing distance education to faculty in ways that encourage participation and demonstrate the potential benefits, it might be advisable to modify the process to provide more support and guidance to the less experienced faculty. This would suggest a possible fifth goal for the process: that of broadening the exposure of faculty to DE in ways that encourage further adoption of distance modalities. International Online Journal of Education and Teaching (IOJET) 2018, 5(4), 753-766. 765 References Baddeley, A. D. (1986). Working memory. Oxford, UK: Oxford University Press. Bandura, A. (1969). Social-learning theory of identificatory processes. In David A. Goslin (Ed.), Handbook of socialization theory and research (pp. 213-262). Chicago, IL: Rand McNally. Berge, Z. L. (2002). Active, interactive, and reflective learning. Quarterly Review of Distance Education. 3, 181-190. Bloom, B. S., Krathwohl, D. R., & Masia, B. B. (1956). Taxonomy of Educational Objectives: The Classification of Educational Goals (Vol. 1). N.p.: David McKay. Blumberg, P. (2009). Maximizing learning through course alignment and experience with different types of knowledge. Innovative Higher Education. 34(2), 93-103. DOI: 10.1007/s10755-009-9095-2 Bonwell, C. C., & Eison, J. A. (1991). Active Learning: Creating Excitement in the Classroom. New York, NY: Wiley. Bruner, J. (1966). Toward a Theory of Instruction. Cambridge, MA: Harvard University Press. Dick, W., Carey, L., & Carey, J. O. (2001). The Systematic Design of Instruction. N.p.: Pearson Higher Ed. Dresner, M., De Rivera, C., Fuccillo, K. K., & Heejun, C. (2014). Improving higher-order thinking and knowledge retention in environmental science teaching. Bioscience. 64(1), 40-48. DOI: 10.1093/biosci/bit005 Guidelines for the evaluation of distance education (on-line learning). (2009). Retrieved from Higher Learning Commission http://download.hlcommission.org/C- RAC_Distance_Ed_Guidelines_7_31_2009.pdf Huang, H. (2002). Toward constructivism for adult learners in online learning environments. British Journal of Educational Technology. 33(1), 27-37. Kauffman, H. (2015). A review of predictive factors of student success in and satisfaction with online learning. Research in Learning Technology. 23(1), 1-13. DOI: https://doi.org/10.3402/rlt.v23.26507 Kuo, Y., Walker, A., Belland, B., & Schroder, K. (2013). A predictive study of student satisfaction in online education programs. The International Review of Research in Open and Distributed Learning. 14(1), 16-39. DOI: http://dx.doi.org/10.19173/irrodl.v14i1.1338 Martin, F., Polly, D., Jokiaho, A., & May, B. (2017). Global standards for enhancing quality in online learning. The Quarterly Review of Distance Education. 18(2), 1-10. Mayer, R. E. (2008). Applying the science of learning: Evidence-based principles for the design of multimedia instruction. American Psychologist. 63(8), 760-769. Mayer, R. E., & Chandler, P. (2001). When learning is just a click away: Does simple user interaction foster deeper understanding of multimedia messages? Journal of Educational Psychology. 93(2), 390-397. Mayer, R. E., Dow, G. T., & Mayer, S. (2003). Multimedia learning in an interactive self- explaining environment: What works in the design of agent-based microworlds? Journal of Educational Psychology. 95(4), 806-812. http://download.hlcommission.org/C-RAC_Distance_Ed_Guidelines_7_31_2009.pdf http://download.hlcommission.org/C-RAC_Distance_Ed_Guidelines_7_31_2009.pdf https://doi.org/10.3402/rlt.v23.26507 http://dx.doi.org/10.19173/irrodl.v14i1.1338 Swann & Cripe 766 Moore, J. C. (2010). A Synthesis of Sloan-C effective practices. Journal of Asynchronous Learning Networks. 14(3), 24-45. Paas, F., Tuovinen, J., Tabbers, H., & van Gerven, P. (2003). Cognitive load measurement as a means to advance cognitive load theory. Educational Psychologist. 38(1), 63-71. Pawlowski, J. M. (2006). Adopting quality standards for education and e-learning. In Ulf- Daniel Ehlers & Jan Marting Pawlowski (Eds.), Handbook on Quality and Standardisation in E-Learning (pp. 65-77). Berlin: Springer. Ramdass, K. & Nemavhola, F. (2018). Quality practices: An open distance learning perspective. Turkish Online Journal of Distance Education. 19(1), 234-246. Reeves, T. (2006). How do you know they are learning?: The importance of alignment in higher education. International Journal of Learning Technology. 2(4), 294-309. DOI: 10.1504/IJLT.2006.011336 Smith, P. L., & Ragan, T. J. (2005). Instructional Design. (3rd ed.). New York, NY: Wiley & Sons. Southard, S. & Mooney, M. (2015). A comparative analysis of distance education quality assurance standards. The Quarterly Review of Distance Education. 16(1), 55-68. Spanjers, I. E., Wouters, P., van Gog, T., & van Merriënboer, J. J. G. (2011). An expertise reversal effect of segmentation in learning from animated worked-out examples. Computers in Human Behavior. 27(1), 46-52. DOI: 10.1016/j.chb.2010.05.011 Spanjers, I. E., Gog, T., & Merriënboer, J. G. (2012). Segmentation of Worked Examples: Effects on Cognitive Load and Learning. Applied Cognitive Psychology. 26(3), 352-358. DOI: 10.1002/acp.1832 Sweller, J., & Cooper, G. A. (1985). The use of worked examples as a substitute for problem solving in learning algebra. Cognition and Instruction. 2(1), 59–89.