SHORT PAPER CAN WEB THIN CLIENTS BE USED TO CREATE FLEXIBLE ASSESSMENT SPACES IN A HIGHER EDUCATION SETTING? Can Web Thin Clients be used to Create Flexible Assessment Spaces in a Higher Education Setting? http://dx.doi.org/10.3991/ijim.v9i4.4681 S. Webb1, M. Malik2 and M. Wilson1 1 University of Portsmouth, Portsmouth, UK 2 University of Portsmouth, Portsmouth, UK Abstract—For many HE institutions, a major factor inhibit- ing the wider deployment of computer-based objective as- sessment is the lack of suitable IT suites. The advent of pervasive wifi within HE campus environments, combined with the development of portable and low-cost browser- based devices, motivates the question of whether existing general-purpose teaching spaces can be used as venues for computer-based assessment. This work describes the results of a pilot study aimed at investigating some of the technical and logistical issues surrounding this question and evaluat- ing student perceptions of this particular implementation of an educational technology. Index Terms—Computer aided instruction, Computer based assessment, Courseware, Electronic learning. I. INTRODUCTION Objective testing, which requires students to provide a response to a question whose correct answer is predeter- mined, is particularly well-suited to forms of computer- based assessment that incorporate automated marking. If the test has been designed correctly then the marking of responses is free from subjective biases. The drawing of questions from large question banks offers an opportunity for students to practice and develop mastery in addition to adding a layer of security to an online test. Furthermore, with online tests students can receive immediate feedback on their performance, regardless of class size. For these and other reasons, in recent years many teachers in higher education have increasingly experimented with delivering computer-based objective assessments [1]. Such testing plays an obvious role in summative assessment [2], but online tests as a type of e-learning can be used in purely formative contexts [3] or as a combination of formative with summative [4]. Online tests are also used extensively for diagnostic testing (see [5] for a discussion of how basic mathematical concepts can be tested via mobile devices). Within many institutions, a limiting factor to increased take-up of computer-based objective assessment is the availability of suitable IT suites: purchasing large numbers of desktop computers is expensive, constructing venues in which to situate those computers is more expensive still. However, the advent of pervasive wifi within HE campus environments, combined with the development of porta- ble, low-cost, browser-based devices, suggests the possi- bility of transforming an existing teaching space into a venue that is appropriate for the delivery of computer- based assessment activities. In this paper we describe the results of a small-scale project that aimed to evaluate some of the technical, logistical and usability issues sur- rounding this possibility. II. THE PROJECT The massification of higher education within the UK and the increasingly diverse student populations entering university has meant that diagnostic testing is increasingly seen to be a useful tool in a lecturer's repertoire. When employed appropriately, and in conjunction with suitable support measures, diagnostic testing can assist in student progression and aid retention [6]. However, diagnostic testing typically needs to take place early in a student's first year of study, ideally during the induction period, and this is a time when an institution's fixed IT suites see par- ticularly heavy demand. This particular constraint moti- vated us to ask whether portable web thin clients might provide a solution. Web thin clients are relatively recent devices that pro- vide a web browser and rely upon web applications to provide general-purpose computation. The Chromebook is perhaps the most common example of a web thin client: it runs ChromeOS as its operating system and, although it has some local storage and can be used in an offline mode, it is primarily designed to be used whilst connected to the internet. The Chromebook can also be used as an ultrathin client for desktop virtualization, giving access to PC or Mac applications. Factors such as affordability, simple device manage- ment and the presence of a physical keyboard – factors that are often missing in tablet devices – have led to the increasing use of Chromebooks in the education sector. Indeed, in the third quarter of 2014, Chromebooks for the first time outsold iPads in US schools [7]. A further ele- ment in their adoption by US schools is that the devices are verified by both Smarter Balanced [8] and PARCC consortia [9] for online student assessments. Take-up in UK schools has been slower than in the US, although there is some evidence of increasing usage in primary and secondary institutions in the UK [10]. Although Chromebooks have been used extensively as secure platforms for online assessment within secondary education and tertiary education, little work has been published on their use within the higher education sector. An internet-based search of the IT and technology en- hanced learning websites of UK universities revealed that several institutions have purchased large numbers of iJIM ‒ Volume 9, Issue 4, 2015 69 SHORT PAPER CAN WEB THIN CLIENTS BE USED TO CREATE FLEXIBLE ASSESSMENT SPACES IN A HIGHER EDUCATION SETTING? Chromebooks and made them available for staff and stu- dents to borrow, but we found no evidence for systematic trials having taken place. This is somewhat puzzling, given the likely expectations of students progressing to higher education from an environment in which Chrome- books are increasingly used. In order to research the po- tential of Chromebooks in our particular question of inter- est – whether web-thin clients make it possible to create, in effect, peripatetic examination and testing spaces – we therefore decided obtained a small number of different models of Chromebook and work with academics and students to determine whether the devices met basic usa- bility and disability requirements. Building on feedback from that initial evaluation we then obtained a larger number (n = 30) of Chromebooks to evaluate their poten- tial for assessment-related tasks. In particular, we investi- gated: their suitability as a secure platform for testing students; their compatibility with delivering Moodle-based tests (our institution deploys the open-source Moodle as its VLE, but the results should independent of the particu- lar quiz engine being used); the administrative overheads involved in switching between an "exam mode" and a "classroom mode" (such a mode being necessary because, as discussed in Section 3.4 below, the economic case for deploying these devices is boosted if they can be used for interactive in-class use); wireless infrastructure require- ments; and logistical issues. III. RESULTS The following subsections present the results of the var- ious different elements of the project, including the out- comes of a pre-pilot phase and a larger pilot in which Chromebooks were used to deliver 15 exam sessions and a large number of in-class activities. A. Usability A small pre-pilot involving level 5 and level 7 engineer- ing students (n = 10) was conducted with three different Chromebook models in order to ascertain whether there were any significant usability issues surrounding their use in computer-based testing. The students were asked to take a sample test, which consisted of questions unrelated to the subject area being taught. The intention was not to test student knowledge, rather their experience of interact- ing with the full range of question types available in Moo- dle. The pilot relied completely on the Chromebook de- vices: no external pointing devices or keyboards were used. After the session, students were asked to complete a short survey relating to their experiences. The results of the survey showed that all students sup- ported the idea of using Chromebooks for CBT. All stu- dents registered a preference for a particular make of de- vice (the one with the largest screen; this device was also deemed to possess a better touchpad and keypad). Howev- er, students were encouraged to take advantage of a free- text field in the survey in order to make additional com- ments, and 40% of them (n = 4) did so. Analysis of these comments showed that some students would prefer access to a traditional mouse, since they were unfamiliar with Chromebook technology and found the experience of scrolling and selecting using a touchpad to be somewhat time consuming. Another comment was that students should be given the opportunity to practice on the devices before taking a test, in order to become familiar with them. This would be possible, for example, if the devices were used during teaching periods for formative in-class activities. Such use would also improve the economic case for purchasing Chromebooks (though note that, by defini- tion, it would be difficult to give students the opportunity to practice taking diagnostic tests if those tests were scheduled early in their transition to university). Some models of Chromebook replace the caps-lock key with a search key; this caused problems when students wanted to use capital letters or entered a password containing capital letters. The provision of an external keyboard overcame this issue. We also commissioned a report on device accessibility by a specialist disability support officer who noted that, at the current time, the main suppliers of assistive and sup- portive software do not support Chromebooks but that the variety of available apps, extensions and built-in system tools can be used to give a reasonable level of support. It was noted, however, that the built-in magnification acces- sibility option in no way replaces specialized software for the visually impaired. Furthermore, one make of device could not easily be opened with one hand since there was no lip; other makes were better on this score. Following this report, and reaction from students, the recommenda- tion was that the accessibility options should be set as the default. Given the encouraging student feedback from this small-scale pilot we decided to source 30 Chromebooks plus flight cases and a smart-charging secure cabinet in order to investigate other issues. B. A secure platform for high-stakes assessment? The educational version of the Chromebook has a blacklist function and a blacklist_exception function; taken together, these functions allow an administrator to block access to all URLs except those specified. We used this as the basis for developing an examination-based Google administration "container". In exam mode the device was locked down: upon logging in, the student was taken directly to an exam landing page and could access only those websites and applications specified in the blacklist_exception. The exam mode also disabled the devices' USB ports (although USB mice and keyboards could still be used) and SD card slot. The project team asked security specialists to attempt to access unauthorized sites while the device was in exam mode. The consensus was that the container provided sufficiently strong security to be employed in summative assessment tasks. The more established online exam platform within the authors' institution consists of a combination of PCs run- ning the open source SafeExamBrowser and an exam theme. The level of security provided by Chromebooks would appear to be at least as high as that given by SafeExamBrowser; students expressed a clear preference for Chromebooks over SafeExamBrowser. C. Compatibility with VLE-based tests The Moodle Quiz engine constitutes the main delivery platform for computer-based assessment within the au- thors' institution. This particular deployment of the Quiz provides test-setters with access to 20 different question types: the standard complement of Moodle question types, several of the question types developed by the Open Uni- versity and subsequently released to the community, plus a question type developed in-house. The project team 70 http://www.i-jim.org SHORT PAPER CAN WEB THIN CLIENTS BE USED TO CREATE FLEXIBLE ASSESSMENT SPACES IN A HIGHER EDUCATION SETTING? confirmed that all question types performed correctly on the Chromebook devices. Existing online examinations therefore required no modification to allow for Chrome- book-based exams. It should be noted, however, that Chromebooks do not support Java applications; this would affect some question types published by the Moodle community. D. Device management Economic considerations dictate that thin client devices are unlikely to be purchased purely for assessment pur- poses. Devices that can be employed for in-class purposes when not being used to deliver assessments constitute a more efficient utilisation of resources. Furthermore, using them in a classroom setting enables both students and academics to gain familiarity with the devices before being faced with them in the high-pressure setting of a high-stakes examination. To this end we developed two Google administration containers in order to support two different modes of operation. The exam mode, which locked the device down, was described in section 3.2. In classroom mode the device could be used freely and with- out restriction; students had access to their usual apps for use in class-based interactive sessions. (We shall publish our experiences of using the device as an interactive mo- bile technology in a classroom setting in a forthcoming paper.) Both staff and students raised privacy concerns regarding their inability to remove personal wifi account details from the devices while in classroom mode. This issue was resolved by adding a setting to the classroom mode profile so that, upon signing out and powering off, all user information and settings are erased. One academic involved in the pilot suggested the de- velopment of a third container to support a restricted mode of operation. The restricted mode would blacklist all web- sites except for those that a teacher planned to use during a session. The intention behind the restricted mode is that students would be unable to use the devices to visit inap- propriate sites during the sessions. This container is cur- rently under development. Switching a device from the normal mode to the exam mode was quick (typically 30 seconds per device) because an automatic login process for wifi was applied. However, although switching a device from the exam mode to the normal mode was found to be relatively straightforward, it was potentially time-consuming. The activity requires an administrator to "move" the devices into the appropriate container so that the appropriate Google policy could be applied. We found that it could take up to three hours for the devices to detect the changed policy. After this time, the devices had to be switched on individually and left for 15 seconds before rebooting. It then required a manual login to the wifi network on each device. It follows that if Chromebooks are to be used in summative assessment situations then those responsible for the management of the devices must be given sufficient notice of require- ments in order to ensure that the Chromebooks are in the correct container at the correct time. E. Wireless infrastructure As is the case with the vast majority of UK universities, the authors' institution deploys eduroam; this service pro- vides users with authenticated network logon and access to the Internet via a single wifi profile and set of creden- tials. An eduroam outage that coincided with a scheduled exam session highlighted the need for an active wifi con- nection when Chromebooks are involved: the exam ses- sion had to be rearranged. Following this episode we in- vestigated whether plug-in wifi routers or a dedicated exam wifi network would mitigate against such conflicts. However, due to restrictions imposed by JANET, the UK supplier of Eduroam, plug-in wifi routers are not an op- tion; and a dedicated network would still be dependent on the institution's infrastructure. We therefore concluded that the adaptation of business processes would be a better approach to minimizing problems caused by maintenance of the wireless infrastructure; in particular, eduroam maintenance could be performed in defined at-risk periods (during which time no exams should be scheduled) while an exams calendar could be published automatically to relevant members of technical support staff to assist with their planning. Network performance was monitored during various Chromebook-based assessment sessions and no problems were visible. The project team also undertook a wifi load test by using all 30 devices in exam mode and working through a specimen Moodle Quiz containing embedded video and audio. The session ran smoothly, with no ob- servable lag when playing media. The wifi load was moni- tored during this session; the network dealt easily with the load. We attempted to profile the session in terms of a local load benchmark. The university library has an above-average client connection count but bandwidth here is low since students are not browsing heavily; in halls of residence students stream HD media services, engage in online gaming, use person-to-person sharing as well as general surfing and so there is high-bandwidth use with fewer connections. Load during the Chromebook test session fell neatly between those extremes. The load test took place in a room with one wireless ac- cess point with the capacity to handle 60 simultaneously connected devices. The infrastructure worked as designed and therefore it is perhaps unsurprising that the test was successful. However, in a real exam situation it is possible that students might bring smart devices that are already connected to eduroam. This would represent an extra load on access points. It would therefore be prudent to ensure that the number of Chromebooks to be used in an exam session should be no more than half the capacity of a room's access point. With this caveat in mind, an eduroam would seem to provide an Internet connection that is as stable as a wired connection. If an institution possesses an enterprise-level wireless infrastructure then the institu- tion's rules, policies and procedures governing online examinations should not require modification to allow for Chromebook-based exams. F. Logistical requirements The project team purchased a lockable charging cabinet that offered a secure location to house the Chromebooks in addition to a power management system that enabled all devices to be charged at the same time. In real-world trials it was found that the Chromebooks could be used actively for ten hours without needing to be recharged (the standby time is of course much longer); this level of battery per- formance meant that academics could have confidence that the devices would be available for use by students throughout a typical working day. The charging cabinet was situated in a central location, from which the Chromebooks had to be transported to a iJIM ‒ Volume 9, Issue 4, 2015 71 SHORT PAPER CAN WEB THIN CLIENTS BE USED TO CREATE FLEXIBLE ASSESSMENT SPACES IN A HIGHER EDUCATION SETTING? number of different locations across the campus. Each Chromebook had a mass of 2.1kg. Although the cabinet was capable of being transported, the combined mass of charging cabinet and a full device load was 131 kg. It was thus deemed to be impractical to transport the cabinet to the examination or classroom venues. Two foam-lined flight cases were therefore purchased to transport the devices from the charging location to the venues where they were to be used. For many institutions, the logistical issues involved in transporting devices from a central location to spatially distributed sites might prove to be the biggest challenge in offering a centrally-supported examination service based on Chromebooks. For those institutions a more feasible and sustainable solution might be to devolve the service to departments and base the charging cabinets and devices closer to the point of use. IV. FUTURE DEVELOPMENT This work has looked at the potential of portable web thin clients to deliver objective test style examinations, which are the most widely implemented types of computer assisted assessment. On the surface, the same devices would appear to be suitable for permitting students to word process their traditional essay-type examinations. The use of word processors in essay-type examinations is well established in law schools in the US, but there are relatively few examples of this practice within UK univer- sities [11]. Given that examinations are stressful for those taking them, and given students' increasing familiarity with keyboards, it seems reasonable to offer students the choice of typing or handwriting their essays. This is planned to be the next stage in our investigation of the use of web thin clients in an examination setting. The current study has shown that the technology itself is unlikely to be an issue in such an investigation; of more concern will be issues of fairness. Will students opting to use a word pro- cessor be at an advantage, or perhaps even at a disad- vantage, over students opting for handwriting? It is known that in a given period of time students will typewrite more words than they will handwrite, and the amount of writing correlates with the essay mark [12]. On the other hand, students are unlikely to have taken a written examination using a computer, and it is possible that their approach to constructing an essay (planning, editing and revising it) will be different and lead to differences that might be picked up by the marker [11]. It is also important to un- derstand, and if necessary account for, the reactions of markers to word processed versus handwritten scripts [13, 14]. These will all be topics of future research. The portability and simplicity of these thin client devic- es opens up a new space in which innovation can occur, outside of the examination setting described above. By putting the devices into classroom mode, and allowing teachers and students the freedom to explore the possibili- ties afforded by the devices, we noted that a variety of technology enhanced active learning (TEAL) techniques were trialled. The devices were used to facilitate online polling, collaborative activities such as wikis, interactive mobile presentations and more. Our observations about this innovation will be published elsewhere. V. CONCLUSION We have shown that, within a suitably equipped wire- less infrastructure, web thin clients can be used to create flexible assessment spaces. The security and management features of such client devices are sufficient to provide an appropriate platform for high-stakes summative assess- ment, and an institution’s existing policies and regulations surrounding online assessments should require no amendments to take account of the fact that wireless as opposed to wired connections are being used. Usability and accessibility options are adequate, although students need to be given the option of using external pointing devices. With the exception of a specific subset of ques- tions, existing question banks can be used without amendment. However, in order to deliver an assessment service based upon web thin clients, particularly if those devices are also to be used for in-class use, care must be taken to understand the implications of using multiple device "con- tainers". In particular, the duration required to swap de- vices from one container to another must be factored in when setting up an online examination. Furthermore, the logistical issues outlined in section 3.6 must be discussed and addressed. REFERENCES [1] L. Gilbert, V. Gale, B. Warburton, and G. Wills, "Report on summative e-assessment quality (REAQ)" Southampton, JISC, Fi- nal report, May 2009 [2] P. Marriott, "Students’ evaluation of the use of online summative assessment on an undergraduate financial accounting module," Brit. J. Educational Tech., vol. 40, no. 2, pp. 237–254, 2009 http://dx.doi.org/10.1111/j.1467-8535.2008.00924.x [3] D.C. Henly, "Use of web-based formative assessment to support student learning in a metabolism/nutrition unit," Europ. J. Dental Educ., vol. 7, pp. 116–122, 2003 http://dx.doi.org/10.1034/j.1600- 0579.2003.00310.x [4] S. Voelkel, "Combining the formative with the summative: the development of a twostage online test to encourage engagement and provide personal feedback in large classes," Res. Learning Tech., vol. 21, 19153, 2013 [5] R.N. Kizito, "Pretesting mathematical concepts with the mobile phone: implications for curriculum design," Int. Rev. Research in Open and Distance Learning, vol. 13, no. 1, pp. 38–55, 2013 [6] J. Hodgen, M. McAlinden, and A. Tomei, "Mathematical transi- tions: a report on the mathematical and statistical needs of students undertaking undergraduate studies in various disciplines", York, Higher Education Academy, The HEA STEM project series, 2014 [7] M. Molnar "Chromebooks ascend in K-12 market to challenge iPads", Education Week, vol. 34, no. 12, pp. 10–11, 2014 [8] Smarter Balanced. (n.d.). "Test-taking devices and approved secure browsers," [Online]. Available: www.smarterbalanced.org/ test-taking-devices-approved-secure-browsers [9] PARCC (2015, January). "Technology guidelines for PARCC assessments version 4.4 – January 2015 update," [Online]. Availa- ble: http://www.parcconline.org/sites/parcc/files/TechnologyGuide lines-PARCCAssessments-January2015.pdf [10] N. Kobie, (2014, August 11). "Who’s buying Chromebooks? American schools," [Online]. Available: www.pcpro.co.uk/ news/education/390214/whos-buying-chromebooks-american- schools [11] N. Mogey and J. Patterson, "Stylistic differences between typed and handwritten essay examinations," in 15th CAA Computer As- sisted Assessment Conf. Ed. D. Whitelock, W. Warburton, G. Wills, and L. Gilbert, University of Southampton, 2012 [12] D.E. Powers (2005) "Wordiness: a selective review of its influ- ence, and suggestions for investigating its relevance in tests re- quiring extended written responses," [Online]. Research Memo- randum, Educational Testing Service. Available: http://www.ets.org/Media/Research/pdf/RM-04-08.pdf [13] D.E. Powers, F. Fowles, M. Farnum and P. Ramsay, "Will they think less of my handwritten essay if others word process theirs? Effects of essay scores of intermingling handwritten and word 72 http://www.i-jim.org SHORT PAPER CAN WEB THIN CLIENTS BE USED TO CREATE FLEXIBLE ASSESSMENT SPACES IN A HIGHER EDUCATION SETTING? processed essays," J. Educ. Measurement, vol. 31, pp. 220–233, 1994 http://dx.doi.org/10.1111/j.1745-3984.1994.tb00444.x [14] N. Mogey, M. Purcell, J. Paterson, and J. Burk (2008) "Handwrit- ing or typing exams – can we give students the choice?" [Online]. Available: www.caaconference.com/pastconferences/2008/procee dings AUTHORS S. Webb received his PhD in theoretical physics from the University of Manchester in 1988. He is a Senior Fel- low of the Higher Education Academy in the UK. In addi- tion to his work as head of operations for Technology Enhanced Learning at the University of Portsmouth (DCQE, University of Portsmouth, Portsmouth PO1 2PR) he is the author of several books on physics and astrono- my. (e-mail: stephen.x.webb@port.ac.uk). M. Malik received his MSc in Communication Sys- tems and Signal Processing from the University of Bristol in 2003 and MA in Online and Distance Learning from the Open University in 2012. He is a Senior Fellow of the Higher Education Academy in the UK. He is currently a Senior Lecturer (School of Engineering, University of Portsmouth, Portsmouth PO1 3DJ) with particular inter- ests in elearning. (e-mail: manish.malik@port.ac.uk). M. Wilson gained a BSc in IT & Society from the Uni- versity of Portsmouth in 2001. He is currently a Senior Educational Technologist (DCQE, University of Ports- mouth, Portsmouth PO1 2PR) specializing in assessment and mobile development. He is an avid supporter of Open Source technology and contributes to various projects, including Moodle. (e-mail: mike.wilson@port.ac.uk). Submitted, 01 May 2015. Published as resubmitted by the authors 20 August 2015. iJIM ‒ Volume 9, Issue 4, 2015 73 iJIM – Vol. 9, No. 4, 2015 Can Web Thin Clients be used to Create Flexible Assessment Spaces in a Higher Education Setting?