wilson.pdf Australian Journal of Educational Technology 2003, 19(3), 311-322 Validation of an Internet delivered and analysed test of cognitive function for use in web based psychology courses Peter H. Wilson RMIT University Paul Maruff Mental Health Research Institute of Victoria In this study we investigated the use of Internet technologies for administering and analysing a test of cognitive function in a web based psychology course. The aim was to determine whether the cognitive test actually measured the same functions when administered in three different settings: a supervised university based laboratory, an unsupervised university based laboratory, and a remote (or off campus) setting. Results showed that the different data collection processes generated similar data sets: the profile of responses over the four different tests of cognitive function were preserved across the different assessment settings, and improvements that occurred with practice were also consistent. Thus, the dataset can be used confidently to model key psychological principles, despite the added experimental noise created by having participants sit the tests in different settings, using different computers. The present findings highlight the potential use of web based technologies for delivering laboratory experiments in core psychology courses. Introduction Scientific experimentation is a cornerstone of education in psychology at a tertiary level. Experimentation forms the basis of the scientist/practitioner model that is endorsed and preferred by professional bodies such as the Australian Psychological Society (APS) and the American Psychological Association (APA). It is also the basis for the development of numerical, statistical, and critical thinking skills implicit in undergraduate training in psychology, skills that equip graduates well in highly competitive job markets, across a range of employment fields. 312 Australian Journal of Educational Technology, 2003, 19(3) It is recognised increasingly that the addition of web based components to university psychology courses offers great advantages to students who study both on and off campus, and also to lecturers who can now develop new and innovative models for course delivery. The ability to disseminate high quality curricula to large numbers of students enables academic units to maximise teaching resources and provides a vehicle to update course content at regular intervals (Vodanovich & Piotrowski, 2001). Considerable development has already taken place in the use of web based applications to deliver lecture and tutorial material to university students generally and psychology students specifically. In contrast, development of laboratory exercises for web based learning modules has been slow. Indeed, this is characteristic not only of psychology but also of other disciplines in science including physics (Forinash & Wisman, 2002) and engineering (Lemckert & Florance, 2002). One of the reasons for the under-development of web based psychology lab exercises is that many of these require that human subjects actually undergo some type of assessment, and in this assessment students can act as both assessor and subject. In the first instance, this allows data to be collected from a sufficient number of participants to ensure that the statistical power of experimental design clearly demonstrates the effect under investigation (Cohen, 1998). A second important aspect of the testing component of psychology lab exercises is that it allows students to develop the interpersonal skills and strategies necessary for the assessment of human subjects. While skills of this type are refined further in higher and professional degrees, for the many psychology students who do not continue with their studies this is the only training or experience they receive. Therefore, if web based psychology courses are to expand, there needs to be some experimental modules in which aspects of thinking or behaviour are assessed in actual human subjects, be they the students themselves or naïve participants. Furthermore, such modules should measure aspects of thinking or behaviour that are relevant to undergraduate training in psychology, be deliverable to off campus students as effectively as to on campus students, and provide a means whereby data collected from different students can be pooled and analysed quickly and then communicated back to students for the purpose of report writing (McGraw, Tew & Williams, 2001). In this manner, the training imperatives of the discipline can be preserved, highlighting most particularly to students the process of experimental research and its associated theoretical and practical issues. In the current study we investigated the potential application of a web delivered computerised test of cognitive functions - specifically attention and memory. We sought to determine whether the cognitive test actually measured the same functions when used in a supervised university based Wilson and Maruff 313 laboratory, an unsupervised university based laboratory, or an off campus setting (that is, by students studying a course by distance education). Importantly, we needed to test whether the different data collection processes were each able to generate similar data sets and, hence, preserve the integrity of the underlying concepts that are the basis for the “class” exercise. To achieve this we determined whether the profiles of responses over the four different tests of cognitive function were preserved across the different assessment settings and, second, whether any improvement that occurred with practice on the tests operated the same way across all settings. Method Participants The data from three different groups of students all enrolled in second year psychology courses were used in this study. The supervised university laboratory group consisted of 21 students (age range 21-35 years, mean age = 24) enrolled at RMIT University. This group was defined by the presence of a laboratory tutor during testing, who was free to answer questions about software installation and file transfer. The unsupervised university laboratory group consisted of 25 students (age range 21-42, mean age = 26) enrolled at La Trobe University. This group was defined by the absence of a laboratory tutor during testing; hence, students did not have the opportunity to ask questions about the use of the technology. The off campus group consisted of 25 students (age range 20-35, mean age = 24), also enrolled at La Trobe University. This group were without any form of technical support, but rather completed the battery by relying entirely on online resources. All students completed their web based laboratory exercises as part of their lecture series in a core second year unit on Cognitive Psychology. Entrance requirements for the psychology degree and the demographics of students enrolled at each university program are regarded as similar. Materials The CogStateTM computerised cognitive test battery was used to measure cognitive function in all students. The battery is used in many psychological applications including the cognitive assessment of pilots (Westerman, Darby, Maruff & Collie, 2001), athletes with sports related head injury (Collie, Darby, & Maruff P, 2001), and the effects of aging (Collie & Maruff, 2002). As well, there is now a growing scientific literature that both validates the test and provides context for the students (Collie, Maruff, Darby & McStephen 2003; Darby, Maruff, Collie & McStephen, 2002). Downloading the battery is free but CogStateTM generally charges a nominal fee for data collation when servicing industry. 314 Australian Journal of Educational Technology, 2003, 19(3) Within the CogStateTM test battery, stimuli for each task consist of playing cards drawn from a standard deck of 52, and all tasks are presented in the context of a game. Stimulus presentation - card type revealed face up - randomises both card selection for each task and correct response for tasks involving binary choices. Before presentation of each task, both written and interactive graphic instructions are presented on screen. These instructions disappear when the participant has made three successive correct responses to practice items, and the scored task then begins. Responses are indicated by pressing one of two keys on the computer keyboard ('d' or 'k') which represent 'yes' or 'no', respectively. All tasks present simple questions requiring a forced choice response – eg., "Is the card red?" A buzzer indicates abnormally slow or anticipatory responses, that is, failures to respond within 3500 ms or responses faster than 100 ms, respectively. Correct responses receive no auditory feedback. Response speed is recorded to the nearest 1 ms, while accuracy is given by the number of correct responses; the former was the focus of this paper. There are minimal or no practice effects and very high test-retest reliability (Collie et al., 2003; Darby et al., 2002). Users can familiarise themselves with the task requirements by practice tests until comfortable and confident with the battery. The four tasks used were: 1. Simple detection task (Detect). A button press is required when a playing card turns face up. 2. Identification task (Identify). A single card is presented face down in the middle of screen. When it turns face up, the person must indicate whether it is red by making a yes or no decision. 3. Matching task (Match). Two packs are used, initially presented face down. When cards appear, the person must make a binary choice depending upon whether the two cards shown are the same colour. 4. Working memory task (Remember). This is a one-back task in which the person is asked to decide whether the currently displayed card is the same as the previously presented card. For each task students are asked to respond as quickly and accurately as possible. The performance measure used is the average reaction time (RT) to trials on which a correct response has been given. Before statistical analysis, this data is normalised using a logarithmic base 10 transformation. Wilson and Maruff 315 Procedure Testing on the CogStateTM battery took place during a laboratory session entitled, The Effect of Attentional Load on Response Time. All subjects were given an information sheet outlining the study, but initially without reference to specific aims and hypotheses. Students completed the CogState battery by running the program on either a laboratory computer or their own computer at home. For those completing the battery in the laboratory, the program was loaded from CD, while for those at home it was downloaded from the web (www.cogstate.com) onto their home PC. In each case, the program was installed according to instructions provided in an executable file. A data file was generated automatically by the download and was sent via email to the laboratory supervisor; this file was safe from editing. The procedure for each group is summarised in Table 1. Table 1: Summary of the administration and reporting procedure used for each group who completed the CogState test battery. Setting Responsible educator Hardware to which test was down-loaded Instruction Results – data transfer Results summary Supervised university lab Lab Tutor Psychology lab computers Complete the CogState test twice in 2 hours Collected by lab tutor from computers Provided by lecturer in lab Unsuper- vised university Tutor Any university computer Complete the CogState tests twice in 2 hours Emailed to tutor Provided by lecturer in class Off campus Lecturer Home or office computer Complete the CogState tests twice in 2 hours Emailed to lecturer Provided in emailed report Data analysis Before any group analysis was performed, each individual’s RT was recorded for each correct response made on each of the four cognitive tasks. These RTs were then normalised using logarithmic base 10 transformation and a mean of the log10RTs was calculated individually for each combination of task (4) and assessment time (2: Time1, Time 2). To determine whether the effect of practice was equivalent across settings, a practice score was computed for each individual by subtracting the average RT for a specific task at Time 1 from the average RT on the same task at Time 2. Thus each individual student provided one practice score for each task. For each task the practice scores were compared between the three settings using a series of one way analysis of variance (ANOVA). Details of ANOVA can be found in Howell (2002). 316 Australian Journal of Educational Technology, 2003, 19(3) Change in performance Figure 1: Group mean (+SD) practice score (in s) for each task performed in a supervised university setting, an unsupervised university setting, and an off campus setting. The second analysis determined whether the pattern of performance on the four CogState tasks was the same for all assessment settings. Therefore, each individual’s mean RT (log10 transformed) averaged over time of assessment was submitted to a 3 (setting) x 4 (task) ANOVA with repeated measures on Supervised uni Unsupervised uni Off campus Supervised uni Unsupervised uni Off campus Supervised uni Unsupervised uni Off campus Supervised uni Unsupervised uni Off campus Detect Identify Match Remember Wilson and Maruff 317 the task factor. The test of the hypothesis concerned the presence of any interaction between task and setting; a significant interaction would indicate that the effect of setting was not uniform across the different tasks. Results Figure 1 shows the average practice effect for each CogState task across the three settings. The mean practice effects were all close to zero and there were no significant differences between settings on any of the cognitive measures (Detect: F(2,68)<1; Identify: (F(2, 68)<1); Match: F(2,68)=1.2, p>0.05; Remember: F(2, 68)<1). Figure 2 shows the interrelationships between the different tests across the three settings, averaged over time. The ANOVA indicated a significant effect for task (F(3,13)=25.1, p<0.001) but no significant effect of setting (F(2,11)=1.1, p>0.05), nor setting by task interaction (F(3,13)<1). Investigation of the significant effect of task with dependent samples t-tests indicated that when averaged across settings, the speed of simple detection was significantly faster than the speed of identification decisions, which, in turn, was significantly faster than the speed of matching decisions. There was no significant difference between the speed of matching and the speed of remembering decisions. Figure 2: Group mean log10RT (s) for each task across the three test settings Supervised uni Unsupervised uni Off campus Detect Match Detect Match Detect Match Identify Remember Identify Remember Identify Remember 318 Australian Journal of Educational Technology, 2003, 19(3) Discussion The results of the analysis indicate that individuals were able to grasp the requirements of the cognitive task and perform it optimally, irrespective of the setting in which the test was downloaded and performed. Furthermore, the pattern of results obtained in the present study is consistent with findings published previously that also show that the CogState performance measures are associated with no practice effects and in which the speed of correct responses is slowed according to the cognitive demands of the different tasks (Collie et al., 2003; Darby et al., 2002). In all cases the speed of simple detection is considerably faster than the speed of identification decisions which is, in turn, faster than the speed of matching decisions. Interestingly, t-test revealed that the speed of matching decisions was not shown to differ from the speed of decisions made from working memory (ie., the remember task). Consideration of the task requirements indicates that in healthy young adults the speed with which objects can be matched in terms of their spatial properties is equivalent to the speed with which they can be matched in time (see also Collie et al., 2003). More generally, results are also consistent with work showing that existing web based technologies are sophisticated enough to permit the timing accuracy necessary to run chronometric experiments (McGraw, Tew & Williams, 2000). Model trends in the data that serve to illustrate key principles are replicable, despite the added experimental error (or noise) created by having participants sit the tests in different settings, using different computer hardware. The present set of results serve to illustrate a number of core principles that fit well with current models of behaviour and their applications in different domains of psychology. These principles include response slowing with increasing task demands, faster decision times for identification decisions than matching decisions, and the equivalence of the thinking time necessary to match stimuli that occur at different points in time and those that occur (concurrently) in space. More specifically, the pattern of results illustrates issues pertinent to experimental psychology (eg., the effect of information load on performance), to cognitive neuropsychology (eg., different processing times indicating the involvement of more complex neurocognitive networks), neuropsychology (e.g., brain disorders in which the interrelationships identified in the current study are disrupted), and to research design and analysis in psychology (e.g., the potential confounding effects of practice, and the statistics necessary to determine whether a given performance measure has changed over time or differs from another measure by virtue of cognitive load) (see also McGraw et al., 2000). Wilson and Maruff 319 Results serve to highlight the efficacy of using interactive lab exercises in experimental psychology delivered via a web based medium. Several aspects of the software technology and computer user interface may explain the success of this approach. In the first instance, it is clear that all students in the unsupervised settings were able to download the CogState software successfully and initiate its application. Informally, no student reported difficulties in this regard, highlighting the use of clear download instructions on the source website. It has been shown that “download delay” can impair learning and performance under some circumstances (Davis & Hantula, 2001); for example, time spent on task learning decreases with large delays, particularly for inexperienced users. Download delay was not, however, a constraining factor in the present study. It is important to note here that most students involved in this study were acquainted with online learning environments (e.g. BlackBoardTM) during their first year of undergraduate study. We believe that basic experience in downloading online resources (eg. plugins like QuickTimeTM) certainly enhanced the ability of students to follow download and software installation instructions in the present study. This type of experience may provide the type of orientation necessary to ensure success when students participate in online laboratories. A formal comparison between experienced and non-experienced online learners would prove instructive in this regard. Secondly, the graphic design of the computer display is an effective medium to present (meaningful) stimuli and to communicate instructions clearly, within the context of a cognitive psychology laboratory. Results show that the integrity of the interface is not compromised in any significant way by remote delivery (see also Najjar, 1998) - the pattern of responding to visual stimuli of varying complexity and decision load is maintained across settings. In other words, the informational load and usability of the stimulus display is maintained despite variations in the immediate learning environment. As a result, participating students are assured of a consistent database that, in turn, enables them to interpret the pattern of results according to the same principles of human information processing. This level of consistency enhances communication between students in local and remote settings, particularly during the report writing phase where data is linked with past research and theoretical arguments are formulated. We also believe that the concurrent use of written (or semantic) and visual prompts served to communicate clearly the types of responses that were required of participants. The transparency of this presentation is consistent with a dual coding model of information processing that finds broad support in the field of cognitive psychology (Paivio, 1991). From the perspective of this model, it is argued that there are two ways to represent 320 Australian Journal of Educational Technology, 2003, 19(3) concepts: (i) through a mental image, and (ii) through a verbal representation. Both forms of representation operate in concert, enhancing the ability to retrieve information. If the function of one system is reduced in some way (perhaps due to incomplete cues or the dominance of one processing modality over another), the other form of representation may compensate, allowing new information to be encoded in a usable and retrievable form. Importantly, presenting task instructions using a dual format is thought to be of particular benefit to naïve participants, the sample of choice in experimental studies (Blake, 1977). Whether the same type of information can be communicated effectively to younger participants (children and young adolescents) using the same web based technologies is an empirical question; this is relevant to laboratories where undergraduate students are recruiting other people online in order to explore developmental issues. All indications are that older children and adolescents can decipher task cues of the type used in this study and make effective use of both semantic and pictorial information (Bjorklund, 2000). It remains to be seen whether more complex task instructions can be communicated equally well as those presented here using web based technologies. We cannot discount the importance of the instructor’s ability to clarify instructions on site and to help encourage participants to maintain their attention and to actively process key task cues (Burns, 1992). The communication of task instructions online may be challenged for decision making tasks involving multiple levels of analysis, from simple feature detection to more elaborate decision making in a real world (or virtual world) setting. This issue awaits further exploration. In sum, the present findings highlight further the potential use of web based delivery in core psychology courses. In addition to its now common use in lecture delivery and presentation of simulations in tutorials, the medium is well suited to the remote delivery of core laboratory work. The technology can clearly communicate task instructions and provide a workable interface with the user, regardless of the type of setting, as well as providing a robust data set. Student participation in the research process and the integrity of the data set, in turn, reinforce principles of research design and theory that are integral to the discipline. Acknowledgements We thank Michael Butson for his insightful comments on earlier versions of this paper. We thank the students of RMIT and La Trobe University for their involvement in the study. Wilson and Maruff 321 References Blake, T. (1977). Motion in instructional media: Some subject-display mode interactions. Perceptual and Motor Skills, 44, 975-985. Bjorklund, D.F. (2000). Children’s thinking: Developmental function and individual differences (3rd ed.). Belmont, CA: Wadsworth. Burns, D.J. (1992). The consequences of generation. Journal of Memory and Language, 31, 615-633. Cohen, J. (1988). Statistical Power Analysis for the Behavioural Sciences. Hillsdale, NJ: Lawrence Erlbaum and Associates. Collie, A., Darby, D.G., & Maruff, P. (2001). Computerised cognitive assessment of athletes with sports related head injury. British Journal of Sports Medicine, 35(5), 297-302. Collie, A., & Maruff, P. (2002). An analysis of systems of classifying mild cognitive impairment in older people. Australian and New Zealand Journal of Psychiatry, 36(1), 133-140. Collie, A., Maruff, P., Darby, D.G., & McStephen, M. (2003). The effects of practice on the test performance of neurologically normal individuals assessed at brief test- retest intervals. Journal of the International Neuropsychological Society, 9(3), 419-428. Darby, D.G., Maruff, P., Collie, A., & McStephen, M. (2002). Mild cognitive impairment can be detected by multiple assessments in a single day. Neurology, 59, 1042-1046. Davis, E.S., & Hantula, D.A. (2001) The effects of download delay on performance and end-user satisfaction in an Internet tutorial. Computers in Human Behavior, 17, 249-268. Forinash, K. & Wisman, R. (2002). Simple internet data collection for physics laboratories. American Journal of Physics, 70, 458. Howell, D. C. (2002). Statistical methods for psychology (5th ed.). Boston: Duxbury. Lemckert, C. & Florance, J. (2002). Real-time internet mediated laboratory experiments for distance education students. British Journal of Educational Technology, 33, 99-102. McGraw, K., Tew, M.D., & Williams, J.E. (2000). The integrity of web-delivered experiments: Can you trust the data? Psychological Science, 11, 502. McGraw, K., Tew, M.D., & Williams, J.E. (2001). Psychology Experiments. TechTrends, 45, 7-8. 322 Australian Journal of Educational Technology, 2003, 19(3) Najjar, L.J. (1998). Principles of educational multimedia user interface design, Human Factors, 40(2), 311-323. Paivio, A. (1991). Dual coding theory: Retrospect and current status. Canadian Journal of Psychology, 45, 255-287. Vodanovich, S. J, & Piotrowski, C. (2001). Internet-Based Instruction: A National Survey of Psychology Faculty. Journal of Instructional Psychology, 28, 253-255. Westerman, R., Darby, D., Maruff, P., & Collie, A. (2001). Computerised cognitive function testing of pilots. Australian Defense Forces Health Services Journal, 2, 29-36. Dr Peter H. Wilson Department of Psychology and Disability Studies RMIT University City Campus PO Box 2476V, Melbourne Vic 3001, Australia Email: peter.h.wilson@rmit.edu.au Paul Maruff The Neurophysiology and Neurovisual Research Unit Mental Health Research Institute of Victoria Parkville, Melbourne, Australia