Australasian Journal of Educational Technology, 2015, 31(6). 685 A student-constructed test learning system: The design, development and evaluation of its pedagogical potential Fu-Yun Yu and Chia-Ling Su National Cheng Kung University, Taiwan Although research has found that student-constructed tests (SCT) have some potential for promoting cognitive elaboration, as yet there are no systems available to support the associated activities. This study designed and developed an SCT learning system guided by the ideology of contribution-based pedagogies, accentuating the principles of adaptability and flexibility. To assess its learning potential, a group of 54 student teachers used the system, and data on their perceptions, generated work, and online log files were collected and analysed. The constant comparative method, descriptive and inferential statistics, and content analysis were employed in the data analysis. Several important findings were obtained. First, a global and macro view accentuating the integration and inter-connectedness of the study material was obtained by the participants based on the SCT activity. Second, a significant proportion of the participants considered SCT as their preferred assessment and learning approach, highlighting its affective and cognitive potential, in comparison to the traditional teacher-constructed tests. Third, SCT encouraged a majority of students to generate questions involving cross-chapter topics and engage in question revision behaviour to different extents, indicative of knowledge integration and elaboration. Significance of this study and suggestions for instructional implementations, online system development, and future study are provided. Introduction Contemporary pedagogy and assessment approaches—student-generated questions and its recent developments The questions and problems used in drill-and-practice sessions, and tests for evaluating students’ level of understanding are mainly developed by subject matter experts and based around aspects of the study content that the experts think of relevance, importance, and interest (English, 1997). However academics, professionals, teachers, and students may view educational relevancy in very distinct ways (Aikenhead, 2008). The idea of enabling students to reflect on and discover what they think is relevant, important, and interesting in the study materials, and then to generate questions based on this for self or peer assessment purposes, is supported by a number of contemporary educational theories and assessment approaches (such as constructivism, self-regulation, self-determination theories and multiple assessments), and thus is attracting increasing attention from researchers and practitioners. This approach, known variably as student-generated questions, problem posing, student item construction, and student-developed assessment items, and hereafter termed SGQ (Yu, Wu, & Hung, 2014), is a notable cognitive and meta-cognitive strategy (Rosenshine, Meister, & Chapman, 1996). The learning benefits of SGQ have been well-documented, and the results of empirical studies on SGQ have generally been positive. These reported benefits include helping students to: • concentrate on the main ideas (Andre & Anderson, 1978), • reflect back on and deeply process the study material (Rosenshine et al, 1996; Yu & Liu, 2005), • develop a better and deepened understanding of the learned material (Brown & Walter, 2005; Lavy & Atara, 2010; Lavy & Bershadsky, 2003; Perez, 1985; Toluk-Ucar, 2009; Wong, 1985), • adopt divergent and flexible thinking modes (Andre & Anderson, 1978; Brown & Walter, 2005; English, 1997; Kojima & Miwa, 2008), • become more active and motivated learners (Chin & Brown, 2002; Lan & Lin, 2011; Barak & Rafaeli, 2004; Wilson, 2004), • engage in group communication (Yu & Liu, 2005), • enhance their abilities in follow-up problem-solving activities (Brown & Walter, 2005; Dori & Herscovitz, 1999; Perez, 1985). Australasian Journal of Educational Technology, 2015, 31(6). 686 To support the classroom use of SGQ and make good use of the various affordances of networked technologies, more than a dozen online learning systems have been developed over the last decade (Yu & Wu, 2012). One step further—from student-generated questions (SGQ) to student-constructed tests (SCT) Although the benefits of SQG with regard to promoting cognitive, affective, and social development are recognised, existing applications mostly treat it as a stand-alone activity. That is, naturally produced artifacts (i.e., student-generated questions), despite their potential for use, are rarely included in other learning activities and their educational potential examined (Yu & Chen, 2014). One promising possibility of such use is SCT. The similarities and differences between SQG and SCT in terms of the associated learning tasks, and the cognitive processes thus enacted by the latter to promote knowledge integration and elaboration, are explained in detail below. In terms of similarities, both SGQ and SCT have question-generation as the focal learning task. With reference to the set of criteria associated with question-generation (e.g., pertaining to the main ideas of the study material, clear wording and expression, accurate answers, appropriate levels of difficulty for the target audience) (Yu & Wu, 2013), question-generation enables students to engage in various cognitive and metacognitive strategies, such as attending to and locating personally deemed important and relevant materials in the study content, monitoring comprehension and assessing personal understanding of the study material, and framing, phrasing, and solving the questions generated (Yu, 2005; Yu & Hung, 2006). Information processing theory and metacognition theory both help to conceptualise and understand why these enacted mental processes (e.g., rehearsal, organization, elaboration, planning, monitoring, and evaluation) may promote cognitive and metacognitive development (Yu & Liu, 2008). SCT, on the other hand, involves an additional set of criteria and learning tasks. Specifically, the additional set of criteria associated with SCT (such as, complete coverage of main ideas, different weighting of test items in accordance to the relative importance of various main ideas covered, and the appropriateness of item sequencing) may stimulate further and deeper cognitive processing on the part of the learners, thus advancing cognitive development. In other words, while working towards the goal of test construction, students are also likely to engage in additional cognitive tasks, such as examining the overall scope of the learned materials, constructing a summary or synthesis, structuring/restructuring the learning material, and assessing the relative importance and difficulty level of different concepts to ensure their adequate representation (e.g., a comprehensive and balanced coverage of all the important topics) in the resulting test. In view of cognitive elaboration theory, these extra steps may prompt further cognitive processing and promote cognitive growth. Theoretical foundations supporting SCT—cognitive elaboration theory Researchers in cognitive psychology have long held that to help cognitive processing and structuring, the learner must engage in some sorts of cognitive elaboration (Wittrock, 1978). Examples of effective cognitive elaboration techniques include: coming up with illustrative explanations or self-derived examples, relating newly received information to previously learned topics of relevance, highlighting distinctive differences among seemingly similar concepts, constructing an overview or synthesis in various formats (e.g., in prose, graphics, or tabular form) (Hoffman, 1997; Reigeluth, 1992, 1999). Studies on elaboration techniques have generally found supportive evidence for its effects on retention of information (Pressley, McDaniel, Turnure, Wood, & Ahmad, 1987; Reigeluth, 1999; Seifert, 1994), reading comprehension (Mohammad, 2011; Oh, 2001), and clarification of the relationships among pieces of information (Wittrock, 1978). As can be expected, question-generation may help students to activate cognitive elaboration techniques, such as providing explanatory descriptions or examples, interconnecting related information, and comparing features among contrasting concepts. In addition to those activated by question-generation, students are also more likely to exercise other elaboration techniques when exposed to SCT. For instance, to facilitate and ensure a comprehensive and balanced coverage of all topics in the study material, a summary of the study content in written or mental form (e.g., organisation chart, outlines and mind map) may be constructed. Also, to ensure that the constructed tests adequately cover the intended cognitive Australasian Journal of Educational Technology, 2015, 31(6). 687 domains of the study material (e.g., remembering, understanding, applying, analysing, evaluating and creating) (Anderson et al., 2001), while recognising that different types of questions are better at assessing different cognitive levels and have different limitations (Gronlund, 1993), organisation strategies, such as a two-way specification table, may be used during the test setting and planning stage. It is extra acts such as these, prompted and mobilised by the SCT activity, that may promote further cognitive processing and development, as implied by cognitive elaboration theory. Despite the potential of SCT for the promotion of the learners’ cognitive processing and growth, to the best of the authors’ knowledge, no systems supporting SCT activity are currently available. Therefore a research project aimed at supporting SCT activities in the classroom was launched in 2011. In the following sections, the guiding ideology and design principles of the online SCT are first explained. An evaluation study to assess its pedagogical potential is then conducted, and the results presented. The design and development of an online SCT learning system The guiding ideology—contribution-based pedagogies (CBP), and the main components of the developed system At the core of CBP are constructivist and community theories of learning. CBP is based on the idea that a constructive way for students to learn is to engage them in creating and sharing learning resources, and thus contributing to each other’s learning (Collis & Moonen, 2001). With the advent of Web 2.0 technologies, CBP has become more popular in educational practice (Hamer et al., 2008). With CBP as the guiding ideology, the proposed SCT highlights designs that support the processes of creating and sharing. In particular, to allow students to be active participants and meaningful contributors to their own and peers’ learning, three main functions are highlighted: test-construction, test-assessment, and test-viewing. Each of these is briefly introduced below. Test-construction To realise CBP while taking advantage of the advantageous features of networked technologies (e.g., fast data processing and instant data display), SCT allows students to construct tests based on self-generated questions and in reference to peer-generated question (i.e., by citing). To construct a test, students first design the overall structure of a test in terms of the number of questions and scoring scheme for each question type (see Figure 1), which can be modified at a later stage, if needed. Figure 1. A screenshot of test structure design Students then proceed to the item compilation stage. By clicking on any question item already stored in the database, as shown in the SGQ window (including both self- and peer-generated questions, shown as individual boxes in the upper-left portion of Figure 2), students can review and revise any question item, if needed, before inclusion. Then, to include items in a test, students simply drag any targeted item(s) from the SGQ window to the SCT window (see the upper-right portion of Figure 2). Similarly, any unwanted item can be dragged from the SCT window back to the SGQ window. Also, at the students’ own discretion, new question item(s) of any question type can be generated at this stage by clicking on the add a question icon ( ) before moving on to the next stage. Australasian Journal of Educational Technology, 2015, 31(6). 688 Figure 2. A screenshot of the item compilation stage of test-construction based on self- and peer- generated questions Finally, at the item sequencing stage and space, students can determine and re-arrange the relative sequence of questions both within and among question types in a test before submission (see Figure 3). Figure 3. A screenshot of the item sequencing stage of test construction Test-assessment To further realise the idea of CBP, a peer-assessment element is included that enables the students to contribute to each other’s learning. Once selecting which test to assess from the list of tests window (Figure 4), students (i.e., the peer-assessors) are directed to the online peer-assessment form, where they can review information related to the targeted test first (top of Figure 5), and then give feedback by completing the form (bottom of Figure 5). In light of the fact that both quantitative ratings and descriptive comments from peers have been found to significantly predict the quality of student’s performance at the task, both individually and collectively, under a peer-assessment arrangement (Yu & Wu, 2013), space is reserved for quantitative and qualitative feedback. Also, to support focused, objective and constructive comments from the peer-assessor, a set of criteria to be considered during test construction are provided for reference. Australasian Journal of Educational Technology, 2015, 31(6). 689 Figure 4. The list of tests window for test-assessment Figure 5. Online peer-assessment form for test-assessment Test-viewing To promote learning by enabling students to learn from observing peers’ work, an observational learning space is created. To take advantage of the power of networked technology (specifically, the huge amount of storage space available), the developed SCT allows students to view not only the finished work, but also the process that leads to the peers’ final work (in this case, both the tests constructed and the assessments given by peers). In other words, different versions of the constructed test (left of Figure 6) and the Australasian Journal of Educational Technology, 2015, 31(6). 690 interactions between the test-constructor and test-assessor (right of Figure 6) with regard to any targeted SCT (middle of Figure 6) are accessible for viewing by individual students. This way, the test revision and negotiation process that leads to the refined and latest version of the peer-constructed test can be examined and possibly used as a model. Figure 6. A screenshot of test-viewing The core design principles—adaptability and flexibility To accommodate individual teachers’ instructional plans, while acknowledging students’ individual differences in learning styles and preferences at different points in time, the principles of flexibility and adaptability are adopted in the developed SCT. In other words, most of the built-in functions and designs in the SCT can be dynamically adjusted in real time or ahead of time. For example, while citing, assessing and viewing peer-generated questions as well as tests are highlighted and supported in SCT, instructors are allowed to set different ranges of reference accessible to specific groups for different learning activities. The instructor may restrict students’ access to peer-generated work in their own class, to other classes enrolling in the same course in the same semester or prior semesters, or provide no access at all (i.e., students construct tests entirely based on self-generated questions), in the phases of test-construction, test- assessment, and test-viewing. In addition, in any of the three main functions (i.e., test-construction, test-assessment, and test-viewing) the access students have can be activated or deactivated by the instructor at any given time. Decisions with regard to this will be dependent on the instructor’s estimation of the process that will provide the optimal learning gains in the available learning time. Moreover, generating questions of different types may engender different cognitive processes and exert different demands on cognitive capacity (Yu & Li, 2011). The proposed SCT allows the instructor to specify any combination of question types for an SCT activity to support the instructional goals for their specific classes. Currently, the most frequently observed questions types in classrooms (including yes/no, multiple- choice, matching, fill-in-the-blank, short-answer, and essay questions) are supported in the proposed SCT. Furthermore, as noted earlier, a set of clear criteria is crucial for objective assessments (Topping & Ehly, 2001). Nonetheless, different wordings and expressions may be more comprehensible or appeal to different groups of learners based on age, educational level, cultural background, and so on. The developed SCT allows individual teachers to import and directly adopt a set of default criteria to be included in the online peer-assessment form (i.e., quantitative feedback rating scale and qualitative feedback/comments, as shown in the middle and bottom-left portions of Figure 5), or add/modify/delete any criterion as appropriate for the target learning group. Finally, studies have confirmed that users’ preference for different identity modes varies across situations (Yu & Liu, 2009). In the developed SCT, the instructor can allow students to select the identity Australasian Journal of Educational Technology, 2015, 31(6). 691 representation modes of their choice (i.e., real-name, nickname, and anonymous) during test-construction and test-assessment (as shown in the middle of Figure 5), so that their psychological needs are attended to. The evaluation study Research questions To assess the pedagogical potential of SCT, an evaluation study was conducted to address the following three main research questions: 1. Does the SCT activity elicit thought and learning processes of a different nature or intensity on the part of the learners, as compared to those engendered by SGQ? And if so, in what ways? 2. Do students hold different preferences with regard to SCT and teacher-constructed tests as an assessment and learning approach? If so, what are their reasons for this? 3. Does the SCT activity promote further knowledge integration and elaboration in addition to that achieved with SQG? If so, in what ways? Participants and implementation procedures Fifty-four student teachers enrolled in an elective course (Instructional Principles) offered in a secondary teacher preparation program at one national university in Taiwan participated in the evaluation study. To benefit from what Pressley and Woloshyn (1995) termed an informed cognitive strategy—to promote the adoption and continued use of the introduced strategy, at the first class session the reasons for incorporating SGQ and SCT in the course were explained, as follows: to help students concentrate on the main ideas of the study material; to accustom them to monitoring their level of understanding while attending lectures; to help them become more proficient at question-generation and test-construction, which are regarded as essential skills that prospective teachers should acquire. To allow students to compare SGQ and SCT activities, while taking into consideration the task complexity of both, the participants first experienced SGQ and were then introduced to SCT at a later stage. A training session with models of appropriate SGQ performance, followed up by guided practice, was arranged at the first class session to equip students with the essential knowledge and skills associated with SGQ and the system. In addition, instructional elements identified by Rosenshine et al. (1996) as effective for SGQ implementation, including a cue card which summarizes the criteria for SGQ performance and the operational procedures for navigating within the system, were arranged for the students’ reference. At the first stage of the study, the students were directed to generate questions around the study content on a chapter-by-chapter basis (one instructional principle per chapter, with a total of eight chapters). As a general routine, following the instructor’s delivery of each instructional principle, the students were given 20 minutes in class to generate two to four questions on areas of the covered content that they regarded as important and relevant. Students were also asked to assess four to six peer-generated questions, and were given access to their peers’ work after class. Opportunities for revision and refinement of questions based on feedback received from the assessor and insights gained from observing peers’ work were also provided at this point. Following each online SGQ learning activity, whole-class feedback with regard to students’ performance at SGQ was given by showcasing three to five students’ work purposively selected by the teacher. At the second stage of the study, and as a culminating learning event, students were directed to construct a test (i.e., SCT) covering all instructional principles in this course on the basis of the questions they had already generated. If deemed necessary, the students could also generate new questions or revise existing ones at this point. To ensure that the students had the key competencies required to achieve this, a training session targeting SCT and associated functions in the system was arranged prior to the activity. Finally, similar to the first stage, students were given chances to access and assess peer-constructed tests after class. A flow diagram summarising the major implementation procedures of the evaluation study is given in Figure 7. Australasian Journal of Educational Technology, 2015, 31(6). 692 Figure 7. Implementation procedures of the evaluation study Measures To address the first and second research questions, students were asked to respond to the following three questions in the last instructional session: 1. What do you think of SGQ and SCT as learning tasks in terms of the learning behaviour and thought processes they respectively helped to activate during the course, and what do you think of any learning effects they individually helped to promote? And in what way did the SCT activity elicit different learning behaviour and thought processes from you, as compared to SGQ? 2. Which do you prefer as an approach for assessing your learning—SCT, traditional teacher- constructed tests, or no difference? Why? 3. Which do you think better promotes your learning of the study materials—SCT, traditional teacher-generated tests, or no difference? Why? In addition to students’ subjective perceptions, all questions contained in the final submitted versions of all constructed tests, together with their revision histories, were retrieved from the online database and analysed to yield objective measures to address the third research question—if and in what ways does SCT activity promote further knowledge integration and elaboration in addition to SGQ? Data analysis In accordance with the focus of the research questions and the nature of the collected data, three data analysis methods were employed: constant comparative method, descriptive and inferential statistics, and content analysis. Each is explained briefly below. First, students’ descriptive responses to the open-ended portion of the three end-of-session questions were analyzed using the constant comparative method developed by Glaser and Strauss (1967). In essence, induction analysis was employed, in which the development of the conceptual categories and relationships among them emerged from the data, and their subsequent extension and refinement were based on a simultaneous comparison of all responses collected (Lincoln & Guba, 1985). The steps suggested by Lincoln and Guba (1985) for processing descriptive data (i.e., unitising and categorising) were followed. Second, descriptive statistics and a chi-square (X2) goodness of fit test were used for the analysis of students’ selection with regard to their preferences for SCT or teacher-constructed tests as an assessment and learning approach. Descriptive statistics was also used to reveal if the SCT activity induced the students to refine their work (indexed as knowledge elaboration). Simply put, the online log files on the revision history of each of the generated questions in SCT were retrieved for each individual participant. The number of questions that were revised after the SCT function was made accessible to the participants was recorded and tallied to reveal the percentage of students who exhibited question revision behaviour, and the percentage of question items in each SCT that were revised. Finally, to reveal if any linkages among instructional principles were present in the SCT (indexed as knowledge integration), content analysis was carried out on the basis of which instructional principle(s) Australasian Journal of Educational Technology, 2015, 31(6). 693 each of the questions in the SCT addressed. Descriptive statistics were then used to reveal the percentage of students whose questions were of a cross-chapter nature, and the percentage of questions in each SCT that were like this was also calculated. Results and discussion of the evaluation study The results in response to each of the three proposed research questions are presented in the following sections. Does the SCT activity elicit thought and learning processes of a different nature or intensity on the part of the learners, as compared to those engendered by SGQ? And if so, in what ways? As noted by the participants, like SGQ, SCT helped to mobilise various cognitive and metacognitive processes (e.g., rehearsal, organisation, monitoring, reflection, adjustment), and led to heightened concentration, greater memorisation, enhanced understanding and better learning of the study material. Nevertheless, nearly 90% of the participants (N = 47, 87.04%) acknowledged that SCT differed from SGQ, with two main features emerging, as follows. First, in comparison to SGQ, the SCT activity helped the students to cultivate a global and macro view that highlighted the integration and inter-connectedness of the entire study material. Specifically, nearly two- thirds of the participants (64.91%) reflected in their written responses that rather than treating concepts, techniques, and methods discussed in respective chapters as individual, un-related parts, SCT provided a special learning opportunity for the fusion, linkage, and integration of those concepts, techniques, and methods located in different chapters. As triggered by the SCT activity, various learning behaviour and process, such as “further reviewing of all course materials and questions already generated, and then reflecting back on the experience as a whole,” and “self-evaluating what have learned as a whole” were exercised frequently, as reported by the participants, to help them gain a more “comprehensive,” “integrative” and “holistic” view of the focal concepts. The other significant feature associated with the SCT activity was related to test construction skills. Specifically, more than half of the participants (57.41%) highlighted “test structure” as one important aspect to be attended to during test construction. Other factors, such as “overall difficult level,” “item sequencing,” “weightings of different chapters,” “total number of questions in reference to the scheduled test administration time,” and “scoring scheme,” were some areas that the students felt needed be taken into account during SCT. Do students hold different preferences with regard to SCT and teacher-constructed tests as an assessment and learning approach? If so, what are their reasons for this? Quantitative data for Question 2 indicated that more than three-quarters of the respondents (77.78%) preferred SCT as the approach to assess their learning. Less than 10% (9.26%) preferred traditional teacher- constructed tests, and 12.96% expressed no preference for either approach. The results from a chi-square goodness of fit test indicated that the distribution was statistically significant at p < .01 (X2=48.11). The constant comparative method conducted on the students’ written responses to Question 2 emphasised the SCT’s affective and cognitive potential. With regard to the affective potential, SCT as being “less stressful” was mentioned by 16 respondents, and it being “novel,” “interesting” and “lively” were identified as reasons for it being favoured by 7 respondents. On the other hand, as for cognitive potential, 24 respondents noted the SCT’s effects in directing their attention to “applying” rather than merely “remembering what was learned.” Twelve respondents stressed the opportunities that SCT provided for “exercising higher-order thinking skills,” including cognitive strategy (e.g., “building linkage to daily life, other subject domains, or future work,” “locating the main ideas of the study content”), metacognitive strategy (e.g., “monitoring personal understanding and comprehension,” “modifying study habits or learning strategy”), “self-regulation” and “reflective thinking.” Finally, 5 respondents pinpointed the “meaningfulness” of SCT, appreciating “the opportunities and experiences SCT provided to practice and master question-generation and test construction skills that teachers are expected to possess.” Australasian Journal of Educational Technology, 2015, 31(6). 694 Quantitative data for Question 3 showed that more than 60% of the respondents (61.11%) felt that the SCT could better promote learning, while less than 10% (9.26%) stated that traditional teacher-constructed tests were better at this, and nearly 30% (29.63%) saw no differences between the two. A chi-square goodness of fit test found that the distribution was statistically significant at p < .01 (X2=22.11). The results from the constant comparative method applied to the students’ written responses to Question 3 basically reflected what has already been reported. In short, the students generally felt that the combined affective and cognitive potential associated with SCT helped to engender a “great sense of achievement” and “heightened learning interest,” which could then facilitate “better learning” and “cognitive development.” Does the SCT activity promote further knowledge integration and elaboration in addition to that achieved with SQG? If so, in what ways? In total, 805 question items were included in the 54 tests constructed by the participants, yielding an average of 14.98 questions per test. The results of the content analysis showed that nearly three-quarters of the participants (n = 40, 74%) had included questions of a cross-chapter nature. About a quarter of the participants (26%) constructed tests consisting of questions purely localised in one instructional principle. Further analysis indicated that among the 40 tests that included cross-chapter items, nearly half of the students (48%) had 10% to 20% of their questions based on cross-chapter topics (see Figure 8). Moreover, more than 20% of the participants had included questions of a cross-chapter nature that accounted for more than 20% of their constructed tests. Figure 8. Percentage of cross-chapter questions that the participants included in their tests The results on revision history from the log files indicated that almost all the participants (n = 53, 98%) engaged in question revision behaviour of some sort (e.g., refinement of wording, elaboration on expression by adding multimedia files or scenarios, and re-sequencing of alternatives within a question). In fact, only 1 student did not revise any of the questions included in his/her constructed test. Further analysis showed that among the 53 constructed tests, more than half of the participants (54%) revised more than half of their questions, and more than a quarter of the participants (26%) revised more than three-quarters (see Figure 9). Moreover, 4 students revised all of the questions in their tests. Figure 9. Percentage of revised questions 0% 10% 20% 30% 40% 50% below 10% 11%~20% 21%~30% 31%~40% 33% 48% 13% 8% 0% 20% 19% 26% 28% 26% Australasian Journal of Educational Technology, 2015, 31(6). 695 Conclusions SGQ is an effective comprehension-fostering and -monitoring strategy, and currently there are more than a dozen SGQ learning systems that are available (Yu & Wu, 2012). In view of cognitive elaboration theory, constructing tests is different from generating questions in terms of the tasks involved and cognitive processes enacted. Noting that no systems supporting SCT activity are available, and that the pedagogical potential of this approach for learning and assessment is yet to be explored and empirically examined, this paper reported the design, development, and evaluation of one such system, as guided by the ideology of CBP and the principles of adaptability and flexibility. Based on the evaluation carried out in this, several important findings were obtained. First, as reflected in the descriptive responses, SCT helped most of the participating undergraduates to attain a more holistic and macro view that focuses on building connections and integrating the concepts located in different chapters, while dealing with various technical issues associated with test construction. Second, a majority of the participating undergraduates preferred SCT over traditional teacher-constructed tests as an assessment and learning strategy, highlighting its affective and cognitive potential. Third, SCT encouraged a significant proportion of participating undergraduates to generate questions involving cross-chapter topics and engage in question revision behaviour of different extents, which are indicative of knowledge integration and elaboration, respectively. In view of the pervasiveness and extensiveness of question revision behaviour among the participants (i.e., enabling almost all participants to engage in question revision behaviour and with the majority revising more than half of their questions), SCT’s facilitative effect with regard to inducing knowledge elaboration cannot be over-emphasised. Significance of this study This study has some significance at the system development, empirical, and instruction levels. First of all, as mentioned, more than a dozen online learning systems have been developed to support SGQ activities over the past decade. Nevertheless, all existing systems focus on the generation of individual question items. To date, no systems have yet been developed that enable students to further construct tests on the basis of the generated questions. The system proposed in the current work, with a sound theoretical basis and design guidelines, is thus the first of its kind. Second, the augmented pedagogical value of SCT for knowledge integration and elaboration on the part of the learner, as confirmed in this study (i.e., enabling the majority of students to engender a global and macro view of the study material, generate questions involving cross-chapter topics, and engage in question revision behaviour), to the best of the authors’ knowledge, was empirically substantiated for the first time in this study. Third, while cognitive elaboration techniques have been found to strengthen knowledge networks, cognitive processing and structuring (Wittrock, 1978), information retention (Pressley et al., 1987; Reigeluth, 1999; Seifert, 1994), and reading comprehension (Mohammad, 2011; Oh, 2001), only a few elaboration techniques and strategies have been suggested and proven effective. With the confirmed effects of SCT for knowledge elaboration as found in this study (e.g., refinement of generated work by adding more explanatory description, including illustrative examples, and resorting to different presentation sequencing), this study helps to expand the limited effective elaboration techniques available for the instructor’s consideration. Finally, the teaching sequence that was carefully devised and clearly delineated in the evaluation study can serve as a valid reference point for instructors interested in integrating SCT in their classrooms. Suggestions and limitations of this study Several suggestions for instructional implementations, online system development, and future research directions are provided, as follows. First of all, SCT is a logical progression and extension of SGQ. In light of the solid empirical evidence supporting SGQ and the newly validated pedagogical value of SCT to further promote knowledge integration and elaboration, instructors are strongly advised to integrate SCT following an SGQ activity. The suggested teaching arrangement is especially pertinent for post-school education, in light of the fact that the integration of practice and theory, and cultivation of reflective thinking, self-regulation, and higher-order thinking supported by SGQ and SCT, is the ultimate goal. Australasian Journal of Educational Technology, 2015, 31(6). 696 Second, system developers interested in the idea and practice of SGQ are encouraged to include an SCT component to allow students to construct tests on the basis of this. It is expected that online systems would facilitate the implementation process and enable students to benefit from the augmented experience of making tests in a more fluid way (as compared to a paper-and pencil format). While this study has obtained some empirical evidence supporting SCT’s pedagogical potential, its subjects were undergraduate students enrolled in a teacher education program. The generalisability of this study to other contexts dealing with subject matters of a different nature should thus be exercised with care. To strengthen its empirical basis, future studies involving other groups of students and employing an experimental research method to examine the comparative effects of SGQ and SCT on learning would be fruitful. Finally, while the results from this study showed that the students preferred the SCT to traditional teacher-constructed tests, and perceived it as better supporting their learning, the comparative effects of SCT and teacher-constructed tests on learning can be tested empirically. Acknowledgements This work was supported by the Ministry of Science and Technology, Taiwan, ROC [NSC 102-2511-S-006 -003 -MY3] References Anderson, L. W., Krathwohl, D. R., Airasian, P. W., Cruikshank, K. A., Mayer, R. E., Pintrich, P. R., & Wittrock, M. C. (2001). A Taxonomy for learning, teaching, and assessing: A revision of Bloom's taxonomy of educational objectives. New York, NY: Pearson, Allyn & Bacon. Andre, M., & Anderson, T. (1978). The development and evaluation of a self-questioning study technique. Reading Research Quarterly, 14(4), 605-622. Retrieved from http://2020engineer.iss.utep.edu/world/Research%20Literature/self- questioning%20study%20techniques.pdf Aikenhead, G. S. (2008, February). Importation of science programs from Euro-American countries into Asian countries and regions: A recipe for colonization? Paper presented at the Conference of Asian Science Education, Kaohsiung, Taiwan. Retrieved from http://www.usask.ca/education/profiles/aikenhead/webpage/importation-science-programs.pdf Barak, M., & Rafaeli, S. (2004). On-line question-posing and peer-assessment as means for web-based knowledge sharing in learning. International Journal of Human-Computer Studies, 61(1), 84-103. doi:10.1016/j.ijhcs.2003.12.005 Brown, S. I., & Walter, M. I. (2005). The art of problem posing (3rd ed.). Mahwah, NJ: Lawrence Erlbaum Associates. Chin, C., & Brown, D. E. (2002). Student-generated questions: A meaningful aspect of learning in science. International Journal of Science Education, 24(5), 521-549. doi:10.1080/09500690110095249 Collis, B., & Moonen, J. (2001). Flexible learning in a digital world: Experiences and expectations. London: Kogan Page. Dori, Y. J., & Herscovitz, O. (1999). Question-posing capability as an alternative evaluation method: Analysis of an environmental case study. Journal of Research in Science Teaching, 36(4), 411-430. doi:10.1002/(SICI)1098- English, L. D. (1997). Promoting a problem-posing classroom. Teaching Children Mathematics, 4(3), 172-179. Glaser, B. G., & Strauss, A. L. (1967). The discovery of grounded theory: Strategies for qualitative research. Chicago, IL: Aldine Transaction. Gronlund, N. (1993). How to make achievement tests and assessments (5th ed.). New York, NY: Allyn and Bacon. Hamer, J. Cutts, Q., Jackova, J., Luxton-Reilly, A., McCartney, R., Purchase, H., Riedesel, C., & Sheard J. (2008). Contributing student pedagogy. ACM SIGCSE Bulletin archive, 40(4), 194-212. doi:10.1145/1473195.1473242 Hoffman, S. (1997). Elaboration theory and hypermedia: Is there a link? Educational Technology, 37(1), 57-64. Retrieved from http://www.etc.edu.cn/articledigest4/foreignarticles/elaboration-theory_and- hypermedia.htm http://www.amazon.com/Taxonomy-Learning-Teaching-Assessing-Educational/dp/080131903X/bigdogsbowlofbis/ http://www.amazon.com/Taxonomy-Learning-Teaching-Assessing-Educational/dp/080131903X/bigdogsbowlofbis/ http://2020engineer.iss.utep.edu/world/Research%20Literature/self-questioning%20study%20techniques.pdf http://2020engineer.iss.utep.edu/world/Research%20Literature/self-questioning%20study%20techniques.pdf http://www.usask.ca/education/profiles/aikenhead/webpage/importation-science-programs.pdf http://www.etc.edu.cn/articledigest4/foreignarticles/elaboration-theory_and-hypermedia.htm http://www.etc.edu.cn/articledigest4/foreignarticles/elaboration-theory_and-hypermedia.htm Australasian Journal of Educational Technology, 2015, 31(6). 697 Kojima, K., & Miwa, K. (2008). A system that facilitates diverse thinking in problem posing. International Journal of Artificial Intelligence in Education, 18(3), 209-236. Retrieved from http://miwalab.cog.human.nagoya-u.ac.jp/Paper/koj08.pdf Lan, Y.-F., & Lin, P.-C. (2011). Evaluation and improvement of student's question-posing ability in a web- based learning environment. Australasian Journal of Educational Technology, 27(4), 581-599. Retrieved from http://www.ascilite.org.au/ajet/ajet27/lan.pdf Lavy, I., & Atara, S. (2010). Engaging in problem posing activities in a dynamic geometry setting and the development of prospective teachers' mathematical knowledge. The Journal of Mathematical Behavior, 29(1), 11-24. doi:10.1016/j.jmathb.2009.12.002 Lavy, I., & Bershadsky, I. (2003). Problem posing via “what if not?” strategy in solid geometry — a case study. The Journal of Mathematical Behavior, 22(4), 369-387. doi:10.1016/j.jmathb.2003.09.007 Lincoln, Y. S., & Guba, E. G. (1985). Naturalistic inquiry. Beverly Hills, CA: Sage Publications. Mohammad, R. (2011). Use of syntactic elaboration techniques to enhance comprehensibility of EST texts. English Language Teaching, 4(1), 11-17. doi:10.5539/elt.v4n1p11 Oh, S.-Y. (2001). Two types of input modification and EFL Reading comprehension: Simplification versus elaboration. TESOL Quarterly, 35(1), 69-97. doi:10.2307/3587860 3 Perez, J. A. (1985). Effects of student-generated problems on problem solving performance (Doctoral dissertation). Teachers College, Columbia University, New York, NY. Retrieved from http://pocketknowledge.tc.columbia.edu/home.php/viewfile/14824 Pressley, M., McDaniel, M. A., Turnure, J. E., Wood, E., & Ahmad, M. (1987). Generation and precision of elaboration: Effects on intentional and incidental learning. Journal of Experimental Psychology: Learning, Memory, and Cognition, 13(2), 291-300. Pressley, M., & Woloshyn, V. E. (1995). Cognitive strategy instruction that really improves children's academic performance (2nd ed.). Cambridge, MA: Brookline Books. Reigeluth, C. M. (1992). Elaborating the elaboration theory. Educational Technology Research & Development, 43(3), 80-86. doi:10.1007/BF02296844 Reigeluth, C. M. (1999). The elaboration theory: Guidance for scope and sequence decisions. In C. M. Reigeluth (Ed.), Instructional-design theories and models: A new paradigm of instructional theory (Vol. II, pp. 425-454). Mahwah, NJ: Lawrence Erlbaum Associates. Rosenshine, B., Meister, C., & Chapman, S. (1996). Teaching students to generate questions: A review of the intervention studies. Review of Educational Research, 66(2), 181-221. Retrieved from http://www.doe.virginia.gov/instruction/virginia_tiered_system_supports/training/higher_ed/tch_stude nts_generate_questions_review.pdf Seifert, T. L. (1994). Enhancing memory for main ideas using elaborative interrogation. Contemporary Educational Psychology, 19(3), 360-366. doi:10.1006/ceps.1994.1026 Toluk-Ucar, Z. (2009). Developing pre-service teachers understanding of fractions through problem posing. Teaching and Teacher Education, 25(1), 166-175. doi:10.1016/j.tate.2008.08.003 Topping, K. J., & Ehly, S. E. (2001). Peer-assisted learning: A framework for consultation. Journal of Educational and Psychological Consultation, 12(2), 113-132. doi:10.1207/S1532768XJEPC1202_03 Wilson, E. V. (2004). ExamNet asynchronous learning network: Augmenting face-to-face courses with student-developed exam questions. Computers & Education, 42(1), 87-107. doi:10.1016/S0360- 1315(03)00066-6 Wittrock, M. C. (1978). The cognitive movement in instruction. Educational Psychology, 13, 15-29. doi:10.1080/00461527809529192 Wong, B. Y. L. (1985). Self-questioning instructional research: A review. Review of Educational Research, 55(2), 227-268. doi:10.3102/00346543055002227 Yu, F. Y. (2005). Promoting metacognitive strategy development through online question-generation instructional approach. Proceedings of International Conference on Computers in Education 2005, Nanyang Technological University, Singapore, 564-571. Yu, F. Y., & Chen, Y. J. (2014). Effects of student-generated questions as the source of online drill-and- practice on learning. British Journal of Educational Technology, 45(2), 316-329. doi:10.1111/bjet.12036 Yu, F. Y., & Hung, C-C. (2006). An empirical analysis of online multiple-choice question-generation learning activity for the enhancement of students’ cognitive strategy development while learning science. Lecture Series on Computer and Computational Sciences: Recent Progress in Computational Sciences and Engineering, Chania, Crete, Greece, 585-588. Yu, F. Y., & Li, M-H (2011). Effects of different types of online student question-generation on learning. Proceedings of International Conference on Computers in Education 2011, Chiang Mai, Thailand, 768-770. http://miwalab.cog.human.nagoya-u.ac.jp/Paper/koj08.pdf http://www.ascilite.org.au/ajet/ajet27/lan.pdf http://pocketknowledge.tc.columbia.edu/home.php/viewfile/14824 http://www.doe.virginia.gov/instruction/virginia_tiered_system_supports/training/higher_ed/tch_students_generate_questions_review.pdf http://www.doe.virginia.gov/instruction/virginia_tiered_system_supports/training/higher_ed/tch_students_generate_questions_review.pdf Australasian Journal of Educational Technology, 2015, 31(6). 698 Yu, F. Y., & Liu, Y. H. (2009). Creating a psychologically safe online space for a student-generated questions learning activity via different identity revelation modes. British Journal of Educational Technology, 40(6), 1109-1123. doi:10.1111/j.1467-8535.2008.00905.x Yu, F. Y., & Liu, Y. H. (2008). The comparative effects of student question-posing and question-answering strategies on promoting college students’ academic achievement, cognitive and metacognitive strategies use. Journal of Education & Psychology, 31(3), 25-52. Yu, F. Y., & Liu, Y. H. (2005). Potential values of incorporating multiple-choice question-construction for physics experimentation instruction. International Journal of Science Education, 27(11), 1319-1335. doi:10.1080/09500690500102854 Yu, F. Y., & Wu, C. P. (2013). Predictive effects of online peer feedback types on performance quality. Educational Technology and Society, 16(1), 332-341. Retrieved from http://www.ifets.info/journals/16_1/29.pdf Yu, F. Y., & Wu. C. P. (2012). Student question-generation: The learning processes involved and their relationships with students’ perceived value. Journal of Research in Education Sciences, 57(4), 135- 162. Yu, F. Y., Wu, C. P., & Hung, C-C. (2014). Are there any joint effects of online student question generation and cooperative learning? The Asia-Pacific Education Researcher, 23(3), 367-378. doi:10.1007/s40299-013-0112-y Corresponding author: Fu-Yun Yu, fuyun.ncku@gmail.com Australasian Journal of Educational Technology © 2015. Please cite as: Yu, F-Y., & Su, C-L. (2015). A student-constructed test learning system: The design, development and evaluation of its pedagogical potential. Australasian Journal of Educational Technology, 31(6), 685-698.