Grounded Theory as an approach to studying students' uses of learning management systems Graham Alsop and Chris Tompsett Learning Technology Research Group, Kingston University email: g.alsop@kingston.ac.uk, c.p.tompsett@kingston.ac.uk This paper presents the first phase of a qualitative study of students' use of a Learning Management System (LMS). A group of students at Kingston University with experience of two different systems were afforded the opportunity to study the relationship between the interface to an LMS and the usability of the system. A 'Grounded Theory' methodology (Glaser and Strauss, 1967) was selected to model the students' framework, as independently as possible from the views of other stakeholders in the LMS. The approach still offered the potential for generalization despite working with a small group of students. In Grounded Theory, data collection and theory-building drive each other reciprocally until a stable 'theory' is reached. In an ideal world this research would be reported at the end of the whole process but it is considered that there is more value in producing less definitive results at this stage rather than a complete theory in one year's time (Oliver, 2000). Even at this stage of the research, it is clear that the students' conception of an LMS is categorically different from that held by the other stakeholders responsible for implementing the system. Identification of these differences and integration of some findings into future developments are critical even if, in the view of the researchers, further development of the 'theory' is required The findings from this phase validate the use of a grounded approach. They suggest that systemic issues regarding the integration of educational activities with other aspects of everyday life are fundamentally more important than issues in interface design. The students make effective use of remote access facilities but only where this offers a clear comparison with activities that already form part of studying at university. Such comparisons are pragmatic and are seldom defined in terms of 'educational' benefit. Introduction The study was designed to examine issues relevant to planning the full-scale implementa- tion of the Learning Management System (LMS) across the university. Two aspects of 63 Graham Alsop and Chris Tompsett Grounded Theory for studying students' uses of learning management systems usability were of particular interest: whether the system could be improved through changes in the design of the interface and whether a student's ability to use an LMS successfully was determined by technical abilities developed either at, or before, university. Students from two courses had experience of using both WebCT and Blackboard. A study using these two groups offered the opportunity to investigate how differences in the interface had affected the usability of the system for students with a relatively high degree of technical competence. An LMS offers different benefits to the various stakeholders in the system. For manage- ment an LMS can provide operational benefits, such as integration with other management systems (student records and finance, etc.) as well as strategic opportunities, for example to increase student numbers by decreasing the time spent by each student within the university. Such benefits do not necessarily offer educational advantage to the students. For a student an LMS would appear to offer few technical advantages beyond those that are available free through other Internet providers offering support for more generic on- line communities. Both WebCT and Blackboard provided similar functions as learning environments accessed through a Web portal. Each offered links to a number of integrated functions supporting document management, both synchronous and asynchronous communication, and integration with email services. Any educational advantages were essentially mediated by the design of the interface. The arrangement and labelling of icons and links to the various facilities controls the students' expectations of what activities they could be engaged in. If the learning management systems were substantially the same, then differences would only be realized if the students could anticipate the benefits that might be available and could successfully navigate to those different services. Simple differences in the usability of the interface could provide significant educational gains. The researchers' own experiences with Blackboard suggested that there was considerable potential for redesign. This research focused on students studying Geographical Information Systems (GIS) in their final year of two courses, either for a B.Sc. or an HND. These students had all used Blackboard during 2001 and WebCT during 1999-2000. WebCT and Blackboard were both offered with the normal suite of services from within the university. Only Blackboard was made available through remote access (with password protection). Although the students were supported by an LMS at each stage, the courses were still structured as conventional undergraduate courses with attendance required for lectures and other workshop activities. Methodology The researchers were aware that their perception of what was critical in the design of an LMS might not be seen as important by the students. The selection of a methodology that was relatively immune to the expectations of the researchers was critical. Selection The first stage in the research was the selection of a methodology appropriate to the outcomes that were required and the constraints of the particular study. The most critical requirement was the importance of capturing the students' own perceptions and understanding of the systems that they were using. 64 ALT-J Volume 10 Number 2 Many research methodologies expect the research to be conducted within an objective, conceptual model that is framed by others external to the subject group. Several stakeholders, other than the students, have an interest in the performance of the LMS: university managers, system designers, system implementers, course lecturers and, of course, ourselves as researchers. Research that is based on an external framework may well provide evidence that a system is not working - but will seldom provide answers as to why or how the system fails to work. This requirement implied that a qualitative approach, grounded in the students' own conceptual frameworks, would be essential. The second requirement was that the methodology had to support the potential of building a theory about what was being studied. This would enable the results of the research to inform the next stage of the development in the use of the LMS. This requirement would exclude any methodologies, such as ethnography, which did not conform to this. A third class of methodologies was excluded by the constraints of the study. Some qualitative approaches use an initial subject-centred qualitative phase as grounds to generate relevant questions for a larger statistical analysis. The timing of the research in two phases would not produce any results over the time-scale available as all the students with experience of both systems would have left by the end of the year and the group would not have been large enough for a statistical analysis. Our principle interest lay in providing a rich description of the range and variety of student perceptions and the factors that could be altered to produce different behaviours. The overall research model selected was Grounded Theory (Glaser and Strauss, 1967; Glaser 1978; Strauss and Corbin, 1990; Miles and Huberman, 1994). A more detailed discussion of this selection process is provided elsewhere (Alsop and Tompsett, 2002). Grounded Theory The Grounded Theory method requires that the full research process is grounded in the data that is collected. As with most qualitative methods it accepts that the researcher cannot remain external to the collection and interpretation of data but is inherently bound up in the process of research. The process of collection and interpretation is cyclical, with low-level analysis of data providing grounds for simple levels of interpretation, which then provide the basis for reinterpretation of existing data and collection of additional data. It is an iterative process that focuses on the subject's perspective but also accepts that the researchers cannot approach the subject without bias or personal viewpoints. The implication that the results will be dependent on the researcher's implicit involvement is managed through careful recording of the process of interpretation, where the researcher's initial interests, subsequent comments and possible interpretations are treated not just as information external to the analysis, but are recorded as data within the project itself. The possibility that different researchers might not agree on results is as a logical consequence of a constructivist epistemology. The rigour of any resulting 'theory' is established by the ability of other researchers to audit the stepwise sequence of interpretation, whilst recognizing that individual bias and interpretation would not necessarily be the same. The degree to which the researchers share (or should share) the same values and frameworks will determine the extent to which different researchers would accept the same conclusions. 65 Graham Alsop and Chris Tompsett Grounded Theory for studying students' uses of learning management systems The development of a theory is cyclical too, using existing data to build a theory and using the developing theory to guide data collection, increasing the depth of interpretation and the coverage of the theory across subjects. The first stage is to generate a catalogue of the terms and concepts used by the subjects (termed 'open coding')- This catalogue is then used to generate an 'axial model' (Strauss and Corbin, 1990: 96), a time-sequenced framework that establishes the interrelationships between the concepts within the open coding. This develops the interpretation of the subjects' perception of the problems that were being solved, for example the conditions controlling what was possible, the choices that were open to them and their assessment of what was successful and what was not. When this appears to be effective in structuring the data collected this model is then summarized by the identification of a single bounding concept, termed the 'core category'. This core category encompasses the framework within which all of the subjects' perceptions can be included. In reality there is a constant cycle between the last two stages; the intention is to develop a theory for which the core category provides a summary that allows all the data to be interpreted within the axial model and hence the core category. The axial model and core category are validated as a theory through a recursive process termed 'theoretical sampling'. In this phase concepts and issues that are already relevant in the current axial model are used to search for new cases to be integrated. The new data that is collected may well introduce new concepts or variables that enrich the open coding stage, provide changes and complexity to the axial model or even require a reconsideration of the core category. When this process stops, a process of uncertain length, the research is considered to be complete and the model is said to have reached 'theoretical saturation'. The outline of the process suggests that existing literature in the field is ignored but this would be illogical if the theory is, itself, to have predictive value. Concepts and models from literature are treated here as frameworks that have been created by other stakeholders outside the immediate focus of the research. There is no direct implication that they should apply to the conceptual framework used by subjects. Existing literature is treated as a further source of material to be used in developing theoretical saturation. Awareness of possible frameworks from the literature, termed 'theoretical sensitivity', allows possible issues to suggest areas for data collection, as long as there is sufficient evidence from what has been collected already. The first phase This paper covers the first stages of the analysis: the open coding, early axial coding and considerations for the core category. Whilst the research process is not complete, the convergence of the cyclical method to a stable theory offers the opportunity to comment on aspects of the developing theory at a stage at which those aspects become sufficiently well defined to be of significance. In this case the requirement to establish a definitive theory must be balanced against the potential utility of that information to stakeholders. Even at this stage of the research, it is clear that the students' conceptions of an LMS are categorically different from those held by the other stakeholders in the system. A delay in publication to develop and refine a 'theory' will have little value if decision-makers proceed to implement large-scale systems unaware that their expectations of how students will use the system is far from reality. 66 ALT-} Volume 10 Number 2 Data collection The particular method for generating data was derived from an earlier qualitative study (Bliss and Ogborn, 1977) that used semi-structured interviews to collect and analyse the experiences of first-year physics undergraduates. Each session was carried out during time allocated to a normal lecture or workshop. An introduction was given to the purpose of the research and why the students had been chosen. The method was described as a technique for collecting their approaches to using the LMS that was designed to avoid interference from the views of other stakeholders in the university's system. Students were given details of the process by which their contributions would remain anonymous and were allowed the option of withdrawing from the process. Students were then asked to write an account, with as much detail as they could remember, of the occasion on which the use of one particular LMS had been 'the most rewarding educational experience'. As they completed their own individual account they were asked to summarize this account with a single word or short phrase. As four or five students finished they were formed into a group and completed the first phase by reviewing, without discussion, the accounts of the other students in the group, summarizing each in turn. They then discussed the accounts between themselves to agree on a common word or phrase to summarize the set of accounts within the group. This is less constrained than a semi-structured interview, but earlier experiments with a similar topic by one of the researchers has demonstrated that it can produce remarkable consistency between individuals and groups. Each of the groups then repeated the cycle with a different starter question - the occasion on which the use of one particular LMS had been 'the worst educational experience'. In general the method worked as anticipated, though not all students confined themselves to single examples. If time allowed, students were then asked to contribute in two further ways to building the model. The first asked for a specific comparison of the different interfaces. The second asked them to identify any issues to which they or their group had failed to refer but which might indicate key issues that should be included in the model within the data collection process. This open structure to the collection of qualitative data was designed to elicit responses that would be framed within language understood by other students. Descriptions written directly for us could be filtered or adapted to make them appropriate for lecturers. Requesting a narrative account supported a simpler transfer to an axial model. The request for individual summaries checks that the accounts make sense to the students and, with a collective discussion, provides a starting point for developing a core category rooted in the subjects' perceptions. The intention was for students to produce an account that was acceptable or understandable by the student community. The data collection process was supplemented with field notes from the researchers throughout the process. It is not possible to collect data and interpret it in separate phases, as interpretation of what is collected cannot be 'turned off': for example the comment 'not exclusively about the technology' was recorded after the first session of data collection even before transcription began. Contemporaneous notes on ideas that occur provide a mechanism for identifying these interpretations and, by recognizing the source, of tracking and auditing the influence of individual bias on the research. Notes are also kept on any _ Graham Alsop and Chris Tompsett Grounded Theory for studying students' uses of learning management systems extraneous issues affecting data collection. In the first session, for example, it was noted that the offer of tea and coffee to students appeared to remove tension. The discussion below focuses on the language used by the students in these accounts. Issues of 'theoretical sensitivity' were excluded at this stage. Open coding The analysis of the full sample collected so far (49 accounts) was conducted using the software package Nudist 4. This provided effective facilities for tracking the progressive searching and coding of accounts, the sorting and resorting of coding groups along with the parallel recording of coding notes. Some of the students followed closely the request to describe a single event with some detail whilst others provided wider but shallower accounts. A typical account (transcribed with minor changes for legibility but with original spellings and expressions recorded) is presented in Figure 1. When using blackboard, groups can be set up so you can communicate with people in group work. This involves talking to the people directly using chat or emailing them, but also being able to exchange files.This is very useful especially as people work in different places and times to others. It also enables people to communicate easily without having t o find the ID numbers. It also means people can continue t o develop work without actually having to meet up to exchange files, which is very useful for people who don't need t o come into the building. Personal summary: communication with others Summaries of this account by other members of group: communication facilities, easy communication, communication tool Group summary (combining all four accounts): communication Figure I: A typical student account Perhaps the best experience of blackboard I have had is that I can get my lecture notes /course materials from home (internet) which has been useful when I have not managed t o get a print-out of the materials when in university. This remote access proved useful when I needed some lecture notes to do a piece of work and I was able t o get hold of, and print the course materials out at home thus meaning I didn't have to go into university to do it Personal Summary: Access from home Summaries of this account by other members of group: Remote access of notes, copy and print the notes, lecture notes Group summary (combining all four accounts): communication Figure 2: A further example of a student account 68 ALT-J Volume 10 Number 2 The consistency within the summaries of the account in Figure 1 was repeated in almost all cases and gave a clear indication that the individual accounts were well understood within the student community (see Figure 2 for an additional example). The consistency of language and terminology used by the students in their individual accounts also provided a strong indication that the particular model of data collection was working effectively. Even the simplest reading of the accounts reveals a terminology and language focused on being able to (or alternatively failing to) access resources in a remote-access mode when the students needed to or preferred to. One researcher specifically notes at this stage: 'stories seem to be about flexibility and choice - students are working in semi-distance mode.' Within this common theme, the different resources available seem equally significant (for example, lecture notes, assignment details, etc.), and the 'reasons' for access were varied. Coding and notes Interpretation (and recording of notes) inevitably occurs as soon as a researcher has access to the data (as with the comment above). Open coding progresses from that point through a structured approach, interleaving analysis and coding of the data collected with recording of notes and commentary. Interpretative comments were attributed to a specific point in the analysis and elaborated using explicit terminology from the cases (for example, 'several students use "choice" words - need, have to, useful, unable, wasting, access (choice), repair (missing lectures)'). This generated, in turn, a further analysis of the cases - for example a search through all the cases for examples where that specific terminology is used and extending the classification to a wider set of cases than those that triggered the initial comment. It was then possible to re-examine these cases in more detail, or juxtapose the new coding with a previous classification - or to review the cases where 'choice' did not seem to play a role. Absence of specific language, where the researcher's personal expectation was not met in reviewing text, was also considered a valid starting point for search, such as, 'No student mentions learning specifically - why?' In such cases searches with no results were also recorded (as with: 'August 1st 2001 searched for "learning" nothing found; August 15th 2001 searched for "learn*" nothing found'). At this stage the study was most remarkable for the absence of significant categories of language. Little comment or reference was made on specific issues that were assumed to be important in the original design, either interface issues or the relevance of technical skills. Remote access, the result of a policy decision at Kingston University, overrode any other consideration of implementation or design. What the data did not say As the students' accounts failed to produce anticipated commentary on either aspect of usability, careful consideration was given to whether this resulted from the design itself. As commented above, the positive and negative accounts (described in what follows as 'open' accounts), omitted almost all terms or issues relating directly to human-computer interface design; only one, triggered by the 'worst experience' question, could be interpreted as a lack of closure (commenting on a lack of confirmation after an assignment was delivered electronically through Blackboard). The extent to which this could be a consequence of the questions that were asked was a consideration but was rejected after analysis of some of 69 Graham A/sof> and Chris Tompsett Grounded Theory for studying students' uses of learning management systems the additional comments that were requested. When asked for comments comparing the two systems, some of the students readily demonstrated an understanding of human- computer interface issues (see Figure 3). Blackboard is much better than WebCT, it has the modules separate in Buttons, calendar t o show the submission date for the assignments or Bb [Blackboard] better than webct - visually a mess - Bb very ordered Figure 3: Examples of students' comments addressing human