olmos.pdf Australasian Journal of Educational Technology 2012, 28(1), 1-15 Academic analytics in a medical curriculum: Enabling educational excellence Martin Olmos and Linda Corrin University of Wollongong An Outstanding Paper Award recipient, ascilite Hobart 2011 Conference The developing field of academic analytics seeks to turn data from educational systems into actionable intelligence for the improvement of teaching and learning. This paper reports on the implementation of analytics in a new medical school with an integrated curriculum and clinical focus. Analytics addressed two challenges in the curriculum: providing evidence of appropriate curriculum coverage and assessing student engagement and equity while on clinical placement. This paper describes the tools and approaches used, and outlines the lessons learnt. These lessons include the risk of a simplistic use of visualisations, their potential to generate important questions, the value of a flexible approach to tool selection, the need for relevant skills, and the importance of keeping the audience central. Although there is much further potential for the school to realise, academic analytics have already been a critical enabler of educational excellence. Introduction The field of academic analytics is concerned with providing actionable intelligence to inform strategies for the enhancement of teaching and learning in higher education (Campbell, De Blois & Oblinger, 2007). The use of data extracted from academic systems to create reports and visualisations enables educators to perform up-to-the- minute curriculum reviews, monitor the impact of changes and guide implementation of new learning activities. The Graduate School of Medicine (GSM) at the University of Wollongong has used academic analytics throughout the development and implementation of its innovative and integrated curriculum. Academic analytics has allowed the GSM to address two main challenges in developing the curriculum: the ability to ensure coverage of content in the curriculum, and the support of student engagement during regional/rural clinical placements. This paper reports on the implementation of the reports and visualisations developed over the past four years and outlines the lessons learnt. The use of academic analytics has proved valuable to the curriculum development and review within the GSM and work continues on the development of more customised reports to address specific issues and areas of future development. Academic analytics: A timely new field The relatively new field of academic analytics has been increasingly discussed in the academic literature. Goldstein (2005, p.14) surveyed 378 higher education executives 2 Australasian Journal of Educational Technology, 2012, 28(1) on their information and analysis needs, finding that the “primary applications of advanced analytics include modeling strategic decisions, studying enrollment trends, and measuring student retention”. They report that management commitment and staff skills, rather than technology, were the critical factors in deploying academic analytics. Oblinger and Campbell (2007) note the increasing accountability requirements on educational institutions and the potential for analytics to improve student retention and graduation levels. Campbell, De Blois and Oblinger (2007, p.44) provide a helpful overview, noting that “with the increased concern for accountability, academic analytics has the potential to create actionable intelligence to improve teaching, learning, and student success”. They outline case studies using analytics for developing predictive models of enrolment and retention and identifying at-risk students. From these cases they highlight three critical success factors: leaders committed to evidence-based decision-making, staff with data analysis skills, and a flexible and effective technology platform. However, they also warn against over simplification and raise a number of issues including privacy, faculty involvement, and data stewardship. Norris et al. (2008, p.56) suggest a process to move from “a culture of information and reporting” towards the use of analytics to inform and enable action. They note the broad range of contexts where analytics can be applied: “clearly, one size does not fit all in action analytics. Most institutions begin predictive/dynamic modeling by focusing on admissions and retention… action analytics is like a smorgasbord of options, all aligned with institutional goals and strategies” (p.50). They also stress the importance of open technology architectures to enable analytics, as well as leaders who are committed to “build organizational capacity, change the organizational culture, and foster new behaviors that both enable and reflect evidence-based decision-making and action” (p.52). They identify four contexts in which these priorities apply: technology, information, analytics, and innovation. Ellaway (2008, p.242), discussing academic analytics in medical education specifically, points out that “being able to account for the execution of medical education is both a professional and ethical concern”. She encourages a critical consideration of analytics’ impact on data collection and analysis, as well as the effort required. Arnold (2010) describes the use of analytics at Purdue University to turn institutional data into actionable intelligence, by identifying at-risk students and guiding them to helpful resources. The field is set to continue growing, with the first International Conference on Learning Analytics and Knowledge held in February 2011. Most recently, the 2011 Horizon Report (Johnson, 2011) identified analytics as a technology to watch in the four to five year adoption horizon. Fortunately, general awareness of analytics and visualisation grew around the same time as the medical school started. In November 2005 Google Analytics was released, showcasing complex and interactive analytics to many (Google, n.d.). In 2007, Dr Hans Rosling gave a popular TED talk demonstrating the power of visualisations to represent large datasets (Rosling, 2007), which developed into a BBC documentary. During the 2009 Web 2.0 Summit conference, Tim O’Reilly stressed that “data analysis, visualization, and other techniques for seeing patterns in data are going to be an increasingly valuable skillset.” (O’Reilly & Battelle, 2009). In late 2010, The Bill & Olmos and Corrin 3 Melinda Gates Foundation announced a $20 million initiative to address Next Generation Learning Challenges, one of them being “Helping institutions, instructors, and students benefit from learning analytics” (EDUCAUSE, 2010). Background and context The Graduate School of Medicine at the University of Wollongong opened its doors to students in 2007 with a bold vision of an innovative curriculum that would seek excellence in medical education, with a particular focus on regional, rural and remote health. The curriculum is distinct from many other medical schools in its attempt to integrate all content in a spiral structure in which issues are revisited throughout the course with increasing levels of understanding and competence required. Two elements in particular contribute to the curriculum’s distinctiveness. Firstly, the curriculum is designed around 93 presenting clinical problems and the body systems to which these relate. It is also driven by a set of learning outcomes grouped in four themes: medical sciences, clinical competencies, personal and professional development, and research and critical analysis. The clinical problems are covered in turn with the aid of specific clinical cases. For example, there is no pathology subject in the course but rather it is taught throughout the program in the context of the clinical problems and body systems. This ensures students learn the various aspects of addressing patients’ presenting problems together and with concrete examples, rather than in a disjoint and abstract manner through separate subjects of anatomy, pathology, paediatrics, etc. Secondly, the curriculum has a very strong clinical focus, with students beginning hospital and general practice placements within the first six weeks of the course. The most significant clinical experience is a 12 month clinical placement in rural and regional areas of the state which starts mid-way through the third year of the program. During this time students are based in a general practice and at the local hospital, but also attend other community health clinics. This allows them to experience and participate in the whole process of medical care. Further, it integrates their learning with practical experience across a wide array of real patients under the supervision of practising doctors. These two critical elements, however, introduce significant challenges to the delivery of the course. Firstly, it is necessary to ensure there is appropriate coverage in the course of the clinical problems, learning outcomes, body systems, as well as clinical and scientific specialties. This has been a critical issue in gaining and maintaining accreditation as a medical school. Further, each of the clinical specialty teams has an interest in ensuring proper coverage of its own area. Clear evidence of coverage is thus essential. Secondly, the many external clinical placements might create a disjointed course if these are not integrated well with the campus-based teaching. Specifically, the year- long placement in a rural or regional setting far from campus raises considerably the level of support required by students. Their experiences need to be linked to the curriculum. A level of equity across students needs to be ensured in the number and type of clinical problems they experience, as well as the students’ level of involvement. 4 Australasian Journal of Educational Technology, 2012, 28(1) In summary, these two elements create two areas of obscurity: what students are taught while on campus, and what they do while away from campus. It is unclear if the school’s curriculum would be feasible without these challenges being overcome. The GSM has used technology extensively to overcome these challenges and in turn implement its vision of excellence in medical education. The online learning environment has been recognised internationally with a 2007 ascilite Award and the 2008 IMS Global Learning Consortium’s Platinum Learning Impact Award. Specifically, academic analytics combined with visualisation have made visible what the school needs to see both in its curriculum content and students’ clinical placements. This is a broader use of analytics than the common applications in finance, research functions, retention, admissions and identifying at-risk students as reported in the literature (Goldstein, 2005; Norris et al., 2008; Arnold, 2010). Simply put, analytics show what students are taught on campus, and what students do while away from campus. Although this paper focuses on these two areas, the GSM uses analytics more broadly in its delivery of the course. The school employs a Data Manager, charged with the collection and analysis of various datasets, including admissions, administrative and financial metrics. Further, the analysis and visualisation of student interactions in discussion forums has already been reported elsewhere (Dawson, Macfadyen & Lockyer, 2009; Dawson, Heathcote & Poole, 2010; Macfadyen & Dawson, 2010). Curriculum content The integrated and outcomes-based curriculum is supported with the use of a learning content management system called Equella. All learning and teaching resources are entered into this system and made available to staff and students. Four item types are defined in the system: • Problem Blueprint: The detailed description of each of the 93 clinical problems, including the learning outcomes it covers as well as the clinical and scientific specialties and body systems to which it relates. • Learning Activity Outline (LAO): A description of the learning outcomes, associated readings, and notes/audio from a session that students attend which is typically led by a presenter (e.g. lectures, clinical demonstrations). • Resource: A digital asset with educational content (e.g. lecture slides, diagrams, readings, animations, movies). • Case-based Learning (CBL): Description of a clinical case and related resources. Each of these is tagged with a common curriculum metadata schema with the following descriptors: • Problem • Medical sciences • Clinical competencies • Personal and professional development • Research and critical analysis • Body system • Science specialty • Clinical specialty Olmos and Corrin 5 This metadata, together with Equella’s flexible XML-based exporting, has allowed the school to generate powerful reports summarising the whole course. For example, Figure 1 visualises the coverage of learning outcomes (represented by the codes on the top row) for the first 18 months in the course. Each cell in the table represents a particular fortnight in the teaching timetable and learning outcome. The darkness of each cell represents the number of learning activities in that fortnight that relate to the learning outcome. The figure makes clear the strong focus on basic medical sciences (represented by the first few learning outcome codes) during this period. The value in each cell can be seen by hovering the mouse pointer over it. Clicking a cell displays a list of the learning activity outlines represented by the cell. The viewer can then click on an individual learning activity outline to get its full details, including the presentation slides and recorded audio where applicable. Figure 1: Coverage of learning outcomes in learning activities for each fortnight Figure 2 is another example, showing the change in coverage from one student cohort to another. Black cells represent no change, bright green a large increase, and bright red a large decrease of coverage for each fortnight/learning outcome pair. Figure 2: Change in coverage of learning outcomes in learning activities for each clinical problem 6 Australasian Journal of Educational Technology, 2012, 28(1) These reports were originally developed by using XSLT to transform the XML of the learning activity outlines’ metadata (exported from Equella) into HTML. A newer release of Equella allowed for the use of the free and open-sourced Business Intelligence Report Tool (BIRT) to query the database and render the visualisations. These visualisations provided evidence which helped the school to satisfy itself and its accreditors that appropriate coverage was achieved in its integrated curriculum. The fact that they are generated from the delivery system with data entered by the academics at the time of preparing the learning activity (rather than a secondary system with data collected long after delivery and in preparation for an audit) gives confidence in the quality and timeliness of the data. Further, the measurement of change in coverage is particularly relevant in a new school, where revisions to the curriculum are common and there is a need to analyse their impact. Review decisions can then be made based on evidence. A moderation process applies to the contribution of LAOs into Equella, which progresses the item from initial entry, to review and approval by an academic coordinator, to a final check by the Educational Technology team, and finally to a live status on a given date when it’s accessible to students. The item’s progress is tracked, and thus analytics can be done at an aggregate level on this process. A moderation progress report was developed which counts the number of items in each of the above steps for each fortnight in the course. This allows academic coordinators to assess readiness leading up to each fortnight. Thus Equella can show the school’s performance in delivering educational content to our students. Clinical experiences GSM students have a range of clinical placements during the course’s four phases, including fortnightly general practice and hospital visits during the first 18 months, seven hospital-based rotations over the next 12 months, a year-long rural or regional clinical placement, and lastly three six-week placements during the fourth phase in a variety of domestic and international settings. A web-based Clinical Log application was developed in order to allow students to record their experiences and reflections, as well as enabling the school to support them in integrating their experiences to the curriculum. It enables recording patients’ gender and age, their presenting complaint (mapped to the curriculum’s clinical problems), the location of the consultation (e.g. campus, general practice, hospital), their level of involvement (e.g. observation, history taking, or examination), and their confidence level at this level of involvement as a self- reported five point scale. Some reflection fields invite students to identify learning needs and required actions. A more detailed description of the Clinical Log can be found in Corrin and Olmos (2010). BIRT was chosen to query the Clinical Log’s relational database and produce a range of reports with specific audiences and purposes. The cohort report aims to summarise an entire cohort’s entries over a particular date range and includes a number of charts and tables, such as: • Number of entries per month, and per confidence level • Scatter plot of the date the student made the entry versus the date they saw the patient • Confidence level distributions per students Olmos and Corrin 7 • Confidence level across time for each level of involvement and location • Number of entries per clinical problem for each age group, gender, location, and involvement Figure 3: Distribution of clinical problems against location The dashboard report aims to highlight potential issues and at-risk students and is designed to be generated regularly. It includes a list of entries with low confidence levels and students with no entries in the last week. Authorised staff can click on an entry to view the full details in the Clinical Log, including the student’s reflections and identified learning needs. They can then insert comments for the student to read. Figure 4: A student’s number of entries per month, stacked by confidence Lastly, the student report summarises an individual student’s entries, reflecting the number and confidence level, and distribution across clinical problems, gender, age group, location, and involvement levels (see Figure 4). This report is made available to 8 Australasian Journal of Educational Technology, 2012, 28(1) students to allow them to see their own experiences. Further, the report was integrated into an assessment involving discussion of specific clinical experiences and it was expected to become a valuable resource to facilitate the interaction between the individual student and their clinical supervisor. As Norris et al. (2008, p.48) noted, “The most powerful action analytics are learner-centric, focusing on issues related to access, affordability, and success for learners at all stages of their learning lives. Over time, these analytics will empower learners to take greater responsibility for their success, in collaboration with parents, teachers, mentors, and employers”. For example, these analytics might empower a student to request exposure to specific clinical problems or higher levels of involvement from their clinical supervisors. Furthermore, they might use it to demonstrate their clinical experience at the start of a placement or beyond their course. Lastly, it was hoped that it would increase students’ motivation to use the Clinical Log. Motion Chart Although the reports mentioned above are very helpful, there remained a challenge of visualising the dataset’s multiple inter-related dimensions at the same time. These dimensions include entry date, frequency of entries, clinical problems, the student’s level of involvement (e.g. observation, history taking, examination) and level of confidence. Specifically, the mean confidence level across time is a natural focus but it needs to be looked at in the context of the level of involvement and the number of entries per period. Further, it’s good to compare how the frequency of entries, confidence and level of involvement vary between students and cohorts as a whole. Figure 5: Motion Chart of clinical problems coverage Google’s interactive Motion Chart tool was used to address this challenge. Motion Charts, popularised by Dr Hans Rosling’s 2007 TED presentation (Rosling, 2007), can render up to six variables concurrently (see Figure 5). Each bubble represents a clinical problem (Figure 5) or a student (Figure 6), with its size representing the number of entries for a given month and its colour representing the student cohort. Confidence is Olmos and Corrin 9 mapped to the y-axis, and level of involvement on the x-axis. The data’s time is mapped onto the time of the animation. Motion Charts also have helpful interactive features. The time slider can be dragged back and forth, and a single student can be highlighted by clicking on a bubble, which will show the values for each of the variables. Playing the animation can display a trail of the bubble’s movements across time, thus showing a student’s progress in involvement and confidence. Figure 6 highlights two students’ different journeys through the course. There are some drawbacks with Motion Charts. They can be somewhat complex to view, as some clarity is traded off to maximise the number of variables rendered. For example, it is generally best to render a single dimension as a line rather than an area. Thus, representing the number of entries as the size of the bubble is not optimal. A simple bar chart would be better to display this single variable across time. Secondly, given that the dataset’s time is mapped onto the animation’s time, it takes time to watch them. If the pattern of a single variable across time is the focus, it is best to map time against the x-axis and use a simple line or bar chart. Lastly, bubbles representing just a few data entries can move quite rapidly in the chart as the means for each month change significantly, and paradoxically the viewer’s attention can be attracted to these. Despite all these issues, the Motion Chart’s core benefit of rendering multiple dimensions in one interactive chart remains. It is thus not seen as a replacement to standard charts but a powerful complement. Figure 6: Motion Chart of students’ paths Tree Map The third phase of the course, when students disperse for a year-long clinical placement in rural and regional settings, is particularly challenging. It is important to compare the medical practices, towns, and regions, as well as the students themselves. 10 Australasian Journal of Educational Technology, 2012, 28(1) This is in order to answer questions such as: • Are students in different regions exposed to different clinical problems? • Do students in a particular practice have lower confidence or involvement levels? Figure 7: Tree map of entries per placement regions However, this hierarchical and heterogeneous dataset is difficult to render using common chart types. Google’s implementation of the Tree Map chart (Figure 7) was used to render it. It displays a set of rectangles with their size and colour mapping to the data, in our case the number of entries and mean confidence respectively. Each rectangle can be clicked to drill down through the hierarchy, from region, town, practice, student, and patient. The relative size and confidence level of each setting is clear at its own level in the hierarchy. For example, Figure 7 shows that Region 1 had the most entries, and its grey colour suggests the mean confidence was not high. Due to the nature of the dataset, the Tree Map’s utility was mixed. This type of chart works best with hierarchies that are wide as well as deep. Often, there is only one general practice in a given town or a single student in a practice. The chart hasn’t progressed onto wide use, but the experiment was worthwhile. Lessons learnt Several lessons have been learnt from the school’s experience in using academic analytics and visualisations. Given this is an implementation study, our findings are somewhat unstructured. They are offered below as a small contribution to a growing and important field, with the caveat that they might not apply across all contexts. Data quality is critical The potential for analytics was implicitly considered while developing the GSM’s systems and data repositories, although the final charts that would eventually be developed were unforeseen. An emphasis was placed on maximising structured data Olmos and Corrin 11 and having discrete vocabularies for important fields. For example, a set of clinical and scientific specialties was defined from which users had to choose rather than allow free text entry. Effective analytics depend on high quality and consistent data. Explore and discover This process was not started with a clear outcome in mind, nor a pre-determined process. The visualisations developed through experimentation from the large, structured datasets. Be flexible and discerning with tools There was no initial commitment to use a particular set of tools. Rather, the best suited tool was selected for each task out of the options available. Sometimes proprietary programs such as Microsoft Excel were used, and other times free and open tools such as BIRT, Gephi and Google’s visualisation APIs were used. For example, the generic BIRT tool helped overcome one of the challenges of academic analytics: accessing multiple datasets from various sources (Norris et al., 2008; Arnold, 2010; Johnson, 2011). The free and open nature of these tools allowed experimentation without significant financial commitment. Further, a toolkit approach enables the best tool to be used for each step of analysis. For example, an analyst might extract a data set with BIRT, do some basic manipulation with Microsoft Excel, and then render it with Gephi or Google’s API. Critically, the choice of tools needs to be guided by the data, audience, and purpose. Just show the data A simple approach of visually representing the data without sophisticated statistical summarising proved very effective. Although sometimes averages were used, the aim was to maximise the data to ink ratio (Tufte, 1983) by rendering on the chart as many data points as possible, relying on the viewer’s pattern perception skills. As the Motion Charts demonstrate, a well designed visualisation can render a large dataset and remain clear: one such chart renders over 33,000 values which are themselves summarising over 337,000 database values. A well designed chart of a good dataset is very powerful, even without advanced statistical summarising of the data: Statistics can reduce large, complex data sets to a few numbers, but this reductive approach can also shear away much of the richness and subtlety in data. Statistical analysis that goes beyond the basics involves an erudite language spoken only by trained specialists, but visual analysis is accessible to a broader audience… In simple terms, information visualization helps us think. (Few, 2009, p.6) Keep the audience central Each analytic tool should be designed with a specific audience and purpose in mind and should be guided by the data available, so as to provide the evidence required to inform the decisions the audience needs to make. Indeed, different reports representing the same dataset may be required to serve different stakeholders. An attractive and technically sophisticated visualisation which doesn’t inform anyone is of little use. Further, users may require some training in using complex visualisations. Visualisations clarify the value of data Both staff and students have informally reported finding the various visualisations helpful in understanding the value of the data that they contributed. Understanding the need for high quality data to produce quality analytics might help motivate them to contribute data, a task which can often be onerous without a clear payoff. Further, visualisations can themselves identify weaknesses in the data. 12 Australasian Journal of Educational Technology, 2012, 28(1) Maximise the audience There is value in opening access to the data where feasible and appropriate. This not only maximises the benefit of analytics work, it also raises awareness of the importance of quality data and the value of analytics in general. Although sensitive data should be handled appropriately, our experience reflects Goldstein’s (2005) finding of no significant adverse impacts of a greater availability of data. Beware the treachery of images A naïve consumption of the visualisations should be avoided. They are often compelling and attractive, but might sometimes be confused with the reality they’re seeking to explain. They are representations of the data, which in turn may or may not represent reality. They should supplement rather than replace direct communication and fact-finding. Analytics as a question-generator Although these visualised analytics no doubt answer many questions, they are typically of a simple nature: when, who, how many. Their greatest value is in enabling the more complex and critical why questions. For example, the Clinical Log motion chart doesn’t tell why a student’s confidence suddenly drops, but the question might remain unasked without it. Leadership support and technical know how The medical school’s leadership has provided strong support for the development efforts. This was critical in sustaining what might otherwise have been seen as a ‘non- core’ activity. Further, having an in-house educational technology team was helpful in understanding both the technology’s potential and the school’s curriculum needs. Much analytics requires a level of technical skills out of reach of most faculties, yet a central (and therefore more distant) university-wide IT department might not see their value to the specific needs of an individual faculty. Goldstein (2005) found training effectiveness, leadership commitment, and staff with analytical skills to be the factors most strongly linked to success. Campbell, De Blois, and Oblinger (2007) also note a leadership committed to evidence-based decision-making and staff skilled in data analysis as critical success factors. Lastly, it is important to have some skills in information design to produce visualisations that provide insight while rendering large datasets. Although these skills are important, it is not critical to have deep expertise in each. Rather, a nimble and generalist approach has proven more useful. As Few (2009, p.2) argues: Computers can’t make sense of data; only people can. More precisely, only people who have developed the necessary data analysis skills can… Ninety percent of the data analysis needed by most organizations can be performed using a simple set of skills that require only a basic and easily mastered understanding of statistics. These analyses involve a set of techniques that use graphs – visual representations of quantitative information – to explore numbers and discover meaningful patterns within them. Mastering these visual analysis techniques allows us to lift the veil that separates us from insight. Track change and flows as well as stocks Although a snapshot of current data is helpful, showing change across time provides a more sophisticated analysis. For example, the visualisations of change in coverage of learning outcomes or clinical problems from one cohort to another (Figure 2) have indicated how the curriculum is developing in its initial years of delivery. A sudden Olmos and Corrin 13 and significant drop in a student’s confidence during clinical placements would be another important change to track. Broad and deep application Although there are areas to which the GSM hasn’t applied analytics, there is indeed value in applying advanced analytics broadly, and specifically in analysing the curriculum. Goldstein (2005) reports this is uncommon, with most application in the areas of central finance, budget and planning, and institutional research functions. Indeed, analytics of the curriculum itself is not mentioned in the paper’s 2011 futuristic scenario. Moving beyond these can bring focus to areas closer to the core of students’ educational experience. Conclusion This paper has reported on the initial implementation of academic analytics and visualisations within a medical school. More detailed research is planned, into the extent of their use and impact, as well as drivers and blocks to their adoption. There is much the school can do in the future to expand and improve the use of academic analytics. For example, integrating assessment results with the Clinical Log dataset would reveal the correlation between confidence and competence. Mapping assessment in a similar way to content would make clearer the integration between the two. Sentiment analysis might prove valuable in analysing students’ reflection comments in the Clinical Log. Although there has been great value in simply rendering the data, the GSM might be in a position to start developing more sophisticated statistical and predictive models given there is now one cohort who has completed the entire course. For example, identifying at-risk students while on their extended rural placement would be valuable. Improvements can be made to the availability of analytics to staff and students and its use for strategic decision making within the school, as well as benchmarking with other medical schools. Academic analytics have played a critical role in enabling the GSM to deliver its vision of excellence in medical education. The wide application of these tools, beyond administrative areas and specifically to the curriculum and students’ clinical experiences, have borne fruit at the core of the quality of students’ learning experience, by providing evidence of curriculum coverage and a window into students’ engagement with clinical experiences. This has helped the school assure itself and its accreditors on the quality of its curriculum and the support and equity of students’ experience while on remote placements. Analytics have been invaluable in mining the considerable data produced by the educational systems. In turn, visualising this data has rendered data in an understandable way to a wide audience, so that the insights available become actionable. Together, analytics and visualisation have helped the school “lift the veil that separates us from insight” and help us think (Few, 2009, p.2, 6). References Arnold, K. (2010). Signals: Applying academic analytics. EDUCAUSE Quarterly, 33(1). http://www.educause.edu/EDUCAUSE+Quarterly/EDUCAUSEQuarterlyMagazineVolum/SignalsAp plyingAcademicAnalyti/199385 14 Australasian Journal of Educational Technology, 2012, 28(1) Campbell, J. P., DeBlois, P. B. & Oblinger, D. G. (2007). Academic analytics: A new tool for a new era. EDUCAUSE Review, 42(4), 40-57. http://www.educause.edu/EDUCAUSE+Review/ EDUCAUSEReviewMagazineVolume42/AcademicAnalyticsANewToolforaN/161749 Corrin, L. & Olmos, M. (2010). Capturing clinical experiences: Supporting medical education through the implementation of an online Clinical Log. In Curriculum, technology & transformation for an unknown future. Proceedings ascilite Sydney 2010 (pp.231-235). http://ascilite.org.au/conferences/sydney10/procs/Corrin-concise.pdf Dawson, S., Heathcote, L. & Poole, G. (2010). Harnessing ICT potential: The adoption and analysis of ICT systems for enhancing the student learning experience. International Journal of Educational Management, 24(2), 116-128. http://dx.doi.org/10.1108/09513541011020936 Dawson, S., Macfadyen. L. & Lockyer, L. (2009). Learning or performance: Predicting drivers of student motivation. In Same places, different spaces. Proceedings ascilite Auckland 2009. http://www.ascilite.org.au/conferences/auckland09/procs/dawson.pdf EDUCAUSE (2010). EDUCAUSE announces $20 million for next generation learning challenges. [viewed 24 Mar 2011] http://www.educause.edu/About+EDUCAUSE/PressReleases/ EDUCAUSEAnnounces20MillionforN/215025 Ellaway, R. (2008). eMedical teacher. Medical Teacher, 30(3), 342-343. http://dx.doi.org/10.1080/01421590802082544 Few, S. (2009). Now you see it: Simple visualization techniques for quantitative analysis. Analytics Press. Goldstein, P. J. (2005). Academic analytics: The uses of management information and technology in higher education. EDUCAUSE. http://net.educause.edu/ir/library/pdf/EKF/EKF0508.pdf Google (n.d.). Google history – Company. [viewed 2 Jun 2011] http://www.google.com/about/corporate/company/history.html Johnson, L., Smith, R., Willis, H., Levine, A. & Haywood, K. (2011). The 2011 Horizon Report. Austin, Texas: NMC. http://www.nmc.org/pdf/2011-Horizon-Report.pdf Macfadyen, L. P. & Dawson, S. (2010). Mining LMS data to develop an “early warning system” for educators: A proof of concept. Computers & Education, 54(2), 588-599. http://dx.doi.org/10.1016/j.compedu.2009.09.008 Norris, D., Baer, L., Leonard, J., Pugliese, L. & Lefrere, P. (2008). Action analytics: Measuring and improving performance that matters in higher education. EDUCAUSE Review, 43(1), 42-67. http://www.educause.edu/EDUCAUSE+Review/EDUCAUSEReviewMagazineVolume43/ ActionAnalyticsMeasuringandImp/162422 O’Reilly, T. & Battelle, J. (2009). Web squared: Web 2.0 five years on. [viewed 10 Mar 2011] http://www.web2summit.com/web2009/public/schedule/detail/10194 Oblinger, D. & Campbell, J. P. (2007). Academic analytics. EDUCAUSE [viewed 23 Jun 2011] http://www.educause.edu/ir/library/pdf/PUB6101.pdf Rosling, H. (2007). Hans Rosling’s new insights on poverty. TED. [viewed 10 Mar 2011] http://www.ted.com/talks/hans_rosling_reveals_new_insights_on_poverty.html Tufte, E. R. (1983). The visual display of quantitative information. Cheshire, Conn. (Box 430, Cheshire 06410): Graphics Press. Olmos and Corrin 15 This article received an Outstanding Paper Award at ascilite Hobart 2011, gaining the additional recognition of publication in AJET (with minor changes). The reference for the Conference version is: Olmos, M. & Corrin, L. (2012). Academic analytics in a medical curriculum: enabling educational excellence. In Changing demands, changing directions. Proceedings ascilite Hobart 2011. http://www.ascilite.org.au/conferences/ hobart11/downloads/papers/Olmos-full.pdf Authors: Mr Martin Olmos, Graduate School of Medicine, University of Wollongong Email: martin.olmos@internode.net Ms Linda Corrin, Graduate School of Medicine, University of Wollongong Email: lcorrin@uow.edu.au Please cite as: Olmos, M. & Corrin, L. (2012). Academic analytics in a medical curriculum: Enabling educational excellence. Australasian Journal of Educational Technology, 28(1), 1-15. http://www.ascilite.org.au/ajet/ajet28/olmos.html