Case Studies Compass: Journal of Learning and Teaching, Vol 15, No 2, 2022 1 Low-stakes, VLE intensive, formative approach to maths teaching as a pedagogic strategy for improving assessment outcomes Susan Force, Mark Goss-Sampson, Sarah Harris School of Science, University of Greenwich, United Kingdom Abstract Functional numeracy skills in the UK workforce have been an issue for at least 20 years and are of mounting concern due to the increasing use of technology in the workplace and the need for mathematical problem solving skills. Surveys reveal anxiety related to maths (MA) has a negative impact on Higher Education (HE) subject choices and subsequent career options. This at a time when Science, Technology, Engineering and Mathematics (STEM) subjects, requiring maths and data handling skills, are in high demand globally. This case study relates the journey from discovering the need for maths support in science students to the use of low-stakes, VLE intensive, formative maths support as a means of overcoming MA and improving student outcomes. 1. Introduction Numeracy skills in the United Kingdom (UK) workforce have been an issue of concern for at least twenty years, following reports that numeracy levels are not adequate for economic survival and that employers are finding it difficult to recruit appropriately numerate employees (Williams et al., 2003; Department for Business Innovation and Skills, 2012; Grayson, 2013). In more recent years, the exponential imposition of technology and digitisation across all economic sectors requires the workforce to master a wide portfolio of competencies, including numeracy and information technology (IT) skills. Lower-skilled employees may now be expected to operate digitised equipment and identify and solve problems by interpreting changes manifesting themselves as deviations from the norm, in often continuous and complex data streams. The National Numeracy Charity (NNC) (2022a) defines numeracy skills as: “The ability to understand and use maths in daily life, at home, work or school. Numeracy doesn't mean complex skills, like algebra, it means being confident enough to use basic maths in real-life situations. Numeracy is as important as literacy - it's sometimes called ‘mathematical literacy’ - and we need both to get on in life.” In 2019, the National Numeracy Day Survey presented 2007 respondents, aged from sixteen to seventy-five years, with questions requiring ‘everyday’ maths skills and questions relating to their feelings about maths. The results indicated that: • 50% of working age adults lack good numeracy skills • 20% respondents scored 80-100%, equivalent to GCSE grade 4/C (a standard pass) and similar to the 2011 Skills for Life survey findings Case Studies Compass: Journal of Learning and Teaching, Vol 15, No 2, 2022 2 • 31% claimed not to be a ‘numbers person’ (more women) • 25% would not apply for a job involving numbers and data • 36% would like to improve their maths and numeracy skills • Many overrated their level of numeracy (National Numeracy, Ipsos Mori and KCL, 2019) These observations are of concern, given the need for functional maths skills in the workforce. Maths anxiety (MA) has been identified as a factor that attenuates numeracy and the development of more advanced mathematics skills. A 2018 survey commissioned by the Maths Anxiety Trust (MAT) indicates that approximately 36% of fifteen- to twenty-four-year- olds and 20% of adults in Great Britain felt anxious about maths with varying degrees of severity (Maths Anxiety Trust and Ipsos Mori, 2018). MAT (2022a) defines maths anxiety as follows: “a negative emotional reaction to mathematics, leading to varying degrees of helplessness, panic and mental disorganisation that arise among some people when faced with a mathematical problem, either in ordinary life or in an academic situation.” MA is characterised by physiological symptoms, such as sweating and raised blood pressure, when someone is confronted with numerical problems. Psychological symptoms include feelings of permanency (it will never improve), isolation (everyone is better than me), nervousness and lack of confidence. Causal factors may relate to poor learning and teaching, lack of consolidation in the classroom, time pressures in tests, fear of exposure and embarrassment and the negative attitude of teachers or parents who may be projecting their own MA (Department for Business Innovation and Skills, 2012; Maths Anxiety Trust, 2022a). According to Sue Johnston-Wilder of the MAT, maths anxiety is an acquired problem, which makes it preventable and treatable (Maths Anxiety Trust, 2022b). Another aspect of maths capability is that of ‘maths self-efficacy’. This is the extent to which students believe in their own ability to solve specific mathematics tasks. It tends to be lower in those who suffer from MA or who score lower marks in maths assessments (Rozgonjuk et al., 2020). As might be expected, higher self-efficacy is associated with better performance in maths. Whilst self-efficacy is associated with less MA and higher levels of maths capability, there are also reports that students overestimate their numeracy skills. This in turn suggests an underestimation or misunderstanding of essential maths skills (OECD, 2013). Because maths capability can be masked by anxiety, the resulting avoidance of education or employment involving maths has far-reaching, long-term implications for education, employment and progression opportunities. For higher education (HE) this can manifest itself as lack of new student registrations and poor retention in science, technology, engineering and mathematics (STEM) subjects, which all require maths skills – this at a time when STEM graduates are needed more than ever in the global workplace (Rozgonjuk et al., 2020). The Jobted website indicates that the demand for chemists, biologists and life science experts is set to rise to deal with rapidly rising global environmental and population issues. The same website lists skills required in applicants for the science industry and these include: Case Studies Compass: Journal of Learning and Teaching, Vol 15, No 2, 2022 3 “1. Data analysis skills: ‘speak the language of science’ (or, in other words, mathematics). The majority of decisions in twenty-first century life are based on data collection, analysis and interpretation. Today’s scientists need to be numerically and statistically literate. 2. The ability to use scientific instruments: such instruments vary enormously in their levels of complexity but they all measure something and collect data. Most are computerised and data collection is managed by specialist software. Scientists need a full working knowledge of the equipment hardware as well as the skills required to use the software followed by the analysis and interpretation of data and its implications.” (Jobted, 2021) Note the reference to mathematics as the language of science and the reliance on data collection and analysis skills. The University of Greenwich Faculty of Engineering and Science (FES) offers undergraduate and postgraduate degree programmes in (STEM) subjects. The FES Life Science programmes attract approximately 200-300 first-year students per annum. Entry requirements stipulate, as a minimum, level 4 GCSE (old grade C) in Maths, which is rated as a ‘standard’ pass and presumed to prepare students for an undergraduate programme in Life Sciences. However, the 2011 Skills for Life Survey suggested that almost one in four students may have numeracy abilities that are below Entry Level 3, as defined below (Department for Business Innovation and Skills, 2012; National Numeracy Charity, 2022b). “Entry Level 3 Understanding information given by numbers, symbols, diagrams and charts for different purposes expressed in graphic, numerical and written forms in different ways. This includes: • dividing two digits by one digit and understanding remainders • comparing weights using standard units For example adults with skills below Entry Level 3 may not be able to understand price labels or pay household bills.” The 2011 survey is now twelve years old and there have been no direct comparisons since; however, the implication is that level 4 GCSE may not confer students with the functional maths skills necessary for engaging with the sciences. The head of maths at a Kent academy recently (Autumn 2021) advised me that university programmes where maths skills are essential should be looking at maths GCSE Level 5 and above. Like other universities, Greenwich offers maths and statistics help (MASH) to all students in the form of drop-in tuition, variously publicised – for example, by the library support services website, the student portal and individual tutors engaged in teaching. Despite this, my own observations and conversations with students indicate reluctance to use the service, Case Studies Compass: Journal of Learning and Teaching, Vol 15, No 2, 2022 4 sometimes illustrated by professions of ignorance of its existence. Some students do admit to feeling intimidated by the thought of explaining their needs to ‘an expert’. 2. Method In 2016, I assumed leadership of an undergraduate study skills module for second-year FES Life Science students. A survey of teaching colleagues revealed that confidence with basic descriptive and inferential statistics was considered the most important study skill that these students should acquire in this module. These sentiments were (and still are) echoed during exam board sessions by external examiners who request “more meaty statistics” and “evidence of statistical interpretation” in final-year projects. Initial attempts to teach descriptive statistics in 2016 revealed that many of these second years lacked confidence about such basic maths concepts as decimals, significant figures, percentages and probability, despite having attained their entry-level requirements of GCSE grade C/level 4 maths. Neither were many able to solve science-related problem-solving scenarios requiring mathematical reasoning. Further questioning revealed a need for additional numeracy support in the first year. Informal conversations with these would-be scientists frequently exposed degrees of MA and accompanying lack of confidence arising from their fear of anything to do with numbers. Agreement having been gained from programme leaders, maths teaching and learning were incorporated into the first-year curriculum, with an associated weighted assessment. A teaching schedule – including decimals, units, maths operator rules, molarity and descriptive statistics – was developed, using pedagogic approaches suggested by the Organisation for Economic Co-operation and Development (OECD) to overcome MA; it included: • Assume nothing concerning previous teaching and learning • Avoid exposure of errors / lack of ability • Provide plenty of low-stakes formative activities (OECD Publishing, 2016) Low-stakes testing in this context refers to a method of assessment allowing students the opportunity to try repeatedly, make mistakes and potentially fail and to learn from those mistakes with minimal academic penalty (Top Hat, 2022). Deliberate emphasis was placed on the value of numeracy skills with respect to academic success and employment, thus relating these skills to long-term projections (Zacharopoulos, Sella and Kadosh, 2021). Since 2017, first-year students have been instructed to take – in the first two weeks of their first term and in a location of their choice – a two-hour, diagnostic online quiz consisting of sixteen questions. They were assured that marks did not count, encouraged just to ‘do their best’ and told that overall results would be used to guide maths teaching content to prepare them for their degree programmes. After the diagnostic, students were advised that the summative assessment at the end of term one would be based on similar questions and advised of a mid-term formative opportunity using the same questions. Case Studies Compass: Journal of Learning and Teaching, Vol 15, No 2, 2022 5 The diagnostic (start-of-term), formative (mid-term) and summative (end-of-term) quizzes were all the same and composed entirely of wild-card questions (Moodle simple question type, Figure 1). These progressed from the use of simple mathematical operators to more complex problem-solving scenarios relevant to their degree programmes (Figure 1). Questions were separated into categories, such as decimals and significant figures, percentages, the rules of mathematical operators and molarity. You are asked to prepare [a] ml of a [b] M (molar) solution of sodium chloride (NaCl). Calculate the cost of the NaCl reagent you will need to use in £ to 2 dp, no units. Molecular wt NaCl = 58.44g Price of NaCl = £[c] Where: [a] = any value between 50 and 950 (0 dp) [b] = any value between 0.2 - 5.0 (2 dp) [c] = any value between 0.05 and 0.20 (2 dp) Formula: [𝑎] 1000 × [𝑏] × 58.44 × [𝑐] Figure 1. Moodle 'Simple Question' type construction. Example scenario requiring functional maths skills. Note the detailed instructions about decimal places and units Figure 2 shows an example question, answer fields and ‘check’ and ‘try again’ options as presented by Moodle to students in the formative version. The question requires the student to process two stages outlined in the first ‘hint’ (figure 2.d). The second hint would be a more detailed description of the stages of calculation. This simple question type can be used to render multiple iterations, depending on the variables, numbering in the 10s to 100s (Figure 3). The formative (revision) quiz settings used the low-stakes approach. This included a generous time allowance, the “interactive with multiple tries” setting and full review options, thus permitting multiple attempts at any question within each attempt at the quiz and access to interim hints for additional support. The multiple iterations prevent students ‘learning’ the answer because each new attempt involves a different set of numbers. Students were reminded throughout the term that their summative assessment would be constructed using the same questions, thus encouraging engagement with the formative quiz (and additional teaching quizzes) and negating the fear of any ‘nasty surprises’ during the summative assessment. Case Studies Compass: Journal of Learning and Teaching, Vol 15, No 2, 2022 6 a: Presentation to the student b: Student enters answer c: After selection of ‘check’ button, a tick appears with the offer to try another similar question Case Studies Compass: Journal of Learning and Teaching, Vol 15, No 2, 2022 7 d: If an incorrect answer is entered and the ‘check’ button selected, a hint is shown and the ‘try again’ button appears. Figure 2. Moodle 'Simple Question' example end-user (student) screens Figure 3. Moodle 'Simple Question' type construction showing example of three iterations from a maximum of thirty. There were no mandatory teaching sessions, in recognition of the broad range of capabilities amongst the cohort. Free-access teaching quizzes for each question category were also provided for additional support. The first question in these quizzes provided teaching materials (e.g. links to videos, web pages and my own screen casts) followed by wild-card questions of increasing difficulty. The same low-stake settings described earlier were applied, with the inclusion of unlimited attempts. Students were instructed to examine their diagnostic test result and make use of these additional teaching resources to improve their outcomes for question categories in which they had achieved only lower scores. Three non-mandatory face-to-face sessions were used to deal with more challenging concepts, such as algebra and molarity. Case Studies Compass: Journal of Learning and Teaching, Vol 15, No 2, 2022 8 The summative assessment at the close of term 1 had a less generous time limit, only a single possible attempt and deferred review options without ‘multiple tries’ settings applied. The provision of multiple iterations made collaboration more difficult, something particularly appropriate during COVID, when assessments were managed off-campus. Ethics approval secured in 2019 permitted the statistical analyses of student maths quiz scores (FES-FREC-18-6.04.11) on the basis that all would be anonymised. Out of 158 students, ninety-seven completed the consent form and agreed to the use of their diagnostic, formative and summative data (scores). The data were analysed in JASP (v0.16.1). JASP (In recognition of Bayesian pioneer Sir Harold Jeffreys), stands for Jeffreys's Amazing Statistics Program and is an open-source project, supported by the University of Amsterdam and designed for students. Its user interface could be considered as an introduction to Statistical Package for Social Sciences (SPSS, IBM) 3. Results Student diagnostic and assessment data was shown not to violate the assumptions of normality or variance. A two-way mixed factor ANOVA was carried out, the 'within subjects' main factor being the type of assessment (diagnostic and summative) and the 'between subjects’ main factor being completion of the formative quiz (Figure 4). Figure 4. Comparison of student (n=97) mean(±CI) diagnostic and summative maths quiz scores (%) categorised by their engagement (l yes) or lack of engagement (¡, no) with a formative quiz. Diagnostic, formative and summative quizzes were exactly the same and consisted of wild-card questions, thus providing multiple iterations of the same quiz questions. There was a significant main effect of assessment type (F(1) = 74.9, p < 0.001. Holm-corrected post hoc testing showed that there was a significant (p < 0.001) increase in performance in the summative assessment (74, SD +/- 24%) compared to the diagnostic assessment scores (52, SD +/- 21%). There was also a significant main effect for completion of the formative quiz (F(1) = 28.7, p < 0.001). A significant test *formative quiz interaction was also found (F(1)=18.6, p <0.001). Simple main effects showed a small but significant increase in summative scores in those students who did not attempt the formative quiz (F(1) = 6.9, p=.1); however, a significantly greater increase in summative scores was seen in those students who did complete the formative quiz (F(1)=131.5, p< 0.001). https://jasp-stats.org/pronamic_faq_category/about-jasp/#:~:text=In%20recognition%20of%20Bayesian%20pioneer,for%20Jeffreys's%20Amazing%20Statistics%20Program. Case Studies Compass: Journal of Learning and Teaching, Vol 15, No 2, 2022 9 4. Discussion The statistical outcomes imply that student maths summative assessment scores were more improved in those who engaged with the formative opportunity than those who did not engage. The mean score of the thirty-one ‘non-engagers’ (M = 44.1%, SD = 18.7%) was significantly lower (t (95) = 2.5, p = 0.02) than that of the sixty-six ‘engagers’ (M = 55.3%, SD = 21.8%) indicating that lower-scoring students did not engage with the formative opportunity despite their apparent need for learning support. Whilst the outcomes of this approach to maths teaching and learning look encouraging, there are confounding factors to consider. Lack of weighting to avoid anxiety in fresher students might have discouraged serious engagement with the diagnostic quiz and so reduced scores, whereas the twenty-five per cent weighting of the summative quiz might have provoked more engagement and thus increased scores. Also, since students would have been registered for a whole term before the summative, any ‘freshers’ anxiety’ might have diminished. The quiz design also needs consideration. It included five simple operator questions and eleven more complex questions, but were there enough questions of sufficiently high complexity to differentiate between student ability at the upper end of scores, particularly in the assessment quiz? It is also clear that students with lower scores in the diagnostic test might need proactive and formalised support. This could be achieved by enforcing engagement with the formative quiz and/or by delivering mandatory teaching sessions. Alternatively, the formative quiz scores could contribute a small portion to the overall score, thus encouraging engagement with formative support. The implication of using low-stakes assessments to improve maths learning for STEM students looks promising and could help improve retention and outcomes for registrants. However, the wider implication for non-STEM University of Greenwich students needs consideration for the reasons stated above re the need for functional numeracy in the workplace. MASH is offered to all students who need help with their assessments, but should we be offering an accredited course for students (and staff?) who can see the value of improving their functional numeracy skills in terms of their personal and work lives? The use of VLE quizzing tools for formative support could be applied to other subjects. As explained previously, external examiners have been suggesting that final-year projects would benefit from more robust processing and interpretation of data. The study skills component for Greenwich second years includes statistics, to prepare students for their final-year project. However, feedback surveys indicate that students are intimidated by statistics and have appreciated the provision of the very basic formative support quizzes offered to date. Future research will address a more comprehensive formative provision. The recent addition of new VLE (Moodle) tools means that we can use a similar approach for the delivery of basic chemistry to life science students. Case Studies Compass: Journal of Learning and Teaching, Vol 15, No 2, 2022 10 Reference list Department for Business Innovation and Skills (2012) ‘The 2011 Skills for Life Survey: A Survey of Literacy, Numeracy and ICT Levels in England.’ BIS, Research Paper 81. London. Available at: www.BIS.gov.uk (Accessed: 21 January 2022). Grayson, R. (2013) Research summary – A brief review of employers’ views on numeracy. Cambridge. Available at: https://www.cambridgeassessment.org.uk/Images/131381- research-summary-a-brief-review-of-employers-views-on-numeracy.pdf (Accessed: 23r January 2022). Jobted (2021) ‘The Science Industry: Jobs, Skills and Job Outlook.’ Jobted. Available at: https://uk.jobted.com/browse/scientific (Accessed: 10 December 2021). Maths Anxiety Trust (2022a) ‘Maths Anxiety Definition.’ Available at: http://mathsanxietytrust.com/ (Accessed: 21 February 2022). Maths Anxiety Trust (2022b) ‘Sue Johnston Wilder.’ Available at: http://mathsanxietytrust.com/johnston-wilder.html (Accessed: 21 February 2022). Maths Anxiety Trust and Ipsos Mori (2018) ‘What is maths anxiety?’ Available at: http://mathsanxietytrust.com/maths-anxiety.html (Accessed: 21 February 2022). National Numeracy Charity (2022a) ‘What is numeracy?’ Available at: https://www.nationalnumeracy.org.uk/what-numeracy (Accessed: 26 January 2022). National Numeracy Charity (2022b) ‘What is the issue?’ Available at: https://www.nationalnumeracy.org.uk/what-numeracy/what-issue (Accessed: 21 February 2022). National Numeracy, Ipsos Mori and KCL (2019) ‘Numerate nation? What the UK thinks about numbers.’ Available at: https://www.nationalnumeracy.org.uk/news/national-numeracy- day-2019-thousands-take-first-step-towards-better-number-skills (Accessed: 22 February 2022). http://www.bis.gov.uk/ https://www.cambridgeassessment.org.uk/Images/131381-research-summary-a-brief-review-of-employers-views-on-numeracy.pdf https://www.cambridgeassessment.org.uk/Images/131381-research-summary-a-brief-review-of-employers-views-on-numeracy.pdf https://uk.jobted.com/browse/scientific http://mathsanxietytrust.com/ http://mathsanxietytrust.com/johnston-wilder.html http://mathsanxietytrust.com/maths-anxiety.html https://www.nationalnumeracy.org.uk/what-numeracy https://www.nationalnumeracy.org.uk/what-numeracy/what-issue https://www.nationalnumeracy.org.uk/news/national-numeracy-day-2019-thousands-take-first-step-towards-better-number-skills https://www.nationalnumeracy.org.uk/news/national-numeracy-day-2019-thousands-take-first-step-towards-better-number-skills Case Studies Compass: Journal of Learning and Teaching, Vol 15, No 2, 2022 11 OECD (2013) PISA 2012 Results: Ready to Learn: Students’ Engagement, Drive and Self- Beliefs (Volume III) PISA. Available at: https://www.oecd-ilibrary.org/education/pisa-2012- results-ready-to-learn-volume-iii/students-drive-and-motivation_9789264201170-7-en (Accessed: 22 February 2022). OECD Publishing (2016) Equations and Inequalities MAKING MATHEMATICS ACCESSIBLE TO ALL. Paris. Available at: https://www.oecd- ilibrary.org/education/equations-and-inequalities_9789264258495-en (Accessed: 18 February 2022). Rozgonjuk, D., Kraav, T., Mikkor, K., Orav-Puurand, K. and Täht, K. (2020) ‘Mathematics anxiety among STEM and social sciences students: the roles of mathematics self-efficacy, and deep and surface approach to learning.’ International Journal of STEM Education, 7(1), 1-11. Available at: https://stemeducationjournal.springeropen.com/articles/10.1186/s40594- 020-00246-z (Accessed: 16 February 2022). Top Hat (2022) ‘Low-Stakes Testing Definition and Meaning.’ Available at: https://tophat.com/glossary/l/low-stakes-testing (Accessed: 16 February 2022). Williams, J. (2003) ‘The Skills for Life survey: A national needs and impact survey of literacy, numeracy and ICT skills.’ Literacy. London. Available at: https://www.gov.uk/government/publications/2011-skills-for-life-survey (Accessed: 13 December 2021). Zacharopoulos, G., Sella, F. and Kadosh, R.C. (2021) ‘The impact of a lack of mathematical education on brain development and future attainment.’ Proceedings of the National Academy of Sciences of the United States of America, 118(24). Available at: https://doi.org/10.1073/pnas.2013155118. https://www.oecd-ilibrary.org/education/pisa-2012-results-ready-to-learn-volume-iii/students-drive-and-motivation_9789264201170-7-en https://www.oecd-ilibrary.org/education/pisa-2012-results-ready-to-learn-volume-iii/students-drive-and-motivation_9789264201170-7-en https://www.oecd-ilibrary.org/education/equations-and-inequalities_9789264258495-en https://www.oecd-ilibrary.org/education/equations-and-inequalities_9789264258495-en https://stemeducationjournal.springeropen.com/articles/10.1186/s40594-020-00246-z https://stemeducationjournal.springeropen.com/articles/10.1186/s40594-020-00246-z https://tophat.com/glossary/l/low-stakes-testing/ https://doi.org/10.1073/pnas.2013155118