tam Australian Journal of Educational Technology Development and evaluation of a computer-based learning pilot project for the teaching of holistic accounting concepts Maureen Tam Lingnan College, Tuen Mun, Hong Kong Stephen Wedd Margaret McKerchar Orange Agricultural College, The University of Sydney This paper reports on the development and evaluation of a computer-based self-instructional program for teaching holistic accounting concepts. It is the aim of the program to focus on the pedagogical use of CBL (computer- based learning) to improve conceptual understanding of accounting principles and to develop higher order thinking skills such as synthesis of information, evaluation and inferential judgements. The project was conducted in two phases, involving firstly the development of a CBL pilot program to present and simulate the linkages and impact of individual business decisions or transactions on the accounting system; and secondly an evaluation of the program to prove that the instructional strategies had led to a better conceptual understanding of accounting principles and greater student motivation. Evaluation results showed that students of the CBL group had a more favourable attitude to learning and their test scores were appreciably better. Project background Our experience of teaching rural accounting subjects at Orange Agricultural College informs us that students are generally not capable of seeing linkages among components in the accounting system. This may be very much due to the fact that accounting principles are usually taught in a compartmentalised way with emphasis on technical mastery of skills through a linear-structured curriculum. This problem is further aggravated by specialised teaching of specific accounting content by individual accounting lecturers. While students may have learned some specific accounting principles very well in isolation, they are not able to see how different concepts are linked to each other and Tam, Wedd and McKerchar 55 how changes in some parts of the accounting system may impact on the system as a whole. It is the holistic understanding that we think is important to accounting teaching. Encel (1976) presented a similar view about accounting teaching when he made the criticism that the objectives of accounting education have concentrated too much on content and preparing students to practice, hence resulting in curricula which were too narrowly specialised and procedural. Similarly, Zeff and Dharan (1994) advocated that the goal of accounting education should be to produce accounting thinkers, rather than accounting technicians. One typical example from our teaching is that while students may know very well technically how to enter various rural business transactions into appropriate accounting journals, they usually fail to predict and assess the impact of these decisions on the business health situation as can always be reflected in various accounting reports including the Profit and Loss Statements, Trading Accounts, and Balance Sheets. It was our quest for teaching analytical thinking and conceptual understanding that gave birth to this project to focus on holistic understanding of the accounting system which is characterised by many intricate linkages and interconnectedness between components. To achieve this teaching goal, traditional instructional methods such as lectures and print-based materials were considered inadequate to demonstrate and simulate the impact of business transactions on the accounting system and present linkages within it. As a result, a computer- based learning (CBL) approach was suggested to make use of the capabilities of the computer to show instant changes and simulate the impact of business transactions by way of clever interactive instructional design. Computer technology, as commended by Beattie (1994), allows the presentation of three-dimensional and dynamic systems to the students, sometimes even putting the parameters of the system under students' control so they can manipulate the system and test hypotheses. These capabilities of computer technology offered us the opportunities to show the dynamic changes in the accounting system caused by business transactions, to allow students to simulate impact and to predict consequences through the manipulation of data and scenarios, and finally to interpret and analyse the information for business decision making and inferential judgements. Australian Journal of Educational Technology, 1997, 13(1) 56 Given the dynamic nature of the accounting system that we want to show, computer technology offers a teaching/learning system that is not otherwise possible by way of a two-dimensional representation on a blackboard or in a textbook. As a result, a computer-based self-instructional program was considered appropriate to provide supplementary tuition about the holistic accounting system through the use of computer simulations and case studies. The development and evaluation process Having selected the CBL format to solve the accounting learning problem that we have identified, the project was undertaken in two phases: Development and Evaluation. The development phase: Major activities in this phase included content analysis, instructional design and computer programming. The project team consisting of an accounting subject expert, instructional designer and programmer met initially to decide precisely on the content to be presented, and analysed it in relation to the teaching objectives that we wanted to achieve. The initial scripting of the program content was conducted by the subject expert. The important task here was to think carefully about what we were trying to teach and what the discrepancy was between the current state of our students' knowledge and our identified ideal. By keeping this focus we were able to avoid designing something that was inappropriate and to be clear about the capabilities of the technology. Further development of the CBL occurred when the instructional designer reformatted the material into screens of information. The challenge here was to capitalise on the capabilities of the computer to present the accounting concepts in an interactive way using screen juxtapositions, timed display, animation, instant feedback and other techniques that were possible with the computer. In the design process, some flexible learning features offered by computer technology, such as learner control of study pathways and pace, were also incorporated. The next stage was computer programming and screen design. After the instructional designer had reproduced the content in the form of screen layouts on paper, the programmer started to work from there using an authoring software. Throughout the programming process, the programmer needed at times to come back to the instructional designer to clarify and discuss ideas, and to adjust plans to accommodate the capabilities and constraints of the authoring program. Formative evaluation and feedback Tam, Wedd and McKerchar 57 from the team on the prototype CBL program were frequent throughout the development process. This helped to identify problem areas, revise learning sequences and make adjustments in the early stages of development. The CBL development has taken us much longer than we thought. It spanned a period of seven months with all team members working on the project on a voluntary basis while attending to other normal duties at work. Piemme (1988) roughly estimated that it may take a programmer 40 hours of work to produce one hour of content. However, the range can be from 50 to 500 hours (Criswell, 1989) for a project team to develop one hour of material for student use. The development process resulted in a pilot program aiming to teach holistic accounting concepts by way of computer simulations and case studies. The program is in three parts. Part One is an optional background module with a test in it to assess whether a student has the prerequisite knowledge to start with. If the test results show that the student needs to acquire some background knowledge about the accounting system, s/he is expected to work through the background module to learn some basic accounting concepts, such as 'the Australian Journal of Educational Technology, 1997, 13(1) 58 accounting equation', and the necessary terminology frequently used in the CBL program. Part Two is a learning module about rural business transactions and how they are linked to various accounting reports. There are two components in Part Two. The first component is a demonstration by the computer in six scenarios about the linkages and impact caused by business transactions on the accounting system as reflected in financial reports such as Profit and Loss Statements, Trading Accounts and Balance Sheets. The preceding and following screen shots were taken from one of the six scenarios which demonstrated the instant impact of a particular business transaction had on the accounting system. The impact was then explained with reference to the accounting equation, showing the relationship between equity, assets and liabilities. The second component is the learner-controlled part in which students are given another six scenarios to predict the linkages and assess changes caused by the business transactions, and to work out the significance of these to the business health situation as interpreted from various accounting Tam, Wedd and McKerchar 59 reports. The six scenarios presented aimed to test students’ understanding of the linkages in the accounting system, reinforcing the concept of the accounting equation and the interconnected changes caused by business transactions. Part Three is a learning module about transaction analysis in relation to the concepts of 'stability' and 'profitability'. Again, students are first presented with instructions and simulations about the analysis in case studies. Afterwards, it is their turn to work out the analysis themselves by measuring the changes caused by business transactions against the concepts of 'stability' and 'profitability' in both long- and short-terms. The evaluation phase: The purpose of the evaluation process was to establish whether the identified learning problem was overcome as a result of the CBL program. Questions such as, 'Do they like it?' or 'Do they find it useful?', while important, are by themselves inadequate. The more fundamental evaluative questions should be: 'What is being learned by students?', 'How will they demonstrate this learning?' and 'Does the CBL program support students in achieving the learning objectives that we want them to achieve?'. Australian Journal of Educational Technology, 1997, 13(1) 60 According to Beattie (1994), a good evaluation needs to establish whether a learning problem is being overcome, and to attempt to investigate deeply the specific learning experiences students are having as a result of the teaching intervention. In other words, in the context of our project, it was important to find out if the CBL program had helped to solve the learning problem that we were concerned about, that is, the lack of holistic understanding of accounting principles in our students. The evaluation of the CBL program took several forms: peer review, student evaluation and pre- and post-tests. An evaluation questionnaire adapted from the guidelines provided by Caffarella (1987) was designed to collect feedback from a small group of lecturers teaching rural accounting subjects at Orange Agricultural College. This peer review was done prior to the evaluation by students with the aim to obtain preliminary feedback about the teaching relevance and effectiveness of the CBL program. After the peer review, anything that was perceived as inappropriate, unclear or incorrect was eliminated and changed as considered necessary by the team. This resulted in a revised version of the CBL program ready for student evaluation. For student evaluation, a questionnaire slightly modified from the one used in the peer review was administered to a group of rural accounting students who were given the CBL program to work through on their own. In the questionnaire, questions were directed towards user's satisfaction and perceived usefulness of the CBL program in helping them to learn the accounting principles presented. Pre- and post-tests to measure what students had learned and hence the teaching effectiveness of the CBL program, were also conducted. The same tests were given to two groups of rural accounting students whose academic abilities and gender were matched to ensure that any discrepancies in the test results would be mainly attributable to the CBL program. One group was used as the control group who did the same pre- and post-tests without working through the CBL program. The two tests, before and after, were almost identical, except that the questions in the post-test were randomised to avoid direct recall of answers from the pre-test. The two tests, each of twenty questions were set by the subject expert to test chiefly conceptual understanding of holistic accounting principles in relation to linkages and impact caused by business transactions on the accounting system as a whole. The tests were eventually marked by another accounting lecturer not involved in the project. Tam, Wedd and McKerchar 61 The purpose of the pre- and post-tests was to ascertain the impact of the CBL package on students’ learning, in particular, their understanding of the holistic aspects of the accounting system. In other words, the research design involved mainly a comparison of CBL instruction versus no CBL instruction, and a measure of the difference. Though limiting it might seem, it was considered somewhat useful to shed light on the value of the CBL package for teaching interconnected accounting concepts. Remember that the focus of this research project was not to revamp or change the entire instruction and curriculum in the current accounting subjects. Rather it was to develop a CBL package that aimed to supplement the current accounting teaching by emphasizing the holistic aspects of the accounting system. Evaluation results Student evaluation Eight students worked through the CBL program and filled out the evaluation questionnaire. The results are summarised as follows: Program content Q1. The content of the CBL program was well- structured and organised. 1 2 3 4 5 Never Sometimes Always Mean = 4.0 sd = 0.189 Q2. The content was appropriate to the aim of the program. 1 2 3 4 5 Never Sometimes Always Mean = 4.750 sd = 0.164 Q3. The amount of information presented was 1 2 3 4 5 Too Little Just right Too much Mean = 3.125 sd= 0.125 Q4. The level of information presented was: 1 2 3 4 5 Too easy Just right Too difficult Mean = 3.000 sd = 0.189 Instructional strategies Q5. The CBL program presented information clearly. 1 2 3 4 5 Never Sometimes Always Mean = 4.250 sd = 0.250 Q6. The program was interesting which motivates learning. 1 2 3 4 5 Never Sometimes Always Mean = 4.125 sd = 0.227 Q7. The program monitored what I have learned 1 2 3 4 5 Never Sometimes Always Mean = 4.000 sd = 0.463 Australian Journal of Educational Technology, 1997, 13(1) 62 Q8. The program made effective use of feedback to help me to learn. 1 2 3 4 5 Never Sometimes Always Mean = 3.875 sd = 0.350 Q9. I could control the learning sequence within the program. 1 2 3 4 5 Never Sometimes Always Mean = 4.250 sd = 0.412 Q10 The use of graphics and colour contributed to effective learning. 1 2 3 4 5 Never Sometimes Always Mean = 4.250 sd = 0.250 Q11. The exercises/activities in the program helped my understanding of concepts. 1 2 3 4 5 Never Sometimes Always Mean = 4.500 sd = 0.267 Q12. The program was so designed that it helped me to learn the accounting principles. 1 2 3 4 5 Never Sometimes Always Mean = 4.500 sd = 0.267 Program techniques Q13. The program took advantage of the interactive capability of the computer 1 2 3 4 5 Never Sometimes Always Mean = 4.250 sd = 0.267 Q14. The screen displays were readable, logically arranged and pleasing to look at 1 2 3 4 5 Never Sometimes Always Mean = 3.750 sd = 0.250 Q15. The program ran properly. 1 2 3 4 5 Never Sometimes Always Mean = 3.750 sd = 0.366 Q16. The program was easy to use. 1 2 3 4 5 Never Sometimes Always Mean = 4.375 sd = 0.263 Q17. The program directions were clear. 1 2 3 4 5 Never Sometimes Always Mean = 4.125 sd = 0.227 Q18. The program made appropriate use of prompts and clues. 1 2 3 4 5 Never Sometimes Always Mean = 3.875 sd = 0.398 Q19. The program used consistent commands and directions throughout 1 2 3 4 5 Never Sometimes Always Mean = 4.500 sd = 0.267 Q20. The program was free from programming errors. 1 2 3 4 5 Never Sometimes Always Mean = 3.250 sd = 0.164 Tam, Wedd and McKerchar 63 Overall evaluation Q21. How long did it take you to work through this program? The range is from 35 to 60 mins. The average = 47.125 mins. Q22. What is your overall evaluation of the program? Did you enjoy working on it? * It was quite enjoyable and very easy to use. * For the stage it is at in development, it was very good. * Once all the hiccups have been taken out, it will be quite beneficial for people wanting to learn about accounting. I enjoyed working with it, it was fun. * Yes, I did enjoy working on it, it was interesting. Pictures and colour will make it better. Good to learn from as it is easy to understand. * Interesting and gives you a chance to confirm knowledge through Question and Answer. * Yes, it helps you along then makes you think about the concepts. * I thought it was good. The graphics keep you interested. The content given was good and easily understood. * Yes, though I thought I rushed a bit due to everyone around me starting a bit earlier - thought I was too slow. Q23. Please identify any parts of the program where you have experienced difficulties, or something is not clear to you. * None * Some questions could have been worded better. * already stated as going through it. * If a question is wrong more explanation, or the chance to do the question again before the right answer is told. * Sometimes didn't give enough time before prompts. If more than one answer doesn't give enough time to complete answer. * Profitability/stability section takes a while to grasp concept. A few exercises showing instead of just explaining could be of use. * The scenarios are hard to read as the boxes and writing are quite small. * It was hard to start to get into the program due to not using other programs before. Q24. Do you prefer the CBL program to other ways of presenting the same accounting principles? Why? (Note: Other ways of presentation may include lectures, notes, study guides, and textbooks). Australian Journal of Educational Technology, 1997, 13(1) 64 * Yes, because it's interesting, quick and fun. * Easy to understand and was at a good pace. * Yes, as you can work at your own pace and go back if need be. * Yes, it is active learning, not just writing down notes and forgetting about them, this involves questions. Good to see where things go in financial statements and in what statements things go. * Yes, can work at own pace without outside interference. Also have a chance through Question and Answer to find out if you have a clear concept. * Yes and No. This is a good overview once the work has been presented slowly. Or it should be a good way for lecturers to use CBL in conjunction with lecturing i.e. the lecture goes through each step as students use computer. * Yes, it keeps you more interested as it's not just reading and there is something else to do and focus on. * I like this way though I would rather use some textbook as you can go back and look at certain parts. Peer review Only two accounting peers took part in the review. While it is not possible to generalise too much from only two responses, the lecturers involved rated the package highly as an effective teaching tool and would consider using the program in their teaching. It is worth noting that both peers agreed unanimously that the program was so designed to achieve its desired level of learning and that the exercises/activities in the program helped the understanding of concepts. (Both items scored 5 - the highest point on the Likert scale.) Pre- and post-tests results The test results are reported in Table 1. Discussion of results The student evaluation results gave a clear and positive message. Students generally enjoyed working through the program and found the CBL presentation effective in teaching rural accounting concepts. The three highest rated positive features of the CBL program were the appropriateness of the content (mean = 4.750), the activities/exercises in the program which had helped understanding of accounting concepts (mean = 4.500), and the fact that the program was so designed to help learning (mean = 4.500). Besides, students also found the CBL program easy to use (mean = 4.375) and the amount and level of information presented were Tam, Wedd and McKerchar 65 Pre-test results Post-test results Difference Group A Student 1 40 46 +6 Student 2 31 31 0 Student 3 28 55 +27 Student 4 38 43 +5 Student 5 50 53 +3 Student 6 70 84 +14 Student 7 71 88 +17 Student 8 44 58 +14 t-value = 3.41 significance = 0.011 Group B Student 1 31 39 +8 Student 2 37 46 +9 Student 3 52 49 -3 Student 4 35 28 -7 Student 5 18 32 +14 t-value = 1.07 significance = 0.347 Table 1: Pre- and post-test results. Group A - students who have worked through the CBL program between the pre- and post-tests. Group B - the control group. found just right (mean = 3.125 and 3.000 indicating appropriateness). The lowest rated item (mean = 3.250) was about programming errors which were eventually taken out to result in a final version after both student and peer reviews. The responses of the two academics involved were equally positive. Although the number of peers involved was small, the review indicated an approval from them about the content and the appropriateness of the CBL program in teaching higher-order accounting concepts. It was encouraging to see that both lecturers were happy to use the program in their teaching and endorsed the program as an effective teaching tool. The results from the pre- and post-tests were particularly pleasing. The difference between the pre- and post-test results within each of the two groups showed that all students (except one) who had worked through the program had improved their test results rather significantly. However, in the control group, the results were mixed. Three out of five students had Australian Journal of Educational Technology, 1997, 13(1) 66 showed some sign of improvement between the tests, but two students were going downhill in the post-test. Conclusions and implications for further research Despite the generally pleasing test results, we could not draw firm conclusions from the tests due to the unequal number of students in the two groups and the fact that positive or negative test results could not be fully explained by the intervention of the CBL program. However, caution had already been exercised to qualify the CBL intervention by using pre- and post-tests that were identical and matched. According to Beattie (1994), the terminology 'pre- and post-' testing is usually reserved for the situation in which the before/after tests are equivalent. Though the pre- and post-tests are valid measures of the learning difference possibly caused by the CBL package, there is a need for further research. The purpose of further research is to investigate the effect of an overall course in rural accountancy and its ability to develop an holistic understanding versus the same course plus the CBL package that substitutes for the traditional teaching of the holistic accounting principles. In other words, the research design will involve a comparison of traditional teaching with CBL teaching of the same accountancy principles, and a measure of their difference in effectiveness. Although the review by students and peers, and the test results indicated an overall positive evaluation of the CBL program, the CBL program was not without flaws. The project team was aware of the need to continuously improve its presentation and design, especially after the CBL program is adopted and integrated into the formal teaching of rural accounting subjects in future at Orange Agricultural College. Continuous development and evaluation of the CBL program is an integral part of the process to gauge its effectiveness in solving the learning problem identified. Acknowledgements This project was funded and supported in 1995-96 by the Rural Industries Research and Development Corporation (RIRDC) and Orange Agricultural College, The University of Sydney. Tam, Wedd and McKerchar 67 References Beattie, K. (1994). How to avoid inadequate evaluation of software for learning. In K. Beattie et al (ed.) Interactive Multimedia in University Education: Designing for Change in Teaching and Learning, Elsevier Science B.V. Caffarella, E. P. (1987). Evaluating the new generation of computer-based instructional software. Educational Technology, 27(4), 19-24. Criswell, E. L. (1989). The Design of Computer-Based Instruction, Macmillan Publishing Company, New York. Encel, S. (1976). Education and politics. In S. D'ruso (ed.) Counterpoints: Critical Writings on Australian Education, Wiley International, Hong Kong. Piemme, T. E. (1988). Computer-assisted learning and evaluation in medicine. JAMA, 260, 367-372. Zeff, S. A. and Dharan, B. G. (1994). Readings and Notes on Financial Accounting: Issues and Controversies, McGraw-Hill, Inc., New York. Maureen Tam Lingnan College, Tuen Mun, Hong Kong. Stephen Wedd and Margaret McKerchar Orange Agricultural College, The University of Sydney.