The importance of cost-benefit analysis (CBA) - a response Steve Draper University of Glasgow email: s.draper@psy.gla.ac.uk In Issue 11.2, Nicol and Coen (2003) argue for developing a cost-benefit analysis model for learning technology. However, there are three substantive issues with this approach. First, it is problematic because it involves comparing different things; second, because by understanding the purpose of the analysis, it may be possible to justify the necessary simplifications for special cases and limited applications while being explicit about the limitations; and third (and conversely), this can expose errors and omissions in the analysis even after accepting its general principles. The need for cost-benefit analysis is not about ignoring distinctions of quality and value and reducing things to a monetary value. It comes from the need to make decisions. In making a decision alternatives are ordered to enable selection, but these are not always equivalent to each other, which makes this problematic. ICT (information and communication technology) research and development activities generally entail a trade-off between time, money and quality. Because budgets are fixed, decision-making tends to weigh money over quality and timeliness, whereas decisions should be made on appropriate educational factors. Is a model for evaluating the costs and benefits of the use of ICT worthwhile? Yes, because it forces attention on the factors that determine educational outcomes. The aim of the model is to calculate quantitatively in order to make decisions between alternatives. The actual but implicit benefit, it is argued, lies in identifying costs and benefits and then measuring them. But the advantage of this approach is not actually quantitative (being able to make close decisions more accurately) but qualitative (identifying what the key issues are). Proponents argue that the use of their model lies in the value and accuracy of the categories or factors used: however, if these do not correspond to sensible and useful divisions in the first place neither will results. Furthermore, their approach does lead to the generation of a list of important factors for 118 ALT-J Volume / / Number 3 users of their model to consider applying. An aspect that could usefully be added would be to enable searching for important factors. After all, as they assert, this is one of the main benefits of the approach. Pessimism about the value of the quantitative aspect of this type of approach is reinforced by Landauer (1995). He argues that there is little evidence of the benefits of IT to the United States economy, except in a few special technical areas, which is in contrast to the kind of clear economic indicators of the benefits of other inventions. If nearly fifty years of IT investment in the USA do not show clear quantitative benefits, then the prospects of measuring these in an ICT project in one university seem remote. My general view is that we need above all to understand the factors involved: we are far from identifying, let alone quantifying, these. CBA forces us to uncover and consider these factors. Is their approach focused enough on this? I criticize their approach only for not going far enough. The general problem with the accountancy categories used in their model is that they try to assign costs under headings, whereas given costs often produce benefits under several headings. Thus the categories used for costs, far from being 'hard', are questionable. The model considers that costs are 'hard' measures (count the money) and benefits are 'soft', therefore difficult to measure. But this fails to bring out the difficulty of inventing and applying categories that provide useful analysis. The model attempts to classify staff time into either teaching or research, whereas in reality many activities contribute to both. We can measure time (like money) accurately, but this should be assigned solely to either teaching or research. Similarly consider, for example, classifying the cost of a student learning to use email. Is this a skill of benefit to their learning needs, relevant for future employment or something of value to their social life? In reality it is valuable for all of these. These examples bring out not only the inappropriateness of apparently obvious cost categories, but also an issue not faced by Nicol and Coen: about how the same action can be classified differently at different times. A student signing up for email training, or an academic going to a conference will very often see the benefit differently before or after going or again at some time later. So too would an analyst, since more information about the true beneficial consequences is available later; yet decisions have to be made on the basis of advance information. Thus these measures are not objective or stable over time. Furthermore, it suggests that CBA done purely to support management decisions may be rather poor at understanding what the real benefits are, and therefore poor at developing understanding of education and at supporting management decisions in the longer term. Overall there are three problems with the approach adopted. First, accountancy is about applying categories that are general and fixed (for example, by tax or laws governing accounts for public companies): these legal requirements in turn stem from forcing companies to apply the same categories in order to allow investors to compare (unlike) companies. Because of the primitive state of CBA of education, we should build cautiously from small cases, embracing new categories as appropriate and acknowledging the deficiencies and time limitations of each. Generalization may only become possible later. Instead the focus should be on identifying the usefulness of each category at a given time. The preference for standard categories, so important in accountancy, is a liability in an educational context. 119 Steve Draper The importance of cost-benefit analysis (CBA) - a response Second, though they say the paper comes from trying to combine an accountant's approach to costs with an educational evaluator's approach to benefits, in fact the accountant too is identifying benefits not costs. It is not difficult to measure how much money has been spent: the difficulty is in identifying what benefits have been bought by it. Categories such as research vs teaching, perhaps even infrastructure vs value-added activity, are categories of benefit bought by money. Third, money is the only cost (in the sense of negative benefit or limited resource) being analysed in their model. If accountants can only measure money, then other cost experts must be brought in to identify and measure the other costs such as staff stress (soon to become a legal liability to organizations), and student time (not paid for, yet a fixed and limited resource, around which the organization's business is structured). My final point considers the issue of learning costs, which is paramount as ICT projects often fail to become embedded; the costs of addressing associated staff development needs were not included. In addition, learning costs are particularly tricky because they change over time. In the USA, there is now an assumption that most students have a basic familiarity with computers. This means that training in basic computer literacy has moved to being a remedial activity for a minority. Learning costs are crucial to analyse because they have a major effect on when a given project may become worthwhile; which disciplines it will and will not work for; and economies of scale: it may work only for large-scale change where a single learning cost gives benefits on repeated occasions. There is no space for more detailed discussion: I have concentrated on the main issues. To support decision-making and so to do anything of practical usefulness to learners, we must be able to reduce all alternatives to a single order of preference. This does not necessarily require expressing their merits in terms of money or any other numerical score; but it does require comparing qualitatively unlike things. Nicol and Coen are tackling one of the most important issues in the education field and one with enormous potential benefits to learners. However the benefits may be a long way off, because the difficulties are substantial. I thus fully endorse their overall aim, but is progress possible? We need to uncover the associated factors and issues. Their method should be modified to stress an open-ended enquiry phase of active search for the important factors in each case and learning costs should be included in the classification. Furthermore it is important to consider whether the model can reason about the cost-time-(learning) quality triangle that implicitly dominates most practical educational decision-making in both research projects and daily practice; or alternatively if you follow Phillips (1996) and Reeves (1992) then you will require it to reason about the quantity-quality-cost trade off. A longer version of this commentary is available at: http://www.psy.gla.ac.ukhstevelqqa.html References Nicol, D. and Coen, M. (2003), 'A model for evaluating the institutional costs and benefits of ICT initiatives in teaching and learning in higher education', ALT-J, 11 (2), 46-60. Landauer, T. K. (1995), The Trouble with Computers: Usefulness, Usability, and Productivity, Cambridge, MA: MIT Press. Phillips, R. (ed.) (1996), Developer's Guide to Interactive Multimedia: A Methodology for 120 ALT-] Volume 11 Number 3 Educational Applications, Perth, Western Australia: Computing Centre, Curtin University of Technology. Reeves, T. C. (1992), 'Evaluating interactive multimedia', Educational Technology, May, 47-52. 121