The importance of cost-benefit analysis:
a response

David Nicol and Michael Coen
University of Strathclyde
email: d.j.nicol@strath.ac.uk

The critique by Draper raises some interesting points that we did not have space to discuss
in our published paper. As he points out, taking a purely quantitative approach to the
evaluation of ICT investments in teaching and learning is wholly inappropriate. However,
in this transitional period, where ICT applications are new and the effects on operational
processes within higher education institutions are unknown, it is not only qualitative issues
that need to be investigated but also the potential changes to the scope and nature of the
costs incurred by institutions. While the small-scale, and localized, introduction of ICT in
teaching might only affect the time and effort of a few individual academics, large-scale
deployment of the same methodology may require substantial institutional investment (for
example, in network infrastructure, hardware, licenses, support staff). The CBA model
encourages institutions to consider and record all the cost implications of their strategies,
not in an attempt to quantify the outputs (benefits) of these new learning processes but to
identify and quantify the inputs to these processes. These quantitative inputs can then be
evaluated in the context of qualitative outputs.

Draper argues that the merit of our model lies in 'the value and accuracy of the categories
or factors . . . [benefits and costs].. . used'. He then notes areas where the model fails to
take into account some important factors (for example, staff stress, student time). While
we agree the factors chosen for the CBA are critical, we did not intend that users of the
model would merely select from our example list of benefits. Instead, we envisaged that a
decision-making group, with representatives of all those with a stake in the CBA, would
discuss and formulate their own benefit types in relation to institutional strategies and
context. A limited set of benefits, those that matter most to the institution, would then be
used to evaluate the investment options. Similarly, institutions would determine what costs
to include within the costing framework. Hence Draper's examples of staff stress and
student time could be accommodated. If, as a consequence of an investment, academic
staff become unduly stressed then this would show up through some indicator of low-staff

122



Au-J Volume 11 Number 3

satisfaction if that were one of the benefit categories. Likewise, if student time is an
important consideration, this could be incorporated into the model by including student
time or student satisfaction as an evaluation criterion on the benefits side.

As Draper highlights, there are difficulties in the model when it comes to allocating cost
information to activities. For example, time spent at a conference might benefit both
teaching and research activities. Although costing staff time is difficult (see Rumble, 1997),
we must attempt at least rough estimates of allocations for different activities. Otherwise
we might as well abandon any attempt to evaluate activities within higher education or to
plan future investments. That said, the model advocates a light touch approach to
estimating staff time, a much less rigorous methodology than that recommended by the
HE Transparency Review (Joint Costing and Pricing Steering Group (JCPSG), 1999). This
type of broad estimating has the inevitable consequence that the 'cost' side of the CBA
exercise cannot claim great accuracy. However, we believe that the process of investigating
and debating the costs is, in itself, useful and that sensitivity analysis can be used to assess
the potential impact that variations in the estimates of staff time will have on the final
CBA evaluation outcomes.

Just as the model does not limit what benefits can be included, the model does not
proscribe what costs should be included in the evaluation. Indeed, the model encourages
institutions to identify all activities that have a bearing on the investment options under
evaluation. Draper is concerned about 'learning costs' such as the basic skills training of
new students. The model represents this as a 'support activity'. A percentage of the total
cost (made up of staff costs, revenue costs, capital costs, etc.) of 'learning' support can be
allocated to the activities associated with each ICT investment option; that percentage
would be determined using a suitable metric (for example, number of students).

It is true that the CBA model attempts to reduce all factors to quantifiable inputs and
measurable, but qualitative, outputs. Users end up with a single figure for each
implementation option, although choice is based on what trade-off between costs and
benefits fits best with institutional priorities. The institution might decide on the lowest
cost, the highest benefit or something in between. However, the process of getting to these
inputs and outputs is much more important than the figures themselves.

While defending the model we acknowledge that many problems remain. In the original
paper we implicitly assumed that large-scale ICT intervention in learning would be 'top-
down', that is, a strategy would be defined and options could be evaluated through CBA
with reference to an institution's strategic objectives (potential benefits). While an
individual's perspective of the benefits of an activity might vary over time (before, during
and after), as Draper notes, we assumed that the long-term strategic objectives expressed in
an institution's learning or e-learning strategy would be stable over time and would provide
an appropriate context for the evaluation of large-scale investments. Moreover, we
assumed that these strategic objectives would have been developed by an institution based
on its understanding of educational processes not, as Draper seems to suggest, that the
CBA evaluation would be used to investigate the learning processes themselves.

However, further work in studying the risks associated with ICT investments in teaching
and learning, has led us to review some of these assumptions (Nicol and Coen, 2003). This
type of 'top-down' approach to strategic management and investment is not the only

123



David Nicol and Michael Caen The importance of cost-benefit analysis: a response

approach that can be taken. Many institutions adopt a 'bottom-up' approach - funding a
range of ICT projects with the intention of gaining practical experience and exploring
(rather than planning) benefits. With this 'thousand flowers bloom' approach institutions
refine their strategies based on the outcomes of pilot implementations and a complex mix
of factors (pedagogical, political, cultural, financial) determine which models are more
widely adopted within the institution.

As Draper points out, we do not yet fully understand all the consequences for institutions
of ICT investment in teaching and learning. Therefore, where 'bottom-up' experimentation
is prevalent, CBA, with its reliance on a set of planned objectives, may not be the most
appropriate tool. Other techniques such as the Balanced Scorecard (Kaplan and Norton,
1996) may provide a better tool for the management and evaluation of ICT investments. In
the Balanced Scorecard methodology a handful of critical perspectives are defined, such as
a 'financial' perspective, a student perspective and a staff perspective. The methodology
advocates the use of performance measurement to facilitate a process of continual
performance review and improvement against each of these perspectives. This type of
management tool may have particular applicability in early, small-scale implementations of
ICT in teaching and learning where costs and benefits are difficult to forecast and
resources (staff time and cash expenditure) are not limited to, or defined by, a strict
budgetary limit. This type of methodology might also address Draper's call for an
alternative management method that is less heavily reliant on accountancy and financial
considerations.

However, at some point each institution, armed with the knowledge gained from its
experimentation, is forced to decide whether to invest potentially large sums of money in
expanding and/or embedding successful models or, in rare cases, transforming the learning
experience and associated institutional processes entirely. In this type of situation much
more structured evaluation, of the type proposed by the Insight CBA model, is required.
The model cannot guarantee that the information on costs and benefits that institutions
feed into it is appropriate, but it does provide a structured, transparent framework through
which institutions can insure that all relevant views have been reflected in the CBA and in
strategic investment decisions.

References

Joint Costing and Pricing Steering Group (JCPSG) (1999), Transparency Review,
Manchester, http://www.jcpsg.ac.uk/transpar/index.htm

Kaplan, R. S. and Norton, D. P. (1996), The Balanced Scorecard: Translating Strategy into
Action, Cambridge, MA: Harvard Business School Press.

Nicol, D. J. and Coen, M. (2003), 'A framework for identifying the risks associated with e-
learning investments FE and HE', University of Strathclyde, unpublished manuscript.

Rumble, G. (1997), The Costs and Economics of Open and Distance Learning, London:
Kogan Page.

124