Microsoft Word - Revised model paper - final 1 Abstract—Undoubtedly Information and communication Technologies (ICT) contribute to development; however there is a need to know how and the extent to which development occurs. Moreover the evaluation of the ICT contribution to development has been challenged from theoretical, ethical and methodological angles. This paper addresses some of these challenges by proposing a model that enables systematic evaluation of the ICT contribution to development. The proposed model is conceptually motivated by Amartya Sen’s capability approach that defines development as freedom. Development is a process that involves the provision of opportunities (capabilities) from an ICT resource, as well as actually exploiting the opportunities to realize development benefits. The conversion of resources to opportunities and opportunities to development benefits is facilitated or inhibited by various contextual factors. Development from the capability perspective is both people- centered and multidimensional. This requires consideration of both instrumental effectiveness and intrinsic importance. Consequently five evaluation dimensions concerning social and economic development are proposed; namely research and education opportunities, healthcare, economic facilities, political freedoms and psychological wellbeing. ICT4D evaluation indicators are suggested for each dimension and a Multi-criteria decision analysis (MCDA) structured evaluation process is proposed to guide the evaluation. The application of a structured evaluation approach is illustrated through the example of an online learning environment at a University in a developing country. Future research is underway to further apply and validate the model in practice. Index Terms—Capability Approach, Development evaluation, ICT contribution to development, ICT Indicators, ICT4D, ICT4D Impact evaluation, Multi-criteria decision analysis. I. INTRODUCTION ARGE Information and communication Technology (ICT) investments especially in developing countries are motivated by the notion that ICT actually contributes to development[1]. Despite the high rates of ICT diffusion and uptake over the years, the equally high failure rates continue to raise skepticism as to whether ICT is actually contributing to development. This explains the increase in ICT evaluation studies aimed at establishing whether and how ICT contributes to development [c.f. 2, 3-7]. Such evaluation facilitates the identification of benefits achieved from investments; advises future investments; enables prioritization; forecasts potential impacts; as well as facilitating accountability exercises [1:628]. The evaluations further provide an understanding of the complexities involved in the translation of an ICT resource into a development benefit. Over the years ICT4D evaluation has moved from focusing on assessing diffusion in terms of availability, access and use of ICT, to measuring benefits and sometimes challenges of ICT [1]. While availability, access and use evaluation approaches are mostly performed at the macro level of analysis; the latter is mostly micro-based. It focuses on individual or community evaluation and is achieved predominantly through qualitative in-depth descriptions. Despite the increase of studies in ICT4D impact evaluation, the contribution ICT makes to development is still elusive [8]. This is evidenced by the existence of calls for such studies [i.e. 1, 9-11], to which the current study aims to respond. One of the concerns is how an objective and structured approach can facilitate the evaluation of the ICT contribution to development. It is noted that while the qualitative evaluation approaches to ICT4D offer rich in-depth explanations of how development has occurred, they are normally difficult to report, and may require longer study periods which are subsequently expensive[8].The difficulty in reporting is especially true for evaluations at macro level, or exercises involving several projects for which in-depth qualitative assessments may not be viable. The evaluation is also challenged by other methodological and ethical factors. Principle among these is how a developmental impact could be attributed to a single intervention since impact occurs after sometime, and there could be other contributing factors. Although this could be addressed through systematic evaluation exercises, such approaches may not be possible in certain instances e.g. due to costs or when the need for impact assessment is realized later in the project lifecycle. Gomez and Pather [8] and Heeks [1] also cite the lack of well- formulated theoretical foundations to devise appropriate impact measures that guide data collection as well as analysis. In addition since studies into the contribution of ICT4D evaluation exercises are in their infancy, there are challenges regarding the availability of data. This calls for structured approaches to facilitate an objective impact evaluation process of the ICT contribution to development [8, 12]. It is envisaged that the structured approach streamlines the data collection and analysis process to ensure that the method is not too simplistic to overlook essential details and neither is it too elaborate to inhibit proper reporting. To contribute to a growing field of ICT4D evaluation, this paper addresses some of these challenges by specifically proposing a model that enables systematic evaluation of the ICT contribution to development based on an indicator-based approach. While the use of indicators in the Towards a Structured Approach for Evaluating the ICT Contribution to Development Florence Nameere Kivunike, Love Ekenberg, Mats Danielson, F. F. Tusubira L 2 evaluation of the ICT contribution to development is still in its infancy, this has been demonstrated in an earlier research which laid a foundation for this extended study [13]. Gigler [14] also demonstrates how an indicator based approach can be used to evaluate people’s perception of the impact of the internet in the different well-being dimensions. He further demonstrates the importance of target beneficiaries in such an evaluation exercise; and the contextual factors determine whether ICTs will or will not contribute to wellbeing. It is also envisaged that a structured approach would facilitate the evaluation of the ICT contribution at a higher (macro/meso) level such as national development goals or strategies at policy level e.g. healthcare delivery, education, universal access etc. According to Walsham [15] evaluations at the strategic policy level are indicative of a move towards inclusive development rather than selective development for only a selected few. A literature review of the current state of evaluating the ICT contribution follows. A discussion of the underlying conceptual foundations applied in this study; as well as the composition and interactions of the proposed model are then presented in the next section. This is followed by an outline of a multi-criteria decision approach that illustrates how the evaluation can be performed. The paper concludes with a discussion of limitations, and recommendations for future works. II. LITERATURE REVIEW – EVALUATING THE ICT CONTRIBUTION TO DEVELOPMENT There is substantial research into the evaluation of the ICT contribution to development, specifically from a development perspective. The main development concepts guiding ICT4D evaluation are Amartya Sen’s [16] capability approach and the sustainable livelihoods approach - SLA [17]. For example Hatakka and Lagsten [18] apply the capability approach to assess how students use internet resources to facilitate their learning. Respondents were masters’ students of an international e-government course that mostly consisted of students from developing countries. The study reveals that Internet resources facilitate student learning at the educational, personal and professional levels. However the students’ choice to exploit/benefit from the resources is restricted by a number of factors including personal interests and motivation, the incentives to use the resources, the applied pedagogical techniques, etc. The main aim of the study was to test the capability approach as a development evaluation approach. Ibrahim-Dasuki et al [3] also apply the freedoms concept of the capability approach as an evaluative space of the developmental impact the electricity pre-paid billing system has had in Nigeria. They generally established that the project had not fully realised its development impact. For while the pre-paid billing system had enabled the freedom of transparency through the elimination of estimated billing, the officials still requested for bribes to have the system installed at the consumers’ premises (pp. 43). Furthermore De’[19] and Madon [7] demonstrate how various components of the capability approach may be applied for the development appraisal of e-government projects in India. De’[19]applies the development concept to evaluate Bhoomi, a land records digitization project. While it was a successfully implemented project that met all the implementation goals, the development benefits towards meeting citizen needs were not so clear. Parkinson and Ramirez [20]also applied the SLA for the assessment of the impact of a telecentre in Colombia on the livelihoods of people within the community. They argue that the SLA facilitates a broader scope of analysis and provides better analytical rigour. It enabled them to establish the kinds of risks and vulnerabilities that people faced, some of the key factors that determined their livelihoods, and how their use of the Internet or other telecentre services link to their livelihood strategies. The majority of the ICT4D evaluation studies cited above apply the capability approach to perform in-depth descriptive analysis. As a point of departure and contribution to this body of knowledge, the model suggested in this study illustrates the use of a structured evaluation approach based on indicators in the evaluation of the ICT contribution to development. III. CONCEPTUAL FOUNDATIONS Drawing on theoretical and conceptual foundations is essential to realizing sound evaluation approaches to support ICT evaluations [1, 8]. This facilitates the understanding of how technology interacts with society to achieve development. ICT4D studies fall within an emergent multidisciplinary field now referred to as “development informatics” that seeks to integrate development theories within information systems, communication studies as well as computer science [21]. This fairly new field resulted from the realization that there is more to ICT4D than just diffusion, adoption and use. The need to establish the real ICT benefits in terms of what they are used for within various contexts called for new approaches. Consequently, there is need for sound theoretical premises as a basis for research on how ICT got integrated and affected people’s everyday lives, businesses as well as national and international development goals. Starting with the ICT4D value chain as a guide, the focus of evaluation in terms of the ICT4D implementation lifecycle in this study is identified. The Capability approach is then applied to facilitate the definition and understanding of what development is and how it is realized. A. The ICT4D Value Chain The ICT4D value chain model [22, 23] facilitates an understanding of ICT4D evaluation. It is based on the standard input-process-output model linking resources and processes to systematically analyze the stages an ICT initiative traverses over time (see Fig. 1). According to the value chain, the input i.e. an ICT4D intervention in combination with fulfilled prerequisites (i.e. policies and implementation skills) will result into a successful deliverable e.g. a telecentre, e-library platform etc. These deliverables once exploited by the target beneficiaries result into outputs, which lead to outcomes and ultimately impact. The realization of outcomes from outputs as well as impact from outcomes is affected by various contextual factors such as skills, institutional barriers and 3 cultural or personal beliefs among others. Over the years interest in the domains along the value chain has shifted from readiness, availability and uptake towards development impact [1, 24]. This shift arises from the need for ICT4D initiatives to demonstrate that they actually contribute to social and economic development. However the challenge in such evaluations is that as one moves from outputs to impact, evaluation becomes more complex since focus shifts from the technology to the development goals. As a result outcomes and impact cannot be attributed to a specific initiative since there are other factors or even initiatives that could have affected the resultant outcome. To address this challenge, it is argued in this paper that rather than aim at proving causality, emphasis should be placed on the contribution an initiative has made on social and economic development [25]. This refers to the change in terms of social and economic development resulting from the presence of that intervention, within the boundaries of the contextual factors. Furthermore focusing on the contribution is appropriate in situations where baseline studies were not performed to facilitate a longitudinal evaluation of the initiatives. Fig. 1. ICT4D value chain adopted from [26] Moreover the impact concepts i.e. outputs, outcomes and impact as per value chain have been severally defined based on various approaches applied to the design and evaluation of projects or programmes in international development. Some of the main and interrelated approaches are the logical framework (LFA) [27], theory of change (including contribution analysis) [28], and results based management (RBM) [29] (see Table I).To some extent all these are theory- based approaches that rely on the theory of change techniques to facilitate the assessment of whether and how an initiative causes or contributes to an impact. They may also apply as underlying guides to the value (or result) chain above to guide program or intervention designs or evaluations [30].This study adopts the output, outcome and impact definitions suggested by the ICT4D value chain since it is a pivotal framework in this study. In addition, the ICT4D value chain definitions are similar to those suggested by the RBM approach that is a widely accepted development evaluation approach. Generally outputs are the immediate results of the program or initiative. These can either be goods or services such as workshops held, information produced or changes in skills etc. In this study ICT4D outputs are the behavioral changes associated with technology use. Heeks cites these as consisting of new information and decisions, new communication patterns and new actions and transactions that an ICT enables. Moreover outputs in telecommunications are similarly defined as information made available and retrievable by computer. Outcomes (purpose) on the other hand are the effects of outputs; in this study they are the direct benefits in terms of measurable (both quantitative and qualitative) benefits as well as costs associated with the outputs. Finally development impact refers to the ICT contribution to the broader development goals: impacts are less tangible. They are the long-term effects of the interventions [24, 31]. The output and outcome definitions adopted in this study are similar to the opportunities and achievements concepts that are discussed in the subsequent section. However the value chain assumes a linear relationship between ICT and development. This does not sufficiently represent the development process, since there are several aspects involved in explaining how and why development would result from an ICT4D initiative[32].For this reason and the need to adequately define what development is and how it is realized in a given context; there is a need to adopt and integrate a development perspective as discussed in the following section. TABLEI OUTPUTS, OUTCOMES AND IMPACT DEFINITIONS Approach Output Outcome Impact Results Based Management [29] Outputs are changes in skills or abilities and capacities of individuals or institutions, or the availability of new products and services that result from the completion of activities Outcomes represent changes in the institutional and behavioral capacities for development conditions that occur between the completion of outputs and the achievement of goals. Impact implies changes in people’s lives. Such changes are positive or negative long- term effects, directly or indirectly, intended or unintended. Logical Framework [33] These are the specific, direct deliverables of the project necessary to achieve the Outcome. The immediate impact on the project area or target group i.e. the change or benefit to be achieved by the project The higher- level identified situation that a project contributes towards achieving. Contribution analysis [25, 34] These are goods and services Outcomes cover the sequence of results (or Impacts are the final or long-term 4 produced by the program. For example checks delivered, advice given, people processed, information provided, reports produced effects) – immediate, intermediate and final outcomes – following the delivery of outputs. consequences. For example environment improved, stronger economy, safer streets, energy saved ICT4D value chain[1] Outputs are the micro level behavioural changes associated with technology use. Outcomes are the wider costs and benefits associated with ICT. Development Impacts are the contribution of the ICT to broader development goals. B. The Capability Approach A development theory perspective facilitates the definition of what constitutes development. For this purpose, Sen’s capability approach [16, 35] is adopted since it facilitates a multi-dimensional, people-centered approach of defining what constitutes development. Development according to Sen is the expansion of freedoms (capabilities or opportunities) to enable people lead the lives they value [16:18]. Development is more than the provision or access to a resource e.g. ICT: it is about what ICT can enable people be or do given their contextual aspects. One of the reasons freedom is central to development is for purposes of evaluation. Sen [16:4] points out that “assessment of progress has to be done primarily in terms of whether the freedoms that people have [or value] are enhanced”. Basically it looks at 1) development in terms of values e.g. being healthy, being educated or being happy; and 2) evaluate how these have been enhanced by, for example, access to the Internet in a given context. The capability approach premise is that a vector of a resource is transformed into a capability set within the restriction of conversion (contextual) factors. The capability set consists of functionings - things one can be or do to obtain the life they value. Simply defined, the capability set is the opportunities a development initiative offers. Achieved functionings on the other hand are the opportunities one chooses to exploit given his/her specific context. The multidimensional nature of the approach argues for a holistic evaluation of wellbeing that is not only limited to income, since wellbeing consists of aspects that income cannot satisfactorily measure [36]. Examples include: greater access to knowledge, better nutrition and health services, more secure livelihoods, security against crime and physical violence, political and cultural freedoms, or participation in community activities. Depending on the nature and purposes of evaluation, one can choose to focus on the opportunities1 (capability set), the achieved functionings, or both. The 1 For this study, opportunities capabilities and capability set are the same construct and may be used interchangeably to refer to a set of valuable functionings that an initiative offers or a person has effective access to. majority of applications of the capability approach focus on opportunities, arguing that while policymakers are mandated to deliver development opportunities, they to a great extent cannot decide how people choose to benefit from them [37]. The capability approach is also concerned about human diversity which results from people’s personal as well as external factors [35]. These factors, referred to as conversion (or contextual) factors determine people’s preferences and choices of the potential functionings. Conversion factors are classified as personal - the individual characteristics such as physical disabilities, motivation, level of education, age, gender and sex; Social factors - the external legalities or societal requirements that may consist of public policies, social or cultural norms and discriminating practices. Another emerging social factor here are the intermediaries e.g. non- government agencies that seek to promote ICT usage[32]. Lastly, environmental aspects focus on location and accessibility to facilities, as well as technical aspects such as quality of service [18, 38]. An individual’s capability set comprises both wellbeing – the opportunities availed for a better life; as well as agency – one’s ability to choose from the availed opportunities based on personal values and circumstances[16]. Agency takes into consideration the active involvement of beneficiaries in their development process; i.e. whether they choose to exploit the available facilities for the improvement of their lives or not depending on what they value, and the circumstances they are in. The following are the multiple evaluation spaces within which policies and initiatives can be evaluated:wellbeing freedom which focuses on the capabilities or opportunities an initiative fosters; wellbeing achievement which is the achieved functionings; agency freedom which evaluates the freedom to achieve whatever a person decides he or she should achieve[39:203]; and finally agency achievement which are the outcomes in terms of one’s values, including those of other people, beings and things[40:341]. IV. PROPOSED ICT4D EVALUATION MODEL As suggested by the capability approach, the realization of development from an initiative is a process that besides the provision of the opportunities (capabilities) also involves the interaction of these capabilities with choice that is influenced by the conversion factors. For example Hatakka and De' [41] analyze a project that successfully set up a distance education system providing opportunities for people to attain formal learning. However factors such as pedagogical training and low computer literacy were ignored, which affected people’s choices regarding the exploitation of the provided opportunities. This highlights two aspects: first, the need to perform a process analysis from capabilities to achieved functionings; and second, the need to explicitly establish the conversion factors that affect people’s choices. Focusing on achieved functioning alone denies one insight into the process that is very essential given that development initiatives are highly contextually dependent. On the other hand, focusing on capabilities alone offers a limited development evaluation that does not actually establish whether development has occurred. 5 Focusing on capabilities alone may also be perceived as techno-centric since evaluation is only performed on the opportunities an initiative can offer and does not investigate whether these were achieved or the factors that influenced this process. In relation to the value chain outputs, outcomes and impact concepts, the ICT contribution to development is evaluated in terms of the outputs contribution to outcomes. The underlying assumption is that exploiting (choosing to use) available opportunities (outputs) will to a great extent contribute to the development achievements (outcome). In essence evaluation can be performed for the contribution the output makes; and the outcome where applicable; which is a process evaluation of how an initiative has contributed to development. A similar assumption is held and proven for the empowerment [42] and the choice [4, 43] frameworks both of which rely on the capability approach as a theoretical foundation. Similarly Garnham [12:33] points out that “[t]he point from a capabilities perspective is the assessment of what contribution the medium makes to enhancing its users’ range of functionings…”. According to the empowerment and choice approaches it is assumed that empowerment results into the realization of development outcomes. Alsop and Heinsohn [42:5] define empowerment as “enhancing an individual’s or group’s capacity to make choices and transform those choices into desired actions and outcomes”. Individuals use their agency (human diversity, personal conversion factors) to explore opportunity structures (capabilities, social factors) resulting into empowerment (presence and exercise of choice) that enables development outcomes. Development outcomes according to Kleine [4:122] are ‘complex to describe’ but consist of choice as the primary outcome, and secondary outcomes which ‘will often be either sketches of overarching aims or limited to aspects relevant to a given context’. These secondary outcomes may consist of goals/aims that individuals or groups value within a given context or achieved functionings – a subset of the capabilities. Based on the above discussion, the constructs of the proposed evaluation model include ICT characteristics, conversion factors, opportunities (capabilities), and achievements (choice, personal or community goals, and achieved functionings) as shown in Figure 2. The ICT characteristics that a resource enables (communication; production, processing and distribution of information) provide opportunities within the limitations of the personal, social and environmental factors. Achievements are the opportunities one chooses to exploit within the restriction of conversion factors, and choice is also explicitly evaluated as one of the achievements [44:74]. Fig. 2. Proposed ICT4D evaluation model adapted from [35] . According to Robeyns and van der Veen [37:44] although governments (development partners, etc.) can provide opportunities, they cannot decide on how people live their lives. It is assumed that if someone’s ability to make choices is increased or strengthened, it will enable the choice of capabilities so that one lives the life they value. Outputs as per these definitions are the opportunities, while outcomes are the achievements. The achievement of certain functionings enables other opportunities: this is shown by the double pointing arrows between outputs and outcomes. For example, sensitization on the benefits of using the Internet empowers individuals to make wise decisions on how to use it. A. ICT4D evaluation criteria - Dimensions and Indicators According to Sen, the expansion of freedom is both the means and end of development. From Sen’s [16] viewpoint,“[t]he intrinsic importance of human freedom [ends], in general, as the preeminent objective of development is strongly supplemented by the instrumental effectiveness [means] of freedoms of particular kinds to promote freedoms of other kinds” (p. xii). An initiative should therefore be evaluated on its ability to increase people’s substantive freedoms such as self-esteem, as well as the instrumental freedoms that contribute to and guarantee people’s substantive freedoms. Consequently, Alkire [45] emphasizes that in defining an evaluative space based on the approach, it is important that considerations of both instrumental effectiveness and intrinsic importance are considered. Sen proposes five instrumental freedoms that enhance people’s capabilities i.e. social opportunities, economic facilities, political freedoms, transparency guarantees and protective security. It is argued that the extent to which these are secured 6 is indicative of the level of an individual, household or community development [3:36]. Since these freedoms are interrelated and supplement each other, earlier studies proposed three (social opportunities, economic facilities and political freedoms) out of the five domains for this study[13, 46]. A fourth dimension, psychological wellbeing is proposed as this evaluates the substantive freedoms such as choice and self esteem [32]. Depending on the nature of the initiative being assessed all or just some of the dimensions may be applied except psychological wellbeing that should be evaluated for all initiatives since it affects the achievements in other dimensions (e.g. as discussed in relation to choice). As Gomez and Pather [8:11] argue, “attention to intangible and unquantifiable aspects (e.g. self-worth and the strengthening of social fabric) that are facilitated indirectly through use, or even the presence of ICTs, will provide a more complete and holistic perspective of ICT impact.” Alkire [47:7] further confirms that these measures “[..] Might be used to provide insights into people’s values and perceptions with respect to other dimensions of interest [..]”. The dimensions are defined as follows: 1) Social opportunities: which refer to arrangements society makes available to enable an individual to live a better life. From the capability perspective, this specifically focuses on education and healthcare 2) Economic facilities: these refer to the opportunities that individuals enjoy utilizing resources for the purpose of consumption, production or exchange. This includes aspects such as productivity, employment, etc. 3) Political freedoms: are the opportunities people have to exercise their political rights e.g. being able to participate in local elections, community development programmes, etc. 4) Psychological wellbeing: refers to the physical, emotional and personal development opportunities. These are mostly a result of using ICT or participating in ICT4D projects. Examples include gaining respect from peers or having an increase in self-esteem. Psychological wellbeing has both substantive and instrumental value that enables people to exploit other opportunities in pursuit of development. It is envisaged that a set of criteria for each of these dimensions could facilitate an evaluation of the ICT contribution to development (see proposal in appendix A). For each dimension, achievements (outcomes), and opportunities (outputs) are proposed. For example it is presumed that to assess whether an initiative has improved access to formal or non-formal education (outcome/achievement) in the research and education dimension; the following opportunities (outputs – what people do) are evaluated:  Accessing information in relevant online resources e.g. research journals, online libraries  Participating in online research collaborations e.g. through discussion forums  Producing and publishing research outputs e.g. journals, patents etc. These are further granulated to define output and outcome indicators (see illustrative example in section IV). The indicators measure whether end users exploit the opportunity in terms of quality and the usage. Quality seeks to establish whether end users actually value the opportunity, which will determine whether it is exploited. On the other hand usage focuses on the level of use. Using the electricity analogy, Roberts points out that ICT is also ubiquitous. Therefore “it is not the electricity or ICTs as such that make the (bulk) impact on economy and society but how they are used to transform organizations, processes and behaviours.”[48:90]. Details of criteria formulation are discussed in another publication [49]. The indicators proposed in this study are mostly qualitative and do not require precise data specifications. It is envisaged that the qualitative assessment facilitates a structured, approach that provides sufficient information to report the ICT contribution to development. Elicitation of data for this approach relies on beneficiaries’ perceptions, which can be imprecise information about how initiatives have been of benefit to peoples’ wellbeing. Moreover the use of structured approaches to evaluate the ICT contribution to development is also recommended as a replacement for access and usage measures which offer little in as far as defining the actual ICT benefits is concerned [12]. V. APPLYING THE EVALUATION MODEL This section presents a structured approach that applies Multi-criteria Decision Analysis (MCDA) techniques to facilitate the evaluation of development initiatives. A. An MCDA approach to evaluate the ICT contribution to development The evaluation and selection of ICT4D initiatives is a complex decision problem that would benefit from the application of MCDA techniques [10]. Besides facilitating multidimensional and multi-stakeholder assessments; MCDA provides a means for handling uncertainty arising from incomplete and vague information. This is a key requirement for the evaluation of the ICT contribution to development which relies on stakeholder value-judgments, perceptions and beliefs of how ICT has affected people’s lives. In addition MCDA techniques offer a structured evaluation process of development outcomes as alternative to the predominately descriptive, and often difficult to report ICT4D evaluation approaches [18]. It further relaxes the requirement of quantitative measures which call for data that is in some cases not accessible, and may be more taxing on the stakeholders. As a structured decision making process the MCDA methodology typically consists of three stages [50]: 1)information gathering or problem structuring – involves the definition of the decision problem to be addressed as well as the criteria and alternatives where necessary, 2) modelling stakeholder preferences - the structured decision problem, i.e., criteria and alternatives are modelled using a decision support tool; and 3) evaluation and comparison of alternatives. While the application of MCDA techniques to decision making situations in the developing country context is 7 appropriate, it is challenged by cultural, organizational, and infrastructural barriers among other factors [51]. Examples specifically include low literacy levels, the lack of electricity, and uneven access to ICT infrastructure as well as elite resistance resulting from leaders being afraid of losing their political position [52]. This calls for appropriation of the MCDA approach and process in a way that takes into account the contextual limitations in the developing country context, and the specific ICT4D evaluation exercise. This section illustrates how an MCDA technique can be applied for the evaluation of an ICT4D initiative. It specifically applies the technique using a subsection of the proposed criteria for the evaluation of the impact of an online learning environment on students’ access to learning. B. The Case: Evaluating the impact of MUELE on students’ access to learning 1) Problem definition and structuring The Makerere University E-Learning Environment (MUELE) is one of several initiatives aimed at leveraging faculty effectiveness and improving access to learning at Makerere University [53]. MUELE is a learning management system (LMS) based on Moodle. It has been in existence since 2009 and boasts of a steady growth of users over the years. For instance the active courses increased from 253 in 2011 to 456 in 2013, while the users increased to 45,000 to date from 20,000 in 2011. Despite this state of progress, the use of MUELE is mostly as a course information repository even after lecturers obtained training in online course authoring and delivery [54]. This has been attributed to the attitude towards the adoption of e-learning, concerns from lecturers regarding the increase in workload resulting from large student numbers, and increased course preparation time. Consequently this illustrative study seeks to establish whether MUELE has improved students’ access to learning. More specifically this sought to establish whether MUELE contributed or did not contribute to improved (access to) learning; and an assessment of how the initiative performed on the different output and outcome indicators; highlighting the most significant outcomes. The criteria consist of the output and outcome indicators relevant for the evaluation of the impact of MUELE on access to learning. This is a subsection of the criteria suggested in appendix A, specifically aimed to evaluate improved access to formal and/or non-formal education.The criteria also include the contextual factors known to have an effect on the use of ICT to support learning. The criteria are summarized in Tables II – IV in Appendix B. 2) Problem modeling and elicitation The proposed criteria consisted of two decision models; the outputs and outcomes decision model. The output model sought (see Figure 3) to establish the perception of students on whether MUELE had improved on their access to learning. On the other hand the outcomes model sought to measure the actual improvement in student learning. The contextual factors had an influence on both models, either facilitating or restricting the improved access to learning. Preference modelling and elicitation considered two aspects: 1) evaluating the relative importance of criteria (eliciting weights); and 2) evaluating the initiative performance against criteria (eliciting scores).  Weight Elicitation:-This is expressed through the assignment of a weight which reflects the importance of one dimension (criteria) relative to the others and can be achieved through various methods [55]. Ideally weight elicitation should be performed for each of the levels of the decision tree hierarchy. This study applied the rank-order approach in which criteria were ranked in order of importance from most to least important including equal ranks, as well as assigning of weights. Rank-ordering was performed for the outcome model and the bottom-level criteria of the output model (output indicators). Equal weights were assumed for the other levels of the hierarchy i.e. outputs and output indicator categories. The weights were developed through consultation with experts in the field – lecturers and MUELE administrators – who assessed the relative importance of the criteria.  Eliciting Scores:- This involved evaluating perceptions of how MUELE had performed on various criteria. Responses were elicited from students who had used MUELE for at least a year or more. Verbal-numerical scales which have been applied in various domains [56- 58] as well as binary (yes/no) scales were used for the elicitation. The verbal-numerical scale is a combination of verbal expressions (e.g. unlikely, strongly agree etc) and the corresponding numerical intervals (see table 5). Since elicitation involved vague and imprecise value judgements of how e-learning had improved learning, it was appropriate to adopt a verbal-numerical scale. While the verbal facilitated stakeholders to vaguely state their preferences, the corresponding numerical ranges were applied for representation and analysis in the decision analysis tool. Studies have established that 8 Fig. 3. Output evaluation model Cr. 10 W. Tot. W:[50.0%, 100.0%] I was mostly just r... Cr. 9 W. Tot. W:[0.0%, 50.0% ] I frequently partic... Cr. 8 W. Tot. W:[0.0%, 50.0%] course discussion... Cr. 7 W. Tot. W:[33.333%, 100.0% ] posts on the discu... Cr. 6 W. Tot. W:[0.0%, 33.333%] encouraged me t... Cr. 5 W. Tot. W:[0.0%, 50.0%] I prefer using cour... Cr. 4 W. Tot. W:[50.0%, 100.0% ] I frequently use th... Cr. 3 W. Tot. W:[33.333%, 50.0%] Course material o... Cr. 2 W. Tot. W:[0.0%, 33.333% ] I think the course ... Cr. 1 W. Tot. W:[33.333%, 50.0%] course material w... Cr. 17 [0.0%, 4.762% ]W. Tot.: W:[0.0%, 14.286%] Traditional face t... Cr. 16 [0.0%, 5.556%]W. Tot.: W:[0.0%, 16.667% ] Mandatory to use ... Cr. 15 [0.0%, 5.556%]W. Tot.: W:[0.0%, 16.667%] Can afford a pers... Cr. 14 [0.0%, 5.556% ]W. Tot.: W:[0.0%, 16.667%] Unreliable or slow... Cr. 13 [0.0%, 5.556%]W. Tot.: W:[0.0%, 16.667%] Limited access ti... Cr. 12 [4.762%, 33.333% ]W. Tot.: W:[14.286%, 100.0%] Personal motivation Cr. 11 [0.0%, 5.556%]W. Tot.: W:[0.0%, 16.667% ] Relevant skills CH7 W: 50.0% Level of use of for... CH5 W: 50.0% Quality of forums CH3 W: 50.0% Use of course mat... CH1 W: 50.0% Quality of course ... CH8 W: 33.333% Contextual Factors CH6 W: 33.333% Participation in di... CH2 W: 33.333% Access to course ... CH4 Improved access t... 9 people assess in terms of words or numbers in varied ways; however the use of a combined verbal-numerical scale is a more effective and simplified elicitation approach [58]. For this study the verbal-numerical scale in Table v was adapted from Budescu et al.[59] mainly because it had been empirically developed and was appropriate to illustrate the use of a structured approach in evaluating development initiatives. TABLE V EXAMPLE OF A VERBAL-NUMERICAL SCALE [59] Verbal Statement Interval range Virtually certain [100-99]% Very likely [98-90]% Likely [89-66]% Neutral [65-33]% Unlikely [32-10]% Very unlikely [9-1]% Exceptionally unlikely [0.9-0]% Since multiple responses were elicited from the students and experts, aggregation was required for the elicited information. The aggregation approach was dependent on the nature of response scales; for example the simple weighted sum approach was applied for the aggregation of the students responses obtained from the verbal-numerical scale [50]. This involved assuming equal weights for each stakeholder and calculating the expected value. The simple weighted sum approach has been used in the aggregation of imprecise values because it has proven to be the most effective aggregation approach [50]. Since the ranking and binary (yes/no) scales are ordinal, the mode was applied as the preferred measure of central tendency was applied to obtain the aggregate value(s) for the analysis [60]. 3) Results: Analysis and Evaluation In this study the DecideIT decision support tool [61-63] was used to analyse and evaluate the decision problem. DecideIT is based on multi attribute value theory [64] and supports both precise and imprecise information. DecideIT supports various data formats i.e. imprecise data in terms of interval values, comparative statements or weights and even precise values. The rank ordered values depicting the relative importance of criteria were modelled as comparative statements, while the student perceptions obtained through the verbal-numerical and binary (yes/no) scales were modelled as intervals, and precise values respectively. Evaluation was performed for each of the models (outputs and outcomes) and the results are discussed below. i. Respondent Demographics Eight (8) experts, 4 male and 4 female were consulted on the ranking of importance of indicators used to evaluate the impact of an e-learning environment on improved students learning. Seven were lecturers, while one of them was an administrator in charge of MUELE. Twenty (20) students 17 male and 3 female participated in the evaluation of the impact of MUELE on improved access to learning. With an exception of 2 students in their second year and 3 who had completed their studies, majority (15) were in their final year of study, and had used MUELE for an overall period of 2 to 4 years. Most of the participants (10) used it 2-3 days a week, 7 used it almost everyday, while 2 rarely used it and 1 used it once a week. Finally while 14 of the student participants were in the 16 to 25 age bracket, the rest were in the 26 to 35 age bracket. Clearly the sample is not representative enough of the student population that uses MUELE. However this was sufficient to illustrate the structured evaluation process. ii. Output Model Evaluation Value Profiling: provides an assessment of how outputs (evaluated in terms of output indicators) have performed in as far as meeting the overall objective is concerned. It assesses the contribution or relevance of the outputs to the overall objective. In this case the quality and level of use of course material are the most significant contributors to improvements in accessing learning materials, while participation in online discussions is average. Finally, satisfaction with the quality of discussion forum posts is the least contributor to improved access to learning (Fig. 4). Evidentily MUELE is mostly used as a course repository as previuosly established [54]. Fig. 4.Performance of individual outputs on improved access to learning. Tornado Graphs: facilitate the identification of the critical issues that have the highest impact on the expected value (Fig. 5). The least contributing (outputs) indicators as per value profiling analysis above i.e. participation in the discussion forums (Cr.7, Cr.9, Cr.10, Cr.8), were the most critical aspects affecting the expected value measure. On the other hand, the high contributors, i.e. the quality and level of use of course material, had the least impact on the expected value. Such information may challenge decision makers to develop strategies for the improvement of the current initiative or streamline the development of future similar initiatives. For example establishing that participation in discussion forums is a critical aspect in realising improved learning through MUELE would challenge the lecturers to actively engage the students in this activity, or to investigate further on why this is an important aspect.   10 Fig. 5. Critical outputs in the realization of student learning.   Expected Value Graph: The expected value interval [0.68 0.84] of the outputs measuresstudent performance in terms of the extent by which access to MUELE has improved access to learning (Fig. 6). This implies that based on the outputs, it is perceived that MUELE had a fairly high potential of improving access to learning with very limited possibility or chances of failure. This serves as confirmation of the ICT potential towards improving access to learning in this particular context. Fig. 6. Expected value graph evaluating the outputs’ contribution to improved access to learning. iii. Outcome Model Evaluation Expected Value Graph: - depicts an expected value interval [0.52 0.92] and the focal point of all interval statements (the 100% contraction value) at 0.73 (Fig. 7). This implies that the different outcomes derived from MUELE effectively contributed to improved access to student learning by a range of [0.52 0.92]. Fig. 7. Expected value graph evaluating the outcomes’ contribution to improved access to learning. Value Profiling: The outcomes to which MUELE most significantly contributed were improvements in student learning, facilitation of student participation in personal learning, and a better chance of obtaining employment (Fig. 8). There was an average effect on the psychological aspects i.e. improved levels of confidence and whether people felt more valued or respected. There was however a low chance that MUELE had a significant negative impact such as affecting concentration or self-discipline, as well as personal health. On the other hand there was a high chance that access to MUELE increased student dependence on computers. Fig. 8. Performance of individual outcomes in terms of outcome indicators. Tornado Graphs:The contextual factors, i.e. relevant skills, limited access to computers, unreliable or slow internet connection, ability to afford a personal computer, as well as the mandatory requirement to use MUELE were the most critical aspects affecting the realisation of improved access to student learning (Fig. 9). The difference in factors affecting the realization of outputs and outcomes is essential for mid- term evaluation; helping implementers address the identified gaps and ensure success of the initiative. 11 Fig. 9.Critical outputs in the realization of student learning. As is seen from the results above, the aim in such an analysis is not necessarily to obtain an aggregated value explaining the overall performance of an initiative. Focus is on facilitating a structured approach to explaining various aspects, such as how an initiative performs on different outputs and outcomes, the most critical factors affecting the realization of the overall objective etc. It is important to note that while the findings in this illustrative example may not be representative of the e- learning status at Makerere University, they are a good illustration of the evaluation process. For example the realization that contextual factors are an essential aspect in meeting the initiative goal will shift focus from just providing the e-learning environment to addressing the most critical contextual factors. Furthermore the low performance of discussion forums will probably encourage further investigation into the pedagogical requirements that would integrate forums into the students’ learning process. The MCDA tool provides a rich, detailed and structured assessment of the different factors that warranty further investigations in its use as an ICT4D evaluation approach. VI. CONCLUSIONS AND FUTURE WORKS This paper proposes and illustrates an MCDA structured approach for the evaluation of the ICT contribution to developmentThe model is based on the capability approach with aspects drawn from the ICT4D value chain as conceptual framework. A major challenge with the capability approach has always been its strong philosophically profound basis, which complicates attempts of its operationalization. The work presented here contributes to the operationalization of the capability approach or more generally applying a development perspective to the evaluation of the ICT contribution to development. However, unlike the existing applications of the approach, the model suggested in this study illustrates the use of indicators in the evaluation of the ICT contribution to development. Moreover the proposed indicator-based evaluation offers more in comparison to the quantitative evaluations of availability and uptake. It is also multi-dimensional, evaluating more than just economic benefits. It explicitly considers the instrumental and substantive ICT benefits, as well as the context in which they should be obtained. It further stresses the need to evaluate psychological wellbeing alongside the other dimensions, because this is both a means and an end in ensuring development. The approach will benefit ICT4D evaluation efforts for which in depth descriptive evaluations are not possible due to various constraints that are budgetary, logistical, or related to insufficiency of data. It may also serve for the comparative evaluation of multiple projects. For instance a subsection of the criteria proposed in this study will also be applied in iMentors2, an EU project developing a platform that will enable donors and development partners to review complete or existing projects to provide policy support and assist programme planning and implementation. The applicability of the model is illustrated through its use in the evaluation of an online learning environment, MUELE aimed to leverage faculty effectiveness and improve access to learning at Makerere University. The illustrative example reveals that such a structured approach can facilitate a sufficient assessment of the performance of development initiative, as well as the most critical factors influencing the attainment of the development goals. The model however does not explicitly address unintended or negative benefits that are prevalent in any development initiative. Future work will seek to address this gap, as well as validate and test the MCDA evaluation model in other ICT4D contexts. REFERENCES [1] R. Heeks, "Do Information and Communication Technologies (ICTs) contribute to development?," Journal of International Development, vol. 22, pp. 625-640, 2010. [2] M. Hatakka, A. Andersson, and Å. Grönlund, "Students' use of one to one laptops: a capability approach analysis," Information Technology & People, vol. 26, pp. 94-112, 2013/04/21/08:37:15 2013. [3] S. Ibrahim-Dasuki, P. Abbott, and A. Kashefi, "The Impact of ICT Investments on Development Using the Capability Approach: The case of the Nigerian Pre-paid Electricity Billing System," The African Journal of Information Systems, vol. 4, 2012. [4] D. Kleine, "The capability approach and the `medium of choice': steps towards conceptualising information and communication technologies for development," Ethics and Information Technology, vol. 13, pp. 119-130, 2013/04/26/13:41:25 2011. [5] H. Grunfeld, S. Hak, and T. Pin, "Understanding benefits realisation of iREACH from a capability approach perspective," Ethics and Information Technology, vol. 13, pp. 151-172, 2013/05/15/09:20:47 2011. 2http://www.gov2u.org/projects/imentors/ 12 [6] J. Johnstone, "Technology as empowerment: a capability approach to computer ethics," Ethics and Information Technology vol. 9, pp. 73-87, 2007. [7] S. Madon, "Evaluating the Developmental Impact of E- Governance Initiatives: An Exploratory Framework," The Electronic Journal on Information Systems in Developing Countries, vol. 20, pp. 1-13, 2004. [8] R. Gomez and S. Pather, "ICT Evaluation: Are We Asking the Right Questions?," The Electronic Journal of Information Systems in Developing Countries, vol. 50, pp. 1-14, 2013/05/04/14:19:04 2012. [9] M. L. Best, D. Thakur, and B. E. Kolko, "The contribution of user- based subsidies to the impact and sustainability of telecenters - the eCenter project in Kyrgyzstan," 2009, pp. 192-200. [10] A. Blake and M. Q. Garzon, "Boundary Objects to Guide Sustainable Technology-Supported Participatory Development for Poverty Alleviation in the Context of Digital Divides," The Electronic Journal of Information Systems in Developing Countries, (EJISDC), vol. 51, pp. 1-25, 2012. [11] H. E. Chew, P. V. Ilavarasan, and M. R. Levy, "The economic impact of Information and Communication Technologies (ICTs) on microenterprises in the context of Development," The Electronic Journal on Information Systems in Developing Countries, vol. 44, pp. 1-19, 2010. [12] N. Garnham, "Amartya Sen's "Capabilities" Approach to the Evaluation of Welfare: Its Application to Communication," The public, vol. 4, pp. 25-34, 1997. [13] F. N. Kivunike, L. Ekenberg, M. Danielson, and F. F. Tusubira, "Using a Multi-Criteria Tool for Selecting ICT Development Initiatives " presented at the IST-Africa 2008 Conference Proceedings Windhoek, Namibia, 2008. [14] B.-S. Gigler. (2011, 29th April 2013). Informational Capabilities— The Missing Link for the Impact of ICT on Development? . Available: http://siteresources.worldbank.org/INFORMATIONANDCOMMU NICATIONANDTECHNOLOGIES/Resources/D4S2P3- BjornGigler.pdf [15] G. Walsham, "Development Informatics in a Changing World: Reflections from ICTD2010/2012," Information Technologies & International Development, vol. 9, pp. pp.-49-54, 2013/05/12/09:07:13 2013. [16] A. Sen, Development as Freedom. . Oxford: Oxford University Press, 1999. [17] R. Chambers and G. Conway. (1992). Sustainable Rural Livelihoods: Practical Concepts for the 21st Century. Available: http://www.ids.ac.uk/publication/sustainable-rural-livelihoods- practical-concepts-for-the-21st-century [18] M. Hatakka and J. Lagsten, "The capability approach as a tool for development evaluation - analyzing students' use of internet resources," Information Technology for Development, vol. 18, pp. 23-41, 2013/04/20/21:41:02 2012. [19] R. De', "Evaluation of E-Government Systems: Project Assessment vs Development Assessment," EGOV 2006, pp. 317-328, 2006. [20] S. Parkinson and R. Ramirez, "Using a Sustainable Livelihoods Approach to Assessing the Impact of ICTs in Development," The Journal of Community Informatics, vol. 2, 2013/05/08/07:41:09 2007. [21] R. Heeks, "Using Competitive Advantage Theory to Analyze IT Sectors in Developing Countries: A Software Industry Case Analysis," Information Technologies and International Development, vol. 3, pp. 5-34, 2007. [22] A. Adamali and B. Lanvin, "E-strategies Monitoring and Evaluation Toolkit," The World Bank, Washington, DC2005. [23] R. Heeks, "Title," unpublished|. [24] R. Gomez and S. Pather, "ICT Evaluation: Are we asking the right questions?," The Electronic Journal on Information Systems in Developing Countries, EJISDC, vol. 50, pp. 1-14, 2012. [25] J. Mayne, "Contribution analysis: Coming of age?," Evaluation, vol. 18, pp. 270-280, 2013/04/15/09:58:34 2012. [26] R. Heeks and A. Molla. (2009, Impact Assessment of ICT- for- Development Projects: A Compendium of Approaches. The Development Informatics Series. [27] DFID, "Title," unpublished|. [28] I. Vogel, "Review of the use of ‘Theory of Change’ in international development " UK Department for International Development (DFID) 2012. [29] UNDG, "Title," unpublished|. [30] J. Mayne, "Best Practices in Results-Based Management: A Review of Experience," United Nations2007. [31] T. Leimbach, S. Kimpeler, M. Pero, I. Bertschek, D. Cerquera, and B. Engelstaetter, "Development of impact measures for e- Infrastructures," European Commission: Information Society and Media Directorate General, Unit F32012. [32] B.-S. Gigler, "Title," unpublished|. [33] DFID. (2011). DFID How to note: guidance on using the revised Logical Framework. Available: https://http://www.gov.uk/government/publications/dfid-how-to- note-guidance-on-using-the-revised-logical-framework [34] J. Mayne, "Title," unpublished|. [35] I. Robeyns, "The Capability Approach: A Theoretical Survey," Journal of Human Development, vol. 6, pp. 93 - 117, 2005. [36] S. Deneulin, "Amartya Sen's Capability Approach to Development and Gaudium Et Spes: On Political Participation and Structural Solidarity," Journal of Catholic Social Thought, vol. 3, pp. 355- 372, 2006. [37] I. Robeyns and R. J. van der Veen, "Sustainable quality of life: Conceptual analysis for policy-relevant empirical specification," Netherlands Environmental Assessment Agency (MNP), Bilthoven and Amsterdam2007. [38] P. Alexander and L. Phahlamohlaka, "Amartya Sen’s Capability Approach applied to Information Systems research," South African Computer Journal, pp. 1-11, 2006. [39] A. Sen, "Well-Being, Agency and Freedom: The Dewey Lectures 1984," The Journal of Philosophy, vol. 82, pp. 169-221, 1985. [40] D. Gasper, "What is the Capability Approach: Its Core, Rationale, Partners and Dangers.," Journal of Socio-Economics, vol. 36, pp. 335-359, 2007. [41] M. Hatakka and R. C. De', "Development, capabilities and technology: an evaluative framework," in Proceedings of the 11th International Conference on Social Implications of Computers in Developing Countries. Partners for Development- ICT. Actors and Actions, Kathmandu, Nepal, , 2011. [42] R. Alsop, J. Holland, and M. Bertelsen. (2006). Empowerment in Practice : From Analysis to Implementation (First printing ed.). Available: https://openknowledge.worldbank.org/bitstream/10986/6980/1/350 320Empowerm1ctice01OFFICIAL0USE1.pdf [43] D. Kleine, "ICT4What? - using the Choice Framework to operationalise the capability approach to development," in 2009 International Conference on Information and Communication Technologies and Development (ICTD), 2009, pp. 108-117. [44] M. Fleurbaey, "Development, capabilities, and freedom," Studies in Comparative International Development, vol. 37, pp. 71-77, 2013/04/23/07:45:38 2002. [45] S. Alkire, "Choosing Dimensions: The Capability Approach and Multidimensional Poverty," in The Many Dimensions of Poverty, Kakwani, Nanak, and J. Silber, Eds., ed: Palgrave-MacMillan, 2008, p. 28. [46] F. N. Kivunike, L. Ekenberg, M. Danielson, and F. F. Tusubira, "Perceptions of the role of ICT on quality of life in rural communities in Uganda," Information Technology for Development, vol. 17, pp. 61-80, 2011. [47] S. Alkire, "Title," unpublished|. [48] S. Roberts, "The Global Information Society: a Statistical View," United Nations, Santiago, Chile2008. [49] F. N. Kivunike, L. Ekenberg, M. Danielson, and F. F. Tusubira, "Developing criteria for the evaluation of the ICT contribution to social and economic development," in Sixth Annual SIG GlobDev Pre-ICIS Workshop, Milan, Italy, 2013, p. 24. [50] R. T. Clemen and T. Reilly, Making Hard Decisions with DecisionTools, second ed. Pacific Grove, CA: Duxbury Press, 2001. [51] C. Karvetsk, J. Lambert, and I. Linkov, "Emergent Conditions and Multiple Criteria Analysis in Infrastructure Prioritization for Developing Countries," Journal of Multi-Criteria Decision Analysis, vol. 16, pp. 125-137, 2009. [52] M. Riabacke, S. Bohman, M. Danielson, L. Ekenberg, W. Faye, and A. Larsson, "Public Decision Making Support: A Developing 13 Country Perspective," in IST-Africa 2010 Durban, South Africa, 2010. [53] Report, "E-Learning report – 2011, Makerere University," Makerere University2011. [54] E. K. Kahiigi, "A Collaborative E-learning Approach: Exploring a Peer Assignment Review Process at the University Level in Uganda," PhD, Department of Computer and Systems Sciences, , Stockholm University, Stockhom, 2013. [55] M. Riabacke, "A Prescriptive Approach to Eliciting Decision Information," Degree of Doctor of Philosophy Doctoral thesis, Faculty of Social Sciences, Department of Computer and Systems Sciences, Stockholm University, Kista, 2012. [56] M. D. Piercey, "Motivated reasoning and verbal vs. numerical probability assessment: Evidence from an accounting context," Organizational Behavior and Human Decision Processes, vol. 108, pp. 330-341, 2013/06/08/13:19:57 2009. [57] B. Rohrmann, "Verbal qualifiers for rating scales: Sociolinguistic considerations and psychometric data," University of Melbourne, Melbourne2007. [58] C. Witteman and S. Renooij, "Evaluation of a verbal - numerical probability scale," International Journal of Approximate Reasoning, vol. 33, pp. 117-131, 2013/06/10/08:07:26 2003. [59] D. V. Budescu, S. Broomell, and H.-H. Por, "Improving Communication of Uncertainty in the Reports of the Intergovernmental Panel on Climate Change," Psychological Science, vol. 20, pp. 299-308, 2013/06/08/13:20:16 2009. [60] S. Manikandan, "Measures of central tendency: Median and mode," Journal of Pharmacology & Pharmacotherapeutics, vol. 2, pp. 214-215, 2013/08/10/12:33:44 2011. [61] M. Danielson, L. Ekenberg, J. Idefeldt, and A. Larsson, "Using a Software Tool for Public Decision Analysis: The Case of Nacka Municipality," Decision Analysis, vol. 4, pp. 76-90, 2013/05/19/13:30:25 2007. [62] M. Danielson, L. Ekenberg, J. Johansson, and A. Larsson, "The DecideIT decision tool," in 3rd International Symposium on Imprecise Probabilities and Their Applications, Lugano, Switzerland, 2003, pp. 204-217. [63] K. Hansson, M. Danielson, and L. Ekenberg, "A framework for evaluation of flood management strategies," Journal of Environmental Management, vol. 86, pp. 465-480, 2008. [64] J. S. Dyer, "MAUT - Multiattribute Utility Theory " in Multiple Criteria Decision Analysis - State of the Art Surveys, J. Figueira, S. Greco, and M. Ehrgott, Eds., ed: Springer, 2005, pp. 3-24. APPENDIX A: ICT4D PROJECTS EVALUATION CRITERIA Dimension Achievements (outcome) Opportunities (outputs) (a) Research & Education Improvement in research quality and innovations Accessing information in relevant online resources e.g. research journals, online libraries Participating in online research collaborations e.g. through discussion forums Producing and publishing research outputs e.g. journals, patents etc. Improved access to formal and/or non-formal education Accessing information in relevant online resources e.g. online courses/tutorials, e-learning platform, research journals, online libraries Participating in ICT-enabled learning forums e.g. discussion forums Producing and publishing learning outputs e.g. journals, patents etc. (b) Healthcare Improved access to health services Accessing health-related information e.g. websites or short text messaging services that share information on good health practice, immunization, or pandemics etc. Remotely consulting medical personnel e.g. through phone calls, video calls etc. Improved delivery of health services Accessing health management information systems e.g. Drug tracking and dispensing systems, patient records management systems Participation in collaborations and co operations among health workers (c) Economic opportunities Improved productivity Accessing information from relevant resources e.g. websites or short text messaging services farming/agricultural resources, SMEs, small scale industries Participation in relevant online communities e.g. farming blogs, content production Improved income (&income generation opportunities) Access to relevant information e.g. new employment opportunities, stocks, investment opportunities, market information etc. Participating in relevant (ICT-related) training & skills development activities e.g. content development, ICT literacy, advanced techniques Performing ICT-related transactions e.g. e-commerce, e-tax, money transfers, remittances (d) Political freedoms Improved participation in local/community or national politics Accessing relevant online resources e.g. e-voting, institutional, community/national websites Participating in local/community or national political activities e.g. elections, debates, radio talk shows etc. Improved national/institutional/community transparency Accessing relevant online resources e.g. budgets on community/national websites, citizen online databases etc. Participating in national/community policing e.g. freely reporting fraud through hotlines, forums on websites Improved institutional/ organizational efficiency Accessing relevant platforms e.g. education management systems, human resource management systems 14 Participate in inter-organizational/institutional networking e.g. exchange of research students Performing relevant transactions e.g. salaries remittances, timetabling, production of reports etc. (e) Personal and psychological wellbeing Individual empowerment Strengthened ability to influence personal choices Perceived improvements in self-esteem and self-confidence Feeling more valued and respected Being able to analyze and solve own problems Improvements in family relationships and social ties Level of use of relevant media and/or applications e.g. online social media- facebook, twitter, mobile phones etc. Quality of relevant media and/or applications to interact with family and friends Having a sense of belonging related to participation in an online group Entertainment and fun Level of access/use to online fun activities e.g. music, movies, or games Level of access to online news updates i.e. local, sports, international news APPENDIX B: PROPOSED EVALUATION CRITERIA FOR THE E-LEARNING CONTRIBUTION TO IMPROVED LEARNING TABLE. III. OUTPUTS MODEL - OUTPUT CATEGORIES AND INDICATORS Indicator Categories Operational Definitions Output Indicators Quality of relevant online resources Perception of qualityof online resource(s) in terms of relevance, and usefulness ; as well as sufficiency in meeting stakeholder needs The course material made available through the online learning environment was very useful in my studies I think the course material offered through the online learning environment was somewhat sufficient to satisfy my learning goals The course material obtained through the online learning environment was always relevant to my learning goals. Level of use of relevant online resources e.g. online courses, e- learning platform A qualitative measure of frequency of use of online resources I frequently refer to/use/apply the course material posted on the online learning environment for my learning needs I prefer accessing/using course material posted on the online learning environment rather than the traditional classroom approach Quality of IT-enabled forum in terms of degree of activity e.g. discussion forums Perception of quality of IT-enabled forum in terms of relevance, and cooperation; as well as perception of ease of use The nature of content posted in the forums hosted on the online learning environment encouraged me to participate more actively in the discussion The posts on the discussion forums somewhat satisfy my learning goals The posted course discussions are extremely relevant for my studies Level of participation in ICT-enabled learning forums A qualitative measure of frequency of participation in discussions on ICT-enabled learning forums I frequently participate in /contribute to the discussion forums relevant to my studies I was mostly just reading messages posted in discussion forums and haven't contributed a lot to the discussions TABLE III OUTCOMES MODEL Outcomes Operational Definition Indicator statements Level of students performance Perception of the level of improvement in student performance I think the use of MUELE helps me to improve my performance in this course unit Efficient and timely feedback Are students getting feedback on their submissions in time? I always obtained an efficient and timely feedback through MUELE Level of student(s) participation in their own learning Degree by which students are taking persona initiative in their learning Using MUELE enables me to participate in my own learning in a better way Chances for (better) employment Degree by which one’s chances of obtaining better employment have increased I think I have better chances of obtaining employment because of the skills I have obtained through the use of MUELE Attainment of new/advanced Has the initiative enabled the participants obtain I was able to obtain advanced skills 15 skills or academic credentials new skills? Ability to participate in a course from anywhere at anytime Does the student have the ability to participate in an online course irrespective of where they are I am able to undertake (participate in) my course from anywhere at anytime Ability to make personal choices Has the participation in e-learning improved one’s ability to make personal choices? Using MUELE (has) strengthened my ability to make personal choices levels of confidence/self esteem Degree by which participation in the initiative has improved confidence/self esteem Participation in an online learning environment has improved my confidence levels Earned respect from peers Degree of increase in value and respect by peers I now feel more valued and respected by my peers because of the skills I have obtained through the use of MUELE Changes in responsibility and demands on the student Degree by which student responsibilities and demands have affected I feel participating in an online course has increased my responsibility and demands on me as a student concentration and self- discipline issues Level of increase of concentration and self- discipline I feel my participation in the online course has negatively affected concentration and self-discipline problems in class Computer dependence Degree of increase of computer dependence The use of the e-learning environment has strongly increased my dependence on computers health concerns The perceived impact of MUELE on personal health I think the use of the e-learning environment will negatively affect my health in the long run TABLE IV: CONTEXTUAL FACTORS AFFECTING THE REALIZATION OF IMPROVED ACCESS TO LEARNING THROUGH MUELE Factor Operational definition Indicators Personal Factors  Relevant skills Whether the possession or not of relevant skills affected one’s access to MUELE I lacked the relevant skills to use the e-learning environment Personal interest Level of personal motivation to exploit the e- learning application I was personally interested in using the e-learning environment to facilitate my learning Afford a personal computer Whether an individual can or cannot afford a personal computer I could afford a personal computer which has strongly contributed to my use of the e-learning environment Social Factors Mandatory to use MUELE Is it compulsory to use MUELE to support learning It was mandatory to use e-learning for the course unit attended knowledge obtained through traditional face to face lectures is sufficient Does the system offer any added value in comparison to the traditional face to face lectures The knowledge I obtained through our traditional face to face lectures is sufficient to meet my learning goals Environmental Factors Access time on the PCs The extent by which available access time on shared PC is sufficient Having limited access time on the computers in the lab limited my use of the e-learning environment Internet connection Whether the quality of the internet connection affects the use of ICT The unreliable/slow internet connection frustrated my use of the e-learning environment