The Journal of Community Informatics 16 (2020), 141-153 ISSN: 1721-4441 141 Notes from the field Lessons from the field: What researchers learned from evaluating ICT platforms for rural development and education The field of information and communication technology for development is a field constantly changing as new ICT tools emerge and new knowledge is gained by field researchers while performing their duties. The research problem: The ICT field is littered with examples of failed projects because field researchers did not know the best way to carry out their work. This paper is about knowledge imparted by six monitoring and evaluation field researchers after working for almost eight years, from 2010 to 2018, in ICT platform (DD) projects. These platforms were deployed across South Africa’s remote rural areas. The work followed interpretivism as its philosophy and was underpinned by qualitative research methods. Written project reports, face-to-face interviews and questionnaires were used to collect data and also to triangulate the findings. Participatory evaluation formed the basis for a complete understanding of the findings. (i) Planning; (ii) Deployment; and iii) Usage were found to be critical elements for a successful implementation of the DD. Although well planned, numerous lessons were still learned for the benefit of future projects. Foko, T., Mahwai, N.J., and Phiri, C.A. (2020). Lessons from the field: What researchers learned from evaluating ICT platforms for rural development and education. The Journal of Community Informatics, 16, 141-153. Copyright (C), 2020 (the author as stated). Licensed under the Creative Commons Attribution- NonCommercial-ShareAlike 2.5. Available at: https://openjournals.uwaterloo.ca/index.php/JoCI/index Thato Foko Nare Joyce Mahwai Charles Acheson Phiri Principal Researcher, Council for Scientific and Industrial Research, South Africa tfoko@csir.co.za Researcher, Council for Scientific and Industrial Research, South Africa nmahwai@csir.co.za Researcher, Council for Scientific and Industrial Research, South Africa cphiri@csir.co.za https://openjournals.uwaterloo.ca/index.php/JoCI/index mailto:tfoko@csir.co.za mailto:nmahwai@csir.co.za mailto:cphiri@csir.co.za The Journal of Community Informatics 16 (2020), 141-153 ISSN: 1721-4441 142 Introduction Numerous research papers have been written about recipients of development projects but few have dealt with field workers and the knowledge they gain when executing any such projects. This paper attempts to address this knowledge gap through the personal feedback of six respondents who, from 2010 to 2018, crisscrossed the country to carry out monitoring and evaluation (M&E) of the Digital Doorway project (DD). The DDs were deployed in many multi-purpose centres and schools across the country. The Council for Scientific and Industrial Research of South Africa (CSIR) researched, deployed, and maintained these DDs on behalf of the funding government departments of Science and Technology (DST), now known as the Department of Science and Innovation (DSI), from 2002 until 2013, and from 2013 up to 2020 the Department of Agriculture and Rural Development, formally known as the Department of Rural Development and Land Reform (DRDLR). The DD is a South African local innovation that supported computer literacy as part of the government’s service delivery policy of providing rural communities with access to ICTs. It was meant to make a fundamental difference in computer literacy and associated skills by providing access to computers, the Internet, and data to South Africans living in rural areas (van der Vyver & Marais, 2013). This ICT platform was based on principles similar to those demonstrated by an Indian project called the ‘hole-in-the-wall’ whose objective was to show that minimally invasive education (MIE) was a viable form of education (Stillman et al., 2012, Mitra et al., 2005). It was initiated to test whether children possess the cognitive ability to acquire functional computer skills without any formal training, with the aim of stimulating their natural curiosity (Smith, 2012). These platforms were deployed in rural and peri-urban areas of the country's nine provinces. During the DDs’ evaluations the respondents, who were field researchers, visited scores of schools and multi-purpose community centres (MPCC) in all provinces and interviewed DD users, community members, and leaders while collecting huge amounts of data. These included participatory evaluation, Outcome Mapping, and narrative enquiry. For these visits, researchers were always paired to ensure that data was corroborated and was of good quality. Many quarterly and annual reports were written and submitted to the departments. Background and Context Before providing the context to this study it is critical to understand our theory of change which was the basis for the work which had been carried out through the many phases of the DD project. For this article we adopted the definition from Comic Relief (2011) where Cathy James defined theory of change as “An ongoing process of reflection to explore change and how it happens – and what that means for the part organisations play in a particular context, sector and/or group of people”. She went further to state other important considerations such as understanding how change comes about and acknowledgement of “the complexity of change: the wider systems and actors that influence it”. Theory of knowledge allows researcher/students to reflect on their learning process, how they interact and use knowledge in the complex world (Mthembu, 2013). In the early 21st century for South Africa to develop it is crucial to harness its human capital potential and to provide ICT access to the poor and marginalised communities who are missing out on the benefits of ICT technologies (Konyana & Konyana, 2013). The legacy of apartheid has left a country with a huge socio-political and economic chasm The Journal of Community Informatics 16 (2020), 141-153 ISSN: 1721-4441 143 where the country’s majority people are unemployed and are still living in abject poverty excluded or marginalised from any meaningful economic activity (Stats SA, 2019). Therefore, most learners in rural communities have little or no access to ICTs resulting in poor learner performance (Leibbrandt, et al. 2010). To address learner performance, the DD project was established by the DSI and later DRDLR, both in collaboration with the CSIR. These DDs were rolled out in public schools, community halls, and multi-purpose centres by the two departments. Elaborate consultations with the relevant stakeholders such as community members and their leaders, school principals, teachers and learners, DD champions, and government employees were held. Many assumptions were made including that through learner’s curiosity, if ICT technologies such as computers with adequate content are deployed, learner and adult groups will use them to discover, with minimal assistance, information from the preloaded content or from online sites where Wi-Fi was available (Smith, 2012). This assumed that people, especially learners, would become self-taught in the use of computers and discover content and information on their own. The purpose of study The respondents, young researchers and post-graduate students, learned a lot from the year 2010 until 2018 when they began crisscrossing the country to carry out the M&E of the DDs so that they could share this knowledge with the world. These students work mostly in unfamiliar environments. Therefore, the purpose was to ascertain what the researchers learned through their interactions with the multiply community stakeholders and school children when evaluating the DDs that were deployed in the rural and semi- rural areas of South Africa. Research problem statement In an ideal world, field researchers would have all the necessary skills to collect perfect data, and equally perfect knowledge of the communities they need to work with when implementing research projects. However, in the real-world researchers, even skilled ones, do not always have an ideal understanding of the communities they work with, and field workers are not the most experienced people in any organisation. In most instances, they need training on how to collect data, engage with communities, develop presentation skills, etc. before they can embark on the journey of data collection. Their learning continues through their interactions with these projects' beneficiaries. It was this new knowledge which the researchers were requested to share with the world. Objectives There are two main objectives of this study: • To interrogate respondents on the lessons they learned during their interactions with the different community members and stakeholders when carrying out the monitoring and evaluation of the DDs since the year 2010 until 2018. • To develop a framework that future researchers, particularly novice researchers, could use as a blueprint for carrying out research in environments they are unfamiliar with. Research question What are the factors that affected the success of the DD project? The answer to this question is provided by the lessons learned by the team. The Journal of Community Informatics 16 (2020), 141-153 ISSN: 1721-4441 144 Scope of the research This paper covers only the information generated by interrogating the field researchers and the reports they assisted in compiling as part of the deliverables at different times during the projects. In scope are the data generated from the respondents’ expressed opinion and experiences, verbally or through written reports. The views of other senior researchers who hardly engaged with communities were not considered for this research because they formed part of the learning by respondent views through commentary and suggestions over the years. The DD in Context The initial project began in 2002 as a joint initiative between the Department of Science and Technology, before the involvement of other departments, with 40 standalone (Fig.1) robust computer terminals, which envisaged a network of over 1000 DDs reaching even the remotest areas of South Africa (Smith et al., 2006). Available on the DD was the Classroom (educational resources) which provided the user with a variety of educational resources. Most of these resources were PDF documents and science simulations. With the new container DDs, the content was accessible by directly engaging with the ICT terminal or via Wi-Fi enabled devices such as tablets, laptops, or smartphones (Walton & Johanson, 2012). Users were provided with access to networking menus with basic text chat which took place between computers. Users could create and post their own content and could view other users' posts. The “New Content” link led to a basic content management system where an administrator could access the DD. Fig. 1: Three Terminal standalone DD Fig. 2: Solar-powered container DDs The later version of the DD was a solar-powered container DD (Fig. 2) consisting of three DDs, one or two printers, Wi-Fi Access point, satellite dish and router, lockable access doors to the rear of the housing, room light with day/night sensor, and 20 Tablets. This version allowed for direct Internet access due to the availability of Wi-Fi. The cost of the Wi-Fi was covered by the participating government departments. The Journal of Community Informatics 16 (2020), 141-153 ISSN: 1721-4441 145 The lessons from the respondents were derived from evaluating the usage of these two different DDs. These evaluations were complex as these DDs were not only deployed in schools but also in inaccessible villages and in community centres. DD facilitators The DDs were run by the DD champions, volunteer facilitators who received minor stipends from the participating departments. Adults and learners from Grade 1 to Grade 12 used these DDs for their different needs but in the main to educate themselves on how to use computers. Participatory evaluation: Benefits and Challenges Why did researchers carry out the evaluations of the DDs? Evaluation is a reflection on the process of how the planning and implementation of certain actions in a project or programme take place; in addition, evaluators reflect on the results of the project, anticipated or otherwise (Springett & Wallerstein, 2008). Evaluations were carried out to improve the performance of programmes and projects. Participatory evaluation is carried out in collaboration with stakeholders, that is, those that benefit from the project, and those that fund or have a close interest in the project (Better Evaluations, 2001, Sufian, Grunbaum, Akintobi & Dozier et.al 2011). Sufian et al. further indicate that participatory evaluation involves not just the interests of stakeholders, but also those of beneficiaries who see value in the project. The participants see the value of collecting quality information and how the project can improve their lives (Springett & Wallerstein, 2008 & Sufian, Grunbaum, Akintobi & Dozier et.al 2015). The benefits of participatory evaluations were summarized by INTRAC, (2017) and Better Evaluation (2001) as follows: (i) Participatory evaluation empowers beneficiaries to understand their situation better and contribute toward the improvement of their lives; (ii) it offers beneficiaries the opportunity to contribute to the formation of decisions which directly influence their lives, (iii) it improves the relationship between the project leader and the beneficiaries, (iv) it improves the quality of information collected and analysed through evaluation because beneficiaries understand the impact of the project/program on their lives and understand their situations better than other stakeholders, (v) beneficiaries were better placed to explain how the project changed their lives and what changed, why and when they started experiencing the difference, and (vi) allows for accessing a wide range of perspectives on the project. Participatory evaluations can be difficult to undertake, as one would expect from such a process that involves many different stakeholders with differing skills. The challenges of a participatory evaluation were summarised by INTRAC (2017), Springett & Wallerstein (2008) and Better Evaluation (2001) as follows: Lack of facilitation skills, need for more evaluation time, beneficiaries not being involved from the beginning, lack of clarity of purpose of participation, lack of cultural and contextual understanding, and the implications of these for the evaluation design, among others. Research design and methodology Value derived from participatory evaluation If the respondents had been involved in the participatory evaluation, Outcome Mapping, and narrative inquiry, why did we concentrate only on participatory evaluation? The basis for this stemmed from the following assertions: The Journal of Community Informatics 16 (2020), 141-153 ISSN: 1721-4441 146 • Because the DD was designed in such a way that knowledge is self-acquired, it was crucial to hear and learn directly from the beneficiaries and respondents to share the lessons from other DDs. • They learned and derived most pleasure by engaging with all groups of stakeholders and beneficiaries in finding solutions to problems; • The participatory evaluation helped DD champions, community leaders and members, government officials, teachers, and even learners to improve project implementation on the go. They were able to analyse problems, obstacles, and bottlenecks and instantly propose solutions. This led in many instances to immediate corrective action and improvements to the implementation of the DD project. • They found it easy to embrace the participatory evaluation method as it generated knowledge which resulted in immediate application of lessons learned. • They realized that this evaluation process provided beneficiaries and stakeholders with the tools to be transformative. • This means that narrative evaluation was infused in the lessons they are providing which will assist other new cohorts of field researchers. Participants in the research The respondents acquired their knowledge by evaluating at least 46 centres across the country. The respondents in this study were six researchers form the CSIR. They were trained on Outcome Mapping, Logical Framework and Narrative Enquiry in order to prepare them to carry out their research fieldwork. This background provides an insight into why they found certain things new to them as they were not specialists in participatory evaluations. At the time of collecting the data, four respondents were still working with the CSIR, while two had left the organisation. Interpretivism philosophy This study uses interpretivism as its philosophy, a social sciences approach which accepts the notion that individuals create meaning within a specific context (Hanson 2008). “Interpretive methods start from the position that our knowledge of reality, including the domain of human action, is a social construction of human actors” (Walsham, 2006). Both Hanson and Walsham agree that interpretivists attempt to discover the way people make sense, within the context of their own environment including its culture, of their own world. Qualitative research methods Qualitative methods were employed together with relevant strategies for data collection. The strategies included keeping journals/notes, reports, and semi-structured questionnaires. For the two researchers that were no longer in the employ of the CSIR, telephonic interviews and questionnaires were administered via the WhatsApp messaging tool. For the four participants who were still with the CSIR face-to-face interviews were undertaken. In order to triangulate the responses a short questionnaire was sent out to the six respondents to verify and to seek clarity on previously provided face-to-face and telephonically given answers. The objectives of all these questions and the ensuing analysis was to assist with compiling a document of lessons learned by the respondents over the eight year period of the The Journal of Community Informatics 16 (2020), 141-153 ISSN: 1721-4441 147 programme. The respondents were allowed to express their feelings and perceptions about the way they had carried out these evaluations. It was important to understand if there were other things they could have done differently had they known better. The responses were coded and led to the themes which will be covered in the findings Data analysis Data collected using the different collection tools were subjected to a thematic analysis. The responses were written on a Microsoft excel spreadsheet and as they were written the big themes were identified. These themes were classified in terms of the number of occasions they were expressed by the respondents and also by question(s). Findings and Analysis The following is based on the results provided by the respondents through the different data collection tools. The format of the questions were based on literature reviews and classified mainly under following four categories: (a) Planning, (b) Deployment, (c) Usage, and (d) Outcomes. The diagram (fig. 3) below is a summary in a diagrammatic form of the study results as expressed by the respondents. Fig 3: Results from the respondents a) Planning The respondents recognised early on that planning was critical for the successful implementation of the project. Planning involved (i) carrying out the baseline study, and (ii) marketing and awareness creation, as shown in fig. 3. These elements were realized through time, identification of stakeholders and participants selection, introductions (teams and individuals), language for engagement, and participant consent. • Time and message: The respondents found that the identification of the start of a project and sticking to that time is critical: everyone must know it and stick to it. This goes with a clear message of why the project is deemed important. They indicated that where these two were at odds with each other the DD project was less successful. • Identification of stakeholders: They found that all identified stakeholders or their representatives should be involved during planning stage. The respondents found that the community’s involvement was critical to the successful implementation of The Journal of Community Informatics 16 (2020), 141-153 ISSN: 1721-4441 148 projects. It was crucial to involve everyone from planning to the end so that during evaluations everyone had the same understanding. • Language: The respondents found that speaking the local language or identifying a community member who is multilingual to fulfill the role of translator was critical. • Consent and incentives: The respondents found that the issue of incentives was not that critical to the communities they worked with because people in the communities found the project to be an incentive on its own. Parents, teachers, and learners believed that the project contributed to their well-being and that the will to participate in the project was self-driven. They discovered that in many cases teachers were forcing learners to participate which is unethical but also did not engender any willingness to fully engage in the DD due to lack of individual’s consent. b) Deployment The respondents identified three main components for a successful DD deployment, and these were: (i) Infrastructure: This included all the ICT Infrastructure and systems such as the hardware, software, networks, etc. which the public needed for the provision of the services. Therefore, the timely installation of the equipment was deemed critical and at the core of this was the functionality of the infrastructure. • Functionality in this case was a subset of this infrastructure and was about the usability of the installed equipment for the benefit of the community members. (ii) Personnel: Communities needed certainty. DDs which were predictably ready for usage were extensively used. Predictability included knowledge of when they were open and closed, and the knowledge that there was readiness on the part of the staff to serve the public while managing these DDs. (iii) Handover: Proper handover of the DD to the recipient communities was crucial because it was usually preceded by much fanfare, marketing, and awareness creation, which all contribute to the knowledge about the existence and purpose of these DDs. The respondents contended that wherever one of these three was missing the likelihood of the project to experience problems increased. c) Usage The respondents found that usage of the DD was a critical and complex element of any well-functioning technology. Usage depended on access and skills. No one had anticipated at the beginning of the project that issues of physical access would become a challenge. After the equipment deployment the problems associated with usage were identified: (i) Access: to the premises where DDs were located (physical access) and access to the actual machines became an unexpected problem. People were employed to open and to manage the daily activities at these premises and to see to it that all machines were always working. However, many people were not able to physically access the infrastructure because of staff absenteeism. To avoid this problem, volunteer staff who lived outside of the areas where the DDs were The Journal of Community Informatics 16 (2020), 141-153 ISSN: 1721-4441 149 deployed should not be hired because once their stipends got depleted some people stop going to work. This issue hindered the use of computers, printers, Wi- Fi, etc. and just the basic provision of services. (ii) Skills: the main aim of the DDs was to empower people to learn to use the computers without any help or with minimal intervention from the champions. However, their lack of skill discouraged several elderly people from using the DDs, especially where there were problems with the champions. d) Outcomes The respondents learned that these DDs were able to achieve the desired outcomes. For example, learners became champions and started teaching their peers on the use of DDs. These learners did not need any encouragement from teachers or champions but wanted to be around the DDs. Due to political tensions in certain areas community members looked to the researchers for a voice of reason and this led to some leaders opposing the use of these DDs. Where there was trust between community members, their leaders and the researchers, engagement was smoother and the property was better protected than in areas where trust was absent. Analysis a) Planning For the successful design and good implementation of any M&E activity planning is critical and is a precondition (UNDP, 2009). The credibility of the project was questioned where some people felt left out. It is at this time that introductions are paramount because everyone knows the go to stakeholder when problems arise. It is crucial to identify the different stakeholders and introduce them to other partners to ensure that there is “a clear understanding of their perceived and stated partnership needs” (World Bank, 2010). Language is another critical element of planning. In many of these development projects, at times field workers do not necessarily speak the local language. It is always important to speak the ‘local language fluently’ although research is still manageable with the assistance of translators (Walsham, 2006). Another critical issue is that of Consent and incentives: Consent forms part of the buying- in to the project and reassures participants from the start about the purpose and confidentiality of the research. This is universally accepted (World Health Organisation, 2013, Walsham, 2006). However, a more controversial issue is that of incentives. Vance (2011) argued that although the issue of incentives was debatable, incentives are the only way to get research participants to give up their time. b) Deployment The respondents showed that deployment needs to be planned for properly. Poor deployment, indicating lack of readiness could lead to the failure or unsustainability of the project (Gorgens & Kusek, 2010). c) Usage Physical access to the infrastructure is easy to manage, but the challenge the respondents identified was that of skills. There was a need to balance the adoption of a flexible approach that supported “an empowerment process of social change” and the need to show “results and fulfill internal rules and regulations” (Holland & Ruedin 2012). It is, The Journal of Community Informatics 16 (2020), 141-153 ISSN: 1721-4441 150 therefore, the work of these champions to guide the elderly whenever they wanted to use the DDs. d) Outcomes The respondents learned that these DDs were able to create their own outcomes. For example, learners became champions and began to teach their peers on the use of DDs. These learners did not need any encouragement from their teachers or the champions but loved being around these DDs. Again, for a successful implementation of a project of this nature it is imperative for the funders, researchers or anyone coming from outside the area to understand the local politics and dynamics. Understanding local politics is critical to the success of any development project (OECD, 2008). Due to political tensions in certain areas community members looked to the researchers for a voice of reason and this led to some leaders opposing the use of these DDs. “Effort to build trust with key stakeholders, to provide public education and to institutionalise forms of public engagement with council have begun to build the layers of public accountability that can best protect the hardware investments of council” concurred Smith (2011). Where there was trust between community members and leaders and the researchers, engagement was smoother, and the property was better protected than in areas where trust was absent. The need for social dialogue where there is a constant conversation between researchers and other stakeholders such as learners, teachers, community leaders, government officials, etc. is critically important (Foko, 2006). Study weaknesses The major weakness of this study was the involvement of the six field researchers as the respondents to the research. Some respondents found it difficult to realise that they were no longer researching the DDs, but the objects of a different study. It was difficult for them to transition from being researchers asking questions to being respondents and answering questions. However, to minimise this weakness several data gathering tools were used, as proposed by Reeves and Hedberg (2003) who noted that this weakness can be lessened by conducting and obtaining qualitative comments from the respondents. Conclusion The lessons imparted by the researchers were crucial to future research as they reflect the experiences gained on the ground while carrying out research. The study shows that regardless of how well a project is implemented, there are numerous lessons which can still be learned, and from which future projects can benefit. Future researchers must be aware that these lessons are generally not new but scattered in many journals and reports. Acknowledgments This work acknowledges the DD project, which is supported by the South African government department and the DD team of Next Generation Enterprises and Institutions, for allowing us to collect data from the participants in the different provinces, enabling us to put our experiences together in the form of this article. Special recognition has to be given to all those the team interacted with in the various rural communities including, among others, government officials, community leaders, local communities, learners and parents and all who have embraced this initiative and became co-creators of their own destiny. Last but not least, our gratitude also goes to the anonymous reviewers whose comments have enhanced the quality of this work. The Journal of Community Informatics 16 (2020), 141-153 ISSN: 1721-4441 151 References Better Evaluation. (2001). Participatory Evaluation. Retrieved from https://www.betterevaluation.org/en/plan/approach/participatory_evaluation Comic Relief. (2011). Theory of Change Review: A report commissioned by Comic Review Viewed 10 April 2019 from http://mande.co.uk/blog/wp-content/uploads/2012/03/2012- Comic-Relief-Theory-of-Change-Review-FINAL.pdf Foko. (2006) The role of computer games and social constructivism in skills development of learners from different educational backgrounds, PhD Dissertation, University of KwaZulu- Natal, Durban, Viewed 10 April 2019 from http://researchspace.ukzn.ac.za/handle/10413/133 Gorgens, M., & Kusek, Z.J. (2010). Making monitoring and evaluation systems work: a capacity development toolkit, Washington, DC: World Bank. Viewed 09 May 2017 from http://documents.worldbank.org/curated/en/708391468331216900/Making-monitoring-and- evaluation-systems-work-a-capacity-development-toolkit Gush, K., & De Villiers, M.R. (2011). Qualitative study on software application usage and user behaviour at South African Digital Doorway sites. Paper presented at the 5th IDIA Conference: ICT for development: people, policy and practice, Lima, Peru. Viewed 10 April 2019 from http://www.developmentinformatics.org/conferences/2011/papers/gush.html Hanson, J.D. (2008). Incorporating the human element: The role of interpretive research in operations management, POM 19th Annual Conference, La Jolla, California, USA, May 9 – 12 May, 2008 Holland, J. and Ruedin, L. (2012). Monitoring and evaluating empowerment processes: Poverty Reduction and Pro-Poor Growth: The Role of Empowerment, OECD, Viewed 10 April 2017 from https://www.oecd.org/dac/povertyreduction/50158246.pdf INTRAC. (2017) Participatory Evaluation. Viewed 20 March 2020, from https://www.intrac.org/wpcms/wp-content/uploads/2017/01/Participatory-evaluation.pdf Konyana, S and Konyana, E.G. (2013). Computerization of Rural Schools in Zimbabwe: Challenges and Opportunities for Sustainable Development: The Case of Chipinge District, South-East Zimbabwe, AJOTE, Vol. 3 (2), iewed 10 April 2019 from https://journal.lib.uoguelph.ca/index.php/ajote/issue/view/162 Leibbrandt, M., Woolard, I., Finn, A., Argent, and J. (2010). Trends in South African Income Distribution and Poverty since the Fall of Apartheid, OECD Social, Employment and Migration Working Papers, No. 101, OECD, from Publishing. http://dx.doi.org/10.1787/5kmms0t7p1ms-en Mitra, S., Dangwal, R., Chatterjee, S., Jha, S., Bisht, R. S. & Kapur, P. (2005). Acquisition of computer literacy on shared public computers: children and the ‘Hole in the wall’, Australasian Journal of Educational Technology, 21, 3, 407–426. Mthembu, Sindi Z., & Mtshali, Fikile G. (2013). Conceptualisation of knowledge construction in community service-learning programmes in nursing education. Curationis, 36(1), 1-10, Viewed 10 April 2019 http://www.scielo.org.za/scielo.php?script=sci_arttext&pid=S2223- 62792013000100017&lng=en&tlng=en. OECD. (2008). The OECD DAC Handbook on Security System Reform: Supporting Security and Justice, OECD Publishing, Paris. DOI: Viewed 10 April 2017 from http://dx.doi.org/10.1787/9789264027862-en https://www.betterevaluation.org/en/plan/approach/participatory_evaluation http://mande.co.uk/blog/wp-content/uploads/2012/03/2012-Comic-Relief-Theory-of-Change-Review-FINAL.pdf http://mande.co.uk/blog/wp-content/uploads/2012/03/2012-Comic-Relief-Theory-of-Change-Review-FINAL.pdf https://www.intrac.org/wpcms/wp-content/uploads/2017/01/Participatory-evaluation.pdf http://dx.doi.org/10.1787/5kmms0t7p1ms-en http://dx.doi.org/10.1787/9789264027862-en The Journal of Community Informatics 16 (2020), 141-153 ISSN: 1721-4441 152 Reeves, T. C., and Hedberg, J. G. (2003). Interactive learning systems evaluation, Englewood Cliffs, NJ: Educational Technology Publications Smith, A. (2012). Enabling technologies: Information and communications technology: Digital Doorway installed Down Under, Viewed 10 April 2019 http://journals.co.za/docserver/fulltext/CSIR_sci/6/3/CSIR_sci_v6_n3_a32.pdf?expires=149 1989247&id=id&accname=guest&checksum=4A7B2FF1AD542F11A246DCC878E8CE80 Smith, A., Cambridge, G. and Gush, K. (2006). Digital doorway computer literacy through unassisted learning in South Africa, Pretoria, Council for Scientific and Industrial Research, Viewed 10 April 2019 https://researchspace.CSIR.co.za/dspace/bitstream/handle/10204/2676/Smith_P_2006.pdf?se quence=1&isAllowed=y Smith, L. (2011) The limits to public participation in strengthening public accountability: A reflection on the ‘Citizens Voice’ initiative in South Africa, Journal for Asian and African Studies, Viewed 10 April 2019 http://journals.sagepub.com/doi/abs/10.1177/0021909611403708 Springett, J. and Wallerstein, N. (2008). Issues in Participatory Evaluation Retrieved from https://www.researchgate.net/publication/292125087_Issues_in_participatory_evaluation Stats SA. (2019). Youth graduate unemployment rate increases in Q1: 2019, Viewed 10 April 2019 http://www.statssa.gov.za/?p=12121 Stillman, L., Herselman, M., Marais, M., Pitse-Boshomane, M., Plantinga, P. & Walton, S. (2012). Digital Doorway: Social-technological innovation for high-needs communities, EJISDC (2012) 50, 2, 1-18, Retrieved from http://www.ejisdc.org/ojs2/index.php/ejisdc/article/viewFile/880/397 Sufian, M., Grunbaum, J.A., Akintobi, T.H, Dozier, A., Eder, M., Jones, S., Mullan, P., Weir, C.R. and White-Cooper, S. (2015). Chapter 7. Program Evaluation and Evaluating Community Engagement Viewed 10 April 2019 https://www.atsdr.cdc.gov/communityengagement/pce_program_background.html World Bank. (2010). Participatory Monitoring and Evaluation, in Topics: Participation and Civic Engagement, Washington D.C.: The World Bank. Viewed 10 April 2019 from http://web.worldbank.org/WBSITE/EXTERNAL/TOPICS/EXTSOCIALDEVELOPMENT/ EXTPCENG/0,,contentMDK:20509352~menuPK:1278203~pagePK:148956~piPK:216618~ theSitePK:410306,00.html UNDP. (2009). Handbook on Planning, monitoring and evaluating for development results, Viewed 10 April 2017 from http://www.undp.org/eo/handbook Van der Vyver A.G and Marais, M A. (2013). Evaluating users’ perceptions of the Digital Doorway: a narrative analysis. Accepted for publication by the Information Technology for Development Vance, M. (2011). Qualitative Research Incentives: 5 Reasons Why More is Better, Market Research Tips, Viewed 10 April 2019 from https://rmsresults.com/2011/03/14/qualitative- research-incentives-5-reasons-why-more-is-better/ Walsham, G. (2006). Interpreting information systems in organizations, Chichester: Wiley. Walton, S. and Johanson, G. (2011). Community Advocates and ICT Diffusion in a Poor Community A Measure of Community Informatics. CIRN Prato Community Informatics Conference 2011. https://www.researchgate.net/publication/292125087_Issues_in_participatory_evaluation http://www.statssa.gov.za/?p=12121 https://www.atsdr.cdc.gov/communityengagement/pce_program_background.html http://web.worldbank.org/WBSITE/EXTERNAL/TOPICS/EXTSOCIALDEVELOPMENT/EXTPCENG/0,,contentMDK:20509352~menuPK:1278203~pagePK:148956~piPK:216618~theSitePK:410306,00.html http://web.worldbank.org/WBSITE/EXTERNAL/TOPICS/EXTSOCIALDEVELOPMENT/EXTPCENG/0,,contentMDK:20509352~menuPK:1278203~pagePK:148956~piPK:216618~theSitePK:410306,00.html http://web.worldbank.org/WBSITE/EXTERNAL/TOPICS/EXTSOCIALDEVELOPMENT/EXTPCENG/0,,contentMDK:20509352~menuPK:1278203~pagePK:148956~piPK:216618~theSitePK:410306,00.html http://www.undp.org/eo/handbook https://rmsresults.com/2011/03/14/qualitative-research-incentives-5-reasons-why-more-is-better/ https://rmsresults.com/category/market-research-tips/ https://rmsresults.com/category/market-research-tips/ The Journal of Community Informatics 16 (2020), 141-153 ISSN: 1721-4441 153 World Health Organisation. (2013). Information for researchers concerning informed decision making, Viewed 10 April 2019 from http://www.who.int/rpc/research_ethics/informed_decision/en/ http://www.who.int/rpc/research_ethics/informed_decision/en/