Cosmopolitan Civil Societies Journal, Vol.4, No.3, 2012 1 ISSN: 1837-5391; http://utsescholarship.lib.uts.edu.au/epress/journals/index.php/mcs CCS Journal is published under the auspices of UTSePress, Sydney, Australia Everyone’s Story Counts: Measuring Social Impact in the Not-for- Profit Sector – an Overview∗ Barbara Bloch University of Technology, Sydney Abstract As a lead-up to a symposium on measuring social impact in the not-for-profit sector, held in November 2011, the author conducted a small number of interviews (20) across the not for profit sector in Sydney, as to their views, concerns and issues regarding the measurement of social impact of their organisations' work within their communities. This paper provides a synthesis of these interviews, as well as situating social impact measurement within a broader economic and political frame. The paper draws the conclusion (confirmed by many at the above symposium) that the concept of collective impact – where a large number of diverse organisations collaborate through, for example, agreed benchmarks of performance and shared measurements, in order to make a difference, and know that they have made a difference to the lives of specific populations in need. Introduction As a lead-up to the Symposium on Social Impact, the Cosmopolitan Civil Societies (CCS) Centre decided to selectively measure the pulse of the not-for-profit (NFP) sector in Sydney and regional NSW, to gain a sense of their views, concerns, frustrations, issues about measuring the social impact of their organisations and beyond. The symposium focused on the question of how to measure the social impact (as opposed to evaluation) of organisations and programs within the local community. The morning consisted of presentations and discussions of current approaches and issues. In the afternoon, small groups attempted to develop some potential solutions and a future research agenda. Twenty interviews were conducted with managers and program leaders in some peak bodies and local community organisations working with migrants, refugees, older people, young people and people with disabilities. The selection, made through a snow-balling technique, was by no means comprehensive, and was mostly located in the community services and arts sectors. The findings are not conclusive. However, their inclusion at the commencement of the symposium provided an empirical basis for the day’s discussions. Overall, this investigation sought to discover what understandings of social impact measurement existed ∗ The title ‘Everyone’s Story Counts’ is a quote from one of the interviewees for this research. It alludes to a basic issue related to measuring social impact, which is how to put into quantifiable or metric terms, the qualitative aspects of the not-for-profit sector’s work. 2 Cosmopolitan Civil Societies Journal, Vol.4, No.3, 2012 among the subjects interviewed and what generic measures of social impact, with a useable metric base, could be developed, which are not based on financial or economic considerations. What follows in this paper is a discussion and contextualisation of the term ‘social impact measurement’, a discussion of ‘collective impact’, a concept whose application is considerably more developed in the United States than in Australia, and a summary and analysis of the key issues raised by the interviewees. The paper concludes that there is an urgent need for governments and other funders of community services, to encourage those in the welfare and community sectors to work together to produce more beneficial outcomes for those individuals and communities most in need, as well as to find ways to share the measurement of this endeavour. Background to Social Impact Measurement An international professional body concerned with Social Impact Assessment (SIA) has formalised principles and guidelines for measuring or assessing social impact. Their definition is broad and encompasses interpretations of SIA both used, and not used, in this article: ‘Social Impact Assessment includes the processes of analysing, monitoring and managing the intended and unintended social consequences, both positive and negative, of planned interventions (policies, programs, plans, projects) and any social change processes invoked by those interventions. Its primary purpose is to bring about a more sustainable and equitable biophysical and human environment’ (Vanclay 1993, p. 5). Social impact can be defined as any intervention that changes the human environment. Measurement of social impact can occur in advance of an activity occurring, for example before construction of a large housing development, or the impact of a major gas pipeline development in a small town, or before the placement of a youth refuge in a suburb. Local government is ‘increasingly recognising the need to assess the impact of new major developments within their boundaries’ (Davies 2004, p. 1). A few councils, like Newcastle City, have established Social Impact Committees to assess prospect development applications. These kinds of assessments are proactive, examining likely consequences before they occur. Cosmopolitan Civil Societies Journal, Vol.4, No.3, 2012 3 On the other hand, the Productivity Commission’s research into the contribution of the Not for Profit (NFP) sector, conducted in 2010, clearly set out four levels for measuring this contribution ‘impact mapping’ – they are the measurement of inputs, outputs, outcomes and impacts. These measurements one might view as reactive, in the sense that they are seeking to categorise impacts that have already occurred. Impacts were distinguished from outcomes in that they were broader; they tried to capture long-term effects on individuals and the community (Productivity Commission 2010, p.34). They also attempted to capture what they called ‘spillover effects’, or the unintended consequences of an activity, program and so on. The Productivity Commission report stated: ‘Measurement of change at the impact level is important to guide the allocation of resources to deliver the highest community wellbeing for the resources used. Disaggregated, ‘impact’ measures also provide information for targeting interventions as well as assessing broader trends in wellbeing’ (2010, p.34). The researchers Lyons and Zappalà (2009), from the Centre for Social Impact at the University of New South Wales, noted three levels of analysis in measuring the impact of the social economy. They started from the macro to the micro, but here we will reverse the order as it seems to better suit the reality of large parts of the NFP sector, that is for those who work in small organisations. So, the ‘smallest’ level, if you like, or most specific, is measuring the impact of individual programs run by particular organisations – this is perhaps evaluation by another name; followed by the meso or middle level of analysis, which concerns the impact of individual organisations, and finally, the macro level of analysis which is ‘identifying and measuring the contribution or impact of the social economy or the slightly smaller nonprofit sector’ (Zappalà & Lyons 2010, p. 3). The discussion about social impact must be situated briefly within the broader context of the evolving role of and expectations on the NFP sector in Australia (as well as in other, particularly Anglo-Saxon countries where neo-liberal policies have had the greatest influence over the past three decades). Barraket (2006, p.1) states it succinctly: ‘…shifts in public policy are changing the community sector’s operating environment. These policy shifts include: new standards of performance measurement; the movement from core funding of organisations to purchasing their services contractually; an emphasis on partnerships between sectors ….and the growth of social policy responses that are focused on improving local communities of place. These changes are driving the developing of new agendas 4 Cosmopolitan Civil Societies Journal, Vol.4, No.3, 2012 within the sector – in terms of competition, accountability, performance and service delivery – while also creating new opportunities to innovate and diversify’. This increasing emphasis on ‘competition, accountability and performance’ has inevitably led to a monetisation of impact measurement; that is measuring economic impact, or economic well-being, based on the dollar and more generally, the quantification of impact measures, for example the increased numbers of families who have access to a particular community service. Aggregation of a number of quantifiable measures will contribute to a picture of a community’s well-being, but it will by no means paint a complete picture. Social impact, as some of our respondents understand it, is more ‘elusive, partly because it is qualitative rather than quantitative, long-term rather than short term, diffuse and multi-layered rather than specific and focused, and probably means different things in different contexts’ (Onyx 2011, p.1). Collective Impact Collective impact, in relation to measuring the work of not-for-profits, is a concept which is yet to be fully realised. In this study, people spoke about it informally in the sense that there was general acknowledgment that no one organisation or program could be responsible on its own for changes in population indicators. Collective Impact then ‘involves a community – say a State, a school district, a hospital system, a mental health delivery system, discerning that change can only be made if the whole sector collaborates as one. Each individual agency abandons their individual organisational agenda in favour of a collective approach to improving a shared goal’ (Douglas 2011). Collective impact, according to the way it is defined and used in the United States, where it is more extensively practised than in Australia, ‘is the commitment of a group of important actors from different sectors to a common agenda for solving a specific problem’ (Kania & Kramer 2011, p. 36). It differs from networks, partnerships, collaborations partly through its scale (large), through having a centralised infrastructure, a dedicated staff and a structured process that leads to a common agenda, shared measurement, continuous communication and mutually reinforcing activities among all participants’ (Kania & Kramer 2011, p.38). Cosmopolitan Civil Societies Journal, Vol.4, No.3, 2012 5 Kania and Kramer (2011) outline the work of Strive, a project seeking a transformation of the educational attainment of the young people of the greater Cincinnati area (USA). More than 300 separate organisations have come together from schools, principal and teacher groups, governments at all levels, philanthropists, local businesses, health professionals, universities and hundreds of different education related non-profits. Collectively they have planned a way forward, agreed benchmarks of performance and shared measurement – in short, they’ve agreed to ‘sing from the same song sheet’. It has required a separately funded secretariat which convenes, enrols and facilitates all the players to work together more harmoniously. The study defined five success criteria of successful Collective Impact projects: • A common agenda – a shared vision for change including a common understanding of the problem being solved • Shared measurement systems – agreement on a common agenda is illusory without agreement on the ways success will be measured and reported • Mutually reinforcing activities – all the various activities, across all the sectors need to be pulling in the same direction, even if they have very different approaches and methods. This is also where the magic of specialisation comes in – everyone is now able to concentrate on their strength • Continuous communication – relationships and communications within the broader community of collaborators need to be well maintained with plenty of time and space for bonding and connecting. If the trust doesn’t develop, the results won’t come – it’s as simple as that. • Backbone support organisation – running a collective impact project requires a particular skill set. It is about creating collaborative spaces, nurturing trust across a broad set of people, keeping the focus clear, continuous reporting and communicating clearly. It requires moral governance of all the players in the project, ensuring that they are on track in terms of their individual contributions and the way they are playing the game together. The significance of ‘shared measurement systems’ is worth noting. These authors state that shared measurement is essential: ‘having a small but comprehensive set of indicators establishes a common language that supports the action framework, measures progress along the common agenda, …. encourages more collaborative problem-solving and becomes the 6 Cosmopolitan Civil Societies Journal, Vol.4, No.3, 2012 platform of an ongoing learning community that gradually increases the effectiveness of all participants’ (Hanleybrown, Kania & Kramer, 2012). Collective action to achieve greater social impact in Australia is in its infancy, albeit with a few exceptions. For example, the Australian Research Alliance for Children and Youth, whose work on improving the well-being of Australian children rests on the necessity of a shared measurement framework in order to achieve collective impact – working across governments and community service organisations (Emerson, 2011). Progressive evaluation of the complex, multi-pronged ‘Keep Them Safe’ program1 of the NSW government is being undertaken for 3 years until 2014. The evaluation’s effectiveness will depend on analysis of data across a number of government and non-government agencies. Social impact will be able to be measured through accumulation and aggregation (and disaggregation) of both quantitative and some qualitative data. Interestingly, the evaluation framework, developed in 2010, is a very comprehensive document, yet it does not mention the term ‘social impact’ (Adamson et al. 2010). Methodology This small research activity was intended to provide an indication of how a selection of NFP organisations in NSW understood and possibly implemented the measurement of social impact. Due to time constraints, only a small number of organisations were contacted. Given the time frame, the rationale for inclusion was based on selecting a diversity of organisations in terms of size, purpose and function. Several peak bodies engaged with policy work were contacted as well as those ‘on the ground’, conducting service delivery. The questions were devised in order to stimulate the most wide-ranging and open discussion on the subject of social impact. The symposium at the Cosmopolitan Civil Societies Research Centre in November 2011 also yielded data for this paper. Analysis was based on the outcomes of discussion in five focus groups. Each group was asked to discuss the following questions: What kinds of measures were needed to assess qualitatively, the work of their organisations? What issues were related to this measurement, in planning and implementation? What resources and tools were 1 Its overall objectives are improving the safety, welfare and wellbeing of children in NSW, particularly those in the child welfare system. Cosmopolitan Civil Societies Journal, Vol.4, No.3, 2012 7 required to measure social impact? What barriers were there to measuring social impact? What broader concepts could be identified in relation to this topic? The Organisations Data was also collected through a questionnaire to be found in the Appendix. The following organisations, mostly based in Sydney, responded to the questions in person, by telephone or by email. They were: • New South Wales Council Of Social Services (NCOSS) • Information Cultural Exchange Parramatta (ICE) • Local Community Services Association (LCSA) • Western Sydney Community Development Forum • South-West Sydney Community Development Organisation • Metro Migrant Resource Centre, Campsie • Parramatta Community Pals • Metro Lands Council • Canterbury City Community Centre, Lakemba • Illawarra Youth Services • Ethnic Communities Council (ECC) • Bankstown Area Multicultural Network (BAMN) Because these organisations perform different kinds of work, the way they understand impact and seek to measure it is also different. The organisations can be grouped into four categories: those that were solely concerned with direct service provision and its impact; those that do a combination of this and community development, peak bodies who are concerned with their own efficacy as ‘peaks’ as well as the sector they represent and those whose work is more geared towards effecting policy. The responses are grouped under four headings, which reflected the questions asked of them and the ensuing themes and issues which emerged. 1. Defining social impact How did the respondents define social impact? Everyone saw it as much broader than program evaluation, that it was a planning and a predictive tool, which inevitably went beyond the stated objectives of a particular program or service. They regarded it as being 8 Cosmopolitan Civil Societies Journal, Vol.4, No.3, 2012 about long term measurement, which has attendant problems like the issue of attribution which is discussed later. Importantly, they saw it as being about trying to capture unintended consequences, or ‘spill-over’ effect. Social impact was about client and community outcomes, and it asked the fundamental question: What is making a difference? It was about planning in order to make fundamental changes, not based on band-aid solutions, according to one interviewee. Some respondents struggled to find ways to describe SI. Yet, it could help determine if ‘what we do makes a difference, help determine where the resources will go, and where the gaps are’. Another interviewee made the point that it was important not to become preoccupied with measurement, at any level. 2. The difference between social impact assessment and program evaluation It is no surprise that it is program evaluation that is the most common form of measurement. It measures the past, it measures the effectiveness of a program and of course, since it is connected with on-going government funding, it tends to cover those areas, or services, that the specific contract is for. It is concerned with showing that the program provided good financial value, it is concerned with outputs and tends not to ask or answer questions like does the program service solve a problem? Are people being helped? What are the outcomes and for whom? In short, it is confined to those that participate in specific programs and is driven largely by the needs of the funding body. ‘Measurement then, exists only if you can touch it, see it and count it’, in the words of one of the interviewees. For example, one community centre executive officer (EO) described a particular program the centre runs, providing gardening services for elderly people who still live at home. The funder wanted to evaluate output – what was the cost of the hours spent per garden, how many hours of service for paid and unpaid staff, but the interviewee also noted a range of outcomes the funder was not interested in measuring, such as the impact on the elderly who have a regular gardener coming to their homes, the impact on the environment as exotic plants get replaced by natives and need less water, the impact on the volunteers who give their time for this activity. Interestingly, the EO concluded this vignette by saying that she was hesitant to inform the funding body about these less tangible outcomes. It is the amalgamation of hundreds of these kinds of specific examples that becomes ‘social’ impact. Cosmopolitan Civil Societies Journal, Vol.4, No.3, 2012 9 3. If you do measure or assess the social impact of your organsiation’s work, how do you do it? In terms of actual practice, no one in this small cohort, used the frameworks for measuring social impact sometimes used within the third sector, like Social Accounting Audits, Logic Models or Social Return on Investment (SROI). Zappalà and Lyons conclude in their paper that very few organisations have adopted either the SAA process or the SROI. And those which have tend to be the larger charities like Mission Australia or the Smith Family. These authors acknowledge that ‘the reality is that all the approaches are resource intensive for nonprofit organisations to implement in terms of the time taken and the money required to either divert existing staff…or employ external specialists to assist them through the process’ (Zappalà & Lyons 2009, p.22). Their paper focused on the need for funding bodies like foundations, companies and philanthropic organisations to include resources (both financial and capacity building) to their grantees to undertake SIA. Additionally, all three levels of government, the major funding source for the majority of the community/welfare sectors, also need to include financial support for the inclusion of various types of evaluation into their funding contracts. In the words of one executive officer interviewed for this research: Funding bodies generally require raw numbers and a little bit of very basic qualitative assessment in some cases, based around client satisfaction. There is nothing longitudinal or more in-depth to look at the depth of impact that a program has had on participants. It would be great if we were able to introduce it, but it’s always a question of time, skills and resources. Despite these limitations, a number of the respondents discussed practices which they identified as attempts to measure ‘social impact’. For example, one fairly large service noted how they collect data on individual clients across their agency, they also get feedback from external agencies they work with, and their clients. This is a form of triangulation, of corroborating evidence – so vital in a research project. They also discussed the difference between services for specific target groups like youth, or the elderly, and those of a community organisation, a peak regional body or neighbourhood centre, whose activities are more complex, varied, and diffuse. It was noted also that tracking the impact on individual clients, even over a longer period of time, does not necessarily change into social impact. The community arts organisation in Western Sydney, which has grown considerably in the past 5-10 years, has also become more sophisticated and systematic in its data collection and impact measurement strategies. They count where and what they can, they gather qualitative 10 Cosmopolitan Civil Societies Journal, Vol.4, No.3, 2012 feedback from different sources, including their external stakeholders, they track web and social media interactions and collect stories to give a richer context. They observed that tracking changes that go beyond the individual also depends on the community and whether it is possible to track these changes, because communities change, change is a constant feature, even at the point of an intervention or a program, which makes certainty difficult. Before leaving this section on practice, it is necessary to briefly discuss Results Based Accountability (RBA). RBA has been adopted ‘by the Human Services departments of the NSW government as its agreed framework of measuring outcomes and applying for funding’ (Keevers et al. 2010, p. 57), a process which has been ongoing since 2005. The Community Services agency of the NSW Department of Family and Community Services (formerly DOCS), is the funding body most mentioned by my respondents. Nevertheless none of the organisations I interviewed were actually using the DOCS version of RBA, to report on their work, although a number of them had attended training in Mark Friedman’s RBA. (Friedman is the American creator of results based accountability). RBA combines a means for coordinating effort with a performance measurement system. It: starts with the outcomes the activity or program is intended to achieve. It then works backwards to identify the resources and processes by which those outcomes can be achieved, and the indicators by which success can been measured. It attempts to distinguish the specific contribution of the organization from other factors that can influence outcomes (Productivity Commission 2010, p. 52). It also distinguishes between performance – dealing with programs, agencies and service systems and population accountability dealing with ‘whole populations in a community, city, county, state or nation’ (Friedman 2005, p. 39). It is beyond the scope of this article to provide a detailed critique of RBA. A few respondents were critical because they saw the RBA processes as limited, they questioned how NGO knowledge and practices could be measured, stating that they thought the indicators were too broad, that the language of the indicators could not cover measurement of the impact of community development, for example. One report is based on a joint Australia Research Council funded project of various university partners and the Illawarra Community Forum, is entitled ‘Practising Social Justice: Measuring what Matters – Locally based community organization and social inclusion’ (2010). This research examined RBA planning processes Cosmopolitan Civil Societies Journal, Vol.4, No.3, 2012 11 in a number of diverse organisations and institutions in the Illawarra community services field. The following questions from this report are pertinent to all measurement frameworks: The authors query: ‘RBA, like any other measuring apparatus makes boundary cuts. So what got included and excluded from mattering when implementing RBA? What happened to the local practices of social inclusion and justice that community sector workers make and re- make in their everyday/everynight work lives?’ (Keevers et al. 2010, p. 72). Their study points to: The need for accountability and planning processes that recognize and are sensitive to the different knowledges and the distinctive roles and contributions of the diversity of organisations that make up the community services industry. Such recognition would mitigate against the ‘one size fits all’ approach that currently mandates common statewide results and performance measures (Keevers et al 2010: 101) The Local Community Services Association (LCSA) utilises a ‘community builders results based performance accountability framework’, which it is rolling out to neighbourhood centres throughout NSW. This forms the basis for the mandatory data reporting for all Community Builder funded services in NSW2 . It is based on Results Based Accountability by measuring "How much do we do?" "How well do we do it?" and "Is any one better off?" for each of the four Service Activity Descriptions – Community Capacity Building, Community Skills Development, Community Sector Development and Community Hub. One of the interviewees noted the need to be careful about the motivations for collecting data – of course organisations want and need to ‘prove impact’, to justify the programs they run rather than critiquing them in order to improve them. That is, in the words of one of my interviewees, ‘everyone wants to show they’re doing a good job’. 4. What issues do you face in trying to measure social impact? This section covers both issues that arose from the initial interviews, followed by a discussion of further points that were raised in the Symposium itself. Overall, interviewees acknowledged that the sector (that is, the community sector) does not have much understanding of social impact but there was a strongly articulated general need to measure 2 The Community Builders is a funding program which aims to strengthen communities across NSW and particularly the disadvantaged groups within them. The program was developed in response to a growing body of evidence that suggests that community strengthening is an effective way of reducing inequality and disadvantage. Community strengthening is about building the capacity of and opportunities for people to actively engage in a wide variety of social, economic, cultural, recreational, learning and civic activities. It also includes supporting the relationships and networks that result from these activities. (NSW Human Services, Community Services, 2012, Community Builders Guidelines http://www2.communitybuilders.nsw.gov.au/program_guidelines_2012_full.pdf ) http://www2.communitybuilders.nsw.gov.au/program_guidelines_2012_full.pdf� 12 Cosmopolitan Civil Societies Journal, Vol.4, No.3, 2012 other things apart from what the funders needed. A manager from one of the interviewed organisations sent me an email, which encapsulated many of the issues people face. She wrote: Indeed we need measures around social impacts - in my experience it's what do we use as indicators, how do we make time for it and commit resources to it and what type of research methods are useful with our target groups? – these are the issues. Attribution The question of attribution was one of the major issues identified in relation to social impact measurement. For example, if decline in youth suicide rates is an indicator of successful social impact within a particular community or cohort, the impact of a specific organisation is very difficult to measure since most service providers work together, in networks and partnerships. This issue is compounded by the fact that, as everyone acknowledged measuring social impact must include an element of longevity to make it valid. It was said that there are not the evaluation tools to determine this attribution. The Productivity Commission research corroborates this, saying that measuring impact rather than outcomes or output means that ‘there are less data available to construct attributable outcome or impact measures’. It adds that ‘outcomes and particularly impacts are usually the result of a much wider array of influences than just the NFP activity’ (Productivity Commission 2010, p.36) – the point made above in relation to youth suicide rates. In general terms, no one organisation or program can be responsible on their own for changes in a population. There are so many independent variables, indicators need to be holistic, but do they then become of no use? Linear Causality A related concern to attribution is that of linear causality – how can one be sure that x caused y, and it was not because of z or a or b. People can be influenced by other factors apart from direct service provision. There was criticism of Logic Models because they assume a linear causality; that is, you work from what you are doing to prove social impact. They assume that ‘one thing leads to another in predictable or highly probable ways that can be written down in the form of a flow chart’ (Lee 2011, p.2). A number of interviewees spoke about the complexity of causality in the social world, that any social problem and its solutions will have ‘uncertain causal relationships’ (ibid). On the other hand, the importance of measuring impact is to discover whether – together – organisations are doing the right things and are making a difference. Cosmopolitan Civil Societies Journal, Vol.4, No.3, 2012 13 Time and data management The time factor was noted frequently as being one of the main impediments. As well, a respondent said: ‘the community care sector is driven by reporting that requires regular outputs, not outcomes. We need to go outside normal data collection methods, to report on people’s strengths, rather than deficit reporting.’ In general there was considerable frustration expressed about various funding bodies’ data requirements. A few questions articulate this: ‘What do you do with all that data we provide for you?’ ‘What happens with all those reports we provide? Do they just sit on someone’s desk?’ It was said that reporting mechanisms are not designed to assist organisations with planning, nor was there the capacity to do analysis over time of programs and services. In almost all the interviews, there were complaints about data collection – the limited resources for organisations to use the data they collect, to be able to identify trends, especially significant in relation to impact, with its need to look longitudinally. Confirming this anecdotal evidence was a nation-wide report commissioned by the Australian National Council on Drugs (ANCD) in 2008, of alcohol and other drugs NGOs. A survey of 71 CEOs of NGOs revealed that they ‘are spending in excess of 473 hours (12.5 weeks of work) over a 12 month period to simply report back to funding bodies’. ….In other words, these organisations reported that they were diverting more and more resources from the front line into “backroom compliance” as they struggle to complete an array of reporting requirements’ (ANCD 2009). Statistics such as these should alert funding bodies as to the efficacy and purposefulness of demanding such onerous reporting regimes onto the over- stretched community sector. The CCS Symposium The afternoon’s group discussions at the symposium in November allowed for corroboration and further exploration of the issues expressed above. A number of themes emerged: • What do organisations need to measure in social impact? Several groups addressed the “what” question and came up with lists of broad concepts to focus measurement; for example health, levels of safety, human capital, social capital, social justice, diversity, empowerment, including measures of change in attitudes and values, measuring impact on government policies and practices. • How do we capture spill-over effects, that is, the wider impacts beyond those anticipated by the organisation or program? 14 Cosmopolitan Civil Societies Journal, Vol.4, No.3, 2012 • Can we convert impact into a monetary base? What happens when we try to do so, what is lost? • There was discussion about tensions between short-term measures imposed by funding bodies and measures which resonate with community members. It was suggested that an audit be made of the differences between measures used for formal program evaluations required by government and other funding sources, and the kinds of measures/data that service providers are interested in gathering. • In relation to the preceding point, it was thought that service organisations should also develop their own measurement tools and strategies, they should be culturally appropriate techniques to obtain meaningful engagement with diverse communities. It was also suggested that an audit be conducted of the difference between measures and data used for formal program evaluations that are required for governments and other funders; and the measures and data which organisations use for their own purposes. • Barriers to measuring social impact were clearly articulated. They included, (and confirm the earlier discussions in this paper): current lack of agreed theoretical/technical frameworks and models; lack of access to analysis of data; short- term funding cycles of government; government counts outputs, mostly financial, not impacts; the economic principle of market competition inhibits collaboration between NGOs; risky to try new approaches and finally – the time it takes not only to conduct a thorough, rigorous study, but also the length of time in years needed for many types of social impact to be realised. • What kind of resources can be made available to measure impact collaboratively and systematically within the sector? For example one group mentioned centralising resources like websites, academic collaboration, accessing data across programs, creating a shared repository of knowledge and critical analysis. Conclusion The next decade will see significant changes in the not-for-profit sector, particularly in relation to how the sector is funded. The NSW Government, following some evidence from Britain and the US, has instigated a pilot project using social impact bonds (SIB)3 3 SIBs, following from their development in the United Kingdom, are a way of reducing public sector funding for social welfare, by encouraging ‘social investment’ by individual investors, superannuation funds, financial institutions and so on, who will receive a ‘dividend’ from the government, if the social program has achieved its outcome/s. (See Centre for Social Impact, 2011, for discussion on the feasibility of the NSW Government adopting the SIB concept.) , as the Cosmopolitan Civil Societies Journal, Vol.4, No.3, 2012 15 basis for funding two community programs; one to reduce the incidence of out-of-home foster care, and the other to improve recidivism rates among criminals. It is too early to make a judgement on the efficacy of this significant shift in the role and responsibility of the state in funding programs for those who are deemed disadvantaged. However, observations from discussions thus far in Australia reveal that the issue of the seemingly obvious relation between the collective and the social, for measuring impact is not seriously being addressed at a governmental level, whose main concern appears to be reduction in tax-payer spending. Nevertheless, on the basis of this small piece of empirical research, as well as some of the literature perused for this article, it could be concluded that there is an urgent need to think about ways to act together to assist those communities and individuals who are doing it tough, as well as to find ways to share the measurement of this endeavour. References Adamson, E, Bromfield, L, Edwards, B, Matthew, G, Hilferty, F, Katz, I, McDonald, M, McHugh, M and Valentine, K. 2010, Keep Them Safe Evaluation Framework, Final Report, Australian Institute of Family Studies and Social Policy Research Centre, University of New South Wales, Sydney. Australian National Council on Drugs 2009, Non-government organisations in the alcohol and other drugs sector, http://www.ancd.org.au/images/PDF/Researchpapers/ rp17_non-government_orgs.pdf accessed 30 May 2012. Barraket, J. 2006, Community Sector Sustainability: research evidence and public policy implications, Victorian Council of Social Services, Melbourne. Centre for Social Impact, February 2011, Report on the NSW Government Social Impact Bond Pilot: Executive Summary, CSI, UNSW, Sydney. Douglas, E. 2011, ‘Collective Impact - a New Approach to Community Collaboration’, Centre for Social Impact UNSW, Sydney. https://secure.csi.edu.au/site/Home/ Blog.aspx?defaultblog=https://blog.csi.edu.au /2011/03/collective-impact- %E2%80%93-a-new-approach-to-community-collaboration-2, accessed 10 September 2012. Emerson, L. 2011, ‘Collective action to improve children’s wellbeing’, www.conferenceonthenet.com/clients/niftey/docs/Emerson.pdf, accessed 21 February 2012. Friedman, M. 2005, Trying Hard is Not Good Enough: How to produce measureable improvements for customers and communities. Victoria, Canada: Trafford Publishing. Hanleybrown, F, Kania, J and Kramer, M. 2012, ‘Channeling Change: Making collective impact work’, in http://www.ssireview.org/blog/entry/channeling_change_ making_collective_impact_work, 26 January, accessed 14 February, 2012. Kania, J and Kramer, M. 2011, ‘Collective impact’, Stanford Social Innovation Review, Winter. Keevers, L, Treleaven, L, Backhouse, H and Darcy, M. 2010, Practising Social Justice: Measuring what matters, Illawarra Forum Inc: Jamberoo NSW. Lee, P. 2011, What’s Wrong with Logic Models? LCSA: Occasional Paper No. 1, Local Community Services Association. http://www.ancd.org.au/images/PDF/Researchpapers/%20rp17_non-government_orgs.pdf� http://www.ancd.org.au/images/PDF/Researchpapers/%20rp17_non-government_orgs.pdf� https://secure.csi.edu.au/site/Home/%20Blog.aspx?defaultblog=https://blog.csi.edu.au%20/2011/03/collective-impact-%E2%80%93-a-new-approach-to-community-collaboration-2� https://secure.csi.edu.au/site/Home/%20Blog.aspx?defaultblog=https://blog.csi.edu.au%20/2011/03/collective-impact-%E2%80%93-a-new-approach-to-community-collaboration-2� https://secure.csi.edu.au/site/Home/%20Blog.aspx?defaultblog=https://blog.csi.edu.au%20/2011/03/collective-impact-%E2%80%93-a-new-approach-to-community-collaboration-2� http://www.conferenceonthenet.com/clients/niftey/docs/Emerson.pdf� http://www.ssireview.org/blog/entry/channeling_change_%20making_collective_impact_work� http://www.ssireview.org/blog/entry/channeling_change_%20making_collective_impact_work� 16 Cosmopolitan Civil Societies Journal, Vol.4, No.3, 2012 Local Community Services Association 2008, ‘Neighbourhood Centres and Results Accountability. A Conversation with Mark Friedman’, Local, The Newsletter for Community Development in NSW, LCSA. Onyx, J. 2011, Measuring Social Impact: a scoping paper, Cosmopolitan Civil Societies Research Centre, University of Technology, Sydney, unpublished. Productivity Commission 2010, Contribution of the Not-for-Profit Sector, Research Report, Canberra. Vanclay, F 2003, ‘International principles for Social Impact Assessment’, Impact Assessment and Project Appraisal, March. Zappalà, G. and Lyons, M. 2009, Recent approaches to measuring social impact in the Third sector: An overview, CSI Background Paper No 5, July, Centre for Social Impact, University of New South Wales. Cosmopolitan Civil Societies Journal, Vol.4, No.3, 2012 17 Appendix – Interview questions 1. In your opinion, what is the difference between social impact assessment and program evaluation? 2. Are you concerned with the social impact of the work of your organisation? 3. Why (or why not)? 4. Do you try to measure/ assess the social impact of your organisation? By qualitative or quantitative means? 5. How do you do this? (or why do you not do this?) 6. What issues do you face in trying to measure social impact? Everyone’s Story Counts: Measuring Social Impact in the Not-for-Profit Sector – an Overview0F( Barbara Bloch Appendix – Interview questions