125 Abstract The goal of this paper is to present and discuss the diffi culties in defi ning and measur- ing innovation. The large number of existing defi nitions is yet failing to make the necessary distinction between innovation and other related concepts. There is no agreement about the most relevant dimensions of innovation and, without a proper defi nition of innovation, measurement is more diffi cult. Several methods and attempts of measure- ment – Awards and Surveys – are identifying different meanings and characteristics of innova- tion. A more in-depth analysis of a pilot exercise in measuring innovation, namely the European Public Sector Innovation Scoreboard 2013, is done in order to examine its methodology and the validity of its fi ndings for the specifi c case of Romania. The need for an innovative public sector is generally recognized, and present measurement attempts are actually overestimating the level of innovation. In order to assess this correctly a clearer and narrower defi nition is needed, the framework of the innovative process needs to be further investigated and research designs have to be improved. Keywords: innovation, measurement, Euro- pean Public Sector Innovation Scoreboard. MEASURING PUBLIC SECTOR INNOVATION Sorin Dan ŞANDOR Sorin Dan ŞANDOR Professor, Department of Public Administration and Management, Faculty of Political, Administrative and Communication Sciences, Babeş-Bolyai University, Cluj-Napoca, Romania Tel.: 0040-264-431.361 E-mail: sandor@fspac.ro Transylvanian Review of Administrative Sciences, No. 54 E/2018 pp. 125-137 DOI:10.24193/tras.54E.8 Published First Online: 2018/06/29 126 1. Defi ning innovation ‘If you can’t measure it, you can’t manage it’ is a widely used but incomplete quote often att ributed to Deming or to Drucker, the original source being – ‘It is wrong to suppose that if you can’t measure it, you can’t manage it’ (Deming, 1994, p. 35). Osborne and Gaebler (1992) use a similar catch phrase in order to emphasize measurement: ‘What gets measured gets done’. Starting from the 1990s, performance indicators became more and more popular and in more and more fi elds of activity. Governments around the world are widely using measurement systems for innova- tion thinking that measuring innovation is a must in order to manage it. The fi rst step of any measurement process is defi ning the concept that has to be measured. Innovation was explored since Schumpeter saw it as a central source for economic development more than a hundred years ago (1911). In Schumpeter’s view innovation was the commercial application of a new idea. Only by the end of the 1980s the fi rst att empts to defi ne innovation in order to be measured were taken. The OECD Working Party of National Experts on Science and Technology Indicators (NESTI) codifi ed innovation in what is known as the Oslo Manual. The fi rst defi nition of in- novation (OECD, 1992) was strictly about technological innovation and comprised two components: product innovation and process innovation. Only the third version of the manual was dedicated to all kinds of innovation (not only technological) and the defi nition became ‘An innovation is the implementation of a new or signifi cantly improved product (good or a service), or process, a new marketing method, or a new organization method in business practices, workplace organization or external rela- tions’ (OECD/Eurostat, 2005, p. 46). Two more components were added: marketing and organization methods as a refl ection of the growing importance of services in the global economy. All the components mentioned by the Oslo Manual are market-ori- ented – and as Gault (2010, p. 155) considers, public sector innovation measurement and analysis has to be quite diff erent from that in the private sector. Katt el et al. (2013) consider that starting from 2000 the literature on public sector innovation intended to move away from the private sector approach but this has been done mostly by adapt- ing it to the public sector realities. A defi nition should present the main characteristics of a phenomenon while clear- ly distinguishing it from other similar phenomena. We identifi ed 11 main characteris- tics that have to be addressed when defi ning innovation. 1. Innovation – is it a Process or a Result? In other words, should we measure the process or its results? Both are of interest, but we should not mix them. NES- TA (Hughes, Moore and Kataria, 2011) approach is clearly process-oriented: it measures wider sector conditions, innovation capability, innovation activity and impact. Other innovation surveys are taking similar approaches. Innovation in the public sector can be defi ned as the process of generating new ideas and im- plementing them to create value for society, covering new or improved processes (internal focus) and services (external focus) (European Commission, 2013). Also, innovation can be defi ned as a change (Hartley, 2005). If we understand innova- 127 tion as the results of the innovation process only the outputs of the innovation ac- tivity would have been of interest. 2. Size – are we discussing small, incremental improvements or are we searching for evolutionary steps? Christensen (1997) advocated for radical or disruptive innova- tions (those with a great impact) – but, instead, more general approaches were pre- ferred – as in European Commission (2011) – ‘any signifi cant improvement’ will do. 3. Novelty – New idea vs. Existing idea. How new should an idea be in order to qual- ify as an innovation? The Oslo Manual, fi rst edition, (OECD, 1992, p. 41) classifi ed novelty as: new to the fi rm, the country or the market, or the world. The fi rst two are used in defi ning innovation, as the last might bring an overlap with invention, but the diff erence between being new to the fi rm and being new to the country (market or sector) is a signifi cant one. Lynn (1997) emphasizes that innovation should be original. Innovation is defi ned by Albury (2005) as ‘creation and implementation’, and by the European Commission (2013) as generation and implementation of new ideas – creation and generation implying a high degree of novelty. Defi nitions that equate innovation with change are not usually specifying the degree of novelty. Other defi nitions stick with new to the fi rm, like West and Farr (1990): ‘new to the relevant unit of adoption’. 4. Adoption – Adaptation or Imitation? If an innovation can be new only to the in- stitution, being already introduced elsewhere, are we discussing about adaptation or imitation? De Lancer Julnes and Gibson (2016) use in their defi nition the term adoption, West and Farr (1990) are talking about introduction and application, Eu- ropean Commission (2013) mentions implementation. We do not have a clear an- swer – is imitation a form of innovation? 5. Intentionality – Intentional or Unintentional? Should innovation be intentional or it may just happen? West and Farr (1990) in their defi nition stress the intentional character of innovation. In some defi nitions a certain amount of intentionality is im- plied, especially when mentioning generation or creation. In other defi nitions there is no hint of intentionality. Innovation as a process supposes intentionality. Many inventions and innovations were adopted against the will of the organizations. 6. Origin – Mandated vs. Voluntary. This is close to the previous characteristic but it is specifi c to public organizations in which change can be imposed from the outside through laws and regulations. If a specifi c law is requiring that an institution has to adopt a specifi c new process can we call that organization as being innovative? 7. Scope – Product, Service, Policy or Process? The Oslo Manual had in its fi rst edition only the fi rst two and added other two components later (marketing and organiza- tion methods). Is this too much already? Galindo-Rueda and Van Cruysen (2016) are reporting about confusions made by managers trying to distinguish between components (especially regarding organizational innovation). Should we go fur- ther than that? Is a defi nition like: ‘Innovation is about the relationship between service providers and their users’ (Hartley, 2005, p. 27) fi tt ing these components or is it adding new ones? What about social innovation (Gault, 2010), or governance 128 innovation (Hartley, 2005) or European Commission’s (2013) innovation through initiatives that foster innovation elsewhere in society. 8. Type – Radical, Architectural, Modular or Incremental? Henderson and Clark (1990) distinguish between the types of innovation according to two diff erent criteria – if the system or the linkages in the system and/or the components (the core concepts) have changed. A radical innovation changes both the system and the components, an architectural innovation addresses only the system, a modular innovation the components and an incremental innovation does not change any of them. Incre- mental innovation may not be considered as an innovation by many defi nitions. 9. Improvement – Do innovations have to be successful? West and Farr (1990) consid- er that innovations have to be ‘designed to signifi cantly benefi t the individual, the group or wider society’ – so intention might be enough. Other defi nitions are asking for some degree of success. European Commission (2013) sees innovation as creat- ing value for society, Innobarometer (2011) demands signifi cant improvements. On the other hand, Hartley (2005, p. 31) considers that you can have ‘innovation but no improvement’, in which we can fi nd innovations unsuccessful or not valuable. 10. Value – How can we defi ne public value? How can we measure it? Cole and Parston (2006), and Bason (2010) found that public value can be found as outcomes, ser- vices, productivity or democracy. The Australian framework for measuring public sector innovation lists as outcomes societal and environmental benefi ts, quality, effi ciency and productivity, employee satisfaction, benefi ts for users, trust and le- gitimacy. As we can see, the list may be extremely long without being exhaustive. There are only few indications on how to measure multidimensional impacts of innovation (Katt el et al., 2013). 11. Sustainability – do one-time initiatives count? All the defi nitions consider that in- novation is a tool to improve the regular services and processes so it has to have a long-lasting eff ect. Moore, Sparrow and Spelman (1997) claim that innovation should be ‘durable enough to appreciably aff ect the operations or character of the organization’. What if some organization fi nds a new way to solve a unique situa- tion – is this innovation or not? 12. Diff usion or Dissemination? Those two terms are often used in an interchangeable way. Still, dissemination is indicating a more active process and it is deemed as an important part of the innovation process. NESTA (Hughes, Moore and Kataria, 2011) is including dissemination as an innovation activity. In the public sector it is important that successful innovations are transferred to other organizations con- cerned, but is really dissemination a part of the innovation process? (Is an organi- zation more innovative if it does more dissemination?) Such characteristics have to be carefully pondered before defi ning innovation. The biggest concern is of confounding innovation with change. De Lancer Julnes and Gib- son (2016, p. 3) defi ne innovation as ‘adoption and implementation of change’. Such a defi nition is too general – any change might be considered an innovation. 129 2. Assessing innovation in the public sector We may distinguish four diff erent ways in assessing innovation: 1. Research on specifi c innovations through case studies; 2. Best practices or Innovation Awards; 3. Managers’ surveys asking about the use of some innovative practices and tech- nologies; 4. Innovation surveys are measuring the innovation state of an entire sector/coun- try, asking about a wide range or all innovation activities and types of innova- tions implemented over a defi ned time period. As Arundel, Bloch and Ferguson (2016) remarked, these are infl uenced by the Oslo Manual and the Community Innovation Surveys based on the Manual. These are subject-based, dedicated to the measurement of the innovation state. The fi rst method is extremely useful when trying to assess the merits of a specifi c innovation, but is not that useful in trying to compare diff erent innovations (as case studies are refl ecting the particular nature of each innovation) and aggregating data in order to assess the degree of innovation in an industry or sector is impossible due to the small number of cases and to their specifi city. Innovation Awards are more useful for a larger number of cases as they have to use the same criteria in assessing diff erent innovations in order to identify the best practices. The scientifi c value of such events is not very high as these awards are mostly celebrations or dissemination vehicles and the data is self-reported by the can- didates. Still the criteria used are instructive. The Innovations in American Government Award organized by the Ash Center for Democratic Governance at Harvard University is using the following criteria: – Novelty, the degree to which the program or initiative demonstrates a leap in cre- ativity; – Eff ectiveness, the degree to which the program or initiative has achieved tangible results; – Signifi cance, the degree to which the program or initiative successfully addresses an important problem of public concern; – Transferability, the degree to which the program or initiative, or aspects of it, show promise of inspiring successful replication by other governmental entities. We can see that innovation has to have a high degree of novelty, has to be success- ful, to address an important issue and to be transferable in order to get the award. A similar initiative took place in Romania, organized by the National Agency for Public Service. Ten criteria with diff erent weights are used. We can see that novelty is less important than diff usion and as important as pre- sentation. Also, the way in which it was done (as in partnership and with citizens’ involvement) is highly valued. There is no indication about the way in which eff ec- tiveness, effi ciency and impact are measured (or if and how the self-reported data is checked). 130 Table 1: National Agency for Public Service Best Practices Award Criteria Weight (%) Novelty 5 Presentation 5 Effectiveness 20 Effi ciency 17.5 Administrative simplifi cation 7.5 Replicability and diffusion 17.5 Partnerships 10 Impact 5 Sustainability 10 Citizens involvement 7.5 Source: National Agency for Public Service (n.d.) Managers’ surveys asking about the use of some innovative practices are used in order to fi nd out if and how some innovations are used in the sector (e.g., a survey on the use of quality management practices). These might be used to shed some light if, to what extent and how some practices are used. Sometimes innovation is seen in such a research as part of a larger picture. United Nations (2015) is assessing the state of e-government, and in the process is also checking for innovative coordination processes and mechanisms for service delivery and citizen engagement and empow- erment. Innovation Surveys for measuring innovation within businesses are becoming more and more popular in the public sector. Arundel, Bloch and Ferguson (2016) list nine diff erent studies done in the last nine years in twelve diff erent countries and one of them in all EU countries. Some of them will be presented in the following section. Hughes, Moore and Kataria (2011) are presenting NESTA – the public sector inno- vation indices (UK). Innovation is defi ned according to the Oslo Manual. NESTA is proposing an innovation index for public sector organizations, based on a scorecard approach using the innovation framework below. Table 2: NESTA – the public sector innovation indices Impact InnovationActivity Innovation Capability Wider Sector Conditions for Innovation Improvement in organizational key performance indicators Accessing new ideas Leadership and culture Incentives Improvement in service evaluation Selecting and developing ideas Management of innovation Autonomy Improvement in effi ciency Implementing ideas Organizational enablers of innovation Leadership and culture Improvement context Diffusing what works Enablers Source: Hughes, Moore and Kataria (2011) The pilot survey showed a high degree of innovation in both sectors surveyed – health and local government. 131 The Australian Public Sector Innovation Indicators (APSII) uses a similar defi ni- tion and a diff erent framework (Australian Government, 2011). Table 3: The Australian Public Sector Innovation Indicators (APSII) Framework Inputs Process Outputs Outcomes Investment in innovation Diffusion of innovation Innovation (activities and implementation) Societal and environmental impacts Human resources and skills for innovation Innovation collaborations Types of innovation Quality, effi ciency and productivity Staff attitudes and attributes for innovation Innovation management practice Innovation novelty Improved employee satisfaction Sources of innovation Innovation culture and leadership Innovation intensity Benefi ts for users Technological infrastructure for innovation Innovation strategy Intangible outputs Other intangible effects (e.g. trust and legitimacy) Environmental conditions User demand and supplier capacity Wider public sector leadership and culture Political and legislative factors Other enablers/barriers to innovation (e.g. research basis, skills shortage) Source: Australian Government (2011) While impact and outcomes are more or less similar, APSII has as outputs diff er- ent characteristics of innovation and as process and inputs we have items related in NESTA to capability. A total of 91.3% of institutions introduced some type of innova- tion over a two year period, mostly (85%) regarding process innovation (Australian Government, 2013). A similar survey is EU’s Innobarometer. Data from this research, but also from other sources of data (Eurostat, World Bank, World Economic Forum, United Nations or OECD), are incorporated into the European Public Sector Innovation Scoreboard. 3. European Public Sector Innovation Scoreboard 2013 Innovation is regarded as ‘any signifi cant improvement in operational practices (either in the product/service range or in support structures)’ (European Commis- sion, 2011, p. 13). The framework used by EPSIS is simpler than NESTA or APSII – en- ablers make activities possible and these activities generate outputs. Some of the criteria might be debatable. 3.2. and 3.3 are not innovations made by institutions but innovations ‘through’ the public sector (European Commission, 2013). Quality of public services and especially the use of e-government are seen both as an enabler and as an innovation. The major fi ndings were (European Commission, 2011, p. 8): – At EU level, two-thirds of public administration institutions introduced a new or signifi cantly improved service in the last 3 years before the survey; – The likelihood of service innovation increased linearly with the size of the insti- tutions. State institutions were just as likely as independent ones to introduce innovations; 132 – The single most important driver of innovation in the public sector was the in- troduction of new laws and regulations, followed by new policy priorities and mandated implementations of an online service provision; – Budget cuts are triggering innovation; – Major barriers are lack of resources and rigid regulatory framework; – Major sources of innovation were: ideas from staff , ideas from management and input from clients or users; – Top-down approach of innovation or innovation support is prevalent; – Positive eff ects of innovation were identifi ed as: improved user access to infor- mation, improved user satisfaction, more targeted services, a faster delivery of services, simplifi ed administration, improved working conditions or employee satisfaction and cost reductions; – New technologies, higher demand from citizens, new policy priorities and new regulations are expected to improve the organizational innovation ability. The results for each country are represented in Figure 1. Green light on one indi- cator means that one country is in the top third scores, yellow light that it is in the second third and red light in the bott om third. We can see that out of the 23 criteria Table 4: EPSIS Framework Enablers Activities Outputs 1.1 HUMAN RESOURCES 2.1 CAPACITIES 3.1 INNOVATORS 1.1.1 Employment share of ‘creative occupations’ 2.1.1 Share of service innovators that innovate in-house 3.1.1 Share of organizations in public admi- nistration with services, communication, process or organizational innovations 1.1.2 Share of employees in public administration with a university degree 2.1.2 Share of process innovators that innovate in-house 3.1.2 Share of ‘New’ services out of all services innovations 1.2 QUALITY OF PUBLIC SERVICES 2.2 DRIVERS AND BARRIERS 3.1.3 Public sector productivity 1.2.1 Government effectiveness 2.2.1 Importance of internal barriers to innovation 3.2 EFFECTS ON BUSINESS PERFORMANCE 1.2.2 Regulatory quality 2.2.2 Importance of external barriers to innovation 3.2.1 Improvements in public services for business 1.2.3 Increased effi ciency of government services due to the use of ICT 2.2.3 Active management involvement in innovation 3.2.2 Impact of innovative public services on business 1.2.4 Online availability of public services 2.2.4 Importance of external knowledge 3.3 GOVERNMENT PROCUREMENT 1.2.5 E-government development index (EGDI) 2.2.5 Share of employees involved in groups that meet regularly to develop innovations 3.3.1 Government procurement as a driver of business innovation 3.3.2 Government procurement of advanced technology products 3.3.3 Importance of innovation in procurement Source: European Commission (2011) 133 Fi gu re 1 : E PS IS S co re ca rd So ur ce : E ur op ea n Co m m iss io n (2 01 1) 134 Romania has four green lights (in university education, share of employees involved in groups that meet regularly to develop innovations, innovators and improved ser- vices), eight yellow lights (especially in capacity and drivers or barriers) and eleven red lights (for the quality of public services – data which is not self-reported in the Innobarometer, internal barriers, new services and procurement). The image is not of a particularly good performance regarding innovation. Only 9.8% of the institutions could have been classifi ed as leading innovators (EU27 – 16.9%) and 36.3% of the in- stitutions (EU27 – 33.7%) are not innovators. There are still some intriguing fi ndings regarding Romania, in which the national fi gures are above the EU27 average: 1. The EU27 share of in-house process innovators is quite high at 76%. For small and medium enterprises the EU27 percentage is at 30.3% (ProInno Europe, 2011). Romania has a greater percentage – 82%. 2. Management is active in promoting innovation – in Romania the percentage of institutions where this is happening is at 46% (EU27 – 41%). 3. In 30% of the Romanian public institutions staff meets regularly to develop inno- vations (EU27 – 22%). 4. The percentage of Romanian public administrations that innovate in services, communication, process or organizational matt ers is 93% (EU27 – 90%). 5. In Romania, innovation has almost only positive eff ects – 94.8%, (EU27 – 93.8%), litt le negative eff ects – 15.6% (EU27 – 30.3%), mainly costs. 6. New or signifi cantly improved services (out of all innovations) in Romania 17% (EU27 – 27%). 7. Improvement in public services for business: in Romania – 32% (EU27 – 20%). But the impact of innovative public services on businesses is reported only by 12% in Romania (EU27 – 15%). These fi ndings appear when we have self-reported data – which might bring a doubt about the results. 4. Conclusions Public administration seems to be a highly innovative sector. The fi ndings are showing us that: – Most of the public institutions are innovating; – In-house innovation is fl ourishing; – Management actively promotes innovation and a large number of employees are involved in innovation; – The positive eff ects of innovation are overwhelming compared to negative ef- fects. Such results seem too good to be true. The public sector is not a laggard sector but neither a top Research and Development company (considering its att ributions). Sev- 135 eral possible methodological fl aws might aff ect the quality of the fi ndings: • Poor defi nition of innovation and of criteria: – Any change can be considered innovation – the formulation ‘any signifi cant new or improved service’ is allowing people to report any change as a signif- icant improvement; – Means or tools are confounded with goals and further with success (e-govern- ment or administrative simplifi cation are just two examples); – Change in the laws and regulations (which we saw that it is the main source of innovation) is actually mandated innovation; when it comes with specifi c indications about the way in which the job has to be done and public institu- tions are just applying the law, we are not discussing about innovation; – Measuring process indicators does not measure outcomes. Like in the case of measuring university education (input), it says nothing about innovation; more than that, in many cases, to have a university education is a condition for entering most public service jobs. • The design of the research itself may be an issue – we have institutions as cases. In some institutions there might be more than one innovation. That’s why bigger institutions seem to be more innovative. When discussing about more than one innovation subjects will have some diffi culty in answering. • There is litt le measurement of the public value created – we have questions ask- ing for some possible eff ects, but this is primarily the subjective opinion of the respondents. There are few user satisfaction surveys made on a regular basis in order to assess an increase in user satisfaction. Also, the phrasing of the questions is actually adding to the eff ects: ‘Have any of your new or signifi cantly improved processes or organizational methods […], had a major positive eff ect by…’ – if several innovations were made in an organization, with diff erent eff ects, we may fi nd more positive eff ects. Arundel, Bloch and Ferguson (2016) consider that measuring outcomes works bett er for single innovations and not very well for multiple innovations. • The use of self-administered questionnaire. Self-reporting is usually prone to in- fl ating the results, especially in areas in which the respondents are more per- sonally involved. Questionnaires addressed to an institution will rarely indicate some problems in its functioning and, even less likely, will indicate that there is a problem with management. The main purpose of the survey seems to confi rm the highly innovative status of the public sector. It may be one of the situations in which ‘the measures themselves are easily distorted to suit reporting purposes’ (Seddon, 2013, p. 71). In the end, we might have found that the public sector is a changing one, but not necessarily an in- novative one. While the innovative state of the public sector might be of concern we should be aware that the public sector is quite heterogeneous. Innovation in public education is diff erent from the one in health; innovation in local public administration is also 136 diff erent from the one in central administration. Treating the entire public sector as a monolith is not methodologically sound. While the use of innovation frameworks is not a problem per se, adding inputs with outcomes in scorecards surely is. More elaborate frameworks should be build and the relationships between inputs, enablers, activities, outputs and outcomes should be carefully analyzed – something that is missing in many of the studies done until now. A clear defi nition of innovation should clearly distinguish innovation from change, especially when we have mandated change (e.g. mandated by laws and regu- lations). More information is needed about the novelty, size and type of innovations. Measurement of outcomes is diffi cult, but we need to know if it is based on subjective opinions or on objective data. In terms of design while we can start with institutions (discuss about inputs or enablers at organizational level) we might try to discuss about each innovation – in order to fi nd specifi c information about each major innovation that is happening in the organization. It is known, at least since 2011 (Galindo-Rueda and van Cruysen, 2016), that cog- nitive testing of the questions of the OECD Community Innovation Surveys (a major source of inspiration for NESTA, APSII or Innobarometer) has shown that many con- cepts are not properly operationalized and measured and, yet, no improvement was made. Still, such eff orts should continue and the recommendations should be applied. References: 1. Albury, D., ‘Fostering Innovation in Public Services’, 2005, Public Money and Manage- ment, vol. 25, no. 1, pp. 51-56. 2. Arundel, A., Bloch, C. and Ferguson, B., ‘Measuring Innovation in the Public Sector’, OECD, [Online] available at htt ps://www.oecd.org/sti/087%20-%20ARUNDEL%20 BLOCH%20Methodologies%20for%20measuring%20innovation%20in%20the%20 public%20sector.pdf, accessed on November 3, 2016. 3. Australian Government, Department of Innovation, Industry, Science and Research, ‘Measuring Innovation in the Public Sector: A Literature Review’, The Australian Pub- lic Sector Innovation Indicators (APSII) Project, 2011. 4. Australian Government, Department of Innovation, Industry, Science and Research, 2013, ‘Summary of Key Findings from the Australian Public Sector Innovation Indica- tors (APSII) Project’, [Online] available at htt ps://industry.gov.au/innovation/public- sectorinnovation/Documents/Summary-of-Key-Findings.pdf, accessed on November 2, 2016. 5. Bason, C., Leading Public Sector Innovation: Co-creating for a Bett er Society, Bristol: Uni- versity of Bristol Policy Press, 2010. 6. Christensen, C.M., The Innovators Dilemma: When New Technologies Cause Great Firms to Fail, Boston: Harvard University Press, 1997. 7. Cole, M. and Parston, G., Unlocking Public Value: A New Model for Achieving High Perfor- mance in Public Service Organizations, New Jersey: John Wiley & Sons, 2006. 8. De Lancer Julnes, P. and Gibson, E., (eds.), Innovation in the Public and Non-profi t Sector: A Public Solutions Handbook, London: Routledge, 2016. 137 9. Deming, W.E., The New Economics for Industry, Government, Education, 2nd edition, Bos- ton: The MIT Press, 1994. 10. European Commission, ‘European Public Sector Innovation Scoreboard 2013 – A Pilot Exercise’, [Online] available at htt p://ec.europa.eu/enterprise/policies/innovation/fi les/ epsis-2013_en.pdf, accessed on November 3, 2016. 11. European Commission, ‘Innobarometer 2010: Analytical Report on Innovation in Pub- lic Administration (Flash Eurobarometer 305)’, Brussels: DG Enterprise, 2011. 12. Galindo-Rueda, F. and Van Cruysen, A., ‘Testing Innovation Survey Concepts, Defi - nitions and Questions: Findings From Cognitive Interviews With Business Managers’, Paris: OECD Publications, 2016. 13. Gault, F., Innovation Strategies for a Global Economy. Development, Implementation, Mea- surement and Management, Cheltenham: Edward Elgar Publishing, 2010. 14. Hartley, J., ‘Innovation in Governance and Public Services: Past and Present’, 2005, Public Money & Management, vol. 25, no. 1, pp. 27-34. 15. Henderson, R.M. and Clark, K.B., ‘Architectural Innovation: The Reconfi guration of Existing Product Technologies and the Failure of Established Firms’, 1990, Administra- tive Science Quarterly, vol. 1, no. 35, pp. 9-30. 16. Hughes, A., Moore, K. and Kataria, N., ‘Innovation in Public Sector Organisations: A Pilot Survey for Measuring Innovation across the Public Sector’, NESTA Index report, 2011. 17. Katt el, R., Cepilovs, A., Drechsler, W., Kalvet, T., Lember, V. and Tõnurist, P., ‘Can We Measure Public Sector Innovation? A Literature Review’, LIPSE Project paper, 2013. 18. Lynn, L., ‘Innovation and the Public Interest. Insights from the Private Sector’, in Alt- shuler, A. and Behn, R.D., (eds.), Innovation in American Government. Challenges, Oppor- tunities, and Dilemmas, Washington, D.C.: Brookings Institution, 1997, pp. 83-103. 19. Moore, M.H., Sparrow, M.K. and Spelman, W., ‘Innovation in Policing: From Produc- tion Line to Jobs Shops’, in Altshuler, A. and Behn, R.D., (eds.), Innovation in American Government, Washington, D.C.: Brookings Institution, 1997, pp. 274-298. 20. OECD, Proposed Guidelines for Collecting and Interpreting Innovation Data, Paris: OECD Publishing, 1992. 21. OECD/Eurostat, Oslo Manual: Guidelines for Collecting and Interpreting Innovation Data, Paris: OECD and Eurostat, 2005. 22. Osborne, D. and Gaebler, T., Reinventing Government: How the Entrepreneurial Spirit is Transforming the Public Sector, Addison-Wesley, 1992. 23. ProInno Europe, ‘Innovation Union Scoreboard 2010 The Innovation Union’s Perfor- mance Scoreboard for Research and Innovation’, Maastricht Economic and Social Re- search and Training Centre on Innovation and Technology, 2011. 24. Schumpeter, J., Theorie der Wirtschaftlichen Entwicklung, Berlin: Dunkler & Humblot, 1911. 25. Seddon, J., Freedom from Command and Control: Rethinking Management for Lean Service, New York: Productivity Press, 2005. 26. United Nations, ‘United Nations E-Government Survey 2014. E-Government for The Future We Want’, Washington, D.C.: United Nations Department of Economic and Social Aff airs, 2015. 27. West, M.A. and Farr, J.L., ‘Innovation at Work’, in West, M.A. and Farr, J.L., (eds.), Innovation and Creativity at Work: Psychological and Organizational Strategies, Chichester: Wiley, 1990, pp. 3-13.