von Groote PM, Skempes D, Bickenbach JE. Evaluation of an implementationstrategy for a World Health Organization (WHO) public health report: The implementation of the International Perspectives on Spinal Cord Injury (IPSCI) in Romania (Original research). SEEJPH 2017, posted: 11October 2017. DOI 10.4119/UNIBI/SEEJPH-2017-151 1 ORIGINAL RESEARCH Evaluation of an implementation strategy for a World Health Organization (WHO) public health report: The implementation of the International Perspectives on Spinal Cord Injury (IPSCI) in Romania Per M. von Groote1-3, Dimitrios Skempes1,2, Jerome E. Bickenbach1,2 1Department of Health Sciences and Health Policy, University of Lucerne, Lucerne, Switzerland; 2Swiss Paraplegic Research, Nottwil, Switzerland; 3Institute of Social and Preventive Medicine (ISPM), University of Bern, Bern, Switzerland Corresponding author: Per M. von Groote, Institute of Social and Preventive Medicine (ISPM), University of Bern; Address: Finkenhubelweg 11, CH-3012, Bern, Switzerland; E-mail: per.vongroote@gmail.com von Groote PM, Skempes D, Bickenbach JE. Evaluation of an implementationstrategy for a World Health Organization (WHO) public health report: The implementation of the International Perspectives on Spinal Cord Injury (IPSCI) in Romania (Original research). SEEJPH 2017, posted: 11October 2017. DOI 10.4119/UNIBI/SEEJPH-2017-151 2 Abstract Aim: This paper aims to evaluate a strategy for the implementation of public health policy recommendations from the World Health Organization’s (WHO) report “International Perspectives on Spinal Cord Injury” in Romania. More specifically, it seeks to: a) evaluate implementation actions with a focus on a number of people reached and status of completion at 12 months follow-up; b) describe implementation activities undertaken in the course of one year, and; c) evaluate perceived barriers and facilitators of implementation at 12 months follow-up. Methods: A cross-sectional design was adopted with two surveys administered in 2014/15 among key implementers in Romania. The questionnaires contained open-ended, multiple choice and 5-point Likert scale questions. Results on the implementation status, implementation activities performed and self-reported barriers and facilitators were analysed and reported using descriptive statistics. Results: Implementation completion rate was 75%, with 4390 persons directly or indirectly benefiting from the implementation-related activities listed in the final implementation plan reporting. A broad range of implementation experiences was reported. Most common activity types were delivery of services, technical trainings, implementation coordination and development meetings. Most useful tools and processes were the Romanian language version summary of the report, educational meetings, and local consensuses processes. Reported outcomes included the direct output produced, evidence of services provided, and individual or organizational level impact. Most barriers were named for the policymakers and academia as stakeholder groups and most facilitating influences for the private sector, with dependence of policymakers on constituency interest scoring highest barrier and the general availability of European Commission and European Structural Funds highest facilitator. Conclusion: The surveys proved to be both feasible and useful tools to expand our understanding of implementation and to supplement the more standard used implementation strategies at country level. Keywords: implementation, implementation strategy, public health report, spinal cord injury, World Health Organization. Conflicts of interest: None. Funding statement: The project received funds from the Swiss-Romanian cooperation programme as part of Swiss contribution to the enlarged EU. Acknowledgments: The authors would specifically like to thank Dr. Cristina Ehrmann- Bostan for her continuous support in analysing the data and preparing display items, and Dr. Jan D. Reinhardt for his conceptual feedback in drafting the manuscript. von Groote PM, Skempes D, Bickenbach JE. Evaluation of an implementationstrategy for a World Health Organization (WHO) public health report: The implementation of the International Perspectives on Spinal Cord Injury (IPSCI) in Romania (Original research). SEEJPH 2017, posted: 11October 2017. DOI 10.4119/UNIBI/SEEJPH-2017-151 3 Introduction Although spinal cord injury (SCI) is a low-incident condition, it can be devastating and costly in both human and social terms (1). SCI can occur as a consequence of trauma, infection, inflammation, degeneration, tumour, or another disease and often results in a significant decline of physical capacity (2). SCI - a principal cause of permanent disability - has become a significant concern for public health mainly because it places substantial socioeconomic burden on affected individuals and their families, communities and the healthcare system (3). It is considered a particularly pervasive stressor as people who sustain these injuries experience profound alterations in almost all aspects of their life (4). However, many of the difficulties experienced by people with SCI do not result from the condition itself, but from inadequate medical care and rehabilitation services, and from barriers in the physical, social and policy environments (5). Implementation of measures aiming at removing barriers to access to healthcare and enhancing the effectiveness of rehabilitation and community reintegration is therefore imperative (1). To help propel the implementation of evidence-informed health care and policy for people with SCI forward, the World Health Organization (WHO) in collaboration with the International Spinal Cord Society in 2013 published a global report titled “International Perspectives on Spinal Cord Injury” (IPSCI) (6). The report assembles and summarizes the best available scientific evidence and information on spinal cord injury together with the lived experience of people with spinal cord injury and makes recommendations for actions that are consistent with the aspirations for inclusion and participation as expressed in the United Nations Convention on the Rights of Persons with Disabilities (CRPD) (1). The CRPD (7) reaffirms the universal human rights and fundamental freedoms of all people with disabilities and calls upon states to secure and promote their inclusion and participation in all aspects of civil, social, economic and community life. Notably, the treaty marks a paradigm shift in understanding disability as the result of physical and social barriers interacting with impairments and health states in a way that deprives people of equal opportunities for societal participation. This view implies that multiple systems and stakeholders from health to social and employment sectors must undertake coordinated actions to translate the normative recommendations of international law into concrete benefits for those living with disability (8). For this reason, the WHO has recognized the necessity to strengthen governments’ capacities in implementing their legal obligations through evidence based programmatic guidance, including guidance on policy implementation. Indeed, while the Convention is among the “most significant policy catalysts” for disability policy at the global level, nonetheless, “the most significant implementation constraints are at the national level” (7,9). To investigate all aspects of implementation, including activities used to put interventions or innovations into practice and contextual factors that influence these activities, one can look toward implementation research (10). This discipline offers insights for selecting evidence- informed policies and interventions, identifying how to implement these in the disability context across populations and resources, and evaluating outcomes. In implementation research, widespread development of programmatic instruments and innovative tools promises to expedite policy implementation in various contexts. These tools are to a large extent tailored to specific purposes and contexts and have limited prospects for large-scale or long-term prospective testing (11). It is now well-established, however, that the transfer of knowledge to support implementation is more complex than it usually appears and is more difficult in the trans-disciplinary domain of public health policy (12,13). Pragmatically, there is no “one-size-fits all” health policy and it would be naive to expect von Groote PM, Skempes D, Bickenbach JE. Evaluation of an implementationstrategy for a World Health Organization (WHO) public health report: The implementation of the International Perspectives on Spinal Cord Injury (IPSCI) in Romania (Original research). SEEJPH 2017, posted: 11October 2017. DOI 10.4119/UNIBI/SEEJPH-2017-151 4 implementation tools to work across different domains of public health policy, from clinical care guidelines to policy recommendations of international public health organizations. Generally, the WHO’s effort to strengthen health policy implementation research and practice has been led by the Alliance for Health Policy and Systems Research with its international academic and civil society partners network (14).The Alliance highlights the value in documenting and analysing implementation experiences and sharing lessons for unravelling the otherwise invisible facets of the complex process of policy implementation and allowing implementing agents, facilitators and ‘champions’ to better understand their practice and realize their roles by reframing their perspective and refocusing their expectations. This will lead to better judgments about whether a particular strategy works or is relevant to other circumstances and situations, leading to measurable improvements in efficient health systems (15). In light of this, the objective of this paper is to evaluate an implementation strategy for the WHO IPSCI report in Romania. The context The research project was led by a partnership between a Romanian non-governmental organization dedicated to delivering health and social services to people with disabilities and a Swiss health research institute specialized in SCI. The partnership organized the development of a strategy to implement the IPSCI report in Romania in August 2012. The strategy consisted of a set of implementation actions or interventions described in a central implementation plan, to work in combination, and administered by a coordinated group of implementers. The resulting implementation activities that are evaluated in the present paper started in March 2014 and lasted for 12 months. Implementation research is by definition a participatory, stakeholder-driven and evidence- informed process (10). Adopting this approach is particularly important in disability research as persons with disabilities have long been denied equal voice in research and policy processes due to power asymmetries and misallocation of technical and financial resources. In this project, the participatory process of developing the implementation strategy and its evaluation involved three main phases: the preparatory phase, the implementation strategy development phase, and the monitoring and evaluation phase. The preparatory phase consisted of a group discussion by the research project team to identify and select mechanisms to develop the strategy. The implementation strategy development phase encompassed focus group interviews (FG) of people with SCI, policy makers, system and service developers and managers, and representatives of NGOs to elicit insights into key implementation considerations, a stakeholder dialogue (SD) with participants from the same pool of FG participants from Romania and international experts to develop an implementation strategy, including the use of tools and processes. The development process was informed by a conceptual framework and guiding principles which have been previously developed by the authors (16). The monitoring and evaluation phase included surveys administered over the course of one year to monitor implementation activities by a core implementation group and evaluate the strategy. The question was now, what actually happened on the ground during 12 months of implementation and in how far the development process infused implementation activities that were successful. More specifically, this paper seeks to: a) evaluate implementation actions with a focus on the number of people reached and the status of completion at 12 months follow-up; b) describe implementation activities undertaken in the course of one year; and c) evaluate perceived barriers and facilitators of implementation at 12 months follow-up. von Groote PM, Skempes D, Bickenbach JE. Evaluation of an implementationstrategy for a World Health Organization (WHO) public health report: The implementation of the International Perspectives on Spinal Cord Injury (IPSCI) in Romania (Original research). SEEJPH 2017, posted: 11October 2017. DOI 10.4119/UNIBI/SEEJPH-2017-151 5 Methods Due to the lack of validated instruments to measure activities using the conceptual implementation framework, and given the research aim of focusing on the tools developed during the SD and documenting their use in implementation activities undertaken, new and fit-for-purpose survey questions were developed by the research project team. As a result, two surveys were developed in the preparatory phase and finalized after the implementation strategy development phase: First, the Online Report Card Survey to document implementation activities throughout the course of one year, and second a One Year SD Follow-up Survey to capture implementation experiences such as perceived barriers and facilitators, among others. In addition, the implementation plan was used as a basis for the summative evaluation of activities at 12 months after the start of implementation. Surveys Development: The Online Report Card Survey questions were first developed by the lead author along the central elements of the comprehensive implementation framework and based on insights from the FGs and SD (16). The survey was independently reviewed by an implementation science expert and an expert on rehabilitation systems and services from the research project team. The survey was tested by a third health scientist who was not involved in the project. The questions were revised based on feedback. The One Year SD Follow-up Survey questions were developed based on the online survey and on first screening of response data to its questions. This survey was reviewed by two team members and reviewer comments were incorporated in the revision. Setup and design: The surveys were self-administered, with both quantitative and qualitative data elements. They contained both open-ended questions and questions with predefined response options ranging from yes/no (‘Did the activity take place in relation to another event or initiative?’) to five-level psychometric scales (‘What tools were used during the implementation activity and how useful where they?’ – ‘very useful’ to ‘not at all useful’). Both surveys were administered in English. The Online Survey was administered beginning after the SD in four waves from March 2014 until February 2015 capturing implementation activities during 3-month reporting periods each. It took approximately 20 minutes to complete each time. The SD follow- up survey was a one time, one year follow up survey to the SD. Components: The Online Survey was composed of nine personal and demographic questions followed by 26 questions categorized by the essential implementation components asking, among others, about the kind of implementation activity, relation to the three central themes identified during the SD (Medical rehabilitation and follow up in the community, Independent living, Employment and Inclusive education), tools used including those introduced during the SD, processes followed, relation of activity to IPSCI recommendation, and perceived receptiveness of audience. In the One Year SD Follow-up Survey participants were asked to judge the extent (0 -5 Likert scale) of hindering and facilitating influence attributes or factors of stakeholder groups had on implementation. These attributes had been jointly identified during the SD and were now being evaluated based on 12 months of implementation experience. Participant recruitment: Participants included a convenience sample of ten residents of Romania, seven who had participated in the SD and three from the focus groups. As described elsewhere, participants of the SD and focus groups had been recruited on a participant roster developed by the researchers to maximize heterogeneity and representativeness. All participants were given an information sheet about the survey and asked to sign a consent form. von Groote PM, Skempes D, Bickenbach JE. Evaluation of an implementationstrategy for a World Health Organization (WHO) public health report: The implementation of the International Perspectives on Spinal Cord Injury (IPSCI) in Romania (Original research). SEEJPH 2017, posted: 11October 2017. DOI 10.4119/UNIBI/SEEJPH-2017-151 6 Data analysis: Qualitative survey data from open-ended questions were content-analysed by two researchers. The specificity and intensity of responses were determined by means of a thematic, open and selective description of meaningful concepts and themes using sentences as units of analysis (17,18). Categories were then produced using inductive reasoning. Conversely, descriptive statistics of quantitative data, such as frequency distributions, were carried out. Implementation plan During the SD, five documents were developed that separately described problems related to the three central themes identified during the SD (i.e., SCI medical rehabilitation and follow up in the community, Independent living, Employment and Inclusive education), options to target these problems, facilitators and barriers by stakeholder groups, and next steps. These documents then served as a baseline analysis for the development of the implementation plan. The final evaluation of the plan was based on the categories ‘number of people reached’ and ‘status’ defined as either ‘completed’ or ‘incomplete’ at 12 months implementation by the core implementation group. Results Implementation plan The implementation plan listed 40potential actions in the categories presentations, publications, report development activities, trainings, services, consultations, conferences, and social events. Actions planned included, among others: the development of a group statement based on IPSCI recommendations, which was disseminated among key stakeholders; a 2-day scientific conference held in Bucharest; a Disability Gala that was broadcasted on national television; an emergency call centre for persons with SCI; an SCI rehabilitation guide; and a meeting with high ranking government officials. Of those listed, 29 actions were rated as “completed” and 11 as “incomplete” (75% completion rate). Those listed as incomplete included also activities already planned or still in progress at 12 months. In total, 4390 persons had directly or indirectly benefited from the implementation related activities listed in the final implementation plan reporting. They were either active participants in activities, such as trainings, or the audience of oral presentations. Implementation activities captured Although the response rate dropped in the Online Report Card Survey, all ten participants responded at least, and often more than, once over the course of the year (10, 8, 3, 4 at time points 1- 4). The one year SD follow up survey to the same pool of core implementers had a response rate of 9/10, one survey was returned incomplete. Overall, respondents seemed to have understood the questions well, as the large majority of open responses were clear and to the point intended. No respondent reported technical problems accessing the online survey platform or the paper based questionnaires. One respondent reported language difficulties and was assisted by a colleague. The Online Report Card Survey captured 36 (14, 12, 5, 5 in time points one to four) implementation activities overall. Table 1 provides an overview of these implementation activities. von Groote PM, Skempes D, Bickenbach JE. Evaluation of an implementationstrategy for a World Health Organization (WHO) public health report: The implementation of the International Perspectives on Spinal Cord Injury (IPSCI) in Romania (Original research). SEEJPH 2017, posted: 11October 2017. DOI 10.4119/UNIBI/SEEJPH-2017-151 7 Table 1. Implementation activities reported Implementation activities (number of reports: 36) Type of activity Percent (number) Delivery of social support services 30.5% (11) ICF training 19.4% (7) Implementation coordination and development meeting 19.4% (7) ICF implementation in support services 5.6% (2) Oral presentation 5.6% (2) Dissemination through personal communication 2.8% (1) Expert workshop 2.8% (1) Guideline development 2.8% (1) Organizing a scientific conference 2.8% (1) Review of current state and report development 2.8% (1) Stakeholder meeting 2.8% (1) Workshop at scientific conference 2.8% (1) Venue or setting Percent (number) Within an organization 36.1% (13) Workshop by invitation only 33.3% (12) Meeting by invitation 27.8% (10) During a scientific conference 8.3% (3) Other 8.3% (3) Within government ministry 2.8% (1) Link to other activity Percent (number) Yes 61.1% (22) No 38.9% (14) Implementation goal Percent (number) Delivery of workshop 22.2% (8) Development implementation content and/or group 19.4% (7) Promotion or dissemination of implementation content 13.9% (5) Professionalization of services 11.1% (4) Social reintegration of wheelchair users 8.3% (3) Implement specialized knowledge 8.3% (3) Improve independence of people with SCI 5.6% (2) Increase awareness 5.6% (2) Improve services and procedures 2.8% (1) Raising level of acceptance and self-competence in PWSCI 2.8% (1) Influencing the revision of disability assessment 2.8% (1) Publish report 2.8% (1) Organizing a conference 2.8% (1) Influence administration of existing services 2.8% (1) Delivery of products and services 2.8% (1) Main implementation theme Percent (number) Independent living 55.6% (20) Medical rehabilitation and follow up in the community 27.8% (10) Employment & inclusive education 16.7% (6) Target audience Percent (number) People with disabilities 50% (18) Disability professionals 33.3% (12) Representatives of government and public authorities 33.3% (12) Civil society 22.2% (8) Health professionals 19.4% (7) Students 11.1% (4) Family members of people with disabilities 8.3% (3) Implementers, implementation and human rights experts 8.3% (3) von Groote PM, Skempes D, Bickenbach JE. Evaluation of an implementationstrategy for a World Health Organization (WHO) public health report: The implementation of the International Perspectives on Spinal Cord Injury (IPSCI) in Romania (Original research). SEEJPH 2017, posted: 11October 2017. DOI 10.4119/UNIBI/SEEJPH-2017-151 8 Pupils and teachers 8.3% (3) Support service professional 8.3% (3) Representatives of international organizations 2.8% (1) Link with IPSCI recommendation Percent (number) 2. Empower people with spinal cord injury and their families 30.6% (11) 3. Challenge negative attitudes to people with spinal cord injury 19.4% (7) 7. Implement recommendations 19.4% (7) 1. Improve health sector response to spinal cord injury 11.1% (4) 5. Support employment and self-employment 11.1% (4) 6. Promote appropriate research and data collection 5.6% (2) 4. Ensure that buildings, transport and information are accessible 2.8% (1) Use of materials and content Percent (number) ICF case studies (distributed) 69.4% (25) Facilitators 66.7% (24) Barriers 63.9% (23) The problem 50% (18) The options 50% (18) Next steps 50% (18) Scientific paper on implementation (distributed) 27.8% (10) Other 16.7% (6) Adaption of content to local context Percent (number) No, the documents were used as they are 36.1% (13) Yes, they were translated further 36.1% (13) Yes, they were shortened 13.9% (5) Other 13.9% (5) Yes, they were rearranged 11.1% (4) Yes, they were rewritten 11.1% (4) Not applicable 5.6% (2) Monitoring tools Percent (number) Longitudinal patient or recipient documentation 30.6% (11) Outcome questionnaire 22.2% (8) Activity documentation 8.3% (3) Group discussion 2.8% (1) Mapping of documents 2.8% (1) Testimonials 2.8% (1) No monitoring of activities 30.6% (11) Receptiveness of audience Percent (number) in favour 66.7% (24) slightly in favour 30.6% (11) neither in favour nor against 2.7% (1) Implementation activities respondents participated in most were by a large margin Social support services (31%), followed by ICF trainings and Implementation coordination and development meetings (both 20%). The majority of implementation activities took place within an organization (34%) and participation was by invitation in 63% of activities. About two thirds of activities were related to other events or projects (63%). Asked to state the explicit goal of the implementation activity they were part of, respondents named the delivery of a workshop or training (n=8) most often, followed by the development of implementation content and / or forming an implementation group (n=5), the promotion or dissemination of implementation content (n=4), and professionalization of services (n=4). In terms of goals targeting the person level, improving independence of people with SCI (n=2), social reintegration of wheelchair users (n=2), their participation in services (n=3), and raising the level of acceptance and self-competence in people with SCI (n=1) were named. von Groote PM, Skempes D, Bickenbach JE. Evaluation of an implementationstrategy for a World Health Organization (WHO) public health report: The implementation of the International Perspectives on Spinal Cord Injury (IPSCI) in Romania (Original research). SEEJPH 2017, posted: 11October 2017. DOI 10.4119/UNIBI/SEEJPH-2017-151 9 Respondents were also asked to what main implementation theme, identified during the SD as main focus of implementation efforts, the activities related to. In 56% of cases and by a large margin these were related to the theme Independent living. In addition, implementation activities mainly related to the IPSCI recommendation Empower people with spinal cord injury and their families (31%). The recommendation Challenge negative attitudes to people with spinal cord injury (20%) and Implement recommendations scored both second highest. Key messages of activities were directed at raising awareness toward problems people with SCI face in terms of accessibility barriers, poor health, denial of rights, and social exclusion. These messages highlighted an inclusive and rights based approach (Obstacles can be overcome and people live independently with the right supports; People with disabilities should be socially and financially independent; People with disabilities have rights they should access). Further key messages targeted the system and service level, calling for an improvement of medical SCI rehabilitation, provision of services based on the ICF approach, and stating that better access to AT and mobility training improves the lives of people with disabilities and the elderly. In addition, employment services should consider all abilities of people with SCI also in relation to their functioning capacity in a specific environment and not only assessed from a medical point of view. Finally, key messages toward implementation stated that successful implementation of IPSCI recommendations would first require a rethinking of legislation and policies on disability in line with CRPD, and that it necessitates joint action by key experts, sustained by policy. The main target audience of activities were people with disabilities followed by disability professionals and representatives from government and public authorities. Out of the seven total documents created or introduced during the SD, the ICF case studies and the facilitators’ document were used most often. In 73% of cases respondents had adapted these documents to the local context, mostly by translation (36%). In terms of processes or techniques used and their usefulness (Figure 1), respondents found in61% of their activities elements of educational meetings or teachings (of health professionals, government employees, people with SCI and families) either fairly useful or very useful as well as local consensus processes (meeting to discuss and agree on implementation goals, steps, etc.) in 47% of cases. Tools rated most useful during implementation activities (Figure2) were the IPSCI summary in Romanian (83% of cases), the IPSCI full report in English (61%), the ICF and own documents or media (53%). Other, very specific WHO media was in the majority of cases not used. Twenty-five out of 36 activities were monitored. About 97% of the target audience reported to have been in favour or slightly in favour (0-5 Likert scale) of the implementation activities. Asked to describe the main outcomes of their activities, respondents named direct output produced, evidence of services provided, and individual or organizational level impact. Activity related output included the development of and promotion of implementation content (n=11), such as an implementation plan, technical information, or a journal article. Also, the organization of an expert group to develop an implementation plan was highlighted as one such direct output. Evidence of service provision (n=13) included the recruitment of clients and services delivered (registration, assessment, program development, training). In addition, some activities were evaluated by participants (n=3) leading to sum scores of how far training participant’s expectations were met. Individual level impact (n=23) was reported as knowledge gain or change of perspective and awareness in the target audiences, including a better understanding of rehabilitation von Groote PM, Skempes D, Bickenbach JE. Evaluation of an implementationstrategy for a World Health Organization (WHO) public health report: The implementation of the International Perspectives on Spinal Cord Injury (IPSCI) in Romania (Original research). SEEJPH 2017, posted: 11October 2017. DOI 10.4119/UNIBI/SEEJPH-2017-151 10 objectives, outcomes and problems by health professionals. Furthermore, specific skills were acquired by the target audience, their independence improved, and their activity and social participation increased. Finally, one respondent named improved working procedures and working tools used within the target organization as a direct organizational level impact (n=1). During the SD participants had listed most anticipated barriers for the stakeholder group policy makers and NGO. Figure 1. Implementation techniques and their perceived usefulness by number of cases von Groote PM, Skempes D, Bickenbach JE. Evaluation of an implementationstrategy for a World Health Organization (WHO) public health report: The implementation of the International Perspectives on Spinal Cord Injury (IPSCI) in Romania (Original research). SEEJPH 2017, posted: 11October 2017. DOI 10.4119/UNIBI/SEEJPH-2017-151 11 Figure 2. Implementation tools and their perceived usefulness by number of cases Respondents also rated the extent of hindering and facilitating factors by stakeholders had on implementation during the last 12 months (Figure 3). These factors had been jointly formulated during the SD and listed most barriers for the policy maker and academia stakeholder group (6 & 4) and most facilitating influences for the private sector (4). Only nine out 26 factors had been rated of no influence and all as either of large or very large influence. The dependence of policy makers on constituency interest scored as highest barrier and the general availability of European Commission and European Structural Funds highest facilitator, possibly counterbalancing the general lack of funds and resources as general barrier. von Groote PM, Skempes D, Bickenbach JE. Evaluation of an implementationstrategy for a World Health Organization (WHO) public health report: The implementation of the International Perspectives on Spinal Cord Injury (IPSCI) in Romania (Original research). SEEJPH 2017, posted: 11October 2017. DOI 10.4119/UNIBI/SEEJPH-2017-151 12 Figure 3. Perceived barriers and facilitators by stakeholder group and in general terms Legend: AC - Academia; DPO – Disabled people’s organization; G- General; NGO – Non- governmental organizations; PM – Policy makers; PS – Private sector; SCI – People with SCI; SPP - Service and product providers. von Groote PM, Skempes D, Bickenbach JE. Evaluation of an implementationstrategy for a World Health Organization (WHO) public health report: The implementation of the International Perspectives on Spinal Cord Injury (IPSCI) in Romania (Original research). SEEJPH 2017, posted: 11October 2017. DOI 10.4119/UNIBI/SEEJPH-2017-151 13 Discussion Summary of main results When summarizing the main results of the surveys it is important to note that multiple and different kinds of implementation activities were undertaken over the course of the monitoring period. These were to a very large extent completed (75%) and had involved over 4000 people. The activities produced direct output, evidence of services provided, and indications of individual or organizational level impact. On closer examination patterns become apparent in the data. A majority of activities were concerned with the delivery of social support services, ICF trainings, and implementation coordination and development meetings. These activities mostly took place within an organization, by invitation and related to other events or projects. Consequently, goals were largely related to improving independence of people with SCI, the delivery of workshops or trainings, and development of implementation content or formation of an implementation group. Subsequently, activities largely related to the overall theme of independent living with key messages of the need for awareness raising, improvement of service provision, and the necessity to coordinate implementation efforts. Evenly matched are the target audiences -- people with disabilities, disability professionals and representatives from government and public authorities. The most useful processes were educational meetings or teachings and local consensus processes. Most notably in terms of tool usefulness is that the IPSCI summary in Romanian, the IPSCI full report in English, the ICF and their own documents and media scored most useful, while WHO media products were least used or useful. Although these tools had been introduced during the SD, implementers resorted to using tools that were more linguistically accessible or their own tools. An additional indication that language accessibility is an important issue is the fact that in one third of cases SD documents were translated for further use in implementation. Finally, the implementers rated substantial hindering and facilitating influences stakeholder groups had on their implementation efforts. Lessons learned Beyond offering insights into actual implementation experiences, we can draw three lessons from the experience that can help in the development and application of an implementation strategy for a WHO public health report. First, we can see that the overall implementation strategy worked in terms of pre-defining activities in a plan and coordinating the implementation groups’ efforts. This was apparent in activity achievement as documented in the implementation plan and established through implementation content and group development meetings as documented in the monitoring survey. Secondly, results indicate that the process of developing the implementation strategy had a positive impact of building the team for the core implementation group, ownership and participation, as well as on focus and the continuation of efforts, and, lastly, on implementation outcomes. Finally, the monitoring mechanism drafted during the implementation strategy development process is feasible, faithful and useful as the surveys were able to display the broad range of implementation experiences with their many facets. This fact underscores the usefulness of the underlying conceptual implementation framework used to map out the survey questions across core implementation components toward planning, administering and monitoring implementation (16). von Groote PM, Skempes D, Bickenbach JE. Evaluation of an implementationstrategy for a World Health Organization (WHO) public health report: The implementation of the International Perspectives on Spinal Cord Injury (IPSCI) in Romania (Original research). SEEJPH 2017, posted: 11October 2017. DOI 10.4119/UNIBI/SEEJPH-2017-151 14 However, the surveys could also benefit from a closer alignment with recently developed surveys in similar contexts (19,20). In particular, this would mean adding survey questions within the Online Report Card Survey assessing the specific impacts the process to develop the strategy had, similar to those survey cycles used in stakeholder dialogue researched by Boyko and colleagues (21). Overall, survey design, analysis and interpretation can be standardized by further application in similar case studies to improve data quality. Results in light of research in the field Waltz and colleagues as part of the Expert Recommendations for Implementing Change (ERIC) study recruited a panel of experts in implementation science to sort 73 implementation strategies and to rate their relative importance and feasibility. The ratings reflect similar processes identified in the present case as the most important and feasible, for instance, identifying barriers and facilitators to implementation, developing stakeholder interrelationships, training and educating stakeholders, and engaging consumers (22). Furthermore, participatory implementation strategy development mechanisms create strong coherence in the implementation group and a shared sense of commitment greatly benefiting outcomes. Findings from the international consortium project ‘EquitAble’ of two European and four African countries highlight very similar lessons learned when developing and implementing a joint project (23). Likewise, identifying implementers during strategy implementation in terms of professional knowledge, involvement in issue, networks, ability to influence, and interpersonal competencies will benefit implementation processes (24,25). Finally, drawing on standard outcome variables proposed for implementation outcome research (26) this study shows that the following are the most relevant: reach in terms of the number of people directly or indirectly involved throughout Romania; adoption and fidelity in terms of level of activity completion; perceived usefulness of implementation tools and processes; and signs of sustainability in terms of technical expertise introduced to services on the ground. Limitations and implications for future research In the present case, much of the implementation success must be attributed to the core implementation group and the influence the individual implementers had in their respective organizations (27). This is particularly evident in the role of main project partners and their effective interplay. Although the core implementation group members were selected from the pool of focus group and SD participants who in turn were invited based on a detailed participants’ recruitment scheme to reach heterogeneity in group composition, the group constitutes a small convenience sample lacking representativeness. This limits the generalisability of results. Expanding on the number of implementers involved will increase reach and generalisability of results. However, inclusion of participants from one particular country only will always introduce a cultural bias. Respondents could have over or underrated specific elements or tools of implementation that are either lacking in their country or are in general under prioritized. In effect, what was accomplished here is a pilot of two surveys that are innovative in their own right and fill an important gap in the toolset of implementation research. Subsequent research using these, or modified versions of these surveys also in other contexts and countries will help to refine the methodology and strengthen the survey approach. Another limitation might be the reporting bias of the implementers. The implementers might have felt obliged to report favourably on implementation interventions within the realm of their own organizations, although specific precautions were made in term of anonymity of responses and disclosure of implementation group composition. It must be kept in mind, however, that the concern here is not so much the accuracy of the reporting, as the self- von Groote PM, Skempes D, Bickenbach JE. Evaluation of an implementationstrategy for a World Health Organization (WHO) public health report: The implementation of the International Perspectives on Spinal Cord Injury (IPSCI) in Romania (Original research). SEEJPH 2017, posted: 11October 2017. DOI 10.4119/UNIBI/SEEJPH-2017-151 15 awareness of the implementers of what they have committed themselves to do. It is extremely difficult to avoid self-serving responses in this context, and independent verification of these results would go far beyond this study. When it comes to evaluation, defining appropriate impact indicators at the start of the project will help measure implementation outcomes and impact in wider contexts in addition to the project-related process and output indicators (28,29). Context data and information on stakeholder influences on implementation could be set in reference to network analysis as it has been applied in health systems research (11,30). Conclusion As WHO’s Alliance for Health Policy and Systems Research has made clear, any effort to strengthen health policy implementation research and practice depends on clear documentation and analysis of the experience of implementers ‘on the ground’. Agents, facilitators and other implementation ‘champions’ have always been the engine of implementation, and it is crucial to understand their motivations, experiences, and self- perception of their implementation roles. In this paper we have presented one important method for achieving this, in the form of surveys used to evaluate an implementation strategy for the WHO IPSCI report in Romania. Despite limitations in this study – described above – it is clear from this initial, piloting of the surveys that they are both feasible and extremely useful tools to supplement the more standard used implementation strategies at country level. References 1. World Health Organization, International Spinal Cord Society. International perspectives on spinal cord injury. Geneva: WHO; 2013. 2. Kirshblum SC, Burns SP, Biering-Sorensen F, Donovan W, Graves DE, Jha A, et al. International standards for neurological classification of spinal cord injury (revised 2011). J Spinal Cord Med 2011;34:535-46. 3. Weerts E, Wyndaele JJ. Accessibility to spinal cord injury care worldwide, the need for poverty reduction. Spinal Cord 2011;49:767. 4. Post MW, van Leeuwen CM. Psychosocial issues in spinal cord injury, a review. Spinal Cord 2012;50:382-9. 5. von Groote PM, Shakespeare T, Officer A. Prevention of spinal cord injury. Inj Prev 2014;20:72. 6. Biering-Sorensen F, Brown DJ, Officer A, Shakespeare T, von Groote P, Wyndaele JJ. IPSCI, a WHO and ISCoS collaboration report. Spinal Cord 2014;52:87. 7. United Nations. Convention on the rights of persons with disabilities, Resolution 61/106. New York, NY: United Nations; 2006. 8. World Health Organization, World Bank. World Report on Disability. Geneva: WHO; 2011. 9. Priestley M. In search of European disability policy, between national and global. ALTER 2007;1:61-74. 10. World Health Organization, Alliance for Health Policy and Systems Research. Implementation research in health, a practical guide. Geneva: WHO; 2013. 11. Contandriopoulos D, Lemire M, Denis JL, Tremblay É. Knowledge exchange processes in organizations and policy arenas, a narrative systematic review of the literature. Milbank Q 2010;88:444-83. 12. Winter S. Implementation, Introduction. In: Peters J, Pierre BG, editors. Handbook of public administration. London: Sage; 2003:205-11. von Groote PM, Skempes D, Bickenbach JE. Evaluation of an implementationstrategy for a World Health Organization (WHO) public health report: The implementation of the International Perspectives on Spinal Cord Injury (IPSCI) in Romania (Original research). SEEJPH 2017, posted: 11October 2017. DOI 10.4119/UNIBI/SEEJPH-2017-151 16 13. Ettelt S, Mays N, Nolte E. Policy learning from abroad, why it is more difficult than it seems. Policy & Politics 2012;40:491-504. 14. World Health Organization - Alliance for Health Policy and Systems Research, Geneva, Switzerland.http://www.who.int/alliance-hpsr/en/(accessed: March 6, 2017). 15. World Health Organization, Alliance for Health Policy and Systems Research. Investing in knowledge for resilient health systems, strategic plan 2016-2020. Geneva: WHO; 2016. 16. von Groote PM, Giustini A, Bickenbach JE. Analysis and implementation of a World Health Organization health report, methodological concepts and strategies. Am J Phys Med Rehabil 2014;93:S12-S26. 17. Zhang Y, Wildemuth B. Qualitative analysis of content. In: Wildemuth B, editor. Applications of Social Research Methods to Questions in Information and Library Science Santa Barbara, CA: Greenwood Press; 2009:308-19. 18. Miles MB, Huberman AM. Qualitative data analysis, An expanded sourcebook. Beverly Hills, California: SAGE; 1995. 19. Lavis JN, Boyko JA, Gauvin F-P. Evaluating deliberative dialogues focussed on healthy public policy. BMC Public Health 2014;14:1. 20. Moat KA, Lavis JN, Clancy SJ, El-Jardali F, Pantoja T. Evidence briefs and deliberative dialogues, perceptions and intentions to act on what was learnt. Bull World Health Organ 2014;92:20-8. 21. Boyko J, Lavis J, Dobbins M. Deliberative dialogues as a strategy for system-level knowledge translation and exchange. Healthcare Policy 2014;9:122-31. 22. Waltz TJ, Powell BJ, Matthieu MM, Damschroder LJ, Chinman MJ, Smith JL, et al. Use of concept mapping to characterize relationships among implementation strategies and assess their feasibility and importance, results from the Expert Recommendations for Implementing Change (ERIC) study. Implement Sci 2015;10:1. 23. MacLachlan M, Amin M, Mji G, Mannan H, McVeigh J, McAuliffe E, et al. Learning from doing the EquitAble project, content, context, process, and impact of a multi- country research project on vulnerable populations in Africa. Afr J Dis 2014;3:1-12. 24. Greenhalgh T, Robert G, Macfarlane F, Bate P, Kyriakidou O. Diffusion of innovations in service organizations, systematic review and recommendations. Milbank Q 2004;82:581-629. 25. Newman J, Cherney A, Head BW. Policy Capacity And Evidence-Based Policy In The Public Service. Public Management Review 2016:1-20. 26. World Health Organization (WHO). A guide to implementation research in the prevention and control of noncommunicable diseases. Geneva: WHO, 2016. 27. Hupe P. What happens on the ground, persistent issues in implementation research. Publ Pol Adm 2014;29:164-82. 28. Fretheim A, Oxman AD, Lavis JN, Lewin S. SUPPORT tools for evidence-informed policymaking in health 18, planning monitoring and evaluation of policies. Health Res Policy Syst 2009;7:S1-S18. 29. Oxman AD, Bjorndal A, Becerra-Posada F, Gibson M, Block MA, Haines A, et al. A framework for mandatory impact evaluation to ensure well informed public policy decisions. Lancet 2010;375:427-31. 30. Blanchet K, James P. How to do (or not to do), a social network analysis in health systems research. Health Policy Plan 2012;27:438-46. ______________________________________________________________________________________ von Groote PM, Skempes D, Bickenbach JE. Evaluation of an implementationstrategy for a World Health Organization (WHO) public health report: The implementation of the International Perspectives on Spinal Cord Injury (IPSCI) in Romania (Original research). SEEJPH 2017, posted: 11October 2017. DOI 10.4119/UNIBI/SEEJPH-2017-151 17 © 2017 von Groote et al; This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. Evaluation of an implementation strategy for a World Health Organization (WHO) public health report: The implementation of the International Perspectives on Spinal Cord Injury (IPSCI) in Romania Abstract Introduction Methods Results Discussion Conclusion