Opinion Section 79 Evaluating the Impact and Value of Competitive Intelligence From The users Perspective - The Case of the National Research Council’s Technical Intelligence Unit Jonathan Calof 1 1 Telfer, Canada Email: Calof@telfer.uottawa.ca Received December 20, accepted December 25 2014 ABSTRACT: Understanding and being able to measure and prove the impact and value of intelligence is of significant importance. The objective of this study was to develop an evaluation instrument that the users of intelligence could fill in that could be used to assess both the impact and value of the intelligence they received. Starting with an evaluation instrument based on lists of benefits identified in the competitive intelligence literature, measures of these benefits and client satisfaction/service quality metrics, the study researchers interviewed clients of one large government competitive technical intelligence organization asking them to articulate the benefits they obtained from the intelligence they received and methods for evaluating these benefits. All users of intelligence identified benefits they had received from the intelligence received. Additional benefits beyond those that are in the current literature were identified by those interviewed. In terms of measurement of these benefits, intelligence users (the clients) understood why hard financial type measures for example ROI or dollar impact on performance was important (especially in their organization) they felt that assessing these for the intelligence they received would be difficult but that softer, more subjective measurement such as extent to which the user agrees that the intelligence provided the intended benefit could be used. Additional perceptual based indicators of service quality and customer satisfaction measures were also suggested by intelligence clients. Based on Available for free online at https://ojs.hh.se/ Journal of Intelligence Studies in Business Vol 4, No 2 (2014) 79-90 mailto:Calof@telfer.uottawa.ca https://ojs.hh.se/ Opinion Section 80 the results of the literature review and interviews, an intelligence evaluation instrument was developed that asks the clients to assess the extent to which they have realized one or more of 27 impacts identified in this study as well as assessing 10 elements of service quality. KEYWORDS: Evaluating intelligence, intelligence impact, CTI, technical intelligence, CTI impact, Case Study 1.0 Introduction - the need to understand the impact of intelligence Evaluating the impact and value of competitive intelligence has been identified as an important intelligence research issue for many years (Blenkhorn and Fleisher 2007; Global Intelligence Alliance 2004; Herring, 2007; Davison, 2001; Kilmetz and Bridge 1999; Lonnqvist and Pirttimaki 2006; Viva Business Intelligence 2000). Fehringer et al. (1996) wrote that “the ability to measure and demonstrate the value of CI has consistently been among the top items on many practitioners wish lists and previous surveys have reflected their desire to be able to demonstrate CI’s contribution to their organization” (Fehringer et al. 1996, 99). Kahaner (1997) warned CI professionals on “the need of showing the added-value of their services to ensure the commitment of top management to support” and almost 20 years later it still remains an important issue as highlighted by Global Intelligence Alliance“ MI professionals have been struggling to answer questions related to the expected value and impact of the MI investment for just about as long as the profession has existed “ (GIA 2014, 4). Given this stream of literature and the weaknesses identified, the objective of the study and the article is to develop an instrument that can be used to measure the impact and value of intelligence for its users. 2.0 Measuring and proving the impact of intelligence – literature review Given the importance of showing the impact of intelligence, it is not surprising that many CI practitioners and researchers have proposed (but not tested) frameworks and approaches for doing so. Herring (1996) was among the first authors who tried to identify relevant criteria for CI measurement. He proposed four types of metrics: revenue increase, cost avoidance, and cost and time savings. The concept behind this approach was that the best way to evaluate impact was to identify what impacts intelligence were supposed to bring to those that tasked the intelligence function (the end users/clients) and then find out if these impacts arose. The key contribution of the Herring paper was the identification of the core benefits that intelligence could bring to its users – increase in the company’s revenues, avoiding costly mistakes, saving time and identifying cost savings. For the development of the instrument for this study, therefore, the literature review and instrument must identify the specific benefits that are supposed to arise from the intelligence produced and develop a way of measuring it. Ten years later, Fehringer et al. (2006) expanded this list of impacts and identified 7 values or impacts of CI - financial goals met, new products or services developed, new or increased revenue, cost savings or avoidance, time savings, profit increases, and actions taken. Similar to the Herring approach this related to the direct impact that the intelligence was to lead to, the reason it had been requested. Fehringer et al. (2006) also defined another measurement category they called assessing CI effectiveness which had six factors (Return on investment, competitive intelligence productivity or output, customer satisfaction, decisions made or supported, new products or services, strategies enhanced). This latter category contained both direct impacts (decisions made or supported, new products or services, strategies enhanced) which in a sense results in 10 direct impacts and an indirect measurement of the benefit of the intelligence – customer satisfaction. This measure has its roots in the service marketing literature which posits that a subjective measure (customer satisfaction) is a good proxy for the quality of the service provided. If the customer was not satisfied with the service received (in this case the intelligence) then it would not have much impact (Anderson et al., 2008; Patterson et al. 1996; Wirtz and Lee 2003). Finally, the study also provided direct hard measures in terms of ROI and productivity. Opinion Section 81 The Fehringer et al. study (2006) made three very important contributions to the development of the intelligence impact instrument. 1. It added additional direct impact factors which intelligence products were designed to create. 2. It added a measurement concept of implying benefit based on satisfaction with the service/product itself, a soft measure. 3. It suggested a direct impact measure such as ROI and productivity. Hard and quantifiable numbers. This concept of using a mix of hard (quantifiable) measures and soft (subjective) measures is not unusual in the CI evaluation literature. For example, in 1998 Simon proposed an evaluation framework that included 21 hard measures and 29 soft measures. These are provided in table 1. A similar approach was taken by McGonagle and Vella (2002 – see table 2). As the impact literature developed, additional direct impact factors and measurements were proposed. For example in 2014, Global Intelligence Alliance, in writing about evaluating intelligence listed 3 broad indicators with several factors that could be measured underlying each. Decision making related indicators: Decision-makers’ perception of the availability of information when it’s needed; MI’s involvement and contribution to different types of business decisions in the company; Financial indicators : Calculated financial worth of case- or project-specific MI efforts; Cost savings through coordinated purchases of information and the elimination of Redundancies; Demonstrated time savings through systematically organized market monitoring ; and, Indicators of a qualitative nature : The status of the company’s MI program as measured against the GIA World Class MI Roadmap; The number of active users of the company’s MI software tool and/or participants in internal events that MI organizes; The size of the internal network of people that are involved in MI activities on a regular basis; The number of requests to the MI team; The number of deliverables (regular and ad hoc) that the MI team produces; The development of the internal NPS score of the MI program; Number of new business ideas generated as a result of MI efforts). There have been many other studies that have looked at evaluating intelligence impact that provided valuable input for the development of a research instrument. Davidson (2001) proposed a formula to calculate the return on CI investment (ROCII) for individual projects. He proposed that CI outputs (or the effects of CI plus decision maker satisfaction) less the monetary value of CI inputs (or costs associated with) are divided by CI inputs to derive the ROCII. This measure of impact combined both hard numbers (the monetary impact of CI) with subjective or soft measures (decision maker satisfaction. Pirttimäki et al. (2006) conducted a case study of a Finnish company they examined how intelligence activities were measured. They identified four categories of measurement: financial (e.g. inputs and outputs ratios), process (e.g. inputs and outputs), learning and growth (e.g. organizational learning, decision making) and customer (e.g. usage of intelligence, satisfaction, resources/time). In all, they identified five objects of measurement and specific indicators for each: intelligence output (number of fulfilled assignments), intelligence input (working hours, total costs of information sources, total costs of using services), satisfaction of information users (surveys, feedback), intelligence usage (intelligence portal usage, number of intelligence requests) and intelligence costs (billing, and reports). Table 1. Hard and Soft Measures of CI Success (Simon, 1998) Hard Measures Soft Measures Costs – CI contribution to the bottom line (input) 1. cost of doing the research 2. cost benefit of CI research 3. financial gain from ideas Quantitative measures (output) Customer usability 1. work habits 2. user friendly reports 3. participation on teams 4. contributions to teams 5. communication skills 6. contact follow-ups Opinion Section 82 1. clients serviced 2. projects completed 3. suggestions submitted 4. suggestions implemented 5. projects assisted 6. number of BI/CI staff 7. staff productivity 8. participants in the CI process (direct and indirect) Quality measures 1. Intelligence product measures 2. accuracy of information (validity and reliability) 3. immediate usability of results (no rework) Time measures 1. ability to produce timely info. 2. efficiency 3. time saved by CI 4. on-time delivery CI practitioner performance measures 1. effective use of resources (resourceful and creative) 2. knowledge of CI methods 3. resourcefulness 7. customer satisfaction ratings 8. understanding Acceptance and alliance measures 1. work climate 2. number of requests for service 3. number of repeated requests for service 4. requests for participation in team meetings 5. referrals from customers 6. further integration of CI projects Unit and personnel effectiveness measures 1. feeling/attitude 2. solicitation for services 3. attitude changes – clients taking you in to confidence or consulting with you 4. customer loyalty rating 5. perception of CI contributions 6. relationship building (sharing of personal information) 7. problem solver perception Personnel development/advancement rewards 1. job effectiveness 2. attendance at CI orientation and training programs (participant or teaching) 3. promotion 4. pay increases 5. work accomplishment acknowledgments CI practitioner performance measures initiative 1. implementation of new ideas 2. degree of supervision required 3. ability to set goals and objectives Total: 21 criteria Total: 29 criteria Table 2. CI Measurement according to McGonagle and Vella (2002) Assignments and Projects 1. Meeting objectives 2. Number completed 3. Number completed on time 4. Number requested 5. Number requested—Increase by End Users 6. Number of follow-up assignments 7. Number of projects assisted 8. Number of suggestions submitted Budget 1. Comparative cost savings—compared with cost of outsider 2. Comparative cost savings—compared with cost of untrained 3. Meeting project and function budget constraints Efficiency 1. Accuracy of analysis 2. Data quality Opinion Section 83 3. First time results (no reworking) 4. Meeting project time line 5. Time for research versus time for response End users 1. Creating compelling reasons to use CI 2. Effectiveness of implementation of findings 3. Meeting needs 4. Number of referrals 5. Number served Feedback 1. [Feedback]—written 2. [Feedback]—oral Financial 1. Cost avoidance 2. Cost savings 3. [Financial] goals met 4. Linking CI to specific investments 5. Linking CI to investments enhancement 6. Linking CI to specific savings from unneeded investments 7. Revenue enhancement 8. Value creation Internal Relationships 1. Building strong with end-users 2. Formulating relevant strategy and tactics 3. Quality of relationship with end-users 4. Quality of participation on cross-functional teams New Products and Services 1. Number developed due to use of CI 2. Cost savings/avoidance in development from use of CI Performance 1. Growth profitable for the unit or firm 2. Impact on strategic direction of unit or firm 3. Market share gains for unit or firm Report and Presentations 1. Number 2. Number of follow-ups 3. Production of actionable CI Sales effectiveness Customer satisfaction 1. Linking to specific customer wins 2. Number of customers retained 3. Number of leads generated 4. Repeat business 5. Improvement in win-loss ratio Surveys 1. [Surveys]—Written 2. [Surveys]—Oral Time 1. Gained by CI input 2. Projects delivered on time 3. Saved by input To summarize, in examining the literature around evaluating intelligence impacts four concepts are identified that impacted this studies evaluation instrument: Opinion Section 84 The concept of an intelligence having a direct impact on an action or decision: The literature has identified many of these direct impacts starting with Herrings (led to revenue increase, led to cost avoidance, led to cost and time savings). In measuring the impact and value of intelligence any instrument designed would need to recognize the actual objective of the intelligence provided. The concept of measurement of impact and value using hard indicators: Most studies reviewed for this paper proposed or identified efficiency and effectiveness measures such as return on investment in the specific project (or unit), amount of revenue arising from the intelligence report and so forth. The concept of measurement of impact and value using soft or perceptual based measure: Decision makers perception of availability of information when it was needed, extent to which they agree that a value was received were found in many studies as well as other soft and perceptual measures. The concept of implied impact based on client satisfaction with the service: Questions such as to what extent where you satisfied with the service?, would you recommend it to someone else? Despite all these concepts and several papers that propose evaluation frameworks and measures, few have tested these measures within an organization. It is this gap as well as the weaknesses identified in the 2014 in the Global Intelligence paper and other articles reviewed that this study sought to address. 3.0 Methodology 3.1 Case Study Design Given the areas of importance and weakness in the CI performance evaluation literature described in 2.0, the objective of the study was to develop an instrument that could be used to measure the benefit’s clients received from the intelligence they received and the value of these benefits. In developing the study methodology access was needed to an organization that had conducted a significant number of intelligence studies and had a broad client base. The higher the number of intelligence products (unit of analysis) the larger the base to draw upon to get client feedback on how the intelligence benefited them and how this benefit could be measured. The author was given access to the competitive intelligence unit of the National Research Council (Canadian Government organization), to their intelligence personnel, past intelligence products and clients. The organization refers to the unit as competitive technical intelligence unit as the unit is producing intelligence within a technical environment. Note that performance being an issue of importance has also been extensively written about in the competitive technical intelligence (CTI) literature as well (Rosenkrans, 1998; Norling et al. 2000; Dollatabady et al. 2011). However, the training the staff received and the projects themselves cover far more than just technical intelligence techniques. In reviewing the intelligence products produced by the unit, the researchers noted that the NRC’s CTI unit produced a broad range of intelligence assessments and products. Studies took anywhere from a day to produce (simple patent scans, market analysis or literature reviews) to multiple months in the case of scenarios and expert panels used for policy development. Clients for the CTI were very broad including Canadian companies, departmental technical officers making investment recommendations (whether the government should provide funding to the venture), research recommendations for government scientists, policy advice and so forth. The following approach was used to develop the instrument for measuring the benefits of intelligence to the end user (client) and their satisfaction with the intelligence. A document was developed (which would be shown to intelligence clients) that identified the benefits of intelligence found in the literature review. The document then had suggestions from the literature regarding how to measure these benefits, providing the participant with both soft and hard measures and finally the document contained a listing of the quality of service/customer satisfaction measures seen in the intelligence literature. A sample of the organizations CTI clients was drawn (sampling methodology is mentioned in the next section), who would be interviewed for their opinions on benefits they received from intelligence and how these benefits could be measured. The research team did not want to be seen as biasing the study towards a priori benefits identified in the literature review but wanted to ensure that as comprehensive a list of benefits and measures from the perspective of the user could be developed. As such, rather than present Opinion Section 85 the document with all the benefits identified in the intelligence literature and measures all of those interviewed were asked to list the benefits that they could recall from the CTI project they had commissioned/received. After describing all benefits, the clients were then shown the intelligence reports that they had received from the NRC’s CTI unit and asked if they could recollect any other benefits. After the respondents had exhausted their recollection of benefits, the researchers then showed the respondent the intelligence benefits portion of the document (appended with any new impacts that the respondent had stated in the interview) and asked again to look at the list and to also indicate the extent to which any of the benefits had been received. The interview would then end with a discussion on how each of the benefits on the list could be accurately measured and the quality of service/customer satisfaction measures. After each interview, the study document was modified with the addition of benefits previously not included in the document and the addition of other measures based on interview results. Any additions to the document were based on two researchers independent review of interview notes. In other words, additions arose only if both researchers reached the same conclusions based on the interview notes. To develop the final survey instrument, those benefits receiving at least one mention in the interviews would be included in the final evaluation instrument and those items which respondents did not list as benefits was removed. In some cases some of the items removed not only did not get a single “vote” but were frequently mentioned as benefits which those interviewed did not feel were an appropriately important benefits of intelligence. While the intelligence literature has identified many direct and indirect benefits of intelligence, in assessing impact and value from the user (clients) perspective, the researchers felt that it was important that the benefits measured be those of importance to the clients themselves. 3.2 Sample Frame In all, clients representing over 50% of the organizations intelligence projects were interviewed for this research. To identify who to interview a two-step process was followed. In the first step which offices to focus on was identified and in the second step selection of clients to interview. The organization has intelligence offices across Canada. Some of these are small offices (one or two intelligence staff) and some are large offices. Five offices were chosen for the study. The offices chosen represented those that produced the highest volume of CTI reports and had been involved in producing CTI the longest. Given that the intent of this study was to develop a comprehensive instrument for measurement of benefits it was felt that offices with higher experience levels and greater number of projects would be appropriate. Second, within each office, the researchers sought to identify the clients that they wanted to interview for the study. Similar to the office selection, experience was used as a basis for the selection of the clients selected for interviews. Clients were chosen based on two factors: Volume of CTI products requested: How many products were requested? Who were the most frequent users of CTI? Scope of CTI products requested: The organization has three levels of CTI products, information reports, CTI brief/insight and CTI assessment. The intent was to interview clients who had requested most if not all of these products. As an example of this selection methodology, one of the offices (call it office 1) was selected as it was one of the oldest offices as of the time of the interviews with one of the largest number of CTI projects completed. The office had 21 clients (people that had requested intelligence reports). In reviewing the type and number of projects ordered by these 21 clients, it was noted that five clients accounted for over half the projects in general and almost all the analysis reports. Accordingly, interviews were scheduled with all 5 clients who collectively represented 60% of all projects done in this office. Sampling in this manner resulted in similar project coverage rates. For office #2, 71% of their projects were covered in the interviews, 77% in office #3 and 100% in office 4. 4.0 Results and discussion Based on the methodology described in section 3, 27 decision impact items and 10 service items were included in the final evaluation questionnaire (appendix A). Based on interviews with the clients, only perceptual measures were used in the final evaluation questionnaire and in particular, a likert evaluation scale of perceptual impact was found to be the best method for measuring impact. Opinion Section 86 Impact factors: Support for many of the impact factors cited in literature reviews arose in the interviews and in fact, all clients interviewed articulated that they had received significant benefits from the CTI products and process. Saving time, saving money, making better recommendations, quicker recommendations, etc., all respondents were easily ably to identify benefits from the intelligence they had received. |Additional impacts were cited that the researchers did not note in the current literature. Service quality/client satisfaction: In all interviews, respondents talked about service quality elements when they talked about the benefits. While the initial study design was to have this brought up by the researcher in the interview when discussing measurement, in all cases the interviewees themselves (the clients) talked about their experience with the intelligence staff before being asked about it . Service quality and satisfaction were evident in statements such as professionalism of the CTI officer, how pleasant they were to deal with, their (the clients) desire to use the service again and how they were recommending CTI services to others. These are all measures that have been examined in the management consulting literature as ways to evaluate the professionalism and effectiveness of consultants and consulting units (see the earlier literature review). These statements provided confirmation on the earlier framework that recommended evaluating the intelligence impacts using service quality and client satisfaction metrics. Measurements, soft versus hard. Clients interviewed stated that use of hard measures such as return on investment, impact on decision, etc. would be difficult, if not impossible to do. The participants felt that the only measures that should be used would be a perceptual measure (subjective questions) about whether they felt they had received the benefit. Although all interview participants told stories about the benefits they received and were insistent that these benefits had been received, when asked if they could quantify the benefit the answer was consistently no. Respondents were aware that harder measures such as return on investment, cost/benefits were critical for their organization but cautioned against it for competitive intelligence. However while they could not quantify the benefit they could provide an indication as to the extent to which they had received the benefit using a Likert scale of 1 (strongly disagree) to 5 (strongly agree). When asked why harder measures could not be used, respondents answers fell into five broad categories: Complexity of the clients decisions making process. While the CTI report was clearly used to help make the decision/policy, their decision making was more complex than reading the CTI report and implementing the report recommendations. No respondent was prepared to say how much of the decision was influenced by the report, only that it was an important element in making decisions and developing policies. Here is one example of this difficulty. In one of the intelligence projects, the client stated that the CTI they received was used to provide an investment recommendation (whether the government should provide funding to the Canadian company that had requested technology funding assistance). The client of the intelligence product (the government officer making the recommendation) talked about how their final decision was based on many factors including the CTI report which provided the market assessment, a technical report provided by technical advisor which assessed the underlying technology, and a business analyst report discussing the strength of the organization that would receive the investment. The CTI report contributed but so did the other reports as well as the officers own experience. Additional value added by the clients to the intelligence: This is a slight extension of the complexity of decision making process. Several respondents stated very strongly that in the end they made the decisions/recommendations based on discussing it with others, doing additional research, etc. Call this client value added activities with the intelligence. Complexity of factors beside the intelligence responsible for success and impact: This was mentioned more when the type of intelligence received was designed to help develop new products/services, reduce costs, make sales, almost dealing with growth. In implementing the intelligence recommendations for example designing policy, strategy, R&D programs there are a lot of other factors that need to line up for success to occur. Thus, directly linking the CTI report in a quantifiable way to the success of the technology investment would not be possible. Further, in trying to quantify costs saved, Opinion Section 87 program benefits arising from the decision itself, or policy benefits (when the CTI report clearly impacted the decision), participants pointed out that policy impacts were too complex to be assessed in this way. In one case program impacts were mediated by government elections wherein the recommended policy was scrapped by a subsequent government. In this case the intelligence had no impact due to change of organization. In another case, the intelligence was not fully implemented – the client decided to adopt some of the intelligence reports recommendations but not all. Temporal orientation of the intelligence: While some intelligence was designed to impact decisions that would get results in a short time frame (under a year) other intelligence projects had a longer time frame. One for example was an intelligence report done for seed research for which whether the benefit is received (market share and sales) will not be known for 40 years. Therefore to link the intelligence with the subsequent research success or commercial success in this case would require waiting 40 years. Organizational politics: Many respondents indicated that politically in their organization it would be unwise to credit too much of a decisions success to anything besides their own skills/expertise. Talk of complexity in measurement of benefits was evident in most interviews with those interviewed providing specific examples to the researchers and challenging them to develop a method that would involve hard measures on direct benefits. For example, one of the clients provided a CTI example and challenged the researcher on how it would be evaluated from a financial/ROI perspective. The CTI developed was a market study which told the officer that a government investment in a technology was sound but that the company was focusing on the wrong market. The report identified other markets. The client then discussed this CTI report with the intended recipient of the government technology investment funds. The investment was approved as the CTI report proved that the technology was sound and the technology was built as per the objectives of the CTI. However, the CTI provided to the company caused them to change their marketing approach. The client challenged the research to identify how the ROI of the CTI would be calculated. Was it the value to me (the client) of a good decision or was it the value to the company that was provided with the money? Is the ROI on this one the money saved by not going to the wrong market? Money gained by going into the correct one? Value of not investing in the wrong company? Another client challenged the researcher on what could be best termed an indirect CTI report benefit. The client had commissioned a CTI report to assist with policy development. While the CTI did land up being used as the basis for policy development (verbatim elements of the CTI report were included in the policy), the government client stated that elements of the CTI report were integrated in a speech the officer made to an industry association and an interview conducted with a national news network. The information was then used by many companies in the industry. Again, the client was clear on the benefits he received from the CTI report but stated that there were additional benefits beyond that intended by the report. In a corporate environment this would be similar to intelligence reports being shared by different divisions or people within the same division and impacting their decisions – whose ROI would you measure? Clients were indicating that the value of CTI was greater than just impact on the policy or decision and while they could subjectively state that they got high value from the intelligence product, they could not quantify it. To conclude this section, based on an extensive literature review and a multi-step methodology that involved extensive interviews with CTI users, a CTI impact evaluation instrument was developed. This instrument identified specific benefits of intelligences and then measures the extent of the benefit were received by asking the client to assess the extent to which the benefit was realized using a five point likert scale. As well, consistent with the consulting and management services literature client satisfaction and other service quality measures were put into the evaluation instrument which was also measured based on client perceptions. This instrument can be used after the CTI project has been done to assess the benefits to the client of the intelligence received. 5.0 Conclusions The intelligence literature notes the importance of proving value and impact of intelligence on the intended user of the intelligence. This study Opinion Section 88 sought to develop an instrument that could be used to measure this impact. Consistent with consulting and intelligence literature, it was found that client perceptions of benefit needs to be used as a primary method of evaluation. Client’s themselves indicated that it would be difficult to use non perception based methods of evaluation. 5.1 Study limitations and areas for future research The results of this study are based on intelligence as conducted in one organization and may not be generalizable to other organizations. In fact, as the unit is a technical intelligence unit, it is uncertain whether the evaluation instrument developed out of the study could be used in a non-technical intelligence organization. Generalizability is further restricted as the list of benefits were driven by the users of intelligence in this organization and perhaps other intelligence organizations have a different focus. While most of the benefits identified in this study are consistent with past research, nevertheless there appears to be organizational nuances to intelligence benefits that may need to be looked at in future studies. Further, even though the literature used in the development of the initial evaluation instrument was global, the evaluation instrument may not be generalizable outside of Canada or even outside this one organization. Accordingly, future studies should attempt to test the instrument developed here. Another area for future study is instrument reliability and validity testing. The instrument should be tested on a broader group with appropriate statistical tests of reliability and validity. Without factor analysis and Cronbach’s alpha it is not possible to state definitively that the instrument is both reliable and valid. While face validity has been established by use of client testing and fit with the existing literature, nevertheless statistical testing is required before the evaluation instrument should be considered acceptable for use. 5.2 Implications for CTI practitioners, policy and other stakeholders Notwithstanding the limitation noted above, the results of this study have significant implications for CTI practitioners as well as policy and other stakeholders. CTI can be assessed without having to wait for the final impacts of the CTI recommendations to arise. For all involved in CTI, it is clear from the results of this study that user perceptual measures should be used. Asking clients to assess on a Likert scale for example the extent to which the CTI provided saved them time in making the decision or helped them gain funds (research funds) is a good way to evaluate CTI impact. Not only is this consistent with the literature but based on the client interviews may be the only method they are prepared to accept. It is undeniable that the evaluation of CTI is a complex task owing to the complexity of both the CTI process and the ensuing client decision making/policy development process. Nevertheless, this study has demonstrated that evaluation can be done, albeit using perceptual measures. 6.0 Acknowledgements Funding for this research was provided by the National Research Council of Canada. The author acknowledges the research support provided by France Bouthillier, McGill University on this research project. Her research skills and insight helped in the development of the study and associated instruments. The author further thanks the reviewers for this paper. The comments provided served to improve and focus the paper. 7.0 References Anderson, Shannon, Lisa Klein Pearo, and Sally K Widener. (2008). Drivers Of Service Satisfaction Linking Customer Satisfaction To The Service Concept And Customer Characteristics. Journal Of Service Research 10, 4, 365-381. Blenkhorn, David, and Craig Fleisher. (2007). Performance Assessment In Competitive Intelligence: An Exploration, Synthesis, And Research Agenda. Journal Of Competitive Intelligence And Management 4, 2, 4-22. Davison, Leigh. (2001). Measuring Competitive Intelligence Effectiveness: Insights From The Advertising Industry. Competitive Intelligence Review 12, 4, 25--38. Dollatabady, Rezaie, Farzaneh Ghandehari, and Farham Amiri. (2011). Analyzing The Impact Of Competitive Intelligence On Innovation At Scientific Research Centers In Isfahan Opinion Section 89 Science And Technology Town. Interdisciplinary Journal Of Contemporary Research In Business 3, 5, 939-947. Fehringer, D., B. Hohhof, and T. Johnson. (2006) State Of The Art Competitive Intelligence. 1st ed. Virginia: Competitive Intelligence Foundation. Global Intelligence Alliance. (2014). Demonstrating the Impact of Market Intelligence. GIA White Paper, GIA. Global Intelligence Alliance. (2004). Measuring the Benefits of Competitive Intelligence. GIA White Paper Series. GIA. Herring, Jan P. (2007). How Much Is Your Competitive Intelligence Worth? Competitive Intelligence Magazine 10, 2: 23. Herring, Jan P. (1996). Measuring the Effectiveness Of Competitive Intelligence. 1st ed. Alexandria, VA: Society of Competitive Intelligence Professionals. Kahaner, Larry. (1997). Competitive Intelligence. 1st ed. New York: Simon & Schuster. Kilmetz, S, and R Bridge. (1999). Gauging the Return On Investment In Competitive Intelligence: A Three-Step Analysis For Executive Decision Makers. Competitive Intelligence Review 10, 1,4-11. Lönnqvist, Antti, and Virpi Pirttimäki. (2006). The Measurement Of Business Intelligence. Information Systems Management 23, 1, 32. McGonagle, John J, and Carolyn M Vella. (2002). Bottom Line Competitive Intelligence. 1st ed. Westport, Conn.: Quorum Books. Norling, Parry M, Jan P Herring, Wayne A Rosenkrans, Marcia Stellpflug, and Stephen B Kaufman. (2000). Putting Competitive Technology Intelligence to Work. Research- Technology Management 43, 5, 23-28. Patterson, Paul G, Lester W Johnson, and Richard A Spreng. (1996). Modeling The Determinants Of Customer Satisfaction For Business-To-Business Professional Services. Journal Of The Academy Of Marketing Science 25, 1, 4-17. Patterson, P. (2000). A contingency approach to modeling satisfaction with management consulting services. Journal of Service Research, 3, 2, 138-153. Pirttimäki, Virpi, Antti Lönnqvist, and Antti Karjaluoto. (2006). Measurement Of Business Intelligence In A Finnish Telecommunications Company. The Electronic Journal Of Knowledge Management 4, 1, 83-90. Rosenkrans, Wayne A. (1998). Past, Present, And Future Directions For Technical Intelligence. Competitive Intelligence Review 9, 2, 34-39. Simon, Neil. (1998). Determining Measures of Success. Competitive Intelligence Magazine 1, 2, 45-48. Viva Business Intelligence Inc. (2000). Measuring the Benefits Of Business Intelligence. Pro-How Papers. Wirtz, Jochen, and Meng Chung Lee. (2003). An Examination Of The Quality And Context- Specific Applicability Of Commonly Used Customer Satisfaction Measures. Journal Of Service Research 5, 4,345-355. Opinion Section 90 Appendix A Final CTI Questionnaire Please note the extent of the benefit using the following scale: 1 2 3 4 5 Strongly disagree strongly agree Item: Benefit to you Benefits Impact on Savings 1. It helped to save time 2. It helped to save money 3. It helped to save resources Impact on Gains 4. It helped me to gain more money 5. It helped me to gain more staff Impact on Decision Making/Recommendation 6. I made my recommendation more rapidly (timeliness) 7. I made a better recommendation (appropriateness) 8. My recommendation was validated (reassurance) 9. It helped to reduce bias(es) in decision making/recommendation 10. It helped to reduce the possibility of errors in my recommendation 11. It helped to pursue opportunities 12. It helped to develop partnerships/collaboration 13. It helped to develop better strategies Impact on Knowledge (cognitive dimension) 14. I became aware of important issues that I was not aware of before 15. I could go further in my thinking 16. It gave me information that I was able to use in future projects 17. It broadened my knowledge 18. It had given me the information required to improve my proposal/project 19. It had given me the information I needed to provide my client with good advice 20. It helped me to identify new markets 21. It helped me to identify new ideas Impact on Perception (affective dimension) 22. It made me more confident on my recommendation 23. It helped to reduce perceived uncertainty 24. It has enabled me to do my job better 25. It helped to reduce risk 26. I could act differently Impact on Service towards Clients 27. It has helped improve service to my clients Appreciation of service quality 1. The reports were easy to read/ consult 2. Staff showed good knowledge of my area/industry 3. Staff understood my problem/issue 4. Staff was flexible in adapting themselves to my requests 5. Staff paid attention to my needs 6. CTI reports were reliable 7. CTI reports were accurate 8. I felt that my needs were dealt with in a timely manner 9. I will recommend the unit to others