Vol11No1Paper6 To cite this article: Rodriguez-Salvador, M. & Castillo-Valdez, P.F. (2021) Integrating science and technology metrics into a competitive technology intelligence methodology. Journal of Intelligence Studies in Business. 11 (1) 69-77. Issue URL: https://ojs.hh.se/index.php/JISIB/article/view/JISIB Vol 11 Nr 1 2021 This article is Open Access, in compliance with Strategy 2 of the 2002 Budapest Open Access Initiative, which states: Scholars need the means to launch a new generation of journals committed to open access, and to help existing journals that elect to make the transition to open access. Because journal articles should be disseminated as widely as possible, these new journals will no longer invoke copyright to restrict access to and use of the material they publish. Instead they will use copyright and other tools to ensure permanent open access to all the articles they publish. Because price is a barrier to access, these new journals will not charge subscription or access fees, and will turn to other methods for covering their expenses. There are many alternative sources of funds for this purpose, including the foundations and governments that fund research, the universities and laboratories that employ researchers, endowments set up by discipline or institution, friends of the cause of open access, profits from the sale of add-ons to the basic texts, funds freed up by the demise or cancellation of journals charging traditional subscription or access fees, or even contributions from the researchers themselves. There is no need to favor one of these solutions over the others for all disciplines or nations, and no need to stop looking for other, creative alternatives. Journal of Intelligence Studies in Business Publication details, including instructions for authors and subscription information: https://ojs.hh.se/index.php/JISIB/index Integrating science and technology metrics into a competitive technology intelligence methodology Marisela Rodriguez-Salvadora,* and Pedro F. Castillo- Valdeza aTecnologico de Monterrey, Av. Eugenio Garza Sada 2501, Col. Tecnológico, Monterrey, N.L. C.P. 64849, México; *marisrod@tec.mx Journal of Intelligence Studies in Business PLEASE SCROLL DOWN FOR ARTICLE Integrating science and technology metrics into a competitive technology intelligence methodology Marisela Rodriguez-Salvadora,* and Pedro F. Castillo-Valdeza aTecnologico de Monterrey, Av. Eugenio Garza Sada 2501, Col. Tecnológico, Monterrey, N.L. C.P. 64849, México *Corresponding author: marisrod@tec.mx Received 27 October 2020 Accepted 29 March 2021 ABSTRACT For years, the appropriate interpretation and application of metrics have enabled scientists to assess science and technology dynamics. Consequently, diverse disciplines have emerged, such as bibliometrics, scientometrics and patentometrics, offering important theoretical and methodological contributions. However, the current accelerated technological advances require researchers to implement a superior approach to detect continuous changes in the external environment identifying opportunities and vulnerabilities to strengthen the decision-making process regarding R&D and innovation. In this context, competitive technology intelligence (CTI) offers a strategic approach based on a continuous cycle where information is transformed into an actionable result. This research provides a broader scope to science and technology metrics, incorporating them into a CTI global methodology of eight steps. Metrics add value throughout the entire CTI process, from project planning to decision-making stages, having the most significant role in the information analysis stage, mainly to process information from sources such as scientific documents, patents, and social networks. Particularly, this approach considers recent studies in CTI in which quantitative tools such as patentometrics and scientometrics were successfully used. This proposal can be applied to predict upcoming technologies, movements of competitors, disrupting activities, market changes, and future trends. Accordingly, this research adds value to the assessment of science and technology dynamics, aiming to improve the decision-making process of R&D and innovation. KEYWORDS Competitive intelligence, competitive technology intelligence, patentometrics, science and technology metrics, scientometrics 1. INTRODUCTION In a global world, technological advances embody new opportunities to be assessed by organizations. Good decision-making requires correctly identifying the technologies to implement, the products to develop and the standards to use. The new forms of collecting, storing, transforming, and processing data have made the information more accessible than ever. Information about the external environment enables leaders to manage their businesses more effectively; however, information is not enough. An adequate assessment of science and technology is fundamental to impact present and future R&D and innovation decisions; therefore, building metrics is essential to obtain accurate results. To facilitate science and technology understanding, diverse disciplines based on metrics analysis have emerged, including scientometrics, patentometrics, and altmetrics. They offer fundamental theoretical and methodological Journal of Intelligence Studies in Business Vol. 11, No. 1 (2021) pp. 69-77 Open Access: Freely available at: https://ojs.hh.se/ 70 contributions involving the use of traditional and non-traditional metrics. Scientometrics is a discipline based on mathematical methods to quantify scientific research literature to reveal the process of scientific development (Qiu et al. 2017). It enables researchers to identify the actors and processes involved in scientific activities, such as authors, research groups, institutions, countries, and their scientific production, to determine the structure, relationships, and research dynamics (Michán L. 2013). Scientometrics deals with scientific information analysis mainly from scientific documents; on the other hand, patentometrics focuses on the analysis of patents. Patents protect inventions developed by companies, institutions, or individuals, and can be interpreted as indicators of invention, and it is possible to create scientific and technological scenarios of countries, industries, and research institutes by analyzing them. Unlike scientific literature, patents have a legal framework that supports them, and the information they contain has a uniform structure, allowing the easy extraction of the information desired. Economic indicators have also been associated with patents, addressing connections between technology and trade. While traditional metrics focus on the data mining of scientific and technological outputs (e.g., scientific papers, patents), non- traditional metrics–also known as alternative metrics or altmetrics (Priem et al. 2012)–are oriented to measure scholarly activities on social networks, blogs, newspaper articles, and web sites, among others (European Union 2017). The consolidation between traditional and non-traditional metrics offers a complementary view to evaluate the dynamism of science and technology, which requires more than one single metric approach (Staudt et al. 2018). Although metrics analyze different aspects of science and technology, they are not enough to continuously monitor the external environment to support strategic R&D and innovation decision-making. By systematically analyzing the external environment, organizations can increase their advantages (Luu, 2015), identify movements of competitors (Rothberg and Erickson 2017), and detect opportunities for growth (Zeid 2014), which represents a crucial factor to survive under the current and future industry global competition (Shaitura et al. 2018). In this context, CTI adds fundamental value since it involves a continuous process based on collecting and analyzing external information transformed into a strategic result (Dou et al. 2019) to anticipate changes in the market and to identify relevant opportunities, supporting the decision-making process for innovation (Du Toit 2015, Rodriguez-Salvador and Lopez- Martinez 2000). Based on studies that apply CTI for science and technology assessment, a global methodology that incorporates metrics into a CTI process is proposed. First, a description of the general context of CTI is presented, then recent CTI studies are identified, and finally, a CTI methodology is proposed revealing the incorporation of metrics. 2. COMPETITIVE TECHNOLOGY INTELLIGENCE R&D fosters knowledge aiming to provide answers to questions from different fields. For this activity to be enhanced, it is necessary to transform information into intelligence and provide conditions to facilitate a continuous flow of knowledge (Amidon Rogers, 1996). Therefore, it is crucial to analyze external data in a timely and proper way to develop business insights and become more competitive (Rodriguez-Salvador 2006, Zeid 2014). Competitive intelligence emerges as an approach to support strategic decision-making. While Fitzpatrick and Burke (2003) define competitive intelligence as a process to collect and analyze external information to increase the advantages and position of an organization, Dou et al. (2019) define it as the ability of an organization to understand its environment effectively and drive strategies accordingly. This ability is sustained by the process of collection, analysis, and dissemination of actionable knowledge. When competitive intelligence is applied to science and technology research, the term CTI arises. CTI pursues timely awareness of scientific and technological events to stay ahead, identifying collaboration prospects, technology knowledge landscapes, and in general, valuable findings to improve R&D and innovation processes (Rodriguez-Salvador et al. 2002). Achieving this involves a methodology which in general starts with a planning stage aligned to the main objectives of the study, continuing with the identification of information sources, aiming to identify the best alternatives to collect the most relevant information on science and technology from different sources, such as experts in the field, 71 documents, patents, and social networks. The next step consists of information collection, where the main goal is not to get the most significant amount of information but to get the most meaningful information possible. This information is processed and prepared for further analysis, where different techniques can be applied (e.g., scientometrics, patentometrics, road-mapping). Dissemination and decision-making comprised the last stages. 3. CTI METHODOLOGY To understand the impacts of metrics on scientific and technological research, an in- depth analysis of studies that examine metrics and apply CTI for science and technology assessment was conducted. Wilsdon et al. (2015) explored the effects of the growing use of metrics to evaluate research, proposing their responsible application and establishing that an appropriate research assessment should include elements such as robustness, transparency, reflexivity, humility, and diversity. Staudt et al. (2018) proposed a set of text- and citation-based metrics to identify high-impact and transformative works. These metrics are categorized into seven types: radical-generative, radical-destructive, risky, multidisciplinary, wide impact, growing impact, and impact (overall). Based on an analysis of the accuracy of 39 metrics, Bollen et al. (2009) showed that a multi-dimensional view is required to measure the impact of scientific research effectively. Finally, Cronin and Sugimoto (2014) emphasized that the web leads to new tools to assess scholar productivity, revealing behaviors and impact that were previously invisible, such as number of mentions, acknowledgments, endorsements, number of downloads, recommendations, blog posts, tweets, and a variety of other metrics that can be utilized. Table 1 Recent studies using quantitative tools under an intelligence approach. Title Year Authors Journal Analysis of the knowledge landscape of 3D bioprinting in Latin America 2019 Marisela Rodriguez-Salvador, Diego Villarreal-Garza, Mario Moisés Alvarez, Grissel Trujillo- de Santiago International Journal of Bioprinting Data analytics for better informed technology & engineering management. 2019 Alan L. Porter IEEE Engineering Management Review Discovering new 3D bioprinting applications: Analyzing the case of optical tissue phantoms. 2019 Luis Hernandez-Quintanar, Marisela Rodriguez-Salvador International Journal of Bioprinting Additive manufacturing in healthcare. 2018 Marisela Rodriguez-Salvador, Leonardo A. Garcia-Garcia Foresight and STI Governance Additive manufacturing knowledge incursion on orthopaedic devices: The case of hand orthoses 2018 Leonardo A. Garcia-Garcia, Marisela Rodriguez-Salvador Proceedings of the 3rd International Conference on Progress in Additive Manufacturing (Pro-AM 2018), Singapore An assessment of technology forecasting: Revisiting earlier analyses on dye- sensitized solar cells (DSSCs). 2018 Ying Huang, Alan L. Porter, Yi Zhang, Xiangpeng Lian, Ying Guo Technological Forecasting and Social Change Competition-driven figures of merit in technology roadmap planning. 2018 Ksenia Smirnova, Alessandro Golkar, Rob Vingerhoeds 2018 IEEE International Systems Engineering Symposium (ISSE) Revealing emerging science and technology research for dentistry applications of 3D bioprinting. 2018 Marisela Rodriguez-Salvador, Laura Ruiz-Cantu International Journal of Bioprinting Uncovering 3D bioprinting research trends: A keyword network mapping analysis 2018 Leonardo A. Garcia-Garcia, Marisela Rodriguez-Salvador International Journal of Bioprinting Scientometric and patentometric analyses to determine the knowledge landscape in innovative technologies: The case of 3D bioprinting. 2017 Marisela Rodriguez-Salvador, Rosa Maria Rio-Belver, Gaizka Garechana-Anacabe PLoS ONE Technology roadmapping for competitive technical intelligence 2016 Yi Zhang, Douglas K.R. Robinson, Alan L. Porter, Donghua Zhu, Guangquan Zhang, Jie Lu Technological Forecasting and Social Change Topic analysis and forecasting for science, technology and innovation: Methodology and a case study focusing on big data research. 2016 Yi Zhang, Guangquan Zhang, Hongshu Chen, Alan Porter, Donghua Zhu, Jie Lu Technological Forecasting and Social Change. Despite the profuse research on metrics, few studies have explored metrics under a strategic view. Recent investigations incorporate scientometrics, patentometrics, and other quantitative tools into a CTI approach using data mining to make science and technology assessment more comprehensible, but they mainly address the analysis of secondary information, such as scientific documents and patents. As an example, Table 1 shows some recent efforts ranging from the analysis of specific fields to those focused on analytic tools. In the first group are new 3D bioprinting applications: optical tissue phantoms, emerging dentistry applications of 3D bioprinting, additive manufacturing for hand orthoses, technology forecasting on dye- sensitized solar cells (DSSCs), competition- driven figures of merit (automotive industry), 3D bioprinting in Latin America, keyword network mapping analysis on 3D bioprinting research, knowledge landscape of 3D bioprinting and forecasting on big data research. In these studies, scientific and technological trends were identified by analyzing documents and patents, including data mining of some of their elements, such as titles, keywords, authors, and affiliations. This revealed the dynamics of intellectual outputs in terms of metrics such as number, evolution, and impact of publications and/or patents by authors, journals, institutions, countries, and areas. On the other hand, Table 1 also exhibits tools to facilitate analysis tasks, particularly data analytics for technology and engineering management and technology road-mapping for competitive technical intelligence where metrics combine qualitative and quantitative data and external actors are participating. Approaches from Porter (2019), Rodriguez- Salvador and Ruiz-Cantu (2019), Hernandez- Quintanar and Rodriguez-Salvador (2019), Huang et al. (2018), Garcia-Garcia and Rodriguez-Salvador (2018a), Rodriguez- Salvador et al. (2017), and Zhang et al. (2016) were considered to develop a new CTI methodology where, unlike current studies, 1) primary and secondary information are considered, 2) quantitative and qualitative metrics are applied and 3) experts participate. This CTI eight-step methodology comprises interdependent phases, receiving continuous feedback. 1. Project planning. The main activities and scope of the CTI project are established, as well as participants, roles, resources, and internal policies. Metrics, in this stage, can be established according to specific key performance indicators depending on the objectives to accomplish. 2. Identification of data sources. This represents the input for further analysis. In this stage, metrics can facilitate the selection of the best information sources. There are two basic types of data sources: primary and secondary. The former is based on the insight of experts, where metrics contribute to identifying experts for feedback purposes, determining their presence in a field in terms of indicators such as the number of paper citations, the number of patents, the ranking of publications, areas of specialization, and network collaborations through affiliation analysis. Commonly, secondary sources include scientific and technical documents, as shown in the studies in Table 1; however, this approach also incorporates strategic information such as industry and market reports. Additionally, social networks, which are rapidly evolving, are considered. In this case, metrics can help identify the quality of sources in terms of features such as their completeness, impact, and prestige. 3. Search strategy design. Establishing a plan and a strategy to retrieve information is essential. This activity should be aligned to the study focus as well as to the characteristics of the specific data sources previously identified. Tools like Delphi studies, focus-groups, and interviews may be considered, each one with specific metrics. For secondary information, particularly from the internet and databases, it is essential to identify the most relevant terms to feed searching queries. For this aim, an in-depth literature review should be executed, not only from scientific publications, technical reports, and patents but also from industry and market reports, among others. Moreover, different queries should be designed to maximize the efficiency of further data collection. 4. Data collection. This focuses on the previous primary and secondary sources identified. Database management systems would be required to access and manipulate large sets of information. This stage also 73 includes normalization and preparation of the information to be processed and analyzed in the following step. 5. Information analysis. While traditional studies are typically focused on only on solving the questions “what?” and “how?” and on the analysis of scientific and technological documents, this stage aims to answer fundamental questions such as “what?” (to develop, incorporate, cancel, allocate), “how?” (human and material resources), “when?”, “where?”, “why?”, “with whom?” and involves industry and market reports in combination with expert views. Scientific literature can be evaluated using metrics such as the number of publications, the growth rate of publications, impact factor of journals, the number of citations, author affiliation, collaboration networks, countries- and institutions-predominance, and areas of specialization by journal, author, institution or country. On the other hand, patents can be analyzed based on metrics such as patent production, patent categorization according to the International Patent Classification (IPC), IPC distribution of inventors, assignees and/or countries, the number of patent families (PFs), patent distribution of assignees and patent legal status (assigned, granted or inactive). To analyze social networks and websites alternative metrics can be incorporated, considering statistics comprising the number of mentions, number of downloads of the documents, and social network interactions. For scientific papers and patents, specialized software, data mining, and algorithms for text mining can be used to apply co-occurrence analysis, and keyword or term clustering to determine behaviors by the output of authors, countries, institutions, journals, and areas. Industry and market reports can be analyzed in terms of well-known metrics such as market share, competitiveness level, consumer behavior, and distribution rate. 6. Feedback from experts. Recommendations from experts constitute a great asset. Compared with other studies where expert participation is scarce or does not exist, this research suggests their participation across the entire CTI process. In this case, metrics can be established to get an expert evaluation of results obtained, particularly for the analysis stage. Interviews and questionnaires throughout the entire CTI process are suggested. Experts can also participate in Delphi studies and focus groups. 7. Validation and delivery of final results. It is crucial to zoom in to examine the accuracy of results obtained; validation should be developed from the initial to the final stages of the methodology. This stage represents the last verification to make final adjustments as needed. Results are then consolidated and prepared for their delivery through a report that can be communicated to the decision-maker and other stakeholders. This report should be aligned to the objectives previously established, adding value to the decision-making process. It is suggested that the content displays quantitative results; for example, figures that show the evolution of the industry, market, technology, product or process, emerging areas, readiness level, its performance, the predominance of areas, and position by authors, countries, companies, collaboration networks, market share, economic feasibility and distribution. Additionally, it is suggested to include qualitative results such as scenarios for industry, market, technology, product, or process. Furthermore, metrics can help to systematically monitor research activities at international, national, regional, and industry levels. It is also fundamental that this report considers user preferences in terms of several aspects such as presentation style, content, and delivery time. 8. Decision making. This step represents the execution of results obtained, where decisions are taken by the people in charge of R&D and innovation. The result obtained previously is transformed into a specific action, and it is an input to decide what should be monitored constantly, which is also a differentiator of other approaches. It is imperative to stimulate debate and discussion, looking for competitive advantages. 4. DISCUSSION The methodology proposed in this paper promotes the integration of metrics through a CTI eight-step process. Like those of Table 1, current studies are mainly focused on a statistical analysis of secondary information, 74 principally through data mining of scientific and patent information applying techniques such as scientometrics and patentometrics. A gap still exists in the application of metrics under a strategic perspective, such as that of CTI, where primary and secondary information can be combined, and quantitative and qualitative metrics are integrated with the insight of experts. Such insight is required to support and validate metrics to ensure the production of reliable outcomes. Figure 1 illustrates how metrics could be integrated into the CTI methodology previously presented, where the principal contribution is for four steps. The second step is to identify the best information sources. The fifth step is where metrics acquire the most critical role of the CTI process, being possible to reveal strategic elements for R&D and innovation as early warnings, technology- and market-lifecycles, potential collaborators, technology impact and so on. The sixth step is where the insight of experts can help to refine results and broaden the scope of the study. And the seventh step involves validation and the elaboration of an executive report, integrating the most relevant insights. Through this CTI global methodology, we aim to fill the gap regarding the current use of metrics. This can be an essential guideline for assessing science and technology evolution supporting strategic planning for R&D and innovation initiatives in technology-based organizations. Leaders should set new directions toward strategic changes to achieve competitive advantages (Verlander 2012). This effort may represent an important alternative to give a better overview of science and technology, giving the possibility of detecting opportunities and threats on time to gain competitive advantages through R&D and innovation. 5. CONCLUSIONS In a globalized world where competition is becoming more complex, professionals should adapt, change, and develop competitive advantages by assessing the evolution of science and technology research. Metrics analysis represents a strong contribution for the research activities to be more Figure 1 Competitive technology intelligence (CTI) eight-step methodology incorporating metrics. 75 understandable. However, they do not offer a complete solution to anticipate and detect continuous changes in the external environment as CTI does. While current research focuses mainly on analyzing secondary information through quantitative metrics, this proposal goes further by considering 1) primary and secondary information, 2) quantitative and qualitative metrics, and 3) insight of experts into a CTI eight-step methodology, with an emphasis on the stages of identification of data sources, information analysis, feedback from experts, and validation and delivery of final results. The proposed methodology can be applied to disclose the dynamics of an industry, market, and a scientific and technological field, predicting new technologies, movements of competitors, disrupting activities, market changes, and future trends. Moreover, this approach can enable the early detection of scientific and technological opportunities or threats by monitoring the competitive environment continuously and supporting the strategic planning of R&D and innovation. Finally, it is relevant to state that the implementation of this global approach requires identifying specific metrics to incorporate according to the objectives to pursue, the industry, and the project context. Furthermore, novel metrics and automation processes to interpret information are continuously emerging. Consequently, it is crucial to keep abreast and include them in future research. ACKNOWLEDGMENTS The authors acknowledge the institutional funding received from Tecnologico de Monterrey, and the National Council of Science and Technology (CONACYT) through a doctoral scholarship. CONFLICT OF INTEREST The authors declare that they do not have any conflicts of interest. 6. REFERENCES Amidon Rogers, Debra M. (1996). The challenge of fifth generation R&D. Research Technology Management, 39(4), 33-41. doi: 10.1080/08956308.1996.11671075. Bollen, J., Van de Sompel, H., Hagberg, A., Chute, R., 2009. A Principal Component Analysis of 39 Scientific Impact Measures. PLoS ONE, 4(6): e6022. doi:10.1371/journal.pone.0006022 Cronin, B., Sugimoto, C.R. (2014). Beyond bibliometrics: Harnessing multidimensional indicators of scholarly impact. Cambridge, United States: The MIT Press. Dou, H., Juillet, A., Clerc, P. (2019). Strategic Intelligence for the Future 1: A New Strategic and Operational Approach. Hoboken, United States: Wiley. Du Toit, A.B. (2015). Competitive intelligence research: an investigation of trends in the literature. Journal of Intelligence Studies in Business. 5(2). doi: 10.37380/jisib.v5i2.127 European Union (2017). Next-generation metrics: Responsible metrics and evaluation for open science. Luxembourg: Publications Office of the European Union. doi: 10.2777/337729. Fitzpatrick, W., Burke, D. (2003). Competitive Intelligence, corporate security and the virtual organization. Advances in Competitiveness Research, 11(1), 20-45. Garcia-Garcia, L. A., Rodriguez-Salvador, M. (2018a). Uncovering 3D bioprinting research trends: A keyword network mapping analysis. International Journal of Bioprinting, 4(2), 147-154. doi: 10.18063/IJB.v4i2.147 Garcia-Garcia, L. A., Rodriguez-Salvador, M. (2018b). Additive manufacturing knowledge incursion on orthopedic devices: The case of hand orthoses. Proceedings of the 3rd International Conference on Progress in Additive Manufacturing (Pro-AM 2018), 571- 576. doi: 10.25341/D4388H Hernandez-Quintanar, L., Rodriguez-Salvador, M. (2019). Discovering New 3D Bioprinting Applications: Analyzing the Case of Optical Tissue Phantoms. International Journal of Bioprinting, 5(1), 178-189. doi: 10.18063/ijb.v5i1.178. Huang, Y., Porter, A. L., Zhang, Y., Lian, X., Guo, Y. (2018). An assessment of technology forecasting: Revisiting earlier analyses on dye-sensitized solar cells (DSSCs). Technological Forecasting and Social Change, 146, 831-843. doi: 10.1016/j.techfore.2018.10.031. Luu, T. T. (2015). From cultural intelligence to supply chain performance. The International Journal of Logistics Management, 27(1). doi: 10.1108/IJLM-01-2014-0009 76 Michán, L., Muñoz-Velasco, I. (2013). Cienciometría para ciencias médicas: definiciones, aplicaciones y perspectivas. Investigación En Educación Médica, 2(6), 100- 106. doi:10.1016/s2007-5057(13)72694-2 Niven, P. R. (2006). Balanced Scorecard Step-by- step: Maximizing Performance and Maintaining Results. Hoboken, United States: Wiley. Porter, A. L. (2019). Data Analytics for Better Informed Technology & Engineering Management. IEEE Engineering Management Review, 47(3), 29-32. doi: 10.1109/EMR.2019.2928265. Priem, J., Piwowar, H., Hemminger, B. (2012). Altmetrics in the wild: Using social media to explore scholarly impact. arXiv:1203.4745 Qiu, J., Zhao, R., Yang, S., Dong, K. (2017). Informetrics: Theory, Methods and Applications. Singapore: Springer. Rodriguez-Salvador, M. (2006). Innovación y creatividad. En Ingeniería concurrente: Una metodología integradora, 137-145. Barcelona, Spain: Universitat Politècnica de Catalunya, Edicions UPC. Rodriguez-Salvador, M., Eddy-Valdez A., Garza- Cavazos, R. (2002). Industry/university cooperative research in competitive technical intelligence: a case of identifying technological trends for a Mexican steel manufacturer. Research Evaluation, 11(3), 165–173. doi: 10.3152/147154402781776835 Rodriguez-Salvador, M., Garcia-Garcia, L. (2018). Additive Manufacturing in Healthcare. Foresight and STI Governance, 12(1), 47-55. doi: 10.17323/2500-2597.2018.1.47.55 Rodriguez-Salvador, M., Lopez-Martinez, R. E. (2000). Cognitive Structure of Research: Scientometric Mapping in Sintered Materials. Research Evaluation, 9(3), 189-200. doi: 10.3152/147154400781777214 Rodriguez-Salvador, M., Rio-Belver, R. M., Garechana-Anacabe, G. (2017). Scientometric and patentometric analyses to determine the knowledge landscape in innovative technologies: The case of 3D bioprinting. PLoS ONE, 12(6): e0180375. doi: 10.1371/journal.pone.0180375 Rodriguez-Salvador, M., Ruiz-Cantu, L. (2019). Revealing Emerging Science and Technology Research for Dentistry Applications of 3D Bioprinting. International Journal of Bioprinting, 5(1), 170-176. doi: 10.18063/ijb.v5i1.170 Rodriguez-Salvador, M., Villarreal-Garza, D., Alvarez, M., Trujillo-de Santiago, G. (2019). Analysis of the Knowledge Landscape of 3D Bioprinting in Latin America. International Journal of Bioprinting, 5(2.2), 16-25. doi:10.18063/ijb.v5i2.3.240 Rothberg, H. N., & Erickson, G. S. (2017). Big data systems: knowledge transfer or intelligence insights? Journal of Knowledge Management, 21(1), 92–112. doi:10.1108/jkm- 07-2015-0300 Shaitura, S.V., Ordov, K.V., Lesnichaya, I.G., Romanova, Y.D., Khachaturova, S.S. (2018). Services and mechanisms of competitive intelligence on the Internet. Revista ESPACIOS. Vol. 39 (No 45). Smirnova, K., Golkar, A., Vingerhoeds, R. (2018). Competition-driven figures of merit in technology roadmap planning. 2018 IEEE International Systems Engineering Symposium (ISSE), 1-6. doi: 10.1109/SysEng.2018.8544407. Staudt, J., Yu, H., Light, R.P., Marschke, G., Börner, K., Weinberg, B. A. (2018) High- impact and transformative science (HITS) metrics: Definition, exemplification, and comparison. PLoS ONE, 13(7): e0200597. doi: 10.1371/journal.pone.0200597. Verlander, E. G. (2012). The Practice of Professional Consulting. San Francisco, United States: Wiley. Wilsdon, J., Allen, L., Belfiore, E., Campbell, P., Curry, S., Hill, S., Jones, R.A.L., Kain, R., Kerridge, S., Thelwall, M., Tinkler, J., Viney, I., Wouters, P., Hill, J., Johnson, B. (2015). The metric tide: report of the independent review of the role of metrics in research assessment and management. London, United Kingdom: Higher Education Funding Council for England. doi: 10.13140/RG.2.1.4929.1363. Zhang, Y., Robinson, D.K.R., Porter, A.L., Zhu, D., Zhang, G., Lu, J. (2016). Technology roadmapping for competitive technical intelligence. Technological Forecasting and Social Change, 110, 175-186. doi: 10.1016/j.techfore.2015.11.029 Zhang, Y., Zhang, G., Chen, H., Porter, A. L., Zhu, D., Lu, J. (2016). Topic Analysis and Forecasting for Science, Technology and Innovation: Methodology and a Case Study 77 focusing on Big Data Research. Technological Forecasting and Social Change. 105, 179-191. doi: 10.1016/j.techfore.2016.01.015. Zeid, A. (2014). Business Transformation: A Roadmap for Maximizing Organizational Insights. Cary, United States: Wiley.