295 American Scientific Research Journal for Engineering, Technology, and Sciences (ASRJETS) ISSN (Print) 2313-4410, ISSN (Online) 2313-4402 © Global Society of Scientific Research and Researchers http://asrjetsjournal.org/ Considerations on the Impact Factor as a Tool in Scientific Assessment Katia Denise Saraiva BrescianiI a *, José Antonio Soares b , Jorge Ramos Do Ó c , Tomás Patrocinio d , Lucas Vinicius Shigaki de Matos e , Maria Luisa Machado Cerdeira f a UNESP, Universidade Estadual Paulista Julio de Mesquita Filho, Departamento de Apoio, Produção e Saúde Animal, Faculdade de Medicina Veterinária. Rua Clóvis Pestana, n. 793, 16050-680, Araçatuba, São Paulo, Brasil. b FKB, Fundação Karnig Bazarian, Faculdades Integradas de Itapetininga, Rodovia Raposo Tavares, km 162, Itapetininga, São Paulo, Brasil.. c UL, Universidade de Lisboa, Instituto de Educação, Alameda da Universidade, 1649-013, Lisboa, Portugal. d UNESP, Universidade Estadual Paulista Julio de Mesquita Filho, Faculdade de Ciências Agrárias e Veterinárias, Jaboticabal. Via de Acesso Professor Paulo Donato Castellane, KM 35 - Vila Industrial, Jaboticabal, São Paulo, 14884-900. a Email: bresciani@fmva.unesp.br b Email: prof.soares@uol.com.br c Email: luisa.cerdeira@ie.ulisboa.pt d Email: jorge.o@ie.ul.pt e Email: patrocinio.tomas@gmail.com f Email: lucasvsm@hotmail.com Abstract This exploratory study aims to wide the debate over the impact factor and its application as a tool in the assessment of science. The analysis of scientific activity has been the subject of much debate, especially in the last decade. The impact factor and the citation frequency have been regularly used in the individual analysis of researchers as well verify the performance of journals, investigators, research centers, graduate programs, institutions and countries. The impact factor of a journal can be highly influenced by few of its articles that are very frequently cited. Some researchers are more concerned about publishing in high impact journals than about doing science itself. ------------------------------------------------------------------------ * Corresponding author. http://asrjetsjournal.org/ mailto:bresciani@fmva.unesp.br mailto:prof.soares@uol.com.br mailto:luisa.cerdeira@ie.ulisboa.pt mailto:jorge.o@ie.ul.pt mailto:patrocinio.tomas@gmail.com mailto:lucasvsm@hotmail.com American Scientific Research Journal for Engineering, Technology, and Sciences (ASRJETS) (2015) Volume 14, No 2, pp 295-303 296 The manner how the scientific research output is assessed by funding agencies, academic institutions, among others, has been the subject of much debate and the current great question is whether the impact factor (IF) is the best indicator to assess the scientific quality or an already obsolete measure. There is one movement of researchers against the impact factor to be used as a qualitative measure of research articles or even as an indicator of the contribution of a scientist for possible hiring, promotion or decision for grant. Then, there is a need of academic community deeply reflect on this important issue. Key words: impact factor; citation database; scientific quality; research outcome evaluation. 1. Introduction The terms “bibliometrics”, “informetrics” and “scientometrics” are derived from the fusion between the suffix “metrics” and bibliography, information and science, respectively [1]. Informetrics is characterized by the study of quantitative aspects of information in any format, regardless of the social group, with incorporation, use and extension of its assessment going further than the limits of bibliometrics and scientometrics [2, 3, 4]. Authors in reference [5] established that this science works with modalities of information and communication production in nonacademic communities, while Authors in reference [6] defines it as an emerging subfield of information science, based on the combination between advanced information retrieval techniques and quantitative studies of the information flow [7]. Bibliometrics is a field of information science which measures, by means of statistical analyses, the scientific and technological investigation outputs [8] in the form of different scientific publications such as papers, books and edited journals [9, 10]. Scientometrics can be defined as the study of the quantitative aspects of science as a subject or economic activity [3], in which analytical numeric techniques are applied to study the science of science [9], or even as the deepening of quantifiable aspects of the scientific knowledge activity like the generation, propagation and use of such information [11]. More recently, the term “webometrics” was added; it consists in the study of web resources and “patentometrics” which, by analyzing patents, measures the technology and innovation degree of a country or industry sector in association with the scientific knowledge [11]. Webometrics relates the major web elements (domains, sites and pages) to their links and matrix of resulting links that involve the whole net extension [12], focusing on quantitative features and aspects of the use of information resources, structures and technologies in the web [13]. Investigators have been interested in the development of the Web impact factor [11]. The bibliometric indicators of scientific production can be subdivided into two groups: quantitative indicators, which correspond to the number of published papers, and those of impact, which are based on the number of citations of published papers. Analysis of registered patents constitutes the main indicator for the technological area. They can be used in scientific politics programs, as well as in studies to assess the latter, measuring the scientific "power" and "prestige" of countries, regions and, in particular, universities or research centers [9,14]. American Scientific Research Journal for Engineering, Technology, and Sciences (ASRJETS) (2015) Volume 14, No 2, pp 295-303 297 In the academic environment, citations can be considered indicators of the researcher's performance and are commonly used to assess the research impact on the scientific community. Once the paper is not cited, it is assumed to not arouse interest in the above-mentioned community [15]. Thus, not being cited shows that the author is not participating in the debate over his/her own area [16]. The journals of greatest impact in each area are generally those that follow the best criteria of quality, prestige and international diffusion; they are extremely rigorous in the analysis of manuscripts received for publication [9]. For the assessment of a researcher's academic excellence, there has been certain convergence among experts in considering the output quality [4]. The analysis of scientific activity has been the subject of much debate, especially in the last decade [17,18]. Indicators such as the impact factor and the citation frequency have been regularly used to verify the performance of journals, investigators, institutions and countries [11]. The current great question is whether the impact factor (IF) is the best indicator to assess the scientific quality or an already obsolete measure [19]. The present study consists in an exploratory study aimed at widening the debate over the impact factor and its application as a tool in the assessment of science. 2. Literature Review The impact factor (IF) was idealized in the classic article by authors in reference [20], with the aim of helping scientists search for bibliographic references for their own investigation and for the recognition of their work. Subsequently, authors in reference [21] suggested that the counting of citations reflects the importance of a journal, even though some periodicals are widely used but rarely cited. That author stated that the IF could be rather useful for librarians in the management of subscriptions and periodical collections and for editors to discover some interesting parameters of their own journal; similarly, the interest of research funding bodies (funding agencies) ends up restimulating the need of authors, librarians and editors to use the IF [4]. Therefore, this indicator has become the most important measure in the assessment of science. The IF is calculated by a formula in which the sum of citations in a certain year is divided by the number of academic articles published in the two previous years [20- 25]. The IF of periodicals selected by Thomson Reuters is published every year in the Journal Citation Reports (JCR) and the citations of each article are provided at the platform Web of Science (WoS). In addition to the IF, other two indicators are annually published in the JCR, according to the periodical's title: the immediacy index and the cited Half-Life. The former is determined by the citation speed of the papers of a journal or periodical. The lower this index, the higher the journal value, since its content is rapidly used by other researchers who transform it into new knowledge. The latter indicator demonstrates the obsolescence degree of a periodical, i.e., the time needed for the papers in this journal to reach 50% citations, which can be related to the validity of such information [26, 9, 6]. Authors in reference [23, 27] criticized the indiscriminate use of the IF and reinforced that the latter was created to compare journals, not authors. American Scientific Research Journal for Engineering, Technology, and Sciences (ASRJETS) (2015) Volume 14, No 2, pp 295-303 298 On the other hand, he stated that there is no better measure of the quality of papers. That author also contended that journals with higher IF are more prestigious, receive a larger number of manuscripts to be considered for publication, but only accept the best ones. Previous studies have evidenced association between the IF and the initial date of publication of periodicals [28], their circulation [29], the number of authors [30], the density of papers [4], and the literature obsolescence rhythm. Systematic comparison and monitoring of the evolution of the scientific production of institutions, research centers or even countries have indicated that the number of published articles may become a very useful and important indicator. The number of citations received by a paper is an indicator of the influence or impact that its content had on the scientific community of that field [9]. The quality and the impact of a paper do not have the same meaning. The former is linked to the scientific content, to the methodology appropriateness, to the clarity of writing and to the originality of the design and conclusions, whereas the second term is related to the influence of the paper on correlated studies at a certain time. Thus, the impact of a study on a certain area consists in an indirect indicator of the study's quality [22, 9]. In light of this discussion, authors in reference [9] cited four limitations to the use of indicators of the impact. The first one is the inexistence of an appropriate model to standardize the citation process; the second one is the fact that citations are from the data bank of Science Citation Index provided by ISI; the third one is related to the use of literature reviews or papers that introduce techniques or methods, which leads to a large number of citations; the fourth limitation is associated with the fact that comparisons among areas or subjects, based exclusively on the number of citations or on the impact factor, are not totally appropriate. There is a strong positive correlation between the number of self-citations and the number of authors in publications. The effect caused by self-citations must be carefully considered before adopting citations as indicators of the scientific impact [31]. Thus, journals that carry the word “review” in their title generally have higher IF because they are more frequently cited, whereas articles contained in journals of social sciences and mathematics, for example, tend to have lower IF since citations tend to occur at a higher frequency for older manuscripts or books, which are not counted. Furthermore, the larger the number of authors in a paper, the larger the number of citations [32]. There is much debate over the subject impact factor, which was initially created to guide scientists in the search for bibliography for their own papers, allowing the interaction among researchers and helping librarians in the decision on which journals to be acquired. Over time, it has become the most important measure of scientific performance applied to journals, articles, scientists, universities, among other things, and this has generated much discussion among the scientific community [19]. In Argentina and Chile, according to reference [33, 34], for the field of Medicine, there are more citations of papers in English than in Spanish, regardless of the quality of the involved articles. American Scientific Research Journal for Engineering, Technology, and Sciences (ASRJETS) (2015) Volume 14, No 2, pp 295-303 299 The IF has been used, at a certain frequency, in the assessment of scientists aimed at promotions and job opportunities [35], in the scientific assessment of institutions [36, 37] and graduate programs [38], in the individual analysis of researchers in reference [30, 39, 40, 41] or for the concession of research grants [42, 14]. The first study of the opinion of scientists on the impact factor was conducted by [19]. A total of 1704 researchers of all fields of knowledge from 86 different countries participated in this survey, which indicated that the more articles they published, the less they believed that the impact factor consisted in a good measure. A possible explanation for this surprising result could be the fact that the most productive scientists do not need this measure. Authors in reference [43] evidenced no positive correlation between the impact factor and the self-citation rate, differently from the findings of [44, 45]. Citations and the impact factor can be raised by means of self-citation using editorial material, i.e., the editor cites articles of his/her own journal, increasing the number of citations [46]. The issue is that there is no penalty for excessive number of self-citations [47], which should not exceed 20% of the total [48]. The impact factor of a journal can be highly influenced by few of its articles that are very frequently cited [49]; it has been proposed that around 20% articles account for 80% of all citations [50]. A study evidenced that the self-citation levels were lower in Egypt, Algeria, Ukraine and Indonesia and higher in China [45]. Investigation into 222 journals from the latter country, including: 76 of agronomy (representing 34.2%), 57 of biology (25.7%), 28 of environmental sciences and technology (12.6%), 15 of forestry (6.8%), 24 academic journals of agrarian universities (10.8%), 9 of aquatic sciences (4.1%) and 13 of livestock and veterinary medicine (5.9%), evidenced that the mean rate of self-citation ranged from 2 to 67% in 2006, from one to 68% in 2007 and from 0 to 67% in 2008 [51]. Authors in reference [52] proposed a ranking of authors according to the H-index, which is the best tool to assess the quality of the individual scientific output of authors [32]. The understanding that studies of science should not follow merely quantitative criteria has changed to the idea that they should be based on bibliometric techniques added of other references and methods, as is the case for studies that describe the history of scientific production, based on the concepts of the archeology of knowledge by Foucalt in reference [10]. Introduction of the sociology and history of science has triggered the approaching between quantitative and qualitative studies [53]. Researchers are more concerned about publishing in high impact journals than about doing science itself [54]. Even though the impact factor has been a good indicator to classify the scientific influence and the level of journals, it is now time for newer indexes to rank periodicals, such as the Eigenfactor, which are possibly less subjected to manipulations [51]. Citations are not the only indicators of scientific quality [55].The manner how the scientific research output is assessed by funding agencies, academic institutions, among others, has been the subject of much debate, especially during the Annual Meeting of the American Society for Cell Biology, in San Francisco, California, in December 2012. American Scientific Research Journal for Engineering, Technology, and Sciences (ASRJETS) (2015) Volume 14, No 2, pp 295-303 300 This group of editors of scientific journals generated a set of recommendations, named DORA (Declaration on Research Assessment). Thus, this movement, formed by more than 150 scientists and 75 scientific organizations, including the American Association for the Advancement of Science (AAAS), aims to eliminate the use of the impact factor of periodicals in the assessment of an individual scientist. This declaration emphasizes the idea that the impact factor should not be used as a qualitative measure of research articles or even as an indicator of the contribution of a scientist for possible hiring, promotion or decision for grant [32]. DORA also provides a list of specific actions directed to improve the form how scientific publications are assessed by funding agencies, institutions, editors, researchers and other organizations [56]. By removing self-citations from a certain body of work, citation indexes will produce more significant results [57]. Citation analysis is a tool researchers have for personal reflection and planning of their research, even though there are risks of inappropriate use [55]. The present study consists in an exploratory study aimed at widening the debate over the impact factor and its application as a tool in the assessment of science. 3. Conclusion There is a need of the researchers think deeply about the impact factor role as a tool to evaluate the scientific activity. References [1] N. SENGUPTA. “Bibliometrics, informetrics, scientometrics and librametrics: an overview.” Libri, vol. 42, n. 2, pp. 99-135, 1992. [2] J. TAGUE-SUTCKIFFE. “An introduction to informetrics.” Information Processing & Management, vol. 28, n.1, pp. 1-3, 1992. [3] C.A. MACIAS-CHAPULA. “O papel da informetria e da cienciometria e sua perspectiva nacional e internacional.” Ciência da Informação, vol. 27, n. 2, pp. 134-140, 1998. [4] L. STREHL. “O fator de impacto do ISI e a avaliação da produção científica: aspectos conceituais e metodológicos.” Revista Ciência da Informação, vol. 34, n. 1, pp. 19-27, jan./apr. 2005. [5] F. HJOREGAARD CHRISTENSEN; P. INGWERSEN. “Data set isolation for bibliometric online analyses of research publications: fundamental methodological issues.” Journal of the American Society for Information Science, vol. 48, n. 3, pp. 205-217, 1997. [6] I. WORMELL. Informetria: explorando bases de dados como instrumentos de análise. Ciência da Informação, vol. 27, n. 2, 1998. Available: [Jun. 16, 2012]. [7] N.A.P. VANTI. “Da bibliometria à webometria: uma exploração conceitual dos mecanismos utilizados para medir o registro da informação e a difusão do conhecimento.” Revista Ciência da Informação, vol. 31, n. 2, pp. 152-162, may/aug, 2002. [8] Y. OKUBO. “Bibliometric indicators and analysis of research systems: methods and examples.” OECD Science, Technology and Industry Working Papers, 1997. American Scientific Research Journal for Engineering, Technology, and Sciences (ASRJETS) (2015) Volume 14, No 2, pp 295-303 301 [9] J.A. SILVA; M.L.P. BIANCHI. “Cientometria: a métrica da ciência.” Paidéia, 2001, vol. 11, n. 21, pp. 5-10. ISSN 0103-863X. Available: [August 20, 2012]. [10] C.A. ARAÚJO. “Bibliometria: evolução histórica e questões atuais.” Em Questão, Porto Alegre, vol. 12, n. 1, pp. 11-32, 2006. [11] V. ROQUE. “Métricas da Informação: O fator de Impacto na prática.” Egitana Sciencia, vol. 6; n. 10, pp. 177-207, 2012. [12] R. ABRAHAM. Webometry: Measuring the complexity of the world wide web. World Futures, v.50, n.1, p.785-791, 1997. [13] L. BJÖRNEBORN; P. INGWERSEN. “Toward a basic framework for webometrics.” Journal of the American Society for Information Science and Technology, vol. 55, n. 14, pp. 1216-1227, 2004. [14] C.J.C. JABBOUR; A.B.L.S. JABBOUR; J.H.C. OLIVEIRA. “The Perception of Brazilian Researchers concerning the Factors that Influence the Citation of their Articles: A Study in the Field of Sustainability.” Serials Review, vol. 39, pp. 1-4, 2013. [15] T.S. KHUN. “The structure of scientific revolutions.” The University of Chigaco Press, 1962. [16] G.L. VOLPATO. Publicação científica. Editora Cultura Acadêmica, 2008. [17] L. BORNMANN; I. NAST; H.D. DANIEL. “Do editors and referees look for signs of scientific misconduct when reviewing manuscripts? A quantitative content analysis of studies that examined review criteria and reasons for accepting and rejecting manuscripts for publication.” Scientometrics, vol. 77, n.3, pp. 415-432, 2008. [18] A.F. MAVROGENIS; P. RUGGIERI; P.J. PAPAGELOPOULOS. “Self-citation in Publishing.” Clinical Orthopaedics and Related Research, vol. 468, pp. 2803–2807, 2010. [19] Psychology, vol.11, pp.95–107, 2011c. [20] G. BUELA-CASAL; I. ZYCH. What do the scientists think about the impact factor? Scientometrics, vol. 92, pp. 281–292, 2012. [21] E. GARFIELD. “Citation indexes for science: a new dimension in documentation through association of ideas.” Science, vol. 122, pp. 108-111, 1955. [22] E. GARFIELD. “Citation analysis as a tool in journal evaluation: journals can be ranked by frequency and impact of citations for science policy studies.” Science, vol. 178, n. 4060, pp. 471-479, 1972. [23] E. GARFIELD. “How can impact factors be improved?” British Medical Journal, vol. 313, pp. 411- 413, 1996. [24] E. GARFIELD. “Journal impact factor: a brief review.” Canadian Medical Association Journal, vol. 161, pp. 979–980, 1999. [25] E. GARFIELD. “The meaning of the impact factor.” International Journal of Clinical and Health Psychology, vol. 3, pp. 363–369, 2003. [26] E. GARFIELD. The history and meaning of the journal impact factor. Journal of the American Medical Association, vol. 295, pp. 90–93, 2006. [27] R. SANCHO. “Indicadores bibliometricos utilizados en la evaluacion de la ciencia y la tecnologia: revision bibliografica”. Revista Española de Documentación Cientifica, vol. 13, n.3/4, pp. 842-865, 1990. [28] E. GARFIELD. The evolution of the science citation index. International Microbiology, vol.10, pp.65– 69, 2007. http://philpapers.org/s/Ralph%20Abraham http://dx.doi.org/10.1080/02604027.1997.9972670 http://philpapers.org/asearch.pl?pubn=World%20Futures American Scientific Research Journal for Engineering, Technology, and Sciences (ASRJETS) (2015) Volume 14, No 2, pp 295-303 302 [29] S.M. DHAWAN; S.K. PHULL; S.P. JAIN. “Selection of scientific journal: a model.” Journal of Documentation, vol. 36, n. 1, pp. 24-32, 1980. [30] B.C. PERITZ. “On the association between journal circulation and impact factor.” Journal of Information Science, vol. 21, n. 1, pp. 63-67, 1995. [31] M. AMIN; M. MABE. “Impact factors: use and abuse.” Perspectives in publising, vol. 63, n. 41, pp. 347-354, 2003. [32] D.W. AKSNES. “A macro study of self-citation.” Scientometrics, vol. 56, n. 2, pp. 235-246, 2003. [33] S. AASE. “What Weight Does Impact Factor Carry?” Journal of the American Dietetic Association, vol.108, n.10, pp.1604-1607, 2008. [34] E.B. ARRIBALZAGA. “El factor de impacto o el impacto de factores.” Revista Chilena de Cirurgıa, vol.57, pp.269–274, 2005. [35] E.B. ARRIBALZAGA. “El factor de impacto: su uso, abuso y mal uso.” Revista Argentina de Cirurgıa, vol. 97, pp. 15–21, 2009. [36] E. BAGATIN; B. GONTIJO. “The expansion of a measure: what is a scientific journal impact factor and how important is it for academic Brazilian dermatologists.” International Journal of Dermatology, vol. 50, pp. 1432–1434, 2011. [37] G. BUELA-CASAL. “Indices de impacto de las revistas cientıficas e indicadores para medir el rendimiento de los investigadores.” Revista de Psicodidáctica, vol. 15, pp. 3–19, 2010a. [38] G. BUELA-CASAL et al. Ranking de 2010 en producción y productividad en investigación de las universidades públicas espanõlas. Psicothema, vol. 23, pp. 527–536, 2011a. [39] G. BUELA-CASAL et al. Rendimiento en el doctorado en función del área de conocimiento. Revista de Psicodidáctica, vol. 16, pp. 181–192, 2011b. [40] G. BUELA-CASAL et al. Ranking de 2009 en investigación de las universidades públicas espanõlas. Psicothema, vol. 22, pp. 171–179, 2010b. [41] G. BUELA-CASAL et al. The h index of the presidents of the American Psychological Association (APA) through journal articles included in the web of science database. International Journal of Clinical and Health, 2011c. [42] I. ZYCH. Comparacion de los criterios para la acreditacion de profesores contratados y Funcionarios. Aula Abierta, vol. 39, pp. 51–62, 2011. [43] A. GALAN-GONZALEZ; I. ZYCH. Análisis de los criterios de la comisio´n nacional evaluadora de La actividad investigadora (CNEAI) para la concesión de los tramos de investigación en educación. Revista de pedagogía, vol. 63, n. 2, pp. 117–139, 2011. [44] A. ANDRADE; R. GONZÁLEZ-JONTE; J.M. CAMPANARIO. “Journals that increase their impact factor at least fourfold in a few years: The role of journal self-citations.” Scientometrics, vol. 80, n. 2, pp. 517– 530, 2009. [45] T.P. KURMIS; A.P. KURMIS, “Self-citation rates among medical imaging journals and a possible association with impact factor.” Radiography, vol. 16, n. 1, p. 21–25, 2010. [46] B. MINASNY; A.E. HARTEMINK; A. MCBRATNEY. “Individual, country and journal self-citation in soil science.” Geoderma, vol.155, pp. 434-438, 2010. http://www.sciencedirect.com/science/journal/10788174/16/1 American Scientific Research Journal for Engineering, Technology, and Sciences (ASRJETS) (2015) Volume 14, No 2, pp 295-303 303 [47] L. GONZÁLEZ; J.M. CAMPANARIO. “Structure of the impact factor of journals included in the social sciences citation index: citations from documents labeled Editorial Material.” Journal of the American Society for Information Science and Technology, vol. 58, pp. 252–262, 2007. [48] B. MINASNY; A. MCBRATNEY; A.E. HARTEMINK. “Soil bibliometrics - How much do we love cite ourselves.” Pedometron, vol. 22, pp. 11-13, 2007. [49] G.L. VOLPATO. Ciência: da filosofia à publicação. 6ª ed. Cultura Acadêmica, 2012. [50] H.A. BUCHTEL; S. DELLA SALA. “Impact factor: Does the 80/20 rule apply to Cortex?” Cortex, vol. 42, pp. 1064–1065, 2006. [51] J.A. FOLEY; S. DELLA SALA. “The impact of self-citation.” Cortex, vol. 46, pp. 802-810, 2010. [52] X. XIAO-DONG; W. YA-WEN. “Journal Self-citation Analysis of Some Chinese Sci-tech Periodicals.” Serials Review, vol. 37, pp. 171–173, 2011. [53] J.E. HIRSCH. “An index to quantify an individual's scientific research output.” Proceedings of the National Academy of Sciences USA. vol. 102, n. 46, pp. 16569–16572, nov. 2005. [54] R.N.M. SANTOS; N.Y. KOBASHI. “Bibliometria, cientometria, infometria: conceitos e aplicações.” Revista Pesquisa Brasileira Ciência da Informação, vol. 2, n. 1, pp. 155-172, 2009. [55] J.A.C. BAUM. “Free-Riding on Power Laws: questioning the validity of the Impact factor as a measure of research quality in organization studies”. Organization, vol.18, pp. 449–466, 2011. [56] J.M. NIGHTINGALE; G. MARSHALL. “Citation analysis as a measure of article quality, journal influence and individual researcher performance.” Radiography, vol. 18, n.2, pp. 60–67, 2012. [57] ALBERTS. “Editorial”. Science, vol . 340, 17 may, 2013. [58] S. CARLEY; A.L. PORTER; J. YOUTIE. “Toward a more precise definition of self-citation.” Scientometrics, vol. 94, pp. 777–780, 2013. http://www.sciencedirect.com/science/article/pii/S1078817411001374 http://www.sciencedirect.com/science/article/pii/S1078817411001374 http://www.sciencedirect.com/science/journal/10788174 http://www.sciencedirect.com/science/journal/10788174/18/2