Microsoft Word - FET-V1N2-p180 Frontiers in Education Technology Vol. 1, No. 2, 2018 www.scholink.org/ojs/index.php/fet ISSN 2576-1846 (Print) ISSN 2576-1854 (Online) 180 Original Paper An International Comparison of Educational Research Review Mechanisms Chiou-Rong Wang Yang1 & Huei Lan Wang2* 1 Department of Health & Social Work, Yu-Da University, Miaoli, Taiwan 2 Nan Hua University, Chiayi, Taiwan * Huei Lan Wang, Nan Hua University, Chiayi, Taiwan Received: July 15, 2018 Accepted: September 25, 2018 Online Published: September 27, 2018 doi:10.22158/fet.v1n2p180 URL: http://dx.doi.org/10.22158/fet.v1n2p180 Abstract This paper explores the current status of educational research review mechanisms, assessment and ethical mechanisms in the UK, USA, Germany, France and Australia, and assesses the merits and shortcomings of the mechanisms and ways in which to improve them in those countries. This exploration is based on the perspective of comparative education and the concept of the academic tribe of T. Becher to assess peer review mechanisms. The research results include, first, the peer review system that was gradually developed in the UK, USA, France and Germany in the 1950s, which became a common mechanism for academic gatekeeping of the quality of educational research. Second, to ensure academic excellence, peer reviewers are required to be fair, professional, and specialized in a variety of “procedural justice” designs. Third, anonymous or not, peer review depends on the subcultures of different countries or academic communities. In the UK and USA, the process tends to be more transparent, and authors can suggest potential reviewers; however, this is not the case in Germany and France. Fourth, for the purpose of improving malpractice in terms of ethics and reliability in peer review, the professional qualifications of reviewers should be evaluated regularly. Finally, the outcome of the comparative study may be used as a reference for improvement of review mechanisms for educational research in Taiwan. Keywords peer review mechanism, educational research www.scholink.org/ojs/index.php/fet Frontiers in Education Technology Vol. 1, No. 2, 2018 181 Published by SCHOLINK INC. 1. Introduction In recent years, countries such as France, Germany, and the USA, etc., have reflected upon the quality of educational research. Criticisms of the quality of educational research include lack of rigor and originality and weak correlations among educational policies, practices and educational research. In order to improve the quality of educational research and to provide a more practical contribution to the enhancement of education, two approaches have been employed to reconstruct the mechanisms that control the quality of educational research: one is the provision of sufficient educational research resources; the other is reasonable and effective regulation of inherent review mechanisms. Peer review, which is referred to as the “academic gatekeeper” and is a means by which to review and allocate funding to academic projects, is a common approach operating in the UK, Germany, France, the USA and Australia. However, problems with this approach remain, such as fairness (Armstrong et al., 1982), gender bias (Lane & Linden, 2009) and integrity issues (Jeffers, 2002; Macrina, 2005). Some have even described the peer-review process as a turf battle, the ultimate prize being knowledge, science or doctrine being published. On the one side, we have the writers and originators of ideas, and on the other, editors and critics (Spier, 2002). Therefore, the question of how to improve and ensure the credibility of review mechanisms is key to assuring the quality of educational research in these countries. In Taiwan, investigation into the fairness of review mechanisms and review ethics can be traced back to 1982. Investigation of research review mechanisms in developed countries by comparative study may contribute to improvement of the practice of this mechanism in Taiwan. Therefore, the purposes of this article are as follows: 1) To understand the development of review mechanisms in relation to educational research in the UK, USA, Germany, France and Australia; 2) To explore approaches to the improvement of review mechanisms in educational research in the UK, USA, Germany, France and Australia; and 3) To synthesize comparative study results as reference points for the improvement of review mechanisms in educational research in Taiwan. 2. Development and Criteria of Review Mechanisms for Educational Research in Various Countries Peer review is a widely-used method for evaluating research papers and proposals nowadays. It is primarily used for manuscripts submitted to periodicals and for research proposals for which authors have applied for funding. Peer review is also employed to evaluate individual researchers and whole institutions. Independent peer reviewers are commissioned to carry out reviews to determine whether a manuscript or proposal meets the quality indicators of the discipline, the funding organization, or the journal in Germany, France, the UK, USA, and Australia. www.scholink.org/ojs/index.php/fet Frontiers in Education Technology Vol. 1, No. 2, 2018 182 Published by SCHOLINK INC. 2.1 Histories and Current Status The peer review mechanism was first developed in connection with the founding of scientific societies in the 17th century, in particular the Royal Society in England. However, unfortunately, the foundation of peer review was not created as a means of scientific quality control; rather, it was a political compromise aimed at guaranteeing self-control in the interests of the establishment of the day. To date, peer review has played an important role in controlling the quality of educational research. Manuscripts and proposals are sent to two or more anonymous peer reviewers who are experts in the same area. Their reviews then form the basis of a decision as to whether to accept the paper or not. Most developed countries, including those referenced above, have established peer review systems to control the quality of research. A brief history and the current status of research review mechanisms in various developed countries, including Germany, France, the UK, USA and Australia, are presented below. 2.1.1 Germany In Germany, the Deutsche Forschungs Gemeinschaft (DFG), founded in 1951, is the academic institute responsible for awarding grants for fundamental research in Germany. The DFG consists of 87 members, including 64 universities, 13 research organizations, 7 research institutes and 3 associations for technological research. For the provision of financial grants for high-quality research projects, the peer review system is very rigorous. The selected reviewers have excellent qualifications, including a doctoral degree and three years of work experience, and serve for a four-year period to ensure the quality of review results, which is based on the standards of scientific, fair and authoritative features. Anonymity is enforced by the review system. In the process, two experts are nominated by the DFG to review research projects. If the review results show a consensus between the two members, the opinions of the reviewers will be reported to the Financial Committee; if not, a third member will review the project. The third reviewer is the key person who plays a decisive role in terms of granting the funding. Joint research projects are encouraged by the DFG. The Collaborative Research Center (CRC) of the DFG is responsible for reviewing joint research projects. Applicants are required to complete a 100-page proposal for review, following which almost 60% of the applicants qualify for a second review opportunity. For the second-round review, the qualified applicants write a 500-page document that is more complete than their submission for the first review round. In the second round review, 80% are successful following the review process and receive grants from the DFG. Then, two members appointed by the DFG, two from the Financial Committee, and ten from the review committee of the CRC spend two days every four years visiting researchers in the field. The tasks of reviewers include reviewing the research proposal, attending the review meeting of the CRC, consulting on the final decision of the Financial Committee, and reporting the final decision to the Review Committee with regards to awarding or refusing the grant. www.scholink.org/ojs/index.php/fet Frontiers in Education Technology Vol. 1, No. 2, 2018 183 Published by SCHOLINK INC. 2.1.2 France The review mechanism in France, the “Comité National D’évaluation de l’enseignement Supérieur” was founded in 1985. Qualitative peer review and panel review are the major means by which quality assurance is ensured in educational research. In 1996, ten survey questions used for reviewing grant proposals were developed: 1) Do they make progress in research quality compared with the level on the agreement made with the Ministry of Education four years ago? 2) Are there any regulations to maintain an appropriate portion of time spent conducting research in the university’s budget? 3) What is the relationship between the university and research organizations outside the university? 4) Does the university provide special support for the leader(s) of research groups? 5) Are there any strengths or weaknesses in the research? 6) What is the function of the doctoral program in relation to research? 7) How does the university participate in local level research activities? 8) How can research results be applied to local development? 9) For how much is the research to be funded and how are the funds to be distributed? 10) Do you value the research resources? (AESEC, 2010) These questions show that there are three key criteria for evaluating research quality: 1) The strength and/or weakness of the research; 2) Research conditions: staff, the funding, the professional growth of researchers, and their relationship with research organizations; and 3) The potential application of research results. After 2007, the European Union (EU) developed European Educational Research Quality Indicators (EERQI). There include five main measures (Gogolin, 2011; Gogolin et al., 2003): 1) Rigor: Are there clear and congruent arguments and appropriate methodology? Are research conclusions fully supported by research evidence and is this evidence reflected sufficiently? 2) Originality: Does the paper have sufficient creativity and innovation, and do its results provide possibilities for developing new knowledge? 3) Significance: Are the issues researched valuable? Do the results show innovation in theoretical and/or (?) methodological approaches? 4) Integrity: Has the author authentically written the paper, and has the author abided by ethics codes and liability regulations? 5) Style: Is the paper’s structure well organized? Is the writing style concise and attractive enough for reading through? The measures include fitness of theoretical basis and methodological application, as well as an appropriate writing style. www.scholink.org/ojs/index.php/fet Frontiers in Education Technology Vol. 1, No. 2, 2018 184 Published by SCHOLINK INC. 2.1.3 The United Kingdom (UK) In the UK, there has been radical debate over issues of efficiency and effectiveness of peer review mechanisms between educational policy researchers and Research Councils UK (RCUK, 2006, p. 13) in the past decade. In the 1990s, the Advisory Board for the Research Councils published the Boden Report (1990), which explored the merits of and improvements needed to the peer review process. In 1995, the Royal Society completed a report discussing the peer review process, which showed the merits of peer review to be as follows: 1) To keep the confidence of the community; 2) To improve research perspectives; 3) To provide accountability; and 4) To offer access to compete appropriately (RCUK, 2006, p. 15). In the report, it appears that there are two shortcomings in educational research: 1) Conservatism negates the appearance of radical perspectives; 2) The burden of peer review (RCUK, 2006, pp. 16-17). The Research Councils announce the results of the institutionalized review process both before and after awarding a grant. Oancea (2009) argues that a good peer review process needs to build in quality criteria for the assessment of educational research in different contexts. For example, for journal publication, the assessment criteria are: relevant, significant and original contribution, accessibility and clarity, high standards of scholarship in argument and interpretation, sound methodological basis, ethics, adequate presentation, good writing style, and fluent language (Oancea, 2009). Criteria used in the assessment of proposals for funding research projects are: relevance, originality, innovation and topicality, scholarly importance, uniqueness, sufficient background information, clear description, concise and appropriate objectives, clear aim and purpose, explicit theoretical and conceptual framework, scientific quality of approach, potential for positive use, social-economic importance, user engagement, good dissemination, contribution for capacity enhancement and benefits for the wider scholarly community, international competitiveness, ethical conduct and awareness of wider ethical implications, proper scientific practice, outstanding ability of the applicants, inclusion of feasibility/achievability, containing evaluation, showing profit, presentation of the application, and corresponding to regulations (Oancea, 2009). Criteria used in the assessment of proposals for the development of practice-based research, and criteria used in project and report evaluations, include good performance in terms of: accountability, effectiveness, quality, innovation, contribution to knowledge, contribution to policy and practice, contribution to research training and professional network, further development of research results, quality of output and effectiveness of dissemination, possibilities for further research, peer appreciation and user satisfaction (Oancea, 2009). www.scholink.org/ojs/index.php/fet Frontiers in Education Technology Vol. 1, No. 2, 2018 185 Published by SCHOLINK INC. 2.1.4 The United States of America (USA) Peer reviews are conducted in the USA by the National Science Foundation, which was founded in 1950 based on the National Science Funding Law, and is the major institute that offers funding for proposals twice a year. The process of review includes the following procedures: 1) Proposer submits a proposal to the NSF. 2) The proposal is reviewed by members of a review committee. 3) The type(s) of review is identified: electronic review, panel review, and special board review. 4) The proposer receives suggestions for proposal improvement (?) from the review committee. 5) The NSF makes a decision to award a grant or not. If the grant amount is over USD $1,500,000.00, it must be awarded by a council member of the NSF. 2.1.5 Australia In Australia, the Australian Research Council, which belongs to the Department of the National Board of Employment and Training (NBEET), is responsible for managing proposals for both scientific research and educational research. The features of the peer review mechanism are a focus on feedback from peer reviewers and the provision of a means by which to appeal reviewer judgments. 2.2 Criticisms of the Peer Review System Peer review is an important mechanism within the scientific research system. It determines which articles will be published in which journals and which projects can be carried out. As peer reviews play such an important role in decision-making for the distribution of funding and for public reward and recognition, peer reviewers have significant effects on the course of scientific research and on the career success of researchers. In view of the power that is thus conferred on peer reviewers, critical voices have been continually heard over the past 30 years. Criticisms include the following: 1) Peer review is too slow and too expensive. 2) Peer review is unfair, because it discriminates against women and young female researchers and favors known figures (Matthew Effect). 3) Peer review is hostile to innovation because it favors established methods and ways of thinking. 4) Peer review is unreliable because reviewers often contradict each other. 5) Peer review opens the door to cronyism in which established researchers favor each other in the anonymity of their review work (Old-Boys-Network) (Reinhart, 2006). 6) Peer reviews lack credibility due to practices whereby assessors are nominated by the applicants themselves. However, in connection with this last criticism, a whole research field was initiated in a study by the US National Science Foundation (NSF) in 1978. The authors came to the conclusion that 50% of a proposal’s success was determined both by random factors and the decisions of reviewers; these conclusions focused on journals, rather than on funding organizations. Above all, three questions are always proposed: www.scholink.org/ojs/index.php/fet Frontiers in Education Technology Vol. 1, No. 2, 2018 186 Published by SCHOLINK INC.  How fair is the process (bias)?  To what extent do the peers agree (reliability)?  Is the approved proposal indeed better than the rejected project (validity)? (Reinhart, 2006) Answers to these three questions remain unclear for now, because the results of peer review are affected by many complex factors. In addition, research on the content of review reports is insufficient by which to evidence proper criticism of peer review. 2.3 Critiques of Educational Research Review Mechanisms in Various Countries Review mechanisms for educational research in the USA and Germany were established earlier, and were later implemented in France. The main reason for this is that the funding principle in the USA is based on capitalism, while products of educational researchers in Germany and France belong to the public, with strong autonomic professional self-regulation. Since the Bologna Declaration to form the EHEA, academic evaluation processes and mechanisms have gradually become more similar to the approaches used in the UK and USA. In particular, research review mechanisms in Germany focus more on mutual communication between the applicant and reviewer, which aims to prevent unfairness in a one-way review. Common challenges in these five countries reflect two problems: first, the reliability and fairness of the review mechanism, and second, limitations related to peer review ethics. 2.4 Reliability and Fairness in Educational Research Review Mechanisms Turner (2003) discusses promoting fair, appropriate, identifiable, timely and helpful (F.A.I.T.H.) core attributes for peer reviews in the Journal of Academic Ethics. There are also stakeholders in the process of peer review, which include the author, reviewer, editors and academic communities. In this process, there are individual responsibilities, technical responsibilities and professional responsibilities. The reviewer’s individual responsibility is to avoid personal preference or confirmatory bias—the tendency to emphasize and believe experiences which support one’s views and to ignore or discredit those which do not (Mahoney, 1977)—while also understanding the research situation and new developments in his or her research domain. The technical responsibility of reviewers includes providing detailed, constructive and deliberate feedback and supporting his or her comments based on related core literature. 2.5 Reliability in the Operation of Peer Review Mechanisms Becher (1989, pp. 61-62) finds that malpractice takes place in the peer review system. According to Becher, there are two common complaints against the peer review system. First, in highly-specialized fields, extensive fair assessment is very limited, because the selection of reviewers is sometimes limited to professional acquaintances. Their review comments may lack professionalism and reliability, or may be a result of selection from a limited pool of experts. These people may resist an opponent’s academic point of view in an irrational way in order to protect their own positions. Second, some people believe the peer review system to be unreliable. Opinions of peer professionals are often in disagreement, and www.scholink.org/ojs/index.php/fet Frontiers in Education Technology Vol. 1, No. 2, 2018 187 Published by SCHOLINK INC. in marginal areas of scientific research, obtaining funding for scientific research to a large extent depends on “chance”. Becher (1989, p. 62) points out, further, that different effects of the peer review system are seen in different disciplines. Principles of the selection of authors from areas different to those of the published article are common in professional journals. There is more inclusiveness in soft disciplines, with inherently permeable boundaries, than in hard disciplines, with firm boundaries. This echoes Bernstein’s (1971, pp. 203-210) investigation of power distribution and the social control of educational knowledge in terms of “classification” and “framing”. Becher (1989, p. 64) received different responses when interviewing experts in the pure sciences and the soft sciences. Experts in pure, hard and applied sciences pointed out that there will be unethical acts in peer reviews—so-called “improper means” or “plagiarization”. They accuse prestigious reviewers of delaying or denying others’ research, but publishing similar papers themselves. This phenomenon of corruption is not common in the soft sciences. Few would conduct research on the same topic, and even within the same theme there exist different views or opinions. These fields of science have more space in which scholars can express their views. It is clear that there are significant differences in the review system between hard and soft sciences. Becher (1989, p. 64) pointed out that limitations of peer review can be seen in book reviews or journal reviews in the public domain, in which the author can deliver his or her opinion, and black box behavior can more easily be seen in this arena than in academic promotion, science funding, honors and awards. The intention of ignorance, exclusion of outstanding researchers, and selection of relatively poor researchers is to consolidate the boundaries of poor academic review processes. Shils (1997, pp. 305-306) mentions that the secrecy and anonymity within the peer review system causes dissatisfaction among reviewed authors. There are also doubts about fairness. Therefore, reforms to the peer review system to limit secrecy and anonymity have been requested. The reason for secrecy in science research is that professors with prominent positions control the consistency of views in the field. A remedial measure would be to let authors know the names of their reviewers, the presumption being that reviewers should take responsibility for their review comments, so that the review is more equitable. Smith (2006) argues that authors know the names of reviewers for the British Medical Journal, and the NSF list the names of all reviewers on their website (NSF, 2006). 3. Conclusions and Recommendations 3.1 Conclusions In summary, the paper draws the following conclusions: 1) The peer review system gradually developed in the UK, USA, France and Germany in the 1950s has become the common mechanism of academic gatekeeping in educational research communities. It is still believed that the function of peer review in academic communities is an appropriate approach for quality management in educational research. www.scholink.org/ojs/index.php/fet Frontiers in Education Technology Vol. 1, No. 2, 2018 188 Published by SCHOLINK INC. 2) The standard procedures of the peer review system are well-established in developed countries such as the USA, Germany, France, UK and Australia. The deliberately-designed peer review system fulfils its function. Specializing in a variety of “procedural justice” designs, peer reviewers are asked to be fair and professional. 3) Whether peer review takes place anonymously or not depends on the subcultures of different countries or academic communities. The UK and the USA tend to list reviewers’ names publicly, but this is not the case in Germany or France, in which countries it is believed that a blind review process is better than a transparent process. 4) There exists malpractice and the problem of reliability in the peer review process. In Germany, reviewers must be qualified to Ph.D. level and have at least three years of related work experience. The service period for reviewers is also limited to a four-year term. However, other countries do not have such regulations. When reviewers cannot exert professional ethics as “self-discipline”, reviewer decisions in such situations have lost their review reliability and validity. 5) Professional qualifications for rigorous reviewer selection are emphasized in the five countries, because professional judgment and argumentation are broadly influenced by many factors, such as the ideology of the peer reviewer arising from conservatism, cognitive cronyism or disciplinary hegemony, chronic levels of disagreement or incompetence of reviewers, misconduct by reviewers, loss of creativity (how is this related?), bias and elitism, especially in relation to minority groups and women. 3.2 Recommendations Based on the conclusions above, this paper makes the following recommendations to improve educational review mechanisms: 1) The elaboration of the peer review: Before establishing a good review mechanism, select professional reviewers with perspectives for the committee as academic gatekeepers. In brief, meta-evaluation of reviewers and committees is needed in the peer review system. 2) Reviewers and committee members are required to be able to practice fully professional argumentation and to have a professional consciousness regarding constructing an epistemic authority. According to Tuner and J. Habermas (2003), the greater the opportunities for communication between researchers and reviewers, the less prejudice, selfishness, and non-professional involvement there will be. In other words, reviewers should exert themselves to play the role of constructors of high-quality knowledge as a community responsibility. 3) Encourage the establishment of a recruiting and nurturing system of cross-disciplinary or interdisciplinary review vision. Most of the malpractice in peer review is due to unconscious bias and/or professional fixation, or a superiority complex. In order to improve the quality of review, it is necessary to establish a database of experts with the ability to review cross-border and to give them greater honor and a review tenure of at least three years or so. www.scholink.org/ojs/index.php/fet Frontiers in Education Technology Vol. 1, No. 2, 2018 189 Published by SCHOLINK INC. 4) Establish a punishment mechanism. Although the number of reviewers is relatively small in Taiwan in comparison with the UK, USA, Germany and France, malpractice still exists in the academic arena. Therefore, it is worth reflecting on the qualifications of reviewers in research communities. References Aesec. (2010). Association des enseignants et chercheurs en sciences de l’education. Retrieved December 20, 2010, from http:// www.aecse.net/ cgi-bin/prog/index.cgi?langue Armstrong, J. S., Starr, M., Ayres, H. F., Robinson, S. M., & Bonder, S. (1982). Is review by peers as fair as it appears. Interfaces, 12(5), 62-74. https://doi.org/10.1287/inte.12.5.62 Becher, T. (1989). Academic tribes and territories. In Intellectual enquiry and the culture of disciplines. Milton Keynes: Society for Research into Higher Education & Open University Press. Bernstein, B. (1971). Class, code and control. In Theoretical studies towards a sociology of language (Vol. 1). London: R.K.P. https://doi.org/10.4324/9780203014035 Gogolin, I. (2011). Quality indicators. Retrieved December 30, 2011, from http://www.eerqi.eu Gogolin, I., Smeyers, P., Garciaa Del Dujo, A., & Rusch-Feja, D. (2003). European social science citation index: A chance for promoting european research? European Educational Research Journal, 2(4), 574-553. https://doi.org/10.2304/eerj.2003.2.4.6 Jeffers, B. R. (2002). Continuing education in research ethics for the clinical nurse. J Contin Educ Nurs, 33(6), 265-269. Lane, J. A., & Linden, D. J. (2009). Is there gender bias in the peer review process at journal of neurophysiology? Journal of Neurophysiology. https://doi.org/10.1152/jn.00196.2009 Macrina, F. L. (2005). Scientific integrity: Text and cases in responsible conduct of research (3rd ed.). Washington, D.C.: ASM Press. Mahoney, M. J. (1977). Publication prejudices: An experimental study of confirmatory bias in the peer review system. Cognitive Therapy and Research, 2(1), 161-175. https://doi.org/10.1007/BF01173636 NSF. (2006). Report to the national science board on the national science research councils. Oancea, A. (2009). Quality criteria for the assessment of educational research in different contexts. Retrieved June 15, 2018, from http://reflectiveteaching.co.uk/media/3.11_Oancea_RB_80_FINAL.pdf RCUK. (2006). Report of the research councils uk efficiency and effectiveness of peer review project. Retrieved June 20, 2018, from http://webarchive.nationalarchives.gov.uk/20090507143607/http://www.rcuk.ac.uk/cmsweb/downl oads/rcuk/documents/rcukprreport.pdf www.scholink.org/ojs/index.php/fet Frontiers in Education Technology Vol. 1, No. 2, 2018 190 Published by SCHOLINK INC. Reinhart, M. (2006). Peer review. Retrieved June 15, 2018, from http://www.research-information.de/iq/iq_inhalt.php?agora/Peer_Review/peer_review_inc.htmlX XXPeer%20Review Shils, E. (1997). Order of learning: Essays on the contemporary university. Piscataway, NJ: Transaction Publishers. Smith, R. (2006). Peer review: A flawed process at the heart of science and journals. Journal of the Royal Society of Medicine, 99(4), 178-182. https://doi.org/10.1177/014107680609900414 Spier, R. (2002). The history of the peer-review process. Trends in Biotechnology, 20(8), 357-358. https://doi.org/10.1016/S0167-7799(02)01985-6 Turner, L. (2003). Promoting f.A.I.T.H. In peer review: Five core attributes of effective peer review. Journal of Academic Ethics, 1(2), 181-188. https://doi.org/10.1023/B:JAET.0000006844.09724.98