Paper title (Paper Title style) Australasian Journal of Information Systems Lamp & Burmeister 2016, Vol 20 Editorial 1 The conduct of a review John Lamp AJIS Editor-in-Chief Deakin Business School Deakin University AJIS.EiC@gmail.com Oliver K. Burmeister School of Computing and Mathematics Charles Sturt University, Bathurst The Australasian Journal of Information Systems (AJIS) has been published since 1993 and is the premier journal in the Information Systems (IS) discipline in this region of the world. That is reflected in its ranking as an “A” journal in both the 2013 Australian Business Deans Council (ABDC) rankings and those of the Australian Council of Professors and Heads of Information Systems (ACPHIS). It also holds the DOAJ Seal for best open access publishing practice. Clearly it is important to guard this ranking and one way of achieving that is with rigorous review of submitted articles. This is a special editorial addressing the review process. The primary purpose of this editorial is to promote a greater consistency and rigour amongst reviewers. A secondary purpose is to help contributing authors better understand the review process that their article will be subjected to. As with other journals, the first point of call after submission is that a Section Editor (SE) reads the article and decides whether it is suitable for AJIS or not. If not, a ‘desk’ rejection follows. If it is suitable, that Section Editor then assigns reviewers. For the purpose of this exercise Dr Burmeister, an AJIS SE, has agreed to have his AJIS publications serve as examples. Of the five articles he has published in AJIS (Bowern, Burmeister, Gotterbarn, & Weckert, 2006; Burmeister, 1995, 2015; Burmeister, Islam, Dayhew, & Crichton, 2015; Simpson, Nevile, & Burmeister, 2003), only the two on scholarship are solo authored. So as not to embarrass co-authors, only the reviews of those two articles are detailed here; those two articles span almost the entire history of AJIS. The first was a scholarship paper published over 20 years ago (Burmeister, 1995), with reviewer responses having to be submitted to the editor via ‘diskette and hard copy’ (letter from the then Editor, R. MacGregor); back then the “A” in AJIS was ‘Australian’, not ‘Australasian’. At that time AJIS did not have its current online submission and review process, and instead manuscripts were mailed to reviewers, who in turn mailed back their recommendations (without including the original manuscript). Instructions to reviewers were simple. A reviewer had to choose one of three options (Accept, Accept with minor modifications, Reject) and had to send comments to the Consulting Editor (now SE). For the latter the instructions were as follows: Please detail on the page attached any praise or specific criticisms about the manuscript. These comments will be released to the authors. Also, if appropriate, please complete the separate page of comments which may be of use to the editors. This information will not be communicated to the authors. Such comments may support any criticisms that you have made above. That 1995 paper had gone to two reviewers, one recommending acceptance and the other acceptance with minor modifications. The process now is similar, but reviews are typically done by either two or three reviewers. As mentioned above, desk rejections are when an editorial decision is made that an article is not suited to the journal. The second scholarship paper (Burmeister, 2015) had that experience. It had previously been submitted to a journal ranked A* in ABDC, but received a desk rejection. It was then reformatted and submitted to Australasian Journal of Information Systems Lamp & Burmeister 2016, Vol 20 Editorial 2 AJIS. Before detailing the review process for this article, it is important to first describe the AJIS process as it is now. Upon submission the Editor-in-Chief firstly engages in a preliminary process, before assigning it to a SE, a domain expert in the area most closely related to the submitted paper. The preliminary process involves ensuring the article matches the scope of the journal, the length is suitable, it adheres to AJIS manuscript guidelines, and checking it for plagiarism using Turnitin. The SE then reads it and rejects it (desk rejection) or assigns it to at least two reviewers, to undergo blind review. The reviewers are volunteers and not paid for their time, hence a first step is to ask potential reviewers if they are willing to engage in the review (it involves sending them the title and abstract). Potential reviewers have two weeks to respond. If, as frequently happens, one or more cannot commit to undertaking the review, then other potential reviewer requests are made by the SE, and those people are again given two weeks to respond. Once two or more have agreed to conduct the review, they are encouraged to perform their review within a six week period. At times that does not happen and in some cases the SE has to find replacement or additional reviewers, which then extends the review period. Upon receiving the reviewer comments, the SE then makes a decision about the article and informs the author(s). That decision is made critically, not just taking reviewer recommendations, nor averaging them. One of four decisions is made: Accept, Revise, Revise and Resubmit, Decline. Revise is not the same as the 1995 example above of ‘Accept with minor modifications’. Instead, only after the revisions are received will the SE make a decision to Accept or not. Returning to Burmeister (2015), it underwent two rounds of review. The first involved three reviewers and resulted in a ‘Resubmit for review’ now called a ‘Revise and resubmit’ decision. The second round involved the same three reviewers and resulted in an ‘Accept’ decision. However, Accept does not mean accept as is, but instead, Accept subject to minor revisions, which are assessed by the SE. A rigorous review process is necessary to ensure the high standards of this A ranked journal are maintained. The following extract is from the first round of review, showing the assessment of one of the three reviewers. Overall assessment: Please provide your overall assessment of the publishability of the manuscript. Please provide written feedback to the author/s and /or editors in the text boxes below: Revisions required Relevance to the Journal: The manuscript falls within the scope of AJIS and is of relevance to the Journal's readership: Highly Relevant Motivation: Why this research is important from theoretical and practical viewpoints. What is the contribution: Adequately motivated Literature Review: Current state of the research in the area… the story so far… forms the basis for the research... may lead in to the theoretical section: Literature review satisfactory Theory: The theoretical development is logically sound and builds on a solidly established framework. E.g., Development of a model or theoretical argument that leads to a series of testable propositions. The paper then sets out to prove or reject these propositions. Alternatively, the paper may build theory based on observation: Poor theoretical development Australasian Journal of Information Systems Lamp & Burmeister 2016, Vol 20 Editorial 3 Research Method: How the research was conducted, that the method is appropriate and justified: Poor research method. Conduct of the Research: How the experiment was conducted or how the data was gathered and manipulated prior to analysis: Satisfactorily conducted Analysis of the Findings: How the data were analysed, appropriate analysis methods and statistics used: Analysis satisfactorily conducted Implications drawn from the Analysis: Do the implications, both theory and practice follow from the results of the analysis? Not over/understated.: Implications are well handled Conclusions and Limitations: What is concluded, and how is it limited by the nature of the experimental procedures or the population from which data was gathered, or from limitations of analytical analyses. Statement of contribution: Conclusions and Limitations satisfactorily drawn. Grammar, Spelling, Format, and Conformance with AJIS standards: Please detail any issues with this aspect of the manuscript in the text box below: Feedback to Authors: The paper reports on a research project investigating the experience of professional doctorate participants in the discipline area of information technology. The paper reports the findings and some conclusions from a survey of a broad range of stakeholders in the delivery of a Doctor of Information Technology at Charles Sturt University (CSU). The paper’s topic and findings are very interesting and well-suited to the readership of AJIS. There are three areas that I think the author(s) might valuably ‘polish’ the work: (1) the literature review is quite good, but the paper does not provide a concise summary of the issues in the area that the literature reviews (it is only a summary statement that is missing), (2) the research method statement is too superficial, with no discussion about how the interviews were coded, analysed, or how the findings emerged from whatever analytical process took place (a more disciplined recounting of the data coding and analysis is warranted), and (3) the analytical findings are not summarised into the concise statement that the abstract implies (again, only a summary statement is required to address this issue). We can see from the above example that there is a standard form that all reviewers are expected to complete. In addition there are comments to the author(s) and optionally, comments to the SE, which are not passed on to the author(s). Use of this format ensures a uniform approach to the review process. Most authors submitting articles to AJIS and elsewhere find it difficult to cope emotionally with rejection and criticism. Therefore we recommend the PIP approach to reviewers; praise, improvement, praise. That is, start and end with something praise-worthy in the article. Reviewers are encouraged to provide constructive feedback, even if they recommend ‘Decline’. AJIS considers this a hallmark of a quality publication and part of the scholarly, collegial process. That is, with appropriate feedback even rejected papers can be reworked and submitted elsewhere, for eventual publication. Ideally the comments to the Australasian Journal of Information Systems Lamp & Burmeister 2016, Vol 20 Editorial 4 author(s) should include advice that can be acted upon, as is seen above, where a suggestion is made to have a summary of the findings. Looking at the published version of the paper, there is a summary table at the end of the paper, which came about as a result of this recommendation from the reviewer. Professor Allen Lee, a keynote speaker at ACIS this year, wrote an excellent article many years ago on the reviewing process, which is worth reading (Lee, 1995). This is also available on the Web at http://www.people.vcu.edu/~aslee/referee.htm Professor Lee mentions making the extent of your expertise clear, and that your comments are framed to be actionable. Reviewers should specify their expertise in their review, eg “This reviewer’s research area is ontology, but focusing on the philosophy of information, not on the use of approaches such as Ontolingua.” Knowing the reviewer’s expertise allows authors to evaluate the relative strengths of the review comments. Providing comments that are actionable makes a review useful, rather than an abstract opinion. For example, stating “the statistical method is inappropriate” is a valid comment, but “analysis using ANOVA would be more appropriate” provides the same information, but in a form that allows the author(s) to not only see the issue that the reviewers have identified, but have some idea what they can do to improve the paper. Figure 1: AJIS Article Throughput 2010 to 2014 If you think of the need for good reviews of papers against the number of papers submitted to AJIS some pretty dramatic issues emerge. Figure 1 puts into a wider context. The profile of AJIS in the world has dramatically changed over the past few years. Figure 1 shows the increase in submissions, and the acceptance rate over the past five years. Submissions have pretty much tripled, and the quality of papers submitted, as shown by the acceptance rate, have also improved. At the time of writing there are five newly received papers, 53 in review and four in copyediting. While that might seem pretty innocuous, the implications are that AJIS is at this time using the services of about 45 Section Editors and over 150 reviewers. AJIS prefers to call upon academics from the region, but does not hesitate to use the services of international experts, especially where sophisticated domain knowledge is required. Despite casting the net wide, it is worth remembering that the Index of Information Systems Journals (Lamp, 2004) lists 700 IS journals at present. That requires a very large number of IS researchers and academics to adequately review submitted articles. 0% 5% 10% 15% 20% 25% 0 50 100 150 200 2010 2011 2012 2013 2014 2015 A cc ep te d N u m b er o f A rt ic le s Year AJIS Article Throughput Submitted Percentage accepted Australasian Journal of Information Systems Lamp & Burmeister 2016, Vol 20 Editorial 5 Participation in the review process is essential for the viability of delivering properly scrutinized research to the research community and to the wider community of practitioners and interested public. Experienced researchers understand that necessity, and the obligation to participate in both ends of the process, however much of a strain it can be. Post-graduate students should also participate as reviewers. Post-graduate students benefit from participation in the review process by gaining a deeper understanding of the process, the standard of submissions before publication and, quite often, exposure to ideas in their area which they had not considered – it is definitely a two way process. Post-graduate students often contribute a fresh approach to a topic, which is a benefit to the journal. Section Editors will ensure a balance of reviewers to deliver high quality reviews to the authors they are assisting. Acting as a reviewer is essential for the continued existence of a journal in which to publish your research and has direct benefits to those reviewing. Your participation is greatly appreciated. References Bowern, M., Burmeister, O. K., Gotterbarn, D., & Weckert, J. (2006). ICT Integrity: Bringing the ACS Code of Ethics up to date. Australasian Journal of Information Systems, 13(2), 168-181. Burmeister, O. K. (1995). Evaluating the factors that facilitate a deep understanding of data analysis. Australasian Journal of Information Systems, 3(1). Burmeister, O. K. (2015). Improving professional IT doctorate completion rates. Australasian Journal of Information Systems, 19, 55-70. doi: 10.3127/ajis.v19i0.1073 Burmeister, O. K., Islam, M. Z., Dayhew, M., & Crichton, M. (2015). Enhancing client welfare through better communication of private mental health data between rural service providers. Australasian Journal of Information Systems, 19, 1-14. doi: http://dx.doi.org/10.3127/ajis.v19i0.1206 Lamp, J. W. (2004). Index of Information Systems Journals. Geelong, Deakin University. [Online] Available: http://lamp.infosys.deakin.edu.au/journals/ Lee, A. S. (1995). Reviewing a Manuscript for Publication. Journal of Operations Management, 13(1), 87-92. Simpson, C., Nevile, L., & Burmeister, O. K. (2003). Doing Ethics: A Universal Technique in an Accessibility Context. Australasian Journal of Information Systems, 10(2). Copyright: © 2016 Lamp & Burmeister. This is an open-access article distributed under the terms of the Creative Commons Attribution-NonCommercial 3.0 Australia License, which permits non-commercial use, distribution, and reproduction in any medium, provided the original author and AJIS are credited. http://creativecommons.org/licenses/by-nc/3.0/au/