Paper title (Paper Title style) Australasian Journal of Information Systems Coulthard & Keller 2016, Vol 20, Research Article Publication, Journal Rankings, Researcher Views 1 Publication Anxiety, Quality, and Journal Rankings: Researcher Views Darryl Coulthard La Trobe University d.coulthard@latrobe.edu.au Susan Keller La Trobe University Abstract Journal ranking systems are increasingly used to measure research performance of academics and universities. A growing number of academic commentators have voiced concerns of possible undesirable outcomes such as increased publication anxiety and an increase in safe and conforming research, but there have been few empirical studies on the possible effects. To address this gap, we surveyed Information Systems (IS) academics who published in one of three key IS conferences in 2013, to gather their views of the effects of journal ranking systems. Overall, we found that the concerns in the literature were strongly reflected in the views of those surveyed. Academics believe the system has greatly increased their publication anxiety. While most believed that the quality of published research had improved, researchers believe the ranking systems inhibit innovative, risky research, and encourages safe, conforming, mainstream research. Keywords: journal rankings; quality; publication anxiety; survey 1 Introduction The rise of neoliberalism and the emphasis on user pays by governments in most developed countries has resulted in reduced funding to public universities over the past 25 years (Bourdieu, 1998; Marginson, 2011). Encouraged by governments, universities have developed market-driven business models and responded to increased pressure to provide quantifiable and marketable evidence of the quality of educational and research outcomes (Hammarfelt & de Rijcke, 2014; Kinman, 2014). Simple metrics such as student satisfaction surveys, research quantum, and citation counts are in demand by government agencies, university marketers and student 'consumers' to demonstrate ‘value for money’, market success and brand differentiation and also feed local and international league tables designed to rank institutions and promote consumer choice and bragging rights (Marginson, 2011). While the validity and veracity of these metrics are questionable, universities nevertheless may have little option but to participate as they compete in a global education market for full fee-paying students and are subject to government requirements (Harvey, 2008). In such as climate, it is unsurprising that the use and importance of journal rankings schemes have increased since they provide a ready proxy for government agencies, funding bodies and universities to determine the quality of research outputs. Such schemes may be convenient for monitoring research performance but they may also be leading to undesirable consequences. There are very few empirical studies, however a growing number of academic commentators have voiced concerns on the adverse effects of journal ranking systems (Rijcke, Wouters, Rushforth, Franssen, & Hammarfelt, 2015). These concerns include: the reduction of research in innovative areas and cross disciplinary work; a narrowing of research methods; encouragement of orthodoxy; and a reduction in research dealing with big questions (Laudel & Gläser, 2014; Lincoln, 2012; Mingers & Willmott, 2013). This may be particularly concerning for Information Systems as it grapples to understand the rapid changes in information technology and its uses and outcomes in contemporary social, organizational and business activities (Cooper & Zmud, 1990; Grover & Lyytinen, 2015). Such technological changes and responses may fall outside well-established research programs and Australasian Journal of Information Systems Coulthard & Keller 2016, Vol 20, Research Article Publication, Journal Rankings, Researcher Views 2 ‘epistemic middle range scripts’ identified by Grover and Lyytinen (2015) in information systems and may require new innovative research programs, concepts and methods. Such innovative programs may necessarily depart from established research paths and the ‘comfort zone’ of IS scholars (Grover and Lyytinen, 2015) and are as a consequence more likely to have poorer, risky publication prospects. In effect, by rewarding orthodox and well-established research and potentially devaluing new research programs and approaches (Mingers & Willmott 2013), the discipline will not only restrict the knowledge it produces, but also its relevance since it will be unwilling to engage in the innovative research topics and methods needed to address important questions in a rapidly changing global environment. In addition to possible undesirable effects on research from a discipline perspective, there is some suggestion of undesirable effects at the individual researcher level too. A growing body of research suggests that academic staff are suffering increased work stress and low levels of psychological health (Kinman, 2014). The use of journal ranking schemes to measure and monitor research output may be adding to such stress and contributing to highly skilled academics leaving higher education institutions (Kinman & Jones, 2003). Universities may well have difficulties recruiting bright graduates and retaining academics. Given the importance of university research to the economic, intellectual, cultural and social development of a country (Rossi & Geuna, 2015, pp. 165-166), any changes that lead to a contraction of research freedoms and loss of research staff would have very serious ramifications and thus warrants further investigation. In sum, there are two areas of concern with the rise of journal ranking systems- its effects on research and the effect on researchers. While academic opinion expressed in journals is overwhelmingly negative concerning these effects, there has been little if any empirical research into the effect on research and the welfare of researchers, and none that we could identify in the Information System area. The purpose of this paper then is to investigate empirically IS researchers’ views on the effects of journal ranking systems on both IS research and IS researchers. To this end, we surveyed Information Systems academics who published in three key IS conferences in 2013, to gather their views on the effects of journal ranking systems. Overall, we found that the concerns in the literature were strongly reflected in the views of those surveyed. Academics believe the system has greatly increased their publication anxiety. While most believed that the quality of published research had improved, researchers believe the ranking systems inhibit innovative, risky research, and encourages safe, conforming, mainstream research. The remainder of the paper is structured as follows. Sections two and three discuss the possible impact of the use of journal ranking systems on the discipline and the individual researcher. We then describe the empirical study and present the findings in sections four and five. This is followed by a discussion of the findings, and finally we conclude the paper by suggesting further research and a stronger role for the IS discipline leadership. 2 Possible Consequences on Research 2.1 The potential effects of ranking on disciplinary research Mingers and Willmott (2013) point out most academic work is now shaped by performance measures. It seems highly improbable that the measurement of research will have no appreciable unintended effect on research itself (van Dalen & Henkens, 2012). Indeed, there are two possible opposing effects. The first possible effect is that ranking will have a positive, salutary effect where competition and measurement breeds and rewards excellence: the top journals will demonstrate exemplary research and rigour that direct junior researchers to the most important areas of research and the methods that are best used to research them. The second is that the ranking systems will result in undesirable effects on research directions, methods and topics as researchers pursue outcomes that are measured rather than outcomes that are desirable from a discipline perspective. In other words, they will pursue whatever is Australasian Journal of Information Systems Coulthard & Keller 2016, Vol 20, Research Article Publication, Journal Rankings, Researcher Views 3 required to publish in high ranking journals but not necessarily pursue high quality innovative research. The pivotal question then is whether the journal ranking systems can promote innovative, ground-breaking research in important areas when the driving force underpinning how the ranking system operates is the use of the system for promotion, tenure, and recruitment and the marketing and advancement of the university. The answer appears gloomy. Published opinion uniformly warn that journal rankings and in particular their coupling to performance management will harm research and in particular will stymie innovation and heighten conformity and ‘incrementalist’ research within the disciplines (Alvesson & Sandberg, 2013; Lincoln, 2012; Macdonald & Kam, 2007; Ozbilgin, 2009; Powell & Woerndl, 2008; Tourish, 2015; van Dalen & Henkens, 2012). The key allegations of harmful outcomes to research include: the end of important but risky or time consuming research; incrementalism and the promotion of orthodoxy, uniformity and reduction of diversity; the reduction of qualitative research; and fewer regional and interdisciplinary studies. Arguably the most serious allegations are the ‘end of important work’ (Powell & Woerndl, 2008; Tourish, 2015) and the rise of ‘incrementalism’ (Alvesson & Sandberg, 2013; Lincoln, 2012). Essentially, the argument is that academics will begin to research areas that they think are publishable, less risky, with good publishable returns for effort rather than in areas that are theoretically important but may be difficult, time consuming, or risky. The fear is that academics will be encouraged to undertake a form of cost benefit analysis for their career advancement and choose areas and methodologies that are least risky and most likely to be published for the least effort. ‘Smart’ research may be to ride on the coat tails of mainstream trends and prevailing orthodoxies, rather than contest these orthodoxies and risk not finding a high ranking publication outlet (Macdonald & Kam, 2007). On the other hand, pursuing new ideas or difficult but important areas is high risk and more likely to be full of ‘cul de sacs’ (Lincoln, 2012). In sum, there is a real fear that blue sky research will be eschewed in favour of risk averse, incremental, orthodox work. As Hassan (2014) quipped, "After we have had the satisfaction of publishing in MISQ or ISR, and at least after making tenure and Full Professor, it is time to start thinking about researching big ideas that make a difference". The problem though is that this may be too late. The full professor, trained in, compliant with and rewarded for his or her orthodoxy may be either unwilling or unable to undertake blue sky research. Grover and Lyytinen (2015) in their recent and important work on “epistemic scripts” demonstrate an already existing tendency in Information Systems towards orthodox, incremental research and the publishing rewards that are attached to such research. Epistemic scripts according to Grover and Lyytinen refers to a ‘middle range theory’ that appropriates an abstract concept from reference or base disciplines and seeks an instance of the concept in an IS related context. This establishes a form of institutionalisation of middle range theories or scripts such that subsequent scholars tinker with the theory and produce a vast body of incremental research that produces publishable work and career security. Following the script, scholars mostly produce knowledge that seeks to get through reviewers looking to check boxes on theory and method. The name of the game is conformity with the received theoretical insights, and the truth is largely determined by the work’s coherence with received theory and less by its novel insight, value, or utility for the field. Overall, examples of how to enact this script with dexterity are not hard to find; they are commonly taught as part of core Ph.D. courses in IS curricula. (Grover & Lyytinen, 2015, p. 275) The history of TAM research epitomises the concept of the script (Benbasat & Barki, 2007; Grover & Lyytinen, 2015). The rise of journal rankings may exacerbate this tendency within the discipline. By virtue of the rankings systems, the importance of the top journals in terms of determining the directions, methods, and topics in a discipline is increased while the importance or impact Australasian Journal of Information Systems Coulthard & Keller 2016, Vol 20, Research Article Publication, Journal Rankings, Researcher Views 4 of lower ranked journals is reduced and thus ranking systems may act to reduce the diversity of journals and promote orthodoxy (Tourish, 2015). For example, Macdonald and Kam (2007) warn that top journals in management studies encourage ‘homogenous research’ that ‘can impose a deadening uniformity on its field of endeavour’ (Harley and Lee, 1997, p. 1434 cited in MacDonald & Kam 2007 p. 647). Lincoln (2012) has also expressed deep concern that qualitative work will suffer as a consequence of journal ranking systems since qualitative work is generally more researcher labour intensive than other methods and has fewer higher ranked outlets. There is some empirical support for a reduction in interpretivism in Information Systems. In our study of three key journals, we found a twelve percent decline in interpretivist research between 2004- 2006 and 2007-2009 and a corresponding fifteen percent increase in positivist studies (Keller & Coulthard, 2013). Others too have argued that under-resourced areas will fail to get the attention they deserve as researchers pursue better resourced areas published in the top journals (Powell & Woerndl, 2008). As early as 2004, Chen and Hirschheim expressed the fear that publication pressure would lead to a reduction in methodological pluralism and less meaningful research in information systems since ‘under the pressure of tenure and promotion, researchers would tend to choose a less time consuming approach - positivism - to avoid perishing’ (p. 225). Methodological pluralism and research diversity are critical for information systems. Information technology and our relations to technology are constantly changing and developing and we must adapt our methodological approaches and research problems accordingly. However, on pragmatic career grounds it may become tempting to simply adopt the current orthodoxy and research problems. It is one thing to advise a candidate to study a safe topic with an orthodox method for their PhD but quite another to advise an academic to do the same until they ‘reach full professorship’. It may be the case that canny scholars have always followed fashion and were incremental in their research rather than iconoclastic and heterodox. Kuhn invented a term for this: ‘normal science’ (Kuhn, 1970). However, the rise of formal ranking systems and associated managerial setting of targets and performance monitoring, make these pressures more pointed, the possible publishing outlets more limited, and the competition for high ranked publications greater. The stakes of the research ‘game’ have increased substantially. A further allegation is that regional studies may suffer in the competition for top journals. IS has been accused of having a North American bias (Gallivan & Benbunan-Fich, 2007) and that North American authors predominate in the top journals (Truex, Cuellar, & Takeda, 2009 , Keller & Coulthard, 2013). Indeed, Tourish (2015) has identified that 31 out of the top 33 journals in the rankings produced by the Association of Business Schools in the UK were of US origin. Research in information systems is highly contextual. A study of an implementation failure in Bangladesh is arguably less interesting to the top western journals, and the majority of its readership, than one in Sydney but even that may pale in interest to a failure in Manhattan. In other disciplines at least, there is evidence that citations decrease with a reference to a specific country in the title (Jacques & Sebire, 2010). Cross-disciplinary work and international collaboration may, in a similar fashion, also suffer (McNay, 1998; Rafols, Leydesdorff, O’Hare, Nightingale, & Stirling, 2012). The problem stems from the ranking and recognition of particular journals within disciplines, institutes, and countries. For instance, if an IS researcher wished to collaborate with an academic in the health sector, a journal would need to be found that ‘counted’ across both disciplines. If the collaboration takes place across institutions or across international borders the search for acceptable outlets becomes even more difficult (Hussain, Liu, Wang, & Zuo, 2015). For Information Systems, this is particularly concerning since information and technology now underpin “most areas of both research and social life,[so] confronting the problématiques of the new century may require a more transdisciplinary perspective on the study of information and systems” (Hovorka, 2010, p. 12). Australasian Journal of Information Systems Coulthard & Keller 2016, Vol 20, Research Article Publication, Journal Rankings, Researcher Views 5 Macdonald and Kam (2007) point out that journal ranking and performance monitoring systems are also likely to place increased pressure on academics to publish in top listed journals – and undervalue, ignore or even deride lesser-ranked journals. This emphasis limits publishing outlets and increases competition among academics for publishing in higher ranked journals and effectively reduces diversity of publication and research areas, since academics are ‘encouraged to write the sort of papers and perform the sort of research these journals favour’ (Macdonald & Kam, 2007, p. 647). Journal ranking systems may therefore inadvertently increase the importance of the top journals in terms of determining the directions, methods, topics, and research thus defining research orthodoxy. This could be exemplary but as we have seen, many argue that it can also be overly conservative and restrictive. Often ground breaking work comes from younger researchers publishing in newer, and therefore lesser ranked journals, that may be among other things nonconformist, exploring new, less established, and emerging research problems not (yet) recognised by the highly ranked journals. This potentially fertile field of innovation and perhaps less polished talent is discouraged through low rankings. Again, in a frontier discipline such as Information Systems that is particularly concerned with the application of innovations in technology this may be particularly detrimental. Finally, journal ranking systems may also act to reduce the diversity of journals by effectively providing barriers to entry of new journals and neutralising the threat of disintermediation of established publishing houses. The internet threatens journals with disintermediation through self or school/university publication or at the very least fierce competition. Scholars could use the web, informal peer networks and social media to find articles of interest and weigh and test their value according to their scientific viewpoint, interests, and specialisations. However, with journal rankings, where something is published has become critical for the researcher (Lamp, 2009). This provides a barrier against disintermediation, and a marketing opportunity for established commercial publishers to trade on the prestige that the rankings systems provide them. Indeed, the top journals will themselves be under increased pressure to maintain their high status and may even restrict papers to increase scarcity. Journals are increasingly branding, promoting, and reporting on the impact of their journals on promotion and tenure (Janvrin, Lim, & Peters, 2015). As Nobel Prize Winner, Randy Schekman angrily exclaimed of the most highly ranked science journals: These journals aggressively curate their brands, in ways more conducive to selling subscriptions than to stimulating the most important research. Like fashion designers who create limited-edition handbags or suits, they know scarcity stokes demand, so they artificially restrict the number of papers they accept (Schekman, 2013). Others have accused journals of developing dual submission policies – inviting articles from highly cited authors to ensure high citation of the journal while a stringent review process ensures high failure rates for other authors (Alvesson & Sandberg, 2013). Journals have also been accused of internal and coercive citation, that is, forcing or otherwise encouraging authors to cite their journal and reviewers (Butler & Spoelstra, 2015; Wilhite & Fong, 2012). Moreover, lower ranked journals may also embark on a form of ‘sanskritisation’ – emulating the topics, methods, and rigour of the top journals to maintain authors and readers; a good thing from the point of maintaining standards but bad from the point of view of diversity. Similarly, researchers will be doing the same as they compete for publication space with each other in the top journals. There may be something of an ‘arms race’ between researchers for publication. While innovation may be put to one side, increased resources – more extensive literature reviews, larger sample sizes, and so on – may come to the fore and more polished, extensive and better-resourced research from established researchers may be increasingly published in the face of competition. 3 Possible Consequences on the Individual Researcher Having detailed possible unintended consequences of journal ranking systems on the discipline we now turn our attention to possible impacts on the individual IS researcher. How Australasian Journal of Information Systems Coulthard & Keller 2016, Vol 20, Research Article Publication, Journal Rankings, Researcher Views 6 the researcher is affected or believes he or she will be affected by the rankings will in turn affect the outcome of the journal ranking system as well as having an impact on the academics themselves. Academics have always had an incentive to publish and experienced a form of ‘publication anxiety’. The catchphrase ‘publish or perish’ encapsulates this anxiety and incentive. We have seen the argument that the increased incentives to publish afforded by the new journal ranking systems may negatively affect the content of research. The question now becomes one as to whether the journal ranking systems produces additional stress and anxiety rather than an additional positive incentive. The potential for additional stress must be considered in the light of the unprecedented level of accountability and transparency of academic activities (Mingers & Willmott, 2013). Many activities have become transparent, accountable and calculable. These activates range from conformance to strategic directions and operational plans of the newly incorporated university, to HR performance plans and appraisals, teaching and learning measures and evaluations, research objectives and KPIs, course compliance monitoring and ‘quality assurance’. Such measures now direct and manage the performance of each academic and the performance of the university overall (Mingers & Willmott, 2013). Within this performative context, increased publication anxiety is likely to provide a powerful social psychological force on academics since employment, job retention, and career prospects are important motivators, and job security has long been recognised as one of the most potent stressors in the workplace (Brief & Atieh, 1987). Whether the pressure to publish and indeed to generate research income is seen as a stressor or a challenge depends in large part on the extent to which a person feels in control of the outcomes (Ganster, Kiersch, Marsh, & Bowen, 2011). Academics may have some control over the number of papers they submit, however, most academics have far less control over the outcomes of publication, or research grant submissions or even what counts as a publication. In Australia, for instance, conference proceedings do not ‘count’ towards research quotas. In the face of fierce competition for journal space and high rejection rates it is reasonable to think there may be a real fear amongst academics that many good research papers are rejected by reviewers who do not like or understand the work (Armstrong, 1997; Smith, 2006), by editorial boards who favour "strong empirical work that informs practice" (George, 2014, p. 1) over carefully considered theory development or possibly because there is insufficient room in the journal and that other papers are deemed superior or more closely aligned with journal, its editors and reviewers. Such factors are largely out of the researcher’s control. Given these pressures it seems likely that scholars will be considering closely what sort of articles get published in the highly ranked journals. Academics may undertake research on topics and use research methods they believe will be rewarded by publication in top journals, or engage in other activities that they perceive will favourably influence editors and reviewers of top journals. In Australia, there is talk of academics seeking collaboration with American academics in order to increase their chances of getting published in highly ranked US journals. Stiff competition for publication may also encourage cronyism and gaming the system (Gallivan & Benbunan-Fich, 2007; Macdonald & Kam, 2007). Knowing, citing, and collaborating with senior editors and established researchers may become increasingly important, if not essential, to publish in the top journals, and ‘dirty tricks’ to increase ones’ publication rate will rise. ‘Dirty tricks’ to improve publication include: self-plagiarism, self- serving citation (i.e., citing for reasons other than scholarly value), ‘salami slicing’ or ‘milking’ (artificially breaking studies into a number of publications to increase publication record), misrepresenting or fabricating results, multiple author cartels and including established researchers on papers where they have done little or no work (Abraham, 2000; Butler & Spoelstra, 2015). Johnston and Riemer (2014) describe these activities as a ‘pernicious’ form of complicity by academics in the journal ranking systems where each academic and manager pursue locally optimal objectives. Australasian Journal of Information Systems Coulthard & Keller 2016, Vol 20, Research Article Publication, Journal Rankings, Researcher Views 7 But we do not want to paint this crucial step of linking simplified measures to operational activity as a management conspiracy. Much more pernicious systemic forces than top-down power are at work here. Academics at all levels themselves subtly conspire in the simplification and reduction of the practice they love. The reason is that once these simplified and supposedly objective measures become embedded, it becomes locally optimal for a wide range of actors to make use of these measures in various political and self-interested ways. Academics present their achievements using them, selection committees compare candidates using them, promotion committees use them to set standards, PhD supervisors use them to focus coursework content and select projects, journal editors use them to position their journals. Each of these actions has local validity derived from the peer review system. However, the unintended effect of the collective use of these metrics as a currency is to increasingly endow them with an objectivity that is unjustified when their use is removed from specific academic contexts (p. 852). This complicity may be driven by publication anxiety generated and exacerbated by the journal ranking system. From this point of view, the potential increase in publication anxiety may well become the central driver of research and publications, and the intrinsic merits of research problems may take a backward step in importance. Similar to our failure to find strong academic support for the journal ranking system of research, we found no academic support for the positive effects of the ranking system on the researchers themselves. Rather, there is some evidence that the pressure to publish is having a deleterious effect on the wellbeing of individual researchers. A recent survey of Dutch medical scientists found that a substantial proportion experienced publication pressure which negatively affected their work (Tijdink, Vergouwen, & Smulders, 2013). While publication pressure may be only one of many pressures facing academics in the current environment it nevertheless causes immense pressure since academics who do not perform well may be ‘performance managed’ out by university managers (Sangster, 2015). This was tragically illustrated by the suicide of a high profile UK academic after he was placed on performance review following a number of failed grant applications (Parr, 2014). Less extreme but still worrying is the growing level of workplace stress amongst academics. For instance, in a recent UK academic well-being survey, more than half of the 14,677 respondents indicated high or very high levels of stress, and almost a third reported they often or always experienced unacceptable levels of stress (Kinman, 2014). Workplace stress is a major contributor to serious health problems (Ganster et al., 2011) and attrition. Indeed, a 2009-10 survey of 1,414 academics from eight institutions in Victoria, Australia, found that less than 30% of respondents would stay at their current institution if they were offered a decent job elsewhere (Fredman & Doughney, 2011). The risk of the loss of highly skilled academics and the possible failure to attract the brightest students should be a concern for governments and society more broadly. Finally, the emphasis on measurement of the performance of the individual researcher clouds the issue of institutional and collegial support for the researcher’s output. It assumes something of a level playing field for researchers and their disciplines. Research is a collective enterprise requiring enormous social capital and physical resources. Such support may range from informal discussions and a strong research culture to access to research grants. Such support would generally be outside the control of the individual researcher but he or she will feel its effects in their performance review. Overall, there are strong arguments that the negative consequences of the emerging journal ranking systems will outweigh the possible benefits such as the identification of exemplary research, clearer career strategies, and more transparent management of recruitment, retention, and rewarding academics by university managers. We further did not find any support in the literature for the use of journal rankings by academics apart from providing guidance to academics of where to publish for furthering their careers ("Senior Scholars' Basket of Journals," n.d.). Australasian Journal of Information Systems Coulthard & Keller 2016, Vol 20, Research Article Publication, Journal Rankings, Researcher Views 8 4 Empirical Study While academic opinion seems uniformly bleak, empirical evidence is scarce (Hammarfelt & de Rijcke, 2014). There have been few attempts to investigate the effects empirically and none that we could identify in the Information Systems area. The assertions of a negative impact may be strong but need to be tested particularly as there appears to have been few political or university administrative changes as a result of academic opinion and argument. The best and most obvious means of testing is to undertake a form of review or content analysis of the change. Perhaps undertaking a content analysis of journal articles by experts utilising various categories such as ‘orthodox’ and ‘innovative’ and chart any change over time. The main problem with this is that by the time negative trends become obvious extensive damage will have already occurred. The views of academics can provide a good indication of the degree and direction that any pressure may be having on research as a consequence of that pressure and whether they feel that have the necessary support and resources to enable them to meet the increased demand for research outputs. While it may be said that their views may be influenced by the critical opinions expressed in journals, the same influence may well lead them to rethink their research for tenure and promotion and thereby bring about the very changes feared by academics. The foregoing suggests a number of empirical questions concerning the effects of journal rankings on research and the researcher. Beginning with research, the first area of questioning concerns what IS researchers think have been the effects of the journal ranking systems on research and on the journals themselves. A second area of investigation concerns whether researchers feel the pressure to publish is increasing as a result of journal rankings and their role in performance management, and whether they feel they have the resources, mainly time and access to research grants, to support them. To undertake the empirical study, we canvassed the views of IS researchers to discover whether they believe that top journals are becoming more important and if they think that the rankings systems strengthen current orthodoxy. And also if they feel under (greater) pressure and whether they believe this pressure is the result of journal rankings. In summary, we asked IS researchers who published a conference paper at one of the three IS conferences their views on these matters in five areas:  researchers’ views on the nature of ‘top journals’;  researchers’ perceptions of how ranking systems have influenced the type of research that is published;  researchers’ views regarding the usefulness of journal ranking systems.  the extent to which researchers feel under pressure to publish; and  the extent to which researchers feel they have the support and resources to undertake research. 4.1 Research Design To obtain a sample of active IS researchers we emailed all authors of research papers published in three information systems conferences during the latter half 2013. These were the Australasian Information Systems Conference (ACIS), the European Conference of Information Systems (ECIS) and the International Conference of Information Systems (ICIS). IS conferences form the ground-spring of research and provide an opportunity to survey a variety of active researchers who are likely to aspire to publishing in highly ranked journals and may already have done so. We also wished to survey researchers from various regions around the world and these three conferences tend to attract researchers who are geographically located close to where the conferences are held. That is, ACIS is usually held in Australia so it tends to attract researchers from Asia, ECIS is usually held in Europe so it tends to attract European scholars and the US based ICIS rotates between North America, Europe Australasian Journal of Information Systems Coulthard & Keller 2016, Vol 20, Research Article Publication, Journal Rankings, Researcher Views 9 and Asia. In 2013, ICIS was held in Europe and thus tended to attract less North American or Australasian researchers. Table 1 shows the totals of emails sent and returned for all three conferences. The survey was internet based and designed in such a way as not to compel people to answer all questions and we include the results of partially completed surveys too. Conference Total Authors Total Successful Emails Total Returned* Return Rate ACIS 377 369 79 21.4 ECIS 585 560 101 18.0 ICIS 657 644 95 14.8 Total 1619 1573 275 17.5 * totally or partially completed Table 1: Survey Return Rates The survey was designed in five sections each with a number of 5 point Likert scale style questions focusing on: (1) pressure to publish; (2) resources available to the researcher; (3) views on the research characteristics of the top journals; (4) views on journal ranking systems and what they contribute; and finally (5) demographic characteristics such as publication history, stage in career and location of the researcher. An initial version of the survey was piloted within our school and improved based on the resultant feedback. We also invited respondents to include additional comments at the end of survey and 51 people, approximately 20 percent, did so. 4.2 Demographics of respondents The demographics for the respondents is shown in Table 2. Overall, 275 respondents answered at least part of the questionnaire. A large number of respondents (40%) have over ten publications and a large number also have not published in top journals (35%). Most of our respondents are male (71%) between the ages of 35 and 55 (53%) and early or mid-career researchers (60%), holding tenured positions (53%). Given that two out of the three conferences we surveyed were hosted in Europe it is not surprising that Europe is the largest group represented by respondents (47%). Australasian Journal of Information Systems Coulthard & Keller 2016, Vol 20, Research Article Publication, Journal Rankings, Researcher Views 10 Publication # None 1 2-4 5-10 >10 n=243 8.81% 10.69% 22.01% 18.24% 40.25% Top journal # None 1 2-4 5-10 >10 n=250 34.97% 11.66% 30.06% 12.88% 10.43% Experience Beginning Early career Mid-career Established n=246 16.15% 29.81% 29.81% 24.22% Position Contract Tenured PhDs / Postdoc Other position* n=247 12.6% 53.4% 28.7% 5.2% Region Europe Nth America Australasi a / Pacific Other region** n=250 46.6% 17.8% 23.5% 12.1% Age <35 35-55 >55 n=247 31.2% 53.4% 15.4% Gender Man Woman Neither n=247 70.9% 28.3% 0.8% * included industry representatives and honorary positions ** other regions included Africa and South America Table 2: Summary of Demographic Data 5 Findings Findings are presented in two sections. Firstly, we detail academics views on the effects of journal rankings on research, and then we present views on the effects of the journal rankings on the individual researcher. 5.1 Views on the Effects of Journal Rankings on Research 5.1.1 Researchers views on the nature of ‘top journals’ While some academics pointed out that top journals did not form a homogenous group and therefore found it difficult to consider top journals as a whole, the qualitative and quantitative responses show that many academics were able to do so. The findings show that most researchers (88%) believe that ‘top journals’ have a strong influence on the direction of the discipline. Table 3 also shows that 60 percent of authors believe top journals publish safe, incremental research which makes modest contributions to knowledge. Around 20 percent believe that new advances are reported first in the top journals but the majority (54%), did not believe so, as summed up by one respondent: They always publish 'safe' research, which can be easily cross-checked, and if you don't mind me saying a bit boring to read. So, in my experience, as the ranking gets higher, the text gets more boring, the research more outdated, and overall less significant or interesting in some way [R8]. Nearly two thirds of the authors believed that ‘mainly quantitative’ research is published. This is consistent with interpretivist fears expressed in the literature of not getting published (Lincoln, 2012). One third of authors believed that the top journals have the most innovative research, however more (42%) did not believe this was the case. In terms of encouraging diversity of publications in top journals, 42% of authors are unconvinced while only 31 percent agreed. Australasian Journal of Information Systems Coulthard & Keller 2016, Vol 20, Research Article Publication, Journal Rankings, Researcher Views 11 Statements about top journals Agree or strongly agree Top journals have a strong influence on the direction of IS research 87.7% Mainly quantitative research gets published in top journals 63.1% Mainly publish safe, incremental research with modest gains in knowledge 59.9% Top journals have the most innovative and ground-breaking research 33.7% (41.7%) Top journals are diverse in the regions and methodologies they publish 31.3% (41.7%) New advances in IS are mostly reported first in top journals 19.9% (54.2%) Notes: Percent in brackets refers to the number who strongly disagree or disagree with the statement. Not all items or questions were completed by all respondents. Unless otherwise stated, the sample size for an item ranged between 270 and 276 responses. Table 3: Authors’ views on the nature of ‘top journals’ Overall, the vast majority of researchers believe top journals strongly influence the direction of the IS discipline. However, it would appear that this influence is towards a research orthodoxy. In one sense, the journal ranking system is doing what was intended, providing guidance but it seems that for IS researchers the direction of that guidance is a narrowing to a particular orthodoxy. As one respondent commented: There is need for more plurality in information systems research. I am not sure if it is the result of journal ranking systems (they exist in other fields after all) or that top journals have a very narrow focus but gradually IS research is losing its potential to participate in public debates on critical contemporary social issues [R51] 5.1.2 Views on the effect of journal rankings on what and how researchers work Authors were asked to comment on their beliefs as to whether journal ranking systems have encouraged, inhibited, or had no effect on the type of IS research published (Table 4). The findings show a strong conservative effect with 82 percent believing that mainstream research is encouraged and 75 percent believing safe research is encouraged. Only 28 percent believed that the ranking system encouraged work on difficult problems while nearly 40 percent believed that the ranking system inhibited such research. One respondent summed up the majority view stating: Establishment of journal rankings drives behaviour of academics: they are looking into currently published research to be able to emulate it and increase chances of publishing. This encourages mediocre but technically well executed research and discourages innovative, risky and unusual research. [R20] It is however, not all one way, the results also show that most academics believe top journals encourage theory development and a significant number believe that the top journals encourage work on emerging issues and important problems (see Table 4). In addition, over half believed that resource intensive research is encouraged. However, a significant minority of nearly one quarter of academics believe that journal rankings inhibit research on emerging issues and important issues. Further, nearly 40% of respondents believe work on ‘difficult problems’ is inhibited. It is difficult to tease out what appears to be contradictory views here; a resounding view that mainstream, safe, conforming research is encouraged and research into difficult problems was not encouraged but at the same time many also believe that research on emerging and important problems is encouraged. We need to undertake further investigation but one resolution is to suggest that because of the power of the ranking system, what is written in the top journals is important and emerging because being in the top journals makes it so. Like Lewis Carroll’s Humpty Dumpty, top journals make ‘important and emerging’ mean what they choose them to mean. Australasian Journal of Information Systems Coulthard & Keller 2016, Vol 20, Research Article Publication, Journal Rankings, Researcher Views 12 Beliefs about affect journal rankings have on nature of research Inhibited Neither Encouraged Mainstream research 1.7 16.2 82.1 Safe, conforming research 6.7 18.8 74.6 Theory development 11.5 23.9 64.5 Resource intensive research 12.3 36.5 51.2 Research on emerging issues 23.2 27.8 48.9 Research on important problems 24.2 33.1 42.8 Research on difficult problems 38.5 33.2 28.3 Table 4: Effect of journal rankings on nature of IS research The survey results suggests that authors believe that journal ranking have influenced the focus of IS research with 57 percent believing it has encouraged research with a business focus (see Table 5). Less emphatically, authors also believe research on both social issues and technical issues is somewhat encouraged by journal ranking systems while applied and practical research is somewhat inhibited. Beliefs about effect journal rankings have on research area Inhibited Neither Encouraged Research on business 12.5% 30.2% 57.3% Research on technical issues 28.8% 32.2% 39.1% Applied and practical research 39.9% 32.2% 27.9% Social issues 21.9% 38.6% 39.5% Table 5: Effect of journal rankings on research area In terms of diversity in regions and methods, Table 6 shows that over a third of surveyed authors believe that research on developing countries and areas outside North America and Europe is inhibited (34% and 37% respectively). Less than 20 percent however believed that such research was encouraged by journal ranking systems. Researchers also believe that quantitative research is the big winner with 78 percent believing ranking schemes encourage this method of research while only 27 percent believe that qualitative research is encouraged. Effect journal rankings have on diversity of regions and methods Inhibited Neither Encouraged Research on developing countries 34.4 47.6 17.9 Research in areas outside North America and Europe 36.8 44.3 18.9 Qualitative research 36.6 36.6 26.7 Quantitative research 2.9 18.8 78.3 Table 6: Effect of journal rankings There were several gender differences identified. Women were significantly more likely to believe that qualitative research was inhibited by journal rankings – 55% compared with 30% of men (χ2=13.955, df2, p <0.001). Women were also significantly more likely to believe that quantitative research was encouraged – 89% compared with 68% (χ2= 6.846, df2, p<0.033). Finally, women were also more likely to believe that technically focused research (57% vs 33%) (χ2=10.999, df2, p<0.004) was encouraged and less likely to believe that applied and practical work was encouraged (42% vs 58%) (χ2=9.102, df2, p<0.011). Australasian Journal of Information Systems Coulthard & Keller 2016, Vol 20, Research Article Publication, Journal Rankings, Researcher Views 13 5.1.3 Beliefs about the use of journal rankings While academics were in general critical of journal rankings effects on research and the pressure they place on academics, only 13 percent believed that they were of little or no value (Table 7). Nearly half (46%) of the academics believed that the rankings had improved the quality of journals. The researchers also saw journal rankings as assisting in determining where to publish (82%), what to read (74%) and helping to avoid unscrupulous journals (60%). Comments on the benefits of ranking systems to academics included: This is an interesting and difficult topic. A key aspect of top journals is to provide rigorous research that provides a solid basis for future research. It is sometimes astonishing how quickly the quality of published research drops when one moves outside of the AIS scholars' basket of journals. There is a lot of research sold as "relevance-oriented" that is just not solid enough to invest in by designing studies that build on those works [R50] It has helped me when filtering papers to reference. If for example, I have two sources providing similar information, the one from a more reputable source is naturally used even though the message might be equally well explained in a journal of less rank. Thus, I think that journal ranking promotes itself, as these papers tend to be more referenced - not necessarily because they contribute more - but since they are safer and more regarded [R38] Beliefs about the use of journal rankings Disagree Neither Agree Assist in determining the journal to publish in 6.1 11.8 82.0 Provides a useful guide to quality literature 12.7 13.5 73.9 Provides little or no value 67.8 19.2 13.1 Assists in avoiding predatory journals 15.1 24.9 60.0 Increased the quality of publications 18.5 35.8 45.7 Primarily of benefit to managers of academics 24.3 30.0 45.7 Table 7: Beliefs about the use of journal rankings Curiously, nearly half (46%) believe that the journal ranking systems are primarily of benefit to managers. 5.2 Views on the Effects of Journal Rankings on the Researcher 5.2.1 Pressure to publish The survey results show that academics feel under enormous pressure to publish. As shown in Table 8, three quarters of academics place themselves under enormous pressure to publish and over 40 percent of academics are often worried that they will not get published and find worry a constant burden. It is as if the anxieties traditionally associated with undertaking a PhD continue on into academic working life. I put myself under enormous pressure to publish 75.8% I am often worried that I will not get my work published 43.4% Worrying about getting published is a constant burden 41.2% Table 8: Felt personal pressure (per cent agree or strongly agree) It is clear that much of the pressure comes from university and the journal ranking systems. As one respondent commented: I think the ranking of the journals now dictate where we need to publish and it’s really, really hard to get published in the top journals. Actually the pressure on writing papers creates heaps of stress. [R6] Australasian Journal of Information Systems Coulthard & Keller 2016, Vol 20, Research Article Publication, Journal Rankings, Researcher Views 14 Indeed, as Table 9 shows, only one in ten academics believe there is little pressure to publish from their university and nearly two thirds did not agree that there was little pressure to publish. Further, 90 percent believed that promotion and tenure were largely achieved by publishing in the top journals. Despite many universities trumpeting the importance of good teaching, only six percent of authors believed that good teaching was the best way to gain promotion or tenure. Academics consider themselves under more pressure to publish than a decade ago with only 5% believing this was not the case. Indeed, 70 percent of authors believe that journal ranking systems are responsible for the increase in pressure to publish and over 80 percent did not believe that journal ranking systems had little influence over their careers. There is little pressure to publish at my university or organisation 10.4% (62.7%*) Promotion and security of tenure is largely achieved by publishing in the top journals 89.5% Academics are under less pressure to publish than a decade ago 5.1% (63.9%) Good teaching not publications is the best way to get promotion or tenure 5.6% (82.7%) Journal ranking systems have led to increased pressure to publish 70.1% Journal ranking systems have little influence on academic careers at my university 11.2% (80.9%) * percentage in brackets refers to the number who strongly disagree or disagree with the statement Table 9: Organisational pressure (per cent agree or strongly agree) It is clear that authors feel under great pressure to publish and that much of this pressure derives from the universities and the importance of publications for promotion and tenure. Pressure to publish may also be exacerbated by a reduction of university sanctioned research areas and outlets. As one respondent stated: It infuriates me that my organisation is only willing to support certain research areas for publication and place incredible pressure on us to publish in high quality journals without additional time and compensation [R2] There were some differences in the level of worry among different researcher groupings. Perhaps not surprisingly, established researchers and researchers who had published more than ten articles were least worried. Consistent with Lincoln’s (2012) views of the effect of journal rankings on interpretivist research, researchers who identified themselves with interpretivism were more worried they would not get their work published. Women were significantly more worried than men that they would not get published (χ2= 17.11, 2df, p<0.000) and were more likely to feel under a constant burden (χ2= 10.386, 2df, p<0.006). Views of organisational pressure were fairly homogenous across different researcher groupings. However, European academics were more likely to agree that journal rankings have increased publication pressure than their North American colleagues (χ2=18.007 df6 p<0.006). This may reflect that over the past decade there has been ‘increased (coercive) institutional pressure to publish in the elite journals’ in Europe (Lyytinen, Baskerville, Iivari, & Te'Eni, 2007, p. 319). Notwithstanding this increased pressure, optimism reigned with most authors confident in their ability to get published (65%) while nearly half (46.3%) were confident that their research would get published in the top journals. There were no significant differences in confidence of publication by region but well established researchers are more confident of being published than other groups (χ2=24.449, df6 p0<.000). Women, on the other hand, were significantly less confident than men (χ2=13.240 df2 p <0.001) with only fifty one percent of women confident that they would get published compared with seventy two percent of men. Australasian Journal of Information Systems Coulthard & Keller 2016, Vol 20, Research Article Publication, Journal Rankings, Researcher Views 15 5.2.2 Resources to undertake research Access to resources and support to undertake research is equivocal (Table 10). The most scarce resource is time, with nearly half the authors believing that they did not have reasonable time to focus on their research; did not have sufficient time to read the literature; or have good access to research only semesters. Nearly seventy percent believed a significant amount of their research was unpaid, that is, they did their research in their own time and in excess of their paid hours. A significant amount of my research is unpaid (in excess of the number of hours I am paid to work) 69.8% I have good access to research-only semesters and study leave 24.5 (52.6*%) I have time to read deeply and widely to keep abreast of the key research developments discipline 23.4 (50.0*%) I believe I have reasonable time to focus on my research to produce quality research 33.7 (47.2*%) *percentage in brackets indicates those who disagree or strongly disagree with the statement Table 10: Time to undertake research (per cent agree or strongly agree) It is perhaps not surprising given the pressure to publish that academics want more time to research and make up time by undertaking a significant amount of their research out of hours. This would suggest that those who can afford to work additional hours will be in a better position to undertake research. Women are significantly more likely to report that they do not have sufficient time to undertake research (χ2=16.1, 2df, p<0.000) or time to read widely and broadly to keep abreast of the field (χ2= 7.51, 2df, p<0.023). This reflects that they tend to be more worried about publication but it is possible that it reflects women generally reporting greater family responsibilities than men (Cha & Weeden, 2014) which reduces the time they have available outside of normal working hours to undertake their research work. North Americans were more likely to agree or strongly agree (52%) that they had time to focus on their research than their Australasian (29%) and European (27%) counterparts (χ2=16.561 df6 p<.011). Academics who had published more than ten articles in top journal were also more likely to believe that they have time to undertake research (χ2=16.781 df8 p<0.032) and access to research semesters (χ2=16.865 df8 p<0.032). Access to grants is also equivocal, with 40 percent believing they had access to research grants but there being over a third of academics believing they did not have reasonable access is hardly an endorsement for good access to grants (see Table 11). Authors with more than ten papers published in the top journals were more likely to agree that they had adequate access (χ2= 21.83, df8, p<0.005). They also tended to believe that their school, department or discipline (47.8%) and their university were supportive (41.9%). However, sizeable proportions of around 30 percent of authors disagreed. The majority of authors did believe that the resources that were available to them were of an international standard (57.7%). The only consistent finding at this aggregate level is that schools and universities need to support research through an increased allocation of time. I believe I have reasonable access to research grants to publish quality research in the top journals 39.9% The university as a whole provides me with a strong supportive research culture 41.9% My school/department/discipline provides me with a strong supportive research culture 47.8% The resources I have available are of an international standard 57.7% Table 11: Resources to undertake research (per cent agree or strongly agree) Australasian Journal of Information Systems Coulthard & Keller 2016, Vol 20, Research Article Publication, Journal Rankings, Researcher Views 16 6 Discussion 6.1 Discussion on findings The findings of this study provide the first empirical support in Information Systems for the concerns expressed more broadly in the literature. In particular, that journal ranking systems are contributing to pressure to publish and publication anxiety (Tijdink et al., 2013) and that most academics surveyed feel they have insufficient time to undertake their research (Kinman & Jones, 2003). Further, the findings also provide empirical support for the assertion that ranking systems have inhibiting effects on innovative, risky research and encourage safe, conforming mainstream orthodoxy (Alvesson & Sandberg, 2013; Lincoln, 2012; Macdonald & Kam, 2007; Ozbilgin, 2009; Powell & Woerndl, 2008; Tourish, 2015; van Dalen & Henkens, 2012). The findings also support allegations in the literature that regional, cross-disciplinary (McNay, 1998; Rafols, Leydesdorff, O’Hare, Nightingale, & Stirling, 2012) and qualitative research is threatened (Lincoln 2012, Chen and Hirschheim 2004). The results show a clear fear that driven by pressure to publish and constrained by lack of time, IS academics will increasingly research what they see being successfully published rather than what is important. Further, researchers believe journal ranking systems and the top journals do influence how and what is researched and while this may provide guidance and instruction, it also provides a model of conformity or model for success. Nevertheless, it is clear that most academics also believe that journal rankings have increased publication quality and significant numbers of academics believe that the top journals do publish important work on emerging issues. Few find that the journal ranking systems are of little or no value. This appears at first to be a contradiction between academics saying, on the one hand; that the top journals are influential and publish work on emerging issues and that journal rankings are useful and have increased quality, while on the other hand giving quite a uniform critical assessment concerning the nature and diversity of research published. However, if we consider top journals and ranking systems as determining value and determining what is published by that fact they are important and influential, then almost by definition they report on emerging important issues. Those important issues are just not innovative or diverse but are safe and conforming. What the respondents meant by increased quality is unclear. However, it does not seem to mean an increase in innovative and ground breaking research. For the respondents, the increase in quality may refer not to more good research being published but simply less bad, poorly conceived and carried out research being published. Most tellingly, while academics find ranking systems useful, almost half think their managers find them more useful. Based on our findings, it would appear that the top journals tell them what is successful and important because that is what the top journals are publishing. There are particular groups who are more affected by publication anxiety and whose publications may fall as a result of the rise of journal rankings. Certainly it appears, in line with the literature (Lincoln, 2012), that qualitative researchers are most concerned. We also found that women were most worried about publication and reported that they had the least time to undertake research. Gaining time to undertake research is clearly an issue for nearly everyone but it may also be an important distinguishing issue between men and women for a career trajectory in academia. Cha and Weeden (2014a) in their study of US Current Population and Labour Force statistics found that: the ability for staff to work longer hours rests on a foundation that is itself highly gendered -- in other words, those employees who work longer hours do so only with the support of other household members, usually women, who shoulder the lion's share of household duties....women are less likely to choose positions where longer hours are embedded in the organisational culture, and the fact they are responsible for the majority of housework and childcare and therefore can't offer as much of their time to outside-hours (and paid) work (Cha & Weeden, 2014b, p. 16) Australasian Journal of Information Systems Coulthard & Keller 2016, Vol 20, Research Article Publication, Journal Rankings, Researcher Views 17 Women may have a strong reason to be worried. It may be the case that the ranking systems overall benefit those with the trifecta of talent, research time, and resources as well as those with better social capital – better research connections with influential researchers, ‘insider knowledge’ on how to get published and to play the ‘game’. 6.2 Cycle of strengthening orthodoxy It is interesting to consider the social psychological role publication anxiety may play in developing or strengthening safe incremental research. Driven by pressure to publish, constrained by lack of time and monitored by performance regimes, it is likely that academics will try to research what they see being successfully published. The chance of publication is naturally greater when they go with, rather than against, the mainstream and hedge their bets accordingly. It is in this way, academics are complicit in the outcome (Johnston & Riemer, 2014). The supply of innovation may be low at the best of times but the journal ranking system may make that supply even lower as academics play ‘safe’ and a vicious cycle arises. An anecdote illustrates this supply side problem. Just before a faculty meeting, when we had just completed our first data run and while waiting for the meeting to start, one of the authors was describing some of the initial findings. A senior academic across the table listened with interest. He then exclaimed in frustration that as a senior editor of an A* ranked publication he found that he rarely received, let alone published, innovative exciting material no matter how many editorials he wrote or who he appointed as associate editors. Since academics do not see much innovation and diversity in the journals they therefore do not believe that is the way to success despite the pronouncement of senior editors. A further break on innovation is that whereas emerging new journals provided an outlet for new, diverse topics (Lamp & Fisher, 2010), the ranking system may raise a significant barrier to entry – new journals will necessarily be unranked and as such more or less worthless to publish in. The systematic effect may be summarised as follows: the rise of performative measures establishes the performance ladder that the academic must follow in order to gain recruitment, tenure and promotion with an increasing emphasis on research output in selected ‘quality journals’. To succeed academics must identify the particular qualities of such journals and reproduce them as closely and efficiently as they can. With one eye on their career and the other eye on their available workload, the academic is likely to make a rational choice of what to research and how. While the intended effect of this may be to provide clarity, objectivity of performative measures and to improve quality by providing exemplary guidance to academic research, the actual effect is increased anxiety and loss of diversity of methods, journals and topics. This should be particularly worrisome for a discipline that is essentially a ‘frontier discipline’ investigating and exploring changes in technology, its use and effects (Yoo, 2010). 7 Conclusions and Further Research Overall, it would seem that academics believe the journal ranking systems and associated performative measures and recruitment and retention processes will lead to the impoverishment of research. Being an academic working towards publishing important work is quite different to publishing to survive. Here again we need to repeat that there has and always will be a tendency towards publishing mainstream research that offers the best opportunity for recruitment, retention and promotion. The point here is that the tendency has intensified in the minds of academics to an imperative. The performative net has tightened and the results strongly support the concerns with the ranking systems expressed by many of critical researchers in the literature. As we have seen too, there is increased stress on researchers, particularly those who work in qualitative areas but also those who may have reduced ability to work additional hours as it Australasian Journal of Information Systems Coulthard & Keller 2016, Vol 20, Research Article Publication, Journal Rankings, Researcher Views 18 would appear that most researchers claim to undertake their research in excess of regular office hours. This may affect women researchers particularly, who also are more worried or stressed over publication. We are aware that we a painting a rather cynical picture of the academic researcher, albeit one under great pressure. Nevertheless, the results are consistent with the view that the journal ranking system is principally external to the processes of research and serves to not only rank journals but to rank academics for recruitment, tenure, and promotion, to rank universities on research league tables, and to secure the ascendancy of established journals. The results suggest that the intrinsic benefits to research are small: a handy guide to the literature and the absence of poor quality papers. Such a view needs to be further examined. Rankings have become so important to academic careers, to universities, and to journals that they need to be examined as a priority to ensure that meritorious and diverse research is found, published, and rewarded. Ranking systems have globalised research and researchers are being measured and compete against their international peers and even across disciplines. The pressures to publish are likely to increase both ‘sharp’ research practices and publication tactics by all who are rewarded by publication or ranking systems namely researchers, reviewers, editors, journals and universities. The concerns and criticisms of journal rankings in the literature are very strongly reflected in the views of the researchers studied. Researchers do fear and perhaps even see a rise in safe, conforming work and the pressures to do so appear enormous. The question is whether academic practices will change in accordance with the concerns expressed in the literature and here in our survey. Key limitations of our study we believe are firstly the charge that ‘academics are simply repeating what they have read in the journals’. That is to say, influential critical researchers have swayed the discipline. Secondly, that criticism of the journal ranking system, a system that is intended to make academic work transparent and accountable, is self-serving. People generally do not like being monitored. There is no getting around these criticisms, but it does not make it untrue and at least gives an indication of the feelings and thoughts of academics on the matter. Most importantly, it is likely that academics will act on their views and become more conservative, incremental and mainstream in their work. Finding an objective or neutral standpoint to these changes is difficult if not impossible at least in the short term and without hindsight. What may be for one researcher an improvement in quality or the establishment of a clear research direction for another may be seen as a reduction in diversity. One cannot simply ask those most senior because, to use the well-worn phrase, the victors write the history. Research into this area will necessarily be partisan but it is no less important for that. Such studies and research can monitor the rise and fall of research methodologies and attempt through content analysis or expert panels to gauge changes in types of research and their theoretical and practical importance and impact. As we have suggested, for the discipline of Information Systems, the reduction of diversity may be particularly worrisome for a discipline engaged in a technological frontier that is constantly moving and which is cross disciplinary in nature. Should each discipline ‘look to their own disciplines’ for their journal rankings, cross disciplinary studies may suffer. Further studies may look on the changing nature of academic work, the rise of performativity and the impact that has on current and future academics. As well as measuring rising or falling levels of work satisfaction, such studies may examine the attractiveness of academic work - does it indeed attract and retain the brightest and the best? Future studies can also investigate the question of the diversity of authorship and diversity of publications. Such evidence that we have suggests a predominance of North American authorship (Truex et al., 2009, Keller & Coulthard, 2013). Will this increase or decrease? Will citations and authorship groupings diversify or become more tightly networked? As the stakes get higher and work performance and promotion more closely related to journal rankings, we may see the rise of plagiarism, false Australasian Journal of Information Systems Coulthard & Keller 2016, Vol 20, Research Article Publication, Journal Rankings, Researcher Views 19 authorship, citation and research networks and other practices aimed at increasing the researcher’s publication standing. Another area for research is whether the ranking systems do stymie new journals and effectively strengthen the brand of existing high ranked journals because researchers gain little institutional benefit for publishing in new journals and where one publishes trumps what one publishes. Certainly our findings suggest that this is likely, but it needs to be further explored. Now that the performative genie is out of the bottle it is difficult to push the genie back in or relax the expectations on academics in the ranking system. Further, despite many calls for better measurement systems, all or most systems are likely to be gamed. Of course, one further alternative is for the interview and promotion panels to read the articles. In summary, future research will involve further investigating:  the positive and negative impact of journal rankings on content and diversity of research;  the effects of journal rankings on research collaboration and publication for different research groups, including gender and region;  the occurrence of ‘sharp’ research practices to increase research publications;  the development of alternative internet publishing forums as outlets for research; and  the effects of performativity on recruitment, retention and advancement of academics. Our empirical study has only been on the supply side of publications – the authors. Studies of the demand side, the journals, publications companies, editors, reviewer, readers, and universities also need to be made. It is likely that similar pressures on the journals to provide ‘quality’ and to increase their rankings is occurring with zeal and as we have seen in the anecdote above, frustration. Academics are not in charge of the system but neither is anyone else fully in charge. Researchers are complicit in the production of orthodoxy (Johnston & Riemer, 2014). However, as we have seen perpetuating orthodoxy is in the interest of the individual researcher and calls for researchers to temper their immediate interests may ring hollow. What is required is a disciplinary approach to reducing the reliance on rankings in the discipline and the encouragement of new journals and new ideas and approaches. It is a matter of academic and disciplinary policy and it would seem an appropriate role for the Association for Information Systems, national bodies, and peak academic forums such as the Australian Professor and Heads of Information Systems to take charge and jointly investigate this matter through supporting research and consultation within the discipline. From such investigations a better system may be created; one controlled by the discipline itself rather than to any particular groupings within or outside of the discipline. References Abraham, P. (2000). Duplicate and salami publications. Journal of Postgraduate Medicine, 46(2), 67-69. Alvesson, M., & Sandberg, J. (2013). Has Management Studies Lost Its Way? Ideas for More Imaginative and Innovative Research: Has Management Studies Lost Its Way? Journal of Management Studies, 50(1), 128-152. doi:10.1111/j.1467-6486.2012.01070.x Benbasat, I., & Barki, H. (2007). Quo vadis TAM? Journal of the Association for Information Systems, 8(4), 7. Bourdieu, P. (1998, December). Utopia of Endless Exploitation: The Essence of Neoliberalism. Le Monde diplomatique. Brief, A. P., & Atieh, J. M. (1987). Studying job stress: Are we making mountains out of molehills? Journal of Organizational Behavior, 8(2), 115-126. doi:10.1002/job.4030080203 Australasian Journal of Information Systems Coulthard & Keller 2016, Vol 20, Research Article Publication, Journal Rankings, Researcher Views 20 Butler, N., & Spoelstra, S. (2015). The dark arts of academia – and why journals must do more to tackle the problem. Retrieved from http://theconversation.com/the-dark-arts-of- academia-and-why-journals-must-do-more-to-tackle-the-problem-35796 Cha, Y., & Weeden, K. A. (2014a). Overwork and the Slow Convergence in the Gender Gap in Wages. American Sociological Review, 79(3), 457-484. doi:10.1177/0003122414528936 Cha, Y., & Weeden, K. A. (2014b). Overwork and the Slow Convergence i n the Gender Gap in Wages. Retrieved from Retreived from http://mypage.iu.edu/~cha5/Youngjoo_Cha_files/Cha_weeden.pdf Chen, W., & Hirschheim, R. (2004). A paradigmatic and methodological examination of information systems research from 1991 to 2001. Information Systems Journal, 14(3), 197-235. doi:10.1111/j.1365-2575.2004.00173.x Cooper, R. B., & Zmud, R. W. (1990). Information technology implementation research: a technological diffusion approach. Management Science, 36(2), 123-139. Fredman, N., & Doughney, J. (2011). Academic dissatisfaction, managerial change and neo- liberalism. Higher Education, 64(1), 41-58. doi:10.1007/s10734-011-9479-y Gallivan, M. J., & Benbunan-Fich, R. (2007). Analyzing IS research productivity: an inclusive approach to global IS scholarship. European Journal of Information Systems, 16(1), 36- 53. doi:10.1057/palgrave.ejis.3000667 Ganster, D. C., Kiersch, C. E., Marsh, R. E., & Bowen, A. (2011). Performance-Based Rewards and Work Stress. Journal of Organizational Behavior Management, 31(4), 221-235. doi:10.1080/01608061.2011.619388 Grover, V., & Lyytinen, K. (2015). New State of Play in Information Systems Research: The Push to the Edges. Mis Quarterly, 39(2), 271-296. Hammarfelt, B., & de Rijcke, S. (2014). Accountability in context: effects of research evaluation systems on publication practices, disciplinary norms, and individual working routines in the faculty of Arts at Uppsala University. Research Evaluation, 24(1), 63-77. doi:10.1093/reseval/rvu029 Harvey, L. (2008). Rankings of Higher Education Institutions: A Critical Review. Quality in Higher Education, 14(3), 187-207. doi:10.1080/13538320802507711 Hassan, N. (2014). Value of IS Research: Is there a Crisis? Communications of the Association for Information Systems, 34 (1). Retrieved from http://aisel.aisnet.org/cais/vol34/iss1/41 Hovorka, D. (2010). Moving Beyond IS Identity: Concepts and Discourses. Paper presented at the 31st International Conference on Information Systems ICIS 2010, St. Louis, United States, 15th December 2010. Hussain, S., Liu, L., Wang, Y., & Zuo, L. (2015). Journal Rankings, Collaborative Research and Publication Strategies: Evidence from China. Accounting Education, 24(3), 233-255. doi:10.1080/09639284.2015.1037776 Jacques, T. S., & Sebire, N. J. (2010). The impact of article titles on citation hits: an analysis of general and specialist medical journals. JRSM Short Reports, 1(1), 2-2. doi:10.1258/shorts.2009.100020 Janvrin, D. J., Lim, J.-H., & Peters, G. F. (2015). The Perceived Impact of Journal of Information Systems on Promotion and Tenure. Journal of Information Systems, 29(1), 73-93. doi:10.2308/isys-50963 Johnston, R., & Riemer, K. (2014). On Putting the Score ahead of the Game. Communications of the Association for Information Systems, 34(1). Retrieved from http://aisel.aisnet.org/cais/vol34/iss1/47 Australasian Journal of Information Systems Coulthard & Keller 2016, Vol 20, Research Article Publication, Journal Rankings, Researcher Views 21 Keller, S., & Coulthard, D. 2013. Charting diversity and change in IS publications: a tri- continental journal analysis, Australasian Journal of Information Systems, 18(1), 5-35. Kinman, G. (2014). Doing More with Less? Work and Wellbeing in Academics. Somatechnics, 4(2), 219-235. doi:10.3366/soma.2014.0129 Kinman, G., & Jones, F. (2003). 'Running Up the Down Escalator': Stressors and strains in UK academics. Quality in Higher Education, 9(1), 21-38. doi:10.1080/13538320308162 Kuhn, T. S. (1970). The structure of scientific revolutions ([2d ed., enl ed.). Chicago: University of Chicago Press. Lamp, J. W. (2009). Journal ranking and the dreams of academics. Online Information Review, 33(4), 827-830. doi:doi:10.1108/14684520910985747 Lamp, J. W., & Fisher, J. (2010). ERA Distribution of Information Systems Journals. Australasian Journal of Information Systems, 16(2). Laudel, G., & Gläser, J. (2014). Beyond breakthrough research: Epistemic properties of research and their consequences for research funding. Research Policy, 43(7), 1204- 1216. doi:10.1016/j.respol.2014.02.006 Lincoln, Y. S. (2012). The Political Economy of Publication: Marketing, Commodification, and Qualitative Scholarly Work. Qualitative Health Research, 22(11), 1451-1459. doi:10.1177/1049732312457713 Lyytinen, K., Baskerville, R., Iivari, J., & Te'Eni, D. (2007). Why the old world cannot publish? Overcoming challenges in publishing high-impact IS research. European Journal of Information Systems(4), 317-326. Macdonald, S., & Kam, J. (2007). Ring a Ring o? Roses: Quality Journals and Gamesmanship in Management Studies. Journal of Management Studies, 44(4), 640-655. doi:10.1111/j.1467-6486.2007.00704.x Marginson, S. (2011). The new world order in higher education: research rankings, outcomes measures, and institutional classifications. In M. Rostan & M. Vaira (Eds.), Questioning Excellence in Higher Education: Policies, Experiences and Challenges in National and Comparative Perspective (pp. 3–20). Rotterdam, The Netherlands: Sense Publishers. McNay, I. (1998). The Research Assessment Exercise (RAE) and after: 'You never know how it will all turn out'. Perspectives: Policy and Practice in Higher Education, 2(1), 19-22. doi:10.1080/713847899 Mingers, J., & Willmott, H. (2013). Taylorizing business school research: On the 'one best way' performative effects of journal ranking lists. Human Relations, 66(8), 1051-1073. doi: 10.1177/0018726712467048 Ozbilgin, M. F. (2009). From Journal Rankings to Making Sense of the World. Academy of Management Learning & Education, 8(1), 113-121. doi:10.5465/amle.2009.37012185 Parr, C. (2014, November 27, 2014). Imperial College London to ‘review procedures’ after death of academic. Retrieved from https://www.timeshighereducation.com/news/imperial-college-london-to-review- procedures-after-death-of-academic/2017188.article Powell, P., & Woerndl, M. (2008). Time to stop researching the important things? European Journal of Information Systems, 17(2), 174-178. doi:10.1057/ejis.2008.11 Rafols, I., Leydesdorff, L., O’Hare, A., Nightingale, P., & Stirling, A. (2012). How journal rankings can suppress interdisciplinary research: A comparison between Innovation Studies and Business & Management. Research Policy, 41(7), 1262-1282. doi:10.1016/j.respol.2012.03.015 Australasian Journal of Information Systems Coulthard & Keller 2016, Vol 20, Research Article Publication, Journal Rankings, Researcher Views 22 Rijcke, S. d., Wouters, P. F., Rushforth, A. D., Franssen, T. P., & Hammarfelt, B. (2015). Evaluation practices and effects of indicator use—a literature review. Research Evaluation. doi:10.1093/reseval/rvv038 Rossi, F., & Geuna, A. (2015). University and the Economy: Edward Elgar Publishing. Sangster, A. (2015). You Cannot Judge a Book by Its Cover: The Problems with Journal Rankings. Accounting Education, 24(3), 175-186. doi:10.1080/09639284.2015.1055929 Schekman, R. (2013, 2013/12/09/). How journals like Nature, Cell and Science are damaging science. The Guardian . Retrieved from http://www.theguardian.com/commentisfree/2013/dec/09/how-journals-nature- science-cell-damage-science Senior Scholars' Basket of Journals. (n.d.). Retrieved from http://aisnet.org/?SeniorScholarBasket Tijdink, J. K., Vergouwen, A. C., & Smulders, Y. M. (2013). Publication pressure and burn out among Dutch medical professors: a nationwide survey. PLoS One, 8(9), e73381. doi:10.1371/journal.pone.0073381 Tourish, D. (2015). Some announcements, reaff irming the critical ethos of Leadership, and what we look for in submissions. Leadership, 11(2), 135- 141. doi:10.1177/1742715015577889 Truex, D., Cuellar, M., & Takeda, H. (2009). Assessing scholarly influence: Using the Hirsch indices to reframe the discourse. Journal of the Association for Information Systems, 10(7), 560-594. van Dalen, H. P., & Henkens, K. (2012). Intended and unintended consequences of a publish- or-perish culture: A worldwide survey. Journal of the American Society for Information Science and Technology, 63(7), 1282-1293. doi:10.1002/asi.22636 Wilhite, A. W., & Fong, E. A. (2012). Coercive Citation in Academic Publishing. Science, 335(6068), 542-543. doi:10.1126/science.1212540 Yoo, Y. (2010). Computing in Everyday Life: A Call for Research on Experiential Computing. Mis Quarterly, 34(2), 213-231. Retrieved from http://aisel.aisnet.org/cgi/viewcontent.cgi?article=2903&context=misq Copyright: © 2016 Coulthard & Keller. This is an open-access article distributed under the terms of the Creative Commons Attribution-NonCommercial 3.0 Australia License, which permits non-commercial use, distribution, and reproduction in any medium, provided the original author and AJIS are credited. http://creativecommons.org/licenses/by-nc/3.0/au/