Microsoft Word - 8 AJIS 14-2.doc Australasian Journal of Information Systems Volume 14 Number 2 June 2007 5 A RANKING LIST FOR INFORMATION SYSTEMS JOURNALS Julie Fisher Monash University, Australia Email: julie.fisher@infotech.monash.edu.au Graeme Shanks University of Melbourne, Australia Email: gshanks@unimelb.edu.au John Lamp Deakin University, Australia Email: John.Lamp@deakin.edu.au ABSTRACT There has been an increasing focus internationally on the quality and impact of research outputs in recent years. Several countries, including the United Kingdom and New Zealand have implemented schemes to base the funding of research on research quality. The Australian government is planning to implement a Research Quality Framework (RQF) in the next few years that will impact greatly on funding of research in Australian universities. A key issue for Australian researchers is how the quality and impact of research is defined and measured in their discipline areas. Although peer review is widely used to assess the quality of research outputs, it is expensive and labour intensive. Other surrogate quality measures are often used. This paper focuses on measuring the quality of research outputs in the information systems discipline. We argue that measures such as citation indexes are inappropriate for information systems and that the publication outlet is a more suitable indicator of quality. We present a ranking list of journals for the information systems discipline, and discuss the approach we have taken in developing the list. We discuss how the ranking list may be used in defining and measuring the quality of information systems research outputs, the limitations inherent in the approach and discuss lessons we have learned in developing the list. Australasian Journal of Information Systems Volume 14 Number 2 June 2007 6 INTRODUCTION The quality and impact of research outputs have received increasing attention internationally in recent years. Governments are concerned that “public money is invested in research of the highest quality that delivers benefits to both research communities and the wider community” (Australian Commonwealth Government 2006, p11). A further concern is to “identify and reward areas of research excellence” (Australian Commonwealth Government 2006, p11) enabling research funding to be more focused. We adopt the definition of research from the OECD (2002) that is used in the proposed Australian Research Quality Framework (RQF), “creative work undertaken on a systematic basis in order to increase the stock of knowledge” (Australian Commonwealth Government 2006, p12). Furthermore the quality of research is defined as its “intrinsic merit and academic impact” (Australian Commonwealth Government 2006, p12). The Australian government has planned to implement the RQF in the next few years and it will greatly impact funding of research in Australian universities. Several other countries, including the United Kingdom and New Zealand have also implemented schemes to base the funding of research on research quality. Defining and measuring the quality of research is a contentious issue and varies considerably between different discipline areas. In this paper we focus on how to define and measure the quality of research in the discipline area of information systems. There are several approaches to defining and measuring the quality of research. The traditional and widely accepted approach is academic peer review. This is used in the highest quality journals and conferences and in research grant assessments, and is based on the critical review of research outputs by peers within the discipline area. Peer review is labour intensive and time consuming so proxy indicators have been used in defining and measuring the quality of research. The most important of these is the quality of the journal (or conference proceedings) in which the research output is published. This may be measured using impact factors or ranking lists. In this paper we argue that for the discipline of information systems, the use of impact factors is inappropriate and we develop a ranked list of information systems journals for use as a proxy for the quality of research outputs. The paper is structured as follows. The next section discusses how the Australian government has proposed to measure research quality within the RQF. We then present a critical assessment of the three main approached to developing journal ranking lists. This is followed by a critical assessment of existing IS journal ranking lists establishing the need for an Australian journal ranking list. We then discuss the approach we have adopted in developing our information systems journal ranking list, and present the list together with evidence for the rankings. Finally we discuss how the ranking list may be used in defining and measuring the quality of information systems research outputs, the limitations inherent in the approach and discuss lessons we have learned in developing the list. RQF QUALITY MEASURES When announcing the RQF, the Honourable Brendan Nelson, then Minister for Education, Science and Training said “Once implemented, the RQF will provide the Australian Government with the basis for redistributing research funding to ensure that areas of the highest quality of research are rewarded.” (Nelson 2005, 3). The focus for the RQF was to be on the quality of the research and its academic and broader impact. Academic impact was described as how well the research was received by our peers and the broader impact was described as the impact and usefulness of the research outcomes on those the research was designed to assist. In 2006 there was a change in Australasian Journal of Information Systems Volume 14 Number 2 June 2007 7 Minister and the new Minister, the Honourable Julie Bishop, reinforced the government’s intention of introducing a scheme and argued that the RQF “would measure quality through a combination of metrics and review by domestic and international peers” Bishop (2006). In order to assess the quality of research outputs, a quality ranking scale of 1 to 5 has been proposed, where a ranking of 5 signifies that “The majority of research outputs were considered to be in at least the top 20% of research in their field internationally, with a significant percentage (>25%) in the top 10%. There was evidence of high international peer esteem and significant impact on the international academic community” (Nelson 2005). The most widely accepted measure of peer esteem and impact is the publication of papers reporting research activity in academic journals. Mort (2004) argues that “the overall quality of the Journal acts as a proxy for the quality of individual papers within that Journal.” In August 2006, the RQF Departmental Advisory Group released the RQF Guiding Principles (DEST, 2006). The main measure of research quality proposed in this document is as follows: 1. Citation measures (where relevant to the discipline) as: 1.1 Citations per publication listed in the Research Grouping’s ‘body of work’; 1.2 The proportion of publications in the ‘body of work’ which have citations that would put them in the top decile for the discipline (to be derived from Thomson Scientific (ISI) citation benchmarks); The RQF Guiding Principles does however provide that: 2. In those disciplines where citation measures cannot be readily applied, an alternative measure should be derived through assessment of selected research output from the Research Grouping’s ‘body of work’. These outputs should be assessed against benchmarks relevant to the discipline (for example, ranked journals, book publishers, conferences, performance venues); In this paper we argue that the use of Thomson Scientific (ISI) citation index is not a useful measure of research quality for the IS discipline and develop a journal ranking list as a step towards an alternative measure in accordance with the RQF Guiding Principles option 2 quoted above. APPROACHES TO DEVELOPING JOURNAL RANKING LISTS A number of approaches have been used to develop journal ranking lists. Three commonly used approaches are the Thomson Scientific (ISI) impact factor, the use of rankings produced by others and polling eminent scholars in the area for their ranking of journals. In this section of the paper we critically assess these three approaches and argue for the development of an Australian IS journal ranking list. Thomson Scientific (ISI) impact factor. The ISI is a measure of the frequency with which the “average article” in a journal has been cited. The impact factor of a journal is calculated by dividing the number of current year citations to the source items published in that journal during the previous two years. The impact factor can be used to provide a gross approximation of the prestige of journals in which individuals have been published (Thomson ISI, 2005). Listing with ISI is not a simple matter of notification. ISI aims “to provide comprehensive coverage of the world’s most important and influential research” (Thomson ISI, 2004). In order to undertake this they are guided by Bradford’s Law (Bradford 1934) which they interpret (Thomson ISI, 2004): Australasian Journal of Information Systems Volume 14 Number 2 June 2007 8 that the core literature for any given scientific discipline was composed of fewer than 1,000 journals. Of this 1,000 journals, there are relatively few with a very strong relevance to the given topic, whereas there are many with a weaker relevance to it. Those with a weak relevance to the given discipline or topic, however, typically have a strong relevance to another discipline. Thus, the core scientific literature can form itself around various topics, with individual journals becoming more or less relevant depending on the topic. Hence ISI confine themselves to tracking approximately 8,700 journals from all fields which they believe accounts for 85% of published research and 95% of cited publications. There is an element of irony in that Bradford’s investigations were motivated by his disquiet over the inadequate coverage of abstracting and indexing services. ISI recognises that this core is not static and their editorial team focuses on evaluating journals on an ongoing basis to determine their eligibility for membership of this core. While they may evaluate around 2,000 new journals per year, only 10-12% become members of the core, and although it is not explicitly stated in their document, it can be assumed that there is an ongoing rate of exit from their core. Publishers may submit journals for evaluation, but inclusion is subject to evaluation by the editorial team. Being listed is not simple, nor is it speedy. The use of ISI as a measure of quality for IS research would have a major impact on the information systems (IS) discipline. Table 1 below shows the number of IS journals by year of first publication and the percentage of those journals which are currently in the ISI index. The table was developed using data from the Index of Information Systems Journals (Lamp 2004) as at October 2006. A number of observations may be made from the table: • Approximately 50% of journals that began publishing 10 years ago (1990-1994) are in ISI • Approximately 80% of journals that began publishing 25 years ago (1975-1979) are in ISI • Approximately 90% of journals that began publishing 40 years ago (1960-1964) are in ISI Number of IS Journals and Percentage in ISI 0 20 40 60 80 100 120 140 19 40 - 1 94 4 19 45 - 1 94 9 19 50 - 1 95 4 19 55 - 1 95 9 19 60 - 1 96 4 19 65 - 1 96 9 19 70 - 1 97 4 19 75 - 1 97 9 19 80 - 1 98 4 19 85 - 1 98 9 19 90 - 1 99 4 19 95 - 1 99 9 20 00 - 2 00 4 Year of first publication 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% No of Journals Percentage in ISI Table 1: Information Systems Journals and the ISI Index Australasian Journal of Information Systems Volume 14 Number 2 June 2007 9 Overall only 36% of IS journals are on the ISI database. It should be noted that no Australian IS journals are presently in the ISI database and neither of the Association for Information Systems (AIS) flagship journals (Journal of the AIS and Communications of the AIS) are in the ISI database. Furthermore, most specialist IS journals have a history of less than 10 years. Clearly, ISI cannot provide a useful measure for RQF assessment of IS research quality as its coverage of the discipline is currently inadequate. Using citation indexing and impact factors derived from them have been criticised from a number of perspectives. Seglen (1997) lists 22 factors which he considers limit the utility of impact factors derived from citation indexing. Tellingly, he also notes that he has found little evidence of the “free ride” belief that publishing in a high impact journal will enhance the impact of an article. In a study focussing on citation indexing and ranking of IS journals Katerattanakul et al. (2003) found inconsistencies between the two measures. The continued validity of many of the assumptions on which links between citation indexing, impact factor and quality are based are also undermined by the growing number of open access journals. Open access publishing is showing up as a major influence on impact factor (Jasco 2006). Furthermore, the choice of which index to use (ISI, Google Scholar, Citeseer, ABI/Inform, or EBSCOhost’s Computer Science Index, to name a few) is a key issue. Using existing ranking lists There are a number of existing journal ranking lists for the information systems discipline and a variety of ways have been used to develop these lists (see Lowry, Romans and Curtis (2004) for a useful summary). Some important existing lists are include the Association for Information Systems (AIS) MIS Journal Ranking List (ISWorld 2006), The Rainer and Miller (2005) ranking list and the Journal Quality list of (Harzing 2006). Several Australian institutions have also developed their own internal IS journal ranking lists which tend to reflect the research proprieties of the institution rather than the broader focus of the whole Australian IS discipline. Both the AIS list and the Rainer and Miller list are composite and based on an average of other lists and present clear evidence for their structure. However both of these lists have a clear North American bias and do not adequately include European and Asia-Pacific journals. The North American bias problem has also been recognised by Katerattanakul and Han (2003) who found that European IS journals are generally underrated as compared with North American journals. This is not unique to the IS discipline: Mort et al. (2004) note that in the discipline of marketing the evaluation of journals has typically emanated from the United States and consequently “reflect the views of American academics”. Mort (2004) goes on to argue that marketing in Australia and New Zealand has continued to develop and mature, reflecting a dynamic fusion of the disciplines’ American, European and Asia Pacific characteristics”. This is similar to the IS discipline. The Journal Quality List is currently in its 23rd edition and collates eighteen different rankings of 909 journals in the field of management, including IS, however it does not attempt to produce a composite ranking list and has a European bias, not adequately including American or Asia-Pacific journals. An additional problem with journal ranking lists is that many were developed some time ago and do not reflect the current quality of journals (Rainer and Miller 2005) or include more recent journals. Furthermore, composite lists that synthesise other lists include some from many years ago with more recent lists, yielding unreliable results. Australasian Journal of Information Systems Volume 14 Number 2 June 2007 10 Polling Eminent Scholars Another common approach to developing journal ranking lists is to ask eminent scholars in the area how they would rank journals in their area. Mort (2004) sought input from the senior marketing academics in Australia and New Zealand on their perceptions of quality of 73 identified journals in the discipline of marketing. This approach has also been used in IS (Walczak, 1999; Whitman et al, 1999; Mylonopoulos and Theoharakis, 2001; Walstrom and Hargrave, 2001; Bharati and Tarasewich, 2002; Peffers and Ya, 2003). There are some difficulties in polling eminent scholars to create journal ranking lists. First, as noted by Uncles (2004), there are problems with drawing up the initial list of journals. The IS studies cited above varied from 23 to 326 journals. Some restricted themselves to refereed publications, while others included professional magazines (Lamp 2006). Another problem Information Systems also faces, is where we draw the boundaries of the discipline. Existing categorial schemes are unhelpful, as they do not adequately cater for the IS discipline (Lamp and Milton, 2003). Even within the discipline, there are differences on the definition of IS; the discipline draws on many reference disciplines and uses many different research methodologies, models and frameworks (Lamp and Milton, 2004). This lack of clear definition may explain the extreme variation in the number of journals used in the studies mentioned above. It also highlights a further problem: given this large range of journals publishing IS research, are we really able to assess the quality of journals such as, for example, IEEE and ACM journals as well as our colleagues in computer science? Uncles (2004) also notes the difficulty respondents face in ranking journals with which they are not familiar. PROCESS FOR DEVELOPING AN AUSTRALIAN IS JOURNAL RANKING Having examined the different approaches to developing an IS journal ranking list we decided on a combination of methods, including the development of a composite ranking list using a set of carefully selected existing lists and seeking feedback and confirmation from senior scholars in the Australian information systems community using email and discussion at a workshop of senior Australian IS academics. The process was undertaken in four stages. Stage one: identification of existing rankings The first stage involved identifying the ranking lists that would be used for the composite list. To do this we drew on our own knowledge of existing ranking lists. We also undertook a general Internet search for information systems journal rankings and finally approaches were made to other leading European and North American professors in the Information Systems. The ranking lists were selected to ensure that both North American and European lists were included and that the lists were recent and reputable. The initial journal ranking lists used to develop the first composite ranking list the rationale for their selection and the approximate number of IS journals each list ranked are described below. • The AIS represents the lead IS body internationally. The ISWorld ranking was created based on other journal rankings that had been published by a reputable IS academics (135 journals ranked). • Vienna University is a very reputable European based university that provides a European perspective. Their ranking includes all journals in the business field including IS (52 journals ranked) Australasian Journal of Information Systems Volume 14 Number 2 June 2007 11 • Lousiana State University (United States) ranking list was developed by a leading North American University (25 journals ranked). • London School of Economics, is a leading European University and their ranking list, provides another European perspective. The London School of Economics has a highly regarded IS Dept (25 journals ranked).. • St Gallen University: HSG-Habilliste this is a journal ranking list for the purpose of getting "Habilitation" at St. Gallen university. This list has been included again because it is from a leading European University (50 journals ranked). • Peffers and Ya (2003) published ranking (50 journals ranked) • Rainer and Miller (2005) published ranking (29 IS journals ranked) Stage two: initial ranking list drawn up The second stage of the process was to develop an initial composite ranking list. This involved the following: • Definitions of three tiers (named Tier 1, Tier 2 and Tier 3) were developed based on the existing IS journal ranking lists. Journals were assigned to one of the tiers based on their average rankings within the selected lists and an initial composite ranking list established. • This ranking was circulated to a small group of senior IS academics within Monash University for feedback and minor modifications were made. Stage three: feedback on the initial ranking list The third stage of the process involved seeking feedback and confirmation from senior scholars in the Australian information systems community. This involved the following: • The initial ranking list was circulated to the Heads of IS academic units and professors within Australia and we sought input from the Australian IS community. • A request for other Australian university rankings was made and three universities provided their rankings. These were incorporated into the composite ranking list and changes were made based on these rankings. • The updated ranking list was circulated within the Australian IS community prior to the Australian Council of Professors and Heads of Information Systems (ACPHIS) workshop held in Canberra in September 2006. Stage four: information and finalisation of ranking list The final stage of the process involved a discussion at the ACPHIS workshop in Canberra about the ranking list and the development process. The workshop was attended by approximately 30 senior Australian IS scholars. A number of outputs resulted from the discussion: • Definitions of tiers were discussed and confirmed. • Labels for Tiers were renamed as Level A, Level B and Non-ranked. • Journals that were considered to be outside the IS discipline were removed from the list (for example IEEE journals, International Journal of Accounting IS and the IBM Systems Journal. This is consistent with the journals Rainer and Miller (2005) identified as being Australasian Journal of Information Systems Volume 14 Number 2 June 2007 12 computer science or management oriented). It should be noted that these journals will be ranked in the ranking lists of other disciplines, for example computer science and software engineering. • It was also agreed that a further journal ranking list relating to electronic commerce (Bharati, & Tarasewich, 2002) would be considered. This was done but most journals in the list were not ranked highly and so not included. The new ranking list was distributed through the Heads of IS academic units and professors email list for comment. It was also made available on the ACPHIS website and the IS community were able to provide further feedback for a number of weeks. While we feel that the journal ranking list is current as at March 2007, we acknowledge that over time journal editorships change, new journals become available and journals change. The journal ranking list should therefore not be regarded as final. The list should be regularly revisit and updated based on changes in other ranking lists used in its development and through feedback within the Australian IS community. The approach we have taken overcomes some of the problems identified by Uncles (2004) and Mort (2004) in relation to the marketing discipline. The initial lists used were drawn from both North America and Europe to ensure the breadth of our discipline was well represented. The process used was an iterative one ensuring there was maximum input from the community. THE AUSTRALIAN INFORMATION SYSTEMS JOURNAL RANKING LIST The final ranking list for IS journals is presented below. Journals within each level are presented in alphabetical order. Evidence for the composite ranking is provided in Appendix B. We have not included the non-ranked journal list as this is large and IS academics will be aiming to publish in the Level A and Level B journals. Level A Journals These are the journals we all should aspire to publish in, but it is not expected that many scholars in Australia will have more than one or two articles in these journals. Those that do will have a deservedly outstanding reputation as the best in their field. Decision Sciences Decision Support Systems European Journal of Information Systems Information Systems Journal Information Systems Research Information and Management Journal of the Association for Information Systems Journal of Management Information Systems Management Science MIS Quarterly Level B Journals These are still top quality journals, and cover a broad range of fields, but they are not strictly in the top level. Good scholars can be expected to publish fairly often at this level. Australasian Journal of Information Systems Australasian Journal of Information Systems Volume 14 Number 2 June 2007 13 Behaviour and Information Technology Communications of the Association for Information Systems Data and Knowledge Engineering Database Electronic Markets Human computer interaction Information and Organisation (formerly, Accounting, Management and IT) Information Systems (Elsevier) Information Technology and People International Journal of Electronic Commerce Journal of the Association for Information Systems Journal of Computer Information Systems Journal of Database Management Journal of IS (ACCT) Journal of Information Technology Journal of Strategic Information Systems Journal of the Operational Research Society Scandinavian Journal of IS Premier Professional Journals These are high quality journals with broad circulation that are highly visible within the practitioner community and can potentially have high impact. Academy of Management Executive Communications of the ACM California Management Review Harvard Business Review Interfaces MIS Quarterly Executive Sloan Management Review DISCUSSION AND CONCLUSION The journals ranking list presented in this paper has been developed specifically for the Australian IS community. It includes a balance between North American and European influences and provides a useful basis for assessing the quality of research output. The ranking list may be used in two main ways. First, papers published in journals ranked in Level A or Level B may be assumed to be of outstanding or top quality respectively and the judgements used in research quality assessment exercises. Second, the list provides a useful guide for academics to target their research publications. There are a number of limitations inherent in the approach we have used to develop the list. First the quality of the list is dependent on the accuracy, completeness and timeliness of the list we have selected as the basis for the development of our composite list. Second, our list is current as of March 2007 but will rapidly become out-of-date unless periodically updated. A mechanism for updating the list needs to be developed. Thirdly, it is difficult for recently initiated journals to join the list until they are well established. There are a number of lessons we have learned in developing the list. First it is important to be very clear about the scope of the IS discipline and to decide which journals may be included and which Australasian Journal of Information Systems Volume 14 Number 2 June 2007 14 will be left to other discipline areas to rank. Second, individuals hold very strong beliefs about the quality of journals in which they publish and clear evidence needs to be provided to justify the list. Third, great care needs to be taken in developing the lists as journals not ranked at Level A or Level B may no longer be desirable places in which to publish for IS researchers. REFERENCES Bharati, P., & Tarasewich, P. (2002). Global perceptions of journals publishing E-Commerce research. Communications of the ACM, 45(5), 21-26. Bishop J (2006) Keynote Address: Knowledge Transfer and Engagement Forum Sydney June, http://www.dest.gov.au/Ministers/Media/Bishop/2006/06/b001160606.asp (accessed July 2006) Bradford, S. C. (1934). “Sources of Information on Specific Subjects.” Engineering 137 85-86 (Reprinted J Information Science (1985) 10(4) 173-180). DEST (2006) RQF Guiding Principles. Canberra, Department of Education, Science and Training (Excerpts from this can be found at http://www.dest.gov.au/sectors/research_sector/policies_issues_reviews/key_issues/research _quality_framework/rqf_development_2006.htm#rqf) Harzing A Journal Quality List [Online] Available: http://www.harzing.com/resources.htm#/jql.htm Accessed October2006 ISWorld Journal Rankings [Online] Available: http://www.isworld.org/csaunders/rankings.htm Accessed Oct 2006 Jacso, P. (2006). “Savvy Searching: Open access to scholarly full-text documents.” Online Information Review 30(5): 587-594. Katerattanakul, P. and B. Han (2003). "Are European I. S. Journals underrated? An answer based on citation analysis." European Journal of Information Systems 12: 60 -- 71. Katerattanakul, P., M. Razi, et al. (2003). “IS journal rankings versus citation analysis: consistency and concerns”. Ninth Americas Conference on Information Systems, Tampa, Florida. Lamp, J. W. Index of Information Systems Journals. Geelong, Deakin University. [Online] Available: http://lamp.infosys.deakin.edu.au/journals/ Accessed: August 2006 Lamp, J. W. (2006). “Recognition as a distinguishing characteristic of IS journals”. Australasian Journal of Information Systems, 13(2), 7-16. Lamp, J. W., & Milton, S. (2003). “An Exploratory Study of Information Systems Subject Indexing”. Fourteenth Australasian Conference on Information Systems, Perth Australia. Lamp, J. W., & Milton, S. (2004). “The Reality of Information Systems Research”. Second Workshop on Foundations of Information Systems, Canberra Australia. Lowry, P., Romans, D. and Curtis, A. (2004) Global Journal Prestige and Supporting Disciplines: a Scientometric Study of Information systems Journals, Journal of the Association for Information Systems 5(2), 29-77 Australasian Journal of Information Systems Volume 14 Number 2 June 2007 15 Mort, G., J. McColl-Kenndy, G. Kiel and G. Soutar (2004). “Perceptions of Marketing Journals by Senior Academics in Australia and New Zealand.” Austral Asian Marketing Journal 12(2): 51-61. Mylonopoulos, N. A., & Theoharakis, V. (2001). “Global Perceptions of IS Journals: Where is the Best IS Research Published?” Comm. ACM, 44(9), 29-33. Nelson Research Quality Framework: Assessing the quality and impact of research in Australia the Preferred Model September http://www.dest.gov.au/sectors/research_sector/policies_issues_reviews/key_issues/research _quality_framework/rqf_preferred_model.htm#The_RQF_Preferred_Model Accessed July 2006 Peffers, K. and T. Ya (2003). “Identifying and evaluating the universe of outlets for information systems research: Ranking the journals.” J Information Technology Theory and Application 5(1): 63-84. Rainer, K.and Miller, M. (2005):"Examining differences across journal rankings.", Communications of the ACM 48.2 91(4). Seglen, P. O. (1997). "Why the impact factor of journals should not be used for evaluating research." British Medical Journal 314(Feb 15): 498-502. Thomson ISI The ISI® Database: The Journal Selection Process [Online] Available http://www.isinet.com/essays/selectionofmaterialforcoverage/199701.html Accessed May 2005 Thomson ISI The ISI Impact Factor [Online] Available http://scientific.thomson.com/free/essays/journalcitationreports/impactfactor/ Accessed August 2006 Uncles, M. (2004). “Journal Rankings: How Much Credence Should We Gives Them?” Australasian Marketing Journal 12(2): 67-72. UQ Business School UQ Business School Journal Rankings [Online] Available: http://www.business.uq.edu.au/research/resources/jnl_rankings.phtml Accessed: October 2006 Walczak, S (1999) “A Re-evaluation of Information Systems Publications Forums” Journal of Computer Information Systems 40(1) 88-97 Walstrom, KA and Hargrave BC (2001) “Forums For Information Systems Scholars: III” Information and Management 39 117-124 Whitman, M. E., Hendrickson, A. R., & Townsend, A. M. (1999). “Research Commentary. Academic Rewards for Teaching, Research, and Service: Data and Discourse”. Information Systems Research, 10(2), 99-109. Australasian Journal of Information Systems Volume 14 Number 2 June 2007 16 APPENDIX A - IS JOURNAL RANKING LISTS USED Details of the ranking lists used to develop our composite ranking list are provided below. ISWorld / AIS (United States-based), the leading international information systems academic web site includes many IS and IS related journals. Their ranking scheme ranks journals individually and 124 journals are ranked. We ranked the top 12% (15) as Tier 1, the next 25% (28) as Tier 2 http://www.isworld.org/csaunders/rankings.htm Vienna University provides a European perspective and has extensive rankings of all journals in the business field http://bach.wu-wien.ac.at/bachapp/cgi-bin/fides/fides.aspx?journal=true;lang=DE Lousianna State University (LSU), United States, provides a North American perspective but with few journals ranked. LSU Journal Ranking - ISDS Department (December 2003) http://www.isworld.org/csaunders/rankings.htm London School of Economics (LSE) provides a European perspective, highly regarded IS Dept. Journals and conferences include only those ranked as ‘first ranked' of which there are 24 only. (Not available on-line - provided by Prof. Leslie Willcocks in September 2006.) APPENDIX B - EVIDENCE FOR COMPOSITE JOURNAL RANKING LIST It should be noted that by the end of Stage four there was a significant convergence of the rankings, that is, there was very little difference in how each of the journals was ranked between the individual lists. For this reason only the rankings where there were 50 or more journals ranked have been included, two are European and two are North American. Evidence for journals ranked in levels A and B is provided below. Level A Journals (listed in alphabetical order) AIS (IS World) Vienna Uni Univer sity of St Gallen Peffers & Ya (2003) Decision Sciences 1 NR B NR Decision Support Systems 1 A NR 1 European Journal of Information Systems 1 A A 1 Information Systems Journal 2 A A 1 Information Systems Research 1 A+ A+ 1 Information and Management 2 A NR 1 Australasian Journal of Information Systems Volume 14 Number 2 June 2007 17 Journal of Management Information Systems 1 A B 1 Management Science 1 A+ A NR MIS Quarterly 1 A+ A 1 Operations Research 2 A A NR Level B Journals (listed in alphabetical order) AIS (IS World) Vienna Uni Univer sity of St Gallen Peffers & Ya (2003) Australian Journal of Information Systems 3 B NR 2 Behaviour and Information Technology 2 B NR 3 Communications of the Association for Information Systems 2 B B 1 Data and Knowledge Engineering NR A A NR Database 2 A NR 1 Electronic Markets 3 A A NR European Journal of Operations Research 3 A A NR Human computer interaction 2 NR NR NR Information and Organisation (formerly, Accounting, Management and IT) NR B NR NR Information Systems (Elesvier) 3 B NR 2 Information Technology and People 2 B NR 2 International Journal of Electronic Commerce 2 A B 2 Australasian Journal of Information Systems Volume 14 Number 2 June 2007 18 Journal of the Association for Information Systems 2 B B 1 Journal of Computer Information Systems 3 NR NR 2 Journal of Database Management 2 NR NR 2 Journal of IS (ACCT) 3 NR NR NR Journal of Information Technology 3 A NR 3 Journal of Strategic Information Systems NR B B 2 Journal of the Operational Research Society NR A NR NR Scandinavian Journal of IS 4 B NR 3