Reviewed Article – Research and Impact A SYSTEMATIC REVIEW OF THE LITERATURE IN EUROPE RELATING TO CLINICAL LEGAL EDUCATION Rachel Dunn, Northumbria University, UK[footnoteRef:1] [1: Rachel is a third year full time PhD student at Northumbria University.] Abstract As my PhD research is European focused, looking at knowledge, skills and attributes development in live client clinics, I wanted to find all the European literature relating to clinical legal education. The aim of this research was to find all of the European literature surrounding clinical legal education available to me, to explore the kind of research published and to identify any gaps in knowledge. With an explosion of literature within the field, and more research undertaken every year, finding the literature which related to my research was challenging. To help aid this work I embarked on a systematic review, building on work by Tribe Mkwebu,[footnoteRef:2] systematically searching for peer reviewed articles. This research was initially presented at the European Network of Clinical Legal Education’s Spring Workshop, 2015, Northumbria University.  This article highlights the journey through this literature. Firstly, it explains what a systematic review is and how it can be used within mixed methods research. It then goes on to outline the methodology used and the number of articles sourced, excluded and synthesised. The analysis shows the amount of papers published before 2015 and their basic content. Finally, I discuss my reflection on the systematic review, what I thought went well and what didn’t, explaining how it was received at the Workshop.  [2: Mkwebu T, 'A Systematic Review of Literature on Clinical Legal Education: A Tool for Researchers in Responding to an Explosion of Clinical Scholarship' (2015) 22 International Journal of Clinical Legal Education 238 ] Keywords Clinical legal education; Europe; systematic review; law clinics; legal education Introduction Clinical legal education (CLE) is a vastly growing area for academic research worldwide. CLE, at its simplest ‘is a method of training law students by putting them in situations where they must apply the legal theory, principles, and doctrines they have studied in the classroom setting.’[footnoteRef:3] This method of teaching is extensively researched and published globally. Doing a basic search for materials online will bring you many articles from the US, UK, Australia and Canada. Other regions of the world, such as Africa and Asia, have begun to publish their research more over the last decade.[footnoteRef:4] The same cannot be said of Europe, this continent seeming to produce less research on their clinics. Thus, it was vital for my PhD research to explore what research is already available and what can be further developed. [3: Bucker A and Woodruff A, 'The Bologna Process and German Legal Education: Developing Professional Competence through Clinical Experiences ' (2008) 9 German Law Journal, p. 578] [4: The fact that clinics have grown in Africa and Asia can be seen in Bloch FS, The global clinical movement: educating lawyers for social justice (Oxford University Press 2010). Furthermore, from my searches I was getting many results for these continents with has led me to draw the conclusion that they have started to publish more. CLE started in Africa in the 1960’s and Asia in the last decade or so. Also, see Mkwebu T, 'A Systematic Review of Literature on Clinical Legal Education: A Tool for Researchers in Responding to an Explosion of Clinical Scholarship' (2015) 22 International Journal of Clinical Legal Education 238] This paper highlights where the European clinical research has come from and the methodology of the systematic review used to find it. The preliminary findings of this systematic review were presented at the European Network of Clinical Legal Education (ENCLE) Spring Workshop, held at Northumbria University in April 2015.[footnoteRef:5] I will refer to this workshop during this paper. It is appreciated that since this research was conducted there have been many more papers published in CLE within Europe. However, I wish for this paper to be used more as a guide of conducting systematic reviews within our field and how it can be used to further our collective research agenda. [5: https://www.northumbria.ac.uk/media/4738635/call-for-interest-encle-osife-newcastle-15-16-april-2015.docx] What is a systematic review? A systematic review, as Khan et al suggest, is, ‘a research article that identifies relevant studies, appraises their quality and summarises their results using scientific methodology.’[footnoteRef:6] The amount of research conducted and published into CLE has increased greatly over the years and the possibility of reading it all is not likely, if not impossible. A systematic review can combine research questions and key words in order to perform a rigorous search for literature, narrowing search results. [6: Kahn KS and others, 'Systematic Reviews to Support Evidence-Based Medicine: How to Review and Apply Findings of Healthcare Research.' (2003) The Royal Society of Medicine Press] There are key phases to a systematic review, formalised through the Cochrane Collaborative over 20 years ago, which were followed during this review. The key phases are: 1) ‘Mapping the field through a scoping review 2) Comprehensive search 3) Quality assessment 4) Data extraction 5) Synthesis 6) Write up.’[footnoteRef:7] [7: Jesson J, Matheson L and Lacey FM, Doing your literature review: traditional and systematic techniques (SAGE 2011) P. 108] The advantages of conducting a systematic review are quite strong, in a methodological sense. For example, it allows a comparison of research papers available in order to assess consistency of the research. Due to its robust nature and its methodological structure results are often more reliable or accurate.[footnoteRef:8] This is due to its scientific nature. Its purpose is to aid a particular question or hypothesis and test it. With my particular question in mind a systematic review appeared a better fit. To attempt to find all of the peer-reviewed literature regarding CLE in Europe, the scientific methodological way of searching, using key words and wide searches, meant that I was more likely to answer my research question, as opposed to using a narrative literature review. As I had searched in such a comprehensive manner, searching all of the databases I did and following the Cochrane Collaborative model, reliability would naturally flow from my results. By looking at all of the articles I could find I reduced the bias, which may sometimes be present in a narrative review, and added strict criteria to the articles used to answer my research question. Having this research question in existence I could keep my searches in line with what I actually wanted to know, creating the rigour needed to not stray from my research area. [8: Greenhalgh T, 'How to read a paper: Papers that summarise other papers (systematic reviews and meta-analyses)' (BMJ, 1997) accessed 28.04.15] There are limitations to this methodology in order to locate relevant literature, which will be discussed in more detail below. It is a particularly time-consuming process, which can take weeks to complete. However, the rigour and comprehensive nature of this research methodology ensures that essentially all the articles in a certain area on the databases are found. The comprehensive nature occurs because I am finding all of the relevant peer-reviewed articles published to answer my research question. When I say a systematic review is rigorous Piper suggests that: ‘When faced with any question, being able to conduct a robust systematic review of the literature is an important skill for any researcher to develop; allowing identification of the current literature, its limitations, quality and potential.’[footnoteRef:9] [9: Piper RJ, 'How to write a systematic literature review: a guide for medical students' (2013) NSAMR, University of Edinburgh, p. 2] This rigour applies to appraising the research as well as finding it. Finding the research and following the set structure means that all literature identified should be in the research area. At the appraisal stage articles are excluded which are not deemed to eligible for inclusion and those left are submitted to a weight of evidence test. This test seeks to assess the quality of the research that has been found from the systematic review, but is important to note that papers are not necessarily excluded due to appraisal.[footnoteRef:10] Rather, sufficient weight should be attached to them in light of the appraisal. [10: Jesson J, Matheson L and Lacey FM, Doing your literature review: traditional and systematic techniques (SAGE 2011) P. 108.p, 114.] Thus, this systematic review is not only identifying the literature already available in Europe, but also its strength, reliability and relevance to the research question. This is why it was chosen for this particular research. Narrative or systematic literature reviews? A narrative literature review differs greatly from a systematic review, and is considered the more traditional approach. They can be more critical, assessing theories, putting the research into context and discussing the background of the research in more depth. Articles are selected by the author, based on his opinions. There are advantages to doing a traditional literature review. For example, they can go into a deeper discussion about the research and its theory, following a trail of different authors, their academic discussions and how their research has developed. It can be argued that this provides a richer literature review, discussing certain elements of research in great depth. Other articles and materials for the review can be found in footnotes and bibliographies, one piece of work providing for another. However, narrative reviews have been criticised by academics. Garg, et al highlight an issue that, ‘It is sometimes uncertain whether the author of a narrative review selectively cited reports that reinforced his or her preconceived ideas or promoted specific views of a topic.’[footnoteRef:11] Thus, due to an author choosing what is included in their review, and having the freedom to exclude materials which may not reinforce their opinion or theory, we may not be getting the entire picture from a narrative review. This is the issue of bias. We only see what the reviewer would like us to see, potentially reducing the reliability of the review. Reviewers, after all, are experts in their field, with opinions about the publications which exist; it can be hard to separate that opinion from a literature review. I am not enforcing that every literature review must be systematic and narrative reviews are always unreliable, but a good, methodologically sound, systematic review eliminates the temptation to exclude material which otherwise should be brought to the attention of the reader. [11: Garg AX, Hackam D and Tonelli M, 'Systematic Review and Meta-analysis: When One Study Is Just not Enough' (2008) 3 Clinical Journal of the American Society of Nephrology, p. 253] Using a systematic review was the most appropriate literature review for my research question. A narrative review would not have produced the same results and I would have missed many articles if I had used this alternative. Using qualitative studies in systematic reviews Systematic reviews, traditionally, contain only quantitative studies within the evaluations of their materials. This is due to their scientific nature, and quantitative studies viewed as supporting a hypothesis or research question more appropriately. However, due to the total amount of papers sourced in this systematic review being conceptual and qualitative in nature it is necessary to discuss the use of these kinds of papers. As Petticrew states, ‘Qualitative research can identify the range and nature of impacts of interventions and can give sense of whether they are rare or common. It can identify unintended, unanticipated impacts.’[footnoteRef:12] He notes that systematic reviews containing qualitative research are becoming more common, and the impact that they can have should not be ignored. Qualitative studies can help us to answer research questions which may not be answered through quantitative studies, and including them in a systematic review may create a more reliable result to the research question. Petticrew maintains his argument with Roberts, advancing that, ‘There is clearly enormous scope for improving the means of accumulating the knowledge gained through qualitative studies.’[footnoteRef:13] Thus, quantitative studies can help to improve the knowledge that is gain through qualitative research, providing more depth to the answer of a research question. The NHS Centre for Reviews and Dissemination now provide, in their guidance of undertaking systematic reviews[footnoteRef:14], the advantages that using qualitative data may provide. Some academics have warmly welcomed this, agreeing that using this kind of data can add a new perspective on systematic reviews, one that quantitative only studies may lack. However, there are still some issues with using this kind of data in a systematic review. Dixon-Woods and Fitzpatrick outline them as: [12: Petticrew M, 'Time to rethink the systematic review catechism? Moving from 'what works' to what happens'' (2015) 4 Systematic Reviews] [13: Petticrew M and Roberts H, Systematic Reviews in the Social Sciences A PRACTICAL GUIDE (2006), p. 71] [14: NHS Centre for Reviews and Dissemination, Undertaking systematic reviews of research on effectiveness: CRD's guidance for those carrying out or commissioning reviews, (2001, 2nd ed.) ] 1) Rigour is important when searching for a systematic review. Searching for qualitative studies can be ‘frustrating’ or difficult 2) A suitable way to appraise qualitative studies has not yet been agreed. The models for quantitative studies are not appropriate 3) How do we synthesise and conduct secondary-summary with qualitative data?[footnoteRef:15] [15: Dixon-Woods M and Fitzpatrick R, 'Qualitative research in systematic reviews' (2001) 323 BMJ Accessed via: http://www.bmj.com/content/323/7316/765 - Last cited 15/05/15] Whilst these are issues for a qualitative systematic review, they are not complete barriers to conducting one. I appreciate that appraising the studies does not have a set methodology, but I believe that if you are clear with your appraising methodology this should not be a huge issue. This systematic review contains only qualitative research, forcing me to exclude the meta-analysis stage of a traditional systematic review. Even though there is no quantitative data to extract and synthesise from the research, there is still a strong opportunity to analyse qualitative data using this method. Petticrew and Roberts highlight that systematic reviews are a great method that ‘can be used to summarize, appraise, and communicate the results and implications of otherwise unmanageable quantities of research.’[footnoteRef:16] However, they add to this by stating that systematic reviews can lack an actual substantive discussion of the literature, and the ‘communication’, element should be greater if systematic reviews are to be ‘really useful’.[footnoteRef:17] This communication element is more prominent in a traditional literature review approach. I hope that the increasing use of qualitative studies in systematic reviews will result in greater communication of the information available for particular research questions. [16: Petticrew M and Roberts H, Systematic Reviews in the Social Sciences A PRACTICAL GUIDE (2006), p. 10] [17: Ibid] As the Cochrane Collaborative follows the key phases, outlined above, I had to modify some phases slightly, particularly the synthesis stage, in order to fit my methodology and the sole use of qualitative research. Aims of this systematic review · To find all of the peer-reviewed and published research available regarding CLE in Europe · Explore what research has been published in this area (area concerning both research and regional) · Analyse the research, looking at whether it is theoretical or empirical, and the reliability attached to it · To identify any gaps in knowledge or further research that can be conducted. Looking at the aims of my systematic review, they were partially met. I did find all of the peer-reviewed and published research available regarding CLE in Europe. I did explore and analyse the research. However, I do not think I can state that I found all of the gaps in knowledge or can definitely say what further research needs to be conducted in this area. This will be discussed in more depth in the section discussing the limitations of this systematic review. Research question and how it was formulated What research has been published in Europe regarding clinical legal education and what is missing? Formulating my research question was not a particularly difficult task. As I wanted to know what research had already been done in Europe generally it was easy to express in a question. I was gathering any articles relating to CLE from the continent, and not narrowing it down to specific CLE issues. Thus, I wanted to gather articles that related to establishing a clinic, sustaining it, teaching methods, social justice aspects, assessment or any other research topics available to CLE. Consequently, my research question became rather broad to capture all of those research areas. Mapping and scoping The first phase of this review was to design a plan to conduct the systematic review. I already knew that there are more articles produced in the UK than there are from Continental Europe. However, I knew that some countries had published research and engaged with CLE more widely than others. Thus, there were some countries highlighted for specific searches with a general European search to catch anything that may be missed. The key words were developed from the research question. ‘Clinical legal education’ was the main search term as this is the most relevant part of the research question. The remaining key words were the countries I had identified as more likely to have published in this area, and then a general Europe search. The ‘sweeping searches’ used more general terms that were wider than the research question, to ensure I had captured all of the materials available to me. The results of these searches have not been included in the final Prisma diagram, as they did not form the main part of the search. They were used as a final check, and did not provide me with any further articles (see appendix 1). I decided that I would not have a time limit, or a start date, to my searches. I wanted to find all of the articles available to me, regardless of when they were published. This worked quite well, as the modern wave of CLE started in the 60’s, mainly in the US, Australia, Canada and the UK.[footnoteRef:18] As Continental Europe mostly did not begin engaging with CLE until later, I knew I wouldn’t have to go too far back. Thus, having a start date for my searches did not really make too much of a difference to narrowing down my searches for Europe. [18: Bloch FS, The global clinical movement: educating lawyers for social justice (Oxford University Press 2010), p.3] Comprehensive search: how I searched for the materials Choosing Databases As this area covers law and education, I choose databases which I knew would provide either these areas, or predominately providing one of them. · Heinonline is a database that covers both these disciplines, with many clinical focused journals subscribed to, rather than just educational. Consequently, most of the articles included in the full text analysis were from this database. · Westlaw is a database that primarily archives legal articles. However, they do bring results up for the International Journal of Clinical Legal Education (IJCLE). Whilst they are not actually subscribed to on this database, it is an open access journal, with all of the archives available online. Thus, the results from these hits were recorded from this database and accessed via the IJCLE website. · LexisNexis is another legal based database. It was included in the database selection to cover all of the main legal databases that are commonly used in legal research. · Lawtel is another legal database, but one that does not commonly archive legal education articles. Again, it was to ensure that the main legal databases had been searched. · SAGE is a socio-legal database, and sometimes legal education is included within socio-legal studies. This database was used to search a much broader variety of articles, rather than strict legal databases. · The Social Science Research Network, again, is a much more boarder database, that does archive some legal education articles. It was chosen in order to conduct another broader scoping look at the articles that are available. Searching In order to find materials on the databases I used the Boolean operators, to narrow down my searches. Boolean operators use simple words, such as AND, NOT and OR, to narrow down results and link search words together. I used the operator AND consistently throughout my search. The knowledge that the majority of material published in this area originates from the US, Australia, Canada and the UK meant that I knew most of the results would originate from these countries. For continental Europe there is a lack of research, with some countries publishing slightly more and some not at all in research outlets. With this knowledge already in place it was vital to narrow down the searches to certain countries within Europe to avoid great volumes of articles from countries that were not relevant. I selected countries that I knew had many clinics or had published limited research on the clinics that had been established recently. Thus, in my search terms I used ‘Clinical legal education AND [selected country]’ to find articles relevant to that region. It is appreciated that not all European countries were included as a search term. From my initial searches I knew that many articles appeared in most searches for any country, even if not the specific country searched. Thus, I searched countries I knew had published on CLE already. Lastly, I would search for ‘Clinical legal education AND Europe’, as a final search to ensure that I hadn’t missed any articles from the continent. It was not practicable to search for every country in Europe separately, but finding countries not searched appear in the results for other multiple times was reassuring and they were added to the results. I did attempt to use NOT as a search operator, but it did not provide desirable results for some databases. During my searches there would be many results relating to clinical medical education and clinic trials in health. They actually formed the majority of my search results for Heinonline and SAGE. When using ‘NOT medical’ or ‘NOT health’ after my usual search term to narrow down results it only brought up more articles rather than producing less. Thus, I stopped using NOT as a search operator and would sift through all of the articles produced for my initial search rather than narrowing them down any further. This was an issue and added time onto the systematic review, but there were not so many results that it became impossible to do in the timeframe allowed. I did not use the search operator OR at all during my searches. There was no specific reason for this, just that I wasn’t searching for multiple research areas of CLE, just any materials from Europe. I could have used ‘Clinical legal education OR law clinics AND [selected country]’, but I felt it would have brought up the same results, which would be added to the duplicate list. For LexisNexis and Westlaw I decided to do a ‘sweeping search’[footnoteRef:19] using different terms to ensure that I had recorded all of the relevant articles from those databases. As Heinonline was my greatest source of articles I did not find this necessary use for these searches, as the majority of articles were produced on my initial searches. The ‘sweeping searches’ ensured that I had searched and recorded all the articles available to me. By using different search terms to ‘clinical legal education’ I was able allow for potential articles with a different description of this method to be found. [19: See Appendix 1] I stopped searching when the records recovered from the searches were displaying irrelevant articles, or the only relevant results were mainly duplicates. Once I started to record many duplicates it was realised that the material available to me was already recorded to be read in full text. I felt that the databases used and the search terms confirmed that I had collected all the material that I possibly could from my institution’s accessible databases. Search terms and databases Databases Searched with Dates: Heinonline - 9/03/15 - 17-03-15 Westlaw - 17/03/15 LexisNexis - 9/03/15 Lawtel - 17/03/15 SAGE - 04/04/15 - 05/04/15 Social Science Research Network (SSRN) - 06/04/15 Search Terms: Clinical legal education AND Poland Clinical legal education AND Germany Clinical legal education AND Croatia Clinical legal education AND UK Clinical legal education AND Russia Clinical legal education AND Ireland Clinical legal education AND Czech Clinical legal education AND France Clinical legal education AND Northern Europe Clinical legal education AND Europe Additional ‘sweeping’ searches for LexisNexis and Westlaw: Law clinics AND Europe Legal Education AND Europe Search limitations Language (Any articles not in English) Databases not subscribed to by my institution Geographic (any research not produced or conducted within Europe) Inclusion/exclusion criteria · Title Many articles were included from their title. It was easy to see articles that were highly relevant to the search as they normally stated the country and kind of CLE it is discussing. For example, it was easy to see from the title, ‘The Next Step Forward: The Development of Clinical Legal Education in Poland through a Clinical Pilot Program in Bialystok[footnoteRef:20]’ is obviously an article regarding CLE in a European country. [20: Skrodzka M, Chia J and Bruce-Jones E, 'The Next Step Forward - The Development of Clinical Legal Education in Poland Through a Clinical Pilot Program in Bialystok' (2008) 2 Columbia Journal of East European Law 56] Most articles results could be excluded by title. They were excluded if it was clear that they didn’t fit into the European criteria and were research produced from outside this continent. A total of 44,171 articles were excluded due to title. Next, articles were excluded if it was clear they were to do with medical education and not legal education. For some databases the majority of articles produced were for medicine and not law. It was very easy to identify which articles were medical and would be excluded by title instantly. · Abstract Occasionally it was necessary to read an article’s abstract in order to determine whether it should be included in the systematic review. Abstracts read had to illustrate that the article was research conducted within Europe and relating to CLE. This was normally fairly easy to identify. If it was not, then the contents page or introduction were read to determine inclusion. This did not, however, occur very often and it was not deemed necessary to make a record of how many articles were included from contents page, etc. They were included under the abstract criteria heading. Altogether 120 articles were excluded by abstract. The reasons for exclusion by abstract varied. The majority (approximately 100 articles) of abstracts excluded were due to the research not having been conducted in Europe. Often, searches would produce articles that mentioned a country in the footnotes or briefly within the main body. These articles obviously would not answer my research question and were not relevant to the systematic review. Articles were also excluded if the abstract did not mention CLE at all, meaning that the research didn’t concern this kind of education, but legal education more generally. · Articles that had to be excluded Some articles had to be excluded for various reasons. If the article’s full text was not available or not subscribed to then it was excluded. If there was no possibility of gaining the full text then it could not be included in the later stages of the systematic review, the synthesis. A total of 29 articles were excluded for unavailability. A small amount of articles were excluded as they were written in a language other than English. There was no means available to translate these articles at the time of the search. There were only 2 articles excluded due to language. Lastly, there were articles excluded as duplicates. These were articles that had already been included to be read in full text and thus were not needed again. The articles excluded during a search were those that were relevant to the country searched. There were a great number of articles produced for medicine and health, and they were not recorded again as duplicates as they were not relevant to my search. Only those to be read in full text were excluded as a duplicate. A total of 34 articles were excluded as a duplicate. · Full text Only 3 articles were excluded after reading full text. These reasons were as follows: · One article was not research in Spain. It was actually research from a US university that had the same name as a Spanish city. Once realised this was not European research it was excluded. · One article was excluded from Germany. When reading this article it came to light that it was only about legal education in Germany and did not mention clinical legal education at all. As it did not mention any clinics, and other articles provided a comprehensive background to legal education in Germany, I felt it was not necessary to include this article in the final results. · An article was excluded after it was realised that it did not really concern CLE, but rather developing an educational system for preparing students for international practice. This was not relevant to this systematic review and was excluded accordingly. As can be seen, there were not many articles excluded during the full text stage of the review. I believe this is because my screening worked so well, and articles that would fit the criteria were chosen carefully. Recording the articles Once I had identified articles I wanted to read in full text, I recorded them on a table in a word document. I recorded the author, date of publication, title, citation and which country the research concerned. Recording the articles in this way made it easier to find them again at a later date, and also to check any potential duplicates from my searches. An example of the final result looked like this: Slyvester, C. And Hall, J. And Hall, E. 2004 Problem-Based Learning and Clinical Legal Education: What can Clinical Educators Learn from PBL International Journal of Clinical Legal Education, Vol. 4, pp 39-63 Int’l J. Clinical Legal Educ. 39 (2004) UK I found that this was the most efficient way of recording the articles. Once I had completed my tables I transferred them to Excel. This meant I could filter articles depending on date or country, for ease of analysis. I only recorded the information of articles I chose to read in full text. Articles that were excluded prior to this were not recorded in more than a numerical figure, as it would have not served me any purpose. Findings The records I identified before screening, for all searches conducted, are as follows: · Heinonline: 23,147 · Lexisnexis: 71 · Lawtell: 3 · Westlaw: 1095 · SAGE: 20,083 · SSRN: 24 I would like to highlight some preliminary observations of the actual records produced rather than the content. As I was searching for certain countries which I knew had already published research, I expected to find more articles than I actually did. This was especially for Poland, Croatia and Ireland. I was not surprised by the results for the UK, as I knew this country would lead the amount of research available to me. The results for Germany were surprising as this is a country which has taken longer than others to engage with CLE.[footnoteRef:21] [21: Tomoszek M, 'The Growth of Legal Clinics in Europe - Faith and Hope, or Evidence and Hard Work?' (2014) 21 International Journal of Clinical Legal Education 93 – this paper highlights, particularly at p.99, how Germany has ‘resisted’ the CLE trend, favouring the more conventional approach to legal education. ] However, I realised that some countries had not produced any research outlets themselves in my results, but rather it was mainly US academics writing about their experiences of CLE in these countries. This is especially the instance for Germany and Russia. Some countries were mixed and some produced all their own research, mixing it comparatively with other countries. This is shown in figures 3 and 4 below, which compares domestic to non domestic articles produced per year. I think I was expecting to find more articles than I actually did in the final result. However, it can’t be forgotten that there were many articles excluded as they weren’t available to me. Had they been, the results would have been different, particularly for the UK. Below is a Prisma flow diagram[footnoteRef:22], showing the different stages of my systematic review and the inclusion/exclusion numbers. I have already dissected above how many articles were excluded for various reasons and why, and it is not necessary to repeat them here. [22: The findings included in this Prisma do not include the records for the additional ‘sweeping searches’. There were no records used, as shown in the table in appendix 1. ] Identification Screening Eligibility Included Records identified through database searching (n = 44,423) Full-text articles excluded, with reasons (n = 3) Studies included in qualitative synthesis (n = 71) Full-text articles assessed for eligibility (n = 74) Records excluded (n = 44,285) Records screened (n = 44,359 ) Records after duplicates/non-English/non-availability removed (n = 65) Studies included in the research for PhD (n =71 ) Figure 1 – PRISMA flow diagram showing the stages of the systematic review Synthesising the quantitative elements of the literature This table shows the search results for each country and the general Europe searches. The figures shown is the total amount of articles for each country across the six databases used. Country searched Articles produced Articles excluded by title Articles excluded by abstract Articles that had to be excluded Articles excluded from full text Articles included in research paper for PhD Croatia 453 445 4 2 0 2 Czech 719 700 11 3 0 5 France 5,906 5,894 10 1 0 1 Germany 6,459 6,442 11 0 1 5 Ireland 3,370 3,349 16 0 0 5 Poland 1,582 1,561 9 4 0 8 Russia 2,120 2,107 6 4 0 3 UK 11,500 11,387 36 43 0 34 Northern Europe 2,520 2,517 3 0 0 0 Europe Generally 9,794 9,769 14 7 1 3 Other articles found during searches, not relevant to country searched and did not appear in the general Europe search 1 2 5 TOTAL ARTICLES INCLUDED 71 Figure 2 – Table displaying the search results per country or region I then decided to put the amount of articles from each country on a map: Figure 3 – Map showing how many articles sourced from each country This helped me to locate where the research had been conducted and which countries had not produced any research from my searches. A systematic review, when completed, gives you a number of articles. It does not present the articles in a way which is easy to look at and dissect where they originate from. As I am not familiar with working with numbers, I felt it necessary to display my results in other ways. A visual aid really helped me to locate which countries had been producing more research than others, if any. It also helped when I was presenting this work for attendees to the ENCLE workshop to see the results in this format, rather than just looking at multiple tables. This map shows a geographical cluster of publication activity, which is not apparent when looking at the results in a table or list format. This map does not, however, show which countries have clinical activity. The fact that I did not find any peer-reviewed articles for certain countries does not mean that they are not engaging with CLE. I know that many European institutions do incorporate some form of CLE into their law programmes, but that is not apparent from this systematic review. It also does not mean that they are not conducting research. It only displays which countries have published peer-reviewed articles, and how many papers for each. Furthermore, this map only shows which countries have produced research or had research conducted in them by domestic or non-domestic authors as a total. It does not distinguish between them. Thus, I was also interested in how many articles had been produced by non-domestic academics compared to domestically produced articles. With my final amount of articles, this is how it looked: Figure 4 – Graph showing the amount of papers published by non-domestic authors From this graph it can be seen that the earlier papers published, there is quite a small difference between the articles published and publications by non-domestic authors. As we get to more recent times, 2008 onwards, there seems to be a big difference between the papers published and publications by non-domestic authors. These graphs, in a way, are in line with my hypothesis. I knew that there wasn’t much peer-reviewed research published within Europe, which focused on CLE. This is for various reasons. The main reason is that CLE was not widely engaged with throughout Continental Europe until the 1990’s.[footnoteRef:23] This boom of CLE started when organisations, such as the ABA or CEELI[footnoteRef:24], worked with European universities to help them incorporate CLE into their legal education. As a result, it is not surprising to see on this graph that the first papers published on European CLE are from non-domestic authors. In some years, such as 1999, there were no papers published by domestic academics in European clinics. Non-domestic authors, especially those from America, would write of their experience helping to set up clinics in Europe and the difficulties they faced. As these clinics become more established we begin to see a rise in the amount of papers published by domestic authors. [23: Bloch FS, The global clinical movement: educating lawyers for social justice (Oxford University Press 2010), Chapter 4.] [24: American Bar Association (ABA) and Central and Easter European Law Initiative (CEELI). ] However, due to the limitations of my systematic review, discussed below, there does seem to be very valid reasons for domestic authors not publishing as much peer-reviewed CLE research, especially in the earlier years. I think these results can also be represented on a graph, comparing papers published by domestic to non-domestic authors, rather than total papers published compared to non-domestic. Representing the results in this way appears like this: Figure 5 – Graph displaying the difference in numbers of papers published by domestic and non-domestic authors Looking at this graph I can conclude that in some years, especially in the earlier years, there were more papers published by non-domestic authors than there were domestic. Whilst this graph is similar to the one above, and the same conclusions can be drawn, seeing it presented visually different on this graph displays the occasional contrasts. I found these differences a particularly interesting aspect of the results, prompting me to draw conclusions as to why. This will be discussed further in the next section, whereby the analysis of the content of the papers can attempt to form a theory to explain this difference. Synthesising the quantitative elements of the literature Synthesising the qualitative literature can be done in a variety of ways, and there have been as many as 12 approaches to qualitative synthesis.[footnoteRef:25] I did not want to go into a large amount of depth of the papers sourced. I wanted to show more broadly what is being published by, and within, European institutions, highlighting areas which could benefit from further research. Thus, a kind of narrative approach was adopted, to ‘describe the scope of existing research.’[footnoteRef:26] [25: Booth A, Papaioannou D and Sutton A, Systematic Approaches to a Successful Literature Review (SAGE 2012), p.127] [26: Ibid., p. 147] There was not a great variety of papers sourced through this systematic review. The kinds of papers found are displayed in the table below: Kind of paper found Number of papers Countries of origin Papers explaining the establishment and running of a clinic 28 Poland (5) Ireland (3) Croatia (2) Czech (2) Norway (1) Netherlands (1) Serbia (1) Romania (2) UK (11) Papers explaining generally legal education and CLE 21 Poland (2) Germany (4) Ireland (2) Italy (1) Czech (2) Russia (3) Netherlands (1) France (1) EU (1) UK (4) Papers discussing ethics, values and access to justice in CLE 6 UK (6) Papers exploring skills and the benefits of CLE 5 Poland (1) Germany (1) Czech (1) UK (2) Papers discussing the integration of CLE into the curriculum 3 UK (3) Papers focusing on theory and practice 3 UK (3) Papers focused on reflection 2 UK (2) Papers focused on assessment 2 UK (2) Papers focused around the clinician 1 UK (1) Figure 6 – Table showing which kinds of papers were being published? By studying this table, we can see where the trends in publications are. It is very common for those working in clinics to write about establishing a clinic and how to run one. However, it was found that these papers were mostly not followed up afterwards, with no papers explaining the development of the clinic or the setbacks. Whilst these papers can be very useful and enlightening to others working in and wanting to establish a clinic, it is also important to discuss if a clinic has failed or any setbacks and difficulties that have been faced. By sharing this information we allow others to learn from our own experiences and to help build collaborations or forums for advice. Clinical legal education, and legal education generally, was also found to be discussed widely in the literature. What are the downfalls of legal education, how can clinic help to fill this gap? These were questions widely addressed in these papers. These are very valuable questions, but ones which have been discussed to a great extent already. This is demonstrated more, the further down we move in the table. Papers discussing specific areas of CLE, such as assessment and integration, become more scarce. Furthermore, these papers mostly come from the UK. As highlighted above, it is necessary for us to be sharing experiences. It may have been that I could not find papers relating to these areas from other European countries, or they may have been in another language. However, I feel it important to emphasise, again, the necessity of us sharing research and experience. Perhaps it is time for us to move away from the general discussion of CLE and focusing on the specifics. Limitations of my systematic review From the ENCLE workshop I learnt that there were some gaps in my research and I had not gathered all of the research that was available within Europe. This was for many reasons. Research may have been published on databases which I did not have access to or which were not in English. Furthermore, I realised that my first attempt at a systematic review, whilst helping to develop my research skills, was not as sophisticated as my later attempts. The naivety which I held during the process highlights a need for reflection and to avoid others making the same mistakes. When I presented my initial findings I faced much opposition, perhaps even hostility, to my results. Participants at the workshop made me aware that they were producing work, and just because I did not come across it in my review did not mean that some countries were not producing anything. There were participants who stated they did write about their clinics, but it was not published in peer-reviewed articles, but rather in blogs, on websites, in university brochures and other unconventional-publishing outlets. It is becoming increasingly popular in this new age of technology to write in places where one may not have traditionally done so. With this mass of grey literature out there I knew I had upset some participants, somewhat accusing them of not conducting research. I was rather taken aback by this. I had not intended to upset anyone or, essentially, call academics from certain countries lazy! I just wondered where their research was and if they were engaging with research at all. They made me aware that there were reasons why they did not write in peer-reviewed journals. This could be because they did not get support from their institutions to write articles or to conduct research. Research can be a time-consuming process, and unless you are allocated time to do it, it is very hard to complete. Some participants did not know how to carry out research appropriate for peer-reviewing, and it was not an important role of their job or even during their university studies. If this presentation highlighted anything for ENCLE, it was that we need to provide support to clinicians in Europe to research and share it in conventional outlets, in order for us to move forward in our clinical movement. With Europe being slower than other regions on the development[footnoteRef:27] of CLE, it is important that we can help facilitate any needs now, to ensure the movement continues to grow. When I say slower, I mean that, for example, the US widely engage with CLE, and it is incorporated in every law school.[footnoteRef:28] Their research into CLE is immense, establishing the US as one of the leaders of CLE. Does this mean that the growth of CLE in developed at a higher rate in the US than it has within Europe? That is a question which will not be answered in this paper; one that needs further research. But from this systematic review results, it would seems that Europe is slower or behind in the incorporation of CLE to their legal programmes, or at least writing about it in peer-reviewed articles. Not only that, from looking at the graphs above (figures 4 and 5), it is important that countries are writing about their own clinical experiences, instead of relying on non-domestic academics to be accurate and fully understand the complexities of another’s legal and political systems. [27: The first clinical boom in Europe started in the 1990’s, compared to places like the U.S. their boom starting in the 1960’s - Bloch FS, The global clinical movement: educating lawyers for social justice (Oxford University Press 2010)] [28: https://www.northumbria.ac.uk/about-us/academic-departments/northumbria-law-school/study/student-law-office/clinical-legal-education/ - cited 02/07/15] Thus, this raises an issue: when do we expect institutions to produce research about their clinics? Does a clinic have to go through adolescence before we can research it? Or is it preferred to publish research regarding the earlier stages of the clinic? Again, these are not questions to be answered in this paper and go beyond the scope of this systematic review. However, it is a consideration when contemplating what and when to publish. A clinic can face many issues when developing a model and attempting to sustain it. The sustainability depends on many factors,[footnoteRef:29] and some institution’s clinics will fail due to these factors, or a lack thereof. At Northumbria, for example, our sustainability is strong. As CLE is such an integral part of our curriculum, it cannot fail. If the clinic were to fail, the whole degree programme would have to undergo a redesign. Other institutions do not have this security. Should we be sharing our failures more readily, in order to inform other institutions of the difficulties of sustaining clinics and how to avoid them? Or is our research more concerned with an established clinic and what makes it a success? These questions will be explored, to an extent, when looking at the papers produced from this systematic review. [29: Mkwebu T, 'A Systematic Review of Literature on Clinical Legal Education: A Tool for Researchers in Responding to an Explosion of Clinical Scholarship' (2015) 22 International Journal of Clinical Legal Education 238 ] Revisiting the research question, whilst I found all of the peer-reviewed articles available to me, the ENCLE workshop highlighted that I cannot boldly state that I found all of the research from Europe, as I have not. I have found all of the research available to me through the databases I searched, in my native language. I was informed that there are publications from countries such as Belarus, which I did not find. With the knowledge now that there are publications I did not unearth, I would conclude that my research question has not been fully met. If my research question had been ‘What peer-reviewed research has been published in Europe, available to me, regarding clinical legal education and what is missing?’ then my results would have been sufficient. It is becoming increasingly more common to include non-published research in a literature review, thus changing the function of a traditional systematic review slightly. Petticrew and Roberts state in their book that: ‘While we also use the phrase systematic ‘‘literature’’ reviews, not all evidence which may be useful for a review will of course appear in the published ‘‘literature.’’ Because of this, the term ‘‘research synthesis’’ (or ‘‘evidence synthesis’’) is becoming increasingly common.’[footnoteRef:30] [30: Petticrew M and Roberts H, Systematic Reviews in the Social Sciences A PRACTICAL GUIDE (2006), p. xiv] It is now highlighted by academics that not only peer-reviewed and/or published research can be used in a systematic review. It is now common, and accepted, that other evidence useful to systematic review research questions can now be included, changing the procedure slightly. With this in mind, for this particular research question it may be more appropriate to move slightly away from a traditional systematic review and more towards a ‘research synthesis’ design. This can result in collecting a broader range of research to add reliability to my results, whilst also ensuring the work that researchers produce in other forums is not forgotten or overlooked as valid research. Even with the limitations I faced during my systematic review, I still believe it was the most appropriate methodology to use, as opposed to a narrative review. I mostly answered my research question, and more sufficiently than it would have been had I done a narrative review. Whilst I acknowledge the benefits of a narrative review, I can draw strong conclusions that it would not have benefited this research. The main benefit I would have gained from the traditional narrative review, compared to that of a systematic review, would have been a wider engagement with the grey literature. Grey literature is now used more widely in research, and a systematic review can overlook these useful sources of information. A disadvantage to the lack of use of the grey literature is that too much reliance is placed on peer-reviewed research, and that other sources of research do not have the same quality. Jeeson et al challenge this, stating ‘The downside of a peer review is that being judged by experts who have established perspectives and paradigms can act as a barrier to publishing new and unconventional ideas.’[footnoteRef:31] This can create what they call a ‘publication bias,’ and using grey literature may help to reduce this bias. However, there is a challenge faced by those conducting systematic reviews, and how exactly they can adopt a method which will be successful to find grey literature. As most electronic databases do not pick up the grey literature, you could be excluding ‘valuable information’[footnoteRef:32] which may be of benefit to the research. [31: Jesson J, Matheson L and Lacey FM, Doing your literature review: traditional and systematic techniques (SAGE 2011) P. 20] [32: Jesson J, Matheson L and Lacey FM, Doing your literature review: traditional and systematic techniques (SAGE 2011) P. 126] Now with the knowledge that there are other publications out there, and a mass of grey literature, I have the means to move forward with this systematic review and look at more unconventional research outlets. Since this work was undertaken I have completed a further systematic review, which forms part of my PhD thesis. I found that that systematic review was much more advanced than this. For example, the duplicates recorded in this systematic review were only duplicates which of the full text articles already selected to be read. In my PhD systematic review duplicates were recorded as any article which had already appeared in my search results. This way I could assess more easily when my searches were becoming saturated, and it gave a more realistic account articles appearing. I hope this work provides guidance for others wishing to undertake similar research and that any hindsight I may have had will inform others. Conclusions Whilst there have been some limitations to this study, some conclusions may still be drawn from the methodological results, as well as the review of the literary content of the paper. From the results it can be seen that Continental Europe is producing more research now, than it previously did. It can also be concluded that there is now more domestic authors publishing research compared to non-domestic. Whilst the limitations discussed above may cause difficulties, I believe that this systematic review has drawn some successful findings as a foundation to help move CLE forward within Europe, highlighting the need for more research in accessible research outlets. Furthermore, whilst the Cochrane Collaborative provided a great foundation for this kind of literature review, the developments in recent studies are welcomed as a way of creating a more modernised form of systematic reviews. We must not forget the unconventional methods of publishing, nor how valuable qualitative data can be in this method. Appendix 1 Results of the sweeping searches: SWEEPING SEARCHES - 14/04/15 Database: Westlaw           Search Term Articles produced Articles included Articles excluded by title Articles that had to be excluded, e.g. due to language. Articles excluded by abstract Articles excluded after reading full text Law clinics AND Europe 108 0 107 1(Duplicate) 0 0 Legal education AND Europe 48 0 47 0 1 0             Total articles used: 0               SWEEPING SEARCHES - 14/04/15 Database: LexisNexis                         Search Term Articles produced Articles included Articles excluded by title Articles that had to be excluded, e.g. due to language. Articles excluded by abstract Articles excluded after reading full text Law clinics AND Europe 11 0 11 0 0 0 Legal education AND Europe 363 0 362 0 1 0             Total articles used: 0 Papers published 1973 1980 1993 1996 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 1 1 1 1 2 1 3 4 2 2 3 6 5 1 8 3 5 4 4 9 5 by non-domestic authors 1973 1980 1993 1996 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 1 1 1 0 0 1 1 2 1 0 0 0 3 0 2 0 1 0 0 0 0 Papers published by domestic authors 1973 1980 1993 1996 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 0 0 0 1 2 0 2 2 1 2 3 6 2 1 7 3 4 4 2 9 5 Papers publishedby non-domestic authors 1973 1980 1993 1996 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 1 1 1 0 0 1 1 2 1 0 0 0 3 0 2 0 1 0 1 0 0 82 Poland 8Germany5 Croatia 2 UK34Russia 3 Ireland5 Czech5France1 NL2 Italy1Romania 2 Serbia1 Europe Generally1 Norway1 Total of articles produced: 71 /docProps/thumbnail.jpeg Slide 6 Poland 8 Germany 5 Croatia 2 UK 34 Russia 3 Ireland 5 Czech 5 France 1 NL 2 Italy 1 Romania 2 Serbia 1 Europe Generally 1 Norway 1 Total of articles produced: 71