Reviewed Article: Research and Impact MEASURING THE IMPACT OF SOCIAL JUSTICE TEACHING: RESEARCH DESIGN AND OVERSIGHT Lisa Radtke Bliss, Sylvia B. Caley and Leslie E. Wolf* Georgia State University, USA Introduction Research and the production of scholarship is a fundamental part of being a legal academic. Such endeavors identify issues and answer questions that further understanding of the law, the profession, and the justice system itself. Research and scholarship in the legal academy traditionally meant the study of law and legal theory. A growing body of legal academics are focusing research and scholarship on legal education itself, as well as research that measures the impact of legal education. The impact of clinical legal education on students’ development of practical and professional skills, as well as the impact on communities that clinics serve are important areas of scholarly inquiry.[footnoteRef:2] This article explores how thoughtfully designed research projects can measure the impact of social justice teaching, using examples and experience gleaned from the evaluation and research component of a medical legal partnership[footnoteRef:3] and its affiliated law school clinic. The article examines principles of good research design, the art of formulating research questions, and the potential uses for resulting data. It also identifies critical steps and issues to consider when developing a research project. [2: *Lisa Radtke Bliss is Associate Dean of Experiential Education, Clinical Professor of Law, and Co-director of the Health Law Partnership Legal Services Clinic at Georgia State University College of Law. Sylvia B. Caley is Clinical Professor of Law, Co-director of the Health Law Partnership Legal Services Clinic and Director of the Health Law Partnership. Leslie E. Wolf is Professor of Law and Director of the Center for Law, Health & Society at Georgia State University College of Law, with a joint appointment in the School of Public Health. See, e.g., Jeanne Charn & Jeffrey Selbin, The Clinic Lab Office, 2013 Wis. L. Rev. 145 (2013); Steven Hartwell, Promoting Moral Development through Experiential Teaching, 1 Clinical L. Rev. 505 (1995); Stefan H. Krieger, The Effect of Clinical Education on Law Student Reasoning: An Empirical Study, 35 Wm. Mitchell L. Rev. 359 (2008); Margaret E. Reuter & Joanne Ingham, The Practice Value of Experiential Legal Education: An Examination of Enrollment Patterns, Course Intensity, and Career Relevance, 22 Clinical L. Rev. 181 (2015); Rebecca Sandefur & Jeffrey Selbin, The Clinic Effect, 16 Clinical L. Rev. 57 (2009).] [3: Medical legal partnership brings together legal and health care providers to holistically address the socio-economic barriers to health. For more information on medical-legal partnerships and the need for such partnerships, see The Need for Medical-Legal Partnership, Nat’l Ctr. for Med.-Legal Partnership, http://medical-legalpartnership.org/need (last visited Aug. 31, 2016).] Part I explains the reasons that engaging in research is a useful, and increasingly necessary, method of measuring the effectiveness and impact of clinical legal education and social justice teaching, and the ways research informs approaches to teaching and client service. It also describes a medical legal partnership (hereinafter “MLP”) known as the Health Law Partnership (hereinafter “the Partnership”),[footnoteRef:4] and the Health Law Partnership Legal Services Clinic (hereinafter “the Partnership Clinic”)[footnoteRef:5] at the University Law School (hereinafter “the University”) and the design and evolution of the research and evaluation component of this project. Part II describes the special needs and concerns that governed the approach to designing and implementing an effective evaluation tool in this context. Part III turns to fundamental principles of research. It identifies important steps to take when establishing a research project, and issues that researchers must confront in doing so, including ethical implications and the need for adherence to international principles of ethical research, as well as university regulations and guidelines. Finally, the article concludes with reflections on the Partnership research project and advice for those considering engaging in similar research, particularly those who wish to evaluate the impact of clinical legal education and justice education. [4: The Health Law Partnership (“the Partnership”) is a community collaboration among the University, the Legal Aid Society, and Children’s Healthcare of City. For the components of the Partnership, see infra Part I. For more information, see The Partnership L. , https://healthlawpartnership.org (last visited Aug. 31, 2016).] [5: The Partnership Legal Services Clinic, Uni. St. U.C.L., http://law.UNI.edu/clinics/the Partnership-legal-services-clinic (last visited Aug. 31, 2016).] Part I – Why Engage in Research and Evaluation? This section explains the reasons that engaging in research and evaluation is increasingly important to law clinics and the benefits to be gained from engaging in research. It argues that clinical educators must not only engage in the rigorous self-evaluation that we ask of our students, but that we must also prioritize evaluation of the impact of our work for several reasons. Primary among those reasons is to ensure we are accomplishing our educational goals and objectives. From the inception of the Partnership and the Partnership Clinic, we committed to a robust research and evaluation agenda. Our work over the last decade informed our understanding of the importance and challenges of conducting this kind of research. Because we draw on these experiences, we provide a brief description of the development of the Partnership project and its research agenda before moving to the broader questions of why and how to measure the impact of education, services, and other activities. A. Designing the Partnership Project Components and Evaluation and Research Conceptualizing and developing the Partnership took place over a protracted period. From the outset, the legal partners, leaders from the Legal Aid Society[footnoteRef:6] (“hereinafter LAS”) and the University were firmly committed to building a partnership between law and medicine in City. It took more than a decade to find the right and willing medical partner. While they searched for the right medical partner, the legal partners built a strong trust relationship and cemented the goals each wanted to achieve from a collaboration specifically designed to combine the expertise of health providers and lawyers to provide more holistic services to low-income health consumers. The overarching goals they identified were to improve health care outcomes for low-income patients while also preparing graduate students in law, medicine, and related health professions to be better problem-solvers. The legal partners recognized that the project needed to include evaluation and research components to assure that goals were met. [6: The Legal Aid Society helps low-income people meet basic needs by providing free legal services and legal education. The organization services clients throughout the Metro Area. For more information, see Legal Aid Soc’y, http://www.CITYlegalaid.org/ (last visited Aug. 31, 2016).] When Children’s Healthcare of City[footnoteRef:7] (hereinafter “Children’s”) agreed to join and create the Health Law Partnership, faculty at the University secured the services of an independent program evaluator to assist in developing metrics targeted at determining whether goals were met. The partners, along with assistance from the program evaluator, developed the following program areas for the Partnership: (1) delivery of direct legal services to low-income children and their families receiving health care services at Children’s,[footnoteRef:8] (2) providing education on two planes: (a) to hospital professionals to familiarize them with the effects social determinants have on health and how the Partnership could assist in addressing health-harming legal needs, and (b) to professional and graduate students in law, medicine, and social work to enhance their education in effective interprofessional collaboration, holistic problem-solving, addressing the social determinants of health, and overall skills development, and (3) engaging in systemic advocacy to address population health issues.[footnoteRef:9] Because so much emphasis was placed on the importance and role of program evaluation and research, the partners agreed to create a fourth program area of research and evaluation to place these functions on equal footing with the other three programmatic components. Once the partners had a vision for the function, the services, and the role for each of the programmatic components, the team set out to develop tools to facilitate data collection and analysis. [7: Children’s Healthcare of City, a non-profit pediatric health system, is the largest provider of healthcare services to children in State. Children’s has 3 hospitals, 27 neighborhood locations, and handles more than 870,000 patient visits annually. Children’s offers access to more than 60 pediatric specialties and programs and is ranked among the top children’s hospitals in the country by U.S. News & World Report. For more information, see Children’s Healthcare City, http://www.choa.org (last visited Aug. 31, 2016).] [8: “Public health legal services” are civil legal services provided by attorneys to low-income persons that, collectively, improve population health. D. Schulman et al., Public Health Legal Services: A New Vision, 15 Geo. J. on Poverty L. & Pol’y 729, 732–33 (2008). Public health legal services are provided to the Partnership clients either through the Partnership Clinic at the University or by Partnership staff attorneys employed by LAS. ] [9: A full explanation of the development and design of the Health Law Partnership and its four components is beyond the scope of this article. For details about the project and its four components, including the delivery of public health legal services, education in the form of professional graduate education, in-service education, systemic advocacy and its public health impact, and program evaluation, research, and scholarship, see Robert Pettignano, Lisa Radtke Bliss & Sylvia Caley, The Health Law Partnership: A Medical-Legal Partnership Strategically Designed to Provide a Coordinated Approach to Public Health Legal Services, Education, Advocacy, Evaluation, Research and Scholarship, 35 J. Legal Med. 57, 57–79 (2014). ] 1. Evaluating Legal Services and Health Provider Satisfaction The first component to be developed was the delivery of direct services. The partners focused on decisions regarding a list of parameters and priorities: · Establishing clients’ financial eligibility for free legal services · Determining how clients would be referred to the Partnership · Developing a plan to capture data on provider referral practices to the Partnership · Establishing the geographic location of children to be served · Identifying the types of legal services to provide to children and families · Identifying clients having types of diagnoses who might be prioritized for legal service · Determining how to capture data on all clients referred for free legal services · Identifying, quantifying, and describing benefits received by children and families as a result of the free legal services to be provided · Determining how to prove that the services provided by the Partnership contributed to improved health outcomes for the children served · Developing a plan to educate health providers to identify and refer eligible children and families for free legal services Using these parameters and priorities as a guide, the team determined the important data points and then developed a lengthy intake questionnaire designed to capture them. The program evaluator and staff built a customized data base to capture all data elements associated with clients’ experiences with the Partnership and developed uniform procedures for interviewing clients to reduce errors, encourage uniformity, and minimize bias.[footnoteRef:10] [10: Staff attorneys serving clients were trained in how to collect the information on the intake questionnaire. Only one person, the Partnership’s office manager, entered all data elements into the database to ensure any errors which occurred would at least be uniform errors.] In addition to the intake questionnaire, the team developed a pre and post survey to be given to the parents/guardians of all children referred to the Partnership for legal services. The pre-survey questions aimed to gather information regarding the parents/guardians perceptions of their own physical health, emotional health, financial well-being, and overall well-being and that of their children. The post survey explored these same areas in order to determine whether there was improvement in self-reported status. The survey also asked whether the parents/guardians felt better equipped to deal with similar problems, at least initially, should they arise in the future. All instruments, including the intake questionnaires and the surveys, received Institutional Review Board (IRB) approval at the University.[footnoteRef:11] While all children and families determined to be eligible for free legal services at the Partnership had the opportunity to participate in the research study, families received Partnership services regardless of their decision to participate or not to participate in the study. Participating families were required to provide informed consent prior to their enrollment in the study. All faculty and staff associated with the Partnership underwent training in Human Subject Research through the Collaborative Institutional Training Initiative.[footnoteRef:12] [11: Institutional Review Board approval is just one part of the legal and ethical requirements for undertaking human subject research. For further discussion, see infra Part III.] [12: Collaborative Institutional Training Initiative, https://www.citiprogram.org (last visited Aug. 31, 2016).] In addition to evaluating the benefits of providing free legal services to low-income children with the hope of improving their health outcomes and those of their parents/guardians, the Partnership also wanted to evaluate the experiences of providers when referring clients to the Partnership. Again, an IRB-approved survey instrument was developed to assess the providers’ satisfaction with both the ease of referral and the receipt of feedback from the Partnership. In addition, survey questions inquired whether referral to the Partnership freed providers to handle matters more within the provider’s scope of practice, and whether the provider perceived that referral of a child or family to the Partnership resulted in any reduction in preventable visits to the Emergency Department, reduced the length of hospitalization, or reduced readmissions to the hospital. The program evaluator distributes the provider survey once annually to all attending physicians, residents, and social workers. Data collected from the surveys have contributed both to program improvement and to publications.[footnoteRef:13] [13: See, e.g., Robert Pettignano, Sylvia Caley & Susan McLaren, The Health Law Partnership: Adding a Lawyer to the Healthcare Team Reduces System Costs and Improves Provider Satisfaction, J. Pub. Health Mgmt. & Prac., July–Aug. 2012, at e1.] 2. The Partnership Clinic and Education for Law, Medical and other Professional Students Experience gleaned from providing free legal services for the purpose of resolving health-harming legal problems, coupled with data on successful outcomes, built credibility for the Partnership, strengthened the Partnership, and created general interest in the concept of MLP. The partners wanted to develop the second component of the Partnership, education of professional graduate students. Development of this component, however, required fund-raising for adequate financial resources to build an in-house clinic at the law school. The data, experiences, and descriptive reports outlining the positive results obtained in the legal services component, along with the evident commitment of the partners, encouraged a major local donor to contribute the funds necessary to develop the Partnership Clinic. Using the same principle applied to the legal services component and the evaluation of provider satisfaction, the partners developed IRB-approved instruments to assess learners’ experiences with the enterprise. The first instrument, which remains in use, is a qualitative post-experience survey originally designed for law students. Perhaps the most significant question on this particular survey asks whether the learners will engage in public interest activities in their professional careers. While faculty in the Partnership Clinic hold no expectations that students enrolled in the clinic will become public interest lawyers upon graduation, the hope is that students will be aware from their experiences in the Partnership Clinic of the negative effects disparity and inequity have on individuals and communities. When lawyers become aware of social injustice and of impediments to access to equal justice and decide to volunteer time and resources, change can happen and injustices can be ameliorated. Our proxy for determining our success in motivating students to get involved in addressing social justice is to ask about their intentions. Four years ago, the Partnership Clinic faculty decided to develop a qualitative survey to evaluate attainment of key values associated with interprofessional clinical legal education.[footnoteRef:14] In developing the instrument, faculty interviewed students, met with educational testing experts, held many working sessions, and tested the instrument on two classes of students completing their semester in the clinic. The instrument remains a work in progress, but the plan is to develop and implement a useful instrument as a first step in developing a longitudinal study to include current students and program graduates. Preliminary information gleaned from this instrument will help to inform whether the goals of the education component are being met and, as necessary, how and where to make adjustments in the curriculum to achieve goals that remain elusive. [14: See Linda Morton et al., Teaching Interdisciplinary Collaboration: Theory, Practice, and Assessment, 13 Quinnipiac Health L.J. 175, 193–196 (2010); Janet Weinstein & Linda Morton, Interdisciplinary Problem Solving Courses as a Context for Nurturing Intrinsic Values, 13 Clinical L. Rev. 839, 862–64 (2007). ] The Partnership has conducted other research projects focused on medical learners specifically. Two different studies have been underway for several years to evaluate changes in awareness and attitudes by medical students and pediatric residents following prescribed exposures to the Partnership and an educational curriculum focusing on the social determinants of health. Additional work is underway, in collaboration with two other MLPs, to develop a new uniform instrument to be used as pre- and post-survey for all learners involved with the MLP.[footnoteRef:15] [15: The Partnership is one of three MLP centers to be awarded a grant from the American Association of Medical Colleges in 2015 entitled Accelerating Health Equity, Advancing through Discovery (AHEAD). The thrust of this three-year grant is to develop and implement uniform metrics across three domains: patient and community; cost savings, institutional benefits, and efficiencies, and learners, including medial students, residents, fellows, and law and social work students. The goal is to determine the influence MLPs have on outcomes and performance. ] 3. Evaluating the Systemic Advocacy Component The systemic advocacy component of the Partnership is carried out through a class taught at the law school entitled Health Legislation and Advocacy. Students enrolled in the class work with non-profit community partners to address issues affecting the health and well-being of State Residents. A pre- and post-survey instrument was developed for use with these students, and it was administered twice with varying success. The next phase of planned research will focus on discreet pieces of legislation enacted into law that were developed in the legislation class to determine whether changes in the law improved the situations the laws were designed to address. For example, the class researched whether increasing the age that children must be placed in booster seats while riding in motor vehicles in State would result in fewer deaths and significant head, neck and back injuries. Compelling arguments were made that State’s law was too lax and that children would be safer if the age was increased. Both the legislature and the governor agreed, and an age increase was enacted into law. Now that the amended law has been in place for a few years, it is possible to develop a new research project to determine whether the new law has contributed to fewer deaths and serious injuries in young children following motor vehicle accidents. Demonstrating a reduction in these types of serious injuries would help to demonstrate that systemic advocacy is a vital component of change within the MLP model. B. Measuring the Impact of Clinical Legal Education While clinical educators know that clinical legal education methods enable students to learn essential knowledge, skills and values, it is difficult to quantify the impact that clinical legal education has on students because the clinical education experience touches so many aspects of professional identity and development. Indeed, scholars have noted that existing research does not help reveal, explain or otherwise inform our understanding of the relationship between clinical legal education and the professional development of law students.[footnoteRef:16] This gap in the research has motivated scholars to explore different aspects of clinical legal education and its effectiveness.[footnoteRef:17] Through formal evaluation processes, we may discover valuable information about the impact of the clinical legal education experience on student learning, development, beliefs, attitudes, and understanding. While the process of measurement can be challenging, the information gained can create new opportunities and should offset the costs.[footnoteRef:18] Without an evaluation process, we have no way of confirming our intuitions about the value of clinical and interprofessional education and of establishing whether we are meeting our teaching and social justice goals, nor are we able to make policy decisions informed by evidence.[footnoteRef:19] The importance of the information that evaluation can provide in the clinical legal education context mandates that we continue efforts to study it. [16: Sandefur & Selbin, supra note 1, at 78.] [17: See, e.g., Stefan H. Krieger, The Effect of Clinical Education on Law Student Reasoning: An Empirical Study, 35 Wm. Mitchell L. Rev. 359, 363 (2008).] [18: But see Deborah L. Rhode, Access to Justice: An Agenda for Legal Education and Research, 62 J. Legal Ed. 531, 542 (2013) (“Compared with other work, empirical research has higher costs and lower rewards [for academics]. It is typically more expensive and time consuming than doctrinal or theoretical scholarship, requires greater interdisciplinary expertise and risks dismissal in some circles as ‘merely descriptive.’”).] [19: See also id. at 532 (discussing need for greater research about access to justice in American in order to inform legal service policies, education, and allocation of resources).] Fundamental to clinical pedagogy is the process of self-reflection, which has long been used as a tool for informally evaluating students’ reactions to the clinical experience. Clinical legal education is rooted in David Kolb’s experiential learning model, which describes a cycle of learning that moves from actual experience, to observation and reflection, to the formation of abstract concepts, to testing and applying the learned theories and concepts to new situations.[footnoteRef:20] Learning through reflection is at the core of clinical pedagogy and is essential to the learning process.[footnoteRef:21] Clinical education emphasizes the importance of continuous self-reflection and critical examination of experiences.[footnoteRef:22] Self-reflection by students occurs in multiple venues during the clinic experience. It can be done through journals, essays, one-on-one meetings with supervisors, in classroom discussions, online discussions, and other contexts. Reflection can also be prompted and measured through formal evaluation processes, and the results used to inform and improve clinical education. [20: Alice Y. Kolb & David A. Kolb, Learning Styles and Learning Spaces: Enhancing Experiential Learning in Higher Education, 4 Acad. Mgmt. Learning & Educ. 193, 194 (2005).] [21: Susan Bryant et al., Learning Goals for Clinical Programs, in Transforming The Education of Lawyers: The Theory And Practice Of Clinical Pedagogy 13, 23–24 (Susan Bryant et al. eds., 2014).] [22: Lisa Radtke Bilss & Donald C. Peters, Delivering Effective Education in In-House Clinics, in Building Best Practices: Transforming Legal Education in a Changing World 188, 202 (Deborah Maranville, Lisa Radtke Bliss, Carolyn Wilkes Kaas & Antoinette Sedillo Lopez eds., 2015). See also Carolyn Grose, Beyond Skills Training, Revisited: The Clinical Education Spiral, 19 Clinical L. Rev. 489, 500 (2013); Kimberly E. O’Leary, Evaluating Law Teaching–Suggestions for Law Professors Who Have Never Used the Clinical Teaching Method, 29 N. Ky. L. Rev. 491, 494–95, 503 (2002).] Through evaluation, teachers and researchers may determine the levels of student understanding and achievement in different concepts, goals, or ideas, and how such understanding may have changed as the result of the experiences students are exposed to in a clinical course. Moreover, asking students self-assessment questions, and engaging in pre-clinic and post-clinic surveys about the student experience forces students to think more deeply about their own experience, thus supplementing their opportunities for reflection, which are a critical part of the learning cycle. Surveys can also measure and track how often students engaged in particular activities, and how well they think they performed them or developed their abilities in a particular skill. Students can be asked about their learning goals and whether they believed that they were able to meet them through the clinical experience. Surveys can explore particular values, attitudes, and ethics of students and how their experience in a clinic may have changed those values, attitudes, or ethics. Evaluation contributes to a better understanding of clinical legal education’s effectiveness. Knowing how the clinical experience has impacted students’ knowledge, skills, and values is valuable information both to understand what clinical education offers as part of a legal education curriculum and to understand where a clinical program may be falling short of its express mission or goals. Evaluation results can also provide concrete information about where a clinical program is meeting its goals. For example, if one goal of a clinic is to help students form an awareness and understanding of social justice and how social factors may affect an individual’s ability to get justice in particular circumstances, students may be surveyed about their awareness and understanding of such issues before taking a clinical course and after taking a clinical course. The resulting survey information is important to clinic teachers. It can be studied and used to inform future iterations of the course and the clinic to ensure that the educational goals outlined for students will be met. Survey data can be used to identify the necessity of making changes to course content and experiences where needed, and to identify those aspects of the course that have been demonstrated to be effective and should be continued. This information is not only important for the success of the program, but it is valuable knowledge for clinic teachers in their own professional development, for the development of clinical legal education, and education in general. A program that is regularly evaluating is also a program that is constantly innovating, because it can be responsive to the feedback that evaluation provides. Research from the Partnership’s IRB approved research project has shown that employing its interprofessional education model that has medical students, law students, social work students and others learning and working together fosters a sense of collaboration and cooperation among them, thus positively influencing the professional attitudes and behaviors of the next generation of providers of health, legal and other related services.[footnoteRef:23] Working across disciplines to achieve a common goal of addressing children’s health and the socioeconomic barriers to health changes preexisting professional attitudes toward collaboration across disciplines. This change creates future professionals who will have knowledge of and access to multiple resources that enable them to better serve their patients and clients. [23: For example, the Partnership Clinic has surveyed medical students to determine whether their exposure to the Partnership model increased their level of awareness that social determinants of health affect health outcomes, their willingness to screen patients to determine whether their health may be affected by health-harming legal problems, and, if patients screen positive, to refer them to the Partnership for evaluation and possible assistance. While the data is not yet complete, early results indicate that exposure to MLP-styled interprofessional education favorably influences medical students’ behavior. As part of the American Association of Medical Colleges grant, supra note 16, the Partnership will be testing a survey instrument designed to measure the attitudes of law and medical students prior to the start of their work with the Partnership Clinic and after completion of their work with the Partnership Clinic in order to identify changes that have occurred in attitudes or beliefs.] Measuring students’ progress, career choices, and social justice engagement after they leave the clinic can also help educators to understand the impact of the clinical experience more generally. For example, some clinics might want to specifically evaluate whether students believe that their participation in a clinic made them more likely to do pro bono work,[footnoteRef:24] to advocate for marginalized populations, or to become public interest lawyers.[footnoteRef:25] Researchers targeting clinical education specifically could also generate new knowledge about the impact of clinical education by identifying the specific legal knowledge, skills and attitudes they hope clinics impart, describe the methods they use for doing so, and measuring how effectively their clinics achieve such goals.[footnoteRef:26] [24: See Sandefur & Selbin, supra note 1, at 90–107 (recounting analysis of findings from data from a national survey of early-career attorneys entitled “After the JD”).] [25: Id.] [26: Id. at 105.] C. Measuring the Impact of Legal Services Provided by Clinics Another critical area of research is a law clinic’s impact on the clients and the communities a clinic serves. Many law school clinics engage in different processes to determine client satisfaction. However, satisfaction is just one measure of impact and effectiveness. Other questions may be asked that allow researchers to understand more completely the impact of legal services and legal interventions on individuals as well as communities. For example, through its evaluation process and the data it generates, the Partnership and the Partnership Clinic have been able to identify the impact that legal interventions have had on particular populations of children, such as those with chronic asthma or those with sickle cell disease.[footnoteRef:27] Researchers can track multiple outcomes that are relevant to the type of service being provided to determine whether and the extent of impact a clinic had on a client’s ability to obtain needed services or benefits, housing, jobs, community support, judgment, improvements in health, and other outcomes, and on a community’s ability to address environmental issues, economic development, human rights and myriad issues that clinics are formed to address. [27: See Robert Pettignano, Lisa Radtke Bliss, Sylvia B. Caley & Susan McLaren, Can Access to a Medical-Legal Partnership Benefit Patients with Asthma Who Live in an Urban Community?, 24 J. Health Care for Poor & Underserved 706, 706 (2013) [hereinafter Pettignano et al., Benefit Patients with Asthma]; Robert Pettignano, Lisa Radtke Bliss & Sylvia Caley, Medical-Legal Partnership: Impact on Patients with Sickle Cell Disease, Pediatrics, Dec. 2011, at e1, e1 [hereinafter Pettignano et al., Patients with Sickle Cell Disease].] The results of research can suggest ways to improve the quality and delivery of client service as well as the ability of clients to connect with the services a clinic provides. Knowledge gained through research can identify the most fruitful referral sources for clients. For instance, if most clients self-identify as having legal problems and self-refer to the clinic for assistance in addressing these problems, the educational and promotional materials used to attract clients will differ substantially from materials employed to educate medical professionals about the clinic’s services. Research can identify target audiences for these materials, identify where and how a clinic should invest resources in client education, and suggest where improvements may be made in the client intake process, overall program awareness, and in-service training of hospital professionals to educate them about legal issues, the eligible client population, and the legal services available. D. The Value of Data on Clinical Education and the Impact of Legal Services The impact of a clinic’s services on clients and community, and the impact on student learning can all be explored through research and evaluation, in order to ensure that clinics are meeting student needs and learning goals, that teachers are meeting their objectives for the clinical learning experience, and that the clinics themselves are achieving their stated mission. Without gathering the relevant data, it is difficult to demonstrate effectiveness, ensure quality, or be alerted to environmental changes that might affect a program’s effectiveness and/or sustainability. Data is useful in identifying strengths, which can be leveraged, and weaknesses, which can be targeted for improvement. Data can be used to assess impact and discoveries yielded can generate the publication of research and scholarship. Publication of research enables the information discovered to be shared with a wider population. It also inures to the benefit of the participating institutions. In fact, the Partnership project has yielded multiple opportunities to share research results, knowledge gained, and to share the development of the project as a model for others to replicate through conference presentations, workshops, consultations, speeches, and publications.[footnoteRef:28] Finally, data is also critically important to institutions and funders that provide resources to support clinical legal education, because it provides evidence of outcomes. [28: John Ammann, Emily A. Benfer, Lisa Bliss, Sylvia Caley, Elizabeth Tobin Tyler & Robert Pettignano, Advancing Health Law & Social Justice in the Clinic, the Classroom and the Community, 21 Annals Health L. 237, 237–56 (2012); Lisa R. Bliss, Robert Pettignano & Sylvia Caley, Bridging the Quality Gap with Medical-Legal Partnerships, 39 Physician’s Executive J. 44, 44–49 (2013); Lisa Bliss & Sylvia Caley, Exploring Cultural Competence in the Context of Medical-Legal Partnerships, Recent Developments in Health Care Law: Culture and Controversy, 25 HEC Forum 14, 14–21 (2012) [hereinafter Bliss et al., Exploring Cultural Competence]; Lisa Bliss, Sylvia Caley & Robert Pettignano, A Model for Interdisciplinary Clinical Education: Medical and Legal Professionals Learning and Working Together to Promote Public Health, 18 Int’l J. Clinical L. Educ. 149, 149–164 (2012) [hereinafter Bliss et al., Interdisciplinary Clinical Education]; Sylvia B. Caley, Lisa Radtke Bliss & Robert Pettignano, Speaking Their Language: Developing a Scorecard for Medical-Legal Partnerships to Balance Quality and Productivity, 5 Int’l J. Health, Wellness & Soc’y 9, 9–17 (2015); Pettignano et al., Benefit Patients with Asthma, supra note 28; Robert Pettignano, Sylvia Caley & Lisa Bliss, A Case for Including Lawyers on the Care Team, Physician Executive J. 34, 34–38 (2011) [hereinafter Pettignano et al., Case for Including Lawyers]; Robert Pettignano, Lisa Radtke Bliss & Sylvia Caley, Collaborative Diagnosis: Interdisciplinary Problem-Solving Between Physicians and Lawyers, 2 Int’l J. Health, Wellness & Soc’y 37, 37–41 (2013); Robert Pettignano, Lisa Bliss & Sylvia Caley, The Health Law Partnership: A Medical-Legal Partnership Strategically Designed to Provide a Coordinated Approach to Public Health Legal Services, Education, Advocacy, Evaluation, Research and Scholarship, 35 J. Legal Med. 57, 57–79 (2014) [hereinafter Pettignano et al., Health Law Partnership].] Part II What Does a Clinical Legal Education Research Project Look Like? A. Social Justice and the Social Determinants of Health Many definitions of social justice exist. A common thread is the focus on equal economic, political, and social rights and opportunities.[footnoteRef:29] Professionals in disciplines such as law, medicine, social work, public health, and ethics have long applied social justice principles to structural problems confronting the communities they serve. Each discipline uses its existing principles and organizational structures to address the needs of its constituencies. Within the healthcare field, there is a growing imperative to achieve health equity by addressing the social determinants of health, in order to improve health outcomes and reduce healthcare costs.[footnoteRef:30] Increasingly, healthcare providers are recognizing that they need to expand the healthcare team to include lawyers as well as problem-solvers from other disciplines in order to successfully address the complex social justice problems fermented by disparity and inequity.[footnoteRef:31] Successfully addressing such problems depends upon educating the next generation of professionals from all disciplines about the principles of social justice, and the tools and collaborative models that can be employed to address social inequity. [29: Pamela Edwards & Sheila Vance, Teaching Social Justice Through Legal Writing, 7 Legal Writing 63, 70 (2001). Two scholars have provided the following definition of social justice issues: “Social justice is the process of remedying oppression, which includes ‘exploitation, marginalization, powerlessness, cultural imperialism, and violence.’ Issues of social justice include problems involving race, ethnicity, and interracial conflict, ‘class conflict, gender distinctions, . . . religious differences,’ and sexual orientation conflicts. Social justice also includes public interest work in its many guises.” Id. at 64.] [30: Council on Cmty. Pediatrics, Am. Acad. of Pediatrics, Poverty and Child Health in the United States, Pediatrics, Apr. 2016, at 1, 1; Michael Marmot, Social Determinants of Health Inequalities, 365 The Lancet 1099, 1099 (2005).] [31: Bliss et al., Exploring Cultural Competence, supra note 29; Bliss et al., Interdisciplinary Clinical Education, supra note 29; Lisa Bliss, Sylvia Caley & Robert Pettignano, An Interdisciplinary Collaborative Approach to Wellness: Adding Lawyers to the Healthcare Team to Provide Integrated Care for Patients, 1 Int’l J. Health, Wellness & Soc’y 129 (2011); Pettignano et al., Benefit Patients with Asthma, supra note 28; Pettignano et al., Patients with Sickle Cell Disease, supra note 28.] Clinical legal education has been recognized as an ideal vehicle to promote teaching and learning of social justice.[footnoteRef:32] Effectively teaching social justice requires the infusion of social justice principles and values not only in law clinics, but throughout the curriculum.[footnoteRef:33] Justice education can equip students with the skills and experiences necessary to promote social change and to increase equality among individuals.[footnoteRef:34] Interprofessional collaboration is a particularly effective way to address the multiple determinants of health and to promote health and social equity. In the United States, the triple aim of providing interprofessional education combined with addressing social justice principles and improving health outcomes is embodied in the medical-legal partnership movement.[footnoteRef:35] Documenting outcomes and performing research that focuses on measuring the success of such collaboration is essential to better understanding the value of this approach. Such research can also ensure quality and continued improvement of the services provided by partnerships, promote the sustainability of partnerships, and encourage the formation of more MLP programs.[footnoteRef:36] [32: Lauren Caraskik, Justice in the Balance: An Evaluation of One Clinic’s Ability to Harmonize Teaching Practical Skills, Ethics and Professionalism with a Social Justice Mission, 16 S. Cal. Rev. L. & Soc. Just. 23, 23 (2006).] [33: See Ammann et al., supra note 29, at 237–56.] [34: Jane H. Aiken, Provocateurs for Justice, 7 Clinical L. Rev. 287, 288 (2001). Scholar Jane Aiken aspires to not only the Partnership students learn about justice through their clinical experience, but to also be a “provocateur” for justice, meaning one who inspires others to action, and who “actively imbues her students with a lifelong learning about justice, prompts them to name injustice, to recognize the role they may play in the perpetuation of injustice and to work toward a legal solution to that injustice.” Id.] [35: The National Center for Medical-Legal Partnership states that across the United States 155 hospitals, 139 health centers, 34 health schools, 126 legal aid agencies, and 52 law schools have some form of medical-legal partnership to address the social determinants of health. Partnerships Across the U.S., Nat’l Ctr. for Med.-Legal Partnership, http://medical-legalpartnership.org/need (last visited Aug. 31, 2016).] [36: Pettignano et al., Health Law Partnership, supra note 29, at 57–79.] B. Measuring the Impact of Social Justice Teaching in Clinical Legal Education While much is written about assessment and evaluation in law school education, experience in measuring the social justice impact of students’ clinical experience is very limited.[footnoteRef:37] To the extent that clinical programs employ the MLP interprofessional model, wide-ranging program research and evaluation focused on determining outcomes is nascent.[footnoteRef:38] Interprofessional learning experiences, in and of themselves, involve challenging and complex relationships particularly because most are created, at least in part, to contribute to transformative change in problem-solving.[footnoteRef:39] These interdisciplinary experiences are created not only to facilitate students’ transition from didactic learning to clinical practice, but also to expose students to complex issues facing our society.[footnoteRef:40] Because of this complexity, developing research projects and creating tools to evaluate effectiveness can be challenging. Moreover, few validated tools exist. Within the clinical legal education model, traditional assessment criteria are insufficient.[footnoteRef:41] Success is not simply mastery of a body of knowledge. The common methodology, testing, is not indicative of development of skills and values. Thus, there is both opportunity and challenge in developing evaluation and research projects employing realistic metrics. [37: Lisa Colarossi & Mary Ann Forgey, Evaluation Study of an Interdisciplinary Social Work and Law Curriculum for Domestic Violence, 42 J. Soc. Work Educ. 307, 307–23 (2006); Suellyn Scarnecchia, An Interdisciplinary Seminar in Child Abuse and Neglect with a Focus on Child Protection Practice, 31 U. Mich. J.L. Reform 33 (1997); Elizabeth Tobin Tyler, Allies Not Adversaries: Teaching Collaboration to the Next Generation of Doctors and Lawyers to Address Social Inequality, 11 J. Health Care L. & Pol’y 249 (2008); Janet Weinstein & Linda Morton, Interdisciplinary Problem Solving Courses as a Context for Nurturing Intrinsic Values, 13 Clinical L. Rev. 839 (2007).] [38: The National Center for Medical-Legal Partnership has developed performance measures to facilitate program evaluation efforts and systematic data collection by MLPs. These performance measures are currently being tested in the field by the members of the National Center. For more information, see Measuring Impact, Nat’l Ctr. for Med.-Legal Partnership, http://medical-legalpartnership.org/resources/measures (last visited Aug. 31, 2016). This effort is not focused on the particular needs of MLPs located within law school clinics. The metrics project is currently under development through a grant from the American Association of Medical Colleges, supra note 16, and includes an investigation of learner outcomes, including outcomes for law students.] [39: P. V. August et al., The T Assessment Tool: A Simple Metric for Assessing Multidisciplinary Graduate Education, 39 J. Nat. Resources & Life Sci. Educ. 15, 16 (2010).] [40: Karim S. Bandali et al., Innovations in Applied Health: Evaluating a Simulation-Enhanced, Interprofessional Curriculum, 34 Med. Tchr. e176, e176–177 (2012), http://dx.doi.org/10.3109/0142159X.2012.642829.] [41: Michael Field & Russell Lee, Assessment of Interdisciplinary Programmes, 27 European J. Educ. 277, 278 (1992).] Clinicians pursue developing MLP clinics not only because they believe in the benefits of interprofessional collaboration, but also because those very clinicians appreciate the complexity of the problems facing the disabled and chronically ill and understand that lack of equity is a root cause of health disparities.[footnoteRef:42] Studies have shown that patient outcomes, quality of care, and patient and provider satisfaction are improved in a collaborative practice setting.[footnoteRef:43] Socio-economic determinants of health affect health outcomes and legal intervention has the potential to effect improvement. Engaging in interprofessional problem solving creates a natural opportunity for students to practice, learn, and have significant impact on those they serve. Developing research protocols to evaluate whether these learning opportunities achieve the goals of improving health outcomes, addressing the social determinants of health, promoting social justice, and educating law students will facilitate further development of MLP law school-based clinics. Key ingredients in determining the effectiveness of interprofessional efforts to address issues of social justice, include recognizing that measuring success is directly related to the learning objectives established in the curriculum; emphasizing the unique goals of the project or endeavor that is the focus of the educational experience; creating an evaluation method that matches the goals; and employing an array of assessment techniques. [footnoteRef:44] [42: Pettignano et al., Case for Including Lawyers, supra note 29, at 34–35.] [43: J. G. Baggs, Intensive Care Unit Use and Collaboration Between Nurses and Physicians, 18 Heart & Lung: J. Critical Care 332 (1989); William A. Knaus, Elizabeth A. Drapter, Douglas P. Wagner & Jack E. Zimmerman, An Evaluation of Outcome from Intensive Care in Major Medical Centers, 104 Annals of Med. 410 (1986); Nurse-Physician Communication Affects Patient Care, 16 Health Affairs 29 (1984).] [44: Field & Lee, supra note 42, at 282. See also Barbara Glesner Fines, Outcomes Assessment for Improving Student Learning, in Building Best Practices: Transforming Legal Education in a Changing World, supra note 23, at 94, 94–100; Judith Welch Wegner, Assessment Plans that Support Student Learning, in Building Best Practices: Transforming Legal Education in a Changing World, supra note 23, at 25, 25–36.] One option for other clinics interested in developing evaluation and research projects is to create an independent research team of students to help with study design and implementation. Using this model, students are able to engage in all of the steps of a well-designed research project. By creating a “research clinic,” law students receive an applied learning experience that is as educational as other clinic models. A research clinic project may also make evaluation much more feasible for a program that may lack the resources to take on this kind of important assessment. Finally, some students may bring valuable skills to such a project that improve it and can be incorporated into future projects. Of course, as described below, a research clinic would need to comply with any research ethics obligations, which may include training for participating students if they are engaged in human subjects research. C. Establishing the Objectives for Measuring Impact and Success The overarching goal of inter-professional collaboration is creating the climate in which participants are able to develop understanding of and embrace differences among the professional groups in order to develop common understanding.[footnoteRef:45] Interprofessional programs have complex outcomes, and selecting metrics is difficult if those outcomes are not clearly identified.[footnoteRef:46] Common themes emerge from these widely held objectives and goals. Most faculties engaging in interprofessional work aspire to develop specific core attributes in their students following their exposure to the work. These core skills include: discipline role clarity; ability to understand the roles of other disciplines; skills in negotiating roles and managing role conflict; developing effective communication skills; developing group process skills, becoming more self-aware, and developing a positive attitude toward collaborative work.[footnoteRef:47] [45: Gunilla Öberg, Facilitating Interdisciplinary Work: Using Quality Assessment to Create Common Ground, 57 Higher Educ. 405, 406 (2009).] [46: Donald E. Stowe & Douglas J. Eder, Interdisciplinary Program Assessment, Issues Integrative Stud. 77, 84-85 (2002).] [47: Colarrossi & Forgey, supra note 38, at 307.] Common threads connect varied disciplines when examining motivations behind developing interdisciplinary educational experiences. Themes include integration of knowledge, need for innovation, development of deductive reasoning, promoting sophisticated analysis, developing the ability to reason by analogy, and, ultimately, development of synthetic thinking.[footnoteRef:48] Developing synthetic thinking involves employing integrative devices of “epistemic frames that enable [students] to articulate 2 or more disciplinary understandings.”[footnoteRef:49] Scholars have identified 21 cognitive skills associated with integrated experiences ranging from the development of critical thinking and recognition of bias to enhanced awareness of ethical issues.[footnoteRef:50] Fundamentally, the common objectives of interprofessional education are to improve the students’ foundation in their chosen disciplines, to advance their overall ability integrate problem-solving skills from other disciplines, and to arrive at a heightened level of critical awareness at the conclusion of the experience.[footnoteRef:51] [48: Veronica Boix Mansilla & Elizabeth Dawes Duraisingh, Targeted Assessment of Students’ Interdisciplinary Work: An Empirically Grounded Framework Proposed, 78 J. Higher Educ. 215, 217–18 (2007).] [49: Id. at 226.] [50: Id. at 217.] [51: Id. at 222–29.] One identifier for determining whether some degree of integration is taking place within an interprofessional experience is to evaluate whether “considered judgment”[footnoteRef:52] is at play. Properties of considered judgment include the ability to weigh options, make compromises, and make decisions from the best available data. Engaging in considered judgment develops disciplinary foundations, promotes assimilation of other disciplines’ problem-solving methods, and instills understanding of the goals, benefits, and limitations of interprofessional practice. Success of an integrative experience may be measured by the “degree to which it achieves its purpose.”[footnoteRef:53] [52: Id. at 229.] [53: Id.] One tool employed by most clinicians in facilitating learning is to require that students engage in some form of regular, reflective journaling. Reflection, facilitated through activities, such as journaling exercises and the practice of reviewing memorialized reflections over time, provides insight into the development of critical awareness. Examining structured reflections permits the opportunity to assess problems identified, options available, choices made, compromises reached, degree of collaboration achieved among the interdisciplinary participants, issues of conflict, ethical dilemmas, and advances in thinking. D. Identifying the Metrics Within discipline-related legal education courses or clinical experiences, the goals of learning and the assessment principles are common and generally agreed upon.[footnoteRef:54] Also familiar is the need for assessment tools to measure change across the experience.[footnoteRef:55] These common, agreed upon features can be incorporated into an evaluation or research protocol. Educators in interprofessional programs share the need to determine whether students’ have developed knowledge of their own discipline’s role, practices, ethics, and duties as well as those of the disciplines of their collaborators. Also important is assessing knowledge attainment of the specific content—health law, tax law, small business issues, or elder issues—and the role of other disciplines in holistically solving problems confronting clients. Assessing the extent to which the students developed skills necessary to perform the work required of the experience is an important component. Did the interprofessional experience enhance their acquisition of interviewing, counseling, negotiation, problem-solving, and/or conflict resolution skills? Were members of the interprofessional teams able to effectively deploy the acquired skills? How did attitudes and understanding of professional values evolve over the learning period? Were client experiences and outcomes improved due to the interdisciplinary collaboration? [54: See Bliss & Peters, supra note 23, at 188–215.] [55: Colarrossi & Forgey, supra note 38, at 309.] Some interprofessional legal education programs utilize validated student evaluations to assess development of knowledge, skills, and values.[footnoteRef:56] Some examples of assessment tools include Attitudes Toward Interdisciplinary Teams Scale, Team Skills Scale (measures improved knowledge about teaming skills, ability to carry out tasks that contribute to quality teaming), and the General Intellectual Skills test.[footnoteRef:57] The Educational Testing Service maintains information of available tests.[footnoteRef:58] Using standardized, validated instruments to document student performance can establish that the interprofessional program at issue is performing well, attaining predetermined benchmarks, and is worthy of retention and support. Standardized instruments permit comparisons with similarly situated programs. [56: Constance L. Coogle, Iris A. Parham, E. Ayn Welleford & F. Ellen Netting, Evaluation of a Distance Learning Course in Geriatric Interdisciplinary Teaming, 28 Educ. Gerontology 791, 796 (2002).] [57: Id.] [58: See Educ. Testing Serv., https://www.ets.org (last visited Aug. 31, 2016).] In interprofessional education programs, qualitative measures may be more useful.[footnoteRef:59] Quantitative measures evaluate attainment of clearly specified learning objectives and a numerical measure is affixed to performance. Assumptions made in statistical analysis can be problematic.[footnoteRef:60] Qualitative measures, on the other hand, may be more flexible. “The complexity of human thought and discourse does not necessarily lend itself to numerical analysis.”[footnoteRef:61] [59: Field & Lee, supra note 42, at 279.] [60: Id.] [61: Id.] Challenges are associated with measuring the impact of social justice teaching. To what degree has a student’s understanding been advanced because of the interdisciplinary experience? What is the value-added achieved by the combination and balancing of multiple disciplinary views? Can students from different disciplines enrolled in an interprofessional experience be evaluated in the same manner? Will assessing students’ performance be useful and relevant to them as they move along their paths to professional identity and completion of their studies?[footnoteRef:62] How do we determine the key features to be evaluated? For instance, should we determine whether differences in problem-solving approaches were managed effectively? [62: Öberg, supra note 46, at 406–09.] E. A Final Word on the Partnership’s Experience with Evaluation and Research While the delivery of free legal services, educational endeavors, and systemic advocacy represent the core programs of the Partnership, program improvement, growth, and sustainability would not have been possible without the evaluation and research component. Engaging in evaluation and research has brought all participants in the Partnership—partners, faculty, staff, contractors, learners, clients, and members of the Partnership Advisory Council[footnoteRef:63]—together in a coordinated effort to prove the utility of the Partnership’s version of the MLP model. In many respects, research and evaluation is the glue that binds the discreet actors into a cohesive and successful collaboration. [63: The founding partners of the Partnership determined during the process of establishing the partnership that creating an advisory council to assist with the development, growth, and sustainability of the project would benefit the endeavor. Initially, members of the Partnership Advisory Council were lawyers practicing in healthcare law at private firms in City, State. Over time, as relationships developed, the Advisory Council membership diversified. Today, membership includes physicians, social workers, nurses, and former students as well as lawyers specializing in healthcare law. The group meets quarterly and has proven to be instrumental in increasing the profile of the Partnership.] Part III Developing an Empirical Research Project Most legal scholars are not trained in empirical research. Legal research generally is a relatively solitary activity – a scholar identifies a question of interests, searches and reviews the literature, and writes. Rarely does a legal scholar report the paths her research took her along and what parts were included or excluded from the final product. Empirical research is quite different. It demands a specific research question matched with an appropriate research methodology that provides data that will answer the question. There is an expectation that the research question and the methodology will be explicitly described when the results are published and that deviations from the methodology will be explicitly described and justified.[footnoteRef:64] In addition, the research typically cannot be commenced until it has been reviewed and approved by an oversight committee.[footnoteRef:65] This section provides guidance for developing a research question and navigating the research oversight system. [64: For a useful resource describing research methods that can fit legal analysis, see Public Health Law Research: Theory and Methods (Alexander C. Wagenaar & Scott Burris eds., 2013). The Robert Wood Johnson Foundation, which has supported the empirical research in public health law, has made some of the material contained in the book available on its website. See PHLR Methods, Pub. Health L. Res., http://publichealthlawresearch.org/resources/methods (last visited Aug. 31, 2016).] [65: The oversight committee may have different names in different countries, although their functions are similar. In the United States, these are referred to as Institutional Review Boards (IRBs), whereas in the UK and other countries these are referred to as research ethics committees (RECs). Having already mentioned the IRB approval for the Partnership research program, we will continue to use this term.] A. Developing a Research Question The first step is to develop a research question. In short, the researcher needs to ask herself what she wants to know. Steven R. Cummings and colleagues developed the FINER criteria for developing a good research question (See Table 1).[footnoteRef:66] According to this approach, a good research question is Feasible, Interesting, Novel, Ethical, and Relevant. [66: Stephen B. Hulley et al., Designing Clinical Research: An Epidemiologic Approach 19–21 tbl2.1 (2nd ed. 2001).] Table 1: FINER Criteria for a Good Research Question From: Stephen B. Hulley et al., Designing Clinical Research: An Epidemiologic Approach 19 (2nd ed. 2001). Reprinted with permission by Wolters Kluwer Feasible Adequate number of subjects Affordable in time and money Manageable in scope Adequate technical expertise Interesting To the investigator Novel Confirms or refutes previous findings Extends previous findings Provides new findings Ethical Relevant To scientific knowledge To future research directions To clinical and health policy Although the authors’ discussion focuses on clinical research – i.e., research designed to test the safety and effectiveness of medical treatment, drugs, devices, and diagnostic tools – the criteria are adaptable to other contexts and serve as a useful guide for developing a good research question for measuring the impact of social justice teaching. Some questions the researcher should be thinking about in developing her research question include (1) who and what she wants to measure; (2) why she wants to measure it (why does the answer matter?); (3) how she is going to measure it; and (4) what is she going to do with the information once she has collected it. This last question is important because many new researchers collect more information than they need. Doing so can be problematic for several reasons. Too much data can be overwhelming and even hamper the ability to make anything of it. Collection of irrelevant data wastes researcher and participant time and resources. Finally, collection of irrelevant personal data can unnecessarily increase the risks to participants. Unfortunately, many of the things that researchers would like to study do not meet the FINER criteria. For example, clinical educators may want to know whether students who participate in social justice clinics are better lawyers. Even if the researcher could define what counts as a “better” or “good” lawyer and measure it, the study would be hard to conduct because not all students participate in social justice clinics and those who do may be different in important ways from those who do not participate. When the researcher faces such an issue, the FINER criteria help point the researcher towards ways of revising the question so that the research can be conducted. (See Table 2).[footnoteRef:67] For example, the question above might be modified to ask whether students who participate in social justice clinics are better lawyers (however that is defined) than those who participate in lawyering skills courses that do not have a social justice component. [67: Id. at 19–21 tbl2.2.] Table 2: The Research Question and Study Plan: Problems and Solutions From: Stephen B. Hulley et al., Designing Clinical Research: An Epidemiologic Approach 21 (2nd ed. 2001). Reprinted with permission by Wolters Kluwer Potential Problem Solutions A. The research question if not FINER 1. Not feasible Too broad Specify a smaller set of variables Narrow the question Not enough subjects available Expand the inclusion criteria Eliminate or modify exclusion criteria Add other sources of subjects Lengthen the time frame for entry into study Use strategies to decrease sample size Methods beyond the skills of the investigator Collaborate with colleagues who have the skills Consult experts and review the literature for alternative methods Learn the skills Too expensive Consider less costly study designs Few subjects and measurements Less extensive measurements Fewer follow-up visits 2. Not interesting, novel, or relevant Consult with mentor Modify the research question 3. Uncertain ethical suitability Consult with institutional review board Modify the research question B. The study plan is vague Write the research question at an early stage Get specific in the one- to two-page study plan How the subjects will be sampled How the variables will be measured In developing the research question, the researcher should be thinking about what kinds of research methods and data match the question she is trying to answer. A discussion of all of the different research methods available is beyond the scope of this article, but a researcher should think about whether she wants quantitative data – e.g., data that reflect that the number of graduates who perform pro bono hours following their experience in social justice clinics vs. those without that experience – or qualitative data – e.g., data that gives a rich description of the value graduates place on their experiences in social justice clinics and how their participation impacts the way they practice law.[footnoteRef:68] Although these are quite different approaches, they need not be considered mutually exclusive. Indeed, mixed-methods approaches that involve both quantitative and qualitative approaches within the same research project can provide a fuller picture than can be obtained with one method alone.[footnoteRef:69] To provide a familiar example, student evaluations typically provide both quantitative and qualitative data. Students usually are asked to rank the professor and the course on a number of metrics on a numeric scale. These responses are relatively easily analyzed because there are limited choices (“closed-questions”) and numeric representations of counts and average ratings have meaning. Because the same questions are asked across all courses, the summary rankings of the professor and course can be compared to others. But student evaluations also typically ask students for comments – open-ended questions that allow students free rein to express their opinions about the professor and the course. These data are not easily analyzed, but help to explain why, for example, students rated a professor highly or poorly. Sometimes, these comments may reveal external influences (e.g., an 8 am Friday class time) that result in low professor and course ratings. [68: For helpful introductions to different types of research, see the public health research methods materials cited supra note 65; Hulley et al., supra note 67. There are numerous resources available that provide more detailed information about specific research methods.] [69: For a comprehensive discussion of the benefits and drawbacks of mixed-methods research, see R. Burke Johnson & Anthony J. Onwuegbuzie, Mixed Methods Research: A Research Paradigm Whose Time Has Come, 33 Educ. Researcher 14, 14–26 (2004).] The researcher also should think about what is known about the question in terms of designing the research. When no research has previously been conducted, any information may be informative. Thus, a quick snapshot may be sufficient. On the other hand, when one issue has been well-studied, it may be more important to look in a more targeted fashion or to build on that research and add a new dimension. The researcher also should think about the skills needed to complete the research effectively. Most law professors–clinical and otherwise–are not trained in empirical research methods. Tools such as SurveyMonkey™ have made it easier for people to create surveys,[footnoteRef:70] but developing quality surveys that are understandable and produce quality data requires considerable skill. Similarly, although numerous statistical calculators are available free on the internet,[footnoteRef:71] the user needs to understand what statistical test is appropriate to what kind of data. Accordingly, the clinician who wants to study the impact of social justice teaching should consider collaborating with people who have the requisite training to ensure that the project provides useful, valid results. Some universities or university departments may provide consultation services for study design and statistical analysis. If such services are not available, it is worth looking for a colleague or a PhD student with the requisite skill set who would be interested in collaborating on the research project. The value of getting input early in the development of the research project cannot be overestimated. Many a research project has failed to reach its full potential because information required for the desired analysis was not collected or an ambiguity in a survey question was not discovered until after the surveys were completed. [70: Survey Monkey, https://www.surveymonkey.com (last visited Aug. 31, 2016).] [71: See, e.g., Statistical Test Calculators, Soc. Sci. Stat., http://www.socscistatistics.com/tests (last visited Aug. 31, 2016); QuickCalcs, GraphPad, http://www.graphpad.com/quickcalcs (last visited Aug. 31, 2016).] Even with careful planning, research can go awry. Research depends on the cooperation of others. In the context of a clinic, researchers may rely on clinic staff to request participation and/or to complete data collection forms. However, those research activities often are in addition to their regular duties. Accordingly, they may occasionally or regularly, depending on their commitment to the project and the exigencies of their work, forget to ask for participation or complete data collection forms. Or they may ask only “cooperative” people to participate, thereby skewing the data. Even when staff members perform all the research tasks perfectly, the people they recruit (e.g., students and clients) may refuse to participate or withdraw from participation after first agreeing to participate, in keeping with their legal and ethical rights (discussed more fully below). Or those participants may agree to participate and simply forget to respond, despite staff members repeated efforts to get them to do so. These kinds of challenges are to be expected at some level in every research project. However, it is important to be aware of them so that planning can help to minimize the negative effects of such issues on the research project. For example, anticipated dropout rates can and should be taken into account in calculating the size of a study sample. Even if research has not been conducted in the study population before, analogous populations can provide a reasonable estimate for dropouts or response rates. Similarly, development of procedures, checklists, and training can increase the consistency of data collection. Another important consideration for the researcher is how the research will be funded. Much research will be conducted in the ordinary course of a legal academic’s scholarship. The benefit to this approach is that the academic typically does not need to search for external funding. The drawback to this approach is that the demands of the clinic and other obligations may leave little time for the research. In addition, there may be no funding for methodological and statistical expertise. External funding – e.g., a grant from a government agency or foundation – can provide important resources to support the project, including funds to pay for methodological expertise. However, external funders may prioritize different questions than the researcher. In addition, if their reporting requirements are frequent and detailed, they can distract from the research or even the main mission of the program. Accordingly, it is essential to think carefully about whether the external funding advances the research goals of the program and that the benefits of the funding received outweigh the burdens involved in maintaining that relationship. The Partnership has taken a hybrid approach. Our clinicians have engaged in empirical research from the beginning of the program as part of their scholarship activities. However, we have also allocated money within the Center for Law, Health and Society budget (with which the Partnership is affiliated within the University) and participated in fundraising with our community partners to support an evaluator for the research program. In addition, the Partnership has applied for and received external funding that has supported a variety of research projects. B. Legal and Ethical Requirements for Research Research involving human beings from simple surveys to complex biomedical clinical trials must comply with applicable ethical and legal standards. Legal academics who are new to empirical research may be unfamiliar with these requirements. Those who are familiar with them often complain about the burden imposed on them and the time wasted in complying with the laws.[footnoteRef:72] Accordingly, it is important to understand how the requirements came into being and why they remain essential ethical requirements. [72: See, e.g., Human Subjects Research Protections: Enhancing Protections for Research Subjects and Reducing Burden, Delay, and Ambiguity for Investigators, 76 Fed. Reg. 44512, 44,512–15 (proposed July 26, 2011) (to be codified at 21 C.F.R pts. 50, 56). The background section of this advance notice of proposed rule-making to amend the Common Rule details some of the criticisms researchers have articulated about the U.S. research oversight regulations. Id.] History. The legal and ethical standards that have come to apply to human subjects research have their history in scandal. The story normally begins with the Nuremberg trial of the Nazi doctors. The doctors conducted a variety of experiments on inmates in the concentration camps during World War II. The experiments can be separated into two categories: (1) research intended to help the Nazi war effort, including experiments on hypothermia and tropical diseases such as malaria, and (2) research intended to support the “Final Solution,” which focused on more efficient ways to render inmates infertile or to kill. In both categories, inmates were physically harmed, often severely, and many died.[footnoteRef:73] The Nuremberg Code, issued as part of the judgment against these doctors is the first widely recognized research ethics code.[footnoteRef:74] The horrors of the Nazi experiments similarly influenced the World Medical Association to issue the Declaration of Helsinki.[footnoteRef:75] The Declaration, which has been updated multiple times since it was first issued in 1964, echoes many of the principles embodied in the Nuremberg Code. [73: For a detailed description of the experiments conducted in the concentration camps, see The Nazi Doctors and the Nuremberg Code: Human Rights in Human Experimentation (George J. Annas & Michael A. Grodin, eds., 1995).] [74: See, e.g., Evelyn Shuster, Fifty Years Later: The Significance of the Nuremberg Code, 337 New Eng. J. Med. 1436, 1436 (1997); Michel Thieren & Alexandre Mauron, Nuremberg Code Turns 60, 85 Bull. World Health Org. 573, 573 (2007), http://www.who.int/bulletin/volumes/85/8/07-045443; Nuremberg Code, U.S. Holocaust Memorial Museum, https://www.ushmm.org/information/exhibitions/online-exhibitions/special-focus/doctors-trial/nuremberg-code (last visited Aug. 31, 2016) [hereinafter Nuremberg Code].] [75: World Med. Ass’n, Declaration of Helsinki: Recommendations Guiding Physicians in Biomedical Research Involving Human Subjects 1-2, http://www.wma.net/en/60about/70history/01declarationHelsinki/index.html.pdf (last visited Aug. 31, 2016). The Declaration is well regard around the world and has been incorporated into the laws and guidelines of many countries. For a full discussion, see Delon Human & Sev S. Fluss, The World Medical Association’s Declaration of Helsinki: Historical and Contemporary Perspectives (2001), http://www.wma.net/en/20activities/10ethics/10helsinki/draft_historical_contemporary_perspectives.pdf. For the text of the current declaration, see Declaration of Helsinki: Ethical Principles for Medical Research Involving Human Subjects, World Med. Ass’n, http://www.wma.net/en/30publications/10policies/b3/index.html (last visited Aug. 31, 2016) [hereinafter Declaration of Helsinki].] Despite its involvement in putting the Nazi doctors on trial, the United States did not adopt its own ethical and legal standards to govern human subjects research until after public revelations about the Tuskegee Syphilis study.[footnoteRef:76] The study involved 400 poor, black men in rural Alabama with syphilis. The men, who were followed for 40 years, not only were not treated for syphilis, even after effective treatment became available, but were actively prevented from receiving treatment. Some of the men’s wives and girlfriends and their children contracted syphilis as a result of this decision. It is not clear that the men were told they had syphilis or even knew that they were in a research study. The National Commission for the Protection of Human Subjects, created in the wake of the scandal, developed ethical principles for governing human subjects research in The Belmont Report.[footnoteRef:77] Those principles are incorporated into the legal requirement set forth in the federal regulations governing human subjects research known as the Common Rule.[footnoteRef:78] [76: U.S. Public Health Service Syphilis Study at Tuskegee, Ctr. for Disease Control & Prevention, http://www.cdc.gov/tuskegee/timeline.htm (last visited Aug. 31, 2016).] [77: Protection of Human Subjects; Belmont Report: Ethical Principles and Guidelines for the Protection of Human Subjects of Research, 44 Fed. Reg. 23,192 (April 18, 1979); Nat’l Comm’n for the Prot. of Human Subjects of Biomedical & Behavioral Research, The Belmont Report: Ethical Principles and Guidelines for the Protection of Human Subjects of Research, U.S. Dep’t Health & Hum. Services (April 18, 1979), http://www.hhs.gov/ohrp/regulations-and-policy/belmont-report [hereinafter The Belmont Report].] [78: Office for Human Research Prot., Federal Policy for the Protection of Human Subjects, U.S. Dep’t Health & Hum. Services, http://www.hhs.gov/ohrp/regulations-and-policy/regulations/common-rule/index.html (last visited Aug 31, 2016). In 2011, the federal government issued an advance notice of proposed rule-making to amend the Common Rule for the first time since 1981. See Human Subjects Research Protections: Enhancing Protections for Research Subjects and Reducing Burden, Delay, and Ambiguity for Investigators, 76 Fed. Reg. at 44,512–15. A notice of proposed rule-making was issued in 2015, but has been widely criticized. See Office for Human Research Prot., NPRM for Revisions to the Common Rule, U.S. Dep’t Health & Hum. Services, http://www.hhs.gov/ohrp/regulations-and-policy/regulations/nprm-home (last visited Aug 31, 2016) [hereinafter Office for Human Research Prot., NPRM for Revisions]. As of this writing, no changes have been made to the Common Rule.] The scandals involving human subjects research have not ended with the adoption of legal and ethical standards. The last decades have surfaced a host of additional research scandals, both old and new. For example, historian Susan Reverby unearthed documents that revealed that, following World War II, the United States engaged in research on sexual transmitted infections (STIs) among prisoners, soldiers, native populations, and children in Guatemala. In these studies, the researchers actively infected the research subjects with STIs.[footnoteRef:79] There are also a range of social-behavioral studies that gave rise to significant ethical concerns. In the Milgram obedience studies, research subjects assigned as “teachers” were instructed by researchers to administer escalating electric shocks to the “learner” for incorrect answer. Many subjects administered the highest level, despite yells of pain from the learner in the next room. Unbeknownst to the subjects, the “learners” were confederates of the researchers and no shocks were actually administered.[footnoteRef:80] In the Stanford Prison Study, young college men were assigned to the roles of guards or prisoners in a simulated prison in the basement of the Stanford psychology department. The study had to be stopped after a few days, when the “guards” became increasingly abusive to the “prisoners,” causing mental distress among them.[footnoteRef:81] [79: Susan M. Reverby, “Normal Exposure” and Inoculation Syphilis: A PHS ‘Tuskegee’ Doctor in Guatemala, 1946–1948, 23 J. Pol’y Hist. 6, 6-28 (2011). A common misconception about the Tuskegee Syphilis study is that the researchers infected the men with syphilis. While untrue in that study, the Guatemala studies demonstrated that the suspicion that the US Public Health Service would do so was not unwarranted. Id. For the Presidential Commission for the Study of Bioethical Issue’s report about the Guatemala studies, available in English and Spanish, see Presidential Comm’n for the Study of Bioethical Issues, “Ethically Impossible” STD Research in Guatemala from 1946 to 1948 (2011), http://bioethics.gov/node/654. The discovery of the Guatemala studies prompted a governmental review of U.S.-supported human subjects research. For the Commission’s report, see Presidential Comm’n for the Study of Bioethical Issues, Moral Science: Protecting Participants in Human Subjects Research (2011), http://bioethics.gov/node/558.] [80: See Saul McLeod, The Milgram Experiment, Simply Psychology (2007), http://www.simplypsychology.org/milgram.html.] [81: See Romesh Ratnesar, The Menace Within, Stan. Mag. (2011), https://alumni.stanford.edu/get/page/magazine/article/?article_id=40741; Philip G. Zimbardo, Stan. Prison Experiment, http://www.prisonexp.org/ (last visited Aug. 31, 2016) (hosted by Philip Zimbardo, the psychology professor who conducted the experiment).] Common principles. Although the legal and ethical standards vary in detail around the world,[footnoteRef:82] there is general agreement about some basic aspects. [82: The U.S. Office for Human Research Protections (OHRP), which is part of the Department of Health and Human Resources, has collected international resources on human subjects research. For more information or to access these materials, see Office for Human Research Prot., International, U.S. Dep’t Health & Hum. Services, http://www.hhs.gov/ohrp/international/index.html (last visited Aug. 31, 2016). In addition to collecting links to US laws, ethical codes like the World Medical Association’s Declaration of Helsinki, Declaration of Helsinki, supra note 76, the OHRP website includes an International Compilation of Human Research Standards, which lists over 1,000 laws, regulations, and guidelines governing human subjects research in 113 countries, as well as international and regional organizations. See Office for Human Research Prot., supra.] First, there is general agreement that research must follow three basic ethical principles: (1) that participants are treated with respect, including seeking informed consent from individual participants (respect for persons); (2) that benefits will be maximized and risks minimized (beneficence); and (3) there will be equitable selection of participants (justice).[footnoteRef:83] [83: See 45 C.F.R. § 46.111 (2016); Nuremberg Code, supra note 75; Declaration of Helsinki, supra note 76; The Belmont Report, supra note 78.] Second, there is general agreement that there should be independent review of a research project before research commences. In the United States, the reviewing body is call an Institutional Review Board (IRB) because they generally are housed at the university, hospital, or other research body that is conducting the research. In other countries, these may be called research ethics committees (REC). The general requirements may be eased if the research presents very little risk to the participants. These circumstances will be specified in the law or ethical code. For example, in the United States, research involving interview and survey methods are generally considered to present minimal risk to participants and may be exempt from the research regulations.[footnoteRef:84] However, these regulations represent a legal floor, and an IRB may determine that greater scrutiny is necessary. For example, an IRB may impose stricter requirements on research on sensitive topics, such as HIV that may be stigmatizing if revealed, even when they use research methods that typically pose little risk. Similarly, IRBs may be concerned about use of student data, which is often protected by law. The flexibility afforded to institutions in applying the regulations also leads to some of the complaints leveled against IRBs. [84: 45 C.F.R. 46.101(b)(2) (2016).] While it is useful to understand the general principles that govern human subjects research, it is essential to understand that rules that apply specifically to the country or countries in which the research is conducted. The U.S. Office for Human Research Protections’ (OHRP) has compiled a list of laws, regulations, and guidelines governing human subjects research in over 100 countries.[footnoteRef:85] This is a helpful reference for the relevant laws in a particular country. However, because the laws are often complex and institutions have their own requirements with respect to training and procedures, new researchers should begin their introduction into these rules through their own institution. Information may be housed in a central or departmental research office, research ethics committee office, or legal counsel’s offices. In addition, other researchers may be helpful in understanding obligations, as well as provide practical advice about how to navigate the oversight requirements. However, as discussed in the next section, some researchers may view of the oversight system as an impediment to research, which may skew their perceptions of the process. Accordingly, it is important to keep in mind, despite frustrations, researchers and research ethics committees share the same goal of protecting the rights and well-being of human subjects while advancing important research. [85: Office for Human Research Prot, supra note 83.] Working within the human subjects oversight system. The literature is full of complaints about the human subjects oversight system.[footnoteRef:86] Researchers complain about inconsistencies among IRB reviews, long delays in receiving requisite approvals, which contribute to research costs, and a lack of transparency in decision-making. Some of these criticisms are valid. IRBs are made up of human beings, who can make mistakes in interpretation. Increased research activity and scrutiny of research oversight without additional investment has led to delays. But the regulations also allow for flexibility and consideration of local conditions, so some differences in reviews may merely illustrate the regulations in action. Moreover, researchers can also contribute to problems in review. Failing to follow instructions, provide requested information, or respond in a timely fashion can substantial delay the review process. [86: The advance notice of proposed rule-making related to the Common Rule summarizes many of the criticisms. See Office for Human Research Prot., NPRM for Revisions, supra note 79.] Researchers need to take the human subjects oversight process as seriously as they take their study design. Conducting research with human beings is a privilege, not a right, and it requires substantial thought. Accordingly, care should be taken to provide the IRB with the information that it needs to do its work. Understanding the reasons for the system helps researchers understand why they are providing the information they are, as well as to help them explain to the IRB why they are pursing the research in the way that they are. It also helps researchers to engage with the IRB when it raises questions or expresses concern about a study. Researchers do not need to capitulate to IRB requests if the research demands otherwise; instead, they should reach out to the committee chair, members, or staff, as permitted by the committee rules, and work together to find a solution that meets the requirements of the research and the human subjects obligations. Such negotiation is likely to be most effective when the researcher understands and respects the legal and ethical requirements with which the IRB must comply and discusses how their research fits within those requirements. Part IV Conclusion As predicted at the inception of the project, the Partnership and the Partnership Clinic have proved to be rich sources of data regarding the effectiveness and impact of interprofessional collaboration and education on students, as well as on the lives of clients and their families, all of whom are low income and most of whom are facing multiple hardships as the result of chronic illness or disease combined with other disparities. By going through the process of identifying the aims of the project, and the questions we wanted to explore, creating and refining survey instruments, and continuing to collect data year after year, we have created a resource of information that has provided multiple benefits to our own project as well as served to educate and inform others. This data has allowed us to assess the impact of our work, to improve referral systems, the services we provide, and the nature of education offered to students, and ultimately to understand whether our dual aims of educating law students about justice and health equity and holistically addressing the health problems of low income children and their families are being met. We encourage other educators to join us in the journey of discovering the power of research to inform the future. 7