Research utilization in higher education rulemaking: A multi-case study of research prevalence, sources, and barriers Journal website: http://epaa.asu.edu/ojs/ Manuscript received: 9/6/2019 Facebook: /EPAAA Revisions received: 3/2/2020 Twitter: @epaa_aape Accepted: 4/3/2020 education policy analysis archives A peer-reviewed, independent, open access, multilingual journal Arizona State University Volume 28 Number 95 June 22, 2020 ISSN 1068-2341 Research Utilization in Higher Education Rulemaking: A Multi-Case Study of Research Prevalence, Sources, and Barriers Rebecca S. Natow Hofstra University United States Citation: Natow, R. S. (2020). Research utilization in higher education rulemaking: A multi-case study of research prevalence, sources, and barriers. Education Policy Analysis Archives, 28(95). https://doi.org/10.14507/epaa.28.5048 Abstract: For stakeholders who would like to see more research as a basis for educational policy, it is important to understand the prevalence of research use and the sources of the studies used by policymakers, as well as the factors that hinder research use in educational policymaking. Through an analysis of regulatory documents and interviews with 34 key informants, this multi-case study examined the prevalence and sources of research utilized in higher education rulemaking, which is the process for developing federal regulations that govern higher education. This study also examined barriers to using research in h igher education rulemaking. Findings indicate that while research has been used in this process, factors other than research were discussed more frequently in final regulations. Barriers to research use in higher education rulemaking included time constraints, unavailability of data, politics, lack of government research capacity, and other disjunctions between the research and policy communities. Moreover, the contexts in which particular rules were http://epaa.asu.edu/ojs/ https://doi.org/10.14507/epaa.28.5048 Education Policy Analysis Archives Vol. 28 No. 95 2 created shaped the prevalence and sources of research use d in the regulations’ development. The article concludes with implications for policy and theory. Keywords: research use; higher education; educational policy; rulemaking; federal policy; United States; case study Utilización de la investigación en la reglamentación de la educación superior: Un estudio de casos múltiples de prevalencia de investigación, fuentes y barreras Resumen: Para las partes interesadas que deseen ver más investigación como base para la política educativa, es importante comprender la prevalencia del uso de la investigación y las fuentes de los estudios utilizados por los formuladores de políticas, así como los factores que dificultan el uso de la investigación en la formulación de políticas educativas. Mediante un análisis de documentos reglamentarios y entrevistas con 34 informantes clave, este estudio de casos múltiples examinó la prevalencia y las fuentes de investigación utilizadas en la reglamentación de la educación superior, que es el proceso para desar rollar regulaciones federales que rigen la educación superior. Este estudio también examinó las barreras para usar la investigación en la elaboración de normas de educación superior. Los resultados indican que, si bien la investigación se ha utilizado en e ste proceso, otros factores además de la investigación se discutieron con mayor frecuencia en los reglamentos finales. Las barreras para el uso de la investigación en la reglamentación de la educación superior incluyeron limitaciones de tiempo, falta de disponibilidad de datos, política, falta de capacidad de investigación del gobierno y otras disyunciones entre las comunidades de investigación y políticas. Además, los contextos en los que se crearon reglas particulares moldearon la prevalencia y las fuentes de investigación utilizadas en el desarrollo de las regulaciones. El artículo concluye con implicaciones para la política y la teoría. Palabras-clave: uso de investigación; educación superior; política educativa; reglamentación política federal; Estados Unidos; caso de e studio Utilização da pesquisa na elaboração de regras do ensino superior: Um estudo de casos múltiplos de prevalência, fontes e barreiras da pesquisa Resumo: Para as partes interessadas que desejam ver mais pesquisas como base para a política educacional, é importante entender a prevalência do uso da pesquisa e as fontes dos estudos utilizados pelos formuladores de políticas, bem como os fatores que dificultam o uso da pesquisa na formulação de políticas educacionais. . Por meio de uma análise de documentos regulatórios e entrevistas com 34 informantes-chave, este estudo de casos múltiplos examinou a prevalência e as fontes de pesquisa utilizadas na elaboração de regras do ensino superior, que é o processo para o desenvolvimento de regulamentos federais que governam o ensino superior. Este estudo também examinou as barreiras ao uso da pesquisa na elaboração de regras do ensino superior. Os resultados indicam que, embora a pesquisa tenha sido usada nesse processo, outros fatores que não a pesquisa foram discutidos com mais frequência nos regulamentos finais. As barreiras ao uso da pesquisa na elaboração de regras do ensino superior incluíam restrições de tempo, indisponibilidade de dados, política, falta de capacidade de pesquisa do governo e outras disjunções entre as comunidades de pesquisa e política. Além disso, os contextos em que regras específicas foram criadas moldaram a prevalência e as fontes de pesquisa usadas no desenvolvimento dos regulamentos. O artigo conclui com implicações para política e teoria. Palavras-chave: uso em pesquisa; ensino superior; política educacional; elaboração de regras; política federal; Estados Unidos; estudo de caso Research Utilization in Higher Education Rulemaking 3 Education Policy Analysis Archives Vol. 28 No. 95 4 Introduction1 In recent decades, educators and policymakers have increasingly called for the use of research to inform educational policy (Cooper, Levin, & Campbell, 2009; Levin, 2013; Lubienski, Scott, & DeBray, 2014; Malin & Lubienski, 2015; Nutley, Walter, & Davies, 2007). These calls reflect a perspective that using findings from research in policy development can “lead to more informed policy, higher-quality decisions, more effective practices, and, in turn, improved outcomes” (Cooper et al., 2009, p. 160). At the same time, observers recognize that research often does not serve as a basis for policy (Asen, Gurke, Solomon, Conners, & Gumm, 2011; Asen, Gurke, Conners, Solomon, & Gumm, 2013; Henig, 2008). Politics, funding resources, and policymakers’ own experiences influence whether and what type of research is conducted or used in policymaking (Alkin & King, 2017; Asen et al., 2013; Levin, 2013; Nutley et al., 2007; Weiss, 1979). For stakeholders who would like to see more research as a basis for educational policy, it is important to understand not only how prevalent the use of research is and the sources of studies that have been used by policymakers, but also the factors that hinder research use in educational policymaking. This study examined the prevalence of and barriers to research utilization in federal higher education rulemaking, which is the process through which the United States Department of Education creates regulatory policy to administer provisions of the Higher Education Act (Lubbers, 2014; Natow, 2017; Pelesh, 1994). Regulations created through the rulemaking process have consequential implications for higher education institutions and students. For example, federal regulatory policy has provided a basis for discharge of student loan debt for borrowers deemed to have been defrauded by their institutions (Kreighbaum, 2019), and noncompliance with federal regulations may result in the loss of eligibility for institutions to participate in federal student financial aid programs (Lederman, 2019). Although previous literature has not examined how research has been used in higher education rulemaking, understanding research utilization in this process is important for several reasons. First, federal law requires government agencies to conduct analyses at different points during the rulemaking process, including regulatory impact assessments and cost-benefit analyses, for many regulations (Belfield, Bowden, & Rodriguez, 2018; Costa, Desmarais, & Hird, 2019; Executive Order 12866, 1993). Rules deemed insufficiently based on evidence are more likely to be struck down by a court (Association of Private Sector Colleges & Universities [APSCU] v. Duncan, 2012; Costa et al., 2019), which can in turn result in confusion about the rule’s status and expenditure of additional government time and resources on redeveloping the regulations (Kelchen, 2019). The study presented here reveals the extent to which research was considered in the creation of several key higher education regulations and the sources of that research. This study also identifies barriers that have obstructed the use of research in the rulemaking process. Understanding factors that hinder research use can help proponents of evidence-based policymaking to develop strategies for overcoming those barriers so that more research may be considered in the creation of these policies. This study also provided an opportunity to examine whether research features prominently in a policymaking process that is intentionally structured to bring research-oriented organizations directly into conversation with government officials for the purpose of drafting policy language. As explained more fully below, the Higher Education Act (2018) requires the Department of Education to meet with stakeholder groups to develop proposed regulations through a process known as negotiated rulemaking when developing regulations that affect federal student financial aid programs (e.g., Natow, 2017; Pelesh, 1994). One common explanation for why research is not used more 1 The research reported here was supported by the Hofstra College of Liberal Arts and Sciences Faculty Research and Development Grant. Research Utilization in Higher Education Rulemaking 5 frequently in policymaking is that researchers and policymakers are part of different “communities” that hold dissimilar values, incentive structures, timelines, and practices regarding research (Caplan, 1979; Henig, 2008; Ness, 2010). The negotiated rulemaking process enables higher education institutions, many of which employ researchers and house research centers, to participate in the policymaking process in a more direct way than other policymaking processes typically allow. Indeed, many recent higher education-negotiated rulemaking sessions have included administrators (and in a few cases faculty) employed by individual colleges and universities as participants (e.g., Program Integrity, 2014b, 2018; Student Assistance General Provisions, 2015b, 2016b; Teacher Preparation Issues, 2014). By involving research-oriented organizations such as universities in a direct and open conversation with other interest groups and federal officials, the higher education rulemaking process provides a unique opportunity to study whether the divide between policymakers and researchers may be bridged by a policymaking process that involves extended public meetings and discussions between these parties. Overview of Federal Rulemaking for Higher Education Federal rulemaking is the process through which a federal government agency creates regulations that implement the provisions of duly enacted authorizing statutes (Administrative Procedure Act, 2018; Kerwin & Furlong, 2011; Office of the Federal Register, 2011). The Higher Education Act (HEA) is an authorizing statute, and federal regulations created under its authority are developed through rulemaking in the U.S. Department of Education (Lubbers, 2014; Natow, 2017; Pelesh, 1994). Under the HEA, rulemaking that affects federal student financial aid programs must include a process known as negotiated rulemaking, during which the agency meets with stakeholders to debate and negotiate the content of a proposed rule. Participants in higher education negotiated rulemaking have included representatives of different types of colleges and universities (including public four-year, public two-year, private nonprofit, and private for-profit institutions), student representatives, consumer advocates, the Department of Education’s negotiator, and depending on the topic, representatives of other stakeholders such as accreditors, the student lending industry, and more. The negotiating team attempts to arrive at a consensus on the language for the proposed regulation. If consensus is attained (that is, if there is unanimous agreement of all negotiators), then the agency uses that content in the Notice of Proposed Rulemaking (NPRM). But if consensus is not reached, the agency alone writes the NPRM. The HEA requires the department to receive feedback from a broad range of stakeholders prior to negotiated rulemaking regarding the matters to be negotiated. These comments may be provided in writing or in the form of statements made at meetings with agency personnel that take place in different regions of the United States (Kerwin & Furlong, 2011; Lubbers, 2014; Natow, 2017; Office of the Federal Register, 2011; Pelesh, 1994). Following negotiated rulemaking, the department publishes an NPRM in the Federal Register. This initiates the comment period, during which stakeholders may provide written comments to the Department of Education regarding the rule. After the closing of the comment period, the department examines the comments, decides whether to make adjustments to the rule’s language based on those comments, and eventually publishes the final rule – including a description of the comments received and the department’s responses to them – in the Federal Register (Kerwin & Furlong, 2011; Lubbers, 2014; Natow, 2017; Office of the Federal Register, 2011; Pelesh, 1994). According to a decades-old executive order, regulations are deemed “significant” when they are projected to affect the economy by an amount of at least $100 million annually, are anticipated to have a negative effect on the economy, will substantially impact government budgets, or will introduce “novel legal or policy issues” (Executive Order 12866, 1993, p. 51738). Rules that fall Education Policy Analysis Archives Vol. 28 No. 95 6 under this definition must undergo regulatory impact assessments and reviews by the Office of Management and Budget (OMB) (Belfield et al., 2018; Executive Order 12866, 1993; Office of the Federal Register, 2011; Sunstein, 2013). Stakeholders may meet with OMB staff to provide their perspectives on rules subject to OMB review (Sunstein, 2013). When conducting regulatory assessments, Executive Order 13563 (2011) requires that agencies “use the best available techniques to quantify anticipated present and future benefits and costs as accurately as possible” (p. 3821). After a regulation is finalized, rulemaking politics may continue. A final rule may be subject to nullification or revision by a legislative action, a future rulemaking, or a court order (Kerwin & Furlong, 2011; Office of the Federal Register, 2011). All of these post-finalization events have occurred with respect to higher education rules in the recent past (e.g., Nolter, 2012; Program Integrity, 2019; Stephan, 2017). The Use of Research and Other Factors in Policymaking Conceptualizing Research Utilization What is research? There are many definitions of research reflected in public policies and the academic literature. Asen et al. (2011) observed that the federal government’s definition of research in the No Child Left Behind (NCLB) law refers to research as “systematic” and “objective,” employing “empirical methods” and “rigorous data analysis” with a preference for randomized controlled trials (No Child Left Behind Act, 2002, as quoted in Asen et al., 2011, pp. 201-202).2 In a separate article, Asen et al. (2013) provided their own definition of research in somewhat broader terms: “empirical findings derived from systematic analysis of information, guided by purposeful research questions and method” (p. 40). Nutley et al. (2007) broadly conceptualized “research as any investigation towards increasing the sum of knowledge based on planned and systematic inquiry,” with no preference for particular types of research methods so long as the methods are appropriate for the what the researcher is studying (pp. 21-22). Penuel and colleagues defined research simply as a “systematic inquiry to answer a specific question” (Penuel et al., 2016, p. 15). Although these definitions differ from each other, they underscore that research involves more than just examining a snapshot of data or mere “number crunching” that is not part of a more systematic analysis; rather, research involves a methodologically justifiable plan and the goal of responding to particular questions or inquiries. How is research used? There are different ways to use research in policymaking. Sometimes, research use is straightforward and easy to identify, as when a policymaker or law cites specific research as a basis for the policy (Alkin & King, 2017; Weiss, 1998). Sometimes the use of research is less observable. For example, learning about the findings of research may help to shift a policymaker’s perspective about an issue, but the policymaker may not mention the research when discussing the issue and may not even be aware that this perspective shift is related to having learned about research findings (Weiss, 1979). Carol Weiss and others have described this as a “conceptual” use of research (Nutley et al., 2007; Tseng, 2012; Weiss, 1979). Political research use – in which research is utilized to support the user’s argument or to promote the user’s ideological or policy position – is a form of research utilization that is particularly notable in policymaking spaces (Ness, 2010; Nutley et al., 2007; Tseng, 2012; Weiss, 1979). Sometimes participants in a policymaking 2 Farley-Ripple, May, Karpyn, Tilley, and McDonough (2018) have noted that NCLB’s successor, the Every Student Succeeds Act (ESSA), also instructs school districts to use evidence-driven practices and offers “a bit more guidance” than NCLB had offered regarding what that means; however, ESSA “is actually less prescriptive than … NCLB” (p. 236). Research Utilization in Higher Education Rulemaking 7 process use research to substantiate their points by indicating that research supports them but do not provide specific citations or other details about the research (Asen et al., 2011). Research might also “inform and enrich” policymaking discussions but ultimately “not drive the outcome” (Head, 2015, p. 474). In their examination of how research use has been defined in the literature, Alkin and King (2017) noted that research need not be identified as the basis for a policy in order for research to have been used, provided that the research was discussed or given consideration in the decision- making process, even if only as “one of multiple influences” (p. 438). Barriers to Research Utilization in Policymaking Despite calls to increase the use of research in policymaking, scholars have documented the infrequency with which research has been used relative to other factors (Asen et al., 2011, 2013; Gollust et al., 2014). For example, in their study of research use by school boards, Asen et al. (2013) found that research was used comparatively less than other factors, such as examples and law. Also, Nutley et al. (2007) wrote that it has been “rare” for policymakers to use “traditional, academic, peer-reviewed literature” in policymaking (p. 62). Although an examination of barriers to research use specifically in federal higher education rulemaking does not appear in prior literature, several studies have documented barriers to research utilization in other contexts. One barrier is when decision makers do not understand research or perceive it to be valuable (Nutley et al., 2007; Penuel et al., 2016). Another is when an organization or agency lacks sufficient resources to consider research in decision making (Farley-Ripple, 2012; Santesso & Tugwell, 2006; Tseng & Nutley, 2014). Time limitations have also been found to hinder research use in decision making. For example, Farley-Ripple (2012) found that school leaders’ myriad responsibilities left them little time “to search, read, and employ education research in their day-to-day work” (p. 799). Additionally, a mismatch often exists between policy timeframes and research timeframes, with the policy world moving at a considerably faster pace than it takes researchers to conduct, write about, and publicize their studies (Henig, 2008; Nutley et al., 2007). The Influence of Context on Research Utilization Surrounding contexts can influence decisions about whether and how to use research in decision making. That is, factors such as the history or culture of a potential research user’s organization, the resources available to a potential research user, and external political pressure can influence whether, how much, and what kinds of research utilization occur (Alkin & King, 2017; Farley-Ripple et al., 2018; Head, 2015; Henig, 2008; Holzer, Lewig, Bromfield, & Arney, 2007; Levin, 2013; Nutley et al., 2007; Santesso & Tugwell, 2006). Organizational structures, such as whether a division specifically dedicated to research exists within an organization, is another context that can influence research utilization (Farley-Ripple et al., 2018). The subject matter of a proposed policy is another example of a context that may influence the use of research in a policy’s creation. As Nutley et al. (2007) wrote, “more radical change requires more extensive debate” (p. 110). This may lead to greater use or scrutiny of research in developing controversial or far-reaching policies. Another important context involves the networks of research users and policymaking participants (Head, 2015). The extent and kinds of evidence used in particular cases are often influenced by the networks and relationships of those involved (Cooper & Levin, 2010; Farley- Ripple & Grajeda, 2020; Head, 2015; Ng-A-Fook, Kane, Butler, Glithero, & Forte, 2015). These relationships may be between researchers and policymakers directly, but often other entities will serve as intermediaries who are neither policymakers nor researchers but work to connect policymakers with useable research findings (Cooper, 2014; Cooper et al., 2009; Cooper & Shewchuk, 2015; Farley-Ripple et al., 2018; Lubienski et al., 2014; Malin & Lubienski, 2015; Naidorf, 2014; Nutley et al., 2007; Rodway, 2015; Scott, Lubienski, DeBray, & Jabbar, 2014; Tseng & Nutley, Education Policy Analysis Archives Vol. 28 No. 95 8 2014). Examples of intermediaries include foundations, policy-oriented think tanks, advocacy organizations, the media, labor unions, and others (Malin & Lubienski, 2015; Ness & Gándara, 2014; Scott et al., 2014). These relationships and networks contribute to “knowledge mobilization,” which is the “dissemination of … research through various channels to its ultimate use (or lack thereof) in policy and practice settings” (Cooper, 2014, p. 4). In contexts where policy actors and researchers network with each other or with intermediaries, research is more likely to be considered (Nutley et al., 2007; Rodway, 2015). Use of Factors Other Than Research in Policymaking Factors other than research have also been used in policymaking, including data and statistics. When these are presented as snapshots of information and not part of a larger inquiry, they are not considered research. Asen et al. (2013) defined “research” and “data” differently in their analysis of school board proceedings, classifying findings from a study that involved methodical data analysis and purposefully designed procedures as research, whereas “measurable quantitative or qualitative information systematically collected to describe a set of conditions or trends” was considered data (p. 40). Similarly, in their study of educational leaders, Penuel et al. (2016) differentiated data from research, noting that the examination of data was “more open-ended” than conducting research, and that the use of data “seldom addresses specific research questions” (p. 15). Examples have also been cited in attempts to influence policymaking (Asen et al., 2013; Simons, 2015). An example is “a specific case or incident” that is “typical or exceptional” of the situation at issue in policymaking (Asen et al., 2013, p. 40). Actual examples are “concrete initiatives and projects” that have actually existed (Simons, 2015, p. 715), and hypothetical examples are situations that are reasonably likely to occur but have not actually occurred. A form of hypothetical example described in OMB guidance permits agencies to consider “plausible scenarios” when conducting cost-benefit analyses if there is “scientific uncertainty” with regard to calculating costs and benefits in regulatory policymaking (U.S. Office of Management & Budget, 2003, p. 39). Other non-research factors used in policymaking include values and beliefs, previous experience or traditions, and laws or public policies (Alkin & King, 2017; Asen et al., 2013; Davies, 2004; Head, 2015; Nutley et al., 2007; Santesso & Tugwell, 2006; Weiss, 1979). Experts’ perspectives have also been considered in policymaking (Davies, 2004). Policymakers have sought expert testimony in legislative hearings (Perna, Orosz, & Kent, 2019), via consulting relationships (Morgan et al., 2006), or in written statements, such as public comments during the rulemaking process (Costa et al., 2019). Expert perspectives are not in themselves a separate category of influence in policymaking but typically fit into one of the other categories: experts may present the results of research or other data, discuss something they learned from previous experience, or speak about their own beliefs or values. The “Two Communities” and Potential for a Bridge Barriers to research use in policymaking often reflect a gulf between researchers and policymakers characterized by a lack of effective communication, divergent cultures, and inadequate understanding of each other’s roles – a phenomenon known as the “two communities” (Caplan, 1979; Farley-Ripple et al., 2018; Ness, 2010; Ness & Gándara, 2014; Nutley et al., 2007). According to this perspective, policymakers and researchers “live in separate worlds with different and often conflicting values, different reward systems, and different languages” (Caplan, 1979, p. 459). Some scholars have critiqued the two-communities perspective as too reductive (Newman, Cherney, & Head, 2016; Nutley et al., 2007; Rich, 1991). However, divides between the research and policy communities have been recognized in the literature (Court & Maxwell, 2005; Ness, 2010), with some Research Utilization in Higher Education Rulemaking 9 observers arguing that differences between the policy and research communities are inevitable and even beneficial (Birnbaum, 2000; Locock & Boaz, 2004). While it may be overly simplistic to state that policymakers and researchers are members of two separate communities, the limited use of research in policymaking and the fact that research utilization has been hindered by fundamental disjunctions between the research and policy worlds indicate that the two-communities perspective should not be too hastily dismissed. Rather than disregard the theory, some scholars have built on it, acknowledging that the framework is not complex enough to capture the entire picture, but also recognizing the existence of a gap between these communities (Farley-Ripple et al., 2018; Holzer et al., 2007). For example, Holzer et al. (2007) advanced a conceptual model that recognized the different “cultures” of researchers, policymakers, and practitioners, but also took account of a variety of different contextual factors, including political, social, organizational, and individual characteristics. There is reason to believe that the federal higher education rulemaking process could serve as a bridge between the research and policy communities. This is because a unique aspect of this process is the frequent use of negotiated rulemaking, which is required in the creation of many federal student financial aid regulations (Higher Education Act, 2018; Pelesh, 1994). As explained above, negotiated rulemaking is a process in which stakeholders and a regulatory agency work together to develop the substance of proposed regulations (Kerwin & Furlong, 2011; Pelesh, 1994). Because some stakeholders of higher education rulemaking are colleges and universities, employees of individual higher education institutions have commonly participated in negotiated rulemaking in the U.S. Department of Education. These institutions employ faculty and other researchers, and they often house research centers and receive research funding from government and other sources. Additionally, Washington, DC-based associations have participated in negotiated rulemaking by nominating individuals for negotiating teams or by having their employees serve as negotiators (Natow, 2017). To the extent that these organizations act as “knowledge brokers” for the higher education policy community, their participation may serve as another link between research and policymaking in this process, as “knowledge brokers” have been “widely recognized as potential levers for bridging the two communities” (Farley-Ripple & Grajeda, 2019, p. 67). For these reasons, higher education rulemaking presents a unique opportunity to determine whether aspects of the “two communities” phenomenon persist in a policymaking process that involves direct participation by research-oriented stakeholders and intermediary organizations. Methods This case study (Creswell & Poth, 2018; Merriam, 1998; Yin, 2014) investigated how research has been used in the development of the following five federal higher education rules: Gainful Employment, Revised Pay as You Earn (REPAYE), Borrower Defense, Teacher Preparation, and Gainful Employment Rescission. This study received Institutional Review Board approval prior to data collection; data sources included documents and interviews. The Case-Study Rules These case studies of higher education rulemaking were purposefully selected via a sampling strategy that combined maximum variation and typical case sampling (Creswell & Poth, 2018; Suri, 2011). This strategy identified regulations developed through the usual process that occurs for higher education rulemaking while leaving room to examine similarities and differences across the diverse subject matters and contexts of these particular rules. Specifically, these five case-study rules were selected because all of them met the following typical case sampling criteria: (1) they were subject to negotiated rulemaking to develop proposed regulatory language; (2) their post-NPRM phase Education Policy Analysis Archives Vol. 28 No. 95 10 involved receiving comments from interested parties; and (3) all of the case-study rules were reviewed by the OMB before they were finalized. These are standard practices for most regulations developed under Title IV of the HEA. However, these rules also differed in key ways, to provide maximum variation among the sample of rules. The differences include subject matter and regulatory history, as explained in the descriptions of the case-study rules immediately below. Gainful Employment Rule. Issued in 2014, this rule was the Obama administration’s second attempt to define the term “gainful employment” that appears in the HEA, which provides that career-focused programs may be eligible to receive federal student financial aid if the programs “prepare students for gainful employment in a recognized occupation” (Higher Education Act, 2018, § 481[b][A][i]). Among other things, this rule required data disclosures and set an acceptable debt-income rate for an institution’s “typical graduate” that “does not exceed 20 percent of his or her discretionary income or 8 percent of his or her total earnings” (U.S. Department of Education, 2014, para. 3). This regulation followed the nullification of the repayment rate provision of the first Gainful Employment rule (issued in 2011) when a court found the rate to be “arbitrary and capricious” because it did not possess a “reasoned basis,” such as expert opinion or studies indicating the rate was appropriate (APSCU v. Duncan, 2012, pp. 30-31; see also Nolter, 2012). REPAYE Rule. This 2015 rule revised a federal income-based student loan repayment program to expand eligibility to a broader set of borrowers and to provide different debt forgiveness qualification criteria for borrowers depending on whether they had only undergraduate debt or debt from graduate school as well. The rule also limited the amount of interest charged on federal loans depending on the borrower’s income (Stratford, 2015; Student Assistance General Provisions, 2015a). The REPAYE negotiated rulemaking committee reached consensus when negotiating the proposed rule (Stratford, 2015); it is the only case-study rule on which consensus was reached. Borrower Defense Rule. This rule was issued in 2016, toward the end of the Obama administration. The rule governed the circumstances under which student loan borrowers could assert, as a defense to debt repayment, fraud on the part of their higher education institutions. The regulation restricted institutions’ ability to require students to agree to arbitration rather than lawsuits for resolving disputes (Kreighbaum, 2016; Student Assistance General Provisions, 2016a). The Trump administration took office not long after this rule was issued. At first, Trump’s Department of Education delayed implementing the rule before a court ruled that the delay was not permissible (Kreighbaum, 2018b). Then in 2019, the department substantially modified the rule and repealed some key aspects of it, including the ban on required arbitration agreements (U.S. Department of Education, 2019). Teacher Preparation Rule. The Teacher Preparation rule was also issued in 2016 toward the end of the Obama administration. This rule set new accountability requirements for teacher preparation programs under Title II of the HEA. For example, the rule would have required teacher education programs to provide the federal government with data on their graduates’ job placement (including the number of alumni working in schools deemed “high-need”) and indicators of academic performance of the students taught by program alumni (Stephan, 2017; Teacher Preparation Issues, 2016). Early in the Trump administration, the Teacher Preparation rule was repealed via a Congressional Review Act resolution before it took effect (Stephan, 2017). Gainful Employment Rescission. In 2019, the Department of Education released a final regulation that rescinded the 2014 Gainful Employment rule (Program Integrity, 2019). This is the only case-study rule that was created by a Republican presidential administration. Research Utilization in Higher Education Rulemaking 11 Documentary Data I obtained copies of each case-study final rule (and any applicable corrections) from the online version of the Federal Register. For each rule, I conducted a line-by-line analysis of the sections that discussed the reasoning behind the regulations. These sections, which I collectively refer to as the reasoning portions of the rules, included the Executive Summary, the Analysis of Comments and Changes, the Regulatory Impact Assessment, and other sections of the final rules that discussed the Department of Education’s rationale for why the regulations contained the provisions that they did (see also Office of the Federal Register, 2011). Because the regulatory text to be codified in the Code of Federal Regulations did not contain background or explanatory information, those sections of the final case-study rules were not included in this line-by-line analysis. I coded each page for the discussion of research and other factors that appeared on the page. These factors fell into six categories: research, data/math, experience, examples, beliefs/opinions, and law. The number of pages in the Federal Register on which each factor appeared was chosen as the unit of measure because page counts in the Federal Register have been used in other studies of regulation to measure such attributes as regulatory complexity and the amount of rulemaking activity (e.g., Acs & Cameron, 2013; Brito & de Rugy, 2008; Cochran, 2001; Kerwin & Furlong, 2011). Pages are also an appropriate measure because the reasoning sections of final rules often contain tables and numbers that would not be conducive to measuring by paragraph or word counts. The definition of research used in this study was broadly conceptualized to include any systematic analysis conducted according to a methodological plan to address a particular area of inquiry (derived from definitions of research found in Asen et al., 2013; Holzer et al., 2007; Nutley et al., 2007; and Penuel et al., 2016). Although this analysis employed a broad definition of research that included diverse methodologies and sources, to be considered research (as opposed to data, examples, or other factors described below), an item must have contained some kind of systematic inquiry and involved analysis that went beyond mere anecdotes or “number crunching.” In keeping with Penuel et al.’s (2016) definition, this study conceived of research as “an activity in which people employ systematic, empirical methods to answer a specific question” (p. 16). Similar to Asen et al. (2013), I applied the data/math code to the discussion of statistics or general trends apart from a broader, systemic analysis. For example, the following statement from the Gainful Employment rule was coded as data/math: “One commenter asserted that the 150 percent normal time graduation rate for public and private non-profit open-enrollment colleges is 28.3 percent and 39.7 percent respectively …” (Program Integrity, 2014a, p. 64903). Also included in the data/math code were discussions of mathematical formulas or principles. Experience was defined as testimonials or first-hand accounts based on “direct observation of or participation in events or activities” (Asen et al., 2013, p. 40), as well as descriptions of traditions, past practice, and other previous experiences (Asen et al., 2013; Davies, 2004). The experience code also included practices that are standard or considered “best practices” in a particular field because such standards and practices are generally developed through previous experience in the field (e.g., Kocbek & Juric, 2010; Rosenthal, 2012). The definition of example was based largely on Asen et al.’s (2013) characterization of the term, which involved “a specific case or incident used to illustrate typical or exceptional characteristics of a topic or issue” (p. 40). Such instances were coded as examples regardless of whether they were real or hypothetical and included discussions of “plausible scenarios” permitted by OMB guidance for regulatory impact assessments (U.S. Office of Management & Budget, 2003, pp. 18, 39; 2011, pp. 14-15). Beliefs/opinions referred to beliefs, values, viewpoints, ideologies, and other subjective perceptions (Davies, 2004; McCright & Dunlap, 2008). Finally, law was defined as rules, legal mandates, public policies, and governmental guidelines for behavior and activities, including statutes, court decisions, regulations, executive orders, agency Education Policy Analysis Archives Vol. 28 No. 95 12 guidance, and other legal instruments issued at any level of government (Asen et al., 2013; Clarke, 2016). I also reviewed negotiated rulemaking records and obtained available reports of research cited in each final rule to identify the types and sources of research discussed in the rules. Interview Data I conducted in-depth, semi-structured interviews with 34 individuals who had some involvement in the Department of Education’s higher education rulemaking process in recent years. Interviewees were purposefully sampled key informants (Marshall, 1996), who had knowledge about one or more of the case-study regulations as well as other rules developed through higher education rulemaking. Interviewees represented a broad range of policy actors, including people with experience working as Department of Education or other federal officials, employees of non-profit organizations, and representatives of state government offices, students, consumers, the lending industry, accreditors, and higher education institutions. Table 1 depicts the categories of professional positions held by this study’s interviewees. Because some interviewees held more than one position through which they worked on rulemaking issues (e.g., a former federal official who later worked at a non-profit organization doing higher education policy work), the total number of positions reflected in Table 1 is greater than the total number of respondents who participated in this study. Table 1 Professional Positions of Interviewees Professional Position (current or former) Number of Interviewees Department of Education staff 7 Other federal staff 5 Higher education institution or association representing same Public four-year college/university 3 Public two-year/community college 3 Private non-profit college/university 6 Private for-profit college/university 2 Financial aid administrator or association representing same 5 Student loan industry/guarantor or association representing same 3 Accreditor or association representing same 2 Student, consumer, student-veteran, or legal aid advocate 6 Think tank staff 4 State-level government staff or association representing same 2 The interview protocol included broad, open-ended questions about respondents’ experiences and observations regarding research use in higher education rulemaking, including how the department and other participants utilized research during rulemaking and whether particular circumstances or other factors made it more or less likely for research to be used. Other interview questions asked about research use in different stages of the rulemaking process, the methodologies of research used, and factors other than research that were used. Because previous studies have not examined research utilization specifically in the higher education rulemaking process, interview questions were informed by the literature on research use prevalence, hindrances, and facilitators in Research Utilization in Higher Education Rulemaking 13 other policymaking contexts (e.g., Asen et al., 2011, 2013; Gollust, 2014; Nutley et al., 2007).3 Data analysis of interviews followed practices described by Miles, Huberman, and Saldaña (2019) and included multiple coding stages, analytic memos, and matrices to categorize, reduce, and visualize findings from the dataset. I used Dedoose qualitative analysis software to code transcripts with a standard coding scheme created with concepts from the literature as well as categories that emerged during interviews. I then exported coded data relevant to research use prevalence and barriers into a spreadsheet and reviewed those excerpts again to identify additional emergent themes and patterns. Results Prevalence, Contexts, and Sources of Research Utilization Prevalence of research utilization. Table 2 indicates the number and percentage of pages in the reasoning portions of the final case-study rules that discussed research and other factors. The Analysis of Comments and Changes section of final rules is unique in that it singularly focuses on responding to comments received during the Notice and Comment period (Natow, 2017; Office of the Federal Register, 2011). For this reason, the number and percentage of pages on which research and other factors were discussed in the case-study rules appear separately from the rest of the reasoning sections in the table. Table 2 is presented with a caveat: There is no requirement that research must be discussed in any particular proportion of pages in final regulations or that research should necessarily be discussed more frequently than other factors in final rules. Moreover, because much of the text of final rules involves summarizing and responding to comments received from the public, it would be expected that those subsections would discuss matters raised in the comments, which may not include research. Thus, the fact that research was discussed in final rules less frequently than other factors does not in itself indicate that research was not considered sufficiently in the development of the rule. However, Table 2 provides important and relevant information for a study of research use in higher education rulemaking. First, information about the frequency with which research and other factors were discussed in final rules allows for a comparison of how often research was featured vis-à-vis other factors. The information in the table also allows for an analysis of how frequently research was discussed in some rules versus others, which is important for understanding how a given rule’s context may influence research use in the rulemaking process. 3 For more information about the interview protocol, see Natow (2020). Table 2 Numbers and Percentages of Pages* in “Reasoning” Portions of the Final Case-Study Rules on Which Research and Other Factors Are Discussed Factor Gainful Employment (2014) REPAYE (2015) Teacher Prep (2016) Borrower Defense (2016) Gainful Employment Rescission (2019) Analysis of Comments & Changes Remainder of the Reasoning Sections Analysis of Comments & Changes Remainder of the Reasoning Sections Analysis of Comments & Changes Remainder of the Reasoning Sections Analysis of Comments & Changes Remainder of the Reasoning Sections Analysis of Comments & Changes Remainder of the Reasoning Sections Research 34/102 (33%) 12/96 (12.5%) 0/19 (0%) 4/13 (31%) 20/90 (22%) 10/33 (30%) 14/119 (12%) 4/27 (15%) 32/40 (80%) 9/23 (39%) Data/Math 85/102 (83%) 91/96 (95%) 12/19 (63%) 13/13 (100%) 53/90 (59%) 33/33 (100%) 31/119 (26%) 25/27 (93%) 38/40 (95%) 23/23 (100%) Examples 100/102 (98%) 32/96 (33%) 19/19 (100%) 9/13 (69%) 88/90 (98%) 29/33 (88%) 107/119 (90%) 21/27 (78%) 39/40 (97.5%) 18/23 (78%) Beliefs/Opinions 99/102 (97%) 24/96 (25%) 18/19 (95%) 7/13 (54%) 84/90 (93%) 31/33 (94%) 110/119 (92%) 14/27 (52%) 40/40 (100%) 20/23 (87%) Experience 54/102 (53%) 11/96 (11.5%) 13/19 (68%) 5/13 (38.5%) 29/90 (32%) 24/33 (73%) 66/119 (55.5%) 13/27 (48%) 34/40 (85%) 14/23 (61%) Law 75/102 (73.5%) 13/96 (13.5%) 14/19 (74%) 7/13 (54%) 66/90 (73%) 23/33 (70%) 107/119 (90%) 17/27 (63%) 29/40 (72.5%) 12/23 (52%) * Page counts include the first and last pages on which the Analysis of Comments & Changes and Remainder of Reasoning Sections appear in each of the final case- study rules. Education Policy Analysis Archives Vol. 28 No. 95 14 http://epaa.asu.edu/ojs/ As Table 2 demonstrates, there was some variation among the case-study rules regarding the proportion of pages and sections in the final regulations that discussed research, data, and other factors. One consistency across all five case-study rules is that research was discussed on fewer pages of the final regulations than all of the other factors reflected in the table. The only final rule in which research was discussed on more than half the pages was the 2019 Gainful Employment rescission, and the majority of research discussed in that rule occurred in the Analysis of Comments and Changes. The 2014 Gainful Employment rule and the Teacher Preparation rule, which both discussed research on just under one-quarter of the pages, cited research more often than REPAYE or Borrower Defense. In three of the five case-study rules, examples were discussed on a larger proportion of pages than any other factor, and examples were mentioned on at least two thirds of pages in all of the final regulations. This is consistent with Asen et al.’s (2013) finding that examples were used more frequently than other types of evidence in their study of local school boards. Also notable is the fact that, in every case-study rule, data or mathematical analyses were discussed on upwards of 90% of the pages of reasoning portions that were not the Analysis of Comments and Changes, and were discussed less frequently in the Analysis of Comments and Changes than in the combined other reasoning sections. This suggests that regulation drafters were eager to cite data or mathematical calculations to justify the rules even when not doing so to respond to public comments. A number of this study’s interviewees made statements indicating they had not observed research playing a large role in higher education rulemaking. One negotiated rulemaking participant stated that “honest to goodness research” has been used in a “precious little” amount during higher education rulemaking. Another negotiator conceded that “you don’t see a lot of research being introduced” into the process, and a representative of community colleges noted that “not that much” research has generally been discussed in the development of the rules. Many other respondents made similar comments. Although research did not appear as frequently as other factors in the final case-study regulations, interviewees and rulemaking documents indicated that research was, in fact, used in the development of these regulations, even if infrequently. Moreover, when research was utilized in higher education rulemaking, it was often used for political purposes, such as to substantiate the arguments or promote the policy preferences of the research users. This “political” use of research is to be expected in policymaking environments such as the higher education rulemaking process (Ness, 2010; Weiss, 1979).4 Contexts of research utilization. Interviewees indicated that a reason why some rules, such as the two Gainful Employment regulations and the Teacher Preparation rule, discussed research on more pages than other rules was related to contextual circumstances specific to those rules. The 2014 Gainful Employment rule was created following a prior version of the rule being largely invalidated by a federal court because an important provision of the rule lacked a “reasoned basis” (APSCU v. Duncan, 2012, pp. 1, 30). Respondents said that this history weighed on federal officials who were involved in creating the later version of the rule. One federal official explained that in that rulemaking, the government was interested in: find[ing] whatever way has the best chance of holding up in court … And so that was like the thrust of our efforts, looking at research and evidence. It wasn’t like, okay, what’s the problem here, or what are we trying to do? It was like, what has the best chance of holding up in court, to meet this ultimate objective? 4 For more about how Weiss’s (1979) typology applies to higher education rulemaking, see Natow (2020). Research Utilization in Higher Education Rulemaking 15 Education Policy Analysis Archives Vol. 28 No. 95 16 Another respondent who had worked for the federal government confirmed: The original rule had been thrown out by the judge because … [the department] had kind of picked the percentage repayment rate below which programs wouldn’t be eligible, and … the judge kind of argued that the standard that they had picked wasn’t grounded in research; it was arbitrary. And so the rule had been kind of thrown out largely on that basis… So [during the 2014 rulemaking] there was … internally a pretty exhaustive review of the research literature on for-profits. Given this history, it is unsurprising that the subsequent Gainful Employment rule and its rescission would include relatively more discussions of research than other rules in the dataset. The context surrounding the Teacher Preparation rule, which did not have an identical history to the Gainful Employment rules but had other characteristics in common with them, may also explain why research was cited on more pages of that regulation than was the case in Borrower Defense or REPAYE. Like Gainful Employment, the Teacher Preparation rule was controversial (Stratford, 2014). Also like Gainful Employment, Teacher Preparation included the possible loss of eligibility to participate in certain federal financial aid programs as a penalty for institutions that did not meet the rule’s requirements (Program Integrity, 2014a; Teacher Preparation Issues, 2016). These contexts influenced policymakers as they developed the rules and used research in the rulemaking process. A federal official interviewed for this study explained: Particularly if you’re going to remove eligibility from an entity or kind of punish a particular entity – I guess that was also kind of a feature of Teacher Prep, but it was really salient in the Gainful Employment rule – then you had to have a kind of an ironclad justification for your metric. These contexts of the Gainful Employment and Teacher Preparation rules help to explain why research featured more prominently in those rulemakings than with the other case-study regulations. Another important context for research use in rulemaking involves the professional positions of the regulatory policymaking participants, as the networks and relationships of researchers, policymakers, and intermediaries can influence the extent and nature of research use in decision making (Cooper & Levin, 2010; Farley-Ripple & Grajeda, 2019; Ng-A-Fook et al., 2015). Table 3 indicates the professional positions of non-federal negotiators – that is, negotiated rulemaking participants who were not representing the Department of Education – in the five case- study rules. The table distinguishes between primary and alternate negotiators, as these different types of participants play different roles in negotiated rulemaking. Generally speaking, alternates participate in consensus voting and debate only when the primary negotiator is absent or otherwise unavailable (Negotiated Rulemaking Committee, 2018). However, alternate negotiators have sometimes served as active participants in higher education rulemaking,5 and as such, their professional backgrounds are important to include in this analysis. Table 3 Professional Positions of Primary and Alternate Non-Federal Negotiators in the Case-Study Rules 5 As one of this study’s interviewees explained, “even when you’re an alternate, you get to sit at the big table because you sort of go in and out, with the primary person and the alternate.” Research Utilization in Higher Education Rulemaking 17 Professional Position Primary Negotiators Alternate Negotiators Total Negotiators Individual college administrator 27 29 56 Student/youth advocacy group representative 6 3 9 State higher education system administrator 5 4 9 Legal assistance attorney 4 4 8 State attorney general’s office 4 4 8 Accrediting agency representative 4 4 8 Lending industry representative 4 4 8 Veterans advocate 3 5 8 Consumer advocate 2 3 5 For-profit higher education group/network administrator 3 2 5 Higher education association employee 3 2 5 Individual higher education student 2 2 4 Private company 3 0 3 Lawyer for for-profit higher education 1 2 3 K-12 teacher 1 1 2 Alternative teacher certification program representative 1 1 2 College or university faculty member 1 1 2 Public K-12 school district representative 1 0 1 Other non-profit representative 0 1 1 Other state-level official 0 1 1 Sources: Program Integrity, 2014b, 2018; Student Assistance General Provisions, 2015b, 2016b; Teacher Preparation Issues, 2014. As indicated in the table, by far the most common professional positions of non-federal negotiators were administrators working in individual higher education institutions. Representatives of advocacy groups were also well represented, as were attorneys of various types and administrators working in higher education systems or networks. The fact that so many negotiated rulemaking participants were not employed by associations or formal networks is noteworthy, because such Education Policy Analysis Archives Vol. 28 No. 95 18 organizations may conduct research or serve as intermediaries between research and policymaking. But even when not serving as negotiators, associations provided support to individual administrators serving on negotiated rulemaking teams. For example, a respondent who was employed by an individual institution described receiving “information” and “research” from a Washington, DC- based association while participating in one of the case-study negotiating committees. Another respondent observed that institutional representatives tend “to stick with the talking points their associations send” during negotiated rulemaking, and that if research findings are not included in those talking points, research is less likely to be introduced by those negotiators. Also noteworthy is the fact that very few college faculty participated in the case-study negotiated rulemakings. Higher education faculty often engage in research as part of their job responsibilities, yet as Table 3 indicates, college administrators have participated in negotiated rulemaking at a much higher rate than faculty. This observation was corroborated by a number of interviewees, who indicated that institutional personnel who participate in rulemaking are often not those who conduct research and are not likely to confer with their colleagues who do conduct research about their participation in the rulemaking process. As one interviewee said, “I think educational institutions are participating in rulemaking as like an industry that’s being regulated. It’s really the business side of educational institutions that are showing up to the table to do that.” Sources of research used. Table 4 displays the sources of research discussed in each final rule.6 The 2014 Gainful Employment rule, that rule’s 2019 rescission, and the Teacher Preparation rule all cited at least 10 academic journal articles in the final regulations – 10 for the 2019 Gainful Employment rescission, 12 for the 2014 Gainful Employment rule, and 20 for the Teacher Preparation rule. This is in contrast to REPAYE, which cited no academic journal articles in its final regulation, and the Borrower Defense rule, which cited one. Although the 2019 rescission of the Gainful Employment rule discussed research on a larger percentage of pages than any other case- study rule (as shown in Table 2), that regulation cited fewer academic journal articles than either the 2014 Gainful Employment rule or the Teacher Preparation rule. The only source of research that was reflected in every case-study rule was federal government research, either from the Department of Education itself or another federal division. Reports from non-profit organizations (including think tanks and associations) were cited in all of the case-study final rules except REPAYE. 6 In addition to these sources, unspecified “research,” “studies,” “analysis,” and the like were discussed in each of the case-study final rules. Because their sources were not identifiable, those discussions of research are not reflected in Table 4. Research Utilization in Higher Education Rulemaking 19 Table 4 Sources of Research Used in Final Rules Source Gainful Employment (2014) REPAYE (2015) Teacher Prep (2016) Borrower Defense (2016) Gainful Employment Rescission (2019) Academic Journal 12 0 20 1 10 Law Review 0 0 1 1 2 ED’s Own Research* 6 1 6 1 5 Non-ED Federal Government 13 1 1 4 8 University-Based Research Center/Working Paper 4 0 6 2 2 Non-Profit Organization 17 0 15 6 29 Collaboration Between Non- Profit Organization and University 0 0 2 0 0 Published Book 2 0 2 0 2 Private Industry 7 0 0 0 1 State-Level Agency 1 0 1 0 0 TOTAL 62 2 54 15 59 * ED is shorthand for the U.S. Department of Education. The figures reflected in this row include research conducted for ED by contractors when published by ED or posted on ED’s website. Respondents offered some insights as to why government and non-profit organization reports were discussed prominently in the case-study regulations while research published in academic journals was cited less frequently. One reason was politics. A long-time observer of higher education rulemaking remarked that public officials tend to seek out research that will substantiate their policy positions,7 and often ideological think tanks8 produce reports on current policy issues 7 This is an example of the political use of research (e.g., Ness, 2010; Weiss, 1979). For more about this and other forms of research use in higher education rulemaking, see Natow (2020). 8 Ness and Gándara (2014) identify “ideological think tanks” as think tanks that are “ideologically driven,” and they provide examples such as the Cato Institute and the Center for American Progress (p. 259). Malin Education Policy Analysis Archives Vol. 28 No. 95 20 that may be useful in that regard. Moreover, a federal official said that government actors are less likely to care about the source of the research as they are about the findings. This respondent said: In government you don’t care who found the finding. You just care that it’s true. And so if we’re showing that the majority of people going to a … for-profit cosmetology program are defaulting on their loans, I don’t really care that [a particular researcher] found that out to be true. What I care is that it’s true. Although the federal official quoted above appeared to be conflating research findings with truth, a crucial point of this respondent’s statement is that sources and authors of research tend to be less important to policymakers than a study’s findings and what the research claims to be “true.” Another reason for the less frequent citation of academic articles in higher education rules relates to how academic reports are written. That is, the language often used to describe research methods and findings in academic articles can make them less understandable to an audience that is not well-versed in research methods or academic jargon. Also, academic articles do not always provide specific policy recommendations or examples of how research findings might be used in practice. As a respondent who worked for the Department of Education explained: I think a lot of studies in academic journals are highly specific to whatever they are studying, and if the researchers themselves have not kind of said something about how they might apply to a federal regulation situation, it’s quite possible they’d never be noticed as possibly relevant. By contrast, think tank reports are more geared toward a policy audience. One negotiated rulemaking participant said that think tanks “drill [research findings] down into these bullets. Boom, boom, boom.” Another negotiated rulemaking participant indicated that think tanks sometimes send their reports directly to their contacts in the policy community, whereas such direct contact with policy actors is less likely in cases of “more scholarly, academic reports where the folks aren’t as connected to the policymakers and advocates operating on the ground.” As Table 4 also indicates, more research published in academic journals appeared in the Teacher Preparation rule than in any other case-study rule; indeed, the Teacher Preparation rule was the only case-study rule for which academic journals were cited more than federal government reports. One federal official offered an explanation for this finding: In Teacher Prep, we had looked at the published research. There was kind of a flurry of publications around 2011 or 2012… looking at trying to develop value-added measures for teacher prep programs and kind of the validity of those measures… There’s a ban on the department collecting any student unit record data9 … So we couldn’t do anything like that ourselves. We couldn’t run any models. We couldn’t somehow get data from teacher prep programs or from states or anything like that because we’re not allowed to have that data. The same official said that the federal government had access to data that government researchers could use for issues related to the Gainful Employment and Borrower Defense rules – such as data and Lubienski (2015) recognize that some of these organizations may serve as “intermediaries that seek to facilitate the use of research by policymakers – albeit often only selected research that supports a particular agenda” (pp. 3-4). 9 This ban was included in the HEA following its reauthorization in 2008 (Miller, 2016). This issue is discussed in more detail in the section of this article below on barriers to research use in higher education rulemaking. Research Utilization in Higher Education Rulemaking 21 from the National Student Loan Data System – so federal officials did more of their own analyses in those rulemakings. These cases illustrate how the sources of research used in policymaking can have more to do with timing and available resources than a desire to use research from particular sources. Other factors. Respondents also discussed factors other than research that they have observed as influential in higher education rulemaking. More frequently than any other factor, interviewees identified the use of data, statistics, and mathematical calculations apart from a systematic study as playing a role in the development of higher education regulations. This included data from the Bureau of Labor Statistics and the Integrated Postsecondary Education Data System as well as gainful employment and other Department of Education data. Also, non-federal negotiated rulemaking participants sometimes discussed data they had gathered themselves, such as the results of a poll conducted of an association’s membership or Department of Education data that non-federal negotiators used to conduct their own statistical analyses. Respondents also indicated that examples were presented in the rulemaking process. As one negotiated rulemaking participant explained, “Honestly, there isn’t always strong quantitative information to support the issues, but sometimes you cite examples.” Respondents also discussed how more subjective and personal factors – including experiences, beliefs, emotions, and political preferences – have influenced the higher education rulemaking process. One negotiated rulemaking observer said, “I would not say research was informing those conversations; there was much more of experience and sort of gut feelings, like philosophical feelings.” Another respondent described how negotiated rulemaking participants’ own past experiences played a role in the process: There’s a lot of experience around those tables. I’ve been doing this for [more than 25] years, so I have, I think, a pretty broad perspective on what happens where the rubber meets the road. In one sense it’s anecdotal, but it’s from a lot of experience and working at different types of schools with all different types of students. Table 2 indicates that experience was discussed on a larger proportion of pages in the Gainful Employment rescission than any other case-study rule. This makes sense given that the gainful employment policy had a relatively long history, which included the 2011 version, a negative court action regarding that rule (see Nolter, 2012), and the development of the 2014 version. The 2019 rule provided reasons to justify the rescission of that rule (Program Integrity, 2019). Thus, the rescission had to reckon with a lot of past experience, particularly the rule’s own regulatory history. As an observer of these and other higher education rulemakings said, “A lot of times it’s history and it’s expertise. It’s not data” that drive the rulemaking. Some interviewees also discussed how the law has influenced decision making in higher education rulemaking. As the authorizing statute, the HEA served as a key consideration in the development of these regulations, as evidenced by numerous citations to the Act in final regulations as well as statements from some interviewees. As one respondent who worked for the federal government said, “I think one constant question is legal authority, and of course any rulemaking must fill in the gaps of the statute, not contradict the statute.” Besides the HEA, other laws and legal matters were sometimes considered in the rulemaking process. For example, participants in the Borrower Defense rulemaking said there was consideration of consumer protection and trade practice laws in the development of that regulation as well as its subsequent modification. Other legal issues discussed during the creation of the case-study rules included statutes of limitations, burdens of proof, and in the words of one negotiated rulemaking participant, information about “judgements and settlements that had occurred between students or student advocate groups and the for-profit sector.” Barriers to Research Utilization in Higher Education Rulemaking Education Policy Analysis Archives Vol. 28 No. 95 22 Respondents also discussed their observations about barriers to using research in the higher education rulemaking process. Several barriers were identified repeatedly by respondents, each of which is described in this section in order of frequency with which they were reported. Although these barriers are presented separately, these descriptions show that there was often overlap between two or more of them. Time constraints. More than three quarters of interviewees discussed issues relating to time constraints in higher education rulemaking as a barrier to using more research in the process. These shortages of time were present in different aspects of the rulemaking process. With regard to negotiated rulemaking, some respondents said the relatively short amount of time in which negotiated rulemaking occurs presented a barrier to research use because conducting original research can take much longer than the time allotted for negotiated rulemaking to take place. As one lending industry representative remarked, “Research, you don’t do overnight.” Relatedly, when data or research was presented during negotiated rulemaking, participants typically had little time to review and consider it. One public university representative said: A unique part of the negotiated rulemaking process is that you don’t usually receive what you’re asking for in enough time to actually use it… And so we got most of that data either after the session was over, so we would have to remember to go back and talk about it at the next session, or even after the rulemaking had concluded in its entirety. Another way time constraints presented a barrier to research utilization was when research was presented or considered during later stages of the rulemaking process. Several respondents indicated that the later research was presented in the process, the more the policy and ideas behind the rule were already crystalized, and therefore it was less likely that research or other factors would prompt changes at that point. For example, one federal official said that by the time meetings with OMB personnel were held, the government’s “ideas are pretty set in stone at that point, and so it would be hard for that kind of research to really have a big impact.” Desired data not available. Also reported by more than three quarters of respondents was the fact that data desired by rulemaking participants were often not available. Several respondents said they observed negotiated rulemaking participants make requests for data from the Department of Education, and that those requests sometimes went unfulfilled or were fulfilled in an untimely manner.10 One negotiated rulemaking participant described this issue: I felt like the difficulty with it was it took the Department of Education a long time to respond to data requests from the committee. I think there’s a lot of reasons for that … but that was a challenge sometimes. So if certain people on the committee had requests, it did take a while to get those requests answered. Notably, the fact that desired data were not readily available is related to the issue of timing described above: if it takes a long while to receive requested data, then there is less time for stakeholders to conduct research with that data during the rulemaking process. Also, the statutory prohibition on the Department of Education from collecting student- level data was discussed by several respondents as a barrier to research use in higher education rulemaking. The HEA states that, with some limited exceptions, the government shall not 10 Some negotiated rulemaking records also reflected these requests. The matter of data requests not always being fulfilled in a timely manner is related to another barrier to research use – lack of government capacity – that is discussed later in this section. Research Utilization in Higher Education Rulemaking 23 “authorize the development, implementation, or maintenance of a Federal database of personally identifiable information on individuals receiving assistance … or otherwise involved in any studies or other collections of data under” the Act (Higher Education Act, 2018, § 1015c[a]). Some respondents said that this statutory prohibition was a barrier to research use in rulemaking because it prevented researchers from having access to relevant data. One negotiated rulemaking participant said that this prohibition meant that the department “doesn’t have perfect data… So they don’t have full information about every borrower. They don’t know which borrowers all have private loans for example.” A respondent who worked for the federal government said that this prohibition contributed to “a lack of available data … in the higher ed space.” Highly politicized process. Policymaking tends to be highly political (Minor & Mutamba, 2017), and higher education rulemaking is no exception. About two-thirds of respondents indicated that the fact that higher education rulemaking is very politicized can present a barrier to the use of research in the process. One way that this barrier manifested itself was when policy actors were so committed to their political positions that they disregarded research that was contrary to these positions. A negotiated rulemaking participant said that in higher education rulemaking, “folks come to the table with very preconceived notions about what they want out of this process and what they’re willing to accept.” This same respondent observed that “not only is [research] produced sparingly [in the process], but when it is produced, it’s mostly ignored.” Another way that politics presented a hindrance to research use was that the political nature of higher education rulemaking caused some participants to be mistrustful of research that was presented in the process (see also Natow, 2020). One federal official said that individuals who attended OMB meetings were often “very interested parties… So the evidence they present is sometimes viewed through a very suspect lens.” Another respondent explained how sometimes this skepticism is warranted: There’s a whole cottage industry – it used to be concentrated in Washington, thanks to the Internet it’s now widespread across the country – of what I call think-tanky stuff, which is nonpeer reviewed, nonrigorous, oftentimes very facade reportage with a preexisting point of view… So there is an ideological tilt to most of the stuff that I think does pass for scientific literature and even those, again, are highly susceptible to selective usage. Thus, the barrier to research use presented by politics was more complicated than simply preventing the consideration of research in higher education rulemaking. Rather, due to politics, research presented in the process was often not trusted or relied upon by policymakers. Mismatches between the research and policy communities. Nearly two-thirds of respondents made statements demonstrating some fundamental differences between the research and policy communities, and they illustrated how these differences presented a barrier to research use in higher education rulemaking. Some of these mismatches overlap with other barriers to research use. For example, the time constraints barrier discussed above reflected a mismatch between the research and policy communities regarding timeframes: the policy world moves quickly, while rigorous research takes more time. Also, research findings have sometimes been presented in ways that are not conducive to being understood by a busy policy audience. As a long-time higher education policy observer said, “Academics speak with great nuance and oftentimes in very turgid and highly technical, very specialized language that doesn’t always – that the regulators can’t necessarily fathom, let alone capture.” Moreover, a negotiated rulemaking participant said that policymakers are more likely to appreciate “high quality talking points... that very busy people can digest, and articulate, and Education Policy Analysis Archives Vol. 28 No. 95 24 analyze.” Academic research does not always present findings in such a format. A respondent who worked for the Department of Education said, “classic academic research is hard for people who are not in the field doing the research to see the connections. It’s helpful for somebody to sort of find that for them.” These mismatches are illustrative of the “two communities” perspective in the literature of research utilization, which posits that researchers and policymakers occupy separate “communities,” and that this is a reason why more research is not used in policymaking (e.g., Caplan, 1979; Ness, 2010). Interestingly, respondents provided statements that demonstrated how this separation of the research and policy communities has not only occurred between organizations who participate in the rulemaking process, but also within them. While higher education institutions were represented on all of the negotiated rulemaking committees, it was not typically the research divisions of those institutions that participated. Rulemaking documents show that, although researchers and faculty members did participate in negotiated rulemaking occasionally, far more often it was financial aid or business officers, senior executives, and other non-research administrators. Also, for at least one of the case-study rules, the research and policy personnel within the Department of Education did not work closely together on the rulemaking until toward the end of the process. A respondent who worked on that rule as a Department of Education official said that one of the department’s other divisions (the Policy and Program Studies Service) was “very research oriented.” But that division provided research support in the development of the rule only “closer to the end of that process.” The same respondent said, “It took us forever to figure out what they were.” These data illustrate how the two-communities phenomenon may exist within a single organization – whether the Department of Education or a university – where the work of personnel active in policy and the work of those active in research do not routinely intersect. Lack of government capacity. About one-fifth of respondents indicated that the federal government has lacked sufficient capacity to respond to data requests and to stay informed about recent research, and that this lack of capacity has presented a hindrance to research use in rulemaking. As explained above, mere data and statistics do not in themselves constitute research, because research involves a systematic inquiry conducted according to a methodological plan (Penuel et al., 2016). But access to relevant data is an important prerequisite to a researcher’s ability to conduct an empirical study. Thus, a lack of government capacity to provide relevant data hinders researchers’ ability to conduct studies relevant to the topics at issue in higher education rulemaking. As mentioned above, stakeholders made data requests of the department that often went unfulfilled during the rulemaking process. Some interviewees said that this may be at least partially due to a lack of capacity within the government to respond to such requests.11 One respondent who had worked for the Department of Education said, “Right now, the capacity constraints at [the department’s Office of Federal Student Aid] are very significant, and getting a data run is not easy.” In a more nuanced explanation, a respondent who had worked for the federal government said: I was going to say there’s not a lot of expertise in the federal government to really incorporate [research], but that’s just not quite what I mean. I think there’s not enough expertise relative to the size of the magnitude of work that they have to process. There’s sort of like a lack of capacity to be able to stay on top of the body of literature that’s relevant for any given issue. 11 Interviewees’ statements about the department’s lack of capacity are consistent with news reporting at the time that the size of the Department of Education’s staff had decreased considerably (see Kreighbaum, 2018a). Research Utilization in Higher Education Rulemaking 25 Discussion and Conclusion This multi-case study was the first to examine the issue of research utilization in federal higher education rulemaking, an important educational policymaking process that has substantial implications for student financial aid, institutional accreditation, and other important policy matters affecting higher education. Federal law requires agencies to conduct regulatory assessments and cost-benefit analyses (Belfield et al., 2018), and to have a “reasoned basis” for their regulations (APSCU v. Duncan, 2012, pp. 1, 30-31). By examining the cases of five higher education regulations, this study found that research was used in the creation of these rules and that references to research appeared in the texts of their final regulations. However, factors other than research – that is, data apart from a formal study, experience, examples, law, and beliefs – were also influential and were discussed on a greater proportion of pages than research in all of the case-study final rules. Also, government research – whether the Department of Education’s own research or that of another federal office – was the only research source cited in all five case-study rules. This study also found that the contexts in which particular rules were created shaped the prevalence and sources of research that were used in the rules’ development. Case-study rules that had a history of litigation or were otherwise controversial (i.e., the two Gainful Employment rules and the Teacher Preparation rule) discussed research on a greater proportion of pages than the other case-study rules. The three controversial rules also cited at least 10 academic journal articles each, while very few academic journals were cited in the other case-study rules. Statements from interviewees suggested that these patterns were related to the histories and subject matters of the regulations: the Gainful Employment rules had a lengthy history of controversy and litigation, the Teacher Preparation rule was also controversial (see also Stratford, 2014), and both policies involved noncompliance penalties that could theoretically jeopardize an institution’s eligibility to receive federal student financial aid funds. Therefore, federal officials anticipated legal challenges for these rules, and they attempted to protect the rules from legal jeopardy by using relatively more research in the development of those regulations. Also, the Department of Education did not have access to student-level data while developing the Teacher Preparation rule, and around that same time, a number of relevant studies had been published in the academic literature. Consequently, the Teacher Preparation rule was the only case-study rule to cite more academic journal articles than government reports. Finally, the positions and networks of rulemaking participants were another important context for research use in this process. Although considerably more negotiators in the case-study rules were employed by individual colleges than by associations or advocacy groups, interested higher education associations sometimes gave support, including information about research, to negotiators, indicating these organizations’ role as intermediaries in the rulemaking process. Several barriers obstructed the use of more research in higher education rulemaking, including time constraints, the unavailability of desired data, the highly political nature of the process, and the lack of government capacity to conduct research and respond to all data requests. Certain disjunctions between the research and policy communities also impeded research use in the process. One disjunction involved timing incongruities between researchers and policymakers: policymakers attend to policy problems more quickly than research on those problems can typically be conducted. Moreover, policymakers tend to prefer reports that provide concise and distilled points and recommendations, whereas academic researchers often write in ways that are more nuanced, highly technical, and contain academic jargon. Although higher education rulemaking proceedings, and negotiated rulemaking in particular, provide a unique opportunity for research organizations to work closely with policymakers in the policy development process, this study found mismatches between researchers and policymakers that are reminiscent of the “two communities” Education Policy Analysis Archives Vol. 28 No. 95 26 perspective (e.g., Caplan, 1979; Ness, 2010), and these gaps existed despite the fact that higher education rulemaking brings research-oriented institutions and policymakers together. Indeed, this study discovered incidents of research-policy disjunctions even within an organization, such as where universities were represented on negotiated rulemaking committees by non-research administrators, or when policymakers at the Department of Education did not work with one of the department’s research offices until near the end of a rulemaking procedure. The findings of this study suggest that the veracity of the “two communities” perspective should not be dismissed, but the theory should be modified to better account for contextual and organizational influences on the willingness and ability of researchers and policy actors to make use of each other’s work. Some prior literature recognized that context plays a role in whether and how research gets used in decision making (Farley-Ripple et al., 2018; Head, 2015; Holzer et al., 2007; Levin, 2013; Nutley et al., 2007). This study of research utilization in higher education rulemaking contributes to that literature by illustrating how localized elements of context can influence the extent to which research is used, the sources of research used, and the barriers to research use in policymaking. These elements of context include the subject matter of a policy (Does research exist on this topic, and if so, who conducted the research?), the resources available to the policymaking entity (Does this agency have access to relevant data? Does it have the capacity to conduct research or respond to data requests?), the history and possible consequences of the policy (Does this policy have a contentious history? Are lawsuits likely to result, and if so, is the policy likely to be jeopardized by not having an evidence-based foundation?), organizational structure and relationships (Is the work of policy personnel separated from the work of research personnel in this organization? Are policy actors and researchers in regular communication with each other? Do policy actors’ professional networks contain “knowledge brokers” [e.g., Rodway, 2015, p. 4]?), and even timing (Has there been any recent research that may be relevant to this policy?). Contexts such as these can influence whether, how much, and what kinds of research are used in the creation of policy, and may also lead members of the often-separate policy and research communities to collaborate. This study has implications for policy and practice that may help to improve the use of research in rulemaking and other educational policymaking processes. One recommendation is for the Department of Education to include a place for researchers on negotiated rulemaking committees (see also Kelchen, 2019). Respondents noted that although this has not occurred often, the times when a researcher was present at negotiated rulemaking were useful in helping the participants make sense of the data and research that were considered during negotiations. Just as the department selects representatives from a number of different stakeholder groups to participate in negotiated rulemaking, it should also appoint experienced researchers as primary and alternate negotiators on future rulemaking panels. Moreover, Congress should eliminate the part of the HEA that bans the Department of Education from gathering student-level data, and that data should be made available to government and outside researchers. This recommendation was made by some of this study’s respondents, and it echoes similar calls from many in the policy community (Kreighbaum, 2017; Miller, 2016). Congress should allocate additional resources to the Department of Education to enhance its capacity to conduct research and respond to data requests via hiring additional research staff, upgrading relevant technologies, and providing enhanced training for staff on research and data use (see also Henig, 2008). For their part, researchers should strive to study policy-relevant topics and include specific policy recommendations in their reports (see also Henig, 2008). Researchers may also provide more visibility to their findings by authoring opinion pieces and other concise publications that are suitable for policy and practitioner audiences. Higher education leaders can assist in this endeavor by providing editorial and public relations support for researchers at their institutions who seek to Research Utilization in Higher Education Rulemaking 27 publicize policy-relevant findings. When institutional executives and other non-research administrators are preparing to participate in policymaking activities, they should consult with researchers within their organizations who have conducted studies on relevant issues. For example, a university administrator planning to participate in negotiated rulemaking should meet with education faculty or other researchers at their institution to discuss how research findings might be relevant and useful to the topics that will be the subject of the rulemaking. To incentivize researchers to connect more with the policy community, higher education leaders should reform faculty and research incentive structures to reward research that targets policymakers (see also Henig, 2008). Higher education institutions might also work with government agencies to provide training for government staff on research methodologies. Despite persistent incongruities between the research and policy communities, one goal that researchers and policymakers often share is the desire to improve the use of research in policymaking. By illuminating the extent of and barriers to research use in higher education rulemaking, the findings presented here can help researchers and policymakers design effective strategies for bringing more and better research into policymaking for the purpose of creating policies that improve education. Acknowledgements The author thanks Dr. Kevin Dougherty and Dr. Richard Sevilla for comments they provided on an earlier draft of this article. References Acs, A., & Cameron, C. M. (2013). Does White House regulatory review produce a chilling effect and “OIRA avoidance” in the agencies? Presidential Studies Quarterly, 43(3), 443-467. Administrative Procedure Act, 5 United State Code §§ 500 et seq. (2018). Alkin, M. C., & King, J. A. (2017). Definitions of evaluation use and misuse, evaluation influence, and factors affecting use. American Journal of Evaluation, 38(3), 434-450. Association of Private Sector Colleges & Universities v. Duncan, Memorandum Opinion, Civil Action 11-1314 (RC) (D.D.C. June 30, 2012). Asen, R., Gurke, D., Solomon, R., Conners, P., & Gumm, E. (2011). “The research says”: Definitions and uses of a key policy term in federal law and local school board deliberations. Argumentation and advocacy, 47, 195-213. Asen, R., Gurke, D., Conners, P., Solomon, R., & Gumm, E. (2013). Research evidence and school board deliberations: Lessons from three Wisconsin school districts. Educational Policy, 27(1), 33-63. Belfield, C. R., Bowden, A. B., & Rodgriguez, V. (2018). Evaluating regulatory impact assessments in education policy. American Journal of Evaluation. https://doi.org/10.1177/1098214018785463 Birnbaum, R. (2000). Policy scholars are from Venus; policy makers are from Mars. The Review of Higher Education, 23(2), 119-132. Brito, J., & De Rugy, V. (2009). Midnight regulations and regulatory review. Administrative Law Review, 61(1), 163-196. Caplan, N. (1979). The two-communities theory and knowledge utilization. American Behavioral Scientist, 22(3), 459-470. Clarke, D. (2016). Law, regulation and strategizing for health. In Strategizing national health in the Education Policy Analysis Archives Vol. 28 No. 95 28 21st century: A handbook. Geneva, Switzerland: World Health Organization. Cochran, J. (2001). The Cinderella constraint: Why regulations increase significantly during post-election quarters. [Unpublished report]. Fairfax, VA: George Mason University. http://mason.gmu.edu/~jcochra1/index_files/SRO.pdf Cooper, A. (2014). The use of online strategies and social media for research dissemination in education. Education Policy Analysis Archives, 22 (88), 1-24. https://doi.org/10.14507/epaa.v22n88.2014 Cooper, A., & Levin, B. (2010). Some Canadian contributions to understanding knowledge mobilisation. Evidence & Policy: A Journal of Research, Debate and Practice, 6(3), 351-369. Cooper, A., Levin, B., & Campbell, C. (2009). The growing (but still limited) importance of evidence in education policy and practice. Journal of Educational Change, 10, 159-171. Cooper, A., & Shewchuk, S. (2015). Knowledge brokers in education: How intermediary organizations are bridging the gap between research, policy and practice internationally. Education Policy Analysis Archives, 23(118), 1-8. https://doi.org/10.14507/epaa.v23.2355 Costa, M., Desmarais, B. A., & Hird, J. A. (2019). Public comments’ influence on science use in U.S. rulemaking: The case of EPA’s National Emission Standards. The American Review of Public Administration, 49(1), 36-50. Court, J., & Maxwell, S. (2005). Policy entrepreneurship for poverty reduction: bridging research and policy in international development. Journal of International Development: The Journal of the Development Studies Association, 17(6), 713-725. Creswell, J. W., & Poth, C. N. (2018). Qualitative inquiry and research design (4th ed.). Thousand Oaks, CA: Sage. Davies, P. (2004). Is evidence-based government possible? Paper presented at the 4th annual Campbell Collaboration Colloquium. Washington, DC. Executive Order No. 12866, Title 3 C.F.R. 638 (1993 comp.). Executive Order No. 13563, Title 3 C.F.R. 215 (2011 comp.). Farley-Ripple, E. N. (2012). Research use in school district central office decision making: A case study. Educational Management Administration & Leadership, 40(6), 786-806. Farley-Ripple, E., & Grajeda, S. (2020). Avenues of influence: An exploration of school-based practitioners as knowledge brokers and mobilizers. In J. Malin & C. Brown (Eds.), The role of knowledge brokers in education: Connecting the dots between research and practice (pp. 65-89). London: Routledge. Farley-Ripple, E., May, H., Karpyn, A., Tilley, K., & McDonough, K. (2018). Rethinking connections between research and practice in education: A conceptual framework. Educational Researcher, 47(4), 235-245. Gollust, S. E., Kite, H. A., Benning, S. J., Callanan, R. A., Weisman, S. R., & Nanney, M. S. (2014). Use of research evidence in state policymaking for childhood obesity prevention in Minnesota. American Journal of Public Health, 104(10), 1894-1900. Head, B. W. (2015). Toward more “evidence-informed” policymaking? Public Administration Review, 76(3), 472-484. Henig, J. R. (2008). Spin cycle: How research is used in policy debates: The case of charter schools. New York, NY: Russell Sage Foundation. Higher Education Act, 20 U.S. Code §§ 1001 et seq. (2018). Holzer, P., Lewig, K., Bromfield, L., & Arney, F. (2007). Research use in the Australian child and welfare sector. Adelaide, Australia: University of South Australia. Research Utilization in Higher Education Rulemaking 29 Kelchen, R. (2019, January 10). The Department of Education needs a researcher on its rule-making panel. Washington Monthly. https://washingtonmonthly.com/2019/01/10/the-department- of-education-needs-a-researcher-on-its-rule-making-panel/ Kerwin, C. M., & Furlong, S. R. (2011). Rulemaking: How government agencies write law and make policy (4th ed.). Washington, DC: CQ Press. Kocbek, A., & Juric, M. B. (2010). Integrating enterprise portal with service oriented architecture. International Journal of Advanced Research in Computer Science, 1(3). Kreighbaum, A. (2016, October 28). Borrower defense rules finalized. Inside Higher Education. https://www.insidehighered.com/news/2016/10/28/education-dept-releases-final-version- defense-repayment-loan-rules Kreighbaum, A. (2017, May 16). Push for “unit records” revived. Inside Higher Education. https://www.insidehighered.com/news/2017/05/16/bipartisan-bill-would-overturn- federal-ban-student-unit-record-database Kreighbaum, A. (2018a, June 13). Under DeVos, a smaller Department of Education. Inside Higher Education. https://www.insidehighered.com/news/2018/06/13/education-department-staff- down-13-percent-trump-administration-began Kreighbaum, A. (2018b, October 13). DeVos won’t seek another delay of Obama borrower defense rule. Inside Higher Education. https://www.insidehighered.com/quicktakes/2018/10/13/devos-wont-seek-another-delay- obama-borrower-defense-rule Kreighbaum, A. (2019, May 28). Turning to courts for loan forgiveness. Inside Higher Education. https://www.insidehighered.com/news/2019/05/28/some-defrauded-borrowers- win-court-debt-relief-claims-pile Lederman, D. (2019, July 24). California students caught in the middle. Inside Higher Education. https://www.insidehighered.com/digital-learning/article/2019/07/24/dispute-over-federal- rules-state-authorization-imperils Levin, B. (2013). To know is not enough: research knowledge and its use. Review of Education, 1(1), 2- 31. Locock, L., & Boaz, A. (2004). Research, policy and practice–worlds apart?. Social Policy and Society, 3(4), 375-384. Lubbers, J. S. (2014, December 5). Enhancing the use of negotiated rulemaking by the U.S. Department of Education. doi:10.2139/ssrn.2764954 Lubienski, D., Scott, J., & DeBray, E. (2014). The politics of research production, promotion, and utilization in educational policy. Educational Policy, 28(2), 131-144. Malin, J. R., & Lubienski, C. (2015). Educational expertise, advocacy, and media influence. Education Policy Analysis Archives, 23(6). https://doi.org/10.14507/epaa.v23.1706 Marshall, M. N. (1996). Sampling for qualitative research. Family Practice, 13(6), 522-526. McCright, A. M., & Dunlap, R. E. (2008). The nature and social bases of progressive social movement ideology: Examining public opinion toward social movements. The Sociological Quarterly, 49(4), 825-848. Merriam, S. B. (1998). Qualitative Research and Case Study Applications in Education. San Francisco, CA: Jossey-Bass. Miles, M. B., Huberman, A. M., & Saldaña, J. (2019). Qualitative data analysis: A methods sourcebook (4th ed.). Thousand Oaks, CA: Sage. Miller, B. (2016, May 23). Building a student-level data system. Washington, DC: Center for American Progress. Education Policy Analysis Archives Vol. 28 No. 95 30 Minor, J. T., & Matamba, C. (2017). Qualitative research and public policy. In P. A. Pasque & V. M. Lechuga (Eds.), Qualitative inquiry in higher education organization and policy research (pp. 131-141). New York, NY: Routledge. Morgan, C. A., Gelles, M. G., Steffian, G., Temporini, H., Fortunati, F., Southwick, S., Feuerstein, S., & Coric, V. (2006). Consulting to government agencies: Indirect assessments. Psychiatry (Edgmont), 3(2), 24. Naidorf, J. C. (2014). Knowledge utility: From social relevance to knowledge mobilization. Education Policy Analysis Archives, 22(89), 1-31. https://doi.org/10.14507/epaa.v22n89.2014 Natow, R. S. (2017). Higher education rulemaking: The politics of creating regulatory policy. Baltimore, MD: Johns Hopkins University Press. Natow, R. S. (2020). Research use and politics in the federal higher education rulemaking process. Educational Policy. DOI: 10.1177/0895904820917363 Negotiated Rulemaking Committee; Negotiator Nominations and Schedule of Committee Meetings – Accreditation and Innovation: Intent to establish negotiated rulemaking committee, 83 Fed. Reg. 51906 (2018). Ness, E. C. (2010). The role of information in the policy process: Implications for the examination of research utilization in higher education policy. In J. C. Smart (Ed.), Higher Education: Handbook of Theory and Research, 25, 1-49. Netherlands: Springer. Ness, E. C. & Gándara, D. (2014). Ideological think tanks in the states: An inventory of their prevalence, networks, and higher education policy activity. Educational Policy, 28(2), 258-280. Newman, J., Cherney, A., & Head, B. W. (2016). Do policy makers use academic research? Reexamining the “two communities” theory of research utilization. Public Administration Review, 76(1), 24-32. Ng-A-Fook, N., Kane, R., Butler, J., Glithero, L., & Forte, R. (2015). Brokering knowledge mobilization networks: Policy reforms, partnerships, and teacher education. Education Policy Analysis Archives, 23 (122), 1-30. https://doi.org/10.14507/epaa.v23.2090 Nolter, C. (2012, July 5). For-profit colleges win in court, face new uncertainty. The Deal Pipeline. Lexis-Nexis Academic. Nutley, S. M., Walter, I., & Davies, H. T. O. (2007). Using evidence: How research can inform public services. Chicago, IL: Policy Press. Office of the Federal Register. (2011, January). A guide to the rulemaking process. https://www.federalregister.gov/uploads/2011/01/the_rulemaking_process.pdf. Pelesh, M. L. (1994). Regulations under the Higher Education Amendments of 1992: A case study in negotiated rulemaking. Law & Contemporary Problems, 57(4), 151-170. Penuel, W. R., Briggs, D. C., Davidson, K. L., Herlihy, C., Sherer, D., Hill, H. C., Farrell, C. C., & Allen, A. (2016, April). Findings from a national study on research use among school and district leaders. [Technical Report No. 1]. National Center for Research in Policy & Practice. Perna, L. W., Orosz, K., & Kent, D. C. (2019). The role and contribution of academic researchers in congressional hearings: A critical discourse analysis. American Educational Research Journal, 56(1), 111-145. Program Integrity: Gainful Employment, Final regulations. 79 Fed. Reg. 64890 (2014a) (to be codified at 34 C.F.R. pts. 600 and 668). Program Integrity: Gainful Employment, Notice of proposed rulemaking. 79 Fed. Reg. 16426 (2014b). Program Integrity: Gainful Employment, Notice of proposed rulemaking. 83 Fed. Reg. 40167 Research Utilization in Higher Education Rulemaking 31 (2018). Program Integrity: Gainful Employment, Final regulations. 84 Fed. Reg. 31392 (2019) (to be codified at 34 C.F.R. pts. 600 and 668). Rich, R. F. (1991). Knowledge creation, diffusion, and utilization: Perspectives of the founding editor of Knowledge. Knowledge, 12(3), 319-337. Rodway, J. (2015). Connecting the dots: Understanding the flow of research knowledge within a research brokering network. Education Policy Analysis Archives, 23 (123), 1-27. https://doi.org/10.14507/epaa.v23.2180 Rosenthal, R. J. (2012). International Sleeve Gastrectomy Expert Panel Consensus Statement: Best practice guidelines based on experience of > 12,000 cases. Surgery for Obesity and Related Diseases, 8(1), 8-19. Santesso, N., & Tugwell, P. (2006). Knowledge translation in developing countries. Journal of Continuing Education in the Health Professions, 26, 87-96. Scott, J., Lubienski, C., DeBray, E., & Jabbar, H. (2014). The intermediary function in evidence production, promotion, and utilization: The case of educational incentives. In K. S. Finnigan & A. J. Daly (Eds.), Using research evidence in education: From the schoolhouse door to Capitol Hill (pp. 69–89). New York, NY: Springer. Simons, M. (2015). Governing education without reform: the power of the example. Discourse: Studies in the Cultural Politics of Education, 36(5), 712-751. Stephan, P. (2017, July 12). Congress rejects teacher preparation reporting rule. The Regulatory Review. https://www.theregreview.org/2017/07/12/stephan-teacher-preparation-programs/ Stratford, M. (2014, April 25). Teacher prep rules are coming. Inside Higher Education. https://www.insidehighered.com/news/2014/04/25/obama-administration-will-move- ahead-controversial-new-rules-teacher-preparation Stratford, M. (2015, May 1). Income-based repayment expansion advances. Inside Higher Education. https://www.insidehighered.com/news/2015/05/01/federal-rule-making-panel-oks-plan- expand-income-based-repayment-program Student Assistance General Provisions, Federal Family Education Loan Program, and William D. Ford Federal Direct Loan Program: Final regulations, 80 Fed. Reg. 67204 (2015a) (to be codified at 34 C.F.R. pts. 668, 682, & 685). Student Assistance General Provisions, Federal Family Education Loan Program, and William D. Ford Federal Direct Loan Program: Notice of proposed rulemaking, 80 Fed. Reg. 39608 (2015b). Student Assistance General Provisions, Federal Perkins Loan Program, Federal Family Education Loan Program, William D. Ford Federal Direct Loan Program, and Teacher Education Assistance for College and Higher Education Grant Program: Final regulations, 81 Fed. Reg. 75926 (2016a) (to be codified at 34 C.F.R. pts. 30, 668, 674, 682, 685, & 686). Student Assistance General Provisions, Federal Perkins Loan Program, Federal Family Education Loan Program, William D. Ford Federal Direct Loan Program, and Teacher Education Assistance for College and Higher Education Grant Program: Notice of proposed rulemaking, 81 Fed. Reg. 39330 (2016b). Sunstein, C. R. (2013). The Office of Information and Regulatory Affairs: Myths and realities. Harvard Law Review, 126, 1838-1878. Suri, H. (2011). Purposeful sampling in qualitative research synthesis. Qualitative Research Journal, 11(2), 63-75. Teacher Preparation Issues: Notice of proposed rulemaking, 79 Fed. Reg. 71820 (2014). Teacher Preparation Issues: Final regulations, 81 Fed. Reg. 75494 (2016) (to be codified at 34 Education Policy Analysis Archives Vol. 28 No. 95 32 C.F.R. pts. 612 & 686). Tseng, V. (2012). The uses of research in policy and practice. Washington, DC: Society for Research in Child Development. Tseng, V., & Nutley, S. (2014). Building the infrastructure to improve the use and usefulness of research in education. In K. S. Finnigan & A. J. Daly (eds.), Using research evidence in education: From the schoolhouse door to Capitol Hill (pp. 163-175). Cham, Switzerland: Springer. U.S. Department of Education. (2014, October 30). Obama administration announces final rules to protect students from poor-performing career college programs. https://www.ed.gov/news/press- releases/obama-administration-announces-final-rules-protect-students-poor-performing- career-college-programs U.S. Department of Education. (2019). Student Assistance General Provisions and Federal Family Education Loan Program. https://www2.ed.gov/policy/highered/reg/hearulemaking/2017/borrower-defense-final- rule.pdf U.S. Office of Management & Budget. (2003). OMB Circular A-4 compliance supplement. Washington, DC: Executive Office of the President, Office of Management & Budget. Weiss, C. H. (1979). The many meanings of research utilization. Public Administration Review, 39(5), 426–431. Weiss, C. H. (1998). Have we learned anything new about the use of evaluation? American Journal of Evaluation, 19(1), 21-33. Yin, R. K. (2014). Case study research: Design and methods. Thousand Oaks, CA: Sage. About the Author Rebecca Natow Hofstra University Rebecca.S.Natow@hofstra.edu https://orcid.org/0000-0003-2948-1904 Rebecca S. Natow is an Assistant Professor of Educational Leadership and Policy at Hofstra University. Her research and teaching are focused on higher education policy, higher education leadership, and qualitative research methods. mailto:Rebecca.S.Natow@hofstra.edu https://orcid.org/0000-0003-2948-1904 Research Utilization in Higher Education Rulemaking 33 Readers are free to copy, display, distribute, and adapt this article, as long as the work is attributed to the author(s) and Education Policy Analysis Archives, the changes are identified, and the same license applies to the derivative work. More details of this Creative Commons license are available at https://creativecommons.org/licenses/by-sa/4.0/. EPAA is published by the Mary Lou Fulton Institute and Graduate School of Education at Arizona State University Articles are indexed in CIRC (Clasificación Integrada de Revistas Científicas, Spain), DIALNET (Spain), Directory of Open Access Journals, EBSCO Education Research Complete, ERIC, Education Full Text (H.W. Wilson), QUALIS A1 (Brazil), SCImago Journal Rank, SCOPUS, SOCOLAR (China). Please send errata notes to Audrey Amrein-Beardsley at audrey.beardsley@asu.edu Join EPAA’s Facebook community at https://www.facebook.com/EPAAAAPE and Twitter feed @epaa_aape. education policy analysis archives ISSN 1068-2341 June 22, 2020 Volume 28 Number 95 https://creativecommons.org/licenses/by-sa/4.0/ http://www.doaj.org/ mailto:audrey.beardsley@asu.edu https://www.facebook.com/EPAAAAPE Education Policy Analysis Archives Vol. 28 No. 95 34 education policy analysis archives editorial board Lead Editor: Audrey Amrein-Beardsley (Arizona State University) Editor Consultor: Gustavo E. Fischman (Arizona State University) Associate Editors: Melanie Bertrand, David Carlson, Lauren Harris, Eugene Judson, Mirka Koro-Ljungberg, Daniel Liou, Scott Marley, Molly Ott, Iveta Silova (Arizona State University) Madelaine Adelman Arizona State University Amy Garrett Dikkers University of North Carolina, Wilmington Gloria M. Rodriguez University of California, Davis Cristina Alfaro San Diego State University Gene V Glass Arizona State University R. Anthony Rolle University of Houston Gary Anderson New York University Ronald Glass University of California, Santa Cruz A. G. Rud Washington State University Michael W. Apple University of Wisconsin, Madison Jacob P. K. Gross University of Louisville Patricia Sánchez University of University of Texas, San Antonio Jeff Bale University of Toronto, Canada Eric M. Haas WestEd Janelle Scott University of California, Berkeley Aaron Benavot SUNY Albany Julian Vasquez Heilig California State University, Sacramento Jack Schneider University of Massachusetts Lowell David C. Berliner Arizona State University Henry Braun Boston College Kimberly Kappler Hewitt University of North Carolina Greensboro Noah Sobe Loyola University Casey Cobb University of Connecticut Aimee Howley Ohio University Nelly P. Stromquist University of Maryland Arnold Danzig San Jose State University Steve Klees University of Maryland Jaekyung Lee SUNY Buffalo Benjamin Superfine University of Illinois, Chicago Linda Darling-Hammond Stanford University Jessica Nina Lester Indiana University Adai Tefera Virginia Commonwealth University Elizabeth H. DeBray University of Georgia Amanda E. Lewis University of Illinois, Chicago A. Chris Torres Michigan State University David E. DeMatthews University of Texas at Austin Chad R. Lochmiller Indiana University Tina Trujillo University of California, Berkeley Chad d'Entremont Rennie Center for Education Research & Policy Christopher Lubienski Indiana University Federico R. Waitoller University of Illinois, Chicago John Diamond University of Wisconsin, Madison Sarah Lubienski Indiana University Larisa Warhol University of Connecticut Matthew Di Carlo Albert Shanker Institute William J. Mathis University of Colorado, Boulder John Weathers University of Colorado, Colorado Springs Sherman Dorn Arizona State University Michele S. Moses University of Colorado, Boulder Kevin Welner University of Colorado, Boulder Michael J. Dumas University of California, Berkeley Julianne Moss Deakin University, Australia Terrence G. Wiley Center for Applied Linguistics Kathy Escamilla University of Colorado, Boulder Sharon Nichols University of Texas, San Antonio John Willinsky Stanford University Yariv Feniger Ben-Gurion University of the Negev Eric Parsons University of Missouri-Columbia Jennifer R. Wolgemuth University of South Florida Melissa Lynn Freeman Adams State College Amanda U. Potterton University of Kentucky Kyo Yamashiro Claremont Graduate University Rachael Gabriel University of Connecticut Susan L. Robertson Bristol University Miri Yemini Tel Aviv University, Israel Research Utilization in Higher Education Rulemaking 35 archivos analíticos de políticas educativas consejo editorial Editores Asociados: Felicitas Acosta (Universidad Nacional de General Sarmiento), Armando Alcántara Santuario (Universidad Nacional Autónoma de México), Ignacio Barrenechea, Jason Beech ( Universidad de San Andrés), Angelica Buendia, (Metropolitan Autonomous University), Alejandra Falabella (Universidad Alberto Hurtado, Chile), Carmuca Gómez-Bueno (Universidad de Granada), Veronica Gottau (Universidad Torcuato Di Tella), Carolina Guzmán-Valenzuela (Universidade de Chile), Antonia Lozano-Díaz (University of Almería), Antonio Luzon, (Universidad de Granada), María Teresa Martín Palomo (University of Almería), María Fernández Mellizo- Soto (Universidad Complutense de Madrid), Tiburcio Moreno (Autonomous Metropolitan University-Cuajimalpa Unit), José Luis Ramírez, (Universidad de Sonora), Axel Rivas (Universidad de San Andrés), César Lorenzo Rodríguez Uribe (Universidad Marista de Guadalajara), Maria Veronica Santelices (Pontificia Universidad Católica de Chile) Claudio Almonacid Universidad Metropolitana de Ciencias de la Educación, Chile Ana María García de Fanelli Centro de Estudios de Estado y Sociedad (CEDES) CONICET, Argentina Miriam Rodríguez Vargas Universidad Autónoma de Tamaulipas, México Miguel Ángel Arias Ortega Universidad Autónoma de la Ciudad de México Juan Carlos González Faraco Universidad de Huelva, España José Gregorio Rodríguez Universidad Nacional de Colombia, Colombia Xavier Besalú Costa Universitat de Girona, España María Clemente Linuesa Universidad de Salamanca, España Mario Rueda Beltrán Instituto de Investigaciones sobre la Universidad y la Educación, UNAM, México Xavier Bonal Sarro Universidad Autónoma de Barcelona, España Jaume Martínez Bonafé Universitat de València, España José Luis San Fabián Maroto Universidad de Oviedo, España Antonio Bolívar Boitia Universidad de Granada, España Alejandro Márquez Jiménez Instituto de Investigaciones sobre la Universidad y la Educación, UNAM, México Jurjo Torres Santomé, Universidad de la Coruña, España José Joaquín Brunner Universidad Diego Portales, Chile María Guadalupe Olivier Tellez, Universidad Pedagógica Nacional, México Yengny Marisol Silva Laya Universidad Iberoamericana, México Damián Canales Sánchez Instituto Nacional para la Evaluación de la Educación, México Miguel Pereyra Universidad de Granada, España Ernesto Treviño Ronzón Universidad Veracruzana, México Gabriela de la Cruz Flores Universidad Nacional Autónoma de México Mónica Pini Universidad Nacional de San Martín, Argentina Ernesto Treviño Villarreal Universidad Diego Portales Santiago, Chile Marco Antonio Delgado Fuentes Universidad Iberoamericana, México Omar Orlando Pulido Chaves Instituto para la Investigación Educativa y el Desarrollo Pedagógico (IDEP) Antoni Verger Planells Universidad Autónoma de Barcelona, España Inés Dussel, DIE-CINVESTAV, México José Ignacio Rivas Flores Universidad de Málaga, España Catalina Wainerman Universidad de San Andrés, Argentina Pedro Flores Crespo Universidad Iberoamericana, México Juan Carlos Yáñez Velazco Universidad de Colima, México javascript:openRTWindow('http://epaa.asu.edu/ojs/about/editorialTeamBio/819') javascript:openRTWindow('http://epaa.asu.edu/ojs/about/editorialTeamBio/820') javascript:openRTWindow('http://epaa.asu.edu/ojs/about/editorialTeamBio/4276') javascript:openRTWindow('http://epaa.asu.edu/ojs/about/editorialTeamBio/1609') javascript:openRTWindow('http://epaa.asu.edu/ojs/about/editorialTeamBio/825') javascript:openRTWindow('http://epaa.asu.edu/ojs/about/editorialTeamBio/797') javascript:openRTWindow('http://epaa.asu.edu/ojs/about/editorialTeamBio/823') javascript:openRTWindow('http://epaa.asu.edu/ojs/about/editorialTeamBio/798') javascript:openRTWindow('http://epaa.asu.edu/ojs/about/editorialTeamBio/555') javascript:openRTWindow('http://epaa.asu.edu/ojs/about/editorialTeamBio/814') javascript:openRTWindow('http://epaa.asu.edu/ojs/about/editorialTeamBio/2703') javascript:openRTWindow('http://epaa.asu.edu/ojs/about/editorialTeamBio/801') javascript:openRTWindow('http://epaa.asu.edu/ojs/about/editorialTeamBio/826') javascript:openRTWindow('http://epaa.asu.edu/ojs/about/editorialTeamBio/802') javascript:openRTWindow('http://epaa.asu.edu/ojs/about/editorialTeamBio/3264') javascript:openRTWindow('http://epaa.asu.edu/ojs/about/editorialTeamBio/804') Education Policy Analysis Archives Vol. 28 No. 95 36 arquivos analíticos de políticas educativas conselho editorial Editor Consultor: Gustavo E. Fischman (Arizona State University) Editoras Associadas: Andréa Barbosa Gouveia (Universidade Federal do Paraná), Kaizo Iwakami Beltrao, (Brazilian School of Public and Private Management - EBAPE/FGVl), Sheizi Calheira de Freitas (Federal University of Bahia), Maria Margarida Machado, (Federal University of Goiás / Universidade Federal de Goiás), Gilberto José Miranda, (Universidade Federal de Uberlândia, Brazil), Marcia Pletsch (Universidade Federal Rural do Rio de Janeiro), Maria Lúcia Rodrigues Muller (Universidade Federal de Mato Grosso e Science), Sandra Regina Sales (Universidade Federal Rural do Rio de Janeiro) Almerindo Afonso Universidade do Minho Portugal Alexandre Fernandez Vaz Universidade Federal de Santa Catarina, Brasil José Augusto Pacheco Universidade do Minho, Portugal Rosanna Maria Barros Sá Universidade do Algarve Portugal Regina Célia Linhares Hostins Universidade do Vale do Itajaí, Brasil Jane Paiva Universidade do Estado do Rio de Janeiro, Brasil Maria Helena Bonilla Universidade Federal da Bahia Brasil Alfredo Macedo Gomes Universidade Federal de Pernambuco Brasil Paulo Alberto Santos Vieira Universidade do Estado de Mato Grosso, Brasil Rosa Maria Bueno Fischer Universidade Federal do Rio Grande do Sul, Brasil Jefferson Mainardes Universidade Estadual de Ponta Grossa, Brasil Fabiany de Cássia Tavares Silva Universidade Federal do Mato Grosso do Sul, Brasil Alice Casimiro Lopes Universidade do Estado do Rio de Janeiro, Brasil Jader Janer Moreira Lopes Universidade Federal Fluminense e Universidade Federal de Juiz de Fora, Brasil António Teodoro Universidade Lusófona Portugal Suzana Feldens Schwertner Centro Universitário Univates Brasil Debora Nunes Universidade Federal do Rio Grande do Norte, Brasil Lílian do Valle Universidade do Estado do Rio de Janeiro, Brasil Geovana Mendonça Lunardi Mendes Universidade do Estado de Santa Catarina Alda Junqueira Marin Pontifícia Universidade Católica de São Paulo, Brasil Alfredo Veiga-Neto Universidade Federal do Rio Grande do Sul, Brasil Flávia Miller Naethe Motta Universidade Federal Rural do Rio de Janeiro, Brasil Dalila Andrade Oliveira Universidade Federal de Minas Gerais, Brasil