Conducting Content Analysis of Documents in Network Research: A Review of Recent Scholarship Complexity, Governance & Networks-Vol 02, Issue 01 (2015) 83–102 83 DOI: 10.7564/14-CGN11 Conducting Content Analysis of Documents in Network Research: A Review of Recent Scholarship Qian Hu, Ph.D. Assistant Professor, School of Public Administration, University of Central Florida, HPA II, Suite 238 Email: Qian.Hu@ucf.edu In this study I examine how researchers have conducted content analysis of documents to collect and analyze network data in public administration. Content analyses of documents help us understand the contexts and meanings of network ties. Researchers can also analyze docu- ments to examine the changes of network ties over time. The types of documents that researchers have used to draw network data are diverse, but the ways they are used are limited. The majority of articles reviewed for this study focused on drawing interorganizational collaboration data, whereas in a few articles content analysis was conducted to draw policy network data. In this study I found that most of the existing network research has converted qualitative data extracted from documents to numerical data for quantitative analysis and has not fully taken advantage of the qualitative data. Researchers can also benefit from providing explicit coding rules and proce- dures to ensure the quality of network data collection. Keywords: Documents, content analysis, network analysis, network scholarship 1. Introduction Social network analysis (SNA) refers to a method and set of tools used to analyze so- cial structures and interactions within social structures, relational processes, and outcomes of these processes (Borgatti, Everett, & Johnson, 2013; Scott, 2013; Wasserman & Faust, 1994). Over the past few decades, SNA has garnered increasing attention from researchers in public administration (Provan & Lemaire, 2012). Researchers applied SNA to examine a wide array of management problems and policy issues. A number of authors reviewed the development of network scholarship in public administration and identified areas for further research (e.g., Berry et al., 2004; Isett, Mergel, LeRoux, Mischen, & Rethemeyer, 2011; Lecy, Mergel, & Schmitz, 2013; Provan, Fish, & Sydow, 2007). A few researchers have called for the use of a mixed-methods approach to enhance the validity and reliability of network research (Berry et al., 2004; Hollstein, 2011; Isett et al., 2011). Yet relatively few studies reviewed the use of specific methods for network data collection and analysis in the public administration literature. 56317_14-11.indd 83 05/10/15 5:09 pm 84 Qian Hu / Conducting Content Analysis of Documents The studies show that use of quantitative approaches such as survey methods still dominates network research, and the use of qualitative approaches remains limited (Hollstein, 2011). Documents, especially government documents such as emergency management plans, can serve as important guidelines for public management practices. In network research, scholars can use documents as primary data sources or supplementary data sources. In this study my aim was to examine the status and potential of using content analysis of documents in network research. Two research questions were addressed: How did researchers conduct content analysis of documents to collect network data? How can researchers better utilize documents and content analysis to contribute to existing network research? I identified 18 articles in public administration journals that had used content analy- sis of documents as their primary network data collection methods. Through a content analysis of the 18 articles, I found that only few researchers have tapped into new forms of documents such as social media data for network data collection. I also found that interorganizational collaborations was the main focus of these studies. In only few stud- ies, the researchers collected longitudinal network data from documents. With regard to the content analysis process, I found that there was not sufficient information about the coding process of content analysis in existing network research. After presenting the find- ings, I  make suggestions about conducting content analysis of documents for network research. The following section discusses the use of documents and content analysis in public administration, in general, and in network research. The data and methods section details the process of selecting the 18 articles and the analyses conducted. The last section reports the key findings and discusses the advantages and caveats of using content analysis of documents for network research. 2. Content Analysis of Documents in Network Research Both quantitative and qualitative approaches are used in network analyses. Quantita- tive approaches have strengths in investigating large-scale network formation and struc- tures, testing theories, and explaining networking patterns. Qualitative approaches allow for exploring new or understudied network phenomena and observing actual networking behaviors, as well as studying network patterns and processes. Qualitative approaches can also help researchers understand actors’ interpretations and perceptions of networks, and explore behind-the-scenes reasons for the formation of network ties and the dynam- ics of networks (Hollstein, 2011). Qualitative data go beyond numerical network data to provide information about why actors participate in networks and about how they interact with other actors and engage in network activities. Qualitative approaches have strengths in providing contextual information and in assisting the interpretation of quantitative net- work measures. Qualitative approaches can also provide more in-depth information about the “content,” “quality,” “meaning,” and “changes” of ties (Edwards, 2010, p. 18). Some researchers suggest that both approaches should be used in mixed-methods designs. They have also shown that mixed-methods designs allow for data triangulation to improve data, explanatory power, and generalizability (Hollstein, 2011). 56317_14-11.indd 84 05/10/15 5:09 pm Qian Hu / Conducting Content Analysis of Documents 85 There is an array of qualitative research strategies for network data collection and analysis. For instance, researchers can use observations, interviews, and documents to collect network data (Hollstein, 2011). In this study I focused on the use of documents and content analysis for network research. Documents, including government plans and reports, organizational publications, journal articles, newspaper articles, and websites, can provide useful information for researchers to address a variety of questions. Content anal- ysis refers to a technique that systematically analyzes textual data for “making replicable and valid inferences from texts (or other meaningful matter) to the contexts of their use” (Krippendorff, 2012, p. 24). Content analysis is an investigative tool that includes the use of explicitly formulated rules and procedures for the coding process (Bowen & Bowen, 2008). When they conduct content analysis of texts, researchers need to develop their coding schemata by clarifying their research purposes and by defining their coding units, coding themes, coding categories, and coding rules. Many public administration scholars analyzed documents to address a variety of questions (Bowen & Bowen, 2008), and many conducted content analyses of publications to examine key research topics and the evolution of research in the field. For example, Bingham and Bowen (1994) conducted a content analysis of a sample of publications from Public Administration Review from 1940 through 1991 to identify “mainstream” public administration topics. Lan and Anders (2000) conducted a content analysis of 634 articles in eight public administration journals from 1993 through 1995 to examine key subjects, academic backgrounds of public administration scholars, epistemological approaches, data collection and analysis methods, and identifiable paradigms. In addition to examining the development of research in the field, scholars also have analyzed govern- ment documents to understand management practices and policy implementation. Yang and Melitski (2007) analyzed 10 state information technology strategic plans to examine the value orientation embedded in these strategic plans. To evaluate the implementation of public management reform, Moynihan (2006) conducted a systematic content analysis of managing for results documents, including government budgets, strategic plans, and performance reports. Yet, these content analyses were conducted to draw conventional management or policy implementation data, which do not involve interactions between actors or relational data. In network research, documents can also serve as primary sources of information or can complement existing network data. Documents and archival data are relatively easy to access and cost less than a large-scale survey approach for data collection (Hollstein, 2011). The availability of large stores of social media data presents new opportunities for researchers to draw unique network data and to explore new research questions (Xu & Li, 2013). Although public administration has seen an increase in the use of docu- ments and content analyses in network research, their uses remain limited, especially compared with the use of surveys and interviews. The content analyses of documents can help researchers identify lists of participants for conducting surveys or qualitative inter- views, draw network connections between actors, and examine network structures (e.g., Kapucu, 2006; Kapucu & Demiroz, 2011). For instance, an analysis of a situation report 56317_14-11.indd 85 05/10/15 5:09 pm 86 Qian Hu / Conducting Content Analysis of Documents of a disaster response may detail interorganizational interactions in response to a disaster (e.g., Kapucu, 2006). An analysis of an inter-local service delivery report may document an intergovernmental collaboration in delivering public services (e.g., Andrew, 2009). An analysis of an organizational directory may present information about the interlocking boards (Vidovich & Currie, 2012). Despite the well-established need for more qualitative network research and the availability of relevant documents in various formats (Berry et al., 2004; Hollstein, 2011), documents remain an underused source of information for network research. Few articles have evaluated the use of documents and content analysis in network research. To fill this gap, in this study I present a systematic review of the existing network research that used documents and content analysis. I do not only evaluate the quality of network data drawn from documents but also identify the potential and challenges in conducting content analysis of documents for network research. 3. Data and Methods The selection of journals and the search for relevant network articles was a collab- orative effort. To identify the network articles that used content analysis of documents and focused on management problems or policy issues, we searched 39 public adminis- tration journals (For a detailed description of the selection of the journals and the search process, see Kapucu, Hu, and Khosa, 2014). We adopted the list of 39 public administra- tion journals based on Bernick and Krueger’s (2010) and Forrester and Watson’s (1994) comprehensive rankings of public administration journals. The journals we selected pub- lish articles on broad topics in public administration or public policy, as well as those on the subfields of public administration, including public budgeting and finance, public personnel administration, and public organization studies (Forrester & Watson, 1994). The list of the 39 journals is presented in Appendix A. In four steps we identified relevant network research articles that analyze substantive management, policy, and governance issues and excluded those that used networks only as a metaphor. First, we searched the titles, abstracts, and keywords of publications in all the 39 journals published from 1997 to 2012, using words “networks,” “network analysis,” “collaboration,” and “collaborative.” In this first step we found 1,279 articles in relevant journals. Second, we read the abstracts of the 1,279 articles to identify those articles that focused on networks. The list was narrowed down to 677 articles focusing on networks, excluding articles that discussed the broader themes of collaborative governance and col- laboration. Third, we identified those articles that used SNA as part of their methodology. This step yielded a total of 136 articles. The last step was to identify those articles that discussed networks related to the contexts of public administration and public policy. We reviewed the articles carefully to ensure that each article had used SNA as part of its meth- ods and discussed topics that were relevant to the public organizational or policy context. Through the systematic search, we identified a total of 76 articles that used SNA in their methods (Kapucu, Hu, & Khosa, 2014). 56317_14-11.indd 86 05/10/15 5:09 pm Qian Hu / Conducting Content Analysis of Documents 87 To select network articles for further analysis in this article, I read all the 76 articles to identity articles in which the researchers used documents and content analysis in their data collections and analyses. I found 18 articles out of the 76 articles. To be included in this study, documents had to have been used to generate primarily network data. The articles that used documents to generate a list of participants for their primary data col- lection, such as interviews and surveys, were excluded. Articles that analyzed documents to produce supplementary organizational attributes data were also excluded. Table 1 lists the journals and the number of network articles that used documents and content analysis to collect primarily network data. The Journal of Public Research and Theory has three articles, followed by Administration & Society, the American Review of Public Adminis- tration, and the Journal of Urban Affairs, each with two articles. I read each of the 18 articles carefully and coded through an open coding process in combination with pre-established coding themes (Bowen & Bowen, 2008). I examined the types of documents considered in each article, the kinds of network data that researchers drew from the documents, and the analysis of the network data that researchers conducted. I conducted open coding to identify the research topics and the types of documents cov- ered in each article by examining research questions, methods, and key findings. During the open coding process, I paid special attention to the title, abstract, and introduction of each article to identify its research focus. As I read the articles, I created tentative labels that summarized what the research was mainly about. For instance, Article A may have focused on economic development issues, while Article B may have examined emergency management. A diverse range of research foci emerged as the coding process continued. I read the data collection sections of each article to identify the data source and the type of Table 1 List of Journals and the number of network articles Journals Number of network articles that used documents and content analysis Administration & Society 2 The American Review of Public Administration 2 Journal of Management Studies 1 Journal of Urban Affairs 2 Journal of Policy Analysis and Management 1 Journal of Public Research and Theory 3 Nonprofit Management and Leadership 1 Nonprofit and Voluntary Sector Review 1 Organization Studies 1 Public Administration Review 1 Public Performance and Management Review 1 Policy Studies Journal 1 Urban Affairs Review 1 Total 18 56317_14-11.indd 87 05/10/15 5:09 pm 88 Qian Hu / Conducting Content Analysis of Documents network data that it discussed. I organized the coded data and saved them in spreadsheets for further comparisons, categorization, and analyses. I used the categories and concepts from existing network research that were already established (e.g., Borgatti et al., 2013; Isett et al., 2011; Provan et al., 2007) to analyze and reorganize the key topics and document types into different categories. I used Borgatti, Everett, and Johnson’s (2013) book, not only because it was the most frequently cited social network analysis book in public administration, but, more important, their book discussed the types of data sources and methods of network data collection. Following Borgatti and his colleagues’ categorizing of one- and two-mode network data, I also coded the network data in the 18 articles into different categories. Similarly, I examined whether the network data involved multiplex ties, whether the data were longitudinal, and whether mixed-methods were used in data collection. These were the pre-established categories used in the coding process to further understand the network data collected through con- tent analysis of documents. I used Provan, Fish, and Sydow’s (2007) review article to help further examine the management and policy domains that the network data cover. I used their definition of “in- terorganizational networks” as another pre-established category to be differentiated from “policy networks.” Interorganizational networks focus on interorganizational interactions in delivering public services to pursue public values (Isett et al., 2011; Provan et al., 2007), whereas policy networks examine the power relationships between policy makers and the relationships between the structural arrangements of policy networks and policy outcomes (Keast, 2014; Rethemeyer & Hatmaker, 2008). Although the research foci for these two types of networks differ, the two types of networks are related to and influence each other. In addition, I used the evaluation criteria from Krippendorff (2012) as the coding themes to evaluate the quality of existing content analysis in network research. The evalu- ation criteria included whether the articles presented information about coding units, cod- ing themes, coding categories, and other coding rules (Krippendorff, 2012), which helped evaluate the reliability and validity of the content analysis. 4. Findings and Discussion In this section I identify the types of documents that were analyzed in the exist- ing network research, present the research foci of these studies, and discuss the types of network data drawn from these documents. Then, I examine the challenges of drawing network data from documents. 5. Types of Documents Scholars used various kinds of organizational documents to collect network data, as seen in Table 2. Government documents or records served as primary sources of informa- tion in 10 out the 18 articles examined here. In six articles, researchers presented network 56317_14-11.indd 88 05/10/15 5:09 pm Qian Hu / Conducting Content Analysis of Documents 89 Table 2 Types of Documents and Data Drawn from the documents Types of Documents Number of Articles Data Drawn from the Documents Organizational documents Interlocal service delivery reports, Con- tract records, Service Catalog; Joint Ser- vice Database 3 • Contractual arrangements in law enforcement (Andrew, 2009) • Local government collaboration in service delivery (LeRoux & Carr, 2010) • Contractual relationships in public transportation (Ponomariov, et al., 2011) Comprehensive Emergency Manage- ment Plans 6 • Lead agencies according to the emergency manage- ment plan (Choi & Brower, 2006) • Key actors and organizational interactions in the planned emergency management network (Kapucu, 2006; Kapucu et al., 2010; Kapucu et al., 2009; Kapucu & Demiroz, 2011; Kapucu & Van Wart, 2006) After-action reports, Situation re- ports from FEMA 5 • Key actors and organizational interactions in the actual emergency response (Kapucu, 2006; Kapucu et al., 2010; Kapucu et al., 2009; Kapucu & Demiroz, 2011; Kapucu & Van Wart, 2006) Organizational pub- lications, reports, di- rectory, and database, and other documents 6 • Presence of collaborations, formal meetings, strategic alliances, and joint projects among colleges (Akkerma et al., 2011) • Organizational involvement in public health cooperation on the United States-Mexico border (Collins-Dogrul, 2012) • Policy actors’ policy preference about Swiss climate policy (Ingold & Varone, 2012) • Nongovernmental organizations’ network affiliations and other attributes data in Mexico (Neal, 2008) • Interlocking ties among companies in Swiss (Ruigrok, Peck, & Keller, 2006) • Professional membership among Danish mayors (Villadsen, 2011) Newspaper articles New York Times, Other newspapers 6 • Key actors and organizational interactions in the actual emergency response (Kapucu, 2006; Kapucu et al., 2010; Kapucu et al., 2009; Kapucu & Demiroz, 2011; Kapucu & Van Wart, 2006) • Policy actors in Germany during the economic reform between 1990 and 1994 (Raab, 2002) Websites Federal Election Commission website, Company websites 3 • Donation and voting patterns (Jasny, 2012) • Interlocking boards between nonprofit and private organizations in Perth, Australia (Vidovich & Currie, 2012) • Professional membership among Danish mayors (Villadsen, 2011) 56317_14-11.indd 89 05/10/15 5:09 pm 90 Qian Hu / Conducting Content Analysis of Documents information in other types of organizational documents, such as publications, reports, di- rectory information, and databases. The researchers analyzed newspaper articles along with other organizational documents to collect information about interorganizational col- laborations in emergency management. Compared with the use of organizational official documents, there is limited use of other sources of documents such as websites and social media data. In only two articles the researchers analyzed websites to collect network in- formation. Despite the rapid growth of social media data, researchers still rely heavily on the content analysis of conventional forms of documents for network data collection in public administration. 6. Research Foci and Network Data Drawn from Documents In recent years, researchers drew various types of network data from documents in public administration to address a variety of questions. As Table 2 shows, there are vari- ous types of network data drawn from documents. The 18 articles included in this study addressed various management and policy issues, ranging from emergency management to economic development, transportation, education, and public health. A wide range of management and policy domains. As Figure 1 shows, among the 18  articles, 6 (33%) were on emergency management. In the field of emergency management, scholars analyzed governmental documents such as comprehensive emergency management plans (CEMPs) of local governments, after-action reports, and situation reports from the Federal Emergency Management Agency (FEMA) to col- lect network data (e.g., Choi & Brower, 2006; Comfort, 2007; Kapucu, 2006; Kapucu, Arslan, & Collins, 2010; Kapucu, Augustin & Garayev, 2009; Kapucu & Demiroz, 2011; Figure 1. Types of network data. 56317_14-11.indd 90 05/10/15 5:09 pm Qian Hu / Conducting Content Analysis of Documents 91 Kapucu &  Van Wart, 2006). Scholars examined CEMPs to identify lists of primary or and support agencies in the face of emergencies. Furthermore, in the analyses of CEMPs researchers examine the planned network of emergency management and the interorgani- zational interactions defined in the documents (Choi & Brower, 2006). Analysis of after- action reports and situation reports can capture the actual response networks, involving the interactions between organizations during and after disasters (Comfort & Kapucu, 2006; Kapucu, 2006). The analysis of newspaper articles such as those from the New York Times can help capture cross-sector collaboration in emergency management, especially the en- gagement of nongovernmental actors in response to disasters. Scholars examined not only the lead organizations but also the structural characteristics of the actual response networks (Kapucu & Demiroz, 2011). In addition, networks may operate differently in urgent situ- ations than in routine environments. Using content analysis to capture how networks re- spond to external environmental uncertainty can help better measure the performance of networks under stress. The relatively high number of articles on emergency management networks can be attributed partly to the availability of government emergency management plans and other government records, as well as the fast-growing research interest in this topic, especially after the terrorist attacks of September 11, 2001. Interlocal public service delivery and intergovernmental collaboration is another area where scholars applied content analysis to collect data. Scholars analyzed govern- mental service records such as interlocal service delivery reports, contract records, service catalogs, and joint service databases to examine intergovernmental and cross-sector col- laborations in a wide variety of areas (Andrew, 2009; LeRoux & Carr, 2010; Ponomar- iov, Kingsley, & Boardman, 2011). Andrew (2009) examined interlocal service delivery reports to collect information about contractual arrangements in law enforcement among 66 local governments in the Orlando-Kissimmee metropolitan area. He examined the contractual ties in various formats—service agreements, contracts and lease agreements, memoranda of understanding, mutual-aid agreements, and informal letters of agreement. He analyzed the patterns of contractual arrangements over five time periods from 1986 to 2003. LeRoux and Carr (2010) analyzed interlocal government agreements among the 44  local governments of Wayne County, in Michigan, in system maintenance functions (e.g., road and bridge construction and repair) and four lifestyle services (e.g., economic development and public safety). In this study the existence of interlocal government agree- ments was considered as a network tie between local governments. Similarly, Ponomariov, Kingsley, and Boardman (2011) examined the contract records of a transportation agency and analyzed the contracting relationships among a state transportation agency, contrac- tors, and subcontractors in engineering design services between 1992 and 2003. These three articles by Andrew (2009), LeRoux and Carr (2010), and Ponomariov et al. (2011) each collected longitudinal network data about interorganizational collaborations. Scholars used other organizational documents such as organizational reports or directories to draw network data on organizational engagement in interorganizational collab- orations. For instance, Akkerma, Torenvlied, and Schalk (2011) examined the written reports and meeting minutes of colleges to identify the presence of collaborations, formal meetings, strategic alliances, and joint projects among five Danish colleges. Collins-Dogrul (2012) 56317_14-11.indd 91 05/10/15 5:09 pm 92 Qian Hu / Conducting Content Analysis of Documents examined organizational databases, publications, and records to investigate organizational involvement in public health cooperation on the United States-Mexico border. Neal (2008) examined the directory of 173 nonprofit organizations in Oaxaca, Mexico, to identify the nonprofits’ connections with government authorities and other nongovernmental organi- zations and the impact of party politics on the nonprofit organizations’ network connec- tions. This study did not specify the types of interactions that nonprofit organizations have with others. Scholars have analyzed documents to examine the interlocking ties among orga- nizations. Ruigrok, Peck, and Keller (2006) examined the annual reports of 217 Swiss companies to understand the interlocking directorships. Vidovich & Currie (2012) researched interlocking boards between nonprofit and private organizations in Perth, Australia, by examining the websites of nonprofit organizations and companies. To examine Danish mayors’ affiliations with professional organizations, Villadsen (2011) examined professional journals and other sources “including municipality associations, the political parties, municipality homepages, ministries, local media, article databases, and directories of business information” (p. 585). Policy is another important domain where researchers used documents to gener- ate network data. Raab (2002) identified the key policy actors in Germany during the economic reform between 1990 and 1994 by examining newspaper articles and other publications. Ingold and Varone (2012) examined official documents (not specified) to understand the preferences of the actors involved in the formation of Swiss climate policies. Jasny (2012) analyzed the websites of the Federal Election Commission and other documents to examine donation behaviors and voting patterns among congressio- nal representatives. Compared with research on interorganizational collaboration, the research on policy networks is limited. Variety of network data drawn from documents. Through content analyses of documents, researchers can draw various kinds of network data and pull both one-mode network data as well as two-mode network data. One-mode data refers to those datasets that include relational data for the same set of actors, whereas two-mode data involve two different sets of actors. As Table 3 shows, the majority of the 18 articles I analyzed focused on one-mode network data, whereas only seven articles collected two-mode data. Although the types of network data are diverse, the study of interorganizational col- laboration is the focus of many studies. In eleven of the 18 articles the researchers presented interorganizational collaboration data in a wide range of management and policy domains. Researchers examined the existence of collaboration ties and organizational involvement in collaborative efforts. The examination of documents allowed researchers to study interor- ganizational collaborations in different formats, ranging from formal contractual relations to informal collaboration agreements, strategic alliances, and joint projects (Akkerman, Torenvlied, & Schalk, 2011; Andrew, 2009). There are fewer studies on the formation and development of policy networks, which may be partially explained by the unavailability of appropriate policy documents and the challenge of coding policy documents. Selecting the appropriate documents might not be easy, as researchers need to justify the selection of particular policy documents instead 56317_14-11.indd 92 05/10/15 5:09 pm T ab le 3 A n A na ly si s of D at a D ra w n fr om t he D oc um en ts A ut ho rs A re a of F oc us M od es o f D at a C on te nt o f N et w or k M ul ti pl ex it y of N et w or k T ie s L on gi - tu di na l da ta M ix ed -m et ho ds A kk er m a, e t al . ( 20 11 ) In te rc ol le ge n et w or ks an d gr ad ua te s at is fa ct io n O ne - m od e an d T w o- m od e P re se nc e of “ co ll ab or at io ns , f or - m al m ee ti ng s, s tr at eg ic a ll ia nc es , an d jo in t p ro je ct s, ” bo ar d an d co m m it te e m em be rs hi p (p . 6 61 ). Y es N o Y es , c ro ss -v al id at ed w it h in te rv ie w d at a A nd re w ( 20 09 ) In te rl oc al g ov er nm en t co ll ab or at io n in p ub li c sa fe ty O ne -m od e P re se nc e of c on tr ac tu al t ie s in va ri ou s fo rm at s N o Y es N o C ho i & B ro w er , 2 00 6; P la nn ed , p er ce iv ed , a nd ac tu al e m er ge nc y m an - ag em en t ne tw or ks O ne -m od e In te ro rg an iz at io na l in te ra ct io ns de fi ne d on t he e m er ge nc y m an - ag em en t pl an s N o N o Y es , c om pa re d w it h su rv ey d at a C ol li ns -D og ru l (2 01 2) C ro ss -b or de r co op er at io n in p ub li c he al th O ne -m od e “A ss oc ia ti on al a nd b eh av io ra l in te ra ct io n of M ex ic an a nd U S or ga ni za ti on ” (p . 1 01 0) . N o Y es Y es , c om pl em en te d w it h in te rv ie w d at a In go ld & V ar on e (2 01 2) P ol ic y br ok er s, th ei r be ha vi or s, a nd im pa ct on p ol ic y m ak in g w it h th e co nt ex t o f w it hi n th e S w is s cl im at e po li cy T w o- m od e P ol ic y pr ef er en ce s of p ol ic y m ak er s N o Y es Y es , c om pl em en te d w it h in te rv ie w d at a Ja sn y (2 01 2) R el at io ns hi p be tw ee n vo ti ng p at te rn s an d do na ti on T w o- m od e V ot in g be ha vi or s of co ng re ss pe rs on s Y es N o N o K ap uc u (2 00 6) ; K ap uc u et a l. ( 20 10 ); K ap uc u et a l. (2 00 9) ; K ap uc u & D em ir oz (2 01 1) ; K ap uc u & V an W ar t (2 00 6) ; In te ro rg an iz at io na l co l- la bo ra ti on s in e m er ge nc y m an ag em en t O ne -m od e T he p re se nc e of i nt er or ga ni - za ti on al i nt er ac ti on s in t he pl an ne d ne tw or ks , a nd t he pr es en ce o f in te ro rg an iz at io na l co ll ab or at io ns i n re sp on se t o di sa st er s N o Y es N o 93 (C o n ti n u es ) 56317_14-11.indd 93 05/10/15 5:09 pm A ut ho rs A re a of F oc us M od es o f D at a C on te nt o f N et w or k M ul ti pl ex it y of N et w or k T ie s L on gi - tu di na l da ta M ix ed -m et ho ds L eR ou x & C ar r (2 01 0) In te rl oc al g ov er nm en t co ll ab or at io n in p ub li c se rv ic e de li ve ry O ne -m od e T he p re se nc e of c ol la bo ra ti on i n ei gh t se rv ic e ar ea s Y es Y es Y es , c om pl em en te d w it h in te rv ie w d at a N ea l (2 00 8) N on pr ofi ts ’ co nn ec ti on s w it h go ve rn m en t au th or i- ti es a nd o th er n on go ve rn - m en ta l or ga ni za ti on s O ne -m od e T he p re se nc e of c ol la bo ra ti on o n pr oj ec ts , m em be rs hi p, i nv ol ve - m en t in s oc ia l m ov em en t, a nd di al og ue . Y es N o N o P on om ar io v et a l. ( 20 11 ) C on tr ac ti ng r el at io ns hi ps in t ra ns po rt at io n O ne -m od e T he e xi st en ce o f co nt ra ct ua l re la ti on sh ip s be tw ee n a go v- er nm en t ag en cy a nd o th er or ga ni za ti on s N o Y es N o R aa b (2 00 2) P ol ic y ne tw or ks i n G er m an y du ri ng t he e co - no m ic r ef or m T w o- m od e K ey p ol ic y ac to rs N o Y es Y es , c om pl em en te d w it h in te rv ie w d at a R ui gr ok e t al ., 20 06 In te rl oc ki ng d ir ec to rs hi ps in S w is s co m pa ni es a nd bo ar ds ’ st ra te gi c de ci si on m ak in g T w o- m od e B oa rd m em be rs ’ af fi li at io n w it h or ga ni za ti on s N o N o Y es , C om pl em en te d w it h su rv ey d at a V id ov ic h & C ur ri e (2 01 2) In te rl oc ki ng d ir ec to rs hi ps T w o- m od e B oa rd m em be rs ’ af fi li at io n w it h or ga ni za ti on s N o N o N o V il la ds en , 2 01 1 D an is h m ay or s’ p ro fe s- si on al m em be rs hi p T w o- m od e M ay er ’s a ffi li at io n w it h pr of es - si on al o rg an iz at io ns N o N o Y es , C om pl em en te d w it h su rv ey d at a 94 T ab le 3 A n A na ly si s of D at a D ra w n fr om t he D oc um en ts ( C o n ti n u ed ) 56317_14-11.indd 94 05/10/15 5:09 pm of others. Furthermore, unlike emergency management plans and interlocal government service agreements, thorough content analysis of documents related to the formation of policy networks may be more complicated. The policy documents can include a large number of policy stakeholders and involve intricate interactions among policy actors, which may require a labor-intensive coding process in order to draw useful network data. Longitudinal network data. Compared with survey methods and qualitative in- terviews, content analysis of documents has strengths in producing longitudinal network data at a reasonable cost, which allows for the examination of the changes in network ties. In Table 4 the strengths and weaknesses of content analysis and survey methods are compared. Collecting longitudinal network data through surveys and interviews can be expensive and time consuming. During a survey study or an interview, respondents may face challenges recalling the history of an event or a collaborative experience. Moreover, it might be difficult to ensure the participation of the same participants in a longitudinal survey or interview study (Marsden, 2011). In contrast, researchers can take advantage of archival data and pull network data for needed time periods. Researchers have collected longitudinal network ties among organizations by examining organizational documents for certain periods of time (e.g., Andrew, 2009; LeRoux & Carr, 2010; Ponomariov et al., 2011). For instance, by examining the interlocal service delivery agreements or reports, researchers can pull interlocal collaboration data from each time period and merge the data into panel network data (Andrew, 2009). Although seven of the 18 articles considered here collected longitudinal data, most of these articles aggregated the data at different points of time, and only few conducted network analysis to examine the change of network ties. Qualitative and quantitative analysis. Data drawn from documents can be ana- lyzed using both quantitative and qualitative analysis methods. The data allow researchers to conduct statistical network analysis. The data can also provide rich contextual infor- mation and the meaning and content of the relational data captured in the documents (Edwards, 2010), though the data are not directly reported by the network participants. Researchers can apply descriptive network measures such as degree centrality and net- work density to identify the key actors and examine the network structural characteristics Table 4 A Comparison between Content Analysis of Documents and Survey Methods Content Analysis of documents Survey Methods Strengths • Provide information about the contexts, the content, quality, and meaning of network ties (Edwards, 2010) • Easy to access and cost less (Hollstein, 2011). • Longitudinal data (Edwards, 2010) • Draw social media data (Groenewegen & Moser, 2014) • Large-scale datasets, and whole-networks (Marsden, 2011) • Easy to analyze (Marsden, 2011) • Standard process and easy to be imple- mented (Marsden, 2011) • Address a variety of network questions (such as intensity of collaborations) in an ef- ficient manner Weakness • Subjective coding process (Edwards, 2010) • Generalizability issues (Edwards, 2010) • Concern about response rates • Lack of contexts (Marsden, 2011) • Attrition for longitudinal study 56317_14-11.indd 95 05/10/15 5:09 pm 96 Qian Hu / Conducting Content Analysis of Documents (e.g., Kapucu & Demiroz,  2011). Researchers can also apply sophisticated statistical models such as actor-oriented models and quadratic assignment procedure regression to understand the collaboration patterns among local governments in public safety (e.g., Andrew, 2009; LeRoux & Carr, 2010). Using qualitative analysis methods, researchers can focus on describing the context, the meaning of network ties, and the process of net- work formation. For instance, after reviewing newspaper articles and other documents, Raab (2002) described the Germany economic reform between 1990 and 1994 and ex- plained the involvement of key policy actors in great detail. Or, researchers can combine both a quantitative analysis approach with a qualitative approach to analyze network data to provide a quantitative description of the actors and the network as well as detailed contextual background information. Most of the existing research has focused more on converting data to numerical forms for quantitative analysis and could take better advan- tage of qualitative data by providing more contextual information and by interpreting the content or meaning of network ties, as Table 4 suggests. 7. Challenges of Drawing Network Data from Documents Applying content analysis to drawing net data from documents can be as challenging as using other quantitative approaches, depending the quantity and types of documents that researchers need to analyze. The meaning of network ties may be not accurately cap- tured through the documentation process, which may lead to differences between what the documents suggest and what the network participants actually mean or would interpret. With survey or documents data, researchers still need to carefully address potential cog- nitive and perceptual biases of third-party interpreters. Furthermore, without systematic coding rules and procedures, the qualitative approach to collecting and analyzing data, including conducting content analysis of documents, may face critiques about its reliabil- ity and generalizability (Bowen & Bowen, 2008). The following will identify some of the areas that deserve more attention to improve the use of documents and content analysis in network research, based on the analysis of 18 instances of network scholarship in public administration. Document sources. Among the 18 articles included in this study, a few did not spec- ify the types of documents that the authors had used to draw their data, which makes it difficult to examine the validity of using the documents for particular research topics and makes it extremely challenging to replicate the study in other contexts. In addition, the majority of the 18 articles used either government documents or other types of traditional publications, including organizational directories, records, and meeting minutes. Few ar- ticles used information from websites, let alone social media data. Yet, many scholars believe that the era of big data presents both “theoretical and methodological opportuni- ties” for network researchers (Groenewegen & Moser, 2014, p. 463). Social media data can be timely and suitable for social network research, as social media data, by nature, involve relational data. On the other hand, the formation and development of network ties in the online community may differ from their counterparts in offline contexts. This 56317_14-11.indd 96 05/10/15 5:09 pm Qian Hu / Conducting Content Analysis of Documents 97 presents new opportunities to test whether existing theories are still valid in explaining the formation and development of online network ties (Groenewegen & Moser, 2014). Social media data, however, exist in vast quantities and can be rife with unwanted information. Computer-aided qualitative data analysis software packages (including various software tools such as NVivo, Atlas ti, and QDA), known as CAQDAS, may become necessary for analyzing the large volume of online data. To take advantage of voluminous amounts of text data, including social media data on the Internet, researchers may need to learn new tools such as NodeXL and Python programming language for data-mining purposes. Coding Process. Most of the 18 articles examined in this study did not discuss the coding process they had used in detail. There is insufficient information in these articles about the coding units, coding themes, coding categories, and other coding rules. The authors do not provide enough information about whether they coded the documents by words, phrases, sentences, paragraphs, or whole texts; neither do they provide information about whether the researchers established their coding schema before or during coding documents. Moreover, few authors mention whether they conducted coding manually or through some computer software. They do not mention r whether the coding was finished by one author or by collaborative efforts either. Even fewer authors reported any inter- coder reliability results. All of these factors may make it difficult for researchers with the same research interests to validate the results or replicate the studies in other contexts. To improve the reliability of the coding process and to strengthen the validity of the study designs, researchers may consider providing more explicit coding rules and procedures for closed-ended content analysis in the text or the appendices. For open-ended coding processes, researchers may provide explicit information about how the open-coding was conducted so that other researchers may better understand the process and replicate the process in similar studies. Furthermore, the availability of a variety of network analysis tools may help researchers reduce labor needed for manual coding. Pfeffer and Carley (2012) applied Rapid Ethnographic Assessment (REA), which originated in anthropology, to social network analysis. REA can be used to “quickly create a socio-cultural profile of a group or region,” using “semi-automated computer-based text-mining and machine learning techniques” (Carley, Bigrigg, & Diallo, 2012, p. 330). Software such as AutoMap can extract network information from unstructured documents through text preprocessing, generating thesauri, formulating linkage rules, and revealing the system structure of texts (Diesner & Carley, 2005). For instance, Morçöl, Vasavada, and Kim (2013) used AutoMap to analyze 354 newspaper articles published between 1990 and 2009 to examine the key policy players in an urban governance network. Network data extracted using AutoMap can also be analyzed by other network analysis tools such as Organization Risk Analyzer (ORA) and UCINET. Quality of the network data. Content analysis of the documents in the 18 articles produced relational data, most of which focused on the existence of interorganiza- tional collaboration ties or affiliation ties. In seven out of the 18 articles the research- ers collected longitudinal network data. A few researchers delved into the network ties and provided more information about the meaning or content of the network ties. 56317_14-11.indd 97 05/10/15 5:09 pm 98 Qian Hu / Conducting Content Analysis of Documents For instance, Andrew (2009) examined the formation of collaboration ties to provide 14  kinds of public services related to public safety. And the format of collaboration ties is not limited to formal contractual ties. This level of detail on the intergovernmen- tal collaboration provides useful contextual information for the network analysis. This type of data would be difficult to collect using quantitative methods such as a survey design. Document data have more potential to be explored to enrich the conversation about the content and quality of network ties. More diverse types of network data such as semantic network data (a network of concepts) and metanetworks data can be ex- tracted from documents (Diesner & Carley, 2005; van Atteveldt, 2012). Furthermore, with the advancement of computer software, archival data could be better utilized for collecting panel network data. In addition, in eight studies the researchers used mixed-methods designs to collect network data, as shown in Table 3. Most of the eight articles used other methods such as surveys or interviews to collect different kinds of data and to complement the network data drawn from documents. Only two articles actually used the interview data or survey data to cross-validate or compare with the network data from documents (e.g., Akkerma et al., 2011; Choi & Brower, 2006). To take advantage of a mixed-methods design, researchers need to better integrate content analysis with other methods. 8. Conclusion This article presents an examination of the use of documents and content analysis in existing network research in the field of public administration. The number of studies using documents and content analysis to draw network data is relatively small. Yet the content analysis of documents can not only produce quantitative network data but also provide unique contextual information about the formation of network ties. Furthermore, documents, especially archival data, can allow researchers to explore the change of net- work ties. Scholars have used documents to extract interorganizational collaboration data and affiliation data and to identify the key policy actors and their policy preferences. More diverse types of network data can be drawn from documents, such as longitudinal network data, to examine the development or evolution of network ties. Most of the existing re- search used formal organizational documents or newspaper data, while social media data remain underexplored. To strengthen the validity and the reliability of the qualitative ap- proach to analyze documents, future researchers may need to provide more explicit coding procedures and rules. This study has some limitations. The results are preliminary and descriptive based on the analysis of 18 network articles. It may raise more questions than it has answered. As the number of studies using documents as the primary source of information con- tinues to grow, a more comprehensive review of qualitative network research will be needed. 56317_14-11.indd 98 05/10/15 5:09 pm Qian Hu / Conducting Content Analysis of Documents 99 References Akkerman, A., Torenvlied, R., & Schalk, J. (2011). Two-level effects of interorganizational network collabora- tion on graduate satisfaction: A comparison of five intercollege networks in Dutch higher education. The American Review of Public Administration, doi: 10.1177/0275074011425418. Andrew, S. A. (2009). Regional integration through contracting networks: An empirical analysis of institu- tional collection action framework. Urban Affairs Review, 44(3), 378–402. Bernick, E., & Krueger, S. (2010). An assessment of journal quality in public administration. International Journal of Public Administration, 33(2), 98–106. Berry, F. S., Brower, R. S., Choi, S. O., Goa, W. X., Jang, H. S., Kwon, M., & Word, J. (2004). Three traditions of network research: What the public management research agenda can learn from other research commu- nities. Public Administration Review, 64(5), 539–552. Bingham, R. D., & Bowen, W. M. (1994). “Mainstream” public administration over time: A topical content analysis of public administration. Public Administration Review, 54(2), 204–208. Borgatti, S. P., Everett, M. G., & Johnson, J. C. (2013). Analyzing social networks. Los Angeles, CA: Sage. Bowen, C. C., & Bowen, W. M. (2008). Content analysis. In G. J. Miller & K. Yang (Eds.). Handbook of re- search methods in public administration (2nd ed., pp. 689–704). Boca Raton, FL: CRC Press. Carley, K. M., Bigrigg, M. W., & Diallo, B. (2012). Data-to-model: A mixed initiative approach for rapid eth- nographic assessment. Computational and Mathematical Organization Theory, 18(3), 300–327. Choi, S. O., & Brower, R. S. (2006). When practice matters more than government plans: A network analysis of local emergency management. Administration and Society, 37(6), 651–678. Collins-Dogrul, J. (2012). Tertius iungens brokerage and transnational intersectoral cooperation. Organization Studies, 33(8), 989–1014. Comfort, L. K. (2007). Crisis management in hindsight: Cognition, communication, coordination, and control. Public Administration Review, 67(1S), 189–197. Comfort, L. K., & Kapucu, N. (2006). Inter-organizational coordination in extreme events: The World Trade Center Attack, September 11, 2001. Natural Hazards, 39(2), 309–327. Diesner, J., & Carley, K. M. (2005). Revealing social structure from texts: Meta-matrix text analysis as a novel method for network text analysis. In V. K. Narayanan & D. J. Armstrong (Eds.), Causal mapping for research in information technology (pp. 81–108). Harrisburg, PA: IGI Global. Edwards, G. (2010). Mixed-methods approach to social network research. Retrieved from http://eprints.ncrm. ac.uk/842/1/Social_Network_analysis_Edwards.pdf Forrester, J. P., & Watson, S. S. (1994). An assessment of public administration journals. Public Administra- tion Review, 54(5), 474–482. Groenewegen, P., & Moser, C. (2014). Online communities: Challenges and opportunities for social network research. In D. J. Brass, G. Labianca, A. Mehra, D. S. Halgin, & S. P. Borgatti, (Eds.), Research in the Sociology of Organizations (Vol. 40). Bradford, UK: Emerald Publishing. Hollstein, B. (2011). Qualitative approaches. In J. Scott & P. J. Carrington (Eds.). The SAGE Handbook of Social network Analysis (pp. 404–416).Thousand Oaks, CA: Sage. Ingold, K., & Varone, F. (2012). Treating policy brokers seriously: Evidence from the climate policy. Journal of Public Administration Research and Theory, 22(2), 319–346. Isett, K. R., Mergel, I. A., LeRoux, K., Mischen, P. A., & Rethemeyer, R. K. (2011). Networks in public ad- ministration scholarship: Understanding where we are and where we need to go. Journal of Public Admin- istration Research and Theory, 21(s1), i157–i173. Jasny, L. (2012). Baseline models for two-mode social network data. Policy Studies Journal, 40(3), 458–491. Kapucu, N. (2006). Interagency communication networks during emergencies boundary spanners in multia- gency coordination. The American Review of Public Administration, 36(2), 207–225. Kapucu, N., & Demiroz, F. (2011). Measuring performance for collaborative public management using net- work analysis methods and tools. Public Performance and Management Review, 34(4), 551–581. 56317_14-11.indd 99 05/10/15 5:09 pm 100 Qian Hu / Conducting Content Analysis of Documents Kapucu, N., Arslan, T., & Collins, M. L. (2010). Examining intergovernmental and interorganizational response to catastrophic disasters toward a network-centered approach. Administration & Society, 42(2), 222–247. Kapucu, N., Augustin, M. A., & Garayev, V. (2009). Interstate partnerships in emergency management: Emer- gency management assistance compact in response to catastrophic disasters. Public Administration Re- view, 69(2), 297–313. Kapucu, N., Hu, Q., & Khosa, S. (2014). The state of network research in public administration. Administra- tion & Society. doi:10.1177/0095399714555752 Kapucu, N., & Van Wart, M. (2006). The evolving role of the public sector in managing catastrophic disasters: Lessons learned. Administration & Society, 38(3), 279–308. Keast, R. (2014). Network theory tracks and trajectories: Where from, where to? In R. Keast, M. Mandell, & R. Agranoff (Eds.). Network theory in the public sector: building new theoretical frameworks (pp. 15–30). New York, NY: Routledge. Krippendorff, K. (2012). Content analysis: An introduction to its methodology (3rd ed.). Thousand Oaks, CA: Sage Publications, Inc. Lan, Z., & Anders, K. K. (2000). A paradigmatic view of contemporary public administration research an empirical test. Administration & society, 32(2), 138–165. Lecy, J. D., Mergel, I. A., & Schmitz, H. P. (2013). Networks in public administration. Public Management Review, 15(1), 1–23. LeRoux, K., & Carr, J. B. (2010). Prospects for centralizing services in an urban county: Evidence from eight self-organized networks of local public services. Journal of Urban Affairs, 32(4), 449–470. Marsden, P. V. (2011). Survey methods for network research. In J. Scott & P. J. Carrington (Eds.). The SAGE Handbook of Social network Analysis (pp. 370–388). Thousand Oaks, CA: Sage. Morçöl, G., Vasavada, T., & Kim, S. (2013). Business improvement districts in urban governance: A longitu- dinal case study. doi:10.1177/0095399712473985 Moynihan, D. P. (2006). Managing for results in state governments: Evaluating a decade of reform. Public Administration Review, 66(1), 77–89. Neal, R. (2008). The importance of the state: Political dimensions of a nonprofit network in Oaxaca, Mexico. Nonprofit and Voluntary Sector Quarterly, 37(3), 492–511. Pfeffer, J., & Carley, K. M. (2012). Rapid modeling and analyzing networks extracted from pre-structured news articles. Computational and Mathematical Organization Theory, 18(3), 280–299. Ponomariov, B., Kingsley, G., & Boardman, C. (2011). Searching for contracting patterns over time: Do prime contractor and subcontractor relations follow similar patterns for professional services provision? Journal of Policy Analysis and Management, 30(3), 581–597. Provan, K. G., Fish, A., & Sydow, J. (2007). Interorganizational networks at the network level: A review of the empirical literature on whole networks. Journal of Management, 33(3), 479–516. Provan, K. G., & Lemaire, R. H. (2012). Core concepts and key ideas for understanding public sector orga- nizational networks: Using research to inform scholarship and practice. Public Administration Review, 72(5), 638–648. Raab, J. (2002). Where do policy networks come from? Journal of Public Administration Research and The- ory, 12(4), 581–622. Rethemeyer, R. K., & Hatmaker, D. M. (2008). Network management reconsidered: An inquiry into manage- ment of network structures in public sector service provision. Journal of Public Administration Research and Theory, 18(4), 617–647. Ruigrok, W., Peck, S. I., & Keller, H. (2006). Board characteristics and involvement in strategic decision mak- ing: Evidence from Swiss companies. Journal of Management Studies, 43(5), 1201–1226. Scott, J. (2013). Social network analysis. (3rd ed.). Los Angeles, CA: Sage. van Atteveldt, W. (2012). Semantic network analysis techniques for extracting, representing, and query- ingmMedia content. Retrieved from https://www.cs.vu.nl/en/Images/WHvanAtteveldt14-10-2008_ tcm75-259647.pdf 56317_14-11.indd 100 05/10/15 5:09 pm Qian Hu / Conducting Content Analysis of Documents 101 Vidovich, L., & Currie, J. (2012). Governance networks: Interlocking directorships of corporate and nonprofit boards. Nonprofit Management and Leadership, 22(4), 507–523. Villadsen, A. R. (2011). Structural embeddedness of political top executives as explanation of policy isomor- phism. Journal of Public Administration Research and Theory, 21(4), 573–599. Wasserman, S., & Faust, K. (1994). Social network analysis: Methods and applications. Cambridge: Cam- bridge University Press. Xu, G., & Li, L. (Eds.). (2013). Social media mining and social network analysis: Emerging research. Her- shey, PA: IGI Global/Information Science Reference. Yang, K., & Melitski, J. (2007). Competing and complementary values in information technology strategic planning: Observations from ten states. Public Performance and Management Review, 30(3), 426–452. 56317_14-11.indd 101 05/10/15 5:09 pm 102 Qian Hu / Conducting Content Analysis of Documents Appendix A List of 39 Public Administration Journals Administration & Society The American Review of Public Administration Australian Journal of Public Administration Canadian Public Administration Evaluation Review Financial Accountability and Management Human Relations International Journal of Public Administration International Review of Administrative Sciences Journal of Accounting and Public Policy Journal of Health Politics, Policy, and Law Journal of Management Studies Journal of Policy Analysis &Management Journal of Public Administration Research &Theory Journal of Public Budgeting, Accounting and Financial Man- agement / Public Budgeting and Financial Management Journal of Public Policy Journal of Urban Affairs Municipal Finance Journal National Tax Journal Nonprofit Management & Leadership Nonprofit & Voluntary Sector Quarterly Organization Studies Policy Sciences Policy Studies Journal Policy Studies Review /Review of Policy Research Political Psychology Political Science Quarterly Public Administration & Development Public Administration: An International Quarterly Public Administration Quarterly Public Administration Review Public Budgeting and Finance Public Finance Quarterly / Public Finance Review Public Performance and Management Review Publius Review of Public Personnel Administration Social Science Quarterly State & Local Government Review Urban Affairs Review 56317_14-11.indd 102 05/10/15 5:09 pm