The U.S. Public Use Census Microdata Files as a Source for the Study of Long-Term Social Change by Steven Ruggles ' Department of History University of Minnesota The United States public use microdata samples are machine-readable hierarchical files consisting of individ- ual-level and household-level records drawn from the federal decennial censuses. Samples covering nine census years between 1880 and 1990 are currently available or in preparation. Taken together, these microdata comprise the richest source of quantitative information on long-term changes in the American population. Because these samples were created at different times by different investigators, however, they have incompatible documentation and a wide variety of record layouts and coding schemes. These differences among the samples inhibit their use as a time-series. At the Social History Research Laboratory of the Univer- sity of Minnesota, we are planning to convert the series of public use samples into a single coherent form. The success of this project will depend on the usefulness of the data scries to a broad range of social scientists. This essay describes the history of the public use samples and some of their potential applications for time-scries analysis, in the hope of stimulating interest and sugges- tions at an early stage of our work. Background Social scientists have increasingly recognized the need to study society as a process. If we confine our analyses to the stale of society at a single moment, we cannot hope to understand the sources of social change. Sociologists, economists and demographers have developed a variety of quantitative data sources to study social change, including retrospective surveys, repetitions of early social surveys, and longitudinal surveys. Although such data sources are essential, they are usually limited to the analysis of changes during the past thirty years. The study of longer term change— over the past 100 or 150 years— has been sharply constrained by the limited availability of consistent data series. Analysts of nine- teenth-century society have often turned to institutional and bureaucratic records, such as those generated by churches and the military, but these sources are typically available only for the distant past and they are limited to the study of specific population subgroups. The decennial census is the most consistent general source of information about the American population over the past two centuries. Quantitative studies of long- term social change have always relied on the published tabulations of the census, but these data have substantial limitations. In each period, the topics addressed by census publications have focussed on contemporary concerns, and these concerns have shifted dramatically over the past century. For example, the early twentieth century census volumes include a wealth of data on immigrants, but virtually nothing on family composition. Moreover, the high costs of tabulation before the intro- duction of modem data processing equipment meant that few cross-classifications of census data were possible, and much of the information collected by the census was never tabulated at all. Even for recent census years, the published census volumes have significant limitations for the study of social change. Despite the dramatic increase in the quantity of published census data in recent years, the census Bureau cannot anticipate all the questions social scientists want to ask. The Census Bureau has addressed these problems by producing individual-level public use samples of the census (U.S. Bureau of the Census 1972, 1973, 1982a, 1989). The first public use sample was created as a byproduct of the 1960 census (U.S. Bureau of the Census 1954). In an effort to meet the needs of scholars who needed specialized tabulations, the Census Bureau created a 1 in ICKX) extract of the basic data tapes they had used to create tabulations for the published census volumes. To preserve confidentiality, the Census Bureau removed names, addresses, and other potentially identi- fying information. The 1960 public use sample was an immediate success. Not only did it allow researchers to make tabulations tailored to their specific research questions, but it also allowed them to apply new methods to the analysis of census data, especially multivariate techniques. But the sample did have two significant limitations. First, the sample size was relatively small. The 1 in 1000 sample density yielded about 180,(XX) person records. Given the modest capacity of computers in 1964, this was a lot of cases, but as researchers began to use the sample for detailed analysis of small population subgroups, its limitations became apparent. Second, the 1960 public use sample provided highly limited geographic informa- tion. In its zeal to preserve confidentiality, the Census Bureau stripped off all information on places below the lASSIST Quarterty state level. This meant, for example, that it was impos- sible to extract a subsample of the New York City population. Both of these problems were addressed by the 1970 public use samples. The 1 in 1000 density of the 1950 sample was increased dramatically; the Census Bureau provided six independent public use samples for 1970, each of which had a 1 in 100 density. Users who re- quired an exceptionally large number of caaes could combine the samples to obtain a six percent density, or about 12 miUion person records. In addition, the 1970 samples provided a variety of alternate geographic codes, although the Census Bureau still did not identify any places of less than 250,000 population. In conjunction with the 1970 public use samples, the Census Bureau released a new version of the 1960 public use sample. They enlarged the sample density from 1 in 1000 to 1 in 100, and at the same time reorganized the coding schemes and record layouts to be compatible with the samples from 1970. This compatibility made it relatively easy for investigators to pool data from 1950 and 1970, and thus incorporate change into their analy- ses. By the late 1970s, the public use samples had become one of the essential tools of American social scientists. It was in this climate that two separate teams of researchers independently came up with the idea of creating histori- cal public use samples for earlier census years. Samuel Preston directed projects at the University of Washington and the University of Pennsylvania to produce a l-in-750 sample of the 1900 census and a l-in-250 sample of the 1910 census (Graham, 1980; Strong et al., 1989). Meanwhile, Halliman Winsborough and a group of others at the University of Wisconsin and the Census Bureau created 1 in 100 samples for the censuses of 1940 and 1950 (U.S. Bureau of the Census 1984a, 1984b). A fifth historical public use sample is now underway. At the University of Minnesota, we are creating a 1 in 100 sample of the 1880 census. That project is about half done, and a preliminary 1 in 1000 subsample is already available (Ruggles and Menard, 1990; Social History Research Laboratory, 1990). In addition, we have applied for funds to create a public use sample of the 1920 census; if that project is funded, the 1920 sample will be complete by 1997. In the meantime, the Census Bureau has released public use samples for the 1980 census, and has scheduled a 1993 release date for samples of the 1990 census (U.S. Bureau of the Census, 1982a, 1989). These samples include greater geographic and subject content detail than either the 1960 or 1970 public use samples. What all this means is that we can anticipate a series of public use microdata samples of the u.s. census covering the years 1880, 1900, 1910, 1920, 1940, 1950, 1950, 1970, 1980 and 1990. This data series will constitute a resource of unprecedented power for the study of long- term social change. The availability of the historical census files is especially important, because few national microdata files of any sort exist for the period before 1960. Furthermore, as one goes farther back in time the published tabulations of the census become increasingly sketchy and the problems of comparability increase. Table 1 summarizes the availability of variables for each of the census years ciurently available or in preparation. Eleven basic questions were asked in all census years, and twenty-two inquiries are available for at least seven of the nine census years. There are a significant number of variables omitted from Table 1 that are available in only one or two census years. Note that in addition to the differences in available variables across census years, there are also multiple versions of the samples for recent years that incorporate slightly differing variables. A detailed discussion of comparability problems can be found in Ruggles (1991). Applications of the Public Use Microdata series The range of potential topics that can be addressed with these data is far too great to describe within the page limitations of this paper. The following paragraphs are intended only to suggest some of the most obvious topics of investigation. 1) Household Composition. American living arrange- ments have been radically transformed since the late nineteenth century. In 1880, for example, 77 percent of the elderly lived with their children or with extended kin, compared with 24 percent in 1980. The frequency of primary individuals has increased about eight-fold, and residence as secondary individuals or extended kin has dropped almost as dramatically. These changes began shortly after the turn of the century, and accelerated after 1940 (Ruggles 198B; Ruggles and King, forthcoming). We are only beginning to understand the dimensions of change in family structure over the past century, and the analysis of the determinants of that transformation has yet to be seriously undertaken. The public use samples are the only detailed national source of information about changing living arrangements in the nineteenth century and first half of the twentieth century. All the public use samples provide sufficient information to construct fully compatible and highly detailed measures of household composition and family interrelationships. 2) Fertility. Between 1850 and 1940 the total fertility rate for While Americans declined from about 5.4 to 2.2 (Coale and zelnik 1963:36). Research on early fertility Summer 1991 trends in America has relied for the most part on child- woman ratios (Forsterand Tucker 1972; Yasuba 1953) and backward projections of age distributions in the published census volumes (coale and zelnick 1953; McClellan and Zeckhauser 1982). Neither of these techniques allows close analysis of marital fertility or fertility differentials. Analyses of fertility using own- child techniques were among the earliest and most fruitful multi-sample studies carried out with the two original pubUc use samples produced by the Census Bureau (e.g. Rindfuss and sweet 1977). The public use microdata series will permit study of differential marital fertility patterns over the period of greatest fertility decline, comparing characteristics such as race, occupa- tional class, region, literacy, size of locality, family structure, and a wide variety of other variables. The richness of these data will greatly enhance our ability to analyze the determinants of early fertility decline in a developed country, and this may in turn lend insight into the onset of fertility control in developing countries. 3) Life Course Analysis. Long-term changes in the timing of major life-course transitions— such as leaving school, leaving home, starting work, marrying, and establishing a separate household— have been studied using both cross-sectional data (ModcU, Furstenberg, and Hershberg 1975) and retrospective survey data (Hogan 1981). Both approaches reveal that American society has become more age-graded during the twentieth century: people tend to pass through the major transitions to adulthood at increasingly prescribed ages and in an increasingly prescribed sequence. Recently, Stevens (1991) suggested that the heterogeneity of the early decades of the twentieth century was a short-term phenomenon brought about by rapid urbanization and immigration from Southern and Eastern Europe. The public use microdata series will provide the opportunity to test this hypothesis through cohort analysis of both the timing of change and of differences among subpopula- tions. 4) Household Economy and Female Labor Force Partici- pation. Much of the research on late nineteenth and early twentieth century social structure has focussed on patterns of employment within the household. Some investigators see a fundamental transformation of the household economy with the rise of wage labor; others fxiint to the continued strength of preindustrial modes of informal family labor (Katz et. al. 1982; Anderson 1971; Barron 1984). Since the existing studies are based on small local samples of census data, regional variation may explain much of difference in interpretation. The hierarchical organization of the proposed census scries is well suited to study of the household economy. Female labor force participation is a closely related and equally controversial issue (Bose 1987; conk 1981; Folbre and Abel 1989: Goldin 1980, 1983; Openheimer 1970; Jaffe 1955). Changes in census definitions of employment and labor force participation have compli- cated such analysis. The public use microdata series will allow researchers to minimize the effects of such changes, since labor force participation can be allocated according to the procedures proposed by Abel and Folbre (1990); such adjustments are impossible with aggregate data. Analysis of the determinants of female labor force participation and child labor during the late nineteenth and twentieth centuries should prove especially reveal- ing. 5) Ethnicity and Immigration. The questions on nativity in the public use samples makes them a rich lode of information for immigration historians. Throughout the period 1880-1970 the census asked about parental birthplaces as well as the respondent's birthplace. Most of the census years also provide information on mother tongue and year of immigration. This makes it possible to analyze patterns of acculturation for a wide variety of cultural groups. Understanding the varied experience of immigrants in the late nineteenth and early twentieth centuries has taken on a special relevance in light of the recent resurgence of immigration. These topics are intended only as representative ex- amples of the sort of research that can be carried out with the public use microdata series. Other key areas of investigation include the transformation of industrial and occupational structure, urbanization, internal migration, nuptiality, and education. The large size of the public use samples increases their versatility by permitting analysis of small population subgroups. Consider, for example, some of the topics addressed by Minnesota graduate students using the historical public use .samples: -the professional ization of nursing - American Indian fertility patterns -race differentials in the living arrangements of the elderly - labor force composition in Minneapolis and St Paul - the adaptation of Scandinavian immigrants -changes in the gender composition of clerical workers -the household structure of early black migrants to lASSIST Quarterly Northern cities -Italian immigration to the Southern U.S. -living arrangements of parentless children These research topics could not be pursued using a general social survey of the scale ordinarily undertaken by academic social scientists. Indeed, even the largest social survey carried out by the government— the Current Population survey — is too small for the detailed analysis of topics like American Indian fertility or the professionalization of nursing. The pubhc use samples are the only general source af microdata with sufficient cases to study such small population subgroups. The large scale of the public use samples also makes them the most suitable source of microdata for policy analysis at the state and local levels. Policy analysts have traditionally focussed on short-run change, but there is increasing recognition of the need to distinguish long- term secular trends from temporary fluctuations. The public use samples also allow policy analysts to set their investigations of State and local conditions in a compara- tive national context In summary, the decennial enumerations of the popula- tion include a great deal of information on demography and socioeconomic structure that can only be taken advantage of through the public use samples. We presently understand just the broad outlines of the social transformation that has taken place since the late nine- teenth century; pubhshed sources provide only limited information on topics such as fertility behavior, urbaniza- tion, immigration, household composition, and occupa- tional structure. The public use microdata series allows the construction of comparable cross-tabulations on a wide range of topics that were not covered by census publications or were incompletely tabulated. Perhaps even more important is the potential for pooled multi- variate analyses opened up by the availability of micro- data, used in combination, the nine data sets spanning a century of cataclysmic social and economic change will comprise our most important resource for the study of changing social structure. Integration of the Public use Microdata Series Despite the enormous potential for time-series analysis of the public use samples, to date only a small proportion of the research based on these data has fully exploited the potential for the study of change over time. Many investigators are using the samples as isolated cross- sections. A preliminary bibliography of recent research using the public use samples compiled by the Social History Data Archives at the University of Minnesota reveals that 178 of 220 studies use only one of the eight public use samples currendy available. It is difficult to use more than one of the public use samples at a time because each sample has a different format, different coding schemes, and different documen- tation. Six separate research teams have been involved in the creation of the samples, and each of them has had their own ideas on how to organize the data. We are faced with eight different occupational classifications with a total of 3200 different categories, and seven incompatible classifications for variables such as birth- place, household relationship, and institution type. In fact, the only variable that is readily comparable across census years is age, and even there the samples differ widely in treatment of missing, illegible, and inconsistent data and in the coding strategy for the very old. Docu- mentation for the eight existing samples is contained in eight separate volumes totaling about 3000 pages. These volumes are for the most part organized differently from one another, and their treatment of comparability issues is often cursory. Only for the 1950 and 1970 public use samples— where the record layout and coding schemes were made to be reasonably compatible— has there been substantial multi-sample research. Indeed, most of the research using more than one public use sample has focussed on these two census years. This suggests that the incompati- bilities of the other samples have been a significant barrier to research on long term social change. The incompatibility of the public use samples in their present form means that multi-sample studies require a large initial investment to prepare the data for use. The number of investigators using multiple public use samples is growing rapidly. Most have proceeded by creating a set of special-purpose semi-compatible extracts containing a limited number of variables and minimal documentation. This ad hoc approach has already led to increasing duplication of effort Moreover, given the complexity of the files and the often subtle differences among them, the potential for eiror is large. The Social History Research Laboratory plans to convert the public use samples for 1880, 1900, 1910, 1940, 1950, 1960, 1970, 1980, and 1990 into a single consistent format and to prepare an integrated set of documentation oriented to the use of the samples as a series. In the long run, we anticipate adding data for all the remaining census years for which individual-level census enumera- tions survive; these years are 1850, 1860, 1870, 1920 and 1930. We are cunently applying for funding to create a sample for 1920, and plan future applications for the 1850, 1860, 1870 and 1930 census years. We already have had extensive experience with the entire series of public use samples. Indeed, the creation of common-format extracts of the samples has been a major preoccupation of the Social History Research Laboratory Summer 1991 Table 1 Summary of Availability of Selected Variables: Public Use Samples, 1880-1990 Blank = variable not available N = Neighborhood samples, 1970 PUS Y = variable available ST == State samples . 1970 Pus C = can be constructed SM = SMSA samp es, 1970 PUS S = sample-line individuals, 1940 and 1950 a = A" sample, 1980 PUMS 5 = five-percent sample onl) , 1970 pus b = 'B" sample, 1980 PUMS 15 = fifteen-percent sample only, 1970 PUS c = C" sample, 1980 PUMS 1880 1900 1910 194C 1950 1960 1970 1980 1990 Geographic Information State Y Y Y Y Y. Y N,ST Y Y(l) urban/Rural residence Y Y Y Y N,ST c Y Farm identifier(2) C Y Y Y Y Y Y Y Y Large cities Y Y Y Y Y SM a,b Y Modified SMA C C C SMA Y Y SMSA SM a,b Y County or county group Y Y Y Y Y SM a,b Y Personal Characteristics Age Y Y Y Y Y Y Y Y Y Sex Y Y Y Y Y Y Y Y Y Race Y Y Y Y Y Y Y Y Y Marital Status(3) Y Y Y Y Y Y Y Y Y Household Relationahip Y Y Y Y Y Y Y Y Y Duration of current Marriage Y Y S(4) Age at First Marriage S Y 5 Y Number of Marrbages Y S(5) S(5) Y 5 Y Married in past year? Y Y Y C,S C,S C C C children ever bom Y Y s s Y Y Y Y children surviving Y Y Surname code Y Y Y Y Subfamily relationships Y C C Y Y Y Y Y Y secondary fam. relationships Y C C Y Y Y Ethnicity and Migration Birthplace (country, stale) Y Y Y Y Y Y Y Y Y citizenship/Naturalization Y Y Y Y 5 Y Y Parental birthplace (country) Y Y Y S S Y 15 Parental birthplace (stale) Y Y Y S S Residence five years ago Y Y 15 Y Y Year of immigration Y Y 5 Y Y Mother tongue Y S S Y 15 Y Y speaks EngUsh? Y Y Y Y Spanish surname Y Y Y Y Y Y Y lASSIST Quarterly Table 1 (oontinued) 1880 1900 1910 1940 1950 1960 1970 1980 1990 Economic status and Employment wage and aalary income Y Y Y Y Y Y Total income Y Y Y Y Y occupation Y Y Y Y Y Y Y Y Y Industry C C Y Y Y Y Y Y Y Home ownership Y Y Y Y Y Y Y Mortgaged? Y Y Y Y Rent/Home value Y Y Y Y Y class of worker Y Y Y Y Y Y Y Period worked in census year Y S Y Y Y Y Hours worked last week Y Y Y Y Y Y Period unemployed Y Y Y Y S Y Year last worked Y Y Y Y Currently unemployed Y Y Y Y Y Y Y Education and veteran status school enrollment Y Y Y Y S Y Y Y Y Can read Y Y Y can write Y Y Y Years of schooling Y s Y Y Y Y Veteran StaUis Y(5) S s Y 15 Y Y 1. Not all geographic information indicated will be avaj lablc for all versions of the 1990 sample. 2. Definition of farm varies. 3. The "separated" category of marital atatus is not available before 1950; however, the similar category of married, spouse absent can be constructed for all census years. 4. Duration of current marital status. 5. The 1940 and 1950 censuses indicated whether married more than once. 6. Civil war veterans only. Summer 1991 over the past five years. These files are custom designed to meet the research and teaching needs of Minnesota faculty and graduate students. Increasingly, we have been receiving requests for common-format extracts from investigators at other institutions. We currently prepare about 25 common-format extracts a month for a broad range of users. In the course of our work, we have become intimately familiar with the intricacies of the public use samples. Our staff has invested hundreds of hours in the reconcili- ation of variables such as occupation and birthplace. It has become obvious, however, that our current proce- dures— which are duplicated at various institutions across the country — are highly inefficient. What is needed is a complete reworking of all the existing public use samples into an integrated format with complete documentation. This would allow most users to con- struct their own specialized extracts, and thus dramati- cally reduce the costs of research. We arc presently in the process of developing a detailed prospectus for the design of such an integrated public use microdata series. It is our hope that prospective users of the data series will provide us with as much feed-back as possible before the design is cast in stone. Copies of the prospectus are available upon request. REFERENCES Abel, Marjorie and Nancy Folbre (1990). "A Methodology for Revising Estimates: Female Market Participation in the U.S. Before 1940." Historical Methods 23: 167-178. American Economic Association (1899). The Federal Census. New York: Macmillan. Estimates ofFertilbty and Population in the United States. Princeton, NJ: Princeton University Press. Conk, Margo A. (1980). The United States Census and Labor Force Change. Ann Arbor: UMl Research Press. (1981). "Accuracy, efficiency, and bias: The interpretation of women's work in the U.S. Census of Occupation, 1890-1940." Historical Methods 14:55-72. Edwards, A.M. (1943). "Comparative occupation statistics for the United States, 1870 to 1940." Sixteenth Census of the United States: 1940, Population. Washington, D.C.: U.S. Government Printing office. Folbre, Nancy and Marjorie Abel (1989). "Women's Work and Women's Households: Gender Bias in the U.S. Census." Social Research 56: 545-569. Forstcr, colin and G.S.L Tucker (1972). Economic Opportunity and While American Fertility Ratios. 1800- 1860. New Haven: Yale University Press. Goldin, Claudia (1980). "The Work and Wages of Single Women, 1870-1920." Journal ofEconomic //wtory 40: 81-88. (1983). "The Changing Economic Role of Women: A Quantitative Approach." Journal of Interdisciplinary History 12: 707-733. x Graham, Stephen N. (1980). 1900 Public Use sample: User's Handbook. Seattle: Center for Demography and Ecology, University of Washington. Hogan, Dennis P., 1981. Transitions and Social Change: The Early Lives of American Men. New York: Academic Press. Anderson, Margo J. (1988). The American Census: A Social History. New Haven: Yale University Press. Anderson, Michael (1971). Family Structure in Nineteenth Century Lancashire. Cambridge, England: Cambridge University Press. Barron, Hal (1984). Those Who Stayed Behind: Rural Society in Nineteenth-Century New England. New York: Cambridge University Press. Bose, C. (1987). "Devaluing Women's work: The Undercount of Women's Work in 1900." In C. Bose, R. Feldberg, and N. Sokoloff (eds). Hidden Aspects of Women's Work. New York: Praeger, 95-1 15. Coale, Ansley J. and Melvin Zclnik (1963). New Jaffe, AJ. (1956). "Trends in the participation of women in the working force. " Monthly Labor Review 79(5): 559-565. Katz, Michael B., Michael J. Doucet, and Mark Stem (1982). The Social Organ ization of Early Industrial Capitalism. Cambridge, Mass.: Harvard University Press. McClelland, Peter D. and Richard J. zeckhauser (1982). Demographic Dimensions of the New Republic: American Interregional Migration, Vital Statistics, and Manumissions, 18(X)-1860. New York: Cambridge University Press. Modell, John, Frank Furstenberg and Theodore Hershbcrg (1975). "Social Change and the Transition to lASSIST Quarterly Adulthood in Historical Perspective," Journal of Family History 1:7-32. Openheimer, Valerie K. (1970). The Female Labor Force in the United States: Demographic and Economic Factors Governing its Growth and Changing Composition. Population Monograph 5, University of California, Berkley. Rindfuss, Ronald R. and James A. Sweet (1977). Postwar Fertility Trends and Differentials in the United States. New York: Academic Press. Ruggles, Steven (1988). "The demography of the unrelated individual, 1900-1950." Demography. 25:4. (1991). "Comparability of the Public Use Files of the U.S. Census of Population. Social Science History. Ruggles, Steven and Mbriam King (forthcoming) Fragmentation of the Family: Living Arrangements in America, 1580-1980. Ruggles, Steven and Russell R. Menard (1990). "A public use sample of the 1880 Census of Population." Historical Methods 23: 104-1 15. (1984a). Census of Population, 1940: Public Use Sample Technical Documentation. Washington, D.C.: U.S. Government Printing office. .(1984b). Census of Population, 1950: Public Use Sample Technical Documentation. Washington, D.C.: U.S. Government Printing Office. _(1989). 1990 Census of Population and Housing. Tabulation and Publication Program. Washington, D.C.: U.S. Government Printing office. Yasuba, Yasukichi (1963). Birth Rates for the white Population of the United States. Baltimore: Johns Hopkins University ' Presented at the lASSIST 91 Conference held in Ed- monton, Alberta, Canada. May 14 - 17, 1991. S. Ruggles, Department of History, University of Minnesota, Minneapolis, MN 55455 ruggles@legohead.histumn.edu (512)624-4081 Stevens, David (1991). "Life-course transitions to adulthood, 1900-1970." Journal of Family History. Strong, Michael A., samuel H. Preston, Ann R. Miller, Mark Hereward, Harold R. Lentzner, Jeffrey R. Seaman, Henry C. williams (1989). User's Guide: Public Use sample, 1910 Census of Population. Philadelphia: Population Studies Center, University of Pennsylvania. U.S. Bureau of the Census (1954). The 1960 Census of Population and Housing: 1960. Two National samples of the Population of the United States. Washington, D.C.: U.S. Government Printing Office. (1972). Public Use samples of Basic Records From the 1970 Census Description and Technical Documentation. Washington D.C.: U.S. Government Printing Office. (1973). Technical Documentalbon for the 1960 Public Use sample. Washington, D.C.: U.S. Government Printing Office. (1982). Public Use samples of Basic Records From the 19B0 Census: Description and Technical Documentation. Washington, D.C.: U.S. Government Printing Office. Summer 1991