iassisl quarterly - 3 Debates and directions in the future of opinion polling data The following two papers were presented at the 1ASSIST '87 conference in a session entitled: The uses of socio- political data. The session focused on the comparability of electoral data, public opinion data and other comparative data projects, including technical and political factors affecting secondary analysis. (Ed. note). by Neil Guppy 1 Department of Anthropology and Sociology University of British Columbia topical issues, and the mass media broadcast polling results in an incessant stream. Daily we see or hear new polling results which reveal how our contemporaries rate the politicians, or the postal service, or the latest soft drink. Recently, pollsters have taken to asking the public how they feel about polls. Since pollsters are concerned with assessing the images and opinions of the population, it is hardly surprising that polls on polling, or surveys on surveys, have increasingly found their way into tne polling literature (see e.g., Roper, 1986; Goyder, 1986). The irony of using polls to evaluate polls is not lost on the pollsters, and a good deal is revealed by these self-assessments. This paper reviews this recent literature in an attempt to gain some leverage on the potential directions of public opinion research in the next decade or so. 1 begin with estimates of the sheer volume of polling data now being collected in different countries. This pervasiveness of polling, though, has generated substantial controversy and conflict in the practice of polling. In an attempt to understand the possible ramifications of current practices and techniques for the future of opinion polling data. I review general criticisms levelled against opinion polling. These criticisms are used to organize a discussion of future directions for both the industry, and by implication, for those who rely on poll-generated data. A social invention of this century, opinion surveys arc now commonplace in liberal democratic societies. Major political parties cannot afford to ignore polling data, market researchers tap public opinion on a plethora of 'This is a revised version of a paper originally presented at the Annual International Association for Social Science Information Service and Technology (IASSIST) Conference held in Vancouver, British Columbia, Canada on May 19-22, 1987 The Prevalence of Polling It is difficult, especially on an international scale, to ascertain exactly how much polling data is currently being collected. No central registry of polling data is available, so a variety of proxy estimates must be employed. I rely on Summer 1987 4 - iassist quarterly three — the percentage of the population reporting participation in opinion research studies, the amount of money spent on polling activities, and the frequency of publication of polling results. One method of assessing the prevalence of polling is to determine how many members of the general public have been involved as respondents in polling or survey research. Table 1: Trends in Respondent Involvement, 1978-1984 Participation 1978 1980 %Ever 47 59 %LastYear 19 25 1982 59 23 1984 54 23 Source: Schleifer, 1986 Table 1 shows trend results from a series of U.S. polls where respondents were asked, first, to report whether they had "ever participated in a survey before", and second, whether they had previously been "interviewed for a survey in the past year" (Schleifer, 1986). Since 1980 the majority of Americans have been contacted at least once, and almost one-quarter have panicipated in at least one poll or survey in the past year. Table 2 presents similar findings reported by Roper (1986). His findings parallel the results reported in Table 1, demonstrating that most Americans now have first-hand experience with polling and survey research. Furthermore, many Americans have multiple experiences as participants. Results from other countries are less systematic. Goyder (1986) reports that Canadians in a mid-sized city have experienced levels of participation roughly equal to the U.S. findings. Table 2: Respondent Involvement in Public Surveys, 1985 Percent Never 41 Once 17 Twice 16 3-5 16 6+ 9 D.K. 1 Source Roper, 1986 Estimates of the amount of money spent on polling are available only for the U.S., where Advertising Age annually reports financial data for polling firms. For the Fiscal year ending in 1985, U.S. research firms in the marketing, advertising, and polling sector billed for $1,785.3 million, up 11.5% from the previous year, and more than three umes the annual rate of inflation. Of this, approximately 78% was from U.S. based work (see Honomichl, 1986). In the mid-1970s, Paleu et al. (1980) reported that the New York Times ran news stories containing polling data on an average of one in even three days. A rough count of news, editorial, and feature stories in the 1985 New York Times Index reveals 278 items under the heading "public opinion polls" (a non-election year in the U.S.). Worcester (1980) reports that in the United Kingdom (as elsewhere), opinion polls dominate the front-page headlines during the build-up to national elections. Polling is pervasive, and every indication suggests that growth has continued to this day. Recent advances in random digit dialing and computer assisted telephone interviewing have served to extend the pollsters' reach even Summer 1987 iassist quarterly - 5 farther. Rather than dulling criticism, this growth has occurred in the face of skeptical commentary. Criticisms of Polling Several very general criticisms of polling are heard frequently. Often the charge is made that polls are, at best, only a superficial barometer of public beliefs. In its strongest guise, this argument suggests polling results are frequently wrong. A weaker version claims opinion polling gives only a perfunctory account of facile opinions. Others argue that polls are invasive, trampling public privacy by asking for personal information (e.g., political preferences). Yet others complain that polls have fundamentally altered the political process such that substantial policy matters are unduly influenced by popular and often ill-informed opinion rather than by thoughtful deliberation. These are very basic arguments on which substantial ink has already been spilled. Rather than add to this area of the debate, I will attempt to look behind some of these general objections to more specific issues in the practice of polling. 1. Distortion — one concern is that people don't give true responses when answering the pollsters' questions (Lewis and Schneider, 1982). Individuals lie, or as the pollsters say, respondents "misreport". Sometimes this appears to be deliberate, as when people are asked whether they voted in the last election (evidence shows that more people claim to vote than actually do vote). On other occasions, it is less certain whether people actually lie or whether they are generally confused; a classic example here is a poll conducted by the German magazine Der Spiegel in which a fictitious cabinet member came sixth in popular rankings, ahead of ten real-life ministers of the crown. 2. Non-attitudes — if pollsters ask people questions about which they have no opinion, some people feel pressure to respond and instant opinions may be invented. Evidence suggests that the more remote an issue is from a respondent, the more random is the response. It is especially on this basis that critics claim polls are superficial. 3. Opinion change — individual attitudes are often not stable or deep-seated. Snap-shots from opinion polls may be as interesting as yesterdays news, but they are known to be poor launching pads for general social forecasts. 4. Issue complexity — few issues are so clear-cut that single attitude questions can capture the essence of the matter. The black and white image of the world that one may acquire from reading opinion poll results does not do justice to the full array of public sentiment. 5. Words and deeds — the ease with which people may express an opinion on a topic is no guarantee of the direction their actions may take. The link between attitude and behaviour has been probed repeatedly, and we still have less than perfect knowledge of when any congruency between the two will hold. 6. Question wording — social scientists have known for some time that subtle changes in question wording can influence response patterns. Asking respondents whether they would "forbid" or "not allow" something leads to very different results, with a swing of some 20% in response frequencies. So too, the sequence of questions in an interview can influence responses. 7. Impersonality — just as students complain that multiple choice exams do not adequately assess the depth of their knowledge, so some Summer 1987 6 - lassist quarterly argue that polls similarly distort reality because people are forced to respond in fixed categories which rarely allow them any self-expression. The frame of reference for the entire polling exercise is determined a priori and this can easily disqualify certain questions and certain responses. 8. Sampling — the ability of samples of several hundred people (up to about 2,500) to accurately reflect the diversity of opinion in an entire nation has often been doubted. Polls seldom tap the rich or the poor in any society, thereby predominantly reflecting the views of the middle class. The more general criticisms, and the eight more specific objections to polling listed immediately above, are likely to continue to surface in debates over polling in the forseeable future. The veracity of these claims is often less compelling than the volume of their elucidation Would suggest. Both polling experts and secondary users of polling data are conversant with the limitations involved. It does not follow from this, however, that these criticisms will gradually dissipate, or even more importantly, that they will have no consequences for the future of opinion polling. Public attitudes about opinion polling data are as likely to influence decisions about the future of polling as they are to influence the future of political parties. What I turn to now is evidence pertaining to criticisms of opinion research in an attempt to develop some perspective on future directions in the polling marketplace. Directions and Tendencies i) Assessments of Accuracy One recurrent question concerning polls is the frequency with which the pollsters accurately reflect public sentiment William Buchanan (1986) has examined the results of election polling in several Western democracies in an attempt to examine the precision of election forecasts based on opinion surveys of voter intentions. In analyzing 155 polls from 68 national elections, he found that on 22 occasions the wrong party was predicted as being victorious. Beyond forecasting the wrong victor in 1 out of 7 attempts, there appears to be no trend of improvement since 1949. Similar findings are reported by Worcester (1980) for the U.K. In general, erroneous forecasts are made by a group of pollsters for particularly close elections. It is not true that the polls are always wrong, although it is the case that when the polls are wrong, they all tend to be wrong. A second issue, linked to accuracy, is the actual reporting of polling results. Here the quesuon is not so much whether the polls are correct or incorrect, but whether they are properly reported by the press. Reporting is crucial, because it is via press reports that most people form their perceptions of the practices of pollsters. Smith and Verrall (1985) undertook a critical evaluation of Australian television coverage of election opinion polls and they claim "poll coverage is extensive, superficial, and inaccurate". Typical errors included "temporal transposition" (incorrectly attributing past or present results to some future point), overgeneralization (extending claims to beyond the sample universe), overstatement (exaggerating the strength of findings), and making ambiguous contrasts (comparisons between poorly conceived groups or time periods). Summer 1987 iassist quarterly - 1 Related to this is yet a third aspect, the completeness of press reports on opinion polls. Here the concern is with whether or not the press meets basic reporting standards so that consumers can make informed judgments regarding polling results. Table 3 contains a listing of the basic standards, showing the percentage of poll reports (either election or non-election polls) which comply with these basic levels of reporting adequacy. As the percentages reveal, the three papers under examination (L.A. Times, Chicago Tribune, Atlantic Constitution) do not do a particularly good job of providing basic information for informed judgments about polling results. These figures are from the 1970s, and there may have been improvements since this time, especially as the media assign specific people to do all their polling reports. No systematic evidence is currently available to assess possible improvements. Table 3: Polling Standards versus Polling Practice 1972-79 % Reported in ElectionStandards Non-Election Sample Size 89 81 Sponsor 80 87 Wording 71 34 Sampling Error 31 2 Population 91 66 Method 62 38 Timing 76 48 N 61 55 Source: Miller and Hurd, 1982. [in Norwegian] Finally, pollsters have asked the public about their perceptions of polling accuracy. Andrew Kohut (President of Gallup) reports that in 1985, 68% of respondents thought the pollsters got election forecasts correct most of the time (Kohut, 1986). This was an increase in public confidence from 57% in 1944. When asked about non-election polls, however, only a slim majority of people felt the polls were generally right in tapping the public mood (52% in 1944; 55% in 1985). Conversely, negative sentiments about non-election polling accuracy seem to have increased, with 21% saying they felt the polls were "not right at all" (up from 12% in 1944). The British appear to be a little more sceptical about polling results, with only 32% believing that "the opinion polls are normally right" (46% thought they were normally wrong, with 22% giving other responses — see Worcester, 1980: 561). If public opinion is as influential as the practice of polling implies, then pollsters need to be conversant with the public images of polling. When gauged by specific measures of accuracy, polling does not have a massive majority of support ii) Bogus Polls Polling got a bad name in the 1936 U.S. presidential election when the Literary Digest, a magazine for affluent Americans, asked readers to write in with their choice for president. On the basis of these responses, the Digest predicted that Alf Landon would win the election, opening itself and the prestige of polls to ridicule when Franklin Roosevelt won by a landslide. Pollsters have insisted that only surveys with "scientifically" selected random samples should be called polls, and for some lime this seemed to be accepted practice. Recently, however, phoney or bogus polls have become more prevalent For example, after the 1980 Carter-Reagan debate, ABC asked viewers to phone and report who they felt won the debate. On the basis of some 727,000 calls. ABC reported that their "poll" showed people felt Reagan had won 2-1. This phenomenon of Summer 1987 [assist quarterly self-selected "samples" appears to be growing in the polling marketplace. QUBE is a more recent invention, allowing cable subscribers to send digital signals back through the video cable to record their "vote" on various issues. Political parties in several countries have taken to doing "surveys" of the public, asking people first to rate the current government, and then to donate money to help the party doing the survey. The prevalence of these bogus surveys is hard to detect, but concern is mounting that sales people are using this technique to identify potential customers. Schleifer (1986) reports that in 1980 some 13% of U.S. respondents reported having been exposed to false surveys, a percentage that increased by 4 points to 17% in 1984. With almost 1 person in 5 being confronted with bogus surveys, the reputation of the industry could be tarnished quickly and decisively. in) Respondent burden Knowing that 1 in 5 people are approached by phoney surveys, one wonders about the frequency with which people have been approached by legitimate pollsters or survey researchers. Estimates vary, as shown in Tables 1 and 2, but over one-half the population in the U.S. appears to have been involved in a survey or poll at some time. Schleifer (1986) reports that of the 23% of respondents who had been involved in a survey in the past year, almost 1 in 5 had participated in four or more polls. These latter individuals are known to survey researchers as "professional respondents" and the inclusion of the same people in multiple surveys has pollsters worried about the "freshness" of their samples. The growth of polling raises the possibility of 'over-kill' — people will be 'turned-ofT polls by too many requests for their help. One way of assessing respondent burden is to examine empirical evidence of possible overexposure. Steeh's (1981) results, shown in Figure 1, chart refusal rates between 1952 and 1980 for two national U.S. samples, both conducted by the University of Michigan Survey Research Center. As the graph shows, refusal rates in both the election and consumer attitude series are rising, although whether or not this is due to respondent burden per se is difficult to determine. As Steeh notes it could be caused by one factor or a combination of factors, including overexposure, disillusionment with the use or accuracy of survey results (see above), or heightened concern about privacy and confidentiality. Goyder and Leiper (1986) report similar trends based on an analysis of polling and survey response rates in the U.K., the U.S., and Canada, and they point to rising criticism of Census practices, especially in Canada and the U.K. iv) Exit polls In 1980, Jimmy Carter conceded defeat before the polls had closed in the American west One reason for this was that the television networks were using exit polls to predict the winner before everyone had had an opportunity to vote. Exit polls (or 'same-day polls' as they are called in Britain) are conducted by standing outside selected polling places and asking those leaving for whom they had voted. Based on these reports, the networks have been able to forecast with accuracy the eventual winner. The State of Washington was so upset with this practice that they banned exit polls, making it illegal for people to conduct surveys within 300 feet of a polling station. The media challenged the law in court, losing an initial verdict and then winning on appeal — the state is currently appealing the appeal. Whatever the eventual outcome, the concern remains that the techniques and the process of polling have fundamentally altered the practice of politics. Whether exit polls actually alter the outcome of elections is debatable (see Sudman, 1986), although they do appear to have a Summer 1987 iassisl quarterly - 9 marginal impact on voter turnout when a landslide has occurred. The key point here, however, is not whether exit polls actually have any effect, but that people believe they have an effect As pollsters themselves have shown, it is the image that is important v) Polling Initiatives Pollsters have recently been expanding their craft at a rapid rate. The use of polls for marketing is an old and established pastime (Labatts Brewery in Canada has opinion data dating back to 1910 in Canada). More recently, polling has had an influence in the courtroom where survey results have been used in judgments over trademark protection (the NFL, Coming Glass Works), advertising claims (Pepsi vs Coke), and jury selection (Ford, IBM, MCI Communications). Furthermore, various departments of government charged with regulatory functions have begun to use polling data to assess the impact of certain initiatives. Listerine was required to engage in corrective advertising to dispel the myth they had created that the mouthwash would prevent people acquiring colds and sore throats. To assess the effectiveness of the correction, the U.S. regulatory agency that was responsible for enforcement used a poll to examine changes in opinion (see Crespi. 1987; Dutka, 1982). As image and knowledge grow in importance, the pollster's craft is more in demand. But as the demand rises, the value of information escalates, and polling agencies in the private marketplace are less willing to freely relinquish their data. Beyond cost, the sheer volume of information frequently makes archiving data a burden to avoid — profit lies with the next project Conclusions U.S. respondents continue to report that they feel polls are "a good thing" (73% in 1944 and 76% in 1985 — see Kohut 1985), although the British are less sanguine, with a majority feeling that polls were "pointless" or "not very accurate" (Worcester, 1980: 560). Potentially, fatal dangers for the polling industry would appear to lurk in the areas of bogus polls and respondent burden. The very prevalence of polling may undermine the craft as individuals feel inundated with strangers asking dubious questions about issues which people increasingly define as nobody else's business (see Goyder and Leiper, 1986 for an analysis of increasing objections to the Census). Serious, but probably not fatal, dangers would appear to lie in the possibility of disastrous election predictions in several countries simultaneously, or the use of polls in a way so as to make people feel their personal freedoms or rights are subverted (as seems to be the case with exit polls). Finally, several signals suggest that more and more public attitude data from polling firms will become off-limits. Currently the vast majority of opinion polling data is not publicly available. Increasingly, polling data will be kept secret as polling agencies realize the economic value of trend projections. As the value of information grows, pollsters will protect their investments and profitability. Several companies in North America now conduct omnibus surveys which they keep confidential. In addition, several countries now have governments collecting general social survey data. Thus there will be less pressure on the pollsters to serve the academic interest by releasing the poll data. Summer 1987 10 — (assist quarterly References Buchanan, William. Election predictions: an empirical assessment. Public opinion quarterly 50(2):222-227, 1986. Dutka, S. Bringing polls to justice. Public opinion October/November, 47-49, 1982. Goyder, John. Surveys on surveys: limitations and potentialities. Public opinion quarterly 50(1):27-41, 1986. Goyder, John and Jean Leiper. The decline in survey response: a social values interpretation, unpublished paper. Waterloo, OnL: University of Waterloo, 1986. Honomichl, Jack. The nation's top marketing/advertising research companies. Advertising age May 19, 1986. Kohut, Andrew. Rating the polls: the views of the media elites and the genera! public. Public opinion quarterly 50(l):l-9, 1986. Lewis, I. A. and W. Schneider. Is the public lying to the pollsters. Public opinion April/May, 1982. Miller. M. and R. Hurd. Conformity to AAPOR standards in newspaper reporting of public opinion polls. Public opinion quarterly 46(2):243-249, 1982. Roper. Burns. Evaluating the polls with poll data. Public opinion quarterly 50(1): 10-16, 1986. Schleifer, Stephen. Trends in attitudes toward and participation in survey research. Public opinion quarterly 50(l):17-26, 1986. Smith. T. and D. Verrall. A critical analysis of Australian television coverage of election opinion polls. Public opinion quarterly *9 2j:58-79, 1985. '..:::. C. Trends in non-response rates. Public opinion quarterly 45:40-57, 1981. Aorcester, Robert. Pollsters, the press, and poiiucal polling in Britain. Public opinion quarterly 44(4): 548-566, 1980. Summer 1987