Schuman, Professor and Research Scientist Emeritus at the University of Michigan, notes in his article, "Sense and Nonsense about Surveys," that surveys have appealed since ancient times to two human propensities: (1) Gathering information by asking questions, and (2) attempting to learn more about one's environment by examining a small part of it (i.e., sampling).
Through an illustrative example of the sampling conducted by the infamous Literary Digest poll prediction of the 1936 U.S. presidential election, Schuman demonstrates both the problems and progress made in survey methodology. On the basis of a very large sample—2 million completed and returned surveys out of 10 million distributed—the Literary Digest had incorrectly predicted that Franklin D. Roosevelt would lose decisively in the election.
"At the same time, pollster George Gallup, using many fewer cases but a better method, accurately predicted that Roosevelt would win. Gallup used quotas in choosing respondents in order to represent different economic strata, whereas the Digest had worked mainly from telephone and automobile ownership lists, which in 1936 were more likely to include wealthy people who were likely to be opposed to Roosevelt," explained Schuman. (There were other sources of bias as well.) As a result, the Digest disappeared from the scene, and Gallup was on his way to becoming a household name.
Schuman highlights two advances in survey methods since the 1930s and 1940s that allowed the modern survey to develop and flourish. "The first is the emergence of probability sampling, which is fundamental to social science research today and helps overcome the problem of biased sampling of respondents. The second relates to the development of greater precision in asking questions in surveys and in replicating and testing the validity of the questions themselves," said Schuman.
Obtaining a good sample depends on more than adhering to principles of good probability sampling (i.e., selecting respondents from the population in proportion to the degree to which they are representative of that population). Well-done surveys also depend on the quality of their implementation. Thus, when members of a population who are selected to be in the sample are not successfully interviewed or do not respond to specific questions, the omissions can seriously compromise the validity and reliability of the survey and are as important as the sample size. Interestingly, however, for mathematical reasons, reasonably accurate estimates can be obtained—from populations of very different sizes—with sample sizes of around 1,000, and only when extraordinary precision is needed do samples need to be much larger.
The "margin of error" plus/minus percentage figures that accompany most media reports of polls reflect the size of the final sample, but they do not reveal the sampling methods or the extent to which the targeted individuals or households were actually included in the final sample. For example, the percentage of people who refuse to take part in a survey is particularly important. In some federal surveys, the percentage is small, within the range of 5 to 10 percent. But even in the best non-government surveys, the refusal rate can reach 25 percent or more, and it can be far larger in the case of poorly executed surveys. The opinions of non-responders in a population may be very different from those of responders, thus their non-inclusion biases results.
Many survey reports are based on such poor sampling procedures that they do not deserve to be taken seriously. This is especially true of reports based on "focus groups," which offer human interest but are subject to vast amounts of error. Internet surveys also cannot represent the general population adequately at present, though this is an area where some serious attempts are being made to compensate for the inherent difficulties and inherent sampling bias.
Schuman also describes several techniques researchers use to improve survey questioning—such as asking several different types of questions about any important issue, varying the form, wording, and context of questions, and introducing comparisons into analyses. These techniques are utilized in order to overcome variations in response that are produced not by the responders actual opinion but by the peculiar form, wording, or frame of reference of questions in a survey. Using several examples from actual surveys, he shows how different ways of asking the same question can produce substantial differences in outcomes.
Despite the methodological advances, some issues still remain. For example, pollsters still face the task of interpreting the meaning of questions and the associations among questions, but this is true in all types of research including those using field observations.
Schuman states that surveys remain our best tool for learning about large populations. "In the end, however, with surveys, as with all research methods, there is no substitute for both care and intelligence in the way evidence is gathered and interpreted. What we learn about society is always mediated by the instruments we use, including our own eyes and ears," concluded Schuman.
The American Sociological Association, founded in 1905, is a non-profit membership association dedicated to serving sociologists in their work, advancing sociology as a science and profession, and promoting the contributions and use of sociology to society.
Journal
سياقات اللغة والدراسات البينية