image: Tosch's Surveyman software addresses four key areas where bias can creep into a survey. view more
Credit: UMass Amherst
AMHERST, Mass. – Doctoral student Emma Tosch of University of Massachusetts Amherst School of Computer Science won a Best Paper award this week at the premier international computer programming language design conference in Portland, Ore., for her work on "Surveyman," a first-of-its-kind software system for designing, deploying and automatically debugging surveys to improve their accuracy and trustworthiness.
Tosch's advisor, professor Emery Berger, says, "Poor quality data can be seriously misleading. Flaws in survey design and deployment threaten the validity of results, making them untrustworthy. Until now, there has been no way to address these shortcomings automatically. But Surveyman, which is a free, publicly available and revolutionary new tool, can identify problems in any survey from the design stage onward. It should completely change the way people create surveys."
Tosch's talk at the Object-oriented Programming, Systems, Languages and Applications (OOPSLA) conference will release Surveyman to assist economists, sociologists, psychologists and professionals in politics, marketing, industry, public health, news and public opinion who increasingly use online surveys to gauge opinion and guide information and marketing campaigns.
From the outset, Surveyman guides the user through a series of steps to create and build a spreadsheet that will become the new survey. The software addresses four key areas where bias can creep into a survey, Berger and Tosch explain. First, it randomizes the survey, changing the order in which questions are asked to avoid order bias. It also provides multiple but equivalent wordings of the same question to prevent word variation bias.
Further, Surveyman conducts a diagnostic sweep to warn the survey author of danger spots such as when certain questions become correlated, or redundant, and should be removed to avoid respondent survey fatigue. Finally, the software can identify random respondent bias that occurs when respondents are not answering truthfully. Berger says, "It will find them and kick them out."
Surveyman will administer an author's survey online and perform diagnostics on the data when results come in, he adds. This work is supported by the National Science Foundation.
###