Abstract:Abstract This article leverages a five-treatment response mode experiment (paper-only, web-only, sequential web-mail, choice, and choice-plus [choice with a promised incentive for responding online]) that was conducted within a nationally representative survey. Because this survey’s sample was drawn from respondents to another nationally representative survey, we have rich frame data that includes multiple indicators of comfort using the internet for our sample members and we can compare their response behavior across two surveys. We find that the paper-only treatment yielded a lower response rate than most of the other treatments, but there were not significant differences between the response rates for the other treatments. Among our mixed-mode treatments, the sequential web-mail treatment had the highest percentage of response by web and the lowest cost per response. When focusing on the subgroups that we expected to be the least—and the most—comfortable with the internet, we found that the paper-only treatment generally performed worse than the others, even among subgroups expected not to be comfortable with the internet. We generally did not find significant differences in the effect of response mode treatment on the response rate or percentage of response by web between the subgroups who were the most and least comfortable with the internet. In terms of the consistency of response mode choice over time, our results suggest that some people respond consistently—but also that response mode preferences are weak enough that they can be influenced by the way in which the modes are offered. We ultimately recommend using a sequential web-mail design to minimize costs while still providing people who cannot or will not respond by web with another response mode option. We also find evidence that there may be a growing lack of interest in responding by paper; more research is needed in this area.

A Comparison of Three Designs for List-style Open-ended Questions in Web Surveys

Visual Design and Cognition in List-Style Open-Ended Questions in Web Probing

Using Placeholder Text in Narrative Open-Ended Questions in Web Surveys

Assessing the effect of questionnaire design on unit and item-nonresponse: evidence from an online experiment

A comparison of question order effects on item-by-item and grid formats: visual layout matters

Relying on External Information Sources When Answering Knowledge Questions in Web Surveys

Data quality comparison between computers and smartphones in different web survey modes and question formats

Concurrent, Web-First, or Web-Only? How Different Mode Sequences Perform in Recruiting Participants for a Self-Administered Mixed-mode Panel Study

Sequential and Concurrent Mixed-Mode Designs: A Tailored Approach

Should We Offer Web, Paper, or Both? A Comparison of Single- and Mixed-Response Mode Treatments in a Mail Survey

The Effects of Questionnaire Length on the Relative Impact of Response Styles in Ambulatory Assessment

A Dynamic Technique for Conducting Online Surveybased Research

From paper to pixels: A comparison of paper and computer formats in psychological assessment

Comparison of Three Navigation Button Designs in Mobile Survey for Older Adults

Online Field Experiments: a Selective Survey of Methods

Improving the Efficiency of Outbound CATI As a Nonresponse Follow-Up Mode in Address-Based Samples: A Quasi-Experimental Evaluation of a Dynamic Adaptive Design

Conducting organizational survey and experimental research online: From convenient to ambitious in study designs, recruiting, and data quality

On Differences between Giving "Middle Answers"or Not in Surveys

Cognitive load reduction strategies in questionnaire design

Ask Me or Tell Me? Enhancing the Effectiveness of Crowdsourced Design Feedback

Questionnaire design from a cross-cultural perspective: an empirical investigation of Koreans and non-Koreans