The Need for a Recurring Large-Scale Benchmarking Survey to Continually Evaluate Sampling Methods and Administration Modes: Lessons from the 2022 Collaborative Midterm Survey

Peter K. Enns,Colleen L. Barry,James N. Druckman,Sergio Garcia-Rios,David C. Wilson,Jonathon P. Schuldt

DOI: https://doi.org/10.48550/arXiv.2407.06090

2024-07-09

Abstract:As survey methods adapt to technological and societal changes, a growing body of research seeks to understand the tradeoffs associated with various sampling methods and administration modes. We show how the NSF-funded 2022 Collaborative Midterm Survey (CMS) can be used as a dynamic and transparent framework for evaluating which sampling approaches - or combination of approaches - are best suited for various research goals. The CMS is ideally suited for this purpose because it includes almost 20,000 respondents interviewed using two administration modes (phone and online) and data drawn from random digit dialing, random address-based sampling, a probability-based panel, two nonprobability panels, and two nonprobability marketplaces. The analysis considers three types of population benchmarks (election data, administrative records, and large government surveys) and focuses on the national-level estimates as well as oversamples in three states (California, Florida, and Wisconsin). In addition to documenting how each of the survey strategies performed, we develop a strategy to assess how different combinations of approaches compare to different population benchmarks in order to guide researchers combining sampling methods and sources. We conclude by providing specific recommendations to public opinion and election survey researchers and demonstrating how our approach could be applied to a large government survey conducted at regular intervals to provide ongoing guidance to researchers, government, businesses, and nonprofits regarding the most appropriate survey sampling and administration methods.

Other Statistics

What problem does this paper attempt to address?

### Problems the paper attempts to solve

This paper aims to solve the problem of how to evaluate the trade - offs between different sampling methods and management models in the process of adapting survey methods to technological and social changes. Specifically, by using the data from the 2022 Collaborative Mid - term Survey (CMS), the paper provides a dynamic and transparent framework to evaluate which sampling methods or combinations of methods are most suitable for different research objectives. The paper mainly focuses on the following points:

1. **Evaluating the accuracy of different sampling methods**:
- The paper evaluates the accuracy of different sampling methods by comparing three types of population benchmarks (election data, administrative records, and large - scale government surveys). These benchmarks include the results of the 2022 mid - term elections, birth records in the past 10 years, and data from the American Community Survey (ACS).

2. **Evaluating the impact of different management models**:
- The paper explores the impact of two management models, telephone surveys and online surveys, on survey results and analyzes the sample performance under different management models.

3. **Evaluating the effectiveness of combinations of different sampling methods**:
- The paper not only evaluates the performance of a single sampling method but also evaluates the effectiveness of combinations of different sampling methods to guide researchers on how to combine multiple sampling methods and sources.

4. **Providing specific suggestions**:
- The paper provides specific suggestions for public opinion and election survey researchers and shows how to apply this method to large - scale government surveys conducted regularly to continuously provide guidance on the most appropriate survey sampling and management models for researchers, governments, enterprises, and non - profit organizations.

### Main contributions of the paper

1. **Dynamic evaluation framework**:
- Provides a dynamic and transparent framework that can continuously evaluate the applicability of different sampling methods and management models as social and technological conditions change.

2. **Multi - dimensional evaluation**:
- Comprehensively evaluates the accuracy of different sampling methods and management models through multiple population benchmarks and data from different geographical regions.

3. **Methodological innovation**:
- Encourages methodological innovation, quickly discloses data and methodological information, and provides methodological guidance for government, enterprise, academic, and policy surveys.

4. **Practical application**:
- Through specific case analysis, shows how to use these data and frameworks to evaluate the "applicability" of different sampling methods and management models and puts forward specific improvement suggestions.

### Conclusion

By comprehensively evaluating the performance of different sampling methods and management models, this paper provides valuable references for researchers. The paper emphasizes that there is no single best survey method, and the best strategy will depend on specific research objectives. Through continuous evaluation and improvement, researchers can better choose survey methods suitable for their research needs.