How do you compare and contrast different survey methods, such as mail, phone, web, or face-to-face?

Mail surveys are often the first entry to customer research, and for good reasons. There is now a wide range of easy to implement tools, they are easy to scale, and they can offer immediately actionable insights. However, drawbacks need to be considered to correctly manage expectations. Surveys are not "real time", and if they aim at assessing a specific experience, the respondent will usually give a feedback that can be less spontaneous that a live one. It is also a good method to validate hypotheses, but not the right for generation of new ideas. The goals of a survey need to be clear from the beginning, and take these limitations into account.

Advantages: Cost-effective: Mail surveys can be relatively inexpensive compared to phone or face-to-face interviews, especially for large samples. Wide geographic reach: You can reach respondents across a broad geographic area without incurring additional travel or phone costs. Respondent convenience: Respondents can complete the survey at their own pace and on their own time, which can increase participation rates. Disadvantages: Low response rates: Mail surveys typically have lower response rates compared to other methods, which can introduce nonresponse bias. Limited question types: Mail surveys are not ideal for complex or open-ended questions, as they lack the flexibility to clarify or probe for additional information.

Advantages: Higher response rates: Compared to mail surveys, phone surveys typically achieve higher response rates due to the interactive nature of the interview. Ability to clarify questions: Interviewers can clarify questions, probe for additional information, and ensure that respondents understand the questions. Faster data collection: Phone surveys can be completed more quickly than mail surveys, allowing for faster data collection and analysis. Disadvantages: Cost: Phone surveys can be more expensive than mail or web surveys, as they require trained interviewers and incur phone charges Interviewer bias: The presence of an interviewer can introduce bias into the responses, as respondents may feel pressured to answer in a certain way

Advantages: Low cost: Web surveys are generally the most cost-effective survey method, as they eliminate printing and mailing costs associated with mail surveys and interviewer costs associated with phone or face-to-face surveys. Fast data collection: Data collection is much faster with web surveys as responses are recorded instantly, allowing for real-time analysis and reporting. Disadvantages: Requires internet access: Web surveys are only accessible to individuals with internet access, potentially excluding certain segments of the population. Potential for coverage error: Depending on how the sample is recruited, web surveys can suffer from coverage error if the sample does not adequately represent the target population.

Here are some key details about web surveys: Accessibility: Web surveys can reach a broad audience as they can be accessed from any device with internet connectivity, allowing for a diverse range of respondents. Cost-Effectiveness: Compared to traditional survey methods (like paper or telephone surveys), web surveys reduce costs related to printing, postage, and manual data entry. Quick Data Collection: Responses can be collected in real-time, allowing researchers to gather and analyze data quickly. Anonymity: Participants often feel more comfortable providing honest feedback online, as web surveys can be designed to be anonymous, which can lead to more candid responses.

The main topic to consider when running a web survey is recruitment of the respondents. Depending on the methodology, safeguards need to be built to ensure the relevance of the answers. Screening questions should be in place to validate the profile of the target. In case of incentive (voucher, etc.) so "quality checks" should be considered, typically repeating at the beginning and a the end of the survey a similar question, to check if the response makes sense and remains consistent. Working with a trustworthy market research agency is advised when the stakes are high and the target hard to reach.

Advantages: Highest response rates: Face-to-face interviews generally have the highest response rates compared to other methods, as personal interaction can encourage participation Ability to clarify questions: Interviewers can clarify questions, probe for deeper responses, and ensure that respondents understand the questions Disadvantages: Most expensive: Face-to-face interviews are the most expensive survey method due to the costs of hiring and training interviewers, travel expenses, and the time required for each interview Potential for interviewer bias: The presence of an interviewer can introduce bias into responses, as respondents may feel pressured to answer in a certain way or may be influenced by interviewer's characteristics

Mail: Typically has the lowest response rates, leading to potential nonresponse bias. Phone: Response rates vary but can be higher than mail surveys, especially with callbacks and incentives. Web: Response rates can be comparable to phone surveys but may vary depending on the population and recruitment method. Face-to-Face: Generally has the highest response rates, but nonresponse bias can still occur if certain groups are difficult to reach.

You can always establish chi-squared analyses to determine if different aspects of your methodology impacted response rates. Additionally, you can use ANOVA or t-tests to see if methodological variables impacted results.

When different methods are used in parallel (phone, web survey, face-to-face...) on a similar topic, it should be expected that results will differ. This is not a problem, but the normal consequence of the differences in methodologies. Typically, when assessing customer satisfaction, email surveys will generate more negative feedback than in-platform surveys (pop ups in your website). One is not more or less true than the other, the difference comes from the timing of the survey, and the framing. Mixed-methods approaches should take this into account, and be clear on the goal of each part of the project, to avoid confusion when analysing the results.