How do you incorporate qualitative feedback into your testing hypothesis formulation?

L'hypothèse doit clairement expliquer le changement de comportement attendu de la part du client ou du prospect, en plus de la variable modifiée et du résultat attendu.

Incorporating qualitative feedback into testing hypothesis formulation involves integrating insights gained from subjective, open-ended responses into the design of testable assumptions. This process entails extracting themes or patterns from qualitative feedback and translating them into specific, measurable hypotheses that can be systematically tested. A testing hypothesis is a formal statement that articulates a predicted relationship or effect between variables, providing a clear framework for empirical investigation and analysis within research or experimentation. When incorporating qualitative feedback, researchers use it to inform and refine the specific elements of their testing hypotheses.

Las hipótesis de prueba también pueden ayudar a las empresas a identificar oportunidades de mejora. Al pensar en cómo los cambios pueden afectar los resultados, las empresas pueden identificar áreas de su sitio web, aplicación o producto que pueden mejorarse. Por ejemplo, una empresa que está experimentando una baja tasa de conversión puede formular una hipótesis de prueba sobre cómo el cambio de diseño de su página de destino puede mejorar la tasa de conversión. Si la hipótesis se prueba y se demuestra que es correcta, la empresa puede implementar el cambio en su página de destino para mejorar los resultados.

A testing hypothesis outlines your expectations for how a change will influence user behavior or key metrics. It sets the stage for systematic experimentation by clearly stating the intervention, the expected quantitative outcome, and the reasoning behind the anticipated effect. This structure ensures the hypothesis is actionable and grounded in a logical prediction.

There are certainly a lot of templates and ideas on how to formulate a hypothesis. I think ultimately a hypothesis has many different functions. It can be considered a decision-support to avoid p-hacking, etc. It can be considered to help documentation to boil down the entire idea in one sentence in a standardised way and it can also be an alignment tool to help all stakeholders understand what you are trying to do.

A simple framework we start with for our hypothesis is the following: If .... then ... as measured by ... I.e. if we change the language of landing page to be directly related to the product a user clicked an ad for, then we will reduce customer confusion as measured by a decrease bounce rate. With a comprehensive hypothesis, I'm make it easy for stakeholders and strategists to align on the problem our users experience, how we plan on resolving it, and what the success will be determined by. This aligns everyone's expectations and removes variability in our interpretation of results.

Hypothesis testing is a way to test the results of an experiment to evaluate if there is an impactful result. The goal is to insure that the results of the experiment have not happened by chance. Example We need to rollout a new feature on the checkout page to add a text box for promotion code. We believe that adding this new feature will increase number of online purchases. Null Hypothesis — There IS NO significant difference between the no. of online purchase transactions from the traditional checkout page and the new checkout page with the new feature Alternate Hypothesis — There IS a significant difference between the no. of online purchase transactions from the traditional checkout page and the new checkout page with the new feature

Ein Fallstrick ist hier häufig, dass zuerst die Testidee generiert wird und DANACH die Hypothese dazu. Es funktioniert aber genau umgekehrt.

Integrate qualitative feedback by identifying recurring themes and insights from user interviews or surveys. Use qualitative data to enrich testing hypotheses by understanding user sentiments, preferences, and pain points. Validate hypotheses by testing against qualitative feedback, ensuring alignment with user expectations and needs.

While qualitative feedback from users might help us size the opportunity of the test, it’s important to also collect feedback from core business stakeholders to determine if a given intervention’s hypothesized lift is worth pursuing or not, i.e. whether or not it is practically significant. A lift metric should be large enough so that we can detect a statistically significant change in behaviour, but also it must align with stakeholder goals such that if statistical significance were detected, it would be meaningful enough to the business to warrant rolling out the change.

Bei qualitativem Feedback durch eine Käuferumfrage beispielsweise ist vor allem der ZEITPUNKT der Befragung entscheidend. Umso länger der Kauf her ist, umso weniger wertvoll ist das Feedback.

Qualitative feedback can be collected through various methods such as interviews, focus groups, open-ended survey questions, observations, and social media listening. Interviews allow for in-depth conversations to gather detailed insights. Focus groups bring together a small group of participants for discussions. Open-ended survey questions give respondents the freedom to express their thoughts. Observations involve directly witnessing behavior or interactions. Social media listening entails monitoring discussions and comments on digital platforms. Each method offers unique opportunities to gather rich qualitative feedback that can inform decision-making and hypothesis formulation.

Es importante tener en cuenta que no existe un método perfecto para recopilar comentarios cualitativos. Cada método tiene sus propias ventajas y desventajas. Las empresas deben elegir el método que mejor se adapte a sus necesidades y objetivos. También es importante recordar que los comentarios cualitativos deben interpretarse con cuidado. Los comentarios pueden estar sesgados o incompletos. Las empresas deben utilizar una variedad de métodos para recopilar comentarios cualitativos y comparar los resultados para obtener una imagen más precisa de las necesidades y preferencias de sus usuarios.

There are many different ways. A nice distinction comes from Refroge where they differentiate between qualitative observational feedback that you can use to understand the behaviour of users (e.g. Heat Mapping Tools, etc.) or qualitative conversational feedback that you can use to understand the attitude of users (e.g. interviews, etc.) Deciding which method to use is very important and an important skill for PMs. But the question often more: "How much discovery is enough?" It should also be pointed out that often there is a lot of feedback already collected in your company. This might be customer support tickets or sales calls or the sales agent at your company that you can interview.

Depending on the size of your organization, you may have the following method available for collecting qualitative feedback: 1. NPS surveys - typically highly structured, designed to be consistent across many touch points to measure Net Promoter Score at enterprise level. 2. In experience feedback surveys: pop up surveys designed to collect feedback in context of the user experience/feature/product flows 3. Using quick testing tools like UserTesting.com to validate new design/concepts 4. In-depth lab/on-site/in-person surveys - Typically user research team will interview customers for detailed feedback prior to launching major releases 5. User replay tools like Quantum metrics, GlassBox, Content Square, Tealeaf to view the user session

To effectively gather qualitative feedback, diversify your sources to obtain a well-rounded understanding of user experiences. Consider implementing interactive tools like live chats or feedback widgets on your website, in addition to traditional methods like surveys and interviews. These tools can provide real-time insights into user behavior and preferences, helping you identify areas for improvement that might not surface in more structured data collection methods.

A self service method of qualitative feedback is the use of session playback tools (i.e. Fullstory and Qualtrics) as they provided a quantitative measure of customer pain points, with a holistic view of where customer breakage points (qualitative data). A well designed user study with specific objectives will allow you to delve into how a specific demographic of users will be helpful in answering questions for your product lines and across your properties. Finding a mix of human insights and data will paint a better picture of your customers and provide ideas of how we can optimize our user experience through the site.

User calls can be an excellent way to really understand the qualitative nuance needed to form hypotheses that go beyond the quantitative data. I generally aim to talk to 10 customers in a specific audience in order to get a wide enough sample of quantitative opinions. By summarizing these results, you can draw conclusions to add into your testing strategy.

Qualitative feedback is super valuable in hypothesis creation. Users’ comments combined with a session replay and quantitative data can be all developers need to fix a bottleneck in consumer’s journey. Don’t discard user’s feedback just because of low number of responses. You can often compliment it with other research methods. Remember about the integrations between the tools. That’s where the magic happens 💫

There are many ways to collect feedback. I would encourage to use more modern tools like discussions on producthunt or reddit (see "gummysearch") to get the view from innovator personas. Often the feedback that is already in the company through sales recordings or customer service interactions is often highly underused.

Analyze qualitative feedback by identifying common themes, patterns, and sentiments. Use techniques like thematic analysis or sentiment analysis. This involves categorizing the feedback, noting frequency of certain types of comments, and interpreting the underlying reasons and emotions.

Es importante tener en cuenta que el análisis temático es una interpretación de los comentarios de los usuarios. Es posible que los comentarios sean sesgados o incompletos, por lo que es importante interpretarlos con cuidado. Al realizar un análisis temático, las empresas deben tener en cuenta los siguientes factores: El propósito del análisis: ¿Qué se espera lograr con el análisis? El tipo de comentarios: ¿Qué tipo de comentarios se están analizando? El tamaño de la muestra: ¿Cuántos comentarios se están analizando? Las habilidades y recursos disponibles: ¿Qué habilidades y recursos están disponibles para realizar el análisis? Al considerar estos factores, las empresas pueden realizar un análisis temático que sea significativo.

I worked with many different tools from grain.com over EnjoyHQ to Dovetail. I would consider these tools more user research tools. But I also worked with more science-related tools like MAXQDA or NVivo. The coding of the parts of the interview is a very interesting skill. I think AI can help a lot and ultimately also shows us that even here when using a method like grounded theory and so on, we can still influence the result that we want to achieve.

I think it is very valuable to use a tool like Dovetail and centrally store all the qualitative feedback. When done correctly and maybe even supported by ResearchOps we can leverage a lot of valuable insights that can then serve as a basis for many hypotheses.

Analyzing qualitative feedback involves organizing, coding, and identifying themes within the data. This can be achieved through methods such as content analysis, thematic analysis, and narrative analysis. Content analysis focuses on identifying patterns within the textual data. Thematic analysis involves systematically organizing and interpreting patterns of meaning within the data. Narrative analysis examines the stories and accounts provided by respondents to identify recurring themes. Utilizing these methods enables a comprehensive understanding of qualitative feedback and facilitates the extraction of valuable insights for decision-making.

To generate hypotheses, qualitative data such as customer feedback, surveys, or maps can provide valuable insights. This data helps identify user problems and needs, forming the basis for hypotheses that can be tested later. However, it is crucial to validate these qualitative insights with quantitative data. By analyzing metrics like conversion rates or bounce rates, one can verify if the identified issues are significant and if the proposed changes could lead to measurable improvements. Combining both data sources ensures well-founded, actionable hypotheses and enhances the decision-making process in A/B testing.

After analyzing qualitative feedback, identify the most impactful insights that align with your strategic goals. Transform these insights into testable hypotheses by focusing on specific user complaints or suggestions. For instance, if several users suggest that a simpler checkout process would enhance their experience, hypothesize that streamlining the number of steps could improve conversion rates. Back this up with qualitative anecdotes and correlate with quantitative data where possible to strengthen your hypothesis before testing.

Formulating testing hypotheses from qualitative feedback involves extracting key insights and patterns from the feedback to inform the creation of specific hypotheses for testing. This process typically includes identifying recurring themes, common pain points, or areas of improvement mentioned in the qualitative feedback. By closely analyzing these elements, it is possible to formulate precise and targeted hypotheses that address the concerns or suggestions highlighted in the qualitative feedback. These hypotheses then become the basis for designing and conducting relevant tests to validate or refute the insights derived from the qualitative data.

As noted by question 1 on hypothesis, our core for testing is to solve a user pain point or to optimize their experiences. Qualitative data broaches the conversation. Then we use data to quantify this observation and analyze the business impact. When an opportunity is presented, we draft potential solutions as a team and thereby create a comprehensive hypothesis. If...we offer this solution for users...then...we will address this pain point...as measured by a key metric. This is how you're able to get vested interest across your UX, Analytics, Strategy, Product, and Sales teams and report on insights learned about your customers.

Es importante tener en cuenta que las hipótesis de prueba no siempre son correctas. Los resultados de las pruebas A/B pueden demostrar que la hipótesis es incorrecta. Sin embargo, incluso si una hipótesis es incorrecta, aún puede proporcionar información valiosa sobre las necesidades y preferencias de los usuarios. Al formular hipótesis de prueba, las empresas deben estar preparadas para aprender de los resultados, incluso si los resultados no son los esperados.

This is the part where creativity and innovation really plays a role. This is also what Stefan Thomke explains in "Experimentation Works". It is the skill of the product manager or the experimentation lead to find a hypothesis that can validate a certain idea or a certain behaviour that we are assuming. Qualitative feedback is just one source of insights that we can use to formulate them. Ultimately it comes down to externalise our assumptions and then create a hypothesis that we then map to an experiment.

The qualitative feedback that you collect flows directly into the hypothesis. The hypothesis is merely a way to express all the assumptions that you have about your customer base or the market. Once you understood the underlying assumptions, formulating the hypothesis is a step to make the different hypotheses comparable and also rank and prioritise them.

Eine Struktur die sich gut eignet ist folgende: Wir erkennen/beobachten, dass [...] Indem wir [...] werden Nutzer [...] und die Kennzahl [...] PS: Immer zuerst die Hypothese, danach das Testdesign.

When using qualitative feedback, avoid confirmation bias by staying open-minded. Be cautious of overgeneralization and ensure all feedback is considered, even dissenting opinions. Provide context to the feedback and establish clear interpretation guidelines to maintain consistency. These steps help mitigate potential pitfalls of qualitative feedback analysis and improve its effectiveness in decision-making processes.

As already written in the comment above, qualitative findings should always be verified by quantitative data in order to see their validity and also their potential effect.

When used in hypothesis formulation, qualitative feedback reveals what users tell us, but should always be used in conjunction with quantitative insights — which reveal user behaviours based on what users aren’t vocalizing explicitly. By only relying on qualitative feedback, we run the risk of response bias, or designing for the needs of the most vocal users only. This can be valuable when designing tests that address the sentiments of specific user segments, but it’s important not to generalize these hypotheses to the behaviour of your overall user base.

I think the Confidence Meter that was introduced by Itamar Gilad illustrates this quite nicely. Most qualitative feedback does not create a lot of confidence, meaning it does not take a lot of risk away from a product decision. The biggest pitfalls is then to not be aware of the fact that most qualitative feedback sits lower in the hierarchy of evidence as for example A/B testing results, etc.

Las empresas pueden tomar las siguientes medidas para mitigar los riesgos de usar retroalimentación cualitativa para generar hipótesis de prueba: Utilizar una variedad de métodos para recopilar retroalimentación cualitativa. Esto ayudará a obtener una imagen más completa de las necesidades y preferencias de los usuarios. Analizar la retroalimentación cualitativa con cuidado. Busque patrones y tendencias en los comentarios. Validar las hipótesis de prueba con pruebas A/B. Las pruebas A/B pueden ayudar a determinar si las hipótesis de prueba son correctas. Al tomar estas medidas, las empresas pueden mejorar la precisión y la fiabilidad de sus hipótesis de prueba.

We should definitely be aware of our biases here. When we are in interviews and hear a customer talk about a challenge or a problem it is just really sticky but in reality it is just one anecdote and we should treat it like this. Getting to hung up on one story/evidence is quite common, thou.

Qualitative feedback helps understand why behind the what/quantitative data. Typically start with quantitative data to identify opportunities e.g. low conversion, high churn, low feature adoption etc. And then apply qualitative feedback in the context to understand the drivers behind the problem you are solving. Also consider source of qualitative feedback. e.g. an evergreen survey link on the site typically attracts negative comments as unhappy customers are more motivated to express frustration proactively. While a prompted survey after customer service calls may be less biased. Keep the context in which feedback was collected.

One of the most valuable ways to use qualitative data while A/B testing, is while iterating on an already running test. While the data from the test can tell you what is happening, most times watching user recordings for your new and losing variants, can explain why users behave the way they do. That's why it's important to make sure you'll be able to identify testing variation on your recording tool, prior to launching a new test.

Qualitative data can be overwhelming to sift through. Using AI / tools to build themes of issues that can be drilled down to specific feedback can be effective way to digest the information in a structured manner.