Quantitative UX Research: What is Hypothesis Testing? How to Test it Step by Step? and Example.
Summary: If you are new to this UX research field or curious to learn about hypothesis testing then this article might be vital for you and In this article, you will be learning the primary stage of what is hypothesis testing in UX research and how to test it step by step with an example.
Definition: A hypothesis is a suggested explanation for a phenomenon. Hypothesis testing is the use of statistics to determine the probability that a given hypothesis is true.
Nature of Hypothesis:
A simple statement of what is supposed to be examined is the hypothesis. Before the study is performed and released publicly in the reporting of the findings, it should be mentioned.
This allows to:
✓ Identify the research objectives.
✓ Identify the key abstract concepts involved in the research.
✓ Identify its relationship to both the problem statement and the literature review.
✓ A problem cannot be scientifically solved unless it is reduced to hypothesis form.
✓ It is a powerful tool for the advancement of knowledge, consistent with existing knowledge, and conducive to further inquiry.
✓ It can be tested — verifiable or falsifiable.
✓ Hypotheses are not moral or ethical questions.
✓ It is neither too specific nor too general.
✓ It is a prediction of consequences.
Type of Hypothesis:
- Null Hypothesis: Null hypothesis is symbolized as H0, These are used when the researcher believes there is no relationship between two variables.
- Alternative Hypothesis: Alternative hypothesis is symbolized as H1 or Ha, is the hypothesis that specifies those values that the researcher believes to hold true and researchers hope that sample data will lead to acceptance of this hypothesis as true.
Steps of Hypothesis Testing:
- Specify the null hypothesis (H0)
- Specify the alternative hypothesis (H1 or Ha)
- Choose the appropriate test (example: chi-square test, T-test, ANOVA, etc)
- Determine your alpha level or Set the Significance level (a)
- Calculate the test statistic and corresponding P-value
- Drawing a conclusion
Few important information you should know before move on to the hypothesis testing.
Population: A population is an entire group that you want to draw conclusions about.
Sample: A sample is a specific group that you will collect data from
Sampling: A sampling method is a procedure for selecting sample elements from a population.
Alpha value or Significance Level: The significance level (denoted by the Greek letter alpha — a) is generally set at 0.05. This means that there is a 5% chance that you will accept your alternative hypothesis when your null hypothesis is actually true. The smaller the significance level, the greater the burden of proof needed to reject the null hypothesis, or in other words, to support the alternative hypothesis.
P-value: The p-value describes the probability of obtaining a sample statistic as or more extreme by chance alone if your null hypothesis is true. This p-value is determined based on the result of your test statistic. Your conclusions about the hypothesis are based on your p-value and your significance level.
Example:
- P-value = 0.01 This will happen 1 in 100 times by pure chance if your null hypothesis is true. Not likely to happen strictly by chance.
Example:
- P-value = 0.75 This will happen 75 in 100 times by pure chance if your null hypothesis is true. Very likely to occur strictly by chance.
- P-value <= significance level (a) then, Reject your null hypothesis in favor of your alternative hypothesis. Your result is statistically significant.
- P-value > significance level (a) then, Fail to reject your null hypothesis. Your result is not statistically significant.
*In order to prove the alternative hypothesis P-value should be less than or equal to significance level (a)
Example: Let’s consider that at Amazon's design team DSP(Amazon Demand-side Platform) because of using the new tool called “Notify” productivity got increased 5% by saving 12,000 minutes a day in our design team and how would you really prove it through statistics that it has a really statistical significance difference because of using the new tool?
- I would suggest to know more about the Notify tool, please click here.
Step 1: Specify the Null Hypothesis (H0): Because of using the Notify tool Amazon design team doesn’t save time in their design process.
Step 2: Specify the Alternative Hypothesis (H1 or Ha): Because of using the Notify tool Amazon design team saves time in their design process.
Step 3: Choose the appropriate test: Here we will be testing our hypothesis by T-test (Dependent or paired one tail).
Why T-test?
Since we are here competing for two continuous variables(Time-on-Task) with the same sample at a different time point(before using Notify and after using Notify) and we are assuming in our H1 that because using Notify has a positive effect on productivity by saving design process time.
Step 4: Determine your alpha level or Set the Significance Level (a): Here the significance level we will be choosing 0.05.
Spet 5: Calculate the Test Statistic and Corresponding P-Value:
so after calculating the test we have gotten the P value=0.03
(I will be sharing a video link soon about how to calculate T-test in a google spreadsheet, till that time I am sharing this below link for your reference.)
Step 5: Drawing a Conclusion: Since our significance level is 0.05 and our P-value is 0.03 which means P-value<Significance level so we will reject our null hypothesis in favor of your alternative hypothesis. So our result is statistically significant. That means there is a significant difference because of using the Notify tool at Amazon design team by saving time in their design process.
Please give me 20 claps if you like it.
Thank you :)
