Validity and Reliability of Qualitative UX Research
Determine the quality of your user experience study and the process.
Summary: A poorly designed research might provide misleading results and destroy your business. Good qualitative research is reliable and valid.
Let’s say you created an impressive UX research report for your new product features, and when you brought it to the management board, they replied, “Your research is not valid.”
I understand the frustration of investing time and energy just to get the result and later find that the outcome is not valuable. I made the same mistake in the past, but it offered an opportunity to learn something new, and here I would like to share some valuable information that will help you justify your UX research results and the research process quality.
Later I realised that doing UX research is insufficient to assist your team and organisation. As a result, we identified another factor: “How can you be confident that your UX research is high quality and people can trust the outcome?” Because incorporating your bad quality research data into your business or product strategy might be really harmful.
“A brilliant solution to the wrong problem can be worse than no solution at all: solve the correct problem.”
― Donald A. Norman,
Now I’ll define two words that describe the quality of your research method and results:
1. Reliability
2. Validity
but what does it mean? ok, let’s understand it deeply.
Reliability
[Good research is consistent from occasion to occasion]
Reliability of a study simply means that you will get the same result if you repeat the study. In other words, findings are not random.
There are four big types of study-design factors:
1. Participant error: Any factor which adversely alters the way in which a participant performs.
Example: For instance, if you do research for your e-commerce website, older people may utilise your product differently due to low vision or other physical/cognitive impediment, therefore low vision or some physical/cognitive impediment is a factor that might lead to participant error.
2. Participant bias: Any factor which produces a false response.
Example: If you do research for your e-commerce website and recruit individuals from the same organisations, participants may be biased during the research process since they don’t want to present the negative aspects of their company. Here people from the same company are a factor that might lead to a participant bias.
3. Researcher error: Any factor which alters the researcher’s interpretations.
Example: After conducting research with five users in one day, you may not be able to ask questions properly with the sixth person since you are exhausted or lacking of energy. As a result, your behaviour may play a role in producing inaccurate research results.
4. Researcher bias: Any factor which induces bias in the researchers’ recording responses.
Example: Sometimes researchers perceive or view things in a different way because they expect a specific outcome and encourage themselves to perceive in that way. As a result, this type of behaviour might lead to researchers’ to get some biased data.
Validity
[Good research is measures accurately what it is intended to measure]
Validity of a study simply means that how accurately you have measures what it claims to measure
There are two big types of study-design errors:.
- Internal validity error
- External validity error
Internal validity error
Internal validity error occurs when a faulty research design leads to a certain reaction or behaviour.
Example 1: If you were doing research for your e-commerce website, you would question your users:
Researcher: “Hello, User 1! Do you believe this design is simple and easy to use?”
User 1: “Yes, I believe I enjoyed it, and it was very simple to use.”
Because participants are conditioned or primed to think of easy when asked this question, they may claim I feel easy to use even if they aren’t.
Example 2: Let’s say you want to see if Design-A or Design-B is the more self-explanatory choice.
If you show each user Design-A first, then Design-B, certain behaviours may arise, and Design-B will most likely be more self-explanatory than Design-A.
While you were demonstrating Design-A, your user participants were already familiar with the testing scenario and the task domain. As a result, internal validity is lacking in this study. (The common solution to this problem is to alternate which site goes first, and have half of the users try site B first.)
External validity error
External validity is about how naturalistic your study is in other words can this research findings be generalised to other relevant settings or groups.
Example 1: If you do research for your e-commerce website to learn how older people use it, but you recruit the audience age range of (25–35 years), the real study data will not be produced by this incorrect sample.
Example 2: Assume you’ve created an application for some field engineers, but you’re bringing them into your usability lab to examine its usability. Do you believe they will use your application in a similar setting?
Not at all, there will be a number of independent variables in the real world that might influence your study in a different way. (Example: Sunlight can effect your design, Reflection is the major reason it is difficult to read a phone screen in bright sunlight)
Conclusion
A research with high reliability but poor validity is one in which you get a pretty excellent measurement of the incorrect item, thus both are necessary.
Now, if someone asks why I should believe your UX research data?
So, as previously said, you must justify one by one why your study is reliable and valid.
In a nutshell, before conveying your research data to the stakeholders, ask yourself these following questions:
- Is there any participant errors?
- Is there any participant bias?
- Is there any researchers error?
- Is there any researchers bias?
- Are you recruiting the actual user or representative of your target audience what you actually supposed to recruit?
- If this research is done by some other researcher in the same environment with new user (but the same type of user group), will it produce the same outcome?
- Can you generalised your research data? in other word if you do it in their (target user group) natural environment, will it produce the same result?
Well, now you have got enough information about a good quality of ux study.
Please comment below if you have any doubts or suggestion.
All the best and Happy Research!
Please applaud if you like it and don’t forget to follow me for your own benefit.
Thank you :)
