Questionnaire design part one (of three): Be kind to participants
Updated: Sep 17, 2018
This is the first in a three-part series covering some best practice hints and tips for online questionnaire design.
Questionnaire design is one of the core skills of every researcher, and most of us think we’re pretty good at it. However, when you’ve been doing something for a long time, it’s easy to fall into habits and take some things for granted. Response rates are falling at an alarming rate, which means we need to take more care of our participants.
We’d love for the whole industry to do cutting edge, implicit research all the time, but sometimes there isn’t the time, budget or need. However, that doesn’t mean that more traditional surveys cannot be improved with a little bit of time and effort.
Over the next three weeks we will lay out key points to consider when designing a questionnaire, based on our popular workshops. There are 3 areas to consider:
1. Your audience: can the people you are speaking to do what you are asking them?
2. Your survey: how do the design, the device, the order, measurement type influence participants?
3. Your questions: how might answer order, scales, frequency and wording bias the answers?
Participants are often overlooked in research - in many cases, surveys are bordering on a traumatic experience for the people taking them. When designing a survey we often focus on giving our clients what they want (which of course we should focus on, as we all want to get paid). Our next priority will be making lives easier for ourselves (think massive grids for easy DP and programming). Then, if we have time, we might consider participants. Of course, there are always trade-offs to be made, but we should always remember survey design is a balancing act between 3 groups of people:
Don’t forget the participants! There are three key kinds of challenges that need to be tackled for participants to provide us with high quality responses from their questionnaires:
1. Decision making constraints impact how people make decisions:
Attention: we cannot process everything we see at the same time - the mind only processes the most important parts
Calculation: we cannot accurately calculate all of the outcomes and consequences of our choices
Myopia constraint: we focus on the here and now, meaning we can only act on how we’re feeling in the present moment.
2. Research barriers influence participants’ responses in the context of research:
Self-knowledge: we don’t always know everything about ourselves to accurately answer survey questions.
Honesty: we might not want to tell a researcher everything about ourselves because it’s embarrassing or painful.
Memory: our recall of past events and experiences can be influenced by characteristics of both the event being recalled and the question asked
3. Survey specific barriers influence participants’ answers in online surveys:
Comprehension: the language used in questions as well as people’s cognitive abilities determine how well people understand survey questions
Cognitive effort: participants may tire and stop investing cognitive effort into surveys, resorting to 'satisficing' (just giving the bare minimum) in the responses they provide.
It is important to be aware of these constraints and barriers when designing questionnaires. Here is a useful list of questions to help you think about the audience you are writing for:
Who is your audience? What sorts of constraints and barriers will apply to that group in particular?
Is the survey too long? Will it realistically hold the attention of your audience?
Can they understand the questions you are asking?
Can the participant be realistically expected to know the answer, and to know themselves well enough?
Does the question ask participants to predict future behaviour?
Does it rely on participant memory?
Will the participant tell the truth?
Would you want to answer these questions? Would you spend 15 minutes of your time doing this?
These questions may seem obvious but when we look at many of the surveys out there, it’s clear our industry is not giving participants the thought and consideration we should. Next week we will be focusing on how to approach the global design of your surveys, before diving into specific question designs in week 3.
We hope you’ve learned something from this post that you can put into practice. These posts are based on an intensive workshop that we run for clients (often for free!). If you are interested in hearing more about this workshop, and others, please contact firstname.lastname@example.org.