Questionnaire design part two: the wrapper
Updated: Sep 17, 2018
Last week we focused on the pitfalls and constraints of your audience and how to treat them better. Understanding of your audiences is key but being able to implement this in a way that allows you to get the right information from the audience is vital.
Surveys typically measure both factual information and subjective states through different question types which all have their specific issues (tune in for next week’s post!). More broadly the flow and method of how the survey is delivered to participants as well as its visual appearance are important to ensure quality data. The survey itself is your key research instrument – a poor questionnaire means the analysis you later do is limited in its use.
There are four key challenges that lie behind any form of survey and how it is designed, and how it can influence participant answers:
Design: both the way the survey is structured online and more visual elements can have a real impact on how participants view and therefore answer online surveys
Response device: a number of device types (laptops, smart phones, tablets etc) will be used to take surveys
Question order: questions can be differently interpreted and answered depending on what precedes and follows them
Explicit vs. implicit measurement: answers gained from explicit questions rather than designed to uncover nonconscious motivations can be very different. We'll cover this area in a separate post, as we have a lot to say!
Let’s look in more detail at each of the areas, and what you should do to combat common issues.
The first thing to remember about design is that there are many elements which essentially come down to personal taste. Therefore, what you are looking to do is create clear and neutral designs, which allow for easy reading.
There are 3 main areas to pay attention to in the visual design of your survey:
Back/foreground design: Keep it clear and readable, with enough white space. Lack of contrast between the background and text can make it difficult to read especially if the font size is small – it’s also good for accessibility. If you are going to use colour be mindful of colour associations, especially across cultures. If in doubt, stick to neutral colours.
Font/typeface: When choosing a font your first consideration should be readability. Beyond that, neutrality is key. Whilst some people like cursive script fonts, they’ll make some people cringe to the point of closing out of a survey.
Layout of questions: In most (but not all) cases avoid grids, but if you have to, try to ensure no scrolling is required. Scrolling vertically or horizontally is very taxing and leads to more drop outs, especially on mobiles. Try to keep multicode lists to one screen, or split them across screens, rather than leaving a list of 50 options on one page.
Surveys were built for desktops, and while we have come a long way in adapting them for laptops, tablets and smartphones it’s often not enough. Long running trackers in particular are hard to adapt and, as an industry, we don’t always make enough effort to do that. New surveys should be written with the end device in mind and if this means having 3 versions of the survey, so be it. Sometimes mobile surveys need to be shorter, so this practice can be a great way to focus on what you really need to ask, leaving out the surplus “nice to knows”.
Before we get down to the detail of how to ask specific questions in an optimal way, we need to consider the order in which we question. You’re probably familiar with the funnel technique (moving from general, less personal topics to more specific, potentially more sensitive ones), but how often do you consciously review the order of your questions?
The sequence in which questions are asked can influence participant answers in two ways:
Serial order effect (location in a sequence of items)
Serial order can operate in at least three ways: by affecting motivation, promoting learning and producing fatigue.
E.g. the beginning of the questionnaire may influence the willingness to respond to the survey, as they shape a participants’ understanding of what the survey is about.
Semantic order effect (location in a sequence of meanings)
Semantic order requires the questionnaire to flow coherently, which usually requires that items on related topics be grouped together.
For example, consider how your answer to both these questions would change, dependent on the order:
Someone who is unhappy with their career might well rate their life happiness level lower if they see the career question first. In some cases, you might want to fix the order of your questions, but in cases like this, where there is a clear bias both ways, then you would probably rotate the order they are asked in (dependent on the rest of the questionnaire).
Although these topics might seem obvious and basic, think about when was the last time you reviewed a tracker or other existing questionnaires consciously and thoroughly? Maybe it’s time for an audit.
These are the three main areas to consider before you start looking at individual questions. Next week we will focus on the main issues with individual question design and how you can mitigate them.
We hope you’ve learned something from this post that you can put into practice. These posts are based on an intensive workshop that we run for clients (often for free!). If you are interested in hearing more about this workshop, and others, please contact firstname.lastname@example.org.