Setup: Implicit Ranking
What is it for?
Getting a highly reliable ranking of a list of statements or images. The statements could be
goals, need states, features that are important in a product category, logo designs, or any
The human brain usually finds it faster and more natural to choose between two options
than to evaluate a single option, so the ranking approach is in general more reliable than a
Motivation Measurement. However it only provides relative agreement rather than
absolute: MM can tell you if 85% of people agree with a statement, but IR can only tell you if
85% of people prefer it to other statements or stimuli in the same list.
What do I need to set up?
You need to specify a question, which will be seen consistently through the whole exercise.
For example, it may be “Which of these words better describes you?” or “Which of these
logos do you prefer?”.
Then, you need a list of goals, need states, statements or other stimuli. The respondent will
see two statements at a time, and quickly choose one. In the first example above, the
statements might be:
The statements should be short, because the respondent needs to read, interpret and
choose between them in a short time, usually between 5 and 8 seconds. All statements
should be reasonably similar in length, otherwise statements that are longer or shorter may
bias the results.
Finally you specify the number of choices that each respondent will make. In most cases, it
is not possible for respondents to see every possible combination of statements: if there are
12 statements, there are 12x11/2 = 66 pairs of statements (or 132 when you take into
account that they could be swapped left and right).
You can choose the length of time allowed to respond to each choice: the default is 8
seconds. The time is expressed in milliseconds, e.g. 8000 milliseconds = 8 seconds.
If the survey will be administered in multiple languages, you should provide translations of
the statements or stimuli into each language, as well as the question.
An instruction screen is usually shown before the exercise. A default set of instructions are
provided but you may wish to update them, for example by putting a customised
screenshot of your own MM instead of a generic screenshot. The instructions should also be
translated if your survey is run in multiple languages.
We recommend no more than 30 choices per respondent. If you have a lot of statements,
you can increase the sample size to make sure that each statement is shown enough times.
For example, if you want to have each statement evaluated 400 times, and you have 25
statements, you could set 30 choices per respondent and choose a sample size of n=400 x
25 / 30 / 2 = 167.
If you have to use longer statements (for example if you are implementing a set of
attitudinal questions for screening or segmentation) you could allow up to 15 seconds, but if
you extend the time much more than this, the time pressure element will become less
If you can use very short, simple statements (for example single words) you can reduce the
time to 5 seconds and make the exercise very efficient.
You can also use images instead of text for the statements – just choose the wording of the
question accordingly. For instance, the question might be “Do you feel positive about this
picture?” if you are testing different imagery for a website. Simple images (like logos) are
usually easy and fast for respondents to evaluate, so you can reduce the time to just 4-5