Survey Priming –How Your Question Order is Giving Your Survey Bias

We all know the dangers of bias. Whether it is caused by the respondent or the researcher, bias has the potential to not only ruin your survey results but to also send your decision makers on the wrong path. What most survey creators don’t take into account is that survey results can become biased through the ordering and structure of a question list. This particularly sneaky form of survey bias is called priming. Today we will take a look at exactly what survey priming is and some tips on how you can avoid it in your research.

Survey Priming Defined

The way a person reacts to something can be heavily influenced by what they have been exposed to recently. This is the premise for advertisements created to for impulse buys. Just think of this the next time you are tempted to buy a chocolate bar at the cash of your corner store. How much of your decision to buy that tasty treat is influenced by the commercial last night showing off the creamy caramel, crunchy peanuts, and chocolate? These commercials have been specifically designed to get you in the mindset to make that purchase.

This tactic is called priming. Basically, priming occurs when a person is exposed to a stimulus, and it influences how they respond to another stimulus. In surveys, priming is not only a bad thing it’s also very tough to avoid. As the respondent answers questions, they will be carrying over those thoughts when they interpret the next question. Let’s look at an example:

QUESTION 1

QUESTION 2

As the respondent answers question 1, they are familiarizing themselves with many of the different resources offered by FluidSurveys. When the respondent gets to question 2 they have been primed to agree more strongly. If the respondent answered question 2 first, their response would more accurately reflect their perception of FluidSurveys’s resources. This is a more blatant example of how priming can effect a survey, but most priming is a lot more subtle and harder to counteract in your survey design. But there are several ways to limit priming when creating your questionnaire.

Tips to Avoiding Priming

1) Order Deliberately: The order of your questions directly impacts the inherent priming bias found in your survey. Take the example above, if we switched the order in which the questions appear, most of the priming bias would be eliminated. The reason for this: ‘Question 1’ is a behavioural based question and ‘Question 2’ is an opinion based question. No matter what the respondent reads, the different types of resources he/she has used in the past will not change.
Keep this in mind when you are ordering your questions per page. Ask yourself which questions are opinion based and place these near the top. That way your respondents can provide their unbiased feelings before giving more factual information.
Also remember to ask your broader opinion questions first and become more specific as the survey goes on. For example, let’s say you wanted to know your respondents’ satisfaction with your company overall and then their satisfaction with specific areas of your company. If you ask about a specific product first, their feelings towards that product will weigh heavily on their answer about the company overall. Instead, ask about the company overall first and then break down into your more specific areas, like product, services and support.

2) Separate Topics into Different Pages: When dividing your question list into topics, it is important to give each different topic their own page. Most respondents treat a new page as a blank slate, erasing from their mind questions on previous pages and assessing the new page as an isolated section. This mental reset can be even more effective if a title is given at the top of the page indicating that the topic has changed.

3) Better Define Your Questions: If you think the previous answer may lead respondents to answer the next question in a particular way, provide extra text that lets them know they should be looking at the question by itself. Let’s say you were running a website feedback survey. In the survey you ask about the site’s ease of navigation, but then want to add a question about the usefulness of the information your site provides. You should be direct about what this question is meant to address. It might be useful to phrase the question like this:
Thinking only of the information provided on our site, how well does our content meet your needs?

4) Randomize the Order of Questions: As we know, priming comes from question ordering. If there is no required order of questions in your survey, it might make sense to randomize the order of those questions. The random ordering will limit any form of priming, giving each question an equal chance of being first read. Here is a video tutorial on how to set this up on FluidSurveys:

Don’t have an account yet? Get started by contacting us or visiting our pricing page!

FluidSurveys Presents

Free Survey Q&A

Join our survey & research expert Rick Penwarden as he answers all of your questions every Wednesday at 1PM EST!


Leave a Reply

Your email address will not be published. Required fields are marked *