Odd or Even? –The Ongoing Debate of Neutral Rating Scales

No, we’re not talking about roulette at the casinos! We’re talking about the number choices provided on a survey question. Since the creation of the rating scale, there has always been disagreement on whether closed-ended survey questions should have a neutral category or should force respondents to take a side. By creating a scale with an odd number of categories, a researcher will be leaving a mid-point, which acts as a neutral option for respondents to select. Without it, respondents will be forced to pick an option on either the lower or higher end of the rating scale. There are different schools of thought on whether an odd or even number of categories is best, but in this article we will discuss the strengths and weaknesses of both.

The Power of Fence Sitters: Why Neutral Responses are Important!

In most situations, having an odd number of categories is usually the more effective way of gathering accurate data. The fact of the matter is many people are legitimately neutral on a subject. Forcing respondents to answer a question on an even scale will bias your end results as truly neutral people must select a category that does not truly represent their opinion.

The question below is a typical likert question asking people to provide their level of agreement with a statement on potato chips. As you can see, the scale has an odd number of categories with a neutral point in the middle. Each category has been allotted a score number, the higher numbers favouring barbeque and the lower numbers favouring salt and vinegar. Taking away the middle category would force respondents to choose between the two flavours, even if they truly do not have a preference of chip type.

Many argue that they can avoid the survey bias caused by an even number of categories by providing an opt-out category for neutral respondents. This means adding a ‘No Opinion’ or ‘N/A’ option at the end of your scale that, if selected, will eliminate that response from the collected answers. However, this would actually result in a more extreme bias in your data analysis. Let’s take a look at why.

Say we reworked our potato chip question to be as follows:

The problem lies in the difference in scoring. The power of neutral responses has been eliminated from the data with the label ‘No Opinion.’ But with this change, we will see a dramatic effect on the intensity of the sample’s overall preference. Let’s say the results for our survey question were as follows:

Strongly Agree = 13 responses
Agree = 22 responses
Neutral = 32 responses
Disagree = 10 responses
Strongly Disagree = 5 responses

If we calculated the average score of our respondents’ agreement with the potato chip statement we would get 3.3 (3=Neutral), but if we used the 4 category rating list and moved all neutrals into the ‘No Opinion’ category, our score would transform to be 2.9 (3=Agree). Evidently, this difference in scoring completely changed the outlook of our results. Omitting neutral answers biases the data to be more in favour of barbeque chips, when in actuality, our sample is much closer to being neutral on the subject.

The other problem with using an opt-out strategy is that some neutral respondents feel obligated to pick a side if there is no midpoint on the scale, whereas others will choose to opt-out. Without being able to control this, the researcher would be at the mercy of respondent discretion, having answers for the same opinion (neutral) in different categories of the rating scales. Instead, the safest route is to keep any normal rating scale odd and therefore safe for the fence sitters.

When to Kick ’em Off the Fence!

However, there are times when an even rating scale can be more effective than one with a neutral point. Sometimes a decision maker will be faced with the dilemma of choosing exclusively one option over another. Say our potato chip company has to discontinue one of its two flavours. As a researcher, we have to acknowledge that the results must fall on one side or the other of the scale. In this case our neutral respondents truly wouldn’t matter to our results. Here is an example of how we can set up our question and its scoring:

In the question above we see that there is six categories and an opt-out choice. The reason for the extra categories is that we want people to make a selection even if they only slightly prefer one brand over another. Making categories with adjectives like “Somewhat,” “Moderately,” or “Slightly” allows fence sitters to take a stand on the question without completely forsaking the other side. Of course, we still have to add an opt-out choice, because we do not want to force a decision on our respondents who are truly neutral.

Moreover, by giving six options of varying degrees of preference, our results will give us both the ability to see the absolute number of respondents who prefer each flavour, as well as a closer measure of the intensity of their preference. The absolute number of respondents’ preference would be calculated by adding the total of respondents that fall on either side of the scale, whereas intensity of preference would be measure by adding the sum total of all response scores. In our example above, the more negative the number means a larger preference for salt and vinegar and the more positive the number means a larger preference for barbeque.

Where do Your Respondents Stand?

We learned that odd and even rating scales have different uses. Odd scales have neutral categories, which serve the dual purpose of eliminating the bias caused by fence sitters and providing more accuracy to your data analysis. This makes odd numbered rating scales the standard for surveying. However, when a decision maker finds themselves in a fork in the road, having to choose between two options in order to move forward, it can make sense to have an even number of categories in your scale.

So now it’s your chance to put this into practice. Create your own FluidSurveys account by visiting our pricing page and design your own rating scales today!

FluidSurveys Presents

Free Survey Q&A

Join our survey & research expert Rick Penwarden as he answers all of your questions every Wednesday at 1PM EST!


5 Comments

  • Ana says:

    If you wanted to use the item “Please select the choice that best reflects your preference between barbecue and salt and vinegar potato chips:” for analysis would you exclude the neutral responses or would you assign it a number such as 0 ?

    • RickPenwarden says:

      Hi Ana!
      If you are talking about the last question item, we actually removed the neutral response because, in this case, neutrality serves no purpose for the decision maker. Either the company will choose to continue with barbecue or salt and vinegar chips. In this question the truly neutral people must give the response ‘Indifferent’. The indifferent option should then hold no score because we don’t want these responses affecting our results. Say we did give ‘Indifferent’ a score of 0, any time we decided to do an average score, our results would be affected by the ‘indifferent’ responses.
      I hope this explanation helps!

  • Kripankar Ali says:

    Rick, How did you arrive at average scores of 3.3 and 2.9?

    • RickPenwarden says:

      Hello Kripankar!

      The average score of 3.3 and 2.9 comes from adding all responses weighted by their scores. In the case of the 3.3 average score:

      Strongly Agree = 13 responses x score 5 = 65
      Agree = 22 responses x score 4 = 88
      Neutral = 32 responses x score 3 = 96
      Disagree = 10 responses x score 2 = 20
      Strongly Disagree = 5 responses x score 1 = 5
      Sum of all Scores = 274 / Total Responses = 82
      Average score = 3.34146

      In the case of the 2.9 value, I remove the neutral. This will change the scoring that is given to each category:

      Strongly Agree = 13 responses x score 4 = 52
      Agree = 22 responses x score 3 = 66
      Disagree = 10 responses x score 2 = 20
      Strongly Disagree = 5 responses x score 1 = 5
      Sum of all Scores = 274 / Total Responses = 50
      Average score = 2.86

      As you can see, the removal of the neutral responses will have a significant impact on the results of the data. Hopefully this helps! Thanks for your question!

  • RickPenwarden says:

    Hi LD71!

    That is a very good way to limit the survey bias in a question.

    I would argue that using multiple monadic ratings is less effective in comparing two opposing objects than using one scale that forces respondents to show their allegiance one way or the other. Consider the fact that customers usually have to make a decision as to which flavour of chips they will buy. Asking a question that makes them choose between the two options on a survey better mimics their experience while shopping at a store. To me this benefit outweighs the extra work on the back end for scoring.

    Unfortunately I have yet to write a university article discussing monadic rating and comparison ratings (AKA: Noncomparative and Comparative scaling). But you have inspired me! I am writing it into my big list of blog ideas!

    Thanks for the input! 🙂

Leave a Reply to Kripankar Ali Cancel reply

Your email address will not be published. Required fields are marked *