Vocalabs Newsletter: Quality Times


This Is Why Survey Design is Hard

In This Issue

This Is Why Survey Design is Hard

Some people will immediately spot the problems I saw with this online survey I got from Discover Card a few months ago, especially those who have some knowledge of User Interface design.

Screen capture from Discover Card survey

Someone thought it would be cute to have the selection buttons shade from red to green. The first problem that jumped out at me is that the grey in the middle for "Neutral" makes it look like the "Neutral" button is disabled (in the screen shot, "Neutral" is selected, which is why its circle is filled in rather than open). It's become a standard part of user interface design to indicate that a control is disabled by greying it out, so at first glance some users might think that Neutral isn't actually an allowed option on this survey.

That's something that could affect the outcome of the survey at the margins. Does it? I have no idea--and I'm guessing that Discover Card didn't calibrate the survey to see if their color choices make a difference. But it's certainly plausible, which is one reason survey design is hard. So many things might affect the results of a survey that you need to be careful to understand the design choices, and make sure that the analysis and decision-making process is robust in the face of subtle survey biases.

There was another problem I spotted with this survey, one which most people won't notice but which 7%-10% of the male population will immediately see (or rather, not see). The particular shades of red and green used in this survey are  hard to distinguish for people with the most common form of color blindness. So for me, and a significant minority of the population, whatever Discover meant to communicate through the colors of the buttons is completely lost because we can't easily tell the difference. There are color palettes out there designed to be accessible to colorblind people, and this is another important detail that good User Interface designers know to watch out for.

(Those with normal color vision will probably be surprised to learn that before I wrote this article, I actually used Photoshop to verify that the colors in the image really are red and green. I really do have that much trouble telling the difference between those particular shades.)

I sometimes like to say that survey design isn't hard, what's hard is dealing with the people who think it's easy. It's not hard to design a reasonable survey, but there are a number of details which can change the outcome. But because designing a survey looks easy, often people will want to make changes without thinking through the implications. This survey is a great example of a seemingly-trivial design choice which might actually impact the data, and which clearly isn't necessary.


Language Log has coined the word "Nerdview" to describe the common mistake of writing in technical terms for a non-technical audience. Nerdviews are all around us, to the point where we hardly even notice them anymore.

For example, when an online store asks you to input the "CVV2" on your credit card, that's a classic nerdview. Unless you're in the payment industry you would have no reason to know what a CVV2 is or how to find it. This is so common that most people have figured out that the CVV2 is the three-digit security code on the back of a credit card--even though most cards don't label the code "CVV2" or give any other indication of what it's for.

Nerdviews almost always lead to a worse customer experience, and should be avoided whenever possible. But oftentimes the people who decide how to communicate important information are the same ones who are experts in their own narrow field. It's hard to step outside your expertise and think like a novice.

My personal favorite nerdview was one I encountered many years ago in a customer feedback project. Our client was a life insurance company, and in their automated customer service system one of the prompts asked, "Do you want to know your withdrawal value or your redemption value?" Unless you are in the insurance business, chances are you don't know that a life insurance policy can have two different values, much less what they mean.

Finding and avoiding nerdviews is almost always worthwhile, but it can be a challenge. Nerdviews tend to become invisible to us once we figure out what they mean. Humans are adaptable, and even if you were mystified the first time a government form asked you for your "DOB", chances are the next time you remembered that it meant "Date of Birth" and hardly even noticed.

Here are some tips for uncovering nerdviews:

  • Don't assume your audience thinks like you, or that you can put yourself in your shoes. Get feedback from actual customers or users.
  • Remember that people adapt to nerdviews quickly, but may ignore or overlook important information if it's not clear. Don't assume that you're communicating well just because your customers or users seem to have figured things out.
  • Communicating clearly is usually more important than technical precision. If you feel compelled to be technically precise when communicating to a non-technical audience, you may be trapped in a nerdview.

Even if you don't think you're a nerd, we are all nerds about something. Remember your audience, and stay away from the nerdview.

Newsletter Archives