The Customer Service Survey
VocaLabs U
Consumerist on Good and Bad Service Factors
Tue - March 18, 2008 01:00 PM
A few weeks ago, I wrote about our survey of good and bad customer service factors, the things which customers feel are important in the quality of their customer service experience.
Ben Popken, the excellent editor of The Consumerist, thought it would be interesting to offer the same survey to his readers. So he did (with my blessing).
After running for several days, he got around 5,000 responses from Consumerist readers (as compared to the 400 we collected from VocaLabs' consumer panelists). The average participant on the Consumerist survey chose only one option on each survey question, in sharp contrast to the VocaLabs panel where the average participant selected about half the options on each question. This suggests that the VocaLabs panelists took more time to answer the question carefully, whereas the Consumerist readers mostly just chose their top answer and stopped there.
But after normalizing the results to adjust for the different number of responses per participant, I found that the survey results were quite similar. On the list of "Good" customer service factors, the differences between the VocaLabs survey results and the Consumerist poll were generally within the margin of error, except that many fewer Consumerist readers chose the "Keep the total call as short as possible" option. My best explanation is that the "short call" factor was rarely people's most important consideration, but often a second- or third-choice option.
There were more differences in the list of "Bad" customer service factors, though most of the differences between our survey and Consumerist readers were still within the margin of error (again, after normalizing the Consumerist results to adjust for the different number of responses per participant). Consumerist readers chose the "The company doesn't do things it promised to do" option relatively more often than VocaLabs panelists, and were less likely to choose long hold times, rude agents, or hard-to-use IVR as important considerations.
Given the differences between the way these two surveys were run, and the fact that the Consumerist ran a wide-open online poll, it's perhaps surprising that the results are as similar as they are. That suggests that consumers are fairly consistent about which customer service issues they care about today.
Posted by Peter Leppik
Factors for Good and Bad Service
Mon - February 25, 2008 02:58 PM
We just finished a Consumer Attitudes survey about what factors are important for good and bad customer service. We're using this data to help refine the metrics we generate on our Service Quality Tracker surveys, and I think a lot of readers will be interested in the results, too.
The survey involved 400 people drawn from our consumer panel, and had a margin of error of +/- 5 percentage points.
One of the questions we asked was which factors people thought were most important in providing good customer service over the phone. They could select as many choices as they wished from a list (the order of the list was random for each participant to avoid bias). The answers were:
1. Make it easy for you to reach a live person (if necessary) 82%
2. Be courteous, polite, and professional 75%
3. Wait only a short time to talk to a human (if needed) 69%
4. Handle your question or problem on the first call 63%
5. Be willing to transfer your call to a supervisor if the agent isn't helping 59% (statistically tied with 4th place)
6. If you have to call more than once for the same problem, be able to talk to the same person each time 49%
7. Follow up (if needed) to let you know that your problem was solved 45% (statistically tied with 6th place)
8. Keep the total call as short as possible 32%
9. Provide an easy-to-use self-service option (where possible) 31% (statistically tied with 8th place)
10. Do more to help than you expected 31% (statistically tied with 8th place)
We also asked participants to choose which factors they feel are most important in providing bad customer service over the phone--as with the good service question, they could select as many choices as they wanted from a random-order list. The choices and results were:
1. When you have to call more than once for the same problem, you have to start over from the beginning each time. The company doesn't keep good enough records of your earlier calls. 68%
2. You're forced to go through repetitive or irrelevant steps to get help 68% (statistically tied with 1st place)
3. The person you spoke to was incompetent, or wasn't trained properly 61%
4. You had to wait too long to talk to a person 60% (statistically tied with 3rd place)
5. The person you spoke to was rude or unprofessional 56% (statistically tied with 4th place)
6. The company doesn't do things it promised to do, like fix a problem, issue a credit, follow up with you, etc. 53% (statistically tied with 5th place)
7. The self-service system was difficult or impossible to use 48% (statistically tied with 6th place)
8. The company refused to admit making a mistake, even when there's obviously a problem 46% (statistically tied with 7th place)
9. The customer service representative won't transfer you to a supervisor when asked 44% (statistically tied with 7th place)
10. The company made it difficult to find the right options to be transferred to a person 44% (statistically tied with 7th place)
11. The quality of the service is inconsistent, and you never know if you will get good or bad service 39%
12. The person you spoke to didn't have the authority to help you 37% (statistically tied with 11th place)
Observations
I found it most interesting that the most popular "good" service options were not mirrored by the most popular "bad" service options. The most common things customers associate with good customer service can be summarized as "make it quick and easy to reach a person, and make that person polite and professional."
In contrast, the top two things customers associate with bad service boil down to "don't make me jump through hoops." Those hoops can be either be having to start over each time (which is more often a complaint with live, rather than automated service), or having to go through needless extra steps (which can be either badly-designed IVR or poorly scripted agents, but most customers probably associate this with having to go through multiple layers of IVR menus).
Making it hard to reach an agent was all the way down at #10 on the list of complaints. Maybe this is because hiding the agent has become so common that it's practically expected rather than a marker of below-par service.
I think in general any factor (good or bad) in the top half of its list is probably worth paying attention to. The top-half "good" factors are most likely to be noticed and appreciated by customers, and the top-half "bad" factors are most likely to be pet peeves. If you're worried that your service is subpar, the first thing to do is make sure you're not making customers jump through hoops, repeat themselves, or start over from scratch each time they call.
If you want to provide excellent service (and you've already eliminated the obvious problems), the first thing to do is make it easy for customers to reach an agent if needed, and that the agents are acting professionally.
Posted by Peter Leppik
One of the questions we asked was which factors people thought were most important in providing good customer service over the phone. They could select as many choices as they wished from a list (the order of the list was random for each participant to avoid bias). The answers were:
1. Make it easy for you to reach a live person (if necessary) 82%
2. Be courteous, polite, and professional 75%
3. Wait only a short time to talk to a human (if needed) 69%
4. Handle your question or problem on the first call 63%
5. Be willing to transfer your call to a supervisor if the agent isn't helping 59% (statistically tied with 4th place)
6. If you have to call more than once for the same problem, be able to talk to the same person each time 49%
7. Follow up (if needed) to let you know that your problem was solved 45% (statistically tied with 6th place)
8. Keep the total call as short as possible 32%
9. Provide an easy-to-use self-service option (where possible) 31% (statistically tied with 8th place)
10. Do more to help than you expected 31% (statistically tied with 8th place)
We also asked participants to choose which factors they feel are most important in providing bad customer service over the phone--as with the good service question, they could select as many choices as they wanted from a random-order list. The choices and results were:
1. When you have to call more than once for the same problem, you have to start over from the beginning each time. The company doesn't keep good enough records of your earlier calls. 68%
2. You're forced to go through repetitive or irrelevant steps to get help 68% (statistically tied with 1st place)
3. The person you spoke to was incompetent, or wasn't trained properly 61%
4. You had to wait too long to talk to a person 60% (statistically tied with 3rd place)
5. The person you spoke to was rude or unprofessional 56% (statistically tied with 4th place)
6. The company doesn't do things it promised to do, like fix a problem, issue a credit, follow up with you, etc. 53% (statistically tied with 5th place)
7. The self-service system was difficult or impossible to use 48% (statistically tied with 6th place)
8. The company refused to admit making a mistake, even when there's obviously a problem 46% (statistically tied with 7th place)
9. The customer service representative won't transfer you to a supervisor when asked 44% (statistically tied with 7th place)
10. The company made it difficult to find the right options to be transferred to a person 44% (statistically tied with 7th place)
11. The quality of the service is inconsistent, and you never know if you will get good or bad service 39%
12. The person you spoke to didn't have the authority to help you 37% (statistically tied with 11th place)
Observations
I found it most interesting that the most popular "good" service options were not mirrored by the most popular "bad" service options. The most common things customers associate with good customer service can be summarized as "make it quick and easy to reach a person, and make that person polite and professional."
In contrast, the top two things customers associate with bad service boil down to "don't make me jump through hoops." Those hoops can be either be having to start over each time (which is more often a complaint with live, rather than automated service), or having to go through needless extra steps (which can be either badly-designed IVR or poorly scripted agents, but most customers probably associate this with having to go through multiple layers of IVR menus).
Making it hard to reach an agent was all the way down at #10 on the list of complaints. Maybe this is because hiding the agent has become so common that it's practically expected rather than a marker of below-par service.
I think in general any factor (good or bad) in the top half of its list is probably worth paying attention to. The top-half "good" factors are most likely to be noticed and appreciated by customers, and the top-half "bad" factors are most likely to be pet peeves. If you're worried that your service is subpar, the first thing to do is make sure you're not making customers jump through hoops, repeat themselves, or start over from scratch each time they call.
If you want to provide excellent service (and you've already eliminated the obvious problems), the first thing to do is make it easy for customers to reach an agent if needed, and that the agents are acting professionally.
Posted by Peter Leppik
Upcoming Mini-Workshop
Mon - November 12, 2007 03:28 PM
We're hosting a Mini-Workshop at our offices in Golden Valley, MN, the morning of December 18th, 2007. The topic will be "Writing and Scripting Surveys for Customer Service: Practical Techniques."
When: 12/18/2007, 9:30 - 11:30 AM
Where: 8421 Wayzata Blvd, Golden Valley, MN, 55426; 2nd floor conference room
Cost: Free (registration required)
Here's the deal: We're revising the course material for our two-day workshop, "Customer Service Surveys: Practical Techniques," and we want to test-drive the new material before we finalize it. We're breaking it into topical chunks, and the first one will be about writing a questionnaire. There's no charge to attend, but we expect brutally honest feedback from everyone who attends. Think of it as like a beta test.
We're trying to make this as hands-on as possible, and right now we're looking at spending about half the time discussing questionnaire design, and the other half actually revising/writing an actual survey.
To attend, please register here. Space is limited, so please register as soon as possible.
Future topics will include methodology; improving response; and analysis.
Posted by Peter Leppik
Where: 8421 Wayzata Blvd, Golden Valley, MN, 55426; 2nd floor conference room
Cost: Free (registration required)
Here's the deal: We're revising the course material for our two-day workshop, "Customer Service Surveys: Practical Techniques," and we want to test-drive the new material before we finalize it. We're breaking it into topical chunks, and the first one will be about writing a questionnaire. There's no charge to attend, but we expect brutally honest feedback from everyone who attends. Think of it as like a beta test.
We're trying to make this as hands-on as possible, and right now we're looking at spending about half the time discussing questionnaire design, and the other half actually revising/writing an actual survey.
To attend, please register here. Space is limited, so please register as soon as possible.
Future topics will include methodology; improving response; and analysis.
Posted by Peter Leppik
Games
Wed - October 3, 2007 04:07 PM
One of the most efficient ways to teach complex ideas and skills is through playing games. That's because games allow you to create a miniature version of the real world (without a lot of the complexity) geared towards the most important elements you're trying to teach.
I'm giving a distilled 90-minute version of the VocaLabs Workshop (Customer Service Surveys: Practical Techniques) at the SOCAP annual conference in Palm Springs next week, and one of my biggest challenges was how to fit the key lessons from two days of hands-on learning into under two hours.
Naturally, I settled on a game.
So if you're going to SOCAP, you'll have the chance to play The Survey Game, where you're in the role of a call center quality manager who has to figure out why your customers aren't happy, and how to fix them. You've got a limited budget and time, and of course the more money you spend on surveys the less is available for actual quality improvements.
The game will teach about survey bias and accuracy, how to allocate scarce resources between data collection and quality improvement, and how to balance expensive but reliable survey methods with inexpensive but unreliable techniques.
Posted by Peter Leppik
Naturally, I settled on a game.
So if you're going to SOCAP, you'll have the chance to play The Survey Game, where you're in the role of a call center quality manager who has to figure out why your customers aren't happy, and how to fix them. You've got a limited budget and time, and of course the more money you spend on surveys the less is available for actual quality improvements.
The game will teach about survey bias and accuracy, how to allocate scarce resources between data collection and quality improvement, and how to balance expensive but reliable survey methods with inexpensive but unreliable techniques.
Posted by Peter Leppik
Correlation
Mon - May 14, 2007 04:14 PM
Survey analysis is an important step in going from raw data to improvements in service levels, and a key part of the analysis is looking for relationships in the data.
Most frequently, the relationships are in the form of correlations. A correlation simply means that X and Y tend to go together in the data: for example, customers who report that they didn't get what they needed during the call tend to have longer calls on average.
Correlations are never perfect, both because the relationships between different variables are very complicated, and because survey data is inherently noisy and imprecise. The quality of a correlation is typically measured as a "p" value, the probability that a given correlation between two variables is the result of random chance. By tradition, a p value of less than 0.05 is usually considered statistically significant.
This creates a problem, though, since automated data analysis makes is possible to test vast numbers of relationships in the data. Our own analysis tool tests hundreds of thousands of possible correlations in a typical survey, which means that it's likely to "find" thousands of bogus correlations which meet the usual test for statistical significance. Because of that problem, we don't flag a correlation as "likely" unless the p value is less than 0.01, and we distinguish the relationships with p < 0.001 as especially likely.
We tend to find several classes of correlations:
1. Obvious and Trivial Correlations
The first group are the correlations which we expect to find, and which we'd be surprised to notfind. For example, people tend to answer "satisfaction" and related questions very similarly: someone who is "very satisfied" with the call overall is likely to respond that he's "very satisfied" with the customer service representative, and "very likely" to remain a customer. At some level, these questions are all asking for the same thing, and customers often don't distinguish between them even when there are qualifiers attached (such as "how satisfied were you with the efficiency of the call?").
2. Interesting but Useless Correlations
We also uncover relationships which aren't necessarily obvious (and which can lend some insight into the data), but which are hard to do anything about. One common correlation is that when we're doing live interviews, the length of the interview call often correlates to the customer's satisfaction. At first this seems weird, until you listen to a few interview recordings and realize that when a customer has a bad experience he usually want to describe it at length. Good experiences, on the other hand, don't usually elicit such a response. This is an interesting result, but it doesn't help us much when we're looking to improve satisfaction.
3. Interesting and Useful Correlations
The really helpful stuff is in the relationships which aren't obvious and which point to ways to improve service levels. For example, when satisfaction is correlated to the type of call (sales vs. support vs. billing), there may be a problem with how certain calls are handled. Or when an expected relationship fails to materialize, that can also point to problems--such as when the customer's stated reason for calling doesn't correlate to the agent queue, that suggests that the IVR is doing a poor job of routing calls.
Posted by Peter Leppik
Correlations are never perfect, both because the relationships between different variables are very complicated, and because survey data is inherently noisy and imprecise. The quality of a correlation is typically measured as a "p" value, the probability that a given correlation between two variables is the result of random chance. By tradition, a p value of less than 0.05 is usually considered statistically significant.
This creates a problem, though, since automated data analysis makes is possible to test vast numbers of relationships in the data. Our own analysis tool tests hundreds of thousands of possible correlations in a typical survey, which means that it's likely to "find" thousands of bogus correlations which meet the usual test for statistical significance. Because of that problem, we don't flag a correlation as "likely" unless the p value is less than 0.01, and we distinguish the relationships with p < 0.001 as especially likely.
We tend to find several classes of correlations:
1. Obvious and Trivial Correlations
The first group are the correlations which we expect to find, and which we'd be surprised to notfind. For example, people tend to answer "satisfaction" and related questions very similarly: someone who is "very satisfied" with the call overall is likely to respond that he's "very satisfied" with the customer service representative, and "very likely" to remain a customer. At some level, these questions are all asking for the same thing, and customers often don't distinguish between them even when there are qualifiers attached (such as "how satisfied were you with the efficiency of the call?").
2. Interesting but Useless Correlations
We also uncover relationships which aren't necessarily obvious (and which can lend some insight into the data), but which are hard to do anything about. One common correlation is that when we're doing live interviews, the length of the interview call often correlates to the customer's satisfaction. At first this seems weird, until you listen to a few interview recordings and realize that when a customer has a bad experience he usually want to describe it at length. Good experiences, on the other hand, don't usually elicit such a response. This is an interesting result, but it doesn't help us much when we're looking to improve satisfaction.
3. Interesting and Useful Correlations
The really helpful stuff is in the relationships which aren't obvious and which point to ways to improve service levels. For example, when satisfaction is correlated to the type of call (sales vs. support vs. billing), there may be a problem with how certain calls are handled. Or when an expected relationship fails to materialize, that can also point to problems--such as when the customer's stated reason for calling doesn't correlate to the agent queue, that suggests that the IVR is doing a poor job of routing calls.
Posted by Peter Leppik
Top Ten Survey Mistakes
Thu - May 10, 2007 02:21 PM
There's lots of common mistakes people make when designing a customer service survey.
Here's my personal list of the top ten:
Posted by Peter Leppik
- Surveying customers on the same phone call as a customer service interaction
- Not allowing a free response on the survey
- Asking vague, unfocused questions instead of specific questions about the experience
- Waiting more than an hour to survey the customer
- Not tying individual survey responses to call recordings and other call data
- Not offering customers a follow-up from a supervisor when there was an unsatisfactory experience
- Not tying survey data to specific business goals
- Asking the same question over and over with only slight variations
- Asking too many questions on the survey
- Thinking of a survey as purely data collection, and forgetting that how the survey is performed will influence your brand image
Posted by Peter Leppik
The Lesson We Didn't Include
Tue - May 8, 2007 02:01 PM
In retrospect, there's one key slide we should have included in our first workshop (it will be in the next one, don't worry):
Stuff To Worry About For Customer Service Surveys
Really Really Difficult And Important Stuff
* Appropriate survey method
* Unbiased sampling technique
* Whether employees manipulate the survey
Stuff Which Is Important But Easy To Get Right
* Wording of survey questions
* Consistency of survey questions and method
* Getting enough responses in your sample
Stuff Which Is Really Not Important At All
* The percent of customers who are surveyed (the absolute number of responses is usually far more important)
* Whether you use a five-, seven-, four-, or nine-point scale for opinion (Likert scale) questions (it's more important to be completely consistent between surveys)
The irony, of course, is that people tend to spend the most time worrying about the bottom two categories (the easy and irrelevant pieces), and relatively little time worrying about the top category (the difficult and important things).
Posted by Peter Leppik
Really Really Difficult And Important Stuff
* Appropriate survey method
* Unbiased sampling technique
* Whether employees manipulate the survey
Stuff Which Is Important But Easy To Get Right
* Wording of survey questions
* Consistency of survey questions and method
* Getting enough responses in your sample
Stuff Which Is Really Not Important At All
* The percent of customers who are surveyed (the absolute number of responses is usually far more important)
* Whether you use a five-, seven-, four-, or nine-point scale for opinion (Likert scale) questions (it's more important to be completely consistent between surveys)
The irony, of course, is that people tend to spend the most time worrying about the bottom two categories (the easy and irrelevant pieces), and relatively little time worrying about the top category (the difficult and important things).
Posted by Peter Leppik
Aftermath
Thu - May 3, 2007 03:28 PM
Our first workshop is over, and there's one thing which is very clear: I am exhausted.
Running this kind of seminar is a lot of work.
We had a split of attendees: about half were people measuring call center quality, and half were people building speech recognition systems (this actually mirrors our customer base pretty well). Unfortunately, the two groups have somewhat divergent needs, so some things which worked for the call center people didn't work for the speech people, and vice-versa. So perhaps we should consider offering two different workshops to meet their different needs.
Beyond that, we have a list of improvements for the next time around:
1. More statistics. I never thought I'd hear anyone ask for that, but everyone said they wanted a more in-depth discussion of statistical methods, in particular ways to analyze data for correlations among the data.
2. More discussion of benchmarks and metrics. Frankly, the details of how a particular metric is constructed aren't that important (it's far more critical to make sure the sampling method is random, and the metric is consistent across surveys), but there are so many metrics and benchmarks out there that there seems to be confusion about what's good and bad.
3. Less time spent designing particular survey questions, and more time spent looking at question banks. It might be a good idea to spend some time critiquing survey questions, either from other surveys or from participants' existing surveys.
Overall, despite the rough edges (which we expected--the first time for anything is never perfect), it was a worthwhile experience for everyone involved. The next one should be even better.
Posted by Peter Leppik
We had a split of attendees: about half were people measuring call center quality, and half were people building speech recognition systems (this actually mirrors our customer base pretty well). Unfortunately, the two groups have somewhat divergent needs, so some things which worked for the call center people didn't work for the speech people, and vice-versa. So perhaps we should consider offering two different workshops to meet their different needs.
Beyond that, we have a list of improvements for the next time around:
1. More statistics. I never thought I'd hear anyone ask for that, but everyone said they wanted a more in-depth discussion of statistical methods, in particular ways to analyze data for correlations among the data.
2. More discussion of benchmarks and metrics. Frankly, the details of how a particular metric is constructed aren't that important (it's far more critical to make sure the sampling method is random, and the metric is consistent across surveys), but there are so many metrics and benchmarks out there that there seems to be confusion about what's good and bad.
3. Less time spent designing particular survey questions, and more time spent looking at question banks. It might be a good idea to spend some time critiquing survey questions, either from other surveys or from participants' existing surveys.
Overall, despite the rough edges (which we expected--the first time for anything is never perfect), it was a worthwhile experience for everyone involved. The next one should be even better.
Posted by Peter Leppik
Ready for Launch
Tue - May 1, 2007 01:58 PM
Our first Interactive Workshop begins tomorrow, and it looks like we're ready to go.
Presentation slides? Check.
Handouts? Check.
Flip charts and markers? Check.
Projector? Check.
Fifty pounds of jellybeans? Uh-oh, hasn't arrived yet.
Posted by Peter Leppik
Handouts? Check.
Flip charts and markers? Check.
Projector? Check.
Fifty pounds of jellybeans? Uh-oh, hasn't arrived yet.
Posted by Peter Leppik
The problem with statistics
Mon - April 16, 2007 10:15 AM
In geometry, there's something called the "left hand rule," which is used to visualize 3D vectors. Unless you use a right-handed coordinate system, in which case you use the right hand rule. It's completely arbitrary, and you find both in computer graphics. Right-handed coordinates are the standard in physics, while left handed (arguably) makes more sense to artists.
Similarly, computer processors have a notion of "endianness". Some systems are big-endian, others are little-endian. The choice is so arbitrary that the name comes from the debate in Gulliver's Travels over whether Lilliputians should crack their eggs from the big end or little end.
Whenever you have two independent groups of people inventing (or using) a complicated tool, they develop different names and conventions for the same things. That can make it difficult to figure out what's really going on. Indeed, it makes it tricky for one group to even discover that the other group exists. (The Wikipedia entry for Right Hand Rule [http://en.wikipedia.org/wiki/Right_hand_rule] doesn't even mention coordinate systems, even though it links to a site that does [http://mathworld.wolfram.com/Right-HandRule.html].)
The problem with statistics is that it's a mathematical tool that every branch of science and engineering uses, and each one uses it differently. The book I learned statistics from is "Statistics for the Behavior Sciences." But it doesn't even mention margin of error, even though that's the most common statistic used to describe opinion polls. And much of behavioral science (psychology) is conducted with surveys! Apparently pollsters and psychologists don't talk much to each other.
Nor is it just psychology that's different. I've been trying to figure out the derivation of Peter's margin of error formulas from last week [http://www.vocalabs.com/resources/blog/C834959743/E20070402150458/index.html]. He has a physics background. Wikipedia is maddeningly inconsistent (no surprise), and even MathWorld [http://mathworld.wolfram.com/] is troublesome-- although more along the lines of "Note that some authors define the term as...."[http://mathworld.wolfram.com/Erf.html].
Statistics are like the water heater in your basement which you install once and don't think about until it breaks. People figure out what formulas they need, and then forget about them. Even statisticians don't get hung up on nomenclature. Which is too bad. The math is hard enough, without having to deal with different--and often conflicting-- definitions for similar things.
One thing you should know is that the formulas in Peter's blog entry are approximations. Instead of 1/sqrt(N), most people would write the margin of error formula as 0.98/sqrt(N). (Our approximation is slightly more conservative.) But if you need that much precision, getting the margin of error right is the least of your worries.
Posted by David Leppik
Whenever you have two independent groups of people inventing (or using) a complicated tool, they develop different names and conventions for the same things. That can make it difficult to figure out what's really going on. Indeed, it makes it tricky for one group to even discover that the other group exists. (The Wikipedia entry for Right Hand Rule [http://en.wikipedia.org/wiki/Right_hand_rule] doesn't even mention coordinate systems, even though it links to a site that does [http://mathworld.wolfram.com/Right-HandRule.html].)
The problem with statistics is that it's a mathematical tool that every branch of science and engineering uses, and each one uses it differently. The book I learned statistics from is "Statistics for the Behavior Sciences." But it doesn't even mention margin of error, even though that's the most common statistic used to describe opinion polls. And much of behavioral science (psychology) is conducted with surveys! Apparently pollsters and psychologists don't talk much to each other.
Nor is it just psychology that's different. I've been trying to figure out the derivation of Peter's margin of error formulas from last week [http://www.vocalabs.com/resources/blog/C834959743/E20070402150458/index.html]. He has a physics background. Wikipedia is maddeningly inconsistent (no surprise), and even MathWorld [http://mathworld.wolfram.com/] is troublesome-- although more along the lines of "Note that some authors define the term as...."[http://mathworld.wolfram.com/Erf.html].
Statistics are like the water heater in your basement which you install once and don't think about until it breaks. People figure out what formulas they need, and then forget about them. Even statisticians don't get hung up on nomenclature. Which is too bad. The math is hard enough, without having to deal with different--and often conflicting-- definitions for similar things.
One thing you should know is that the formulas in Peter's blog entry are approximations. Instead of 1/sqrt(N), most people would write the margin of error formula as 0.98/sqrt(N). (Our approximation is slightly more conservative.) But if you need that much precision, getting the margin of error right is the least of your worries.
Posted by David Leppik
Customer service questionnaires as storytelling
Thu - April 12, 2007 01:44 PM
I'm working on our upcoming workshop and there are a lot of things to cover. My wife can't believe we have enough material for two days, but the fact is that there's far more than we can squeeze in. So I'll be blogging about a few ideas that catch my attention that we may not have time to cover. (Peter's already done this in a recent entry.)
One thing that's occurred to me is that a customer service questionnaire should be as similar as possible to how the respondent would tell the story to a friend. It's the respondent's life, and much of what we humans spend our time doing is telling and retelling our stories. When a survey leads us away from our story, we disengage from the survey. That leads to less accurate and less heartfelt responses.
If you watch a lot of interviews on television or listen to them on the radio, you know that a bad interview consists of questions which bore the interviewee or highlight the gap between how the interviewer and the interviewee see things. A good interview, on the other hand, consists of a mix of questions which allow the interviewee to tell a favorite story (in your book, you describe..., explain what that's about) with the occasional engaging question which make the interviewee think about things (s)he's an expert in, but in a new light.
With a survey, it's impossible to give everyone that expert, personalized experience. But there are a few ways you can craft it to engage the storytelling instinct.
Posted by David Leppik
If you watch a lot of interviews on television or listen to them on the radio, you know that a bad interview consists of questions which bore the interviewee or highlight the gap between how the interviewer and the interviewee see things. A good interview, on the other hand, consists of a mix of questions which allow the interviewee to tell a favorite story (in your book, you describe..., explain what that's about) with the occasional engaging question which make the interviewee think about things (s)he's an expert in, but in a new light.
With a survey, it's impossible to give everyone that expert, personalized experience. But there are a few ways you can craft it to engage the storytelling instinct.
- Consider the hook. There are three types of customer service experiences: good, bad, and indifferent. By far, indifferent is the most common, but you get a more nuanced view if you can draw out something good or bad from the experience. Questions like "what's the best thing you remember about..." or "if you could change one thing..." draw out the emotional center of the story, along with memories of the experience.
- Consider the plot. How is the story structured? Chronologically? Thematically? One of the classic techniques that lawyers use in cross-examination is to ask questions out of order, in order to confuse the witness. Often surveys do this unintentionally, with the same effect. Good surveys will often include memory-jogging factual questions simply to prepare the respondent for the real questions.
- Consider the audience. If the survey is delivered verbally, the responses are taylored to the questioner. If it is written, the responses are taylored to whomever is likely to read them. The same questions can yield significantly different responses based on the audience. If the interviewer has a slight Indian accent, you can bet that you'll get a lot fewer complaints about incomprehensible call center agents than if the interviewer has a midwestern accent. Sometimes respondents intentionally exaggerate for effect, but usually it's subconscious.
Posted by David Leppik
Customer Service Surveys are Different
Fri - April 6, 2007 04:09 PM
It's interesting to survey the academic literature about survey methods with an eye to the call center. It turns out that relatively little formal research has been done about how to survey for customer service quality.
Most of the research has been done about political polling and psychological research, with some views of market research surveys.
Customer service is unique from these other types of surveys, and is easier in many ways:
There are also unique aspects to a customer service survey which are harder than more traditional surveys:
These differences do have some practical implications. For example, where someone designing a political poll will place a lot of emphasis on carefully wording the survey questions and figuring out who's a likely voter, in a customer service survey we're going to look at the client's internal process for using the data, consistency and bias in the survey process, and getting meaningful data to decision-makers in real time.
In my world, just collecting data isn't enough. We have to make the data fit into the client's internal processes, make it actionable, and deliver it now.
Posted by Peter Leppik
Customer service is unique from these other types of surveys, and is easier in many ways:
- The population to survey is very well-defined: people who contacted customer service. There's no need to sample the general population.
- The survey is usually asking about a specific event which has already occurred, unlike political polling or market research which ask about future behavior and opinions about hypothetical products or situations.
- The survey usually is trying to track changes over time or between different groups of customers, rather than predict the outcome of a future event. So predictive ability (which is all-important in a political poll) isn't a big deal. Instead, consistency is crucial.
There are also unique aspects to a customer service survey which are harder than more traditional surveys:
- Customer service surveys are often more tactical than other surveys, which is to say that they're looking for specific problems which need to be fixed right away. This means that real-time reporting and alerting is essential. A tabulated report summarizing data from six weeks ago is stale.
- Part of the role of a customer service survey is to enhance the company's image in the customer's eyes. That means that the customer has to think the survey is effective and worthwhile. With other surveys, you don't usually care what the participant thinks about the survey itself.
- Most of the value in a customer service survey is in the ability to match individual customers' opinions with records of how that particular call was handled, in order to find ways to improve service. Most surveys focus on just the high-level statistics, and don't bring external data into the survey. But if you can't match the customer complaint to the call recording, agent name, call classification, etc., you can't figure out where the problems are.
These differences do have some practical implications. For example, where someone designing a political poll will place a lot of emphasis on carefully wording the survey questions and figuring out who's a likely voter, in a customer service survey we're going to look at the client's internal process for using the data, consistency and bias in the survey process, and getting meaningful data to decision-makers in real time.
In my world, just collecting data isn't enough. We have to make the data fit into the client's internal processes, make it actionable, and deliver it now.
Posted by Peter Leppik
Margin of Error
Mon - April 2, 2007 03:04 PM
There's really no excuse for not calculating the margin of error in a survey, since the calculation is so simple and it's so important to understand the accuracy of your data.
The formula is MOE = 1/sqrt(N) where N is the number of survey responses. If you punch this into a calculator or spreadsheet, the result will be a fraction, which you can also express as a percent.
For example, with 100 survey responses, the margin of error is 1/sqrt(100) = 1/10 = .1 = 10%.
Similarly, with 400 survey responses, the margin of error is 5%. Quadrupling the number of responses cuts the margin of error in half.
This formula is used all the time in error analysis, but it's actually an approximation which doesn't work for answers which are given on only a very small fraction of the surveys.
When less than five percent of the people gave a particular answer, the margin of error for that answer is approximately 2*sqrt(P/N), where P is the fraction of participants who gave that answer (the exact formula in all cases is 2*sqrt[P(1-P)/N] ).
For example, if you surveyed 1,000 people about their religion, and 10 (1%) answered "Jedi," then the margin of error for the percentage of Jedi in the survey is 2*sqrt(.01/1000) = 0.6%. This is actually quite a bit smaller than the margin of error given by the more common formula (3.2% in this case). It turns out that the common 1/sqrt(N) formula is the largestmargin of error possible for any response on the survey.
Posted by Peter Leppik
UPDATE (4/12/07): There was some confusion about the approximate margin of error formula when P is small, so I reworded that section. I hope it's clear now.
For example, with 100 survey responses, the margin of error is 1/sqrt(100) = 1/10 = .1 = 10%.
Similarly, with 400 survey responses, the margin of error is 5%. Quadrupling the number of responses cuts the margin of error in half.
This formula is used all the time in error analysis, but it's actually an approximation which doesn't work for answers which are given on only a very small fraction of the surveys.
When less than five percent of the people gave a particular answer, the margin of error for that answer is approximately 2*sqrt(P/N), where P is the fraction of participants who gave that answer (the exact formula in all cases is 2*sqrt[P(1-P)/N] ).
For example, if you surveyed 1,000 people about their religion, and 10 (1%) answered "Jedi," then the margin of error for the percentage of Jedi in the survey is 2*sqrt(.01/1000) = 0.6%. This is actually quite a bit smaller than the margin of error given by the more common formula (3.2% in this case). It turns out that the common 1/sqrt(N) formula is the largestmargin of error possible for any response on the survey.
Posted by Peter Leppik
UPDATE (4/12/07): There was some confusion about the approximate margin of error formula when P is small, so I reworded that section. I hope it's clear now.
Survey Mistakes
Wed - March 28, 2007 02:52 PM
We talk a lot about the margin of error in a survey, as well as the sample bias inherent in the survey method. The former is the inherent uncertainty because you can't survey 100% of customers, and the latter is a systematic bias coming from the way participants are selected or the survey is administered.
There are lots of other ways surveys can be wrong, though. For any survey there's a certain percentage of surveys which don't get filled out correctly.
With a survey where the participant fills out the answers directly, mistakes include:
These errors can be very difficult to quantify, since there's usually no (economical) way to go back to the participants and ask them what they reallymeant. As a rule of thumb, we generally figure that a couple percent of questions are answered incorrectly--and we work hard to try to craft a survey which minimizes participants' confusion.
When there's a live interviewer, you can get a better handle on whether a survey question is confusing or doesn't apply, but there are other kinds of mistakes:
If there's a recording of the interview (and there should alwaysbe a recording of the interview!), most of these mistakes can be caught by carefully reviewing the recording. One advantage of a live interview is that you actually can go back and find mistakes, rather than just having to guess. While it isn't practical to review every interview, by reviewing a sample of recordings and tracking the number of mistakes, you can get a handle on how often various kinds of mistakes are made.
You can never expect 100% perfection in a survey, but it's important to understand where there might be errors and what the limitations of the data are.
Posted by Peter Leppik
With a survey where the participant fills out the answers directly, mistakes include:
- Misunderstanding the question.
- Understanding the question, but accidentally choosing the wrong option.
- Being forced to answer a question which doesn't apply (if the survey forces a participant to answer a question to continue when the question doesn't apply, most people will just make something up).
These errors can be very difficult to quantify, since there's usually no (economical) way to go back to the participants and ask them what they reallymeant. As a rule of thumb, we generally figure that a couple percent of questions are answered incorrectly--and we work hard to try to craft a survey which minimizes participants' confusion.
When there's a live interviewer, you can get a better handle on whether a survey question is confusing or doesn't apply, but there are other kinds of mistakes:
- Accidentally marking a survey answer differently than the participant answered
- Changing the wording of a survey question
- Prompting the participant for a particular answer
- Failing to clarify an ambiguous answer
If there's a recording of the interview (and there should alwaysbe a recording of the interview!), most of these mistakes can be caught by carefully reviewing the recording. One advantage of a live interview is that you actually can go back and find mistakes, rather than just having to guess. While it isn't practical to review every interview, by reviewing a sample of recordings and tracking the number of mistakes, you can get a handle on how often various kinds of mistakes are made.
You can never expect 100% perfection in a survey, but it's important to understand where there might be errors and what the limitations of the data are.
Posted by Peter Leppik
Common survey problems
Tue - March 13, 2007 03:58 PM
Want a quick assessment of how good a survey is? It turns out that the same problems tend to crop up over and over again.
Here's a quick checklist of the most common problems with customer service surveys:
-
Survey Design
- Are the survey questions biased?
- Are the survey questions ambiguous? In an interview, do participants ask for clarification?
- Is the survey too long or redundant?
- Are the management goals of the survey ambiguous or poorly understood?
-
Methodology
- Can the survey be manipulated, especially if employees are given bonuses based on survey results?
- Are certain customers systematically excluded from the survey? For example, are there conditions under which the customer cannot take the survey?
- Is there too much delay between the customer interaction and when the survey is administered?
- Is is difficult or impossible to analyze survey data based on other data about the customer (i.e. customer type, what the customer called about, who the customer spoke to, recording of the phone call, etc.)?
-
Analysis
- Is there a lack of context for the survey data such as historical survey data, industry data, etc.?
- Is the error analysis missing, incomplete, presented out of context, or otherwise inadequate? Error analysis should include not just a calculation of the margin of error, but also look at sources of bias and potential problems with the survey questions and method.
- Is the analyst interpreting the language of the survey the same way that participants do? For example, "satisfied" can mean many different things to different people.
-
Follow-Through
- Is there a failure to regularly generate action-items for improving service based on survey results?
- Are there key decision makers who don't believe in the survey process?
- Are there employees (or managers) who don't take the survey seriously?
- Is upper-level management failing to support the survey?
Posted by Peter Leppik