The Customer Service Survey
VocaLabs' weblog providing news and commentary on the challenges of providing good customer service.
Yes, 20-40 Percentage Points
Friday - March 10, 2006 11:52 AM in
Earlier this week, I published a summary of the relative cost and accuracy of different survey methods. One method, the end-of-call survey (where a caller is asked to stay on the line at the end of a call to take a survey) is very inexpensive but also extremely inaccurate; I estimated that the sample bias is 20-40 percentage points.
What that means in practice is that an end-of-call survey might show a company having (to make up a number) a 95% customer satisfaction rate, when the real number is 50% to 60% as measured by a less-biased technique.
Since then, I've been asked a couple of times if that 20-40 point bias is real. It sounds so big (and it is) that people instinctively questioned whether the bias could really be that great.
Well, as it happens, we do have a couple of clients who have done end-of-call surveys (not through VocaLabs--I have someself-respect!), so I was able to do some informal comparisons of similar surveys using end-of-call surveys and other techniques.
And in everycase I looked at, satisfaction-type questions scored at least 20 points higher in an end-of-call survey than with other survey methods. Some questions came out as much as 40 or even 50 points higher. These were not even cases where we would expect to see extreme bias in an end-of-call survey (i.e. where there is a known and severe service problem combined with agents known to be manipulating the survey). These were the kinds of operations which I would consider "typical" of end-of-call survey installations.
So the answer is: Yes, end-of-call surveys really are that bad, and the data exists to prove it. The data I used for my comparison is proprietary to VocaLabs clients, but it isn't hard to make this comparison if a company already is doing end-of-call surveys. We just run an Express Feedback (or similar survey) for a month and compare the two surveys.
I feel pretty strongly about the problems with end-of-call surveys, as you might have guessed. So if any call center out there is doing end-of-call surveys and wants to do this experiment, I'll even pay for the Express Feedback survey. Just drop me an e-mail.
Posted by Peter Leppik
Since then, I've been asked a couple of times if that 20-40 point bias is real. It sounds so big (and it is) that people instinctively questioned whether the bias could really be that great.
Well, as it happens, we do have a couple of clients who have done end-of-call surveys (not through VocaLabs--I have someself-respect!), so I was able to do some informal comparisons of similar surveys using end-of-call surveys and other techniques.
And in everycase I looked at, satisfaction-type questions scored at least 20 points higher in an end-of-call survey than with other survey methods. Some questions came out as much as 40 or even 50 points higher. These were not even cases where we would expect to see extreme bias in an end-of-call survey (i.e. where there is a known and severe service problem combined with agents known to be manipulating the survey). These were the kinds of operations which I would consider "typical" of end-of-call survey installations.
So the answer is: Yes, end-of-call surveys really are that bad, and the data exists to prove it. The data I used for my comparison is proprietary to VocaLabs clients, but it isn't hard to make this comparison if a company already is doing end-of-call surveys. We just run an Express Feedback (or similar survey) for a month and compare the two surveys.
I feel pretty strongly about the problems with end-of-call surveys, as you might have guessed. So if any call center out there is doing end-of-call surveys and wants to do this experiment, I'll even pay for the Express Feedback survey. Just drop me an e-mail.
Posted by Peter Leppik
Posted at 11:52 AM by | | | |