The Customer Service Survey
Size Matters, But So Does Technique
Mon - March 6, 2006 02:21 PM in
Survey size and quality collide in the budget. Any effort to measure customer satisfaction has to simultaneously gather enough data to be useful, using a technique which doesn't distort the results, and still live within a budget.
Let's suppose that you've got a budget of $10,000 to survey callers to a call center. There are several choices for how to spend this money:
In the analysis above, the cost/survey is the range of prices we think customers are being charged in the "real world" for each type of survey service. From that, we estimated the number of survey responses the $10,000 budget could buy (not counting any setup fees), using a midrange cost per survey. The margin of error is the statistical sampling error based on the number of surveys, and the estimated bias is our estimate of how much inherent bias there is in each survey technique.
When people decide on a survey method and budget, they tend to focus on the Margin of Error, since it is based on a mathematical formula and can be calculated very precisely. And as you can see, the larger the survey, the lower the margin of error.
But just as important is the inherent bias in the survey technique. This bias can't be calculated, and it can be measured only with great difficulty (the estimates I gave are based on our experience and experience our clients have had). And since the bias is inherent in the survey technique, the only way to reduce the bias is to change survey techniques.
When you look at both the margin of error and the inherent bias in the survey technique, you see that simply going for the biggest possible survey (i.e. the one with the lowest cost per response) is a big mistake. Having a statistical margin of error in the 1% range sounds nice, except that the technique is so badly biased that your survey might be inherently off by 40 percentage points. You are measuring exactly the wrong thing with extreme precision.
In this example, your best bet is probably either a phone follow-up survey or an immediate follow-up survey. If your budget is bigger or smaller, the optimal survey method might be different. If your budget is bigger, you probably get more bang for the buck by going for an inherently more accurate survey technique (immediate follow-up) than spending the money on a bigger survey with a less-accurate method.
On the other hand, if your budget is smaller, you may need to switch to a less-accurate survey method in order to get enough responses, since there's no point in using an extremely accurate survey method if your budget doesn't allow enough surveys to get a small margin of error.
And there may be other considerations, too: how important is it to get the data in hours instead of in days or weeks? How important is it to have a particular report format, or additional data like call recordings and the name of the agent the caller talked to? Different survey methods can deliver different levels of depth, and whatever method you choose has to provide the type of data you need.
Posted by Peter Leppik
- Automated End-of-Call Survey
- This is the least expensive method for surveying callers to a call center, but this technique distorts the data very badly (to the extent that we advise clients that it may be better to not survey at all than do end-of-call surveys). The problem is that this method requires customers to stay on the phone at the end of the call to take the survey, and upset customers usually hang up before the survey. As a result, you only survey happy customers.
Cost/Survey: $1-$2
Budgeted Surveys: 8,000
Margin of Error: 1.1%
Estimated Bias: 20%-40% - Postcard Follow-Up Survey
- Mailing survey postcards to customers after they hang up avoids most of the bias of the end-of-call survey, but by the time customers get the postcards in the mail, they've often forgotten key details of the customer service call. As a result, you tend to get data on how satisfied customers are with the company overall, rather than anything you can attribute to a particular phone call or customer service rep. In addition, it takes several weeks to collect and tabulate the surveys.
Cost/Survey: $2-$10
Budgeted Surveys: 2,500
Margin of Error: 2.0%
Estimated Bias: 10%-20% - Phone Follow-Up Survey
- Phoning customers back within a couple days is similar to a mail survey in that by the time you call the customer they have often forgotten details of the call. But it is somewhat better, since you can usually reach customers within 24 hours instead of several days, and the data is available in a week or so, instead of several weeks.
Cost/Survey: $5-$40
Budgeted Surveys: 750
Margin of Error: 3.7%
Estimated Bias: 5%-10% - Immediate Follow-Up Survey
- Best of all is calling customers back immediately after the call, while their memory is still fresh and you can get instant feedback on how well customer service is performing. With our Express Feedback survey technique, we typically call customers back within five minutes of the time they hang up from the call center, which lets us ask very detailed questions about how the call went, and get the data to our clients fast enough for immediate action.
Cost/Survey: $13-$30
Budgeted Surveys: 500
Margin of Error: 4.4%
Estimated Bias: 2%-5%
In the analysis above, the cost/survey is the range of prices we think customers are being charged in the "real world" for each type of survey service. From that, we estimated the number of survey responses the $10,000 budget could buy (not counting any setup fees), using a midrange cost per survey. The margin of error is the statistical sampling error based on the number of surveys, and the estimated bias is our estimate of how much inherent bias there is in each survey technique.
When people decide on a survey method and budget, they tend to focus on the Margin of Error, since it is based on a mathematical formula and can be calculated very precisely. And as you can see, the larger the survey, the lower the margin of error.
But just as important is the inherent bias in the survey technique. This bias can't be calculated, and it can be measured only with great difficulty (the estimates I gave are based on our experience and experience our clients have had). And since the bias is inherent in the survey technique, the only way to reduce the bias is to change survey techniques.
When you look at both the margin of error and the inherent bias in the survey technique, you see that simply going for the biggest possible survey (i.e. the one with the lowest cost per response) is a big mistake. Having a statistical margin of error in the 1% range sounds nice, except that the technique is so badly biased that your survey might be inherently off by 40 percentage points. You are measuring exactly the wrong thing with extreme precision.
In this example, your best bet is probably either a phone follow-up survey or an immediate follow-up survey. If your budget is bigger or smaller, the optimal survey method might be different. If your budget is bigger, you probably get more bang for the buck by going for an inherently more accurate survey technique (immediate follow-up) than spending the money on a bigger survey with a less-accurate method.
On the other hand, if your budget is smaller, you may need to switch to a less-accurate survey method in order to get enough responses, since there's no point in using an extremely accurate survey method if your budget doesn't allow enough surveys to get a small margin of error.
And there may be other considerations, too: how important is it to get the data in hours instead of in days or weeks? How important is it to have a particular report format, or additional data like call recordings and the name of the agent the caller talked to? Different survey methods can deliver different levels of depth, and whatever method you choose has to provide the type of data you need.
Posted by Peter Leppik
Posted at 02:21 PM | | | | |