The Customer Service Survey
VocaLabs' weblog providing news and commentary on the challenges of providing good customer service.
Survivor Bias
Wednesday - November 02, 2005 12:33 PM in
Every day in World War II, bombers would fly from England to drop bombs on targets in continental Europe. Some would never return, some others would make it home on a wing and a prayer, shot full of holes from German fighters and anti-aircraft guns.
In order to improve the odds of survival, the allies decided to improve the armor plating on their bombers. But since weight is important on airplanes, they only wanted to add armor to those places on the airplane where it would actually help. So the engineers began a research project to determine where on the aircraft it would be most useful to have additional armor.
They diligently measured the locations of bullet holes on damaged bombers, compiled the statistical data (today we would say they built a database, but back then it was done by hand), and discovered clear patterns: bombers were much more likely to have bullet holes in the wings, tail surfaces, and in the tail gunner's position. Holes in the cockpit and fuel tanks were relatively rare.
So the engineers made their recommendation: add armor plating to the wings, tail surfaces, and tail gunner's position, since those were the locations on the plane most likely to be hit by German fire.
Then a statistician looked at the data, and realized that the engineers had come to exactly the wrong conclusion: because the engineers could only examine bombers which made it home safely, the right thing to do was to put armor plating where the engineers didn'tfind find any bullet holes, over the fuel tanks and cockpit.
In other words, the reason no bombers had holes in the fuel tanks or cockpit was because any plane hit on those areas never made it home to be examined. Those were critical areas, as compared to the wings and tail, where the bomber was more likely to survive being hit.
In order to improve the odds of survival, the allies decided to improve the armor plating on their bombers. But since weight is important on airplanes, they only wanted to add armor to those places on the airplane where it would actually help. So the engineers began a research project to determine where on the aircraft it would be most useful to have additional armor.
They diligently measured the locations of bullet holes on damaged bombers, compiled the statistical data (today we would say they built a database, but back then it was done by hand), and discovered clear patterns: bombers were much more likely to have bullet holes in the wings, tail surfaces, and in the tail gunner's position. Holes in the cockpit and fuel tanks were relatively rare.
So the engineers made their recommendation: add armor plating to the wings, tail surfaces, and tail gunner's position, since those were the locations on the plane most likely to be hit by German fire.
Then a statistician looked at the data, and realized that the engineers had come to exactly the wrong conclusion: because the engineers could only examine bombers which made it home safely, the right thing to do was to put armor plating where the engineers didn'tfind find any bullet holes, over the fuel tanks and cockpit.
In other words, the reason no bombers had holes in the fuel tanks or cockpit was because any plane hit on those areas never made it home to be examined. Those were critical areas, as compared to the wings and tail, where the bomber was more likely to survive being hit.
This is called survivor bias, and it happens all the time in customer satisfaction surveys. Survivor bias can be subtle and insidious, and in the worst cases it can lead to data which is completely misleading. Just like the bombers in WWII, survivor bias can lead you to try to fix things which aren't broken, and ignore huge problems which get overlooked by the survey.
The worst example of survivor bias in a satisfaction comes in an end-of-call survey. This is a survey administered at the end of a call, usually by an automated system, before the caller hangs up. End-of-call surveys completely exclude the following groups of people:
In other words, end-of-call surveys exclude most people who had a bad call. You are, in essence, only surveying the satisfied callers. The data you get may be very gratifying, but it won't help much when trying to improve service levels. (Because of these shortcomings, end-of-call surveys are one of the few methods for measuring customer service quality which we do not recommend clients use.)
But survivor bias creeps into other survey methods as well. For example, in a traditional follow-up survey the caller is contacted by phone or mail between one and seven days after the end of the call and asked to complete a survey. Since you're not relying on the caller to stay on the phone to take the survey you avoid the worst problems of an end-of-call survey, but you still exclude:
This is a whole lot better (and not nearly as open to manipulation by agents) than the end-of-call survey, but you're still excluding important groups of people. For example, if you're American Express, and an important part of your customer base is traveling professionals and executives, you probably don't want to exclude those people from your survey. (Follow-up surveys have other important limitations, such as the fact that people often forget what happened on the call shortly after they hang up, but that's a topic for another day.)
It is hard to completely eliminate survivor bias, since in any survey there will always be some fraction of people who drop out before the end of the survey, and there will always be some people who can't be bothered to take a survey or who don't take it seriously. But some techniques are clearly better than others, and it is important to give thought to the details of how a survey is administered. The important questions to ask are:
Posted by Peter Leppik
The worst example of survivor bias in a satisfaction comes in an end-of-call survey. This is a survey administered at the end of a call, usually by an automated system, before the caller hangs up. End-of-call surveys completely exclude the following groups of people:
- Anyone who gets trapped in an automated system--for example, people who choose the wrong option and can't figure out how to back up.
- Anyone who abandons the call while waiting for an agent.
- Anyone who forgets to hang on at the end of the call to take a survey.
- Anyone who hangs up when the agent says "Goodbye." It doesn't take agents long to figure out that if they don't want a caller taking the survey, all they have to do is say "Goodbye" and wait a few seconds for the caller to hang up first.
In other words, end-of-call surveys exclude most people who had a bad call. You are, in essence, only surveying the satisfied callers. The data you get may be very gratifying, but it won't help much when trying to improve service levels. (Because of these shortcomings, end-of-call surveys are one of the few methods for measuring customer service quality which we do not recommend clients use.)
But survivor bias creeps into other survey methods as well. For example, in a traditional follow-up survey the caller is contacted by phone or mail between one and seven days after the end of the call and asked to complete a survey. Since you're not relying on the caller to stay on the phone to take the survey you avoid the worst problems of an end-of-call survey, but you still exclude:
- Anyone who you couldn't identify during the call. For example, people who couldn't figure out how to enter their account number into an automated system.
- Non-customers, and customers whose contact information (address or phone number) is incorrect.
- People who are unlikely to be available to take a survey 1-7 days after the call, for example, people who travel a lot and might not be home when contacted with the survey.
This is a whole lot better (and not nearly as open to manipulation by agents) than the end-of-call survey, but you're still excluding important groups of people. For example, if you're American Express, and an important part of your customer base is traveling professionals and executives, you probably don't want to exclude those people from your survey. (Follow-up surveys have other important limitations, such as the fact that people often forget what happened on the call shortly after they hang up, but that's a topic for another day.)
It is hard to completely eliminate survivor bias, since in any survey there will always be some fraction of people who drop out before the end of the survey, and there will always be some people who can't be bothered to take a survey or who don't take it seriously. But some techniques are clearly better than others, and it is important to give thought to the details of how a survey is administered. The important questions to ask are:
- What events or actions might prevent someone from completing the survey?
- Will certain people be more likely to be excluded than others?
- Are the excluded people likely to answer survey questions differently than everyone else?
- How can the excluded people be included, or failing that, how can you measure the differences between excluded and included people?
Posted by Peter Leppik
Posted at 12:33 PM by | | | |