The Customer Service Survey
VocaLabs' weblog providing news and commentary on the challenges of providing good customer service.
I was 83.2% Satisfied!
Monday - October 31, 2005 03:29 PM in
One of the presentations I attended during last week's Nuance "Conversations" conclave in Arizona was a keynote presentation by a representative of "The Hartford". As I have said many times, it is difficult to be overly critical of any effort to improve the caller experience, but some methods simply work better than others and there is a certain amount of "art" in both the creation and interpretation of customer satisfaction data.
I was completely in sync with everything the speaker discussed surrounding the importance of measuring customer care quality until she described a particular piece of their satisfaction measurement methodology. What The Hartford does is employ a follow up study in which they ask callers if they would like to participate in a short survey on completion of their call, and within a very few minutes after hang up, the customer receives a phone call with an automated voice asking the survey questions and the respondent uses speech technology to give answers. The positive is that the opt in request and the immediate feedback addresses the problems of both participation bias and the brain fade that occurs because callers forget detail very quickly. But a particular problem jumped out at me. Callers are asked to rate their level of satisfaction on a scale of 1-10 and agents are measured, rewarded or re-trained on the numeric grade they receive from callers. The difficulty is that humans have a fundamental issue with a breakdown this fine that is compounded because there is no specific meaning as to what constitutes being satisfied. We've written about this problem a number of times, and it boils down to this: Satisfied in the mind of the consumer is not a positive rating overall. It is a neutral term akin to what you say to the server in a restaurant when the service wasn't anything special but you don't wish to have to defend a negative impression. So when The Hartford asks "On a scale of 1-10, how satisfied were you?" They fail to provide a definition of what neutral (presumably #5) really is. Without this frame of reference, the survey taker is left with a guess as to where to begin. Added to this problem is that humans can't really break their opinion down to that fine a granularity. They don't know if they were 67% satisfied or 82% satisfied, and so human nature takes over with a thought process along these lines: If service was memorably amazing, they might enter a top score. More than expected, and they want to say so, but is that a 9 an 8 a 7 or a 6? If service was ok and the caller was satisfied do they enter a 5 or should it be a 6? If disappointed was that a 4, 3 or 2?
Evidence suggests that the human brain will simplify a 10 point scale into 10 for outstanding, 7 above expectation, 5 as ok but nothing special, 3 as disappointing and 1 or zero as unacceptable. Thus The Hartford may well be making decisions on agent performance (10's get bonuses, 9's don't) on data that is too imprecise for statistical reliability.
This is exactly why VocaLabs uses a simple letter grade of A thru D for satisfaction scores in our SectorPulse studies. It is simply a more reasonable reflection of opinion than a too fine breakdown which is prone to individual subjective interpretation.
Posted by Rick Rappe
Evidence suggests that the human brain will simplify a 10 point scale into 10 for outstanding, 7 above expectation, 5 as ok but nothing special, 3 as disappointing and 1 or zero as unacceptable. Thus The Hartford may well be making decisions on agent performance (10's get bonuses, 9's don't) on data that is too imprecise for statistical reliability.
This is exactly why VocaLabs uses a simple letter grade of A thru D for satisfaction scores in our SectorPulse studies. It is simply a more reasonable reflection of opinion than a too fine breakdown which is prone to individual subjective interpretation.
Posted by Rick Rappe
Posted at 03:29 PM by | | | |