The Customer Service Survey
Things We've Learned
How to Design a Phone Menu
Wed - March 26, 2008 03:55 PM
The conventional wisdom for decades has been that the ideal phone menu (aka IVR tree, aka callflow) should have no more than five options at each level. Any more than that, it was thought, would overtax the short term memory of the caller and lead to confusion and poor experiences.
Never mind the clearly observable fact that menus designed according to this rule of thumb still led to confusion and poor experiences.
Broad vs. Deep
As far as I know, however, nobody had ever experimentally tested the question of how broad (=lots of choices at each level) or deep (=multiple levels and submenus) a design should be for a given application. Until now.
A recent article in the journal Human Factors, A Comparison of Broad Verses Deep Auditory Menu Structure, by Patrick Commarford and James Lewis (both of IBM) et al. tackles the question of broad vs. deep menus head-on and comes up with the surprising result that (at least for the application they tested) it works better to give callers a single menu with lots of options than to give callers fewer options in each menu but have submenus.
In other words, the "no more than five options" rule of thumb is wrong.
The particular application Commarford and Lewis used for this experiment is navigating e-mail through a speech application. They identified 11 different functions (Next, Previous, Delete, Reply, etc.), and built a "Broad" and a "Deep" version of the e-mail system. The "Broad" menu provided all options in a single menu, and the "Deep" version offered four top-level choices (Listen, Respond, Distribution, and Details) with the 11 options contained in the four top-level choices.
There were some design oddities about the "Deep" menus: for example, the "Delete" function was under the "Respond" menu (I can easily write a whole blog entry about the uniquely user-centric design process which led to this, ahem, unique design decision). In the final analysis, though, I think it's fair and reasonable to conclude that for this application a broad design is probably superior to a deep design, even though that leads to more than five choices in a single menu.
What Does It Mean
Some people may read this result to mean "make the menu as flat as possible," presumably by putting every option into one single uber-menu, but I think what it really means is "there's nothing magical about five menu options." There will always be a design decision about how broad or deep to make a tree, unless the system has few enough choices to put in a single menu. If the IVR needs to route the call among hundreds of different destinations or provide several dozen functions, a single menu will simply be unwieldy (of course, one thing to look at carefully in those situations is whether the total number of functions or destinations can be consolidated).
The key to usability is making sure that the menu choices are obvious (which may require including some choices in more than one menu, if they don't fit cleanly in one place), and that the caller's time and mental effort is respected. That means striking a balance between broad and deep, and the right balance will be different in every situation.
The real lesson of this research is that real-world usability testing is likely to show that general rules of thumb break down when applied to a specific design. At the end of the day, usability testing is critical to making sure that the design is right.
Posted by Peter Leppik
Broad vs. Deep
As far as I know, however, nobody had ever experimentally tested the question of how broad (=lots of choices at each level) or deep (=multiple levels and submenus) a design should be for a given application. Until now.
A recent article in the journal Human Factors, A Comparison of Broad Verses Deep Auditory Menu Structure, by Patrick Commarford and James Lewis (both of IBM) et al. tackles the question of broad vs. deep menus head-on and comes up with the surprising result that (at least for the application they tested) it works better to give callers a single menu with lots of options than to give callers fewer options in each menu but have submenus.
In other words, the "no more than five options" rule of thumb is wrong.
The particular application Commarford and Lewis used for this experiment is navigating e-mail through a speech application. They identified 11 different functions (Next, Previous, Delete, Reply, etc.), and built a "Broad" and a "Deep" version of the e-mail system. The "Broad" menu provided all options in a single menu, and the "Deep" version offered four top-level choices (Listen, Respond, Distribution, and Details) with the 11 options contained in the four top-level choices.
There were some design oddities about the "Deep" menus: for example, the "Delete" function was under the "Respond" menu (I can easily write a whole blog entry about the uniquely user-centric design process which led to this, ahem, unique design decision). In the final analysis, though, I think it's fair and reasonable to conclude that for this application a broad design is probably superior to a deep design, even though that leads to more than five choices in a single menu.
What Does It Mean
Some people may read this result to mean "make the menu as flat as possible," presumably by putting every option into one single uber-menu, but I think what it really means is "there's nothing magical about five menu options." There will always be a design decision about how broad or deep to make a tree, unless the system has few enough choices to put in a single menu. If the IVR needs to route the call among hundreds of different destinations or provide several dozen functions, a single menu will simply be unwieldy (of course, one thing to look at carefully in those situations is whether the total number of functions or destinations can be consolidated).
The key to usability is making sure that the menu choices are obvious (which may require including some choices in more than one menu, if they don't fit cleanly in one place), and that the caller's time and mental effort is respected. That means striking a balance between broad and deep, and the right balance will be different in every situation.
The real lesson of this research is that real-world usability testing is likely to show that general rules of thumb break down when applied to a specific design. At the end of the day, usability testing is critical to making sure that the design is right.
Posted by Peter Leppik
Simple Solutions
Mon - March 24, 2008 02:52 PM
There are some lessons which I need to learn over and over and over. The old engineering advice to KISS your problems ("Keep It Simple, Stupid") is one of them.
As a nerd and a geek, it's always tempting to reach for the technological fix for any problem. It always seems simplest to just propose something like, "You can just encapsulate the data in your XML file, and we'll process it and include it in our report. Of course, your telecom system will have to populate the field from an internal org chart....."
The problem, in this case, was how to include the name of an agent's supervisor in our database of scores from a Quality Assurance review. We already receive the call recording and the name of the agent from the client's telecom system, but the name of the supervisor isn't available in any system currently interfaced to the telecom system.
We started to research options, such as building a client-specific web interface for maintaining an organizational chart in our database, or auto-populating from a periodic batch dump, either of which would require several days (at a minimum) to develop and test, and some unknown amount of ongoing maintenance.
Then, "Hey, why not add one question to the QA review for the reviewer to enter the agent's supervisor? It's a small call center, they've only got a handful of supervisors, and the reviewer sits on the same floor as the agents--they've got better information about the org chart than anyone." A five minute change which, while not fully automated, will give exactly the results the client wants with almost zero headache.
KISS, indeed.
Posted by Peter Leppik
The problem, in this case, was how to include the name of an agent's supervisor in our database of scores from a Quality Assurance review. We already receive the call recording and the name of the agent from the client's telecom system, but the name of the supervisor isn't available in any system currently interfaced to the telecom system.
We started to research options, such as building a client-specific web interface for maintaining an organizational chart in our database, or auto-populating from a periodic batch dump, either of which would require several days (at a minimum) to develop and test, and some unknown amount of ongoing maintenance.
Then, "Hey, why not add one question to the QA review for the reviewer to enter the agent's supervisor? It's a small call center, they've only got a handful of supervisors, and the reviewer sits on the same floor as the agents--they've got better information about the org chart than anyone." A five minute change which, while not fully automated, will give exactly the results the client wants with almost zero headache.
KISS, indeed.
Posted by Peter Leppik
Factors for Good and Bad Service
Thu - February 21, 2008 01:41 PM
We're running a consumer attitudes survey right now to collect data about what consumers like and dislike about phone service these days.
Since the survey is still ongoing, I don't have final results, and I haven't had a chance to analyze anything yet, but here are a couple of interesting tidbits.
We gave participants a list of factors which they think are important for providing good customer service. The number one choice? "Make it easy to reach a live person (if necessary)," selected by 79% of participants so far. The #2 choice is "Be courteous, polite, and professional," chosen by 73% (participants are allowed to choose more than one option).
So far, nothing too surprising, though it is interesting to note that making it easy to reach a person was selected more often than "Wait only a short time to talk to a person" (selected by 67%). Consumers seem to be saying it's less frustrating to wait on hold than having to hunt for the option to talk to an agent.
Another question asked participants to choose factors which are most important in bad service. The top two options (a statistical tie at about 68% each) were "When you have to call more than once for the same problem, you have to start over from scratch each time," and "You're forced to go through repetitive or irrelevant steps to get help."
Given that these choices beat out even such dramatic problems as "The person you spoke to was rude or unprofessional," and "The company doesn't do things it promised to do, like fix a problem, issue a credit, etc." is really intriguing, and I'm guessing that the top two choices won out because they are much more common problems. Companies have (for the most part) gotten pretty good with call monitoring, coaching, and other QA problems, so rude agents and broken promises don't happen as often as they once did.
On the other hand, the top two bad service issues have the common thread of being systemic to the operations of a call center. The roots of these problems are buried in a company's disinterest in looking at its customer service from a customer's perspective, so nobody ever asks key questions like, "How does this call come across to the customer from end-to-end, when I consider the IVR, agents, transfers, etc. as a whole rather than individually," and "Is this step necessary from the customer's perspective, or is it just a hoop we're making him jump through for the convenience of the company?"
I'll have more on this survey data in a few days when it's completed.
Posted by Peter Leppik
We gave participants a list of factors which they think are important for providing good customer service. The number one choice? "Make it easy to reach a live person (if necessary)," selected by 79% of participants so far. The #2 choice is "Be courteous, polite, and professional," chosen by 73% (participants are allowed to choose more than one option).
So far, nothing too surprising, though it is interesting to note that making it easy to reach a person was selected more often than "Wait only a short time to talk to a person" (selected by 67%). Consumers seem to be saying it's less frustrating to wait on hold than having to hunt for the option to talk to an agent.
Another question asked participants to choose factors which are most important in bad service. The top two options (a statistical tie at about 68% each) were "When you have to call more than once for the same problem, you have to start over from scratch each time," and "You're forced to go through repetitive or irrelevant steps to get help."
Given that these choices beat out even such dramatic problems as "The person you spoke to was rude or unprofessional," and "The company doesn't do things it promised to do, like fix a problem, issue a credit, etc." is really intriguing, and I'm guessing that the top two choices won out because they are much more common problems. Companies have (for the most part) gotten pretty good with call monitoring, coaching, and other QA problems, so rude agents and broken promises don't happen as often as they once did.
On the other hand, the top two bad service issues have the common thread of being systemic to the operations of a call center. The roots of these problems are buried in a company's disinterest in looking at its customer service from a customer's perspective, so nobody ever asks key questions like, "How does this call come across to the customer from end-to-end, when I consider the IVR, agents, transfers, etc. as a whole rather than individually," and "Is this step necessary from the customer's perspective, or is it just a hoop we're making him jump through for the convenience of the company?"
I'll have more on this survey data in a few days when it's completed.
Posted by Peter Leppik
Confusing Statistics
Thu - June 28, 2007 10:49 AM
My formal education is in physics, and I have considerable professional experience in surveying and finance. One common thread between all these fields is the importance of statistics.
I've noticed that many different fields developed the same statistical methods independently from each other, and as a result, the same statistical concepts are called different things in different fields, using different mathematical symbols and notation.
For example, the same statistical distribution is often called a "Gaussian Distribution" by physicists, a "Bell Shaped Curve" by social scientists, and a "Normal Distribution" by mathematicians. In other words, one calls it by the name of one of the guys who developed the math, another uses a descriptive term, and the third makes a value judgment.
(Thanks to David for this particular example).
Consider also the use of the terms "average," "mean," and "expectation value" to refer to identical statistical concepts.
In many cases, the same statistical methods were first developed by guys in the Renaissance trying to figure out how to win at dice, rediscovered in the 19th century by physicists working on thermodynamics, and then discovered a third time by social scientists.
The result is unnecessary confusion and overlap: a "Statistics for Finance" textbook will contain much the same material as a "Statistics for Physics" text, but because they use different names and notation, it may take a while for a student to realize that he's re-learning the same concepts.
Posted by Peter Leppik
For example, the same statistical distribution is often called a "Gaussian Distribution" by physicists, a "Bell Shaped Curve" by social scientists, and a "Normal Distribution" by mathematicians. In other words, one calls it by the name of one of the guys who developed the math, another uses a descriptive term, and the third makes a value judgment.
(Thanks to David for this particular example).
Consider also the use of the terms "average," "mean," and "expectation value" to refer to identical statistical concepts.
In many cases, the same statistical methods were first developed by guys in the Renaissance trying to figure out how to win at dice, rediscovered in the 19th century by physicists working on thermodynamics, and then discovered a third time by social scientists.
The result is unnecessary confusion and overlap: a "Statistics for Finance" textbook will contain much the same material as a "Statistics for Physics" text, but because they use different names and notation, it may take a while for a student to realize that he's re-learning the same concepts.
Posted by Peter Leppik
Ten Dumb Things about Customer Service
Mon - March 12, 2007 03:42 PM
Companies do a lot of inexplicable things when it comes to running their customer service operations.
Here are ten common practices which industry outsiders often don't believe can possibly be true (in no particular order):
Posted by Peter Leppik
- If you hear "this call may be recorded for quality assurance," you can be assured that nobody ever listens to the vast majority of call recordings.
- Most recording systems aren't even capable of recording the entire call, and only catch the live agent portion.
- Many call center agents are paid more to hang up on you than to take care of your problem.
- Most automated customer service systems are never tested for ease-of-use, even after customers complain.
- Most companies at least review satisfaction survey results, but very few ever make any changes based on the feedback in the survey.
- In many companies, the call center does not communicate with other departments: Marketing doesn't inform customer service about new promotions or ads, product development doesn't hear about customer complaints, etc.
- Many companies view customer service as a "weed-out" department, where inexperienced employees are either quickly promoted (if they show promise) or fired (if they don't). Few companies have a meaningful career path which stays inside customer service.
- Senior management is often genuinely surprised to learn that the front-line customer service representatives are inexperienced, unqualified, and/or unmotivated.
- The vast majority of corporate executives believe that their company provides above-average customer service.
- Live agents and automated customer service are usually managed completely separately, even though a typical customer involves both. Nobody ever reviews an entire call from end to end.
Posted by Peter Leppik
Control
Thu - February 22, 2007 11:04 PM
I'm at SpeechTEK West this week, and I spent much of the day today in an all-day brainstorming session with a group of the top user interface designers in the world of automated customer service.
Nominally, the topic was how to improve error handling in speech recognition systems.
But during the session, something which has been banging around in the back of my head for a long time finally crystalized: The key to excellent customer service is letting the customer control the call.
The biggest customer complaints about bad service--especially bad automatedcustomer service--generally revolve around the company trying to force the customer down a path the customer doesn't want to follow. Roadblocks to reaching a live agent are the biggie, of course. But overly-scripted agents, asking irrelevant questions, and even unexpectedly transferring to an agent when the self-service system fails are all in the same category.
Before a customer calls, he already knows what he wants to accomplish, and (often) an idea of the steps he needs to follow to get that task done. Maybe the customer's idea of how to get the task done isn't really practical, but the idea exists, and anything which pushes the customer off his mental "script" of the call is going to cause frustration. If he's calling about a billing problem, he expects to have to talk to a human and explain the problem. If he's calling to find the hours of a local bank branch, he doesn't expect to have to enter an account number.
The challenge for the company is to handle that customer's call efficiently without knowing in advance why the customer called. What's more, the customer is a whole lot smarter than the automated customer service system which (in the majority of cases) will first attempt to handle the call.
So by letting the customer keep control of the call, the company is both taking advantage of the customer's knowledge of why he called, and also the customer's superior intelligence in how to handle the task.
In many cases, the philosophy of letting the customer control the call is mostly, well, philosophical. An automated call will always progress from state to state based on customer inputs; and customers will often want to do things which are impossible or against business rules. But there are some practical implications:
* Don't try to force customers down a particular path. Encourage where appropriate, but always leave an out: "You can make your balance transfer in our self-service system, or dial zero at any time to connect to an agent. Let's begin...."
* Make sure there's always a way to back up, start over, or bail out.
* Ask for information only when needed.
* Enlist the customer's help by explaining what's going on and why. For example, if the speech recognition is having a hard time, don't just drop the call into an agent queue, give the customer a choice: "I'm sorry we're having so much trouble on this call. If you want to start over with a live agent, just press zero at any time. Otherwise, we can keep trying self-service...."
Posted by Peter Leppik
But during the session, something which has been banging around in the back of my head for a long time finally crystalized: The key to excellent customer service is letting the customer control the call.
The biggest customer complaints about bad service--especially bad automatedcustomer service--generally revolve around the company trying to force the customer down a path the customer doesn't want to follow. Roadblocks to reaching a live agent are the biggie, of course. But overly-scripted agents, asking irrelevant questions, and even unexpectedly transferring to an agent when the self-service system fails are all in the same category.
Before a customer calls, he already knows what he wants to accomplish, and (often) an idea of the steps he needs to follow to get that task done. Maybe the customer's idea of how to get the task done isn't really practical, but the idea exists, and anything which pushes the customer off his mental "script" of the call is going to cause frustration. If he's calling about a billing problem, he expects to have to talk to a human and explain the problem. If he's calling to find the hours of a local bank branch, he doesn't expect to have to enter an account number.
The challenge for the company is to handle that customer's call efficiently without knowing in advance why the customer called. What's more, the customer is a whole lot smarter than the automated customer service system which (in the majority of cases) will first attempt to handle the call.
So by letting the customer keep control of the call, the company is both taking advantage of the customer's knowledge of why he called, and also the customer's superior intelligence in how to handle the task.
In many cases, the philosophy of letting the customer control the call is mostly, well, philosophical. An automated call will always progress from state to state based on customer inputs; and customers will often want to do things which are impossible or against business rules. But there are some practical implications:
* Don't try to force customers down a particular path. Encourage where appropriate, but always leave an out: "You can make your balance transfer in our self-service system, or dial zero at any time to connect to an agent. Let's begin...."
* Make sure there's always a way to back up, start over, or bail out.
* Ask for information only when needed.
* Enlist the customer's help by explaining what's going on and why. For example, if the speech recognition is having a hard time, don't just drop the call into an agent queue, give the customer a choice: "I'm sorry we're having so much trouble on this call. If you want to start over with a live agent, just press zero at any time. Otherwise, we can keep trying self-service...."
Posted by Peter Leppik
Effective Surveys
Fri - February 9, 2007 01:57 PM
I've been asked to do a presentation for the local chapter of SOCAP (that's the Society of Consumer Affairs Professionals) in six weeks or so. The topic will be how to make surveys more effective.
The undercurrent, according to Mary Leary, the charming head of programming for the Minneapolis SOCAP chapter, is that a lot of their members have become frustrated with how survey data is being used (or rather, not being used) in their organizations. People go through significant effort to collect and compile survey data and look for ways to improve, only to have their results routinely ignored by decision-makers who have the ability to act on the survey results.
This is a problem I've observed a lot, and the bad news is that there's no simple solution.
In order for a survey to be effective, it has to be part of a much larger feedback loop, which I made the focus of Chapter 2 of Gourmet Customer Service. The steps are:
1) Set Goals
2) Gather Data
3) Take Action
4) Validate Improvements
5) Go back to step 1
The surveys take place in steps 2 and 4 (validation means measuring whether the action in step 3 had any actual effect). The challenge is that this loop has to be something that the organization as a whole is committed to. Steps 1 and 2 are often as far as anyone gets, because if there's no commitment to change, it's always easier to do nothing than something. If there's a short-term push for improvement then a company may get as far as step 3, but without validation there's no way to know if the changes actually improved things, and without closing the loop by setting new goals, any improvement is likely to be short-lived.
The good news is that this process isn't rocket science. The magic comes in the commitment to make changes, measure results, and then go back and make more changes--there's nothing special about the expertise of the consultants or the three-letter acronyms used. Over time, even a marginally competent department will figure out which changes are improving things and which aren't, and improve the overall service level.
So hard work and corporate commitment? Yes. Special techniques and magic consulting wizards? Not necessary.
Posted by Peter Leppik
This is a problem I've observed a lot, and the bad news is that there's no simple solution.
In order for a survey to be effective, it has to be part of a much larger feedback loop, which I made the focus of Chapter 2 of Gourmet Customer Service. The steps are:
1) Set Goals
2) Gather Data
3) Take Action
4) Validate Improvements
5) Go back to step 1
The surveys take place in steps 2 and 4 (validation means measuring whether the action in step 3 had any actual effect). The challenge is that this loop has to be something that the organization as a whole is committed to. Steps 1 and 2 are often as far as anyone gets, because if there's no commitment to change, it's always easier to do nothing than something. If there's a short-term push for improvement then a company may get as far as step 3, but without validation there's no way to know if the changes actually improved things, and without closing the loop by setting new goals, any improvement is likely to be short-lived.
The good news is that this process isn't rocket science. The magic comes in the commitment to make changes, measure results, and then go back and make more changes--there's nothing special about the expertise of the consultants or the three-letter acronyms used. Over time, even a marginally competent department will figure out which changes are improving things and which aren't, and improve the overall service level.
So hard work and corporate commitment? Yes. Special techniques and magic consulting wizards? Not necessary.
Posted by Peter Leppik
SectorPulse, the Background
Tue - January 16, 2007 01:21 PM
As most readers know, one of our key services is the use of our constantly renewing pool of "panelists" to act as surrogate customers by calling into a customer service operation, performing some task, having the call recorded, and completing a feedback survey on the experience. That panelist pool contains about 80,000 people and early on we realized these panelists are also real consumers. And most, when completing the demographic profile on sign up, have told us they own a cell phone. So under SectorPulse-Wireless, we ask that whenever a panelist needs to call their carrier for whatever reason, they call through VocaLabs. Before being connected, we ask a series of survey questions; and immediately on completion of their service call, we survey a second time asking many of the same questions.
This before/after survey allows a way to measure the impact a specific call has on consumer opinion, likelihood of switching vendors and other loyalty and service quality issues. While we have the capability, for security, we do not record SectorPulse calls unless prior arrangements have been made with a specific client.
Our Press Releases on SectorPulse show a head to head comparison in the categories of overall caller satisfaction as well as the probability of completing a callers business without having to call back. But we also track five additional categories: Automation is the percent of callers able to complete their business without speaking with a live agent. Caller Frustration is a measure of the difficulty in reaching live help. Call time, or more accurately named "Transaction time" measures the total amount of time including any multiple calls required to complete a piece of business. And we track customer loyalty both pre and post phone call.
A summary of the comparative results in these latter five categories is a part of the available Executive Summary, and in varying levels of depth, we can also provide historical data, and consumer feedback to better understand why/how caller opinions get formed plus the impact good/or poor service has on company revenues.
The same criteria apply to SectorPulse-Financial that tracks customer service quality among major US banks plus PayPal.
Feel free to call or drop us a note any time on how SectorPulse might be used and how we can aid in making good service a more likely result.
Posted by Rick Rappe
Our Press Releases on SectorPulse show a head to head comparison in the categories of overall caller satisfaction as well as the probability of completing a callers business without having to call back. But we also track five additional categories: Automation is the percent of callers able to complete their business without speaking with a live agent. Caller Frustration is a measure of the difficulty in reaching live help. Call time, or more accurately named "Transaction time" measures the total amount of time including any multiple calls required to complete a piece of business. And we track customer loyalty both pre and post phone call.
A summary of the comparative results in these latter five categories is a part of the available Executive Summary, and in varying levels of depth, we can also provide historical data, and consumer feedback to better understand why/how caller opinions get formed plus the impact good/or poor service has on company revenues.
The same criteria apply to SectorPulse-Financial that tracks customer service quality among major US banks plus PayPal.
Feel free to call or drop us a note any time on how SectorPulse might be used and how we can aid in making good service a more likely result.
Posted by Rick Rappe
Net Promoter- The Ultimate Question
Thu - October 12, 2006 02:07 PM
You needn't be a greybeard in business to agree with the experience that business fads come and go. I'll go out on a limb and say that there really hasn't been much new written on the importance of service since Dale Carnegie wrote "How to Win Friends and Influence People" what 80 years ago?
Over the last perhaps two years there has been considerable buzz resulting from author Fred Reichfield's book "Ultimate Question". The gist is that the best measure of company performance can be had by asking the simple question: "Would you recommend company X to others?"
Great question. We've been asking it for years, but missed the opportunity to get our 15 minutes of fame by writing a book about it.
The topic came up again in a conversation yesterday, with the unfortunate theme that asking this question was all the customer satisfaction surveying that needs to be done. Sigh... Would that this were true.
Had we written the book, we would have said that "Would you recommend..." is a fine high level indicator of company image, but that it is an inadequate measure of customer satisfaction. While it answers how customers feel, it doesn't tell you what you did, right or wrong to cause the opinion to form, why it happened and when it happened. Without knowing what, how, why and when, how can you improve?
Posted by Rick Rappe
Great question. We've been asking it for years, but missed the opportunity to get our 15 minutes of fame by writing a book about it.
The topic came up again in a conversation yesterday, with the unfortunate theme that asking this question was all the customer satisfaction surveying that needs to be done. Sigh... Would that this were true.
Had we written the book, we would have said that "Would you recommend..." is a fine high level indicator of company image, but that it is an inadequate measure of customer satisfaction. While it answers how customers feel, it doesn't tell you what you did, right or wrong to cause the opinion to form, why it happened and when it happened. Without knowing what, how, why and when, how can you improve?
Posted by Rick Rappe
"Cecil, Connect Me To Kellogg-257 Please."*
Wed - September 20, 2006 04:02 PM
I read an interesting article this morning on SpeechTek's online magazine titled "Is Paul English Right?" by Wes Hayden, CEO of Genesys Telecommunications Labs.
Peter called it a classic example of a self-serving editorial, but as the VocaLabs sales guy, I've no problem with promoting one's own opinion, especially because Mr. Hayden answers the question as Yes, Paul is right. I agree.
My problem is that the author skirts around WHY Paul is right, and he makes some questionable comments in the process.
People have no problem with self service customer care so long as it meets their needs. It is bad IVR that is the cause of the problem. No vendor including Genesys is blame free. This is a subject I've written and spoken about many times: "If you build a system to save money over serving the caller, it will do neither. But if you build it with a caller focus, it will do both."
In attempting to explain the "symbiotic" relationship that should exist between the IVR and the human CSR, he uses the recent Citibank "dial 0"-get a human advertising campaign as both evidence of consumer backlash against self service customer care and as evidence of this symbiotic relationship. Perhaps this wasn't such a good example. Recall that VocaLabs has a constantly running survey on service quality among the nationwide banks, and Citibank has consistently had the lowest overall satisfaction scores.
During the period that the dial "0"- get a human ad campaign was running, all the banks except Citi improved their satisfaction scores, Citi's dropped such that caller satisfaction with Citi was actually worse after a service call than immediately before. I have not investigated deeply enough to make an absolute statement, but my sense is that the ad campaign raised caller expectations, and when they actually called and found service wasn't any better, Citi customers were more disappointed than had the campaign never run. (In fact, I suspect that more callers tried to reach a live operator after seeing the ad and Citi failed to add enough people to the phones to allow for it.)
The author then moves on to tout how speech technology allows the humanization of the computer voice, inducing the caller to stay with the machine. And he accepts this as a positive. Not necessarily. There is a quite strong argument that giving the machine human attributes can be a bad idea; that it insults the caller since they know they are dealing with a machine incapable of real empathy, and that humanization actually distracts from the caller completing their business with a minimum of fuss.
He then closes by comparing our resistance to talking to a machine to a half century ago when there was some resistance to dial telephones replacing speaking the phone number to the local operator to be connected. We think this is a poor analogy, as Peter said to me this AM: "If only speech recognition DID work as well as dial phones, then there would be no need for a Paul English."
[*In the 50's our small town operator was a lady named Cecil Stuart. The town was so small Cecil could recognize voices and where the call was coming from. Didn't really need numbers. All I had to do was pick up the phone and ask for Grandma, and she'd know what connection to make. We knew enough not to call after Cecil went to bed unless it was a real emergency because she'd tell our parents.]
Posted by Rick Rappe
My problem is that the author skirts around WHY Paul is right, and he makes some questionable comments in the process.
People have no problem with self service customer care so long as it meets their needs. It is bad IVR that is the cause of the problem. No vendor including Genesys is blame free. This is a subject I've written and spoken about many times: "If you build a system to save money over serving the caller, it will do neither. But if you build it with a caller focus, it will do both."
In attempting to explain the "symbiotic" relationship that should exist between the IVR and the human CSR, he uses the recent Citibank "dial 0"-get a human advertising campaign as both evidence of consumer backlash against self service customer care and as evidence of this symbiotic relationship. Perhaps this wasn't such a good example. Recall that VocaLabs has a constantly running survey on service quality among the nationwide banks, and Citibank has consistently had the lowest overall satisfaction scores.
During the period that the dial "0"- get a human ad campaign was running, all the banks except Citi improved their satisfaction scores, Citi's dropped such that caller satisfaction with Citi was actually worse after a service call than immediately before. I have not investigated deeply enough to make an absolute statement, but my sense is that the ad campaign raised caller expectations, and when they actually called and found service wasn't any better, Citi customers were more disappointed than had the campaign never run. (In fact, I suspect that more callers tried to reach a live operator after seeing the ad and Citi failed to add enough people to the phones to allow for it.)
The author then moves on to tout how speech technology allows the humanization of the computer voice, inducing the caller to stay with the machine. And he accepts this as a positive. Not necessarily. There is a quite strong argument that giving the machine human attributes can be a bad idea; that it insults the caller since they know they are dealing with a machine incapable of real empathy, and that humanization actually distracts from the caller completing their business with a minimum of fuss.
He then closes by comparing our resistance to talking to a machine to a half century ago when there was some resistance to dial telephones replacing speaking the phone number to the local operator to be connected. We think this is a poor analogy, as Peter said to me this AM: "If only speech recognition DID work as well as dial phones, then there would be no need for a Paul English."
[*In the 50's our small town operator was a lady named Cecil Stuart. The town was so small Cecil could recognize voices and where the call was coming from. Didn't really need numbers. All I had to do was pick up the phone and ask for Grandma, and she'd know what connection to make. We knew enough not to call after Cecil went to bed unless it was a real emergency because she'd tell our parents.]
Posted by Rick Rappe
Fool Me Once, But Not Twice
Thu - August 24, 2006 06:37 PM
I just finished sitting in on a Webinar presented by Nortel and featuring their relationship with Witness Systems. The title "Learning How to Measure Customer Loyalty" had caught my attention. In sum, the presentation didn't exactly match the title, but I do have some observations to share.
I want to like the products of Witness Systems. They haven't rested on their laurels, quickly realizing that sales of call recording technology stalled because it has been too human resource intense to listen to many thousands of contact center calls in order to find problems. In truth, I regularly hear that companies actually only monitor 1-5% of calls, and in more than a few cases, not even that.
So Witness has continued to add value to their product with capabilities such as matching the screen images the CSR was looking at with the call recording. Their systems now integrate with CRM technology so that reports can be generated including client data. They now include the ability to monitor e-mails as well as phone calls, and are even beginning to deploy speech analytics to search through the call recordings for key words and phrases to lower the need for a human to monitor every recording. Impressive stuff.
But when the Witness speaker got to that point in his talk about the importance of including caller surveys in the quality assurance process, that's when I fell off the bus. Statisticians call it "confirmation bias" which simply means that if we hear or read something that fits our belief system, we easily assimilate the new information. But if we hear something that doesn't readily fit, we tend to discard the data. A corollary to confirmation bias is that we might be willing to accept the pronouncements of a speaker until they say one incorrect thing on any subject for which we hold strong opinions or good knowledge. Our brains say: "Oops, if I can't trust the speaker on this thing I know about, what else has he/she said that might also be wrong?
In this case, one of the opening slides and remarks was that a satisfaction survey can't tell you about brand loyalty. Horse apples. Of course it can, you just need to know what and how to ask the questions. So my warning antenna was already up and when the speaker talked of the importance of surveys in the quality assurance process, I began to soften. But he quickly blew it by recommending the end of call survey (which Witness has the technical ability to provide), without any concession to the fact that this is a terrible method by which to gauge true caller opinion. (It excludes anyone who hangs up or the CSR steers away and so tends to skew satisfaction scores upward by an estimated 20-40%.)
That's too bad. I want to continue to think highly of Witness Systems technology, but since I can't trust them on this point, where else might I be wrong about them?
Posted by Rick Rappe
So Witness has continued to add value to their product with capabilities such as matching the screen images the CSR was looking at with the call recording. Their systems now integrate with CRM technology so that reports can be generated including client data. They now include the ability to monitor e-mails as well as phone calls, and are even beginning to deploy speech analytics to search through the call recordings for key words and phrases to lower the need for a human to monitor every recording. Impressive stuff.
But when the Witness speaker got to that point in his talk about the importance of including caller surveys in the quality assurance process, that's when I fell off the bus. Statisticians call it "confirmation bias" which simply means that if we hear or read something that fits our belief system, we easily assimilate the new information. But if we hear something that doesn't readily fit, we tend to discard the data. A corollary to confirmation bias is that we might be willing to accept the pronouncements of a speaker until they say one incorrect thing on any subject for which we hold strong opinions or good knowledge. Our brains say: "Oops, if I can't trust the speaker on this thing I know about, what else has he/she said that might also be wrong?
In this case, one of the opening slides and remarks was that a satisfaction survey can't tell you about brand loyalty. Horse apples. Of course it can, you just need to know what and how to ask the questions. So my warning antenna was already up and when the speaker talked of the importance of surveys in the quality assurance process, I began to soften. But he quickly blew it by recommending the end of call survey (which Witness has the technical ability to provide), without any concession to the fact that this is a terrible method by which to gauge true caller opinion. (It excludes anyone who hangs up or the CSR steers away and so tends to skew satisfaction scores upward by an estimated 20-40%.)
That's too bad. I want to continue to think highly of Witness Systems technology, but since I can't trust them on this point, where else might I be wrong about them?
Posted by Rick Rappe
Oh Great! Still More Departments That Won't Talk to Each Other.
Fri - August 18, 2006 04:35 PM
I might be competing with the announcement Peter just made, but I wanted to get a new entry in as Peter will be out of the office for much of next week and the blog access software is on his laptop, so no new entries until the 24th.
A few weeks ago, we were advised that we shouldn't be focusing our sales efforts on the call center market, but rather on the marketing department. The idea being that our research provides customer insights perhaps of more interest there than to the contact center. Made sense to at least pursue a parallel sales track, so I joined the American Marketing Association and attended the first local gathering last night.
This was a networking gathering of several marketing oriented membership groups, and what struck me was how segmented "marketing" has become. I won't get all the association acronyms right, so won't try. But as I understood it, there are associations for PR professionals, another for advertising, another for marketing copy writers, another for marketing managers, another for consumer affairs and still another for "sales and marketing". Whew. In any given day, I do ALL those things. Now, I can concede that in some organizations there needs to be a differentiation between sales and marketing. I can also argue that everybody in a company is, or should be, a marketer of their business. We're in a more specialized business world today, but aren't six different local association chapters to break up what used to be a single discipline a bit extreme?
Or am I just behind the times?
Posted by Rick Rappe
This was a networking gathering of several marketing oriented membership groups, and what struck me was how segmented "marketing" has become. I won't get all the association acronyms right, so won't try. But as I understood it, there are associations for PR professionals, another for advertising, another for marketing copy writers, another for marketing managers, another for consumer affairs and still another for "sales and marketing". Whew. In any given day, I do ALL those things. Now, I can concede that in some organizations there needs to be a differentiation between sales and marketing. I can also argue that everybody in a company is, or should be, a marketer of their business. We're in a more specialized business world today, but aren't six different local association chapters to break up what used to be a single discipline a bit extreme?
Or am I just behind the times?
Posted by Rick Rappe
The Message is Getting Out
Tue - August 15, 2006 02:37 PM
Besides Peter, I also moderated a panel earlier in the SpeechTek convention on the latest in Voice User Interface (VUI) design of speech recognition systems. There was a short presentation on dialog phrasing with an emphasis on using language that is actually useful for the caller instead of being there to mollify the designer. (The caller doesn't care about "listen carefully as our menu options have changed"...just provide the instructions, etc. etc.) Another speaker outlined a design that included voice print authentication for the US Justice dept. A third speaker charged with improving the 1-800-ask-USPS IVR spoke of the importance of satisfaction testing, and Melissa Dougherty of Voicepartners wrapped up the presentations emphasizing a holistic approach to customer care that includes executive buy in to increase the odds of building caller centric designs.
That is a vast difference from just a few years ago when VUI designers felt testing for caller ease of use and satisfaction was either an afterthought, or not considered necessary.
I wrapped up the discussion by including the admonition that while pleased that the industry is now recognizing the importance of user focus and thorough design tests of IVR applications, it is important to understand how-what-when and why to test so as to avoid misleading results. I pointed out that surveying and consumer feedback assessment is a science itself and used the examples of the end-of-call survey that inflates satisfaction scores* and that later feedback studies are limited in the delivery of actionable improvements because humans forget details so quickly.
*Sample bias arises because dissatisfied callers who hang up never get to the survey, and agents can manipulate who participates. The result is our estimation that the end-of-call survey inflates true caller satisfaction scores by 20-40 percent.
Posted by Rick Rappe
[UPDATE: 8/25/06 Corrected the name of Melissa's company.]
I wrapped up the discussion by including the admonition that while pleased that the industry is now recognizing the importance of user focus and thorough design tests of IVR applications, it is important to understand how-what-when and why to test so as to avoid misleading results. I pointed out that surveying and consumer feedback assessment is a science itself and used the examples of the end-of-call survey that inflates satisfaction scores* and that later feedback studies are limited in the delivery of actionable improvements because humans forget details so quickly.
*Sample bias arises because dissatisfied callers who hang up never get to the survey, and agents can manipulate who participates. The result is our estimation that the end-of-call survey inflates true caller satisfaction scores by 20-40 percent.
Posted by Rick Rappe
[UPDATE: 8/25/06 Corrected the name of Melissa's company.]
What Do You Call a Room Full of Engineers?
Fri - August 11, 2006 09:51 AM
Well, we beat the terrorism threat by a few hours and made it home from the Speech Technology conference in NYC almost on time, and one of my comments to Peter was that we got at least a week's worth of blog entries out of our visit, which I'll share over the next few days.
Peter was on a panel discussion/session the last hour of the last day of the conference, and as I sat in, I was surprised by the turn out. There were at least 75 Speech Recognition system designers in attendance. The discussions were relevant, but when the moderator brought up the topic of Paul English's "standards" idea (see Peter's earlier entry), I wanted to shout NO! Asking 75 engineers in one room to discuss the practicality of design standards, let alone what those standards should be was an invitation to disaster. Sure enough, 20 minutes later after some pointed remarks, no one could even agree on the definition of a "standard" let alone what they should be. Reminded me of the line: "Ask an engineer what time it is, and he'll build you a watch." With 75 designers in the room, I give the moderator credit for calling a cease fire and ending the session in time for us to catch our flight.
More thoughts from SpeechTek later. Stay tuned.
Posted by Rick Rappe
More thoughts from SpeechTek later. Stay tuned.
Posted by Rick Rappe
Doing the Wrong Things Excellently
Wed - August 2, 2006 11:53 AM
One of the difficulties in selling usability and customer satisfaction testing, particularly for automated self service customer care systems, is getting clients to understand that there are different test types depending on what answers are needed.
One variation on testing for technical performance is becoming increasingly important as self service systems advance in capability and also complexity. As callers progress through several layers of system prompts to accomplish their task, the number of options offered also increase, such as asking for a live operator at any stage, or finishing one task and needing to back up to accomplish another one. Traversal testing is the process of insuring that any action at any prompt does what it is supposed to.
Time and again we encounter systems that work fine technically, but that fail to meet caller needs. It is more often bad design, not faulty technical performance that frustrates callers, and Traversal Testing tells you little to nothing about whether a technically functioning system is actually serving the caller well.
For example, IQ Services does automated testing of self-service systems, and they have many very happy clients. Their special area of expertise is system monitoring for technical performance. Are all the trunk phone lines working? Is the link between the system and the billing computer for access to customer records efficiently communicating the data?
IQS has just announced a new traversal test service they call Feature/Function Testing. Good for them. But they crank up the marketing hyperbole a bit too much when they claim that: "Feature/Function Testing will give you confidence that...your customers will have the best possible experience using your system."
We have tested hundreds of systems for usability, and so say with confidence that a system can be doing the wrong things just fine. In order to truly provide the best customer experience, you need to performa a variety of tests, including performance testing and traversal testing, and also usability testing with live callers, and ongoing caller surveys.
Anything less is like looking at the world through a paper-towel tube.
Posted by Rick Rappe
Time and again we encounter systems that work fine technically, but that fail to meet caller needs. It is more often bad design, not faulty technical performance that frustrates callers, and Traversal Testing tells you little to nothing about whether a technically functioning system is actually serving the caller well.
For example, IQ Services does automated testing of self-service systems, and they have many very happy clients. Their special area of expertise is system monitoring for technical performance. Are all the trunk phone lines working? Is the link between the system and the billing computer for access to customer records efficiently communicating the data?
IQS has just announced a new traversal test service they call Feature/Function Testing. Good for them. But they crank up the marketing hyperbole a bit too much when they claim that: "Feature/Function Testing will give you confidence that...your customers will have the best possible experience using your system."
We have tested hundreds of systems for usability, and so say with confidence that a system can be doing the wrong things just fine. In order to truly provide the best customer experience, you need to performa a variety of tests, including performance testing and traversal testing, and also usability testing with live callers, and ongoing caller surveys.
Anything less is like looking at the world through a paper-towel tube.
Posted by Rick Rappe