The Customer Service Survey
VocaLabs' weblog providing news and commentary on the challenges of providing good customer service.
Why Do I Love Apple?
Thu - May 1, 2008 01:04 PM in
I've been out of the blogging loop for a couple weeks, thanks to some travel and lots of sick kids at home. So, apologies to our loyal reader (Hi, Mom!) for the long dry spell.
I stumbled across an interesting analysis yesterday on the Equity Private blog, where the anonymous author wonders why she loves Apple so much. After a lengthy analysis in which she talks about how many other companies try to maintain market share by building customer-unfriendly barriers around their products, the summary captures it perfectly:
Many big consumer-oriented companies can learn a lot from this statement.
Posted by Peter Leppik
Tax Day
Tue - April 15, 2008 10:21 AM in
It's April 15th once again, here in the United States that's Tax Day.
The worst IVR system we ever tested was an IRS system called "TeleTax," which is still in use: 800-829-4477. We did this test as part of the calibration of our process many years ago, so it wasn't a client project (and hence I'm not under NDA about it).
TeleTax lets callers listen to prerecorded informational messages on about 100 topics, and the interface is almost completely unusable. In order to hear a given message, you need to know a three-digit code identifying the particular recording you want. For example, "100" plays a recording listing the kinds of IRS help which are available to taxpayers.
If you don't happen to know the code for the information you want, there is an index available--but just navigating to the index takes over two minutes thanks to all the helpful hints about getting information from the IRS' web site (not a bad strategy, actually). No live help is available on the TeleTax line.
Worse, the system appears to have been generally neglected (again, not surprising, since I doubt many taxpayers actually find this a useful resource) with some bugs and general degradation. When I entered "253" to access a recording about alternative tax forms, I got the message "That is not a valid entry. Please enter your social security number." Apparently at some point, "2" was turned into a universal option for transferring out to the refund status line--probably when someone discovered that the vast majority of callers to TeleTax actually wanted to check on their tax refund and called the wrong number--but the entries were not reindexed to take this change into account.
So no matter how bad the IVR jail at your least-favorite bank or airline, you can take comfort in this: the IRS is worse.
Posted by Peter Leppik
TeleTax lets callers listen to prerecorded informational messages on about 100 topics, and the interface is almost completely unusable. In order to hear a given message, you need to know a three-digit code identifying the particular recording you want. For example, "100" plays a recording listing the kinds of IRS help which are available to taxpayers.
If you don't happen to know the code for the information you want, there is an index available--but just navigating to the index takes over two minutes thanks to all the helpful hints about getting information from the IRS' web site (not a bad strategy, actually). No live help is available on the TeleTax line.
Worse, the system appears to have been generally neglected (again, not surprising, since I doubt many taxpayers actually find this a useful resource) with some bugs and general degradation. When I entered "253" to access a recording about alternative tax forms, I got the message "That is not a valid entry. Please enter your social security number." Apparently at some point, "2" was turned into a universal option for transferring out to the refund status line--probably when someone discovered that the vast majority of callers to TeleTax actually wanted to check on their tax refund and called the wrong number--but the entries were not reindexed to take this change into account.
So no matter how bad the IVR jail at your least-favorite bank or airline, you can take comfort in this: the IRS is worse.
Posted by Peter Leppik
Will The Airline Industry Put Itself Out of Business with Bad Service?
Fri - April 11, 2008 02:11 PM in
Yesterday, I took a sympathetic view towards all the American Airlines employees who are put in the miserable position today of trying to deal with the hundreds of thousands of passengers stranded by American's maintenance issues (which have--so far--cancelled over 3,000 flights this week).
Jeff Jarvis writes this morning that the airline business is fundamentally broken: the experience has become so completely and uniformly miserable that customers will disappear unless something radical happens.
I have to agree with Jeff on many points. I should be an ideal customer for air travel: I love to fly, I have a pilot's license, I often have out-of-state business, and I enjoy visiting other cities. But the past few years I've been avoiding air travel whenever possible. I haven't taken a family vacation in six years where we had to fly anywhere (instead, we go places we can drive or take a train), and I try to minimize business travel involving flying anywhere.
Commercial air travel has simply become a miserable experience. From check-in to security to the packed planes, the process is impersonal, unpleasant, and dehumanizing; and that's when nothing goes wrong. Things get even worse when flights are delayed or cancelled, or your plans change.
I lay the blame for this state of affairs squarely at the feet of the airlines themselves. For the past 30 years, they've been training travelers that the only thing that matters is price--despite the fact that many people can and do pay a premium for premium service (and the airlines depend on these customers to make ends meet). At the end of the day, though, there's only so much you can do to provide good service when two-thirds of the passengers on a given flight will always go for the cheapest ticket, no matter what.
Now that the price of jet fuel is soaring, the airlines are discovering just what this lowest-fare mentality is doing to them. They have very little ability to raise prices without losing huge market share, and four airlines have gone bankrupt in just the past two weeks.
As for myself, I gladly pay a 50% premium to fly on my favorite airline, Midwest, whenever I can. Those are the only flights I don't dread.
Posted by Peter Leppik
I have to agree with Jeff on many points. I should be an ideal customer for air travel: I love to fly, I have a pilot's license, I often have out-of-state business, and I enjoy visiting other cities. But the past few years I've been avoiding air travel whenever possible. I haven't taken a family vacation in six years where we had to fly anywhere (instead, we go places we can drive or take a train), and I try to minimize business travel involving flying anywhere.
Commercial air travel has simply become a miserable experience. From check-in to security to the packed planes, the process is impersonal, unpleasant, and dehumanizing; and that's when nothing goes wrong. Things get even worse when flights are delayed or cancelled, or your plans change.
I lay the blame for this state of affairs squarely at the feet of the airlines themselves. For the past 30 years, they've been training travelers that the only thing that matters is price--despite the fact that many people can and do pay a premium for premium service (and the airlines depend on these customers to make ends meet). At the end of the day, though, there's only so much you can do to provide good service when two-thirds of the passengers on a given flight will always go for the cheapest ticket, no matter what.
Now that the price of jet fuel is soaring, the airlines are discovering just what this lowest-fare mentality is doing to them. They have very little ability to raise prices without losing huge market share, and four airlines have gone bankrupt in just the past two weeks.
As for myself, I gladly pay a 50% premium to fly on my favorite airline, Midwest, whenever I can. Those are the only flights I don't dread.
Posted by Peter Leppik
Customer Service Nightmare
Thu - April 10, 2008 02:40 PM in
American Airlines has cancelled well over 2,000 flights over the past three days, after the FAA discovered that required maintenance had not been performed in a number of the airline's jets. That led to an immediate inspection of a huge fraction of the fleet, forcing many planes to be grounded until maintenance and inspections can be completed.
In this instance (unlike winter storms), the damage inflicted to American is entirely self-inflicted, so there's a limit to how sympathetic one can be to the company's plight. Still, this has to be the very definition of a call center manager's nightmare: hundreds of thousands of angry customers, systemwide disruption, and no way to help most people. American simply doesn't have the capacity to get all those stuck people to their destinations in a timely manner, even if it somehow miraculously got all the planes in the air immediately.
So to anyone working in American's call centers this week, you've got my sympathy. Good luck.
Posted by Peter Leppik
So to anyone working in American's call centers this week, you've got my sympathy. Good luck.
Posted by Peter Leppik
My brilliant moneymaking idea
Mon - April 7, 2008 02:31 PM in
I have an idea for a gizmo which will make me millions. It's a variation of those toy voice distorters which can make you sound like Darth Vader, a woman (if a man), or a man (if a woman).
My voice distorter will take any voice, no matter how bored-sounding, and make it sound like the speaker is Excited! and Enthused!
The market is call centers and telemarketers, which seem to have a terrible time training their agents to not sound like they're reading a script while they're reading a script.
It's the worst with telemarketers: I can usually tell within the first two words if a caller is trying to sell me something, just from the tone of voice. From that instant, I'm not hearing anything the telemarketer is saying, just looking for a pause big enough to say, "Sorrynotinterested" and hang up.
Going the other way, when I call customer service it's frustrating greeted by a bored, robotic-sounding agent, especially if he's asking irrelevant questions. It feels impersonal, and sends the message that the company doesn't care what I'm trying to say.
These aren't new problems, but apparently all attempts to train bored, scripted customer service reps to not sound bored and scripted have failed.
Given the usual approach in the customer service business of applying technological band-aids to cover up people problems, my Enthusiasm Machine can't fail to be a commercial success.
Posted! by! Peter! Leppik!
The market is call centers and telemarketers, which seem to have a terrible time training their agents to not sound like they're reading a script while they're reading a script.
It's the worst with telemarketers: I can usually tell within the first two words if a caller is trying to sell me something, just from the tone of voice. From that instant, I'm not hearing anything the telemarketer is saying, just looking for a pause big enough to say, "Sorrynotinterested" and hang up.
Going the other way, when I call customer service it's frustrating greeted by a bored, robotic-sounding agent, especially if he's asking irrelevant questions. It feels impersonal, and sends the message that the company doesn't care what I'm trying to say.
These aren't new problems, but apparently all attempts to train bored, scripted customer service reps to not sound bored and scripted have failed.
Given the usual approach in the customer service business of applying technological band-aids to cover up people problems, my Enthusiasm Machine can't fail to be a commercial success.
Posted! by! Peter! Leppik!
Today's Funny Tech Support Stories
Thu - April 3, 2008 01:31 PM in
Every now and then it's good to step back from the consumer horror stories about bad service and share some stories from the other side of the phone.
So here's a set of tech support stories from The Daily WTF. Next time you're stuck in voicemail jail, just remember that big companies don't have a monopoly on cluelessness.
Posted by Peter Leppik
Posted by Peter Leppik
As if wedding planning wasn't stressful enough....
Wed - April 2, 2008 02:40 PM in
It's been a long time since I got married, but I was always under the impression that part of the point of the preparations and what-not is to make every bride feel like she gets to be a princess for a short while. Of course, some brides take this too far (hence the "Bridezilla" phenomenon).
Someone at David's Bridal apparently didn't get the memo, though. Read on for the full story....
Posted by Peter Leppik
Posted by Peter Leppik
Newsletter #38 Available
Tue - April 1, 2008 02:09 PM in
The 38th issue of Quality Times, VocaLabs' quasi-periodic newsletter, is now available on our website.
In this issue, we discuss the recent research results about building broad vs. deep IVR menus.
Posted by Peter Leppik
Posted by Peter Leppik
User-Centric Design (to a fault)
Fri - March 28, 2008 01:35 PM in
A couple days ago, I wrote about a research paper comparing broad vs. deep menu designs for a phone menu. One quirk of the research was the uniquely user-centered design process which led to some rather (ahem) unique decisions.
The application was a system for browsing e-mail over the phone, and had eleven different functions for dealing with each message: Next, Previous, Repeat, Reply, Reply to All, Forward, Delete, List Recipients, Add Sender (to address book), Mark Unread, and Time and Date. The researchers were interested in comparing the usability of giving the caller all eleven options in a single menu (a "broad" design) vs. dividing the options into submenus (a "deep" design).
The menu structure they came up with for the "deep" design was:
Listen to Messages: Next, Previous, Repeat
Respond: Reply, Reply to All, Forward, Delete
Distribution: List Recipients, Add Sender
Message Details: Mark Unread, Time and Date
So if you wanted to go to the next message, you would first have to say, "Listen to Messages" and then choose the "Next" option from that submenu.
My immediate reaction to this design was along the lines of, "WTF? Why is 'Delete' in a submenu called 'Respond'? I would never look for the 'Delete' option under the 'Respond' menu, and that's probably one of the most used functions. What sane VUI designer would build this menu tree?"
It turns out that this menu structure wasn't built by any VUI design (sane or otherwise), but rather through a uniquely user-centric process which illustrates a hazard of blindly applying user data to the design process. Here's what they did:
Step 1: 26 users were asked to organize the eleven functions into logical groupings of five or fewer functions. Each user's groupings were analyzed, and an aggregated grouping was generated with the following groups:
1. Delete, Forward, Reply, Reply to All
2. Repeat, Next, Previous
3. Mark Unread, Time and Date
4. List Recipients, Add Sender
So far so good, though there's no reason why some functions (especially heavily-used ones like "delete") can't be included in multiple groups.
Step 2: 101 users (not the same ones as in Step 1) were given the four groups from Step 1 and asked to suggest a label for each group. The responses were compiled, and the researchers identified the most common suggestions for each group:
1. Delete, Forward, Reply, Reply to All: "Action" (volunteered 22 times), "Respond" (volunteered 15 times)
2. Repeat, Next, Previous: "Navigate" (volunteered 22 times), "Listen to Messages" (volunteered 15 times)
3. Mark Unread, Time and Date: "Message Details" (volunteered 10 times), "Status" (volunteered 10 times), "Miscellaneous" (volunteered 5 times), "Options" (volunteered 5 times)
4. List Recipients, Add Sender: "Address Book" (volunteered 9 times), "Distribution" (volunteered 9 times)
Here we start to see the beginnings of trouble. First, while the paper's authors don't disclose the exact wording of the instructions to the survey participants, it appears that they asked participants to "label" the group; in other words, offer a short description. However, that's the opposite of what a user of the application needs to do: a user needs to take a set of labels and guess which label contains a given function, rather than take a group of functions and describe them with a label.
As descriptive terms for the groups, the labels are fine. As guideposts to the functions contained in each group, many of the labels fall short.
The other problem is that none of the labels were volunteered by more than one in four participants. This should have been a red flag that the labeling isn't obvious, has no user consensus, and needs to be treated with some care. There's no evidence that the paper's authors did anything more than accept the survey results at face value.
Step 3: 155 users (not the same as in Step 1 or Step 2) were given the most common labels for each group, and asked to choose the most appropriate label for each group. The most popular label was used in the application:
1. Delete, Forward, Reply, Reply to All: "Respond" (66%) beat "Action" (34%)
2. Repeat, Next, Previous: "Listen to Messages" (83%) beat "Navigate" (17%)
3. Mark Unread, Time and Date: "Message Details" (40%) beat "Status" (25%), "Options" (26%) and "Miscellaneous" (10%)
4. List Recipients, Add Sender: "Distribution" (64%) beat "Address Book" (36%)
This is where "Delete" managed to get in a menu called "Respond:" because it was grouped with two other functions which are variations on "Reply," most survey respondents thought "Respond" was a more "appropriate" label for the group than "Action."
Just as in Step 2, part of the problem is that the survey participants were asked to do the wrong thing: they were asked to choose the "most appropriate" label, not which label they thought a given function belonged under. The end result is perverse, even though none of the steps sound (on the surface) unreasonable.
User Centric Design
The lesson we take from this is that while user input and surveys are critical input to the design process, they cannot replace the design process. Surveys are valuable tools, but they are not perfect, and require some intelligence and interpretation. Most importantly, they require a healthy skepticism.
In this situation, while I applaud the effort to gather lots of data about user preferences, nobody ever stopped and asked questions like "Does this result make sense?" "Did we apply the survey properly?" and "Did we ask the right questions?"
There were also some artificial constraints applied to the design, such as requiring every function to belong to exactly one submenu. A human designer would probably have elevated the most common functions to top-level status and push others into logical subgroups, maybe something like this:
Delete
Next
Repeat
Previous
Respond: Reply, Reply to All, Forward
Info: Add Sender, List Recipients, Mark Unread, Time and Date
(this particular grouping mildly violates the "no more than five options" limit at the top menu, but only mildly since "Next" and "Previous" aren't offered when listening to the last or first message. A professional could probably dream up something equally functional which doesn't violate the "five options" rule)
It would be interesting to know how the broad vs. deep applications would have compared with a more reasonable "deep" design. Unfortunately, we may never know the answer, since that will require re-doing much of the work behind this paper.
Posted by Peter Leppik
The menu structure they came up with for the "deep" design was:
Listen to Messages: Next, Previous, Repeat
Respond: Reply, Reply to All, Forward, Delete
Distribution: List Recipients, Add Sender
Message Details: Mark Unread, Time and Date
So if you wanted to go to the next message, you would first have to say, "Listen to Messages" and then choose the "Next" option from that submenu.
My immediate reaction to this design was along the lines of, "WTF? Why is 'Delete' in a submenu called 'Respond'? I would never look for the 'Delete' option under the 'Respond' menu, and that's probably one of the most used functions. What sane VUI designer would build this menu tree?"
It turns out that this menu structure wasn't built by any VUI design (sane or otherwise), but rather through a uniquely user-centric process which illustrates a hazard of blindly applying user data to the design process. Here's what they did:
Step 1: 26 users were asked to organize the eleven functions into logical groupings of five or fewer functions. Each user's groupings were analyzed, and an aggregated grouping was generated with the following groups:
1. Delete, Forward, Reply, Reply to All
2. Repeat, Next, Previous
3. Mark Unread, Time and Date
4. List Recipients, Add Sender
So far so good, though there's no reason why some functions (especially heavily-used ones like "delete") can't be included in multiple groups.
Step 2: 101 users (not the same ones as in Step 1) were given the four groups from Step 1 and asked to suggest a label for each group. The responses were compiled, and the researchers identified the most common suggestions for each group:
1. Delete, Forward, Reply, Reply to All: "Action" (volunteered 22 times), "Respond" (volunteered 15 times)
2. Repeat, Next, Previous: "Navigate" (volunteered 22 times), "Listen to Messages" (volunteered 15 times)
3. Mark Unread, Time and Date: "Message Details" (volunteered 10 times), "Status" (volunteered 10 times), "Miscellaneous" (volunteered 5 times), "Options" (volunteered 5 times)
4. List Recipients, Add Sender: "Address Book" (volunteered 9 times), "Distribution" (volunteered 9 times)
Here we start to see the beginnings of trouble. First, while the paper's authors don't disclose the exact wording of the instructions to the survey participants, it appears that they asked participants to "label" the group; in other words, offer a short description. However, that's the opposite of what a user of the application needs to do: a user needs to take a set of labels and guess which label contains a given function, rather than take a group of functions and describe them with a label.
As descriptive terms for the groups, the labels are fine. As guideposts to the functions contained in each group, many of the labels fall short.
The other problem is that none of the labels were volunteered by more than one in four participants. This should have been a red flag that the labeling isn't obvious, has no user consensus, and needs to be treated with some care. There's no evidence that the paper's authors did anything more than accept the survey results at face value.
Step 3: 155 users (not the same as in Step 1 or Step 2) were given the most common labels for each group, and asked to choose the most appropriate label for each group. The most popular label was used in the application:
1. Delete, Forward, Reply, Reply to All: "Respond" (66%) beat "Action" (34%)
2. Repeat, Next, Previous: "Listen to Messages" (83%) beat "Navigate" (17%)
3. Mark Unread, Time and Date: "Message Details" (40%) beat "Status" (25%), "Options" (26%) and "Miscellaneous" (10%)
4. List Recipients, Add Sender: "Distribution" (64%) beat "Address Book" (36%)
This is where "Delete" managed to get in a menu called "Respond:" because it was grouped with two other functions which are variations on "Reply," most survey respondents thought "Respond" was a more "appropriate" label for the group than "Action."
Just as in Step 2, part of the problem is that the survey participants were asked to do the wrong thing: they were asked to choose the "most appropriate" label, not which label they thought a given function belonged under. The end result is perverse, even though none of the steps sound (on the surface) unreasonable.
User Centric Design
The lesson we take from this is that while user input and surveys are critical input to the design process, they cannot replace the design process. Surveys are valuable tools, but they are not perfect, and require some intelligence and interpretation. Most importantly, they require a healthy skepticism.
In this situation, while I applaud the effort to gather lots of data about user preferences, nobody ever stopped and asked questions like "Does this result make sense?" "Did we apply the survey properly?" and "Did we ask the right questions?"
There were also some artificial constraints applied to the design, such as requiring every function to belong to exactly one submenu. A human designer would probably have elevated the most common functions to top-level status and push others into logical subgroups, maybe something like this:
Delete
Next
Repeat
Previous
Respond: Reply, Reply to All, Forward
Info: Add Sender, List Recipients, Mark Unread, Time and Date
(this particular grouping mildly violates the "no more than five options" limit at the top menu, but only mildly since "Next" and "Previous" aren't offered when listening to the last or first message. A professional could probably dream up something equally functional which doesn't violate the "five options" rule)
It would be interesting to know how the broad vs. deep applications would have compared with a more reasonable "deep" design. Unfortunately, we may never know the answer, since that will require re-doing much of the work behind this paper.
Posted by Peter Leppik
How to Design a Phone Menu
Wed - March 26, 2008 03:55 PM in
The conventional wisdom for decades has been that the ideal phone menu (aka IVR tree, aka callflow) should have no more than five options at each level. Any more than that, it was thought, would overtax the short term memory of the caller and lead to confusion and poor experiences.
Never mind the clearly observable fact that menus designed according to this rule of thumb still led to confusion and poor experiences.
Broad vs. Deep
As far as I know, however, nobody had ever experimentally tested the question of how broad (=lots of choices at each level) or deep (=multiple levels and submenus) a design should be for a given application. Until now.
A recent article in the journal Human Factors, A Comparison of Broad Verses Deep Auditory Menu Structure, by Patrick Commarford and James Lewis (both of IBM) et al. tackles the question of broad vs. deep menus head-on and comes up with the surprising result that (at least for the application they tested) it works better to give callers a single menu with lots of options than to give callers fewer options in each menu but have submenus.
In other words, the "no more than five options" rule of thumb is wrong.
The particular application Commarford and Lewis used for this experiment is navigating e-mail through a speech application. They identified 11 different functions (Next, Previous, Delete, Reply, etc.), and built a "Broad" and a "Deep" version of the e-mail system. The "Broad" menu provided all options in a single menu, and the "Deep" version offered four top-level choices (Listen, Respond, Distribution, and Details) with the 11 options contained in the four top-level choices.
There were some design oddities about the "Deep" menus: for example, the "Delete" function was under the "Respond" menu (I can easily write a whole blog entry about the uniquely user-centric design process which led to this, ahem, unique design decision). In the final analysis, though, I think it's fair and reasonable to conclude that for this application a broad design is probably superior to a deep design, even though that leads to more than five choices in a single menu.
What Does It Mean
Some people may read this result to mean "make the menu as flat as possible," presumably by putting every option into one single uber-menu, but I think what it really means is "there's nothing magical about five menu options." There will always be a design decision about how broad or deep to make a tree, unless the system has few enough choices to put in a single menu. If the IVR needs to route the call among hundreds of different destinations or provide several dozen functions, a single menu will simply be unwieldy (of course, one thing to look at carefully in those situations is whether the total number of functions or destinations can be consolidated).
The key to usability is making sure that the menu choices are obvious (which may require including some choices in more than one menu, if they don't fit cleanly in one place), and that the caller's time and mental effort is respected. That means striking a balance between broad and deep, and the right balance will be different in every situation.
The real lesson of this research is that real-world usability testing is likely to show that general rules of thumb break down when applied to a specific design. At the end of the day, usability testing is critical to making sure that the design is right.
Posted by Peter Leppik
Broad vs. Deep
As far as I know, however, nobody had ever experimentally tested the question of how broad (=lots of choices at each level) or deep (=multiple levels and submenus) a design should be for a given application. Until now.
A recent article in the journal Human Factors, A Comparison of Broad Verses Deep Auditory Menu Structure, by Patrick Commarford and James Lewis (both of IBM) et al. tackles the question of broad vs. deep menus head-on and comes up with the surprising result that (at least for the application they tested) it works better to give callers a single menu with lots of options than to give callers fewer options in each menu but have submenus.
In other words, the "no more than five options" rule of thumb is wrong.
The particular application Commarford and Lewis used for this experiment is navigating e-mail through a speech application. They identified 11 different functions (Next, Previous, Delete, Reply, etc.), and built a "Broad" and a "Deep" version of the e-mail system. The "Broad" menu provided all options in a single menu, and the "Deep" version offered four top-level choices (Listen, Respond, Distribution, and Details) with the 11 options contained in the four top-level choices.
There were some design oddities about the "Deep" menus: for example, the "Delete" function was under the "Respond" menu (I can easily write a whole blog entry about the uniquely user-centric design process which led to this, ahem, unique design decision). In the final analysis, though, I think it's fair and reasonable to conclude that for this application a broad design is probably superior to a deep design, even though that leads to more than five choices in a single menu.
What Does It Mean
Some people may read this result to mean "make the menu as flat as possible," presumably by putting every option into one single uber-menu, but I think what it really means is "there's nothing magical about five menu options." There will always be a design decision about how broad or deep to make a tree, unless the system has few enough choices to put in a single menu. If the IVR needs to route the call among hundreds of different destinations or provide several dozen functions, a single menu will simply be unwieldy (of course, one thing to look at carefully in those situations is whether the total number of functions or destinations can be consolidated).
The key to usability is making sure that the menu choices are obvious (which may require including some choices in more than one menu, if they don't fit cleanly in one place), and that the caller's time and mental effort is respected. That means striking a balance between broad and deep, and the right balance will be different in every situation.
The real lesson of this research is that real-world usability testing is likely to show that general rules of thumb break down when applied to a specific design. At the end of the day, usability testing is critical to making sure that the design is right.
Posted by Peter Leppik
Updating Service Quality Tracker
Tue - March 25, 2008 01:43 PM in
We've updated our Service Quality Tracker survey for tech support at different PC vendors, and we're starting to gradually roll out the new survey to participating consumers.
This is a major overhaul: we've expanded the survey from about a half-dozen questions to over 20, and we've added a long list of new metrics. Some of the new metrics are based on input from our clients, and we're also developing a consumer-focused metric based on the factors which consumers say are important to providing good service.
Some of the new things we're tracking include hold time, IVR problems, preferred communication channel, type of call, the type of resolution (if any) to the customer's problem, and whether the company attempted to upsell the customer.
This is in addition to the things we've long been tracking with our old Service Quality Tracker and the discontinued SectorPulse survey: satisfaction, loyalty, automation rate, completion rate, consumer frustration, and call duration.
Watch this space for more updates as things keep changing.
Posted by Peter Leppik
Some of the new things we're tracking include hold time, IVR problems, preferred communication channel, type of call, the type of resolution (if any) to the customer's problem, and whether the company attempted to upsell the customer.
This is in addition to the things we've long been tracking with our old Service Quality Tracker and the discontinued SectorPulse survey: satisfaction, loyalty, automation rate, completion rate, consumer frustration, and call duration.
Watch this space for more updates as things keep changing.
Posted by Peter Leppik
Simple Solutions
Mon - March 24, 2008 02:52 PM in
There are some lessons which I need to learn over and over and over. The old engineering advice to KISS your problems ("Keep It Simple, Stupid") is one of them.
As a nerd and a geek, it's always tempting to reach for the technological fix for any problem. It always seems simplest to just propose something like, "You can just encapsulate the data in your XML file, and we'll process it and include it in our report. Of course, your telecom system will have to populate the field from an internal org chart....."
The problem, in this case, was how to include the name of an agent's supervisor in our database of scores from a Quality Assurance review. We already receive the call recording and the name of the agent from the client's telecom system, but the name of the supervisor isn't available in any system currently interfaced to the telecom system.
We started to research options, such as building a client-specific web interface for maintaining an organizational chart in our database, or auto-populating from a periodic batch dump, either of which would require several days (at a minimum) to develop and test, and some unknown amount of ongoing maintenance.
Then, "Hey, why not add one question to the QA review for the reviewer to enter the agent's supervisor? It's a small call center, they've only got a handful of supervisors, and the reviewer sits on the same floor as the agents--they've got better information about the org chart than anyone." A five minute change which, while not fully automated, will give exactly the results the client wants with almost zero headache.
KISS, indeed.
Posted by Peter Leppik
The problem, in this case, was how to include the name of an agent's supervisor in our database of scores from a Quality Assurance review. We already receive the call recording and the name of the agent from the client's telecom system, but the name of the supervisor isn't available in any system currently interfaced to the telecom system.
We started to research options, such as building a client-specific web interface for maintaining an organizational chart in our database, or auto-populating from a periodic batch dump, either of which would require several days (at a minimum) to develop and test, and some unknown amount of ongoing maintenance.
Then, "Hey, why not add one question to the QA review for the reviewer to enter the agent's supervisor? It's a small call center, they've only got a handful of supervisors, and the reviewer sits on the same floor as the agents--they've got better information about the org chart than anyone." A five minute change which, while not fully automated, will give exactly the results the client wants with almost zero headache.
KISS, indeed.
Posted by Peter Leppik
Judge Rules Management Training Materials "Aggressively Vapid"
Fri - March 21, 2008 01:11 PM in
Anyone who has spent any time working in a large company has probably been through some sort of training program. Sometimes it's useful, more often it's an excuse to get paid for eating bagels and drinking coffee for a day.
One such program became the subject of a copyright dispute when a group of training consultants quit their jobs to form their own company and develop their own training materials. The original employer sued, alleging that the breakaway employees violated the copyright on the materials.
The judge disagreed, apparently ruling that copyright law applies to "original creative works," and that the management training materials were neither original nor creative:
"These works exemplify the sorts of training programs that serve as fodder for sardonic workplace humor that has given rise to the popular television show The Office and the movie Office Space. They are aggressively vapid-hundreds of pages filled with generalizations, platitudes, and observations of the obvious."
All across the country, office workers applaud Judge Young for recognizing the, well, obvious.
Posted by Peter Leppik
The judge disagreed, apparently ruling that copyright law applies to "original creative works," and that the management training materials were neither original nor creative:
"These works exemplify the sorts of training programs that serve as fodder for sardonic workplace humor that has given rise to the popular television show The Office and the movie Office Space. They are aggressively vapid-hundreds of pages filled with generalizations, platitudes, and observations of the obvious."
All across the country, office workers applaud Judge Young for recognizing the, well, obvious.
Posted by Peter Leppik
Extra! Extra! Customers Still Fed Up
Thu - March 20, 2008 11:01 AM in
"Customers are angry about customer service" is a staple of consumer reporting. If I passed along every one of those articles, this would be the "Customers Angry About Service" blog.
Still, it's been a while since I've linked to one of those articles, so it's about time for another. Today's choice comes from the Christian Science Monitor, and features consumers complaining about clueless customer service reps, useless automation, and phone reps based overseas. There's also the hopeful conclusion about how more companies are starting to get the message and focus on service.
Enjoy!
Posted by Peter Leppik
Enjoy!
Posted by Peter Leppik
Consumerist on Good and Bad Service Factors
Tue - March 18, 2008 01:00 PM in
A few weeks ago, I wrote about our survey of good and bad customer service factors, the things which customers feel are important in the quality of their customer service experience.
Ben Popken, the excellent editor of The Consumerist, thought it would be interesting to offer the same survey to his readers. So he did (with my blessing).
After running for several days, he got around 5,000 responses from Consumerist readers (as compared to the 400 we collected from VocaLabs' consumer panelists). The average participant on the Consumerist survey chose only one option on each survey question, in sharp contrast to the VocaLabs panel where the average participant selected about half the options on each question. This suggests that the VocaLabs panelists took more time to answer the question carefully, whereas the Consumerist readers mostly just chose their top answer and stopped there.
But after normalizing the results to adjust for the different number of responses per participant, I found that the survey results were quite similar. On the list of "Good" customer service factors, the differences between the VocaLabs survey results and the Consumerist poll were generally within the margin of error, except that many fewer Consumerist readers chose the "Keep the total call as short as possible" option. My best explanation is that the "short call" factor was rarely people's most important consideration, but often a second- or third-choice option.
There were more differences in the list of "Bad" customer service factors, though most of the differences between our survey and Consumerist readers were still within the margin of error (again, after normalizing the Consumerist results to adjust for the different number of responses per participant). Consumerist readers chose the "The company doesn't do things it promised to do" option relatively more often than VocaLabs panelists, and were less likely to choose long hold times, rude agents, or hard-to-use IVR as important considerations.
Given the differences between the way these two surveys were run, and the fact that the Consumerist ran a wide-open online poll, it's perhaps surprising that the results are as similar as they are. That suggests that consumers are fairly consistent about which customer service issues they care about today.
Posted by Peter Leppik