Observations of Edward in the Wild, Part II


In my previous blog post, I discussed some of the conclusions we have been able to draw from two months of recorded user interactions with Edward, a concierge and service chatbot deployed at the Radisson Blu Edwardian hotel chain in London.  This post is a continuation of those observations.

A Significant Portion of Interaction Can be Handled with Self-Service

While analyzing the data from Edward, we coded each request with a request type.  We observed from this effort that 75% of the sentences texted to Edward could be deflected from the front desk staff. Most of these (59% of the total) were divided between pleasantries that Edward could respond to automatically, general FAQs that could evoke a canned response, and personalized account questions that required a dip into CRM data associated with that phone number but still could be fully automated.  An additional 16% were service requests that could be forwarded to appropriate staff members such as Housekeeping or Maintenance without involving the front desk staff.  This is a strong argument in favor of the success of the Edward deployment; it freed up those behind the check-in desk to handle the individuals who elected to call down or to bring their business directly to the desk in person, while providing such a satisfying user experience that some guests have left tips for Edward, and even nominating him for Employee of the Month, mistaking him for a member of the staff.

Edward chatbot for self-service

Chatbots and Humans are Partners in the Customer Experience

Edward could not do what he does without assistance, however.  As I mentioned in my last post, Aspect strongly advocates the practice of standing up a chatbot that can hand the conversation over to a human when that is called for.  No bot can answer every possible input without help; some requests are too nuanced to be easily recognized by a nonhuman intelligence, and some requests are outside of the scope of the bot’s store of answers or actions.  The ideal way to provide human backup is to forward requests to a human agent who can respond directly in the same channel, as the Edwardian hotel staff can do.  In fact, one very reasonable approach to bot-building that my colleague Tobias Goebel has written about is to create a limited-scope bot as an initial offering, with everything beyond a very small number of questions handled by live human staff responding as if they were the bot.  This “Wizard of Oz” arrangement provides an excellent opportunity to collect data on the nature of requests to expect in this new channel for your business, and then each iterative version of the bot can slowly take over more and more of the questions that fall into the self-service category, until the human staff is only handling those interactions for which they are truly required.

We do not, however, suggest a lack of transparency where a bot’s response is presented as that of a human agent.  While bots are becoming more and more competent, they will still occasionally fall short of true human competence at understanding natural language. This could adversely affect the customer experience if she believes her conversational partner is human – more so than if she knows it is a bot, of whom she has slightly lower expectations.

It’s not Just Processing a Sentence; It’s Having a Dialogue

The practice of creating conversational user interfaces (CUIs) needs to recognize that natural language understanding cannot operate as if each sentence were an isolated event.  In the Edward data, 13% of the utterances were not autonomous; they were either providing context to a request in another utterance, they were making reference to previous steps in the dialogue by use of a pronoun or another anaphora, or they were cancelling a previously made request (sending something like “never mind – I’m good”).  A true CUI needs to be able to handle each of these:

  • In the case of a cancellation, there needs to be access to context or history so that an action can be removed from a queue or undone.
  • When pronouns or other referring expressions are used, a sophisticated NLU component needs to be able to resolve the reference to the relevant entity from the previous dialogue.
  • When context is provided for a request, this context needs to be both used for any anaphora resolution (above), and stored as information to possibly pass on to a human agent as part of the same complex request.

Data Makes Us All Smarter

Looking at these data helps everyone in the bot universe: bot creators, bot buyers, and the bots themselves.  We learn from the mistakes and realize what to do better next time.  We evolve and become more competent at providing the ideal customer experience.  I look forward to what I am going to learn from the next bot.