Variations, Abstractions and Why Natural Language Understanding is So Much More than Keyword Spotting

avatar

There are some who believe that writing an ITR application with keyword-spotting is much easier than writing one that leverages Natural Language Understanding (NLU). But even if your objective is just to pull out specific words and phrases from the user input, a keyword-spotting approach may actually make your job harder. Suppose, for example, you want to react to questions from customers about ordering products. How many forms of “order” are there to recognize?

  • I would like to order an Agent Carter t-shirt…
  • I ordered a model lightstaber on Thursday…
  • I am ordering a TARDIS keychain, and I wanted to know…
  • If someone orders the Enterprise kit today, will…

This gets worse when dealing with irregular verbs whose inflected forms don’t all share the same stem (break, breaks, broke, broken) and in other, more morphologically-rich languages than English, where each conjugation of the verb might have a different spelling.SodaDrinking Fountain

Inflections, of course, are far from the only variations one has to accommodate. My friends and I (being linguistic nerds) never tire of discussing the differences in our regional dialects. Even non-nerds recognize that when you speak the same language as someone else, this does not necessarily mean you use the same words for the same concepts. The perennial favorite for comparison is the term for a carbonated non-alcoholic drink. “Pop,” say my friends from the Midwest. “Soda,” retort my East Coast friends. “Coke,” say the folks who grew up down South. I smile at all of them, having grown up close enough to Boston to have heard the uniquely Bostonian term that no one else uses: “Tonic!” Everyone looks at me like I’m crazy, but it’s as classic as “packy” (liquor store) and “bubbler” (water fountain), and some of us don’t realize until the first odd look that we’re using a term that is not universally recognized.

Why is this relevant in a self-service ITR application? If you want to give your conversational system the ability to avoid the chatbot equivalent of the odd, confused look, then you either need to write your keyword-spotting to cover all of the regional variations of a term or phrase on top of any inflections that might change the way it appears – or you need to step away from specifying the item of interest as a sequence of characters, and instead think of it as an abstract semantic concept. Coke, pop, soda, and tonic all semantically represent the same concept, like British crisps and American chips. If you can specify the concept as an abstraction, by which you mean all expressions that represent this abstraction, then you can specify a single cue for action, and it is the job of the language model inside the Aspect NLU Engine to handle the details of what that cue may actually look like.

An added benefit is multi-linguality. The abstractions in Aspect NLU’s language model are not restricted to English; one identifier represents the same semantic concept in every language in the model. To adapt your ITR to work in another language, all that may be required is for someone to write out the prompts or responses in that other language, because the target concepts are specified in an interlingual form.

All this does not touch on unlocking the deeper capabilities of NLU, which has the possibility of going beyond triggers based on specified concepts to extracting entities and their roles from a deeper analysis of the sentence. (What was ordered? When was the order placed? By what method was this ordered?) We are also working toward a built-in spelling normalization that will handle non-standard spellings within the NLU so that the ITR script designer does not need to try and anticipate typos. So, is it easier to respond to specific concepts using NLU versus keyword-spotting? It depends on what you consider “easy.” It might depend on where you’re from; maybe that word does not mean the same to you as it does to me.

avatar

Lisa Michaud

Lisa Michaud is the Director of Natural Language Processing (NLP) at Aspect. She has been centrally involved in the integration of NLP components into Aspect’s product suite for customer engagement and the architecting of our Interactive Text Response (chatbot) technology. She has 20 years of research experience in the field of Natural Language Processing / Computational Linguistics and pursues diverse interests in user modeling, dialogue, parsing, generation, and the analysis of non-grammatical text.She holds a PhD in Computer Science and has been published in multiple international journals, workshops, and conferences in the fields of user-adaptive interaction and Computational Linguistics.
avatar