Many people worldwide start their day with the questions: «Alexa, what is the weather?» or «Siri, where are the traffic jams in the city today?». Surprisingly, they get answers to these questions from digital assistants. How do they know the weather in which city and traffic jams on which section of the road users are interested? The answer is simple: thanks to NLU – natural language understanding. Let’s talk about what is NLU.
The definition of natural language understanding
NLU is the field of artificial intelligence (AI), which uses special software to translate natural human words (in the form of text or speech) into a format that machines can interpret. In short, natural language understanding provides communication between a person and a computer.
Such comprehension of different human languages allows the computer to interpret commands without the formalized syntax of computer languages. Thanks to NLU, a machine can answer a person in the language he understands.
Nowadays, natural language understanding allows the creation of voicebots and business chatbots that contact the community without supervision. The most successful IT-corporation globally (Google, Microsoft, Amazon) regularly implement large-scale NLU projects.
How does NLU work?
Natural language understanding processes information to comprehend its meaning using tools that transform human talk into ontology with a clear structure. It’s a data model that consists of semantics and pragmatics definitions. The fundamental principles of NLU:
- Word-Sense Disambiguation is the procedure of determining the meaning of words according to the context. During word-sense disambiguation, often use the part of speech taggers to contextualize individual words. The two most common supervised algorithms of word-sense disambiguation are support vector machines and memory-based learning.
- Named entity recognition is the procedure of identification of «named entities» (people, important locations) and «numeric entities» (numbers, currencies, and percentages). Named entity recognition selects the main concepts and references in the text, recognizes named and numeric entities, and divides them into groups: places, dates, companies, people, and others. NER task execution is provided by controlled models based on grammar rules.
To understand what is NLU, consider an example. The user searches for information about the Hawaii island trip with overnight camping on July 15. The system can break down this data into different categories: ship tickets [intent] / need: book a campsite [intent] / Hawaii [place] / July 15 [date].
Main differences between NLP and NLU
It is easier to understand what is NLU, if we compare it with NLP. NLP works with simple sentences; it can’t analyze real intent or complex sentences. Let’s look at an example:
We have a sentence: I need some cake. On the one hand, different NLP tools help process such information. On the one hand, NLU recognizes that I’m looking for something sweet or I need to take a break to replenish my energy.
Syntaxes and semantic
Natural languages processing is closely related to syntax; it focuses on the structure of languages and grammar aspects. NLP uses tokenization, lemmatization, and stemming methods to extract data from a particular text.
Natural language understanding works with the meaning of language; it does not consider word-formation or punctuation in a sentence. The primary goal of NLU is to determine and analyze the speaker’s real intentions.
Processing Data and Understanding Data
The NLP helps to organize the information base. When a person says a command, the system breaks it into small parts (tokens) and begins processing.
At the same time, NLU decrypts the meaning of the request. It uses many techniques, including sentiment analysis and sarcasm detection, to understand the sentence’s meaning. It will select the request based on his understanding of the underlying meaning.
Translation and transcreation
Translation and transcreation are entirely different concepts, NLP approach for translations, but NLU is a necessary part of transcreation.
Translation means searching for the exact analog of a word in another language, but it fails when it comes to phrases and idioms. In such a case, it’s better to use transcreation, which conveys the sentence’s meaning in the targeted language without a word-by-word translation.
Examples of natural language understanding
Specialists identify areas where NLU use in applications that cooperate with human language:
- Automatic Ticket Routing. The most common variant of using natural language understanding is customer service automation. With different text analysis techniques, the computer defines the essence of each customer support request and directs them to authorized departments. It helps workers not waste time opening each ticket and first considering priority applications.
- Machine Translation. The competent translation of written text or speech is one of the essential tasks of NLP and NLU. Such services as Google Translate and Microsoft Translator manipulate complex algorithms based on linguistic norms and AI machine training. The user can type words or upload documents of any size and receive the correct translation by preserving the context and semantic information. Google Translate has a function called Optical character recognition that allows the system gets data from images, analyze and translate it.
- Automated Reasoning. It’s the area of cognitive science that helps automatically prove theorems in mathematics or make a medical diagnosis after processing the initial data. This technology allows the system to think logically and draw conclusions by induction. A computer collects and analyzes data and then generates inferences; an experience often helps formulate a medical diagnosis.
- Question Answering. Speech recognition uses natural language understanding methods to understand questions asked in human language. Technology allows a computer or smartphone user to receive answers in his native language instead of demonstrating the list of most frequent responses. When you communicate with virtual assistants, NLU helps him comprehend the question and choose the correct answer, taking into account recognized objects and the context of past affirmations.
It’s not a complete list of areas where we can use NLU.
Future expectations
The main task of researchers for the coming years is to create a chatbot for communication with a person on equal terms. That’s why specialists develop chatbots that can process native language.
The global goal is to make an intelligent agent who can understand human speech and respond to it correctly; it will be possible only if NLU, NLG, and NLP work together. Indeed, NLP is a combination of NLU and NLG.
Despite all prospects, NLP & NLU have to overcome many difficulties in the future to teach the system not only to understand people but also to interact with them.