1. Description
This article presents an example of how to use spaCy, the leading open source Python library for NLP, to extract meaning from user input.
2. How to extract meaning from user input
Extracting meaning from user input programmatically can be very challenging, but not impossible. Obviously, you can't rely on the meaning of individual words in a sentence - the same word can mean different things depending on its syntactic function in a particular sentence. This can best be understood by example. Please see the following two sentences:
I’d like to order a cake.
I want to cancel my order. In both utterances, you can see the word "order". In each case, however, it has a different syntactic function and has a different meaning. In the first case, "order" is an action (transitive) verb acting on the noun "cake" - the direct object of the sentence. In contrast, "order" in the second utterance is the noun that receives the action of the sentence—that is, it acts as the direct object of the sentence, of which "cancel" is the transmitting verb.
Linguistic features of words in a sentence (such as transitive verbs or direct objects in the example above) are also called linguistic features. spaCy automatically assigns linguistic features to each token in a sentence to which the spaCy text processing pipeline is applied. Then, analyzing linguistic features can help identify the meaning of the words in this particular sentence. We will discuss how to use linguistic features for meaning extraction tasks later in this article in the Using Linguistic Features in NLP section.
3. Prepare your work environment
To follow the code presented in this article, you need the following software components installed on your computer:
Pretrained spaCy English model
You can find installation instructions on the respective sites. The quickest way to make sure your environment is ready is to enter the following line of code in your Python session:
import spacy
nlp = spacy.load('en') If everything worked, you should have no error messages.
4. Using Linguistic Features in NLP
Features such as part-of-speech tagging and syntactic dependency tags are specifically designed to support the development of applications that can intelligently process raw text. The following script illustrates how to use spaCy to extract linguistic features for each word in a sentence:
import spacy
nlp = spacy.load('en')
doc = nlp(u'I have to send them a notification.')
for token in doc:
print(token.text, token.pos_, token.tag_, token.dep_)In the script above, you extract and output coarse-grained part-of-speech tags (pos_), fine-grained part-of-speech tags (tag_), and grammatical dependency tags (dep_) for each token in the submitted sentence. Therefore, the script should provide the following output (listed for readability):
I PRON PRP nsubj
have VERB VBP ROOT
to PART TO aux
send VERB VB xcomp
them PRON PRP dative
a DET DT det
notification NOUN NN dobj
. PUNCT . Punct If you're not familiar with spaCy, the above output of fine-grained part-of-speech tags and grammatical dependency tags in the third and fourth columns respectively may seem a bit confusing. To understand the meaning of the values in these columns, you can view spacy's documentation at Data formats spaCy API Documentation or use the spacy.explain() function, which returns a description of a given language feature. In the loop below, you output a description of the fine-grained part-of-speech tag for each token in the example sentence:
for token in doc:
print(token.text, spacy.explain(token.tag_)) This should give you the following output:
I pronoun, personal
have verb, non-3rd person singular present
to infinitival to
send verb, base form
them pronoun, personal
a determiner
notification noun, singular or mass
. punctuation mark, sentence closer Likewise, you can use the spacy.explain() function to get an explanation of coarse-grained part-of-speech tags and grammatical dependency tags.
5. Extracting intentions from utterances
Let's now look at an example of how language features can be used to extract meaning from user input. Suppose the intent needs to be extracted from the submitted statement. For example, a user of a food ordering chatbot submits the following statement:
I want to order a photo cake. Obviously, the words "order" and "cake" best describe the intent expressed in this sentence. In this particular case, these words denote transitive verbs and direct objects, respectively. In fact, passing predicate/direct object pairs is most descriptive in determining the intent expressed in a request utterance in most cases. From a diagram, this might look like this:
In many request utterances, the transitive predicate and its direct object best describe the intent of the phrase.
The operations described in the diagram above can be easily performed in a Python script using spaCy as follows:
import spacy
nlp = spacy.load('en')
doc = nlp(u'I want to order a photo cake.')
for token in doc:
if token.dep_ == 'dobj':
print(token.head.text + token.text.capitalize()) In this script, a text processing pipeline is applied to an example sentence, and then iterates through the tokens, looking for tokens whose dependency tag is dobj . Once you've found it, you can determine the corresponding transitive verb by getting the syntactic header of the direct object. Finally, connect the transitive verb and its direct object, expressing the intent as a single word (this is often a requirement for processing scripts).
Therefore, the script should generate:
orderCake In a real application, users might use a broad set of phrases for each intent. This means that real applications must recognize synonymous phrases in user input. For details on these, you can check out my new book , Natural Language Processing with Python, which includes many examples on using spaCy to perform different NLP tasks.
Also, a real-world example of what intent extraction techniques might be used in practice can be found in Generating Intents and Entities for Oracle Digital Assistant Skills article I recently wrote for Oracle Magazine .