AAAI 2018 analysis

word embedding

Learning Sentiment-Specific Word Embedding via Global Sentiment Representation

Context-based word embedding learning approaches can model rich semantic and syntactic information.

However, it is problematic for sentiment analysis because the words with similar contexts but opposite sentiment polarities, such as good and bad, are mapped into close word vectors in the embedding space.

Recently, some sentiment embedding learning methods have been proposed, but most of them are designed to work well on sentence-level texts.

Directly applying those models to document-level texts often leads to unsatisfied results.

To address this issue, we present a sentiment-specific word embedding learning architecture that utilizes local context informationas well as global sentiment representation.

The architecture is applicable for both sentence-level and document-level texts.

We take global sentiment representation as a simple average of word embeddings in the text, and use a corruption strategy as a sentiment-dependent regularization.

Extensive experiments conducted on several benchmark datasets demonstrate that the proposed architecture outperforms the state-of-the-art methods for sentiment classification.

"I represent the embedded learn specific emotional words by global sentiment."

Embedded learning methods can model a wealth of semantic and syntactic information based on the context of the word.

However, for analyzing emotional be problematic, because the insertion space having a similar but opposite sentiment polarity context words (e.g., good and wrong words) is mapped into a closed word vector.

In recent years, it made a number of emotions embedded learning, but most of the methods is to play a role in the sentence-level text.

These models are applied directly to the document-level text often leads to unsatisfactory results.

To solve this problem, we propose a specific emotion words embedded learning architecture that takes advantage of local background information as well as global sentiment expressed.

This architecture is suitable for sentence-level and document-level text.

We will characterize the global sentiment as the simple average of the embedded text and corruption strategy as a mood-dependent normalization.

A large number of experiments carried out on a plurality of reference data sets show that the most advanced sentiment classification method proposed architecture is superior.

Using k-Way Co-Occurrences for Learning Word Embeddings

Co-occurrences between two words provide useful insights into the semantics of those words. Consequently, numerous prior work on word embedding learning has used co-occurrences between two words as the training signal for learning word embeddings. However, in natural language texts it is common for multiple words to be related and co-occurring in the same context. We extend the notion of co-occurrences to cover k(≥2)-way co-occurrences among a set of k-words. Specifically, we prove a theoretical relationship between the joint probability of k(≥2) words, and the sum of l_2 norms of their embeddings. Next, we propose a learning objective motivated by our theoretical result that utilize k-way co-occurrences for learning word embeddings. Our experimental results show that the derived theoretical relationship does indeed hold empirically, and despite data sparsity, for some smaller k(≤5) values, k-way embeddings perform comparably or better than 2-way embeddings in a range of tasks.

Semantic Structure-Based Word Embedding by Incorporating Concept Convergence and Word Divergence

Representing the semantics of words is a fundamental task in text processing.

Several research studies have shown that text and knowledge bases (KBs) are complementary sources for word embedding learning.

Most existing methods only consider relationships within word-pairs in the usage of KBs.

We argue that the structural information of well-organized words within the KBs is able to convey more effective and stable knowledge in capturing semantics of words.

In this paper, we propose a semantic structure-based word embedding method, and introduce concept convergence and word divergence to reveal semantic structures in the word embedding learning process.

To assess the effectiveness of our method, we use WordNet for training and conduct extensive experiments on word similarity, word analogy, text classification and query expansion.

The experimental results show that our method outperforms state-of-the-art methods, including the methods trained solely on the corpus, and others trained on the corpus and the KBs.

"Based on the concept of convergence and divergence of the word semantic structure embedding"

The semantics of the word represent a fundamental role in text processing.

Some studies suggest text and knowledge base (KBS) is the word of embedding supplemental source of learning.

When using KBS, most existing methods only consider the word of the internal relations of children.

We believe that well-organized knowledge base word of structural information can be more effective, more stable transfer capture the semantics of the word of knowledge.

This paper presents a method of embedding based semantic structure, and introduces the concept of convergence and divergence to reveal the word semantic structures embedded in the learning process.

To assess the effectiveness of this method, we use WordNet been trained, and the similarity of the word, the word analogy, text classification and query expansion and other aspects of a wide range of experiments.

The experimental results show that our method is superior to the most advanced methods, including training only in the corpus, and a method in the training corpus and KBS.

Spectral Word Embedding with Negative Sampling

In this work, we investigate word embedding algorithms in the context of natural language processing. In particular, we examine the notion of ``negative examples'', the unobserved or insignificant word-context co-occurrences, in spectral methods. we provide a new formulation for the word embedding problem by proposing a new intuitive objective function that perfectly justifies the use of negative examples. In fact, our algorithm not only learns from the important word-context co-occurrences, but also it learns from the abundance of unobserved or insignificant co-occurrences to improve the distribution of words in the latent embedded space. We analyze the algorithm theoretically and provide an optimal solution for the problem using spectral analysis. We have trained various word embedding algorithms on articles of Wikipedia with 2.1 billion tokens and show that negative sampling can boost the quality of spectral methods. Our algorithm provides results as good as the state-of-the-art but in a much faster and efficient way.

Chinese LIWC Lexicon Expansion via Hierarchical Classification of Word Embeddings with Sememe Attention

Linguistic Inquiry and Word Count (LIWC) is a word counting software tool which has been used for quantitative text analysis in many fields.

Due to its success and popularity, the core lexicon has been translated into Chinese and many other languages.

However, the lexicon only contains several thousand of words, which is deficient compared with the number of common words in Chinese.

Current approaches often require manually expanding the lexicon, but it often takes too much time and requires linguistic experts to extend the lexicon.

To address this issue, we propose to expand the LIWC lexicon automatically.

Specifically, we consider it as a hierarchical classification problem and utilize the Sequence-to-Sequence model to classify words in the lexicon.

Moreover, we use the sememe information with the attention mechanism to capture the exact meanings of a word, so that we can expand a more precise and comprehensive lexicon.

The experimental results show that our model has a better understanding of word meanings with the help of sememes and achieves significant and consistent improvements compared with the state-of-the-art methods.

The source code of this paper can be obtained from https://github.com/thunlp/Auto_CLIWC.

"Chinese vocabulary development based on phrases embedded hierarchical classification of"

Query language and word count (LIWC) is the word count tool in many areas of software used for quantitative analysis of the text.

Due to its success and popularity of core vocabulary it has been translated into Chinese and many other languages.

However, only a few thousand vocabulary words, compared with the Chinese words, which is a deficiency.

Current methods typically require manually extend the vocabulary, but it usually takes a lot of time and requires language experts to expand vocabulary.

To solve this problem, we propose an automatic extension LIWC dictionary.

Specifically, we will issue it as a hierarchical classification, using sequence - a sequence of words classification model.

In addition, we also use bits of information have righteousness attention mechanism to capture the exact meaning of a word to extend a more accurate and comprehensive vocabulary.

Experimental results show that our model in the original sense of help, have a better understanding of the meaning of a word, compared with the most advanced method to obtain significant and consistent improvement.

From the source code herein may https://github.com/thunlp/auto_cliwc obtained.

Training and Evaluating Improved Dependency-Based Word Embeddings

Word embedding has been widely used in many natural language processing tasks. In this paper, we focus on learning word embeddings through selective higher-order relationships in sentences to improve the embeddings to be less sensitive to local context and more accurate in capturing semantic compositionality. We present a novel multi-order dependency-based strategy to composite and represent the context under several essential constraints. In order to realize selective learning from the word contexts, we automatically assign the strengths of different dependencies between co-occurred words in the stochastic gradient descent process. We evaluate and analyze our proposed approach using several direct and indirect tasks for word embeddings. Experimental results demonstrate that our embeddings are competitive to or better than state-of-the-art methods and significantly outperform other methods in terms of context stability. The output weights and representations of dependencies obtained in our embedding model conform to most of the linguistic characteristics and are valuable for many downstream tasks.

word representation

Learning Multimodal Word Representation via Dynamic Fusion Methods

Multimodal models have been proven to outperform text-based models on learning semantic word representations. Almost all previous multimodal models typically treat the representations from different modalities equally. However, it is obvious that information from different modalities contributes differently to the meaning of words. This motivates us to build a multimodal model that can dynamically fuse the semantic representations from different modalities according to different types of words. To that end, we propose three novel dynamic fusion methods to assign importance weights to each modality, in which weights are learned under the weak supervision of word association pairs. The extensive experiments have demonstrated that the proposed methods outperform strong unimodal baselines and state-of-the-art multimodal models.

Representing the semantics of words is a long-standing problem for the natural language processing community.

Most methods compute word semantics given their textual context in large corpora.

More recently, researchers attempted to integrate perceptual and visual features.

Most of these works consider the visual appearance of objects to enhance word representations but they ignore the visual environment and context in which objects appear.

We propose to unify text-based techniques with vision-based techniques by simultaneously leveraging textual and visual context to learn multimodal word embeddings.

We explore various choices for what can serve as a visual context and present an end-to-end method to integrate visual context elements in a multimodal skip-gram model.

We provide experiments and extensive analysis of the obtained results.

"Multimodal word-based visual representation of the context of learning."

Semantic representation of natural language word processing industry long-standing problem.

Most of the word semantic context calculation method according to a large corpus of text.

Recently, researchers have tried to integrate perception and visual features.

Most of these works consider the visual appearance of an object in order to enhance the representation of words, but they ignore the visual environment and context object appears.

We recommend that based on technology and unified vision-based technology, while taking advantage of text and visual context to learn multimodal word embedded in the text.

We explored various options as a visual context, and presents a visual context elements will be integrated into the end-to-multi-hop mode diagram model.

The results we have conducted experiments and extensive analysis.