【NLP】【五】gensim之Word2Vec

【一】整体流程综述

gensim底层封装了Google的Word2Vec的c接口,借此实现了word2vec。使用gensim接口非常方便,整体流程如下:

1. 数据预处理(分词后的数据)

2. 数据读取

3.模型定义与训练

4.模型保存与加载

5.模型使用(相似度计算,词向量获取)

【二】gensim提供的word2vec主要功能

【三】gensim接口使用示例

1. 使用jieba进行分词。

文本数据:《人民的名义》的小说原文作为语料 

百度云盘:https://pan.baidu.com/s/1ggA4QwN

# -*- coding:utf-8 -*-
import jieba

def preprocess_in_the_name_of_people():
    with open("in_the_name_of_people.txt",mode='rb') as f:
        doc = f.read()
        doc_cut = jieba.cut(doc)
        result = ' '.join(doc_cut)
        result = result.encode('utf-8')
        with open("in_the_name_of_people_cut.txt",mode='wb') as f2:
            f2.write(result)

2. 使用原始text8.zip进行词向量训练

from gensim.models import word2vec
# 引入日志配置
import logging

logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', level=logging.INFO)

def train_text8():
    sent = word2vec.Text8Corpus(fname="text8")
    model = word2vec.Word2Vec(sentences=sent)
    model.save("text8.model")

注意。这里是解压后的文件,不是zip包

3. 使用Text8Corpus 接口加载数据

def train_in_the_name_of_people():
    sent = word2vec.Text8Corpus(fname="in_the_name_of_people_cut.txt")
    model = word2vec.Word2Vec(sentences=sent)
    model.save("in_the_name_of_people.model")

4. 使用 LineSentence 接口加载数据

def train_line_sentence():
    with open("in_the_name_of_people_cut.txt", mode='rb') as f:
        # 传递open的fd
        sent = word2vec.LineSentence(f)
        model = word2vec.Word2Vec(sentences=sent)
        model.save("line_sentnce.model")

5. 使用 PathLineSentences 接口加载数据

def train_PathLineSentences():
    # 传递目录,遍历目录下的所有文件
    sent = word2vec.PathLineSentences("in_the_name_of_people")
    model = word2vec.Word2Vec(sentences=sent)
    model.save("PathLineSentences.model")

6. 数据加载与训练分开

def train_left():
    sent = word2vec.Text8Corpus(fname="in_the_name_of_people_cut.txt")
    # 定义模型
    model = word2vec.Word2Vec()
    # 构造词典
    model.build_vocab(sentences=sent)
    # 模型训练
    model.train(sentences=sent,total_examples = model.corpus_count,epochs = model.iter)
    model.save("left.model")

7. 模型加载与使用

model = word2vec.Word2Vec.load("text8.model")
print(model.similarity("eat","food"))
print(model.similarity("cat","dog"))
print(model.similarity("man","woman"))
print(model.most_similar("man"))
print(model.wv.most_similar(positive=['woman', 'king'], negative=['man'],topn=1))

model2 = word2vec.Word2Vec.load("in_the_name_of_people.model")
print(model2.most_similar("吃饭"))
print(model2.similarity("省长","省委书记"))

model2 = word2vec.Word2Vec.load("line_sentnce.model")
print(model2.similarity("李达康","市委书记"))


top3 = model2.wv.similar_by_word(word="李达康",topn=3)
print(top3)

model2 = word2vec.Word2Vec.load("PathLineSentences.model")
print(model2.similarity("李达康","书记"))
print(model2.wv.similarity("李达康","书记"))
print(model2.wv.doesnt_match(words=["李达康","高育良","赵立春"]))

model = word2vec.Word2Vec.load("left.model")
print(model.similarity("李达康","书记"))

结果如下:

0.5434648
0.8383337
0.7435267
[('woman', 0.7435266971588135), ('girl', 0.6460582613945007), ('creature', 0.589219868183136), ('person', 0.570125937461853), ('evil', 0.5688984990119934), ('god', 0.5465947389602661), ('boy', 0.544859766960144), ('bride', 0.5401148796081543), ('soul', 0.5365912914276123), ('stranger', 0.531282901763916)]
[('queen', 0.7230167388916016)]
[('只能', 0.9983761310577393), ('招待所', 0.9983713626861572), ('深深', 0.9983667135238647), ('干警', 0.9983251094818115), ('警察', 0.9983127117156982), ('公安', 0.9983105659484863), ('赵德汉', 0.9982908964157104), ('似乎', 0.9982795715332031), ('一场', 0.9982751607894897), ('才能', 0.9982657432556152)]
0.97394305
0.99191403
[('新', 0.9974302053451538), ('赵立春', 0.9974139928817749), ('谈一谈', 0.9971731901168823)]
0.91472965
0.91472965
高育良
0.88518995

8. 参考链接

https://github.com/RaRe-Technologies/gensim

https://github.com/RaRe-Technologies/gensim/blob/develop/docs/notebooks/word2vec.ipynb

猜你喜欢

转载自my.oschina.net/u/3800567/blog/2875805