02-NLP-gensim中文处理案例

word2vec训练中文模型

 

1.准备数据与预处理

首先需要一份比较大的中文语料数据，可以考虑中文的维基百科（也可以试试搜狗的新闻语料库）。中文维基百科的打包文件地址为 
https://dumps.wikimedia.org/zhwiki/latest/zhwiki-latest-pages-articles.xml.bz2

中文维基百科的数据不是太大，xml的压缩文件大约1G左右。首先用 process_wiki_data.py处理这个XML压缩文件，执行：python process_wiki_data.py zhwiki-latest-pages-articles.xml.bz2 wiki.zh.text

 

1. #!/usr/bin/env python
2. # -*- coding: utf-8 -*-
3. # process_wiki_data.py 用于解析XML，将XML的wiki数据转换为text格式
5. import logging
6. import os.path
7. import sys
9. from gensim.corpora import WikiCorpus
11. if __name__ == '__main__':
12. program = os.path.basename(sys.argv[0])
13. logger = logging.getLogger(program)
15. logging.basicConfig(format='%(asctime)s: %(levelname)s: %(message)s')
16. logging.root.setLevel(level=logging.INFO)
17. logger.info("running %s" % ' '.join(sys.argv))
19. # check and process input arguments
20. if len(sys.argv) < 3:
21. print globals()['__doc__'] % locals()
22. sys.exit(1)
23. inp, outp = sys.argv[1:3]
24. space = " "
25. i = 0
27. output = open(outp, 'w')
28. wiki = WikiCorpus(inp, lemmatize=False, dictionary={})
29. for text in wiki.get_texts():
30. output.write(space.join(text) + "\n")
31. i = i + 1
32. if (i % 10000 == 0):
33. logger.info("Saved " + str(i) + " articles")
35. output.close()
36. logger.info("Finished Saved " + str(i) + " articles")

得到信息：

 

1. 2016-08-11 20:39:22,739: INFO: running process_wiki.py zhwiki-latest-pages-articles.xml.bz2 wiki.zh.text
2. 2016-08-11 20:40:08,329: INFO: Saved 10000 articles
3. 2016-08-11 20:40:45,501: INFO: Saved 20000 articles
4. 2016-08-11 20:41:23,659: INFO: Saved 30000 articles
5. 2016-08-11 20:42:01,748: INFO: Saved 40000 articles
6. 2016-08-11 20:42:33,779: INFO: Saved 50000 articles
7. ......
8. 2016-08-11 20:55:23,094: INFO: Saved 200000 articles
9. 2016-08-11 20:56:14,692: INFO: Saved 210000 articles
10. 2016-08-11 20:57:04,614: INFO: Saved 220000 articles
11. 2016-08-11 20:57:57,979: INFO: Saved 230000 articles
12. 2016-08-11 20:58:16,621: INFO: finished iterating over Wikipedia corpus of 232894 documents with 51603419 positions (total 2581444 articles, 62177405 positions before pruning articles shorter than 50 words)
13. 2016-08-11 20:58:16,622: INFO: Finished Saved 232894 articles

Python的话可用jieba完成分词，生成分词文件wiki.zh.text.seg 
接着用word2vec工具训练： 
python train_word2vec_model.py wiki.zh.text.seg wiki.zh.text.model wiki.zh.text.vector

 

1. #!/usr/bin/env python
2. # -*- coding: utf-8 -*-
3. # train_word2vec_model.py用于训练模型
5. import logging
6. import os.path
7. import sys
8. import multiprocessing
10. from gensim.corpora import WikiCorpus
11. from gensim.models import Word2Vec
12. from gensim.models.word2vec import LineSentence
14. if __name__ == '__main__':
15. program = os.path.basename(sys.argv[0])
16. logger = logging.getLogger(program)
18. logging.basicConfig(format='%(asctime)s: %(levelname)s: %(message)s')
19. logging.root.setLevel(level=logging.INFO)
20. logger.info("running %s" % ' '.join(sys.argv))
22. # check and process input arguments
23. if len(sys.argv) < 4:
24. print globals()['__doc__'] % locals()
25. sys.exit(1)
26. inp, outp1, outp2 = sys.argv[1:4]
28. model = Word2Vec(LineSentence(inp), size=400, window=5, min_count=5,
29. workers=multiprocessing.cpu_count())
31. # trim unneeded model memory = use(much) less RAM
32. #model.init_sims(replace=True)
33. model.save(outp1)
34. model.save_word2vec_format(outp2, binary=False)

运行信息

 

1. 2016-08-12 09:50:02,586: INFO: running python train_word2vec_model.py wiki.zh.text.seg wiki.zh.text.model wiki.zh.text.vector
2. 2016-08-12 09:50:02,592: INFO: collecting all words and their counts
3. 2016-08-12 09:50:02,592: INFO: PROGRESS: at sentence #0, processed 0 words and 0 word types
4. 2016-08-12 09:50:12,476: INFO: PROGRESS: at sentence #10000, processed 12914562 words and 254662 word types
5. 2016-08-12 09:50:20,215: INFO: PROGRESS: at sentence #20000, processed 22308801 words and 373573 word types
6. 2016-08-12 09:50:28,448: INFO: PROGRESS: at sentence #30000, processed 30724902 words and 460837 word types
7. ...
8. 2016-08-12 09:52:03,498: INFO: PROGRESS: at sentence #210000, processed 143804601 words and 1483608 word types
9. 2016-08-12 09:52:07,772: INFO: PROGRESS: at sentence #220000, processed 149352283 words and 1521199 word types
10. 2016-08-12 09:52:11,639: INFO: PROGRESS: at sentence #230000, processed 154741839 words and 1563584 word types
11. 2016-08-12 09:52:12,746: INFO: collected 1575172 word types from a corpus of 156430908 words and 232894 sentences
12. 2016-08-12 09:52:13,672: INFO: total 278291 word types after removing those with count<5
13. 2016-08-12 09:52:13,673: INFO: constructing a huffman tree from 278291 words
14. 2016-08-12 09:52:29,323: INFO: built huffman tree with maximum node depth 25
15. 2016-08-12 09:52:29,683: INFO: resetting layer weights
16. 2016-08-12 09:52:38,805: INFO: training model with 4 workers on 278291 vocabulary and 400 features, using 'skipgram'=1 'hierarchical softmax'=1 'subsample'=0 and 'negative sampling'=0
17. 2016-08-12 09:52:49,504: INFO: PROGRESS: at 0.10% words, alpha 0.02500, 15008 words/s
18. 2016-08-12 09:52:51,935: INFO: PROGRESS: at 0.38% words, alpha 0.02500, 44434 words/s
19. 2016-08-12 09:52:54,779: INFO: PROGRESS: at 0.56% words, alpha 0.02500, 53965 words/s
20. 2016-08-12 09:52:57,240: INFO: PROGRESS: at 0.62% words, alpha 0.02491, 52116 words/s
21. 2016-08-12 09:52:58,823: INFO: PROGRESS: at 0.72% words, alpha 0.02494, 55804 words/s
22. 2016-08-12 09:53:03,649: INFO: PROGRESS: at 0.94% words, alpha 0.02486, 58277 words/s
23. 2016-08-12 09:53:07,357: INFO: PROGRESS: at 1.03% words, alpha 0.02479, 56036 words/s
24. ......
25. 2016-08-12 19:22:09,002: INFO: PROGRESS: at 98.38% words, alpha 0.00044, 85936 words/s
26. 2016-08-12 19:22:10,321: INFO: PROGRESS: at 98.50% words, alpha 0.00044, 85971 words/s
27. 2016-08-12 19:22:11,934: INFO: PROGRESS: at 98.55% words, alpha 0.00039, 85940 words/s
28. 2016-08-12 19:22:13,384: INFO: PROGRESS: at 98.65% words, alpha 0.00036, 85960 words/s
29. 2016-08-12 19:22:13,883: INFO: training on 152625573 words took 1775.1s, 85982 words/s
30. 2016-08-12 19:22:13,883: INFO: saving Word2Vec object under wiki.zh.text.model, separately None
31. 2016-08-12 19:22:13,884: INFO: not storing attribute syn0norm
32. 2016-08-12 19:22:13,884: INFO: storing numpy array 'syn0' to wiki.zh.text.model.syn0.npy
33. 2016-08-12 19:22:20,797: INFO: storing numpy array 'syn1' to wiki.zh.text.model.syn1.npy
34. 2016-08-12 19:22:40,667: INFO: storing 278291x400 projection weights into wiki.zh.text.vector

测试模型效果：

 

1. In [1]: import gensim
3. In [2]: model = gensim.models.Word2Vec.load("wiki.zh.text.model")
5. In [3]: model.most_similar(u"足球")
6. Out[3]:
7. [(u'\u8054\u8d5b', 0.6553816199302673),
8. (u'\u7532\u7ea7', 0.6530429720878601),
9. (u'\u7bee\u7403', 0.5967546701431274),
10. (u'\u4ff1\u4e50\u90e8', 0.5872289538383484),
11. (u'\u4e59\u7ea7', 0.5840631723403931),
12. (u'\u8db3\u7403\u961f', 0.5560152530670166),
13. (u'\u4e9a\u8db3\u8054', 0.5308005809783936),
14. (u'allsvenskan', 0.5249762535095215),
15. (u'\u4ee3\u8868\u961f', 0.5214947462081909),
16. (u'\u7532\u7ec4', 0.5177896022796631)]
18. In [4]: result = model.most_similar(u"足球")
20. In [5]: for e in result:
21. print e[0], e[1]
22. ....:
23. 联赛 0.65538161993
24. 甲级 0.653042972088
25. 篮球 0.596754670143
26. 俱乐部 0.587228953838
27. 乙级 0.58406317234
28. 足球队 0.556015253067
29. 亚足联 0.530800580978
30. allsvenskan 0.52497625351
31. 代表队 0.521494746208
32. 甲组 0.51778960228
34. In [6]: result = model.most_similar(u"男人")
36. In [7]: for e in result:
37. print e[0], e[1]
38. ....:
39. 女人 0.77537125349
40. 家伙 0.617369174957
41. 妈妈 0.567102909088
42. 漂亮 0.560832381248
43. 잘했어 0.540875017643
44. 谎言 0.538448691368
45. 爸爸 0.53660941124
46. 傻瓜 0.535608053207
47. 예쁘다 0.535151124001
48. mc刘 0.529670000076
50. In [8]: result = model.most_similar(u"女人")
52. In [9]: for e in result:
53. print e[0], e[1]
54. ....:
55. 男人 0.77537125349
56. 我的某 0.589010596275
57. 妈妈 0.576344847679
58. 잘했어 0.562340974808
59. 美丽 0.555426716805
60. 爸爸 0.543958246708
61. 新娘 0.543640494347
62. 谎言 0.540272831917
63. 妞儿 0.531066179276
64. 老婆 0.528521537781
66. In [10]: result = model.most_similar(u"青蛙")
68. In [11]: for e in result:
69. print e[0], e[1]
70. ....:
71. 老鼠 0.559612870216
72. 乌龟 0.489831030369
73. 蜥蜴 0.478990525007
74. 猫 0.46728849411
75. 鳄鱼 0.461885392666
76. 蟾蜍 0.448014199734
77. 猴子 0.436584025621
78. 白雪公主 0.434905380011
79. 蚯蚓 0.433413207531
80. 螃蟹 0.4314712286
82. In [12]: result = model.most_similar(u"姨夫")
84. In [13]: for e in result:
85. print e[0], e[1]
86. ....:
87. 堂伯 0.583935439587
88. 祖父 0.574735701084
89. 妃所生 0.569327116013
90. 内弟 0.562012672424
91. 早卒 0.558042645454
92. 曕 0.553856015205
93. 胤祯 0.553288519382
94. 陈潜 0.550716996193
95. 愔之 0.550510883331
96. 叔父 0.550032019615
98. In [14]: result = model.most_similar(u"衣服")
100. In [15]: for e in result:
101. print e[0], e[1]
102. ....:
103. 鞋子 0.686688780785
104. 穿着 0.672499775887
105. 衣物 0.67173999548
106. 大衣 0.667605519295
107. 裤子 0.662670075893
108. 内裤 0.662210345268
109. 裙子 0.659705817699
110. 西装 0.648508131504
111. 洋装 0.647238850594
112. 围裙 0.642895817757
114. In [16]: result = model.most_similar(u"公安局")
116. In [17]: for e in result:
117. print e[0], e[1]
118. ....:
119. 司法局 0.730189085007
120. 公安厅 0.634275555611
121. 公安 0.612798035145
122. 房管局 0.597343325615
123. 商业局 0.597183346748
124. 军管会 0.59476184845
125. 体育局 0.59283208847
126. 财政局 0.588721752167
127. 戒毒所 0.575558543205
128. 新闻办 0.573395550251
130. In [18]: result = model.most_similar(u"铁道部")
132. In [19]: for e in result:
133. print e[0], e[1]
134. ....:
135. 盛光祖 0.565509021282
136. 交通部 0.548688530922
137. 批复 0.546967327595
138. 刘志军 0.541010737419
139. 立项 0.517836689949
140. 报送 0.510296344757
141. 计委 0.508456230164
142. 水利部 0.503531932831
143. 国务院 0.503227233887
144. 经贸委 0.50156635046
146. In [20]: result = model.most_similar(u"清华大学")
148. In [21]: for e in result:
149. print e[0], e[1]
150. ....:
151. 北京大学 0.763922810555
152. 化学系 0.724210739136
153. 物理系 0.694550514221
154. 数学系 0.684280991554
155. 中山大学 0.677202701569
156. 复旦 0.657914161682
157. 师范大学 0.656435549259
158. 哲学系 0.654701948166
159. 生物系 0.654403865337
160. 中文系 0.653147578239
162. In [22]: result = model.most_similar(u"卫视")
164. In [23]: for e in result:
165. print e[0], e[1]
166. ....:
167. 湖南 0.676812887192
168. 中文台 0.626506924629
169. 収蔵 0.621356606483
170. 黄金档 0.582251906395
171. cctv 0.536769032478
172. 安徽 0.536752820015
173. 非同凡响 0.534517168999
174. 唱响 0.533438682556
175. 最强音 0.532605051994
176. 金鹰 0.531676828861
178. In [26]: result = model.most_similar(u"林丹")
180. In [27]: for e in result:
181. print e[0], e[1]
182. ....:
183. 黄综翰 0.538035452366
184. 蒋燕皎 0.52646958828
185. 刘鑫 0.522252976894
186. 韩晶娜 0.516120731831
187. 王晓理 0.512289524078
188. 王适 0.508560419083
189. 杨影 0.508159279823
190. 陈跃 0.507353425026
191. 龚智超 0.503159761429
192. 李敬元 0.50262516737
194. In [28]: result = model.most_similar(u"语言学")
196. In [29]: for e in result:
197. print e[0], e[1]
198. ....:
199. 社会学 0.632598280907
200. 人类学 0.623406708241
201. 历史学 0.618442356586
202. 比较文学 0.604823827744
203. 心理学 0.600066184998
204. 人文科学 0.577783346176
205. 社会心理学 0.575571238995
206. 政治学 0.574541330338
207. 地理学 0.573896467686
208. 哲学 0.573873817921
210. In [30]: result = model.most_similar(u"计算机")
212. In [31]: for e in result:
213. print e[0], e[1]
214. ....:
215. 自动化 0.674171924591
216. 应用 0.614087462425
217. 自动化系 0.611132860184
218. 材料科学 0.607891201973
219. 集成电路 0.600370049477
220. 技术 0.597518980503
221. 电子学 0.591316461563
222. 建模 0.577238917351
223. 工程学 0.572855889797
224. 微电子 0.570086717606
226. In [32]: model.similarity(u"计算机", u"自动化")
227. Out[32]: 0.67417196002404789
229. In [33]: model.similarity(u"女人", u"男人")
230. Out[33]: 0.77537125129824813
232. In [34]: model.doesnt_match(u"早餐 晚餐 午餐 中心".split())
233. Out[34]: u'\u4e2d\u5fc3'
235. In [35]: print model.doesnt_match(u"早餐 晚餐 午餐 中心".split())
236. 中心

+