Detailed use in Chinese word segmentation based on jieba package in python (2)

Detailed use in Chinese word segmentation based on jieba package in python (2)

01. Preface

Detailed use in Chinese word segmentation based on jieba package in python (1) Some basic content of jieba word segmentation has been introduced, and now I will introduce it.

02. Keyword extraction

02.01Keyword extraction based on TF-IDF algorithm

import jieba.analyse

• jieba.analyse.extract_tags(sentence, topK=20, withWeight=False,
  allowPOS=())
  It should be noted that:
  1.sentence is the text to be extracted
  2.topK is to return several keywords with the largest TF/IDF weight , the default value is 20
  3.withWeight is whether to return the keyword weight value together, the default value is False
  4.allowPOS only includes words with the specified part of speech, the default value is empty, that is, no filtering
• jieba.analyse.TFIDF(idf_path=None) Create a new TFIDF instance, idf_path is the IDF frequency file

code example

#!/usr/bin/env python
# -*- coding: utf-8 -*-
# @Date    : 2018-05-05 22:15:13
# @Author  : JackPI ([email protected])
# @Link    : https://blog.csdn.net/meiqi0538
# @Version : $Id$
import jieba
import jieba.analyse
#读取文件,返回一个字符串，使用utf-8编码方式读取，该文档位于此python同以及目录下
content  = open('人民的名义.txt','r',encoding='utf-8').read()
tags = jieba.analyse.extract_tags(content,topK=10) 
print(",".join(tags))

operation result

Building prefix dict from the default dictionary ...
Dumping model to file cache C:\Users\JACKPI~1\AppData\Local\Temp\jieba.cache
Loading model cost 1.280 seconds.
Prefix dict has been built succesfully.
侯亮,李达康,高育良,祁同伟,高小琴,瑞金,陈海,老师,丁义珍,成功
[Finished in 5.9s]

The Inverse Document Frequency (IDF) text corpus used for keyword extraction can be switched to the path of a custom corpus

• Usage: jieba.analyse.set_idf_path(file_name) # file_name is the path of the custom corpus
• Custom Corpus Example
  劳动防护 13.900677652
勞動防護 13.900677652
生化学 13.900677652
生化學 13.900677652
奥萨贝尔 13.900677652
奧薩貝爾 13.900677652
考察队员 13.900677652
考察隊員 13.900677652
岗上 11.5027823792
崗上 11.5027823792
倒车档 12.2912397395
倒車檔 12.2912397395
编译 9.21854642485
編譯 9.21854642485
蝶泳 11.1926274509
外委 11.8212361103
• Usage example

import jieba
import jieba.analyse
#读取文件,返回一个字符串，使用utf-8编码方式读取，该文档位于此python同以及目录下
content  = open('idf.txt.big','r',encoding='utf-8').read()
tags = jieba.analyse.extract_tags(content, topK=10)
print(",".join(tags))

result

Building prefix dict from the default dictionary ...
Loading model from cache C:\Users\JACKPI~1\AppData\Local\Temp\jieba.cache
Loading model cost 1.186 seconds.
Prefix dict has been built succesfully.
13.2075304714,13.900677652,12.8020653633,12.5143832909,12.2912397395,12.1089181827,11.9547675029,11.8212361103,11.7034530746,11.598092559
[Finished in 20.9s]

The Stop Words text corpus used for keyword extraction can be switched to the path of a custom corpus

• Usage: jieba.analyse.set_stop_words(file_name) # file_name is the path of the custom corpus
• Example of custom corpus:

!
"
#
$
%
&
'
(
)
*
+
,
-
--
.
..
...
......
...................
./
.一
记者
数
年
月
日
时
分
秒
/
//
0
1
2
3
4

• Usage example

import jieba
import jieba.analyse
#读取文件,返回一个字符串，使用utf-8编码方式读取，该文档位于此python同以及目录下
content  = open(u'人民的名义.txt','r',encoding='utf-8').read()
jieba.analyse.set_stop_words("stopwords.txt")
tags = jieba.analyse.extract_tags(content, topK=10)
print(",".join(tags))

result

Building prefix dict from the default dictionary ...
Loading model from cache C:\Users\JACKPI~1\AppData\Local\Temp\jieba.cache
Loading model cost 1.316 seconds.
Prefix dict has been built succesfully.
侯亮,李达康,高育良,祁同伟,高小琴,瑞金,陈海,老师,丁义珍,成功
[Finished in 5.2s]

Example of keyword weight value returned with keywords

import jieba
import jieba.analyse
#读取文件,返回一个字符串，使用utf-8编码方式读取，该文档位于此python同以及目录下
content  = open(u'人民的名义.txt','r',encoding='utf-8').read()
jieba.analyse.set_stop_words("stopwords.txt")
tags = jieba.analyse.extract_tags(content, topK=10,withWeight=True)
for tag in tags:
    print("tag:%s\t\t weight:%f"%(tag[0],tag[1]))

result

Building prefix dict from the default dictionary ...
Loading model from cache C:\Users\JACKPI~1\AppData\Local\Temp\jieba.cache
Loading model cost 1.115 seconds.
Prefix dict has been built succesfully.
tag:侯亮       weight:0.257260
tag:李达康      weight:0.143901
tag:高育良      weight:0.108856
tag:祁同伟      weight:0.098479
tag:高小琴      weight:0.062259
tag:瑞金       weight:0.060405
tag:陈海       weight:0.054036
tag:老师       weight:0.051980
tag:丁义珍      weight:0.049729
tag:成功       weight:0.046647
[Finished in 5.3s]

02.02 Part-of-speech tagging

• jieba.posseg.POSTokenizer(tokenizer=None) Create a new custom tokenizer. The tokenizer
  parameter can specify the jieba.Tokenizer tokenizer used internally. jieba.posseg.dt is the default POS tagging tokenizer.
• Label the part of speech of each word after sentence segmentation, using a notation method compatible with ictclas.
• Usage example

>>> import jieba.posseg as pseg
>>> words = pseg.cut("我爱北京天安门")
>>> for word, flag in words:
...    print('%s %s' % (word, flag))
...
我 r
爱 v
北京 ns
天安门 ns

part-of-speech table

part-of-speech encoding	part-of-speech name	annotation
Ag	morpheme	Adjective morphemes. The adjective code is a, and the morpheme code g is preceded by an A.
a	form word	Take the first letter of the English adjective adjective.
ad	adverb	Adjectives that are directly adverbial. The adjective code a and the adverb code d are combined.
an	noun	Adjectives that function as nouns. The adjective code a and the noun code n are combined.
b	distinguishing word	Take the initials of the Chinese character "bie".
c	conjunction	Take the first letter of the English conjunction conjunction.
dg	paramorpheme	Adverbial morphemes. The adverb code is d, and the morpheme code g is preceded by a D.
d	adverb	Take the 2nd letter of adverb because the 1st letter is already used for an adjective.
e	interjection	Take the first letter of the English interjection exclamation.
f	Position of the word	Take the Chinese character "square"
g	morpheme	Most morphemes can be used as the "root" of compound words, taking the initials of the Chinese character "root".
h	preceding ingredient	Take the first letter of the English head.
i	idiom	Take the first letter of the English idiom idiom.
j	Abbreviation	Take the initials of the Chinese character "Jian".
k	followed by ingredients
l	idioms	The idiom has not yet become an idiom, and it is a bit "temporary", taking the initials of "pro".
m	numeral	Take the third letter of English numeral, n, u have other uses.
Ng	noun	Noun morphemes. The noun code is n, and the morpheme code g is preceded by N.
n	noun	Take the first letter of the English noun noun.
no	Personal name	The noun code n is combined with the initials of "ren".
ns	Place name	The noun code n and the location word code s are combined.
nt	Institutional groups	The initial consonant of "group" is t, and the noun codes n and t are combined together.
nz	other proper names	The first letter of the initials of "special" is z, and the noun codes n and z are combined together.
O	Onomatopoeia	Take the first letter of the English onomatopoeia onomatopoeia.
p	preposition	Take the first letter of the English preposition prepositional.
q	量词	取英语 quantity的第1个字母。
r	代词	取英语代词 pronoun的第2个字母,因p已用于介词。
s	处所词	取英语 space的第1个字母。
tg	时语素	时间词性语素。时间词代码为 t,在语素的代码g前面置以T。
t	时间词	取英语 time的第1个字母。
u	助词	取英语助词 auxiliary
vg	动语素	动词性语素。动词代码为 v。在语素的代码g前面置以V。
v	动词	取英语动词 verb的第一个字母
vd	副动词	直接作状语的动词。动词和副词的代码并在一起。
vn	名动词	指具有名词功能的动词。动词和名词的代码并在一起。
w	标点符号
x	非语素字	非语素字只是一个符号，字母 x通常用于代表未知数、符号。
y	语气词	取汉字“语”的声母。
z	状态词	取汉字“状”的声母的前一个字母
un	未知词	不可识别词及用户自定义词组。取英文Unkonwn首两个字母。(非北大标准，CSW分词中定义)

02.03并行分词

• 原理：将目标文本按行分隔后，把各行文本分配到多个 Python 进程并行分词，然后归并结果，从而获得分词速度的可观提升
• 基于 python 自带的 multiprocessing 模块，目前暂不支持 Windows
• 用法
  jieba.enable_parallel(4) # 开启并行分词模式，参数为并行进程数
jieba.disable_parallel() # 关闭并行分词模式
  官方使用案例

import sys
import time
sys.path.append("../../")
import jieba

jieba.enable_parallel()

url = sys.argv[1]
content = open(url,"rb").read()
t1 = time.time()
words = "/ ".join(jieba.cut(content))

t2 = time.time()
tm_cost = t2-t1

log_f = open("1.log","wb")
log_f.write(words.encode('utf-8'))

print('speed %s bytes/second' % (len(content)/tm_cost))

• 注意：并行分词仅支持默认分词器 jieba.dt 和 jieba.posseg.dt。

02.04Tokenize：返回词语在原文的起止位置

注意，输入参数只接受 unicode
默认模式

import jieba
import jieba.analyse
result = jieba.tokenize(u'永和服装饰品有限公司')
for tk in result:
    print("word %s\t\t start: %d \t\t end:%d" % (tk[0],tk[1],tk[2]))

结果

Building prefix dict from the default dictionary ...
Loading model from cache C:\Users\JACKPI~1\AppData\Local\Temp\jieba.cache
Loading model cost 1.054 seconds.
Prefix dict has been built succesfully.
word 永和      start: 0        end:2
word 服装      start: 2        end:4
word 饰品      start: 4        end:6
word 有限公司        start: 6        end:10
[Finished in 3.3s]

• 搜索模式

result = jieba.tokenize(u'永和服装饰品有限公司', mode='search')
for tk in result:
    print("word %s\t\t start: %d \t\t end:%d" % (tk[0],tk[1],tk[2]))

结果

word 永和                start: 0                end:2
word 服装                start: 2                end:4
word 饰品                start: 4                end:6
word 有限                start: 6                end:8
word 公司                start: 8                end:10
word 有限公司            start: 6                end:10

02.05ChineseAnalyzer for Whoosh 搜索引擎

• 引用： from jieba.analyse import ChineseAnalyzer
• 官方案例

# -*- coding: UTF-8 -*-
from __future__ import unicode_literals
import sys,os
sys.path.append("../")
from whoosh.index import create_in,open_dir
from whoosh.fields import *
from whoosh.qparser import QueryParser

from jieba.analyse import ChineseAnalyzer

analyzer = ChineseAnalyzer()

schema = Schema(title=TEXT(stored=True), path=ID(stored=True), content=TEXT(stored=True, analyzer=analyzer))
if not os.path.exists("tmp"):
    os.mkdir("tmp")

ix = create_in("tmp", schema) # for create new index
#ix = open_dir("tmp") # for read only
writer = ix.writer()

writer.add_document(
    title="document1",
    path="/a",
    content="This is the first document we’ve added!"
)

writer.add_document(
    title="document2",
    path="/b",
    content="The second one 你 中文测试中文 is even more interesting! 吃水果"
)

writer.add_document(
    title="document3",
    path="/c",
    content="买水果然后来世博园。"
)

writer.add_document(
    title="document4",
    path="/c",
    content="工信处女干事每月经过下属科室都要亲口交代24口交换机等技术性器件的安装工作"
)

writer.add_document(
    title="document4",
    path="/c",
    content="咱俩交换一下吧。"
)

writer.commit()
searcher = ix.searcher()
parser = QueryParser("content", schema=ix.schema)

for keyword in ("水果世博园","你","first","中文","交换机","交换"):
    print("result of ",keyword)
    q = parser.parse(keyword)
    results = searcher.search(q)
    for hit in results:
        print(hit.highlights("content"))
    print("="*10)

for t in analyzer("我的好朋友是李明;我爱北京天安门;IBM和Microsoft; I have a dream. this is intetesting and interested me a lot"):
    print(t.text)

03.延迟加载

jieba 采用延迟加载，import jieba 和 jieba.Tokenizer() 不会立即触发词典的加载，一旦有必要才开始加载词典构建前缀字典。如果你想手工初始 jieba，也可以手动初始化。

import jieba
jieba.initialize()  # 手动初始化（可选）

官方使用案例

#encoding=utf-8
from __future__ import print_function
import sys
sys.path.append("../")
import jieba

def cuttest(test_sent):
    result = jieba.cut(test_sent)
    print("  ".join(result))

def testcase():
    cuttest("这是一个伸手不见五指的黑夜。我叫孙悟空，我爱北京，我爱Python和C++。")
    cuttest("我不喜欢日本和服。")
    cuttest("雷猴回归人间。")
    cuttest("工信处女干事每月经过下属科室都要亲口交代24口交换机等技术性器件的安装工作")
    cuttest("我需要廉租房")
    cuttest("永和服装饰品有限公司")
    cuttest("我爱北京天安门")
    cuttest("abc")
    cuttest("隐马尔可夫")
    cuttest("雷猴是个好网站")

if __name__ == "__main__":
    testcase()
    jieba.set_dictionary("foobar.txt")
    print("================================")
    testcase()

04.其他词典

1.占用内存较小的词典文件 https://github.com/fxsjy/jieba/raw/master/extra_dict/dict.txt.small
2.支持繁体分词更好的词典文件 https://github.com/fxsjy/jieba/raw/master/extra_dict/dict.txt.big
下载你所需要的词典，然后覆盖 jieba/dict.txt 即可；或者用 jieba.set_dictionary('data/dict.txt.big')

写在最后

由于jieba分词的内容比较多，功能也是比较强大的，笔者只是针对官方的文档进行了一定的解释。有对自然语言处理的可以关注个人订阅号，这里有关于自然语言处理、机器学习等学习资料。