Chapter two
Rationlism & Empiricism
Rationalism and empiricism
Rationalism (processed in the human way of doing lexical, syntactic, semantic analysis)
Natural Language Processing
use meaning, grammatical structure plus the state when analyzing the content of meaning.
The problem here? ? ? ?
Syntax accommodative? ? ? Can not accommodate does not comply with the syntax of the sentence.
Syntax rigor? ? ? Reject illegal sentence.
Simple sentence requires a lot of analysis to explain? ? ? Nani? ? ?
Empiricism (Big Data)
instead of the human knowledge of the language with plenty of corpora.
Use machine learning to identify important structures and relationships.
The problem here? ? ? ?
We need to rely on a large number of effective corpora, rather than the people's point of view.
What is a word? ? ?
= Things Separated by Tokens White Space (separated by a space, and that Chinese do ???).
Hyphennation (ampersands in English -)
(Database & the Data-Base) Chinese do not have this problem
Case
( "at The Bath at The Shop & Bath shop) Chinese do not have this problem
Morphology
Retrieval & Retrieve & retrieved & Retrieving
punctuation
'honest' & honest! & honest
course, the above-mentioned problems are also the most simple processing scheme, ignoring case, ignoring punctuation.
English only need to follow the space can be word, Chinese word segmentation is a very troublesome thing, the following example with java jieba word:
package com.lbh.jieba;
import com.huaban.analysis.jieba.JiebaSegmenter;
import java.util.List;
public class WordsCounter {
public static void main(String[] args) {
String testText = "夜色笼罩大地，有着一道道的篝火自山林间燃烧起来，宛如夜空的星辰。\n" +
"\n" +
" 周元，左丘青鱼，绿萝等人围在一簇篝火旁。\n" +
"\n" +
" 望着眼前的周元，左丘青鱼与绿萝此时还稍微的有点不真实的感觉，特别是当她们在感受到周围那许多天渊域的强者在面对着周元都是显得有些敬畏的态度时，那种不真实就变得更为的强烈了。\n" +
"\n" +
" “周元，这些年原来你是去了混元天吗？还加入了天渊域？”绿萝小声的问道。\n" +
"\n" +
" 周元笑着点点头。\n" +
"\n" +
" “我听见他们，似乎称你为元老？这是你的外号吗？”左丘青鱼有些敏锐的把握住其他人对周元的称呼，元老这个称谓，不论在哪里显然都不是能够随随便便达到的。\n" +
"\n" +
" 而且，在来到古源天之前，她们也恶补了其余诸天的诸多知识，所以她们知道天渊域，同样也知道天渊域的元老代表着什么。\n" +
"\n" +
" 正因为知道这代表着什么，所以左丘青鱼才不敢那么去想，反而宁愿是相信这是一种特殊的外号。\n" +
"\n" +
" 周元笑了笑，倒并没有隐瞒的意思：“我现在的确是天渊域的元老...而且也正是你所想的那种，并非是什么外号。”\n" +
"\n" +
" 于是下一刻，左丘青鱼与绿萝便是有些呆滞的望着他。\n" +
"\n" +
" 这种震撼比周元赶走那童鹤还要来得强烈。\n" +
"\n" +
" 左丘青鱼艰难的看向一旁的秦莲等人，后者展现出来的源气威压，比此前那骑着巨鹤的白发男子还要更强，这般人物即便是在天渊域中必然也是最顶尖的。\n" +
"\n" +
" “他是在开玩笑吧？”左丘青鱼呐呐道。\n" +
"\n" +
" 秦莲见状，也是忍不住的一笑，旋即正色道：“周元的确是我们天渊域五大元老之一，并非是玩笑。”\n" +
"\n" +
" 左丘青鱼嘴角都是微微抽了一下：“可是...据我所知，天渊域的元老，可都是法域强者啊？！”\n" +
"\n" +
" 法域境，在她们百花仙宗，也就唯有宗主一人而已！\n" +
"\n" +
" 那可是苍玄天内顶尖的强者了，虽说如今的周元比当初也是厉害了许多，可显然与法域境还有着天大的差距。\n" +
"\n" +
" “周元是苍渊大尊的亲传弟子，按照大尊所立的规矩，他有资格成为天渊域的元老。”秦莲解释道，她也是知晓周元这天阳境的实力成为元老是何等让人感到不可思议的事情，所以对于左丘青鱼，绿萝的反应很是理解。\n" +
"\n" +
" “大尊的亲传弟子...”\n" +
"\n" +
" 左丘青鱼与绿萝面面相觑，最终苦笑一声，原来这家伙有这等际遇，怪不得了。\n" +
"\n" +
" 大尊啊，那可是圣者境，他们苍玄天中，当苍玄老祖陨落后，似乎就再未曾出现过圣者了。\n" +
"\n" +
" “别说我了，说说你们吧...你们没跟苍玄天的大部队在一起吗？为何会独自的跑出来？”周元摆了摆手，有些疑惑的问道。\n" +
"\n" +
" 听到他这话，左丘青鱼与绿萝脸颊都是变得黯淡下来。\n" +
"\n" +
" “唉，别提了，苍玄天的大部队在进入古源天没多久，便是出现了巨大的变故。”\n" +
"\n" +
" 绿萝咬了咬牙，有些气愤的道：“都是圣宫那些混蛋！”\n" +
"\n" +
" “怎么回事？”周元眼神微凝，圣宫二字落在耳中，也是引得他眼眸深处有着寒气涌现，当年的那些恩怨，可并不会随着时间的流逝就消失，反而是变得更为的浓烈了。\n" +
"\n" +
" 毕竟他会离开苍玄天，远走混元天从某种意义来说，也是因为圣宫的逼迫。\n" +
"\n" +
" 当然，更重要的是，他们还是夭夭沉睡至今的罪魁祸首！\n" +
"\n" +
" 左丘青鱼轻叹一口气，缓缓道：“此事恐怕还得从你当年那里说起...当初你将苍玄圣印破碎，圣印碎片洒落于整个苍玄天，此后的这些年，各方势力都是在找寻以及争夺，为此倒是掀起过不少的战争。”\n" +
"\n" +
" 周元闻言，同样是暗叹了一口气，苍玄圣印这等至宝的诱惑力太大了，当其化为碎片散开时，难免会引动许多的贪婪，可这在当初也是没办法的事情，如果他不这么做，苍玄圣印必定会落在圣元宫主的手中，到了那个时候，苍玄天才是会有大劫难。\n" +
"\n" +
" “不过好在除开圣宫的五大圣宗还比较理智，他们也是在搜集着圣印碎片，同时皆是在戒备着圣宫。”\n" +
"\n" +
" “毕竟圣宫那位圣元宫主，如今算是苍玄天唯一的一位伪圣...”\n" +
"\n" +
" “他的伤好了？”周元皱眉，他记得当年那一场大战，那圣元宫主应该是被重创了。\n" +
"\n" +
" 左丘青鱼摇摇头：“从局势来看，应该并未彻底恢复，不然的话苍玄天早就有大变故了...但即便如此，圣宫如今在苍玄天的威势依旧愈发的惊人。”\n" +
"\n" +
" “在这些年圣印碎片的找寻中，圣宫与五大圣宗屡屡发生冲突，也爆发了无数的战斗，双方皆有不小的损伤。”\n" +
"\n" +
" 说着，左丘青鱼的神色变得有些沉重起来：“但这种情况随着时间的推移对于五大圣宗而言则是有些不利起来，我曾听宗主说，圣元宫主的恢复在加快。”\n" +
"\n" +
" 周元缓缓的道：“五大圣宗没有考虑联手剿灭圣宫吗？”\n" +
"\n" +
" 圣宫虽强，但如果五大圣宗齐心协力的话，未必会输吧？\n" +
"\n" +
" 绿萝道：“五大圣宗虽然在面对圣宫这上面保持着一致，但毕竟不可能真的完全齐心，想要剿灭圣宫，就得对付圣元宫主，即便那一位如今未曾恢复，可毕竟是伪圣...”\n" +
"\n" +
" “若是将他逼得急了，五大圣宗的法域强者，恐怕起码会陨落一半。”\n" +
"\n" +
" “这个代价，太过的惨重，就算是五大圣宗也不敢轻易的发动。”\n" +
"\n" +
" “所以如今五大圣宗的策略，还是以搜集苍玄圣印碎片为主，只要这些圣印碎片不会被圣元宫主全部的夺得，那么他就不可能成为真正的圣者。”\n" +
"\n" +
" 周元轻轻点头，有些可惜，其实现在是解决圣元宫主最好的时机，但五大圣宗显然不可能承受那么巨大的代价，毕竟人人都有私心，一旦自家的法域强者陨落，那将会是毁灭性的代价，说不得到时候还会跌落圣宗之位。\n" +
"\n" +
" “这些就是如今苍玄天内的局势了，反正一团乱。”\n" +
"\n" +
" “至于我们为什么会单独出现这里...”\n" +
"\n" +
" 左丘青鱼咬了咬银牙，娇媚的脸蛋上有些愤怒涌现：“那当然也是圣宫做的好事！”\n" +
"\n" +
" “在我们苍玄天的大部队进入古源天的第一时刻，那圣宫便是与我们五大圣宗的人马起了冲突，但当时并没有爆发战斗，圣宫的人马选择了撤退。”\n" +
"\n" +
" “我们以为他们暂时的选择了息事宁人...”\n" +
"\n" +
" “可没想到的是，七天之后，圣宫夜袭我们苍玄天大本营...苍玄天各方势力死伤惨重，节节败退。”\n" +
"\n" +
" “圣宫实力虽强，可也没强到能够打败五大圣宗合力的程度吧？”周元有些惊疑的道。\n" +
"\n" +
" 五大圣宗此次所派出的人马，必然也是最为精锐的强者，怎会如此不堪一击？\n" +
"\n" +
" 左丘青鱼与绿萝对视一眼，两女的脸色都是在此时变得格外冰寒下来。\n" +
"\n" +
" “因为那圣宫不知从何请来了一支圣族的队伍...”\n" +
"\n" +
" 篝火旁，原本还有着的一些吵杂声也是在此时悄无声息的消失了。\n" +
"\n" +
" 周元的神色，也是渐渐的阴沉。";
JiebaSegmenter jiebaSegmenter = new JiebaSegmenter();
List<String> strings = jiebaSegmenter.sentenceProcess(testText);
System.out.println(strings);
}
}
The code can be realized a simple word, word need be selected according to the specific scenario.
Stuttered word JiebaSegmenter provides a sentenceProcess method, word, word stammer supports three modes, the interest can go to learn. '
I first saw an example of "mathematical beauty," a book of this mode my first reaction was remembered before.
"Tsinghua University" is a word or "Tsinghua University" and "University."
This need to look at different scenarios, and for the search engines, they are sure will be able to divide the word apart to imagine and use the "Tsinghua University" found less than "Tsinghua University" This scene related information.
Frequency statistics only need to traverse the list can own their own statistics, this time the advantage of python is manifested, there is no more convenient than Counter statistical tools ( possibly ignorance ).
After the word you need to do another thing, word frequency statistics. Suppose a word w, its word frequency is generally referred to as:
F（w）
The general word frequency distribution function is:
F (r) = C / r ^ a, where a is approximately equal to, C equal to about 0.1;
