网址链接：http://nbviewer.jupyter.org/gist/boskaiolo/cc3e1341f59bfbd02726
First, fix the verbosity of the logger. In this example we're logging only warnings, but for a better debug, uprint all the INFOs.
import logging
logging.basicConfig(format='%(levelname)s : %(message)s', level=logging.WARNING)
logging.root.level = logging.WARNING
Now, it's time to get some textual data. We're gonna use the 20 newsgroups dataset (more info here: http://qwone.com/~jason/20Newsgroups). As stated by its creators, it is a collection of approximately 20,000 newsgroup documents, partitioned (nearly) evenly across 20 different newsgroups.
To make things more real, we're remving email headers, footers (like signatures) and quoted messages.
from sklearn import datasets
news_dataset = datasets.fetch_20newsgroups(subset='all', remove=('headers', 'footers', 'quotes'))
# A list of text document is contained in the data variable
documents = news_dataset.data
print "In the dataset there are", len(documents), "textual documents"
print "And this is the first one:\n", documents[0]
We do now have a collection of documents. Let's start with some preprocessing steps. At first, we're gonna import all the modules we need. Then, we define a word tokenizer (https://en.wikipedia.org/wiki/Tokenization_(lexical_analysis)) with stopword removal (common words like "the", "are" and "and" are excuded from the processing, since they don't have discriminative power and they just increase the processing complexity).
import gensim
from gensim.utils import simple_preprocess
from gensim.parsing.preprocessing import STOPWORDS
def tokenize(text):
return [token for token in gensim.utils.simple_preprocess(text) if token not in gensim.parsing.preprocessing.STOPWORDS]
print "After the tokenizer, the previous document becomes:\n", tokenize(documents[0])
Next step: tokenise all the documents and build a count dictionary, that contains the count of the tokens over the complete text corpus.
processed_docs = [tokenize(doc) for doc in documents]
word_count_dict = gensim.corpora.Dictionary(processed_docs)
print "In the corpus there are", len(word_count_dict), "unique tokens"
We might want to further lower the complexity of the process, removing all the very rare tokens (the ones appearing in less than 20 documents) and the very popular ones (the ones appearing in more than 20% documents; in our case circa 4'000)
word_count_dict.filter_extremes(no_below=20, no_above=0.1) # word must appear >10 times, and no more than 20% documents
print "After filtering, in the corpus there are only", len(word_count_dict), "unique tokens"
Let's not build the bag of words representation (https://en.wikipedia.org/wiki/Bag-of-words_model) of the text documents, to create a nice vector space model (https://en.wikipedia.org/wiki/Vector_space_model). Within this methaphor, a vector lists the multiplicity of the tokens appearing in the document. The vector is indexed by the dictionary of tokens, previously built. Note that, since a restricted subset of words appears in each document, this vector is often represented in a sparse way.
bag_of_words_corpus = [word_count_dict.doc2bow(pdoc) for pdoc in processed_docs]
bow_doc1 = bag_of_words_corpus[0]
print "Bag of words representation of the first document (tuples are composed by token_id and multiplicity):\n", bow_doc1
print
for i in range(5):
print "In the document, topic_id {} (word \"{}\") appears {} time[s]".format(bow_doc1[i][0], word_count_dict[bow_doc1[i][0]], bow_doc1[i][1])
print "..."
Now, finally, the core algorithm of the analysis: LDA. Gensim offers two implementations: a monocore one, and a multicore. We use the monocore one, setting the number of topics equal to 10 (you can change it, and check the results). Try themulticore to prove the speedup!
# LDA mono-core
lda_model = gensim.models.LdaModel(bag_of_words_corpus, num_topics=10, id2word=word_count_dict, passes=5)
# LDA multicore (in this configuration, defaulty, uses n_cores-1)
# lda_model = gensim.models.LdaMulticore(bag_of_words_corpus, num_topics=10, id2word=word_count_dict, passes=5)
Here's a list of the words (and their relative weights) for each topic:
_ = lda_model.print_topics(-1)
Let's print now the topics composition, and their scores, for the first document. You will see that only few topics are represented; the others have a nil score.
for index, score in sorted(lda_model[bag_of_words_corpus[0]], key=lambda tup: -1*tup[1]):
print "Score: {}\t Topic: {}".format(score, lda_model.print_topic(index, 10))
That's wonderful! LDA is able to understand that the article is about a team game, hockey, even though the work hockey never appears in the document. Checking the ground truth for that document (the newsgroup category) it's actually correct! It was posted in sport/hockey category. Other topics, if any, account for less than 5%, so they have to be considered marginals (dirt).
news_dataset.target_names[news_dataset.target[0]]
So far, we have dealt with documents contained in the training set. What if we need to process an unseed document? Fortunately, we don't need to re-train the system (wasting lots of time), as we can just infer its topics.
unseen_document = "In my spare time I either play badmington or drive my car"
print "The unseen document is composed by the following text:", unseen_document
print
bow_vector = word_count_dict.doc2bow(tokenize(unseen_document))
for index, score in sorted(lda_model[bow_vector], key=lambda tup: -1*tup[1]):
print "Score: {}\t Topic: {}".format(score, lda_model.print_topic(index, 5))
print "Log perplexity of the model is", lda_model.log_perplexity(bag_of_words_corpus)