NLP natural language processing data preparation and application practical python, detailed tutorial
In the field of NLP, the quality and preparation of data is critical to the performance of the model. Therefore, before performing NLP tasks, we must collect and prepare corresponding data. This article will show how to collect and prepare data, and use Python to build and train a simple language model.
- Collect corpus
First, we need to choose a suitable corpus to train our language model. Here, we choose the relatively common and open source Chinese Wikipedia as our corpus, you can download it with the following code:
!wget https://dumps.wikimedia.org/zhwiki/latest/zhwiki-latest-pages-articles.xml.bz2
- data cleaning
After obtaining the original corpus, we need to clean it to remove unnecessary labels, symbols and other useless information. The following is a simple code