Combat: Python framework Scrapy distributed crawler will learn to create a search engine

• Chapter 1 Introduction Course

  Introduction course objectives, through the course content can be learned, and the development of the former system requires knowledge

  •  1-1 python to create a distributed crawler search engine About Look

• Development environment set up under Chapter 2 windows

  Introduction of project development need to install the software developer, python and virtual virtualenv virtualenvwrapper to install and use, and finally describes the simple use of pycharm and navicat

  •  2-1 pycharm simple to install and use
  •  Installation and use of 2-2 mysql and navicat
  •  Installation python2 and python3 at 2-3 windows and linux
  •  2-4 installation and configuration of the virtual environment

• Chapter 3 Basics reptile Review

  Introduces the basics of reptiles need to use the development of reptiles, including what to do, regular expressions, depth-first and breadth-first algorithm and implementation, reptiles url deduplication strategy to completely clear the difference between unicode and utf8 coding and applications.

  •  What do reptiles technology selection 3-1
  •  3-2 regex -1
  •  3-3 regex -2
  •  3-4 regular expression -3
  •  3-5 depth-first and breadth-first principle
  •  3-6 url deduplication method
  •  3-7 and completely clear unicode utf8 encoding

• Chapter 4 scrapy website crawling well-known technical articles

  Build scrapy development environment This chapter describes the commonly used commands and project directory structure scrapy analysis, this chapter will explain in detail the use of xpath and css selector. Then complete all the articles by crawling spider scrapy provided. After the item and then explain in detail Loader item extraction is accomplished using a specific field pipeline scrapy separately provided to save the data file and json mysql database. ...

  •  4-1 Solution article on the website can not be accessed (Note Before learning of this chapter) 
  •  4-2 scrapy installation directory structure and presentation
  •  4-3 pycharm debugging scrapy execution process
  •  4-4 xpath usage --1
  •  4-5 xpath usage - 2
  •  4-6 xpath usage --3
  •  4-7 css Selector Implementation field parsing --1
  •  Field Selector Implementation 4-8 css parsing - 2
  •  4-9 all articles written spider crawling jobbole of 1 -
  •  4-10 write spider crawling jobbole of all articles - 2
  •  4-11 items designed --1
  •  4-12 items Design - 2
  •  4-13 items designed --3
  •  Table 4-14 Data Design and save the item to json file
  •  4-15 by pipeline to save data mysql - 1
  •  4-16 by pipeline to save data mysql - 2
  •  4-17 scrapy item loader mechanism --1
  •  4-18 scrapy item loader mechanism - 2