foreword
Hello everyone, today we are going to crawl the list of colleges and universities on a certain website, and get the list of colleges and universities, members and content numbers. However, we found that this website has one more verification than we usually do. Let's see how I solved it.
environment use
- python 3.9
- pycharm
module use
- requests
module introduction
- requests
requests is a very practical Python HTTP client library. It is often used when crawlers and test servers respond to data. requests is a third-party library in Python language, which is specially used to send HTTP requests. It is much simpler to use than urllib.
- parcel
parsel is a third-party library of python, which is equivalent to css selector + xpa