What is the difference between the web page source code and the code opened in the developer tools? How should we choose when crawling web pages?
I hope you won’t waste your time reading this article of mine and can learn something and grow from it! ! ! Chong Chong Chong (ง •_•)ง
Everyone has different places where they have doubts. Some people are creating web pages, and some are crawling web information. One carrot and another is a pit. Today I will analyze the situation that occurs when web crawling and deal with it.
Let’s briefly introduce these two terms.
Web page source code:
Web page source code is the composition of the code of a web page. When we open a web page, the
frame source code
can be said to be the web page source code, but the rendering effect is different, and the information inside is relatively more than the web page source code.
Developer tools:
It is the result of rendering all source codes. The web page source code and framework code are combined and rendered to us. The structure inside is relatively clear.
The source code of the web page is the code of the parent web page.
There is a node in the web page called iframe, which is a sub-frame, which is equivalent to the sub-page of the web page. Its structure is exactly the same as that of the external web page. The frame source code is the source code of this sub-web page. .
What our crawler gets here is the web page source code. Sometimes when we use requests to extract the web page source code, we find that the data we need to extract is not included. Don’t panic. The information you need is in the frame source code of the web page. This can be considered With a one-minute anti-crawling mechanism, we can crawl the framework source code.
You can right-click the framework source code to view it, and then look for the corresponding URL
, or you can use developer tools to capture and process the package.