I want to introduce the meaning of inclusion and ranking first, and then introduce the role of robots protocol.
I wonder if you have noticed the following situations. When you want to know something, you may first Baidu it to see if you have the information you need.
For example, search for "Where is Shanghai the most fun on Baidu?
You can see that the first few places are basically advertising spots. This involves Baidu's bidding rankings and natural rankings. I will share them in my blog in the future.
But turn to the back , You can see that there are no “ads” tails behind these later articles. These articles are the posts that are included by Baidu and have rankings on the Baidu homepage. This leads to today’s question-what is inclusion, ranking, and robots What does the agreement have to do?
There are countless articles created on the Internet every day, so there will be many websites that publish such articles about "Where is Shanghai the most fun", especially some tourism websites may have more. So why do we search for "Where is Shanghai the most fun" on Baidu. Except for those posts with advertising spots, these posts without Baidu advertising spots will appear on the homepage? In fact, these posts are included by Baidu, and only after they are included can they be ranked on the homepage by Baidu.
For example, it's like selecting an outstanding class leader of the year in a class. The prerequisite is that you must be a class leader, you can be the class leader or the party secretary, etc., so that you have the prerequisite to be selected as an outstanding class leader. It can be expressed in terms of the included terms that these are the class leaders who were included. Then, with the status of class leader, through the voting of class students and teachers, the two recognized as outstanding class leaders were selected. In terms of ranking, it can be said that these two students will become outstanding class leaders if they have a ranking. Those who are class leaders but have not been rated as outstanding class leaders may still become outstanding class leaders in the next year, that is, they will have a ranking. Students who are not class leaders may also become class leaders the next year, that is, be included.
Therefore, through examples, we can know that there is not necessarily a ranking if there is a ranking, and a ranking will definitely be included by Baidu, and the ranking is not fixed, and the inclusion is not fixed.
Here is an interspersed introduction on how to check whether your posts are included. Open Baidu and enter the URL in the Baidu search box. If it can be found, it has been included, otherwise it has not been included. A small number of posts can be checked in this way. If the data is large, you must use tools. There are many such tools on the Internet. You can search for some by yourself. The ranking can also be checked manually or with tools.
Finally, let's talk about the robots protocol. Baidu is a big spider. It crawls a large number of websites and posts every day, and collects posts that it finds good. It selects high-quality posts from the included posts and displays them to users on the Baidu homepage. Users can be interested after searching. Click to browse through the posts. So what content does Baidu crawl when it crawls the site? The robots protocol is the protocol written by the website, which stipulates what content Baidu can and cannot crawl on the website. Of course, this is only an agreement. Baidu does not fully comply with it, but it basically complies with it.
For example, check Taobao's robots agreement. It stipulates that Baidu cannot crawl, but you can still search Taobao on Baidu. As the largest Chinese search engine in China, Baidu is not convinced if it can't even search Taobao. Users will feel that it can't even search Taobao, so Baidu does not fully comply with the robots agreement.
User-agent: Baiduspider
Disallow: /
User-agent: baiduspider
Disallow: /
It can be seen from Taobao's robots agreement that it can basically be summarized into three words, namely User-Agent, Allow, and Disallow. And a symbol slash (/).
Let me introduce what they mean.
User-Agent: Represents a robot
Such as User-Agent: Baiduspider means Baidu spider robot.
Allow: Indicates the pages that the robot is allowed to visit.
Allow: / means that the entire website is allowed.
Disallow: Indicates pages that robots are not allowed to access.
Disallow: / means to block the entire website
The following is an interception of a part of the robots agreement of Baixing.com. We can analyze the specific meaning together.
User-Agent: Mediapartners-Google #用户：谷歌机器人
Allow: / #允许谷歌爬取百姓网
User-Agent: AdsBot-Google #用户：谷歌机器人
Allow: / #允许谷歌爬取百姓网
User-agent: Yahoo! Slurp China # 用户雅虎机器人
Disallow: / #不允许雅虎爬取百姓网
User-Agent: * #允许所有的搜索引擎可以按照以下限制语法进行合理的抓取网站中的文件、目录。
Disallow: /*?*#禁止访问网站中所有包含问号 (?) 的网址。
Disallow: /*%3F*#禁止访问网站中所有包含%3F的网站
Disallow: /autocomplete/ #禁止访问此目录以及其中的所有内容
Disallow: /arch/ #禁止访问此目录以及其中的所有内容
Disallow: /*/t*.html #禁止访问此网站中所有文件夹下以t字母开始命名的html文件
Disallow: /gifts/murmur#禁止访问murmur目录下的所有文件
This is how robots are used. Set rules as needed to allow crawlers to crawl more effective content on the website, thereby increasing the number of posts on the website.
Summary: Write the robots agreement first, and the spider crawls the website. Only then can the posts in the website be included, and then it is possible to rank on the Baidu homepage. Users can click on keywords to see your posts on the homepage, and then click With traffic, users can continue to have follow-up customer acquisition, conversion rate, retention rate, etc. through clicking.