Today, still learning the webmagic reptile, found yesterday not to crawl URLs, content is not right, re-find a web site crawling, rearrange ideas and found this site, divided into three types of links, advice, counseling and what remember, you need to determine the type, then you can assign Id, then add url queue.
A major difficulty in this web site, is a POST paging is passed by value, and then dynamically modify the content of the page, so today re-learn, and acquire content through POST, benefits greatly, busy day, forget to write a blog, late so write so it.
Attach additional learning website, speak well
The key code is:
Request request = new Request("");
request.setMethod(HttpConstant.Method.POST);
request.setRequestBody (HttpRequestBody.json ( "{ 'ID':. 1}", "UTF-.
8")); ----------------
Copyright: This is CSDN blogger "simple." original article, follow the CC 4.0 BY-SA copyright agreement, reproduced, please attach the original source link and this statement.
Original link: https: //blog.csdn.net/weixin_42244754/article/details/81566590