The json crawler gets the list data incomplete and has been resolved

The original link data is relatively large, but the data obtained through jsoup is almost only more than 2000.

 Document document = Jsoup.connect(url)
         .timeout(4000)
         .ignoreContentType(true)
         .userAgent("Mozilla\" to \"Mozilla/5.0 (Windows NT 10.0; WOW64; rv:50.0)")
         .method(Connection.Method.POST)
         .get();

There are two possible causes.
①Set the timeout timeout limit larger, in milliseconds, 10s should be long enough.
② In addition, maxBodySize(0)set it to 0 to get data with unlimited response length.

 Document document = Jsoup.connect(url)
         .timeout(10000)
         .ignoreContentType(true)
         .userAgent("Mozilla\" to \"Mozilla/5.0 (Windows NT 10.0; WOW64; rv:50.0)")
         .method(Connection.Method.POST)
         .maxBodySize(0)
         .get();

Guess you like

Origin blog.csdn.net/qq_41885819/article/details/111017207