Writing crawler programs using Scala and Sttp libraries

The following is a video crawler program written using Scala and Sttp library. The program uses proxy to obtain the IP. Note that this example needs to find a specific video link on and then pass it to crawlVideothe function.

import scala.util.{Failure, Success}
import scala.concurrent.{Future, ExecutionContext}
import sttp.client3._
​
object FacebookCrawler {
​
  def main(args: Array[String]): Unit = {
    val proxyUrl = ""
    val facebookUrl = ""
    val videoUrl = "your_video_url_here" // 请将此处更改为你要爬虫的视频链接
​
    val sttpBackend = new BlockingSttpBackend(executionContext)
    val client = new SttpClient(sttpBackend)
​
    val proxyResponse: Future[Either[String, String]] = client.send(get(proxyUrl)).map(_.body)
    val videoResponse: Future[Either[String, String]] = client.send(get(videoUrl).header("Referer", facebookUrl)).map(_.body)
​
    for {
      proxy <- proxyResponse
      video <- videoResponse
    } yield {
      println("IP: " + proxy)
      println("视频内容: " + video)
    }
  }
​
  def getProxy(client: SttpClient[Future, Nothing], executionContext: ExecutionContext): Future[Either[String, String]] = {
    client.send(get("")).map(_.body)
  }
​
  def crawlVideo(client: SttpClient[Future, Nothing], executionContext: ExecutionContext, videoUrl: String): Future[Either[String, String]] = {
    client.send(get(videoUrl).header("Referer", "")).map(_.body)
  }
}

In this example, we first obtain an IP address and then use this IP address to send a request to get the video. You will need to adjust the code according to your specific needs. In actual applications, you may need to handle errors, add exception handling, add logs, etc.