The cURL library can simply and effectively grab web pages, you only need to run a script, then analyze the web pages you grab, and then you can get the data you want in a program. Whether you want to fetch partial data from a link, or fetch an XML file and import it into a database, or simply fetch web content, cURL is a powerful PHP library. This article mainly describes how to use this PHP library.
<?php
// 初始化一个cURL对象
$curl = curl_init();
// 设置您需要抓取的URL
curl_setopt($curl, CURLOPT_URL, 'http://www.example.com');
// 设置header
curl_setopt($curl, CURLOPT_HEADER, 1);
// 设置cURL参数，要求结果保存到字符串中还是输出到屏幕上
curl_setopt($curl, CURLOPT_RETURNTRANSFER, 1);
// 运行cURL，请求网页
$data = curl_exec($curl);
// 关闭URL请求
curl_close($curl);
// 显示获得的数据
var_dump($data);
?>
How to POST data The above is the code to grab the webpage, and the following is the POST data to a certain webpage. Suppose we have a URL http://www.111cn.cn/sendSMS.php that handles forms, which can accept two form fields, one is the phone number and the other is the text message content. The sample code is as follows:
<?php
$phoneNumber = '13912345678';
$message = 'This message was generated by curl and php';
$curlPost = 'pNUMBER=' . urlencode($phoneNumber) . '&MESSAGE=' . urlencode($message) . '&SUBMIT=Send';
$ch = curl_init();curl_setopt($ch, CURLOPT_URL, 'http://www.example.com/sendSMS.php');
curl_setopt($ch, CURLOPT_HEADER, 1);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, $curlPost);
$data = curl_exec();
curl_close($ch);
?>
From the above program, we can see that use CURLOPT_POST to set the POST method of the HTTP protocol instead of the GET method, and then set the POST data with CURLOPT_POSTFIELDS.
About proxy servers
Here is an example of how to use a proxy server, the code is very simple, so I don't need to say more:
<?php
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, 'http://www.example.com');
curl_setopt($ch, CURLOPT_HEADER, 1);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_HTTPPROXYTUNNEL, 1);
curl_setopt($ch, CURLOPT_PROXY, 'fakeproxy.com:1080');
curl_setopt($ch, CURLOPT_PROXYUSERPWD, 'user:password');
$data = curl_exec();
curl_close($ch);
?>
About SSL and Cookies
Regarding the SSL or HTTPS protocol, you only need to change the http:// in the CURLOPT_URL connection to https://. Of course, there is another parameter called CURLOPT_SSL_VERIFYHOST that can be set to verify the site.
Regarding cookies, you need to understand the following three parameters:
1. CURLOPT_COOKIE: Set a cookie in the face-to-face session
2. CURLOPT_COOKIEJAR: Save a cookie when the session ends
3. CURLOPT_COOKIEFILE: Cookie file.
HTTP server authentication
Finally, let's take a look at the HTTP server authentication situation:
<?php
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, 'http://www.example.com');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_HTTPAUTH, CURLAUTH_BASIC);
curl_setopt(CURLOPT_USERPWD, '[username]:[password]')
$data = curl_exec();
curl_close($ch);
?>