English
English
With the explosive growth of information on the internet, quickly and accurately extracting target data from a large number of web pages has become a key concern for developers. PHP, as a widely used backend language, combined with the phpSpider crawler framework, can simplify the process of web data collection and improve efficiency.
This article will guide you through installing phpSpider, writing crawler scripts, and demonstrate how to locate and extract key information from web pages with practical examples.
phpSpider is an open-source crawler framework based on PHP and is easy to install. Simply run the following command via Composer:
<span class="fun">composer require php-spider/phpspider</span>
After installation, create a file named spider.php, include the autoloader, and instantiate a crawler object:
<?php
require 'vendor/autoload.php';
<p>use phpspider\core\phpspider;</p>
<p>// Create crawler instance<br>
$spider = new phpspider();</p>
<p>// Set the starting URL for the crawler<br>
$spider->add_start_url('<a rel="noopener" target="_new" class="" href="http://www.example.com">http://www.example.com</a>');</p>
<p>// Define the callback function to extract page content<br>
$spider->on_extract_page = function ($page, $data) {<br>
// Write extraction logic here, using regex, XPath, or CSS selectors to extract data<br>
return $data;<br>
};</p>
<p>// Start the crawler<br>
$spider->start();<br>
Inside the callback function, use CSS selectors to quickly locate the page title and main content, for example:
$spider->on_extract_page = function ($page, $data) {
$title = $page['raw']['headers']['title'][0];
$content = $page['raw']['content'];
$data['title'] = $title;
$data['content'] = strip_tags($content);
return $data;
};
This accesses the raw page content to extract the title and plain text content, meeting basic data scraping needs.
Extracted data can be saved to files or databases. Here’s an example saving data into a text file:
$spider->on_extract_page = function ($page, $data) {
$title = $page['raw']['headers']['title'][0];
$content = $page['raw']['content'];
$data['content'] = strip_tags($content);
// Append data to a file
file_put_contents('extracted_data.txt', var_export($data, true), FILE_APPEND);
return $data;
};
After completing the code, execute the following command in the terminal:
<span class="fun">php spider.php</span>
The program will automatically crawl web pages starting from the specified URL and extract and save information according to the defined rules.
Using PHP and phpSpider, developers can quickly build powerful web crawlers to automate data extraction from massive numbers of web pages. With minimal code and simple configuration, you can precisely locate and extract target information, greatly improving data acquisition efficiency. phpSpider also supports many advanced features, suitable for customized development in various scenarios.
