English
English
In web development, cURL is a very commonly used tool that enables communication with servers through various protocols such as HTTP and FTP. In PHP, the cURL extension is widely used to retrieve remote page content, submit form data, and more. Usually, cURL defaults to using PHP’s own User-Agent, but when simulating browser requests, we need to disguise the request as coming from a browser, and one of the most common ways is to set a custom User-Agent.
User-Agent is an HTTP header sent by the client (usually a browser) when making a request to the server. It is typically used to identify the source of the request and details about the client, including browser type, operating system version, and more. Through the User-Agent, the server can return different content based on the client device and browser.
For example, when a browser accesses a webpage, the HTTP request header contains a User-Agent field similar to the following:
<span><span><span class="hljs-keyword">User</span></span><span><span class="hljs-operator">-</span></span><span>Agent: Mozilla</span><span><span class="hljs-operator">/</span></span><span><span class="hljs-number">5.0</span></span><span> (Windows NT </span><span><span class="hljs-number">10.0</span></span><span>; Win64; x64) AppleWebKit</span><span><span class="hljs-operator">/</span></span><span><span class="hljs-number">537.36</span></span><span> (KHTML, </span><span><span class="hljs-keyword">like</span></span><span> Gecko) Chrome</span><span><span class="hljs-operator">/</span></span><span><span class="hljs-number">91.0</span></span><span><span class="hljs-number">.4472</span></span><span><span class="hljs-number">.124</span></span><span> Safari</span><span><span class="hljs-operator">/</span></span><span><span class="hljs-number">537.36</span></span><span>
</span></span>When simulating browser requests using the cURL extension, you can set a custom User-Agent using the CURLOPT_USERAGENT option in the curl_setopt() function.
In the example below, we use curl_setopt() to set the User-Agent, simulating a request from the Chrome browser:
<span><span><span class="hljs-title function_ invoke__">curl_setopt</span></span><span>(</span><span><span class="hljs-variable">$ch</span></span><span>, CURLOPT_USERAGENT, </span><span><span class="hljs-string">"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36"</span></span><span>);
</span></span>The User-Agent string in the code above simulates a request from Chrome running on a Windows operating system.
The following PHP code demonstrates how to set a User-Agent in a cURL request to simulate a browser request:
<span><span><span class="hljs-variable">$ch</span></span><span> = </span><span><span class="hljs-title function_ invoke__">curl_init</span></span><span>();
</span><span><span class="hljs-title function_ invoke__">curl_setopt</span></span><span>(</span><span><span class="hljs-variable">$ch</span></span><span>, CURLOPT_URL, </span><span><span class="hljs-string">"https://www.example.com"</span></span><span>); </span><span><span class="hljs-comment">// Target URL</span></span><span>
</span><span><span class="hljs-title function_ invoke__">curl_setopt</span></span><span>(</span><span><span class="hljs-variable">$ch</span></span><span>, CURLOPT_USERAGENT, </span><span><span class="hljs-string">"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36"</span></span><span>); </span><span><span class="hljs-comment">// Simulate browser User-Agent</span></span><span>
</span><span><span class="hljs-title function_ invoke__">curl_setopt</span></span><span>(</span><span><span class="hljs-variable">$ch</span></span><span>, CURLOPT_RETURNTRANSFER, </span><span><span class="hljs-literal">true</span></span><span>); </span><span><span class="hljs-comment">// Return response content instead of outputting directly</span></span><span>
<p></span>$response = curl_exec($ch); // Execute request<br>
curl_close($ch); // Close cURL session</p>
<p>echo $response; // Output response content<br>
</span>To improve the simulation effect of crawlers, you may need to set more complex User-Agent strings or even randomly switch between multiple User-Agent values. Below are examples of User-Agent strings for some common browsers:
Google Chrome:
<span><span><span class="hljs-type">Mozilla</span></span><span><span class="hljs-regexp">/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/</span></span><span><span class="hljs-number">537.36</span></span><span> (</span><span><span class="hljs-type">KHTML</span></span><span>, like </span><span><span class="hljs-type">Gecko</span></span><span>) </span><span><span class="hljs-type">Chrome</span></span><span><span class="hljs-regexp">/91.0.4472.124 Safari/</span></span><span><span class="hljs-number">537.36</span></span><span>
</span></span>Mozilla Firefox:
<span><span><span class="hljs-type">Mozilla</span></span><span><span class="hljs-regexp">/5.0 (Windows NT 10.0; Win64; x64) Gecko/</span></span><span><span class="hljs-number">20100101</span></span><span> </span><span><span class="hljs-type">Firefox</span></span><span><span class="hljs-operator">/</span></span><span><span class="hljs-number">89.0</span></span><span>
</span></span>Safari (Mac):
<span><span><span class="hljs-type">Mozilla</span></span><span><span class="hljs-regexp">/5.0 (Macintosh; Intel Mac OS X 10_15_4) AppleWebKit/</span></span><span><span class="hljs-number">537.36</span></span><span> (</span><span><span class="hljs-type">KHTML</span></span><span>, like </span><span><span class="hljs-type">Gecko</span></span><span>) </span><span><span class="hljs-type">Version</span></span><span><span class="hljs-regexp">/13.1 Safari/</span></span><span><span class="hljs-number">537.36</span></span><span>
</span></span>You can modify the User-Agent string according to your needs to make the request appear as if it comes from different devices or browsers.
With PHP’s cURL extension, we can easily simulate browser requests by setting custom User-Agent headers to disguise the request source. This is very useful for web scraping, API requests, and other operations, helping to bypass simple anti-crawling mechanisms and avoid being detected as a bot by using the default User-Agent.
By using the methods above, you can flexibly set and adjust User-Agent strings to simulate various browsers or devices to obtain webpage content.
