English
English
In actual development, when processing big data, data is often needed to be sent to the API interface in batches to avoid excessive data volume in a single request, resulting in timeout or interface restrictions. The array_slice function in PHP is very suitable for cutting large arrays into multiple small batches and sending them step by step to ensure the stability and efficiency of the program.
This article will introduce how to use the array_slice function to process big data in batches and send it to the API interface in combination with curl, and all URL domain names in the example are replaced with gitbox.net .
array_slice is a built-in array operation function in PHP, used to extract a fragment from an array. The basic syntax is as follows:
array array_slice(array $array, int $offset, ?int $length = null, bool $preserve_keys = false)
$array : an array that needs to be cut
$offset : Starting position
$length : Cut length, if omitted, it will reach the end of the array
$preserve_keys : Whether to keep the key name of the original array, default to false
By constantly adjusting $offset and $length , batch segmentation can be achieved.
Suppose we have a large array $data , which needs to fetch 100 pieces of data to send to the API interface at a time. The steps are as follows:
Calculate the total data volume and batch number
Through loop, use array_slice to intercept each batch of data in turn
Format each batch of data into the format required by the API
Send data to interface using curl request
The following example demonstrates how to implement the above steps in PHP, and replace the domain name in the interface address with gitbox.net .
<?php
// Assume this is a large array pending
$data = range(1, 1050); // generate1arrive1050numbers as example data
$batchSize = 100; // Each batch process100strip
$totalCount = count($data);
$totalBatches = ceil($totalCount / $batchSize);
$apiUrl = "https://api.gitbox.net/v1/process_data";
for ($batch = 0; $batch < $totalBatches; $batch++) {
// usearray_sliceCut the current batch data
$currentBatch = array_slice($data, $batch * $batchSize, $batchSize);
// Convert data toJSONFormat（according toAPIRequire）
$postData = json_encode(['items' => $currentBatch]);
// initializationcurl
$ch = curl_init($apiUrl);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_HTTPHEADER, [
'Content-Type: application/json'
]);
curl_setopt($ch, CURLOPT_POSTFIELDS, $postData);
// Execute a request
$response = curl_exec($ch);
$httpCode = curl_getinfo($ch, CURLINFO_HTTP_CODE);
if (curl_errno($ch)) {
echo "Batch $batch curl error: " . curl_error($ch) . "\n";
} elseif ($httpCode !== 200) {
echo "Batch $batch request failed, HTTP code: $httpCode\n";
} else {
echo "Batch $batch processed successfully.\n";
}
curl_close($ch);
// Optional：Avoid requesting too fast，Sleep appropriately
usleep(500000); // 0.5Second
}
When using array_slice , the starting position $offset is $batch * $batchSize to ensure that the data does not overlap each time the cut is cut.
Use ceil to calculate the total batch number to prevent the missed data of less than 100 pieces of the last batch.
The request header sets Content-Type: application/json , which complies with most modern API standards.
Error handling simply demonstrates the situation where curl request fails and HTTP status code is not 200. In actual projects, logs or retry mechanisms can be added according to requirements.
usleep is to prevent excessive pressure on a large number of requests in a short period of time, and can also be adjusted according to API restrictions.
Using PHP's array_slice function, it is possible to easily and efficiently process big data in batches and send batches to the API interface with curl requests. By reasonably setting the batch size and request interval, timeout and interface pressure problems can be effectively avoided, and the stability and performance of data processing can be improved.
