504 Gateway error on huge sql query in php

Kaleem Qasim :

Hello I've a project in which I'm storing some data from xml file into my database, which the data is so huge that it takes some time, few data is inserted like 30thousands and then I get 504 gateway timeout. I've tried increasing max_execution_time from php.ini but that didnt work. Im using cpanel

myscript

$xml=simplexml_load_file($url) or die("Error: Cannot create object");
        $totalproducts = $xml->shop->offers->offer->count();
        for($x=0 ; $x < $totalproducts; $x++){
            if(!empty($xml->shop->offers->offer[$x])){
                $name = $xml->shop->offers->offer[$x]->name;
                $price = $xml->shop->offers->offer[$x]->price;
                $description = $xml->shop->offers->offer[$x]->description;
                // $description = preg_replace('#[^a-zA-Z0-9 ]#', "", $description);
                $description = str_replace('"', "", $description);
                $description = str_replace("'", "", $description);
                // $description = mysqli_real_escape_string($description);
                $imglink = $xml->shop->offers->offer[$x]->picture;
                $url = $xml->shop->offers->offer[$x]->url;
                $catname = 'shoes';
                $program = $_POST['programname'];
                $sql = "INSERT into products(categoryname,productname,productdiscription,price,url,program,image) VALUES('$catname','$name','$description','$price','$url','$program', '$imglink')";
                if ($conn->query($sql) === TRUE) {
                    // echo "New record created successfully";
                } else {
                    // echo "Error: " . $sql . "<br>" . $conn->error;
                }
            }
        }
FarZan Takhsha :

I think problem comes from "simplexml_load_file()" function. probably loading xml takes much time. If you sure about the function, I add a few code line to prevent timeout, but not decreasing time:

<?php
if (isset($_GET['offset'])){$offset=$_GET['offset'];}
else {$offset=0;}

$xml=simplexml_load_file($url) or die("Error: Cannot create object");
        $totalproducts = $xml->shop->offers->offer->count();
        for($x=$offset ; ($x < $totalproducts && ($x < (($offset+1)*100)); $x++){
            if(!empty($xml->shop->offers->offer[$x])){
                $name = $xml->shop->offers->offer[$x]->name;
                $price = $xml->shop->offers->offer[$x]->price;
                $description = $xml->shop->offers->offer[$x]->description;
                // $description = preg_replace('#[^a-zA-Z0-9 ]#', "", $description);
                $description = str_replace('"', "", $description);
                $description = str_replace("'", "", $description);
                // $description = mysqli_real_escape_string($description);
                $imglink = $xml->shop->offers->offer[$x]->picture;
                $url = $xml->shop->offers->offer[$x]->url;
                $catname = 'shoes';
                $program = $_POST['programname'];
                $sql = "INSERT into products(categoryname,productname,productdiscription,price,url,program,image) VALUES('$catname','$name','$description','$price','$url','$program', '$imglink')";
                if ($conn->query($sql) === TRUE) {
                    // echo "New record created successfully";
                } else {
                    // echo "Error: " . $sql . "<br>" . $conn->error;
                }
            }
        }
Header("Location: yourpage.php?offset=".($offset+100));
echo ($offset+100).' items inserted';
?>

This splits inserting to 100 items by 100 items. you can change 100 to more items at each time.

Guess you like

Origin http://43.154.161.224:23101/article/api/json?id=342660&siteId=1