Laravel command-line tool as much as thread synchronization large quantities of data DB connection confusion Solutions

Multi-process synchronization in mind a high-volume data

Background: Since the company's user ID was incomplete and therefore needs data from the Group of sync once the user mark, the user data source is a micro letter, of the order of 1.5 million, the group users of the order of six million

Group finalized the program is off to a query interface, the frequency of concurrent access is not limited, in that order of magnitude of it, we traverse the company's users to query the identity of the user to update

Project uses laravel, I wrote a command-line script, open 15 process to run

 

Because of the time using PHP's multi-process pcntl_fork

The core code is as follows:

The main processes are:

First, the main process allocation userid to the respective sub-processes, is used here redis queues to store, then fork a child process, pop queue acquired assigned userid range of the child, and then check the library assembly information request group the API, designated user based on the return value .

Are being given because Laravel package redis and DB uses a singleton, all multiple processes much money and redis DB:

redis：Error while reading line from the server

mysql:Packets out of order. Expected 1 received 19

Ah, it is multi-process reads a single case caused by the same connection, the solution:

redis: Use of native PHP redis connection:

        $redis = new \Redis();
        $re1 = $redis->connect(env('REDIS_HOST'),env('REDIS_PORT'));
        $re2 = $redis->auth(env('REDIS_PASSWORD'));

　　

mysql: Use laravel the purge and reconnect reconnetc

            DB::purge('mysql');
            DB::reconnect('mysql');

　　

 

多线程大批量数据问题解决^_^