php queue uses php-resque (1)
php queue uses php-resque (2)
php queue uses php-resque (3) - by supervisor
-- general description --
the following is an online Chinese The description of
www.fzb.me/2015-7-31-beanstalkd-faq.html
is divided into our common server and client.
The server side is beanstalkd, which is used by many people. In fact, it is more appropriate to call it a message queue.
It is different from the php-resque we introduced earlier:
1. php-resque does not distinguish between obvious client and server, but beanstalkd distinguishes it, which is more in line with our common way of thinking.
2. php-resque only needs to install redis and run it, while beanstalkd is a server written by itself and has nothing to do with other software and does not need it.
3. php-resque cannot be persisted, while beanstalkd can, but it is better not to use it because it is too slow.
4. Both php-resque and beanstalkd are lightweight and easy to use.
5. The most important php-resque cannot use the delay queue (or needs to be supplemented by other components), but beanstalkd can.
6. php-resque has not been updated for a long time, and beanstalkd was last updated in composer in 2015, which is quite a lot.
7. php-resque has no priority, while beanstalkd has priority, but I don't use it either.
8. In order to prevent a consumer from occupying the task for a long time but cannot process it, Beanstalkd sets a timeout time for the reserve operation. If the consumer cannot complete the job within the specified time, the job will be migrated back to the READY state for other consumers to execute.
Introducing the delay queue, I can put a message, but hope that the message will actually enter the queue at a certain point in time, and thus be taken out.
In a normal queue, if you put a message into the queue, you cannot control it. As long as there is no other message in front of it, it will be taken out immediately.
The client can choose multiple languages, and php also has multiple class libraries, but the most downloaded from the composer repository is pda/pheanstalk
========================= =====
Server description:
1. A message has four states: READY, RESERVED, DELAYED, and BURIED.
2. When the producer puts a message directly, the message is in the READY state.
3. If you choose to delay the put, the message will arrive first. DELAYED state, wait until the time elapses before migrating to READY state. The task status will change from READY to RESERVED (scheduled), and others will not be able to get it. When PUT generates a message, it carries ttr (time to run). If the consumer does not send a delete, release or buried command within this time. The task will automatically return to the READY state, and others can continue to obtain it. (Note: if the consumer returns the release command or does not return, it will return to the READY/DELAYED state and can be re-consumed!! This may not be what we want)
4. If the current READY message is obtained, the state of the message will be Migrate to RESERVED so that no other process can manipulate the message. Make sure to use only.
5. BURIED This interface can be used to prevent a message from being called multiple times.
6. The program has a delete operation. After a message is deleted, it disappears completely. It does not belong to any of the above 4 cases. In general, it should delete the message after it is obtained, and it must not be omitted. .
Beanstalkd's job status is diversified, supports task priority (priority), delay (delay), timeout retransmission (time-to-run) and reserved (buried), can well support distributed background tasks and timing task processing.
Its internal implementation uses libevent, and a lightweight communication protocol similar to memcached is used between the server and the client, so it has high performance.
Regarding the memory used by the server, it only depends on the size of the queue and is not limited.
Disadvantages: There is no maximum memory control. If there are messages piled up or the business usage is wrong, the memory surge will bring down the machine
--server installation--
complete installation steps, it must be a Unix-like system.
yum install beanstalkd
service beanstalkd start
The above writing method will read the following configuration files.
Another way to start the configuration file /etc/sysconfig/beanstalkd
is to start directly and set the parameters yourself.
When -b is enabled to start persistence, it will flush the binlog to the hard disk every N seconds (N can be configured to 0), similar to The AOF persistence method of redis is generally not needed.
-- Server monitoring --
there are many
https://github.com/kr/beanstalkd/wiki/Tools
I picked the first download. Extremely easy to
use , no installation required.
Just unzip the file and execute the command
./beanstalkd-cli stats
or
./beanstalkd-cli monitor
but it is not very intuitive.
-- A php web monitoring console for server monitoring, great--
how to use, first, the computer must follow composer
and then,
composer create-project ptrofimov/beanstalk_console -s dev target directory
This target directory is usually located in the web root directory . The console is successfully installed with one command.
Then,
open the browser
http://domain name/target directory/public/index.php and
the magic console will appear.
php is great!
-- client installation --
composer install
"pda/pheanstalk": "3.1.0"
-- client code --
for simplicity, write it in a file, the real project must be separated
<?php
namespace app\test\controller;
use Pheanstalk\Pheanstalk;
/**
* This is the facade program.
* Before use, the beanstalkd service needs to be started locally.
* I use the framework to execute, if there is no framework, I need to write two php files.
*
* @author Administrator
*
*/
class Beanstalkd
{
/**
* This is adding a message to the queue.
*/
public function add() {
$beanstalkd = new Pheanstalk('127.0.0.1', '11300');
//This is the message data, in this demo, type cannot be omitted. Distinguish task types.
$data = array(
'type' => mt_rand(1,2),
'mobile' => '13051662435',
'id' => mt_rand(100,200),
'time' => date("Y-m-d H:i:s"),
);
$delay = (int) strtotime($data['time']) - time();
$delay=0;
// Put the message into the queue, 1024 is the priority
//$delay is very important, assuming 10 is set, the message will not be put into the queue until 10 seconds later! Very useful.
$beanstalkd->useTube( Worker::queue_name )
->put(serialize($data), 1024, $delay);
}
/**
* This is the running queue manager.
*/
public function run()
{
$worker = new Worker();
$worker->run();
}
}
/**
* This is the real task execution 1
* @author Administrator
*
*/
class WorkerJob1
{
public function excute($data)
{
echo "---\n";
echo ('job:type1:'.print_r($data, 1));
echo "---\n";
}
}
/**
* This is the real task execution 2
* @author Administrator
*
*/
class WorkerJob2
{
public function excute($data)
{
echo "---\n";
echo ('job:type2:'.print_r($data, 1));
echo "---\n";
}
}
/**
* This is the queue manager daemon process, it is best to use supervisord to support it.
And it should be executed in the background of linux
* @author Administrator
*
*/
class Worker {
private $job1; //task object
private $job2; //this is the task object
private $pheanstalk;//This is the service
// Set the queue name. start at will
const queue_name ="queue1";
public function __construct() {
$this->log('starting');
$this->pheanstalk = new Pheanstalk('127.0.0.1:11300');
$this->job1 = new WorkerJob1();
$this->job2 = new WorkerJob2();
}
public function excute($data)
{
if ($data['type']==1) {
$this->job1->excute($data);
}elseif ($data['type']==2) {
$this->job2->excute($data);
}
}
public function run() {
$this->log('starting to run');
while(true) {
$job = $this->pheanstalk->watch( self::queue_name )->ignore('default')->reserve();
$job_data = unserialize( $job->getData());
// This is what really works.
$this->excute($job_data);
//Do not forget to delete.
$this->pheanstalk->delete($job);
$memory = memory_get_usage();
// This is an insurance measure. can also be removed.
if($memory > 100000000) {
$this->log('exiting run due to memory limit');
exit;
}
}
}
private function log($txt) {
echo "worker: ".$txt."\n";
}
}
-- browser output--
