1. Problem
The customer conducts stress tests on distributed tasks and finds that there will be repeated assignments when assigning tasks.
2. Analysis
1. The client application is developed based on our framework. The distributed tasks use the framework-integrated Quartz for task scheduling. The client application test environment is deployed in clusters of multiple servers. Therefore, Quartz is required to be deployed in clusters, otherwise the Quartz jobs under multiple servers will be There is a problem of order grabbing, so first, verify whether the Quartz cluster configuration is correct and effective;
2. Check the Job task to see if there is an order grabbing situation inside the Job. After analysis, it is found that the Job interface currently implemented by AbstractQuartzService has a potential risk. When the scheduled Job is not executed this time, it will be started again next time. When the job is called, there will be task competition;
3. The customer application is based on our development framework. The startup of the distributed task is started through the quartz plug-in. When testing, it is necessary to simulate the startup process of the production environment for testing.
3. Verify the Quartz cluster process
3.1 , Quartz configuration
3.1.1 quartz_client.properties
#============================================================================
# Configure Main Scheduler Properties
#============================================================================
org.quartz.scheduler.instanceName = LmsScheduler
org.quartz.scheduler.instanceId = AUTO
#============================================================================
# Configure ThreadPool
#============================================================================
org.quartz.threadPool.class = org.quartz.simpl.SimpleThreadPool
org.quartz.threadPool.threadCount = 3
org.quartz.threadPool.threadPriority = 5
#============================================================================
# Configure JobStore
#============================================================================
org.quartz.jobStore.misfireThreshold = 60000
# org.quartz.jobStore.class = org.quartz.simpl.RAMJobStore
org.quartz.jobStore.class = org.quartz.impl.jdbcjobstore.JobStoreTX
org.quartz.jobStore.driverDelegateClass = org.quartz.impl.jdbcjobstore.oracle.OracleDelegate
org.quartz.jobStore.useProperties = false
org.quartz.jobStore.dataSource = myDS
org.quartz.jobStore.tablePrefix = SYS_TA_
org.quartz.jobStore.isClustered = true
#============================================================================
# Configure Datasources
#============================================================================
org.quartz.dataSource.myDS.driver = oracle.jdbc.driver.OracleDriver
org.quartz.dataSource.myDS.URL = jdbc:oracle:thin:@localhost:1521:orcl
org.quartz.dataSource.myDS.user = test
org.quartz.dataSource.myDS.password = test
org.quartz.dataSource.myDS.maxConnections = 5
#============================================================================
# Configure Plugins
#============================================================================
org.quartz.plugin.jobInitializer.class = org.quartz.plugins.xml.JobInitializationPlugin
org.quartz.plugin.jobInitializer.fileNames = conf/schedule/quartz_jobs.xml
org.quartz.plugin.jobInitializer.overWriteExistingJobs = true
org.quartz.plugin.jobInitializer.failOnFileNotFound = true
org.quartz.plugin.jobInitializer.scanInterval = 0
org.quartz.plugin.jobInitializer.wrapInUserTransaction = false
illustrate:
1. Pay attention to the content marked in red;
2. Quartz cluster configuration, the configuration of the cluster is required to be placed in a shared database, the example is to use oracle , and the database configuration table corresponding to quartz needs to be created in advance;
3. Due to the quartz-1.6.4 version integrated by the framework , the following configuration has no corresponding implementation class, and the test is upgraded to quartz-1.7.1 version
org.quartz.jobStore.driverDelegateClass = org.quartz.impl.jdbcjobstore.oracle.OracleDelegate
3.1.2 quartz_jobs
<?xml version='1.0' encoding='utf-8'?>
<quartz xmlns="http://www.opensymphony.com/quartz/JobSchedulingData"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.opensymphony.com/quartz/JobSchedulingData
http://www.opensymphony.com/quartz/xml/job_scheduling_data_1_5.xsd"
version="1.5">
<job>
<job-detail>
<name>simpleJob</name>
<group>simpleJob</group>
<description>test_dispatcher</description>
<job-class>com.itown.dtask.service.SimpleQuartJob</job-class>
<volatility>false</volatility>
<durability>false</durability>
<recover>false</recover>
</job-detail>
<trigger>
<simple>
<name>simpleJob</name>
<group>simpleJob</group>
<job-name>simpleJob</job-name>
<job-group>simpleJob</job-group>
<repeat-count>-1</repeat-count>
<repeat-interval>10000</repeat-interval>
</simple>
</trigger>
</job>
</quartz>
3.2 , write test class
3.2.1 Job implementation class
public class SimpleQuartJob implements StatefulJob{
public void execute(JobExecutionContext jec) throws JobExecutionException {
System.out.println("SimpleQuartJob is running when " + new Date());
try {
Thread.sleep(15000);
} catch (InterruptedException ex) {
System.out.println(ex.getMessage());
}
}
}
Description: This class implements the StatefulJob interface.
3.2.2 QuartzPluginStarter
public class QuartzPluginStarter {
private static Scheduler scheduler = null;
private static final Logger logger = Logger.getLogger(QuartzPlugin.class.getName());
public static void main(String[] args){
try {
// read from the path of the configuration file
String fileName = "quartz_client.properties";
File file = new File(RunModeManager.getInstance().getWorkPath(), "conf" + File.separator + "schedule" + File.separator + fileName);
logger.info(" Load configuration file , file=" + file.toString());
if (!file.exists()) {
logger.warning(String.format("The scheduler configuration file [%s] does not exist. ", file));
return;
}
SchedulerFactory sf = new StdSchedulerFactory(file.getAbsolutePath());
scheduler = sf.getScheduler();
scheduler.start();
logger.info("SchedulerFactory started");
} catch (SchedulerException ex) {
logger.log(Level.SEVERE, null, ex);
}
}
}
Description: This class simulates the implementation of QuartzPlugin , which is the same as the mechanism for starting quartz in the production environment .
3.3 , cluster test
1. Start a QuartzPluginStarter ( A ), observe the execution of the Job , and print a record each time it is executed normally;
2. Start another QuartzPluginStarter ( B ), start multiple Quartz in a simulated cluster environment , and observe the execution of the two programs A and B at the same time. After the cluster configuration is correct, the normal result is that only one of A or B is executing;
3. If B executes, A waits, then stops B and observes whether A starts to execute. If it starts to execute, it verifies the high availability of the Quartz cluster.
Test conclusion: The verification results are as expected. In the Quartz cluster environment, only one Quartz node takes effect. When there is a problem with the enabled Quartz , the Quartz cluster will start other Quartz nodes to achieve high availability.
4. Follow-up work
1. Check and confirm whether the Quartz configuration of the customer application is correct;
2. Modify the interface implemented by AbstractQuartzService from the original implementation of Job to the implementation of the StatefulJob interface.
5. Reference
http://www.quartz-scheduler.org/
http://www.ibm.com/developerworks/cn/opensource/os-cn-quartz/index.html