Xuecheng Online (Video Processing - Demand Analysis: xxl-job)

demand analysis

Job Fragmentation Scheme

• After the task is successfully added, the task to be processed will be added to the pending task table. Now start multiple executor instances to query these pending tasks. How to ensure that multiple executors will not execute the task repeatedly?
• In the test in the previous section, each executor received a broadcast task with two parameters, the fragment number and the total number of fragments
  • When each executor fetches a task from the data table, it can be fetched in 任务idpairs分片总数模 . If it is equal to the shard sequence number of the executor, the task will be executed
  • For example
    • 1 % 2 = 1, actuator 2 executes
    • 2 % 2 = 0, actuator 1 executes
    • 3 % 2 = 1, actuator 2 executes
    • 4 % 2 = 1, actuator 1 executes
    • and so on

 

Ensure tasks are not repeated

• Through the job sharding scheme, it is ensured that the tasks allocated between the executors are not executed repeatedly
• But if the same executor has not finished processing a video, and at this time the scheduling center requests scheduling again, what should I do in order not to process the same video repeatedly?
• First configure the scheduling expiration policy
  • Scheduling expiration policy:
    • Ignore: After the schedule expires, ignore the expired task and recalculate the next trigger time from the current time;
    • Execute once immediately: After the schedule expires, execute once immediately, and recalculate the next trigger time from the current time;
• Here we choose 忽略, if it is executed once immediately, it may be scheduled repeatedly
• Second, we are looking at blocking handling strategies.
• The blocking processing strategy is when the current executor is executing the task and the scheduling center requests scheduling, how to deal with it at this time
  • Blocking processing strategy: the processing strategy when the scheduling is too intensive for the executor to process in time;
    • Stand-alone serial (default): After the scheduling request enters the single-machine executor, the scheduling request enters the FIFO queue and runs in serial mode;
    • Discard subsequent scheduling: After the scheduling request enters the stand-alone executor, it is found that the executor has a running scheduling task, and this request will be discarded and marked as failed;
    • Scheduling before coverage: After the scheduling request enters the stand-alone executor, if the executor finds that there is a running scheduling task, it will terminate the running scheduling task and clear the queue, and then run the local scheduling task;
• Choose here 丢弃后续调度to avoid repeated scheduling
• Finally, it is necessary to pay attention to ensure that the task is processed 幂等性, what is it 任务的幂等性?
  • The idempotency of the task means that no matter how many times the data is operated, the result of the operation is always consistent.
• The executor receives the scheduling request to execute the task. There must be a way to judge whether the task has been processed.
• What is idempotency?
  • It describes one and multiple requests for a resource, which should have the same result for the resource itself
• Idempotence is to solve the problem of repeated submission, such as: malicious swiping, repeated payment, etc.
• Commonly used solutions to idempotence
  1. Database constraints, such as: unique index, primary key
  2. Optimistic lock, long-term user database, when updating data, update according to the state of optimistic lock
  3. Unique serial number, the operation passes a unique serial number, if it is judged to be equal to the serial number during the operation, then execute
• Here we add a status processing field to the video processing table in the database. After the video processing is completed, the updated status is completed. Before executing the video, judge whether the status is complete. If it is completed, it will not be processed.

Business Process

• The fragmentation scheme is determined, and the business process of humming video upload and processing is sorted out below
• The video upload is successful, and a record is added to the video pending table. The detailed process of video processing is as follows

 

 

1. Task scheduling center broadcasts job shards
2. The executor receives broadcast job fragments and reads pending tasks from the database
3. The executor downloads the files to be processed according to the task content MinIO
4. The executor starts multiple threads to process tasks
5. The task processing is completed, and the processed video is uploaded to MinIO
6. The task processing result will be updated. If the video processing is completed, in addition to updating the task processing result, the access address of the file will also be updated to the task processing table and file, and finally the task completion record will be written into the history table

• The following is the list of pending tasks