Processing of parallel tasks in Camunda

In workflow orchestration, sometimes multiple tasks or sub-processes are run at the same time. The default mode is to run serially, but in order to improve performance, we hope to run in parallel. I also conducted some tests and found that this parallel operation is not so simple.

For example, we now define a process, which is very simple, that is, to call an Http interface. Then define another process, refer to the sub-process just defined in this process, and then run this sub-process multiple times. The effect we hope to achieve is that this sub-process can run in parallel.

First of all, we simply define an Http interface. After the interface receives the Request, it simply sleeps for 10 seconds and then returns. This interface is very simple, the code is not attached.

Then we define a simple process, which contains a Service Task, which calls this Http interface. The BPMN content of this process is as follows:

<?xml version="1.0" encoding="UTF-8"?>
<bpmn2:definitions xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:bpmn2="http://www.omg.org/spec/BPMN/20100524/MODEL" xmlns:bpmndi="http://www.omg.org/spec/BPMN/20100524/DI" xmlns:dc="http://www.omg.org/spec/DD/20100524/DC" xmlns:di="http://www.omg.org/spec/DD/20100524/DI" xmlns:camunda="http://camunda.org/schema/1.0/bpmn" id="sample-diagram" targetNamespace="http://bpmn.io/schema/bpmn" xsi:schemaLocation="http://www.omg.org/spec/BPMN/20100524/MODEL BPMN20.xsd">
  <bpmn2:process id="Process_12345" name="Test_12345" isExecutable="true">
    <bpmn2:startEvent id="StartEvent_1">
      <bpmn2:outgoing>Flow_1lqf9ii</bpmn2:outgoing>
    </bpmn2:startEvent>
    <bpmn2:sequenceFlow id="Flow_1lqf9ii" sourceRef="StartEvent_1" targetRef="Activity_1jsutqa" />
    <bpmn2:endEvent id="Event_03vs1kv" camunda:asyncBefore="true">
      <bpmn2:incoming>Flow_179keqc</bpmn2:incoming>
    </bpmn2:endEvent>
    <bpmn2:sequenceFlow id="Flow_179keqc" sourceRef="Activity_1jsutqa" targetRef="Event_03vs1kv" />
    <bpmn2:serviceTask id="Activity_1jsutqa" name="调用HTTP接口" camunda:resultVariable="newNEType">
      <bpmn2:extensionElements>
        <camunda:connector>
          <camunda:inputOutput>
            <camunda:inputParameter name="url">http://127.0.0.1:7777/getDeviceList</camunda:inputParameter>
            <camunda:inputParameter name="method">POST</camunda:inputParameter>
            <camunda:inputParameter name="headers">
              <camunda:map>
                <camunda:entry key="Accept">application/json</camunda:entry>
                <camunda:entry key="Content-Type">application/json</camunda:entry>
              </camunda:map>
            </camunda:inputParameter>
            <camunda:inputParameter name="payload">{"neType": "${execution.getVariable("neType")}",
"test": "${execution.getVariable("test")}"}</camunda:inputParameter>
          </camunda:inputOutput>
          <camunda:connectorId>http-connector</camunda:connectorId>
        </camunda:connector>
      </bpmn2:extensionElements>
      <bpmn2:incoming>Flow_1lqf9ii</bpmn2:incoming>
      <bpmn2:outgoing>Flow_179keqc</bpmn2:outgoing>
    </bpmn2:serviceTask>
  </bpmn2:process>
  <bpmndi:BPMNDiagram id="BPMNDiagram_1">
    <bpmndi:BPMNPlane id="BPMNPlane_1" bpmnElement="Process_12345">
      <bpmndi:BPMNShape id="_BPMNShape_StartEvent_2" bpmnElement="StartEvent_1">
        <dc:Bounds x="192" y="102" width="36" height="36" />
      </bpmndi:BPMNShape>
      <bpmndi:BPMNShape id="Event_03vs1kv_di" bpmnElement="Event_03vs1kv">
        <dc:Bounds x="432" y="102" width="36" height="36" />
      </bpmndi:BPMNShape>
      <bpmndi:BPMNShape id="Activity_1fzq8lj_di" bpmnElement="Activity_1jsutqa">
        <dc:Bounds x="280" y="80" width="100" height="80" />
        <bpmndi:BPMNLabel />
      </bpmndi:BPMNShape>
      <bpmndi:BPMNEdge id="Flow_1lqf9ii_di" bpmnElement="Flow_1lqf9ii">
        <di:waypoint x="228" y="120" />
        <di:waypoint x="280" y="120" />
      </bpmndi:BPMNEdge>
      <bpmndi:BPMNEdge id="Flow_179keqc_di" bpmnElement="Flow_179keqc">
        <di:waypoint x="380" y="120" />
        <di:waypoint x="432" y="120" />
      </bpmndi:BPMNEdge>
    </bpmndi:BPMNPlane>
  </bpmndi:BPMNDiagram>
</bpmn2:definitions>

Then we define another process, which contains a Call Activity, which refers to the process we just defined. In order to allow this Call Activity to run multiple times, we need to define an inputCollection variable. For example, I define a Script Task in the workflow, and then create an InputCollection through JavaScript:

var ArrayList = Java.type("java.util.ArrayList")
var neTypes = new ArrayList();
neTypes.add('AMF');
neTypes.add('SMF');
neTypes.add('UPF');
neTypes.add('ENUM');
neTypes.add('SPG');
neTypes.add('PCC');
neTypes;

Then create a Call Activity, select the type in the configuration to run multiple instances in parallel, that is, a small icon with three vertical lines, and then set the Collection in the Multi-Instance property to ${neTypes}, which is the variable output by the Script Task just now. Enter neType in Element Variable, so that when the workflow is running, it will traverse the elements in the Collection, assign it as neType, and then pass it to the Call Activity for execution. In addition, the Asynchronous before option in Multi-Instance should also be checked, but the Exclusive option should not be checked. In this way, when the process is executed before the Call Activity, it will pause, and then create a thread task to run the Call Activity asynchronously. If you don't set it like this, even if we set the Call Activity to run in parallel, it is still executed serially. As shown below:

 Here is the BPMN code for this process definition:

<?xml version="1.0" encoding="UTF-8"?>
<bpmn2:definitions xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:bpmn2="http://www.omg.org/spec/BPMN/20100524/MODEL" xmlns:bpmndi="http://www.omg.org/spec/BPMN/20100524/DI" xmlns:dc="http://www.omg.org/spec/DD/20100524/DC" xmlns:di="http://www.omg.org/spec/DD/20100524/DI" xmlns:camunda="http://camunda.org/schema/1.0/bpmn" id="sample-diagram" targetNamespace="http://bpmn.io/schema/bpmn" xsi:schemaLocation="http://www.omg.org/spec/BPMN/20100524/MODEL BPMN20.xsd">
  <bpmn2:process id="Process_12345" name="Test_12345" isExecutable="true">
    <bpmn2:startEvent id="StartEvent_1">
      <bpmn2:outgoing>Flow_1lqf9ii</bpmn2:outgoing>
    </bpmn2:startEvent>
    <bpmn2:sequenceFlow id="Flow_1lqf9ii" sourceRef="StartEvent_1" targetRef="Activity_1jsutqa" />
    <bpmn2:endEvent id="Event_03vs1kv" camunda:asyncBefore="true">
      <bpmn2:incoming>Flow_179keqc</bpmn2:incoming>
    </bpmn2:endEvent>
    <bpmn2:sequenceFlow id="Flow_179keqc" sourceRef="Activity_1jsutqa" targetRef="Event_03vs1kv" />
    <bpmn2:serviceTask id="Activity_1jsutqa" name="调用HTTP接口" camunda:resultVariable="newNEType">
      <bpmn2:extensionElements>
        <camunda:connector>
          <camunda:inputOutput>
            <camunda:inputParameter name="url">http://127.0.0.1:7777/getDeviceList</camunda:inputParameter>
            <camunda:inputParameter name="method">POST</camunda:inputParameter>
            <camunda:inputParameter name="headers">
              <camunda:map>
                <camunda:entry key="Accept">application/json</camunda:entry>
                <camunda:entry key="Content-Type">application/json</camunda:entry>
              </camunda:map>
            </camunda:inputParameter>
            <camunda:inputParameter name="payload">{"neType": "${execution.getVariable("neType")}",
"test": "${execution.getVariable("test")}"}</camunda:inputParameter>
          </camunda:inputOutput>
          <camunda:connectorId>http-connector</camunda:connectorId>
        </camunda:connector>
      </bpmn2:extensionElements>
      <bpmn2:incoming>Flow_1lqf9ii</bpmn2:incoming>
      <bpmn2:outgoing>Flow_179keqc</bpmn2:outgoing>
    </bpmn2:serviceTask>
  </bpmn2:process>
  <bpmndi:BPMNDiagram id="BPMNDiagram_1">
    <bpmndi:BPMNPlane id="BPMNPlane_1" bpmnElement="Process_12345">
      <bpmndi:BPMNShape id="_BPMNShape_StartEvent_2" bpmnElement="StartEvent_1">
        <dc:Bounds x="192" y="102" width="36" height="36" />
      </bpmndi:BPMNShape>
      <bpmndi:BPMNShape id="Event_03vs1kv_di" bpmnElement="Event_03vs1kv">
        <dc:Bounds x="432" y="102" width="36" height="36" />
      </bpmndi:BPMNShape>
      <bpmndi:BPMNShape id="Activity_1fzq8lj_di" bpmnElement="Activity_1jsutqa">
        <dc:Bounds x="280" y="80" width="100" height="80" />
        <bpmndi:BPMNLabel />
      </bpmndi:BPMNShape>
      <bpmndi:BPMNEdge id="Flow_1lqf9ii_di" bpmnElement="Flow_1lqf9ii">
        <di:waypoint x="228" y="120" />
        <di:waypoint x="280" y="120" />
      </bpmndi:BPMNEdge>
      <bpmndi:BPMNEdge id="Flow_179keqc_di" bpmnElement="Flow_179keqc">
        <di:waypoint x="380" y="120" />
        <di:waypoint x="432" y="120" />
      </bpmndi:BPMNEdge>
    </bpmndi:BPMNPlane>
  </bpmndi:BPMNDiagram>
</bpmn2:definitions>

Another thing to note is that in the process referenced by the Call Activity, Asynchronous before and Exclusive also need to be set in the end event, otherwise the number of times the Call Activity will run will be different from what we expected, and there will be more. According to the explanation on the Internet, since there may be multiple threads running this subprocess at the same time, Camunda only allows only one thread to be active in the same process, so only one thread runs successfully, and other threads will retry to run, so We will observe that there are more runs than we expected. Setting Async before in the End envent can prevent the thread from running repeatedly.

Run the process, and from the call log printed by the HTTP interface, you can see that the sub-processes are indeed running in parallel.

In the actual project, I also found a strange problem, that is, I run multiple sub-processes in Call Activity in parallel, but one of the sub-processes will always run without ending, and other sub-processes do not have this problem. At that time, I thought about it for a long time and found no problem. Later, I observed this sub-process. It took a long time to run, because the data to be processed was more than other sub-processes, and it took about 5 minutes to run once. I checked the configuration description on the official website again, and found that there is a configuration parameter in the task execution which is lock-time-in-millis, and the default configuration is exactly 5 minutes, so it is likely that the task has not been executed for more than 5 minutes. Ended, so Camunda started another task to execute repeatedly, so the phenomenon is as if this task has been running. After finding the problem, I changed this configuration to 15 minutes, which solved the problem. Of course, according to the explanation on the official website, another better solution is that when there is a task that needs to run for a long time, it is best to change it to an external task and let it be executed externally. After the external execution is completed, send a message to the task. .