Parallel and Distributed Computing Chapter 3 Process-Level Parallelism: MPI Programming

Parallel and Distributed Computing Chapter 3 Process-Level Parallelism: MPI Programming

3.1 The concept of process-level parallelism

3.1.1 Multi-process working mode

master-slave mode

• Divide a task to be solved into a main task (main process) and some slave tasks (sub-processes)
• The main process is responsible for decomposing tasks, dispatching and collecting the solution results of each sub-task and finally summarizing them to obtain the final solution to the problem.
• Each sub-process receives messages from the main process; performs its own calculations in parallel; and then sends back its respective calculation results to the main process.

Single control flow multiple data flow mode

• First, the data needs to be pre-allocated to each calculation process.
• Then each computing process completes its own computing tasks in parallel, including data exchange between processes during the computing process (implementation of communication synchronization)
• Finally, the calculation results are gathered together.

Inter-Process Communication
Inter-Process Communication (IPC, Inter-Process Communication) allows programmers to coordinate different processes so that they can run simultaneously in an operating system and communicate with each other. Transfer and exchange information.

3.1.2 Message passing model

• In the message passing model, parallel processes exchange data by passing messages to each other
• IPC methods such as pipes, message queues, and sockets are all message passing

** SEND & RECEIVE**

• Send/receive (send/receive) is a pair of the most basic messaging primitives
  • Pipe’s write/read operation (using pipes to realize communication between parent and child processes)< /span>
  • Send/receive operations on Message Queue or Socket
• Most of the messaging methods we have access to can be abstracted into send/receive operations.

Communication mechanism: synchronous & asynchronous
• In messaging, synchronous and asynchronous are two different ways the sender handles communication

• Synchronous: The message sender waits for the message receiver to complete reception and response, and then
starts processing other work
• Asynchronous : The sender does not wait for the response after sending the message and starts processing directly
Other work

Comparison between MPI and OPENMP

Characteristics of MPI programs

• Users must exchange data between processors by explicitly sending and receiving messages.
• Each parallel process has its own independent address space.
• Parallel computing has large granularity and is particularly suitable for large-scale scalable parallel algorithms.

3.2 OPENMPI

MPI function	Function
MPI_Init(&argc, &argv)	Notifies the MPI system to perform all initialization settings.
MPI_Finalize()	Clean up the MPI environment. After calling this function, no other MPI functions can be called.
MPI_Comm_rank(MPI_COMM_WORLD, &rank)	Get the current process ID.
MPI_Comm_size(MPI_COMM_WORLD, &size)	Get the number of available processes.

3.2.1 MPI communication function SEND & RECEIVE

A message is like a letter

• The content of the message, that is, the content of the letter, is called the message buffer (Message Buffer) in MPI. The message buffer is identified by the triplet <starting address, data number, data type>
• The sender and receiver of the message, that is, the address of the letter, becomes the message envelope (Message Envelop) in MPI. The message envelope consists of the triplet <source/target process, message label, communication domain> Identifies MPI_Send (buf, count, datatype, dest, tag, comm)

Why do you need message tags?
When the sender sends two consecutive messages of the same type to the same receiver, without message tags, the receiver will not be able to distinguish the two messages

type of data

• One of the features of MPI is that all communication functions take a data type parameter to describe the type of data sent and received.
• You can create your own data types
• The purpose of using data type parameters is that in heterogeneous environments, such as heterogeneous computing and systems with different byte storage orders or different basic data type lengths, when the data type is specified, MPI can internally convert it into the corresponding word number of nodes to ensure smooth communication.

COMMUNICATOR

• Processes can be divided into groups, with each group handling a specific task
• Every message should be sent and received in the same context
• Such a grouping, together with the messaging context, is called a communicator
• MPI_COMM_WORLD is the default communicator. If there is no need for grouping, you can use this communicator directly.

MPI_STATUS
When the receiver calls MPI_Recv, it will provide an MPI_Status structure as a parameter. This structure contains some information about the message:
• Sender rank, in MPI_SOURCE field (stat.MPI_SOURCE)
• Message tag, in MPI_TAG (stat.MPI_TAG)
• Message length, via MPI_Get_count(MPI_Statusstatus, MPI_Datatype datatype, int count) function readout

3.2.2 Point-to-point communication diagram

Blocking communication (blocking send)

Blocking send means that the sending function does not return until the data is copied from the location specified in the function parameter list, so the data can be changed after the sending function is called without affecting the original message.

MPI’s four point-to-point communication modes
Communication Mode refers to buffer management and synchronization between the sender and receiver.

Synchronous communication
• The sending process returns correctly only when the corresponding receiving process has been started.
• Therefore, after the synchronous send returns, it means that all the data in the send buffer has been cached by the system buffer and the sending has started.
• When the synchronous send returns, the send buffer can be released or reused

Buffered communication
Sending in buffered communication mode can be executed regardless of whether the receiving operation has been started, but the user program needs to apply for a large enough buffer in advance, which is implemented through MPI_Buffer_attch and through MPI_Buffer_detach Buffer for recycling applications.

Standard communication

Ready Communication
• A send operation is sent only after the corresponding receive operation of the receiving process has started.
• When a send operation is started but the corresponding receive has not been started, an error occurs in the send operation.
• The special feature of ready communication mode is that the receiving operation must be started before the sending operation.

Conditions for blocking communication return:
• The communication operation has been completed, that is, the message has been sent or received;
• The called buffer is available. If it is a sending operation, the buffer can be updated by other operations; if it is a receiving operation, the data in the buffer is complete and can be referenced correctly.

Non-blocking communication

The non-blocking communication function returns immediately. In this asynchronous situation, we do not know whether the sending/receiving is successful, so a Request object is needed as Handler (handle) to check the data sending status. MPI also provides detection of the completion of non-blocking communication. There are two main types: MPI_Wait function and MPI_Test function.

3.2.3 N V N (cluster communication)

Collective Communications is a global communication operation that all processes in a process group participate in. Two classification methods: functional classification and direction classification
Functional classification
• Communication function: mainly completes the transmission of data within the group
• Aggregation function: Complete certain operations on given data based on communication
• Synchronization function: Achieve consistency in the execution progress of all processes in the group

Direction classification
• One-to-many communication: One process sends messages to all other processes. The process responsible for sending messages is called
Root process .
• Many-to-one communication: One process is responsible for receiving messages from all other processes. This receiving process is also called
Root process.
• Many-to-many communication: Each process sends or receives messages to all other processes.

Set communication synchronization
Set communication introduces the concept of synchronization points (Barriers) between processes. This means that all processes must first reach a synchronization point when executing code before they can continue to execute subsequent code

If you need to synchronize your program explicitly, you can also use the barrier directive: MPI_Barrier(MPI_Comm communicator)

Broadcast MPI_Bcast

Broadcast is used to pass data from the root node to all processes in the communicator

Distribution MPI_Scatter
Distribution is to send each part of a set of data from the root node to each process in the communicator. The data received by each process can be different a>

Gather MPI_Gather

Global collection VS global exchange

Reduction REDUCE
Reduction is a concept in functional programming. It accepts a set of data and outputs a smaller set of numbers, such as sum([1,2,3, 4])=10, MPI provides functions to perform reduction operations between processes

MPI protocol operations	Function
MPI_MAX	Returns the largest element
MPI_MIN	Returns the smallest element
MPI_SUM	Sum the elements
MPI_PROD	Multiply all elements
MPI_LAND	Performs a logical AND operation on elements
MPI_LORS	Performs a logical OR operation on the elements
MPI_BAND	Performs a bitwise AND on each element's bits
MPI_BOR	Performs a bitwise OR operation on the bits of the elements
MPI_MAXLOC	Returns the maximum value and the rank of the process where it resides
MPI_MINLOC	Returns the minimum value and the rank of the process where it is located

3.2.4 Unilateral communication

• Unilateral communication is also called Remote Memory Access (RMA)
• Whether point-to-point communication or collective communication, the sender and receiver need to cooperate , is a message passing method based on synchronization
• Unilateral communication decouples the two operations of data transfer and synchronization. Each process exposes a part of the memory to other processes, and other processes can access the memory at will. area, transmit data without synchronization
• When using unilateral communication, you must first create a window (MPI_Win), and then perform unilateral communication operations on the window

Create and destroy windows

One-sided communication operation PUT

One-sided communication operation get

One-sided communication operation Atomic addition