Introduction to Parallel and Distributed Computing (2) Programming Model and Hardware Model

Section 2 Programming Model and Hardware Model

2.1 Parallel model and structure

2.1.1 System Layers (not important)

1. Parallel program
2. Compiler, operating environment
3. operating system
4. Microstructure (hardware equipment)

2.1.2 Three parallel programming models (important)

The three models differ in the abstract structure of the program communication shown to the programmer

Different programming models will affect programmers' thinking when programming

Shared address space（SAS Model）

• Each thread passesRead and write shared variablesTo communicate
• Shared variables are like a big bulletin board

• A few examples (understand the main idea of ​​SAS, it is better to understand the computer architecture below)

• Non-uniform Memory Access

Message passing

• Each thread operates in its own private address space
• Threads interact by sending and receiving messages

Commonly used libraries: MPI

The hardware does not need to load and store cross-program information across the entire system, only need to be able to exchange information

• Note that the abstraction of message passing can be implemented on a machine where the hardware executes the shared address space, or SAS can be executed on a machine that does not support the shared address space.
  • The correlation between the program model and the machine category is vague. Just remember what is a programming model and what is a hardware implementation.

Data parallel

• The historical DP is to perform the same operation on an array (for example, the Clay supercomputer is a vector processor)
  • For example, the matlab operation C=A+B, where A, B, and C are all vectors
• Now generally adopt the programming form of SPMD (specifically as follows)
  • map(function,collection)
    • The function processes each element independently, and the function may be a complex logical sequence (thereby using a unified function to distinguish and process different elements)
    • At the end of the map, synchronization is implicit
      • When the function has acted on all elements of the collection, map returns

The structure that the programming model imposes on the program

SAS: There are few mandatory requirements for the structure

MP: highly structured communication

DP: Very rigid computing structure

• The DP program performs the same function on different data elements of the same set

Modern practice (mixed programming model)

• More common examples
  • Use shared address space between internal multiple cores of the same node
  • Use massage passing at the node level

2.1.3 Three machine structures (not important)

The machine structure usually reflects the capabilities of the hardware device

The machine structure is an abstract form
presented by the hardware to the software (I don’t understand it very well, the original sentence is "Abstraction presented by the hardware to low-level software")

The following things are more ICS, I don’t understand anyway

Flynn classification

• XIYD means X instruction Y Data, where X and Y are single or multiple
  • There are four types of SISD MISD SIMD MIMD

SISD

Example: Single-core computer-one data pool, one instruction pool, linear processing tasks

SIMD

MISD

MIMD

• Further breakdown of MIMD

  • Shared memory structure

• Distributed memory structure

• Hybrid structure

• Grid structure
  • Distributed heterogeneous resources connected by local area network and/or wide area network