Reference: "PCIe-Multicast (Multicast) Implementation"
PCIe multicast concept
The traditional PCIe system is limited to single-target transmission. This transmission mode is centered on the host, so that the host can only communicate with one of many sub-devices at a given time.
Now more and more applications need to send data to multiple sub-devices at the same time. Perhaps the host is not just one, and even one sub-device needs to send data to the host to other sub-devices, and PCIE multicast has emerged.
Multicast refers to sending information to multiple targets or groups at the same time. It allows each host to communicate with multiple sub-devices at a certain time and send the same data packet to multiple sub-devices at the same time. Compared with traditional transmission, multicast The model efficiency is greatly improved, so it is widely used in engineering realization.
The multicast function of PCIe Multicast is implemented on the Switch. For example, PEX8749, PEX8796, etc.
PCIe multicast implementation
Multicast allows software to simultaneously write the same data to a group of multiple targets. The memory write TLP of PEX 8796 is addressed to the MC address range (MC BAR), and after multicast is enabled, PEX 8796 will automatically generate and send the original copy to the TLP of one or more target ports (called MC copy TLP) ). MC address space is divided into MC group (MCG), which is defined by MC base address and MC index position. Each PEX 8796 port can choose to receive MC Copy TLP by choosing to belong to MCG. The corresponding MC receive bit. If needed, you can use "Block All" to block MC TLP. The MC overlay bar can be used to replace the address of the original MC TLP with the unicast address space if the endpoint does not support MC.
The implementation of PCIE multicast depends on the configuration of the PCIe Multicast Capability structure. The PCIeMulitcastCapability structure is as follows:
- PCIe_Extended_Capability_Header: Contains extend_capability_ID, version, and the next capability offset.
- MC_Control_and_Capability register: bit31 is multicast_enable_bit; Bit30-22: reserved as 0; Bit21-16 multicast_groups number setting, users can define the number of multicast groups according to their needs. When the value is 0, it means there is only one group. When the value is N, group=N+ I; Bit15. Bit0 is the capability setting, which is set by the manufacturer and cannot be modified.
- MC_Base_Address register: Bit 63-l2: broadcast base address; Bit 11-6: reserved as 0; Bit 5-0: Index_position multicast memory space, Memeory size is 2 Index_Posiztion. Index_position<12 is invalid.
- MC_Receive register: Bit 63-0, each bit corresponds to a multicast group for receiving data, for example, if bit3 is set to 1, it indicates that the device wants to accept the third multicast group data.
- MC_block_All register: Bit63-0, each bit corresponds to a group, used to mask data, for example, if bit3 is set to 1, all data from the third group must be masked for this device.
- MC_Block_untransated register: Bit 63-0, each bit corresponds to a group, used to shield data packets containing non-transparent addresses, for example, if bit3 is set to 1, all data from the third group of the device must be shielded.
- MC_OVERLAY_BAR register: This register is used to change the access address in the multicast TLP, Bit5-0, the size of the relocation access address space, if the value is less than 6, the address change mechanism is invalid; Bit63-6: the redirected address, such as the original TLP The access address is 0x2000. If the redirect address is set to 0x1000, the device will change the access address to 0x1000 after receiving this TLP.
PCIe multicast access conditions
By setting the MC Control register and MC BaseAddress register, we divide the group address space into N groups of address spaces based on the base address, and the size of each group address space is 2^MC_Index_Position.
The conditions for triggering PCIe_Multicast are:
- MC_Enable is set.
- TLP is a write request.
- The destination address falls within the multicast address range.
Under this premise, the multicast will be activated. If the receiving device MC_Receive_bit matches the multicast group, the device can receive the multicast TLP.
PCIe multicast example
A system host is connected to 3 devices through pcie switch. In the address map, we can see that there are four PCIe ports. Assume that Port0 is up_stream_port, Port1, 2, and 3 are devices. The address map is as follows:
Port 0：F7800000 ～ F7AFFFFF
Port 1：F7A00000 ～ F7AFFFFF
Port 2：F7900000 ～ F79FFFFF
Port 3：F7800000 ～ F78FFFFFSet the settings as follows:
- The MC_Control_Capability register of each port is 0x8002803F, 3 groups.
- The MC Baseaddress register of each port is set to 0x7800014, the base address is 0x7800000, and the group address size is 1MB.
- The MC Receive register of each port is set to 0x7. In this way, each downstream port can receive the TLP from the host.
PCIe multicast practice
JD4000 has 19 chips in total (3536*6 + 3519*12 + 6678), the tree structure under pci is as follows:
lspci -t -vxxx
-[0000:00]-+-00.0 Intel Corporation Sky Lake Host Bridge/DRAM Registers
+-01.0-[01-14]----00.0-[02-14]--+-04.0-[03]----00.0 Device 19e5:3536
| +-05.0-[04]----00.0 Device 19e5:3536
| +-06.0-[05]----00.0 Device 19e5:3536
| +-07.0-[06]----00.0 Device 19e5:3536
| +-08.0-[07]----00.0 Device 19e5:3519
| +-09.0-[08]----00.0 Device 19e5:3519
| +-0a.0-[09]----00.0 Device 19e5:3519
| +-0b.0-[0a]----00.0 Device 19e5:3519
| +-0c.0-[0b]----00.0 Device 19e5:3519
| +-0d.0-[0c]----00.0 Device 19e5:3519
| +-0e.0-[0d]----00.0 Device 19e5:3519
| +-0f.0-[0e]----00.0 Device 19e5:3519
| +-10.0-[0f]----00.0 Device 19e5:3519
| +-11.0-[10]----00.0 Device 19e5:3519
| +-12.0-[11]----00.0 Device 19e5:3536
| +-14.0-[12]----00.0 Device 19e5:3519
| +-15.0-[13]----00.0 Device 19e5:3519
| \-16.0-[14]----00.0 Device 19e5:3536That is, there are 20 PCI bridges, 01:00.0 is the upstream of the bridges such as 02:04.0~02:16.0.
To put it simply, the address map size on the 00:01.0 PCI bridge is 1536M.
A total of 18 ports can be seen in the address map.
BUS PORT BAR
02:16.0 22 0x2058000000-0x205bffffff 0x205c000000-0x205fffffff
02:15.0 21 0x2054000000-0x2057ffffff
02:14.0 20 0x2050000000-0x2053ffffff
02:12.0 18 0x2048000000-0x204bffffff 0x204c000000-0x204fffffff
02:11.0 17 0x2044000000-0x2047ffffff
02:10.0 16 0x2040000000-0x2043ffffff
02:07.0 7 0x2018000000-0x201bffffff 0x201c000000-0x201fffffff
02:0e.0 14 0x2038000000-0x203bffffff
02:0f.0 15 0x203c000000-0x203fffffff
02:06.0 6 0x2010000000-0x2013ffffff 0x2014000000-0x2017ffffff
02:0c.0 12 0x2030000000-0x2033ffffff
02:0d.0 13 0x2034000000-0x2037ffffff
02:05.0 5 0x2008000000-0x200bffffff 0x200c000000-0x200fffffff
02:0a.0 10 0x2028000000-0x202bffffff
02:0b.0 11 0x202c000000-0x202fffffff
02:04.0 4 0x2000000000-0x2003ffffff 0x2004000000-0x2007ffffff
02:09.0 9 0x2024000000-0x2027ffffff Roughly, the address map size on the entire 00:01.0 PCI bridge can be divided into 24 blocks. (12*3519 + 6*3536*2)
The size of each map address is 64M (1536M / 24).
Set the settings as follows, and set the relevant multicast registers through setpci:
The MC_Control_Capability register of each port is 0x8017803F, 24 groups
setpci -s 01:00.0 0x334.l=0x8017803F
The MC_Base_address register of each port is set to 0x00000020 0000001A, the base address is 0x0000002000000000, and the group address size is 64MB.
setpci -s 01:00.0 0x338.l=0x0000001A
setpci -s 01:00.0 0x33C.l=0x00000020
The MC_Receive register of each port is set to 0x1FFFFFF (24 ports). In this way, each downstream port can receive the TLP from the host.
setpci -s 01:00.0 0x340.l=0x1FFFFFF
setpci -s 01:00.0 0x344.l=0x0