When the same ring is used by multiple producers, and one producer is on the non-real-time core and the other is on the real-time core, it is possible that the real-time core waits for the end of the non-real-time core enqueue.
As described in the DPDK documentation, each core will update ring->prod_tail through the function __rte_ring_update_tail(), indicating the end of enqueue. A core can only be updated when ring->prod_tail is equal to the local prod_head. In the figure above, only core1 can be updated successfully, and core2 needs to wait for the update of core1 to complete. If core1 is a non-real-time core and is cut off before updating ring->prod_tail, the waiting time may be longer.
rte_ring_mp_enqueue_bulk
-> rte_ring_mp_enqueue_bulk_elem
-> __rte_ring_do_enqueue_elem
-> __rte_ring_update_tail
https://elixir.bootlin.com/dpdk/v21.11/source/lib/ring/rte_ring_c11_pvt.h#L15
The figure shows that the non-real-time core is cut off, and the real-time core is waiting for the end of the prod_tail update of the non-real-time core ring.
solution
-
Use hardware to schedule, and some chips will have their own scheduling or queue processing modules.
-
The communication between the non-real-time core and the real-time core is realized by using different queues. For example, one queue is used for non-real-time verification and dispatch to real-time verification, and another queue is used for real-time verification and dispatch to non-real-time verification. The communication between real-time cores still uses the original queue. There are several implementation methods for the new queue:
a. Handwritten ring buffer, you can refer to the ring buffer implementation of linux, such as https://lwn.net/Articles/340443/. Improvement: The best thing to save in the ring is the offset, not the entire object, and the performance is better. Note that the ring buffer of linux or the ring implemented by itself is generally not sync. For example, the sender sends a semaphore to ask the receiver to fetch the packet. Although the semaphore has passed, the packet may not actually arrive in the queue.
b. The boost::lockfree::queue implementation of the C++ boost library, the ring of the boost library is sync, that is, after the push to the ring returns, the peer must be able to pop.
c. Use DPDK's memif to realize communication.
performance comparison
Hardware scheduling > DPDK queue > handwritten ring buffer > boost ring buffer > DPDK memif.
If the queue is only used by one core, there is no need to consider whether to lock or not to program, just use the C++ default push and pop functions to put it in the queue.
Original link: https://mp.weixin.qq.com/s/g4Dbw8X8bylUZfid8eluOA
Learn more dpdk videos
Dpdk/network protocol stack/vpp/OvS/DDos/NFV/virtualization/high performance expert learning address: https://ke.qq.com/course/5066203?flowToken=1043799
DPDK development learning materials, Teaching videos and learning roadmap sharing can be added to learning exchanges if necessary. q Junyang 909332607 Get it