Common problems with Linux read-write locks
Read-write lock is another way of synchronizing between multiple threads besides mutex locks.
Multiple threads are safe for read operations of a shared variable, but write operations a> is unsafe. If you are in a scenario where there are many reads and few writes, then using a mutex will prevent a large number of thread-safe read operations. In this scenario, a design such as read-write lock was born.
The characteristics of the read-write lock are shown in the following table. To sum up, reading is not mutually exclusive, Read and write are mutually exclusive, Write and write are mutually exclusive.
| read | Write | |
|---|---|---|
| read | Not mutually exclusive | mutually exclusive |
| Write | mutually exclusive | mutually exclusive |
It seems that such a good design does have many misunderstandings in actual use. Master Chen Shuo once gave his suggestions in his book<<Linux多线程服务端编程>>, don’t Use read-write locks. Why is this so? Let’s go through it one by one.
Correct use of read-write locks
The first error-prone part of read-write locks is that shared data may be modified where the read lock is held. It may not be error-prone for some relatively simple methods, but it is also error-prone in nested calling scenarios. For example, in the following example, the read method holds the read lock, but operator4 will modify the shared variable. Since operator4 has a deep call depth, it may be prone to errors.
//operator4会修改共享变量
void operation4();
{
//...
}
void operation3()
{
operation4();
}
void operation2()
{
operation3();
}
void read() {
std::shared_lock<std::shared_mutex> lock(mtx);
operation1();
}
Read-write lock performance overhead
read-write lock is more complex in design than mutex lock, so its The internal locking and unlocking logic is also more complex than a mutex lock.
The following is the data structure of glibc read-write lock. It can be speculated that during the lock and unlock process, the number of readers and writers needs to be updated, and Mutex locks do not require such operations.
struct __pthread_rwlock_arch_t
{
unsigned int __readers;
unsigned int __writers;
unsigned int __wrphase_futex;
unsigned int __writers_futex;
unsigned int __pad3;
unsigned int __pad4;
int __cur_writer;
int __shared;
unsigned long int __pad1;
unsigned long int __pad2;
/* FLAGS must stay at this position in the structure to maintain
binary compatibility. */
unsigned int __flags;
};
The following example uses a mutex lock and a read-write lock to repeatedly lock and unlock a critical section. Because the critical section has no content, the overhead is basically on locking and unlocking the lock.
//g++ test1.cpp -o test1
#include <pthread.h>
#include <iostream>
#include <unistd.h>
pthread_mutex_t mutex;
int i = 0;
void *thread_func(void* args) {
int j;
for(j=0; j<10000000; j++) {
pthread_mutex_lock(&mutex);
// test
pthread_mutex_unlock(&mutex);
}
pthread_exit((void *)0);
}
int main(void) {
pthread_t id1;
pthread_t id2;
pthread_t id3;
pthread_t id4;
pthread_mutex_init(&mutex, NULL);
pthread_create(&id1, NULL, thread_func, (void *)0);
pthread_create(&id2, NULL, thread_func, (void *)0);
pthread_create(&id3, NULL, thread_func, (void *)0);
pthread_create(&id4, NULL, thread_func, (void *)0);
pthread_join(id1, NULL);
pthread_join(id2, NULL);
pthread_join(id3, NULL);
pthread_join(id4, NULL);
pthread_mutex_destroy(&mutex);
}
//g++ test2.cpp -o test2
#include <pthread.h>
#include <iostream>
#include <unistd.h>
pthread_rwlock_t rwlock;
int i = 0;
void *thread_func(void* args) {
int j;
for(j=0; j<10000000; j++) {
pthread_rwlock_rdlock(&rwlock);
//test2
pthread_rwlock_unlock(&rwlock);
}
pthread_exit((void *)0);
}
int main(void) {
pthread_t id1;
pthread_t id2;
pthread_t id3;
pthread_t id4;
pthread_rwlock_init(&rwlock, NULL);
pthread_create(&id1, NULL, thread_func, (void *)0);
pthread_create(&id2, NULL, thread_func, (void *)0);
pthread_create(&id3, NULL, thread_func, (void *)0);
pthread_create(&id4, NULL, thread_func, (void *)0);
pthread_join(id1, NULL);
pthread_join(id2, NULL);
pthread_join(id3, NULL);
pthread_join(id4, NULL);
pthread_rwlock_destroy(&rwlock);
}
[root@localhost test1]# time ./test1
real 0m2.531s
user 0m5.175s
sys 0m4.200s
[root@localhost test1]# time ./test2
real 0m4.490s
user 0m17.626s
sys 0m0.004s
It can be seen that from the perspective of locking and unlocking, the performance of mutex locks is better than that of read-write locks.
Of course, during the test here, the contents of the critical section are spatiotemporal. If the critical section is larger, the performance of the read-write lock may be better than that of the mutex lock.
However, in multi-threaded programming, we always try to reduce the size of the critical section as much as possible, so many times, read-write locks are not as efficient as imagined.
Read-write locks can easily cause deadlocks
As mentioned earlier, the design of read-write lock is produced in the scenario where reads more and writes less. However, in such a scenario , it is easy to cause starvation of write operations. Because there are too many read operations, the write operation cannot obtain the lock, causing the write operation to be blocked.
Therefore, writing operations to acquire locks usually have high priority.
Such a setting will cause a deadlock in the following scenario. Assume that there are threads A, B and locks, and they are executed in the following sequence:
- 1. Thread A applies for a read lock;
- 2. Thread B applies for a write lock;
- 3. Thread A applies for the read lock again;
In step 2, when thread B applies for the write lock, thread A has not yet released the read lock, so it needs to wait. In step 3, thread B is applying for the write lock, so thread A's application for the read lock will be blocked, and it will fall into a deadlock state.
The following uses shared_mutex of c++17 to simulate such a scenario.
#include <iostream>
#include <thread>
#include <mutex>
#include <condition_variable>
#include <shared_mutex>
void print() {
std::cout << "\n";
}
template<typename T, typename... Args>
void print(T&& first, Args&& ...args) {
std::cout << first << " ";
print(std::forward<Args>(args)...);
}
std::shared_mutex mtx;
int step = 0;
std::mutex cond_mtx;
std::condition_variable cond;
void read() {
//step0: 读锁
std::shared_lock<std::shared_mutex> lock(mtx);
std::unique_lock<std::mutex> uniqueLock(cond_mtx);
print("read lock 1");
//通知step0结束
++step;
cond.notify_all();
//等待step1: 写锁 结束
cond.wait(uniqueLock, []{
return step == 2;
});
uniqueLock.unlock();
//step2: 再次读锁
std::shared_lock<std::shared_mutex> lock1(mtx);
print("read lock 2");
}
void write() {
//等待step0: 读锁 结束
std::unique_lock<std::mutex> uniqueLock(cond_mtx);
cond.wait(uniqueLock, []{
return step == 1;
});
uniqueLock.unlock();
//step1: 写锁
std::lock_guard<std::shared_mutex> lock(mtx);
uniqueLock.lock();
print("write lock");
//通知step1结束
++step;
cond.notify_all();
uniqueLock.unlock();
}
int main() {
std::thread t_read{
read};
std::thread t_write{
write};
t_read.join();
t_write.join();
return 0;
}
You can test it using the online version below.
The output of the online version is as follows. The program was killed due to deadlock execution timeout.
Killed - processing time exceeded
Program terminated with signal: SIGKILL
The cause of the deadlock is that thread 1 and thread 2 are waiting for each other.
For glibc's read-write lock, it provides read priority and write priorityAttributes of .
Usepthread_rwlockattr_setkind_np method to set the attributes of the read-write lock. It has the following properties:
- PTHREAD_RWLOCK_PREFER_READER_NP, //Reader priority (that is, when a read lock and a write lock are requested at the same time, the thread requesting the read lock obtains the lock first)
- PTHREAD_RWLOCK_PREFER_WRITER_NP, //Don't be fooled by the name, it is also the reader's priority
- PTHREAD_RWLOCK_PREFER_WRITER_NONRECURSIVE_NP, //Writers take precedence (that is, when a read lock and a write lock are requested at the same time, the thread requesting the write lock obtains the lock first)
- PTHREAD_RWLOCK_DEFAULT_NP = PTHREAD_RWLOCK_PREFER_READER_NP //Default, reader priority
glibc’s read-write lock mode is read-first. The following uses read first and write first respectively for testing.
- Write first
#include <iostream>
#include <pthread.h>
#include <unistd.h>
pthread_rwlock_t m_lock;
pthread_rwlockattr_t attr;
int A = 0, B = 0;
// thread1
void* threadFunc1(void* p)
{
printf("thread 1 running..\n");
pthread_rwlock_rdlock(&m_lock);
printf("thread 1 read source A=%d\n", A);
usleep(3000000); // 等待3s，此时线程2大概率会被唤醒并申请写锁
pthread_rwlock_rdlock(&m_lock);
printf("thread 1 read source B=%d\n", B);
//释放读锁
pthread_rwlock_unlock(&m_lock);
pthread_rwlock_unlock(&m_lock);
return NULL;
}
//thread2
void* threadFunc2(void* p)
{
printf("thread 2 running..\n");
pthread_rwlock_wrlock(&m_lock);
A = 1;
B = 1;
printf("thread 2 write source A and B\n");
//释放写锁
pthread_rwlock_unlock(&m_lock);
return NULL;
}
int main()
{
pthread_rwlockattr_init(&attr);
pthread_rwlockattr_setkind_np(&attr, PTHREAD_RWLOCK_PREFER_WRITER_NONRECURSIVE_NP);//设置写锁优先级高
//初始化读写锁
if (pthread_rwlock_init(&m_lock, &attr) != 0)
{
printf("init rwlock failed\n");
return -1;
}
//初始化线程
pthread_t hThread1;
pthread_t hThread2;
if (pthread_create(&hThread1, NULL, &threadFunc1, NULL) != 0)
{
printf("create thread 1 failed\n");
return -1;
}
usleep(1000000);
if (pthread_create(&hThread2, NULL, &threadFunc2, NULL) != 0)
{
printf("create thread 2 failed\n");
return -1;
}
pthread_join(hThread1, NULL);
pthread_join(hThread2, NULL);
pthread_rwlock_destroy(&m_lock);
return 0;
}
Setting write priority can lead to deadlock.
- read first
#include <iostream>
#include <pthread.h>
#include <unistd.h>
pthread_rwlock_t m_lock;
pthread_rwlockattr_t attr;
int A = 0, B = 0;
// thread1
void* threadFunc1(void* p)
{
printf("thread 1 running..\n");
pthread_rwlock_rdlock(&m_lock);
printf("thread 1 read source A=%d\n", A);
usleep(3000000); // 等待3s，此时线程2大概率会被唤醒并申请写锁
pthread_rwlock_rdlock(&m_lock);
printf("thread 1 read source B=%d\n", B);
//释放读锁
pthread_rwlock_unlock(&m_lock);
pthread_rwlock_unlock(&m_lock);
return NULL;
}
//thread2
void* threadFunc2(void* p)
{
printf("thread 2 running..\n");
pthread_rwlock_wrlock(&m_lock);
A = 1;
B = 1;
printf("thread 2 write source A and B\n");
//释放写锁
pthread_rwlock_unlock(&m_lock);
return NULL;
}
int main()
{
pthread_rwlockattr_init(&attr);
pthread_rwlockattr_setkind_np(&attr, PTHREAD_RWLOCK_PREFER_READER_NP);
//初始化读写锁
if (pthread_rwlock_init(&m_lock, &attr) != 0)
{
printf("init rwlock failed\n");
return -1;
}
//初始化线程
pthread_t hThread1;
pthread_t hThread2;
if (pthread_create(&hThread1, NULL, &threadFunc1, NULL) != 0)
{
printf("create thread 1 failed\n");
return -1;
}
usleep(1000000);
if (pthread_create(&hThread2, NULL, &threadFunc2, NULL) != 0)
{
printf("create thread 2 failed\n");
return -1;
}
pthread_join(hThread1, NULL);
pthread_join(hThread2, NULL);
pthread_rwlock_destroy(&m_lock);
return 0;
}
If read first, there is no deadlock problem and it can be executed normally.
thread 1 running..
thread 1 read source A=0
thread 2 running..
thread 1 read source B=0
thread 2 write source A and B
Through the above experiment, you need to be very cautious when the reader lock needs to re-enter. Once the attribute of the read-write lock is write priority, a deadlock is likely to occur.
Summarize
- Read-write locks are suitable for scenarios where there is more reading and less writing. In this scenario, there may be some performance gains.
- There are some traps in the use of read-write locks. Usually, try to use mutex locks instead of read-write locks.