Spring Boot+SQL/JPA combat pessimistic locking and optimistic locking

Original address: Spring Boot+SQL/JPA combat pessimistic lock and optimistic lock

---

Recently, I have encountered concurrency problems in the company's business, and it is still a very common concurrency problem, which is considered a low-level mistake. Since the company's business is relatively complex and not suitable for public disclosure, here is a very common business to restore the scene, and at the same time introduce how pessimistic locks and optimistic locks solve such concurrency problems.

The company's business is the most common "order + account" problem. After solving the company's problem, I turned around and thought that my blog project Fame also has the same problem (although the traffic does not need to consider the concurrency problem at all...) , then I'll take this as an example.

business recovery

The first environment is: Spring Boot 2.1.0 + data-jpa + mysql + lombok

Database Design

For a blog system with comment function, there are usually two tables: 1. Article table 2. Comment table. In the article table, in addition to saving some article information, etc., there is also a field to save the number of comments. We design a minimal table structure to restore this business scenario.

article article table

field	Types of	Remark
id	INT	auto increment primary key id
title	VARCHAR	article title
comment_count	INT	The number of comments on the article

comment comment form

field	Types of	Remark
id	INT	auto increment primary key id
article_id	INT	commented article id
content	VARCHAR	comments

When a user comments, 1. Get the article according to the article id 2. Insert a comment record 3. The number of comments on the article is increased and saved

Code

First, introduce the corresponding dependencies in maven

<parent>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-parent</artifactId>
    <version>2.1.0.RELEASE</version>
    <relativePath/> <!-- lookup parent from repository -->
</parent>

<dependencies>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-web</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-data-jpa</artifactId>
    </dependency>
    <dependency>
        <groupId>mysql</groupId>
        <artifactId>mysql-connector-java</artifactId>
    </dependency>
    <dependency>
        <groupId>org.projectlombok</groupId>
        <artifactId>lombok</artifactId>
        <optional>true</optional>
    </dependency>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-test</artifactId>
        <scope>test</scope>
    </dependency>
</dependencies>

Then write the entity class corresponding to the database

@Data
@Entity
public class Article {
    @Id
    @GeneratedValue(strategy = GenerationType.IDENTITY)
    private Long id;

    private String title;

    private Long commentCount;
}

@Data
@Entity
public class Comment {
    @Id
    @GeneratedValue(strategy = GenerationType.IDENTITY)
    private Long id;

    private Long articleId;

    private String content;
}

Then create the Repository corresponding to these two entity classes. Since spring-jpa-data CrudRepositoryhas already helped us implement the most common CRUD operations, our Repository only needs to inherit the CrudRepositoryinterface and nothing else needs to be done.

public interface ArticleRepository extends CrudRepository<Article, Long> {
}

public interface CommentRepository extends CrudRepository<Comment, Long> {
}

Then we simply implement the Controller interface and the Service implementation class.

@Slf4j
@RestController
public class CommentController {

    @Autowired
    private CommentService commentService;

    @PostMapping("comment")
    public String comment(Long articleId, String content) {
        try {
            commentService.postComment(articleId, content);
        } catch (Exception e) {
            log.error("{}", e);
            return "error: " + e.getMessage();
        }
        return "success";
    }
}

@Slf4j
@Service
public class CommentService {
    @Autowired
    private ArticleRepository articleRepository;

    @Autowired
    private CommentRepository commentRepository;

    public void postComment(Long articleId, String content) {
        Optional<Article> articleOptional = articleRepository.findById(articleId);
        if (!articleOptional.isPresent()) {
            throw new RuntimeException("没有对应的文章");
        }
        Article article = articleOptional.get();

        Comment comment = new Comment();
        comment.setArticleId(articleId);
        comment.setContent(content);
        commentRepository.save(comment);

        article.setCommentCount(article.getCommentCount() + 1);
        articleRepository.save(article);
    }
}

Concurrency problem analysis

The process of this simple comment function can be seen from the code implementation just now. When the user initiates a comment request, the entity class of the corresponding article is found from the database Article, and then the corresponding comment entity class is generated according to the article information Comment, and inserted into In the database, then increase the number of comments of the article, and then update the revised article to the database. The whole process is as follows.

There is a problem in this process. When there are multiple users concurrently commenting at the same time, they simultaneously enter step 1 to get the Article, then insert the corresponding Comment, and finally update the number of comments in step 3 and save it to the database. It's just because they obtained the Articles in step 1 at the same time, so their Article.commentCount values ​​are the same, then the Article.commentCount+1 saved in step 3 is also the same, then the original number of comments that should be +3 is only added 1.

Let's try it with the test case code

@RunWith(SpringRunner.class)
@SpringBootTest(classes = LockAndTransactionApplication.class, webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT)
public class CommentControllerTests {
    @Autowired
    private TestRestTemplate testRestTemplate;

    @Test
    public void concurrentComment() {
        String url = "http://localhost:9090/comment";
        for (int i = 0; i < 100; i++) {
            int finalI = i;
            new Thread(() -> {
                MultiValueMap<String, String> params = new LinkedMultiValueMap<>();
                params.add("articleId", "1");
                params.add("content", "测试内容" + finalI);
                String result = testRestTemplate.postForObject(url, params, String.class);
            }).start();
        }

    }
}

Here we open 100 threads and send comment requests at the same time, and the corresponding article id is 1.

Before sending the request, the database data is

select * from article

select count(*) comment_count from comment

After sending the request, the database data is

select * from article

select count(*) comment_count from comment

Obviously, the value of comment_count in the article table is not 100. This value is not necessarily 14 in my picture, but it must be no greater than 100, and the number of comment tables must be equal to 100.

This shows the concurrency problem mentioned at the beginning of the article. This kind of problem is actually very common. As long as there is a system with a process like the above comment function, you must be careful to avoid this problem.

The following is an example to show how to prevent concurrent data problems through pessimistic locking and optimistic locking. At the same time, the SQL solution and JPA's own solution are given. The SQL solution can be used in "any system" or even any language, while the JPA solution is very fast. If you happen to be using JPA too, you can simply use optimistic or pessimistic locking. Finally, we will compare some differences between optimistic locks and pessimistic locks according to the business.

Pessimistic locks solve concurrency problems

Pessimistic locks, as the name implies, are pessimistic that the data they operate will be operated by other threads, so they must monopolize the data themselves, which can be understood as "exclusive locks". In java, synchronizedand ReentrantLockwait locks are pessimistic locks, and table locks, row locks, read-write locks, etc. in the database are also pessimistic locks.

Using SQL to solve concurrency problems

Row lock is to lock this row of data when operating data. Other threads must wait to read and write, but other data in the same table can still be operated by other threads. As long as you add it after the sql to be queried for update, you can lock the row of the query. In particular, pay attention to the fact that the query condition must be an index column. If it is not an index, it will become a table lock, locking the entire table.

Now, modify it on the basis of the original code, and first ArticleRepositoryadd a manual sql query method.

public interface ArticleRepository extends CrudRepository<Article, Long> {
    @Query(value = "select * from article a where a.id = :id for update", nativeQuery = true)
    Optional<Article> findArticleForUpdate(Long id);
}

Then change CommentServicethe query method used in the original findByIdto our custom method

public class CommentService {
    ...
    
    public void postComment(Long articleId, String content) {
        // Optional<Article> articleOptional = articleRepository.findById(articleId);
        Optional<Article> articleOptional = articleRepository.findArticleForUpdate(articleId);
    
    	...
    }
}

In this way, we found out Articlethat other threads cannot acquire and modify it before we commit it to the transaction, which ensures that only one thread can operate the corresponding data at the same time.

Now test it again with the test case, article.comment_countthe value must be 100.

Use JPA's own row lock to solve concurrency problems

For the addition after sql mentioned just now for update, JPA provides a more elegant way, that is @Lock, annotation. The parameters of this annotation can be passed in the desired lock level.

Now ArticleRepositoryadd the JPA lock method in , where the LockModeType.PESSIMISTIC_WRITEparameter is the row lock.

public interface ArticleRepository extends CrudRepository<Article, Long> {
    ...
    
    @Lock(value = LockModeType.PESSIMISTIC_WRITE)
    @Query("select a from Article a where a.id = :id")
    Optional<Article> findArticleWithPessimisticLock(Long id);
}

Similarly, as long as CommentServicethe query method is changed to here findArticleWithPessimisticLock(), and then the test case is tested, there will be no concurrency problems. And at this time, I looked at the console print information and found that the sql query was actually added for update, but JPA added it for us.

Optimistic locking solves concurrency problems

Optimistic locking, as the name implies, is very optimistic. It thinks that the resources obtained by itself will not be operated by other threads, so it is not locked. It is only to determine whether the data has been modified when inserting into the database. Therefore, pessimistic locking restricts other threads, while optimistic locking restricts itself. Although his name has a lock, it is not actually locked. It is only to determine how to operate at the final operation.

Optimistic locking is usually a version number mechanism or a CAS algorithm

Using SQL to implement version numbers to solve concurrency problems

The version number mechanism is to add a field to the database as a version number, for example, we add a field version. Then when you get it at this time Article, you will bring a version number. For example, the version you get is 1, and then you Articleoperate on this one pass, and you will insert it into the database after the operation. I found that the Articleversion in the database is 2, which is different from the version in my hand, which means that the version in my hand is Articlenot the latest, so it cannot be placed in the database. This avoids the problem of data conflict during concurrency.

So we now add a field version to the article table

article article table

field	Types of	Remark
version	INT DEFAULT 0	version number

Then the corresponding entity class also adds the version field

@Data
@Entity
public class Article {
	...
    
    private Long version;
}

Then ArticleRepositoryadd the update method. Note that this is the update method, which is different from the query method when pessimistic locks are added.

public interface ArticleRepository extends CrudRepository<Article, Long> {
    @Modifying
    @Query(value = "update article set comment_count = :commentCount, version = version + 1 where id = :id and version = :version", nativeQuery = true)
    int updateArticleWithVersion(Long id, Long commentCount, Long version);
}

You can see that the where of the update has a condition for judging the version, and will set version = version + 1. This ensures that the data will only be updated if the version number in the database is the same as the version number of the entity class to be updated.

Then CommentServicemodify the code slightly.

// CommentService
public void postComment(Long articleId, String content) {
    Optional<Article> articleOptional = articleRepository.findById(articleId);

    ...	

    int count = articleRepository.updateArticleWithVersion(article.getId(), article.getCommentCount() + 1, article.getVersion());
    if (count == 0) {
        throw new RuntimeException("服务器繁忙,更新数据失败");
    }
    // articleRepository.save(article);
}

First of all Article, the query method only needs ordinary findById()methods without any locks.

Then Articleuse the newly added updateArticleWithVersion()method when updating. You can see that this method has a return value. This return value represents the number of updated database rows. If the value is 0, it means that there are no rows that meet the conditions that can be updated.

After that, we can decide how to deal with it. Here is a direct rollback. Spring will help us roll back the previous data operations and cancel all operations this time to ensure data consistency .

Now use the test case to test

select * from article

select count(*) comment_count from comment

Now I see Articlethat the number of comment_count and Commentthe number in it is not 100, but the value of these two must be the same . Because if there is a conflict in the data of the table when we were dealing with it Article, then it will not be updated to the database. At this time, an exception is thrown to make the transaction roll back, so as to ensure that it will not be inserted Articlewhen there is no update, and it is solved. CommentThe problem of inconsistent data.

This direct rollback processing method has poor user experience. Generally speaking, if it is judged Articlethat the number of updates is 0, it will try to query the information from the database again and modify it again, and try to update the data again. until updated. Of course, it will not be an operation such as a wireless loop. An online line will be set. For example, if it is not possible to query, modify, and update the loop three times, an exception will be thrown at this time.

Use JPA to implement version now to solve concurrency problems

JPA has a way to implement pessimistic locks, and naturally there are also optimistic locks. Now we use the methods that come with JPA to implement optimistic locks.

First Articleadd an annotation to the version field of the entity class @Version. Let's look at the annotation of the source code. You can see that some of it is written:

The following types are supported for version properties: int, Integer, short, Short, long, Long, java.sql.Timestamp.

The note says that the type of the version number supports three basic data types of int, short, long and their wrapper classes and Timestamp, we are using the Long type now.

@Data
@Entity
public class Article {
    ...
    
    @Version
    private Long version;
}

Then, we just need to CommentServicemodify the comment process in it back to the business code that "will trigger concurrency problems" at the beginning. It shows that this optimistic locking implementation of JPA is non-intrusive.

// CommentService
public void postComment(Long articleId, String content) {
    Optional<Article> articleOptional = articleRepository.findById(articleId);
    ...

    article.setCommentCount(article.getCommentCount() + 1);
    articleRepository.save(article);
}

As before, use test cases to test whether concurrency problems can be prevented.

select * from article

select count(*) comment_count from comment

ArticleThe number of comment_count and the same in the same Commentis not 100, but the two values ​​are definitely the same. Take a look at IDEA's console and you will find ObjectOptimisticLockingFailureExceptionexceptions thrown by the system.

This is similar to our own implementation of optimistic locking just now. If the data is not successfully updated, an exception will be thrown and the data will be rolled back to ensure data consistency. If you want to realize the retry process to catch ObjectOptimisticLockingFailureExceptionthis exception, you usually use AOP + custom annotations to implement a global general retry mechanism. Here is to expand according to the specific business situation. If you want to know more, you can search for the solution by yourself. .

Pessimistic locking and optimistic locking comparison

Pessimistic locks are suitable for scenarios with more writes and fewer reads . Because the thread will monopolize this resource when it is used, in the example of this article, it is an article with a certain id. If there are a large number of comment operations, it is suitable to use pessimistic lock. Otherwise, if the user just browses the article and has no comments, Using pessimistic locking will often lock, which increases the resource consumption of locking and unlocking.

Optimistic locking is suitable for scenarios where write less and read more . Since optimistic locks will be rolled back or retried when a conflict occurs, if the number of write requests is large, conflicts will often occur, and frequent rollbacks and retries will consume a lot of system resources.

Therefore, pessimistic locks and optimistic locks are not absolutely good or bad , and which method to use must be decided based on specific business conditions. In addition, it is also mentioned in the Alibaba development manual:

If the probability of each access conflict is less than 20%, optimistic locking is recommended, otherwise pessimistic locking is used. The number of retries for optimistic locking shall not be less than 3 times.

Alibaba recommends using the value of the conflict probability of 20% as the dividing line to decide the use of optimistic locks and pessimistic locks. Although this value is not absolute, it is also a good reference as summarized by Alibaba's bigwigs.

{{o.name}}

{{m.name}}