The reason for writing this article is that when I completed the business requirements during my work yesterday, I wrote my own code according to the requirements of the team leader. As a result, when the team leader reviewed later, he found a lot of deficiencies, so I wrote this article. blog for the record.
Without further ado, let's start with the original code.
public class DocValuesWarmUpListener extends AbstractSolrEventListener{
private static final Logger logger = LoggerFactory.getLogger(DocValuesWarmUpListener.class);
private List<String> fields;
public DocValuesWarmUpListener(SolrCore core) {
super(core);
}
@Override
public void init(NamedList args) {
super.init(args);
fields = new ArrayList<>();
NamedList o;
if(args != null) {
o = (NamedList) args.get("fields");
if(o!=null && o instanceof NamedList) {
for (int i = 0; i < o.size(); i++) {
fields.add((String) o.getVal(i));
}
}
}
}
@Override
public void newSearcher(SolrIndexSearcher newSearcher, SolrIndexSearcher currentSearcher) {
long start = System.currentTimeMillis();
ExecutorService executorService = Executors.newFixedThreadPool(fields.size());
try {
//The fields required for sorting are warmed up with a thread pool and multiple threads
for(int i=0;i<fields.size();i++) {
executorService.execute(new WarmUpDocValueThread(fields.get(i),newSearcher));
}
} catch (Exception e) {
throw new IllegalStateException(e);
}finally {
logger.info("warming up consumes ["+(System.currentTimeMillis()-start)+"] Millis");
}
}
class WarmUpDocValueThread implements Runnable{
private String field;
private SolrIndexSearcher newSearcher;
public WarmUpDocValueThread(String field,SolrIndexSearcher newSearcher) {
this.field = field;
this.newSearcher = newSearcher;
}
@Override
public void run() {
try {
DocValues.getSorted(this.newSearcher.getLeafReader(), this.field);
} catch (IOException e) {
e.printStackTrace ();
}
}
}
}
Here first briefly explain the business requirements of this code.
When the business side conducts a query, it is found that the time jitter of a query statement is very large, the longest time may be 6000ms, and the fast time is about 400ms. Therefore, the group leader's analysis is that the query statement involves sorting, and after Solr's periodic softCommit, the entire docValue needs to be re-merged, which will cause periodic jitter. Therefore, it is necessary to warm up the docValue after softComiit.
After determining the logic, let me start to complete this logic. The whole code logic is not complicated, just write an eventListener, initialize the configuration of solrconfig.xml in the init() method, and warm up the docValue in the newSearcher() method.
So I wrote the above version. When the team leader reviewed it, I found a lot of deficiencies. I will record it now and try to avoid it in the future.
1: In the case of multi-thread concurrency, the blockage of the main thread is not set, and in the case of pre-heating docValue by multiple threads at the same time, the main thread does not wait for the pre-heating to complete and continues to execute directly. In response to this problem, in the first version of the modification, after multi-threading is enabled, thread.join() is added to complete the waiting of the main thread, and it is modified to countDownLatch at the suggestion of the team leader. By the way, here is the difference between countDownLatch and join usage.
In essence, countDownLatch and join both achieve the same purpose, that is, when multi-threaded concurrent programming, when each thread needs to wait for specific other thread tasks to complete before executing, the above two methods are used. The specific usage will not be repeated here, but different usage scenarios will be discussed.
When using join, the blocking thread must wait for the task of other threads to complete before continuing to execute, while when using countDownLatch, the execution degree of other threads can be arbitrarily specified.
For example, at this time, there are three threads A, B, and C to execute tasks. Suppose that the three thread tasks are all executed with a count of 100, and the A thread must wait until the B and C counts are completed before executing. At this time, use join or countDownLatch. Able to complete task requirements. However, if the task is changed, A thread must wait until B and C count to 50 before starting to execute. At this time, join cannot complete this requirement, but it can be achieved when countDownLatch is used, that is, first
CountDownLatch cdl = new CountDownLatch(2) //A thread cld.await() //When thread B completes 50 counts cld.countDown() //When the C thread completes 50 counts cld.countDown()
The above code can achieve the requirements. The above is only pseudo code. In actual programming, pay attention to writing cld.countDown() in finally to avoid deadlock caused by a thread crash.
The second deficiency lies in the degree of configurability of the code. The first version of the code writes the code parameters too rigidly, and it is difficult to reuse the code in the future. After being prompted by the team leader, I read the source code of AbstractSolrEventListener and found that there is an init method, you can read the configuration parameters of the schema.xml file, resolutely override this method, and configure the fields that need to be warmed up in the xml file.
This article mainly records this experience.