"Druid source code analysis and study (including detailed monitoring of egg design ideas)."

druid source code analysis and study (including detailed monitoring of the design ideas of eggs)

Blog Category:

 
• druid
• java
• database
• Source

druid java database 

    Druid is Alibaba's database connection pool tool, yesterday suddenly want to learn about Ali's druid source code, so the download of a moment. Also more than two hours cursory looked at some middle point of knowledge have not seen, do not know, now check BAIDU learning. Briefly summarize, summarize edge side continue to look at the code, it is estimated a lot of mistakes, please correct me!

    Prior to yourself, looking to find druid source code analysis, actually search on BAIDU no information, but describes how to configure, to fend for themselves over here. Here are the details do not say, focus on the general direction and design ideas.

1. Monitoring is born, so when, and how to monitor it?
    Simple operation usually involves a database datasource, connection, preparedstatement, ResultSet other things, if I want to monitor these, some are bound to build the proxy class.
   Our operating by proxy class is completed, the process is completed, the generated data monitoring.
   druid known as the monitoring and health monitoring function is a needle, we must attack the saying goes, there is no gap would create a gap, so we'll build a proxy class, between classes and the proxy agency management is the gap. The proxy object is bound to hold the proxy object.
    public interface PreparedStatementProxy extends PreparedStatement, StatementProxy
    as the implementation class: PreparedStatementProxyImpl, will hold a java.sql.PreparedStatement.
    Just look at its query method:
    public ResultSet executeQuery () throws SQLException {
.....
        return createChain () preparedStatement_executeQuery (this) ;. // chain produced was filtered by the filter chain execution.
    }
  FilterChainImpl are:

Java code  

1. public ResultSetProxy preparedStatement_executeQuery(PreparedStatementProxy statement) throws SQLException {  
2.     if (this.pos < filterSize) {  
3.         return nextFilter().preparedStatement_executeQuery(this, statement);  
4.     }  
5.     ResultSet resultSet = statement.getRawObject().executeQuery();  
6.     return wrap(statement, resultSet);  
7. }  

    The method described above: before executing the query, the processing chain to be filtered, etc. finished processing, and then executed by the Statement, finished execution, the ResultSet obtained after a, generates a proxy class final package returned.

2. Talk about statistical filter it, just a ring on the filter chain
   this mode to see the most is configured in web.xml filter filter, you configure several filters filter, are realized dofilter () method, so they organized into the filter, the container and the like on the List filterchain in, it can be executed cyclically dofilter to deal with some stuff. It can be seen filterchain hold all of the filter, that filter after the execution, to tell filterchain to the next, so fiter must hold filterchain. But filter need not always hold filterchain objects, only temporary hold for a while, so filterchain as a parameter passed in the method. This also shows that the filter may be part of the chain, the chain may be another link in the same time, who will do.
This process, it points like the observer pattern, but also as a callback mode.
   Take a simple method of connecting a statistical StatFilter submitted: (StatFilter in Super Multi filtration method, various database operations accounting; of course logfilter in the same defense SQL injection attacks WallFilter, estimated as: )

Java code  

1. @Override  
2. public void connection_commit(FilterChain chain, ConnectionProxy connection) throws SQLException {  
3.     chain.connection_commit(connection);  
4.     JdbcDataSourceStat dataSourceStat = chain.getDataSource().getDataSourceStat();  
5.     dataSourceStat.getConnectionStat().incrementConnectionCommitCount();  
6. }  

  First, let the chain go for step (nextFilter is beginning to work, all of the filter have finished the job, you really commit it) and then, to commit operation count data sources be increased.

Java code  

1. public void connection_commit(ConnectionProxy connection) throws SQLException {  
2.     if (this.pos < filterSize) {  
3.         nextFilter () connection_commit (. the this, Connection); next // make a work  
4.         return;  
5.     }  
6.     connection.getRawObject () the commit ();. // all be over before they really commit. This connection is also a proxy, which allows real java.sql.connection submitted.  
7. }  
8. private Filter nextFilter() {  
9.     Filter filter = getFilters().get(pos++);  
10.     return filter;  
11. }  

DataSourceProxyConfig in a private final List <Filter> filters = new ArrayList <Filter> (); // common arraylist is let filter.

The above two points together, is the original perform a database operation, now run to the agent class, go through the execution of a statistical filters, after then actually perform database operations. Transparent to the end user.

3. The statistical record of how stuff it?
In the stat package inside, just find an object to see it. For example: JdbcStatementStat
..........................

Java code  

1. Private  Final AtomicLong createCount =  new new AtomicLong ( 0);                                      // execution count of createStatement  
2. Private  Final AtomicLong prepareCount =  new new AtomicLong ( 0);                                      // execution count of parepareStatement  
3. Private  Final AtomicLong prepareCallCount =  new new AtomicLong ( 0);                                      // execution count of preCall  
4. private final AtomicLong    closeCount       = new AtomicLong(0);   

.....................................
Oh wow, a lot of statistics counters, all of AtomicLong call it incrementAndGet () method, that is, thread synchronization, increase the count. But in TableStat is an ordinary int, huh, huh.

4. The blue words to talk about the top of the execution
ResultSet resultSet = statement.getRawObject () executeQuery ();.
     PreparedStatementProxy agent of an object that implements java.sql.statement interface, so it executes the query using that object. That object is what it?
     By the way, it is this: DruidPooledPreparedStatement, look out, this guy is a proxy to someone else, because it uses stat to do a query, just before and after the agent did some other things.

Java code  

1. public ResultSet executeQuery() throws SQLException {  
2.     checkOpen();  
3.     incrementExecuteCount ();  
4.     transactionRecord(sql);  
5.     oracleSetRowPrefetch();  
6.     conn.beforeExecute();  
7.     try {  
8.         ResultSet rs = stmt.executeQuery();  
9.         if (rs == null) {  
10.             return null;  
11.         }  
12.         DruidPooledResultSet poolableResultSet = new DruidPooledResultSet(this, rs);  
13.         addResultSetTrace(poolableResultSet);  
14.         return poolableResultSet;  
15.     } catch (Throwable t) {  
16.         throw checkException(t);  
17.     } finally {  
18.         conn.afterExecute();  
19.     }  
20. }  

    Where: addResultSetTrace is the result of a query on the List <ResultSet> resultSetTrace; in. Why?

5. look at a few Holder is doing?
What DruidConnectionHolder there? When new DruidPooledConnection, just use this holder.

Java code  

1.     private final DruidAbstractDataSource       dataSource;  
2.     private final Connection                    conn;  
3.     private final List<ConnectionEventListener> connectionEventListeners = new CopyOnWriteArrayList<ConnectionEventListener>();  
4.     private final List<StatementEventListener>  statementEventListeners  = new CopyOnWriteArrayList<StatementEventListener>();  
5.     PreparedStatementPool statementPool Private;    // This is a pool LRU algorithm. The following is put PreparedStatementHolder !!!!  
6.     private final List<Statement>               statementTrace           = new ArrayList<Statement>(2);  
7.   
8. PreparedStatementHolder in it? When new DruidPooledPreparedStatement, just use this holder.  
9.     private final PreparedStatementKey key;  
10.     private final PreparedStatement    statement;  

Holder is what holds the name of view, DruidConnectionHolder necessarily hold Connection, PreparedStatementHolder necessarily hold PreparedStatement. DruidConnectionHolder course, also holds a part of the connection of the PreparedStatement class. Through several calls relationship, you can almost guess the design:
Generally, we connect with the object, and then generate statment objects, and then execute SQL-like, when we do something like the statistics before and after the implementation of an object, then a proxy object to do, for example, in front of filterchain proxy object. But if calls to other objects, trying some of the association between them stuff preserved, such as a PreparedStatement under all connections, it would need a holder objects to help. Maybe you can put a bunch of other stuff gave him the object, but this is not clear, too messy.
Probably holder can also be called a design pattern. Of course proxy also, remember that's it called handler, and have behavioral sense! !
The DruidPooledConnection PreparedStatement prepareStatement (String sql) method, is to look at the pool there (stmtHolder = holder.getStatementPool () get ( key);.), No word was new PreparedStatementHolder (key, conn.prepareStatement (sql )); , any vessel to take the memory.

Probably this relationship: DruidPooledConnection -> DruidConnectionHolder -> ConnectionProxy -> filterChain --- connection.

6. talk about LRU cache used above, is to keep the pool PreparedStatementHolder.
  One kind LinkedHashMap it, his realization of what removeEldestEntry method on it, throw away the capacity to achieve most of OLD.

Java code  

1. public class LRUCache extends LinkedHashMap<PreparedStatementKey, PreparedStatementHolder> {  
2.     private static final long serialVersionUID = 1L;  
3.     public LRUCache(int maxSize){  
4.         super(maxSize, 0.75f, true);  
5.     }  
6.     protected boolean removeEldestEntry(Entry<PreparedStatementKey, PreparedStatementHolder> eldest) {  
7.         boolean remove = (size() > dataSource.getMaxPoolPreparedStatementPerConnectionSize());  
8.         if (remove) {  
9.             closeRemovedStatement(eldest.getValue());  
10.         }  
11.         return remove;  
12.     }  
13. }  

7.MOCK packages
  that do not understand, online search a bit, saying that some of the fraud, and facilitate the testing of the object, which implements the relevant interface, so it can be something as fake use. In particular, the real thing is not convenient to use, or slow, or have other undesirable situation.

8, sqlPaser
  main function package under the druid addition to this, are introduced over, say SQL parser, but no time to read, read the next time to add it.

9.connectPool connection pool
  connection pool is the highlight of the course, simple to mention, is mainly used ReentrantLock lock, there are two conditions notEmpty empty, the thread connecting production and consumption in the wake wait for a connection on two conditions. Connection pool is determined by the data source, the two classes of the pool depends on the specific bag with DruidAbstractDataSource DruidDataSource.
Wow, these two classes is very large, first looked at the property, there are a lot of count, time, and some default value, pay attention to the set of fields, such as the Map <DruidPooledConnection, Object> activeConnections
Private volatile DruidConnectionHolder [] Connections; like . Which has some threads.
9.1 create a connection
  to look DruidDataSource in the CreateConnectionThread are doing, first of a number of conditions, such as the following codes (omitting unimportant code), connection too much time waiting in the empty condition, and so is the empty run, now do not rush to create a connection, wait!
// prevent the creation of connections exceeds the number maxActive

Java code  

1.         if (activeCount + poolingCount >= maxActive) {  
2.             empty.await();  
3.             continue;  
4.         }  
5. try {  
6.     connection = createPhysicalConnection();  
7.     setFailContinuous ( false);  
8. boolean result = put(connection);  

Behind is to create a physical connection, and then put it, this is likely to be put in place pond, then a closer look. The following few major, behind the instructions are written in:
    Holder = new new DruidConnectionHolder (DruidDataSource.this, physicalConnectionInfo); // generate a connection Holder
            Connections [poolingCount] = Holder; // Is not that pool it? It is a DruidConnectionHolder of the Array.
            incrementPoolingCount (); // count plus pond 1.
            notEmpty.signal (); // issued by a non-empty signal on a non-empty condition of all the waiting threads, you may move up.
            ++ notEmptySignalCount;
9.2 using the connection
    then who wait on notEmpty conditions? We checked and found that the method DruidConnectionHolder takeLast (), awaiting the poolingCount when the number is zero. It goes to show that use a threaded connection, and when there is no connection, just waiting for the pictures. If the pool has connected it? Executes the following statement:
        decrementPoolingCount (); // decrement the count of connections in the pool, at least one of a course away.
        DruidConnectionHolder last = connections [poolingCount]; // just take is the last cell.
        connections [poolingCount] = null; // the last to become null.
    Way to see who is using takeLast (), find the find is getPooledConnection (), from the name to know is the main method of getting links. A closer look at getPooledConnection in turn calls getConnection (), which is inserted inside the filter chain, it really is for statistical born, have a record. Look carefully filterChain.dataSource_connect () parameters is this, explain it to pass and went in, that this filterChain does not belong to any datasource, this tool can be a data source, it can be that a data source. Which specific filter, temporary pass.
    When we design a filter chain, if our function is to serve the people, it shows that clients come to pass. Instead setpropety set relationship. How does one design a complex code? Of course, there is a very abstract mind, and clear thinking.
9.2 to reduce connection
is created in the vicinity there is a connection thread DestroyConnectionThread (), to see it
tracking inside, destroyTask.run (); -----> shrink ( true); see the name contraction Well, that may be connected idle too more, it reduced chant.
In the shrink () method, the focus there is the following statement, on the back analysis:
Final CheckCount int = poolingCount - minIdle; // pool number - the minimum number of idle, the feeling is one of the conditions contracted thing.

Java code  

1. for (DruidConnectionHolder Item: evictList) { // here are placed recoverable  
2.     Connection connection = item.getConnection();  
3.     JdbcUtils.close (Connection); // close these connected.  
4.     destroyCount.incrementAndGet();  
5. }  

9.3 initialization method init ()
is who is using these two threads do? Find it and found a thread is created with the contraction of the thread is void init () to call, see the name to know that this is the main method of system startup.
the init void () {
     initFromSPIServiceLoader (); // Load the ServiceLoader Filters from SPI, the spi not introduced, the analysis dubbo's another post, which has been introduced SPI. Because it is in the configuration of a filter in filterchain (List <Filter>)

Java code  

1. = Connections  new new DruidConnectionHolder [for maxActive]; // new connection pool, the maximum number of active connections maxActive.  
2.                 for ( int I =  0, size = getInitialSize (); I <size; I ++) { // placed in connection pool  
3.                     PhysicalConnectionInfo pyConnectInfo = createPhysicalConnection();  
4.                     DruidConnectionHolder holder = new DruidConnectionHolder(this, pyConnectInfo);  
5.                     connections[poolingCount] = holder;  
6.                     incrementPoolingCount ();  
7.                 }  
8.             createAndLogThread (); // see the name of the log, do not read  
9.             createAndStartCreatorThread (); // create a thread connections, has been at work, the pool is full of wait states.  
10.             createAndStartDestroyThread (); // shrinking pool of threads have been working.  
11.   
12.             initedLatch.await (); // main thread waits 0 before the counter.  
13.     init = true;  
14. }  

There is a point on the knowledge. CountDownLatch initedLatch = new CountDownLatch (2) ; called countdown synchronizer. Current sync number 2, after becomes 0, the main thread to run, otherwise it has been waiting.
In connection thread creation and contraction of the pool in both initedLatch.countDown () ;, a total of just two, then the main thread is waiting for the above two threads are running and we have to run it, just set init state identified as true. It seems to understand nothing wrong with myself.

Look create a single thread, which has a countDown method:

Java code  

1. protected void createAndStartCreatorThread() {  
2.     if (createScheduler == null) {  
3.         String threadName = "Druid-ConnectionPool-Create-" + System.identityHashCode(this);  
4.         createConnectionThread = new CreateConnectionThread(threadName);  
5.         createConnectionThread.start();  
6.         return;  
7.     }  
8.     initedLatch.countDown (); // if createScheduler directly -1;  
9. }  
10. public class CreateConnectionThread extends Thread {  
11.     public void run() {  
12.         initedLatch.countDown (); // If not, there is at 1;  

10. Summary:

Read the source code? How do we actually improve? We can also make such good stuff right?
1. The first thing you will master enough of the basics, such as would be used in multi-threaded, concurrent contents of the bag, even rarely used containers. Description of data read very thick book or seen a lot of java source code.
2. In-depth study of foreign source. I found some similar approach, appear in this product, also appear in the product, of course, not necessarily identical. Hadoop such as asynchronous and synchronous transfer dubbo almost, what fastfail was also used in the spike.
3. read the source code of the two, you should be able to abstract approach to the problem, so when their own encounter similar situations, we can immediately apply.
[Eggs]  
basically read through the main code, to monitor and students a deeper understanding of the phrase, we choose one point, in-depth experience the filter chain mode it can be regarded as the paste of eggs. This stuff is really in a lot of places, so how do we use it? When used, there should be three objects, which are objects to be processed, the filter, the filter chain, previous post I make a point, the nature of the object is a combination of relationships, the combination is the most important. Then the inclusion relation between these two objects, reference relations, as well as the main approach should be how to design, and why such a design it?
Is a basic unit of the filter, characterized by: that it does not refer to the filter chain, since it can belong to different filter chain. It does not deal with a reference object because it can handle this object can also handle that object. Therefore, in these three, the filter filtration method, the two additional objects will pass, without generating other attributes and methods. dofilter parameters of the filter web is req and res chain and three.
Chains are generally available filter assembly, the filter is a basic unit, the configuration of the chain to give the user the opportunity, after the filter may be increased, but this is not a function of the user can add to the mix through mechanisms of spi.
Chain is a filter combiner, which is characterized by: when generating the filter, the filter must be substantially passed in, and reference relationships are stable, but because there are a plurality of filters to the chain, so that they have a discharge container . Therefore, a container holding the filter chain, init when placed in one filter. Filters are executed one by one to string together, so there is also a location information, go there now perform a a. Then a moment's thought, object filtering process chains, some processing to the first two, to deal with some of the first five, and some processing to the first three. That multi-threaded problem came. Check codes (protected int pos = 0; if (this.pos <filterSize) donext), a count is not positioned pos opinion threadlocal ah, look public FilterChainImpl (DataSourceProxy dataSource), the constructor is passed a data source, Therefore, a data source does not count the incoming pointers, will pass over the filter container. I want to check back in the filter web source, what is inside the container, positioning is what? Is not threadlocal variables?
(Look up later in the tomcat found here: org.apache.catalina.core.ApplicationFilterChain the properties private ApplicationFilterConfig [] filters = new ApplicationFilterConfig [0]; private int pos = 0; and afterFilter beforeFilter also contains other functions, these authors certainly learn to filtering code, and I'm too young, and to see the little things)
A bit complicated, take a look at FilterChainImpl, which does not hold the filter unit, and important method which, getFilters, nextFilter, are coming from the dataSource in the construction chain. What is a bit like it? As before you go to a restaurant for dinner, now let you take food restaurants processing. Ah makes sense, because the filter chain rule does not have to hold the basic unit of ah, like bubble sort is not necessarily held by collating elements, it can be abstracted out alone. Since the filter chain contains a non-threaded pos, then each filter chain are one-time use, or pos pointer into chaos. So to find what works in all new FilterChainImpl (), we found that there are four pool bag, the other in the proxy bag, just to prove that they must attack said earlier, came up with a proxy, to be inserted between the agent and the agent chain filter, or whatever you want. For example ClobProxyImpl have createChain () method, and each clob operation should call createChain, means that each operation is a new filter chain, then the aforementioned pos counts is only their own method, there will be no sharing the conflict took place.

Let us look at management ideas, three main elements (objects to be processed, the filter, the filter chain), wherein the filter chain is split, leaving only the filtering rules and counters. Filter container to hold the object to be processed, and processing variations. Each filter chain is a proxy object that is accurate to say that each specific filtration methods to be used for each proxy object temporarily create out. Li a further idea, we look at a filter, to watch interface objects, found that there are a lot of methods to connect process, there are a lot of methods resultset process, there are a lot of methods statemente process with process .. ...

Oh wow, too much, all of them here, each filter is very large, very round, heavyweight objects, only one. A data source holding several filters, filter chain is not an elongated shape, but chunky shape. And your any object (of course, a proxy object myself) any one of a simple operation, generates a filter chain. So you should not hold the filter chain filter, each holding a simple count pos is very reasonable, filtering chain is a lightweight object.

And then extended to think about, if we start to think of a function, for example, such a monitoring function, we can be sure knows how to filter the way, as we can so organize the relationships between these objects? And so there is nothing further tissue remodeling could it be?
We may have a number associated with the connection of the filter, and a number of related resultset filter; from another point of view, we have a statistical filter, a filter log. We also filter chain rule, we have a lot of objects to be filtered, you can break up such rules and filter chain filter container chain are separated. Try combination, you can be combined by connecting (connection statistics, logs connected together) may be combined (connected to the log with the log resultset) functionally. So we pluggable either the entire statistics, either the entire log. If another combination, log statistics and log as long as we can connect, as long as you can query and statistics. But if you want to connect to statistics, logs to be queried, then the druid is currently not supported. This is something like split into pieces, and then the feeling of building blocks.
Another example, now by the datasource to hold the filter vessel. Then the size of the filter is only different data sources. Some filters is connect, and some is resultset, all the subordinate object data source is the same filter container, in this area there is no demand for personalization of it? For example, I just want to clob do not want to log statistics, createChain time is not what parameters you can configure it?

In addition, when looking at FilterChainImpl, in addition to its large number of methods to filter, there are several wrap method. The wrap is just the original object is to generate a proxy object filtering gap while the conditions are generated filterchain pass it, so that each proxy object execution method may be a new new filterchain out. When a specific method in each proxy object, call filterchain to deal with, the agent himself again to filterchain, filterchain implementation process, but also with the proxy object getRawObject get the proxy object to perform a final service. If the proxy object is continuously performed three different methods, the first time that a new FilterChain, a chain will be finished processing recycleFilterChain (chain); pos reset count back to two methods or use of the proxy object holds filterchain, but the count becomes 0.

Haircut then: I have an original object, produced by FilterChainImpl proxy object, the proxy object is then passed to a specific method FilterChainImpl, the method which was filtered before removing the original object to use. Analogy again: I have a stainless steel cup, to let others wrapped in mud, has become a mud cup, let someone else do the firing, coloring, characterization and other treatment. Eventually became fine ceramic cups, of course, drink plenty of water features or rely on the original stainless steel cup to achieve, because it is not without stainless steel kitchen knife. Of course, the knife may be wrapped mud, do firing, coloring, etc. characterization, and finally became a ceramic tool. Mud pack and firing, coloring, of course, is portrayed different business, but you can open a shop to do.

Then the full sum up:
About druid, configure a data source, which holds a pile of filter components, hold a connection pool, connection pool has two main threads running. When I do anything on any object, give me a packaged, re-create or reset a misplaced chain and filtering process and then perform an operation, it is Jiang Zi.

• View Image Attachment