Apache-Ignite-2.0.0持久化

16.持久化

16.1.持久化存储

16.1.1.摘要

JCache提供了javax.cache.integration.CacheLoader和javax.cache.integration.CacheWriterAPI，他们分别用于底层持久化存储的通读和通写（比如RDBMS中的Oracle或者MySQL，以及NoSQL数据库中的MongoDB或者CouchDB）。 
虽然Ignite可以单独地配置CacheRLoader和CacheWriter，但是在两个单独的类中实现事务化存储是非常尴尬的，因为多个load和put操作需要在同一个事务中的同一个连接中共享状态。为了缓解这个问题，Ignite提供了·org.apacche.ignite.cache.store.CacheStore·接口，他同时扩展了CacheLoader和CacheWriter。

事务 
CacheStore是完整事务性的，他会自动地融入当前的缓存事务。 
CacheJdbcPojoStore 
Ignite附带了他自己的CacheJdbcPojoStore,他会自动地建立Java POJO和数据库模式之间的映射，可以参照3.13.自动持久化章节。

16.1.2.通读和通写

当希望通读和通写行为时，提供一个正确的缓存存储的实现是很重要的。通读意味着当缓存无效时会从底层的持久化存储中读取，通写意味着当缓存更新时会自动地进行持久化。所有的通读和通写都会参与整体的缓存事务以及作为一个整体提交或者回滚。 
要配置通读和通写，需要实现CacheStore接口以及设置CacheConfiguration中cacheStoreFactory的readThrough和writeThrough属性，下面的例子会有说明。

16.1.3.后写缓存

在一个简单的通写模式中每个缓存的put和remove操作都会涉及一个持久化存储的请求，因此整个缓存更新的持续时间可能是相对比较长的。另外，密集的缓存更新频率也会导致非常高的存储负载。 
对于这种情况，Ignite提供了一个选项来执行异步化的持久化存储更新，也叫做后写，这个方式的主要概念是累加更新操作然后作为一个批量操作异步化地刷入持久化存储中。真实的数据持久化可以被基于时间的事件触发（数据输入的最大时间受到队列的限制），也可以被队列的大小触发（当队列大小达到一个限值），或者通过两者的组合触发，这时任何事件都会触发刷新。

更新顺序 
对于后写的方式只有数据的最后一次更新会被写入底层存储。如果键为key1的缓存数据分别依次地更新为值value1、value2和value3，那么只有(key1,value3)对这一个存储请求会被传播到持久化存储。 
更新性能 
批量的存储操作通常比按顺序的单一存储操作更有效率，因此可以通过开启后写模式的批量操作来利用这个特性。简单类型（put和remove）的简单顺序更新操作可以被组合成一个批量操作。比如，连续地往缓存中加入(key1,value1),(key2,value2),(key3,value3)可以通过一个单一的CacheStore.putAll(...)操作批量处理。

后写缓存可以通过CacheConfiguration.setWriteBehindEnabled(boolean)配置项来开启，下面的3.12.6.配置章节显示了一个完整的配置属性列表来进行后写缓存行为的定制。

16.1.4.CacheStore

Ignite中的CacheStore接口用于向底层的数据存储写入或者加载数据。除了标准的JCache加载和存储方法，他还引入了最终事务划界以及从底层数据存储批量载入数据的能力。

loadCache() 
CacheStore.loadCache()方法可以加载缓存，即使没有传入要加载的所有键，它通常用于启动时缓存的热加载，但是也可以在缓存加载完之后的任何时间点调用。 
在每一个相关的集群节点，IgniteCache.loadCache()方法会分配给CacheStore.loadCache()方法，如果只想在本地节点上进行加载，可以用IgniteCache.localLoadCache()方法。

对于分区缓存，不管是主节点还是备份节点，如果键没有被映射到该节点，会被缓存自动丢弃。

load(), write(), delete() 
当IgniteCache接口的get,put,remove方法被调用时，相对应的CacheStore的load(),write()和delete()方法会被调用，当与单个缓存数据工作时，这些方法会用于启用通读和通写行为。

loadAll(), writeAll(), deleteAll() 
当IgniteCache接口的getAll,putAll,removeAll方法被调用时，相对应的CacheStore的loadAll(),writeAll()和deleteAll()方法会被调用，当与多个缓存数据工作时，这些方法会用于启用通读和通写行为，他们通常用批量操作的方式实现以提供更好的性能。

CacheStoreAdapter提供了loadAll(),writeAll()和deleteAll()方法的默认实现，他只是简单地对键进行一个一个地迭代。

sessionEnd() 
Ignite有一个存储会话的概念，他可以跨越不止一个的缓存存储操作，会话对于事务非常有用。 
对于原子化的缓存，sessionEnd()方法会在每个CacheStore方法完成之后被调用，对于事务化的缓存，不管是在底层持久化存储进行提交或者回滚多个操作，sessionEnd()方法都会在每个事务结束后被调用。

CacheStoreAdapater提供了sessionEnd()方法的默认的空实现。 
Cassandra Cache Store 
Ignite提供了将Apache Cassandra作为内存网格级CacheStore的开箱即用的集成，要了解更多的信息，可以查看相关的文档。

16.1.5.CacheStoreSession

缓存存储会话的主要目的是当CacheStore用于事务中时在多个存储操作中持有一个上下文。比如，如果使用JDBC，可以通过CacheStoreSession.attach()方法保存数据库的连接，然后可以在CacheStore.sessionEnd(boolean)方法中提交这个连接。 
CacheStoreSession可以通过@GridCacheStoreSessionResource注解注入自定义的缓存存储实现中。

16.1.6.CacheStore示例

下面是几个不同场景的缓存存储的实现，注意事务化的实现用还是没用事务。 
JDBC非事务：

1. public class CacheJdbcPersonStore extends CacheStoreAdapter<Long, Person> {
2. // This mehtod is called whenever "get(...)" methods are called on IgniteCache.
3. @Override public Person load(Long key) {
4. try (Connection conn = connection()) {
5. try (PreparedStatement st = conn.prepareStatement("select * from PERSONS where id=?")) {
6. st.setLong(1, key);
8. ResultSet rs = st.executeQuery();
10. return rs.next() ? new Person(rs.getLong(1), rs.getString(2), rs.getString(3)) : null;
11. }
12. }
13. catch (SQLException e) {
14. throw new CacheLoaderException("Failed to load: " + key, e);
15. }
16. }
18. // This mehtod is called whenever "put(...)" methods are called on IgniteCache.
19. @Override public void write(Cache.Entry<Long, Person> entry) {
20. try (Connection conn = connection()) {
21. // Syntax of MERGE statement is database specific and should be adopted for your database.
22. // If your database does not support MERGE statement then use sequentially update, insert statements.
23. try (PreparedStatement st = conn.prepareStatement(
24. "merge into PERSONS (id, firstName, lastName) key (id) VALUES (?, ?, ?)")) {
25. for (Cache.Entry<Long, Person> entry : entries) {
26. Person val = entry.getValue();
28. st.setLong(1, entry.getKey());
29. st.setString(2, val.getFirstName());
30. st.setString(3, val.getLastName());
32. st.executeUpdate();
33. }
34. }
35. }
36. catch (SQLException e) {
37. throw new CacheWriterException("Failed to write [key=" + key + ", val=" + val + ']', e);
38. }
39. }
41. // This mehtod is called whenever "remove(...)" methods are called on IgniteCache.
42. @Override public void delete(Object key) {
43. try (Connection conn = connection()) {
44. try (PreparedStatement st = conn.prepareStatement("delete from PERSONS where id=?")) {
45. st.setLong(1, (Long)key);
47. st.executeUpdate();
48. }
49. }
50. catch (SQLException e) {
51. throw new CacheWriterException("Failed to delete: " + key, e);
52. }
53. }
55. // This mehtod is called whenever "loadCache()" and "localLoadCache()"
56. // methods are called on IgniteCache. It is used for bulk-loading the cache.
57. // If you don't need to bulk-load the cache, skip this method.
58. @Override public void loadCache(IgniteBiInClosure<Long, Person> clo, Object... args) {
59. if (args == null || args.length == 0 || args[0] == null)
60. throw new CacheLoaderException("Expected entry count parameter is not provided.");
62. final int entryCnt = (Integer)args[0];
64. try (Connection conn = connection()) {
65. try (PreparedStatement st = conn.prepareStatement("select * from PERSONS")) {
66. try (ResultSet rs = st.executeQuery()) {
67. int cnt = 0;
69. while (cnt < entryCnt && rs.next()) {
70. Person person = new Person(rs.getLong(1), rs.getString(2), rs.getString(3));
72. clo.apply(person.getId(), person);
74. cnt++;
75. }
76. }
77. }
78. }
79. catch (SQLException e) {
80. throw new CacheLoaderException("Failed to load values from cache store.", e);
81. }
82. }
84. // Open JDBC connection.
85. private Connection connection() throws SQLException {
86. // Open connection to your RDBMS systems (Oracle, MySQL, Postgres, DB2, Microsoft SQL, etc.)
87. // In this example we use H2 Database for simplification.
88. Connection conn = DriverManager.getConnection("jdbc:h2:mem:example;DB_CLOSE_DELAY=-1");
90. conn.setAutoCommit(true);
92. return conn;
93. }
94. }

JDBC事务：

1. public class CacheJdbcPersonStore extends CacheStoreAdapter<Long, Person> {
2. /** Auto-injected store session. */
3. @CacheStoreSessionResource
4. private CacheStoreSession ses;
6. // Complete transaction or simply close connection if there is no transaction.
7. @Override public void sessionEnd(boolean commit) {
8. try (Connection conn = ses.getAttached()) {
9. if (conn != null && ses.isWithinTransaction()) {
10. if (commit)
11. conn.commit();
12. else
13. conn.rollback();
14. }
15. }
16. catch (SQLException e) {
17. throw new CacheWriterException("Failed to end store session.", e);
18. }
19. }
21. // This mehtod is called whenever "get(...)" methods are called on IgniteCache.
22. @Override public Person load(Long key) {
23. try (Connection conn = connection()) {
24. try (PreparedStatement st = conn.prepareStatement("select * from PERSONS where id=?")) {
25. st.setLong(1, key);
27. ResultSet rs = st.executeQuery();
29. return rs.next() ? new Person(rs.getLong(1), rs.getString(2), rs.getString(3)) : null;
30. }
31. }
32. catch (SQLException e) {
33. throw new CacheLoaderException("Failed to load: " + key, e);
34. }
35. }
37. // This mehtod is called whenever "put(...)" methods are called on IgniteCache.
38. @Override public void write(Cache.Entry<Long, Person> entry) {
39. try (Connection conn = connection()) {
40. // Syntax of MERGE statement is database specific and should be adopted for your database.
41. // If your database does not support MERGE statement then use sequentially update, insert statements.
42. try (PreparedStatement st = conn.prepareStatement(
43. "merge into PERSONS (id, firstName, lastName) key (id) VALUES (?, ?, ?)")) {
44. for (Cache.Entry<Long, Person> entry : entries) {
45. Person val = entry.getValue();
47. st.setLong(1, entry.getKey());
48. st.setString(2, val.getFirstName());
49. st.setString(3, val.getLastName());
51. st.executeUpdate();
52. }
53. }
54. }
55. catch (SQLException e) {
56. throw new CacheWriterException("Failed to write [key=" + key + ", val=" + val + ']', e);
57. }
58. }
60. // This mehtod is called whenever "remove(...)" methods are called on IgniteCache.
61. @Override public void delete(Object key) {
62. try (Connection conn = connection()) {
63. try (PreparedStatement st = conn.prepareStatement("delete from PERSONS where id=?")) {
64. st.setLong(1, (Long)key);
66. st.executeUpdate();
67. }
68. }
69. catch (SQLException e) {
70. throw new CacheWriterException("Failed to delete: " + key, e);
71. }
72. }
74. // This mehtod is called whenever "loadCache()" and "localLoadCache()"
75. // methods are called on IgniteCache. It is used for bulk-loading the cache.
76. // If you don't need to bulk-load the cache, skip this method.
77. @Override public void loadCache(IgniteBiInClosure<Long, Person> clo, Object... args) {
78. if (args == null || args.length == 0 || args[0] == null)
79. throw new CacheLoaderException("Expected entry count parameter is not provided.");
81. final int entryCnt = (Integer)args[0];
83. try (Connection conn = connection()) {
84. try (PreparedStatement st = conn.prepareStatement("select * from PERSONS")) {
85. try (ResultSet rs = st.executeQuery()) {
86. int cnt = 0;
88. while (cnt < entryCnt && rs.next()) {
89. Person person = new Person(rs.getLong(1), rs.getString(2), rs.getString(3));
91. clo.apply(person.getId(), person);
93. cnt++;
94. }
95. }
96. }
97. }
98. catch (SQLException e) {
99. throw new CacheLoaderException("Failed to load values from cache store.", e);
100. }
101. }
103. // Opens JDBC connection and attaches it to the ongoing
104. // session if within a transaction.
105. private Connection connection() throws SQLException {
106. if (ses.isWithinTransaction()) {
107. Connection conn = ses.getAttached();
109. if (conn == null) {
110. conn = openConnection(false);
112. // Store connection in the session, so it can be accessed
113. // for other operations within the same transaction.
114. ses.attach(conn);
115. }
117. return conn;
118. }
119. // Transaction can be null in case of simple load or put operation.
120. else
121. return openConnection(true);
122. }
124. // Opens JDBC connection.
125. private Connection openConnection(boolean autocommit) throws SQLException {
126. // Open connection to your RDBMS systems (Oracle, MySQL, Postgres, DB2, Microsoft SQL, etc.)
127. // In this example we use H2 Database for simplification.
128. Connection conn = DriverManager.getConnection("jdbc:h2:mem:example;DB_CLOSE_DELAY=-1");
130. conn.setAutoCommit(autocommit);
132. return conn;
133. }
134. }

JDBC批量操作

1. public class CacheJdbcPersonStore extends CacheStore<Long, Person> {
2. // Skip single operations and open connection methods.
3. // You can copy them from jdbc non-transactional or jdbc transactional examples.
4. ...
6. // This mehtod is called whenever "getAll(...)" methods are called on IgniteCache.
7. @Override public Map<K, V> loadAll(Iterable<Long> keys) {
8. try (Connection conn = connection()) {
9. try (PreparedStatement st = conn.prepareStatement(
10. "select firstName, lastName from PERSONS where id=?")) {
11. Map<K, V> loaded = new HashMap<>();
13. for (Long key : keys) {
14. st.setLong(1, key);
16. try(ResultSet rs = st.executeQuery()) {
17. if (rs.next())
18. loaded.put(key, new Person(key, rs.getString(1), rs.getString(2));
19. }
20. }
22. return loaded;
23. }
24. }
25. catch (SQLException e) {
26. throw new CacheLoaderException("Failed to loadAll: " + keys, e);
27. }
28. }
30. // This mehtod is called whenever "putAll(...)" methods are called on IgniteCache.
31. @Override public void writeAll(Collection<Cache.Entry<Long, Person>> entries) {
32. try (Connection conn = connection()) {
33. // Syntax of MERGE statement is database specific and should be adopted for your database.
34. // If your database does not support MERGE statement then use sequentially update, insert statements.
35. try (PreparedStatement st = conn.prepareStatement(
36. "merge into PERSONS (id, firstName, lastName) key (id) VALUES (?, ?, ?)")) {
37. for (Cache.Entry<Long, Person> entry : entries) {
38. Person val = entry.getValue();
40. st.setLong(1, entry.getKey());
41. st.setString(2, val.getFirstName());
42. st.setString(3, val.getLastName());
44. st.addBatch();
45. }
47. st.executeBatch();
48. }
49. }
50. catch (SQLException e) {
51. throw new CacheWriterException("Failed to writeAll: " + entries, e);
52. }
53. }
55. // This mehtod is called whenever "removeAll(...)" methods are called on IgniteCache.
56. @Override public void deleteAll(Collection<Long> keys) {
57. try (Connection conn = connection()) {
58. try (PreparedStatement st = conn.prepareStatement("delete from PERSONS where id=?")) {
59. for (Long key : keys) {
60. st.setLong(1, key);
62. st.addBatch();
63. }
65. st.executeBatch();
66. }
67. }
68. catch (SQLException e) {
69. throw new CacheWriterException("Failed to deleteAll: " + keys, e);
70. }
71. }
72. }

16.1.7.配置

下面的配置参数可以通过CacheConfiguration用于启用以及配置通写缓存：

setter方法	描述	默认值
setWriteBehindEnabled(boolean)	设置后写是否启用的标志	false
setWriteBehindFlushSize(int)	后写缓存的最大值，如果超过了这个限值，所有的缓存数据都会被刷入缓存存储然后写缓存被清空。如果值为0，刷新操作将会依据刷新频率间隔，注意不能将写缓存大小和刷新频率都设置为0	10240
setWriteBehindFlushFrequency(long)	后写缓存的刷新频率，单位为毫秒，该值定义了从对缓存对象进行插入/删除和当相应的操作被施加到缓存存储的时刻之间的最大时间间隔。如果值为0，刷新会依据写缓存大小，注意不能将写缓存大小和刷新频率都设置为0	5000
setWriteBehindFlushThreadCount(int)	执行缓存刷新的线程数	1
setWriteBehindBatchSize(int)	后写缓存存储操作的操作数最大值	512

CacheStore接口可以在IgniteConfiguration上通过一个工厂进行设置，就和CacheLoader和CacheWriter同样的方式。

对于分布式缓存的配置，Factory应该是可序列化的。

XML：

1. <bean class="org.apache.ignite.configuration.IgniteConfiguration">
2. ...
3. <property name="cacheConfiguration">
4. <list>
5. <bean class="org.apache.ignite.configuration.CacheConfiguration">
6. ...
7. <property name="cacheStoreFactory">
8. <bean class="javax.cache.configuration.FactoryBuilder" factory-method="factoryOf">
9. <constructor-arg value="foo.bar.MyPersonStore"/>
10. </bean>
11. </property>
12. <property name="readThrough" value="true"/>
13. <property name="writeThrough" value="true"/>
14. </bean>
15. </list>
16. </property>
17. ...
18. </bean>

Java：

1. IgniteConfiguration cfg = new IgniteConfiguration();
3. CacheConfiguration<Long, Person> cacheCfg = new CacheConfiguration<>();
5. cacheCfg.setCacheStoreFactory(FactoryBuilder.factoryOf(MyPersonStore.class));
6. cacheCfg.setReadThrough(true);
7. cacheCfg.setWriteThrough(true);
9. cfg.setCacheConfiguration(cacheCfg);
11. // Start Ignite node.
12. Ignition.start(cfg);

16.1.8.CacheJdbcBlobStore

CacheJdbcBlobStore实现基于JDBC，这个实现将对象以BLOB的格式存储在底层数据库中。存储会在数据库中创建名为ENTRIES的表来存储数据，表具有key和val两个字段。 如果提供了定制的DDL和DML语句，表和字段的名字要和所有的语句一致以及参数的顺序也要保留。 使用CacheJdbcBlobStoreFactory工厂来向CacheConfiguration传入CacheJdbcBlobStore:

Spring:

1. <bean id= "simpleDataSource" class="org.h2.jdbcx.JdbcDataSource"/>
3. <bean id="ignite.cfg" class="org.apache.ignite.configuration.IgniteConfiguration">
4. ...
5. <property name="cacheConfiguration">
6. <list>
7. <bean class="org.apache.ignite.configuration.CacheConfiguration">
8. ...
9. <property name="cacheStoreFactory">
10. <bean class="org.apache.ignite.cache.store.jdbc.CacheJdbcBlobStoreFactory">
11. <property name="user" value = "user" />
12. <property name="dataSourceBean" value = "simpleDataSource" />
13. </bean>
14. </property>
15. </bean>
16. </list>
17. </property>
18. ...
19. </bean>

16.1.9.CacheJdbcPojoStore

CacheJdbcPojoStore实现基于JDBC和基于反射的POJO，这个实现将对象用基于反射的Java Bean映射描述的形式存储在底层数据库中。 
使用CacheJdbcPojoStoreFactory工厂来向CacheConfiguration传入CacheJdbcPojoStore: 
Spring:

1. <bean id= "simpleDataSource" class="org.h2.jdbcx.JdbcDataSource"/>
3. <bean id="ignite.cfg" class="org.apache.ignite.configuration.IgniteConfiguration">
4. ...
5. <property name="cacheConfiguration">
6. <list>
7. <bean class="org.apache.ignite.configuration.CacheConfiguration">
8. ...
9. <property name="cacheStoreFactory">
10. <bean class="org.apache.ignite.cache.store.jdbc.CacheJdbcPojoStoreFactory">
11. <property name="dataSourceBean" value = "simpleDataSource" />
12. </bean>
13. </property>
14. </bean>
15. </list>
16. </property>
17. </bean>

16.1.10.CacheHibernateBlobStore

CacheHibernateBlobStore实现基于Hibernate,这个实现将对象以BLOB的格式存储在底层数据库中。 使用CacheHibernateBlobStoreFactory工厂来向CacheConfiguration传入CacheHibernateBlobStore: Spring:

1. <bean id="ignite.cfg" class="org.apache.ignite.configuration.IgniteConfiguration">
2. ...
3. <property name="cacheConfiguration">
4. <list>
5. <bean class="org.apache.ignite.configuration.CacheConfiguration">
6. <bean class="org.apache.ignite.cache.store.hibernate.CacheHibernateBlobStoreFactory">
7. <property name="hibernateProperties">
8. <props>
9. <prop key="connection.url">jdbc:h2:mem:</prop>
10. <prop key="hbm2ddl.auto">update</prop>
11. <prop key="show_sql">true</prop>
12. </props>
13. </property>
14. </bean>
15. </list>
16. </property>
17. ...
18. </bean>