Ceph buffer pool management

Introduction to the buffer pool

Principle buffer pool

Cache hierarchy characteristics in ceph is officially released version of the F ceph among. The so-called cache hierarchy is in fact faster disk (usually ssd) to create a storage pool on. This is then placed at the front end of the storage pool of a conventional copy pool or pool acts as the erasure code cache. So that all client I / O operations are first processed by the buffer pool, then after the data is written back to the existing data in the storage pool so that the client can enjoy better performance in the cache pool, and the final data is written back to the regular pool.

In general, the buffer pool will be built on the more expensive or faster ssd disk, so as to provide better I / O performance. The buffer pool back-end storage pool by ordinary hdd disk components. In this type of setup, the client I / O requests submitted to the buffer pool, whether it is a read or write operation, it can request immediately get a response. Faster cache layer is provided in response to client requests. Over time, the cache write back all the data layer backend memory storage, so that it can cache the new request from the client. Data migration between the buffer layer and the data layer are triggered automatically and transparent to the client.

Operating mode buffer pool

Buffer pool can be configured in two modes:

• writeback mode: When the buffer pool is configured to writeback mode, the write data to the client buffer pool, and then immediately receives a write acknowledgment. Buffer pool based on flushing / evicting us to set policy, and migrate data from the buffer pool to the back-end storage pool, and eventually remove it from the cache by the cache agent layer. When processing a read operation from the client, first migrate from the proxy cache hierarchy data from the storage layer to the buffer layer, and then return it to the client. Only when the data is no longer active or become cold data, and then delete it from the cache pool.

• read-only mode: When the buffer pool is configured to read-only mode, the client process only applies to read operations. Client write operations do not involve cache hierarchy, all clients write is done on the storage layer. When processing the read operation from a client, the proxy cache hierarchy requested data is copied from the storage layer to the buffer layer. Based on our strategy for the buffer pool configuration, the inactive object is removed from the buffer pool. This method is very suitable for a plurality of client needs to read a large amount of data similar scene.

Configuring cache pool

1. Create a buffer pool

Suppose we have a basis of ssd crush_rule, we create a storage pool, you can use the crush rule. Based on how to create a ssd's crush_ruledirect reference to "crushmap intelligent disk groups" .

Here is an example of a creation based ssd storage pool:

# 假设ssd_rule即基于ssd的cursh_rule
ceph osd create pool cache 64 64 ssd_rule

2. Set the buffer layer

# 将上面创建的cache池绑定至存储池的前端，cephfs_data即为我们的后端存储池
ceph osd tier add cephfs_data cache

# 设置缓存模式为writeback
ceph osd tier cache-mode cache writeback

# 将所有客户端请求从标准池引导至缓存池
ceph osd tier set-overlay cephfs_data cache

At this point, we are viewing the storage pool and buffer pool information, see relevant cache configuration information:

root@ceph:~# ceph osd dump |egrep 'cephfs_data|cache'
pool 2 'cephfs_data' replicated size 3 min_size 1 crush_rule 0 object_hash rjenkins pg_num 128 pgp_num 128 last_change 3079 lfor 3079/3079 flags hashpspool tiers 4 read_tier 4 write_tier 4 stripe_width 0 application cephfs
pool 4 'cache' replicated size 3 min_size 1 crush_rule 1 object_hash rjenkins pg_num 64 pgp_num 64 last_change 3100 lfor 3079/3079 flags hashpspool,incomplete_clones tier_of 2 cache_mode writeback target_bytes 1099511627776 hit_set bloom{false_positive_probability: 0.05, target_size: 0, seed: 0} 0s x0 decay_rate 0 search_last_n 0 stripe_width 0

3. The buffer layer related parameter description

1. Production environment for deployment, currently only bloom filters data structure (see the meaning of official document, it seems only support this kind of filter):

ceph osd pool set cache hit_set_type bloom

1. When you set when the data buffer pool of the number of bytes or how many objects are reached, the agency began to refresh the cache hierarchy objects from the buffer pool to the back-end storage pool and expulsion:

# 当缓存池中的数据量达到1TB时开始刷盘并驱逐
ceph osd pool set cache target_max_bytes 1099511627776

# 当缓存池中的对象个数达到100万时开始刷盘并驱逐
ceph osd pool set cache target_max_objects 10000000

1. Brush object defines buffer layer to the storage layer or expulsion time:

ceph osd pool set cache cache_min_flush_age 600
ceph osd pool set cache cache_min_evict_age 600 

1. When the buffer pool dirty defined object (an object to be modified) accounted for much reached, the agent begins cache hierarchy brush object from the cache memory layer to layer:

# 当脏对象占比达到10%时开始刷盘
ceph osd pool set cache cache_target_dirty_ratio 0.4
# 当脏对象占比达到60%时开始高速刷盘
cache_target_dirty_high_ratio: 0.6

1. When the amount reaches a certain percentage of the buffer pool of its total, hierarchical caching proxy will expel the object in order to maintain the available capacity (the limit is reached, it is felt that buffer pool is full), then will unmodified (clean ) Object brush plate:

ceph osd pool set cache cache_target_full_ratio 0.8

4. Test buffer pool

After configured the buffer pool, we can minimize the time of its first eviction object is set to 60s:

ceph osd pool set cache cache_min_evict_age 60
ceph osd pool set cache cache_min_flush_age 60

We then write data to a storage pool

rados -p cephfs_data put test /tmp/test.txt

View storage pool, then should not be able to view the data, see the buffer pool, you can see the data stored in the buffer pool:

rados -p cephfs_data ls |grep test
rados -p cache ls |grep test

Etc. After 60s, the brush plate data, this time to see the data in a storage pool, the buffer pool, i.e., the data eviction.

Delete cache pool

It should be noted that, according to the different types of buffer pool, delete the buffer pool different methods.

1. Remove the read-only cache pool

Since the read-only cache does not have the modified data, and therefore can disable and delete it, without losing any recent changes to the cache object.

1. The cache mode personality is none to disable caching:

ceph osd tier cache-mode cache none

1. Delete cache pool:

# 解除绑定
ceph osd tier remove cephfs_data cache

2. Delete the writeback cache pool

Since the write-back cache may have modified data, steps must be taken to ensure that any recent changes before you disable and delete the cache, the cache is not lost objects.

1. The cache mode is changed to forward to the new and modified objects to refresh the back-end storage pool:

ceph osd tier cache-mode cache forward

1. Check the buffer pool to ensure that all objects are refreshed (which may need some time):

rados -p cache ls 

1. If the buffer pool is still an object, you can also manually refresh:

rados -p cache cache-flush-evict-all

1. Deleted covering layer, so that the client no longer directs traffic to the cache:

ceph osd tier remove-overlay cephfs_data

1. Unbind storage pool buffer pool:

ceph osd tier remove cephfs_data cache

Reference: https://blog.csdn.net/weixin_34364135/article/details/87528549