Corresponding PPT: ceph source code io reading and writing process analysis serial talk - CSDN download
1. Introduction to OSD module
1.1 Message Encapsulation : Send and receive information on the OSD.
cluster_messenger - communicates with other OSDs and monitors
client_messenger - communicates with clients
1.2 Message scheduling :
Dispatcher class, mainly responsible for message classification
1.3 Work queue:
1.3.1 OpWQ: handles ops (from clients) and sub ops (from other OSDs). Runs in the op_tp thread pool.
1.3.2 PeeringWQ: Processes peering tasks and runs in the op_tp thread pool.
1.3.3 CommandWQ: Process cmd commands and run in command_tp.
1.3.4 RecoveryWQ: Data recovery, running on recovery_tp.
1.3.5 SnapTrimWQ: Snapshot related, running on disk_tp.
1.3.6 ScrubWQ: scrub, running on disk_tp.
1.3.7 ScrubFinalizeWQ: scrub, running on disk_tp.
1.3.8 RepScrubWQ: scrub, running on disk_tp.
1.3.9 RemoveWQ: Remove the old pg directory. Running on disk_tp.
1.4 Thread pool:
There are 4 OSD thread pools:
1.4.1 op_tp: Handling ops and sub ops
1.4.2 recovery_tp: process recovery tasks
1.4.3 disk_tp: handles disk-intensive tasks
1.4.4 command_tp: processing commands
1.5 Main objects:
ObjectStore *store;
OSDSuperblock superblock; mainly the version number and other information
OSDMapRef osdmap;
1.6 Main Operation Process: Reference Articles
1.6.1 The client initiates the request process
1.6.2 The op_tp thread processes data reading
1.6.3 Processing of Object Operations
1.6.4 Handling of modification operations
1.6.5 Log writing
1.6.6 Write operation processing
1.6.7 The sync process of the transaction
1.6.8 Log Recovery
1.7 Overall processing diagram
Ceph OSD uses journaling filesystems such as Btrfs and XFS. Before committing data to alternate storage, Ceph first writes the data to a separate storage area called a journal, which is a small buffer on the same mechanical disk (such as OSD) or a different SSD disk or partition size partition, or even a file on the file system. In this mechanism, all Ceph writes go to the log first and then to the standby storage, as shown in the figure below.
Author: Enlightenment Cloud
Link: https://www.zhihu.com/question/21718731/answer/561372178
2. The general process and storage form of data written by the client
2.1 Read and write framework
2.2 Client write process
There are generally two methods when using rbd on the client side:
- The first is Kernel rbd. After creating the rbd device, map the rbd device into the kernel to form a virtual block device. At this time, this block device is the same as other general block devices. The general device file is /dev/rbd0, and this block device is used directly in the future. The file is enough, you can format /dev/rbd0 and mount it to a certain directory, or you can use it directly as a raw device. At this time, the operation of the rbd device is carried out through the kernel rbd operation method.
- The second is the librbd way. After the rbd device is created, the librbd and librados libraries can be used to access and manage the block device. This method does not map to the kernel, and directly calls the interface provided by librbd to access and manage rbd devices, but does not generate block device files on the client side.
The process of applying writing to the rbd block device:
- The application calls the librbd interface or writes binary blocks to the linux kernel virtual block device. Take librbd as an example below.
- librbd divides binary blocks into blocks, the default block size is 4M, each block has a name and becomes an object
- librbd calls librados to write objects to the Ceph cluster
- librados writes divided binary data blocks to the main OSD (first establishes a TCP/IP connection, then sends a message to the OSD, and the OSD writes it to its disk after receiving it)
- The primary OSD is responsible for writing replicas to one or more secondary OSDs simultaneously. Note that when writing to the journal (Journal), it will be returned. Therefore, if SSD is used as the Journal, the response speed can be improved, so that the server can quickly synchronize with the client and return the write result (ack).
- When the writing of the primary and secondary OSDs is completed, the primary OSD returns the writing success to the client.
- When the data in the Journal is successfully written to the disk after a period of time (maybe a few seconds), Ceph notifies the client that the data is successfully written to the disk (commit) through an event. At this time, the client can completely write the data in the cache. cleared.
- By default, the Ceph client will cache written data until it receives a commit notification from the cluster. If the OSD fails during this phase (between the write method returning and the commit notification) causing data to fail to be written to the filesystem, Ceph will allow the client to redo the uncommitted operation (replay). Therefore, the PG has a state called replay: "The placement group is waiting for clients to replay operations after an OSD crashed.".
That is, the file system is responsible for file processing, librbd is responsible for block processing, librados is responsible for object processing, and OSD is responsible for writing data to Journal and disk.
2.3 RBD storage format
As shown in the figure below, the form of data seen by components/users at different levels in the Ceph system is different:
- What the Ceph client sees is a complete contiguous block of binary data, the size of which is the set size or the size of the resize created when the RBD image is created, and the client can write binary data from the beginning or from a certain position.
- librados is responsible for creating objects in RADOS, whose size is determined by the order of the pool. By default, order = 22, and the size of the object is 4MB; and it is responsible for striping the binary block passed in by the client into several stripes (stripe).
- librados controls which stripe is written by which OSD (stripe---write which--->object---located in which--->OSD)
- The OSD is responsible for creating files in the file system and writing data passed in from librados.
Ceph client writes binary data to an RBD image (assuming the pool has 3 copies):
(1) The Ceph client calls librados to create an RBD image. At this time, it does not allocate storage space, but creates several metadata objects to save metadata information.
(2) The Ceph client calls librados to start writing data. librados computes stripes, objects, etc., and then starts writing the first stripe to a specific target object.
(3) librados calculates the main OSD ID corresponding to the object according to the CRUSH algorithm, and sends the binary data to it.
(4) The main OSD is responsible for calling the file system interface to write binary data to the file on the disk (each object corresponds to a file, and the content of the file is one or more stripes).
(5) After the main ODS completes data writing, it uses CRUSH to calculate the positions of the second OSD (secondary OSD) and the third OSD (tertiary OSD), and then copies the objects to these two OSDs. After all are completed, it reports back to the ceph client that the object has been saved.
(6) Then write the second strip until all writing is complete. After all is completed, librados should also do metadata updates, such as writing new size and so on.
Full process ( source ):
The osd ( object store daemon ) daemon ( daemon ) is a special kind of process that runs in the background.
This process has the characteristics of strong consistency:
- Ceph's read and write operations use the Primary-Replica model. The Client only initiates read and write requests to the Primary of the OSD set corresponding to the Object, which ensures strong data consistency.
- Since each Object has only one Primary OSD, the updates to the Object are all sequential, and there is no synchronization problem.
- When the Primary receives the write request of the Object, it is responsible for sending the data to other Replicas. As long as the data is stored on all OSDs, the Primary responds to the write request of the Object, which ensures the consistency of the replicas. This also brings some side effects. Compared to storage systems that only implement eventual consistency, such as Swift, Ceph is not written until all three copies are written, which increases write latency in the event of disk corruption.
On the OSD, after receiving the data storage instruction, it will generate 2~3 disk seek operations:
- Record the write operation to the OSD Journal file (Journal is to ensure the atomicity of the write operation).
- Update the write operation to the file corresponding to the Object.
- Log write operations to the PG Log file.
3. Client request process
RADOS read object process
RADOS write object operation process
example:
#!/usr/bin/env python
import sys,rados,rbd
def connectceph():
cluster = rados.Rados(conffile = '/root/xuyanjiangtest/ceph-0.94.3/src/ceph.conf')
cluster.connect()
ioctx = cluster.open_ioctx('mypool')
rbd_inst = rbd.RBD()
size = 4*1024**3 #4 GiB
rbd_inst.create(ioctx,'myimage',size)
image = rbd.Image(ioctx,'myimage')
data = 'foo'* 200
image.write(data,0)
image.close()
ioctx.close()
cluster.shutdown()
if __name__ == "__main__":
connectceph()
1. First cluster = rados.Rados(conffile = 'ceph.conf'), use the current ceph configuration file to create a rados, here is mainly to parse the cluster configuration parameters in ceph.conf. Then save the values of these parameters in rados.
2. cluster.connect() , a radosclient structure will be created here, and this structure will mainly include several functional modules:
Message management module Messager, data processing module Objector, finisher thread module.
3. ioctx = cluster.open_ioctx('mypool'), create an ioctx for a storage pool named mypool, the radosclient and Objector modules will be specified in the ioctx, and the information of mypool, including the parameters of the pool, will also be recorded.
4. rbd_inst.create(ioctx,'myimage',size), create an rbd device named myimage, and then write data to this device.
5. image = rbd.Image(ioctx,'myimage'), create an image structure, where the structure associates myimage with ioctx, and ioctx can be found directly through the image structure later. Here, the ioctx will be copied into two copies, which are divided into data_ioctx and md_ctx. See clearly, one is used to process the storage data of rbd, and the other is used to process the management data of rbd.
flow chart:
1. image.write(data, 0), which starts the life of a write request through image. Two basic elements of the request are specified here: buffer=data and offset=0. From here, I entered the world of ceph, which is also the world of c++.
Converted from image.write(data,0) to the Image::write() function in the librbd.cc file, let's take a look at the main implementation of this function
ssize_t Image::write(uint64_t ofs, size_t len, bufferlist& bl)
{
ImageCtx *ictx = (ImageCtx *)ctx;
int r = librbd::write(ictx, ofs, len, bl.c_str(), 0);
return r;
}2. This function directly distributes the function to librbd::wrte. Follow along to see the implementation in librbd::write. The specific implementation of this function is in the internal.cc file.
ssize_t write(ImageCtx *ictx, uint64_t off, size_t len, const char *buf, int op_flags)
{
Context *ctx = new C_SafeCond(&mylock, &cond, &done, &ret); //---a
AioCompletion *c = aio_create_completion_internal(ctx, rbd_ctx_cb);//---b
r = aio_write(ictx, off, mylen, buf, c, op_flags); //---c
while (!done)
cond.Wait(mylock); // ---d
}---a. This sentence needs to apply for a callback operation for this operation. The so-called callback is some finishing work, signal wake-up processing.
---b. This sentence is to apply for an operation to be performed when io is completed. When io is completed, the rbd_ctx_cb function will be called, which will continue to call ctx->complete().
---c. The function aio_write will continue to process this request.
---d. When sentence c sends this io to osd, and osd has not requested to complete the processing, it will wait on d until the bottom layer processes the request, call back the AioCompletion applied by b, and continue to call ctx- in a >complete(), wake up the waiting signal here, and then the program continues down.
3. Let's see what aio_write will do when it gets the requested offset and buffer?
int aio_write(ImageCtx *ictx, uint64_t off, size_t len, const char *buf, AioCompletion *c, int op_flags)
{
//将请求按着object进行拆分
vector<ObjectExtent> extents;
if (len > 0)
{
Striper::file_to_extents(ictx->cct, ictx->format_string, &ictx->layout, off,
clip_len,0, extents); //---a
}
//处理每一个object上的请求数据
for (vector<ObjectExtent>::iterator p = extents.begin(); p != extents.end(); ++p)
{
C_AioWrite *req_comp = new C_AioWrite(cct, c); //---b
AioWrite *req = new AioWrite(ictx, p->oid.name, p->objectno, p- >offset,bl,…..,
req_comp); //---c
r = req->send(); //---d
}
}According to the size of the request, the request needs to be divided according to the object, which is processed by the function file_to_extents. After the processing is completed, it is stored in the extents according to the object. There are many functions with the same name in file_to_extents(). The main content of these functions does one thing, which is the splitting of the original request.
An rbd device is composed of many objects, that is, the rbd device is divided into blocks, each block is called an object, and the size of each object is 4M by default, or you can specify it yourself. The file_to_extents function maps this large request to the object, and splits it into many small requests as shown in the figure below. The result of the final mapping is stored in the ObjectExtent.
The original offset refers to the offset in rbd (the location where rbd is written), and after file_to_extents, it is converted into the internal offset offset0 of one or more objects. After this conversion, a batch of requests in this object are processed.
4. Go back to the aio_write function, you need to process each object request after splitting.
---b. Apply for a callback handler for the write request.
---c. According to the request inside the object, create a structure called AioWrite.
---d. Send the req of this AioWrite to send().
5. Here AioWrite is inherited from AbstractWrite, AbstractWrite is inherited from AioRequest class, the send method is defined in AbstractWrite class, see the specific content of send.
int AbstractWrite::send() {
if (send_pre()) //---a
}
#进入send_pre()函数中
bool AbstractWrite::send_pre()
{
m_state = LIBRBD_AIO_WRITE_PRE; // ----a
FunctionContext *ctx = //----b
new FunctionContext( boost::bind(&AioRequest::complete, this, _1));
m_ictx->object_map.aio_update（ctx）; //-----c
}---a. Modify m_state to LIBRBD_AIO_WRITE_PRE.
---b. Apply for a callback function and actually call AioRequest::complete()
---c. Start to issue the request of object_map.aio_update. This is a function of status update, not a very important link. I won't say more here. When the update request is completed, it will automatically call back to the callback function applied by b.
6. Go to the AioRequest::complete() function.
void AioRequest::complete(int r)
{
if (should_complete(r)) //---a
}---a.should_complete function is a pure virtual function, which needs to be implemented in the inherited class AbstractWrite, to 7. Look at AbstractWrite:: should_complete()
AbstractWrite:: should_complete()
bool AbstractWrite::should_complete(int r)
{
switch (m_state)
{
case LIBRBD_AIO_WRITE_PRE: //----a
{
send_write(); //----b----a. In send_pre, the state of m_state has been set to LIBRBD_AIO_WRITE_PRE, so this branch will be taken.
----b. In the send_write() function, processing will continue,
7.1. Let's look at the send_write function
void AbstractWrite::send_write()
{
m_state = LIBRBD_AIO_WRITE_FLAT; //----a
add_write_ops(&m_write); // ----b
int r = m_ictx->data_ctx.aio_operate(m_oid, rados_completion, &m_write);
}---a. Reset the state of m_state to LIBRBD_AIO_WRITE_FLAT.
---b. Fill m_write and convert the request into m_write.
---c. Issue m_write and use data_ctx.aio_operate function to process. Continue to call the io_ctx_impl->aio_operate() function, and continue to call objecter->mutate().
8. object->mutate()
ceph_tid_t mutate(……..)
{
Op *o = prepare_mutate_op(oid, oloc, op, snapc, mtime, flags,
onack, oncommit, objver);//----d
return op_submit(o);
}---d. Convert the request into an Op request, and continue to use op_submit to issue the request. Continue to call _op_submit_with_budget in op_submit to process the request. Continue to call _op_submit processing.
8.1 Processing of _op_submit. Worth a look here
ceph_tid_t Objecter::_op_submit(Op *op, RWLock::Context& lc)
{
check_for_latest_map = _calc_target(&op->target, &op->last_force_resend)； //---a
int r = _get_session(op->target.osd, &s, lc); //---b
_session_op_assign(s, op); //----c _send_op(op, m); //----d
}----a. _calc_target, by calculating the saved osd of the current object, and then saving the main osd in the target, the rbd write data is sent to the main osd first, and the main osd then sends the data to other replica osd. I won't say much about how to select the relationship between the osd set and the main osd. The principle of this process has been described in "Ceph's Data Storage Road (3)", and the code part is not difficult to understand.
----b. _get_session, this function is used to establish communication with the main osd. After the communication is established, it can be sent to the main osd through this channel. Let's see how this function handles
9. _get_session
int Objecter::_get_session(int osd, OSDSession **session, RWLock::Context& lc)
{
map<int,OSDSession*>::iterator p = osd_sessions.find(osd); //----a
if (p != osd_sessions.end()) {
auto s = p->second;
s->get();
*session = s;
return 0;
}
OSDSession *s = new OSDSession(cct, osd); //----b
osd_sessions[osd] = s;//--c
s->con = messenger->get_connection(osdmap->get_inst(osd));//-d
……
}----a. First, check whether there is a connection in osd_sessions that can be used directly. The first communication does not exist.
----b. Re-apply for an OSDSession, and use osd and other information to initialize.
---c. Add the newly applied OSDSession to osd_sessions and save it for next use.
----d. Call the get_connection method of messager. In this method, continue to find a way to establish a connection with the target osd.
10. messager is implemented by the subclass simpleMessager, let's take a look at the implementation method of get_connection in SimpleMessager
ConnectionRef SimpleMessenger::get_connection(const entity_inst_t& dest)
{
Pipe *pipe = _lookup_pipe(dest.addr); //-----a
if (pipe) {
} else {
pipe = connect_rank(dest.addr, dest.name.type(), NULL, NULL); //----b
}
}----a. First of all, we need to find this pipe, the first communication, naturally this pipe does not exist.
----b. connect_rank will be created according to the addr of this target osd. See what connect_rank does.
11. SimpleMessenger::connect_rank
Pipe *SimpleMessenger::connect_rank(const entity_addr_t& addr, int type, PipeConnection *con, Message *first)
{
Pipe *pipe = new Pipe(this, Pipe::STATE_CONNECTING, static_cast<PipeConnection*>(con)); //----a
pipe->set_peer_type(type); //----b
pipe->set_peer_addr(addr); //----c
pipe->policy = get_policy(type); //----d
pipe->start_writer(); //----e
return pipe; //----f
}----a. First, you need to create this pipe, and associate the pipe with the pipecon.
----b,----c,-----d. All are to set some parameters.
----e. Start the writing thread of the pipe. Here, the processing function of the writing thread of the pipe is pipe->writer(), which will try to connect to the osd. And establish a socket connection channel.
According to the current resource statistics, the write request can find or create an OSDSession according to the target main osd. This OSDSession will have a Pipe structure that manages the data channel, and then there is a processing thread writer that sends messages in this structure. This thread will Maintain socket communication with the target osd.
12. Create and obtain these resources, then go back to the _op_submit function
ceph_tid_t Objecter::_op_submit(Op *op, RWLock::Context& lc)
{
check_for_latest_map = _calc_target(&op->target, &op->last_force_resend)； //---a
int r = _get_session(op->target.osd, &s, lc); //---b
_session_op_assign(s, op); //----c
MOSDOp *m = _prepare_osd_op(op); //-----d
_send_op(op, m); //----e
}---c, bind the current op request to this session, and know which session to use to send the request when sending the request later.
--d, convert op to MOSDop, which will be processed with MOSDOp as the object later.
---e, _send_op will send this MOSDOp according to the previously established communication channel. Call op->session->con->send_message(m) in _send_op, this method will call SimpleMessager->send_message(m), then call _send_message(), and then call submit_message(). In submit_message, the previous pipe will be found, Then call the pipe->send method, and finally send it to the target osd through the thread of the pipe->writer.
Since then, the client waits for the osd processing to complete and return the result.
1. Look at the rados structure in the upper left corner, first create an io environment, create rados information, and structure the data in the configuration file into rados.
2. Create a radosclient client structure based on rados, which includes three important modules, finiser callback processing thread, Messager message processing structure, and Objector data processing structure.
The final data is to be encapsulated into a message sent to the target osd through the Messager.
3. Create an ioctx with radosclient based on the pool information, where the pool-related information is included, and then the information will be used in data processing after obtaining the information.
4. Next, the ioctx will be copied to imagectx, which will become the data_ioctx and md_ioctx data processing channels, and finally the imagectx will be encapsulated into the image structure. All subsequent writes will go through this image. Following the structure of the image, you can find the data structure that was created earlier and can be used.
5. Read and write operations are performed through the image in the upper right corner. When the object of the read and write operations is an image, the image will start to process the request, and then the request will be processed and split into the request of the object object. After splitting, it will be handed over to the objector for processing to find the target osd. Of course, the crush algorithm is used here to find the set of target osd and the main osd.
6. Encapsulate the request op into a MOSDOp message, and then hand it over to SimpleMessager for processing. SimpleMessager will try to search in the existing osd_session. If the corresponding session is not found, it will recreate an OSDSession and create a data channel pipe for this OSDSession , save the data channel in SimpleMessager, you can use it next time.
7. The pipe will establish a Socket communication channel with the target osd, and the pipe will have a dedicated writer thread writer to be responsible for socket communication. In the thread writer, the target ip will be connected first to establish communication. After the message is received from SimpleMessager, it will be saved to the outq queue of the pipe. Another purpose of the writer thread is to monitor the outq queue. When there is a message in the queue waiting to be sent, it will write the message to the socket and send it to the target OSD.
8. After waiting for the OSD to process the data message, it will call back, feedback the execution result, and then inform the caller of the result step by step.
Fourth, Ceph reading process
OSD-side read message distribution process
OSD side read operation processing flow
Overall flow chart:
int read(inodeno_t ino,
file_layout_t *layout,
snapid_t snap,
uint64_t offset,
uint64_t len,
bufferlist *bl, // ptr to data
int flags,
Context *onfinish,
int op_flags = 0) --------------------------------Filer.hStriper::file_to_extents(cct, ino, layout, offset, len, truncate_size, extents);//Convert the length and offset of the data to be read into the object to be accessed, extents follow the concept of the brtfs file system
objecter->sg_read_trunc (extents, snap, bl, flags, truncate_size, truncate_seq, onfinish, op_flags);//Initiate a request to osd
For read operations:
1. The client directly calculates the main osd to which the stored data belongs, and sends a message directly to the main osd.
2. After the main osd receives the message, it can call Filestore to directly read the content in the main pg in the underlying file system and return it to the client. The specific calling function is implemented in ReplicatedPG::do_osd_ops.
CEPH_OSD_OP_MAPEXT||CEPH_OSD_OP_SPARSE_READ
r = osd->store->fiemap(coll, soid, op.extent.offset, op.extent.length, bl);
CEPH_OSD_OP_READ
r = pgbackend->objects_read_sync(soid, miter->first, miter->second, &tmpbl);
Five, Ceph writing process
OSD side write operation process flow
For write operations, it is much more complicated to ensure the synchronization of data writing:
1. First the client will send the data to the main osd,
2. The master osd also needs to preprocess the write operation first. After completion, it needs to send a write message to other slave osd, so that they can make changes to the copy pg.
3. After completing the write operation from osd to Journal through FileJournal, send a message to tell the main osd that it is completed, and enter 5
4. When the master osd receives all the messages from the osd to complete the write operation, it will complete its own write operation to the Journal through the FileJournal. After completion, the client will be notified that the write operation has been completed.
5. The main osd, starts working from the osd thread and calls Filestore to write the data in the Journal to the underlying file system.
The written logic flow chart is shown in the figure:
From the figure, we can see that the write operation is divided into the following steps:
1. The OSD::op_tp thread is taken out of the OSD::op_wq and operated as described in the figure at the beginning of this article. The specific code flow is
Create callback classes C_OSD_OpCommit and C_OSD_OpApplied in ReplicatePG::apply_repop
The callback class C_JournaledAhead is created in FileStore::queue_transactions
2. The FileJournal::write_thread thread is taken out of FileJournal::writeq to operate, mainly to write data to a specific journal, the specific code flow:
3. The Journal::Finisher.finisher_thread thread is taken out of the Journal::Finisher.finish_queue for operation, and by calling the callback function FileStore:_journaled_ahead left by C_JournalAhead, the thread starts to work two things: first enter the underlying FileStore::op_wq notification starts Write, re-entry FileStore::ondisk_finisher.finisher_queue notification can return. Specific code flow:
4. The FileStore::ondisk_finisher.finisher_thread thread is taken out of FileStore::ondisk_finisher.finisher_queue for operation, and the client is notified that the write operation is successful by calling the callback function ReplicatePG::op_commit left by C_OSD_OpCommit
5. The FileStore::op_tp thread pool takes out operations from FileStore::op_wq (the OP_WQ here inherits the parent class ThreadPool::WorkQueue and rewrites functions such as _process and _process_finish, so unlike OSD::op_wq, it has its own workflow), call FileStore::_do_op first, and call FileStore::_finish_op when done.
6. The FileStore::op_finisher.finisher_thread thread is taken out of FileStore::op_finisher.finisher_queue for operation, and the notification data is readable by calling the callback function ReplicatePG::op_applied left by C_OSD_OpApplied.
For the sentence-by-sentence analysis of the source code of the specific OSD, you can refer to a Xiaojiang's blog post