TB-level NFS smooth migration of data series

ccdbfs cluster services company originally offered coming down the line, we need to migrate completed within a month, and the subsequent offer steady, high availability NFS service.

20TB estimated data rate is assumed copies 100MB / s.

20 * 1024 * 1024 / 100 ≈ 209715.2 s
209715.2 / (60 * 60) ≈ 58 h

At least take two or three days. In fact, between the two copies of NFS clusters transmission rate not so fast, but in the case of a large number of small files, slower, down to 10MB / s it is very normal. How smooth, how to move, how to efficiently destructive? Service structures, risk plan, double the cluster and so are worth careful consideration.

What is NFS?

Distributed File System: Distributed file system, DFS, also known as Network File System: Network File System. File sharing that allows multiple hosts on a network file system that allows multiple users to share files across multiple machines and storage space.

Smooth migration of heavy and difficult

• Adaptogens business scenario, there are a large number of small files.
• Can random read and write.
• It can be mounted directly to use.
• High availability, stability, part of the machine fault can zero impact on the business can recover quickly stop.
• Online service has been running, how to achieve customer complete without a sense of migration.
• R & D personnel from the cluster machine operation and maintenance convenience.

The family planning

Plan describes in detail the following points in the series:

• "Several NFS system contrast, selection reasons"
• "MooseFS deployment and pit row" on this blog omitted, internal and external structures are different with some information confidential, you can find someone else's instructions or view the official website of the network manual.
• "TB-level NFS smooth design and implementation of data migration."
• "MooseFS cluster failover operation and maintenance."

Notice

The final program selection is MooseFS, and the whole room is mixed cloth, tested and found more than metalogger + Multi backup copies goal, does not even affect literacy rate, this is a surprise after a use.

Smooth migration program will be a detailed description of the design process, and why the choice of several different schemes Why abandon "design and implementation of a smooth migration NFS TB level data". Finalized the whole room inotify + change_file.list + [delta replication customized single client customized] + diff remaining data copy program.