In the modern information world, we have many photos, documents, videos, etc. to store. I'm sure almost everyone has tried using Google Drive or Dropbox.
Functional Requirements
1. Users can upload and download files from any device. 2. User can update the file by replacing the same file name. 3. Users can share files and folders with others via email links. 4. Users can delete files and folders from any device at any time. 5. Deleted files and folders will be stored in the Recycle Bin for 15 days. 6. The system should support offline editing. Users can add/modify file and folder names while offline and changes will be synced with the remote server once the network is restored. 7. Users can perform file version management to restore previous versions of files. (The system can support multiple versions of the same file that has been changed multiple times, and the bandwidth and required space will increase significantly) 8. The system can support file and folder synchronization across devices. 9. The system allows each user to upload files up to 10 GB for free. 10. The system will provide different market plans for different storage sizes.
non-functional requirements
1. The system must be highly reliable. No uploaded files must be lost. Users can recover their important files. 2. The system can support 24 hours a day, 7 days a week. 3. Users can upgrade their plans at any time and use the system immediately. 4. Users must be able to easily integrate their systems with other applications. For example, the system supports document sharing through other applications. 5. The system can provide delayed or concurrent utilization (which means repeated uploading and downloading of complete files from the cloud in an efficient way).
Storage estimate
1. Number of users = 500 million 2. Number of active users = 100 million 3. Average number of files stored by user = 3004. Average size of each file = 200 KB 5. Total number of files = 500 million * 300 = 150 billion 6. Required Total storage space = 150 billion * 200 KB = 30 PB
System Components
1. Client (installed on your desktop or mobile app to access storage-related applications) • Monitors workspace folders on user machines and maintains consistency by synchronizing files with remote servers. Therefore, it must consist of 4 basic components, namely monitor, chunker, indexer and internal database . • The monitor will watch the local workspace folder and notify the indexer of any actions performed by the user (adding, deleting, replacing, and updating files or folders), and track any changes from other devices that are broadcast by the synchronization service. • Chunker breaks the file into smaller chunks and reconstructs the file from these chunks so that the complete file can be transferred without any missing chunks. The chunking algorithm will detect user-modified parts of the file and transfer only the modified parts to cloud storage, saving bandwidth and synchronization time. • The indexer will process events from the monitor and update the internal metadata database. Once a block has been successfully downloaded or uploaded from cloud storage, the indexer will communicate with the remote sync service, broadcast the changes to other clients, and update the remote metadata database. • The internal metadata database will keep track of all files, blocks, any updated versions, and their location in the file system. •The client application will communicate with the backend cloud storage server by requesting upload, download and edit APIs. The client also interacts with the Remote Sync service to handle any file metadata updates such as file name, size, modification date, etc. 2. Metadata database• Maintain version and metadata information about files, blocks, users, devices, workspaces, and storage locations. •The metadata database can be a relational database, such as MySQL, or a NoSQL database service, such as DynamoDB. • Synchronization services provide a consistent view of files when multiple users are working on the same file at the same time. •Since NoSQL data stores do not support ACID properties, programming code combines ACID properties with the NoSQL database for synchronization of the service's logic. 3. Synchronization service • Update files or folders created by the client. •Synchronize the client's local database with information stored in the remote metadata database. • Use HTTP long polling to get responses from cloud storage, or send files and updates to cloud storage after being offline for a period of time, and once offline again, a notification will be sent to all devices or users. •The synchronization service also attempts to transfer less data between the client and cloud storage to achieve faster response times, therefore using a differentiation algorithm to reduce the amount of data that needs to be synchronized. Instead of transferring the entire file from client to server, the difference between two versions of the file is transferred. Only changed parts (chunks) are transferred. This reduces end-user bandwidth consumption and
Cloud data storage. The server and client will calculate the hash to see if the modified block is updated. This process is called data deduplication.
1. Message Queuing Service • This is a message middleware that handles the number of read and write requests. •A scalable message queue service that supports asynchronous communication between clients and synchronization services. •Availability and reliability must be designed to be optimal for the Message Queuing service. • Request Queue : This queue will be shared among all clients. When the client performs any update and request, this request is sent to the Message Queuing service and then further processed by the synchronization service and finally the metadata database is updated. • Response Queue : Each client has an associated response queue as each client has a separate response queue. • Once a file is updated, the synchronization service notifies all response queues about the changes, and the response queues then notify each client of the changes. 2. Cloud/Block Storage • Stores blocks of files • Uses Amazon S3 service • Client will use API service to interact with cloud storage
• File processing workflow : Client A uploads chunks to cloud storage, then updates metadata and commits changes, then gets confirmation. The server will notify clients B and C about the change. Clients B and C receive metadata changes and download updated chunks. • Data deduplication : Eliminate duplicate copies of data. This also reduces the same data transfer (number of bytes sent) to increase transfer rates. For each incoming block, a hash is tagged, and that hash is calculated and compared to all other hashes of existing blocks. a) Post-processing deduplication : New blocks are stored on the storage device and the blocks will be analyzed to understand if they are duplicates. The advantage is that the client does not need to wait for the hash calculation or lookup to complete before storing the data, which improves storage performance. The disadvantage is that unnecessary duplicate data will be stored in a short period of time, and bandwidth will be transmitted and consumed. b) Inline deduplication : Deduplication hash calculations can be performed in real time so that the system can identify duplicate blocks and then only store non-existent blocks. The advantage is optimized network and storage usage. The disadvantage is that it consumes user time and reduces storage performance.
1. Caching • There are two types of caches in Google Drive/Dropbox system design • Cache for block storage • Cache using ready solutions like Memcache which can store the entire block with its corresponding ID/hash and block servers • Can adopt a least recently used (LRU) caching policy. 2. Load Balancer (LB) • Between clients and block servers • Between clients and metadata servers • Evenly distributes incoming requests between backend servers 3. Security/Permissions and File Sharing • Uploaded/downloaded files will be synchronized with the remote server • Multiple operations on a single file are not allowed (concurrency issues) • If some connection issues occur during a process, the client must re-upload or download the entire file, or resume Download or upload chunks.
System API
1.Upload(string uploadToken, fileInfo file, userInfo user)2.Edit(string authToken, fileInfo file, userInfo user)3.Delete(string authToken, fileInfo file, userInfo user)4.Download(string authToken, fileInfo file, userInfo user)5.GenerateToken (string userName, string password)
High level system architecture
Detailed system architecture
More exciting~