Object Storage
basic concept
There are three types of mainstream storage: a storage block, and the object file storage memory
- NAS (file storage): Network Attached storage, providing storage capabilities and file system web server, the client can access the file system on the NAS, you can also upload and download files, using protocols: SMB, NFS and AFS such as Network File System agreement for the client is the file server on the network.
- SAN (block storage): Storage Area Network, and the difference between NAS is SAN provides only a block storage, and the file system abstraction to the client to manage the use of protocol: Fibre \ Channel, iSCSI \ ATA over Ethnet and HyperSCSI, for customers it is the end of a disk, you can format it, create file systems and mount.
- Object Store: refers to object-oriented objects, advantages collection and file storage block, has fast access speed, large capacity and other characteristics. And easy to match cloud computing, it is a new network storage architecture.
Object storage system (Object-Based Storage System) that combines the advantages of NAS and SAN, but also has direct access to high-speed SAN and NAS data sharing and other advantages, providing high reliability, cross-platform data sharing and secure storage Architecture.
In order to better illustrate the differences of the three, I have an analogy, suppose you have three people want to get from A to B, now there are three modes of transportation. A selected car, bus selection acetate, propionate selected subway. Block storage like cars, fast speed, but small capacity (sedan only take a few people); file storage similar to the bus, slow (there is a bus station and traffic lights need to be considered), but larger capacity (and more can not sit less people); object storage is similar to the subway, high speed, large capacity.
Different data management
- NAS, data in the form of a file of management.
- SAN, data management in the form of data blocks, each block has its own address, no additional background information.
- Object storage, by way of the data management object, one object comprises: data, metadata, global identifier.
Data objects are typically unstructured data, such as: photos, videos or other documents; metadata objects it refers to the associated description of the object, such as: size of the picture, the document owner, etc.; object id is a global unique identifier used to distinguish objects.
Different ways of accessing data
- NAS, access files stored on a remote server via network protocols such as NFS
- SAN, access to data blocks on a SAN address of the data block by
- Object storage through REST web services to access the object
Object storage, easy way to access the object, the object is to be operated through the REST interface with HTTP verbs (GET, POST, PUT, DELETE, etc.) described operations. In addition, there is an access method is to use the client's major cloud providers to operate object.
For example: Amazon's s3cmd, Ali cloud osscmd / ossutil, Tencent cloud coscmd and so on.
Object storage advantages and disadvantages
Let me talk about the advantages, probably before the next mention:
- High scalability: object storage can extend the capacity of tens to hundreds of EB, to take advantage of high-density memory;
- Efficiency: a flat structure, the directory system is not complex effects on performance;
- You need to migrate: object storage system is a scale, as capacity increases, the data is automatically distributed to all nodes according to an algorithm stored objects;
- High security: authentication key object storage usually with HTTP calls itself provides object storage to provide data access; easy access: not only supports HTTP (S) protocol, using the REST API invoke and retrieve data, also increases the NFS and SMB support; cost is relatively low: compared to bulk storage, object store is most cost-effective type of data storage, and cloud computing with, the properties of the object stored in this play the most.
Mention Cons:
- Final consistency: Due to the different positions of the different nodes, the data synchronization may be some time delay or mistake;
- Not easy to do database: objects stored more suitable for storing those files were little changed even the same, and for applications such as database storage needs directly with each other bare disk mapping, or block storage is more appropriate.
Single object storage architecture to achieve
Single object storage architecture
go language
package main
import (
"io"
"net/http"
"os"
"log"
"strings"
)
func main() {
http.HandleFunc("/objects/",Handler)
println("server...")
log.Fatal(http.ListenAndServe("127.0.0.1:8006", nil))
}
func Handler(w http.ResponseWriter, r *http.Request){
println(r)
m := r.Method
if m == http.MethodPut{
Put(w,r)
return
}
if m == http.MethodGet{
Get(w,r)
return
}
w.WriteHeader(http.StatusMethodNotAllowed)
}
func Put(w http.ResponseWriter,r *http.Request){
//C:\Users\Administrator\go\src\awesomeProject\test_file
f,e := os.Create(("C:/Users/Administrator/go/src/awesomeProject/test_file"+"/objects/"+strings.Split(r.URL.EscapedPath(),"/")[2]))
if e != nil {
log.Println(e)
w.WriteHeader(http.StatusInternalServerError)
return
}
defer f.Close()
io.Copy(f,r.Body)
}
func Get(w http.ResponseWriter,r *http.Request){
f,e := os.Open(("C:/Users/Administrator/go/src/awesomeProject/test_file"+"/objects/"+strings.Split(r.URL.EscapedPath(),"/")[2]))
if e != nil {
log.Println(e)
w.WriteHeader(http.StatusNotFound)
return
}
defer f.Close()
io.Copy(w,f)
}
Detailed
main function, register a HTTP handler and start listening port.
Http.HandleFunc role is to register HTTP handlers Handler, if accessed by the client machine's HTTP service and to "/ objects /" at the beginning, then the request will be responsible for handling Handler.
http.ListenAndServer formal listening port, under normal circumstances would have been listening, non-normal circumstances, log.Fatal will for error and exit the program.
http.HandleFunc("/objects/",Handler)
println("server...")
log.Fatal(http.ListenAndServe("127.0.0.1:8006", nil))
Handler function, HTTP request and the most important response Response, Request parameter, depending on the mode of the client request, performs various processing functions: Put and Get functions Functions.
func Handler(w http.ResponseWriter, r *http.Request){
println(r)
m := r.Method
if m == http.MethodPut{
Put(w,r)
return
}
if m == http.MethodGet{
Get(w,r)
return
}
w.WriteHeader(http.StatusMethodNotAllowed)
}
Put function, r.URL variable record URL HTTP request, EscapedPath method for obtaining the results of the subsequent escape path portion, the path is in the form: / objects / <object_name>, and a function-divided strings.Split / objects / < object_name>, divided into "", "objects", <object_name>, go to the third element of the array is <object_name>, os.Create f create the same file in the root directory of the local storage file system, create success will r.Body write to the file f with io.Copy.
func Put(w http.ResponseWriter,r *http.Request){
//C:\Users\Administrator\go\src\awesomeProject\test_file
f,e := os.Create(("C:/Users/Administrator/go/src/awesomeProject/test_file"+"/objects/"+strings.Split(r.URL.EscapedPath(),"/")[2]))
if e != nil {
log.Println(e)
w.WriteHeader(http.StatusInternalServerError)
return
}
defer f.Close()
io.Copy(f,r.Body)
}
Get a similar function with the Put function.
This article is distributed object storage - to achieve the first chapter summarizes the principles, architecture and language Go