"Data-intensive applications system design" study notes - Chapter Four

Data encoding and Evolution

A data encoding format

Two procedures in representation of the data :

1. Memory, stored in object, structures, lists, arrays, trees, hash tables and other data structures. Access to the cpu optimized.

2. When data is written to a file or transmitted through a network, encodes it into a sequence of bytes.

Two kinds of line transformation process is called serialization and deserialization (encoding and decoding).

Built-in programming language coding scheme disadvantages :

1. Access difficulty between different languages.

2. Decoding requires any class can be instantiated, can lead to security problems, such as remote execution of arbitrary code.

3. Forward and backward compatible with difficulties.

4. low efficiency.

The main formats : JSON, XML, CSV

Binary coding : Thtift, Protocol Buffers, Avro

Second, the data flow mode

1, the database based on the data stream

Backward compatibility and forward compatibility.

Note saved unknown field.

Archive storage: create a database snapshot, such as backup and loaded into the data warehouse or, with the latest model code.

2, service-based data flow

Discloses API server over the network, the client can request the server to connect to the API, the server public API is called service.

The client has the form: web browser, application, js in Ajax.

SOA : Service-Oriented Architecture (micro-services). Client server itself may be another service. The large applications into smaller services by functional area.

The key service-oriented architecture design goal is to make the service available through independent deployment and evolution, to make the application easier to change and maintain. Old and new versions of servers and clients can be run simultaneously. ,

REST : HTTP-based design principles. Emphasis on simple data format, use the URL to identify resources. According to the design principles of REST API is called RESTful.

The SOAP : XML-based protocol used to send an API request. API uses SOAP Web service is called WSDL (web services description language).

RPC : trying to make a service request sent to the remote network looks identical to calling the programming language in the same process in the function or method.

3, message passing based on the data stream

Agent message (the message queue, the messaging middleware): https://zhuanlan.zhihu.com/p/46201859

Message broker advantages :

• If the recipient is not available or overloaded, acts like a buffer.

• Automatic re-send the message to the process to prevent the collapse of the message is lost.

• Preventing the sender needs to know the IP and port of the receiver.

• It supports a message to multiple recipients.