What is the stick package / unpacking
A full service may be split into a plurality of TCP packets transmitted, there may be a plurality of smaller packets are encapsulated into a large packet transmission, and this is the TCP packet unpacking problems.
Stick package / unpacking issue a general approach, there are four:
Processing fixed length data segments, less than the median space filled.
Message header + message body, message header generally contains the length of the message body, the message type and other information, the actual message body data volume.
Special characters (eg: carriage return) as the end of the message data, in order to achieve sub-message data.
Complex application layer protocol used in this way is relatively small, the coupling network and application layers. The above object of the four ways is to accurately separate the data stream for further analysis processing, the custom protocols, a second embodiment used more, because it can satisfy demand customized protocol development, such as from specific data packets can be placed in the body defining Netty protocol message, message header into other variables (e.g., if necessary: message type, which may correspond to the specific long link netty maintenance heartbeat message, the client request message, the service end of the message processing result, etc.).
Short connection
Connection -> Data transmission -> close the connection
, such as HTTP is stateless short link, the browser and the server once for each HTTP operation, you establish a connection, but the task will end disconnected.
Because the data is received after the connection is disconnected, the data receiving processing each time there will be no contact. This is also one of the reasons the stateless protocol of HTTP.
Long connection
Connection -> transmit data -> remain connected -> transmit data -> ... -> until one connection is closed, multiple client connection is closed.
After establishing a long connection refers to the SOCKET connection regardless of whether the connection is maintained, but less secure.
When connected with a long and short connection?
Long connections used for frequent operation, point to point communications, and the number of connections can not be too often. Each TCP connection requires three-way handshake,
it will take time, if each operation are the first connection, then operate if the processing speed will be much lower, so after each operation will
continue to develop, to send packets directly when processing times OK, do not establish a TCP connection. For example: with a long connection to the database connection, if the
connection frequent communication with the short will cause socket errors, but frequently socket creation is a waste of resources.
And like WEB site http services are generally short link because a long connection to the server, it will consume some resources, like WEB network
station so often thousands or even millions of clients will be more connected with short connection Some provincial resources, if connected with a long and successful at the same time
many thousands of users, if each user occupies one connection, then it can be imagined. So concurrent capacity, but each user does not need frequent
case of need to use short even better fan operation.
Examples of single-end customers
Client:
public static void main(String[] args) throws InterruptedException {
System.out.println("客户端已经启动....");
// 创建负责接收客户端连接
NioEventLoopGroup pGroup = new NioEventLoopGroup();
Bootstrap b = new Bootstrap();
b.group(pGroup).channel(NioSocketChannel.class).handler(new ChannelInitializer<SocketChannel>() {
@Override
protected void initChannel(SocketChannel sc) throws Exception {
sc.pipeline().addLast(new StringDecoder());
sc.pipeline().addLast(new ClientHandler());
}
});
ChannelFuture cf = b.connect("127.0.0.1", 8080).sync();
cf.channel().writeAndFlush(Unpooled.wrappedBuffer("itmayiedu".getBytes()));
cf.channel().writeAndFlush(Unpooled.wrappedBuffer("itmayiedu".getBytes()));
// 等待客户端端口号关闭
cf.channel().closeFuture().sync();
pGroup.shutdownGracefully();
}
public class ClientHandler extends ChannelHandlerAdapter {
@Override
public void channelRead(ChannelHandlerContext ctx, Object msg) throws Exception {
System.out.println("客户端消息:"+msg.toString());
}
}
Server:
public static void main(String[] args) {
try {
System.out.println("服务器端启动");
NioEventLoopGroup pGroup=new NioEventLoopGroup();
NioEventLoopGroup pGroup2=new NioEventLoopGroup();
//创建铺助类
ServerBootstrap serverBootstrap = new ServerBootstrap();
serverBootstrap.group(pGroup, pGroup2).channel(NioServerSocketChannel.class).option(ChannelOption.SO_BACKLOG, 1024)
.option(ChannelOption.SO_SNDBUF,32*1024).option(ChannelOption.SO_RCVBUF, 32*1024)
.childHandler(new ChannelInitializer<io.netty.channel.socket.SocketChannel>() {
@Override
protected void initChannel(io.netty.channel.socket.SocketChannel soc) throws Exception {
soc.pipeline().addLast(new StringDecoder());
soc.pipeline().addLast(new ServerHandler());
};
});
ChannelFuture sync = serverBootstrap.bind(8080).sync();
sync.channel().closeFuture().sync();
pGroup.shutdownGracefully();
pGroup2.shutdownGracefully();
}catch(Exception e) {
e.printStackTrace();
}
}
public class ServerHandler extends ChannelHandlerAdapter {
@Override
public void channelRead(ChannelHandlerContext ctx, Object msg) throws Exception {
super.channelRead(ctx, msg);
System.out.println("服务端消息："+msg.toString());
//ctx.channel().writeAndFlush("serverHandler"+System.currentTimeMillis());
//把消息往下一个Handler传
ctx.fireChannelRead(msg);
}
}
Multi-client
Server:
private Bootstrap bootstrap=new Bootstrap();
private final AtomicInteger index=new AtomicInteger();
//对话对象组
private List<Channel> channels=new ArrayList<Channel>();
public void init(int count){
bootstrap.group(new NioEventLoopGroup());
bootstrap.channel(NioSocketChannel.class);
bootstrap.handler(new ChannelInitializer<Channel>() {
@Override
protected void initChannel(Channel ch) throws Exception {
ch.pipeline().addLast(new StringEncoder());
ch.pipeline().addLast(new StringDecoder());
ch.pipeline().addLast(new ServerHandlerMuch());
}
});
for (int i = 0; i < count; i++) {
Channel channel=bootstrap.connect(new InetSocketAddress("127.0.0.1",8080)).channel();
channels.add(channel);
}
}
//获取可用的channel
private Channel next(){
return getChannel(0);
}
private Channel getChannel(int count) {
Channel channel = channels.get(Math.abs(this.index.getAndIncrement()%channels.size()));
if(count>=channels.size()) {
throw new RuntimeException("没有足够的管道");
}
if(!channel.isActive()) {
//重连
reconnect(channel);
//尝试获取下一个channel
return getChannel(++count);
}
return channel;
}
private void reconnect(Channel channel) {
synchronized (channel) {
int index=channels.indexOf(channel);
bootstrap.connect(new InetSocketAddress("127.0.0.1",8080)).channel();
channels.set(index, channel);
}
}
public static void main(String[] args) {
NetServerMuch netServerMuch=new NetServerMuch();
netServerMuch.init(10);
Scanner scanner = new Scanner(System.in);
while(true) {
System.out.println("请输入：");
String next = scanner.next();
try {
Channel next2 = netServerMuch.next();
next2.writeAndFlush(next);
}catch (Exception e) {
e.printStackTrace();
}
}
}
Serialization agreement with custom serialization protocol
Sequence definition
Serialization (serialization) is the target sequence into a binary form (a byte array), typically also referred to as a sequence of coding (the Encode), is mainly used for network transmission, data persistence and the like;
deserialization (deserialization) is an array of bytes from the restored network, disk, etc. to read the original object, in order to carry out the subsequent operations, generally referred to deserialize decoding (the decode), for decoding the main object of transmission networks, in order to complete the remote call.
Serialization agreement "originator"
I know the first serialization protocol Java serialization mechanism is provided by default, need to serialize a Java object only needs to implement Serializable / Externalizable interfaces and generate a sequence of ID, it will be able to pass this class and ObjectInput ObjectOutput serialization and deserialization
Java disadvantage serialized
java provide their own serialization and also very simple to use, but the remote service invocation rarely use it mainly has the following disadvantages:
Unable to cross-language: I think that for the development of Java serialization is fatal "mistake" because the byte array of Java serialization, other languages can not be deserialized. ;
Stream serialized too large relative to the current mainstream :: serialization protocol, the stream is too Java serialization;
difference sequence of performance: Since Java serialization using the IO synchronous blocking, with respect to the current mainstream serialization protocol, its efficiency is very poor.
The key factors affecting the performance of the serialization
stream size serialized (network bandwidth);
the sequence of performance (CPU resource consumption);
support cross-language (docking and heterogeneous systems development language switching).
netty serialization
Objects
public class TransportObject implements Serializable {
/**
*
*/
private static final long serialVersionUID = -4128002851096643251L;
private String name;
private int id;
private List list = new ArrayList();
public String getName() {
return name;
}
public void setName(String name) {
this.name = name;
}
public int getId() {
return id;
}
public void setId(int id) {
this.id = id;
}
public List<String> getList() {
return list;
}
public void setList(List<String> list) {
this.list = list;
}
@Override
public String toString() {
return "TransportObject{" +
"name='" + name + '\'' +
", id=" + id +
", list=" + list +
'}';
}
}
Server
public class TranspoertObjectServer {
private static int port = 7777;
public static void start() {
ServerBootstrap bootstrap = new ServerBootstrap();
EventLoopGroup boss = new NioEventLoopGroup();
EventLoopGroup worker = new NioEventLoopGroup();
try {
bootstrap.group(boss, worker);
bootstrap.channel(NioServerSocketChannel.class);
bootstrap.option(ChannelOption.SO_BACKLOG, 2048);
bootstrap.childOption(ChannelOption.SO_KEEPALIVE, true);
bootstrap.childOption(ChannelOption.TCP_NODELAY, true);
bootstrap.childHandler(new ChannelInitializer<Channel>() {
@Override
protected void initChannel(Channel ch) throws Exception {
ch.pipeline().addLast(new ObjectEncoder());
ch.pipeline().addLast(new ObjectDecoder(Integer.MAX_VALUE, ClassResolvers.cacheDisabled(null)));
ch.pipeline().addLast(new ObjectServerHandler());
}
});
ChannelFuture channelFuture = bootstrap.bind(port);
System.out.println("server start");
channelFuture.channel().closeFuture().sync();
} catch (Exception e) {
e.printStackTrace();
} finally {
boss.shutdownGracefully();
worker.shutdownGracefully();
}
}
public static void main(String[] args) {
start();
}
}
public class ObjectServerHandler extends ChannelInboundHandlerAdapter {
@Override
public void channelRead(ChannelHandlerContext ctx, Object msg) throws Exception {
System.out.println(msg);
ctx.writeAndFlush(msg);
}
@Override
public void channelInactive(ChannelHandlerContext ctx) throws Exception {
super.channelInactive(ctx);
}
@Override
public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause) throws Exception {
System.out.println(cause.getMessage());
ctx.close();
// super.exceptionCaught(ctx, cause);
}
}
Client
public class TranspoertOjectClient {
private static int port = 7777;
private static String ip = "localhost";
public static void connect(){
Bootstrap bootstrap = new Bootstrap();
EventLoopGroup worker = new NioEventLoopGroup();
try{
bootstrap.group(worker);
bootstrap.channel(NioSocketChannel.class);
bootstrap.handler(new ChannelInitializer<Channel>() {
@Override
protected void initChannel(Channel ch) throws Exception {
ch.pipeline().addLast(new ObjectEncoder());
ch.pipeline().addLast(new ObjectDecoder(Integer.MAX_VALUE,
ClassResolvers.cacheDisabled(null)));
ch.pipeline().addLast(new ObjectClientHandler());
}
});
ChannelFuture channelFuture = bootstrap.connect(ip, port).sync();
channelFuture.channel().closeFuture().sync();
}catch (Exception e){
e.printStackTrace();
}finally {
worker.shutdownGracefully();
}
}
public static void main(String[] args) {
connect();
}
}
public class ObjectClientHandler extends ChannelInboundHandlerAdapter {
private TransportObject getTransportObject(){
TransportObject to = new TransportObject();
to.setId(10001);
to.setName("xiaowang");
to.setList(Arrays.asList("zhangsan","lisi","wangwu"));
return to;
}
@Override
public void channelActive(ChannelHandlerContext ctx) throws Exception {
System.out.println("active");
super.channelActive(ctx);
//发送消息给服务端
ctx.writeAndFlush(getTransportObject());
}
@Override
public void channelInactive(ChannelHandlerContext ctx) throws Exception {
super.channelInactive(ctx);
}
@Override
public void channelRead(ChannelHandlerContext ctx, Object msg) throws Exception {
System.out.println(msg);
ctx.close();
}
@Override
public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause) throws Exception {
//super.exceptionCaught(ctx, cause);
System.out.println(cause.getMessage());
ctx.close();
}
}
XML
(1) the definition:
XML (Extensible Markup Language) is a common serialization and de-serialization protocol, it has a long history, from version 1.0 in 1998, is widely used today.
(2) the advantage of
the man-machine readability
can specify the name of the element or property
(3) disadvantages
serialized data structure contains only the data itself, and the like, and a program type identifier does not include set information.
There must be a class XmlSerializer serialized by default constructor.
Only public properties and sequence fields
can serialization method
file large, complex file format, representing the bandwidth of the transmission
(4) using the scene
as a profile data store
real-time data conversion
JSON
(1) Definition:
the JSON (the JavaScript Object Notation, the JS object tag) is a lightweight data-interchange format. It is based on ECMAScript (w3c developed js specification) of a subset, JSON uses nothing to do with the programming language in text format, but also used the Class C language (including C, C ++, C #, Java, JavaScript, Perl, Python , etc.) habits, concise and clear hierarchy make JSON an ideal data-interchange language.
(2) the advantages of
front high compatibility
data format is simple, easy to read
sequence data after small, good scalability, compatibility
compared to XML, the protocol which is relatively simple, analytical faster
(3) disadvantages
data descriptive than XML difference
is not suitable for the performance requirements for the case where ms level
extra space overhead is relatively large
(4) suitable for scene (alternatively XML)
across a firewall access
high adjustable requirements case
-based Web browser Ajax request
transfer amount data corresponding small, relatively low real-time requirements (for example, second level) service
Fast Json
(1) defined
Fastjson is well-written in a high-performance Java language features JSON library. It uses a "rapid and orderly assumed to match" algorithm, the JSON Parse performance to the extreme.
(2) the advantage of
the interface is easy to use
currently the fastest java language json libraries
(3) shortcomings of
too much emphasis on fast, and deviate from the "standard" and the functional
code quality is not high, incomplete documentation
(4) applies scenes
agreement interactive
Web output
Android client
Thrift
(1) Definition:
the Thrift and not just a sequence of protocol, but a frame RPC. It lets you select the category of the transmission of the communication protocol between the client and the server, namely the text (text) and binary (binary) transmission protocol, in order to save bandwidth and provide transmission efficiency, using binary type of transport protocol in general.
(2) the advantages of
small volume after serialization, speed
supports multiple languages and rich data types
for data fields deletions compatibility with strong
support for binary compression-encoded
(3) disadvantages
user less
across firewall access time, unsafe
relatively difficult is not readable, debugging code
can not be combined with other transport layer protocols (such as HTTP)
does not support the persistence layer directly read and write data, that is not suitable for data persistence serialization protocol
(4) applicable scene
RPC distributed system solutions
Euro
(1) Definition:
Avro belonging to a sub Apache Hadoop. Avro provides two serial format: JSON format or Binary format. Binary format and Protobuf rival in the space overhead and analytical performance, Avro produced solved JSON lengthy and no IDL problems
(2) the advantage of
support for rich data types
simple dynamic language binding function of
self-descriptive attributes
improved data analysis speed
fast compressible binary data
can be remote procedure call RPC
support cross-language programming
(3) disadvantages
for users accustomed to statically typed language is not intuitive
(4) applies scenes
do in Hadoop's Hive, Pig and MapReduce persistence Data Format
protobuf
(1) define the
protocol buffers from open source by Google, tried and tested within Google. It .proto file data structure to be described, the data corresponding to the object may be generated POJO Protobuf structures and related methods and properties through code generation tool.
(2) the advantage of
the serialized stream small, the performance of the high
structured data storage format (XML JSON, etc.)
by sequential identification field can be achieved protocol forward compatibility
easier to manage structured document and Maintenance
(3) disadvantages
need depends on the tools generate code
languages supported by relatively few official only supports the Java, C ++, Python
(4) applicable scene
of the high performance requirements of RPC calls
with good cross-firewall access properties
suitable for application layer object persistence
