The previous article introduced the construction process of sproto , this article introduced how to use sproto
The A side actively sends a request to the B side: call request_encode to encode the lua table, and then pack it with sproto.pack.
The B side receives the request from the A side: unpack with sproto.unpack, and then call request_decode to decode it into a lua table.
End B sends a return packet to end A: encode the lua table with response_encode, and then pack it with sproto.pack.
End A receives the return packet from end B: unpack it with sproto.unpack, and then call request_decode to decode it into a lua table.
Whether it is request_encode or response_encode, the encode interface of the c layer will eventually be called, and both request_decode and response_decode will call the decode interface of the c layer. Encode is responsible for encoding the Lua data table into binary data blocks, while decode is responsible for decoding. The two are complementary operations. Similarly, pack and unpack are complementary operations.
-- lualib/sproto.lua
function sproto:request_encode(protoname, tbl)
...
return core.encode(request,tbl) , p.tag
end
function sproto:response_encode(protoname, tbl)
...
return core.encode(response,tbl)
end
function sproto:request_decode(protoname, ...)
...
return core.decode(request,...) , p.name
end
function sproto:response_decode(protoname, ...)
...
return core.decode(response,...)
end
sproto.pack = core.pack
sproto.unpack = core.unpack1. encode
First put an example (available on github), which will be used when analyzing the source code:
person { name = "Alice" , age = 13, marital = false }
03 00 (fn = 3)
00 00 (id = 0, value in data part)
1C 00 (id = 1, value = 13)
02 00 (id = 2, value = false)
05 00 00 00 (sizeof "Alice")
41 6C 69 63 65 ("Alice")The purpose of encode is to convert the data in the lua table into the type in c according to the specified protocol type, and then encode into a string of binary data blocks according to a specific format.
The final call to sproto_encode api encoding has 5 parameters: st, sproto specifies the type of c structure; buffer, size, the buffer and size of the encoding result, if the buffer is not enough, the buffer will be expanded and re-encoded; cb, corresponding to lsproto The encode api in .c is a c interface, responsible for obtaining the value of the specified key in the lua table, or the value of the specified index position in the array; ud, additional information, contains the virtual stack used for interaction between lua and c, and the corresponding in sproto Type c structure, etc.
In lines 3-6, the encoding result is divided into two parts: header and data. The header length is fixed, equal to the total number of 2 bytes + the number of fields * 2 bytes each field length. As shown in the figure below: the header pointer points to the first address of the buffer, data points to the header+header_sz position, and then
When encoding each field information, the data pointer will move backward, but the header pointer will remain unchanged.
Lines 63-65, pack the total number of fields in big-endian format with a length of 2 bytes (03 00 in the example), data points to header+header_sz, and finally use memmove to connect the header and data block together.
The next step is to encode each field data and do different processing according to the field type:
On lines 11-13, if it is an array, call encode_array, which will be introduced later.
On lines 33-37, if it is a string or a custom type, call encode_object encoding, which will be introduced later.
Lines 16-32, if it is an integer or boolean type, call cb (encode in lsproto.c) to get the value of the corresponding field name in the lua table, and save it to args.value (that is, u). In line 21, the variable value is equal to (the original value + 1)*2, because the encoded 0 has a special function, in order to distinguish the original value of 0.
Line 58-59, finally encode the value in big-endian format with 2 bytes, and store it in the location specified by the header. For example, in the example, 1C 00,(13+1)*2=28=1C, 02 00,(0+1)*2=2=02, note: false in lua will be encoded as 0, and true will be encoded as 1. If it is an array, string or custom type, the value is 0, and the encoded value is 00 00, which means the value is in the data part.
Lines 47-56, if some tags are not set, the tag information needs to be encoded into the header.
// lualib/sproto/sproto.c
int sproto_encode(const struct sproto_type *st, void * buffer, int size, sproto_callback cb, void *ud) {
uint8_t * header = buffer;
uint8_t * data;
int header_sz = SIZEOF_HEADER + st->maxn * SIZEOF_FIELD;
data = header + header_sz;
...
for (i=0;i<st->n;i++) {
struct field *f = &st->f[i];
int type = f->type;
if (type & SPROTO_TARRAY) {
args.type = type & ~SPROTO_TARRAY;
sz = encode_array(cb, &args, data, size);
} else {
switch(type) {
case SPROTO_TINTEGER:
case SPROTO_TBOOLEAN: {
sz = cb(&args);
if (sz == sizeof(uint32_t)) {
if (u.u32 < 0x7fff) {
value = (u.u32+1) * 2;
sz = 2; // sz can be any number > 0
} else {
sz = encode_integer(u.u32, data, size);
}
} else if (sz == sizeof(uint64_t)) {
sz= encode_uint64(u.u64, data, size);
} else {
return -1;
}
break;
}
case SPROTO_TSTRUCT:
case SPROTO_TSTRING:
sz = encode_object(cb, &args, data, size);
break;
}
if (sz > 0) {
uint8_t * record;
int tag;
if (value == 0) {
data += sz;
size -= sz;
}
record = header+SIZEOF_HEADER+SIZEOF_FIELD*index;
tag = f->tag - lasttag - 1;
if (tag > 0) {
// skip tag
tag = (tag - 1) * 2 + 1;
if (tag > 0xffff)
return -1;
record[0] = tag & 0xff;
record[1] = (tag >> 8) & 0xff;
++index;
record += SIZEOF_FIELD;
}
++index;
record[0] = value & 0xff;
record[1] = (value >> 8) & 0xff;
lasttag = f->tag;
}
}
header[0] = index & 0xff;
header[1] = (index >> 8) & 0xff; datasz = data - (header+header_sz); data = header +header_sz; memmove(header + SIZEOF_HEADER + index * SIZEOF_FIELD, data, datasz);
}If it is a string or a custom type, call encode_object encoding. The four parameters are: cb, which is the encode interface in lsproto.c; args, additional parameters; data, the buffer for storing the encoding results, consisting of 4 bytes of length + specific Data composition; size, buffer length
In line 9, fill the length of 4 bytes to the first address of data, as in the example05 00 00 00
In line 5, the data is stored from data+SIZEOF_LENGTH, and the first 4 bytes store the data length
On line 26, if it is a string, copy the string to the specified position, such as in the example41 6C 69 63 65("Alice")
On line 31, if it is a custom type, call sproto_encode recursively on the subtype again
// lualib-src/sproto/sproto.c
static int
encode_object(sproto_callback cb, struct sproto_arg *args, uint8_t *data, int size) {
int sz;
args->value = data+SIZEOF_LENGTH;
args->length = size-SIZEOF_LENGTH;
sz = cb(args);
...
return fill_size(data, sz);
}
static inline int
fill_size(uint8_t * data, int sz) {
data[0] = sz & 0xff;
data[1] = (sz >> 8) & 0xff;
data[2] = (sz >> 16) & 0xff;
data[3] = (sz >> 24) & 0xff;
return sz + SIZEOF_LENGTH;
}
// lualib-src/sproto/lsproto.c
static int
encode(const struct sproto_arg *args) {
...
case SPROTO_TSTRING: {
memcpy(args->value, str, sz);
...
}
case SPROTO_TSTRUCT: {
...
r = sproto_encode(args->subtype, args->value, args->length, encode, &sub);
}
}If it is an array type, call encode_array to encode, traverse the array, encode each element, and also encode the data length into 4 bytes to fill in the front. E.g:
children = {
{ name = "Alice" , age = 13 },
{ name = "Carol" , age = 5 },
}
26 00 00 00 (sizeof children)
0F 00 00 00 (sizeof child 1)
02 00 (fn = 2)
00 00 (id = 0, value in data part)
1C 00 (id = 1, value = 13)
05 00 00 00 (sizeof "Alice")
41 6C 69 63 65 ("Alice")
0F 00 00 00 (sizeof child 2)
02 00 (fn = 2)
00 00 (id = 0, value in data part)
0C 00 (id = 1, value = 5)
05 00 00 00 (sizeof "Carol")
43 61 72 6F 6C ("Carol")
Note: If the array element is an integer, an extra byte will be used between the length and the data to mark whether it is a small integer (less than 2^32) or a large integer. Small integers are stored in 4 bytes (32 bits), and large integers Use 8 bytes (64 bits) to store, for example:
numbers = { 1,2,3,4,5 }
15 00 00 00 (sizeof numbers)
04 ( sizeof int32 )
01 00 00 00 (1)
02 00 00 00 (2)
03 00 00 00 (3)
04 00 00 00 (4)
05 00 00 00 (5)
Summary: The encoded binary data block consists of two parts: header and data. The header contains the total number of fields and the value of each field. The data part is composed of length and specific values. If the field value is 0, it means that the data is in the data part (array, string or custom type); if the last digit of the field value is 1, it means that the field has no data; otherwise the field value can be directly converted to the corresponding lua data (integer or boolean type) ).
2. decode
Understand the encode encoding process, the decode decoding process is the inverse process of encoding, the binary data block is decoded into a lua table. 5 parameters: st, sproto type c structure; data and size, binary data block and length to be decoded; cb, a c interface, that is, decode in lsproto.c, responsible for pushing c type data to the lua virtual stack , And then used by the lua layer; ud, additional parameters, including the lua virtual stack needed in cb.
Lines 9-12, get the first two bytes to indicate the total number of fields fn, stream points to the head, datastream points to the data block
Line 17, decode each field
On line 20, get the value of the field. If the last digit of value is 1, it means that there is no data for the following value/2 tags (lines 22-25);
Line 26, calculate the actual value of value, currentdata points to the current data block (line 27). If it is less than 0, it means array, string or custom type, indicating that the data is in the data part, calculate the data length sz, and then move the datastream to the position of the data block corresponding to the next field (lines 28-33).
Lines 34-37, find out the field information corresponding to the tag, assign it to args, and perform the corresponding conversion according to the args information when calling cb.
Lines 61-66, if it is an integer or boolean type, value is the data itself, call cb to set the location of the specified key in the specified table of the lua virtual stack.
Lines 49-58, if it is a string or a custom type, first get the data from the data part (line 52), and then call cb.
Line 39-42, if it is array type, call decode_array to decode
// lualib-src/sproto/sproto.c
int
sproto_decode(const struct sproto_type *st, const void * data, int size, sproto_callback cb, void *ud) {
struct sproto_arg args;
int total = size;
uint8_t * stream;
uint8_t * datastream;
stream = (void *)data;
fn = toword(stream);
stream += SIZEOF_HEADER;
size -= SIZEOF_HEADER ;
datastream = stream + fn * SIZEOF_FIELD;
size -= fn * SIZEOF_FIELD;
args.ud = ud;
tag = -1;
for (i=0;i<fn;i++) {
uint8_t * currentdata;
struct field * f;
int value = toword(stream + i * SIZEOF_FIELD);
++ tag;
if (value & 1) {
tag += value/2;
continue;
}
value = value/2 - 1;
currentdata = datastream;
if (value < 0) {
uint32_t sz;
sz = todword(datastream);
datastream += sz+SIZEOF_LENGTH;
size -= sz+SIZEOF_LENGTH;
}
f = findtag(st, tag);
args.tagname = f->name;
...
if (value < 0) {
if (f->type & SPROTO_TARRAY) {
if (decode_array(cb, &args, currentdata)) {
return -1;
}
} else {
switch (f->type) {
case SPROTO_TINTEGER: {
...
break;
}
case SPROTO_TSTRING:
case SPROTO_TSTRUCT: {
uint32_t sz = todword(currentdata);
args.value = currentdata+SIZEOF_LENGTH;
args.length = sz;
if (cb(&args))
return -1;
break;
}
}
} else if (f->type != SPROTO_TINTEGER && f->type != SPROTO_TBOOLEAN) {
return -1;
} else {
uint64_t v = value;
args.value = &v;
args.length = sizeof(v);
cb(&args);
}
}
return total - size;
}3. Pack and unpack
After encoding the lua table into a specific binary data block, pack it with pack. The principle is: every 8 bytes form a group, and after packing, it is composed of the first byte + bytes whose original data is not 0. When each bit of the first byte is 0, the original byte is 0. , Otherwise it is a byte to follow. When the first byte is FF, it has special meaning. Assuming the next byte is N, it means that the next (N+1)*8 bytes are all original data. E.g:
unpacked (hex): 08 00 00 00 03 00 02 00 19 00 00 00 aa 01 00 00
packed (hex): 51 08 03 02 31 19 aa 0151 = 0101 0001, counting from right to left, indicating that the 1, 5, and 7 positions of the group are 08, 03, and 02 at a time, and the rest are 0.
Call sproto_pack to pack, 4 parameters: srcv, srcsz original data block and length; bufferv, bufsz store the buffer and length of the packed data.
Lines 5-6, ff_srcstart, ff_desstart point to the source and destination addresses represented by ff respectively
Line 11, pack 8 in groups
Lines 17-19, less than 8, fill with 0
Line 22, call pack_seg, pack it into a specific format, and store it in the buffer
On lines 33 and 40, if ff_n>0, call write_ff, repackage according to the meaning of ff, and then store it in the buffer.
int
sproto_pack(const void * srcv, int srcsz, void * bufferv, int bufsz) {
uint8_t tmp[8];
int i;
const uint8_t * ff_srcstart = NULL;
uint8_t * ff_desstart = NULL;
int ff_n = 0;
int size = 0;
const uint8_t * src = srcv;
uint8_t * buffer = bufferv;
for (i=0;i<srcsz;i+=8) {
int n;
int padding = i+8 - srcsz;
if (padding > 0) {
int j;
memcpy(tmp, src, 8-padding);
for (j=0;j<padding;j++) {
tmp[7-j] = 0;
}
src = tmp;
}
n = pack_seg(src, buffer, bufsz, ff_n);
bufsz -= n;
if (n == 10) {
// first FF
ff_srcstart = src;
ff_desstart = buffer;
ff_n = 1;
} else if (n==8 && ff_n>0) {
++ff_n;
if (ff_n == 256) {
if (bufsz >= 0) {
write_ff(ff_srcstart, ff_desstart, 256*8);
}
ff_n = 0;
}
} else {
if (ff_n > 0) {
if (bufsz >= 0) {
write_ff(ff_srcstart, ff_desstart, ff_n*8);
}
ff_n = 0;
}
}
src += 8;
buffer += n;
size += n;
}
if(bufsz >= 0){
if(ff_n == 1)
write_ff(ff_srcstart, ff_desstart, 8);
else if (ff_n > 1)
write_ff(ff_srcstart, ff_desstart, srcsz - (intptr_t)(ff_srcstart - (const uint8_t*)srcv));
}
return size;
}After understanding the packing principle, unpacking is the inverse process of packing and it becomes very easy. Call sproto_unpack to unpack:
On lines 11-27, if the first byte is ff, calculate the number n of bytes that can be copied directly, and then copy it to the buffer.
Lines 30-50, calculate each bit of the first byte (8 bits in total), if it is 1, copy the following byte to buffer (lines 32-41); otherwise, set buffer to 0 (42- 49 lines).
// lualib-src/sproto/sproto.c
int
sproto_unpack(const void * srcv, int srcsz, void * bufferv, int bufsz) {
const uint8_t * src = srcv;
uint8_t * buffer = bufferv;
int size = 0;
while (srcsz > 0) {
uint8_t header = src[0];
--srcsz;
++src;
if (header == 0xff) {
int n;
if (srcsz < 0) {
return -1;
}
n = (src[0] + 1) * 8;
if (srcsz < n + 1)
return -1;
srcsz -= n + 1;
++src;
if (bufsz >= n) {
memcpy(buffer, src, n);
}
bufsz -= n;
buffer += n;
src += n;
size += n;
} else {
int i;
for (i=0;i<8;i++) {
int nz = (header >> i) & 1;
if (nz) {
if (srcsz < 0)
return -1;
if (bufsz > 0) {
*buffer = *src;
--bufsz;
++buffer;
}
++src;
--srcsz;
} else {
if (bufsz > 0) {
*buffer = 0;
--bufsz;
++buffer;
}
}
++size;
}
}
}
return size;
}This is the end of this article,
On January 13/14, 2021, I will open a four-hour skynet training camp , which is two weeks later. Registration is now open. Those who are interested in game development can subscribe.
The training camp content is roughly as follows:
1. Multi-core concurrent programming
2. Message queue, thread pool
3. Actor message scheduling
4. Network module implementation
5. Time wheel timer implementation
6. Lua/c interface programming
7. Skynet programming essentials
8. Demo demonstrates actor programming thinking
I look forward to everyone working together to create a technological feast for game development.
With the registration screenshots, you can enter the group 973961276 to receive the recording and broadcasting of the last skynet training camp and the preview materials for this period!