提高node.js内部使用http请求性能

这两天刚刚接触node.js，有业务会遇到。通过测试请求单个url，不同的文件大小对性能影响较大。使用strace -p pid -c 统计，futex 90%。后置api带宽没有跑起来,前置web 进程100%，AB测试性能很低。

通过node.js profile，得知性能主要在使用LazyCompile: *afterTransform _stream_transform.js:73:24。使用Transform Stream，阻塞向后置api的请求。

[C++]:
   ticks  total  nonlib   name
  18218   48.3%   48.3%  void node::Buffer::(anonymous namespace)::StringSlice<(node::encoding)1>(v8::FunctionCallbackInfo<v8::Value> const&)
   2029    5.4%    5.4%  __pthread_cond_signal
   1979    5.2%    5.3%  __memcpy_sse2_unaligned_erms
    499    1.3%    1.3%  v8::internal::IncrementalMarking::Step(unsigned long, v8::internal::IncrementalMarking::CompletionAction, v8::internal::IncrementalMarking::ForceCompletionAction, v8::internal::StepOrigin)

[Summary]:
   ticks  total  nonlib   name
   5582   14.8%   14.8%  JavaScript
  31081   82.4%   82.5%  C++
   4545   12.1%   12.1%  GC
      8    0.0%          Shared libraries
   1028    2.7%          Unaccounted

 [C++ entry points]:
   ticks    cpp   total   name
  19788   77.0%   52.5%  v8::internal::Builtin_HandleApiCall(int, v8::internal::Object**, v8::internal::Isolate*)
   1958    7.6%    5.2%  __pthread_cond_signal
   1376    5.4%    3.6%  v8::internal::Runtime_StringBuilderConcat(int, v8::internal::Object**, v8::internal::Isolate*)
    467    1.8%    1.2%  v8::internal::Builtin_ArrayBufferConstructor_ConstructStub(int, v8::internal::Object**, v8::internal::Isolate*)




[Bottom up (heavy) profile]:
  Note: percentage shows a share of a particular caller in the total
  amount of its parent calls.
  Callers occupying less than 1.0% are not shown.

   ticks parent  name
  18218   48.3%  void node::Buffer::(anonymous namespace)::StringSlice<(node::encoding)1>(v8::FunctionCallbackInfo<v8::Value> const&)
  18218  100.0%    v8::internal::Builtin_HandleApiCall(int, v8::internal::Object**, v8::internal::Isolate*)
  18218  100.0%      LazyCompile: ~stringSlice buffer.js:555:21
  15147   83.1%        LazyCompile: *Buffer.toString buffer.js:609:37
  14753   97.4%          LazyCompile: *StringDecoder.write string_decoder.js:70:41
  14753  100.0%            LazyCompile: *afterTransform _stream_transform.js:73:24
    391    2.6%          LazyCompile: ~utf8Text string_decoder.js:198:18
    391  100.0%            LazyCompile: ~StringDecoder.write string_decoder.js:70:41
   2350   12.9%        LazyCompile: *StringDecoder.write string_decoder.js:70:41
   2350  100.0%          LazyCompile: *afterTransform _stream_transform.js:73:24
   2188   93.1%            LazyCompile: *PassThrough._transform _stream_passthrough.js:41:44
    162    6.9%            LazyCompile: ~PassThrough._transform _stream_passthrough.js:41:44
    558    3.1%        LazyCompile: ~Buffer.toString buffer.js:609:37
    558  100.0%          LazyCompile: ~utf8Text string_decoder.js:198:18
    558  100.0%            LazyCompile: ~StringDecoder.write string_decoder.js:70:41

   2029    5.4%  __pthread_cond_signal
   1746   86.1%    LazyCompile: *callback zlib.js:447:20
    106    5.2%    LazyCompile: *_transform zlib.js:360:48
     83   78.3%      LazyCompile: *Transform._write _stream_transform.js:164:38
     83  100.0%        LazyCompile: *ondata _stream_readable.js:636:18
     83  100.0%          LazyCompile: *emit events.js:156:44
     83  100.0%            LazyCompile: *afterTransform _stream_transform.js:73:24
     13   12.3%      LazyCompile: *emit events.js:156:44
     13  100.0%        LazyCompile: *afterWrite _stream_writable.js:460:20
     13  100.0%          LazyCompile: *onwrite _stream_writable.js:432:17
     13  100.0%            LazyCompile: *afterTransform _stream_transform.js:73:24
     10    9.4%      LazyCompile: *prefinish _stream_transform.js:135:19
     10  100.0%        LazyCompile: *emit events.js:156:44
     10  100.0%          LazyCompile: *afterWrite _stream_writable.js:460:20
     10  100.0%            LazyCompile: *onwrite _stream_writable.js:432:17
     91    4.5%    LazyCompile: *_flush zlib.js:301:40
     91  100.0%      LazyCompile: *prefinish _stream_transform.js:135:19
     91  100.0%        LazyCompile: *emit events.js:156:44
     91  100.0%          LazyCompile: *afterWrite _stream_writable.js:460:20
     91  100.0%            LazyCompile: *onwrite _stream_writable.js:432:17





 1979    5.2%  __memcpy_sse2_unaligned_erms
   1237   62.5%    v8::internal::Runtime_StringBuilderConcat(int, v8::internal::Object**, v8::internal::Isolate*)
   1237  100.0%      LazyCompile: *DoJoin native array.js:94:16
   1171   94.7%        LazyCompile: *Join native array.js:119:14
    904   77.2%          LazyCompile: *InnerArrayJoin native array.js:274:24
    904  100.0%            LazyCompile: *join native array.js:287:46
    267   22.8%          LazyCompile: ~InnerArrayJoin native array.js:274:24
    160   59.9%            LazyCompile: ~join native array.js:287:46
    107   40.1%            LazyCompile: *join native array.js:287:46
     66    5.3%        LazyCompile: ~Join native array.js:119:14
     66  100.0%          LazyCompile: ~InnerArrayJoin native array.js:274:24
     66  100.0%            LazyCompile: ~join native array.js:287:46



1132    3.0%  LoadIC: A load IC from the snapshot
    188   16.6%    LazyCompile: *addChunk _stream_readable.js:261:18
    152   80.9%      LazyCompile: *afterTransform _stream_transform.js:73:24
    130   85.5%        LazyCompile: *PassThrough._transform _stream_passthrough.js:41:44
    130  100.0%          LazyCompile: *Transform._write _stream_transform.js:164:38
    130  100.0%            LazyCompile: *ondata _stream_readable.js:636:18
     22   14.5%        LazyCompile: ~PassThrough._transform _stream_passthrough.js:41:44
     22  100.0%          LazyCompile: *Transform._write _stream_transform.js:164:38
     22  100.0%            LazyCompile: *ondata _stream_readable.js:636:18
     32   17.0%      LazyCompile: *callback zlib.js:447:20
      3    1.6%      LazyCompile: *onread net.js:589:16
    142   12.5%    LazyCompile: *ondata _stream_readable.js:636:18
    142  100.0%      LazyCompile: *emit events.js:156:44
     78   54.9%        LazyCompile: *afterTransform _stream_transform.js:73:24
     75   96.2%          LazyCompile: *transform /home/ikh/Documents/node.js/testdemo/node_modules/got/index.js:267:12
     75  100.0%            LazyCompile: *Transform._write _stream_transform.js:164:38
      3    3.8%          LazyCompile: ~transform /home/ikh/Documents/node.js/testdemo/node_modules/got/index.js:267:12
      2   66.7%            LazyCompile: *Transform._write _stream_transform.js:164:38
      1   33.3%            LazyCompile: *Transform._read _stream_transform.js:181:37
     63   44.4%        LazyCompile: *addChunk _stream_readable.js:261:18
     63  100.0%          LazyCompile: *callback zlib.js:447:20
    141   12.5%    LazyCompile: *Readable.read _stream_readable.js:365:35
     65   46.1%      LazyCompile: *addChunk _stream_readable.js:261:18
     59   90.8%        LazyCompile: *afterTransform _stream_transform.js:73:24
     55   93.2%          LazyCompile: *PassThrough._transform _stream_passthrough.js:41:44
     55  100.0%            LazyCompile: *Transform._write _stream_transform.js:164:38
      4    6.8%          LazyCompile: ~PassThrough._transform _stream_passthrough.js:41:44
      4  100.0%            LazyCompile: *Transform._write _stream_transform.js:164:38
      5    7.7%        LazyCompile: *callback zlib.js:447:20
      1    1.5%        LazyCompile: ~readableAddChunk _stream_readable.js:216:26
      1  100.0%          LazyCompile: ~Readable.push _stream_readable.js:191:35
      1  100.0%            LazyCompile: ~onread net.js:589:16
     46   32.6%      LazyCompile: *<anonymous> _stream_readable.js:700:18
     46  100.0%        LazyCompile: *emitNone events.js:104:18
     46  100.0%          LazyCompile: *afterWrite _stream_writable.js:460:20
     46  100.0%            LazyCompile: *_tickCallback internal/process/next_tick.js:151:25
     14    9.9%      LazyCompile: *afterTransform _stream_transform.js:73:24
     14  100.0%        LazyCompile: *transform /home/ikh/Documents/node.js/testdemo/node_modules/got/index.js:267:12
     14  100.0%          LazyCompile: *Transform._write _stream_transform.js:164:38
     14  100.0%            LazyCompile: *ondata _stream_readable.js:636:18
      5    3.5%      LazyCompile: ~resume_ _stream_readable.js:819:17
      5  100.0%        LazyCompile: *_tickCallback internal/process/next_tick.js:151:25
      3    2.1%      LazyCompile: *emitReadable _stream_readable.js:498:22
      2   66.7%        LazyCompile: *done _stream_transform.js:203:14
      1   50.0%          LazyCompile: *flush /home/ikh/Documents/node.js/testdemo/node_modules/got/index.js:284:8
      1  100.0%            LazyCompile: *prefinish _stream_transform.js:135:19
      1   50.0%          LazyCompile: *callback zlib.js:447:20
      1   33.3%        LazyCompile: ~onEofChunk _stream_readable.js:480:20
      1  100.0%          LazyCompile: *Writable.write _stream_writable.js:264:36
      1  100.0%            LazyCompile: *flush /home/ikh/Documents/node.js/testdemo/node_modules/got/index.js:284:8
      2    1.4%      LazyCompile: ~read _http_incoming.js:91:47

最后的代码修改:

'use strict';
const http = require('http');
const got = require('got');

const hostname = '0.0.0.0';
const port = 3000;

async function GetUrl(url) {
    try {
        // decompress Transform Stream(解决对后置api请求少的问题) 
        // encoding cpu 100%
        // 以下没有strace -f
        /**
         * % time     seconds  usecs/call     calls    errors syscall
        ------ ----------- ----------- --------- --------- ----------------
        18.80    0.988517          48     20714           read
        15.30    0.804408         108      7457           close
        15.26    0.802015          25     32431      7287 epoll_ctl
        15.21    0.799565          99      8100      8100 connect
        12.58    0.661128          81      8137           write
         9.06    0.476106          59      8100           socket
         8.98    0.472091          42     11122       142 futex
         3.15    0.165562          23      7200           getsockopt
         0.59    0.030772          34       909         9 accept4
         0.32    0.016636          49       338           mmap
         0.31    0.016210          42       388           brk
         0.27    0.014293         476        30           epoll_wait
         0.14    0.007439          46       163           munmap
         0.04    0.002271          26        88           getpid
         0.00    0.000026          13         2           mprotect
        ------ ----------- ----------- --------- --------- ----------------
        100.00    5.257039                105179     15538 total

         */
        const response = await got(url,{decompress:false,encoding:null}); 
        //await got('http://192.168.31.176:8080/6.txt');
        return response.body.toString('utf-8');
        // return response.body;
            //=> '<!doctype html> ...'
    } catch (error) {
        console.log(error.response.body);
        //=> 'Internal server error ...'
    }
}

const server = http.createServer(async(req, res) => {
    let startTime = new Date().getTime();
    // http://172.16.7.20:808/3.txt size:600kb other url size:1kb
    GetUrl(`http://172.16.7.20:808/3.txt`); 
    GetUrl('http://172.16.7.20:808/8.txt'); 
    GetUrl('http://172.16.7.20:808/9.txt');
    GetUrl('http://172.16.7.20:808/8.txt');
    GetUrl('http://172.16.7.20:808/8.txt');
    GetUrl('http://172.16.7.20:808/8.txt');
    GetUrl('http://172.16.7.20:808/8.txt');
    GetUrl('http://172.16.7.20:808/8.txt');
    GetUrl('http://172.16.7.20:808/8.txt');

   
    res.statusCode = 200;
    res.setHeader('Content-Type', 'text/plain');
    res.end('Hello World\n'+`${new Date().toUTCString()}`);
});

server.listen(port, hostname, () => {
  console.log(`Server running at http://${hostname}:${port}/${Date.now()}`);
});

使用strace -f -p pid 包含子线程的情况，futex还是争抢，有可能是多个请求导致争抢过多。