Application, split and analysis of logs in node JS projects

1. Logs in nodeJS

1. The log is as follows:

• Access log access log, the servermost important log
• Custom logs, including custom events, error records, etc.
• nodejs File operations,nodejs stream
• Development and use of log function
• Log file splitting, log content analysis
• To log to a file, not stored mysqland redisthe

2. A simple application of the log, the code is as follows:

// process.stdin.pipe(process.stdout)

// const http = require('http')
// const server = http.createServer((req, res) => {
    
    
//   if (req.method === 'POST') {
    
    
//     req.pipe(res) 
//   }
// })

// server.listen(8000)


// 复制文件
// const fs = require('fs')
// const path = require('path')

// const fileName1 = path.resolve(__dirname, 'data.txt')
// const fileName2 = path.resolve(__dirname, 'data-back.txt')

// const readStream = fs.createReadStream(fileName1)
// const writeStream = fs.createWriteStream(fileName2)

// readStream.pipe(writeStream)
// readStream.on('data', chunk => {
    
    
//   console.log(chunk.toString())
// })
// readStream.on('end', () => {
    
    
//   console.log('copy done')
// })


const http = require('http')
const fs = require('fs')
const path = require('path')
const fileName1 = path.resolve(__dirname, 'data.txt')

const server = http.createServer((req, res) => {
    
    
  if (req.method === 'GET') {
    
    
    const readStream = fs.createReadStream(fileName1)
    readStream.pipe(res)
  }
})

server.listen(8344)

3. Log node JSapplication programs, as follows:

• log.js, The code is as follows:

    const fs = require('fs')
    const path = require('path')
    
    // 写日志
    function writeLog(writeStram, log) {
        
        
      writeStram.write(log + '\n') 
    }
    
    // 生成 write stream
    function createWriteStream(fileName) {
        
        
      const fullFileName = path.join(__dirname, '../', '../', 'logs', fileName)
      const writeStream = fs.createWriteStream(fullFileName, {
        
        
        flags: 'a'
      })
      return writeStream
    }
    
    // 写访问日志
    const accessWriteStream = createWriteStream('access.log')
    function access(log) {
        
        
      writeLog(accessWriteStream, log)
    }
    
    module.exports = {
        
        
      access
    }
  

• app.js, Part of the code is as follows:

  // 记录 access log
  access(`${
          
          req.method} -- ${
          
          req.url} -- ${
          
          req.headers['user-agent']} --  ${
          
          Date.now()}`)
  

4. IO The performance bottleneck of the operation is as follows:

• IO, Including network IOand filesIO
• Compared to the CPUcomputing and memory read and write, IOthe prominent feature is the slow
• Increase under the limited hardware resources IOof operational efficiency

5. The log split is as follows:

• The content of the log will accumulate slowly, and it is not easy to deal with it in a file
• Divide log files by time, such as 2019-02-10.access.log
• Implementation: Linuxthe crontabtask, that scheduled task

6. crontab,As follows:

• Set timed tasks, format:**** command
• The access.logcopy and rename2019-02-10.access.log
• Empty access.logfile, continue to accumulate log

7. Log split node JSapplication project, copy.shthe code is as follows:

#!/bin/sh
cd /node-blog/blog1
cp access.log $(date + %Y-%m-%d).access.log
echo "" > access.log

8. The log analysis is as follows:

• As for access.loglogging, analysis of Chromethe proportion of
• Logs are stored in rows, one row is one log
• Use nodejsof readline, based on streamhigh efficiency

9. Log analysis node JSapplication in the project, readline.jsthe code is as follows:

const fs = require('fs')
const path = require('path')
const readline = require('readline')

// 文件名
const fileName = path.join(__dirname, '../', '../', 'logs', 'access.log')
// 创建 read stream
const readStream = fs.createReadStream(fileName)

// 创建 readLine 对象
const rl = readline.createInterface({
    
    
  input: readStream
})

let chromeNum = 0
let sum = 0

// 逐行读取
rl.on('line', (lineData) => {
    
    
  if (!lineData) {
    
    
    return
  }

  // 记录总行数
  sum++

  const arr = lineData.split(' -- ')
  if (arr[2] && arr[2].indexOf('Chrome') > 0) {
    
    
    // 累加 Chrome 的数量
    chromeNum++
  }
})


// 监听读取完成
rl.on('close', () => {
    
    
  console.log('chrome 占比：', + chromeNum / sum)
})