SpringBoot + DingTalk == Java application monitoring alarm

Background: In a large-scale system, online exceptions are an inevitable result, so how can we find problems with application operation as early as possible and deal with them in time to avoid expanding the scope of influence?

The answer is to monitor the application and alert on error messages.

This article uses SpringBoot + DingTalk to realize application monitoring and alarm, of course, it can also use enterprise WeChat, SMS notification, etc.

DingTalk Development Platform-Custom Robot Access Documentation

After looking at DingTalk's documentation, we can find that the function of message notification can be realized by calling the Webhook address to send the alarm message to the group chat.

Implementation steps

Get the calling address

• First create a DingTalk group and create a custom robot

Choose one of the security settings to strengthen security and prevent the leaked Webhook address from being sent indiscriminately

• Created successfully, copy the Webhook address, you will need to use it later

Create a SpringBoot application

• Project structure

WarnService

Define the interface for reporting error information

Error messages will be added to the queue MonitorMessageQueue

interface WarnService {
    fun reportErrorMsg(moduleName: String, msg: String)

}

@Service
class WarnServiceImpl : WarnService {
    @Autowired
    private lateinit var monitorMessageQueue: MonitorMessageQueue
    override fun reportErrorMsg(moduleName: String, msg: String) {
        monitorMessageQueue.add(MessageDto().apply {
            this.moduleName = moduleName
            this.content = msg
            this.timestamp = System.currentTimeMillis()
        })
    }
复制代码

MonitorMessageQueue queue

Methods provided by the queue

• start: start the daemon thread
• drain: wait for the timeout to return to the queue element, here set a 30-second timeout to return
• add: add an element to the queue

@Component
@Scope("singleton")
class MonitorMessageQueue {
    private val queue: BlockingQueue<MessageDto> = LinkedBlockingQueue()
    private val logger = LoggerFactory.getLogger(MonitorMessageQueue::class.java)

    @Autowired
    private lateinit var sendService: DataSendService

    @PostConstruct
    private fun start() {
        logger.info("MonitorMessageQueue start")
        val thread = Thread(DataSendThread(this, sendService), "monitor_thread_0")
        thread.isDaemon = true
        thread.start()
    }

    //每个机器人每分钟最多发送20条消息到群里，如果超过20条，会限流10分钟。
    fun drain(): ArrayList<MessageDto> {
        val bulkData = ArrayList<MessageDto>()
        Queues.drain(queue, bulkData, Int.MAX_VALUE, 30, TimeUnit.SECONDS)
        return bulkData
    }

    fun add(message: MessageDto) {
        queue.add(message)
    }
}
复制代码

Alert thread

Monitor the MonitorMessageQueue queue, group and summarize messages, and call the sending service to send messages

class DataSendThread(private val queue: MonitorMessageQueue, private val sendService: DataSendService) : Runnable {
    private val sendCount = AtomicLong(0)
    private val stop = false
    private val logger = LoggerFactory.getLogger(DataSendThread::class.java)
    override fun run() {
        while (!stop) {
            val list = queue.drain()
            if (list.isNullOrEmpty()) {
                logger.info("queue isEmpty")
                return
            }
            val format = SimpleDateFormat("yyyy-MM-dd HH:mm:ss")
            val mid = UUID.randomUUID().toString().replace("-", "")
            val stringBuilder = StringBuilder("[${format.format(System.currentTimeMillis())}][APP监控报警]")
            stringBuilder.append("\n");
            list.groupBy { it.moduleName }.map {
                stringBuilder.append("${it.key}(${it.value.size}次)")
                stringBuilder.append("\n")
                stringBuilder.append(it.value.firstOrNull()?.content ?: "")
                stringBuilder.append("\n")
            }
            stringBuilder.append("http://127.0.0.1/monitor/detail?mid=${mid}")
            sendService.send(stringBuilder.toString())
            logger.info("send success:${sendCount.addAndGet(1)}")
        }
    }
}
复制代码

DataSendService

Process the signature, and actually call the Webhook to send the alarm information to the DingTalk group chat

interface DataSendService {
    fun send(content: String)
}

@Service
class DataSendServiceImpl : DataSendService {
    @Autowired
    private lateinit var restTemplate: RestTemplate

    private fun url(timestamp: Long, sign: String): String {
        return "https://oapi.dingtalk.com/robot/send?access_token=xxxxxxx&timestamp=${timestamp}&sign=$sign"

    }

    override fun send(content: String) {
        val timestamp = System.currentTimeMillis()
        println(
            restTemplate.postForObject(
                url(timestamp, calcSign(timestamp)), mapOf(
                    "msgtype" to "text",
                    "text" to mapOf(
                        "content" to content
                    )
                ),
                String::class.java
            )
        )
    }


    private fun calcSign(timestamp: Long): String {
        val secret = "xxxxxxx"

        val stringToSign = """
            $timestamp
            $secret
            """.trimIndent()
        val mac = Mac.getInstance("HmacSHA256")
        mac.init(SecretKeySpec(secret.toByteArray(charset("UTF-8")), "HmacSHA256"))
        val signData = mac.doFinal(stringToSign.toByteArray(charset("UTF-8")))
        return URLEncoder.encode(String(Base64.getEncoder().encode(signData)), "UTF-8")
    }
}
复制代码

run test cases

@SpringBootTest
internal class WarnServiceImplTest {
    @Autowired
    private lateinit var warnService: WarnService

    @Test
    fun reportErrorMsg() {
        while (true) {
            for (i in 1..((1000 * Math.random()).toInt())) {
                warnService.reportErrorMsg("app-test1", "too many error")
            }
            for (i in (1..((1000 * Math.random()).toInt()))) {
                warnService.reportErrorMsg("app-test2", "too many error")
            }
            Thread.sleep(1000 * 30)
        }
    }
}
复制代码