1 Introduction
LogSearch is a log monitoring tool integrated in the release package of HDP. It is divided into two parts: logsearch portal (hereinafter referred to as portal) and logsearch logfeeder (referred to as logfeeder). LogSearch depends on Solr (on ambari it's called Ambari Infra). Its workflow:
logfeeder monitors the corresponding log files and stores them in Solr, and users can query the logs of each component through the portal.
It meets the requirements of most of us for logs, such as: statistics by time period, statistics of each alarm level, chart display, user login information statistics, etc.
Compared with ELK, the configuration is quite simple for users who use HDP. no code development
2. Prepare
In this example, we take the LogTest component as an example to show how to add the log of the logTest component to logsearch.
2.1 Code preparation
Our service is called LogTest, and its code is also very simple, which is to continuously generate logs. Its code is as follows:
package com.test.logtest;
import org.apache.log4j.Logger;
public class LogTestMain {
private static Logger logger = Logger.getLogger(LogTestMain.class);
private static long count = 0L;
public static void main(String[] args){
while(true){
count++;
logger.fatal("This is a fatal log. count = " + count);
logger.error("This is an error log. count = " + count);
logger.warn("This is a warn log. count = " + count);
logger.info("This is an info log. count = " + count);
logger.debug("This is a debug log. count = " + count);
try {
Thread.sleep(30000);
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
}
Pack it up and set it up as required.
3. Add new components.
Because logsearch can only monitor services managed by ambari, we need to add the logtest service to ambari
- Add the logtest service to ambari hosting
(The new service is a function of ambari, not specific, you can search it yourself)
- When logtest is added to ambari, its directory format is as follows:
[root@hdp91 services]# tree LOGTEST/
LOGTEST/
├── configuration
│ └── logtest-logsearch-conf.xml
├── metainfo.xml
└── package
├── archive.zip
└── scripts
└── logtest.py
Among them, logtest-logsearch-conf.xml is the file required by logsearch, and metainfo.xml is created when logtest is added to ambari. We need to configure these two files.
- metainfo.xml configuration
<metainfo>
<schemaVersion>2.0</schemaVersion>
<services>
<service>
<name>LOGTEST</name>
<displayName>LOGTEST</displayName>
<version>1.0.0</version>
<comment>LOGTEST is a test log sample</comment>
<components>
<component>
<name>Log Test</name>
<displayName>LogTest</displayName>
<category>MASTER</category>
<cardinality>1+</cardinality>
<commandScript>
<script>scripts/logtest.py</script>
<scriptType>PYTHON</scriptType>
<timeout>10000</timeout>
</commandScript>
<!-- 这个logs标签很重要。如果没有，logseach 的portal界面测无法显示 -->
<logs>
<log>
<logId>logtest</logId>
<primary>true</primary>
</log>
</logs>
</component>
</components>
</service>
</services>
</metainfo>The metainfo.xml configuration file is read and displayed by ambari-server. Among them, this tag is more important, and it will be associated with the subsequent logtest-logsearch-conf.xml configuration file. So it must be configured.
Note: I don’t know why, but this part was omitted in the official description of hdp, which caused me to be unable to see the logs of the logtest service on the web page of logsearch when I configured it at the beginning.
- Configure logtest-logsearch-conf.xml
<configuration supports_final="false"
supports_adding_forbidden="true">
<!-- 服务名称 -->
<property>
<name>service_name</name><!--这一项的值，不要改它 -->
<display-name>Service name</display-name> <!--这个例子，依照zookeeper的配置完成的，zk这一项也是这个名字，可以不用管它 -->
<description>Service name for Logsearch Portal (label)</description>
<value>LogTest</value> <!--服务名称 -->
<on-ambari-upgrade add="true" />
</property>
<property>
<name>component_mappings</name> <!-- 这个参数的值，不要改动 -->
<display-name>Component mapping</display-name>
<description>Logsearch component logid mapping list (e.g.:
COMPONENT1:logid1,logid2;COMPONENT2:logid3)</description>
<value>LOGTEST:logtest</value> <!--前面一个为组件名称，后面一个为组件的值，后面一个值比较重要, 与下面的type值，还有metainfo.xml的logid的值保持一致 -->
<on-ambari-upgrade add="true" />
</property>
<property>
<name>content</name>
<display-name>Logfeeder Config</display-name>
<description>Metadata jinja template for Logfeeder which contains grok
patterns for reading service specific logs.</description>
<value>
{
"input": [
{
"type": "logtest",
"rowtype": "service",
"path": "{
{
default('/configurations/logtest_env/lt_log_dir', '/var/log/logtest')}}/logtest*.log"
}
],
"filter": [
{
"filter": "grok",
"conditions": {
"fields": {
"type": [
"logtest"
]
}
},
"log4j_format": "%d{ISO8601} - %-5p [%t:%C{1}@%L] - %m%n",
"multiline_pattern": "^(%{TIMESTAMP_ISO8601:logtime})",
"message_pattern": "(?m)^%{TIMESTAMP_ISO8601:logtime}%{SPACE}-%{SPACE}%{LOGLEVEL:level}%{SPACE}\\[%{DATA:thread_name}\\@%{INT:line_number}\\]%{SPACE}-%{SPACE}%{GREEDYDATA:log_message}",
"post_map_values": {
"logtime": {
"map_date": {
"target_date_pattern": "yyyy-MM-dd HH:mm:ss,SSS"
}
}
}
}
]
}
</value>
<value-attributes>
<type>content</type>
<show-property-name>false</show-property-name>
</value-attributes>
<on-ambari-upgrade add="true" />
</property>
</configuration>
This file is read by logsearch. This will configure the location of the monitoring file, the format of the log, the name of the monitoring log, etc. This document can be divided into three parts. HDP has a place to explain this configuration (you can refer to https://community.hortonworks.com/articles/105297/adding-new-service-component-to-log-search-in-amba.html , the following are some of the combined documents illustrate.
Service_name : Define the service name
component_mappings : Define each component of the service, you need to pay attention to this configuration item:
<value>LOGTEST:logtest</value> This configuration item is very important, an explanation in front of it
Logsearch component logid mapping list (e.g.: COMPONENT1:logid1,logid2;COMPONENT2:logid3)
It can be seen that LOGTEST is the name of the component, and logtest is logid, which is the one we configured in metainfo.xml
- content : This item is to configure the log information corresponding to each logid. It can be divided into two parts: input and filter
. For input, there are the following items: type, rowtype and path. The meanings of these parameters are as follows:
| parameter | illustrate |
|---|---|
| type | logid mentioned in metainfo.xml and component-mapping |
| rowtype | set to "service" |
| path | The path where the log is located, and the name of the log |
The second piece is "filter", which is responsible for filtering logs and other information (because many are similar, so I copied it directly). For the meaning of each parameter, please refer to:
https://github.com/apache/ambari/blob/trunk/ambari-logsearch/ambari-logsearch-logfeeder/docs/filter.md
For configuration files like xxx-logsearch.xml, it is actually relatively clear. For ambari, our logtest is a Service, so it first needs to define the service_name name, which is our first item. A service may be composed of multiple components, like the Service of HDFS, which is composed of Namenode and DataNode. Therefore, after we define the service, we need to define the component and tell logsearch clearly which components the Service is composed of. , which is the second component-mapping we defined. When these definitions are completed, we need to tell logsearch where to find these files and how to filter them. That is the third item, which is the configuration information of log4j of each component.
4. Summary:
- Simple configuration (only need to configure two files metainfo.xml and xxx-logsearch.xml)
- During the whole process, you must pay attention to the value of logid, which is responsible for concatenating metainfo.xml and xxx-logsearch.xml.