Task running log record is quite necessary, and we want to see the log when the input is not a simple matter, although the point of view logs and stdout in flink ui, once the log becomes large, this view will be extremely nausea.
flink conf provided and logback log4j configuration file, the default is to use log4j, log4j because the company does not support authentication mailbox, so here the choice of excellent logback.
In the lib folder is not logback dependent jar package, so it is necessary to increase
logback-access-1.2.3.jar
logback-classic-1.2.3.jar
logback-core-1.2.3.jar
mail-1.4.7.jar
Then modify logback-console.xml logback.xml logback-yarn.xml three file conf
Mainly due to increase in mail appender
<!-- 邮件发送的appender -->
<appender name="Email" class="ch.qos.logback.classic.net.SMTPAppender">
<smtpHost>smtp.partner.outlook.cn</smtpHost>
<smtpPort>587</smtpPort>
<username>[email protected]</username>
<password></password>
<asynchronousSending>false</asynchronousSending>
<SSL>false</SSL>
<STARTTLS>true</STARTTLS>
<to>xx@xx.com</to>
<from>xx@xx.com</from>
<subject>flink异常邮件</subject>
<!-- html格式 -->
<layout class="ch.qos.logback.classic.html.HTMLLayout">
<Pattern>%date%level%thread%logger{0}%line%message</Pattern>
</layout>
<! - grade filter used here matches the specified level before transmission ->
<filter class = "ch.qos.logback.classic.filter.LevelFilter">
<Level> ERROR </ Level>
</ filter>
<! - - Email only 40 per log entries ->
<class cyclicBufferTracker = "ch.qos.logback.core.spi.CyclicBufferTracker">
<bufferSize> 40 </ bufferSize>
</ cyclicBufferTracker>
</ the appender>
<root level="INFO">
<appender-ref ref="Email"/>
</root>
Such flink configuration is complete.
In the development and application of the time, if you want to log information, you need to add slf4j and logback dependent and the exclusion of log4j dependency (remember), then the error information is recorded, it will be directly sent to the e-mail configuration
Normally, a few mistakes occasionally issued to complete no problem, but once the more serious errors, such as a component fails, you will need to send out the message may have hundreds of thousands of real-time task, it is likely the company mail ringing off the hook.
Based on this situation, we will be able to transfer large buffersize play a role, but not solve the underlying.