programming principles

Reasonable business logic design can avoid 80% of the problems, and the remaining 20% ​​requires some extra work and auxiliary manual processing.

Some principles of program design are summarized as follows:

1. internal design principles
   1. For very serious exceptions that are considered intolerable, interrupt the program;
   2. For tolerable exceptions, exception capture and error log prompts are required. If you scan redis, if there is a problem with a key, it should not affect the processing of other data;
   3. The call of each module inside the program should comprehensively judge the return value and deal with it accordingly. If the shell script uploads hdfs, if the upload fails, the SUCCESS logo cannot be generated;
      
      
2. Processing principle between multiple jobs
   
   1. The downstream should believe the returned status code agreed by the upstream according to the upstream agreement. If the result is found to be inconsistent with the status code, the upstream should modify the process. If the offline hdfs has the SUCCESS flag, but there is no data or there is too little data, you should let the upstream modify it and ensure that there is no problem with the data when there is SUCCESS.
   2. When the downstream uses the upstream results or data, it needs to judge the status code. For example, the SUCCESS flag must be checked for offline data using upstream data.
      
      
3. Monitoring principles of HDFS offline data
       If the development of the program has followed the above two principles, there are still some problems that are difficult to guarantee. For example, there is no problem with the entire business logic design, but there is a problem with the content of the data, which is difficult for the program to cover. At this time, it is necessary to monitor the data, generate alarms in time, and find problems earlier.
       The design of this module adopts the principle of twenty-eight:
           for 80% of the data monitoring methods, try to adopt a unified monitoring method, so that there will be higher scalability in the future. For example, a unified program can be used to monitor the size of the hdfs path, put it into the escort (monitoring platform), and the escort will generate an alarm;
           for 20% of the data monitoring, if the unified monitoring method cannot achieve the purpose, according to the specific business monitoring settings. If you want to cut off the follow-up process after there is a problem with the data, you need to do a separate task for data verification.