ELK study notes the Elasticsearch delete the specified date data (+ script timed tasks)

0x00 Overview

Currently ES version is 6.3.x, after a long period of accumulation, within the ES more and more data, you need to delete the log before the specified log.

For example, only requires 60 days of the log, then it would drop all indexes and logs 60 days before.

 

Before the day XX 0x01 ES delete indexes and data

# ! / Bin / SH 
# ES version 6.3.x 


function delIndex () 
{ 
        # Index Enter the name you want to delete here, for example, to delete filebeat-xxx index log, here to enter 'filebeat-' 
        index_name = $ 1 # the number of days here enter data retention, for example, you want to keep the last 60 days of data, here to enter 60 
        savedays = $ 2 # $ 3 variable is the number of days, for example, you want to keep 60 days of data, enter here 90, it is to be deleted first 60 to 90 days of data of this time, 60 <= 90 the while [savedays -le $ $. 3 ] 
        do # where es is the index date format, some 2019.11.26, some 2019-11-26 
                format_day = ' % the Y .% m.% D ' # format_day ='% Y-M-% D% ' # timestamp part by the index date command here, e.g. 2019.11.26 or 2019-11-26 
                sevendayago = `date -d "
        
        
        
                
                
                
                - $ {} savedays Day " + $ {}` format_day 
                
                # index up a complete data es Here, the data format or filebeat-2019.11.26-2019-11-26 filebeat 
                index = $ $ index_name sevendayago
                 # echo $ sevendayago 
                echo index $ 
                curl -XDELETE " http://127.0.0.1:9200/${index} " 
                # Exit 0 
                
                # delete index finished day 60, day 61 to start deleting the index until the first 90 days 
                savedays = `expr savedays. 1 + $ ` 
        DONE 
} 

# before deleting index close to the index, and then delete the index 
# logic logically consistent with the above to delete 
function closeIndex () 
{ 
        index_name=$1
        savedays=$2
        while [ $savedays -le $3 ]
        do

                format_day='%Y.%m.%d'
                #format_day='%Y-%m-%d'
                sevendayago=`date -d "-${savedays} day " +${format_day}`
                index=$index_name$sevendayago
                #echo $sevendayago
                echo $index
                curl -XPOST "http://127.0.0.1:9200/${index}/_close?pretty"
                #exit 0
                expr $ = `savedays savedays. 1 + ` 
        DONE 
} 

# close to Day 60 Day 90 Index 
closeIndex 'filebeat-' 60 90 # Remove the 60th day to the 90th day index 
deleteIndex 'filebeat-' 60 90

 

0x03 summary

The above shell script content can be made by adding crontab regular tasks.