Solution to Segments overlap in Kylin

Our company has Hadoop and Spark clusters that have begun to take shape, used for offline data statistics and analysis. Kylin was introduced at the beginning of this year for mature business calculations and calculations. Of course, Hbase is used to store Kylin’s result data. Due to the rapid growth of business data scale, log processing is getting slower and slower, so some hardware upgrades were carried out yesterday. During the upgrade process, Hadoop and other clusters were directly shut down, but a serious flaw occurred. HDFS was shut down first, and then Hbase was shut down. As a result, there may be data in HBASE that was not refreshed to the disk. Today, KYLIN incremental calculations were performed. Sometimes, it all failed. When manually refreshing, it kept reporting: Segments overlap error. After several twists and turns, I finally got it right. I recorded the whole process for reference.

(1) Error expression form

2017-11-01 22:51:08,285 ERROR [http-bio-7070-exec-12] controller.CubeController:296 : Segments overlap: cube_logs_app_event_lodate_eventkey_channel_version_param[201710310000
00_20171101000000] and cube_logs_app_event_lodate_eventkey_channel_version_param[20171031000000_20171101000000]
java.lang.IllegalStateException: Segments overlap: cube_logs_app_event_lodate_eventkey_channel_version_param[20171031000000_20171101000000] and cube_logs_app_event_lodate_eve
ntkey_channel_version_param[20171031000000_20171101000000]
        at org.apache.kylin.cube.CubeValidator.validate(CubeValidator.java:85)
        at org.apache.kylin.cube.CubeManager.updateCubeWithRetry(CubeManager.java:401)
        at org.apache.kylin.cube.CubeManager.updateCube(CubeManager.java:353)
        at org.apache.kylin.cube.CubeManager.appendSegment(CubeManager.java:472)
        at org.apache.kylin.cube.CubeManager.appendSegment(CubeManager.java:459)
        at org.apache.kylin.rest.service.JobService.submitJob(JobService.java:210)

Note: Prioritize using the Action "reload metadata" under the system tab of kylin's WEB console to load the data. If it does not work, go to the following process.

(2) Processing process

I was completely confused at first. I couldn’t see any failed or processing JOBs in the Monitor panel of KYLIN’s WEB console, but the same error was prompted after repeated Rebuild, Merage, Refresh, etc. Next, I looked for help from Du Niang and Google, but no valid information was found. The first thing I thought of was to delete the Segments and then BUILD again, so I found the Rest API interface to operate. The command is as follows:

curl -X DELETE -H "Content-Type:application/json;charset=UTF-8" -H "Authorization: Basic YWRtaW416S1lMSU4="    http://HOST55:7070/kylin/api/cubes/cube_logs_advert_logdate_advert_id_terminal_type/segs/20171031000000_20171101000000

But the above failed. Only valid Segments can be deleted, and only the short ones at the head or tail can be deleted. Like the Segment in an abnormal state here (kylin's metadata exists, but the corresponding table and storage space do not exist in HBASE). is 0) cannot be deleted.

Next, consider whether the metadata can be deleted directly.

Directly back up metadata: ./bin/metastore.sh backup (This step is too important, otherwise KYLIN’s data will be lost accidentally)

Check metadata: ./bin/metastore.sh clean

Clear invalid data: ./bin/metastore.sh clean --delete true

After the above operations, the problem remains the same, without any change.

Next, the only option is to consider manually modifying the metadata and then restore it.

Execute the following commands to find possible error points:

fgrep -r 20171031000000_20171101000000 ./$KYLIN_HOME/meta_backups_OK/meta_2017_11_01_19_30_49

Find the following data at the end of the file $KYLIN_HOME/meta_backups/meta_2017_11_01_19_30_49/cube]$vi cube_logs_advert_logdate_advert_id_terminal_type.json, delete it directly and then save it.

, {
    "uuid" : "32eb368f-d75f-4ea8-951f-0ace62958215",
    "name" : "20171031000000_20171101000000",
    "storage_location_identifier" : "KYLIN_2AOZN2GZX3",
    "date_range_start" : 1509408000000,
    "date_range_end" : 1509494400000,
    "source_offset_start" : 0,
    "source_offset_end" : 0,
    "status" : "NEW",
    "size_kb" : 0,
    "input_records" : 0,
    "input_records_size" : 0,
    "last_build_time" : 0,
    "last_build_job_id" : null,
    "create_time_utc" : 1509492601706,
    "cuboid_shard_nums" : { },
    "total_shards" : 0,
    "blackout_cuboids" : [ ],
    "binary_signature" : null,
    "dictionaries" : null,
    "snapshots" : null,
    "rowkey_stats" : [ ]
  }

The next step is to clean up KYLIN's metadata and then restore it with the modified metadata.

./bin/metastore.sh reset

./bin/metastore.sh restore $KYLIN_HOME/meta_backups/meta_xxxx_xx_xx_xx_xx_xx

At this point, the problem has been solved, and KYLIN can happily rebuild yesterday's problem data.

Solution to Segments overlap in Kylin

Guess you like