druid 实践总结

【1】No TaskLocation avaiable for task

druid 从kafka接入数据，需启动启动相关task，在向overlord提交相关任务之后，任务却无法正确创建task,错误异常如下：

2018-08-07T08:29:53,494 INFO [KafkaIndexTaskClient-timeon.behavior-1] io.druid.indexing.kafka.KafkaIndexTaskClient - No TaskLocation available for task [index_kafka_timeon.behavior_a87911f9690f740_anbnjdma], this task may not have been assigned to a worker yet or may have already completed
2018-08-07T08:29:53,495 INFO [KafkaIndexTaskClient-timeon.behavior-2] io.druid.indexing.kafka.KafkaIndexTaskClient - No TaskLocation available for task [index_kafka_timeon.behavior_27a7a856452c340_ejnolfok], this task may not have been assigned to a worker yet or may have already completed
2018-08-07T08:29:53,497 INFO [KafkaIndexTaskClient-timeon.behavior-0] io.druid.indexing.kafka.KafkaIndexTaskClient - No TaskLocation available for task [index_kafka_timeon.behavior_11e103733926db2_pafpbmki], this task may not have been assigned to a worker yet or may have already completed

此时，druid middlemanager节点无法创建相关task,需上条middlemanage节点的-Xms -Xmx 的值，并kill 并重启启动。

在middlemanage的配置文件runtime.properties的中需注意 druid.processing.buffer.sizeBytes 和javaOpts -Xmx

(druid.processing.numThreads+3) *druid.processing.buffer.sizeBytes < druid.indexer.runner.javaOpts -Xmx

【2】druid删除datasource某个interval之间的数据,例如删除aaaa datasource 中intervals=[1970-01-01T00:00:00.000Z_2018-09-21T00:09:00.000Z]

curl  -X 'DELETE' "http://druid-a:8081/druid/coordinator/v1/datasources/aaaa"

curl -X  'DELETE' 'http://druid-a:8081/druid/coordinator/v1/datasources/aaaa/intervals/1970-01-01T00:00:00.000Z_2018-09-21T00:09:00.000Z'

curl -X 'POST' "http://druid-a:8081/druid/coordinator/v1/datasources/aaaa"

【3】druid deep storage 改为s3

将druid的deep storage 改为s3时，需要改druid的相关参数配置。

common.runtime.properties

druid.extensions.loadList=["mysql-metadata-storage","druid-kafka-indexing-service","kafka-emitter","druid-datasketches","druid-s3-extensions"]


# For S3:
druid.storage.type=s3
druid.storage.bucket=buckt-name      # 不用s3://开头
druid.storage.baseKey=druid/segments
druid.s3.accessKey=xxxxxxxxxxxxx
druid.s3.secretKey=sxxxxxxxxxxxxxxxxxx



# For S3  indexer log:
druid.indexer.logs.type=s3
druid.indexer.logs.s3Bucket=buckt-name      # 不用s3://开头
druid.indexer.logs.s3Prefix=druid/indexing-logs

1: 在druid.extensions.loadList中增加"druid-s3-extensions"

2:配置deep storage 为s3，需要配置bucket ,basekey,accessKey,secretKey 。

3、配置indexer log存储地址为s3 。

注意：默认情况下，druid.extensions.loadList默认加载了"druid-hdfs-storage"，如果不去掉此依赖的话，会存现如下错误：

去掉 "druid-hdfs-storage"之后，druid可正常写入s3。

猜你喜欢