Druid集群配置

其他 2018-07-09 09:38:17 阅读次数: 0


2016年11月18日 15:19:19
阅读数：4355

一、默认端口



二、集群配置思想

（一）本集群有五个节点（master, slave1,slave2,slave3,slave4），节点安排情况如下：

master节点：Mysql server, coordinator node, overlord node 

slave1节点:   historical node， middleManager node

slave2节点:   historical node， middleManager node

slave3节点:   broker node 

slave4节点:   realtime node(没有配置，空着)


（二）配置deep storage（保存冷数据）为HDFS


三、集群配置

（一）配置所有节点（master, slave1,slave2,slave3,slave4）的_common/common.runtime.properties文件

  1. #
  2. # Licensed to Metamarkets Group Inc. (Metamarkets) under one
  3. # or more contributor license agreements. See the NOTICE file
  4. # distributed with this work for additional information
  5. # regarding copyright ownership. Metamarkets licenses this file
  6. # to you under the Apache License, Version 2.0 (the
  7. # "License"); you may not use this file except in compliance
  8. # with the License. You may obtain a copy of the License at
  9. #
  10. # http: //www.apache.org/licenses/LICENSE-2.0
  11. #
  12. # Unless required by applicable law or agreed to in writing,
  13. # software distributed under the License is distributed on an
  14. # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
  15. # KIND, either express or implied. See the License for the
  16. # specific language governing permissions and limitations
  17. # under the License.
  18. #
  20. #
  21. # Extensions
  22. #
  24. # This is not the full list of Druid extensions, but common ones that people often use. You may need to change this list
  25. # based on your particular setup.
  26. #
  27. # Licensed to Metamarkets Group Inc. (Metamarkets) under one
  28. # or more contributor license agreements. See the NOTICE file
  29. # distributed with this work for additional information
  30. # regarding copyright ownership. Metamarkets licenses this file
  31. # to you under the Apache License, Version 2.0 (the
  32. # "License"); you may not use this file except in compliance
  33. # with the License. You may obtain a copy of the License at
  34. #
  35. # http: //www.apache.org/licenses/LICENSE-2.0
  36. #
  37. # Unless required by applicable law or agreed to in writing,
  38. # software distributed under the License is distributed on an
  39. # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
  40. # KIND, either express or implied. See the License for the
  41. # specific language governing permissions and limitations
  42. # under the License.
  43. #
  45. #
  46. # Extensions
  47. #
  49. # This is not the full list of Druid extensions, but common ones that people often use. You may need to change this list
  50. # based on your particular setup.
  51. druid.extensions.loadList=[ "mysql-metadata-storage", "druid-hdfs-storage"]
  54. # If you have a different version of Hadoop, place your Hadoop client jar files in your hadoop-dependencies directory
  55. # and uncomment the line below to point to your directory.
  56. #druid.extensions.hadoopDependenciesDir=/my/dir/hadoop-dependencies
  58. #
  59. # Logging
  60. #
  62. # Log all runtime properties on startup. Disable to avoid logging properties on startup:
  63. druid.startup.logging.logProperties= true
  65. #
  66. # Zookeeper
  67. #
  69. druid.zk.service.host=master: 2181,slave1: 2181,slave2: 2181,slave3: 2181,slave4: 2181
  70. #druid.zk.service.host=master: 2181
  71. druid.zk.paths.base=/druid
  73. #
  74. # Metadata storage
  75. #
  77. # For Derby server on your Druid Coordinator (only viable in a cluster with a single Coordinator, no fail-over):
  78. #druid.metadata.storage.type=derby
  79. #druid.metadata.storage.connector.connectURI=jdbc:derby: //metadata.store.ip:1527/var/druid/metadata.db;create=true
  80. #druid.metadata.storage.connector.host=metadata.store.ip
  81. #druid.metadata.storage.connector.port= 1527
  83. # For MySQL:
  84. druid.metadata.storage.type=mysql
  85. druid.metadata.storage.connector.connectURI=jdbc:mysql: //localhost:3306/druid?characterEncoding=UTF-8
  86. druid.metadata.storage.connector.user=***
  87. druid.metadata.storage.connector.password=***
  89. # For PostgreSQL (make sure to additionally include the Postgres extension):
  90. #druid.metadata.storage.type=postgresql
  91. #druid.metadata.storage.connector.connectURI=jdbc:postgresql: //db.example.com:5432/druid
  92. #druid.metadata.storage.connector.user=...
  93. #druid.metadata.storage.connector.password=...
  95. #
  96. # Deep storage
  97. #
  99. # For local disk (only viable in a cluster if this is a network mount):
  100. #druid.storage.type=local
  101. #druid.storage.storageDirectory=var/druid/segments
  103. # For HDFS (make sure to include the HDFS extension and that your Hadoop config files in the cp):
  104. druid.storage.type=hdfs
  105. druid.storage.storageDirectory=hdfs: //master:9000/druid/segments
  107. # For S3:
  108. #druid.storage.type=s3
  109. #druid.storage.bucket=your-bucket
  110. #druid.storage.baseKey=druid/segments
  111. #druid.s3.accessKey=...
  112. #druid.s3.secretKey=...
  114. #
  115. # Indexing service logs
  116. #
  118. # For local disk (only viable in a cluster if this is a network mount):
  119. #druid.indexer.logs.type=file
  120. #druid.indexer.logs.directory=var/druid/indexing-logs
  122. # For HDFS (make sure to include the HDFS extension and that your Hadoop config files in the cp):
  123. druid.indexer.logs.type=hdfs
  124. druid.indexer.logs.directory=hdfs: //master:9000/druid/indexing-logs
  126. # For S3:
  127. #druid.indexer.logs.type=s3
  128. #druid.indexer.logs.s3Bucket=your-bucket
  129. #druid.indexer.logs.s3Prefix=druid/indexing-logs
  131. #
  132. # Service discovery
  133. #
  135. druid.selectors.indexing.serviceName=druid/overlord
  136. druid.selectors.coordinator.serviceName=druid/coordinator
  138. #
  139. # Monitoring
  140. #
  142. druid.monitoring.monitors=[ "com.metamx.metrics.JvmMonitor"]
  143. druid.emitter=logging
  144. druid.emitter.logging.logLevel=info

（二）配置master节点的coordinator（在coordinator/runtime.properties文件里） 

  1. druid.service=druid/coordinator
  2. druid.port= 8081
  4. druid.coordinator.startDelay=PT30S
  5. druid.coordinator.period=PT30S
  7. #Below it 's append by fenghui for test
  8. druid.host=master



      配置master节点的 overlord（在overlord/runtime.properties文件里） 

  1. druid.service=druid/overlord
  2. druid.port= 8090
  4. #druid.indexer.queue.startDelay=PT30S
  6. druid.indexer.runner.type=remote
  7. druid.indexer.storage.type=metadata
  9. #below it is append by fenghui for test!
  10. druid.host=master
  12. #druid.indexer.logs.type=hdfs
  13. #druid.indexer.logs.directory=/druid/indexing-logs

（三）配置slave1节点的historical （在historical /runtime.properties文件里） 

  1. druid.service=druid/historical
  2. druid.port= 8083
  4. # HTTP server threads
  5. druid.server.http.numThreads= 25
  7. # Processing threads and buffers
  8. druid.processing.buffer.sizeBytes= 6870912
  9. druid.processing.numThreads= 7
  11. # Segment storage
  12. druid.segmentCache.locations=[{ "path": "var/druid/segment-cache", "maxSize"\: 130000000000}]
  13. druid.server.maxSize= 130000000000
  17. druid.host=slave1
  20. druid.historical.cache.useCache= false
  21. druid.historical.cache.populateCache= false


    配置slave1节点的middleManager （在middleManager /runtime.properties文件里） 

  1. druid.service=druid/middleManager
  2. druid.port= 8091
  4. # Number of tasks per middleManager
  5. druid.worker.capacity= 3
  7. # Task launch parameters
  8. druid.indexer.runner.javaOpts=-server -Xmx256m -Duser.timezone=UTC -Dfile.encoding=UTF- 8 -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager
  9. druid.indexer.task.baseTaskDir=var/druid/task
  11. # HTTP server threads
  12. druid.server.http.numThreads= 25
  14. # Processing threads and buffers
  15. druid.processing.buffer.sizeBytes= 65536
  16. druid.processing.numThreads= 2
  18. # Hadoop indexing
  19. druid.indexer.task.hadoopWorkingPath=var/druid/hadoop-tmp
  20. druid.indexer.task.defaultHadoopCoordinates=[ "org.apache.hadoop:hadoop-client:2.3.0"]
  24. #below it 's append by fenghui for test
  25. druid.host=slave1
  27. #druid.indexer.logs.type=hdfs
  28. #druid.indexer.logs.directory=/druid/indexing-logs

（四）配置slave2节点（同slave1节点）


（五）配置slave1节点的broker（在broker/runtime.properties文件里） 

  1. druid.service=druid/broker
  2. druid.port= 8082
  4. # HTTP server threads
  5. druid.broker.http.numConnections= 5
  6. druid.server.http.numThreads= 25
  8. # Processing threads and buffers
  9. druid.processing.buffer.sizeBytes= 32768
  10. druid.processing.numThreads= 2
  12. # Query cache
  13. druid.broker.cache.useCache= true
  14. druid.broker.cache.populateCache= true
  15. druid.cache.type=local
  16. druid.cache.sizeInBytes= 2000000000
  18. druid.host=slave3


三、启动集群命令

    分别到每个节点（master, slave1,slave2,slave3,slave4）去启动相应的服务，命令如下（下面是启动broker的命令）：     java -Xmx256m -Duser.timezone=UTC -Dfile.encoding=UTF-8 -classpath conf/druid/_common:conf/druid/broker:lib/* io.druid.cli.Main server broker &


四、验证集群

（一）打开web端

http://master:8090/console.html



http://master:8081/#/




（二）下载测试文件，把文件解压到每个节点（master, slave1,slave2,slave3,slave4）的druid-0.9.1.1目录下，然后再file目录下执行 bash submit_task.sh命令，可以在http://master:8090/console.html下查看状态

    在file目录下执行bash query_task.sh命令





参考文件：

druid.io_druid.io本地集群搭建 / 扩展集群搭建

http://druid.io/docs/latest/tutorials/cluster.html

一、默认端口



二、集群配置思想

（一）本集群有五个节点（master, slave1,slave2,slave3,slave4），节点安排情况如下：

master节点：Mysql server, coordinator node, overlord node 

slave1节点:   historical node， middleManager node

slave2节点:   historical node， middleManager node

slave3节点:   broker node 

slave4节点:   realtime node(没有配置，空着)


（二）配置deep storage（保存冷数据）为HDFS


三、集群配置

（一）配置所有节点（master, slave1,slave2,slave3,slave4）的_common/common.runtime.properties文件

  1. #
  2. # Licensed to Metamarkets Group Inc. (Metamarkets) under one
  3. # or more contributor license agreements. See the NOTICE file
  4. # distributed with this work for additional information
  5. # regarding copyright ownership. Metamarkets licenses this file
  6. # to you under the Apache License, Version 2.0 (the
  7. # "License"); you may not use this file except in compliance
  8. # with the License. You may obtain a copy of the License at
  9. #
  10. # http: //www.apache.org/licenses/LICENSE-2.0
  11. #
  12. # Unless required by applicable law or agreed to in writing,
  13. # software distributed under the License is distributed on an
  14. # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
  15. # KIND, either express or implied. See the License for the
  16. # specific language governing permissions and limitations
  17. # under the License.
  18. #
  20. #
  21. # Extensions
  22. #
  24. # This is not the full list of Druid extensions, but common ones that people often use. You may need to change this list
  25. # based on your particular setup.
  26. #
  27. # Licensed to Metamarkets Group Inc. (Metamarkets) under one
  28. # or more contributor license agreements. See the NOTICE file
  29. # distributed with this work for additional information
  30. # regarding copyright ownership. Metamarkets licenses this file
  31. # to you under the Apache License, Version 2.0 (the
  32. # "License"); you may not use this file except in compliance
  33. # with the License. You may obtain a copy of the License at
  34. #
  35. # http: //www.apache.org/licenses/LICENSE-2.0
  36. #
  37. # Unless required by applicable law or agreed to in writing,
  38. # software distributed under the License is distributed on an
  39. # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
  40. # KIND, either express or implied. See the License for the
  41. # specific language governing permissions and limitations
  42. # under the License.
  43. #
  45. #
  46. # Extensions
  47. #
  49. # This is not the full list of Druid extensions, but common ones that people often use. You may need to change this list
  50. # based on your particular setup.
  51. druid.extensions.loadList=[ "mysql-metadata-storage", "druid-hdfs-storage"]
  54. # If you have a different version of Hadoop, place your Hadoop client jar files in your hadoop-dependencies directory
  55. # and uncomment the line below to point to your directory.
  56. #druid.extensions.hadoopDependenciesDir=/my/dir/hadoop-dependencies
  58. #
  59. # Logging
  60. #
  62. # Log all runtime properties on startup. Disable to avoid logging properties on startup:
  63. druid.startup.logging.logProperties= true
  65. #
  66. # Zookeeper
  67. #
  69. druid.zk.service.host=master: 2181,slave1: 2181,slave2: 2181,slave3: 2181,slave4: 2181
  70. #druid.zk.service.host=master: 2181
  71. druid.zk.paths.base=/druid
  73. #
  74. # Metadata storage
  75. #
  77. # For Derby server on your Druid Coordinator (only viable in a cluster with a single Coordinator, no fail-over):
  78. #druid.metadata.storage.type=derby
  79. #druid.metadata.storage.connector.connectURI=jdbc:derby: //metadata.store.ip:1527/var/druid/metadata.db;create=true
  80. #druid.metadata.storage.connector.host=metadata.store.ip
  81. #druid.metadata.storage.connector.port= 1527
  83. # For MySQL:
  84. druid.metadata.storage.type=mysql
  85. druid.metadata.storage.connector.connectURI=jdbc:mysql: //localhost:3306/druid?characterEncoding=UTF-8
  86. druid.metadata.storage.connector.user=***
  87. druid.metadata.storage.connector.password=***
  89. # For PostgreSQL (make sure to additionally include the Postgres extension):
  90. #druid.metadata.storage.type=postgresql
  91. #druid.metadata.storage.connector.connectURI=jdbc:postgresql: //db.example.com:5432/druid
  92. #druid.metadata.storage.connector.user=...
  93. #druid.metadata.storage.connector.password=...
  95. #
  96. # Deep storage
  97. #
  99. # For local disk (only viable in a cluster if this is a network mount):
  100. #druid.storage.type=local
  101. #druid.storage.storageDirectory=var/druid/segments
  103. # For HDFS (make sure to include the HDFS extension and that your Hadoop config files in the cp):
  104. druid.storage.type=hdfs
  105. druid.storage.storageDirectory=hdfs: //master:9000/druid/segments
  107. # For S3:
  108. #druid.storage.type=s3
  109. #druid.storage.bucket=your-bucket
  110. #druid.storage.baseKey=druid/segments
  111. #druid.s3.accessKey=...
  112. #druid.s3.secretKey=...
  114. #
  115. # Indexing service logs
  116. #
  118. # For local disk (only viable in a cluster if this is a network mount):
  119. #druid.indexer.logs.type=file
  120. #druid.indexer.logs.directory=var/druid/indexing-logs
  122. # For HDFS (make sure to include the HDFS extension and that your Hadoop config files in the cp):
  123. druid.indexer.logs.type=hdfs
  124. druid.indexer.logs.directory=hdfs: //master:9000/druid/indexing-logs
  126. # For S3:
  127. #druid.indexer.logs.type=s3
  128. #druid.indexer.logs.s3Bucket=your-bucket
  129. #druid.indexer.logs.s3Prefix=druid/indexing-logs
  131. #
  132. # Service discovery
  133. #
  135. druid.selectors.indexing.serviceName=druid/overlord
  136. druid.selectors.coordinator.serviceName=druid/coordinator
  138. #
  139. # Monitoring
  140. #
  142. druid.monitoring.monitors=[ "com.metamx.metrics.JvmMonitor"]
  143. druid.emitter=logging
  144. druid.emitter.logging.logLevel=info

（二）配置master节点的coordinator（在coordinator/runtime.properties文件里） 

  1. druid.service=druid/coordinator
  2. druid.port= 8081
  4. druid.coordinator.startDelay=PT30S
  5. druid.coordinator.period=PT30S
  7. #Below it 's append by fenghui for test
  8. druid.host=master



      配置master节点的 overlord（在overlord/runtime.properties文件里） 

  1. druid.service=druid/overlord
  2. druid.port= 8090
  4. #druid.indexer.queue.startDelay=PT30S
  6. druid.indexer.runner.type=remote
  7. druid.indexer.storage.type=metadata
  9. #below it is append by fenghui for test!
  10. druid.host=master
  12. #druid.indexer.logs.type=hdfs
  13. #druid.indexer.logs.directory=/druid/indexing-logs

（三）配置slave1节点的historical （在historical /runtime.properties文件里） 

  1. druid.service=druid/historical
  2. druid.port= 8083
  4. # HTTP server threads
  5. druid.server.http.numThreads= 25
  7. # Processing threads and buffers
  8. druid.processing.buffer.sizeBytes= 6870912
  9. druid.processing.numThreads= 7
  11. # Segment storage
  12. druid.segmentCache.locations=[{ "path": "var/druid/segment-cache", "maxSize"\: 130000000000}]
  13. druid.server.maxSize= 130000000000
  17. druid.host=slave1
  20. druid.historical.cache.useCache= false
  21. druid.historical.cache.populateCache= false


    配置slave1节点的middleManager （在middleManager /runtime.properties文件里） 

  1. druid.service=druid/middleManager
  2. druid.port= 8091
  4. # Number of tasks per middleManager
  5. druid.worker.capacity= 3
  7. # Task launch parameters
  8. druid.indexer.runner.javaOpts=-server -Xmx256m -Duser.timezone=UTC -Dfile.encoding=UTF- 8 -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager
  9. druid.indexer.task.baseTaskDir=var/druid/task
  11. # HTTP server threads
  12. druid.server.http.numThreads= 25
  14. # Processing threads and buffers
  15. druid.processing.buffer.sizeBytes= 65536
  16. druid.processing.numThreads= 2
  18. # Hadoop indexing
  19. druid.indexer.task.hadoopWorkingPath=var/druid/hadoop-tmp
  20. druid.indexer.task.defaultHadoopCoordinates=[ "org.apache.hadoop:hadoop-client:2.3.0"]
  24. #below it 's append by fenghui for test
  25. druid.host=slave1
  27. #druid.indexer.logs.type=hdfs
  28. #druid.indexer.logs.directory=/druid/indexing-logs

（四）配置slave2节点（同slave1节点）


（五）配置slave1节点的broker（在broker/runtime.properties文件里） 

  1. druid.service=druid/broker
  2. druid.port= 8082
  4. # HTTP server threads
  5. druid.broker.http.numConnections= 5
  6. druid.server.http.numThreads= 25
  8. # Processing threads and buffers
  9. druid.processing.buffer.sizeBytes= 32768
  10. druid.processing.numThreads= 2
  12. # Query cache
  13. druid.broker.cache.useCache= true
  14. druid.broker.cache.populateCache= true
  15. druid.cache.type=local
  16. druid.cache.sizeInBytes= 2000000000
  18. druid.host=slave3


三、启动集群命令

    分别到每个节点（master, slave1,slave2,slave3,slave4）去启动相应的服务，命令如下（下面是启动broker的命令）：     java -Xmx256m -Duser.timezone=UTC -Dfile.encoding=UTF-8 -classpath conf/druid/_common:conf/druid/broker:lib/* io.druid.cli.Main server broker &


四、验证集群

（一）打开web端

http://master:8090/console.html



http://master:8081/#/




（二）下载测试文件，把文件解压到每个节点（master, slave1,slave2,slave3,slave4）的druid-0.9.1.1目录下，然后再file目录下执行 bash submit_task.sh命令，可以在http://master:8090/console.html下查看状态

    在file目录下执行bash query_task.sh命令





参考文件：

druid.io_druid.io本地集群搭建 / 扩展集群搭建

http://druid.io/docs/latest/tutorials/cluster.html

猜你喜欢

转载自blog.csdn.net/wangshuminjava/article/details/80870157
今日推荐
Apache Doris 2.0.10 版本正式发布！
开源日报 | 大模型开战；大模型独角兽被曝卖身；周鸿祎建议谷歌开源所有产品；最大开源AI社区提供1000万美元共享GPU
开源日报 | Chrome内置Gemini的意义不在于Gemini；中国AI追随之路的五大误区；ECharts创始人“下海”养鱼；谷歌I/O开发者大会什么都有，只是没有惊喜
微软回应中国区AI团队“打包赴美”传闻
基于大语言模型的开源知识库问答系统 MaxKB GitHub Star 数量突破 5,000 个！
美国拟限制 AI 大模型出口中国和俄罗斯
苹果将与 OpenAI 达成协议，将 ChatGPT 应用于 iPhone
周排行
阿里云短信服务平台注册
Windows下的字符串处理(1)
sqoop: mysql导入数据到hdfs, hive, hbase
commons.lang中常用的工具类
离线安装PostgreSQL11.6
使用PyTorch简单实现卷积神经网络模型
一文彻底搞定谱聚类
一道面试题引发的血案
One Chat for Mac(聊天工具)
TCP/IP的底层队列是如何实现的？
每日归档
更多
2024-05-17(34)
2024-05-16(6)
2024-05-15(24)
2024-05-14(0)
2024-05-13(18)
2024-05-12(0)
2024-05-11(38)
2024-05-10(38)
2024-05-09(35)
2024-05-08(42)

代码天地 © 2018 codetd.com

境外服务器   最新电视剧2022