Apache Druid0.15.0 installation

Others 2019-08-01 21:05:36 views: null

Druid0.15.0 installation documentation

1 cluster planning

  • Master Coordinator and comprising Overlord, 4-core 16G * 2;
  • Historical data and comprising MiddleManager, 16 nuclear 64G * 3;
  • Broker and query comprising Router, 4-core 16G * 1.
1.1 Hadoop profile settings

The storage installation as HDFS into three data nodes, / data1 / druid / druid-0.15.0 / conf / druid / cluster / _common directory, corresponding soft profile directory hadoop of this step in order to identify Hadoop HA mode, otherwise use HDFS does not recognize the depth of the storage path.

ln -s /usr/hdp/2.6.5.0-292/hadoop/conf hadoop-xml
1.2 jdk1.8 installation, will be omitted here.
1.3 data datanode HDFS as a node, is omitted here
1.4 common configuration
  1. log4j2.xml Configuration
<Configuration status="WARN">
    <Properties>
        <Property name="log.path">/data1/druid/log</Property>
    </Properties>
    <Appenders>
        <Console name="Console" target="SYSTEM_OUT">
            <PatternLayout pattern="%d{ISO8601} %p [%t] %c - %m%n"/>
        </Console>
        <File name="log" fileName="${log.path}/one.log" append="false">
            <PatternLayout pattern="[%d{yyyy-MM-dd HH:mm:ss:SSS}] [%p] - %l - %m%n"/>
        </File>
        <RollingFile name="RollingFileInfo" fileName="${log.path}/druid-data.log"
                     filePattern="${log.path}/druid-data-%d{yyyy-MM-dd}-%i.out">
            <ThresholdFilter level="info" onMatch="ACCEPT" onMismatch="DENY"/>
            <PatternLayout pattern="[%d{yyyy-MM-dd HH:mm:ss:SSS}] [%p] - %l - %m%n"/>
            <Policies>
                <TimeBasedTriggeringPolicy modulate="true" interval="1"/>
                <SizeBasedTriggeringPolicy size="100 MB"/>
            </Policies>

        </RollingFile>
    </Appenders>
    <Loggers>
        <Root level="info">
            <AppenderRef ref="Console"/>
            <appender-ref ref="RollingFileInfo"/>
            <appender-ref ref="log"/>
        </Root>
    </Loggers>
</Configuration>

 

  1. common.runtime.properties configuration, druid.host druid into the machine where hostname
druid.extensions.loadList=["druid-kafka-eight", "druid-histogram", "druid-datasketches", "mysql-metadata-storage","druid-hdfs-storage","druid-kafka-extraction-namespace","druid-kafka-indexing-service"]
druid.extensions.directory=/data1/druid/druid-0.15.0/extensions
# If you have a different version of Hadoop, place your Hadoop client jar files in your hadoop-dependencies directory
# and uncomment the line below to point to your directory.
druid.extensions.hadoopDependenciesDir=/data1/druid/druid-0.15.0/hadoop-dependencies


#
# Hostname
#
druid.host=bd-prod-slave06
#
# Logging
# Log all runtime properties on startup. Disable to avoid logging properties on startup:
druid.startup.logging.logProperties=true

#
# Zookeeper
#

druid.zk.service.host=bd-prod-master01:2181,bd-prod-master02:2181,bd-prod-slave01:2181
druid.zk.paths.base=/druid

#
# Metadata storage
#

# For Derby server on your Druid Coordinator (only viable in a cluster with a single Coordinator, no fail-over):
# druid.metadata.storage.type=derby
# druid.metadata.storage.connector.connectURI=jdbc:derby://localhost:1527/var/druid/metadata.db;create=true
# druid.metadata.storage.connector.host=localhost
# druid.metadata.storage.connector.port=1527

# For MySQL (make sure to include the MySQL JDBC driver on the classpath):
druid.metadata.storage.type=mysql
druid.metadata.storage.connector.connectURI=jdbc:mysql://bd-prod-master01:3306/druid?useSSL=false&amp;useUnicode=true&amp;characterEncoding=UTF-8
druid.metadata.storage.connector.user=user
druid.metadata.storage.connector.password=password

# For PostgreSQL:
#druid.metadata.storage.type=postgresql
#druid.metadata.storage.connector.connectURI=jdbc:postgresql://db.example.com:5432/druid
#druid.metadata.storage.connector.user=...
#druid.metadata.storage.connector.password=...

#
# Deep storage
#

# For local disk (only viable in a cluster if this is a network mount):
# druid.storage.type=local
# druid.storage.storageDirectory=var/druid/segments

# For HDFS:
druid.storage.type=hdfs
druid.storage.storageDirectory=hdfs://bd-prod/druid/segments

# For S3:
#druid.storage.type=s3
#druid.storage.bucket=your-bucket
#druid.storage.baseKey=druid/segments
#druid.s3.accessKey=...
#druid.s3.secretKey=...

#
# Indexing service logs
#

# For local disk (only viable in a cluster if this is a network mount):
# druid.indexer.logs.type=file
# druid.indexer.logs.directory=var/druid/indexing-logs

# For HDFS:
druid.indexer.logs.type=hdfs
druid.indexer.logs.directory=hdfs://bd-prod/druid/indexing-logs

# For S3:
#druid.indexer.logs.type=s3
#druid.indexer.logs.s3Bucket=your-bucket
#druid.indexer.logs.s3Prefix=druid/indexing-logs

#
# Service discovery
#

druid.selectors.indexing.serviceName=druid/overlord
druid.selectors.coordinator.serviceName=druid/coordinator

#
# Monitoring
#

druid.monitoring.monitors=["org.apache.druid.java.util.metrics.JvmMonitor"]
druid.emitter=noop
druid.emitter.logging.logLevel=info

# Storage type of double columns
# ommiting this will lead to index double as float at the storage layer

druid.indexing.doubleStorage=double

#
# Security
#
druid.server.hiddenProperties=["druid.s3.accessKey","druid.s3.secretKey","druid.metadata.storage.connector.password"]


#
# SQL
#
druid.sql.enable=true

#
# Lookups
#
druid.lookup.enableLookupSyncOnStartup=false

 

2.data node

Into the node data, modify the druid.host;

2.1 historical
  1. /data1/druid/druid-0.15.0/conf/druid/cluster/data/historical/jvm.config
-server
-Xms8g
-Xmx8g
-XX:MaxDirectMemorySize=12g
-XX:+ExitOnOutOfMemoryError
-Duser.timezone=UTC+0800
-Dfile.encoding=UTF-8
-Djava.io.tmpdir=/tmp
-Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager

 

  1. /data1/druid/druid-0.15.0/conf/druid/cluster/data/historical/runtime.properties
druid.service=druid/historical
druid.plaintextPort=9088
druid.segmentCache.numLoadingThreads=16
# HTTP server threads
druid.server.http.numThreads=60

# Processing threads and buffers
druid.processing.buffer.sizeBytes=500000000
druid.processing.numMergeBuffers=4
druid.processing.numThreads=16
druid.processing.tmpDir=/data1/druid/processing

# Segment storage
druid.segmentCache.locations=[{"path":"/data1/druid/segment-cache","maxSize":300000000000}]
druid.server.maxSize=300000000000

# Query cache
druid.historical.cache.useCache=true
druid.historical.cache.populateCache=true
druid.cache.type=caffeine
druid.cache.sizeInBytes=256000000

 

2.2 middleManager
  1. /data1/druid/druid-0.15.0/conf/druid/cluster/data/middleManager/jvm.config
-server
-Xms128m
-Xmx128m
-XX:+ExitOnOutOfMemoryError
-Duser.timezone=UTC+0800
-Dfile.encoding=UTF-8
-Djava.io.tmpdir=/tmp
-Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager

 

  1. /data1/druid/druid-0.15.0/conf/druid/cluster/data/middleManager/runtime.properties
druid.service=druid/middleManager
druid.plaintextPort=8091

# Number of tasks per middleManager
druid.worker.capacity=4

# Task launch parameters
druid.indexer.runner.javaOpts=-server -Xms1g -Xmx1g -XX:MaxDirectMemorySize=1g -Duser.timezone=UTC+0800 -Dfile.encoding=UTF-8 -XX:+ExitOnOutOfMemoryError -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager
druid.indexer.task.baseTaskDir=/data1/druid/task

# HTTP server threads
druid.server.http.numThreads=60

# Processing threads and buffers on Peons
druid.indexer.fork.property.druid.processing.numMergeBuffers=2
druid.indexer.fork.property.druid.processing.buffer.sizeBytes=100000000
druid.indexer.fork.property.druid.processing.numThreads=4

# Hadoop indexing
druid.indexer.task.hadoopWorkingPath=/data1/druid/hadoop-tmp

 

2.3 startup command
 nohup ./bin/start-cluster-data-server >/dev/null 2>&1 &

3 master node

Enter the master node, modify the druid.host common options;

3.1 coordinator-overlord
  1. /data1/druid/druid-0.15.0/conf/druid/cluster/master/coordinator-overlord/jvm.config
-server
-Xms12g
-Xmx12g
-XX:+ExitOnOutOfMemoryError
-XX:+UseG1GC
-Duser.timezone=UTC+0800
-Dfile.encoding=UTF-8
-Djava.io.tmpdir=/tmp
-Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager
-Dderby.stream.error.file=/data1/druid/derby.log

 

  1. /data1/druid/druid-0.15.0/conf/druid/cluster/master/coordinator-overlord/runtime.properties
druid.service=druid/coordinator
druid.plaintextPort=9181

druid.coordinator.startDelay=PT10S
druid.coordinator.period=PT5S

# Run the overlord service in the coordinator process
druid.coordinator.asOverlord.enabled=true
druid.coordinator.asOverlord.overlordService=druid/overlord

druid.indexer.queue.startDelay=PT5S

druid.indexer.runner.type=remote
druid.indexer.storage.type=metadata

 

3.2 startup command
 nohup ./bin/start-cluster-master-no-zk-server >/dev/null 2>&1 &

4 query node

Enter the query node, modify the druid.host common options;

4.1 broker
  1. /data1/druid/druid-0.15.0/conf/druid/cluster/query/broker/jvm.config
-server
-Xms12g
-Xmx12g
-XX:MaxDirectMemorySize=6g
-XX:+ExitOnOutOfMemoryError
-Duser.timezone=UTC+0800
-Dfile.encoding=UTF-8
-Djava.io.tmpdir=/tmp
-Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager

 

  1. /data1/druid/druid-0.15.0/conf/druid/cluster/query/broker/runtime.properties
druid.service=druid/broker
druid.plaintextPort=8182

# HTTP server settings
druid.server.http.numThreads=60

# HTTP client settings
druid.broker.http.numConnections=50
druid.broker.http.maxQueuedBytes=10000000

# Processing threads and buffers
druid.processing.buffer.sizeBytes=500000000
druid.processing.numMergeBuffers=6
druid.processing.numThreads=1
druid.processing.tmpDir=/data1/druid/processing

# Query cache disabled -- push down caching and merging instead
druid.broker.cache.useCache=true
druid.broker.cache.populateCache=true

 

4.2 router
  1. /data1/druid/druid-0.15.0/conf/druid/cluster/query/router/jvm.config
-server
-Xms1g
-Xmx1g
-XX:+UseG1GC
-XX:MaxDirectMemorySize=256m
-XX:+ExitOnOutOfMemoryError
-Duser.timezone=UTC+0800
-Dfile.encoding=UTF-8
-Djava.io.tmpdir=/tmp
-Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager

 

  1. /data1/druid/druid-0.15.0/conf/druid/cluster/query/router/runtime.properties
druid.service=druid/router
druid.plaintextPort=8888

# HTTP proxy
druid.router.http.numConnections=50
druid.router.http.readTimeout=PT5M
druid.router.http.numMaxThreads=100
druid.server.http.numThreads=100

# Service discovery
druid.router.defaultBrokerServiceName=druid/broker
druid.router.coordinatorServiceName=druid/coordinator

# Management proxy to coordinator / overlord: required for unified web console.
druid.router.managementProxy.enabled=true

 

4.3 startup command
nohup ./bin/start-cluster-query-server >/dev/null 2>&1 &

Guess you like

Origin www.cnblogs.com/ChouYarn/p/11282909.html
Recommended
Ranking
#2019110700005
What materials and procedures are required for patent transfer
What is the blockchain Ethereum triplet state root transaction root receipt root
Front-end study notes 04 --- About the insertion of html pictures and videos
Documents required for the filing of WeChat Mini Programs in special industries, the filing process of WeChat Mini Programs in special industries, how to file WeChat Mini Programs in special industries
2017 Qingdao-site tournament I The Squared Mosquito Coil
[BZOJ3165][HEOI2013]Segment (line segment tree without marking)
Kettle series: KettleEasyExpand, an open source Kettle universal plugin by Ma Jinju
The latest tutorial on making framework for iOS
DAX Section 6: Statistical Functions
Daily
More
2024-05-14(9)
2024-05-13(8)
2024-05-12(28)
2024-05-11(32)
2024-05-10(34)
2024-05-09(32)
2024-05-08(18)
2024-05-07(34)
2024-05-06(6)
2024-05-05(0)

Code World

© 2018 codetd.com