hadoop hbase集群环境信息整理

1、硬件及系统版本信息

系统	Linux version 2.6.32-431.el6.x86_64
cpu	Intel x86_64 2.3GHz 40核(逻辑)
内存	256G
硬盘	共12块SAS，每块5.5T
网卡	1000Mb/s，网络延时<0.5ms

2、机器列表

主机

x015015

x015016

x015017

x015018

x015026

x015027

x015028

x015029

x015030

x015031

x015032

y015015

y015016

y015017

y015018

3、系统参数设置

系统配置项	值
最大文件数	655350
最大进程数	655350
swap分区	关闭
预读缓存区大小	2048K
系统磁盘IO调度算法	CFQ
ntp同步	是

4、软件准备

软件	版本
JDK	1.7.0_65-b17
zookeeper	zookeeper-3.4.6
hadoop	2.5.2
hbase	1.1.2

5、安装用户、组

用户名	hadoop
所属组	hero
家目录	/home/hadoop

6、JVM配置

角色	堆大小	JVM参数
NameNode	32G	-Xms32G -Xmx32G -XX:PermSize=256M -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=65 -XX:+CMSParallelRemarkEnabled -XX:+CMSConcurrentMTEnabled -XX:ParallelGCThreads=40 -XX:+DisableExplicitGC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -verbose:gc -XX:OnOutOfMemoryError=kill -9 %p -XX:+UseCMSCompactAtFullCollection -XX:CMSFullGCsBeforeCompaction=1 -XX:+CMSScavengeBeforeRemark -XX:+HeapDumpOnOutOfMemoryError -Xloggc:${hadoop.home}/hadoop-$hadoop_ident_string-$command-$hostname. gc.log
DataNode	16G	基本配置同上
JournalNode	2G	基本配置同上
ZKFC	2G	基本配置同上
HMaster	32G	基本配置同上
RegionServer	64G	基本配置同上

7、hadoop core-site.xml

<name>dfs.permissions</name>

<value>false</value>

</property>

<name>fs.defaultFS</name>

<value>hdfs://mgfscluster</value>

</property>

<name>io.file.buffer.size</name>

</property>

<name>io.bytes.per.checksum</name>

</property>

<name>hadoop.tmp.dir</name>

<name>/home/hadoop/tmp</name>

</property>

<name>ha.zookeeper.quorum</name>

</property>

<name>ha.zookeeper.parent-znode</name>

<value>/mgfscluster/hadoop-ha</value>

</property>

<name>fs.trash.interval</name>

</property>

<name>fs.trash.checkpoint.interval</name>

</property>

<name>ipc.server.listen.queue.size</name>

</property>

<name>file.stream-buffer-size</name>

</property>

<name>file.bytes-per-checksum</name>

</property>

<name>file.blocksize</name>

</property>

<name>io.compression.codecs</name>

<value>org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.DefaultCodec,org.apache.hadoop.io.compress.BZip2Codec,org.apache.hadoop.io.compress.SnappyCodec</value>

</property>

</configuration>

8、hadoop hdfs-site.xml

<name>dfs.nameservices</name>

<value>mgfscluster</value>

</property>

<name>dfs.ha.namenodes.mgfscluster</name>

</property>

<name>dfs.namenode.rpc-address.mgfscluster.nn1</name>

</property>

<name>dfs.namenode.rpc-address.mgfscluster.nn2</name>

</property>

<name>dfs.namenode.http-address.mgfscluster.nn1</name>

</property>

<name>dfs.namenode.http-address.mgfscluster.nn2</name>

</property>

<name>dfs.journalnode.edits.dir</name>

<value>/home/hadoop/data/hdfs/journal/</value>

</property>

<name>dfs.namenode.shared.edits.dir.mgfscluster.nn1</name>

<value>qjournal://x015026:8485;x015027:8485;x015028:8485/mgfscluster</value>

</property>

<name>dfs.namenode.shared.edits.dir.mgfscluster.nn2</name>

<value>qjournal://x015026:8485;x015027:8485;x015028:8485/mgfscluster</value>

</property>

<name>dfs.client.failover.proxy.provider.mgfscluster</name>

<value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>

</property>

<name>dfs.ha.fencing.methods</name>

<value>sshfence</value>

</property>

<name>dfs.ha.fencing.ssh.private-key-files</name>

<value>/home/hadoop/.ssh/id_rsa</value>

</property>

<name>dfs.namenode.name.dir</name>

<value>file:///home/hadoop/data/hdfs/nn0</value>

</property>

<name>dfs.datanode.data.dir</name>

<value>file:///home/hadoop/data/hdfs/dd0,file:///home/hadoop/data/hdfs/dd1,file:///home/hadoop/data/hdfs/dd2,file:///home/hadoop/data/hdfs/dd3,file:///home/hadoop/data/hdfs/dd4,file:///home/hadoop/data/hdfs/dd5,file:///home/hadoop/data/hdfs/dd6,file:///home/hadoop/data/hdfs/dd7,file:///home/hadoop/data/hdfs/dd8,file:///home/hadoop/data/hdfs/dd9,file:///home/hadoop/data/hdfs/dd10,file:///home/hadoop/data/hdfs/dd11</value>

</property>

<name>dfs.ha.automatic-failover.enabled</name>

</property>

<name>dfs.namenode.handler.count</name>

</property>

<name>dfs.datanode.handler.count</name>

</property>

<name>dfs.datanode.du.reserved</name>

</property>

<name>dfs.blockreport.initialDelay</name>

</property>

<name>dfs.datanode.max.transfer.threads</name>

</property>

<name>dfs.heartbeat.interval</name>

</property>

<name>dfs.namenode.stale.datanode.interval</name>

</property>

<name>dfs.blocksize</name>

</property>

<name>dfs.datanode.failed.volumes.tolerated</name>

</property>

<name>dfs.datanode.balance.bandwidthPerSec</name>

</property>

<name>dfs.stream-buffer-size</name>

</property>

<name>dfs.bytes-per-checksum</name>

</property>

<name>dfs.client-write-packet-size</name>

</property>

</configuration>

9、hbase hbase-site.xml

<name>hbase.master</name>

</property>

<name>hbase.rootdir</name>

<value>hdfs://mgfscluster/hbase</value>

</property>

<name>hbase.cluster.distributed</name>

</property>

<name>hbase.zookeeper.quorum</name>

</property>

<name>hbase.zookeeper.property.dataDir</name>

<value>/home/hadoop/data/hbase/zookeeper</value>

</property>

<name>zookeeper.znode.parent</name>

<value>/hbasecluster</value>

</property>

<name>hbase.tmp.dir</name>

<value>/home/hadoop/data/hbase/tmp</value>

</property>

<name>hbase.fs.tmp.dir</name>

<value>/home/hadoop/data/hbase/tmp/hbase-staging</value>

</property>

<name>hbase.local.dir</name>

<value>/home/hadoop/data/hbase/local</value>

</property>

<name>hbase.master.logcleaner.ttl</name>

</property>

<name>hbase.regionserver.logroll.period</name>

</property>

<name>hbase.regionserver.global.memstore.size</name>

</property>

<name>hbase.regionserver.global.memstore.size.lower.limit</name>

</property>

<name>hbase.regionserver.region.split.policy</name>

<value>org.apache.hadoop.hbase.regionserver.IncreasingToUpperBoundRegionSplitPolicy</value>

</property>

<name>hbase.regionserver.regionSplitLimit</name>

</property>

<name>zookeeper.session.timeout</name>

</property>

<name>hbase.normalizer.enabled</name>

</property>

<name>hbase.normalizer.period</name>

</property>

<name>hbase.server.thread.wakefrequency</name>

</property>

<name>hbase.server.versionfile.writeattempts</name>

</property>

<name>hbase.hregion.memstore.flush.size</name>

</property>

<name>hbase.hregion.memstore.block.multiplier</name>

</property>

<name>hbase.hregion.memstore.mslab.enabled</name>

</property>

<name>hbase.hregion.max.filesize</name>

</property>

<name>hbase.hregion.majorcompaction</name>

</property>

<name>hbase.hstore.compactionThreshold</name>

</property>

<name>hbase.hstore.flusher.count</name>

</property>

<name>hbase.hstore.blockingStoreFiles</name>

</property>

<name>hbase.hstore.blockingWaitTime</name>

</property>

<name>hbase.hstore.compaction.min</name>

</property>

<name>hbase.hstore.compaction.max</name>

</property>

<name>hbase.hstore.compaction.min.size</name>

</property>

<name>hbase.hstore.compaction.ratio</name>

</property>

<name>hbase.regionserver.thread.compaction.throttle</name>

</property>

<name>hbase.hstore.compaction.kv.max</name>

</property>

<name>hbase.storescanner.parallel.seek.enable</name>

</property>

<name>hbase.storescanner.parallel.seek.threads</name>

</property>

<name>hfile.block.cache.size</name>

</property>

<name>hbase.rpc.timeout</name>

</property>

<name>hbase.server.compactchecker.interval.multiplier</name>

</property>

<name>hbase.security.authentication</name>

<value>simple</value>

</property>

<name>hbase.regionserver.storefile.refresh.period</name>

</property>

<name>hbase.region.replica.replication.enabled</name>

</property>

<name>hbase.replication</name>

</property>

<name>hbase.ipc.warn.response.time</name>

</property>

<name>hbase.ipc.warn.response.size</name>

</property>

<name>hbase.quota.enabled</name>

</property>

<name>hbase.regionserver.handler.count</name>

</property>

<name>hbase.snapshot.enabled</name>

</property>

<name>hbase.rest.port</name>

</property>

<name>hbase.rest.info.port</name>

</property>

<name>>hbase.rest.readonly</name>

<value>false</value>

</property>

<name>hbase.rest.threads.min</name>

</property>

<name>hbase.thrift.minWorkerThreads</name>

</property>

<name>hbase.thrift.info.port</name>

</property>

<name>thrift.accept-backlog</name>

</property>

</configuration>

hadoop hbase集群环境信息整理

猜你喜欢