MapReduce parameter tuning
Set file compression (compress input files, Map output files, and reduce output files)
- mapreduce.map.output.compress
HDFS：
dfs.block.size- Mapreduce:
io.sort.mb
io.sort.spill.percent
mapred.local.dir
mapred.map.tasks & mapred.tasktracker.map.tasks.maximum
mapred.reduce.tasks & mapred.tasktracker.reduce.tasks.maximum
mapred.reduce.max.attempts
mapred.reduce.parallel.copies
mapreduce.reduce.shuffle.maxfetchfailures
mapred.child.java.opts
mapred.reduce.tasks.speculative.execution
mapred.compress.map.output & mapred.map.output.compression.codec
mapred.reduce.slowstart.completed.maps
Let's talk about the highlight, which is the parameters of these NBs in mapred. I believe everyone has already understood the prerequisite knowledge (if you don't know the operation mechanism of mapred, it is meaningless to look at this...), first the data needs to be mapped, then merged, then the reduce process is copied, and finally the reduce is performed. The general term for merge and copy can be shuffle. Before you start a job, hadoop needs to know how many map and reduce processes you want to start. If you start with default parameters, there is only one map thread by default. (reduce may also be a..) This speed is very slow.设置map启动个数的参数是mapred.map.tasks，reduce则是mapred.reduce.tasks. These two parameters can be said to be the parameters that play a dominant role in the performance of the entire cluster, and debugging basically revolves around these two parameters. So what do you want to ask about the two parameters that can be modified back and forth? In fact, the setting ratio of these two parameters also directly affects the settings of other parameters.mapred.tasktracker.map.tasks.maximumFirst and foremostmapred.tasktracker.reduce.tasks.maximum. Because these two parameters set the maximum number of map and reduce that can run simultaneously on a server. Now let's assume that a cluster has one namenode and 8 datanodes, which is a very objective cluster. We assume that the above data are all three backups, so the local data rate is 3/8. Suppose you set map.tasks=128 and recur.tasks=64, then your corresponding two maximums should be 16 and 8 or higher, respectively. Because this ensures that all your map and reduce tasks are started at the same time, if you set the maximum reduce to 7, then you will get very bad results, because the number of reduce that can be run by 8 machines at the same time is 56, which is 8 processes worse than the 64 you set. These eight processes will be in the pending state, and will not be able to make up for running until some running reducers are completed, which will greatly increase the running time. Of course, this is not the bigger the better, because the map coexists with the reduce process for a long time, and the coexistence time depends on the settings you set.mapred.reduce.slowstart.completed.maps, if you set it to 0.6. Then the reduce will enter the running state after the map is 60% complete. Therefore, if the map and reduce parameters you set are very large, it will inevitably cause map and reduce to compete for resources, resulting in starvation of some processes, timeout errors, and the most likelysocket.timeouterror is that the network is too busy. Therefore, these need to be added and subtracted appropriately according to the performance of the cluster to achieve the best results. So, what is the best ratio between map and reduce? The apache official website gave us some suggestions, such as setting reduce and map, there is a specific formula between them. But the actual situation can't always be applied with formulas (otherwise there would be no need for a system engineer...). In general, after you set the number of map and reduce processes, you can view the map and reduce progress through the page entry of hadoop's mapred ( http://namenode:50030/jobdetai.jps )如果你发现reduce在33%时，map正好提早一点点到100%，那么这将是最佳的配比, because reduce is at 33% When the copy phase is completed, that is to say, the map needs to complete all map tasks and prepare the data before reduce reaches 33%. Never let reduce wait, but let map finish first.
OK! After this key point is finished, we will look at two closely related parameters,io.sort.mbandmapred.child.java.opts. Because each map or reduce process is a task, it will start a JVM, so in fact, java.opts is also related to the number of map and reduce you start and some other JVM-sensitive parameters. Since the task runs in the JVM, the sort.mb I will mention here is also allocated in the JVM. This value is used to set the available buffer size of a map sort. If the map is sorted in memory When the result reaches a certain value, it will be spilled into the hard disk. Specifically, this value is equal to mbio.sort.spill.percent.. According to the usual setting method, in order to maximize the performance of the JVM, the maximum available memory of the JVM is generally set to twice the amount of memory set by mb. So what is the amount of mb memory set based on? It is mainly related to the amount of result data of one of your maps. If the result data volume of a map is 600M, then if you set mb io.sort.spill.percent.=200M, then the spill will enter the hard disk three times, and then the data will be taken out from the hard disk for copy after the map is completed. Therefore, if the mb setting is 600M, then there is no need for this hard disk access, which saves a lot of time. But the biggest problem is that it consumes a lot of memory. If mb is 600M, then jvm.opts will need to be set to more than 1G. Then, according to the above example, if you start 16 maps and 8 reducers at the same time, then your memory should be at least 24G. Therefore, the settings here should also be careful, because after all, your server has to run many other services.
mapred.reduce.parallel.copiesAll ofmapreduce.reduce.shuffle.maxfetchfailuresthese parameters have some influence on the network. The first is the maximum number of parallel copy threads that reduce can perform. These threads will fetch map results from different datanodes at the same time, and the second is that too many error retries are a problem that reduces performance for many of our applications. Because generally a job is retried once without success, basically no matter how the retry is unsuccessful, it doesn’t matter if the retry is unsuccessful. The key is that this retry consumes a lot of system resources, making other threads possible. It also entered the retry state because of starvation, which is a vicious circle. If your network is really a bottleneck and cannot reach the Gigabit network, then it is recommended to open themapred.compress.map.outputcompression option and configure themapred.map.output.compression.codeccompression encoding format. Generally, snappy is used because this format is relatively fast for compression and decompression. In addition如果你的集群是异构的, some machines have good performance and some are poor, so it is recommended to turn onmapred.reduce.tasks.speculative.executionspeculative execution, which is conducive to optimizing process allocation and improving cluster performance.
source:https://blog.csdn.net/yaoxiaochuang/article/details/50959158 - Mapreduce performance tuning 2
Hive parameter tuning
- Hive achieves parallel processing by dividing the query into one or more MapReduce tasks. Each task may have multiple mapper and reducer tasks, at least some of which can be executed in parallel.
- Determining the optimal number of mappers and reducers depends on a number of variables, such as the amount of input data and the type of operations to be performed on that data.
- It is necessary to maintain balance. For big data systems such as Spark/Hadoop, the amount of data is not terrible. What is terrible is that the data is skewed and the operations processed by each node are not balanced.
If there are too many mapper or reducer tasks, it will cause too much overhead in the startup phase, scheduling and running the job; if the number is set too small, it may not take full advantage of the inherent parallelism of the cluster.
The number of reducers of the Job submitted by mapred.reduce.tasks
, using the configuration of Hadoop Client.- hive.mapred.mode
Map/Redure mode, if set to strict, will prohibit 3 types of queries:
1. The where filter condition of the partition table must contain the partition field;
2. For the query using the order by statement, limit must be used Statement (order by statement will distribute all the result set data to the same reducer for processing, adding limit statement can prevent the reducer from executing for a long time)
3. The query that restricts the Cartesian product is the where statement , without the on statement.
'nonstrict' hive.merge.mapfiles
merge small files at the end of the Map-only task.
Whether to enable merging of map-side small files, when the Hive input consists of many small files, since each small file will start a map task, if the file is too small, It will make the map task startup and initialization time longer than the logic processing time, resulting in waste of resources and even OOM. For this reason, when we start a task and find that the amount of input data is small but the number of tasks is large, we need to pay attention to input merging in the front-end of Map. Of course, when we write data to a table, we also need to pay attention to the output file size
trueWhether hive.merge.mapredfiles
is enabled to merge Map/Reduce small files, that is, whether to merge small files at the end of the Map-Reduce task
false- Whether hive.exec.parallel enables
concurrent submission of map/reduce jobs.
false - hive.limit.optimize.enable
When using the LIMIT statement, it can sample the data source, avoid executing the entire query statement, and then return partial results.
However , this feature has the disadvantage that it is possible that the useful data in the input will never be retrieved. processed. - hive.exec.reducers.bytes.per.reducer
The average load bytes per reducer.
1000000000 - hive.exec.reducers.max
sets the upper limit of the number of reducers, which can prevent a query from consuming too many reducer resources. For the setting of the value of this attribute, a suggested calculation formula is as follows:
(The total number of reduce slots in the cluster *1.5) / (average number of queries in execution) The
multiple of 1.5 is an empirical factor to prevent underutilization of the cluster.
999 - hive.exec.rowoffset
hive provides 2 kinds of virtual columns: one for the input filename to be divided, and one for the offset within the block within the file. These virtual columns can be used to diagnose queries when hive produces unexpected or null results. With these "fields" the user can see which file or even which data caused the problem:
SELECT
INPUT_FILE_NAME,
BLOCK_OFFSET_INSIDE_FILE,
ROW_OFFSET_INSIDE_BLOCK,
line
FROM hive_text
WHERE line LIKE '%hive%' LIMIT 2;
true - A special optimization of hive.multigroupby.singlemr
, whether to assemble multiple group by operations in a query into a single MapReduce task.
false Whether hive.exec.dynamic.partition enables
dynamic partitioning.
falseAfter hive.exec.dynamic.partition.mode enables
dynamic partitioning, the dynamic partitioning mode has two optional values, strict and nonstrict. Strict requires at least one static partition column, but nonstrict does not.
strict
The maximum number of dynamic partitions allowed by hive.exec.max.dynamic.partitions .
1000hive.exec.max.dynamic.partitions.pernode
The maximum number of dynamic partitions allowed by a single reduce node.
100- hive.exec.default.partition.name
The name of the default dynamic partition, which is used when the dynamic partition column is '' or null. ''
' HIVE_DEFAULT_PARTITION ' hive.exec.mode.local.auto
determines whether Hive should automatically run locally (on GateWay) based on input file size
truehive.exec.mode.local.auto.inputbytes.max
If hive.exec.mode.local.auto is true, when the input file size is smaller than this threshold, it can automatically run in local mode, the default is 128 MB.
134217728Lhive.exec.mode.local.auto.tasks.max
If hive.exec.mode.local.auto is true, when Hive Tasks (Hadoop Jobs) is less than this threshold, it can automatically run in local mode.
4Whether hive.auto.convert.join
automatically converts the Common Join on the Reduce side into a Map Join according to the size of the input small table, so as to speed up the Join speed of the large table associated with the small table.
falsehive.mapred.local.mem
The maximum amount of memory for Mapper/Reducer in local mode, in bytes, 0 means no limit.
0hive.exec.scratchdir
HDFS path to store execution plans of different map/reduce stages and intermediate output results of these stages.
/tmp/<user.name>/hivehive.metastore.warehouse.dir
The default data file storage path for Hive, usually a path writable by HDFS.
"hive.groupby.skewindata
determines whether group by operations support skewed data.
falsehive.default.fileformat
Hive's default output file format, the same as the one specified when creating the table, the options are 'TextFile', 'SequenceFile' or 'RCFile'.
'TextFile'hive.map.aggr
determines whether aggregation operations can be performed on the Map side
truehive.join.emit.interval
The emission interval for Hive Join operations, in milliseconds.
1000hive.mapjoin.cache.numrows
Number of rows cached by Hive Map Join.
25000hive.groupby.mapaggr.checkinterval Check interval
for Map aggregation of Group By operations, in milliseconds.
100000hive.map.aggr.hash.percentmemory
The memory percentage of the virtual machine occupied by hash storage aggregated on the Hive Map side.
0.5hive.map.aggr.hash.min.reduction
The minimum reduce ratio of hash storage for Hive Map-side aggregation.
0.5hive.merge.size.per.task
The size of the merged file of each task, the number of reducers is determined according to this size, the default is 256 M.
256000000hive.merge.smallfiles.avgsize
The average size of the small file group to be merged, the default is 16 M.
16000000mapred.min.split.size
Minimum input split size for Map Reduce Job, use the same configuration as Hadoop Client.
1Whether hive.mergejob.maponly
enables Map Only Merge Job.
truehive.mapjoin.maxsize
Maximum number of rows processed by Map Join. If the number of rows is exceeded, the Map Join process will exit abnormally.
1000000hive.mapjoin.check.memory.rows
sets how many rows to check the memory size. If it exceeds hive.mapjoin.localtask.max.memory.usage, it will exit abnormally and Map Join will fail.
100000Whether hive.optimize.groupby
optimizes group by.
trueWhether hive.optimize.bucketmapjoin
optimizes bucket map join.
false
source: https://www.cnblogs.com/binguo2008/p/7487782.html