Pig
42.在master节点安装Pig Clients，打开Linux Shell以MapReduce模式启动它的Grunt，将启动命令和启动结果显示如下。
简写：
[root@master ~]# pig
WARNING: Use "yarn jar" to launch YARNapplications.
17/05/07 07:58:29 INFO pig.ExecTypeProvider: TryingExecType : LOCAL
17/05/07 07:58:29 INFO pig.ExecTypeProvider: TryingExecType : MAPREDUCE
17/05/07 07:58:29 INFO pig.ExecTypeProvider: PickedMAPREDUCE as the ExecType
2017-05-07 07:58:29,081 [main] INFO org.apache.pig.Main - Apache Pig version0.15.0.2.4.3.0-227 (rexported) compiled Sep 10 2016, 00:14:52
2017-05-07 07:58:29,081 [main] INFO org.apache.pig.Main - Logging error messagesto: /root/pig_1494143909080.log
2017-05-07 07:58:29,104 [main] INFO org.apache.pig.impl.util.Utils - Defaultbootup file /root/.pigbootup not found
2017-05-07 07:58:29,507 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine -Connecting to hadoop file system at: hdfs://master:8020
2017-05-07 07:58:30,427 [main] INFO org.apache.pig.PigServer - Pig Script ID forthe session: PIG-default-a45e5d92-ef27-4629-8326-66cbf6605e8e
2017-05-07 07:58:30,870 [main] INFO org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl - Timelineservice address: http://slaver1:8188/ws/v1/timeline/
2017-05-07 07:58:30,977 [main] INFO org.apache.pig.backend.hadoop.ATSService -Created ATS Hook
grunt>
原形：
[root@master ~]# pig -x mapreduce
WARNING: Use "yarn jar" to launch YARNapplications.
17/05/07 08:00:45 INFO pig.ExecTypeProvider: TryingExecType : LOCAL
17/05/07 08:00:46 INFO pig.ExecTypeProvider: TryingExecType : MAPREDUCE
17/05/07 08:00:46 INFO pig.ExecTypeProvider: PickedMAPREDUCE as the ExecType
2017-05-07 08:00:46,060 [main] INFO org.apache.pig.Main - Apache Pig version0.15.0.2.4.3.0-227 (rexported) compiled Sep 10 2016, 00:14:52
2017-05-07 08:00:46,060 [main] INFO org.apache.pig.Main - Logging error messagesto: /root/pig_1494144046058.log
2017-05-07 08:00:46,086 [main] INFO org.apache.pig.impl.util.Utils - Defaultbootup file /root/.pigbootup not found
2017-05-07 08:00:46,540 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine -Connecting to hadoop file system at: hdfs://master:8020
2017-05-07 08:00:47,459 [main] INFO org.apache.pig.PigServer - Pig Script ID forthe session: PIG-default-8abb37de-5d81-487e-99f0-d1fb8eceac03
2017-05-07 08:00:47,913 [main] INFO org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl - Timelineservice address: http://slaver1:8188/ws/v1/timeline/
2017-05-07 08:00:48,019 [main] INFO org.apache.pig.backend.hadoop.ATSService -Created ATS Hook
grunt>
43.在master节点安装Pig Clients，打开Linux Shell以Local模式启动它的Grunt，将启动命令和启动结果显示如下。
[root@master ~]# pig -x local
WARNING: Use "yarn jar" to launch YARNapplications.
17/05/07 08:00:21 INFO pig.ExecTypeProvider: TryingExecType : LOCAL
17/05/07 08:00:21 INFO pig.ExecTypeProvider: PickedLOCAL as the ExecType
2017-05-07 08:00:21,794 [main] INFO org.apache.pig.Main - Apache Pig version0.15.0.2.4.3.0-227 (rexported) compiled Sep 10 2016, 00:14:52
2017-05-07 08:00:21,794 [main] INFO org.apache.pig.Main - Logging error messagesto: /root/pig_1494144021792.log
2017-05-07 08:00:21,814 [main] INFO org.apache.pig.impl.util.Utils - Defaultbootup file /root/.pigbootup not found
2017-05-07 08:00:21,974 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine- Connecting to hadoop file system at: file:///
2017-05-07 08:00:22,194 [main] INFO org.apache.pig.PigServer - Pig Script ID forthe session: PIG-default-edcec26c-6956-4050-97c9-1c1806d1c853
2017-05-07 08:00:22,622 [main] INFO org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl - Timelineservice address: http://slaver1:8188/ws/v1/timeline/
2017-05-07 08:00:23,221 [main] INFO org.apache.pig.backend.hadoop.ATSService -Created ATS Hook
grunt>
44.使用Pig工具在Local模式计算系统日志access_log.txt中的IP的点击数，要求使用GROUP BY语句按照IP进行分组，通过FOREACH运算符，对关系的列进行迭代，统计每个分组的总行数，最后使用DUMP 语句查询统计结果。将查询命令和查询结果显示如下。
[root@master ~]# pig
WARNING: Use "yarn jar" to launch YARNapplications.
17/05/07 08:03:59 INFO pig.ExecTypeProvider: TryingExecType : LOCAL
17/05/07 08:03:59 INFO pig.ExecTypeProvider: TryingExecType : MAPREDUCE
17/05/07 08:03:59 INFO pig.ExecTypeProvider: PickedMAPREDUCE as the ExecType
2017-05-07 08:03:59,198 [main] INFO org.apache.pig.Main - Apache Pig version0.15.0.2.4.3.0-227 (rexported) compiled Sep 10 2016, 00:14:52
2017-05-07 08:03:59,198 [main] INFO org.apache.pig.Main - Logging error messagesto: /root/pig_1494144239196.log
2017-05-07 08:03:59,220 [main] INFO org.apache.pig.impl.util.Utils - Default bootupfile /root/.pigbootup not found
2017-05-07 08:03:59,618 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine -Connecting to hadoop file system at: hdfs://master:8020
2017-05-07 08:04:00,528 [main] INFO org.apache.pig.PigServer - Pig Script ID forthe session: PIG-default-bdf6e873-56a4-479d-b13f-3244ce895852
2017-05-07 08:04:00,991 [main] INFO org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl - Timelineservice address: http://slaver1:8188/ws/v1/timeline/
2017-05-07 08:04:01,098 [main] INFO org.apache.pig.backend.hadoop.ATSService -Created ATS Hook
grunt> copyFromLocal /root/access_log.txt/user/root/input/log1.txt
grunt> A =LOAD '/user/root/input/log1.txt' USINGPigStorage (' ') AS (ip,others);
grunt> group_ip =group A by ip;
grunt> result =foreach group_ip generategroup,COUNT(A);
grunt> dump result;
2017-05-07 08:09:16,681 [main] INFO org.apache.pig.tools.pigstats.ScriptState -Pig features used in the script: GROUP_BY
2017-05-07 08:09:16,718 [main] INFO org.apache.pig.data.SchemaTupleBackend - Key[pig.schematuple] was not set... will not generate code.
2017-05-07 08:09:16,756 [main] INFO org.apache.pig.newplan.logical.optimizer.LogicalPlanOptimizer -{RULES_ENABLED=[AddForEach, ColumnMapKeyPrune, ConstantCalculator,GroupByConstParallelSetter, LimitOptimizer, LoadTypeCastInserter, MergeFilter,MergeForEach, PartitionFilterOptimizer, PredicatePushdownOptimizer,PushDownForEachFlatten, PushUpFilter, SplitFilter, StreamTypeCastInserter]}
2017-05-07 08:09:16,896 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler- File concatenation threshold: 100 optimistic? false
2017-05-07 08:09:16,917 [main] INFO org.apache.pig.backend.hadoop.executionengine.util.CombinerOptimizerUtil- Choosing to move algebraic foreach to combiner
2017-05-07 08:09:16,949 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer- MR plan size before optimization: 1
2017-05-07 08:09:16,949 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer- MR plan size after optimization: 1
2017-05-07 08:09:17,135 [main] INFO org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl - Timelineservice address: http://slaver1:8188/ws/v1/timeline/
2017-05-07 08:09:17,143 [main] INFO org.apache.hadoop.yarn.client.RMProxy -Connecting to ResourceManager at slaver1/10.0.0.15:8050
2017-05-07 08:09:17,432 [main] INFO org.apache.pig.tools.pigstats.mapreduce.MRScriptState - Pig scriptsettings are added to the job
2017-05-07 08:09:17,438 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler- mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
2017-05-07 08:09:17,440 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler- Reduce phase detected, estimating # of required reducers.
2017-05-07 08:09:17,441 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler- Using reducer estimator:org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.InputSizeReducerEstimator
2017-05-07 08:09:17,448 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.InputSizeReducerEstimator- BytesPerReducer=1000000000 maxReducers=999 totalInputFileSize=3045
2017-05-07 08:09:17,449 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler- Setting Parallelism to 1
2017-05-07 08:09:17,449 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler- This job cannot be converted run in-process
2017-05-07 08:09:17,962 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler- Added jar file:/usr/hdp/2.4.3.0-227/pig/pig-0.15.0.2.4.3.0-227-core-h2.jar toDistributedCache through/tmp/temp-2081003050/tmp-1335603865/pig-0.15.0.2.4.3.0-227-core-h2.jar
2017-05-07 08:09:18,350 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler- Added jar file:/usr/hdp/2.4.3.0-227/pig/lib/automaton-1.11-8.jar toDistributedCache through/tmp/temp-2081003050/tmp1196315125/automaton-1.11-8.jar
2017-05-07 08:09:18,646 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler- Added jar file:/usr/hdp/2.4.3.0-227/pig/lib/antlr-runtime-3.4.jar toDistributedCache through/tmp/temp-2081003050/tmp1640187430/antlr-runtime-3.4.jar
2017-05-07 08:09:18,929 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler- Added jar file:/usr/hdp/2.4.3.0-227/hadoop-mapreduce/joda-time-2.9.4.jar toDistributedCache through /tmp/temp-2081003050/tmp247039283/joda-time-2.9.4.jar
2017-05-07 08:09:18,962 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler- Setting up single store job
2017-05-07 08:09:18,968 [main] INFO org.apache.pig.data.SchemaTupleFrontend - Key[pig.schematuple] is false, will not generate code.
2017-05-07 08:09:18,968 [main] INFO org.apache.pig.data.SchemaTupleFrontend -Starting process to move generated code to distributed cacche
2017-05-07 08:09:18,968 [main] INFO org.apache.pig.data.SchemaTupleFrontend -Setting key [pig.schematuple.classes] with classes to deserialize []
2017-05-07 08:09:19,046 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher- 1 map-reduce job(s) waiting for submission.
2017-05-07 08:09:19,138 [JobControl] INFO org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl - Timelineservice address: http://slaver1:8188/ws/v1/timeline/
2017-05-07 08:09:19,138 [JobControl] INFO org.apache.hadoop.yarn.client.RMProxy -Connecting to ResourceManager at slaver1/10.0.0.15:8050
2017-05-07 08:09:19,540 [JobControl] WARN org.apache.hadoop.mapreduce.JobResourceUploader- No job jar file set. User classes maynot be found. See Job or Job#setJar(String).
2017-05-07 08:09:19,600 [JobControl] INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total inputpaths to process : 1
2017-05-07 08:09:19,600 [JobControl] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Totalinput paths to process : 1
2017-05-07 08:09:19,619 [JobControl] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Totalinput paths (combined) to process : 1
2017-05-07 08:09:20,198 [JobControl] INFO org.apache.hadoop.mapreduce.JobSubmitter -number of splits:1
2017-05-07 08:09:20,553 [JobControl] INFO org.apache.hadoop.mapreduce.JobSubmitter -Submitting tokens for job: job_1494143770260_0002
2017-05-07 08:09:20,687 [JobControl] INFO org.apache.hadoop.mapred.YARNRunner - Job jaris not present. Not adding any jar to the list of resources.
2017-05-07 08:09:21,386 [JobControl] INFO org.apache.hadoop.yarn.client.api.impl.YarnClientImpl - Submittedapplication application_1494143770260_0002
2017-05-07 08:09:21,446 [JobControl] INFO org.apache.hadoop.mapreduce.Job - The url totrack the job: http://slaver1:8088/proxy/application_1494143770260_0002/
2017-05-07 08:09:21,447 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher- HadoopJobId: job_1494143770260_0002
2017-05-07 08:09:21,447 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher- Processing aliases A,group_ip,result
2017-05-07 08:09:21,447 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher- detailed locations: M: A[1,3],result[3,8],group_ip[2,10] C:result[3,8],group_ip[2,10] R: result[3,8]
2017-05-07 08:09:21,464 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher- 0% complete
2017-05-07 08:09:21,464 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher- Running jobs are [job_1494143770260_0002]
2017-05-07 08:10:28,914 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher- 50% complete
2017-05-07 08:10:28,914 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher- Running jobs are [job_1494143770260_0002]
2017-05-07 08:10:38,929 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher- Running jobs are [job_1494143770260_0002]
2017-05-07 08:10:42,044 [main] INFO org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl- Timeline service address: http://slaver1:8188/ws/v1/timeline/
2017-05-07 08:10:42,045 [main] INFO org.apache.hadoop.yarn.client.RMProxy -Connecting to ResourceManager at slaver1/10.0.0.15:8050
2017-05-07 08:10:42,056 [main] INFO org.apache.hadoop.mapred.ClientServiceDelegate - Application state iscompleted. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
2017-05-07 08:10:43,592 [main] INFO org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl- Timeline service address: http://slaver1:8188/ws/v1/timeline/
2017-05-07 08:10:43,593 [main] INFO org.apache.hadoop.yarn.client.RMProxy -Connecting to ResourceManager at slaver1/10.0.0.15:8050
2017-05-07 08:10:43,600 [main] INFO org.apache.hadoop.mapred.ClientServiceDelegate- Application state is completed. FinalApplicationStatus=SUCCEEDED. Redirectingto job history server
2017-05-07 08:10:43,744 [main] INFO org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl - Timelineservice address: http://slaver1:8188/ws/v1/timeline/
2017-05-07 08:10:43,745 [main] INFO org.apache.hadoop.yarn.client.RMProxy -Connecting to ResourceManager at slaver1/10.0.0.15:8050
2017-05-07 08:10:43,750 [main] INFO org.apache.hadoop.mapred.ClientServiceDelegate- Application state is completed. FinalApplicationStatus=SUCCEEDED. Redirectingto job history server
2017-05-07 08:10:43,818 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher- 100% complete
2017-05-07 08:10:43,820 [main] INFO org.apache.pig.tools.pigstats.mapreduce.SimplePigStats - ScriptStatistics:
HadoopVersion PigVersion UserId StartedAt FinishedAt Features
2.7.1.2.4.3.0-227 0.15.0.2.4.3.0-227 root 2017-05-07 08:09:17 2017-05-07 08:10:43 GROUP_BY
Success!
Job Stats (time in seconds):
JobId Maps Reduces MaxMapTime MinMapTime AvgMapTime MedianMapTime MaxReduceTime MinReduceTime AvgReduceTime MedianReducetime Alias Feature Outputs
job_1494143770260_0002 1 1 3 3 3 3 7 7 7 7 A,group_ip,result GROUP_BY,COMBINER hdfs://master:8020/tmp/temp-2081003050/tmp270373100,
Input(s):
Successfully read 10 records (3415 bytes) from:"/user/root/input/log1.txt"
Output(s):
Successfully stored 4 records (85 bytes) in:"hdfs://master:8020/tmp/temp-2081003050/tmp270373100"
Counters:
Total records written : 4
Total bytes written : 85
Spillable Memory Manager spill count : 0
Total bags proactively spilled: 0
Total records proactively spilled: 0
Job DAG:
job_1494143770260_0002
2017-05-07 08:10:43,893 [main] INFO org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl - Timelineservice address: http://slaver1:8188/ws/v1/timeline/
2017-05-07 08:10:43,893 [main] INFO org.apache.hadoop.yarn.client.RMProxy -Connecting to ResourceManager at slaver1/10.0.0.15:8050
2017-05-07 08:10:43,898 [main] INFO org.apache.hadoop.mapred.ClientServiceDelegate - Application state iscompleted. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
2017-05-07 08:10:44,022 [main] INFO org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl - Timelineservice address: http://slaver1:8188/ws/v1/timeline/
2017-05-07 08:10:44,022 [main] INFO org.apache.hadoop.yarn.client.RMProxy -Connecting to ResourceManager at slaver1/10.0.0.15:8050
2017-05-07 08:10:44,027 [main] INFO org.apache.hadoop.mapred.ClientServiceDelegate - Application state iscompleted. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
2017-05-07 08:10:44,130 [main] INFO org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl - Timelineservice address: http://slaver1:8188/ws/v1/timeline/
2017-05-07 08:10:44,131 [main] INFO org.apache.hadoop.yarn.client.RMProxy -Connecting to ResourceManager at slaver1/10.0.0.15:8050
2017-05-07 08:10:44,137 [main] INFO org.apache.hadoop.mapred.ClientServiceDelegate - Application state iscompleted. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
2017-05-07 08:10:44,175 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher- Success!
2017-05-07 08:10:44,179 [main] INFO org.apache.pig.data.SchemaTupleBackend - Key[pig.schematuple] was not set... will not generate code.
2017-05-07 08:10:44,190 [main] INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total inputpaths to process : 1
2017-05-07 08:10:44,190 [main] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Totalinput paths to process : 1
(112.97.24.243,7)
(208.115.113.82,1)
(220.181.94.221,1)
(220.181.108.151,1)
45.使用Pig工具计算天气数据集temperature.txt中年度最高气温，要求使用GROUP BY语句按照year进行分组，通过FOREACH运算符，对关系的列进行迭代，统计每个分组的最大值，最后使用DUMP 语句查询计算结果。将以上查询命令和查询结果显示如下。
[root@master ~]# pig
WARNING: Use "yarn jar" to launch YARNapplications.
17/05/07 07:58:29 INFO pig.ExecTypeProvider: TryingExecType : LOCAL
17/05/07 07:58:29 INFO pig.ExecTypeProvider: TryingExecType : MAPREDUCE
17/05/07 07:58:29 INFO pig.ExecTypeProvider: PickedMAPREDUCE as the ExecType
2017-05-07 07:58:29,081 [main] INFO org.apache.pig.Main - Apache Pig version0.15.0.2.4.3.0-227 (rexported) compiled Sep 10 2016, 00:14:52
2017-05-07 07:58:29,081 [main] INFO org.apache.pig.Main - Logging error messagesto: /root/pig_1494143909080.log
2017-05-07 07:58:29,104 [main] INFO org.apache.pig.impl.util.Utils - Defaultbootup file /root/.pigbootup not found
2017-05-07 07:58:29,507 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine -Connecting to hadoop file system at: hdfs://master:8020
2017-05-07 07:58:30,427 [main] INFO org.apache.pig.PigServer - Pig Script ID forthe session: PIG-default-a45e5d92-ef27-4629-8326-66cbf6605e8e
2017-05-07 07:58:30,870 [main] INFO org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl- Timeline service address: http://slaver1:8188/ws/v1/timeline/
2017-05-07 07:58:30,977 [main] INFO org.apache.pig.backend.hadoop.ATSService -Created ATS Hook
grunt> copyFromLocal /root/temp.txt/user/root/temp.txt
grunt> A = LOAD '/user/root/temp.txt' USINGPigStorage(' ')AS (year:int,temperature:int);
grunt> B = GROUP A BY year;
grunt> C = FOREACH B GENERATEgroup,MAX(A.temperature);
grunt> dump C;
2017-05-07 08:23:04,298 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pigfeatures used in the script: GROUP_BY
2017-05-07 08:23:04,326 [main] INFO org.apache.pig.data.SchemaTupleBackend - Key[pig.schematuple] was not set... will not generate code.
2017-05-07 08:23:04,326 [main] INFO org.apache.pig.newplan.logical.optimizer.LogicalPlanOptimizer- {RULES_ENABLED=[AddForEach, ColumnMapKeyPrune, ConstantCalculator,GroupByConstParallelSetter, LimitOptimizer, LoadTypeCastInserter, MergeFilter,MergeForEach, PartitionFilterOptimizer, PredicatePushdownOptimizer,PushDownForEachFlatten, PushUpFilter, SplitFilter, StreamTypeCastInserter]}
2017-05-07 08:23:04,335 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler- File concatenation threshold: 100 optimistic? false
2017-05-07 08:23:04,337 [main] INFO org.apache.pig.backend.hadoop.executionengine.util.CombinerOptimizerUtil- Choosing to move algebraic foreach to combiner
2017-05-07 08:23:04,339 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer- MR plan size before optimization: 1
2017-05-07 08:23:04,339 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer- MR plan size after optimization: 1
2017-05-07 08:23:04,425 [main] INFO org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl- Timeline service address: http://slaver1:8188/ws/v1/timeline/
2017-05-07 08:23:04,425 [main] INFO org.apache.hadoop.yarn.client.RMProxy -Connecting to ResourceManager at slaver1/10.0.0.15:8050
2017-05-07 08:23:04,432 [main] INFO org.apache.pig.tools.pigstats.mapreduce.MRScriptState- Pig script settings are added to the job
2017-05-07 08:23:04,433 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler- mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
2017-05-07 08:23:04,433 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler- Reduce phase detected, estimating # of required reducers.
2017-05-07 08:23:04,433 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler- Using reducer estimator:org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.InputSizeReducerEstimator
2017-05-07 08:23:04,435 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.InputSizeReducerEstimator- BytesPerReducer=1000000000 maxReducers=999 totalInputFileSize=55
2017-05-07 08:23:04,435 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler- Setting Parallelism to 1
2017-05-07 08:23:04,435 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler- This job cannot be converted run in-process
2017-05-07 08:23:04,476 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler- Added jar file:/usr/hdp/2.4.3.0-227/pig/pig-0.15.0.2.4.3.0-227-core-h2.jar toDistributedCache through/tmp/temp-2081003050/tmp455800525/pig-0.15.0.2.4.3.0-227-core-h2.jar
2017-05-07 08:23:04,491 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler- Added jar file:/usr/hdp/2.4.3.0-227/pig/lib/automaton-1.11-8.jar toDistributedCache through/tmp/temp-2081003050/tmp1148421651/automaton-1.11-8.jar
2017-05-07 08:23:04,505 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler- Added jar file:/usr/hdp/2.4.3.0-227/pig/lib/antlr-runtime-3.4.jar toDistributedCache through/tmp/temp-2081003050/tmp280400162/antlr-runtime-3.4.jar
2017-05-07 08:23:04,930 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler- Added jar file:/usr/hdp/2.4.3.0-227/hadoop-mapreduce/joda-time-2.9.4.jar toDistributedCache through /tmp/temp-2081003050/tmp93917590/joda-time-2.9.4.jar
2017-05-07 08:23:04,949 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler- Setting up single store job
2017-05-07 08:23:04,950 [main] INFO org.apache.pig.data.SchemaTupleFrontend - Key[pig.schematuple] is false, will not generate code.
2017-05-07 08:23:04,950 [main] INFO org.apache.pig.data.SchemaTupleFrontend -Starting process to move generated code to distributed cacche
2017-05-07 08:23:04,950 [main] INFO org.apache.pig.data.SchemaTupleFrontend -Setting key [pig.schematuple.classes] with classes to deserialize []
2017-05-07 08:23:04,977 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher- 1 map-reduce job(s) waiting for submission.
2017-05-07 08:23:05,050 [JobControl] INFO org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl- Timeline service address: http://slaver1:8188/ws/v1/timeline/
2017-05-07 08:23:05,050 [JobControl] INFO org.apache.hadoop.yarn.client.RMProxy -Connecting to ResourceManager at slaver1/10.0.0.15:8050
2017-05-07 08:23:05,077 [JobControl] WARN org.apache.hadoop.mapreduce.JobResourceUploader - No job jar fileset. User classes may not be found. SeeJob or Job#setJar(String).
2017-05-07 08:23:05,114 [JobControl] INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat- Total input paths to process : 1
2017-05-07 08:23:05,114 [JobControl] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Totalinput paths to process : 1
2017-05-07 08:23:05,116 [JobControl] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Totalinput paths (combined) to process : 1
2017-05-07 08:23:05,155 [JobControl] INFO org.apache.hadoop.mapreduce.JobSubmitter -number of splits:1
2017-05-07 08:23:05,193 [JobControl] INFO org.apache.hadoop.mapreduce.JobSubmitter -Submitting tokens for job: job_1494143770260_0003
2017-05-07 08:23:05,196 [JobControl] INFO org.apache.hadoop.mapred.YARNRunner - Job jaris not present. Not adding any jar to the list of resources.
2017-05-07 08:23:05,444 [JobControl] INFO org.apache.hadoop.yarn.client.api.impl.YarnClientImpl - Submittedapplication application_1494143770260_0003
2017-05-07 08:23:05,449 [JobControl] INFO org.apache.hadoop.mapreduce.Job - The url totrack the job: http://slaver1:8088/proxy/application_1494143770260_0003/
2017-05-07 08:23:05,479 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher- HadoopJobId: job_1494143770260_0003
2017-05-07 08:23:05,479 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher- Processing aliases A,B,C
2017-05-07 08:23:05,479 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher- detailed locations: M: A[4,4],A[-1,-1],C[6,4],B[5,4] C: C[6,4],B[5,4] R:C[6,4]
2017-05-07 08:23:05,488 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher- 0% complete
2017-05-07 08:23:05,488 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher- Running jobs are [job_1494143770260_0003]
2017-05-07 08:27:59,719 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher- 50% complete
2017-05-07 08:27:59,719 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher- Running jobs are [job_1494143770260_0003]
2017-05-07 08:30:37,027 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher- Running jobs are [job_1494143770260_0003]
2017-05-07 08:30:43,127 [main] INFO org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl - Timelineservice address: http://slaver1:8188/ws/v1/timeline/
2017-05-07 08:30:43,128 [main] INFO org.apache.hadoop.yarn.client.RMProxy - Connectingto ResourceManager at slaver1/10.0.0.15:8050
2017-05-07 08:30:43,191 [main] INFO org.apache.hadoop.mapred.ClientServiceDelegate - Application state iscompleted. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
2017-05-07 08:30:44,506 [main] INFO org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl - Timelineservice address: http://slaver1:8188/ws/v1/timeline/
2017-05-07 08:30:44,506 [main] INFO org.apache.hadoop.yarn.client.RMProxy -Connecting to ResourceManager at slaver1/10.0.0.15:8050
2017-05-07 08:30:44,520 [main] INFO org.apache.hadoop.mapred.ClientServiceDelegate - Application state iscompleted. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
2017-05-07 08:30:44,663 [main] INFO org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl- Timeline service address: http://slaver1:8188/ws/v1/timeline/
2017-05-07 08:30:44,664 [main] INFO org.apache.hadoop.yarn.client.RMProxy -Connecting to ResourceManager at slaver1/10.0.0.15:8050
2017-05-07 08:30:44,669 [main] INFO org.apache.hadoop.mapred.ClientServiceDelegate - Application state iscompleted. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
2017-05-07 08:30:44,721 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher- 100% complete
2017-05-07 08:30:44,721 [main] INFO org.apache.pig.tools.pigstats.mapreduce.SimplePigStats - ScriptStatistics:
HadoopVersion PigVersion UserId StartedAt FinishedAt Features
2.7.1.2.4.3.0-227 0.15.0.2.4.3.0-227 root 2017-05-07 08:23:04 2017-05-07 08:30:44 GROUP_BY
Success!
Job Stats (time in seconds):
JobId Maps Reduces MaxMapTime MinMapTime AvgMapTime MedianMapTime MaxReduceTime MinReduceTime AvgReduceTime MedianReducetime AliasFeature Outputs
job_1494143770260_0003 1 1 284 284 284 284 149 149 149 149 A,B,C GROUP_BY,COMBINER hdfs://master:8020/tmp/temp-2081003050/tmp-111966269,
Input(s):
Successfully read 7 records (419 bytes) from:"/user/root/temp.txt"
Output(s):
Successfully stored 2 records (18 bytes) in:"hdfs://master:8020/tmp/temp-2081003050/tmp-111966269"
Counters:
Total records written : 2
Total bytes written : 18
Spillable Memory Manager spill count : 0
Total bags proactively spilled: 0
Total records proactively spilled: 0
Job DAG:
job_1494143770260_0003
2017-05-07 08:30:44,784 [main] INFO org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl - Timelineservice address: http://slaver1:8188/ws/v1/timeline/
2017-05-07 08:30:44,784 [main] INFO org.apache.hadoop.yarn.client.RMProxy -Connecting to ResourceManager at slaver1/10.0.0.15:8050
2017-05-07 08:30:44,789 [main] INFO org.apache.hadoop.mapred.ClientServiceDelegate- Application state is completed. FinalApplicationStatus=SUCCEEDED. Redirectingto job history server
2017-05-07 08:30:44,892 [main] INFO org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl - Timeline serviceaddress: http://slaver1:8188/ws/v1/timeline/
2017-05-07 08:30:44,892 [main] INFO org.apache.hadoop.yarn.client.RMProxy -Connecting to ResourceManager at slaver1/10.0.0.15:8050
2017-05-07 08:30:44,897 [main] INFO org.apache.hadoop.mapred.ClientServiceDelegate- Application state is completed. FinalApplicationStatus=SUCCEEDED. Redirectingto job history server
2017-05-07 08:30:44,990 [main] INFO org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl - Timelineservice address: http://slaver1:8188/ws/v1/timeline/
2017-05-07 08:30:44,991 [main] INFO org.apache.hadoop.yarn.client.RMProxy -Connecting to ResourceManager at slaver1/10.0.0.15:8050
2017-05-07 08:30:44,996 [main] INFO org.apache.hadoop.mapred.ClientServiceDelegate - Application state iscompleted. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
2017-05-07 08:30:45,029 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher- Success!
2017-05-07 08:30:45,031 [main] INFO org.apache.pig.data.SchemaTupleBackend - Key[pig.schematuple] was not set... will not generate code.
2017-05-07 08:30:45,034 [main] INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total inputpaths to process : 1
2017-05-07 08:30:45,034 [main] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil- Total input paths to process : 1
(1990,34)
(1991,35)
46.使用Pig工具统计数据集ip_to_country中每个国家的IP地址数。要求使用GROUP BY语句按照国家进行分组，通过FOREACH运算符，对关系的列进行迭代，统计每个分组的IP地址数目，最后将统计结果保存到/data/pig/output目录中，并查看数据结果。将以上操作命令和查询结果显示如下。
[root@master ~]# pig
WARNING: Use "yarn jar" to launch YARNapplications.
17/05/07 07:58:29 INFO pig.ExecTypeProvider: TryingExecType : LOCAL
17/05/07 07:58:29 INFO pig.ExecTypeProvider: TryingExecType : MAPREDUCE
17/05/07 07:58:29 INFO pig.ExecTypeProvider: PickedMAPREDUCE as the ExecType
2017-05-07 07:58:29,081 [main] INFO org.apache.pig.Main - Apache Pig version0.15.0.2.4.3.0-227 (rexported) compiled Sep 10 2016, 00:14:52
2017-05-07 07:58:29,081 [main] INFO org.apache.pig.Main - Logging error messagesto: /root/pig_1494143909080.log
2017-05-07 07:58:29,104 [main] INFO org.apache.pig.impl.util.Utils - Defaultbootup file /root/.pigbootup not found
2017-05-07 07:58:29,507 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine- Connecting to hadoop file system at: hdfs://master:8020
2017-05-07 07:58:30,427 [main] INFO org.apache.pig.PigServer - Pig Script ID forthe session: PIG-default-a45e5d92-ef27-4629-8326-66cbf6605e8e
2017-05-07 07:58:30,870 [main] INFO org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl - Timelineservice address: http://slaver1:8188/ws/v1/timeline/
2017-05-07 07:58:30,977 [main] INFO org.apache.pig.backend.hadoop.ATSService -Created ATS Hook
grunt> copyFromLocal /root/ip_to_country.txt/user/root/ip_to_country.txt
grunt> ip_countries = LOAD'/user/root/ip_to_country.txt' AS (ip: chararray, country:chararray);
grunt> country_grpd = GROUP ip_countries BYcountry;
grunt> country_counts = FOREACH country_grpdGENERATE FLATTEN(group),COUNT(ip_countries) as counts;
grunt> STORE country_counts INTO'/data/pig/output';
2017-05-07 08:36:28,897 [main] INFO org.apache.pig.tools.pigstats.ScriptState -Pig features used in the script: GROUP_BY
2017-05-07 08:36:28,921 [main] INFO org.apache.pig.data.SchemaTupleBackend - Key[pig.schematuple] was not set... will not generate code.
2017-05-07 08:36:28,922 [main] INFO org.apache.pig.newplan.logical.optimizer.LogicalPlanOptimizer -{RULES_ENABLED=[AddForEach, ColumnMapKeyPrune, ConstantCalculator,GroupByConstParallelSetter, LimitOptimizer, LoadTypeCastInserter, MergeFilter,MergeForEach, PartitionFilterOptimizer, PredicatePushdownOptimizer,PushDownForEachFlatten, PushUpFilter, SplitFilter, StreamTypeCastInserter]}
2017-05-07 08:36:28,925 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler- File concatenation threshold: 100 optimistic? false
2017-05-07 08:36:28,926 [main] INFO org.apache.pig.backend.hadoop.executionengine.util.CombinerOptimizerUtil- Choosing to move algebraic foreach to combiner
2017-05-07 08:36:28,927 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer- MR plan size before optimization: 1
2017-05-07 08:36:28,927 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer- MR plan size after optimization: 1
2017-05-07 08:36:29,009 [main] INFO org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl - Timelineservice address: http://slaver1:8188/ws/v1/timeline/
2017-05-07 08:36:29,009 [main] INFO org.apache.hadoop.yarn.client.RMProxy -Connecting to ResourceManager at slaver1/10.0.0.15:8050
2017-05-07 08:36:29,013 [main] INFO org.apache.pig.tools.pigstats.mapreduce.MRScriptState - Pig scriptsettings are added to the job
2017-05-07 08:36:29,014 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler- mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
2017-05-07 08:36:29,014 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler- Reduce phase detected, estimating # of required reducers.
2017-05-07 08:36:29,014 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler- Using reducer estimator:org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.InputSizeReducerEstimator
2017-05-07 08:36:29,015 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.InputSizeReducerEstimator- BytesPerReducer=1000000000 maxReducers=999 totalInputFileSize=75728
2017-05-07 08:36:29,015 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler- Setting Parallelism to 1
2017-05-07 08:36:29,015 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler- This job cannot be converted run in-process
2017-05-07 08:36:29,057 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler- Added jar file:/usr/hdp/2.4.3.0-227/pig/pig-0.15.0.2.4.3.0-227-core-h2.jar toDistributedCache through/tmp/temp-2081003050/tmp1402100776/pig-0.15.0.2.4.3.0-227-core-h2.jar
2017-05-07 08:36:29,118 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler- Added jar file:/usr/hdp/2.4.3.0-227/pig/lib/automaton-1.11-8.jar toDistributedCache through/tmp/temp-2081003050/tmp1729569612/automaton-1.11-8.jar
2017-05-07 08:36:29,143 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler- Added jar file:/usr/hdp/2.4.3.0-227/pig/lib/antlr-runtime-3.4.jar toDistributedCache through/tmp/temp-2081003050/tmp658892872/antlr-runtime-3.4.jar
2017-05-07 08:36:29,200 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler- Added jar file:/usr/hdp/2.4.3.0-227/hadoop-mapreduce/joda-time-2.9.4.jar toDistributedCache through /tmp/temp-2081003050/tmp68205245/joda-time-2.9.4.jar
2017-05-07 08:36:29,210 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler- Setting up single store job
2017-05-07 08:36:29,211 [main] INFO org.apache.pig.data.SchemaTupleFrontend - Key[pig.schematuple] is false, will not generate code.
2017-05-07 08:36:29,211 [main] INFO org.apache.pig.data.SchemaTupleFrontend -Starting process to move generated code to distributed cacche
2017-05-07 08:36:29,211 [main] INFO org.apache.pig.data.SchemaTupleFrontend -Setting key [pig.schematuple.classes] with classes to deserialize []
2017-05-07 08:36:29,229 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher- 1 map-reduce job(s) waiting for submission.
2017-05-07 08:36:29,300 [JobControl] INFO org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl - Timelineservice address: http://slaver1:8188/ws/v1/timeline/
2017-05-07 08:36:29,301 [JobControl] INFO org.apache.hadoop.yarn.client.RMProxy -Connecting to ResourceManager at slaver1/10.0.0.15:8050
2017-05-07 08:36:29,354 [JobControl] WARN org.apache.hadoop.mapreduce.JobResourceUploader- No job jar file set. User classes maynot be found. See Job or Job#setJar(String).
2017-05-07 08:36:29,387 [JobControl] INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total inputpaths to process : 1
2017-05-07 08:36:29,387 [JobControl] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Totalinput paths to process : 1
2017-05-07 08:36:29,389 [JobControl] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total inputpaths (combined) to process : 1
2017-05-07 08:36:29,443 [JobControl] INFO org.apache.hadoop.mapreduce.JobSubmitter -number of splits:1
2017-05-07 08:36:29,491 [JobControl] INFO org.apache.hadoop.mapreduce.JobSubmitter -Submitting tokens for job: job_1494145664092_0001
2017-05-07 08:36:29,493 [JobControl] INFO org.apache.hadoop.mapred.YARNRunner - Job jaris not present. Not adding any jar to the list of resources.
2017-05-07 08:36:29,769 [JobControl] INFO org.apache.hadoop.yarn.client.api.impl.YarnClientImpl- Submitted application application_1494145664092_0001
2017-05-07 08:36:29,774 [JobControl] INFO org.apache.hadoop.mapreduce.Job - The url totrack the job: http://slaver1:8088/proxy/application_1494145664092_0001/
2017-05-07 08:36:29,774 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher- HadoopJobId: job_1494145664092_0001
2017-05-07 08:36:29,774 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher- Processing aliases country_counts,country_grpd,ip_countries
2017-05-07 08:36:29,774 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher- detailed locations: M:ip_countries[7,15],ip_countries[-1,-1],country_counts[9,17],country_grpd[8,15]C: country_counts[9,17],country_grpd[8,15] R: country_counts[9,17]
2017-05-07 08:36:29,781 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher- 0% complete
2017-05-07 08:36:29,781 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher- Running jobs are [job_1494145664092_0001]
2017-05-07 08:36:41,876 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher- 50% complete
2017-05-07 08:36:41,876 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher- Running jobs are [job_1494145664092_0001]
2017-05-07 08:36:49,387 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher- Running jobs are [job_1494145664092_0001]
2017-05-07 08:36:49,969 [main] INFO org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl - Timelineservice address: http://slaver1:8188/ws/v1/timeline/
2017-05-07 08:36:49,969 [main] INFO org.apache.hadoop.yarn.client.RMProxy -Connecting to ResourceManager at slaver1/10.0.0.15:8050
2017-05-07 08:36:49,976 [main] INFO org.apache.hadoop.mapred.ClientServiceDelegate - Application state iscompleted. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
2017-05-07 08:36:50,266 [main] INFO org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl - Timelineservice address: http://slaver1:8188/ws/v1/timeline/
2017-05-07 08:36:50,266 [main] INFO org.apache.hadoop.yarn.client.RMProxy - Connectingto ResourceManager at slaver1/10.0.0.15:8050
2017-05-07 08:36:50,272 [main] INFO org.apache.hadoop.mapred.ClientServiceDelegate - Application state iscompleted. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
2017-05-07 08:36:50,377 [main] INFO org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl - Timelineservice address: http://slaver1:8188/ws/v1/timeline/
2017-05-07 08:36:50,377 [main] INFO org.apache.hadoop.yarn.client.RMProxy -Connecting to ResourceManager at slaver1/10.0.0.15:8050
2017-05-07 08:36:50,382 [main] INFO org.apache.hadoop.mapred.ClientServiceDelegate - Application state iscompleted. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
2017-05-07 08:36:50,416 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher- 100% complete
2017-05-07 08:36:50,417 [main] INFO org.apache.pig.tools.pigstats.mapreduce.SimplePigStats - ScriptStatistics:
HadoopVersion PigVersion UserId StartedAt FinishedAt Features
2.7.1.2.4.3.0-227 0.15.0.2.4.3.0-227 root 2017-05-07 08:36:29 2017-05-07 08:36:50 GROUP_BY
Success!
Job Stats (time in seconds):
JobId Maps Reduces MaxMapTime MinMapTime AvgMapTime MedianMapTime MaxReduceTime MinReduceTime AvgReduceTime MedianReducetime Alias Feature Outputs
job_1494145664092_0001 1 1 3 3 3 3 3 3 3 3 country_counts,country_grpd,ip_countries GROUP_BY,COMBINER /data/pig/output,
Input(s):
Successfully read 3000 records (76101 bytes) from:"/user/root/ip_to_country.txt"
Output(s):
Successfully stored 98 records (1207 bytes) in:"/data/pig/output"
Counters:
Total records written : 98
Total bytes written : 1207
Spillable Memory Manager spill count : 0
Total bags proactively spilled: 0
Total records proactively spilled: 0
Job DAG:
job_1494145664092_0001
2017-05-07 08:36:50,485 [main] INFO org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl- Timeline service address: http://slaver1:8188/ws/v1/timeline/
2017-05-07 08:36:50,485 [main] INFO org.apache.hadoop.yarn.client.RMProxy -Connecting to ResourceManager at slaver1/10.0.0.15:8050
2017-05-07 08:36:50,490 [main] INFO org.apache.hadoop.mapred.ClientServiceDelegate - Application state iscompleted. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
2017-05-07 08:36:50,589 [main] INFO org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl- Timeline service address: http://slaver1:8188/ws/v1/timeline/
2017-05-07 08:36:50,590 [main] INFO org.apache.hadoop.yarn.client.RMProxy -Connecting to ResourceManager at slaver1/10.0.0.15:8050
2017-05-07 08:36:50,595 [main] INFO org.apache.hadoop.mapred.ClientServiceDelegate - Application state iscompleted. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
2017-05-07 08:36:50,694 [main] INFO org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl - Timelineservice address: http://slaver1:8188/ws/v1/timeline/
2017-05-07 08:36:50,694 [main] INFO org.apache.hadoop.yarn.client.RMProxy -Connecting to ResourceManager at slaver1/10.0.0.15:8050
2017-05-07 08:36:50,699 [main] INFO org.apache.hadoop.mapred.ClientServiceDelegate- Application state is completed. FinalApplicationStatus=SUCCEEDED. Redirectingto job history server
2017-05-07 08:36:50,734 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher- Success!
grunt> cat /data/pig/output/part-r-00000
Iraq 1
Oman 1
Peru 3
Chile 7
China 252
Egypt 6
Gabon 1
India 30
Italy 43
Japan 177
Macau 1
Nepal 1
Qatar 1
Spain 21
Yemen 2
Angola 2
Brazil 38
Canada 75
Europe 34
France 58
Greece 6
Israel 6
Kuwait 5
Latvia 1
Mexico 23
Norway 18
Poland 15
Serbia 1
Sweden 17
Taiwan 26
Turkey 16
Albania 1
Algeria 2
Austria 14
Bahrain 1
Belarus 1
Belgium 14
Croatia 2
Denmark 11
Ecuador 3
Estonia 2
Finland 13
Germany 89
Hungary 2
Iceland 1
Ireland 5
Morocco 19
Nigeria 1
Romania 13
Senegal 1
Tunisia 3
Ukraine 10
Uruguay 2
Vietnam 13
Barbados 1
Botswana 1
Bulgaria 6
Colombia 21
Malaysia 8
Pakistan 4
Portugal 3
Slovenia 2
Thailand 10
Argentina 13
Australia 68
Guatemala 1
Hong Kong 8
Indonesia 29
Lithuania 6
Macedonia 1
Mauritius 10
Singapore 5
Venezuela 4
Azerbaijan 1
Costa Rica 2
Kazakhstan 3
Martinique 1
Uzbekistan 1
Netherlands 28
New Zealand 9
Philippines 7
Switzerland 15
Saudi Arabia 4
South Africa 20
United States 1379
Czech Republic 7
United Kingdom 93
Anonymous Proxy 1
Dominican Republic 1
Korea, Republic of 70
Russian Federation 36
Satellite Provider 2
Moldova, Republic of 1
Syrian Arab Republic 1
United Arab Emirates 2
Bosnia and Herzegovina 1
Iran, Islamic Republic of 2
Tanzania, United Republic of 1