hadoop异常 error in shuffle in fetcher

错误信息:Error:org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in shuffle in fetcher#5
解决方法：限制reduce的shuffle内存使用
hive：set mapreduce.reduce.shuffle.memory.limit.percent=0.1;
MR：job.getConfiguration().setStrings(“mapreduce.reduce.shuffle.memory.limit.percent”, “0.1”);
原理分析：reduce会在map执行到一定比例启动多个fetch线程去拉取map的输出结果，放到reduce的内存、磁盘中，然后进行merge。当数据量大时，拉取到内存的数据就会引起OOM，所以此时要减少fetch占内存的百分比，将fetch的数据直接放在磁盘上。
mapreduce.reduce.shuffle.memory.limit.percent：每个fetch取到的map输出的大小能够占的内存比的大小。默认是0.25。因此实际每个fetcher的输出能放在内存的大小是reducer的java heap size*0.9*0.25。

----

Task ID:

  task_1465962505870_540836_r_000713



URL:

  http://bigdata-arch-nm00.bh:18088/taskdetails.jsp?jobid=job_1465962505870_540836&tipid=task_1465962505870_540836_r_000713

-----

Diagnostic Messages for this Task:

Error: org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in shuffle in fetcher#5

    at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:121)

    at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:380)

    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:162)

    at java.security.AccessController.doPrivileged(Native Method)

    at javax.security.auth.Subject.doAs(Subject.java:415)

    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)

    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:157)

Caused by: java.lang.OutOfMemoryError: Java heap space

    at org.apache.hadoop.io.BoundedByteArrayOutputStream.<init>(BoundedByteArrayOutputStream.java:56)

    at org.apache.hadoop.io.BoundedByteArrayOutputStream.<init>(BoundedByteArrayOutputStream.java:46)

    at org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput.<init>(InMemoryMapOutput.java:63)

    at org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl.unconditionalReserve(MergeManagerImpl.java:297)

    at org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl.reserve(MergeManagerImpl.java:287)

    at org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyMapOutput(Fetcher.java:411)

    at org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:341)

    at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:165)



Container killed by the ApplicationMaster.

Container killed on request. Exit code is 143





FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask

MapReduce Jobs Launched: