执行hive内存溢出
错误日志:
Diagnostic Messages for this Task:
Error: java.lang.OutOfMemoryError: Java heap spaceat org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.shuffleInMemory(ReduceTask.java:1703)
at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getMapOutput(ReduceTask.java:1563)
at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.copyOutput(ReduceTask.java:1401)
at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.run(ReduceTask.java:1333)
FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask
MapReduce Jobs Launched:
Job 0: Map: 10 Reduce: 3 Cumulative CPU: 3035.3 sec HDFS Read: 2528366258 HDFS Write: 1209449 FAIL
Total MapReduce CPU Time Spent: 50 minutes 35 seconds 300 msec
解决方法:
Solution:
Currently hive.map.aggr.hash.percentmemory is set to 0.5. Try setting it to a lower value. i.e "set hive.map.aggr.hash.percentmemory = 0.25;"
-----
声明:该文观点仅代表作者本人,牛骨文系教育信息发布平台,牛骨文仅提供信息存储空间服务。