当前位置 博文首页 > 文章内容

    HIVE报错:Error: Error while processing statement: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask (state=08S01,code=2)

    作者: 栏目:未分类 时间:2020-10-07 15:00:22

    本站于2023年9月4日。收到“大连君*****咨询有限公司”通知
    说我们IIS7站长博客,有一篇博文用了他们的图片。
    要求我们给他们一张图片6000元。要不然法院告我们

    为避免不必要的麻烦,IIS7站长博客,全站内容图片下架、并积极应诉
    博文内容全部不再显示,请需要相关资讯的站长朋友到必应搜索。谢谢!

    另祝:版权碰瓷诈骗团伙,早日弃暗投明。

    相关新闻:借版权之名、行诈骗之实,周某因犯诈骗罪被判处有期徒刑十一年六个月

    叹!百花齐放的时代,渐行渐远!



    执行 insert into table video_orc select * from video_ori;时报错

    查看hive日志发现具体报错信息如下:

    2020-10-07T09:33:11,117  INFO [HiveServer2-Background-Pool: Thread-241] ql.Driver: Concurrency mode is disabled, not creating a lock manager
    2020-10-07T09:33:11,119 ERROR [HiveServer2-Background-Pool: Thread-241] operation.Operation: Error running hive query:
    org.apache.hive.service.cli.HiveSQLException: Error while processing statement: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
            at org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:335) ~[hive-service-3.1.2.jar:3.1.2]
            at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:226) ~[hive-service-3.1.2.jar:3.1.2]
            at org.apache.hive.service.cli.operation.SQLOperation.access$700(SQLOperation.java:87) ~[hive-service-3.1.2.jar:3.1.2]
            at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:316) ~[hive-service-3.1.2.jar:3.1.2]
            at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_212]
            at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_212]
            at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) ~[hadoop-common-3.1.3.jar:?]
            at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:329) ~[hive-service-3.1.2.jar:3.1.2]
            at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_212]
            at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_212]
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212]
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212]
            at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212]
    2020-10-07T09:33:11,120  INFO [HiveServer2-Handler-Pool: Thread-52] conf.HiveConf: Using the default value passed in for log id: 544f2b3d-7a09-44a1-bf44-f25c7b2ad6e4

    分析:

    经过各个方法实验,无法找到具体报错信息,无法判断具体原因

    偶然看到一个贴子说把同样的sql放到hive shell里面执行(本来我是在beeline -u jdbc:hive2://Linux201:10000 -n zls下执行,得到的报错信息不完全)

    即直接在hive(defualt)下执行

    发现报错信息如下:

    running beyond virtual memory错误。

    提示如下:

    Container [pid=28920,containerID=container_1389136889967_0001_01_000121] is running beyond virtual memory limits. Current usage: 1.2 GB of 1 GB physical memory used; 2.2 GB of 2.1 GB virtual memory used. Killing container.…………

    原因:从机上运行的Container试图使用过多的内存,而被NodeManager kill掉了。

    解决:加内存

    在hadoop的配置文件mapred-site.xml中设置map和reduce任务的内存配置如下:(value中实际配置的内存需要根据自己机器内存大小及应用情况进行修改)

    <property>
      <name>mapreduce.map.memory.mb</name>
      <value>1536</value>
    </property>
    <property>
      <name>mapreduce.map.java.opts</name>
      <value>-Xmx1024M</value>
    </property>
    <property>
      <name>mapreduce.reduce.memory.mb</name>
      <value>3072</value>
    </property>
    <property>
      <name>mapreduce.reduce.java.opts</name>
      <value>-Xmx2560M</value>
    </property>

    关掉集群,分发配置,启动集群

    再次运行,?

    总结:没啥好总结的