hadoop-006完全分布式问题有哪些

小编给大家分享一下hadoop-006完全分布式问题有哪些，希望大家阅读完这篇文章之后都有所收获，下面让我们一起去探讨吧！

成都创新互联公司是一家专业提供柞水企业网站建设,专注与成都网站制作、做网站、外贸营销网站建设、H5开发、小程序制作等业务。10年已为柞水众多企业、政府机构等服务。创新互联专业网站制作公司优惠进行中。

一、运行mapreduce 任务，但是在yarn里看不到任务。

bin/yarn jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.2.jar wordcount /test /out1

我配置的resourceManager 为 http://192.168.31.136:8088

原因是在 mapred-site.xml 没有配置

mapreduce.framework.name
yarn

二、配置好问题一后，运行mr任务提示The auxService:mapreduce_shuffle does not exist

原因是在yarn-site.xml 没有配置yarn.nodemanager.aux-services 节点

yarn.nodemanager.aux-services
mapreduce_shuffle

报错如下

16/11/29 23:10:45 INFO mapreduce.Job: Task Id : attempt_1480432102879_0001_m_000000_2, Status : FAILED
Container launch failed for container_e02_1480432102879_0001_01_000004 : org.apache.hadoop.yarn.exceptions.InvalidAuxServiceException: The auxService:mapreduce_shuffle does not exist
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateException(SerializedExceptionPBImpl.java:168)
        at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.deSerialize(SerializedExceptionPBImpl.java:106)
        at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:155)
        at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:375)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

-----------------------------------------------------------------------------------------------------------------

配置文件记录

已提前搭建好三个节点的zookeeper 小集群，配置可以实现 HDFS　ＨＡ和　ＹＡＲＮ　ＨＡ

一、hadoop-env.sh 修改了

export JAVA_HOME=/usr/lib/jvm/jdk8/jdk1.8.0_111

二、yarn-env.sh 修改了

export JAVA_HOME=/usr/lib/jvm/jdk8/jdk1.8.0_111

三、core-site.xml

      fs.defaultFS
      hdfs://mycluster



      dfs.journalnode.edits.dir
      /home/jxlgzwh/hadoop-2.7.2/data/jn




       hadoop.tmp.dir
       /home/jxlgzwh/hadoop-2.7.2/data/tmp


       ha.zookeeper.quorum
       master:2181,slave01:2181,slave02:2181

四、hdfs-site.xml

      dfs.nameservices
      mycluster



      dfs.ha.namenodes.mycluster
      nn1,nn2




      dfs.namenode.rpc-address.mycluster.nn1
      master:8020


      dfs.namenode.rpc-address.mycluster.nn2
      slave01:8020




      dfs.namenode.http-address.mycluster.nn1
      master:50070


      dfs.namenode.http-address.mycluster.nn2
      slave01:50070



      dfs.namenode.shared.edits.dir
      qjournal://master:8485;slave01:8485;slave02:8485/mycluster



      dfs.client.failover.proxy.provider.mycluster
      org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider



      dfs.ha.fencing.methods
      sshfence



      dfs.ha.fencing.ssh.private-key-files
      /home/jxlgzwh/.ssh/id_dsa



       dfs.replication
       3



       dfs.permissions.enabled
       false



       dfs.ha.automatic-failover.enabled
       true

五、mapred-site.xml

mapreduce.framework.name
yarn

六、slaves

192.168.31.136
192.168.31.130
192.168.31.229

七、yarn-site.xml

yarn.resourcemanager.ha.enabled
true

yarn.resourcemanager.cluster-id
cluster1

yarn.resourcemanager.ha.rm-ids
rm1,rm2

yarn.resourcemanager.hostname.rm1
master

yarn.resourcemanager.hostname.rm2
slave01

yarn.resourcemanager.webapp.address.rm1
master:8088

yarn.resourcemanager.webapp.address.rm2
slave01:8088

yarn.resourcemanager.zk-address
master:2181,slave01:2181,slave02:2181

yarn.resourcemanager.recovery.enabled
true

yarn.resourcemanager.zk-state-store.parent-path
/rmstore

yarn.resourcemanager.store.class
org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore

yarn.nodemanager.recovery.enabled
true

yarn.nodemanager.address
0.0.0.0:45454

yarn.nodemanager.recovery.dir
/home/jxlgzwh/hadoop-2.7.2/data/tmp/yarn-nm-recovery

yarn.nodemanager.aux-services
mapreduce_shuffle

八、配置 /etc/hosts 文件并配置 ssh免密码登录

192.168.31.136 master.com master
192.168.31.130 slave01
192.168.31.229 slave02

看完了这篇文章，相信你对“hadoop-006完全分布式问题有哪些”有了一定的了解，如果想了解更多相关知识，欢迎关注创新互联行业资讯频道，感谢各位的阅读！

文章标题：hadoop-006完全分布式问题有哪些
链接URL：http://cdweb.net/article/iehcej.html

NEWS

网站建设资讯

hadoop-006完全分布式问题有哪些

其他资讯