hadoop yarn jobhistoryserver 配置

hadoop1.x之前的版本中可以开启50030端口,查看历史作业的运行日志,包括mr日志和自定义日志,但是hadoop2.x 是用MRv2(yarn)作为作业运行服务,代替50030端口的是一个jobhistory服务.jobhistory记录下已运行完的MapReduce作业信息并存放在指定的HDFS目录下，默认情况下是没有启动的，需要配置完后手工启动服务。

1、编辑mapred-site.xml文件:

<configuration>
        <property>
                <name>mapreduce.framework.name</name>
                <value>yarn</value>
        </property>
        <property>
                <name>mapreduce.jobhistory.address</name>
                <value>10.4.1.1:10020</value>
        </property>
        <property>
                <name>mapreduce.jobhistory.webapp.address</name>
                <value>10.4.1.1:19888</value>
        </property>
        <property>
                <name>mapreduce.jobhistory.joblist.cache.size</name>
                <value>1000</value>
                <description>default 20000</description>
        </property>
        <property>
                <name>mapred.child.java.opts</name>
                <value>-Xmx512m</value>
        </property>
        <property>
                <name>mapreduce.jobhistory.cleaner.enable</name>
                <value>true</value>
        </property>
        <property>
                <name>mapreduce.jobhistory.cleaner.interval-ms</name>
                <value>86400000</value>
                <description>the job history cleaner checks for files to delete, in milliseconds. Default 86400000 (one day). Files are only deleted if they are older than</description>
        </property>
        <property>
                <name>mapreduce.jobhistory.max-age-ms</name>
                <value>432000000</value>
        <description>Job history files older than this many milliseconds will be deleted when the history cleaner runs. Defaults to 604800000 (1 week)</description>
        </property>

2、启动history-server Hadoop启动jobhistoryserver来实现web查看作业的历史运行情况，由于在启动hdfs和Yarn进程之后，jobhistoryserver进程并没有启动，需要手动启动， 启动的方法是通过(注意：必须是两个命令)： ./mr-jobhistory-daemon.sh start historyserver ./yarn-daemon.sh start timelineserver

启动完成后，查看进程
org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer
org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer

3、验证

我们在Spark上以YARN方式启动一个任务，然后通过Hadoop YARN来查看日志

3.1 YARN模式下启动SPARK案例

bin/spark-submit  --class  org.apache.spark.examples.SparkPi 
--master yarn-cluster 
--num-executors 3 
--driver-memory 1g 
--executor-memory 1g 
--executor-cores 1 
lib/spark-examples*.jar  10

3.1 查看YARN-Cluster，并查看到刚才执行的任务，然后history->logs 就可以查看日志了