凌云的博客

行胜于言

Hadoop 编译以及安装

分类:hadoop| 发布时间:2018-04-13 09:48:00


概述

本文主要描述了在 Ubuntu 14.04 系统中如何通过 hadoop 源码进行编译以及安装 hadoop。

下载 hadoop 源码

在这里可以下载 hadoop 的最新源码,目前最新 Release 版本为 3.1。

https://hadoop.apache.org/releases.html

将源码包下载解压后,里面有个 BUILDING.txt 的文件介绍了怎么编译, 我这里重复下同时把遇到的问题以及解决方法记录下。

安装依赖

  • Java 这里推荐安装 Oracle Java
    $ sudo apt-get purge openjdk*
    $ sudo apt-get install software-properties-common
    $ sudo add-apt-repository ppa:webupd8team/java
    $ sudo apt-get update
    $ sudo apt-get install oracle-java8-installer
    
  • Maven
    $ sudo apt-get -y install maven
    
  • Native libraries
    $ sudo apt-get -y install build-essential autoconf automake libtool cmake zlib1g-dev pkg-config libssl-dev
    
  • ProtocolBuffer 2.5.0 (required)
    $ sudo apt-get -y install protobuf-compiler
    

编译分发包

mvn package -Pdist,native -DskipTests -Dtar

mvn 版本不对

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-enforcer-plugin:3.0.0-M1:enforce (default) on project hadoop-assemblies: Some Enforcer rules have failed. Look above for specific messages explaining why the rule failed. -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-enforcer-plugin:3.0.0-M1:enforce (default) on project hadoop-assemblies: Some Enforcer rules have failed. Look above for specific messages explaining why the rule failed.

这里报不符合 maven-enforcer-plugin 的某些规则,往上查看输出:

[DEBUG] Adding failure due to exception
org.apache.maven.enforcer.rule.api.EnforcerRuleException: Detected Maven Version: 3.0.5 is not in the allowed range [3.3.0,).
    at org.apache.maven.plugins.enforcer.AbstractVersionEnforcer.enforceVersion(AbstractVersionEnforcer.java:106)
    at org.apache.maven.plugins.enforcer.RequireMavenVersion.execute(RequireMavenVersion.java:47)
    at org.apache.maven.plugins.enforcer.EnforceMojo.execute(EnforceMojo.java:202)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:101)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:209)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:84)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:59)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.singleThreadedBuild(LifecycleStarter.java:183)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:161)
    at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:320)
    at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:156)
    at org.apache.maven.cli.MavenCli.execute(MavenCli.java:537)
    at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:196)
    at org.apache.maven.cli.MavenCli.main(MavenCli.java:141)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
[DEBUG] Executing rule: org.apache.maven.plugins.enforcer.RequireJavaVersion

maven.enforcer 要求 mvn 版本为 3.3.0。

查看本机安装的 mvn 版本(Ubuntu 14.04)

mvn -version

发现本机安装的版本为 3.0.5。

查看 hadoop-assemblies/pom.xml,在这里限制了 mvn 的版本

<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-enforcer-plugin</artifactId>
<inherited>false</inherited>
<configuration>
  <rules>
    <requireMavenVersion>
      <version>${enforced.maven.version}</version>
    </requireMavenVersion>
    <requireJavaVersion>
      <version>${enforced.java.version}</version>
    </requireJavaVersion>
  </rules>
</configuration>

搜下 enforced.maven.version 发现在 hadoop-project/pom.xml 定义了这个变量:

<enforced.maven.version>[3.3.0,)</enforced.maven.version>

将这里的 3.3.0 改为 3.0.5(当然也可以安装 3.3.0 版本的 mvn),重新执行命令

mvn package -Pdist,native -DskipTests -Dtar

编译完成后,分发包在

hadoop-dist/target/hadoop-3.1.0.tar.gz

将其解压到合适的目录

设置环境变量

接下来需要设置相关的环境变量(根据实际路径进行设置)

export JAVA_HOME=/usr/lib/jvm/java-8-oracle/
export HADOOP_INSTALL=/home/ubuntu/hadoop-3.1.0
export PATH=$PATH:$HADOOP_INSTALL/bin:$HADOOP_INSTALL/sbin

可以将其追加到

  • bash
~/.bashrc
  • zsh
~/.zshrc