准备

  1. JDK 1.8
  2. Maven 3.5.4
  3. Ant 1.9.15
  4. Hadoop 2.7.7
  5. Protobuf 2.5.0

配置运行环境

安装系统依赖包

1
sudo apt-get install libxrender1 libxtst6 libxi6 g++ autoconf automake libtool cmake zlib1g-dev pkg-config libssl-dev -y

下载完 JdkantMaven 后,将压缩包解压到 /opt/java/opt/maven/opt/ant/ 目录下。

Jdk

1
2
sudo mkdir -p /opt/java 
sudo tar -zxvf jdk-8u271-linux-x64.tar.gz -C /opt/java/

Maven

1
2
sudo mkdir -p /opt/maven
sudo tar -zxvf apache-maven-3.5.4-bin.tar.gz -C /opt/maven/

Ant

1
2
sudo mkdir -p /opt/ant
sudo tar -zxvf apache-ant-1.9.15-bin.tar.gz -C /opt/ant/

配置环境变量

1
sudo vim /etc/profile

在文件尾部追加以下内容

1
2
3
4
5
export JAVA_HOME=/opt/java/jdk1.8.0_271
export JRE_HOME=$JAVA_HOME/jre
export MAVEN_HOME=/opt/maven/apache-maven-3.5.4
export ANT_HOME=/opt/ant/apache-ant-1.9.15
PATH=$PATH:$JAVA_HOME/bin:$JRE_HOME/bin:$MAVEN_HOME/bin:$ANT_HOME/bin

使配置文件生效

1
source /etc/profile

终端输入 java -version

1
2
3
java version "1.8.0_271"
Java(TM) SE Runtime Environment (build 1.8.0_271-b09)
Java HotSpot(TM) 64-Bit Server VM (build 25.271-b09, mixed mode)

终端输入 mvn -version

1
2
3
4
5
Apache Maven 3.5.4 (1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T11:33:14-07:00)
Maven home: /opt/maven/apache-maven-3.5.4
Java version: 1.8.0_271, vendor: Oracle Corporation, runtime: /opt/java/jdk1.8.0_271/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "4.4.0-186-generic", arch: "amd64", family: "unix"

使用阿里云镜像仓库

修改 maven 配置文件,在 mirrors 标签下新增 阿里云镜像仓库配置

1
2
3
4
5
6
<mirror>
<id>nexus-aliyun</id>
<mirrorOf>central</mirrorOf>
<name>Nexus aliyun</name>
<url>http://maven.aliyun.com/nexus/content/groups/public/</url>
</mirror>

编译 protobuf

protobuf 解压到 /opt/protobuf 目录下

1
2
sudo mkdir -p /opt/protobuf
sudo tar -zxvf protobuf-2.5.0.tar.gz -C /opt/protobuf/

开始编译 protobuf

1
2
3
4
cd  /opt/protobuf/protobuf-2.5.0
sudo ./configure --prefix=/usr/local
sudo make
sudo make install

终端输入 protoc --version

libprotoc 2.5.0

编译 hadoop

解压 hadoop

1
2
sudo mkdir -p /opt/hadoop
sudo tar -zxvf hadoop-rel-release-2.7.7.tar.gz -C /opt/hadoop/

进入 hadoop 根目录,并使用 maven package 编译打包

我们把提前准备好的tomcat6.0.48文件放到下载目录,不然的话编译的时候去官网下载,那速度。。。,而且有可能会一直下载是吧。

1
2
3
4
mkdir -p /opt/hadoop/hadoop-2.7.7/hadoop-common-project/hadoop-kms/downloads
mkdir -p /opt/hadoop/hadoop-2.7.7/hadoop-hdfs-project/hadoop-hdfs-httpfs/downloads
cp apache-tomcat-6.0.48.tar.gz /opt/hadoop/hadoop-2.7.7/hadoop-common-project/hadoop-kms/downloads/
cp apache-tomcat-6.0.48.tar.gz /opt/hadoop/hadoop-2.7.7/hadoop-hdfs-project/hadoop-hdfs-httpfs/downloads/
1
2
3
4
5
cd /opt/hadoop
sudo chown -R $USER:$GROUPS .
mv hadoop-rel-release-2.7.7 hadoop-2.7.7
cd hadoop-2.7.7
mvn package -Pdist,native -DskipTests -Dtar
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Apache Hadoop Main 2.7.7 ........................... SUCCESS [ 2.163 s]
[INFO] Apache Hadoop Build Tools .......................... SUCCESS [ 2.272 s]
[INFO] Apache Hadoop Project POM .......................... SUCCESS [ 1.461 s]
[INFO] Apache Hadoop Annotations .......................... SUCCESS [ 3.528 s]
[INFO] Apache Hadoop Assemblies ........................... SUCCESS [ 0.403 s]
[INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [ 2.059 s]
[INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [ 4.722 s]
[INFO] Apache Hadoop MiniKDC .............................. SUCCESS [ 6.255 s]
[INFO] Apache Hadoop Auth ................................. SUCCESS [ 5.903 s]
[INFO] Apache Hadoop Auth Examples ........................ SUCCESS [ 4.657 s]
[INFO] Apache Hadoop Common ............................... SUCCESS [01:20 min]
[INFO] Apache Hadoop NFS .................................. SUCCESS [ 6.239 s]
[INFO] Apache Hadoop KMS .................................. SUCCESS [ 11.640 s]
[INFO] Apache Hadoop Common Project ....................... SUCCESS [ 0.124 s]
[INFO] Apache Hadoop HDFS ................................. SUCCESS [01:48 min]
[INFO] Apache Hadoop HttpFS ............................... SUCCESS [ 19.675 s]
[INFO] Apache Hadoop HDFS BookKeeper Journal .............. SUCCESS [ 48.737 s]
[INFO] Apache Hadoop HDFS-NFS ............................. SUCCESS [ 4.538 s]
[INFO] Apache Hadoop HDFS Project ......................... SUCCESS [ 0.118 s]
[INFO] hadoop-yarn ........................................ SUCCESS [ 0.133 s]
[INFO] hadoop-yarn-api .................................... SUCCESS [ 37.421 s]
[INFO] hadoop-yarn-common ................................. SUCCESS [02:46 min]
[INFO] hadoop-yarn-server ................................. SUCCESS [ 0.118 s]
[INFO] hadoop-yarn-server-common .......................... SUCCESS [ 10.927 s]
[INFO] hadoop-yarn-server-nodemanager ..................... SUCCESS [ 20.784 s]
[INFO] hadoop-yarn-server-web-proxy ....................... SUCCESS [ 3.615 s]
[INFO] hadoop-yarn-server-applicationhistoryservice ....... SUCCESS [ 7.805 s]
[INFO] hadoop-yarn-server-resourcemanager ................. SUCCESS [ 20.598 s]
[INFO] hadoop-yarn-server-tests ........................... SUCCESS [ 5.801 s]
[INFO] hadoop-yarn-client ................................. SUCCESS [ 6.577 s]
[INFO] hadoop-yarn-server-sharedcachemanager .............. SUCCESS [ 3.816 s]
[INFO] hadoop-yarn-applications ........................... SUCCESS [ 0.104 s]
[INFO] hadoop-yarn-applications-distributedshell .......... SUCCESS [ 3.102 s]
[INFO] hadoop-yarn-applications-unmanaged-am-launcher ..... SUCCESS [ 2.308 s]
[INFO] hadoop-yarn-site ................................... SUCCESS [ 0.078 s]
[INFO] hadoop-yarn-registry ............................... SUCCESS [ 5.831 s]
[INFO] hadoop-yarn-project ................................ SUCCESS [ 3.839 s]
[INFO] hadoop-mapreduce-client ............................ SUCCESS [ 0.234 s]
[INFO] hadoop-mapreduce-client-core ....................... SUCCESS [ 20.956 s]
[INFO] hadoop-mapreduce-client-common ..................... SUCCESS [ 17.338 s]
[INFO] hadoop-mapreduce-client-shuffle .................... SUCCESS [ 3.806 s]
[INFO] hadoop-mapreduce-client-app ........................ SUCCESS [ 9.730 s]
[INFO] hadoop-mapreduce-client-hs ......................... SUCCESS [ 6.573 s]
[INFO] hadoop-mapreduce-client-jobclient .................. SUCCESS [ 21.295 s]
[INFO] hadoop-mapreduce-client-hs-plugins ................. SUCCESS [ 2.017 s]
[INFO] Apache Hadoop MapReduce Examples ................... SUCCESS [ 6.246 s]
[INFO] hadoop-mapreduce ................................... SUCCESS [ 2.363 s]
[INFO] Apache Hadoop MapReduce Streaming .................. SUCCESS [ 11.421 s]
[INFO] Apache Hadoop Distributed Copy ..................... SUCCESS [ 13.918 s]
[INFO] Apache Hadoop Archives ............................. SUCCESS [ 2.359 s]
[INFO] Apache Hadoop Rumen ................................ SUCCESS [ 5.266 s]
[INFO] Apache Hadoop Gridmix .............................. SUCCESS [ 4.152 s]
[INFO] Apache Hadoop Data Join ............................ SUCCESS [ 2.902 s]
[INFO] Apache Hadoop Ant Tasks ............................ SUCCESS [ 2.081 s]
[INFO] Apache Hadoop Extras ............................... SUCCESS [ 3.052 s]
[INFO] Apache Hadoop Pipes ................................ SUCCESS [ 7.431 s]
[INFO] Apache Hadoop OpenStack support .................... SUCCESS [ 5.002 s]
[INFO] Apache Hadoop Amazon Web Services support .......... SUCCESS [02:46 min]
[INFO] Apache Hadoop Azure support ........................ SUCCESS [ 26.999 s]
[INFO] Apache Hadoop Client ............................... SUCCESS [ 7.450 s]
[INFO] Apache Hadoop Mini-Cluster ......................... SUCCESS [ 1.024 s]
[INFO] Apache Hadoop Scheduler Load Simulator ............. SUCCESS [ 5.816 s]
[INFO] Apache Hadoop Tools Dist ........................... SUCCESS [ 7.048 s]
[INFO] Apache Hadoop Tools ................................ SUCCESS [ 0.055 s]
[INFO] Apache Hadoop Distribution 2.7.7 ................... SUCCESS [ 35.905 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 16:55 min
[INFO] Finished at: 2021-01-10T21:51:46-08:00
[INFO] ------------------------------------------------------------------------