基于zookeeper的Hadoop集群搭建详细步骤

目录

一、一些基本概念

二、集群配置图

三、Hadoop高可用集群配置步骤

1.在第一台虚拟机解压hadoop-3.1.3.tar.gz到/opt/soft/目录

2.修改文件名、属主和属组

3.配置windows四台虚拟机的ip映射

4.修改hadoop配置文件

(1)hadoop-env.sh

(2)workers

(3)crore-site.xml

(4)hdfs-site.xml

(5)mapred-site.xml

(6)yarn-site.xml

5.拷贝hadoop到其他三台虚拟机

6.分配环境变量

7.重启环境变量，检验四台虚拟机安装是否成功

四、首次启动hadoop集群 

1.高可用启动之前要启动zookeeper

2.三台机器启动JournalNode

3.第一台机器格式化

4.第一台机器启动namenode

5.在第二台机器同步namenode信息

6.第二台机器启动namenode

7.每台机器查看namenode的状态都是standby

8.关闭所有的与dfs有关的服务

9.格式化zookeeper

10.zkCli.sh

11.启动dfs

12.查看namenode节点状态

13.打开网页登录查看

14.每台虚拟机下载主备切换工具

15.启动yarn

16.有resourcemanager的主机名登录8088端口

17.查看resourcemanager节点状态

18.关闭集群

---

zookeeper集群的安装步骤参考博文《搭建zookeeper高可用集群详细步骤》，注意主机名，这里换了主机名ant161=ant165;ant162=ant166;ant163=ant167;ant164=ant168

一、一些基本概念

JournalNode的作用

Hadoop集群中的DFSZKFailoverController进程的作用

二、集群配置图

ant161	ant162	ant163	ant164
NameNode	NameNode
DataNode	DataNode	DataNode	DataNode
NodeManager	NodeManager	NodeManager	NodeManager
		ResourceManager	ResourceManager
JournalNode监控NameNode是否同步	JournalNode	JournalNode
DFSZKFailoverController监控NameNode是否存活	DFSZKFailoverController
zookeeper0	zookeeper1	zookeeper2
JobHistory

三、Hadoop高可用集群配置步骤

1.在第一台虚拟机解压hadoop-3.1.3.tar.gz到/opt/soft/目录

[root@ant161 install]# tar -zxf ./hadoop-3.1.3.tar.gz -C /opt/soft/

2.修改文件名、属主和属组

[root@ant161 soft]# mv ./hadoop-3.1.3/ hadoop313[root@ant161 soft]# chown -R root:root ./hadoop313/

3.配置windows四台虚拟机的ip映射

C:\Windows\System32\drivers\etc目录下的host文件，添加以下的主机ip配置

4.修改hadoop配置文件

(1)hadoop-env.sh

# The java implementation to use. By default, this environment
# variable is REQUIRED on ALL platforms except OS X!
export JAVA_HOME=/opt/soft/jdk180
export HDFS_NAMENODE_USER=root
export HDFS_DATANODE_USER=root
export HDFS_SECONDARYNAMENODE_USER=root
export HDFS_JOURNALNODE_USER=root
export HDFS_ZKFC_USER=root
export YARN_RESOURCEMANAGER_USER=root
export YARN_NODEMANAGER_USER=root

(2)workers

        输入四台虚拟机主机名

ant161
ant162
ant163
ant164

(3)crore-site.xml

<configuration><property><name>fs.defaultFS</name><value>hdfs://gky</value><description>逻辑名称，必须与hdfs-site.cml中的dfs.nameservices值保持一致</description></property><property><name>hadoop.tmp.dir</name><value>/opt/soft/hadoop313/tmpdata</value><description>namenode上本地的hadoop临时文件夹</description></property><property><name>hadoop.http.staticuser.user</name><value>root</value><description>默认用户</description></property><property><name>hadoop.proxyuser.root.hosts</name><value>*</value><description></description></property><property><name>hadoop.proxyuser.root.groups</name><value>*</value><description></description></property><property><name>io.file.buffer.size</name><value>131072</value><description>读写文件的buffer大小为：128K</description></property><property><name>ha.zookeeper.quorum</name><value>ant161:2181,ant162:2181,ant163:2181</value><description></description></property><property><name>ha.zookeeper.session-timeout.ms</name><value>10000</value><description>hadoop链接zookeeper的超时时长设置为10s</description></property>
</configuration>

(4)hdfs-site.xml

<configuration><property><name>dfs.replication</name><value>3</value><description>Hadoop中每一个block的备份数</description></property><property><name>dfs.namenode.name.dir</name><value>/opt/soft/hadoop313/data/dfs/name</value><description>namenode上存储hdfs名字空间元数据目录</description></property><property><name>dfs.datanode.data.dir</name><value>/opt/soft/hadoop313/data/dfs/data</value><description>datanode上数据块的物理存储位置</description></property><property><name>dfs.namenode.secondary.http-address</name><value>ant161:9869</value><description></description></property><property><name>dfs.nameservices</name><value>gky</value><description>指定hdfs的nameservice,需要和core-site.xml中保持一致</description></property><property><name>dfs.ha.namenodes.gky</name><value>nn1,nn2</value><description>gky为集群的逻辑名称，映射两个namenode逻辑名</description></property><property><name>dfs.namenode.rpc-address.gky.nn1</name><value>ant161:9000</value><description>namenode1的RPC通信地址</description></property><property><name>dfs.namenode.http-address.gky.nn1</name><value>ant161:9870</value><description>namenode1的http通信地址</description></property><property><name>dfs.namenode.rpc-address.gky.nn2</name><value>ant162:9000</value><description>namenode2的RPC通信地址</description></property><property><name>dfs.namenode.http-address.gky.nn2</name><value>ant162:9870</value><description>namenode2的http通信地址</description></property><property><name>dfs.namenode.shared.edits.dir</name><value>qjournal://ant161:8485;ant162:8485;ant163:8485/gky</value><description>指定NameNode的edits元数据的共享存储位置(JournalNode列表)</description></property><property><name>dfs.journalnode.edits.dir</name><value>/opt/soft/hadoop313/data/journaldata</value><description>指定JournalNode在本地磁盘存放数据的位置</description></property><!-- 容错 --><property><name>dfs.ha.automatic-failover.enabled</name><value>true</value><description>启用NameNode故障自动切换</description></property><property><name>dfs.client.failover.proxy.provider.gky</name><value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value><description>失败后自动切换的实现方式</description></property><property><name>dfs.ha.fencing.methods</name><value>sshfence</value><description>防止脑裂的处理</description></property><property><name>dfs.ha.fencing.ssh.private-key-files</name><value>/root/.ssh/id_rsa</value><description>使用sshfence隔离机制时需要ssh免登陆</description></property><property><name>dfs.permissions.enabled</name><value>false</value><description>关闭HDFS操作权限验证</description></property><property><name>dfs.image.transfer.bandwidthPerSec</name><value>1048576</value><description>1M</description></property><property><name>dfs.block.scanner.volume.bytes.per.second</name><value>1048576</value><description>如果该值为0，则DataNode的块扫描程序将被禁用。如果这是正数，则这是DataNode的块扫描程序将尝试从每个卷扫描的每秒字节数。</description></property>
</configuration>

(5)mapred-site.xml

<property><name>mapreduce.framework.name</name><value>yarn</value><description>job执行框架： local, classic or yarn</description>
</property>
<property><name>mapreduce.application.classpath</name><value>/opt/soft/hadoop313/etc/hadoop:/opt/soft/hadoop313/share/hadoop/common/lib/*:/opt/soft/hadoop313/share/hadoop/common/*:/opt/soft/hadoop313/share/hadoop/hdfs/*:/opt/soft/hadoop313/share/hadoop/hdfs/lib/*:/opt/soft/hadoop313/share/hadoop/mapreduce/*:/opt/soft/hadoop313/share/hadoop/mapreduce/lib/*:/opt/soft/hadoop313/share/hadoop/yarn/*:/opt/soft/hadoop313/share/hadoop/yarn/lib/*</value><description></description>
</property>
<property><name>mapreduce.jobhistory.address</name><value>ant161:10020</value></property><property><name>mapreduce.jobhistory.webapp.address</name><value>ant161:19888</value></property>
<property><name>mapreduce.map.memory.mb</name><value>1024</value><description>设置map阶段的task工作内存</description>
</property>
<property><name>mapreduce.reduce.memory.mb</name><value>2048</value><description>设置reduce阶段的task工作内存</description>
</property>

(6)yarn-site.xml

<configuration><property><name>yarn.resourcemanager.ha.enabled</name><value>true</value><description>开启resourcemanager高可用</description></property><property><name>yarn.resourcemanager.cluster-id</name><value>yrcabc</value><description>指定yarn集群中的id</description></property><property><name>yarn.resourcemanager.ha.rm-ids</name><value>rm1,rm2</value><description>指定resourcemanager的名字</description></property><property><name>yarn.resourcemanager.hostname.rm1</name><value>ant163</value><description>设置rm1的名字</description></property><property><name>yarn.resourcemanager.hostname.rm2</name><value>ant164</value><description>设置rm2的名字</description></property><property><name>yarn.resourcemanager.webapp.address.rm1</name><value>ant163:8088</value><description></description></property><property><name>yarn.resourcemanager.webapp.address.rm2</name><value>ant164:8088</value><description></description></property><property><name>yarn.resourcemanager.zk-address</name><value>ant161:2181,ant162:2181,ant163:2181</value><description>指定zookeeper集群地址</description></property><property><name>yarn.nodemanager.aux-services</name><value>mapreduce_shuffle</value><description>运行mapreduce程序必须配置的附属服务</description></property><property><name>yarn.nodemanager.local-dirs</name><value>/opt/soft/hadoop313/tmpdata/yarn/local</value><description>nodemanager本地存储目录</description></property><property><name>yarn.nodemanager.log-dirs</name><value>/opt/soft/hadoop313/tmpdata/yarn/log</value><description>nodemanager本地日志目录</description></property><property><name>yarn.nodemanager.resource.memory-mb</name><value>2048</value><description>resource进程的工作内存</description></property><property><name>yarn.nodemanager.resource.cpu-vcores</name><value>2</value><description>resource工作中所能使用机器的内核数</description></property><!--下面三个配置在公司要删除--><property><name>yarn.scheduler.minimum-allocation-mb</name><value>256</value><description></description></property><property><name>yarn.log-aggregation-enable</name><value>true</value><description></description></property><property><name>yarn.log-aggregation.retain-seconds</name><value>86400</value><description>日志保留多少秒</description></property><property><name>yarn.nodemanager.vmem-check-enabled</name><value>false</value><description></description></property><property><name>yarn.application.classpath</name><value>/opt/soft/hadoop313/etc/hadoop:/opt/soft/hadoop313/share/hadoop/common/lib/*:/opt/soft/hadoop313/share/hadoop/common/*:/opt/soft/hadoop313/share/hadoop/hdfs/*:/opt/soft/hadoop313/share/hadoop/hdfs/lib/*:/opt/soft/hadoop313/share/hadoop/mapreduce/*:/opt/soft/hadoop313/share/hadoop/mapreduce/lib/*:/opt/soft/hadoop313/share/hadoop/yarn/*:/opt/soft/hadoop313/share/hadoop/yarn/lib/*</value></property><property><name>yarn.nodemanager.env-whitelist</name><value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME</value></property>
</configuration>

5.拷贝hadoop到其他三台虚拟机

[root@ant161 soft]# scp -r ./hadoop313/ root@ant162:/opt/soft/
[root@ant161 soft]# scp -r ./hadoop313/ root@ant163:/opt/soft/
[root@ant161 soft]# scp -r ./hadoop313/ root@ant164:/opt/soft/

6.分配环境变量

[root@ant161 soft]# scp /etc/profile root@ant162:/etc/
profile                                                                                                                          100% 2202     1.4MB/s   00:00    
[root@ant161 soft]# scp /etc/profile root@ant163:/etc/
profile                                                                                                                          100% 2202     1.4MB/s   00:00    
[root@ant161 soft]# scp /etc/profile root@ant164:/etc/
profile                                              

7.重启环境变量，检验四台虚拟机安装是否成功

source /etc/profilehadoop hadoop version

四、首次启动hadoop集群 

1.高可用启动之前要启动zookeeper

[root@ant161 soft]# /opt/shell/zkop.sh start
------------ ant161 zookeeper -----------
JMX enabled by default
Using config: /opt/soft/zk345/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
------------ ant162 zookeeper -----------
JMX enabled by default
Using config: /opt/soft/zk345/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
------------ ant163 zookeeper -----------
JMX enabled by default
Using config: /opt/soft/zk345/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED[root@ant161 soft]# /opt/shell/showjps.sh 
---------------- ant161 服务启动状态 -----------------
2532 QuorumPeerMain
2582 Jps
---------------- ant162 服务启动状态 -----------------
2283 QuorumPeerMain
2335 Jps
---------------- ant163 服务启动状态 -----------------
2305 Jps
2259 QuorumPeerMain
---------------- ant164 服务启动状态 -----------------
2233 Jps[root@ant161 soft]# /opt/shell/zkop.sh status
------------ ant161 zookeeper -----------
JMX enabled by default
Using config: /opt/soft/zk345/bin/../conf/zoo.cfg
Mode: follower
------------ ant162 zookeeper -----------
JMX enabled by default
Using config: /opt/soft/zk345/bin/../conf/zoo.cfg
Mode: leader
------------ ant163 zookeeper -----------
JMX enabled by default
Using config: /opt/soft/zk345/bin/../conf/zoo.cfg
Mode: follower

2.三台机器启动JournalNode

[root@ant161 soft]# hdfs --daemon start journalnode
WARNING: /opt/soft/hadoop313/logs does not exist. Creating.[root@ant162 soft]# hdfs --daemon start journalnode
WARNING: /opt/soft/hadoop313/logs does not exist. Creating.[root@ant163 soft]# hdfs --daemon start journalnode
WARNING: /opt/soft/hadoop313/logs does not exist. Creating.

3.第一台机器格式化

[root@ant161 soft]# hdfs namenode -format

4.第一台机器启动namenode

[root@ant161 hadoop]# hdfs --daemon start namenode

5.在第二台机器同步namenode信息

[root @ant162 soft]# hdfs namenode -bootstrapStandby

6.第二台机器启动namenode

[root @ant162 soft]# hdfs --daemon start namenode

7.每台机器查看namenode的状态都是standby

[root @ant161 soft]# hdfs haadmin -getServiceState nn1
standby
[root @ant161 soft]#  hdfs haadmin -getServiceState nn2
standby

8.关闭所有的与dfs有关的服务

[root @ant161 soft]# stop-dfs.sh

9.格式化zookeeper

[root @ant161 soft]# hdfs zkfc -formatZK

10.zkCli.sh

11.启动dfs

12.查看namenode节点状态

[root@ant161 soft]# hdfs haadmin -getServiceState nn1
standby
[root@ant161 soft]# hdfs haadmin -getServiceState nn2
active

13.打开网页登录查看

14.每台虚拟机下载主备切换工具

[root@ant161 soft]# yum install psmisc -y

此时如果停止active那一台的namenode，7218是active那一台机器namenode的进程号

[root@ant162 soft]# kill -9 7218

此时active那一台网页无法连接，另一台没有关闭namenode的机器的网页变为active

再重新启动关闭的namenode，网页端的两个网址刷新，就会发现刚刚变为active的那一台机器，还是active状态，而另一台重启的机器就编程了standby等待状态

15.启动yarn

16.有resourcemanager的主机名登录8088端口

哪一台机器是active的状态，就会自动跳转到那一台机器的主机名

17.查看resourcemanager节点状态

[root@ant161 soft]# yarn rmadmin -getServiceState rm1
active
[root@ant161 soft]# yarn rmadmin -getServiceState rm2
standby

18.关闭集群

(1)关闭dfs

[root@ant161 soft]# stop-dfs.sh

(2)关闭yarn

[root@ant161 soft]# stop-yarn.sh

(3)关闭journalnode

[root@ant161 soft]# hdfs --daemon stop journalnode

(4)关闭zookeeper

[root@ant161 soft]# /opt/shell/zkop.sh stop