云服务器centos 安装hadoop集群

百度 搜索 云服务器centos 安装hadoop

创建Hadoop用户
sudo useradd hadoop -m -s /bin/bash
sudo passwd hadoop 
123456

下载Hadoop
wget https://mirrors.tuna.tsinghua.edu.cn/apache/hadoop/common/hadoop-3.2.4/hadoop-3.2.4.tar.gz 

解压并移动Hadoop到指定目录
tar -zxvf hadoop-3.2.4.tar.gz
sudo mv hadoop-3.2.4 /usr/local/hadoop

配置环境变量
sudo su - hadoop  # 切换到hadoop用户
vi ~/.bash_profile
export HADOOP_HOME=/usr/local/hadoop
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
source ~/.bash_profile

配置Hadoop配置文件
cd /usr/local/hadoop/etc/hadoop
sudo mkdir -p /usr/local/hadoop/dfs/name /usr/local/hadoop/dfs/data  
sudo chown -R hadoop:hadoop /usr/local/hadoop/dfs  

# core-site.xml
<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://节点:9000</value>
    </property>
</configuration>

# yarn-site.xml (Hadoop 3)
<configuration>  
    <property>  
        <name>yarn.resourcemanager.hostname</name>  
        <value>节点</value>  
    </property>  
</configuration> 

# hdfs-site.xml
<configuration>
    <property>
        <name>dfs.replication</name>
        <value>3</value>
    </property>
    <property> 
        <name>dfs.namenode.name.dir</name> 
        <value>file:/usr/local/hadoop/dfs/name</value> 
    </property> 
    <property> 
        <name>dfs.datanode.data.dir</name> 
        <value>file:/usr/local/hadoop/dfs/data</value> 
    </property> 
</configuration> 

 格式化HDFS并启动Hadoop服务
hdfs namenode -format  # 只应在第一次运行，或者在删除namenode数据目录后重新运行  
start-dfs.sh  # 启动HDFS  
start-yarn.sh  # 启动YARN

http://节点:50070/
http://节点:8088/

问题一
hadoop is not in the sudoers file.  This incident will be reported.
sudo visudo
hadoop ALL=(ALL:ALL) ALL

问题二
Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).
# 在主节点生成密钥
ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
# 将公钥复制到所有节点（包括自己）
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
chmod 600 ~/.ssh/authorized_keys
# 测试SSH连接
ssh localhost

问题三
localhost: ERROR: JAVA_HOME is not set and could not be found.
cd /usr/local/hadoop/etc/hadoop
sudo vim hadoop-env.sh
export JAVA_HOME=/usr/local/jdk1.8.0_202

问题四
Unit ssh.service could not be found.
Couldn't resolve host name for http://mirrorlist.centos.org/?release=8&arch=x86_64&repo=AppStream&infra=stock [Could not resolve host: mirrorlist.centos.org]