Shell脚本之——Hadoop3单机版安装
目录
1.解压
2.文件重命名
3.配置环境变量
4.hadoop-env.sh
5.core-site.xml
6. hdfs-site.xml
7. mapred-site.xml
8.yarn-site.xml
9.完整脚本代码(注意修改主机名)
10.重启环境变量
11.初始化
12.启动服务
13.jps查询节点
1.解压
tar -zxf /opt/install/hadoop-3.1.3.tar.gz -C /opt/soft/
2.文件重命名
mv /opt/soft/hadoop-3.1.3 /opt/soft/hadoop313
3.配置环境变量
sed -i '73a\export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop' /etc/profilesed -i '73a\export HADOOP_LIBEXEC_DIR=$HADOOP_HOME/libexec' /etc/profilesed -i '73a\export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native' /etc/profilesed -i '73a\export HADOOP_INSTALL=$HADOOP_HOME' /etc/profilesed -i '73a\export HADOOP_YARN_HOME=$HADOOP_HOME' /etc/profilesed -i '73a\export HADOOP_HDFS_HOME=$HADOOP_HOME' /etc/profilesed -i '73a\export HADOOP_COMMON_HOME=$HADOOP_HOME' /etc/profilesed -i '73a\export HADOOP_MAPRED_HOME=$HADOOP_HOME' /etc/profilesed -i '73a\export YARN_NODEMANAGER_USER=root' /etc/profilesed -i '73a\export YARN_RESOURCEMANAGER_USER=root' /etc/profilesed -i '73a\export HDFS_ZKFC_USER=root' /etc/profilesed -i '73a\export HDFS_JOURNALNODE_USER=root' /etc/profilesed -i '73a\export HDFS_SECONDARYNAMENODE_USER=root' /etc/profilesed -i '73a\export HDFS_DATANODE_USER=root' /etc/profilesed -i '73a\export HDFS_NAMENODE_USER=root' /etc/profilesed -i '73a\export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$HADOOP_HOME/lib' /etc/profilesed -i '73a\export HADOOP_HOME=/opt/soft/hadoop313' /etc/profilesed -i '73a\#HADOOP_HOME' /etc/profile
4.hadoop-env.sh
#hadoop-env.shsed -i '53a\export JAVA_HOME=/opt/soft/jdk180' /opt/soft/hadoop313/etc/hadoop/hadoop-env.sh
5.core-site.xml
#core-site.xmlsed -i '19a\ </property>' /opt/soft/hadoop313/etc/hadoop/core-site.xmlsed -i '19a\ <value>*</value>' /opt/soft/hadoop313/etc/hadoop/core-site.xmlsed -i '19a\ <name>hadoop.proxyuser.root.groups</name>' /opt/soft/hadoop313/etc/hadoop/core-site.xmlsed -i '19a\ <property>' /opt/soft/hadoop313/etc/hadoop/core-site.xmlsed -i '19a\ </property>' /opt/soft/hadoop313/etc/hadoop/core-site.xmlsed -i '19a\ <value>*</value>' /opt/soft/hadoop313/etc/hadoop/core-site.xmlsed -i '19a\ <name>hadoop.proxyuser.root.hosts</name>' /opt/soft/hadoop313/etc/hadoop/core-site.xmlsed -i '19a\ <property>' /opt/soft/hadoop313/etc/hadoop/core-site.xmlsed -i '19a\ </property>' /opt/soft/hadoop313/etc/hadoop/core-site.xmlsed -i '19a\ <description>读写队列缓存:128k</description>' /opt/soft/hadoop313/etc/hadoop/core-site.xmlsed -i '19a\ <value>131072</value>' /opt/soft/hadoop313/etc/hadoop/core-site.xmlsed -i '19a\ <name>io.file.buffer.size</name>' /opt/soft/hadoop313/etc/hadoop/core-site.xmlsed -i '19a\ <property>' /opt/soft/hadoop313/etc/hadoop/core-site.xmlsed -i '19a\ </property>' /opt/soft/hadoop313/etc/hadoop/core-site.xmlsed -i '19a\ <value>root</value>' /opt/soft/hadoop313/etc/hadoop/core-site.xmlsed -i '19a\ <name>hadoop.http.staticuser.user</name>' /opt/soft/hadoop313/etc/hadoop/core-site.xmlsed -i '19a\ <property>' /opt/soft/hadoop313/etc/hadoop/core-site.xmlsed -i '19a\ </property>' /opt/soft/hadoop313/etc/hadoop/core-site.xmlsed -i '19a\ <description>namenode上本地的hadoop临时文件夹</description>' /opt/soft/hadoop313/etc/hadoop/core-site.xmlsed -i '19a\ <value>/opt/soft/hadoop313/data</value>' /opt/soft/hadoop313/etc/hadoop/core-site.xmlsed -i '19a\ <name>hadoop.tmp.dir</name>' /opt/soft/hadoop313/etc/hadoop/core-site.xmlsed -i '19a\ <property>' /opt/soft/hadoop313/etc/hadoop/core-site.xmlsed -i '19a\ </property>' /opt/soft/hadoop313/etc/hadoop/core-site.xmlsed -i '19a\ <value>hdfs://cp145:9000</value>' /opt/soft/hadoop313/etc/hadoop/core-site.xmlsed -i '19a\ <name>fs.defaultFS</name>' /opt/soft/hadoop313/etc/hadoop/core-site.xmlsed -i '19a\ <property>' /opt/soft/hadoop313/etc/hadoop/core-site.xml
6. hdfs-site.xml
#hdfs-site.xmlsed -i '19a\ </property>' /opt/soft/hadoop313/etc/hadoop/hdfs-site.xmlsed -i '19a\ <description>关闭权限验证</description>' /opt/soft/hadoop313/etc/hadoop/hdfs-site.xmlsed -i '19a\ <value>false</value>' /opt/soft/hadoop313/etc/hadoop/hdfs-site.xmlsed -i '19a\ <name>dfs.permissions.enabled</name>' /opt/soft/hadoop313/etc/hadoop/hdfs-site.xmlsed -i '19a\ <property>' /opt/soft/hadoop313/etc/hadoop/hdfs-site.xmlsed -i '19a\ </property>' /opt/soft/hadoop313/etc/hadoop/hdfs-site.xmlsed -i '19a\ <description>datanode上数据块的物理存储位置目录</description>' /opt/soft/hadoop313/etc/hadoop/hdfs-site.xmlsed -i '19a\ <value>/opt/soft/hadoop313/data/dfs/data</value>' /opt/soft/hadoop313/etc/hadoop/hdfs-site.xmlsed -i '19a\ <name>dfs.datanode.data.dir</name>' /opt/soft/hadoop313/etc/hadoop/hdfs-site.xmlsed -i '19a\ <property>' /opt/soft/hadoop313/etc/hadoop/hdfs-site.xmlsed -i '19a\ </property>' /opt/soft/hadoop313/etc/hadoop/hdfs-site.xmlsed -i '19a\ <description>namenode上存储hdfsq名字空间元数据的目录</description>' /opt/soft/hadoop313/etc/hadoop/hdfs-site.xmlsed -i '19a\ <value>/opt/soft/hadoop313/data/dfs/name</value>' /opt/soft/hadoop313/etc/hadoop/hdfs-site.xmlsed -i '19a\ <name>dfs.namenode.name.dir</name>' /opt/soft/hadoop313/etc/hadoop/hdfs-site.xmlsed -i '19a\ <property>' /opt/soft/hadoop313/etc/hadoop/hdfs-site.xmlsed -i '19a\ </property>' /opt/soft/hadoop313/etc/hadoop/hdfs-site.xmlsed -i '19a\ <description>hadoop中每一个block文件的备份数量</description>' /opt/soft/hadoop313/etc/hadoop/hdfs-site.xmlsed -i '19a\ <value>1</value>' /opt/soft/hadoop313/etc/hadoop/hdfs-site.xmlsed -i '19a\ <name>dfs.replication</name>' /opt/soft/hadoop313/etc/hadoop/hdfs-site.xmlsed -i '19a\ <property>' /opt/soft/hadoop313/etc/hadoop/hdfs-site.xml
7. mapred-site.xml
#mapred-site.xmlsed -i '19a\ </property>' /opt/soft/hadoop313/etc/hadoop/mapred-site.xmlsed -i '19a\ <value>1024</value>' /opt/soft/hadoop313/etc/hadoop/mapred-site.xmlsed -i '19a\ <name>mapreduce.reduce.memory.mb</name>' /opt/soft/hadoop313/etc/hadoop/mapred-site.xmlsed -i '19a\ <property>' /opt/soft/hadoop313/etc/hadoop/mapred-site.xmlsed -i '19a\ </property>' /opt/soft/hadoop313/etc/hadoop/mapred-site.xmlsed -i '19a\ <value>1024</value>' /opt/soft/hadoop313/etc/hadoop/mapred-site.xmlsed -i '19a\ <name>mapreduce.map.memory.mb</name>' /opt/soft/hadoop313/etc/hadoop/mapred-site.xmlsed -i '19a\ <property>' /opt/soft/hadoop313/etc/hadoop/mapred-site.xmlsed -i '19a\ </property>' /opt/soft/hadoop313/etc/hadoop/mapred-site.xmlsed -i '19a\ <value>cp145:19888</value>' /opt/soft/hadoop313/etc/hadoop/mapred-site.xmlsed -i '19a\ <name>mapreduce.jobhistory.webapp.address</name>' /opt/soft/hadoop313/etc/hadoop/mapred-site.xmlsed -i '19a\ <property>' /opt/soft/hadoop313/etc/hadoop/mapred-site.xmlsed -i '19a\ </property>' /opt/soft/hadoop313/etc/hadoop/mapred-site.xmlsed -i '19a\ <value>cp145:10020</value>' /opt/soft/hadoop313/etc/hadoop/mapred-site.xmlsed -i '19a\ <name>mapreduce.jobhistory.address</name>' /opt/soft/hadoop313/etc/hadoop/mapred-site.xmlsed -i '19a\ <property>' /opt/soft/hadoop313/etc/hadoop/mapred-site.xmlsed -i '19a\ </property>' /opt/soft/hadoop313/etc/hadoop/mapred-site.xmlsed -i '19a\ <value>/opt/soft/hadoop313/etc/hadoop:/opt/soft/hadoop313/share/hadoop/common/lib/*:/opt/soft/hadoop313/share/hadoop/common/*:/opt/soft/hadoop313/share/hadoop/hdfs/*:/opt/soft/hadoop313/share/hadoop/hdfs/lib/*:/opt/soft/hadoop313/share/hadoop/mapreduce/*:/opt/soft/hadoop313/share/hadoop/mapreduce/lib/*:/opt/soft/hadoop313/share/hadoop/yarn/*:/opt/soft/hadoop313/share/hadoop/yarn/lib/*</value>' /opt/soft/hadoop313/etc/hadoop/mapred-site.xmlsed -i '19a\ <name>mapreduce.application.classpath</name>' /opt/soft/hadoop313/etc/hadoop/mapred-site.xmlsed -i '19a\ <property>' /opt/soft/hadoop313/etc/hadoop/mapred-site.xmlsed -i '19a\ </property>' /opt/soft/hadoop313/etc/hadoop/mapred-site.xmlsed -i '19a\ <final>true</final>' /opt/soft/hadoop313/etc/hadoop/mapred-site.xmlsed -i '19a\ <description>job执行框架:local,classic or yarn</description>' /opt/soft/hadoop313/etc/hadoop/mapred-site.xmlsed -i '19a\ <value>yarn</value>' /opt/soft/hadoop313/etc/hadoop/mapred-site.xmlsed -i '19a\ <name>mapreduce.framework.name</name>' /opt/soft/hadoop313/etc/hadoop/mapred-site.xmlsed -i '19a\ <property>' /opt/soft/hadoop313/etc/hadoop/mapred-site.xml
8.yarn-site.xml
#yarn-site.xmlsed -i '17a\ </property>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xmlsed -i '17a\ <value>/opt/soft/hadoop313/yarndata/log</value>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xmlsed -i '17a\ <name>yarn.nodemanager.log-dirs</name>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xmlsed -i '17a\ <property>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xmlsed -i '17a\ </property>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xmlsed -i '17a\ <value>/opt/soft/hadoop313/yarndata/yarn</value>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xmlsed -i '17a\ <name>yarn.nodemanager.local-dirs</name>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xmlsed -i '17a\ <property>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xmlsed -i '17a\ </property>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xmlsed -i '17a\ <value>mapreduce_shuffle</value>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xmlsed -i '17a\ <name>yarn.nodemanager.aux-services</name>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xmlsed -i '17a\ <property>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xmlsed -i '17a\ </property>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xmlsed -i '17a\ <value>cp145:8042</value>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xmlsed -i '17a\ <name>yarn.nodemanager.webapp.address</name>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xmlsed -i '17a\ <property>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xmlsed -i '17a\ </property>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xmlsed -i '17a\ <value>cp145:8050</value>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xmlsed -i '17a\ <name>yarn.nodemanager.address</name>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xmlsed -i '17a\ <property>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xmlsed -i '17a\ </property>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xmlsed -i '17a\ <value>cp145:8040</value>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xmlsed -i '17a\ <name>yarn.nodemanager.localizer.address</name>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xmlsed -i '17a\ <property>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xmlsed -i '17a\ </property>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xmlsed -i '17a\ <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler</value>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xmlsed -i '17a\ <name>yarn.resourcemanager.scheduler.class</name>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xmlsed -i '17a\ <property>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xmlsed -i '17a\ </property>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xmlsed -i '17a\ <value>20000</value>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xmlsed -i '17a\ <name>yarn.resourcemanager.connect.retry-interval.ms</name>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xmlsed -i '17a\ <property>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xml
9.完整脚本代码(注意修改主机名)
#! /bin/bash
echo 'auto install hadoop begining....'#global var
hadoop=trueif [ "$hadoop" = true ];thenecho 'hadoop install set true'echo 'setup hadoop 3'tar -zxf /opt/install/hadoop-3.1.3.tar.gz -C /opt/soft/mv /opt/soft/hadoop-3.1.3 /opt/soft/hadoop313sed -i '73a\export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop' /etc/profilesed -i '73a\export HADOOP_LIBEXEC_DIR=$HADOOP_HOME/libexec' /etc/profilesed -i '73a\export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native' /etc/profilesed -i '73a\export HADOOP_INSTALL=$HADOOP_HOME' /etc/profilesed -i '73a\export HADOOP_YARN_HOME=$HADOOP_HOME' /etc/profilesed -i '73a\export HADOOP_HDFS_HOME=$HADOOP_HOME' /etc/profilesed -i '73a\export HADOOP_COMMON_HOME=$HADOOP_HOME' /etc/profilesed -i '73a\export HADOOP_MAPRED_HOME=$HADOOP_HOME' /etc/profilesed -i '73a\export YARN_NODEMANAGER_USER=root' /etc/profilesed -i '73a\export YARN_RESOURCEMANAGER_USER=root' /etc/profilesed -i '73a\export HDFS_ZKFC_USER=root' /etc/profilesed -i '73a\export HDFS_JOURNALNODE_USER=root' /etc/profilesed -i '73a\export HDFS_SECONDARYNAMENODE_USER=root' /etc/profilesed -i '73a\export HDFS_DATANODE_USER=root' /etc/profilesed -i '73a\export HDFS_NAMENODE_USER=root' /etc/profilesed -i '73a\export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$HADOOP_HOME/lib' /etc/profilesed -i '73a\export HADOOP_HOME=/opt/soft/hadoop313' /etc/profilesed -i '73a\#HADOOP_HOME' /etc/profile#hadoop-env.shsed -i '53a\export JAVA_HOME=/opt/soft/jdk180' /opt/soft/hadoop313/etc/hadoop/hadoop-env.sh#core-site.xmlsed -i '19a\ </property>' /opt/soft/hadoop313/etc/hadoop/core-site.xmlsed -i '19a\ <value>*</value>' /opt/soft/hadoop313/etc/hadoop/core-site.xmlsed -i '19a\ <name>hadoop.proxyuser.root.groups</name>' /opt/soft/hadoop313/etc/hadoop/core-site.xmlsed -i '19a\ <property>' /opt/soft/hadoop313/etc/hadoop/core-site.xmlsed -i '19a\ </property>' /opt/soft/hadoop313/etc/hadoop/core-site.xmlsed -i '19a\ <value>*</value>' /opt/soft/hadoop313/etc/hadoop/core-site.xmlsed -i '19a\ <name>hadoop.proxyuser.root.hosts</name>' /opt/soft/hadoop313/etc/hadoop/core-site.xmlsed -i '19a\ <property>' /opt/soft/hadoop313/etc/hadoop/core-site.xmlsed -i '19a\ </property>' /opt/soft/hadoop313/etc/hadoop/core-site.xmlsed -i '19a\ <description>读写队列缓存:128k</description>' /opt/soft/hadoop313/etc/hadoop/core-site.xmlsed -i '19a\ <value>131072</value>' /opt/soft/hadoop313/etc/hadoop/core-site.xmlsed -i '19a\ <name>io.file.buffer.size</name>' /opt/soft/hadoop313/etc/hadoop/core-site.xmlsed -i '19a\ <property>' /opt/soft/hadoop313/etc/hadoop/core-site.xmlsed -i '19a\ </property>' /opt/soft/hadoop313/etc/hadoop/core-site.xmlsed -i '19a\ <value>root</value>' /opt/soft/hadoop313/etc/hadoop/core-site.xmlsed -i '19a\ <name>hadoop.http.staticuser.user</name>' /opt/soft/hadoop313/etc/hadoop/core-site.xmlsed -i '19a\ <property>' /opt/soft/hadoop313/etc/hadoop/core-site.xmlsed -i '19a\ </property>' /opt/soft/hadoop313/etc/hadoop/core-site.xmlsed -i '19a\ <description>namenode上本地的hadoop临时文件夹</description>' /opt/soft/hadoop313/etc/hadoop/core-site.xmlsed -i '19a\ <value>/opt/soft/hadoop313/data</value>' /opt/soft/hadoop313/etc/hadoop/core-site.xmlsed -i '19a\ <name>hadoop.tmp.dir</name>' /opt/soft/hadoop313/etc/hadoop/core-site.xmlsed -i '19a\ <property>' /opt/soft/hadoop313/etc/hadoop/core-site.xmlsed -i '19a\ </property>' /opt/soft/hadoop313/etc/hadoop/core-site.xmlsed -i '19a\ <value>hdfs://cp145:9000</value>' /opt/soft/hadoop313/etc/hadoop/core-site.xmlsed -i '19a\ <name>fs.defaultFS</name>' /opt/soft/hadoop313/etc/hadoop/core-site.xmlsed -i '19a\ <property>' /opt/soft/hadoop313/etc/hadoop/core-site.xml#hdfs-site.xmlsed -i '19a\ </property>' /opt/soft/hadoop313/etc/hadoop/hdfs-site.xmlsed -i '19a\ <description>关闭权限验证</description>' /opt/soft/hadoop313/etc/hadoop/hdfs-site.xmlsed -i '19a\ <value>false</value>' /opt/soft/hadoop313/etc/hadoop/hdfs-site.xmlsed -i '19a\ <name>dfs.permissions.enabled</name>' /opt/soft/hadoop313/etc/hadoop/hdfs-site.xmlsed -i '19a\ <property>' /opt/soft/hadoop313/etc/hadoop/hdfs-site.xmlsed -i '19a\ </property>' /opt/soft/hadoop313/etc/hadoop/hdfs-site.xmlsed -i '19a\ <description>datanode上数据块的物理存储位置目录</description>' /opt/soft/hadoop313/etc/hadoop/hdfs-site.xmlsed -i '19a\ <value>/opt/soft/hadoop313/data/dfs/data</value>' /opt/soft/hadoop313/etc/hadoop/hdfs-site.xmlsed -i '19a\ <name>dfs.datanode.data.dir</name>' /opt/soft/hadoop313/etc/hadoop/hdfs-site.xmlsed -i '19a\ <property>' /opt/soft/hadoop313/etc/hadoop/hdfs-site.xmlsed -i '19a\ </property>' /opt/soft/hadoop313/etc/hadoop/hdfs-site.xmlsed -i '19a\ <description>namenode上存储hdfsq名字空间元数据的目录</description>' /opt/soft/hadoop313/etc/hadoop/hdfs-site.xmlsed -i '19a\ <value>/opt/soft/hadoop313/data/dfs/name</value>' /opt/soft/hadoop313/etc/hadoop/hdfs-site.xmlsed -i '19a\ <name>dfs.namenode.name.dir</name>' /opt/soft/hadoop313/etc/hadoop/hdfs-site.xmlsed -i '19a\ <property>' /opt/soft/hadoop313/etc/hadoop/hdfs-site.xmlsed -i '19a\ </property>' /opt/soft/hadoop313/etc/hadoop/hdfs-site.xmlsed -i '19a\ <description>hadoop中每一个block文件的备份数量</description>' /opt/soft/hadoop313/etc/hadoop/hdfs-site.xmlsed -i '19a\ <value>1</value>' /opt/soft/hadoop313/etc/hadoop/hdfs-site.xmlsed -i '19a\ <name>dfs.replication</name>' /opt/soft/hadoop313/etc/hadoop/hdfs-site.xmlsed -i '19a\ <property>' /opt/soft/hadoop313/etc/hadoop/hdfs-site.xml#mapred-site.xmlsed -i '19a\ </property>' /opt/soft/hadoop313/etc/hadoop/mapred-site.xmlsed -i '19a\ <value>1024</value>' /opt/soft/hadoop313/etc/hadoop/mapred-site.xmlsed -i '19a\ <name>mapreduce.reduce.memory.mb</name>' /opt/soft/hadoop313/etc/hadoop/mapred-site.xmlsed -i '19a\ <property>' /opt/soft/hadoop313/etc/hadoop/mapred-site.xmlsed -i '19a\ </property>' /opt/soft/hadoop313/etc/hadoop/mapred-site.xmlsed -i '19a\ <value>1024</value>' /opt/soft/hadoop313/etc/hadoop/mapred-site.xmlsed -i '19a\ <name>mapreduce.map.memory.mb</name>' /opt/soft/hadoop313/etc/hadoop/mapred-site.xmlsed -i '19a\ <property>' /opt/soft/hadoop313/etc/hadoop/mapred-site.xmlsed -i '19a\ </property>' /opt/soft/hadoop313/etc/hadoop/mapred-site.xmlsed -i '19a\ <value>cp145:19888</value>' /opt/soft/hadoop313/etc/hadoop/mapred-site.xmlsed -i '19a\ <name>mapreduce.jobhistory.webapp.address</name>' /opt/soft/hadoop313/etc/hadoop/mapred-site.xmlsed -i '19a\ <property>' /opt/soft/hadoop313/etc/hadoop/mapred-site.xmlsed -i '19a\ </property>' /opt/soft/hadoop313/etc/hadoop/mapred-site.xmlsed -i '19a\ <value>cp145:10020</value>' /opt/soft/hadoop313/etc/hadoop/mapred-site.xmlsed -i '19a\ <name>mapreduce.jobhistory.address</name>' /opt/soft/hadoop313/etc/hadoop/mapred-site.xmlsed -i '19a\ <property>' /opt/soft/hadoop313/etc/hadoop/mapred-site.xmlsed -i '19a\ </property>' /opt/soft/hadoop313/etc/hadoop/mapred-site.xmlsed -i '19a\ <value>/opt/soft/hadoop313/etc/hadoop:/opt/soft/hadoop313/share/hadoop/common/lib/*:/opt/soft/hadoop313/share/hadoop/common/*:/opt/soft/hadoop313/share/hadoop/hdfs/*:/opt/soft/hadoop313/share/hadoop/hdfs/lib/*:/opt/soft/hadoop313/share/hadoop/mapreduce/*:/opt/soft/hadoop313/share/hadoop/mapreduce/lib/*:/opt/soft/hadoop313/share/hadoop/yarn/*:/opt/soft/hadoop313/share/hadoop/yarn/lib/*</value>' /opt/soft/hadoop313/etc/hadoop/mapred-site.xmlsed -i '19a\ <name>mapreduce.application.classpath</name>' /opt/soft/hadoop313/etc/hadoop/mapred-site.xmlsed -i '19a\ <property>' /opt/soft/hadoop313/etc/hadoop/mapred-site.xmlsed -i '19a\ </property>' /opt/soft/hadoop313/etc/hadoop/mapred-site.xmlsed -i '19a\ <final>true</final>' /opt/soft/hadoop313/etc/hadoop/mapred-site.xmlsed -i '19a\ <description>job执行框架:local,classic or yarn</description>' /opt/soft/hadoop313/etc/hadoop/mapred-site.xmlsed -i '19a\ <value>yarn</value>' /opt/soft/hadoop313/etc/hadoop/mapred-site.xmlsed -i '19a\ <name>mapreduce.framework.name</name>' /opt/soft/hadoop313/etc/hadoop/mapred-site.xmlsed -i '19a\ <property>' /opt/soft/hadoop313/etc/hadoop/mapred-site.xml#yarn-site.xmlsed -i '17a\ </property>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xmlsed -i '17a\ <value>/opt/soft/hadoop313/yarndata/log</value>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xmlsed -i '17a\ <name>yarn.nodemanager.log-dirs</name>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xmlsed -i '17a\ <property>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xmlsed -i '17a\ </property>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xmlsed -i '17a\ <value>/opt/soft/hadoop313/yarndata/yarn</value>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xmlsed -i '17a\ <name>yarn.nodemanager.local-dirs</name>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xmlsed -i '17a\ <property>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xmlsed -i '17a\ </property>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xmlsed -i '17a\ <value>mapreduce_shuffle</value>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xmlsed -i '17a\ <name>yarn.nodemanager.aux-services</name>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xmlsed -i '17a\ <property>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xmlsed -i '17a\ </property>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xmlsed -i '17a\ <value>cp145:8042</value>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xmlsed -i '17a\ <name>yarn.nodemanager.webapp.address</name>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xmlsed -i '17a\ <property>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xmlsed -i '17a\ </property>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xmlsed -i '17a\ <value>cp145:8050</value>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xmlsed -i '17a\ <name>yarn.nodemanager.address</name>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xmlsed -i '17a\ <property>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xmlsed -i '17a\ </property>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xmlsed -i '17a\ <value>cp145:8040</value>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xmlsed -i '17a\ <name>yarn.nodemanager.localizer.address</name>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xmlsed -i '17a\ <property>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xmlsed -i '17a\ </property>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xmlsed -i '17a\ <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler</value>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xmlsed -i '17a\ <name>yarn.resourcemanager.scheduler.class</name>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xmlsed -i '17a\ <property>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xmlsed -i '17a\ </property>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xmlsed -i '17a\ <value>20000</value>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xmlsed -i '17a\ <name>yarn.resourcemanager.connect.retry-interval.ms</name>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xmlsed -i '17a\ <property>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xmlecho 'setup hadoop 3 success!!!'
fi
10.重启环境变量
[root@lxm148 opt]# source /etc/profile
11.初始化
[root@lxm148 yarndata]# hdfs namenode -format
12.启动服务
[root@lxm148 yarndata]# start-all.sh
13.jps查询节点
相关文章:

Shell脚本之——Hadoop3单机版安装
目录 1.解压 2.文件重命名 3.配置环境变量 4.hadoop-env.sh 5.core-site.xml 6. hdfs-site.xml 7. mapred-site.xml 8.yarn-site.xml 9.完整脚本代码(注意修改主机名) 10.重启环境变量 11.初始化 12.启动服务 13.jps查询节点 1.解压 tar -zxf /opt/install/hadoo…...

代码随想录NO39 |0-1背包问题理论基础 416.分割等和子集
0-1背包问题理论基础 分割等和子集1. 0-1背包问题理论基础(二维数组实现)2. 0-1背包问题理论基础 二(一维数组实现)1. 0-1背包问题理论基础(二维数组实现) 背包问题一般分为这几种: 0-1背包问题:有n件物品和一个最多能背重量为w…...

FITC-PEG-FA,荧光素-聚乙二醇-叶酸,FA-PEG-FITC,实验室科研试剂,提供质量检测
FITC-PEG-FA,荧光素-聚乙二醇-叶酸 中文名称:荧光素-聚乙二醇-叶酸 英文名称:FITC-PEG-FA 英文别名:Fluorescein-PEG-Folic Acid 性状:基于不同的分子量,呈白色/类白色固体,或粘稠液体。 溶…...

简洁易懂:源码+实战讲解Redisson并发锁及看门狗自动续期
1 缘起 有一次同事问Redisson存储的键是否为hash? 我当时,没有看Redisson的相关源码,只知道应用, 所以没有办法回答,于是开始看看Redisson实现的源码, 顺便写了一个单机Redisson测试, 发现Redi…...

TCP 三次握手和四次挥手
✏️作者:银河罐头 📋系列专栏:JavaEE 🌲“种一棵树最好的时间是十年前,其次是现在” 目录TCP 建立连接(三次握手)为啥不能是 4 次?为啥不能是 2 次?三次握手的意义:TCP 断开连接(四…...

JavaWeb复习
JavaWeb复习一.概述1.概念2.B/S和C/S 架构二.HTTP通信协议概述1.概念2.HTTP1.0 与 HTTP1.1 版本3.HTTP 协议组成4.常见状态码5.GET 与 POST 请求方式三.Tomcat1.Web服务器介绍2.安装(Windows)3.Tomcat目录结构4.server.xml部分配置解释四.Servlet1.概念2…...

P14 PyTorch AutoGrad
前言:激活函数与loss的梯度PyTorch 提供了Auto Grad 功能,这里系统讲解一下torch.autograd.grad系统的工作原理,了解graph 结构目录:1: require_grad False2: require_grad True3: 多层bakcward 原理4: in…...

前端报表如何实现无预览打印解决方案或静默打印
在前端开发中,除了将数据呈现后,我们往往需要为用户提供,打印,导出等能力,导出是为了存档或是二次分析,而打印则因为很多单据需要打印出来作为主要的单据来进行下一环节的票据支撑, 而前端打印可…...
Operating System Course 2 - My OS
Computer Startup process上一篇:http://t.csdn.cn/XfUKt 讲到这个启动设备的第一个扇区:引导扇区。那么引导扇区的代码长什么样子?这里得看引导扇区代码源文件bootsect.s(.s后缀文件为用汇编语言编写的源代码文件)。另…...

离散数学 课时一 命题逻辑的基本概念
1 命题 1、命题:可以判断其真值的陈述句 2、真值:真或者假(1或者0) 3、真命题:真值为真的命题 4、假命题:真值为假的命题 5、原子命题:不可以再被分解成更简单的命题 6、复合命题:由原子命题通过联结词联结…...

Word文档带有权限密码怎么办?
Word文档的权限密码指的是什么?其实这是Word文档的保护方法之一,具体指Word文档的编辑、修改受到了限制,需要输入密码才能进行。 设置了权限密码的Word文档还是可以直接打开,只有当需要编辑或者修改内容的时候,才会发…...

C++多态
1. 多态的概念1.1 概念多态的概念:通俗来说,就是多种形态,具体点就是去完成某个行为,当不同的对象去完成时会产生出不同的状态举个例子:比如买票这个行为,当普通人买票时,是全价买票;…...
访问学者如何申请美国J1签证?
一、申请美国J1签证的步骤: 第一步:填写I901表。 填写I901表会收取SERVIS费用180美元,可以用VISA/Master卡直接网上支付。填完后打印收据单或者存成PDF后续再打印,记下I901收据编号。 第二步:DS-160表填写。 填写DS-…...

使用gitlab ci/cd来发布一个.net 项目
gitlab runner的安装和基本使用:https://bear-coding.blog.csdn.net/article/details/120591711安装并给项目配置完gitlab runner后再操作后面步骤。实现目标:master分支代码有变更的时候自动构建build。当开发人员在gitlab上给项目打一个tag标签分支的时候自动触发…...
笔试题-2023-蔚来-数字芯片设计【纯净题目版】
回到首页:2023 数字IC设计秋招复盘——数十家公司笔试题、面试实录 推荐内容:数字IC设计学习比较实用的资料推荐 题目背景 笔试时间:2022.08.24应聘岗位:校招-芯片逻辑综合工程师-智能硬件笔试时长:90min笔试平台:nowcoder牛客网题目类型:不定项选择题(15道)、填空题…...

ThreadLocal 详解
ThreadLocal简介JDK源码对ThreadLocal类的注释如下:ThreadLocal提供线程局部变量,使得每个线程都有自己的、独立初始化的变量副本ThreadLocal实例通常是类中的private static字段,用于将状态与线程相关联,如用户ID、事务ID只要线程…...

【Java 面试合集】重写以及重载有什么区别能简单说说嘛
重写以及重载有什么区别能简单说说嘛 前述 这是一道非常基础的面试题,我们在回答的过程中一定要逐一横向比较。 从方法的 修饰符,返回值,方法名,含义,参数等方面进行逐一分析来比较不同。 话不多话,看下…...
到底什么是股票委托接口?
在量化股票市场上,常见的股票委托接口其实有着不一样的交集,就拿股票交易接口,在量化股票跟程序化交易中,有共同之处就是在于直接委托执行下单,并且能很快的就能够将策略输出在账户持仓数据中,继续缓存下来…...
Linux驱动:VPU
1. 前言 限于作者能力水平,本文可能存在谬误,因此而给读者带来的损失,作者不做任何承诺。 2. 概述 VPU 是用来进行图像、视频数据进行硬件编、解码的硬件模块。内部集成了 Encoder、Decoder 功能部件进行图像、视频数据进行硬件编、解码&a…...

简介Servlet
目录 一、maven中心库 二、简介Servlet 三、实现Servlet动态页面 1、创建一个maven项目 2、引入依赖 3、创建目录结构 4、编写Servlet代码 5、打包 6、部署 7、验证程序 四、Servlet的运行原理 五、Tomcat伪代码 1、Tomcat初始化 a、让Tomcat先从指定的目录…...

eNSP-Cloud(实现本地电脑与eNSP内设备之间通信)
说明: 想象一下,你正在用eNSP搭建一个虚拟的网络世界,里面有虚拟的路由器、交换机、电脑(PC)等等。这些设备都在你的电脑里面“运行”,它们之间可以互相通信,就像一个封闭的小王国。 但是&#…...
应用升级/灾备测试时使用guarantee 闪回点迅速回退
1.场景 应用要升级,当升级失败时,数据库回退到升级前. 要测试系统,测试完成后,数据库要回退到测试前。 相对于RMAN恢复需要很长时间, 数据库闪回只需要几分钟。 2.技术实现 数据库设置 2个db_recovery参数 创建guarantee闪回点,不需要开启数据库闪回。…...

python/java环境配置
环境变量放一起 python: 1.首先下载Python Python下载地址:Download Python | Python.org downloads ---windows -- 64 2.安装Python 下面两个,然后自定义,全选 可以把前4个选上 3.环境配置 1)搜高级系统设置 2…...

STM32标准库-DMA直接存储器存取
文章目录 一、DMA1.1简介1.2存储器映像1.3DMA框图1.4DMA基本结构1.5DMA请求1.6数据宽度与对齐1.7数据转运DMA1.8ADC扫描模式DMA 二、数据转运DMA2.1接线图2.2代码2.3相关API 一、DMA 1.1简介 DMA(Direct Memory Access)直接存储器存取 DMA可以提供外设…...
【论文笔记】若干矿井粉尘检测算法概述
总的来说,传统机器学习、传统机器学习与深度学习的结合、LSTM等算法所需要的数据集来源于矿井传感器测量的粉尘浓度,通过建立回归模型来预测未来矿井的粉尘浓度。传统机器学习算法性能易受数据中极端值的影响。YOLO等计算机视觉算法所需要的数据集来源于…...
安卓基础(aar)
重新设置java21的环境,临时设置 $env:JAVA_HOME "D:\Android Studio\jbr" 查看当前环境变量 JAVA_HOME 的值 echo $env:JAVA_HOME 构建ARR文件 ./gradlew :private-lib:assembleRelease 目录是这样的: MyApp/ ├── app/ …...

以光量子为例,详解量子获取方式
光量子技术获取量子比特可在室温下进行。该方式有望通过与名为硅光子学(silicon photonics)的光波导(optical waveguide)芯片制造技术和光纤等光通信技术相结合来实现量子计算机。量子力学中,光既是波又是粒子。光子本…...
【Java学习笔记】BigInteger 和 BigDecimal 类
BigInteger 和 BigDecimal 类 二者共有的常见方法 方法功能add加subtract减multiply乘divide除 注意点:传参类型必须是类对象 一、BigInteger 1. 作用:适合保存比较大的整型数 2. 使用说明 创建BigInteger对象 传入字符串 3. 代码示例 import j…...
Java + Spring Boot + Mybatis 实现批量插入
在 Java 中使用 Spring Boot 和 MyBatis 实现批量插入可以通过以下步骤完成。这里提供两种常用方法:使用 MyBatis 的 <foreach> 标签和批处理模式(ExecutorType.BATCH)。 方法一:使用 XML 的 <foreach> 标签ÿ…...

算法:模拟
1.替换所有的问号 1576. 替换所有的问号 - 力扣(LeetCode) 遍历字符串:通过外层循环逐一检查每个字符。遇到 ? 时处理: 内层循环遍历小写字母(a 到 z)。对每个字母检查是否满足: 与…...