Hive本地开发/学习环境配置

前提

hive依赖hadoop的相关组件，需要启动Hadoop的相关组件。

Hive 版本：3.1.3
Hadoop版本：3.3.4

hive-env.sh

export HADOOP_HOME=$HADOOP_HOME
export HIVE_CONF_DIR=/usr/local/Cellar/hive/3.1.3/libexec/conf
export HIVE_AUX_JARS_PATH=/usr/local/Cellar/hive/3.1.3/libexec/lib

hive-site.xml

<configuration><property><name>javax.jdo.option.ConnectionURL</name><value>jdbc:mysql://localhost:3306/hive?createDatabaseIfNotExist=true&amp;useSSL=false</value><description>JDBC connect string for a JDBC metastore</description></property><property><name>javax.jdo.option.ConnectionDriverName</name><value>com.mysql.jdbc.Driver</value><description>Driver class name for a JDBC metastore</description></property><property><name>javax.jdo.option.ConnectionUserName</name><value>root</value><description>username to use against metastore database</description></property><property><name>javax.jdo.option.ConnectionPassword</name><value>root</value><description>password to use against metastore database</description></property><property><name>hive.cli.print.header</name><value>true</value></property><property><name>hive.cli.print.current.db</name><value>true</value></property><property><name>hive.metastore.schema.verification</name><value>false</value></property><!-- hiveserver2运行绑定的host --><property><name>hive.server2.thrift.bind.host</name><value>localhost</value></property><!-- 远程模式部署metastore 的地址 --><property><name>hive.metastore.uris</name><value>thrift://localhost:9083</value></property><!-- 配置hiveserver2服务启动问题（不配置，则hiveserver无法监听服务，会导致beeline客户端无法连接） --><property><name>hive.metastore.event.db.notification.api.auth</name><value>false</value></property>
</configuration>

hive初始化

需要先启动Hadoop的相关组件服务。
schematool -dbType mysql -initSchema --verbose

注意：一定要先-dbType mysql然后才-initSchema，否则会出现连接MySQL的错误，

Metastore connection URL:	 jdbc:mysql://localhost:3306/hive?createDatabaseIfNotExist=true&useSSL=false
Metastore Connection Driver :	 com.mysql.jdbc.Driver
Metastore connection User:	 root
org.apache.hadoop.hive.metastore.HiveMetaException: Failed to get schema version.
Underlying cause: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException : Communications link failureThe last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
SQL Error code: 0
org.apache.hadoop.hive.metastore.HiveMetaException: Failed to get schema version.at org.apache.hadoop.hive.metastore.tools.HiveSchemaHelper.getConnectionToMetastore(HiveSchemaHelper.java:94)at org.apache.hive.beeline.HiveSchemaTool.getConnectionToMetastore(HiveSchemaTool.java:169)at org.apache.hive.beeline.HiveSchemaTool.testConnectionToMetastore(HiveSchemaTool.java:475)at org.apache.hive.beeline.HiveSchemaTool.doInit(HiveSchemaTool.java:581)at org.apache.hive.beeline.HiveSchemaTool.doInit(HiveSchemaTool.java:567)at org.apache.hive.beeline.HiveSchemaTool.main(HiveSchemaTool.java:1517)at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)at java.lang.reflect.Method.invoke(Method.java:498)at org.apache.hadoop.util.RunJar.run(RunJar.java:323)at org.apache.hadoop.util.RunJar.main(RunJar.java:236)
Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failureThe last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)at java.lang.reflect.Constructor.newInstance(Constructor.java:423)at com.mysql.jdbc.Util.handleNewInstance(Util.java:403)at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:990)at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:335)at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2187)at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2220)at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2015)at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:768)at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47)at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)at java.lang.reflect.Constructor.newInstance(Constructor.java:423)at com.mysql.jdbc.Util.handleNewInstance(Util.java:403)at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:385)at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:323)at java.sql.DriverManager.getConnection(DriverManager.java:664)at java.sql.DriverManager.getConnection(DriverManager.java:247)at org.apache.hadoop.hive.metastore.tools.HiveSchemaHelper.getConnectionToMetastore(HiveSchemaHelper.java:88)... 11 more
Caused by: java.net.ConnectException: Connection refused (Connection refused)at java.net.PlainSocketImpl.socketConnect(Native Method)at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)at java.net.Socket.connect(Socket.java:606)at com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:211)at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:299)... 26 more
*** schemaTool failed ***

metastore服务启动

进入hive的bin目录，执行命令hive --service metastore ,或者使用nohup命令进行后台启动，nohup hive --service metastore &，这样就启动了metastore服务，可以通过jps命令查看，是否存在RunJar的进程，存在则说明启动成功，再通过hive客户端连接即可。

hiveserver2服务启动

该服务依赖于metastore服务，所以，在启动hiveserver2之前，一定确保已经启动了metastore服务，这时，可以通过hive --service hiveserver2命令启动服务，当然同样可以通过nohup命令进行后台启动，nohup hive --service hiveserver2 &，启动后，通过jps命令查看服务进程，此时会多出一个RunJar的进程，即为hiverserver2的服务。

通过beeline客户端进行连接，此时还没有真正连接上。

还需要在当前客户端窗口输入hiveserver2的连接信息，! connect jdbc:hive2://localhost:10000，hive2默认的端口为10000，此时，会提示让你输入用户名名，然后再输入密码。

Error:Could not open client transport with JDBC Uri: jdbc:hive2://localhost:10000/default:java.net.ConnectException refused

出现上述错误

需要在hive-site。添加如下配置

<property><name>hive.metastore.event.db.notification.api.auth</name><value>false</value>
</property>　

然后重新启动相关服务，再使用beeline客户端连接，出现了如下错误：

beeline> !connect jdbc:hive2://localhost:10000

Connecting to jdbc:hive2://localhost:10000

Enter username for jdbc:hive2://localhost:10000: yuxiao

Enter password for jdbc:hive2://localhost:10000:

23/05/01 18:14:01 [main]: WARN jdbc.HiveConnection: Failed to connect to localhost:10000

Error: Could not open client transport with JDBC Uri: jdbc:hive2://localhost:10000: Failed to open new session: java.lang.RuntimeException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException): User: yuxiao is not allowed to impersonate yuxiao (state=08S01,code=0)

这个问题是由于不允许你当前登录的用户访问hadoop，所以需要调整hadoop的core-site.xml的配置，添加相关proxuser的访问即可，

<property><name>hadoop.proxyuser.用户名.hosts</name><value>*</value></property><property><name>hadoop.proxyuser.用户名.groups</name><value>*</value></property>

将配置文件的用户修改上hive不允许访问的用户名，配置文件修改后，重新启动hadoop的dfs服务，

再使用beeline客户端访问，用户名输入之前提示不允许访问的用户名，密码直接回车即可，这样就可以正常访问hive了，如下图所示：

查看数据库：

hive webui配置

hiveserver2服务具有可视化界面，查看相关信息，需要在hive-site.xml文件配置webui，如下所示：

<property><name>hive.server2.webui.host</name><value>127.0.0.1</value>
</property>
<property><name>hive.server2.webui.port</name><value>10002</value>
</property>

在启动hiveserver2服务后，访问10002端口查看webui，如下：