The most detailed and comprehensive hive installation document in the history of the whole network. Xiaobai can also install it by hand (recommended Collection)

Big data learning monk 2022-02-13 07:18:33 阅读数:703

detailed comprehensive hive installation document

Detailed and comprehensive Hive Installation process and environment configuration documents , Welcome to collect

1. Environmental preparation

Be careful : The following steps are in root Permission to operate

1.1 Virtual machine cluster preparation

Clone three virtual machines (linux01、linux02、linux03),
Modify the network configuration of the virtual machine

vi /etc/sysconfig/network-scripts/ifcfg-eth0

Configure the network of the corresponding host IP

vi /etc/udev/rules.d/70-persistent-net.rules

Permanently change the host name

 vi /etc/sysconfig/network

Configure domain mapping

vi /etc/hosts
[[email protected] /]# cat /etc/hosts
192.168.133.3 linux01
192.168.133.4 linux02
192.168.133.5 linux03
192.168.133.6 linux04
192.168.133.7 linux05

Turn off firewall
service iptables stop

View firewall status
service iptables status

iptables: Firewall is not running.

1)linux01、linux02、linux03 The corresponding memory of the host is :10G、2G、2G
2) stay linux01 Of /opt Create... On the directory apps and software Catalog

1.2 SSH Password free login

To configure linux01 Yes linux01、linux02、linux03 Secret free login of three hosts .
(1) Generate public and private keys :

[[email protected] .ssh]$ ssh-keygen -t rsa

Then knock ( Three carriage returns ), Two files will be generated id_rsa( Private key )、id_rsa.pub( Public key )
(2) Copy the public key to the target machine for password free login

[[email protected] .ssh]$ ssh-copy-id linux01
[[email protected] .ssh]$ ssh-copy-id linux02
[[email protected] .ssh]$ ssh-copy-id linux03

1.3 install JDK( Three stations )

1) stay linux01 Of /opt Create under directory apps and software Folder

[[email protected] opt]# mkdir apps
[[email protected] opt]# mkdir software

2) use SecureCRT take jdk-8u144-linux-x64.tar.gz Import to linux01 Of /opt/software Under the table of contents
3) stay Linux Under the system opt Check the directory to see if the package was imported successfully

[[email protected] software]$ ls
jdk-8u144-linux-x64.tar.gz

4) decompression JDK To /opt/apps Under the table of contents

[[email protected] software]$ tar -zxvf jdk-8u144-linux-x64.tar.gz -C /opt/apps/

5) To configure JDK environment variable
(1) First get JDK route

[[email protected] jdk1.8.0_144]$ pwd
/opt/apps/jdk1.8.0_144
(2) open /etc/profile file
[[email protected] software]$ vi /etc/profile

stay profile Add at the end of the file JDK route

#JAVA_HOME
export JAVA_HOME=/opt/apps/jdk1.8.0_144
export PATH=$PATH:$JAVA_HOME/bin

(3) Exit after saving

:wq

(4) Let the modified file take effect

[[email protected] jdk1.8.0_144]$ source /etc/profile

6) test JDK Is the installation successful

[[email protected] jdk1.8.0_144]# java -version
java version "1.8.0_144"

7) take linux01 Medium JDK And environment variables are distributed to linux02、linux03 Two hosts

[[email protected] opt]# xsync /opt/apps/
[[email protected] opt]# xsync /etc/profile

Respectively in linux02、linux03 On source once

[[email protected] ~]$ source /etc/profile
[[email protected] ~]# source /etc/profile

1.4 close SELINUX

Enhanced security Linux(Security-Enhanced Linux) abbreviation SELinux, It's a Linux
The kernel module , It's also Linux A security subsystem of .

SELinux Its structure and configuration are very complicated , So in order to avoid all kinds of mistakes , Proposed closure , There are two closing methods :
(1) Temporarily Closed :

[[email protected] ~]# setenforce 0

But this method is only effective for the current startup , It will fail after restarting the machine , The second method is recommended .
(2) Permanent ban
Modify the configuration file /etc/selinux/config

[[email protected] ~]# vim /etc/selinux/config
vi /etc/selinux/config
take SELINUX=enforcing Change it to SELINUX=disabled
SELINUX=disabled

(3) Sync /etc/selinux/config The configuration file

[[email protected] ~]# xsync /etc/selinux/config

(4) restart linux01、linux02、linux03 host

[[email protected] ~]# reboot
[[email protected] ~]# reboot
[[email protected] ~]# reboot

1.5 Hadoop Cluster configuration , And start the hdfs colony

hadoop See this address for cluster configuration : The most detailed and comprehensive Hadoop colony |HDFS Cluster installation documentation

1.6 install Mysql service

mysql Service detailed installation address : Detailed and comprehensive Linux Lower installation mysql Client server

After installation, you need to open the remote connection permission

1.6.1 First , stay mysql Installed on the machine , use mysql Command line client login

mysql -uroot -p The password you set

1.6.2 stay mysql In the command line interactive interface , Enter the following command

 mysql > grant all privileges on *.* to 'root'@'%' identified by '123456' with grant option;
mysql > flush privileges;

2 Hive install

2.1 Upload Hive Installation package

rz apache-hive-2.3.5.tar.gz

2.2 decompression hive Install the package to the specified directory

tar -zxf apache-hive-2.3.5.tar.gz -C /opt/apps/

2.3 modify hive The configuration file

2.3.1 modify hive-env.sh

go in conf Under the table of contents

cd /opt/bigdata/hive-2.3.5/conf

copy One copy hive-env.sh.template Rename it to hive-env.sh

cp hive-env.sh.template hive-env.sh

modify hive-env.sh

vi hive-env.sh

add to hadoop Address and hive conf Address

# Set HADOOP_HOME to point to a specific hadoop install directory
HADOOP_HOME=/usr/apps/hadoop-2.8.5/
# Hive Configuration Directory can be controlled by:
export HIVE_CONF_DIR=/opt/bigdata/hive-2.3.5/conf/

2.3.2 newly build hive-site.xml

<configuration>
<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:mysql://linux01:3306/hive?createDatabaseIfNotExist=true&amp;useSSL=false</value>
</property>
<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>com.mysql.jdbc.Driver</value>
</property>
<property>
<name>javax.jdo.option.ConnectionUserName</name>
<value>root</value>
</property>
<property>
<name>javax.jdo.option.ConnectionPassword</name>
<value> Their own mysql password </value>
</property>
<property>
<name>hive.metastore.warehouse.dir</name>
<value>/user/hive/warehouse</value>
</property>
<property>
<name>hive.exec.scratchdir</name>
<value>/user/hive/tmp</value>
</property>
<property>
<name>hive.querylog.location</name>
<value>/user/hive/log</value>
</property>
<property>
<name>hive.server2.thrift.port</name>
<value>10000</value>
</property>
<property>
<name>hive.server2.thrift.bind.host</name>
<value>0.0.0.0</value>
</property>
<property>
<name>hive.server2.webui.host</name>
<value>0.0.0.0</value>
</property>
<property>
<name>hive.server2.webui.port</name>
<value>10002</value>
</property>
<property>
<name>hive.server2.long.polling.timeout</name>
<value>5000</value>
</property>
<property>
<name>hive.server2.enable.doAs</name>
<value>true</value>
</property>
<property>
<name>datanucleus.autoCreateSchema </name>
<value>false</value>
</property>
<property>
<name>datanucleus.fixedDatastore </name>
<value>true</value>
</property>
<property>
<name>hive.execution.engine</name>
<value>mr</value>
</property>
<property>
<name>hive.zookeeper.quorum</name>
<value>linux01,linux02,linux03</value>
</property>
</configuration>

2.4 Need to upload mysql The driver jar Package to hive Of lib Next

 link :https://pan.baidu.com/s/1sFU2zf_4qQBeZ2OATjV3KQ
Extraction code :u3ch

2.5 Configure environment variables

In order to operate hive Convenient command
The root directory Run the code , take hadoop Configure the location into the environment variable

vi /etc/profile
export JAVA_HOME=//opt/bigdata/jdk1.8.0_141
export HADOOP_HOME=/opt/bigdata/hadoop-2.8.5
export HIVE_HOME=/opt/bigdata/hive-2.3.5
export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$HIVE_HOME/bin

2.6 initialization hive Metadata information to mysql

${HIVE_HOME}/bin/ schematool -initSchema -dbType mysql

2.7 server2 When connecting remotely, you need to modify hadoop User operation permissions of the configuration file

cd /opt/bigdata/hadoop-2.8.5/etc/hadoop/
vi core-site.xml
<property>
<name>dfs.permissions.enabled</name>
<value>false</value>
</property>
<property>
<name>hadoop.proxyuser.root.hosts</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.root.groups</name>
<value>*</value>
</property>
copyright:author[Big data learning monk],Please bring the original link to reprint, thank you. https://en.javamana.com/2022/02/202202130718307420.html