Hadoop Optimize

1、HDFS Multi directory

1.1 NameNode Can be configured into multiple local directories , And each directory holds the same content , Thus increasing reliability .

• hdfs-site.xml Add the following

  <property>
  <name>dfs.namenode.name.dir</name>
  <value>file://${hadoop.tmp.dir}/dfs/name1,file://${hadoop.tmp.dir}/dfs/name2</value>
  </property>
  

• Shut down the cluster

• Delete the of each node data and logs Catalog

• Format cluster

  hdfs namenode -format
  

• restart hdfs

  start-dfs.sh
  

• Check hadoop Installation directory /opt/module/hadoop-3.1.3/data/dfs, Will find name1 and name2 And the two contents are the same .

1.2 DataNode It can be configured into multiple directories , Each directory stores different data （ Data is not a copy ）

• hdfs-site.xml Add the following

  <property>
  <name>dfs.datanode.data.dir</name>
  <value>file://${hadoop.tmp.dir}/dfs/data1,file://${hadoop.tmp.dir}/dfs/data2</value>
  </property>
  

• Shut down the cluster

• Delete the of each node data and logs Catalog

• Format cluster

  hdfs namenode -format
  

• restart hdfs

  start-dfs.sh
  

• Check hadoop Installation directory /opt/module/hadoop-3.1.3/data/dfs, After uploading the file, you will find data1 and data2 And the two contents are different .

1.3 Inter disk data balancing for cluster data balancing

Production environment , Due to the lack of hard disk space , You often need to add a hard disk . When the newly loaded hard disk has no data , You can execute the disk data balancing command .（Hadoop3.x New characteristics ）

1. Generate a balanced plan （ We only have one disk , No plan will be generated ）

   hdfs diskbalancer -plan hadoop103
   

2. Execute a balanced plan

   hdfs diskbalancer -execute hadoop103.plan.json
   

3. View the execution of the current balancing task

   hdfs diskbalancer -query hadoop103
   

4. Cancel the balancing task

   hdfs diskbalancer -cancel hadoop103.plan.json
   

2、HDFS Expand and shrink

2.1 White list

Hosts on the white list IP The address can , Used to store data .

1. Create a white list 、 The blacklist

   touch /opt/module/hadoop-3.1.3/etc/hadoop/whitelist
   touch /opt/module/hadoop-3.1.3/etc/hadoop/blacklist
   

   stay whitelist Add the following host name , If the node of the cluster is 102 103

   hadoop102
   hadoop103
   

2. stay hdfs-site.xml Add the following

   <!-- White list -->
   <property>
   <name>dfs.hosts</name>
   <value>/opt/module/hadoop-3.1.3/etc/hadoop/whitelist</value>
   </property>
   <!-- The blacklist -->
   <property>
   <name>dfs.hosts.exclude</name>
   <value>/opt/module/hadoop-3.1.3/etc/hadoop/blacklist</value>
   </property>
   

3. Distribution profile whitelist,blacklist,hdfs-site.xml

4. The first time you add a whitelist, you must restart the cluster , Not for the first time , Just refresh NameNode The node can be

   hdfs dfsadmin -refreshNodes
   

5. View results

6. Modify the white list twice , increase hadoop104 , distribution

7. Refresh NameNode

2.2 Service the new server

Environmental preparation ：

1. stay hadoop100 Clone another one on the host hadoop105 host

2. modify IP Address and host name

3. Copy hadoop102 Of /opt/module Contents and /etc/profile.d/my_env.sh To hadoop105 and source

4. Delete hadoop105 On Hadoop Historical data of ,data and log data

   Otherwise, an error will be reported ： The reason is that the node id The same conflict , This id It's just data Directory . After deleting him , restart 105 Of hdfs Will be based on hdfs-site.xml The configuration information in automatically creates a new data and logs Catalog .

5. To configure hadoop102 and hadoop103 To hadoop105 Of ssh No secret login

6. Direct start DataNode, You can associate to a cluster

7. In the white list whitelist add hadoop104、hadoop105, And restart the cluster

8. Distribute to all nodes , Include hadoop105

9. Refresh NameNode

   hdfs dfsadmin -refreshNodes
   

2.3 Data balance between servers

1. Turn on the data equalization command

   start-balancer.sh -threshold 10
   

   For parameters 10, Represents that the disk space utilization of each node in the cluster does not differ by more than 10%, It can be adjusted according to the actual situation .

2. Stop data balancing command

   stop-balancer.sh
   

   because HDFS You need to start a separate Rebalance Server To execute Rebalance operation , So try not to be in NameNode On the implementation start-balancer.sh, Instead, find a spare machine .

2.4 Blacklist retired servers

Hosts on the blacklist IP The address cannot be , Used to store data .

In the enterprise , Configure the blacklist , Used to retire servers .

1. edit /opt/module/hadoop-3.1.3/etc/hadoop In the catalog blacklist file , Add ready to retire hadoop105

2. distribution blacklist

3. The first time you add a blacklist, you must restart the cluster , Not for the first time , Just refresh NameNode The node can be

4. Check Web browser , The state of the decommissioned node is decommission in progress（ The retired ）, Indicates that the data node is copying blocks to other nodes

   

5. Wait for the status of the decommissioned node to be decommissioned（ All the blocks have been copied ）, Stop the node and the node explorer .

   Be careful ： If the copy number is 3, The nodes in service are less than or equal to 3, You can't retire successfully , You need to modify the number of copies before you can retire .

   

6. If the data is unbalanced , Cluster rebalancing can be achieved by command .

   start-balancer.sh -threshold 10