MelodyYN 2022-02-13 08:26:33 阅读数:252
hdfs-site.xml Add the following
<property>
<name>dfs.namenode.name.dir</name>
<value>file://${hadoop.tmp.dir}/dfs/name1,file://${hadoop.tmp.dir}/dfs/name2</value>
</property>
Shut down the cluster
Delete the of each node data and logs Catalog
Format cluster
hdfs namenode -format
restart hdfs
start-dfs.sh
Check hadoop Installation directory /opt/module/hadoop-3.1.3/data/dfs, Will find name1 and name2 And the two contents are the same .
hdfs-site.xml Add the following
<property>
<name>dfs.datanode.data.dir</name>
<value>file://${hadoop.tmp.dir}/dfs/data1,file://${hadoop.tmp.dir}/dfs/data2</value>
</property>
Shut down the cluster
Delete the of each node data and logs Catalog
Format cluster
hdfs namenode -format
restart hdfs
start-dfs.sh
Check hadoop Installation directory /opt/module/hadoop-3.1.3/data/dfs, After uploading the file, you will find data1 and data2 And the two contents are different .
Production environment , Due to the lack of hard disk space , You often need to add a hard disk . When the newly loaded hard disk has no data , You can execute the disk data balancing command .(Hadoop3.x New characteristics )
Generate a balanced plan ( We only have one disk , No plan will be generated )
hdfs diskbalancer -plan hadoop103
Execute a balanced plan
hdfs diskbalancer -execute hadoop103.plan.json
View the execution of the current balancing task
hdfs diskbalancer -query hadoop103
Cancel the balancing task
hdfs diskbalancer -cancel hadoop103.plan.json
Hosts on the white list IP The address can , Used to store data .
Create a white list 、 The blacklist
touch /opt/module/hadoop-3.1.3/etc/hadoop/whitelist
touch /opt/module/hadoop-3.1.3/etc/hadoop/blacklist
stay whitelist Add the following host name , If the node of the cluster is 102 103
hadoop102
hadoop103
stay hdfs-site.xml Add the following
<!-- White list -->
<property>
<name>dfs.hosts</name>
<value>/opt/module/hadoop-3.1.3/etc/hadoop/whitelist</value>
</property>
<!-- The blacklist -->
<property>
<name>dfs.hosts.exclude</name>
<value>/opt/module/hadoop-3.1.3/etc/hadoop/blacklist</value>
</property>
Distribution profile whitelist,blacklist,hdfs-site.xml
The first time you add a whitelist, you must restart the cluster , Not for the first time , Just refresh NameNode The node can be
hdfs dfsadmin -refreshNodes
View results
Modify the white list twice , increase hadoop104 , distribution
Refresh NameNode
Environmental preparation :
stay hadoop100 Clone another one on the host hadoop105 host
modify IP Address and host name
Copy hadoop102 Of /opt/module Contents and /etc/profile.d/my_env.sh To hadoop105 and source
Delete hadoop105 On Hadoop Historical data of ,data and log data
Otherwise, an error will be reported : The reason is that the node id The same conflict , This id It's just data Directory . After deleting him , restart 105 Of hdfs Will be based on hdfs-site.xml The configuration information in automatically creates a new data and logs Catalog .
To configure hadoop102 and hadoop103 To hadoop105 Of ssh No secret login
Direct start DataNode, You can associate to a cluster
In the white list whitelist add hadoop104、hadoop105, And restart the cluster
Distribute to all nodes , Include hadoop105
Refresh NameNode
hdfs dfsadmin -refreshNodes
Turn on the data equalization command
start-balancer.sh -threshold 10
For parameters 10, Represents that the disk space utilization of each node in the cluster does not differ by more than 10%, It can be adjusted according to the actual situation .
Stop data balancing command
stop-balancer.sh
because HDFS You need to start a separate Rebalance Server To execute Rebalance operation , So try not to be in NameNode On the implementation start-balancer.sh, Instead, find a spare machine .
Hosts on the blacklist IP The address cannot be , Used to store data .
In the enterprise , Configure the blacklist , Used to retire servers .
edit /opt/module/hadoop-3.1.3/etc/hadoop In the catalog blacklist file , Add ready to retire hadoop105
distribution blacklist
The first time you add a blacklist, you must restart the cluster , Not for the first time , Just refresh NameNode The node can be
Check Web browser , The state of the decommissioned node is decommission in progress( The retired ), Indicates that the data node is copying blocks to other nodes
Wait for the status of the decommissioned node to be decommissioned( All the blocks have been copied ), Stop the node and the node explorer .
Be careful : If the copy number is 3, The nodes in service are less than or equal to 3, You can't retire successfully , You need to modify the number of copies before you can retire .
If the data is unbalanced , Cluster rebalancing can be achieved by command .
start-balancer.sh -threshold 10
copyright:author[MelodyYN],Please bring the original link to reprint, thank you. https://en.javamana.com/2022/02/202202130826314813.html