3. HDFS upload and download process

Code sheep 2022-02-13 07:07:45 阅读数:579

hdfs upload download process

3.HDFS Upload | Download process

1.HDFS Basic knowledge of
2.HDFS operation
3.HDFS Upload | Download process
 Insert picture description here

3.1 Working mechanism – Roles and responsibilities

The main character :NameNode( The core ) Is access HDFS The only access
Maintain and manage file system metadata ( Namespace directory tree structure 、 Location information of files and blocks 、 Access rights and other information

 Insert picture description here

From the role :DataNode
Responsible for specific data block storage , Its number determines HDFS The overall data storage capacity of the cluster .
Primary role, secondary role :SecondaryNameNode

 Insert picture description here

3.2 Working mechanism – Upload file process ( Write file process )
What's up? pipeline The Conduit ( transport )

Pipeline The Conduit , yes HDFS A data transmission method used in the process of uploading files and writing data
The client writes the data block to the first data node , After the first data node saves the data, the block is copied to the second data node , After the latter is saved, it is copied to the third data node

pipeline The benefits of transmission

Sequential delivery can make full use of the bandwidth of each machine , Avoid network bottlenecks and high latency connections , Minimize the delay of pushing all data , All data copies can be completed simultaneously in a short time

 Insert picture description here

The core concept –ACK Respond, respond ( Ensure data transmission security 、 No loss
ACK (Acknowledge character) It's the confirmation character , In data communication , A transmission type control character sent by the receiver to the sender . Indicates that the data sent has been confirmed to be received correctly
stay HDFS pipeline In the process of pipeline data transmission , The opposite direction of transmission will be ACK check , Ensure data transmission security
️ Expand your knowledge : Rack perception , The operation and maintenance personnel write paths to the equipment through scripts to determine the location of each other

 Insert picture description here

 give an example : Upload a 300MB The file of
With 128MB Divide into pieces , Can be divided into 3 block (128MB、128MB、44MB)
#1、 Client to NameNode Send data upload request , An important information contained in this request is the length of the file .300MB
The process :HDFS Client creation FileSystem Instance object of DistributedFileSystem
(`FileSystem Encapsulates the methods related to file system operation `)
And then call DFS In the object create() Method , adopt -RPC request -NameNode(NN) create a file
#2、NameNode The client will do a series of checks after receiving the request 
The process :NN Perform various checks and judgments
a) Does the file exist , If there is , Report errors
b) Whether the parent directory of the uploaded file exists , If it doesn't exist , Report errors
c) Whether the client has the permission to create the file and other contents
#3、 Inspection passed NN A record will be recorded for this request , return FSDataOutputStream The output stream object is used to write data to the client 'NameNode After passing the check, it will return... To the client - Store node information and block information -')
Storage node information : Where to save ?( Default 3, return ABC3 Node information )
DataNode Copy backup strategy (`3 Replica storage policy - Nearby principle -`) as follows :( Through rack awareness )
(1) Consider network transmission efficiency , The first priority is the one closest to the client dataNode( Later, it is called dataNode by dataNodeA, The following analogy )
(2) Efficiency and reliability , the second DataNode Will consider a different rack from the first one dataNode
(3) The first 3 individual dataNode Usually in the second DataNode Pick another node on your rack
Block information : save n block ( nX3=3, On data )
blk_1 Before storage 128M The data of , 0-128
blk_2 After storage 128M The data of , 128-256
blk_3 Store the remaining 44M data ,256-300
(' Physical chunks : Real segmentation , take 200M It's divided into 128M+72M. The two blocks are stored separately ')

 Insert picture description here

' Start transfer block '#4、 The client receives the third step NameNode After the build response of pipeline The Conduit 
Client request 3 platform DN One of them A Upload data ( It's essentially a RPC call , establish pipeline),A Upon receipt of the request, the call continues B, then B call C, Will the whole pipeline Build complete , After completion, a response will be returned to the client in reverse order

 Insert picture description here

#5、 According to the id Upload files in sequence and 【 Physical chunks 】
Start real file upload , Physical segmentation of data while uploading ( cut 2 Time )
Split in blocks for the first time , read out 0-128M As the first block block(' Through cursor seek Locate read data ')
The second time, the block is divided into data packets packet( Default 64K) And generate an internal data queue data queue( Reply queue waiting for reply , Sort )

 Insert picture description here

#6、 Client pass FSDataOutputStream The output stream starts uploading data 
When the first stage DataNode received packet after , First, it will be put into memory , Then save a copy of the data in memory to the local , A copy is passed on to the next person connected to him datanode(A->B->C), Complete the transfer of DN wait for ACK Information (` Data protection `)
(' When one block Once the transmission is complete , The client calls again pipeline The Conduit , For the second block Conduct 【 Physical chunks 】, request namenode Upload the second block Of packet, And pass it to each node in turn . That is, repetition 4-5-6 step ')
(` Data protection `` In the process OutputStream Also maintains an internal packet queue to wait DataNode Receipt confirmation of receipt , It's called the confirmation queue (ack queue), Only received pipeline All in DataNode After confirming the information , The packet will be removed from the confirmation queue `
#7、 After the client completes data writing , stay FSDataOutputStream Call... On the output stream close() Method shut down 
#8、DFS contact NameNode Tell them that the file is written , wait for NameNode confirm 

 Insert picture description here

# reflection : In the process of uploading , What if some nodes fail to upload ?
In the process of file upload, if the block upload of one node fails , Then a retry will be made immediately , If the retry fails , The failed node will be removed from pipeline Middle elimination , And report the failed node to NameNode
️ During transmission , because NameNode It is known that the file consists of those blocks
(`DataStream Request to allocate data block `)
Therefore, just wait for the smallest copy block to return successfully ACK Information
The minimum copy is determined by the parameter dfs.namenode.replication.min Appoint , The default is 1
(` It means as long as there is 1 Copies can be uploaded successfully , Because the remaining copies can be copied according to the first copy `)
If 3 Both nodes failed , Will return to NameNode apply 3 Nodes , Rebuild pipeline
Ensure that at least one copy is uploaded successfully , Other replicas will be replicated asynchronously when the cluster is idle
# reflection : Why is the upload speed of small files slow , Shouldn't it be smaller and faster ?
In the process of uploading , There are many steps (1-2-3-4 and 7-8) Running process and file size ( And steps 5-6 The number of repetitions is related to ) irrelevant
for example , Go to the vegetable market to buy vegetables from home , The time spent has nothing to do with the price of the dish , The price of the dish depends on how many comparisons are made
3.3 Working mechanism – Download file process ( Read file process )
# Process steps :
1、HDFS Client creation FileSystem Instance object of DistributedFileSystem(`FileSystem Encapsulates the methods related to file system operation `)
2、 call DFS Object's open() Method to open the file you want to read
3、 Determine block location , Find the nearest block of data ----DFS Use RPC call namenode To determine the block location of the first few blocks in the file ( Read in batches ) Information (` Applied to rack sensing , Find the copy close to the client -- Distance in the network topology , Refers to the number of switches that communicate with each other `) For each piece ,namenode Returns a with all copies of the block datanode Location address list , And the address list is sorted , Those close to the network topology of the client are ranked first
4、DFS take FSDataInputStream The input stream is returned to the client for reading
5、 The client is in FSDataInputStream Call... On the input stream read() Method . then , Stored DataNode Address of the InputStream Connect to the nearest... Of the first block in the file DataNode. Data from DataNode Back to the client , As a result, the client can call... Repeatedly on the stream read()
6、 When the block ends ,InputStream Close with DataNode The connection of , Then look for the best of the next block datanode. These operations are transparent to the user . So the user feels that it has been reading a continuous stream
(' Go back to -- Read -- Piece of data , In the local ( client ) Splicing , When the client reads data from the stream , Will also ask as needed NameNode To retrieve the next batch of data blocks DataNode Location information ')
# To ensure data integrity , Need to verify (CRC check )
7、 Once the client has finished reading , That's right FSDataInputStream call close() Method
3.4 Working mechanism –NN and DN Communication mechanism between
 The communication mechanism is the heartbeat mechanism between the master and the slave (` Newspaper activity , confirm DN Whether there is `), Data block reporting mechanism (` Confirm the data status `# heartbeat :
DataNode On a regular basis (`dfs.heartbeat.interval Configuration Item Configuration , The default is 3 second `) towards NameNode Send a heartbeat , If NameNode I haven't received... For a long time DataNode The heartbeat sent , NameNode Think it's time to DataNode invalid
# Data block reporting mechanism :
DataNode Regularly to NameNode Report the data block information held by yourself , The time interval of the report is taken as a parameter dfs.blockreport.intervalMsec, If the parameter is not configured, the default value is 6 Hours
Every time it starts ,DN You need to NN Register yourself and report the data block information you hold (' I've dealt with NN In my duties , Not persistent in every file , Where each block is located DN Location information for '
copyright:author[Code sheep],Please bring the original link to reprint, thank you. https://en.javamana.com/2022/02/202202130707407762.html