Relationship between yarn working mechanism and HDFS, yarn and MapReduce

MelodyYN 2022-02-13 08:26:39 阅读数:72

relationship yarn working mechanism hdfs

1、Yarn Infrastructure

​ YARN Mainly by ResourceManager、NodeManager、ApplicationMaster and Container Etc . The functions of each component are as follows :

 Insert picture description here

2、Yarn Working mechanism

 Insert picture description here

2.1 The whole process of homework submission is explained in detail

(1) Homework submission

  1. Client call job.waitForCompletion Method , Submit... To the entire cluster MapReduce Homework .
  2. Client towards RM Apply for an assignment id(Application).
  3. RM to Client Return to the job Submit path and job of resource id.
  4. Client Submit jar package 、 Slice information and configuration files to the specified resource submission path .
  5. Client After submitting resources , towards RM Apply to run MrAppMaster.

(2) Job initialization

  1. When RM received Client After request , Will be job Add to capacity scheduler .
  2. Some free NM Take it Job.
  3. The NM establish Container, And produce MRAppmaster.
  4. download Client Commit resources to local .

(3) Task assignment

  1. MrAppMaster towards RM Apply to run multiple MapTask Task resources .
  2. RM Will run MapTask The task is assigned to the other two NodeManager, The other two NodeManager Pick up tasks and create containers .

(4) Task run

  1. MR To receive the task from two NodeManager Send program startup script , these two items. NodeManager To start, respectively, MapTask,MapTask Sort the data partition .
  2. MrAppMaster Wait for all MapTask After running , towards RM Apply for containers , function ReduceTask.
  3. ReduceTask towards MapTask Get the data of the corresponding partition .
  4. After the program runs ,MR Will send to RM Apply to cancel yourself .

2.2 During job submission HDFS and MapReduce

Detailed process explanation :
 Insert picture description here

3、HDFS、Yarn、MapReduce The relationship between the three

 Insert picture description here

copyright:author[MelodyYN],Please bring the original link to reprint, thank you.