{"id":77,"date":"2024-01-27T10:13:41","date_gmt":"2024-01-27T10:13:41","guid":{"rendered":"https:\/\/oceanwp.org\/travel\/metus-vitae-pharetra-auctor-copy\/"},"modified":"2024-01-27T10:24:56","modified_gmt":"2024-01-27T10:24:56","slug":"interdum-magna-augue-eget","status":"publish","type":"post","link":"https:\/\/techgurudehradun.in\/blog\/?p=77","title":{"rendered":"In Apache Spark Worker nodes and Executors are same or different?"},"content":{"rendered":"

In the context of Apache Spark, worker nodes and executors are not the same; they are different components of a Spark cluster.<\/p>\n

    \n
  1. Worker Nodes:<\/strong><\/li>\n<\/ol>\n
      \n
    • Worker nodes are also known as slave nodes in a Spark cluster.<\/li>\n
    • These nodes are responsible for running the tasks assigned by the Spark driver program.<\/li>\n
    • Worker nodes manage the resources (CPU, memory, etc.) and execute the tasks on behalf of the driver.<\/li>\n
    • They can be part of a cluster managed by a cluster manager such as Apache Mesos, Hadoop YARN, or Spark’s built-in standalone cluster manager.<\/li>\n<\/ul>\n
        \n
      1. Executors:<\/strong><\/li>\n<\/ol>\n
          \n
        • Executors are a specific component of a worker node. Each worker node can have multiple executors.<\/li>\n
        • Executors are responsible for executing the tasks within a Spark application. They run in separate JVMs.<\/li>\n
        • Executors are created and managed by the Spark cluster manager (e.g., YARN, Mesos, or Spark’s standalone cluster manager) and are allocated resources from the worker nodes.<\/li>\n
        • Multiple executors can run on a single worker node, and they are used to parallelize the processing of tasks within a Spark application.<\/li>\n<\/ul>\n

          Working Process :<\/strong><\/p>\n

            \n
          1. Let\u2019s say a user submits a job using \u201cspark-submit\u201d.<\/li>\n
          2. \u201cspark-submit\u201d will in-turn launch the Driver which will execute the main() method of our code.<\/li>\n
          3. Driver contacts the cluster manager and requests for resources to launch the Executors.<\/li>\n
          4. The cluster manager launches the Executors on behalf of the Driver.<\/li>\n
          5. Once the Executors are launched, they establish a direct connection with the Driver.<\/li>\n
          6. The driver determines the total number of Tasks by checking the Lineage.<\/li>\n
          7. The driver creates the Logical and Physical Plan.<\/li>\n
          8. Once the Physical Plan is generated, Spark allocates the Tasks to the Executors.<\/li>\n
          9. Task runs on Executor and each Task upon completion returns the result to the Driver.<\/li>\n
          10. Finally, when all Task is completed, the main() method running in the Driver exits, i.e. main() method invokes sparkContext.stop().<\/li>\n
          11. Finally, Spark releases all the resources from the Cluster Manager.<\/li>\n<\/ol>\n","protected":false},"excerpt":{"rendered":"

            In the context of Apache Spark, worker nodes and executors are not the same; they are different components of a Spark cluster. Worker Nodes: Worker nodes are also known as slave nodes in a Spark cluster. These nodes are responsible for running the tasks assigned by the Spark driver program. Worker nodes manage the resources […]<\/p>\n","protected":false},"author":1,"featured_media":361,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[13,3],"tags":[12,11],"_links":{"self":[{"href":"https:\/\/techgurudehradun.in\/blog\/index.php?rest_route=\/wp\/v2\/posts\/77"}],"collection":[{"href":"https:\/\/techgurudehradun.in\/blog\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/techgurudehradun.in\/blog\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/techgurudehradun.in\/blog\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/techgurudehradun.in\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=77"}],"version-history":[{"count":4,"href":"https:\/\/techgurudehradun.in\/blog\/index.php?rest_route=\/wp\/v2\/posts\/77\/revisions"}],"predecessor-version":[{"id":359,"href":"https:\/\/techgurudehradun.in\/blog\/index.php?rest_route=\/wp\/v2\/posts\/77\/revisions\/359"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/techgurudehradun.in\/blog\/index.php?rest_route=\/wp\/v2\/media\/361"}],"wp:attachment":[{"href":"https:\/\/techgurudehradun.in\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=77"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/techgurudehradun.in\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=77"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/techgurudehradun.in\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=77"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}