Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-45498

Followup: Ignore task completion from old stage after retrying indeterminate stages

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    Description

      With SPARK-45182, we added a fix for not letting laggard tasks of the older attempts of the indeterminate stage from marking the partition has completed in map output tracker.

      When a task completes, DAG scheduler also notifies all the tasksets of the stage about that partition being completed. Tasksets would not schedule such task if they are not already scheduled. This is not correct for indeterminate stage, since we want to re-run all the tasks on re-attempt

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            mayurb31 Mayur Bhosale
            mayurb31 Mayur Bhosale
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment