Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 2.8.0, 3.0.0-alpha1
    • scheduler
    • None
    • Reviewed

    Description

      Implement a FairOrderingPolicy which prefers to allocate to SchedulerProcesses with least current usage, very similar to the FairScheduler's FairSharePolicy.

      The Policy will offer allocations to applications in a queue in order of least resources used, and preempt applications in reverse order (from most resources used). This will include conditional support for sizeBasedWeight style adjustment

      Optionally, based on a conditional configuration to enable sizeBasedWeight (default false), an adjustment to boost larger applications (to offset the natural preference for smaller applications) will adjust the resource usage value based on demand, dividing it by the below value:

      Math.log1p(app memory demand) / Math.log(2);

      In cases where the above is indeterminate (two applications are equal after this comparison), behavior falls back to comparison based on the application id, which is generally lexically FIFO for that comparison

      Attachments

        1. YARN-3319.13.patch
          9 kB
          Craig Welch
        2. YARN-3319.14.patch
          7 kB
          Craig Welch
        3. YARN-3319.17.patch
          10 kB
          Craig Welch
        4. YARN-3319.35.patch
          18 kB
          Craig Welch
        5. YARN-3319.39.patch
          18 kB
          Craig Welch
        6. YARN-3319.45.patch
          10 kB
          Craig Welch
        7. YARN-3319.47.patch
          10 kB
          Craig Welch
        8. YARN-3319.53.patch
          10 kB
          Craig Welch
        9. YARN-3319.58.patch
          13 kB
          Craig Welch
        10. YARN-3319.70.patch
          20 kB
          Craig Welch
        11. YARN-3319.71.patch
          22 kB
          Craig Welch
        12. YARN-3319.72.patch
          22 kB
          Craig Welch
        13. YARN-3319.73.patch
          22 kB
          Craig Welch
        14. YARN-3319.74.patch
          23 kB
          Craig Welch
        15. YARN-3319.75.patch
          23 kB
          Craig Welch

        Issue Links

          Activity

            People

              cwelch Craig Welch
              cwelch Craig Welch
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: