Replace DispatchQueue and BookingQueue with HealthyThreadPool #1035
Conversation
Sorting jobs only by priority causes a situation where low priority jobs can get starved by a constant flow of high priority jobs. The new formula adds a modifier to the sorting rank to take into account the number of cores the job is requesting and also the number of days the job is waiting on the queue. Priorities numbers over 200 will mostly override the formula and work as a priority only based scheduling. sort = priority + (100 * (1 - (job.cores/job.int_min_cores))) + (age in days) Besides that, also take layer_int_cores_min into account when filtering folder_resourse limitations to avoid allocating more cores than the folder limits. (cherry picked from commit 566411aeeddc60983a30eabe121fd03263d05525)
This reverts commit 2eb4936
Queues will not inherit from ThreadPoolExecutor, instead they will manage an instance of HealthThreadPool, which is a threadPoolExecutor that handles healthChecks, termination and repeated tasks. With this the Booking queue should be able to self-heal when locked threads happen.
|
I still have some merge issues causing test errors to work on. |
Use a guava cache to store only the last version of a HostReport per host.
|
I'd like to clarify three things. Handle Repeated TasksThis PR will handle repeated Host-reports by always keeping the latest version. The root cause of this situation is the host-report-queue contention. Quoting myself from #1008,
With the default settings, if Cuebot is connected from a several 100 to 1,000 RQDs,
So the solution is obvious, prevent host-report-queue contention at all.
I posted the example settings in #1008. If it is still not enough, the system needs an extra Cuebot instance to handle the number of RQDs. Eventually PostgreSQL would be the bottleneck. In that case, the system needs to adjust the RQD host-report interval to reduce the number of host-reports. Healthy checkHandling repeated task and healthy check are not able to fix the root cause, queue contention. It might be able to reduce some of repeated host-reports, or discard the filled queue. But it is temporary, very short time. If the host-report-queue filled up one time, it will always happen in the system. Because of the combination - the number of host-report-queue threads, the HikariCP PostgreSQL connection, and the number of RQDs. Could you try #1008 with the example settings? locked threads?Could you elaborate on what is the
|
This solution is only "obvious" if we consider tasks will always finish in a timely fashion, which was not the case for our situation (See root cause explained bellow)
Just to be clear, I don't think this PR show be used in favor of #1008. IMW both are complimentary. We're currently running more than 5k rqds with this configuration:
My bad for mentioning locked threads without a context. There was a condition on the server->rqd communication where threads would get locked waiting for a grpc response without a timeout and it got fixed by #994. The changes on this PR, besides reorganizing the threadpools, worked as a temporary fix while we searched for the root cause, which wasn't contention on our case as the number of threads and Hiraki limits have already been tuned to our needs, but the mentioned starving condition. The proposal of this PR is to not only have a "self-healing" threadpool as a protection for future starving conditions, but also reorganize the threadpools to facilitate maintenance. |
I'm very happy to hear that! That is pretty much the same as ours. 5 Cuebots + 5.5K RQDs, and counting. |
|
@bcipriano Can I get your review on this? |
cuebot/src/main/java/com/imageworks/spcue/dispatcher/HostReportQueue.java
Outdated
Show resolved
Hide resolved
cuebot/src/main/java/com/imageworks/spcue/dispatcher/HealthyThreadPool.java
Show resolved
Hide resolved
31945a4
to
c966d73
Compare
Test doesn't make sense with the new threadpool and will also cause problems whenever an user changes a config property.
|
Alright, unit tests are passing now. @bcipriano Approval still missing. |
|
@bcipriano Can you we consider this approved? |
|
@bcipriano Can we have your final approval on this. This PR is causing several conflicts in our end and would be good to have merged ASAP |
…ySoftwareFoundation#1035) * Update dispatchQuery to use min_cores Sorting jobs only by priority causes a situation where low priority jobs can get starved by a constant flow of high priority jobs. The new formula adds a modifier to the sorting rank to take into account the number of cores the job is requesting and also the number of days the job is waiting on the queue. Priorities numbers over 200 will mostly override the formula and work as a priority only based scheduling. sort = priority + (100 * (1 - (job.cores/job.int_min_cores))) + (age in days) Besides that, also take layer_int_cores_min into account when filtering folder_resourse limitations to avoid allocating more cores than the folder limits. (cherry picked from commit 566411aeeddc60983a30eabe121fd03263d05525) * Revert "Update dispatchQuery to use min_cores" This reverts commit 2eb4936 * Replace DispatchQueue and BookingQueue with HealthyThreadPool Queues will not inherit from ThreadPoolExecutor, instead they will manage an instance of HealthThreadPool, which is a threadPoolExecutor that handles healthChecks, termination and repeated tasks. With this the Booking queue should be able to self-heal when locked threads happen. * Remove trackit reference * Refactor HostReportQueue to use guava Cache Use a guava cache to store only the last version of a HostReport per host. * Configure HostReportQueue on opencue.properties * Fix unit tests * Fix unit tests * This unit tests is not actually testing anything useful Test doesn't make sense with the new threadpool and will also cause problems whenever an user changes a config property. Co-authored-by: Roula O'Regan <[email protected]>
…ySoftwareFoundation#1035) * Update dispatchQuery to use min_cores Sorting jobs only by priority causes a situation where low priority jobs can get starved by a constant flow of high priority jobs. The new formula adds a modifier to the sorting rank to take into account the number of cores the job is requesting and also the number of days the job is waiting on the queue. Priorities numbers over 200 will mostly override the formula and work as a priority only based scheduling. sort = priority + (100 * (1 - (job.cores/job.int_min_cores))) + (age in days) Besides that, also take layer_int_cores_min into account when filtering folder_resourse limitations to avoid allocating more cores than the folder limits. (cherry picked from commit 566411aeeddc60983a30eabe121fd03263d05525) * Revert "Update dispatchQuery to use min_cores" This reverts commit 2eb4936 * Replace DispatchQueue and BookingQueue with HealthyThreadPool Queues will not inherit from ThreadPoolExecutor, instead they will manage an instance of HealthThreadPool, which is a threadPoolExecutor that handles healthChecks, termination and repeated tasks. With this the Booking queue should be able to self-heal when locked threads happen. * Remove trackit reference * Refactor HostReportQueue to use guava Cache Use a guava cache to store only the last version of a HostReport per host. * Configure HostReportQueue on opencue.properties * Fix unit tests * Fix unit tests * This unit tests is not actually testing anything useful Test doesn't make sense with the new threadpool and will also cause problems whenever an user changes a config property. Co-authored-by: Roula O'Regan <[email protected]>
…ySoftwareFoundation#1035) * Update dispatchQuery to use min_cores Sorting jobs only by priority causes a situation where low priority jobs can get starved by a constant flow of high priority jobs. The new formula adds a modifier to the sorting rank to take into account the number of cores the job is requesting and also the number of days the job is waiting on the queue. Priorities numbers over 200 will mostly override the formula and work as a priority only based scheduling. sort = priority + (100 * (1 - (job.cores/job.int_min_cores))) + (age in days) Besides that, also take layer_int_cores_min into account when filtering folder_resourse limitations to avoid allocating more cores than the folder limits. (cherry picked from commit 566411aeeddc60983a30eabe121fd03263d05525) * Revert "Update dispatchQuery to use min_cores" This reverts commit 2eb4936 * Replace DispatchQueue and BookingQueue with HealthyThreadPool Queues will not inherit from ThreadPoolExecutor, instead they will manage an instance of HealthThreadPool, which is a threadPoolExecutor that handles healthChecks, termination and repeated tasks. With this the Booking queue should be able to self-heal when locked threads happen. * Remove trackit reference * Refactor HostReportQueue to use guava Cache Use a guava cache to store only the last version of a HostReport per host. * Configure HostReportQueue on opencue.properties * Fix unit tests * Fix unit tests * This unit tests is not actually testing anything useful Test doesn't make sense with the new threadpool and will also cause problems whenever an user changes a config property. Co-authored-by: Roula O'Regan <[email protected]>
Replace DispatchQueue and BookingQueue with HealthyThreadPool
Queues will not inherit from ThreadPoolExecutor, instead they will manage
an instance of HealthThreadPool, which is a threadPoolExecutor that handles
healthChecks, termination and repeated tasks. With this the Booking queue
should be able to self-heal when locked threads happen.