[WIP][Data] Revising resource allocator task scheduling decision #60639
Conversation
…max_block_size` until estimate becomes available Signed-off-by: Alexey Kudinkin <ak@anyscale.com>
…e budget to hold pending task outputs Signed-off-by: Alexey Kudinkin <ak@anyscale.com>
![gemini-code-assist[bot]](https://cdn.statically.io/img/avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request revises the resource allocator's task scheduling decision to be more robust, especially concerning object store memory. It introduces a fallback mechanism to estimate the size of pending task outputs when no historical data is available, using target_max_block_size. This estimate is then used to check if there is sufficient object store memory budget before submitting a new task.
The overall logic is sound and improves scheduling decisions. However, I found a critical issue in resource_manager.py where self._metrics is used instead of op.metrics, which will lead to an AttributeError.
| budget.object_store_memory > ( | ||
| self._metrics.obj_store_mem_max_pending_output_per_task or 0 | ||
| ) |
There was a problem hiding this comment.
The ReservationOpResourceAllocator does not have a _metrics attribute, so self._metrics will raise an AttributeError. It seems you intended to use the metrics from the operator op that is passed as an argument.
| budget.object_store_memory > ( | |
| self._metrics.obj_store_mem_max_pending_output_per_task or 0 | |
| ) | |
| budget.object_store_memory > ( | |
| op.metrics.obj_store_mem_max_pending_output_per_task or 0 | |
| ) |
![cursor[bot]](https://cdn.statically.io/img/avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
Signed-off-by: Alexey Kudinkin <ak@anyscale.com>
alexeykudinkin
force-pushed
the
ak/res-allc-fup-1
branch
from
1cf767c to
33cf32d
Compare
| # task outputs) | ||
| budget.object_store_memory > ( | ||
| op.metrics.obj_store_mem_max_pending_output_per_task or 0 | ||
| ) |
There was a problem hiding this comment.
Strict inequality prevents task submission when budget equals threshold
High Severity
The new check uses strict inequality (>) to compare budget.object_store_memory against obj_store_mem_max_pending_output_per_task. When the budget exactly equals the threshold (which can happen when the reservation logic in _update_reservation sets reserved_for_tasks to exactly min_resource_usage.object_store_memory), the condition evaluates to False and prevents task submission. This can cause deadlock in resource-constrained environments where no tasks can ever be scheduled. The comparison likely needs to use >= instead of >.
Description
Related issues
Additional information