[Pools] Support remote job controller. #6591
Conversation
|
/smoke-test |
…to pool-in-remote
|
/smoke-test |
cg505
left a comment
There was a problem hiding this comment.
The scheduling logic is increasingly complicated.
We should have some comments that explain how it works with pools.
Can we do an audit of these existing comments.
| self.graph = nx.DiGraph() | ||
| self.name: Optional[str] = None | ||
| self.policy_applied: bool = False | ||
| self.pool: Optional[str] = None |
There was a problem hiding this comment.
Since we do not serialize this, we need to make sure it is handled correctly by RESTful admin policy. See #6348.
| if current_state == state.ManagedJobScheduleState.ALIVE_WAITING: | ||
| if not _can_lauch_in_alive_job(): | ||
| if not (controller_utils.can_provision() or | ||
| actual_pool is not None): |
There was a problem hiding this comment.
Why do we care about the pool here?
There was a problem hiding this comment.
Because pool jobs will not launch. This helps it to get scheduled when there is no slot for provisioning but still slot for new job controller process
| # first. The requirements to launch in an alive job are more | ||
| # lenient, so there is no way that we wouldn't be able to launch | ||
| # an ALIVE_WAITING job, but we would be able to launch a WAITING | ||
| # job. |
There was a problem hiding this comment.
This comment is no longer true, right? There could be a WAITING job using a pool that can start (doesn't need a LAUNCHING slot), but the ALIVE_WAITING job we fetch is not on a pool and cannot go to LAUNCHING.
|
/smoke-test |
cg505
left a comment
There was a problem hiding this comment.
We will fix scheduler logic in a follow-up PR
|
/smoke-test |
1 similar comment
|
/smoke-test |
|
/smoke-test |
|
/smoke-test -k test_skyserve_new_autoscaler_update |
|
/smoke-test -k test_skyserve_new_autoscaler_update --kubernetes |
This reverts commit 8f22215.
|
/smoke-test -k test_skyserve_new_autoscaler_update |
* init * check pools * resources management * fix * fix test * tracking #procs * refactor * fix * fix * fix * fix endpoint * fix ready replicas * backoff 1 * fix leakage * locks * ux * ux * dont check can_provision for pool jobs * fix * comments * nit * nit * try fix autoscaler * Revert "try fix autoscaler" This reverts commit 8f22215. * new resource limitation
This PR supports running pools on remote job controller. It also adds resources management across pool & jobs.
Tested (run the relevant ones):
bash format.sh/smoke-test(CI) orpytest tests/test_smoke.py(local)/smoke-test -k test_name(CI) orpytest tests/test_smoke.py::test_name(local)/quicktest-core(CI) orpytest tests/smoke_tests/test_backward_compat.py(local)