/quicktest-core
/smoke-test
/smoke-test --serve
/smoke-test --kubernetes

smoketest, and smoketest serve pass

@kevinmingtarja I think I'll have to upgrade the api server version (or maybe skylet version?). I am getting backwards compatibility issues because skylet doesn't have the Serve implementation for grpc (nevertheless, is_grpc_enabled is True, naturally because we implemented it for autostop)

smoketest, and smoketest serve pass

@kevinmingtarja I think I'll have to upgrade the api server version (or maybe skylet version?). I am getting backwards compatibility issues because skylet doesn't have the Serve implementation for grpc (nevertheless, is_grpc_enabled is True, naturally because we implemented it for autostop)

Yeah good catch, I'm also facing that problem in my PR, and I think upgrading the skylet version is not enough, because we don't auto-upgrade skylet/clusters that are launched by an older API server (we have to re-run sky start IIRC). So I think we have two options:

1. We catch grpc UNIMPLEMENTED errors, and fallback to legacy SSH codegen execution

except grpc.RpcError as e:
            if e.code() == grpc.StatusCode.UNIMPLEMENTED:

2. We need a way to detect if a skylet/cluster is deployed by an older API server, and have a way to auto-upgrade Skylet, so that it can always get the latest code. Thinking out loud here, maybe we should store the skylet version on the API server, similar to how in k8s you can see what version of kubelet your nodes are running.

I was planning on just doing (1) for now in #6647, and leave (2) for a follow up PR.

1. We catch grpc UNIMPLEMENTED errors, and fallback to legacy SSH codegen execution

except grpc.RpcError as e:
            if e.code() == grpc.StatusCode.UNIMPLEMENTED:

I was planning on just doing (1) for now in #6647, and leave (2) for a follow up PR.

Got it. Also, I feel it will be helpful for us to have some kind of standards on catching errors that arise with grpc.
I've identified four common errors:

• grpc.FutureTimeoutError
• grpc.RpcError
• exceptions.FetchClusterInfoError (this comes from handle.get_grpc_channel() which routes to like a cluster status updater that will fail when cluster is unavailable)
• exceptions.SkyletInternalError

The existing codepath exposes CommandError and FetchClusterInfoError, but handling them is extremely inconsistent (some errors are wrapped as RuntimeExceptions while others are just exposed), which makes it really confusing to develop.

/smoke-test --serve

/smoke-test --kubernetes
/smoke-test --jobs-consolidation

/smoke-test --kubernetes /smoke-test --jobs-consolidation

I saw that the --jobs-consolidation test failure is related to sky.exceptions.CloudError: grpc error (FutureTimeoutError). FYI I also saw those and made it more robust in my PR.

/smoke-test -k test_launch_fast_with_autostop --aws --jobs-consolidation

/smoke-test --serve

/quicktest-core

/smoke-test --serve --grpc

/smoke-test --serve --kubernetes --grpc
/smoke-test --serve --kubernetes

/quicktest-core

rerunning failed smoke tests (flaky, due to provisioning errors which are unrelated)

smoke-tests passing (current failures due to provisioning errors/manual cancel) linking buildkite results below (commit to pre-review commit, but review only was concerned about renaming, etc):
https://buildkite.com/skypilot-1/smoke-tests/builds/3364
https://buildkite.com/skypilot-1/smoke-tests/builds/3364