This PR introduces a gRPC service interface for Skylet's autostop configuration functionality, replacing the previous mechanism of generating code using AutostopCodeGen and invoking it over SSH.

We started with autostop as it serves as a good opportunity to start migrating to gRPC while minimizing risk, as there's only two functions (SetAutostop and IsAutostopping). We will quickly follow up with the rest of the places that are still using codegen (JobLibCodeGen, ManagedJobCodeGen, ServeCodeGen, RayCodegen).

The way this works is by establishing an SSH tunnel using local port forwarding from the API server to the Skylet, and then creating a gRPC channel over this encrypted tunnel. We store the local port number and the PID in CloudVmRayResourceHandle (which is persisted to the db, and lasts for the lifetime of the cluster) so that we can reuse this tunnel across multiple requests.

Given that this is the first time we're using gRPC in this codebase, there are a few things that first need to be decided:

1. Directory structure for the protobuf files, as pointed out by [Core] Add gRPC autostop service to skylet #6574 (comment)
2. Process for generating and committing the generated python code, and ensuring the runtime grpc and protobuf version is compatible with the generated code

Tested (run the relevant ones):

• Code formatting: install pre-commit (auto-check on commit) or bash format.sh
• Any manual or new tests for this PR (please specify below)
• All smoke tests: /smoke-test (CI) or pytest tests/test_smoke.py (local)
• Relevant individual tests: /smoke-test -k test_name (CI) or pytest tests/test_smoke.py::test_name (local)
  • Still investigating one smoke test failure - False alarm, an additional log line from the port forward command caused some output matching in our smoke test to fail, reduced that log line to debug level so it wouldn't break this
• Backward compatibility: /quicktest-core (CI) or pytest tests/smoke_tests/test_backward_compat.py (local)
  • When a new API server talks to a remote cluster spun up before this commit, the skylet will not have a gRPC server listening, so we will fall back to the old way of ssh + codegen. Tested and verified in https://buildkite.com/skypilot-1/quicktest-core/builds/829#0198873e-20a8-470e-ba5b-976fe5935a65/57-1332
  • For old API server with new Skylet, it will just keep on using the old codegen over SSH execution, so no breaking changes too here

Performance improvements:

Ran a simple test (gist) on a 2 vCPU 4 GB RAM machine which schedules 50 autostop requests in parallel.

% sky launch -c bench --infra aws -y prometheus.yaml; sleep 30; for i in {1..50}; do; sh -c 'sky autostop -i 60 --down -y bench &'; done;

Before: The cluster would hang (and go to INIT state) as it would reach 100% CPU and memory usage

After: No noticeable CPU/memory usage spikes