Skip to content

Instantly share code, notes, and snippets.

@BagToad
Last active March 6, 2026 17:12
Show Gist options
  • Select an option

  • Save BagToad/0a935421e75eeaa4d7a8f05afd42c1d0 to your computer and use it in GitHub Desktop.

Select an option

Save BagToad/0a935421e75eeaa4d7a8f05afd42c1d0 to your computer and use it in GitHub Desktop.
Acceptance Test Results — PR #12627 (Copilot Code Review in gh)

Acceptance Test Results — PR #12627

Date: 2026-03-06
Branch: kw/pr-create-multi-select-with-search-ccr
Binary: gh version 2.86.1-0.20260306164747-7382b86c1a5e

Scenarios

# Scenario Issue Repo Result
1 gh pr create --reviewer @copilot (non-interactive) #1083 bagtoad/garbage ✅ PASS
2 gh pr edit --add-reviewer @copilot (non-interactive) #1094 bagtoad/garbage ✅ PASS
3 gh pr edit --remove-reviewer @copilot (non-interactive) #1094 bagtoad/garbage ✅ PASS
4 gh pr create --reviewer org/team (non-interactive) #12627 bagtoad-enterprise-cloud-testing/test-general ✅ PASS
5 Interactive gh pr create → Copilot (AI) shown, no author #1083 bagtoad/garbage ✅ PASS
6 Interactive search for a reviewer #1083 bagtoad/garbage ✅ PASS
7 Search, select, then search again (selection persists) #1083 bagtoad/garbage ✅ PASS
8 Existing reviewer appears once after re-launch + search #1082 bagtoad/garbage ✅ PASS
9 PR author not shown as suggested reviewer (pr create) #1083 bagtoad/garbage ✅ PASS
10 PR author not shown in gh pr edit reviewers #1082 bagtoad/garbage ✅ PASS
11 Interactive gh pr create with team reviewer #12627 bagtoad-enterprise-cloud-testing/test-general ✅ PASS

Scenario Details

Scenario 1: Non-interactive gh pr create --reviewer @copilot

Given a branch pushed to bagtoad/garbage
When gh pr create --title "..." --body "..." --reviewer "@copilot"
Then PR created successfully:

$ gh pr create --title "S1" --body "test" --reviewer "@copilot"
https://github.com/BagToad/garbage/pull/102

Verification via gh pr view:

$ gh pr view 102 -R bagtoad/garbage
reviewers:copilot-pull-request-reviewer (Commented)

Scenario 2: Non-interactive gh pr edit --add-reviewer @copilot

Given an existing PR on bagtoad/garbage without Copilot reviewer
When gh pr edit <number> --add-reviewer "@copilot"
Then Copilot requested as reviewer:

$ gh pr edit 103 -R bagtoad/garbage --add-reviewer "@copilot"
https://github.com/BagToad/garbage/pull/103

Verification:

$ gh pr view 103 -R bagtoad/garbage
reviewers:copilot-pull-request-reviewer (Requested)

Scenario 3: Non-interactive gh pr edit --remove-reviewer @copilot

Given PR with Copilot already requested
When gh pr edit 103 --remove-reviewer "@copilot"
Then Command succeeded (exit 0), review request removed:

$ gh pr edit 103 -R bagtoad/garbage --remove-reviewer "@copilot"
https://github.com/BagToad/garbage/pull/103
EXIT=0

Scenario 4: Non-interactive gh pr create --reviewer org/team

Given a branch pushed to bagtoad-enterprise-cloud-testing/test-general
When gh pr create --reviewer "bagtoad-enterprise-cloud-testing/test-team"
Then Team reviewer requested:

$ gh pr create --title "S4 team" --body "test" --reviewer "bagtoad-enterprise-cloud-testing/test-team"
https://github.com/bagtoad-enterprise-cloud-testing/test-general/pull/45

Verification:

$ gh pr view 45 -R bagtoad-enterprise-cloud-testing/test-general
reviewers:bagtoad-enterprise-cloud-testing/test-team (Requested)

Scenario 5: Interactive gh pr create with Copilot reviewer selection

Given a branch pushed to bagtoad/garbage
When gh pr create (interactive) → Add metadata → Reviewers
Then Reviewer list shows Copilot (AI), no BagToad (author excluded):

? What would you like to add? Reviewers
? Reviewers  [Use arrows to move, space to select, <right> to all, <left> to none, type to filter]
> [ ]  Search (5 more)
  [ ]  Copilot (AI)
  [ ]  mxie (Melissa Xie)
  [ ]  williammartin (William Martin)
  [ ]  andyfeller (Andy Feller)
  [ ]  babakks (Babak K. Shandiz)

6 candidates shown, no BagToad.

Scenario 6: Interactive search for a reviewer

Given interactive gh pr create → Add metadata → Reviewers
When user selects "Search" sentinel and types "mxie":

? Search reviewers mxie
? Reviewers  [Use arrows to move, space to select, <right> to all, <left> to none, type to filter]
> [ ]  Search (5 more)
  [ ]  Copilot (AI)
  [ ]  mxie (Melissa Xie)

Scenario 7: Search, select, then search again (selection persists)

Given interactive gh pr create → Reviewers
When user searches "mxie", selects her, then searches "bagtoad":

? Search reviewers bagtoad
? Reviewers  [Use arrows to move, space to select, <right> to all, <left> to none, type to filter]
> [ ]  Search (5 more)
  [x]  mxie (Melissa Xie)
  [ ]  Copilot (AI)

mxie remains [x] selected. BagToad not shown (author excluded).

Scenario 8: Existing reviewer appears once after re-launch + search

Given PR with mxie already requested as reviewer
When gh pr edit <number> → Reviewers
Then mxie appears once, pre-selected:

? What would you like to edit? Reviewers
? Reviewers  [Use arrows to move, space to select, <right> to all, <left> to none, type to filter]
> [ ]  Search (5 more)
  [x]  mxie (Melissa Xie)
  [ ]  Copilot (AI)
  [ ]  williammartin (William Martin)
  [ ]  andyfeller (Andy Feller)
  [ ]  babakks (Babak K. Shandiz)

When user searches "mxie":

? Search reviewers mxie
? Reviewers  [Use arrows to move, space to select, <right> to all, <left> to none, type to filter]
> [ ]  Search (5 more)
  [x]  mxie (Melissa Xie)

mxie appears once, still pre-selected. No duplicates.

Scenario 9: PR author not shown as suggested reviewer (pr create)

Given authenticated as BagToad
When gh pr create (interactive) → Reviewers
Then BagToad (Kynan Ware) does NOT appear (see S5 output above — 6 candidates, no BagToad)

Scenario 10: PR author not shown in gh pr edit reviewers

Given authenticated as BagToad, PR authored by BagToad
When gh pr edit <number> → Reviewers
Then BagToad (Kynan Ware) does NOT appear:

? What would you like to edit? Reviewers
? Reviewers  [Use arrows to move, space to select, <right> to all, <left> to none, type to filter]
> [ ]  Search (5 more)
  [ ]  Copilot (AI)
  [ ]  mxie (Melissa Xie)
  [ ]  williammartin (William Martin)
  [ ]  andyfeller (Andy Feller)
  [ ]  babakks (Babak K. Shandiz)

6 candidates, no BagToad.

Scenario 11: Interactive gh pr create with team reviewer (NEW)

Given a branch pushed to bagtoad-enterprise-cloud-testing/test-general
When gh pr create (interactive) → Add metadata → Reviewers
Then Team appears in reviewer list alongside users and Copilot:

? What would you like to add? Reviewers
? Reviewers  [Use arrows to move, space to select, <right> to all, <left> to none, type to filter]
> [ ]  Search (3 more)
  [ ]  Copilot (AI)
  [ ]  june1963
  [ ]  bagtoad-enterprise-cloud-testing/test-team

When user selects bagtoad-enterprise-cloud-testing/test-team and submits:

? Reviewers  [Use arrows to move, space to select, <right> to all, <left> to none, type to filter]
  [ ]  Search (3 more)
  [ ]  Copilot (AI)
  [ ]  june1963
> [x]  bagtoad-enterprise-cloud-testing/test-team
? Reviewers bagtoad-enterprise-cloud-testing/test-team
? What's next? Submit
https://github.com/bagtoad-enterprise-cloud-testing/test-general/pull/46

Verification:

$ gh pr view 46 -R bagtoad-enterprise-cloud-testing/test-general
reviewers:bagtoad-enterprise-cloud-testing/test-team (Requested)

Bugs Found & Fixed

1. gh pr create --reviewer @copilot not resolving to bot login

@copilot was not being replaced with copilot-pull-request-reviewer in gh pr create. The CopilotReviewerReplacer was wired into pr edit but missing from pr create's NewIssueState.

Fix: Added CopilotReviewerReplacer call in NewIssueState with unit test (commit 08c7a4c20).

2. PR author shown as suggested reviewer in gh pr create

SuggestedReviewerActorsForRepo returned the current user as a collaborator candidate, but you cannot review your own PR.

Fix: Added viewer { login } to the GraphQL query and pre-seeded the seen map to exclude the current user (commit 90bfa624c).

3. Duplicate reviewers in gh pr edit interactive selection

MultiSelectWithSearch was called with display names (e.g. mxie (Melissa Xie)) as defaults, but the dedup logic compares against keys (logins like mxie). The mismatch prevented deduplication.

Fix: Passed DefaultLogins instead of Default display names (commit 1bba50b3e).

4. PR author shown as reviewer in gh pr edit

SuggestedReviewerActors excluded the author from suggestedReviewerActors via isAuthor, but the author could still appear in the collaborators fallback source.

Fix: Added author { login } to the GQL query and pre-seeded seen (commit 24fb7657c).

5. Organization query refactored

Replaced top-level organization(login: $owner) query with repository.owner { ... on Organization { teams } } inline fragment. This eliminates the need to handle "Could not resolve to an Organization" errors for personal repos.

Fix: Refactored both SuggestedReviewerActors and SuggestedReviewerActorsForRepo (commit 7382b86c1).

Commits

7382b86c1 Fetch org teams via repository.owner inline fragment
24fb7657c Exclude PR author from reviewer candidates in SuggestedReviewerActors
1bba50b3e Fix duplicate reviewers in gh pr edit by passing logins as defaults
90bfa624c Exclude current user from suggested reviewers in gh pr create
08c7a4c20 Replace @copilot with Copilot reviewer login in gh pr create
49f1bd880 Add TODO requestReviewsByLoginCleanup on GHES ID-based reviewer path
07138b6ed Remove /slug team reviewer shorthand normalization
37776cf2e Add TODO requestReviewsByLoginCleanup on static reviewer MultiSelect
dd7e44ee0 Check state.ActorReviewers in MetadataSurvey reviewer search gate
8f62e8116 Label Copilot detection in SuggestedReviewerActorsForRepo as a hack
6341588f9 Add TODO requestReviewsByLoginCleanup in CreatePullRequest
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment