Skip to content

Fix target group instances registration when TG created#302

Merged
k8s-ci-robot merged 1 commit into
kubernetes:masterfrom
abubyr:fix-tg-registration
Jun 1, 2022
Merged

Fix target group instances registration when TG created#302
k8s-ci-robot merged 1 commit into
kubernetes:masterfrom
abubyr:fix-tg-registration

Conversation

@abubyr

@abubyr abubyr commented Jan 28, 2022

Copy link
Copy Markdown
Contributor

In case of creating a new TG removed return dropped off #this function
just before instance registration code. Daemon restart was needed to
call ensureTargetGroup() again and reach out registration code
(as targetGroup != nil, TG already created)

Change-Id: I57717602624d9632049596e6928ac4671387905d

What type of PR is this?
/kind bug

What this PR does / why we need it: fix of instances registration in AWS NLB target group during its creation

Which issue(s) this PR fixes:

Fixes #301

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

NONE

In case of creating a new TG removed return dropped off this function
just before instance registration code. Daemon restart was needed to
call ensureTargetGroup() again and reach out registration code
(as targetGroup != nil, TG already created)

Change-Id: I57717602624d9632049596e6928ac4671387905d
@k8s-ci-robot k8s-ci-robot added the kind/bug Categorizes issue or PR as related to a bug. label Jan 28, 2022
@k8s-ci-robot

Copy link
Copy Markdown
Contributor

Thanks for your pull request. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please follow instructions at https://git.k8s.io/community/CLA.md#the-contributor-license-agreement to sign the CLA.

It may take a couple minutes for the CLA signature to be fully registered; after that, please reply here with a new comment and we'll verify. Thanks.


Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@linux-foundation-easycla

linux-foundation-easycla Bot commented Jan 28, 2022

Copy link
Copy Markdown

CLA Signed

The committers are authorized under a signed CLA.

@k8s-ci-robot k8s-ci-robot added do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. labels Jan 28, 2022
@k8s-ci-robot

Copy link
Copy Markdown
Contributor

Welcome @abubyr!

It looks like this is your first PR to kubernetes/cloud-provider-aws 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes/cloud-provider-aws has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

@k8s-ci-robot k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Jan 28, 2022
@k8s-ci-robot

Copy link
Copy Markdown
Contributor

Hi @abubyr. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot k8s-ci-robot added the size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. label Jan 28, 2022
@k8s-ci-robot k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. release-note-none Denotes a PR that doesn't merit a release note. and removed cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. labels Jan 28, 2022

tg := result.TargetGroups[0]
return tg, nil
targetGroup = result.TargetGroups[0]

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems like a simple fix. I wonder, @nckturner or @M00nF1sh, maybe this was coded this way originally because we wanted to sleep some period of time after creation of a TargetGroup before attempting to register expected instances in the TargetGroup below on original line 632?

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I encountered the same issue with 1.23.0 today where the target group had no target registered.
Restarting the controller seems to have correctly updated the target group.

The return might be there to make a sleep, but I am not sure we are doing the reconcile loop again unless there is a change in the service.

I will try your version and see if it's happening again.

@glebiller

Copy link
Copy Markdown

I just tried this fix using a cluster that is created every morning for multiple days in a row. It's correctly populating the target groups when first deploying the cloud controller \o/.
With the latest released version, the cloud controller needs to be restarted in order to have the target groups filled.

@olemarkus

Copy link
Copy Markdown
Member

/ok-to-test

@k8s-ci-robot k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels May 28, 2022
@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jun 1, 2022
@k8s-ci-robot

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: olemarkus

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 1, 2022
@k8s-ci-robot k8s-ci-robot merged commit bb211bf into kubernetes:master Jun 1, 2022
@nullzone

Copy link
Copy Markdown

Hello,

This issue is still happening in 1.24.3 (causing a lot of issues and forcing us to move to CLB which the consequent reduction in the features like the possibility of assigning fixed IPs in the LBs).

Any idea when this merge could get accepted as part of a following release?

@olemarkus

olemarkus commented Aug 16, 2022

Copy link
Copy Markdown
Member

This PR has been released. Make sure you run cloud controller manager 1.24.1 or later

@mariusmarais

mariusmarais commented Aug 16, 2022

Copy link
Copy Markdown

@olemarkus There is no 1.24.1 in the Github Releases for this project, only 1.24.0 on 13 May. Is there another place to look for release info?

Edit: Apologies for asking (un-) related project questions, but I see you're active on kOps as well...

How does one instruct kOps 1.24.1 running Kubernetes 1.24.3 to use cloud-provider-aws 1.24.1? Wait for a kOps release?

@olemarkus

Copy link
Copy Markdown
Member

Try setting something like this

spec:
  cloudControllerManager:
    image:   registry.k8s.io/provider-aws/cloud-controller-manager:v1.24.1

I'll see to this being bumped in kOps shortly.

@olemarkus

Copy link
Copy Markdown
Member

See kubernetes/kops#14130

@JoelSpeed

Copy link
Copy Markdown
Contributor

I recently came across this and I think I've found the original issue, https://github.com/kubernetes/cloud-provider-aws/pull/293/files#diff-70a4ebdf516a65404dc3c3b3ba9925fc62d0f85bd75e272a4f8ddb4b7b8ab852L620-L623

These lines were dropped, I'm guessing unintentionally and meant that the chunking work also got broken. IMO we should be reverting this PR and reinstating these lines, the end of this block should be like below, as it was originally

		tg := result.TargetGroups[0]
		tgARN := aws.StringValue(tg.TargetGroupArn)
		if err := c.ensureTargetGroupTargets(tgARN, expectedTargets, nil); err != nil {
			return nil, err
		}
		return tg, nil

@JoelSpeed

Copy link
Copy Markdown
Contributor

I created #527 to restore the behaviour and re-separate the create and update behaviours. This will save us some API calls when creating services and re-distinguish the difference between create and update behaviour in the ensureTargetGroups function

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/bug Categorizes issue or PR as related to a bug. lgtm "Looks good to me", indicates that a PR is ready to be merged. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. release-note-none Denotes a PR that doesn't merit a release note. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Target group has empty instances list after creation

9 participants