Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Endpoints controller is dumb #14710

Closed
smarterclayton opened this issue Jun 16, 2017 · 5 comments
Closed

Endpoints controller is dumb #14710

smarterclayton opened this issue Jun 16, 2017 · 5 comments
Assignees
@smarterclayton @aveshagarwal @derekwaynecarr
Labels
area/performance priority/P1
Milestone
3.6.0

Comments

@smarterclayton
Copy link
Contributor

smarterclayton commented Jun 16, 2017 •
edited
Loading

On 3.5 It looks like it's refreshing every 30s, and it's doing a live GET before update, and somehow we're generating 100 GET calls / second on a set of 10k endpoints.
@smarterclayton smarterclayton added this to the 3.6.0 milestone Jun 16, 2017
@smarterclayton smarterclayton self-assigned this Jun 16, 2017
@smarterclayton
Copy link
Contributor Author

@derekwaynecarr if i don't get a chance to look at this someone should

@smarterclayton
Copy link
Contributor Author

Basically - endpoints controller is horribly broken, upstream we completely rewrote it (as a result of this issue), and we may backport if we still see issues at extreme scale in 3.6.

@derekwaynecarr derekwaynecarr self-assigned this Jul 5, 2017
@derekwaynecarr
Copy link
Member

we can have someone look at backport if needed.

pertinent upstreams:
kubernetes/kubernetes#47731
kubernetes/kubernetes#47788

@derekwaynecarr
Copy link
Member

@aveshagarwal -- can you help put picks together in origin for the referenced PRs in case we see this in 3.6?

@aveshagarwal aveshagarwal mentioned this issue Jul 5, 2017
Merged
@aveshagarwal
Copy link
Contributor

#15067

@aveshagarwal aveshagarwal mentioned this issue Jul 19, 2017
Closed
@joelsmith joelsmith mentioned this issue Sep 27, 2017
Merged
openshift-merge-robot added a commit that referenced this issue Sep 29, 2017
@openshift-merge-robot
Merge pull request #16575 from joelsmith/release-3.6
f7a367c 
Automatic merge from submit-queue.

Cherry-pick #15067 endpoints fixes.

cc @sjenning @eparis @derekwaynecarr

This is a cherry-pick of #15067 to prevent too-frequent processing of endpoint updates.

https://bugzilla.redhat.com/show_bug.cgi?id=1481603

Here's the original text of #15067 as written by @aveshagarwal:

> Fixes #14710
> 
> Note: I had to manually merge changes and due to this, the commits might not match exactly. 
> 
> It has following 2 complete upstream PRs: 
> kubernetes/kubernetes#47731  (with some modifications/additions).
> kubernetes/kubernetes#47788
> 
> And one partial PR to make above PRs compile:
> kubernetes/kubernetes#44408

(Edited to fix bz link -- originally sent with wrong bug)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@smarterclayton smarterclayton

@aveshagarwal aveshagarwal

@derekwaynecarr derekwaynecarr

Labels
area/performance priority/P1
Projects
None yet
Milestone
3.6.0
Development

No branches or pull requests
3 participants
@smarterclayton @aveshagarwal @derekwaynecarr