Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cache mutation bug #15324

Closed
bparees opened this issue Jul 19, 2017 · 9 comments
Closed

Cache mutation bug #15324

bparees opened this issue Jul 19, 2017 · 9 comments
Assignees
@mfojtik
Labels
component/kubernetes kind/test-flake Categorizes issue or PR as related to test flakes. priority/P0

Comments

@bparees
Copy link
Contributor

bparees commented Jul 19, 2017

https://ci.openshift.redhat.com/jenkins/job/test_pull_request_origin/3256/

I0719 02:12:14.503031   10463 stateful_set.go:452] Syncing StatefulSet default/mysql with 1 pods
CACHE *v1beta1.StatefulSet[4] ALTERED!

Cache mutations are not flakes. They are P0 bugs and you found one. Open it up!

was run against post-rebase code.
@bparees
Copy link
Contributor Author

bparees commented Jul 19, 2017

Note there are two other failures in that run also, that make it appear as if the server went away:

=== BEGIN TEST CASE ===
hack/test-cmd.sh:126: executing 'oc delete project 'cmd-admin'' expecting success
FAILURE after 0.684s: hack/test-cmd.sh:126: executing 'oc delete project 'cmd-admin'' expecting success: the command returned the wrong error code
There was no output from the command.
Standard error from the command:
The connection to the server 172.17.0.2:28443 was refused - did you specify the right host or port?
=== END TEST CASE ===

probably unrelated, but also probably rebase related?

@mfojtik
Copy link
Contributor

mfojtik commented Jul 19, 2017

seems like upstream bug?

@deads2k
Copy link
Contributor

deads2k commented Jul 19, 2017

seems like upstream bug?

Yes, but it will block our CI until it's fixed.

@deads2k
Copy link
Contributor

deads2k commented Jul 19, 2017

probably unrelated, but also probably rebase related?

Same run? The cache mutation kills the server.

@mfojtik
Copy link
Contributor

mfojtik commented Jul 19, 2017

attempt to fix this: #15328

@bparees
Copy link
Contributor Author

bparees commented Jul 19, 2017 via email

@liggitt
Copy link
Contributor

liggitt commented Jul 19, 2017 •
edited
Loading

weird, the diff showed kind and apiVersion fields got populated, and nothing else changed...

I bet passing an object to an encoder sets those fields.

@jsafrane jsafrane mentioned this issue Jul 19, 2017
Merged
@jsafrane jsafrane added the kind/test-flake Categorizes issue or PR as related to test flakes. label Jul 19, 2017
@deads2k deads2k assigned mfojtik and unassigned deads2k Jul 19, 2017
@deads2k
Copy link
Contributor

deads2k commented Jul 19, 2017

attempt to fix this: #15328

Assigned you since you have a pull.

@mfojtik mfojtik mentioned this issue Jul 20, 2017
Merged
k8s-github-robot pushed a commit to kubernetes/kubernetes that referenced this issue Jul 21, 2017
Merge pull request #49285 from mfojtik/fix-statefull-mutation
29cc1aa 
Automatic merge from submit-queue (batch tested with PRs 49328, 49285, 49307, 49127, 49163)

fix mutation in statefulset sync

Original issue: openshift/origin#15324

Seems like something inside `UpdateStatefulSet` mutates the StatefulSet. We might need to dig deeper to find the exact location where the non-copy mutation happens.
@0xmichalis
Copy link
Contributor

This is fixed and @mfojtik said he will open a follow-up issue upstream to track investigation on this.

@mfojtik mfojtik mentioned this issue Jul 21, 2017
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mfojtik mfojtik

Labels
component/kubernetes kind/test-flake Categorizes issue or PR as related to test flakes. priority/P0
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@mfojtik @liggitt @jsafrane @0xmichalis @bparees @deads2k