application redeployment is getting failed for OpenShift v3.7.0 #1130
Comments
|
Found this issue in regression test execution |
hrishin
changed the title
|
Confirmed, I was able to reproduce the bug you reported. Here are some pod logs: #2nd Deployment Surprisingly, application pod comes back into a steady state after some time. Need to check why this behaviour is happening. |
|
I am also running into this issue in a OpenShift 3.7 cluster. Observations:
|
|
Is this something related to OpenShift 3.7.0 or fmp version? Wha tI mean is can I use the current version of fmp with OpenShift 3.6.0 without any problem or I'll face the same problem? |
|
As far as I know, this problem is specific to using fmp with Openshift 3.7.0 . It will work fine with previous Openshift version. If you use the current version of fmp with Openshift 3.6.0, ideally it should work. Thanks |
|
@lordofthejars This is a OpenShift 3.7 issue for me. It's actually a major blocker and preventing us from moving to OpenShift 3.7. In OpenShift 3.6, it works fine. |
|
I tried doing this with older versions of Fabric8 maven plugin but faced a similar issue. I don't think this has something to do with Fabric8 maven plugin. On OpenShift 3.7, people are facing these kinds of issues, one of them can be found here. |
|
As @rohanKanojia said, I've been having this issue too. In fact, I think I already encountered during a live coding presentation back in November but forgot about it. My planned workaround for this is to switch to binary deployments. E.g. do |
|
I have done some deep diving into this and I am fairly certain that it is triggered by F-M-P target "fabric8:apply". I am not saying it is the F-M-P itself, but perhaps one of its dependencies that interact with OpenShift. I can see in the log that when I bump the image version (fabric8:resource and fabric8:build) on top of an existing DC that has been deployed and run a fabric8:apply, a lot of things happen but what I managed to extract from the server-side logs is the following:
and when looking at rest.go:91 you can see that it will go through all the triggers to verify that it can indeed be triggered (https://github.com/openshift/origin/blob/12d2744c60e3f7a27d6b76a06d5b4d2a5f40e246/pkg/apps/registry/instantiate/rest.go#L91). The instantiate log-entry above also seems to point at resource being a OC 3.6 - which is a version below 3.7. This looks like it's due to a dependency of F-M-P that doesn't use OC 3.7? When I do the same thing via the web console - that is, do a fabric8:build to push up a new version of the image - and manually bump the version of the image via the UI dropdown and save, it does not trigger a re-deployment and I cannot find it in the logs. |
|
@hrishin @rohanKanojia What seems to be new in 3.7 is that if the container image reference is updated, it will lead to a redeployment being triggered. This is not due to the ImageTrigger - which I have disabled in my case - which acts based on ImageStream changes. If the image trigger reference is updated, it works just like in 3.6 and only executes once there is a rollout. This is what happens when you modify the image reference via the Web Console - it only changes the trigger section's image reference. On the other hand, F-M-P changes both the Image Trigger section as well as the Container Image reference which leads to an automatic deploy (which fails) in 3.7. I think we need to change the DeploymentConfig generation in F-M-P such that it only updates the trigger section if there is one and thereby does not "auto-deploy". The deployment can then be triggered by oc rollout which will resolve the trigger Image reference. I don't have a solution for cases where there is no trigger. |
…enShift v3.7.0 + Added a workaround to deal with ImageTriggers. + Added flag fabric8.openshift.trimImageInContainerSpec which would trim image in container spec
Fixes #1130 Application redeployment is getting failed on OpenShift v3.7.0
|
hey @valdar it's not yet available. Will make it available soon, in a day or two max. |
|
Hey @valdar FMP 3.5.35 has been released yesterday. It contains the workaround for redeployment on Openshift 3.7.x You should hit for Openshift 3.7.x Thanks |
…enShift v3.7.0 + Added a workaround to deal with ImageTriggers. + Added flag fabric8.openshift.trimImageInContainerSpec which would trim image in container spec + Updated documentation related to flag. Backporting changes to accomodate: OSFUSE-718: [OSO][OCP 3.7] f-m-p redeployments failing to deploy
|
Sorry for the noise here, I know this is closed, but -- would using |
|
Yes, It would work on Openshift 3.6 also. |
|
Thanks! |
|
Hey @hrishin @rohanKanojia @mojsha @Ladicek, Just figured out that now redeployment is working on openshift 3.7 without the flag also. Can you please cross check. I think the updated the binary. Not sure. Thanks |
|
Is that with some newer OCP version that perhaps backported the bugfix? |
|
Here is the PR link openshift/origin#18524 which resolved this problem I think. |
|
It seems to be backported to Origin 3.7.2 too, so not just OCP: https://github.com/openshift/origin/releases/tag/v3.7.2 At this point, I think the best course of action would be:
WDYT? |
|
@Ladicek Not sure why you want to remove this flag? The flag is doing something good, namely doesn't add unnecessary information. If you use a DC along with an ImageTrigger, you will get that field populated once the DC is executed. Populating that field in this scenario has no meaning, and as we saw in this case, 3.7 it caused problems. This is also why the OpenShift developer in that thread (openshift/origin#18406 (comment)) says:
Please do not remove this flag - if someone does not need it, simply don't use it. But for the abovementioned use-case this makes sense. |
|
Can you please help me with the link of the boosters you are talking about to remove the flags? |
|
Sorry guys, when I said "remove the flag", I didn't mean "remove it from FMP", I meant "remove it from RHOAR quickstarts (aka boosters)". We enable it by default in there, which I think we no longer have to do (if the fix in OpenShift actually fixes the problem), and it actually causes us some issues (which I'm sure @cescoffier would be happy to tell you about :-) ). |
|
Hey @Ladicek Are you talking about https://github.com/openshiftio/booster-parent/blob/master/pom.xml |
|
Yes. Did I just open a can of worms I didn't really want to touch? |
|
No, i just want to confirm that this parent pom has something to do with https://github.com/snowdrop/spring-boot-crud-booster. Because we are using this booster for our regression tests and if by default the flag is added than my findings may be wrong. |
|
Indeed the project https://github.com/snowdrop/spring-boot-crud-booster inherits from https://github.com/openshiftio/booster-parent/blob/master/pom.xml (indirectly through https://github.com/snowdrop/spring-boot-booster-parent/blob/master/pom.xml), so it has the property enabled by default. |
|
I'm late on this, @Ladicek. Your plan looks like a winner in my book. :) |
|
I no longer recall and don't even want to :-D |
Description
If f8-m-p redeploy the existing application on OpenShift v3.7 then Deployment is not happening correctly.
Its unable to pull the images and status appears ImgPullErr for a long time. After some time with 4-5 retries, it's able to pull the image and deployment happen.
Info
This might be an issue with F8-M-P or OpenShift.
mvn -v) :Kubernetes / OpenShift setup and version : OpenShift v3.7.0
If it's a bug, how to reproduce :
mvn install or mvn fabric8:deploymvn insall or mvn fabric8:deployThe text was updated successfully, but these errors were encountered: