Allow retry action of pulling image in build process #13380

guangxuli · 2017-03-14T00:22:33Z

command :
[cloud@centos ~]$ oc new-build https://github.com/openshift/ruby-hello-world --source-image=openshift/jenkins-1-centos7 --source-image-path=/var/lib/jenkins:tmp --allow-missing-images=false

result:

--> Found Docker image bce81d0 (2 weeks old) from Docker Hub for "centos/ruby-22-centos7"

Ruby 2.2 
-------- 
Platform for building and running Ruby 2.2 applications

Tags: builder, ruby, ruby22

* An image stream will be created as "ruby-22-centos7:latest" that will track the source image
* A Docker build using source code from https://github.com/openshift/ruby-hello-world will be created
  * The resulting image will be pushed to image stream "ruby-hello-world:latest"
  * Every time "ruby-22-centos7:latest" changes a new build will be triggered


--> Creating resources with label build=ruby-hello-world ...
    imagestream "ruby-22-centos7" created
    imagestream "ruby-hello-world" created
    buildconfig "ruby-hello-world" created
    imagestream "jenkins-1-centos7" created
--> Success
    Build configuration "ruby-hello-world" created and build triggered.
    Run 'oc logs -f bc/ruby-hello-world' to stream the build progress.

Log:

[cloud@centos ~]$ oc logs -f bc/ruby-hello-world
Cloning "https://github.com/openshift/ruby-hello-world" ...
	Commit:	022d87e4160c00274b63cdad7c238b5c6a299265 (Merge pull request #58 from junaruga/feature/fix-for-ruby24)
	Author:	Ben Parees <[email protected]>
	Date:	Fri Mar 3 15:29:12 2017 -0500
Pulling image "openshift/jenkins-1-centos7@sha256:1fc2d985e743c43ac32344d4987778647621c68810d2d25551736f40de34b2ce" ...
error: build error: error pulling image openshift/jenkins-1-centos7@sha256:1fc2d985e743c43ac32344d4987778647621c68810d2d25551736f40de34b2ce: Get https://registry-1.docker.io/v2/: read tcp 192.168.122.251:60528->50.17.62.194:443: read: **connection reset by peer**

Since the error connection reset by peer had been handled in build pushing process,
I think we should add retry action of pulling image like what pushing image does. Test result seems acceptable which always successful after retrying pull image 2~3 time in my workaround.

/cc @bparees

bparees · 2017-03-15T15:47:24Z

pkg/build/builder/dockerutil.go

+	if err := RetryImageAction(client, opts, authConfig); err != nil {
+		return "", err
+	} else {
+		return digestWriter.Digest, nil
 	}


no else block needed, just "return digestWriter.Digest,nil"

bparees · 2017-03-15T15:47:53Z

[test]

bparees · 2017-03-15T15:48:04Z

[testextended][extended:core(builds)]

guangxuli · 2017-03-16T01:55:57Z

@bparees

test falilure seems a flake


[ERROR] PID 12365: hack/lib/build/environment.sh:137: `return "${exitcode}"` exited with status 2.
13:02:27 [INFO] 		Stack Trace: 
13:02:27 [INFO] 		  1: hack/lib/build/environment.sh:137: `return "${exitcode}"`
13:02:27 [INFO] 		  2: hack/lib/build/environment.sh:217: os::build::environment::withsource
13:02:27 [INFO] 		  3: hack/env:36: os::build::environment::run
13:02:27 [INFO]   Exiting with code 2.
13:02:27 ++ export status=FAILURE
13:02:27 ++ status=FAILURE
13:02:27 + set +o xtrace
13:02:27 
########## FINISHED STAGE: FAILURE: RUN CHECK AND VERIFY TASKS ##########

bparees · 2017-03-16T02:40:26Z

That's not the failure, the failure was in a push image test, which given the area of your changes is probably not a flake. Ben Parees | OpenShift

…

On Mar 15, 2017 9:56 PM, "Guangxu Li" ***@***.***> wrote: @bparees <https://github.com/bparees> test falilure seems a flake [ERROR] PID 12365: hack/lib/build/environment.sh:137: `return "${exitcode}"` exited with status 2. 13:02:27 [INFO] Stack Trace: 13:02:27 [INFO] 1: hack/lib/build/environment.sh:137: `return "${exitcode}"` 13:02:27 [INFO] 2: hack/lib/build/environment.sh:217: os::build::environment::withsource 13:02:27 [INFO] 3: hack/env:36: os::build::environment::run 13:02:27 [INFO] Exiting with code 2. 13:02:27 ++ export status=FAILURE 13:02:27 ++ status=FAILURE 13:02:27 + set +o xtrace 13:02:27 ########## FINISHED STAGE: FAILURE: RUN CHECK AND VERIFY TASKS ########## — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#13380 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AEvl3lEfFFFghnckMjTL78S93q4Io-91ks5rmJazgaJpZM4Mb-wS> .

guangxuli · 2017-03-16T07:03:46Z

@bparees sorry, in my local test I did not check the test error carefully. RetryImageActionshould return error after retry out max times. Fixed.

bparees · 2017-03-16T15:22:58Z

pkg/build/builder/dockerutil_test.go

@@ -141,8 +141,8 @@ func TestTagImage(t *testing.T) {
 func TestPushImage(t *testing.T) {


please add a TestPullImage equivalent to this test.

bparees · 2017-03-16T15:23:47Z

pkg/build/builder/dockerutil.go

+		time.Sleep(DefaultPushOrPullRetryDelay)
+	}
+
+	return fmt.Errorf("After retry %d times, %s image still failed", DefaultPushOrPullRetryDelay, actionName)


retry->retrying

guangxuli · 2017-03-17T13:24:17Z

@bparees Test case addressed, ptal.

bparees

one last nit.

bparees · 2017-03-17T14:07:39Z

pkg/build/builder/dockerutil_test.go

+	fakeDocker := NewFakeDockerClient()
+	fakeDocker.pullImageFunc = fakePullImageFunc
+	testAuth := docker.AuthConfiguration{
+		Username:      "usernname_foo_bar",


username_foo_bar

bparees · 2017-03-17T14:08:31Z

[testextended][extended:core(builds)]

bparees · 2017-03-17T14:08:36Z

[test]

openshift-bot · 2017-03-17T14:13:46Z

Evaluated for origin testextended up to c4afcce

delete else block return error after retry maxtimes add test case for pull image fix username

guangxuli · 2017-03-17T15:35:44Z

@bparees conformance test fail, the failure cause seems not related to what this PR does.

bparees · 2017-03-17T15:39:06Z

agree, looks like a flake.

openshift-bot · 2017-03-17T15:53:39Z

continuous-integration/openshift-jenkins/testextended SUCCESS (https://ci.openshift.redhat.com/jenkins/job/test_pull_request_origin_extended/48/) (Base Commit: 75c6750) (Extended Tests: core(builds))

bparees · 2017-03-17T16:43:23Z

[merge]

openshift-bot · 2017-03-17T16:45:26Z

Evaluated for origin merge up to ca07fa9

openshift-bot · 2017-03-17T16:45:27Z

[Test]ing while waiting on the merge queue

openshift-bot · 2017-03-17T16:53:27Z

Evaluated for origin test up to ca07fa9

openshift-bot · 2017-03-17T18:16:06Z

continuous-integration/openshift-jenkins/test SUCCESS (https://ci.openshift.redhat.com/jenkins/job/test_pull_request_origin/288/) (Base Commit: fae18e0)

openshift-bot · 2017-03-17T23:05:23Z

continuous-integration/openshift-jenkins/merge SUCCESS (https://ci.openshift.redhat.com/jenkins/job/merge_pull_request_origin/131/) (Base Commit: 65a0cbf) (Image: devenv-rhel7_6083)

openshift-bot · 2017-03-17T23:13:17Z

Origin Action Required: Please contact #openshift-dev to have this pull request manually reviewed and tested

pweil- added the priority/P2 label Mar 14, 2017

pweil- assigned bparees Mar 14, 2017

bparees requested changes Mar 15, 2017

View reviewed changes

guangxuli force-pushed the fix_docker_pull_fail branch from be2e5e6 to bce7095 Compare March 16, 2017 06:54

bparees requested changes Mar 16, 2017

View reviewed changes

guangxuli force-pushed the fix_docker_pull_fail branch from bce7095 to c4afcce Compare March 17, 2017 12:54

bparees requested changes Mar 17, 2017

View reviewed changes

add retry action of pulling image

ca07fa9

delete else block return error after retry maxtimes add test case for pull image fix username

guangxuli force-pushed the fix_docker_pull_fail branch from c4afcce to ca07fa9 Compare March 17, 2017 15:22

openshift-bot merged commit bd0bf10 into openshift:master Mar 18, 2017

smarterclayton mentioned this pull request Mar 22, 2017

Too much whitespace in new-app description #13492

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow retry action of pulling image in build process #13380

Allow retry action of pulling image in build process #13380

guangxuli commented Mar 14, 2017 •

edited

Loading

bparees Mar 15, 2017

guangxuli Mar 16, 2017

bparees commented Mar 15, 2017

bparees commented Mar 15, 2017

guangxuli commented Mar 16, 2017

bparees commented Mar 16, 2017 via email

guangxuli commented Mar 16, 2017

bparees Mar 16, 2017

guangxuli Mar 17, 2017

bparees Mar 16, 2017

guangxuli commented Mar 17, 2017

bparees left a comment

bparees Mar 17, 2017

guangxuli Mar 17, 2017

bparees commented Mar 17, 2017

bparees commented Mar 17, 2017

openshift-bot commented Mar 17, 2017

guangxuli commented Mar 17, 2017

bparees commented Mar 17, 2017

openshift-bot commented Mar 17, 2017

bparees commented Mar 17, 2017

openshift-bot commented Mar 17, 2017

openshift-bot commented Mar 17, 2017

openshift-bot commented Mar 17, 2017

openshift-bot commented Mar 17, 2017

openshift-bot commented Mar 17, 2017 •

edited

Loading

openshift-bot commented Mar 17, 2017

		@@ -141,8 +141,8 @@ func TestTagImage(t *testing.T) {
		func TestPushImage(t *testing.T) {

Allow retry action of pulling image in build process #13380

Allow retry action of pulling image in build process #13380

Conversation

guangxuli commented Mar 14, 2017 • edited Loading

bparees Mar 15, 2017

Choose a reason for hiding this comment

guangxuli Mar 16, 2017

Choose a reason for hiding this comment

bparees commented Mar 15, 2017

bparees commented Mar 15, 2017

guangxuli commented Mar 16, 2017

bparees commented Mar 16, 2017 via email

guangxuli commented Mar 16, 2017

bparees Mar 16, 2017

Choose a reason for hiding this comment

guangxuli Mar 17, 2017

Choose a reason for hiding this comment

bparees Mar 16, 2017

Choose a reason for hiding this comment

guangxuli commented Mar 17, 2017

bparees left a comment

Choose a reason for hiding this comment

bparees Mar 17, 2017

Choose a reason for hiding this comment

guangxuli Mar 17, 2017

Choose a reason for hiding this comment

bparees commented Mar 17, 2017

bparees commented Mar 17, 2017

openshift-bot commented Mar 17, 2017

guangxuli commented Mar 17, 2017

bparees commented Mar 17, 2017

openshift-bot commented Mar 17, 2017

bparees commented Mar 17, 2017

openshift-bot commented Mar 17, 2017

openshift-bot commented Mar 17, 2017

openshift-bot commented Mar 17, 2017

openshift-bot commented Mar 17, 2017

openshift-bot commented Mar 17, 2017 • edited Loading

openshift-bot commented Mar 17, 2017

guangxuli commented Mar 14, 2017 •

edited

Loading

openshift-bot commented Mar 17, 2017 •

edited

Loading