test_pull_request_origin_extended_conformance_install_update failing across the board #17263

sjenning · 2017-11-10T17:49:51Z

test_pull_request_origin_extended_conformance_install_update started failing across the board about 2hrs ago

https://openshift-gce-devel.appspot.com/builds/origin-ci-test/pr-logs/directory/test_pull_request_origin_extended_conformance_install_update

@simo5

sjenning · 2017-11-10T17:51:15Z

Fails here

Creating output directory: /tmp/tito
Tagging new version of openshift-ansible: 3.7.5-1 -> 3.7.6-1
Traceback (most recent call last):
  File "/usr/bin/tito", line 23, in <module>
    CLI().main(sys.argv[1:])
  File "/usr/lib/python2.7/site-packages/tito/cli.py", line 202, in main
    return module.main(argv)
  File "/usr/lib/python2.7/site-packages/tito/cli.py", line 666, in main
    return tagger.run(self.options)
  File "/usr/lib/python2.7/site-packages/tito/tagger/main.py", line 114, in run
    self._tag_release()
  File "/usr/lib/python2.7/site-packages/tito/tagger/main.py", line 136, in _tag_release
    self._check_tag_does_not_exist(self._get_new_tag(new_version))
  File "/usr/lib/python2.7/site-packages/tito/tagger/main.py", line 501, in _check_tag_does_not_exist
    raise Exception("Tag %s already exists!" % new_tag)
Exception: Tag openshift-ansible-3.7.6-1 already exists!
++ export status=FAILURE
++ status=FAILURE
+ set +o xtrace
########## FINISHED STAGE: FAILURE: BUILD AN OPENSHIFT-ANSIBLE RELEASE [00h 00m 02s] ##########

smarterclayton · 2017-11-10T18:03:47Z

@sdodson @michaelgugino @abutcher

michaelgugino · 2017-11-10T18:39:44Z

@smunilla Linking you to this issue.

jupierce · 2017-11-10T23:03:55Z

Hopefully fixed now. Branching 3.7 in openshift-ansible yesterday caused this. The normal CD build of 3.7 in the release-3.7 branch created the 3.7.6-1 tag. The openshift-ansible.spec in master was still at 3.7.4-1. I've bumped the master openshift-ansible.spec to 3.8 to avoid this.

smarterclayton · 2017-11-11T01:55:46Z

Install update is still broken, might be that openshift-ansible tag has the wrong release https://openshift-gce-devel.appspot.com/build/origin-ci-test/pr-logs/pull/17268/test_pull_request_origin_extended_conformance_install_update/9241/ On Nov 10, 2017, at 6:04 PM, Justin Pierce <[email protected]> wrote: Hopefully fixed now. Branching 3.7 in openshift-ansible yesterday caused this. The normal CD build of 3.7 in the release-3.7 branch created the 3.7.6-1 tag. The openshift-ansible.spec in master was still at 3.7.4-1. I've bumped the master openshift-ansible.spec to 3.8 to avoid this. — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#17263 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABG_p26o9Kg4-mWnKHyT-CuteSSWQUyXks5s1NZdgaJpZM4QZ5YZ> .

simo5 · 2017-11-11T13:32:33Z

Still broken on the next step:

######### STARTING STAGE: INSTALL ANSIBLE ATOMIC-OPENSHIFT-UTILS ##########
+ [[ -s /var/lib/jenkins/jobs/test_pull_request_origin_extended_conformance_install_update/workspace/activate ]]
+ source /var/lib/jenkins/jobs/test_pull_request_origin_extended_conformance_install_update/workspace/activate
++ export VIRTUAL_ENV=/var/lib/jenkins/origin-ci-tool/b4433dfdce6a5fba26d100d9416d78fd95716382
++ VIRTUAL_ENV=/var/lib/jenkins/origin-ci-tool/b4433dfdce6a5fba26d100d9416d78fd95716382
++ export PATH=/var/lib/jenkins/origin-ci-tool/b4433dfdce6a5fba26d100d9416d78fd95716382/bin:/sbin:/usr/sbin:/bin:/usr/bin
++ PATH=/var/lib/jenkins/origin-ci-tool/b4433dfdce6a5fba26d100d9416d78fd95716382/bin:/sbin:/usr/sbin:/bin:/usr/bin
++ unset PYTHON_HOME
++ export OCT_CONFIG_HOME=/var/lib/jenkins/jobs/test_pull_request_origin_extended_conformance_install_update/workspace/.config
++ OCT_CONFIG_HOME=/var/lib/jenkins/jobs/test_pull_request_origin_extended_conformance_install_update/workspace/.config
++ mktemp
+ script=/tmp/tmp.KroIrSOmRy
+ cat
+ chmod +x /tmp/tmp.KroIrSOmRy
+ scp -F ./.config/origin-ci-tool/inventory/.ssh_config /tmp/tmp.KroIrSOmRy openshiftdevel:/tmp/tmp.KroIrSOmRy
+ ssh -F ./.config/origin-ci-tool/inventory/.ssh_config -t openshiftdevel 'bash -l -c "timeout 600 /tmp/tmp.KroIrSOmRy"'
+ cd /data/src/github.com/openshift/aos-cd-jobs
+ pkg_name=origin
+ echo origin
+ echo 'openshift-ansible openshift-ansible-callback-plugins openshift-ansible-docs openshift-ansible-filter-plugins openshift-ansible-lookup-plugins openshift-ansible-playbooks openshift-ansible-roles'
++ cat OPENSHIFT_ANSIBLE_BUILT_VERSION
+ sudo python sjb/hack/determine_install_upgrade_version.py atomic-openshift-utils-3.8.1-1.git.0.d2eee04.el7.noarch --dependency_branch master
[ERROR] Can not determine install and upgrade version for the `atomic-openshift-utils` package
++ export status=FAILURE
++ status=FAILURE
+ set +o xtrace
########## FINISHED STAGE: FAILURE: INSTALL ANSIBLE ATOMIC-OPENSHIFT-UTILS [00h 01m 01s] ##########

0xmichalis · 2017-11-13T13:19:35Z

+ sudo python sjb/hack/determine_install_upgrade_version.py atomic-openshift-utils-3.8.1-1.git.0.d2eee04.el7.noarch --dependency_branch master
[ERROR] Can not determine install and upgrade version for the `atomic-openshift-utils` package

stevekuznetsov · 2017-11-13T15:42:14Z

Adding 3.7 repos to the test AMIs in openshift/origin-ci-tool@2453a17

Will need to build a new base AMI... which is currently blocked. Will figure out what we need to do there.

stevekuznetsov · 2017-11-13T15:54:39Z

New base AMIs blocked on #17268

jhadvig · 2017-11-13T15:59:40Z

FYI: The issue with the determine_install_upgrade_version.py is that we are building o-a with 3.8 tag

....       
atomic-openshift-utils.noarch                 3.5.145-1.git.0.e1e330f.el7                                                  rhel-7-server-ose-3.5-rpms            
atomic-openshift-utils.noarch                 3.6.8-1.git.0.8e26f8c.el7                                                    centos-paas-sig-openshift-origin36-rpms
atomic-openshift-utils.noarch                 3.6.153-1.el7                                                                centos-paas-sig-openshift-origin36-rpms
atomic-openshift-utils.noarch                 3.6.173.0.3-1.el7                                                            centos-paas-sig-openshift-origin36-rpms
atomic-openshift-utils.noarch                 3.6.173.0.60-1.el7                                                           centos-paas-sig-openshift-origin36-rpms
atomic-openshift-utils.noarch                 3.6.173.0.71-1.git.0.23eee8c.el7                                             rhel-7-server-ose-3.6-rpms            
atomic-openshift-utils.noarch                 3.8.1-1.git.0.0ef0a1e.el7                                                    openshift-ansible-local-release

so we are missing the 3.7 which would be installed. That's why the script fails

stevekuznetsov · 2017-11-13T16:00:06Z

Right -- adding the repos for 3.7 in the commit above adds the 3.7 version that we need to be present

0xmichalis · 2017-11-13T16:05:21Z

Adding 3.7 repos to the test AMIs in openshift/origin-ci-tool@2453a17

Is there any documentation about places that need to change once we cut a new branch?

stevekuznetsov · 2017-11-13T16:07:07Z

@Kargakis unfortunately no. We had a card on our board to tackle it, but there are a lot of moving parts

sdodson · 2017-11-13T21:40:24Z

That particular failure should be fixed now.

https://ci.openshift.redhat.com/jenkins/job/test_pull_request_openshift_ansible_extended_conformance_install_update/3467 at least got past installing the 3.7 installer as part of master install_upgrade

sjenning · 2017-11-14T00:52:40Z

It appears we get farther now. New fail point is

INSTALLER STATUS ***************************************************************
Initialization             : Complete
Health Check               : In Progress
	This phase can be restarted by running: playbooks/byo/openshift-checks/pre-install.yml



Failure summary:


  1. Hosts:    localhost
     Play:     OpenShift Health Checks
     Task:     Run health checks (install) - EL
     Message:  One or more checks failed
     Details:  check "docker_image_availability":
               One or more required container images are not available:
                   openshift/origin-deployer:v3.7.0,
                   openshift/origin-docker-registry:v3.7.0,
                   openshift/origin-haproxy-router:v3.7.0,
                   openshift/origin-pod:v3.7.0
               Checked with: skopeo inspect [--tls-verify=false] [--creds=<user>:<pass>] docker://<registry>/<image>
               Default registries searched: docker.io
               

The execution of "/usr/share/ansible/openshift-ansible/playbooks/byo/config.yml" includes checks designed to fail early if the requirements of the playbook are not met. One or more of these checks failed. To disregard these results,explicitly disable checks by setting an Ansible variable:
   openshift_disable_check=docker_image_availability
Failing check names are shown in the failure details above. Some checks may be configurable by variables if your requirements are different from the defaults; consult check documentation.
Variables can be set in the inventory or passed on the command line using the -e flag to ansible-playbook.
++ export status=FAILURE
++ status=FAILURE

stevekuznetsov · 2017-11-14T15:57:23Z

@smarterclayton please confirm but my understanding is that this job will remain broken until we actually cut a 3.7 release and push out the images, etc

sdodson · 2017-11-14T16:13:46Z

That's my understanding of the situation as well.

stevekuznetsov · 2017-11-14T16:26:00Z

Lowering priority as it is not a queue blocker

jsafrane · 2017-11-22T14:55:53Z

This has flaked three times in a row for #16538 that updates just /example directory.

stevekuznetsov · 2017-11-22T16:40:54Z

@jsafrane the jobs is 100% broken but is not required for any merges. Just running in the background.

stevekuznetsov · 2017-11-22T16:42:35Z

It's also not set up to run by default, so it's the /retest that is running it for your PR.

jsafrane · 2017-11-22T18:19:51Z

Oh, ok. I miss the tiny Required label in test results window we/they have in Kubernetes :-)

stevekuznetsov · 2017-11-22T19:15:53Z

Yeah, we considered it -- even k/k may get rid of it in the future :)

0xmichalis · 2017-11-23T01:47:00Z

We will likely end up with requiring all statuses that are present on a commit.

smarterclayton · 2018-01-24T18:44:10Z

This is not happening anymore.

sjenning assigned smarterclayton Nov 10, 2017

simo5 mentioned this issue Nov 11, 2017

Implement a way to time out tokens based on (in)activity #17161

Closed

0xmichalis added the priority/P0 label Nov 13, 2017

sjenning mentioned this issue Nov 13, 2017

check RepoTags len before accessing #17258

Merged

stevekuznetsov added priority/P1 and removed priority/P0 labels Nov 14, 2017

pweil- added component/internal-tools kind/bug Categorizes issue or PR as related to a bug. labels Nov 14, 2017

jsafrane mentioned this issue Nov 22, 2017

Add provisioner template for local storage #16538

Merged

php-coder mentioned this issue Dec 4, 2017

UPSTREAM: 56503: MustRunAsNonRoot should reject a pod if it has non-numeric USER #17512

Merged

dcbw mentioned this issue Dec 5, 2017

sdn: fix locking of egress DNS policy updates #17584

Merged

stevekuznetsov mentioned this issue Dec 5, 2017

CI: [ERROR] Can not determine install and upgrade version for the atomic-openshift-utils package #17594

Closed

ingvagabund mentioned this issue Dec 5, 2017

Extended.[k8s.io] Service endpoints latency should not be very high [Conformance] #15024

Closed

smarterclayton closed this as completed Jan 24, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

test_pull_request_origin_extended_conformance_install_update failing across the board #17263

test_pull_request_origin_extended_conformance_install_update failing across the board #17263

sjenning commented Nov 10, 2017

sjenning commented Nov 10, 2017

smarterclayton commented Nov 10, 2017

michaelgugino commented Nov 10, 2017

jupierce commented Nov 10, 2017

smarterclayton commented Nov 11, 2017 via email

simo5 commented Nov 11, 2017 •

edited by 0xmichalis

Loading

0xmichalis commented Nov 13, 2017 •

edited

Loading

stevekuznetsov commented Nov 13, 2017

stevekuznetsov commented Nov 13, 2017

jhadvig commented Nov 13, 2017

stevekuznetsov commented Nov 13, 2017

0xmichalis commented Nov 13, 2017

stevekuznetsov commented Nov 13, 2017

sdodson commented Nov 13, 2017

sjenning commented Nov 14, 2017

stevekuznetsov commented Nov 14, 2017

sdodson commented Nov 14, 2017

stevekuznetsov commented Nov 14, 2017

jsafrane commented Nov 22, 2017

stevekuznetsov commented Nov 22, 2017

stevekuznetsov commented Nov 22, 2017

jsafrane commented Nov 22, 2017

stevekuznetsov commented Nov 22, 2017

0xmichalis commented Nov 23, 2017

smarterclayton commented Jan 24, 2018

test_pull_request_origin_extended_conformance_install_update failing across the board #17263

test_pull_request_origin_extended_conformance_install_update failing across the board #17263

Comments

sjenning commented Nov 10, 2017

sjenning commented Nov 10, 2017

smarterclayton commented Nov 10, 2017

michaelgugino commented Nov 10, 2017

jupierce commented Nov 10, 2017

smarterclayton commented Nov 11, 2017 via email

simo5 commented Nov 11, 2017 • edited by 0xmichalis Loading

0xmichalis commented Nov 13, 2017 • edited Loading

stevekuznetsov commented Nov 13, 2017

stevekuznetsov commented Nov 13, 2017

jhadvig commented Nov 13, 2017

stevekuznetsov commented Nov 13, 2017

0xmichalis commented Nov 13, 2017

stevekuznetsov commented Nov 13, 2017

sdodson commented Nov 13, 2017

sjenning commented Nov 14, 2017

stevekuznetsov commented Nov 14, 2017

sdodson commented Nov 14, 2017

stevekuznetsov commented Nov 14, 2017

jsafrane commented Nov 22, 2017

stevekuznetsov commented Nov 22, 2017

stevekuznetsov commented Nov 22, 2017

jsafrane commented Nov 22, 2017

stevekuznetsov commented Nov 22, 2017

0xmichalis commented Nov 23, 2017

smarterclayton commented Jan 24, 2018

simo5 commented Nov 11, 2017 •

edited by 0xmichalis

Loading

0xmichalis commented Nov 13, 2017 •

edited

Loading