Wednesday, 2020-08-05

*** fdaencar_ has quit IRC00:05
rlandycloudnull: still around?00:14
cloudnullyup00:14
cloudnullwhat's going on ?00:14
rlandycloudnull: container builds in rhos-16.2 fail with00:15
rlandy ERROR! the role 'tripleo-container-image-build' was not found in /tmp/tripleo47g9n6wp/roles:/root/.ansible/roles:/usr/share/ansible/roles:/etc/ansible/roles:/usr/share/openstack-tripleo-validations/roles:/tmp/tripleo47g9n6wp[00m00:15
rlandyso  ... here is what I see00:15
rlandy(they work fine on rhos-17)00:15
rlandyhttp://git.app.eng.bos.redhat.com/git/python-tripleoclient.git/tree/tripleoclient/utils.py?h=rhos-17.0-trunk-patches00:15
rlandyhttp://git.app.eng.bos.redhat.com/git/python-tripleoclient.git/tree/tripleoclient/utils.py?h=rhos-17.0-trunk-patches#n54500:16
rlandyvs00:16
rlandyhttp://git.app.eng.bos.redhat.com/git/python-tripleoclient.git/tree/tripleoclient/utils.py?h=rhos-16.2-trunk-patches#n28300:16
rlandyit looks like the ANSIBLE_ROLES_PATH is read from ^^00:16
rlandywhy is https://github.com/openstack/tripleo-ci/blob/master/roles/build-containers/templates/build.sh.j2#L47 not overriding it?00:17
rlandyor ...00:17
rlandyshould the 16.2 ANSIBLE_ROLES_PATH be updated to match 17?00:17
rlandy(where 17 is master and 16.2 is stable/train)00:17
rlandycloudnull: if it's too late for all this, we can pick it up in the morning00:18
* cloudnull looking 00:18
cloudnullit seems like the role is missing.00:19
* cloudnull still looking 00:19
cloudnullmaybe we're missing a review for the downstream 16.2 sync?00:19
rlandyit's in tripleo-ansible00:19
rlandyhttp://git.app.eng.bos.redhat.com/git/tripleo-ansible.git/tree/tripleo_ansible/roles?h=rhos-16.2-trunk-patches00:20
rlandy^^ role is there00:20
rlandytripleo-container-image-build00:20
cloudnullok00:20
cloudnullthen it must be the ANSIBLE_ROLES_PATH settings00:21
rlandyit should be installed in https://github.com/openstack/tripleo-ci/blob/master/roles/build-containers/tasks/venv_setup.yml#L4200:21
rlandyin venv00:21
rlandyso either ANSIBLE_ROLES_PATH never gets overriden00:21
rlandyor it needs to be updated to match master (17)00:21
cloudnullhowever, all tripleo-ansible roles should be installed under /usr/share/ansible/roles/00:21
rlandyhow does that installation happen?00:22
rlandyI see the install to venv00:22
cloudnulloh, its installed in a venv.00:22
cloudnullthe package puts the role in /usr/share/ansible/roles00:22
cloudnullin a venv it'd be something like $VENV_PATH/usr/share/ansible/roles00:23
rlandyhttps://github.com/openstack/tripleo-ci/blob/master/roles/build-containers/tasks/venv_setup.yml#L4400:23
rlandy^^ in venv00:23
* cloudnull thought we had some of that logic somewhere already ?00:23
rlandyhttp://git.app.eng.bos.redhat.com/git/tripleo-ansible.git/tree/setup.cfg?h=rhos-16.2-trunk-patches#n2300:24
rlandyif we do a package install00:24
rlandyha - so we install tripleo-ansible ... looking00:25
cloudnull++ that bit will transfer the roles00:25
rlandy | LOOP [Install Buildah and ansible]00:26
cloudnullspecifically https://github.com/openstack/tripleo-ansible/blob/stable/train/setup.cfg#L23-L2700:26
rlandybut not tripleo-ansible00:26
cloudnullhowever, the usr/ path is padded within a venv00:26
rlandyright - so - we could install tripleo-ansible ...00:26
rlandyfrom package but we're doing a source install00:27
rlandyand  ANSIBLE_ROLES_PATH should be overriden to read the venv00:27
rlandybut I can test this with a package install00:27
cloudnull++00:28
* cloudnull looking for what we did in quickstart to solve this 00:28
cloudnullwe had some logic that was in one the scripts (which i thought was in quickstart) but i cant seem to find it00:31
rlandyI'll try the package install - we can touch base tomorrow00:34
cloudnulli just built a quick venv with tripleo-ansible to test. so to make it go you'll need to extend the role path with $VIRTUAL_ENV/share/ansible/roles00:34
cloudnullwhen the venv is activated, VIRTUAL_ENV should automatically be set .00:35
cloudnullyou may also wanted to extend the various plugins path too - - with $VIRTUAL_ENV/share/ansible/plugins/00:36
cloudnullbut i think just addeding the role path will make it go00:37
openstackgerritRonelle Landy proposed openstack/tripleo-ci master: WIP: Testing change of roles path  https://review.opendev.org/74478600:40
cloudnullrlandy nice ^00:42
cloudnullI'm going to take off, lets sync up tomorrow ?00:42
rlandycloudnull: yep - have a good night00:45
cloudnullyou too00:45
*** rlandy has quit IRC00:50
openstackgerritRonelle Landy proposed openstack/tripleo-ci master: WIP: Testing change of roles path  https://review.opendev.org/74478600:58
*** fmount|afk has quit IRC01:09
*** fmount has joined #tripleo01:09
*** gthiemonge has quit IRC01:24
*** gthiemonge has joined #tripleo01:24
*** rfolco has quit IRC01:39
*** ekultails has quit IRC01:57
*** d34dh0r53 has quit IRC02:01
*** d34dh0r53 has joined #tripleo02:04
openstackgerritChandan Kumar (raukadah) proposed openstack/ansible-role-collect-logs master: Added sova pattern for failed containers  https://review.opendev.org/74468802:13
*** apetrich has quit IRC02:14
*** ramishra has joined #tripleo02:24
openstackgerritChandan Kumar (raukadah) proposed openstack/tripleo-quickstart master: [DNM] ironic full coverage  https://review.opendev.org/74481902:25
*** psahoo has joined #tripleo02:39
openstackgerritChandan Kumar (raukadah) proposed openstack/ansible-role-collect-logs master: Added sova pattern for failed containers  https://review.opendev.org/74468802:55
*** rcernin has joined #tripleo03:07
*** skramaja has joined #tripleo03:21
*** psachin has joined #tripleo03:34
*** skramaja has quit IRC03:36
*** skramaja has joined #tripleo03:36
*** tkajinam has quit IRC03:51
*** tkajinam has joined #tripleo03:52
*** ramishra has quit IRC04:16
*** jfrancoa has joined #tripleo04:30
*** udesale has joined #tripleo04:33
*** evrardjp has quit IRC04:33
*** evrardjp has joined #tripleo04:33
*** ykarel|away has joined #tripleo04:43
*** ramishra has joined #tripleo04:44
*** ykarel|away is now known as ykarel04:46
openstackgerritRonelle Landy proposed openstack/tripleo-ci master: WIP: Testing change of roles path  https://review.opendev.org/74478604:48
openstackgerritMerged openstack/tripleo-common stable/train: Handle redirects for blobs better  https://review.opendev.org/74446704:51
*** saneax has joined #tripleo04:52
*** raukadah is now known as chkumar|rover04:54
*** ratailor has joined #tripleo04:56
*** ratailor has quit IRC04:59
*** ratailor has joined #tripleo04:59
*** lbragstad_ has joined #tripleo05:04
*** rh-jelabarre has quit IRC05:05
*** lbragstad has quit IRC05:07
*** marios has joined #tripleo05:08
*** marios is now known as marios|ruck05:08
*** waleedm has joined #tripleo05:12
*** waleedm_ has joined #tripleo05:12
*** psahoo has quit IRC05:23
*** ykarel has quit IRC05:24
*** ykarel has joined #tripleo05:29
*** lmiccini has joined #tripleo05:29
*** waleedm_ has quit IRC05:37
*** waleedm__ has joined #tripleo05:37
*** skramaja_ has joined #tripleo05:39
*** tremble has quit IRC05:39
*** skramaja has quit IRC05:39
openstackgerritCédric Jeanneret (Tengu) proposed openstack/tripleo-ansible master: Fix podman version  https://review.opendev.org/74473205:40
Tengucloudnull, mwhahaha -^ this is probably the easiest, cleanest way for this issue.05:41
*** udesale_ has joined #tripleo05:45
*** udesale has quit IRC05:47
*** psahoo has joined #tripleo05:50
openstackgerrityatin proposed openstack/tripleo-quickstart master: [Ussuri/Train] ansible-2.9.11 testing  https://review.opendev.org/72221806:14
*** skramaja_ has quit IRC06:19
*** skramaja has joined #tripleo06:19
*** udesale_ has quit IRC06:24
*** maciejjozefczyk has joined #tripleo06:28
openstackgerritRabi Mishra proposed openstack/tripleo-common master: Don't pass dry_run to image upload tasks  https://review.opendev.org/74482406:31
openstackgerrityatin proposed openstack/tripleo-common stable/train: [DNM] Test ansible-2.9.11 Train  https://review.opendev.org/74482506:31
*** jcapitao has joined #tripleo06:32
openstackgerrityatin proposed openstack/tripleo-common stable/ussuri: [DNM] Test ansible-2.9.11 Ussuri  https://review.opendev.org/74482606:34
*** marios|ruck has quit IRC06:40
*** marios has joined #tripleo06:44
*** marios is now known as marios|ruck06:44
*** udesale has joined #tripleo06:52
*** rcernin has quit IRC06:58
*** rcernin_ has joined #tripleo06:59
*** ntt_ has joined #tripleo07:00
*** rcernin_ has quit IRC07:05
*** apetrich has joined #tripleo07:06
*** rcernin has joined #tripleo07:06
*** slaweq has joined #tripleo07:07
*** tremble has joined #tripleo07:08
*** skramaja has quit IRC07:13
*** psahoo has quit IRC07:21
openstackgerritCédric Jeanneret (Tengu) proposed openstack/tripleo-validations master: New validation: Ensure sVirt is active  https://review.opendev.org/74483507:28
*** bogdando has joined #tripleo07:30
openstackgerritRabi Mishra proposed openstack/python-tripleoclient master: Don't build image upload tasks when dry_run is True  https://review.opendev.org/74483607:33
openstackgerritRabi Mishra proposed openstack/tripleo-common master: Don't build image upload tasks when dry_run=True  https://review.opendev.org/74482407:34
*** psahoo has joined #tripleo07:34
dciabrinmarios|ruck, morning :) https://review.opendev.org/#/c/725782/ to fix half of tripleo-ci-centos-8-standalone-upgrade problems07:37
*** tosky has joined #tripleo07:38
marios|ruckthanks dciabrin will check in a bit07:39
dciabrinmarios|ruck, thx!07:39
*** suuuper has joined #tripleo07:45
*** psahoo has quit IRC07:52
*** jpena|off is now known as jpena07:54
*** skramaja has joined #tripleo07:56
*** ntt_ has quit IRC07:58
*** ntt_ has joined #tripleo07:58
*** jpich has joined #tripleo08:00
*** gfidente|afk is now known as gfidente08:03
*** dtantsur|afk is now known as dtantsur08:04
*** psahoo has joined #tripleo08:08
*** thegodfather has joined #tripleo08:09
*** pkopec has joined #tripleo08:11
openstackgerritDamien Ciabrini proposed openstack/tripleo-quickstart-extras master: Pass proper VIPs on Standalone  https://review.opendev.org/72578208:12
*** lucasagomes has joined #tripleo08:15
*** xek has joined #tripleo08:21
*** tosky has quit IRC08:26
*** tosky has joined #tripleo08:27
*** thegodfather has quit IRC08:27
openstackgerritBernard Cafarelli proposed openstack/tripleo-validations stable/ussuri: Fix neutron_sanity_check for ML2/OVS overcloud  https://review.opendev.org/74485608:43
openstackgerritBernard Cafarelli proposed openstack/tripleo-validations stable/train: Fix neutron_sanity_check for ML2/OVS overcloud  https://review.opendev.org/74485708:43
*** holser has joined #tripleo08:48
*** derekh has joined #tripleo08:51
*** sshnaidm|afk is now known as sshnaidm08:56
*** devep has joined #tripleo08:57
*** devep has quit IRC08:59
*** devep has joined #tripleo08:59
openstackgerritDaniel Bengtsson proposed openstack/python-tripleoclient master: Stop configuring install_command in tox.  https://review.opendev.org/69350208:59
*** rcernin has quit IRC09:04
openstackgerritSagi Shnaidman proposed openstack/tripleo-ansible stable/ussuri: Fix introspection restart with unlocking node  https://review.opendev.org/74486009:04
*** rcernin has joined #tripleo09:04
*** devep_ has joined #tripleo09:04
*** rcernin has quit IRC09:05
*** rcernin has joined #tripleo09:05
openstackgerritZhu Sheng Li proposed openstack/tripleo-heat-templates stable/train: Add non-string value support for CephAnsibleEnvironmentVariables  https://review.opendev.org/74486109:06
*** devep__ has joined #tripleo09:07
*** devep has quit IRC09:07
*** devep__ has quit IRC09:07
*** devep_ has quit IRC09:09
openstackgerritCédric Jeanneret (Tengu) proposed openstack/tripleo-validations master: New validation: Ensure sVirt is active  https://review.opendev.org/74483509:17
openstackgerrityatin proposed openstack/tripleo-quickstart-extras master: Honour standalone updates too for container-prep  https://review.opendev.org/74486409:18
*** jpich has quit IRC09:26
*** jpich has joined #tripleo09:27
*** ntt__ has joined #tripleo09:28
*** ntt_ has quit IRC09:28
openstackgerrityatin proposed openstack/tripleo-quickstart master: Switch Standalone environment to centos8  https://review.opendev.org/74391909:34
mrungehi there, anyone up for a quick review? https://review.opendev.org/#/c/734009/09:39
mrungealready has a +209:39
openstackgerritCédric Jeanneret (Tengu) proposed openstack/tripleo-validations master: New validation: Ensure sVirt is active  https://review.opendev.org/74483509:52
openstackgerritDamien Ciabrini proposed openstack/tripleo-quickstart-extras master: Pass proper VIPs on Standalone  https://review.opendev.org/72578210:14
*** tkajinam has quit IRC10:15
*** marios|ruck is now known as marios|ruck|afk10:16
*** rcernin has quit IRC10:23
*** xek has quit IRC10:24
openstackgerritChandan Kumar (raukadah) proposed openstack/ansible-role-collect-logs master: Added sova pattern for failed containers  https://review.opendev.org/74468810:25
*** marios|ruck|afk is now known as marios|ruck10:33
openstackgerritamolkahat proposed openstack/tripleo-quickstart master: Added ironic specific variables in the featureset060 file.  https://review.opendev.org/73170410:34
openstackgerritEmma Foley proposed openstack/tripleo-heat-templates master: [WIP][collectd][ansible] Add template to deploy collectd using ansible  https://review.opendev.org/73710010:34
openstackgerritamolkahat proposed openstack/openstack-tempest-skiplist master: Adding ironic chessis_node_list test to skiplist.  https://review.opendev.org/74488210:41
openstackgerritamolkahat proposed openstack/tripleo-ci master: [DNM] Making tempest_public_net_name variable ovarridable.  https://review.opendev.org/72387610:43
openstackgerritLukas Bezdicka proposed openstack/tripleo-upgrade master: WIP: test vm_ip cleanup  https://review.opendev.org/74429310:45
openstackgerritLukas Bezdicka proposed openstack/tripleo-upgrade master: WIP: upgrade converge cleanup  https://review.opendev.org/74488410:45
openstackgerritSergii Golovatiuk proposed openstack/tripleo-heat-templates stable/ussuri: Fix HA resource restart when no replicas are running  https://review.opendev.org/74467610:51
openstackgerritMarios Andreou proposed openstack/tripleo-ci master: Make centos-8 standalone-upgrade-ussuri voting and gating  https://review.opendev.org/74241810:54
openstackgerritMarios Andreou proposed openstack/tripleo-ci master: Centos8 undercloud upgrade - rename & split ussuri voting/gating  https://review.opendev.org/74241910:54
openstackgerritMarios Andreou proposed openstack/tripleo-ci master: Make train centos-8 scen0 overcloud update job voting/gating  https://review.opendev.org/74242010:54
openstackgerritCédric Jeanneret (Tengu) proposed openstack/tripleo-validations master: New validation: Ensure sVirt is active  https://review.opendev.org/74483511:02
*** lmiccini has quit IRC11:04
openstackgerritMerged openstack/tripleo-ci master: Add buildimage hardened job to buildimages project template  https://review.opendev.org/74266211:11
openstackgerritCédric Jeanneret (Tengu) proposed openstack/tripleo-validations master: Correct role name in generated playbook  https://review.opendev.org/74488611:15
openstackgerritCédric Jeanneret (Tengu) proposed openstack/tripleo-validations master: Correct role name in generated playbook  https://review.opendev.org/74488611:17
openstackgerritCédric Jeanneret (Tengu) proposed openstack/tripleo-validations master: New validation: Ensure sVirt is active  https://review.opendev.org/74483511:17
*** lmiccini has joined #tripleo11:19
*** raildo has joined #tripleo11:21
*** psahoo has quit IRC11:27
openstackgerritCédric Jeanneret (Tengu) proposed openstack/tripleo-validations master: Correct role name in generated playbook  https://review.opendev.org/74488611:30
Tenguok, last spam hopefully -.-. doing 10 things in // is bad.11:32
openstackgerritCédric Jeanneret (Tengu) proposed openstack/tripleo-validations master: New validation: Ensure sVirt is active  https://review.opendev.org/74483511:32
*** jpena is now known as jpena|lunch11:39
*** xek has joined #tripleo11:40
*** jpich has quit IRC11:41
*** jcapitao is now known as jcapitao_lunch11:42
*** psahoo has joined #tripleo11:42
*** jpich has joined #tripleo11:42
*** rfolco has joined #tripleo11:51
*** ekultails has joined #tripleo11:53
*** aedc has joined #tripleo11:56
*** aedc has quit IRC11:56
*** xek has quit IRC11:56
*** pcaruana has quit IRC11:57
*** rcernin has joined #tripleo11:57
*** rh-jelabarre has joined #tripleo11:58
*** rh-jelabarre has quit IRC11:58
*** rh-jelabarre has joined #tripleo11:59
*** rcernin has quit IRC12:02
*** rlandy has joined #tripleo12:03
*** fdaencar_ has joined #tripleo12:04
*** pcaruana has joined #tripleo12:12
openstackgerritCédric Jeanneret (Tengu) proposed openstack/tripleo-heat-templates stable/ussuri: Modify how libvirt related containers use SELinux  https://review.opendev.org/73947112:12
openstackgerritCédric Jeanneret (Tengu) proposed openstack/tripleo-heat-templates stable/train: Modify how libvirt related containers use SELinux  https://review.opendev.org/73948112:12
openstackgerritCédric Jeanneret (Tengu) proposed openstack/tripleo-heat-templates master: Modify how libvirt related containers use SELinux  https://review.opendev.org/73617312:12
openstackgerritZhu Sheng Li proposed openstack/tripleo-heat-templates stable/train: Add non-string value support for CephAnsibleEnvironmentVariables  https://review.opendev.org/74486112:15
*** suuuper has quit IRC12:17
*** suuuper has joined #tripleo12:20
*** psahoo has quit IRC12:22
openstackgerritamolkahat proposed openstack/openstack-tempest-skiplist master: Adding ironic chessis_node_list test to skiplist.  https://review.opendev.org/74488212:23
*** derekh has quit IRC12:24
openstackgerritamolkahat proposed openstack/openstack-tempest-skiplist master: Adding ironic chessis_node_list test to skiplist.  https://review.opendev.org/74488212:25
openstackgerritamolkahat proposed openstack/openstack-tempest-skiplist master: Adding ironic chessis_node_list test to skiplist.  https://review.opendev.org/74488212:30
*** psahoo has joined #tripleo12:33
*** ade_lee has quit IRC12:33
*** ade_lee has joined #tripleo12:34
*** udesale has quit IRC12:37
*** udesale has joined #tripleo12:38
cloudnullmornings all12:46
*** jpena|lunch is now known as jpena12:46
cloudnullrlandy when you're up and about, and fully caffeinated (no rush) let me know. I'd be good to sync up regarding those 16.x VENV issues you were seeing last night.12:47
cloudnullTengu looking12:49
*** dprince has joined #tripleo12:49
rlandycloudnull: hey  - so the symlink hack worked12:51
rlandyalthough it shouldn't be needed12:52
rlandyI'm updating the 16.2 hash to make sure I got the latest rebased patches12:52
rlandyalso has to set selinux to premissive - was not needed for 17 either ...12:53
rlandybut on the upside, the build step is still running :)12:53
openstackgerritBogdan Dobrelya (bogdando) proposed openstack/tripleo-common master: Less workers, longer exp. failback upon reconnects  https://review.opendev.org/74370412:53
rlandywill ping you with latest results12:53
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/python-tripleoclient stable/train: Rename UpgradeConvergeOvercloud into UpgradeConverge.  https://review.opendev.org/74091312:54
bogdandomwhahaha: ^^12:55
bogdandomwhahaha: https://bugs.launchpad.net/tripleo/+bug/1889122/comments/1812:59
openstackLaunchpad bug 1889122 in tripleo "mirror timeouts in upstream causing undercloud and standalone failures" [Critical,Triaged] - Assigned to chandan kumar (chkumar246)12:59
bogdandooops wrong patch linked12:59
*** rcernin has joined #tripleo13:01
openstackgerritLukas Bezdicka proposed openstack/tripleo-upgrade master: Remove skip ceph-validations and --playbook for system_upgrade.  https://review.opendev.org/74449613:01
openstackgerritLukas Bezdicka proposed openstack/tripleo-upgrade master: Evacuate computes before RHEL upgrade  https://review.opendev.org/74395113:01
openstackgerritLukas Bezdicka proposed openstack/tripleo-upgrade master: Refactor FFWD run to run on 1/3rd of controlplane  https://review.opendev.org/74395213:01
*** jcapitao_lunch is now known as jcapitao13:06
*** derekh has joined #tripleo13:09
openstackgerritChandan Kumar (raukadah) proposed openstack/ansible-role-collect-logs master: Added sova pattern for failed containers  https://review.opendev.org/74468813:11
*** iurygregory has quit IRC13:12
*** rcernin has quit IRC13:15
*** xek has joined #tripleo13:18
openstackgerritCédric Jeanneret (Tengu) proposed openstack/tripleo-validations master: New validation: Ensure sVirt is active  https://review.opendev.org/74483513:18
openstackgerritCédric Jeanneret (Tengu) proposed openstack/tripleo-validations master: Add new dependency for the tests: lxml  https://review.opendev.org/74490313:18
*** iurygregory has joined #tripleo13:18
bogdandomwhahaha: I think https://review.opendev.org/#/c/743629/12/tripleo_common/image/image_uploader.py@294 (and https://review.opendev.org/#/c/743629/12/tripleo_common/image/image_uploader.py@1810 ?) should also "attempt a single re-authentication if the initial request                                                                       |   fails with a 401", see13:20
bogdandohttp://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_d11/743260/3/gate/tripleo-ci-centos-7-standalone-upgrade-train/d11c9ef/logs/undercloud/home/zuul/standalone_deploy.log - there is "unescaped" 401 in the _inspect13:20
openstackgerritwes hayutin proposed openstack/puppet-tripleo stable/train: Fix up cluster_setup_extras usage  https://review.opendev.org/74419213:21
bogdandoor it spams reauth attemts full ahead13:21
*** udesale has quit IRC13:21
*** udesale has joined #tripleo13:22
rlandycloudnull: ok - we did better this time ... build 108 out of 109 containers ... https://sf.hosted.upshift.rdu2.redhat.com/logs/02/207802/12/check/tripleo-build-containers-ubi-8-internal-rhel-8-rhos-16.2/a57518d/logs/build.log13:25
openstackgerritMerged openstack/tripleo-ansible stable/ussuri: Fix introspection restart with unlocking node  https://review.opendev.org/74486013:30
*** ayoung has joined #tripleo13:32
mwhahahabogdando: the problem with the whole thing is the stupid 5 minute timeout on requests, unfortunately we don't get the luxury to just do a single one.  ideally we would be able to limit the tokens but the scope is per image so we end up having to auth alot13:33
mwhahahabogdando: we need less jobs, not to make this slower13:33
bogdandomwhahaha: sure thing, but that's a different aspect13:34
bogdandothat I'm talking about is too much reconnects upon 40113:34
bogdandothat needs to be fixed13:34
mwhahahano because we're getting nailed because the volume of requests from the mirror13:34
mwhahahawe're partially hurting ourselves by funneling through a single ip13:35
mwhahaha429 may not just be us13:35
*** raildo has quit IRC13:35
bogdandohm, shouldn't re-auth happen just not too often to not trigger rate-limiting?13:35
mwhahahawho says it's re-auth13:35
bogdandosee the log13:35
bogdandoor re _inspect13:35
mwhahahai will eventually13:35
bogdandowhatever13:35
mwhahahabut i think ti's not as simple as you think13:35
bogdandothat's for sure :)13:36
cloudnullrlandy woot!13:36
bogdandojust noticed there is a comment related13:36
* cloudnull looks at logs what 110 wasn't built 13:36
cloudnull**10913:36
bogdandomwhahaha: https://opendev.org/openstack/tripleo-common/src/branch/master/tripleo_common/image/image_uploader.py#L30413:36
bogdandoso may be similar approach is needed every time we call that check_status and expect 40113:37
bogdando?13:37
bogdandoor at least retry less often for that expecptional case13:37
*** raildo has joined #tripleo13:38
cloudnullrlandy https://pasted.tech/pastes/c8d6d0335aaa78d8fcfeb358c82662a821cfd739.raw -- there was a missing layer?13:39
bogdandomwhahaha: sorry If I'm just telling nonsense :D13:39
bogdandosome code there still hasn't been wrapped around my head13:40
rlandycloudnull: how does that happen - that we miss a layer? I'm running again with an updated tripleo-ci-testing hash13:40
mwhahahabogdando: i'll look later after i catch up with things13:40
bogdandok thanks13:40
cloudnullI'm not quite sure why a layer would be missing.13:41
cloudnullthat's pretty suspect.13:41
rlandychecking build tree13:41
*** ratailor has quit IRC13:42
cloudnullhttps://sf.hosted.upshift.rdu2.redhat.com/logs/02/207802/12/check/tripleo-build-containers-ubi-8-internal-rhel-8-rhos-16.2/a57518d/logs/buildah-builds/container-builds/856761bf-8e84-4b0e-9dcd-bc1161983a3e/base/os/designate-base/designate-producer/designate-producer-build.log - the container where it choked has nothing eventful in the logs.13:42
*** pcaruana has quit IRC13:42
*** raildo has quit IRC13:43
*** raildo has joined #tripleo13:43
rlandycloudnull: I'm rerunning with an updated hash - let's see if it dies again13:43
rlandyalso I know there are some outstanding stable/train patches - but I can't use them until they are merged and rebased downstream13:44
*** raildo has quit IRC13:44
*** raildo has joined #tripleo13:44
cloudnullI think most of them have been merged upstream (checks)13:46
*** raildo_ has joined #tripleo13:46
cloudnulldo you know if we're doing a bulk import for 16.2 or should we cherry-pick some of these extra changes?13:47
*** raildo_ has quit IRC13:50
Tengucloudnull: would you be able to check https://review.opendev.org/#/c/743990/ when you have some time? owalsh - would be nice having you on that one as well :)13:51
openstackgerritSagi Shnaidman proposed openstack/ansible-role-collect-logs master: Add patterns for sova  https://review.opendev.org/74466913:52
*** pcaruana has joined #tripleo13:54
*** TrevorV has joined #tripleo13:56
cloudnullTengu on it14:03
Tenguthanks :)14:04
owalshcloudnull, Tengu: could you hold on that for a minute?14:05
cloudnullowalsh ++14:05
*** xek has quit IRC14:06
mwhahahabogdando: so i think a better solution is that if we get a 429 to pause a long time and retry slowly14:06
mwhahahawe don't respect 429s14:06
bogdandomwhahaha: yeah, something like that!14:06
mwhahahaso they just trigger a flood of retries14:06
bogdandobut in the logs there was 40114:06
mwhahahayea cause teh reauth keeps trying i think14:07
bogdandoah14:07
mwhahahai'm going to dig a bit more into the logs and throw up a patch today14:07
bogdandomwhahaha.TNA.guru++ (like TLA but for numbers)14:07
Tenguowalsh: ah, thanks for the leftover - missed it -.-.14:07
*** dprince has quit IRC14:08
mwhahahabogdando: this also comes from having to lookup tags. i think in the upstream they are universal (downstream not so much) so we likely could cheat a bit upstream and skip this part and only discover once14:09
mwhahahaor not at all14:09
mwhahahaweshay|ruck: are container tags consistent upstream for a given promotion? or can they vary14:09
weshay|ruckmwhahaha, they are all tagged w/ the md5 sum of the dlrn_repo14:13
mwhahahaweshay|ruck: can we cheat and not have to do discovery upstream?14:13
*** skramaja has quit IRC14:13
mwhahahae.g. get a hash and put it in the container-image-prepare.yaml14:13
weshay|ruckmwhahaha, they should also be tagged w/ current-tripleo14:13
mwhahahaand drop tags_from_label14:13
* weshay|ruck checks14:13
mwhahahaok i know the bug then14:14
mwhahahawe specify a tag and tag_from_label14:14
mwhahahatag_From_label triggers inspection of tags14:14
* mwhahaha shall fix14:14
mwhahahai hit this downstream all the time14:14
mwhahahaif tag is specify, tag_from_label should be ignored14:14
mwhahahaso much for being productive this week14:15
weshay|rucktripleomaster/centos-binary-tripleoclient:current-tripleo14:15
weshay|ruckDIGEST:sha256:0a2586238bdb96eb531d9b631e949b122f727f44cf2cd6aeaeef4286ac593f8614:15
weshay|rucktripleomaster/centos-binary-tripleoclient:1454d01ab7b923258838c6f1863a6c8014:15
weshay|ruckDIGEST:sha256:0a2586238bdb96eb531d9b631e949b122f727f44cf2cd6aeaeef4286ac593f8614:15
weshay|ruckhttps://trunk.rdoproject.org/centos8-master/current-tripleo/delorean.repo.md514:16
mwhahahahttp://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_d11/743260/3/gate/tripleo-ci-centos-7-standalone-upgrade-train/d11c9ef/logs/undercloud/home/zuul/containers-prepare-parameters.yaml14:16
weshay|ruckmwhahaha, not sure I totally grok.. yet.. did I answer your question?14:17
mwhahahayea you did14:17
*** waleedm__ has quit IRC14:20
*** waleedm has quit IRC14:20
openstackgerritMaciej Józefczyk proposed openstack/tripleo-ansible master: Set the OpenFlow protocol while upgrading OVS from 2.11  https://review.opendev.org/74491014:23
*** sshnaidm is now known as sshnaidm|afk14:24
openstackgerritCédric Jeanneret (Tengu) proposed openstack/tripleo-common master: Correct the inventory generation  https://review.opendev.org/74399014:25
Tenguowalsh: updated - just tested tox + live env, seems to work as expected. hopefully this is the last time I have to dig in this thing X(14:25
bogdandomwhahaha:  > if tag is specify, tag_from_label should be ignored14:25
bogdandomakes absolute sense14:25
mwhahahapretty much. i also spotted another issue14:26
mwhahahawe use 16 threads for the label lookups14:26
mwhahahathat needs to be reduced or limited for things liek docker.io/quay.io14:26
* mwhahaha is patching14:27
openstackgerritCédric Jeanneret (Tengu) proposed openstack/tripleo-common master: Correct the inventory generation  https://review.opendev.org/74399014:28
Tenguowalsh: ah, and moving the block doesn't do anything. I was probably lost in the unit-testing -^14:28
owalshTengu: ack, was worried that could have issues for the special keys such as _meta and overcloud but probably just evals to false14:29
openstackgerritAlex Schultz proposed openstack/tripleo-common master: Reduce workers for label lookups  https://review.opendev.org/74491314:29
mwhahahabogdando: -^ will help, i'll fix the tags thing in a different patch14:30
Tenguowalsh: yeah - basically no effect, according to the unit tests14:30
rlandycloudnull: well .. it's reproducible ... :rhel-binary-designate-base-working-container-83: error extracting layer "76fdb57b1ddb3c1742b97909522f282b29e8a9424ceeb7083ed3df555f3e699a": no such file or director ... https://sf.hosted.upshift.rdu2.redhat.com/logs/02/207802/12/check/tripleo-build-containers-ubi-8-internal-rhel-8-rhos-16.2/a487b1c/logs/build.log14:30
Tenguowalsh: that damned vars() function and its vars representation.....14:30
cloudnullrlandy well thats no good14:31
owalshTengu: yea errr probably my bad14:31
Tenguowalsh: (won't say but.... yeah ;) that's why I wanted you to have a look ;))14:31
owalshI blame python and ducks14:32
Tenguhehehe14:32
*** redrobot has joined #tripleo14:32
cloudnullrlandy are we able to hold an environment where this is happening?14:33
Tenguowalsh: I was first like "what's going on, why is it not presenting that container_cli.... oh... oooohhhh nasty python" :)14:33
cloudnullsomething we can login to14:33
openstackgerritMaciej Józefczyk proposed openstack/tripleo-ansible master: Set the OpenFlow protocol while upgrading OVS from 2.11  https://review.opendev.org/74491014:33
Tenguanyway. corrected.14:33
rlandycloudnull: yep - will get that14:33
owalshTengu++ ack14:34
*** lmiccini has quit IRC14:34
owalshTengu: ansible_ssh_user being set per service is really noisy too, but lets deal with that in a follow up patch14:36
Tenguowalsh: yeah - also my thought, but I didn't have the courage to dig in it14:36
owalshTengu: just always been that way AFAICT14:37
Tenguyeah, probably, or at least should have been ;)14:37
Tengugetting the "container_cli" is THE thing, since many validations rely on it.14:37
Tenguthat's all I care for right now - and getting it in train and osp-16.1.2 asap14:38
owalshTengu: ack, LGTM14:38
openstackgerritCédric Jeanneret (Tengu) proposed openstack/tripleo-validations master: Add new dependency for the tests: lxml  https://review.opendev.org/74490314:44
Tengugrumpf...14:44
openstackgerritAlex Schultz proposed openstack/tripleo-common master: Stop doing tag_from_label lookup with tag  https://review.opendev.org/74491814:44
mwhahahabogdando: -^14:44
bogdandofruitful, thanks!14:46
*** lmiccini has joined #tripleo14:49
*** xek has joined #tripleo14:54
openstackgerritCédric Jeanneret (Tengu) proposed openstack/tripleo-validations master: Updated "overcloud" to "allovercloud" group  https://review.opendev.org/74492214:55
*** psachin has quit IRC14:57
weshay|ruckmwhahaha, should we get your container lookup/upload patches cherry-picked now for ussuri / train perhaps get more results?14:57
mwhahahai can pre-emptively cp once they pass master14:57
mwhahahai don't want to unless i know they are good14:58
*** ayoung has quit IRC15:11
*** ayoung has joined #tripleo15:12
*** toure|gone is now known as toure15:19
rlandycloudnull: pls send me your keys - I have a node held running container builds15:20
*** waleedm has joined #tripleo15:20
*** waleedm__ has joined #tripleo15:20
*** ykarel is now known as ykarel|away15:21
*** ayoung has quit IRC15:22
*** ayoung has joined #tripleo15:24
cloudnullrlandy - https://github.com/cloudnull.keys15:24
rlandycloudnull: thanks - see pvt15:25
*** chkumar|rover is now known as raukadah15:26
*** tosky has quit IRC15:31
*** lmiccini has quit IRC15:31
*** aedc has joined #tripleo15:34
*** aedc has quit IRC15:36
*** pcaruana has quit IRC15:38
*** tremble has quit IRC15:38
*** lbragstad_ is now known as lbragstad15:42
*** suuuper has quit IRC15:42
*** slaweq_ has joined #tripleo15:45
*** slaweq has quit IRC15:45
*** slaweq_ is now known as slaweq15:45
openstackgerritwes hayutin proposed openstack/ansible-role-collect-logs master: add provision failures in deployment  https://review.opendev.org/74493115:49
*** jcapitao has quit IRC15:50
*** pcaruana has joined #tripleo15:51
*** ntt__ has quit IRC15:52
*** marios|ruck is now known as marios|out15:54
*** ayoung has quit IRC15:54
*** pkopec has quit IRC15:55
*** ayoung has joined #tripleo15:57
*** xek_ has joined #tripleo15:57
*** ykarel|away has quit IRC15:58
*** maciejjozefczyk has quit IRC15:59
*** xek has quit IRC16:00
*** psahoo has quit IRC16:00
*** lucasagomes has quit IRC16:00
openstackgerritJulia Kreger proposed openstack/tripleo-common master: Change default DHCP timeout for ironic-python-agent  https://review.opendev.org/74493916:05
openstackgerritJulia Kreger proposed openstack/tripleo-common master: Change default DHCP timeout for ironic-python-agent  https://review.opendev.org/74493916:06
*** bogdando has quit IRC16:08
*** jpena is now known as jpena|off16:12
*** jmasud has quit IRC16:14
weshay|ruckmwhahaha, https://review.opendev.org/#/c/744939/16:15
*** jmasud has joined #tripleo16:16
*** mcornea has joined #tripleo16:21
*** 18VABDA9G has joined #tripleo16:25
*** 92AAA3746 has joined #tripleo16:25
*** marios|out has quit IRC16:26
*** waleedm has quit IRC16:29
*** waleedm__ has quit IRC16:29
*** tosky has joined #tripleo16:29
*** udesale has quit IRC16:32
*** dtantsur is now known as dtantsur|afk16:39
*** maciejjozefczyk has joined #tripleo16:42
*** gfidente is now known as gfidente|afk16:44
*** mcornea has quit IRC16:57
*** derekh has quit IRC17:00
openstackgerritAlex Schultz proposed openstack/tripleo-common stable/ussuri: Reduce workers for label lookups  https://review.opendev.org/74495217:05
openstackgerritAlex Schultz proposed openstack/tripleo-common stable/ussuri: Stop doing tag_from_label lookup with tag  https://review.opendev.org/74495317:05
openstackgerritAlex Schultz proposed openstack/tripleo-common stable/train: Reduce workers for label lookups  https://review.opendev.org/74495417:06
openstackgerritAlex Schultz proposed openstack/tripleo-common stable/train: Stop doing tag_from_label lookup with tag  https://review.opendev.org/74495517:06
mwhahahaweshay|ruck: -^ pre-emptive backports17:06
weshay|ruckyou rock17:06
weshay|ruckmwhahaha++17:06
*** jpich has quit IRC17:16
*** sshnaidm|afk is now known as sshnaidm17:19
openstackgerritSagi Shnaidman proposed openstack/ansible-role-collect-logs master: Added sova pattern for failed containers  https://review.opendev.org/74468817:37
openstackgerritKevin Carter (cloudnull) proposed openstack/tripleo-heat-templates master: [WIP] Remove YAQL queries for RabbitCookie  https://review.opendev.org/74495717:42
openstackgerritLukas Bezdicka proposed openstack/tripleo-upgrade stable/ussuri: Remove skip ceph-validations and --playbook for system_upgrade.  https://review.opendev.org/74495917:55
openstackgerritLukas Bezdicka proposed openstack/tripleo-upgrade stable/train: Remove skip ceph-validations and --playbook for system_upgrade.  https://review.opendev.org/74496017:56
*** dciabrin has quit IRC17:57
*** dciabrin has joined #tripleo17:58
*** raildo has quit IRC18:03
*** raildo has joined #tripleo18:08
*** raildo_ has joined #tripleo18:11
*** raildo_ has quit IRC18:13
openstackgerritMerged openstack/tripleo-heat-templates stable/ussuri: Add CephAnsibleRepo warning to make this validation more flexible  https://review.opendev.org/74446918:14
openstackgerritMerged openstack/tripleo-upgrade master: Add BootParams service to all roles  https://review.opendev.org/74438818:14
*** rh-jelabarre has quit IRC18:16
*** rh-jelabarre has joined #tripleo18:16
*** ayoung has quit IRC18:20
*** ayoung has joined #tripleo18:23
*** aedc has joined #tripleo18:30
*** aedc has quit IRC18:30
openstackgerritMerged openstack/tripleo-heat-templates stable/train: [TRAIN-ONLY] Set the right container_client when set/unset noout  https://review.opendev.org/74401818:34
openstackgerritMerged openstack/tripleo-heat-templates master: Add support for Gnocchi NFS Backend  https://review.opendev.org/73400918:34
openstackgerritMerged openstack/puppet-tripleo stable/train: Make promote timeout configurable  https://review.opendev.org/74237618:34
openstackgerritMerged openstack/puppet-tripleo stable/train: Ensure post-save certmonger scripts target the right HA container  https://review.opendev.org/74236218:34
openstackgerritMerged openstack/tripleo-heat-templates master: Align kernel args for system upgrade using leapp  https://review.opendev.org/74262518:34
openstackgerritMerged openstack/tripleo-upgrade master: Remove skip ceph-validations and --playbook for system_upgrade.  https://review.opendev.org/74449618:34
openstackgerritMerged openstack/tripleo-heat-templates stable/ussuri: Fix HA resource restart when no replicas are running  https://review.opendev.org/74467618:34
openstackgerritMerged openstack/tripleo-common stable/train: Don't disable centos-rabbitmq-38 repo in base image  https://review.opendev.org/74261318:34
*** waleedm_ has joined #tripleo18:40
*** waleedm has joined #tripleo18:41
*** 92AAA3746 has quit IRC18:44
openstackgerritBrent Eagles proposed openstack/tripleo-ansible master: WIP: add unit tests for tripleo_ovs_package module  https://review.opendev.org/74383618:44
*** 18VABDA9G has quit IRC18:44
*** raildo has quit IRC18:46
*** raildo has joined #tripleo18:48
mwhahahaweshay|ruck: https://review.opendev.org/#/c/744913/ plz4me18:54
weshay|ruckmwhahaha, done18:55
openstackgerritSergii Golovatiuk proposed openstack/tripleo-ci master: Make ussuri scen0 multinode upgrade job voting and gating  https://review.opendev.org/74218218:56
weshay|ruckshort queue.. /me grabs lunch and will get these through..18:56
weshay|ruckholser, if you want do a depends on mwhahaha's patches18:57
*** stevebaker has joined #tripleo18:57
weshay|ruckfor train, ussuri18:57
holserweshay|ruck sure, if/when it fails18:59
holsernow I crossed fingers19:00
holser:D19:00
*** xek_ has quit IRC19:06
openstackgerritOliver Walsh proposed openstack/tripleo-heat-templates master: WIP: refactor nova db config  https://review.opendev.org/71855219:09
openstackgerritPaul Leimer proposed openstack/tripleo-heat-templates master: Mount libpod container volume into collectd container  https://review.opendev.org/74497719:25
*** tosky has quit IRC19:39
openstackgerritMerged openstack/tripleo-common stable/train: Add update and post update tasks to step generated file.  https://review.opendev.org/74414120:09
*** raildo has quit IRC20:40
*** raildo has joined #tripleo20:40
*** cfontain has joined #tripleo20:43
*** jidar has quit IRC20:51
*** jidar has joined #tripleo20:52
*** raildo has quit IRC20:57
*** maciejjozefczyk has quit IRC21:00
openstackgerritAlex Schultz proposed openstack/tripleo-heat-templates stable/ussuri: Stop using a conditional for role tasks  https://review.opendev.org/74498821:07
openstackgerritAlex Schultz proposed openstack/tripleo-heat-templates stable/train: Stop using a conditional for role tasks  https://review.opendev.org/74498921:07
openstackgerritMerged openstack/tripleo-common master: Reduce workers for label lookups  https://review.opendev.org/74491321:07
*** ayoung has quit IRC21:10
*** slaweq has quit IRC21:18
*** waleedm has quit IRC21:18
*** waleedm_ has quit IRC21:18
*** xek has joined #tripleo21:19
*** saneax has quit IRC21:24
*** tosky has joined #tripleo21:25
*** gthiemonge has quit IRC21:25
*** gthiemonge has joined #tripleo21:26
*** rcernin has joined #tripleo21:36
*** xek has quit IRC21:47
*** TrevorV has quit IRC21:57
*** cfontain has quit IRC22:02
*** cfontain has joined #tripleo22:02
*** rcernin has quit IRC22:03
*** rcernin has joined #tripleo22:17
*** rlandy is now known as rlandy|bbl22:17
*** rcernin has quit IRC22:40
*** rcernin has joined #tripleo22:40
*** rcernin has quit IRC22:40
*** rcernin has joined #tripleo22:44
*** artom has quit IRC22:45
*** artom has joined #tripleo22:46
*** jbadiapa has joined #tripleo22:49
*** jbadiapa has quit IRC22:50
*** tkajinam has joined #tripleo23:00
*** tosky has quit IRC23:14
*** ayoung has joined #tripleo23:26
*** irclogbot_0 has quit IRC23:34
*** irclogbot_3 has joined #tripleo23:38
openstackgerritTakashi Kajinami proposed openstack/python-tripleoclient master: Fix ignored parameters in create_deployment_plan  https://review.opendev.org/73767723:52
*** rlandy|bbl is now known as rlandy23:53

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!