Wednesday, 2020-05-27

*** rlandy has quit IRC00:56
*** Goneri has quit IRC01:03
*** rlandy has joined #oooq01:05
rlandyweshay|ruck: hi .. bm ussuri job is deploying the overcloud now01:06
rlandyit justfailed01:06
rlandysorry - wrong job01:06
rlandydeploy is still running01:06
*** rlandy has quit IRC01:11
*** Tengu has quit IRC01:38
*** Tengu has joined #oooq01:40
*** Tengu has quit IRC01:50
*** Tengu has joined #oooq01:52
*** gthiemon1e has joined #oooq01:55
*** gthiemonge has quit IRC01:56
*** jmasud has joined #oooq02:05
*** Tengu has quit IRC02:22
*** Tengu has joined #oooq02:23
*** gthiemon1e has quit IRC02:28
*** gthiemonge has joined #oooq02:29
*** jmasud has quit IRC02:46
*** jmasud has joined #oooq02:48
*** jmasud has quit IRC03:04
*** jmasud has joined #oooq03:05
*** jmasud has quit IRC03:07
*** jmasud has joined #oooq03:09
*** ysandeep|away is now known as ysandeep03:38
*** jmasud has quit IRC03:52
*** jmasud has joined #oooq03:53
*** jmasud has quit IRC04:17
*** skramaja has joined #oooq04:18
*** ratailor has joined #oooq04:36
*** saneax has joined #oooq04:49
*** marios has joined #oooq05:00
*** jmasud has joined #oooq05:14
*** jfrancoa has joined #oooq05:40
*** udesale has joined #oooq05:40
*** ykarel|away is now known as ykarel05:44
*** jmasud has quit IRC05:54
*** dpawlik has joined #oooq06:05
*** dpawlik has quit IRC06:12
*** dpawlik has joined #oooq06:12
*** jmasud has joined #oooq06:15
*** jmasud has quit IRC06:28
*** jbadiapa has joined #oooq07:07
*** ccamacho has joined #oooq07:11
*** tosky has joined #oooq07:26
*** amoralej|off is now known as amoralej07:31
*** ratailor has quit IRC07:38
*** ratailor has joined #oooq07:38
*** jbadiapa has quit IRC07:43
arxcruzmarios: chandankumar https://review.rdoproject.org/r/#/c/26990/ and https://review.rdoproject.org/r/#/c/26991/ when you guys have time :)07:46
*** bogdando has joined #oooq07:49
*** jpena|off is now known as jpena07:51
mariosarxcruz: https://review.rdoproject.org/r/#/c/26991/ needs rebase07:54
arxcruzmarios: of course it does07:54
arxcruzlol07:54
mariosrfolco|rover: i don't see a retro in calendar for today07:55
mariosarxcruz: ;)07:55
arxcruzmarios: where are the periodic now ?08:01
mariosarxcruz: what do you mean08:03
mariosarxcruz: ah maybe you mean https://github.com/rdo-infra/rdo-jobs/blob/master/zuul.d/integration-pipeline-master.yaml ?08:04
arxcruzmarios: yeah, turns out i won't need https://review.rdoproject.org/r/#/c/26991/08:05
arxcruzi can add all in the one single patch08:05
mariosarxcruz: yeah i forgot rfolco|rover moved that stuff revoted08:05
arxcruzmarios: then give your +2 on https://review.rdoproject.org/r/#/c/26990/08:20
mariosarxcruz: make me08:21
arxcruzsudo marios: then give your +2 on https://review.rdoproject.org/r/#/c/26990/08:21
mariosarxcruz is not in the sudoers file. This incident will be reported08:22
arxcruzmarios: https://xkcd.com/838/08:23
mariosarxcruz: :D08:24
arxcruzzbr: around ?08:25
marioschandankumar: any idea why we have non-voting for buildimage-hardened? looks pretty green https://zuul.openstack.org/builds?job_name=tripleo-buildimage-overcloud-hardened-full-centos-8&job_name=tripleo-buildimage-overcloud-hardened-full-centos-8-ussuri08:29
*** dtantsur|afk is now known as dtantsur08:32
*** ysandeep is now known as ysandeep|lunch08:46
*** jmasud has joined #oooq08:56
chandankumarmarios, we fixed this job recently so it is nv, we will make it voting soo08:59
marioschandankumar: i added it in https://review.opendev.org/#/c/730763/09:00
marioschandankumar: posting update now09:00
chandankumaraye09:00
marioschandankumar: its already voting in tripleo-common gate apparently https://opendev.org/openstack/tripleo-common/src/commit/df6008b2f412a475c603bf9a4062eb60fdb57c8d/zuul.d/layout.yaml#L6809:01
marioschandankumar: well not voting but is in gate09:01
chandankumarmarios, let me know when you update the patch09:02
marioschandankumar: just did09:02
*** pojadhav has quit IRC09:03
chandankumarmarios, thanks, will take a look09:04
marioschandankumar: thx09:05
*** sshnaidm|afk is now known as sshnaidm09:26
bogdandohi folks, what is the magic here, in 730860? https://review.opendev.org/#/c/730860/1/roles/standalone/templates/standalone.sh.j2 looks different to that tripleo_deploy.sh contains in CI jobs09:28
bogdandowhat is the right place to fix the command?09:28
bogdandoso CI creates that tripleo_deploy from https://opendev.org/openstack/tripleo-operator-ansible/src/branch/master/roles/tripleo_deploy/tasks/main.yml09:30
bogdandonot sure why I don't see ansible -vv there :/09:30
bogdandotripleo_deploy_debug seems set to true09:30
bogdandonm, figured that out09:32
*** ykarel is now known as ykarel|lunch09:35
*** ysandeep|lunch is now known as ysandeep09:39
*** chem has quit IRC09:39
*** ccamacho has quit IRC09:49
*** jmasud has quit IRC09:55
*** pojadhav has joined #oooq10:02
*** chem has joined #oooq10:27
*** udesale_ has joined #oooq10:33
chandankumarFYI ussuri sc10 is busted https://review.opendev.org/#/c/731149/ will fix it10:34
*** udesale has quit IRC10:36
*** ykarel|lunch is now known as ykarel10:36
bhagyashris|ruckmarios, sshnaidm ^ your vote will appreciate!!10:41
mariosack bhagyashris|ruck10:44
bhagyashris|ruckmarios, thanks!10:44
sshnaidmmarios, commented on https://review.opendev.org/#/c/73076311:01
mariossshnaidm: ack thanks will check11:01
ysandeeprfolco|rover, retrospective meeting invite missing.11:13
pojadhavrfolco|rover, yup for me too.11:24
marios10:55 < marios> rfolco|rover: i don't see a retro in calendar for today11:25
mariosme too :)11:25
*** sshnaidm is now known as sshnaidm|afk11:25
*** jpena is now known as jpena|lunch11:31
*** matbu has quit IRC11:41
*** matbu has joined #oooq11:42
bhagyashris|ruckweshay|ruck, Hi can you please merge this https://review.opendev.org/#/c/731149/ as this is blocking stable/ussri patches in check pipeline11:44
rfolco|roverysandeep, pojadhav marios I saw that yesterday. However, who owns the retro meeting is weshay|ruck11:47
mariosexcuses rfolco|rover tsk tsk tsk11:48
mariosshame11:48
rfolco|rovermarios, lol11:48
rfolco|rovermarios, good morning for you too11:48
mariosrfolco|rover: :)11:48
weshay|ruckrfolco|rover, marios we'll split th planning meeting tomorrow into retro and planning.. not much to cover in planning11:48
weshay|ruckwas about to email folks11:48
mariosweshay|ruck: k11:49
rfolco|roverweshay|ruck, tripleo-quickstart-promote-train-current-tripleo-delorean-minimal failed :(11:50
weshay|ruckrfolco|rover, k.. don't worry about it.. it will pass.. just need to keep rerunning it11:52
weshay|ruckrfolco|rover, need a bug on the fs039 issue w/ prmotion blocker11:52
rfolco|rovermy bug is that one that bhagyashris|ruck removed promotion-blocker tag coz it was not part of criteria11:53
rfolco|roverweshay|ruck, https://bugs.launchpad.net/tripleo/+bug/188023611:53
openstackLaunchpad bug 1880236 in tripleo "fs039 train: deploy freeipa failed" [Critical,Triaged]11:53
chandankumarmarios, sshnaidm|afk need some help at this part https://review.opendev.org/#/c/718946/24/roles/oooci-build-images/tasks/image_sanity.yaml@4511:55
chandankumarhitting module failure https://19a821469f9827248c75-0a60532523bd9fb5f045ad0db727d57f.ssl.cf1.rackcdn.com/718946/24/check/tripleo-buildimage-overcloud-full-centos-8/5aa6308/job-output.txt11:55
chandankumardo I need to remove pipefail from there?11:55
marioschandankumar: looking11:57
*** rlandy has joined #oooq11:58
rlandyrfolco|rover; hey - do we have a retrospective meeting today?11:59
rfolco|roverrlandy, no, we'll combine into planning mtg tomorrow11:59
marioschandankumar: if you suspect the pipefail you can remove it i think let me comment (you don't need cat | grep)11:59
*** ykarel is now known as ykarel|afk11:59
rfolco|roverweshay|ruck will send out an email rlandy11:59
rlandyrfolco|rover: thanks11:59
*** pojadhav is now known as pojadhav|brb12:00
chandankumarmarios, ok thanks!12:00
marioschandankumar: commented12:00
weshay|ruckmarios, thank you !! merged https://review.opendev.org/#/c/722790/12:08
weshay|ruckrfolco|rover, chandankumar /me runs a promotion for ussuri now12:08
weshay|ruckovb needs an update12:08
rfolco|roverweshay|ruck, ack12:09
chandankumarweshay|ruck, sure12:14
*** amoralej is now known as amoralej|lunch12:16
rlandypojadhav|brb: hey - ping me when you are  ready to meet12:19
weshay|ruckchandankumar, let's chat for a minute12:21
weshay|ruckwhen you have time12:21
chandankumarweshay|ruck, yes12:22
chandankumarweshay|ruck, gmeet link12:22
weshay|ruckchandankumar, https://meet.google.com/emi-sbfg-uda12:22
*** pojadhav|brb is now known as pojadhav12:27
pojadhavrlandy, lets meet12:28
*** ratailor has quit IRC12:28
rlandypojadhav: https://meet.google.com/ocj-dnao-nus12:29
*** ykarel|afk is now known as ykarel12:41
*** Goneri has joined #oooq12:42
*** sshnaidm|afk is now known as sshnaidm12:45
rlandypojadhav: https://review.opendev.org/#/c/731189/12:51
rlandyysandeep: hi - sorry - never got to the failing scenario 7 test yesterday - where are we with that?12:52
chandankumarweshay|ruck, bhagyashris|ruck https://review.rdoproject.org/r/27799 testing ussuri fs01 here12:52
rlandystill need debug?12:52
bhagyashris|ruckchandankumar, ack12:52
ysandeeprlandy, No worries i got hold of a node and taking yatin's and damien(rhos-pidone) help12:53
rlandyysandeep: k - great - ping if you need me12:53
*** amoralej|lunch is now known as amoralej13:04
weshay|ruckpromoting ussuri13:05
weshay|ruckakahat++ https://amolkahat.github.io/deploy-standalone-tripleo-using-tripleo-operator-ansible.html13:06
ysandeepakahat++ nice blog buddy!13:08
ysandeeprlandy, scenario10 jobs are green - https://code.engineering.redhat.com/gerrit/#/c/200673 , Please add to your review list for today13:10
rlandymarios: hey - we are going to have to change rhos-17 work to point to branch stable/ussuri rather than master13:10
rlandyysandeep: looks good - +2'ed - pls add a job to add scenario010 to the integration line13:11
ysandeeprlandy, ack will post soon..13:12
mariosrlandy: o/ ack13:12
mariosrlandy: ysandeep: i started working on standalone-upgrade added a new task https://tree.taiga.io/project/tripleo-ci-board/task/177213:12
rlandymarios: not today :)13:12
ysandeepmarios, ack13:12
mariosweshay|ruck: btw we don't have centos8 standalone-upgrade o_O we should probably add that too (noticed it today working on rhel8/osp17 one)13:13
ysandeepmarios, https://code.engineering.redhat.com/gerrit/#/c/200673/ need workflow.. whenever time permits.13:13
rlandymarios: ysandeep: working on ipa-multinode in osp https://code.engineering.redhat.com/gerrit/#/c/201122/13:13
mariosbrb coffee refill13:13
rlandythere is some work there we may need for multinode in general13:13
rlandyit's at the standalone deploy - debugging that13:14
rlandymarios: when back - pls vote on ysandeep's change https://code.engineering.redhat.com/gerrit/#/c/200673 so we can merge if you are happy13:14
ysandeeprlandy, i am working on multinode https://code.engineering.redhat.com/gerrit/#/c/201375/ - config , and testing multinode here https://code.engineering.redhat.com/gerrit/#/c/201376/ for now..13:14
weshay|ruckmarios, there are instructions on how to add your blog to planet tripleo right?13:15
weshay|ruckI saw it.. but can't find it atm.. can you please point akahat at it13:15
weshay|ruckand /me so I can note it13:15
ysandeeprlandy sry correct testpatch for multinode- https://code.engineering.redhat.com/gerrit/#/c/201382/ - currently failing on emit_releases_file.py  - https://sf.hosted.upshift.rdu2.redhat.com/logs/82/201382/1/check/periodic-tripleo-ci-rhel-8-multinode-1ctlr-featureset010-rhos-17/eff9e9e/job-output.txt will check once I am free from scenario713:19
mariosweshay|ruck: sure... i have to remember how :) let me see if i can find it13:20
mariosrlandy: ysandeep: ack will check13:21
weshay|ruckmarios, thanks13:22
mariosakahat: weshay|ruck: ha its in our repo :) found that i think it should be it: https://review.opendev.org/#/c/332187/1/scripts/website/planet.config.ini13:22
mariosakahat: weshay|ruck: so you need to add a xml feed to your site, then add that to the config ^^^ unless things have changed since then ...13:23
*** ccamacho has joined #oooq13:25
mariosysandeep: rlandy: checking now13:25
akahatweshay|ruck, ysandeep marios thanks :)13:30
weshay|ruckrlandy, thanks for the bm runs.. passed twice.. so feeling good about that13:32
weshay|ruckchandankumar, to_build patch?13:32
rlandyweshay|ruck: ack - ok13:32
chandankumarweshay|ruck, currently running without to_build https://review.opendev.org/#/c/718946/ here13:33
weshay|ruckchandankumar, right.. but we also talked about switching check jobs -> to_build=true right?13:35
chandankumarweshay|ruck, patch is coming sir13:35
chandankumarweshay|ruck, https://review.rdoproject.org/r/2780113:38
weshay|ruckchandankumar, thanks.. thinking we should just merge right.. can't make things worse :)13:39
chandankumarweshay|ruck, please merge that13:39
*** TrevorV has joined #oooq13:42
chandankumarrlandy, https://review.opendev.org/#/c/729541/13:43
pojadhavrlandy, https://review.opendev.org/#/c/729907/13:52
pojadhavrlandy, what can be done for failing "RDO Third Party CI"13:52
rlandynot much of it's unrelated to your job13:52
rfolco|roverchandankumar, is this the right place to modify cirros image for rdo jobs ? https://github.com/openstack/tripleo-quickstart-extras/blob/b24b18b507f903518d4b52dd72bf9b91f4e1eb15/playbooks/tempest.yml#L6813:55
pojadhavrlandy, ok13:55
chandankumarrfolco|rover, yes13:56
rfolco|roverchandankumar, so there is no way to specify in the job definition... this is kind of hardcoded13:56
pojadhavrlandy, I am done with this - https://review.opendev.org/#/c/731206/13:57
*** pojadhav is now known as pojadhav|brb13:57
chandankumarrfolco|rover, you can tweak it by moving this var to featureset override13:57
rlandyok - i'll look after meeting13:57
pojadhav|brbrlandy, sure ;)13:58
rfolco|roverchandankumar, yeah, let me propose this13:58
rfolco|roverchandankumar, thx13:58
*** jbadiapa has joined #oooq14:00
*** jpena|lunch is now known as jpena14:04
chandankumarakahat, nice blog14:09
chandankumarlooking forward for followup blogs14:09
arxcruzchandankumar: soniya29 just find out that the validate job is passing even when the yaml file is invalid, funny part is that locally it fails, but on the job it pass :/ trying to debug the reason14:15
soniya29arxcruz: let me try14:18
weshay|ruckarxcruz, saw your prayer :) https://review.opendev.org/#/c/724256/14:20
weshay|ruckglad it merged for ya14:20
chandankumararxcruz, job link or review link?14:20
arxcruzweshay|ruck: yeah, almost one week trying to make it get merged14:23
arxcruzchandankumar: any job, it's always passing, and today i was creating the module for the skiplist, try to run the validate command found it was failing, but the job was passing14:23
arxcruzchandankumar: for example https://c016f1744540d2b796f4-bac515511eb85ae193e5d3c2e4dbaa93.ssl.cf2.rackcdn.com/731151/10/check/openstack-tempest-skiplist-validate/76b287b/job-output.txt14:27
*** skramaja_ has joined #oooq14:27
*** skramaja has quit IRC14:28
chandankumararxcruz, do you have error which you got in local testing?14:30
chandankumarit is passing in the job14:31
arxcruzchandankumar: not a valid value for dictionary value @ data['known_failures'][0]['deployment']14:31
arxcruzchandankumar: yeah, that's what i'm investigating, it's passing on the job, but not locally14:31
chandankumarrfolco|rover, 2020-05-27 14:01:31.040 | ln: target '/etc/puppet/modules/' is not a directory: No such file or directory14:41
chandankumarrfolco|rover, https://logserver.rdoproject.org/99/27799/2/check/periodic-tripleo-centos-8-buildimage-overcloud-full-ussuri/314a6f7/overcloud-full.log14:41
chandankumarrecent error on build images14:41
chandankumarknown issue?14:41
rfolco|roverlet me check if its the same one14:41
rfolco|roverwe do have a bug for build images14:42
mariosweshay|ruck: chandankumar: is there a reason we have voting: false for tripleo-buildimage-ironic-python-agent-centos-7 (looks ok https://zuul.openstack.org/builds?job_name=tripleo-buildimage-ironic-python-agent-centos-7 ) ... also do you know why we don't have a -train for that one? (I am planning on making it voting and adding -train in next iteration of https://review.opendev.org/#/c/730763/ )14:43
rfolco|roverchandankumar, new issue :(14:44
chandankumarrfolco|rover, let me do a recheck14:45
chandankumarand then confirm14:45
chandankumarhttps://review.rdoproject.org/r/#/c/27799/ running here14:45
rfolco|roverchandankumar, https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-latest-released&job_name=periodic-tripleo-centos-8-buildimage-overcloud-full-ussuri14:45
rfolco|roverchandankumar, all green so far14:45
chandankumarmarios, we never added train, we were testing on master only14:47
chandankumarlater on weshay|ruck added one image build job for train when we found some issue on c7 train14:47
marioschandankumar: but we have for example tripleo-buildimage-overcloud-full-centos-7-train14:48
chandankumaryes  tripleo-buildimage-overcloud-full-centos-7-train got added when we saw some issue on c7 train14:48
weshay|ruckchandankumar, marios k.. just make sure the branched versions of jobs are not in templates.. just the layout.yaml for ci repos14:48
mariosweshay|ruck: chandankumar: k... going to add it into the review, and you can comment there i will remove if we don't want it14:49
weshay|ruckor else they trigger in on the wrong repos... like tripleo repos14:49
mariosweshay|ruck: i'm splitting into two templates like we have for containers (tripleo-ci-buildimage-jobs template and tripleo-buildimage-jobs )14:49
rlandypojadhav|brb: w+'ed https://review.opendev.org/#/c/729907/15:02
mariossshnaidm: thanks for reviews updated https://review.opendev.org/#/c/730763/ whenever you next have some reviews time15:03
marioschandankumar: weshay|ruck: rlandy: *all please check when you have time ^^15:03
rlandymarios: +2 on https://review.opendev.org/#/c/730763 - one question in line15:07
weshay|ruckhttps://review.rdoproject.org/zuul/builds?job_name=tripleo-ci-centos-8-ovb-1ctlr_1comp-featureset001&job_name=tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001&branch=master15:08
rlandypojadhav|brb: hmmm ... we didn't get the train job triggering15:08
rlandywe may need to do the depends the other way around15:09
weshay|ruckvs. https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-master&job_name=periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-master15:09
rlandyor create a third job15:09
weshay|ruckchandankumar, rfolco|rover bhagyashris|ruck we need master ovb jobs resolved.. not going to promote when the current build is working so well15:09
rlandyysandeep: hey - how are things with the baremetal work?15:09
weshay|ruckchandankumar, what was left to do w/ tripleo-ci-centos-8-scenario010-standalone15:15
weshay|ruckfor ussuri15:15
weshay|ruckthere was one patch right?15:15
weshay|ruckbhagyashris|ruck, you had a patch on that ^15:16
weshay|ruckhttps://review.opendev.org/#/c/731149/15:16
rfolco|roverweshay|ruck, ussuri fs020 and fs001 are failing tempest, not the same ones.... do you think an updated cirros would help?15:19
mariosrlandy: thanks for reviewing replied15:20
weshay|ruckrfolco|rover, link?15:21
rfolco|roverhttps://logserver.rdoproject.org/openstack-periodic-latest-released/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-ussuri/c153ad4/logs/undercloud/var/log/tempest/stestr_results.html.gz15:21
rfolco|roverhttps://logserver.rdoproject.org/openstack-periodic-latest-released/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-ussuri/29aadb0/logs/undercloud/var/log/tempest/stestr_results.html.gz15:21
rfolco|roverand scen010 has this silly error: /home/zuul/tempest-setup.sh: line 26: /home/zuul/overcloudrc: No such file or directory15:21
rfolco|roverhttps://logserver.rdoproject.org/openstack-periodic-latest-released/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario010-standalone-ussuri/49d3f5b/logs/undercloud/home/zuul/tempest.log.txt.gz15:22
weshay|ruckrfolco|rover, ah k.. so the very latest run for fs001 got to tempest15:24
weshay|ruckk15:24
rfolco|roverweshay|ruck, and my comment is based on another issue.... but I think this may make sense...15:25
rfolco|roverhttps://trello.com/c/OnJrKpfY/1495-cixlp1878248tripleociproa-networksecgrouptest-failing-on-fs020-stein15:25
weshay|ruckrfolco|rover, openstack-ansible-tempest should be pulling a very recent cirros15:25
weshay|ruckrfolco|rover, I would skip list tempest.api.object_storage.test_object_formpost15:25
weshay|ruckhttps://logserver.rdoproject.org/openstack-periodic-latest-released/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-ussuri/29aadb0/logs/undercloud/var/log/tempest/stestr_results.html.gz15:25
weshay|ruckrfolco|rover, it's probably worth a testproject too... maybe we can flip it to green15:26
weshay|ruckrfolco|rover, could be a issue w/ the deployment15:26
weshay|ruck and scen010 has this silly error: /home/zuul/tempest-setup.sh: line 26: /home/zuul/overcloudrc: No such file or directory15:26
weshay|ruckthis ^15:26
weshay|ruckrfolco|rover, if that file was not written15:26
rfolco|roverweshay|ruck, ok, i proposed both https://review.opendev.org/731232 and https://review.opendev.org/731233 to override default cirros15:26
*** ykarel is now known as ykarel|away15:27
weshay|ruckarxcruz, standalone I think uses a clouds.yaml right?15:27
weshay|rucknot overcloudrc15:27
arxcruzweshay|ruck: yes15:28
arxcruzclouds.yaml15:28
weshay|ruckrfolco|rover, work w/ arxcruz on the cirros image.. I think both those patches are not needed.. we're using cirros 5.0 from openstack-ansible-tempest15:28
weshay|ruckarxcruz, so what's up w/ https://logserver.rdoproject.org/openstack-periodic-latest-released/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario010-standalone-ussuri/49d3f5b/logs/undercloud/home/zuul/tempest.log.txt.gz15:29
arxcruzweshay|ruck: checking15:29
rfolco|roverweshay|ruck, arxcruz apparently its using cirros 4.0 2020-05-27 13:05:04.524 258643 DEBUG tempest [-] image.http_image               = http://images.rdoproject.org/cirros/cirros-0.4.0-x86_64-disk.img15:30
arxcruzrfolco|rover: weshay|ruck first of all, it's not using os_tempest, it's using validate-tempest15:30
arxcruz:D15:30
rfolco|roverarxcruz, from which release os_tempest should be used ?15:32
arxcruzrfolco|rover: i think stein15:32
arxcruzchandankumar: am i correct ?15:32
arxcruzrfolco|rover: ussuri for sure uses os_tempest :)15:33
sshnaidmmarios, commented on https://review.opendev.org/#/c/73076315:33
rfolco|roverarxcruz, chandankumar: does this override what os_tempest has ? https://github.com/openstack/tripleo-quickstart-extras/blob/master/playbooks/tempest.yml#L65-L6815:34
rfolco|rovercoz I see cirros 4.0 everywhere15:35
arxcruzrfolco|rover: yes15:36
arxcruzit does15:36
arxcruzwe did it because cirros offical repo was always offline15:37
weshay|ruckarxcruz, wait.. usuri jobs are not using os_tempest?15:37
rfolco|roverarxcruz, chandankumar so please review my initial propose on https://review.opendev.org/731232 and https://review.opendev.org/73123315:37
* weshay|ruck looks15:37
rfolco|roverthis may require some changes15:38
arxcruzweshay|ruck: at least scenario10-standalone doesn't15:38
weshay|ruckya. that's in progress15:38
weshay|ruckhttps://review.opendev.org/#/c/731149/15:39
arxcruzrfolco|rover: lgtm15:39
weshay|ruckarxcruz, and what about the overcloudrc vs. clouds.yaml15:41
weshay|ruckarxcruz, rfolco|rover just a reminder that will flip to os_tempest soon https://tree.taiga.io/project/tripleo-ci-board/us/1587?milestone=26474115:43
arxcruzweshay|ruck: need to set in the job the variable tempest_os_cloud: standalone15:44
arxcruzto use os_cloud15:44
weshay|ruckrfolco|rover, take care of that one please ^15:44
weshay|ruckrfolco|rover, check and periodic15:44
rfolco|roverweshay|ruck, arxcruz for all jobs ?15:46
rfolco|roverall branches ?15:46
arxcruzrfolco|rover: it's set on fs05215:46
arxcruzif standalone uses it it's fine to use i guess15:46
weshay|ruckrfolco|rover, need a bug as well.. because I need to make ussuri nv15:48
weshay|ruckfor scenario01015:48
rfolco|roverweshay|ruck, ok let me file a bug for scen10 ussuri15:49
mariosack sshnaidm thanks will check in the morning got to go15:50
*** marios has quit IRC15:50
bhagyashris|ruckrfolco|rover, there is one patch chandankumar has submitted to port the sc10 jobs to os_tempest15:53
bhagyashris|ruckrfolco|rover, https://review.opendev.org/#/c/729541/15:53
rfolco|roverhmm I don't see clouds.yaml issue being fixed there15:55
rfolco|roverchandankumar, with that patch we get clouds.yaml fixed for free ?15:55
rfolco|roverthis error https://logserver.rdoproject.org/openstack-periodic-latest-released/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario010-standalone-ussuri/49d3f5b/logs/undercloud/home/zuul/tempest.log.txt.gz15:55
bhagyashris|ruckrfolco|rover, ah ok..15:57
rfolco|roverbhagyashris|ruck, unless chandankumar confirms that with that fix, switching to os_tempest will fix the clouds.yaml, two rabbits in one shot15:57
bhagyashris|ruckrfolco|rover, hahaha ok .. one thing we can test sc10 on testproject by Depends-On...15:59
rfolco|roverok will file a new bug for clouds.yaml, if its the same, we mark as invalid16:00
*** ccamacho has quit IRC16:00
bhagyashris|ruckok np16:00
*** skramaja_ has quit IRC16:09
ysandeeprlandy, sorry missed your pings earlier, i was in a meeting with damien and ykarel|away for troubleshooting that scenario0716:17
ysandeeprlandy, Turns out rabbitmq can't start properly in some sceanrios when ipv6 is entirely disabled on the system. We opened a bz https://bugs.launchpad.net/tripleo/+bug/1880967 but turns out its because of downstream nodepool customization. Luckily someone posted a review on what we need recently - https://code.engineering.redhat.com/gerrit/#/c/201431/116:17
openstackLaunchpad bug 1880967 in tripleo "RabbitMQ-specific IPv6 sysctl flags were dropped when migrating to the kernel-baremetal-ansible service" [High,In progress] - Assigned to Damien Ciabrini (dciabrin)16:17
chandankumarrfolco|rover, what is cloudname issue?16:18
arxcruzchandankumar: i already explain to him, he's submitting the patch :)16:19
rlandyysandeep: that review won't help you16:21
rlandytristanC posted that review after our conversation on ipv616:21
ysandeeprlandy, why?16:21
rlandyysandeep: you are using an upstream-rhel-8 node16:22
rlandynot a a rhel-8 cloud node16:22
rlandyysandeep: this works for me for ipa on multinode ...16:22
rlandyhttps://code.engineering.redhat.com/gerrit/#/c/201122/11/playbooks/rhel-enable-ipv6.yaml16:22
rlandyhttps://code.engineering.redhat.com/gerrit/#/c/201122/11/playbooks/configure-rhel.yaml16:23
rlandyysandeep: ^^ I reconfigured the configure-rhel playbook to include that16:23
rlandyit should fix your ipv6 problems16:23
bhagyashris|ruckchandankumar, https://logserver.rdoproject.org/openstack-periodic-latest-released/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario010-standalone-ussuri/49d3f5b/logs/undercloud/home/zuul/tempest.log.txt.gz16:24
ysandeeprlandy, thanks! looking16:24
chandankumarbhagyashris|ruck, I know the issue16:25
chandankumarbhagyashris|ruck, https://github.com/rdo-infra/rdo-jobs/blob/master/zuul.d/standalone-jobs.yaml#L401 and https://github.com/rdo-infra/rdo-jobs/blob/master/zuul.d/standalone-jobs.yaml#L422 job definition needs to be updated with this patch https://review.opendev.org/#/c/729541/16:26
*** dtantsur is now known as dtantsur|afk16:26
chandankumarbhagyashris|ruck, https://review.opendev.org/#/c/729541/6/zuul.d/standalone-jobs.yaml16:27
weshay|ruckarxcruz, rfolco|rover reminder... scenario010 useds fs62 not 5216:27
*** bogdando has quit IRC16:27
chandankumarweshay|ruck, rfolco|rover arxcruz ^^ above check16:27
* chandankumar will send patch in another hour16:28
*** jmasud has joined #oooq16:30
arxcruzweshay|ruck: yes, we are aware16:31
arxcruzchandankumar: rfolco|rover is already on it ;)16:31
weshay|ruckgreat.. thanks fellas16:31
weshay|ruckarxcruz, rfolco|rover https://review.opendev.org/#/c/731265/16:32
weshay|ruckchandankumar, ^16:32
*** udesale_ has quit IRC16:32
chandankumarweshay|ruck, sc10 is working fine16:33
chandankumarit is not failing16:33
ysandeeprlandy, do you know where exactly we are setting these parameters in downstream in the first place - https://sf.hosted.upshift.rdu2.redhat.com/logs/37/200637/4/check/periodic-tripleo-ci-rhel-8-scenario007-standalone-rhos-17/1d63d36/logs/undercloud/etc/sysctl.conf16:34
weshay|ruckchandankumar, zuul begs to differ https://zuul.openstack.org/builds?job_name=tripleo-ci-centos-8-scenario010-standalone&branch=stable%2Fussuri16:34
chandankumarhttps://review.opendev.org/731149 will fix that16:34
rlandyysandeep: will explain at the meeting tomorrow16:35
rlandyhad a chat with ianw16:35
chandankumarweshay|ruck, rfolco|rover https://review.opendev.org/73114916:35
*** pojadhav|brb is now known as pojadhav16:35
rfolco|roverchandankumar, weshay|ruck your patches don't add tempest_os_cloud: standalone to fs06216:36
rfolco|roverI am submiting one for it16:36
weshay|ruckchandankumar, k.. tested where?16:36
* weshay|ruck looking at hackmd bhagyashris|ruck ^16:36
chandankumarweshay|ruck, I know that issue16:37
chandankumarweshay|ruck, it works16:37
chandankumarweshay|ruck, you can add a dnm patch16:38
weshay|ruckhttps://review.rdoproject.org/r/#/c/27804/16:38
chandankumarweshay|ruck, https://review.opendev.org/#/c/731149/ this one16:38
ysandeeprlandy, ack thanks! will catch up with you tomorrow on this along with BM work, In the meantime will add the change you mentioned to confirm it works.16:39
weshay|ruckchandankumar, scenario010 is not in the jobs there16:39
weshay|ruckchandankumar, so yes.. I believe you.. ok.. this fixes it.. but just in case I HAVE to mark it non-voting and have that queued up as well16:39
weshay|ruckit's blocking backports16:39
chandankumarweshay|ruck, ok16:39
weshay|ruckchandankumar, because I don't see a working scenario010 anywhere16:40
ysandeepweshay|ruck, its working in downstream :)16:45
weshay|ruckysandeep, /me looks16:45
ysandeephttps://code.engineering.redhat.com/gerrit/#/c/200673/ , testproject https://code.engineering.redhat.com/gerrit/#/c/200674/ .. we are already on os_tempest.16:45
weshay|ruckysandeep, sure enough https://sf.hosted.upshift.rdu2.redhat.com/logs/74/200674/4/check/periodic-tripleo-ci-rhel-8-scenario010-standalone-network-rhos-17/4f1155f/16:46
ysandeepweshay|ruck, https://code.engineering.redhat.com/gerrit/#/c/200673/4/zuul.d/standalone-jobs.yaml this will give info on what we did.16:46
weshay|ruckysandeep, http://tripleo-cockpit.usersys.redhat.com/d/Wj-ao_sWz/full-component-pipeline?orgId=116:47
weshay|ruckya.. and you can see it's failing in the upstream ussuri right next to it16:47
ysandeepweshay|ruck, btw this side to side looks is awesome :)16:48
weshay|ruckysandeep, it has to be the job def..16:48
weshay|ruckrfolco|rover, may want to compare the internal job def w/ the ussuri job def16:48
weshay|ruckysandeep, rfolco|rover ya.. also working in master16:48
ysandeepweshay|ruck, ack o/ see you tomorrow .. have a nice day ahead!16:49
*** ysandeep is now known as ysandeep|away16:49
weshay|ruckysandeep|away, 0/16:49
weshay|ruckrfolco|rover, train promoted in rdo https://ci.centos.org/view/rdo/view/promotion-pipeline/job/rdo_trunk-promote-train-current-tripleo/16:51
rfolco|roverweshay|ruck, great, I can breath until next wed16:51
weshay|ruckrfolco|rover, well the promoter failed :)16:52
weshay|rucksorry looking at old log16:53
*** yolanda has quit IRC16:53
weshay|ruck2020-05-27 16:11:04,008 9406 INFO     promoter SUCCESS promoting centos7-train current-tripleo as current-tripleo-rdo (commit: ba27b93bcaf31634d02a40682c62c741c304a687, distro: 33622c19034bc1338acc6704c1e13657912787bb, timestamp=1590528398)16:53
rfolco|roverweshay|ruck, ok, there is always a fine print in comercials16:53
weshay|ruckhttp://rhos-release.virt.bos.redhat.com:3030/rhosp16:53
weshay|rucklooks good16:53
weshay|ruckrfolco|rover, want me to push the skiplist chagne for master?16:54
rfolco|roverweshay|ruck, do if you have time for it.... I am finishing the fs062 check16:54
weshay|ruckk16:55
rfolco|roverotherwise I'll do it in a few16:55
rfolco|rover https://review.opendev.org/731276 Use clouds.yaml for fs06216:56
rfolco|roverchandankumar, arxcruz weshay|ruck16:57
*** jbadiapa has quit IRC16:58
rfolco|roverweshay|ruck, skip list for fs001 and fs020 ?16:59
weshay|ruckoh.. were we looking at 020?16:59
rfolco|roverlet me look again17:00
rfolco|roverussuri right ?17:00
weshay|ruckrfolco|rover, ussuri just promoted17:00
* weshay|ruck worried about master17:00
rfolco|roverok17:00
*** amoralej is now known as amoralej|off17:01
weshay|ruckfs001 is working in the master component pipeline17:01
rfolco|roverfs020 is failing on master.. https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-master&job_name=periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-master17:02
rfolco|rover39, 35 failed too17:03
rfolco|roverand full tempest failed last run17:03
weshay|ruckrfolco|rover, master pipeline just triggered17:03
weshay|ruckso perhaps we have the patches we need now17:03
weshay|ruckwill let it run17:03
weshay|ruckthe gate is running pretty well17:03
rfolco|roverTest Group/Test caseCountPassFailErrorSkipView17:04
rfolco|roverneutron_tempest_plugin.scenario.test_trunk.TrunkTest17:04
rfolco|roverfamiliar?17:04
rfolco|roverhttps://logserver.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-full-tempest-scenario-master/cc034ac/logs/undercloud/var/log/tempest/stestr_results.html.gz17:04
weshay|ruckrfolco|rover, nope.. haven't seen that one17:05
weshay|ruckRuntimeError: Timed out waiting for port 'ce6044d6-b31e-4b1c-815e-49c84a8af7e2' to transition to get status 'ACTIVE'.17:05
bhagyashris|ruckweshay|ruck, rfolco|rover from my observation fs001 is blocking promotion on master ....17:05
weshay|ruckrfolco|rover, not enough consistent fails to write it up yet thoughhttps://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-master&job_name=periodic-tripleo-ci-centos-8-standalone-full-tempest-scenario-master17:05
weshay|ruckhttps://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-master&job_name=periodic-tripleo-ci-centos-8-standalone-full-tempest-scenario-master17:05
weshay|ruckbhagyashris|ruck, yes.. :)17:06
weshay|ruckbhagyashris|ruck, I'm hoping the success here http://dashboard-ci.tripleo.org/d/FE8Hf29Wz/component-pipeline?orgId=117:06
weshay|ruckpulls in some patches to promotion jobs .. it's running now17:06
weshay|ruckthe baremetal component promoted today17:06
bhagyashris|ruckweshay|ruck, and from last two days i worried about it i saw your comment on my test patch and then i am hoping  for the best in next run17:07
weshay|ruckbhagyashris|ruck, rfolco|rover and had been failing the last seven days17:07
bhagyashris|ruckweshay|ruck, rfolco|rover jfyi regarding stein promotion fs020 and fs035 is also failing because of Permission denied (publickey,gssapi-keyex,gssapi-with-mic). issue otherwise other jobs are doing good17:09
weshay|ruckbhagyashris|ruck, k.. I might as well review those branches as well.. while master runs17:10
* weshay|ruck looks17:10
weshay|ruckthanks bhagyashris|ruck17:10
bhagyashris|ruckweshay|ruck, https://hackmd.io/2MdkNAUuT7aBcM0Yck4xnw?both#May-27th17:10
*** jpena is now known as jpena|off17:10
bhagyashris|ruckhere i have given details17:10
weshay|ruckk.. thanks17:12
weshay|ruckrfolco|rover, waiving fs020/39 in stein17:31
rlandyweshay|ruck: 1-on-1?17:31
*** dpawlik has quit IRC17:59
*** chem has quit IRC18:12
*** chem has joined #oooq18:15
chandankumarrfolco|rover, can you pass me the all the patches here for tempest to take a look?18:18
chandankumarweshay|ruck, rlandy https://review.opendev.org/729541 please merge this one18:19
rfolco|roverchandankumar, https://review.opendev.org/#/c/731232/, https://review.opendev.org/#/c/731233/ and https://review.opendev.org/#/c/731276/18:21
chandankumarrfolco|rover, thanks, will take care of that18:21
rfolco|roverchandankumar, please see comment about run_tempest missing in the last one... arxcruz might be right18:21
zbrrlandy: re ansible-lint and missing modules, I managed to raise https://github.com/ansible/ansible-lint/pull/801 -- now we need to wait for reviews. still i am confident that we will see this fixed18:24
zbronce done, we will no longer need to hack ANSIBLE_LIBRARY18:24
rlandyzbr: nice - thanks for taking this on18:47
*** jschlueter has joined #oooq18:47
weshay|ruckchandankumar, ok.. on my list18:48
weshay|ruckchandankumar, watching https://zuul.openstack.org/stream/3620123e2f9946c9a41b73e9601efd77?logfile=console.log18:48
*** jmasud has quit IRC18:51
*** jmasud has joined #oooq18:52
chandankumarweshay|ruck, sorry which patch?18:59
weshay|ruckhttps://review.opendev.org/72954119:02
*** jmasud has quit IRC19:04
*** jmasud has joined #oooq19:05
*** jmasud has quit IRC19:12
*** sshnaidm is now known as sshnaidm|afk19:12
*** jfrancoa has quit IRC19:13
*** jmasud has joined #oooq19:15
*** rlandy is now known as rlandy|biab19:23
*** jmasud has quit IRC19:30
weshay|ruckrfolco|rover, bhagyashris|ruck stein is promoting19:34
rfolco|roverweshay|ruck, nice19:34
weshay|ruckchandankumar, ur patch worked :)19:35
weshay|ruckwf https://review.opendev.org/#/c/729541/ it's master though19:35
weshay|ruckstill watching ussuri19:35
chandankumarweshay|ruck, Dhanyabad sir :-)19:35
chandankumarweshay|ruck, for ussuri you need bhagyashris|ruck patch19:35
weshay|ruckchandankumar, aye.. watching that through the gate queue19:35
weshay|ruckwill be a bit..19:36
* chandankumar prepares one more patch19:36
weshay|ruckbig queues.. but they are running well19:36
*** apetrich has quit IRC19:38
chandankumarrfolco|rover, weshay|ruck https://review.rdoproject.org/r/2780519:41
chandankumarfor sc10 sync with upstream19:41
*** jmasud has joined #oooq19:46
*** jmasud has quit IRC20:20
weshay|ruckspacex in 11min https://www.spacelaunchschedule.com/launch-schedule/launch-details.php?id=109320:21
arxcruzweshay|ruck: rfolco|rover https://review.opendev.org/#/c/731276 is gtg if this is an emergency21:19
weshay|ruckthanks arxcruz21:19
rfolco|roverthx arxcruz21:21
*** TrevorV has quit IRC21:32
*** saneax has quit IRC21:34
*** jmasud has joined #oooq21:46
*** jmasud has quit IRC21:50
*** jmasud has joined #oooq21:51
*** apetrich has joined #oooq21:53
*** jmasud has joined #oooq22:01
weshay|ruckrlandy|biab, rfolco|rover when ur back https://bugs.launchpad.net/tripleo/+bug/188102422:04
openstackLaunchpad bug 1881024 in tripleo "ussuri component jobs using images from master " [Critical,Triaged]22:04
*** rlandy|biab is now known as rlandy22:04
rlandyoops22:04
rlandyforgot to nick22:04
weshay|rucknp22:04
weshay|rucknot sure if that's the fix to be honest22:05
rlandylooking22:05
*** jmasud has quit IRC22:05
rfolco|roverisn't branch-override ?22:06
weshay|ruckya.. I think so22:07
rfolco|roverno... ignore me22:07
rlandyweshay|ruck: so that's not going to change the images used22:07
rfolco|roverwait...22:07
rlandythe images are from the release file22:07
weshay|ruckrlandy, the images are correct.. in the config/release22:07
weshay|ruckrlandy, confirmed those are fine22:07
weshay|ruckit's the component job def22:07
weshay|ruckperiodic promotions jobs are fine.. and the config is fine22:08
rlandyeither way - one of those jobs has an error in definition22:08
weshay|ruck2020-05-27 16:19:26.445468 | primary | + BRANCH_OVERRIDE=stable/ussuri22:08
weshay|ruckneeds to be that22:08
weshay|ruckupdated patch22:12
rlandyweshay|ruck: you cloud keep the previous parenting22:16
rlandycould22:16
rlandyshouldn't matter22:16
weshay|ruckya.. I think either will work22:16
rlandyweshay|ruck: mass update all ussuri component jobs?22:17
weshay|ruckrlandy, /me checks22:20
weshay|ruckwas looking.. but wasn't convinced it needed it22:21
* weshay|ruck looks22:21
weshay|ruckrlandy, hrm.22:21
weshay|ruckthey do22:21
weshay|ruckhttps://logserver.rdoproject.org/openstack-component-octavia/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-multinode-1ctlr-featureset010-octavia-ussuri/7ff9f94/job-output.txt22:21
weshay|ruck--extra-vars @/home/zuul/workspace/.quickstart/config/release/tripleo-ci/CentOS-8/master.yml22:21
weshay|ruckawesome :(22:21
rlandyweshay|ruck: someone can take that as an item next sprint22:22
rlandyie: tomorrow22:22
weshay|ruckI'll do it now22:22
rlandyweshay|ruck: want help?22:22
weshay|ruckjust validation22:22
rlandyok- I'm here22:22
weshay|ruckrlandy, can you check     name: periodic-tripleo-centos-8-ussuri-component-baremetal-promote-consistent-to-component-ci-testing22:24
weshay|ruckrlandy, that the promotion jobs are working properly.. I think they are based on the dlrn hashes that are getting promoted22:24
weshay|ruckbut good to double check22:24
rlandylooking22:27
weshay|ruckrlandy, you may also want to double check internal zuul config22:28
rlandy https://trunk.rdoproject.org/api-centos8-ussuri22:28
rlandylooks right22:28
rlandy "https://trunk.rdoproject.org/centos8-ussuri/component/baremetal/16/a7/16a734a0ea2c51085cf3417e6b8f7a91b36b16e1_2795a5bb",22:28
rlandyweshay|ruck: ^^ that look sright22:29
rlandyno branch involved in promote jobs22:29
rlandyweshay|ruck: question - how come the integration lines work w/o branch override?22:29
weshay|ruckhttps://review.rdoproject.org/r/2780722:30
weshay|ruckrlandy, ^22:30
weshay|ruckrlandy, internal is correct /home/zuul/workspace/.quickstart/bin/ansible-playbook /home/zuul/workspace/.quickstart/playbooks/collect-logs.yml -vvvv --extra-vars @/home/zuul/workspace/.quickstart/config/release/tripleo-ci/RedHat-8/rhos-17.yml22:31
weshay|ruckrlandy, /me looks at integration22:32
weshay|ruckrlandy, they have it https://review.rdoproject.org/r/gitweb?p=rdo-jobs.git;a=blob;f=zuul.d/integration-jobs-ussuri.yaml#l19822:33
rlandyok - good22:33
weshay|ruckhttps://review.rdoproject.org/r/#/c/27807/22:34
rlandyweshay|ruck: https://review.rdoproject.org/r/#/c/27807/ - looks like you got them all22:34
rlandy+2 thre22:34
rlandythere22:34
weshay|ruckthanks22:34
rlandynice catch22:36
*** jmasud has joined #oooq22:54
*** jmasud has quit IRC22:59
*** tosky has quit IRC23:30
*** Goneri has quit IRC23:49

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!