Monday, 2018-09-17

hubbot1FAILING CHECK JOBS on stable/pike: tripleo-ci-centos-7-scenario004-multinode-oooq-container @ https://review.openstack.org/602248, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario004-multinode-oooq-container @ https://review.openstack.org/567224, stable/rocky: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7 (2 more messages)01:46
*** ykarel has joined #oooq03:31
*** udesale has joined #oooq03:34
*** skramaja has joined #oooq03:37
hubbot1FAILING CHECK JOBS on stable/pike: tripleo-ci-centos-7-scenario004-multinode-oooq-container @ https://review.openstack.org/602248, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario004-multinode-oooq-container @ https://review.openstack.org/567224, stable/rocky: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7 (2 more messages)03:46
*** udesale has quit IRC04:03
*** udesale has joined #oooq04:24
*** udesale has quit IRC04:25
*** udesale has joined #oooq04:25
*** ykarel has quit IRC05:22
*** jbadiapa has joined #oooq05:25
*** quique|rover|off is now known as quiquell|rover05:30
*** jtomasek has joined #oooq05:34
chkumar|ruckquiquell|rover: Good morning :-)05:42
chkumar|ruckquiquell|rover: I am taking a look at periodic jobs05:42
*** ykarel has joined #oooq05:42
quiquell|roverchkumar|ruck: good morning05:44
*** jtomasek has quit IRC05:44
quiquell|roverstill no promotions :-/05:44
quiquell|roverThis is impossible05:44
quiquell|roverchkumar|ruck: jobs are running now05:45
*** jtomasek has joined #oooq05:45
hubbot1FAILING CHECK JOBS on stable/pike: tripleo-ci-centos-7-scenario004-multinode-oooq-container @ https://review.openstack.org/602248, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario004-multinode-oooq-container @ https://review.openstack.org/567224, stable/rocky: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7 (2 more messages)05:46
chkumar|ruckquiquell|rover: fs019 is failing on master and rocky with overcloud deploy step05:49
quiquell|roverchkumar|ruck: Yep, on friday we saw the problem05:51
quiquell|roverchkumar|ruck: Didn't have time to find a soltuion05:51
chkumar|ruckquiquell|rover: do we have open bug for fs-1905:51
quiquell|roverchkumar|ruck: Let's check queens05:52
quiquell|roverchkumar|ruck: yep05:52
quiquell|roverhttps://bugs.launchpad.net/tripleo/+bug/179229605:52
openstackLaunchpad bug 1792296 in tripleo "Overcloude deploy error:Timed out waiting for messages from Execution" [Critical,Triaged] - Assigned to Quique Llorente (quiquell)05:52
quiquell|roverchkumar|ruck: I will find a solution today05:52
quiquell|roverchkumar|ruck: They has post a solution05:53
quiquell|roverchkumar|ruck: We are also affected at master/rocky by this (is intermitent)05:53
quiquell|roverhttps://bugs.launchpad.net/tripleo/+bug/179234305:54
openstackLaunchpad bug 1792343 in tripleo "[tripleo] rocky baremetal deployment fails with jq: error: Could not open file /var/lib/heat-config/deployed/<id>.notify.json" [Critical,Triaged] - Assigned to Quique Llorente (quiquell)05:54
chkumar|ruckquiquell|rover: so there are two bugs affecting fs019 ?05:54
chkumar|ruckfor master and rocky05:54
quiquell|roverchkumar|ruck: fs019 only affected by the first one05:55
quiquell|roverchkumar|ruck: The other is intermitent issue we can see at different featuresets05:55
quiquell|rover In case you find it as ruck05:55
*** saneax has joined #oooq05:55
*** quiquell|rover has quit IRC05:56
chkumar|ruckquiquell|rover: fs017 tempest telemtry issue https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/legacy-periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset017-master/90fd211/logs/undercloud/home/zuul/tempest.log.txt.gz05:56
*** quiquell has joined #oooq05:56
chkumar|ruckfiling a bz05:56
*** quiquell is now known as quiquell|rover05:56
quiquell|roverchkumar|ruck: What bz ?05:56
chkumar|ruckquiquell|rover: https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/legacy-periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset017-master/90fd211/logs/undercloud/home/zuul/tempest.log.txt.gz05:57
chkumar|ruckquiquell|rover: telemetry tests failure05:57
quiquell|roverchkumar|ruck: Yep, ykarel told me so on friday05:57
quiquell|roverchkumar|ruck: But Don't know why we didn't have a queens promotion05:57
chkumar|ruckykarel: do we have fix up for telemetry issue from ceilometer side05:57
quiquell|roverchkumar|ruck: All is suppose to be in place05:57
quiquell|roverchkumar|ruck: fs001 queens has fail :-/05:58
chkumar|ruckquiquell|rover: fs01 failed on rocky also checking05:59
ykarelchkumar|ruck, yes, sileht proposed upstream05:59
ykarelchkumar|ruck, is bug created for that issue ^^05:59
chkumar|ruckykarel: do we have a bug?05:59
quiquell|roverchkumar|ruck: Ok, open a bug if necessary, going to fix the master fs019 issue06:00
ykarelchkumar|ruck, no idea06:00
chkumar|ruckquiquell|rover: ok opening a bz06:00
ykarelchkumar|ruck, prposed fix was https://review.openstack.org/#/c/602625/ but it's failing in ci06:00
chkumar|ruckykarel: quiquell|rover https://review.openstack.org/#/c/602625/ andhttps://review.openstack.org/#/c/602624/106:00
chkumar|rucktwo fixes06:01
*** quiquell|rover is now known as quique|rover06:01
chkumar|ruckykarel: it might also needs fixes in puppet-ceilometer?06:01
chkumar|ruckhttps://review.openstack.org/#/c/602624/106:01
*** quique|rover is now known as quiquell|rover06:01
quiquell|roverchkumar|ruck: Add them to the etherpad to know what we have to focus on today06:02
ykarelchkumar|ruck, not sure about that06:02
chkumar|ruckquiquell|rover: sure06:02
ykarelbut good to create lp and set promotion-blocker06:02
*** holser|ptg has joined #oooq06:03
ykarelchkumar|ruck, many tests failed, not sure if ur recheck will help there06:03
ykarelon the ceilometer patch06:03
ykarelunit tests needs to be fixed there06:04
*** holser|p_ has joined #oooq06:05
*** jfrancoa has joined #oooq06:05
*** holser|ptg has quit IRC06:05
chkumar|ruckykarel: quiquell|rover https://bugs.launchpad.net/tripleo/+bug/179286206:08
openstackLaunchpad bug 1792862 in tripleo "[master] Telemetry Tempest integration tests failed giving Unable to complete operation on subnet b13c76ec-c851-48d0-91a9-245a2fdcad9b: One or more ports have an IP allocation from this subnet." [Critical,Triaged] - Assigned to Mehdi Abaakouk (sileht)06:08
quiquell|roverchkumar|ruck: ack06:08
*** jaganathan has joined #oooq06:11
ykarelchkumar|ruck, updated ceilometer patch: https://review.openstack.org/#/c/602624/, now should be ok06:16
*** rnoriega has quit IRC06:17
*** apetrich has joined #oooq06:18
chkumar|ruckquiquell|rover: on fs01 for queens and rocky I am seeing two issues related to ironic06:18
chkumar|ruckquiquell|rover: https://logs.rdoproject.org/openstack-periodic-24hr/git.openstack.org/openstack-infra/tripleo-ci/master/legacy-periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens/59f5f7c/logs/undercloud/var/log/ironic/app.log.txt.gz#_2018-09-16_06_30_14_14706:18
chkumar|ruckquiquell|rover: https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/legacy-periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-rocky/4820d62/logs/undercloud/var/log/extra/docker/containers/ironic_pxe_http/log/ironic/ironic-conductor.log.txt.gz#_2018-09-17_02_06_28_64406:19
chkumar|ruckquiquell|rover: opening two seperate bugs for both06:20
*** rnoriega has joined #oooq06:20
quiquell|roverchkumar|ruck: Feels related but ok06:20
*** quiquell|rover is now known as quique|rover|brb06:37
*** udesale has quit IRC06:38
*** udesale has joined #oooq06:40
*** udesale has quit IRC06:45
*** saneax has quit IRC07:04
*** quique|rover|brb is now known as quiquell|rover07:07
*** holser|p_ is now known as holser_07:24
*** ykarel is now known as ykarel|lunch07:45
hubbot1FAILING CHECK JOBS on stable/pike: tripleo-ci-centos-7-scenario004-multinode-oooq-container @ https://review.openstack.org/602248, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario004-multinode-oooq-container @ https://review.openstack.org/567224, stable/rocky: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7 (2 more messages)07:46
*** kopecmartin has joined #oooq07:52
quiquell|roverchkumar|ruck: queens fs001 has pass now https://logs.rdoproject.org/openstack-periodic-24hr/git.openstack.org/openstack-infra/tripleo-ci/master/legacy-periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens/2f0399c/08:07
chkumar|ruckquiquell|rover: for queens no jobs are blocking now i think08:09
chkumar|ruckfor promotion08:09
quiquell|roverchkumar|ruck: fs020 has to finish but looks good08:09
quiquell|roverchkumar|ruck: Going to cry if we have a queens promotion08:10
chkumar|ruckquiquell|rover: yes same here08:10
quiquell|roverchkumar|ruck: Going to close this https://bugs.launchpad.net/tripleo/+bug/179287208:14
openstackLaunchpad bug 1792872 in tripleo "[queens] overcloud prepare image failed by giving IronicAction.node.set_provision_state failed: 'NoneType' object has no attribute '__getitem_" [Critical,Triaged] - Assigned to Quique Llorente (quiquell)08:14
chkumar|ruckquiquell|rover: ack08:17
quiquell|roverchkumar|ruck: let's wait, maybe it's rdo infra issue08:18
chkumar|ruckack08:23
*** udesale has joined #oooq08:25
*** udesale has quit IRC08:27
*** udesale has joined #oooq08:28
*** udesale has quit IRC08:29
*** udesale has joined #oooq08:29
*** udesale has quit IRC08:29
*** udesale has joined #oooq08:30
*** bogdando has joined #oooq08:32
*** saneax has joined #oooq08:35
*** tosky has joined #oooq08:41
ssbarneahi! I am back!08:46
*** udesale has quit IRC08:47
chkumar|ruckquiquell|rover: I think we can add this also http://grafana.openstack.org/d/ACtl1JSmz/afs?orgId=1 in ruck rover dashboard for afs mirror status08:49
*** udesale has joined #oooq08:55
quiquell|roverchkumar|ruck: Agree08:56
chkumar|ruckquiquell|rover: I will check with fedora guys how they track the mirror sync status09:00
*** udesale has quit IRC09:01
quiquell|roverchkumar|ruck: The only thin we need from this dashboard is the mirror.centos time or do we also need fedora's ?09:02
quiquell|roverchkumar|ruck: queens fs020 pass :-), we have to see a promotion09:09
quiquell|roverchkumar|ruck: Yippiii!!! http://dashboard-ci.tripleo.org/d/cEEjGFFmz/cockpit?orgId=1&panelId=117&fullscreen09:10
quiquell|roverafter near a month :-)09:10
*** ykarel|lunch is now known as ykarel09:12
quiquell|roverchkumar|ruck: What are we missing for pike ? https://logs.rdoproject.org/openstack-periodic-24hr/git.openstack.org/openstack-infra/tripleo-ci/master/legacy-periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset019-pike/c396a8a/logs/undercloud/home/zuul/tempest.log.txt.gz09:13
quiquell|roverchkumar|ruck: fs019 is still failing at SharesCephFSTest09:14
quiquell|roverchkumar|ruck: fs050 is broken too at master, going to open lp and dig into it09:21
ssbarneaquiquell|rover: chkumar|ruck : i am still catching up with emails but if there is anything I can help with,  feel free to ping me.09:28
quiquell|roverssbarnea: Welcome back, we have being on-fire all the week, didn't even have time to check UUID issue :-/09:30
ssbarneaquiquell|rover: sure. i am sure that catching up will take me a lot of time.09:31
chkumar|ruckquiquell|rover: checking fs05009:31
chkumar|ruckquiquell|rover: fs019 for manila na?09:32
quiquell|roverchkumar|ruck: Looks like wrong ansible version09:32
quiquell|roverchkumar|ruck: what do you mean ?09:32
chkumar|ruckquiquell|rover: sorry ignore09:32
chkumar|ruckquiquell|rover: for fs019 manila pike https://review.openstack.org/601522 this review will do the job09:33
quiquell|roverchkumar|ruck: huff still not merged :-(09:35
quiquell|roverchkumar|ruck: I remember big backport needed for pike :09:35
chkumar|ruckquiquell|rover: yes it is still a workaround09:35
quiquell|roverchkumar|ruck: Let's try to merge stuff now that people are coming back09:35
chkumar|ruckquiquell|rover: is it a known issue http://logs.openstack.org/22/601522/4/check/tripleo-ci-centos-7-containerized-undercloud-upgrades/afb75fd/logs/undercloud/home/zuul/undercloud_upgrade.log.txt.gz#_2018-09-16_11_16_5509:37
chkumar|ruck?09:37
*** udesale has joined #oooq09:38
hubbot1FAILING CHECK JOBS on stable/pike: tripleo-ci-centos-7-scenario004-multinode-oooq-container @ https://review.openstack.org/602248, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario004-multinode-oooq-container @ https://review.openstack.org/567224, stable/rocky: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7 (2 more messages)09:46
*** arxcruz|ptg is now known as arxcruz10:11
*** kopecmartin has quit IRC10:15
*** dtantsur|afk is now known as dtantsur10:18
*** quiquell|rover is now known as quique|rover|afk10:19
quique|rover|afkjrist: unit test ar passing locally...10:25
ssbarneaquique|rover|afk: regarding UID, bdobrelya found another way to avoid it, so if we merge his patch, I would abandon mine. https://review.openstack.org/60201510:27
quique|rover|afkssbarnea: Cool thanks10:28
ssbarneaat least his patch is easier to review and lower risks10:28
quique|rover|afkssbarnea: LÃlet's wait for results, I diit out of nowhere10:29
*** jfrancoa has quit IRC10:35
*** jfrancoa has joined #oooq10:36
*** quique|rover|afk is now known as quiquell|rover10:49
*** panda has joined #oooq10:58
quiquell|roverchkumar|ruck: Can force run featureset019 at RDO '11:07
quiquell|rover?11:07
quiquell|roverchkumar|ruck: You have do something similar before for rocky and irnonic issue11:07
*** kopecmartin has joined #oooq11:15
chkumar|ruckquiquell|rover: let me try that11:18
chkumar|ruckquiquell|rover: fs019 is running here https://review.rdoproject.org/zuul/stream.html?uuid=506957ea0b7a44f6a63a17387625b129&logfile=console.log11:18
chkumar|ruckquiquell|rover *fs00111:19
chkumar|ruckquiquell|rover: once it finishes I will trigger fs01911:19
*** kopecmartin has quit IRC11:19
quiquell|roverchkumar|ruck: ack11:19
chkumar|ruckquiquell|rover: fs019 for rocky?11:23
quiquell|roverfs019 for master, to check that mirror is ok now11:24
chkumar|ruckquiquell|rover: ack11:24
quiquell|roverchkumar|ruck: thanks so much11:24
quiquell|roverchkumar|ruck: update CIX new / triage with the telemetry issue11:32
chkumar|ruckquiquell|rover: yup sure11:32
chkumar|ruckquiquell|rover: what to do two ironic related bugs11:32
chkumar|ruckquiquell|rover: fs01 rocky passed https://logs.rdoproject.org/43/13943/29/check/legacy-periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-rocky/506957e/11:33
quiquell|roverchkumar|ruck: Puff don't know, let's just say that they are related to RDO infra issues11:34
quiquell|roverchkumar|ruck: And that we are so lucky :-/11:34
quiquell|roverchkumar|ruck: They will say what to do with those11:34
*** kopecmartin has joined #oooq11:34
chkumar|ruckquiquell|rover: do we have any upstream review to link with fso19?11:34
chkumar|ruckwhich is not yet merged?11:34
quiquell|roverchkumar|ruck: https://review.openstack.org/60305411:36
chkumar|ruckquiquell|rover: https://review.rdoproject.org/r/#/c/13943/ added here11:39
chkumar|rucklet'see how it goes11:39
quiquell|roverchkumar|ruck: yep, can you add this to the launchpad or CIX card ?11:40
arxcruzykarel: chkumar|ruck https://review.rdoproject.org/r/#/c/16216/11:42
arxcruz\o/11:42
quiquell|roverchkumar|ruck: I not able to go to CIX can you cover it ?11:42
chkumar|ruckquiquell|rover: yup11:42
quiquell|roverchkumar|ruck: Will try to join as soon I finish here11:42
chkumar|ruckquiquell|rover: do we have bz for mirror issue?11:42
quiquell|roverYep11:43
quiquell|roverIs the old one https://bugs.launchpad.net/tripleo/+bug/179229611:43
openstackLaunchpad bug 1792296 in tripleo "Overcloude deploy error:Timed out waiting for messages from Execution" [Critical,In progress] - Assigned to Quique Llorente (quiquell)11:43
chkumar|ruckquiquell|rover: added rdo review here11:44
quiquell|roverchkumar|ruck: I think I have fill in all the stuff neded at the CIX card11:45
quiquell|roverchkumar|ruck: Check them to see if we have averything I have like 15 minutes or so11:45
hubbot1FAILING CHECK JOBS on stable/pike: tripleo-ci-centos-7-scenario004-multinode-oooq-container @ https://review.openstack.org/602248, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-queens, tripleo-ci-centos-7-scenario004-multinode-oooq-container @ https://review.openstack.org/567224, stable/rocky: tripleo-ci- (2 more messages)11:46
chkumar|ruckquiquell|rover: what is the duplicate card of this one https://trello.com/c/lxhDNvX4/744-cixlp1792296tripleociproa-overcloude-deploy-errortimed-out-waiting-for-messages-from-execution ?11:46
quiquell|roverchkumar|ruck: https://trello.com/c/zKpzUWCY/740-cixlptripleociproa-overcloude-deploy-errortimed-out-waiting-for-messages-from-execution11:47
*** zul has joined #oooq11:48
*** kopecmartin has quit IRC11:51
quiquell|roverssbarnea: you were not already back ?11:51
quiquell|roverssbarnea: ups old chat history, sorry :-)11:52
*** rfolco has joined #oooq12:00
*** panda has quit IRC12:00
weshayssbarnea, welcome back12:03
ssbarneaweshay: thanks!12:04
quiquell|roverchkumar|ruck: I am able to make it to CIX12:04
*** trown|outtypewww is now known as trown12:06
weshaychkumar|ruck, question to you.. https://trello.com/c/OpWmOkAZ/742-cixlp1792862tripleociproa-master-telemetry-tempest-integration-tests-failed-giving-unable-to-complete-operation-on-subnet-b13c7612:09
weshaychkumar|ruck, I don't know anything about the bug/fix but wondering why the issue was not caught by devstack... maybe the devstack tempest was updated?12:10
chkumar|ruckweshay: it is caught in POI12:10
weshayah k12:11
chkumar|ruckweshay: quiquell|rover https://review.rdoproject.org/r/1394312:12
chkumar|ruckhere we are testing the failed jobs12:12
ssbarneais sagi still away this week?12:12
*** skramaja has quit IRC12:13
weshaychkumar|ruck, quiquell|rover queens promoted.. that's great :)12:18
quiquell|roverweshay: We were watching it with pop corns12:21
chkumar|ruckweshay: https://review.rdoproject.org/etherpad/p/ruckrover-sprint1912:28
weshaychkumar|ruck, quiquell|rover great job.. help real quick find the card that is associated w/ fs019 failing in rocky12:30
chkumar|ruckweshay: quiquell|rover fs019 for rocky is passing https://review.rdoproject.org/zuul/builds.html?job_name=legacy-periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset019-queens12:31
chkumar|rucksorry12:31
quiquell|roverweshay: This week was a pain in the ass12:31
chkumar|rucksorry wrong info12:32
weshayquiquell|rover, chkumar|ruck ya.. I know .. thank you for staying on guard :)12:32
quiquell|roverweshay: Can we skipp scalation ?12:32
weshayquiquell|rover, chkumar|ruck ya.. I'll cover you guys12:32
quiquell|rovercool thanks12:33
*** rlandy has joined #oooq12:33
*** quiquell|rover is now known as quique|rover|lch12:33
* chkumar|ruck headed home12:34
quique|rover|lchssbarnea: This rings any bell ? https://ci.centos.org/job/tripleo-quickstart-gate-master-delorean-quick-basic/6964/consoleFull12:34
weshayquique|rover|lch, you posted a queens job12:35
weshayfor rocky :P12:35
quique|rover|lchweshay: where ?12:36
weshay<chkumar|ruck> weshay: quiquell|rover fs019 for rocky is passing https://review.rdoproject.org/zuul/builds.html?job_name=legacy-periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset019-queens12:36
arxcruzweshay: fs019 is scenario004 right ?12:36
weshayaye12:37
arxcruzweshay: chkumar|ruck i talked with gouthamr last week, he will send the patches to fix it12:37
arxcruzif it's related to manila12:37
quique|rover|lchchkumar|ruck: Do you have around the review you did to force run rocky job ?12:37
ssbarneaquique|rover|lch: chkumar|ruck : https://review.openstack.org/#/c/596841/ -- maybe a +W on it?12:38
weshayquique|rover|lch, chkumar|ruck so fs019 / scen004 https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/legacy-periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset019-rocky/3f03ed5/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz12:43
* weshay doesn't see the error yet12:44
weshayit's a puppet error12:44
weshayjust a firewall timeout? https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/legacy-periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset019-rocky/3f03ed5/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz#_2018-09-17_09_03_2112:46
weshayThe WebSocket timed out before the Workflow completed.12:46
*** udesale has quit IRC12:54
*** kopecmartin has joined #oooq12:55
*** kopecmartin has quit IRC12:55
*** kopecmartin has joined #oooq12:55
*** kopecmartin has quit IRC12:56
*** kopecmartin has joined #oooq12:56
mariosscrum?13:01
weshaylet's do it!13:01
weshayssbarnea, rlandy rfolco mtg13:02
*** toure|gone is now known as toure13:03
mariosssbarnea: congrats !13:03
*** quique|rover|lch is now known as quiquell|rover13:05
*** thrash|g0ne is now known as thrash13:12
quiquell|roverssbarnea: What happend ?13:13
quiquell|rovermarios: ?13:13
ssbarneaquiquell|rover: I got married last week.13:15
*** panda has joined #oooq13:16
quiquell|roverssbarnea: Ahh congratulations !!!13:16
ykarelssbarnea, Congrats!!13:17
weshaypanda, rlandy https://review.openstack.org/#/c/589448/3313:22
*** vinaykns has joined #oooq13:23
arxcruzssbarnea: congrats!!!13:28
rfolcossbarnea, best wishes, congrats!13:31
toskyoh, congratulations!13:32
quiquell|roverpanda, rlandy: We need to put this at te-broker https://review.openstack.org/#/c/58848813:34
quiquell|roverpanda, rlandy: We have miss it after the last reconstruction13:34
rlandyquiquell|rover: I'm with sagi on this .. "did you test it live on te-broker? worth to do it before we approve it"13:35
rlandyhard to approve unless we know it works13:35
rlandywe would need a live test :(13:36
rlandythe way te-broker works,13:36
rlandywe would have to edit on the instance itself13:36
quiquell|roverrlandy: So I apply the patch directly at te-broker ?13:36
quiquell|roverand try it ?13:36
quiquell|roverOr I start my own te-broker ?13:36
rlandyquiquell|rover: well, usually we would make the edit in /opt/stack on the actual te-broker instance13:37
rlandyand see if it works13:37
rlandyrevert of it does not13:37
quiquell|roverrlandy: will do13:37
rlandyte-broker gets refreshed from master every nigh13:37
rlandynight13:37
quiquell|roverrlandy: The review looks good ? I have add the UTF part13:37
rlandyquiquell|rover: afaict13:37
rlandypanda would be good to ask13:38
rlandyhe write the bmc log code13:38
rlandyif you can live test it, I can +2 it13:38
quiquell|roverrlandy: is panda around ?13:38
rlandyquiquell|rover: irc says he is13:38
quiquell|roverwill try to apply the change13:39
rlandybut you can't always believe what you see on line these days :)13:39
quiquell|roverrlandy: I like this way :-) realtime status is a sh...13:39
pandaI'm still coping with the effect of constant air conditioning at PTG13:39
ssbarneachkumar|ruck: I seen IronicAction.node.set_provision_state failed: 'NoneType' object has no attribute '__getitem__' (HTTP 500) on http://logs.rdoproject.org/15/602015/1/openstack-check/legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/f45dce8/logs/undercloud/home/zuul/overcloud_prep_images.log.txt.gz - is this known issue?13:40
quiquell|roverpanda: Are you throwing blood from your ears ?13:40
quiquell|roverssbarnea: Yep, RDO infra issue13:40
weshayquiquell|rover, panda http://38.145.33.166/console-logs/13:40
quiquell|roverpanda: Going to apply your te-broker patch https://review.openstack.org/#/c/588488 to te-broker directly13:41
quiquell|roverweshay: we are talking about it13:41
weshayhas that code been removed or you are waiting to redeploy it based on the review13:41
weshayquiquell|rover, ya13:41
pandaquiquell|rover: weshay the code should already be there13:41
quiquell|roverweshay: te-broker was re-constructed last week, so we lose that13:41
weshayquiquell|rover, ya13:41
quiquell|roverpanda: So I apply to live te-broker to test it ?13:42
rlandywhy was that code not in the automatic re-apply?13:42
rlandypanda: ^^ th elive apply was my suggestion13:42
quiquell|roverrlandy: Does not have to be merged ?13:42
rlandyquiquell|rover: it would have to be merge to re-apply every night with the refresh13:42
rlandyto just test it, you can manually apply13:43
rlandyand see what explodes13:43
quiquell|roverrlandy: so how did it work before, because the review was not merged by the logs were there13:43
quiquell|rovers/by/but/13:43
quiquell|roverok liked to work at redhat, going to break te-broker13:43
rlandyquiquell|rover: idk- that is what I was asking above13:44
pandaI applied the code live on te broker some days ago, but now that te-broken has been provisioned with our scripts, the tripleo-ci repo gets refreshed every 10 minutes, so to test any local modification we need to suspend the automatic refresh13:44
rlandyhow was that not in the rebuilt te-broker to begin with?13:44
pandaquiquell|rover: ^13:44
rlandyof we rebuild from our code base13:44
quiquell|roverpanda: How do I suspend automatic refresh, the ansible crontab line ?13:45
pandaquiquell|rover: root crontab13:45
panda*/5 * * * * flock -n -E 2 /var/lock/ansible-pull /usr/local/bin/ansible_pull.sh13:45
quiquell|roverpanda: ack13:46
hubbot1FAILING CHECK JOBS on stable/pike: tripleo-ci-centos-7-scenario004-multinode-oooq-container @ https://review.openstack.org/602248, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-queens, tripleo-ci-centos-7-scenario004-multinode-oooq-container @ https://review.openstack.org/567224, stable/rocky: tripleo-ci- (2 more messages)13:46
pandammmhhh13:46
pandawait13:46
quiquell|roverdone13:46
quiquell|rover:-//13:46
pandayep, ok13:47
pandathat's it13:47
quiquell|roverpanda: also the dashboard-ci part was missing https://review.rdoproject.org/r/#/c/16284/13:48
weshaypanda, rlandy you guys looking at that patch?13:51
weshayhttps://logs.rdoproject.org/48/589448/33/openstack-check/legacy-tripleo-ci-centos-7-container-to-container-upgrades-master/459ae33/job-output.txt.gz#_2018-09-13_14_34_26_33077713:51
* weshay wondering why 2018-09-13 14:34:26.330777 | primary | TASK [undercloud-setup : Run the TripleO-CI VXLAN networking script] ***********13:51
weshayis running on an undercloud upgrade job13:51
pandaweshay: is some of the task running, or they are just all skipped ?13:52
weshaypanda, it's running13:53
weshayhttps://logs.rdoproject.org/48/589448/33/openstack-check/legacy-tripleo-ci-centos-7-container-to-container-upgrades-master/459ae33/logs/undercloud/home/zuul/vxlan_networking.sh.log.txt.gz13:53
rlandyyes13:53
rlandythat vxlan networking taks has been broke forever13:53
quiquell|roverHumm rfolco you are looking something regarding this ?13:53
quiquell|roverWait imean jfrancoa13:53
quiquell|roverjfrancoa is debugging vxlan networking issues with upgrades at RDO13:54
rlandyI remember asking ssbarnea about this error a couple weeks ago13:56
rlandythere was a related bug I think13:56
quiquell|roverrlandy: ykarel say something about version of ovs13:56
rlandyquiquell|rover: truth is, we should not need that at all with reparenting13:56
jfrancoaquiquell|rover: weshay hey, yes..I'm trying to find the source of the problem and I'm somehow close. the vxlan_networking fails constantly in the upgrades jobs because we're not setting up the repos for the subnodes, only in the undercloud. So when it tries to install ovs in the subnode-1, it's a really old version and the script breaks13:57
weshayjfrancoa, rlandy panda is that job even valid?13:57
ykarelquiquell|rover, rlandy afair repos were not configured on overcloud nodes, so ovs was installed from centos base,13:57
weshaythat is a multinode container upgrade job13:57
rlandythe script doesn't pass on the undercloud either13:57
weshayjfrancoa, which is duped by fs051 isn't it?13:58
jfrancoaweshay: well, it should..it's a multinode containers upgrade with several services enabled, but it has been failing for very long13:58
*** agopi is now known as agopi|afk|ncsu|c13:58
*** agopi|afk|ncsu|c is now known as agopi|afk|car-fa13:58
*** agopi|afk|car-fa is now known as agopi|afk13:58
rlandyhttps://logs.rdoproject.org/48/589448/33/openstack-check/legacy-tripleo-ci-centos-7-container-to-container-upgrades-master/459ae33/logs/undercloud/home/zuul/vxlan_networking.sh.log.txt.gz#_2018-09-13_14_34_3013:58
jfrancoaweshay: fs051 is keystone only, which is not very realistic. it's good to test the workflow, but we were having this other one running in RDO to test the upgrade enabling other services13:58
weshayjfrancoa, we have a new plan of attack w/ regards to update/upgrade testing13:59
weshaymarios, had the blueprint approved13:59
jfrancoarlandy: exactly, it you see any other job which runs it (and it's not a mixed_upgrade job), it would install 2.0.9 version13:59
weshayjfrancoa, my thought is to nuke that job13:59
rlandyjfrancoa: I'd like to nuke this altogether14:00
rlandynot the job, the vxlan networking14:00
jfrancoaweshay: holser_ mentioned something about that, it seems there were some conversations about updates/upgrades CI jobs in the PTG14:00
rlandyit's only a legacy thing14:00
weshayjfrancoa, ya.. the vxlan script is legacy..14:00
holser_jfrancoa ... wait for chem14:00
chkumar|ruckquiquell|rover: https://review.rdoproject.org/r/#/c/13943/14:00
weshaythat job is not providing value.. ----> NUKE14:01
chkumar|ruckquiquell|rover: https://review.rdoproject.org/zuul/stream.html?uuid=7dee50d4a27c41de81350d1dc14df9d0&logfile=console.log14:01
jfrancoaweshay: anyway, the same is happening in the ffu one, which does provide value14:01
weshayrlandy, rfolco do we have any tht patches on top of https://review.openstack.org/#/c/589448/33 to test?14:02
weshayjfrancoa, point me at that job please14:02
quiquell|roverchkumar|ruck: Have already pass introspection, so we are good14:02
jfrancoaweshay: rlandy: so, we  can get rid entirely of this vxlan_networking thing? I tried disabling it in the featurset, but it didnt work (probably multinode-rdo took precendence for the variables) https://review.openstack.org/#/c/600816/14:02
chkumar|ruckweshay: fs019 rocky I am already testing it there in rdo noop jobs14:02
weshaychkumar|ruck, perfect.. thanks14:02
jfrancoaweshay: https://logs.rdoproject.org/16/563616/12/openstack-check/legacy-tripleo-ci-centos-7-multinode-1ctlr-featureset036-oc-ffu-queens/811195e/14:03
quiquell|roverWe are not launching RDO CI at rocky THT :-/14:03
jfrancoaweshay: it's a little bit old, but I guess it's still happening, if it's happening in the other RDO upgrades jobs14:03
rlandyweshay: not that I know of - I have tht patches on top of my work to test - can probably reuse14:04
quiquell|roverpanda: Do I have to do anything else ? I still don't see any logs here http://38.145.33.166/console-logs/14:04
rlandyjfrancoa: not quite yet14:04
weshayjfrancoa, that job last ran on 9/414:04
weshayjfrancoa, it last passed on 7/1114:04
pandaquiquell|rover: CONSOLE_LOG_PATH is missing14:04
chkumar|ruckssbarnea: this is the bug https://bugs.launchpad.net/tripleo/+bug/179287214:05
openstackLaunchpad bug 1792872 in tripleo "[queens] overcloud prepare image failed by giving IronicAction.node.set_provision_state failed: 'NoneType' object has no attribute '__getitem_" [Critical,Triaged] - Assigned to Quique Llorente (quiquell)14:05
rlandyjfrancoa: we are desperately trying to get rid of legacy. the non-legacy stuff uses a different role to set bridge14:05
mariosweshay: sorry was on the phone checking14:05
rlandyfor the moment, we still need it14:05
pandaquiquell|rover: then you'll have to wait for a test env to be destroyed14:05
mariosweshay: which blueprint?14:05
weshaymarios, standalone upgrade14:05
chkumar|ruckarxcruz: manila issue is handled , only pike and ocata is blocked due to undercloud upgrade job failure14:05
mariosweshay: didn't know we had one ;)14:05
chkumar|ruckalso we need to work on removing ocata jobs14:06
mariosweshay: i just checked the spec i need to update it https://review.openstack.org/#/c/579854/ some last comments and remove the WIP14:06
weshaymarios, jfrancoa imho.. we kill 3rd party upgrade jobs, focus on blueprint work14:06
marioswe can land it this week weshay14:06
weshay+114:06
weshayrlandy, panda jfrancoa for the context of the patch.. I'm ignoring the non-voting upgrade job14:07
rlandyweshay:ack - I left a similar comment on the patch14:07
rlandywhen I reviewed it before14:07
jfrancoaweshay: you mean marios' blueprint? and the idea would be enabling these legacy jobs in the standalone deployment? In my opinion, it would be helpful having a FFU job upstream, as well as a closer-to-reality upgrades job14:08
quiquell|roverpanda: /var/www/html/tebroker/console-logs/ this is the path ?14:08
pandaquiquell|rover: yes14:08
weshayjfrancoa, /me firing up blue jeans for a few minutes to discuss upgrades w/ jfrancoa14:08
weshayanyone is welcome14:08
weshayhttps://bluejeans.com/u/whayutin/14:09
weshayjfrancoa, have a sec?14:09
rlandyquiquell|rover: weshay: for https://review.openstack.org/#/c/589448/33 - the reviews listed in the comment, can they be abandoned?14:09
jfrancoaweshay: joining14:09
rlandyit says this review includes them14:09
weshayrlandy, the reviews?14:09
weshayin the commit msg?14:09
rlandyThis is include the following reviews from [1]: - https://review.openstack.org/#/c/585714 - https://review.openstack.org/#/c/587340 - https://review.openstack.org/#/c/587360 - https://review.openstack.org/#/c/58734414:09
weshayplease update that.. rlandy  :)14:09
weshaythanks14:09
rlandyweshay: well, I am not sure what the update should be - which is why I am asking14:10
rlandydo we abandon those revuews14:10
rlandyreviews14:10
rlandyand remove them from the comment?14:10
rlandyor actually consider them?14:10
quiquell|roverrlandy: I think so, we are not going to go back to them, and I put back the comments on them in the all-in-one14:11
quiquell|roverrlandy: drop them14:11
rlandyquiquell|rover: fine, I will abandon them and update the commit message14:12
rlandyweshay: ^^14:12
quiquell|roverrlandy: btw, we are forcing vxlan at common.yaml, don't know if this is going to interfiere with future RDO jobs running new workflow14:12
rlandyquiquell|rover: what do you mean by forcing?14:13
rlandythe new workflow set that to false14:13
rlandyie;running our role14:13
quiquell|roverrlandy: and also multinode-rdocloud.yaml is putting them to true, so don't know wich one take preference14:14
rlandyquiquell|rover: right now, in rdo cloud,we still run this vxlan networking role14:15
rlandywhen we reparent we should not - and rely on the one from zuul-jobs I think14:15
rlandywe will see that when we reparent14:15
quiquell|roverrlandy: ok, then we will ahve to remove those false from anywhere14:16
quiquell|roverpanda: console-logs working now14:19
pandaquiquell|rover: ok, you want to patch the review to include CONSOLE_LOG_PATH ?14:19
quiquell|roverpanda: Have already add it to the review14:19
quiquell|roverpanda: I think the review have all the stuff now, take a look14:20
quiquell|roverok drop now14:21
rlandyquiquell|rover: can we vote on ^^ yet?14:21
rlandyte-broker tested?14:21
*** quiquell|rover is now known as quique|rover|off14:22
quique|rover|offrlandy: working fine14:22
quique|rover|offykarel: is happy now14:22
quique|rover|offhttp://38.145.33.166/console-logs/14:22
ykarel:)14:22
rlandywith live patch?14:22
quique|rover|offrlandy: yep14:22
rlandyok - voting14:22
rlandyhttp://logs.openstack.org/45/602445/2/check/tripleo-ci-centos-7-containerized-undercloud-upgrades/ba720d5/job-output.txt.gz#_2018-09-17_13_45_42_73386314:27
rlandycontaierized_upgrades is going to fail removal job14:27
*** dtrainor has joined #oooq14:30
rlandyhttp://logs.openstack.org/45/602445/2/check/tripleo-ci-centos-7-containerized-undercloud-upgrades/ba720d5/logs/undercloud/home/zuul/undercloud_upgrade.log.txt.gz#_2018-09-17_13_45_4014:31
rlandy^^ worth rechecking?14:31
rlandyjfrancoa: ^^?14:33
*** ykarel is now known as ykarel|away14:33
rlandytripleo-ci-centos-7-containerized-undercloud-upgradesSUCCESS in 1h 43m 18s - passed in last run14:34
weshayrlandy, panda have a master and pike tht test patch on 58944814:51
weshayvisible in https://review.openstack.org/#/q/topic:toci_jobtype+(status:open+OR+status:merged)14:51
weshaypanda, please vote before the end of your day14:51
*** ykarel|away has quit IRC14:53
chkumar|ruckis someoone working on this https://ci.centos.org/job/tripleo-quickstart-gate-master-delorean-quick-basic/7017/ to make it passing?14:55
weshaychkumar|ruck, https://ci.centos.org/job/tripleo-quickstart-gate-master-delorean-quick-basic/14:56
weshaychkumar|ruck, pip errors? https://ci.centos.org/job/tripleo-quickstart-gate-master-delorean-quick-basic/7049/console14:57
weshayor clone14:57
chkumar|ruckweshay: chekcing14:59
arxcruzweshay: chkumar|ruck will we have scrum today ?15:01
chkumar|ruckarxcruz: yes,15:02
arxcruzk15:02
weshayarxcruz, chkumar|ruck ya15:02
weshaypanda, opt/stack/new?15:03
weshaypanda, ??15:04
pandaweshay:  sorry, what's the context ?15:05
weshaypanda, your patch https://review.openstack.org/#/c/602673/15:06
chkumar|ruckweshay: will check tomorrow15:06
weshaychkumar|ruck, k15:06
weshaypanda, where is that file logged?15:06
weshayzuul-variables.yaml15:07
pandadumpend in tripleo_root15:07
weshaypanda, show me the log15:07
pandaweshay: the file is not logged. And yes probably tripleo_root is not the right location to put that file15:08
pandaweshay: I'll move it to local_working_dir15:09
weshaypanda, either way.. just log it15:11
weshaybut agree.. local_working_dir15:11
*** thrash has quit IRC15:20
rlandy ResourceInError: resources.NovaCompute: Went to status ERROR due to "Message: No valid host was found. There are not enough hosts available., Code: 500"15:28
rlandyugh this again - in pike :(15:28
rlandyweshay: tripleo-ci-centos-7-containerized-undercloud-upgrades killed us in  https://review.openstack.org/#/c/602445/ - is there a way to recheck while it's busy with the current run?15:30
rlandymy recheck did not take15:30
arxcruzchkumar|ruck: weshay: which bj ?15:30
chkumar|ruckarxcruz: https://bluejeans.com/7050859455/https://bluejeans.com/7050859455/15:30
chkumar|ruckhttps://bluejeans.com/7050859455/15:30
chkumar|ruckkopecmartin: weshay ^^15:31
weshayjoining15:31
weshaysory15:31
chkumar|ruckweshay: fs019 rokcy faild https://logs.rdoproject.org/43/13943/31/check/legacy-periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset019-rocky/7dee50d/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz#_2018-09-17_14_15_3115:32
hubbot1FAILING CHECK JOBS on stable/pike: tripleo-ci-centos-7-scenario004-multinode-oooq-container @ https://review.openstack.org/602248, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-queens, tripleo-ci-centos-7-scenario004-multinode-oooq-container @ https://review.openstack.org/567224, stable/rocky: tripleo-ci- (2 more messages)15:46
bogdandoweshay, ssbarnea: PTAL https://review.openstack.org/#/c/576746/15:48
bogdandoweshay: one more https://review.openstack.org/#/c/592996/15:49
bogdandoaaand https://review.openstack.org/#/c/595265/15:49
bogdandothere is also https://review.openstack.org/#/c/595566/ which boosts provisioning by 100% or so15:50
bogdandonot for CI tho, just for users15:50
*** vinaykns has quit IRC15:55
*** saneax has quit IRC15:56
*** ykarel has joined #oooq15:59
weshayrlandy, ok.. caught up w/ you..  running recheck again should rekick it from scratch16:11
weshayrlandy, I just did it.. watching16:11
rlandyno dice when I did it16:12
weshayhrm.. k16:12
rlandybut maybe it likes you better16:12
*** marios is now known as marios|hometime16:12
rlandyweshay: tbh, the actual failure worries me more16:12
rlandyit passed on the last run16:12
rlandyso I doubt it's related16:12
*** marios|hometime is now known as marios16:13
weshayrlandy, hrm.. I think tripleo-ci-centos-7-containerized-undercloud-upgrades may need to go nv16:13
weshayquique|rover|off, chkumar|ruck fyi ^16:13
weshayrlandy, oh.. wait16:13
weshaywe have a patch on that16:13
weshaysec16:13
rlandyoh great16:13
weshayhow r the gate jobs at 97%16:13
*** holser_ has quit IRC16:14
rlandystatistics ?!?16:15
rlandycan you in patch - can I depend on it?16:16
weshayrlandy, https://review.openstack.org/#/c/602550/16:16
weshayhttps://review.openstack.org/#/c/603026/16:16
weshayand finally https://review.openstack.org/#/c/590774/16:16
weshayrlandy, in reverse order16:16
rlandyhmmm ... well not sure it's worth depending on those pathces16:21
rlandythey may not merge16:21
weshayya16:24
rlandybut if I don;t ... the test will fail16:27
rlandyworth the recheck?16:27
rlandyweshay: ack - ok I think we need to go non-voting here until these patches merge. I can put in the change for your review16:28
weshayrlandy, you can dep on the tht change16:28
weshaythat will merge16:28
rlandyis that enough?16:28
rlandybesides other jobs will fail here16:29
weshayrlandy, the job passes there16:29
rlandyI see16:29
weshayrlandy, worth a shot imho16:29
* rlandy see how close those gates are to passing16:30
rlandyhmmm ... indeterminate16:30
*** jfrancoa has quit IRC16:30
rlandyok - let's try the depends16:30
rlandyhere we go again16:32
weshaybogdando, when did vars_prompt become a thing16:32
weshaypretty cool16:32
bogdandoindeed16:32
weshaypanda, rlandy check this out https://docs.ansible.com/ansible/2.6/user_guide/playbooks_prompts.html16:33
bogdandoweshay: did you manage to deploy f28 with quickstart libvirt?16:33
bogdandoI think I did16:33
bogdandofrom the cloud image16:33
weshaybogdando, I've gotten to the python-tripleoclient16:33
bogdandowith https://review.openstack.org/#/c/603069/16:34
weshaybogdando, which cloud image did you use?16:34
bogdandoweshay: https://github.com/bogdando/oooq-warp/blob/3e6dc629df0d3e6a3f99a164d4e836abe2b1e6a1/vars/fedora28.yaml#L1016:35
rlandykind of cool - not sure we ever ask the users for their input though16:35
weshaybogdando, I wish there was idempotent command for that16:35
rlandyreproducer maybe16:35
weshayrlandy, reproduceer16:35
weshaymaybe16:35
weshay:)16:35
bogdandoweshay: I think the whole thing is wrapped with idempotency16:35
rlandylol - agreed16:35
weshaynot sure if that is > than bash16:35
weshaybogdando, bah16:35
bogdandolike then you run provision based on tags/skip16:35
weshaybogdando, on my list today16:36
bogdandoor images expiring rules16:36
bogdandoIIUC16:36
bogdandoso basically that section never runs on a mutated qcow216:36
bogdandoit's either fully skipped, or always applies from the scratch16:36
bogdando(IIUC :D)16:37
bogdandoso it's fully atomic with other virt-customize commands around16:37
weshaybogdando, is there some doc on https://review.openstack.org/#/c/595566/3/roles/modify-image/tasks/libguestfs.yml16:38
weshayforce_tcg16:38
* weshay is googling16:38
weshaygot it http://libguestfs.org/guestfs.3.html#force_tcg16:38
bogdandoweshay: that thing WFM16:39
bogdandoI put some benchmarking results into comments16:39
bogdandosee 23. Aug 14:2916:39
bogdandooops, not really, it's 15:01 weshay16:40
*** bogdando has quit IRC16:48
*** ykarel is now known as ykarel|away16:50
*** kopecmartin has quit IRC16:51
*** dtantsur is now known as dtantsur|afk16:54
weshayrlandy, so https://trello.com/c/TVsZ3Ut6/877-clean-up-rdo-sf-legacy-code-after-zuulv3-migration is up 2 date and accurate ?17:10
*** vinaykns has joined #oooq17:10
weshaypanda, this needs QE https://trello.com/c/JN4UIjWv/854-replace-zuulchanges-handling-in-build-test-packages-role17:10
*** trown is now known as trown|lunch17:18
rlandyweshay: well, depends if we consider panda's patch to be part of it17:25
rlandythe lagacy zuul vars17:25
rlandyI see it has its own card though17:25
*** dsneddon has quit IRC17:30
*** dsneddon has joined #oooq17:30
weshayplaybooks/tripleo-ci/vars/common.yaml17:34
weshayLine 23:17:34
weshaytoci_gate_test.sh.j2 L130 uses nodes_file17:34
weshaywhile17:34
weshaytoci_quickstart.sh.j2 L125 uses nodes_args17:34
weshaynodes_args includes "--extra-vars" to make the logic around string manipulation easier and keep consistent with the rest of extra-vars...17:34
weshayhttps://review.openstack.org/#/c/58944817:35
weshayrlandy, fyi https://review.openstack.org/#/c/603187/1/zuul.d/undercloud-jobs.yaml17:36
rlandyyes  thank you17:36
*** tbarron_ is now known as tbarron17:41
hubbot1FAILING CHECK JOBS on stable/pike: tripleo-ci-centos-7-scenario004-multinode-oooq-container @ https://review.openstack.org/602248, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-queens, tripleo-ci-centos-7-scenario004-multinode-oooq-container @ https://review.openstack.org/567224, stable/rocky: tripleo-ci- (2 more messages)17:46
rlandyweshay: you mentioned that the  container image prepare command changed. Is that merged/updated in CI yet?18:24
weshayrlandy, oh ya18:24
weshayrlandy, there have been some changes that happened w/o widespread knowledge18:24
weshayrlandy, read through that image-modify role18:25
rlandyweshay: still working on rhos-14 in the background18:25
weshayk18:25
rlandylooking  - I think I have a few fixes18:25
*** trown|lunch is now known as trown18:32
rlandyhmmm ... don;t see any changes in CI18:41
rlandyugh - no dice on depends-on THT review18:51
rlandyrebased on nv job18:53
*** ykarel|away has quit IRC18:53
weshayrlandy, /me shows u19:01
weshaysec19:01
weshayrlandy, https://github.com/openstack/ansible-role-tripleo-modify-image19:02
weshayhttps://review.openstack.org/#/q/status:open+project:openstack/tripleo-docs+branch:master+topic:bp/container-prepare-workflow19:02
weshayrlandy, and ALL https://review.openstack.org/#/q/status:merged+topic:bp/container-prepare-workflow19:03
*** zul has quit IRC19:08
*** jtomasek has quit IRC19:17
*** jtomasek has joined #oooq19:20
rlandyoh I see19:21
*** zul has joined #oooq19:25
rfolcotripleo-ci-centos-7-containerized-undercloud-upgrades is problematic weshay ?19:28
rlandyrfolco: ack19:29
rlandy<weshay> rlandy, fyi https://review.openstack.org/#/c/603187/1/zuul.d/undercloud-jobs.yaml19:29
weshaysec19:30
rfolcorlandy, thanks19:30
rlandymirror vexxhost???19:31
rlandyhttp://logs.openstack.org/45/602445/5/check/tripleo-ci-centos-7-containers-multinode/c01eace/logs/subnode-2/home/zuul/repo_setup.log.txt.gz#_2018-09-17_19_22_1319:32
rlandyrfolco: weshay: ^^ fyi19:33
rlandyfailing repo setup19:33
rlandychecking iwth sf-dfg19:35
*** jtomasek has quit IRC19:37
*** jtomasek has joined #oooq19:40
weshayrlandy, I just wrote up a blocking bug19:41
weshayrlandy, /me decides to speak w/ bugs19:41
weshayrlandy, vs.. english19:41
rlandyweshay: probably a wise switch ... english communication got no response for me19:43
rlandyoh my gosh, will we ever get this patch through ?????19:44
weshayrlandy, we might as well chat on blue19:45
weshayI have a question for you as well19:46
hubbot1FAILING CHECK JOBS on stable/pike: tripleo-ci-centos-7-scenario004-multinode-oooq-container @ https://review.openstack.org/602248, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens, tripleo-ci-centos-7-scenario004-multinode-oooq-container @ https://review.openstack.org/567224, stable/ocata: legacy-tripleo-ci-centos-7 (2 more messages)19:46
rlandyok - joining blue19:46
weshayrlandy, https://bugs.launchpad.net/tripleo/+bug/179300619:53
openstackLaunchpad bug 1793006 in tripleo "yum repo failures in upstream mirrors" [Critical,Triaged]19:53
*** jtomasek has quit IRC20:22
*** zul has quit IRC20:32
*** agopi|afk is now known as agopi21:06
*** trown is now known as trown|outtypewww21:07
hubbot1FAILING CHECK JOBS on stable/pike: tripleo-ci-centos-7-scenario004-multinode-oooq-container @ https://review.openstack.org/602248, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens, tripleo-ci-centos-7-scenario004-multinode-oooq-container @ https://review.openstack.org/567224, stable/ocata: legacy-tripleo-ci-centos-7 (2 more messages)21:46
rlandyweshay: hi21:56
weshayrlandy, hey21:56
rlandyweshay: https://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/tripleo-quickstart-rhos-14-passed-p1-baremetal-hp_dl360_envE-single_nic_vlans/7/console got as far as the tempest run - so we are getting there ...21:56
rlandyalso21:56
* rlandy copies ...21:56
weshaywow21:56
weshayrlandy, oh.. it was trying to setup tempest on the container21:57
weshayError: image rhosp14/centos-binary-tempest:current-tripleo not found21:57
rlandyhttps://paste.fedoraproject.org/paste/wLXlXMjIhL0CjM49dlP-JQ21:57
rlandy^^ we need something like that for the changes21:57
weshayI wonder if internal is building that yet21:57
rlandywill test out and update21:57
rlandyI will ask21:58
rlandyanyways - will work on the change thing22:00
rlandybut that's the logic22:00
rlandyrerunning rhos-14 with tempest settings changes22:20
*** vinaykns has quit IRC22:34
*** rlandy is now known as rlandy|bbl22:50
*** tosky has quit IRC23:23
hubbot1FAILING CHECK JOBS on stable/pike: tripleo-ci-centos-7-scenario004-multinode-oooq-container @ https://review.openstack.org/602248, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens, tripleo-ci-centos-7-scenario004-multinode-oooq-container @ https://review.openstack.org/567224, stable/ocata: legacy-tripleo-ci-centos-7 (2 more messages)23:46
*** dsneddon has quit IRC23:52

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!