Thursday, 2020-03-05

*** Tengu has quit IRC00:20
*** Tengu has joined #oooq00:22
*** dsneddon has quit IRC00:40
*** dsneddon has joined #oooq00:45
*** ChanServ sets mode: +o panda00:46
*** jmasud has joined #oooq00:47
*** panda changes topic to "Docs: https://docs.openstack.org/tripleo-quickstart/latest/ || 5th Mar promoter patch to review https://review.rdoproject.org/r/24721 || next promoter patch in the chain https://review.rdoproject.org/r/24774"00:47
*** panda sets mode: -o panda00:47
*** jmasud has quit IRC00:54
*** jmasud has joined #oooq00:56
*** Goneri has quit IRC02:24
*** sanjayu_ has joined #oooq02:42
*** sanjayu_ has quit IRC02:53
*** raukadah is now known as chandankumar03:02
*** dsneddon has quit IRC03:51
*** apetrich has quit IRC04:00
*** udesale has joined #oooq04:09
*** rlandy|bbl is now known as rlandy04:29
*** skramaja has joined #oooq04:29
rlandychandankumar: hey ... https://review.rdoproject.org/r/#/c/25618/ should be in a mergeable state now. Note the change to scenario010. Had mixed results with scenario00704:51
rlandyputting up a testproject to retest that04:52
chandankumarrlandy, will take care of that04:52
rlandyhttps://review.opendev.org/#/c/708599/ is in gate now04:53
rlandygoing to sleep - back in 7 hours04:53
*** rlandy has quit IRC04:53
*** ykarel|away is now known as ykarel04:55
*** udesale has quit IRC05:31
*** udesale has joined #oooq05:39
*** dsneddon has joined #oooq06:12
chandankumarykarel, can you check this job https://logserver.rdoproject.org/12/25612/7/check/distgit-tripleo-master-testing-centos-8-standalone-scenario010/1efbfb9/logs/undercloud/var/log/extra/package-list-installed.txt.gz06:25
chandankumarI am not sure packages from temp repo got installed or not06:25
ykarelchandankumar, looking06:28
ykarelchandankumar, u mean gating-repo or temp repo?06:28
*** marios has joined #oooq06:29
chandankumarykarel, temp repo06:29
ykarelchandankumar, in logs i see @temp-distgit06:29
ykarelso should be used06:29
ykarelany specific example u looking for?06:29
ykarelwhich is not updated06:29
chandankumarykarel, I want to make sure unpinned packages get used06:30
ykarelthose worked in my tests both on host and containers06:30
ykarelbut if u doubt on some specific package i can check again06:30
chandankumarok coming to office ttyl06:31
ykarelack06:31
*** weshay|ruck has quit IRC06:36
*** saneax has joined #oooq06:36
*** weshay has joined #oooq06:36
*** weshay has quit IRC06:40
*** weshay has joined #oooq06:40
*** jfrancoa has joined #oooq06:52
*** dsneddon has quit IRC07:01
*** ccamacho has quit IRC07:02
*** weshay has quit IRC07:13
*** weshay_ has joined #oooq07:16
*** jbadiapa has joined #oooq07:24
*** weshay_ has quit IRC07:34
*** weshay_ has joined #oooq07:34
*** ykarel is now known as ykarel|lunch07:35
*** weshay_ has quit IRC07:39
*** migi has quit IRC07:40
*** tesseract has joined #oooq07:52
*** weshay_ has joined #oooq07:56
*** ccamacho has joined #oooq08:11
*** weshay_ has quit IRC08:24
*** weshay has joined #oooq08:25
*** jpena|off is now known as jpena08:26
*** saneax has quit IRC08:29
*** saneax has joined #oooq08:30
*** ykarel|lunch is now known as ykarel08:32
*** amoralej|off is now known as amoralej08:45
*** bogdando has joined #oooq08:46
*** marios has quit IRC08:56
*** pojadhav is now known as pojadhav|lunch09:01
*** pojadhav|lunch is now known as pojadhav09:01
pandaarxcruz|rover: can't stop thinking now about a parody itaalian song from about the same period. It goes "I suffer from stress, yeah I suffer from stress // I'm tired and out of shape // I play in a boy band, I play in a boy band // There must be some mistake"09:05
arxcruz|roverpanda: where can I find this preciousity?09:06
*** irclogbot_2 has quit IRC09:08
pandaarxcruz|rover: Yuo asked for it, you've got it. https://www.youtube.com/watch?v=Z1UO91TmOsY09:08
arxcruz|roverOMG, I can watch youtube video directly from glowing-bear09:13
arxcruz|roverthanks dmellado for the tip :D09:13
dmelladoxD09:13
dmellado\o/09:13
dmelladoglad you liked it09:14
arxcruz|roverpanda: I have to say, I only know Laura Pausini italian09:14
pandaarxcruz|rover: ....09:14
arxcruz|roverI mean, I only understand italian from Laura Pausini songs :D09:15
arxcruz|roverand Renato Russo in Italian, which is pretty good09:15
*** apetrich has joined #oooq09:24
*** tosky has joined #oooq09:25
*** holser has joined #oooq09:28
*** sshnaidm|afk is now known as sshnaidm09:37
arxcruz|roverapetrich: ping09:39
arxcruz|roverapetrich: are you aware of this mistral error? https://logserver.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset002-master-upload/e7bbed3/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz09:40
apetricharxcruz|rover, heya09:40
apetricharxcruz|rover, how big is that file? Still loading09:41
arxcruz|roverapetrich: not sure...09:41
*** irclogbot_2 has joined #oooq09:43
apetricharxcruz|rover, what is that file? still loading09:43
arxcruz|roverapetrich: really?09:43
arxcruz|roverapetrich: http://paste.openstack.org/show/790326/09:44
apetrichjust finished09:44
amoralejarxcruz|rover, i've finished moving deps for centos8-master, let me know if you find any issue09:44
arxcruz|roveramoralej: ok i'll09:45
apetrich1.6Mb is not that big. weird09:45
*** openstackstatus has quit IRC09:45
apetricharxcruz|rover, so. kind of. We seen those but I'm not sure we have a patch for it09:49
arxcruz|roverapetrich: I see this on the job yesterday, the same job today fails in another way, but I'm not sure yet, if was before or after mistral run09:50
sshnaidmchandankumar, hi09:50
chandankumarsshnaidm, Hello09:51
chandankumarsshnaidm, https://review.rdoproject.org/r/#/c/25689/09:53
sshnaidmchandankumar, are 007 and 010 working on c8?09:55
chandankumarsshnaidm, yes09:55
sshnaidmcool09:55
*** bogdando has quit IRC09:55
chandankumarsshnaidm, https://review.opendev.org/711287 and https://review.opendev.org/71133009:55
*** bogdando has joined #oooq09:56
arxcruz|roverapetrich: so, it fails today before the mistral, so not sure if it was fixed, I'm running on testproject right now to check10:01
*** derekh has joined #oooq10:13
sshnaidmpanda, can you take a look please at http://38.102.83.109/centos7_queens.log - there is "Exception: Named Hashes Changed!"10:17
*** marios has joined #oooq10:17
arxcruz|roverapetrich: can you check if there's a patch already? Otherwise i'll open a bug10:22
arxcruz|rovermarios: regarding https://logserver.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset002-master-upload/e7bbed3/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz did you open a bug?10:22
apetricharxcruz|rover, sure. one minute10:22
arxcruz|roveri see it in some runs yesterday, today it fails in a different step, before mistral runs so, not sure if it's persistent, i'm running the job again in testproject to check10:23
*** dtantsur|afk is now known as dtantsur10:24
mariosarxcruz|rover: looking10:28
mariosarxcruz|rover: (but if it isn't on the etherpad no)10:29
apetricharxcruz|rover, yeah I think we need a bug10:29
mariosarxcruz|rover: yeah it looks like that one10:29
apetrichlol marios10:29
mariosarxcruz|rover: https://bugs.launchpad.net/tripleo/+bug/186603110:29
openstackLaunchpad bug 1866031 in tripleo "periodic centos7 fs2 upload/fs1 master fails overcloud deploy pcs "create constraint failed"" [Critical,Fix released] - Assigned to Sagi (Sergey) Shnaidman (sshnaidm)10:29
mariosapetrich: :) which bit10:31
arxcruz|rovermarios: apetrich sshnaidm okay, so, I did not see this today, however, the job fails before this step, so I'm running right now in testproject10:31
mariosapetrich: arxcruz|rover: confirmed same i see 2020-03-03 03:31:09 |     Error: /Stage[main]/Tripleo::Profile::Pacemaker::Ovn_dbs_bundle/Pacemaker::Constraint::Order[ip-192.168.24.16-with-ovndb_servers]/Pcmk_constraint[order-ovn-dbs-bundle-ip-192.168.24.16]/ensure: change from 'absent' to 'present' failed: pcs -f /var/lib/pacemaker/cib/puppet-cib-backup20200303-10-1ku4ixs create constraint failed: Error:10:31
mariosResource 'ip-192.168.24.16' does not exist10:31
mariosapetrich: arxcruz|rover: fix from sshnaidm https://review.opendev.org/#/c/711201/10:31
pandasshnaidm: it's a check to be use taht nothing modified the dlrnapi while we are promoting. It's either a) a known bug I thought was fixed, b) something really chenged dlrnapi while you were promoting. Investigating.10:36
pandasshnaidm: s/use/sure10:36
pandasshnaidm: yep, bug, it was fixed while I was refactoring, so maybe I did not ported the fix correctly. The previous promotion did not update the global promotions status. It's generally harmless, if you rerun, the status will be consiste again, since it's per-invocation. I'll try to fix it.10:46
*** udesale has quit IRC10:49
chandankumarsshnaidm, good to go https://review.opendev.org/#/c/710961/ minion one10:55
sshnaidmpanda, ack, thanks11:01
*** jmasud has quit IRC11:04
*** jmasud has joined #oooq11:04
*** chem has quit IRC11:35
*** tesseract-RH has joined #oooq11:40
*** tesseract has quit IRC11:43
chandankumarsshnaidm, where we are testing image build changes11:54
chandankumar?11:54
chandankumarmay be job link would be helpful11:55
chandankumaror is it current ovb wip patch in rdo-jobs?11:55
sshnaidmchandankumar, just remove here to_build: false https://review.rdoproject.org/r/#/c/25332/36/zuul.d/ovb-jobs.yaml11:55
chandankumarsshnaidm, ok will get to that11:56
chandankumarbrb11:58
*** marios is now known as marios|call12:01
marios|callarxcruz|rover: weshay: we have sync?12:01
arxcruz|rovermarios|call: yes, we are waiting for you12:01
marios|callwut, did i join wrong one? from calendar meet.google.com/ixv-xbhp-xzg12:01
arxcruz|rovermarios|call: https://meet.google.com/ixv-xbhp-xzg12:01
*** rlandy has joined #oooq12:02
rlandychandankumar: hey  ... merging https://review.rdoproject.org/r/#/c/25689/12:05
*** amoralej is now known as amoralej|lunch12:06
rlandyoh we should remove the old jobs12:07
rlandyI'll fix that12:08
*** jpena is now known as jpena|lunch12:09
*** dpawlik has quit IRC12:14
*** dpawlik has joined #oooq12:15
rlandychandankumar: pls review https://review.opendev.org/#/c/711287/12:15
rlandyie: vote12:15
*** dpawlik has quit IRC12:22
chandankumarrlandy, done12:23
chandankumarrlandy, ah, I forgot12:23
rlandychandankumar: https://review.rdoproject.org/r/#/c/25700/12:23
rlandychandankumar: I checked with weshay about fs002 - he said it could be removed but I just want to confirm that12:24
rlandy^^ patch can merge12:24
chandankumarrlandy, ok12:24
chandankumarsshnaidm, ^^12:24
chandankumarrlandy, sshnaidm https://review.rdoproject.org/r/#/c/25699/12:24
rlandychandankumar: ^^ let's use that review to add new jobs to criteria12:25
rlandyadd all the ones we added12:25
chandankumarexcept ovb we got all the jobs I think12:26
rlandychandankumar: fs002? yeah so I don;t know if we are still promoting anything12:26
rlandyie: changing the hash12:26
rlandyif yes, I would leave fs002 - will confimr12:26
chandankumarrlandy, arxcruz|rover filed a bug related to fs0212:27
rlandychandankumar: yep, according to meeting, sshnaidm fixed it12:27
rlandyif it's the same one12:27
arxcruz|roverchandankumar: there is already a bug...12:27
chandankumarrlandy, not yet12:28
chandankumarrlandy, https://review.opendev.org/#/c/711081/ going to work on it tonight to make compatile to c812:28
ykarelchandankumar, need to hold https://review.opendev.org/#/c/711287/3..5/zuul.d/standalone-jobs.yaml12:29
ykarelwrong parent12:29
chandankumarone min fixing it12:29
ykarelack12:30
chandankumarykarel, done12:30
chandankumarykarel, https://review.opendev.org/#/c/711287/12:31
chandankumarsshnaidm, https://review.opendev.org/#/c/711287/ +2 and +w12:31
ykarelok now12:31
chandankumarrlandy, rlandy https://review.rdoproject.org/r/#/c/25698/1/zuul.d/projects.yaml is this one correct way of testing component jobs?12:32
rlandychandankumar: you don't want force_periodic12:33
rlandyit should use current tripleo for other components12:34
rlandyalso promote_source should be defined12:34
chandankumarpromote_source: component-ci-testing will work?12:34
*** dpawlik has joined #oooq12:36
chandankumarrlandy, https://review.opendev.org/#/c/711408/ good to go12:46
rlandysec - adding jobs12:47
chandankumararxcruz|rover, weshay rlandy https://bugs.launchpad.net/tripleo/+bug/186618413:05
openstackLaunchpad bug 1866184 in tripleo "centos-8 multinode overcloud deploy failed with heat_resource_tree_params = heat_resource_tree['parameters'] KeyError: 'parameters'" [Critical,Confirmed]13:05
chandankumarykarel, ^^13:05
weshaychandankumar, please join13:06
weshaypanda, please join13:06
ykarelchandankumar, ack13:06
arxcruz|roverchandankumar:  thanks, do you have a patch already ? just to add on the rr hackmd13:08
chandankumararxcruz|rover, nope13:08
*** udesale has joined #oooq13:14
*** cgoncalves has quit IRC13:41
weshayrlandy, check if container-builds are usng ceph repos from rdo13:44
*** cgoncalves has joined #oooq13:45
weshayhttps://hackmd.io/HrQd03c9SxOMtFPFrq50tg?both13:48
weshaychandankumar, https://review.opendev.org/71132213:49
chandankumarweshay, https://bugs.launchpad.net/tripleo/+bug/186618413:52
openstackLaunchpad bug 1866184 in tripleo "centos-8 multinode overcloud deploy failed with heat_resource_tree_params = heat_resource_tree['parameters'] KeyError: 'parameters'" [Critical,Confirmed]13:52
chandankumarweshay, https://review.rdoproject.org/r/2561213:59
chandankumarrlandy, add the require jobs definition here https://review.rdoproject.org/r/#/c/25659/  and add the respective jobs here by editing .zuul.yaml file in this review https://review.rdoproject.org/r/#/c/25612/ and you might also need to rebase this review https://review.rdoproject.org/r/#/c/25551/14:04
chandankumarand we are done14:04
chandankumarrlandy, please update the review only when jobs finishes14:05
chandankumarit takes approx 6 hours to complete14:05
rlandyok- will check it out - thanks14:05
rlandychandankumar: can you take a look at the standalone tempest review ... jobs runs 6 hours - no logs14:17
rlandyjob times out in three places14:17
chandankumarrlandy, sure, link please14:17
rlandyhttps://review.rdoproject.org/r/#/c/25325/14:17
chandankumarrlandy, log collection is also timing out sudo unbound-control dump_cache14:19
chandankumarsudo unbound-control dump_cache14:19
chandankumar2020-03-03 21:43:11.398921 | TASK [warn when collect logs timed out (SIGTERM or SIGKILL used)]14:19
chandankumar2020-03-03 21:43:11.476915 | ERROR: Collect logs timed out14:19
*** marios has joined #oooq14:25
*** marios|call has quit IRC14:25
rlandychandankumar: job times out in three places14:25
*** ykarel is now known as ykarel|away14:27
chandankumararxcruz|rover, rlandy I have updated the concurrency let's see how it goes14:27
chandankumarmoved to 414:27
chandankumarhttps://review.rdoproject.org/r/#/c/25325/14:28
rlandychandankumar: thanks - will watch it14:28
* chandankumar headed home14:28
*** jpena|lunch is now known as jpena14:29
*** amoralej|lunch is now known as amoralej14:31
*** Goneri has joined #oooq14:31
*** TrevorV has joined #oooq14:32
*** chem has joined #oooq14:33
*** cgoncalves has quit IRC14:39
*** cgoncalves has joined #oooq14:40
rlandyweshay: you moving the taiga tasks or should I ?14:49
weshayrlandy, it's tricky.. there's a button for it.. let's do it together later so you see how14:50
rlandyweshay: ack14:50
weshaysshnaidm++ re: vexx perf email14:51
arxcruz|roverweshay: what's the rhel8 issue you're talking about ?14:51
weshayarxcruz|rover, aye. ./me looks sec14:51
arxcruz|roveri see the rados one, that is waiting to get promoted14:51
rlandysshnaidm: what's the deal with scenario012? https://github.com/openstack/tripleo-heat-templates/blob/master/ci/environments/scenario012-standalone.yaml - still relevant for master?14:52
rlandywith novaless deploy14:52
sshnaidmrlandy, this is a poor job14:52
sshnaidmrlandy, tried to make it working, but there is some bad stuff in heat templates probably14:53
rlandysshnaidm: ok - so we'll leave it for the baremetal component now ...14:53
rlandywhen there is some value, we can test it there14:53
sshnaidmrlandy, I think it might be fine to have it, but somebody more familiar with tht should fix it finally14:53
rlandythat's where we'll get the biggest value14:53
rlandysshnaidm: stevebaker?14:53
sshnaidmrlandy, I think more about EmilienM :)14:54
sshnaidmrlandy, I think he was already looking into it once, EmilienM right? (about standalone012 job)14:54
rlandyit's been failing for a while - so really we are no worse off14:55
EmilienMhi14:55
rlandybut it would be great to have that in the baremetal component pipeline14:55
rlandyEmilienM: hi ...14:55
rlandyscenario01214:55
rlandyis there any worth in moving that to centos-8?14:56
weshayarxcruz|rover, checking the latest deployments.. looks like we get through the deployment and just fail on the known bug..14:56
weshayarxcruz|rover, so I guess we're good14:56
rlandywith the new noveless deploy?14:56
EmilienMI haven't looked at standalone012 for a long time14:56
weshayarxcruz|rover, it will fail until we promote.. which will be a little while14:56
rlandyEmilienM: so it's been non-voting forever - so we really haven't  had coverage there14:57
arxcruz|roverweshay: ack14:57
rlandyEmilienM: it's a test we considered adding to the baremetal component pipeline14:58
weshayhelp w/ https://review.opendev.org/#/q/topic:centos-8-jobs+(status:open+OR+status:merged)14:58
rlandybut it's not essential if we have OVB there14:58
weshaysshnaidm, https://review.opendev.org/#/c/711320/15:00
weshaysshnaidm, rlandy https://review.opendev.org/#/c/710810/15:03
*** d0ugal has joined #oooq15:07
rlandyweshay: comments added https://review.opendev.org/#/c/71081015:08
*** jaosorior has quit IRC15:20
weshayrhanks15:32
weshaythanks15:32
weshaysshnaidm, bug on fs001 coming up?15:33
sshnaidmweshay, yeah15:33
weshaythanks15:33
sshnaidmweshay, https://bugs.launchpad.net/tripleo/+bug/186620215:37
openstackLaunchpad bug 1866202 in tripleo "OVB on centos8 fails because of networking failures" [Critical,Triaged]15:37
mariosrlandy: so how does the coordination look like on https://hackmd.io/HrQd03c9SxOMtFPFrq50tg?both do you have sync calls? /me wondering where /how to get involved15:55
weshaymarios, we have nothing for scenario010-ovn-provider-standalone15:56
weshayhttps://hackmd.io/HrQd03c9SxOMtFPFrq50tg?view#Standalone15:56
rlandymarios: we were syncing every morning15:56
rlandyI'll set that meeting up again15:56
mariosweshay: rlandy: ack15:58
rlandymarios: I can also take you through the component pipeline and you can help there15:58
mariosrlandy: ack let me look at scen10 and i can join your next sync call and take from there.15:59
rlandymarios: ack16:00
rlandyweshay: setting up 1Pm UTC daily sync calls again - know that clashes with some of your other meetings. If you want to change that time, let me know16:01
*** Trevor_V has joined #oooq16:01
*** dsneddon has joined #oooq16:01
mariossshnaidm: minor addition to README about ssh tunnel (I wasted some time on that today might be useful for others, like use RDO vm for grafana development) https://review.rdoproject.org/r/2570616:04
*** TrevorV has quit IRC16:05
mariosweshay: but scen10 ovn not for periodic right just check?16:06
mariosweshay: currently no periodic centos7 afaics16:06
weshaymarios, any voting check job requires a periodic16:06
rlandymarios: chandankumar: weshay: panda: sync meeting invite sent for this sprint16:06
mariosweshay: its non voting so i assumed it was not in gate16:07
weshaymarios, k.. let's convert to centos-8 and add a periodic16:07
mariosweshay: and so i was not suprised to not find a periodic for it16:07
mariosweshay: ack so we do want it in gate & periodic for c816:07
weshaymarios, add it to criteria but # it out as non-voting atm16:08
mariosweshay: ack when i get there ^16:08
weshaymarios, we want it in check and periodic, not gate16:08
mariosweshay: voting in check but not in gate?16:08
weshayis it voting or non-voting in check atm?16:08
mariosweshay: non16:08
weshayok.. so leave it non-voting16:09
sshnaidmmarios, ack, btw you can just remove "127.0.0.1" here: https://github.com/rdo-infra/ci-config/blob/master/ci-scripts/infra-setup/roles/rrcockpit/files/docker-compose.yml#L50:L5016:09
mariosweshay: oooci-jobs.sh:  *** COLLECTED URLs ***  http://zuul.openstack.org/job/tripleo-ci-centos-7-scenario010-ovn-provider-standalone http://codesearch.openstack.org/?q=tripleo-ci-centos-7-scenario010-ovn-provider-standalone https://github.com/openstack/tripleo-ci/blob/master/zuul.d/standalone-jobs.yaml#L595 http://zuul.openstack.org/builds?job_name=tripleo-ci-centos-7-scenario010-ovn-provider-standalone16:09
mariosweshay: sorry wrong paste16:09
mariosweshay: that https://github.com/openstack/tripleo-ci/blob/7c68cda8b4390a58b4bdbbfed2a2fd87584d25f6/zuul.d/standalone-jobs.yaml#L59616:09
marios(non voting)16:09
sshnaidmmarios, and then go to http://public_ip:808016:09
mariossshnaidm: oh nice, that would have been easier ;)16:10
mariossshnaidm: but i thin kyou'd still need the tunnel for the port from floating ip16:10
weshaymarios, is it clear or need clarification?16:10
sshnaidmmarios, no, just connect to floating ip, 8080 need to be opened by sec group16:11
mariosweshay: well :D some conflicting info, but i guess you want it in check only?16:11
mariosweshay: as is currently?16:11
weshayin nonvoting-check, in periodic, no-gate job, in criteria but remarked out w/ a #non-voting atm comment16:12
mariosweshay: or non voting in check, like curently but with periodic not in creiteria16:12
mariosack weshay ok ^16:12
weshaymarios, people add jobs .. on or off the team.. and fail to add something to periodic16:12
weshayfucks the system up :)16:12
mariosweshay: hmm... the job is in pretty bad shape by the way no green at all http://zuul.openstack.org/builds?job_name=tripleo-ci-centos-7-scenario010-ovn-provider-standalone16:14
mariosmaybe thats why its not gating/voting16:14
weshayit's because there is no periodic16:15
weshaythis happens all the time16:16
weshayalmost 100% when someone not on our team adds jobs16:16
chandankumarrlandy, weshay panda I tested the emit release patch here https://review.rdoproject.org/r/#/c/25694/16:16
chandankumarbut periodic-tripleo-ci-centos-8-singlenode-featureset027-masterTIMED_OUT in 4h 56m 16s16:17
chandankumarperiodic-tripleo-ci-centos-8-multinode-1ctlr-featureset010-masterSUCCESS in 1h 47m 14s16:17
chandankumarsshnaidm, can you take a look at collect logs https://logserver.rdoproject.org/94/25694/1/check/periodic-tripleo-ci-centos-8-singlenode-featureset027-master/782dec9/job-output.txt16:18
chandankumarafter timeout logs are not getting collected16:18
rlandyok16:18
chandankumarwarn when collect logs timed out (SIGTERM or SIGKILL used16:18
chandankumarseen in full tempest and these jobs16:18
weshaychandankumar, that is the fastest fs10/containers-multinode I've ever seen16:19
weshayI hope that happens upstream16:19
weshaychandankumar, so we clear to merge https://review.opendev.org/#/c/710441/16:20
weshayif hte timeout was logs.. I would say yes16:20
chandankumarok16:20
weshaychandankumar, please vote16:20
weshaywf if you have the votes16:21
weshaypanda, ^16:21
rlandyhttps://review.opendev.org/#/c/711305/ - should that be w+'ed?16:27
rlandychandankumar: ^^16:28
chandankumarrlandy, yes, as this is a dependent on emit release16:28
chandankumarrlandy, will I not?16:28
weshaychandankumar, fyi http://codesearch.openstack.org/?q=tripleo-ci-centos-7-containers-undercloud-minion&i=nope&files=&repos=16:29
weshaychandankumar, want me to update those repos?16:29
chandankumarweshay, I will take care of that tomorrow16:29
weshaychandankumar, I'll do it now16:29
chandankumarweshay, ok16:29
sshnaidmchandankumar, this jobs is messed up completely16:30
chandankumarsshnaidm, sorry which one minion?16:30
mariosrlandy: i see you're working on the scen10 standalone... https://review.opendev.org/#/c/711287/6/zuul.d/standalone-jobs.yaml  any idea what the difference with the ovn-provider one is ?16:30
* marios looking at definitions cant see it16:30
sshnaidmchandankumar, seems like failed in sudo unbound-control dump_cache, if it doesn't happen again just ignore it16:30
rlandychandankumar: https://review.opendev.org/#/c/710441/ is not in16:30
mariosrlandy: same featureset and vars afaics https://github.com/openstack/tripleo-ci/blob/7c68cda8b4390a58b4bdbbfed2a2fd87584d25f6/zuul.d/standalone-jobs.yaml#L575-L59516:31
mariosonly the tempest differs16:31
rlandythe depends-on is the wrong way around16:31
mariosweshay: do you know (difference between scen10 and scen10 ovn)16:31
rlandychandankumar: pls check that16:31
rlandymarios: getting16:31
weshaymarios, brent put up that job... I don't have much background on it16:32
chandankumarrlandy, un workflowed that16:32
rlandychandankumar: we put the depends-on the wrong way :(16:32
mariosweshay: k i can try reach out to him too.. but afaics they have same featureset, same standalone environment file. only tempest differs16:33
chandankumarfixing that16:33
mariosweshay: https://github.com/openstack/tripleo-ci/blob/7c68cda8b4390a58b4bdbbfed2a2fd87584d25f6/zuul.d/standalone-jobs.yaml#L575-L59516:33
rlandymarios: take a look at ^^16:34
chandankumarrlandy, fixed16:34
mariosrlandy: yeah?16:34
* marios playing spot the difference 16:35
mariostempest differs yes but something else i missed rlandy ?16:35
rlandymarios: yep - seems like just tempest16:36
mariosrlandy: ack thanks for sanity check i can reach out to beagles16:36
mariosbeagles: (we are trying to understand difference between scen10 standalone and scen10 ovn-provider standalone.. looks like just tempest difference? do we really need both jobs?16:36
chandankumarrlandy, is it about scenario 10?16:37
marioschandankumar: yes16:37
chandankumarrlandy, there we run validate-tempest16:37
rlandymarios: those jobs are a bit of a mystery - if you compare rdo and check - they are actually different16:37
chandankumarso stats might vary16:37
rlandyyou'll note I changed the centos-8 one16:37
beaglesmarios, yes we need both because the tempest configs are distinct and afaik we aren't able to run tempest twice in the same CI job16:37
mariosrlandy: chandankumar: k i'll post the centos8 see what happens and same time try to understand the difference or if we can combine the two jobs eventually16:37
chandankumarrlandy, only fs I thinj16:37
mariosbeagles: ack thanks for confirming16:37
rlandychandankumar: yeah ...16:38
marioswell there we go then16:38
rlandyI changed it to 062 for two reasons16:38
rlandy1. carlos commented16:38
rlandy2. ykarel noticed the jobs was missing pieces16:38
rlandy062 seemed more complete but old16:38
rlandymarios: ^ may need an update in general16:39
mariosrlandy: ack gonna start with job definition + testproject run and see where it breaks ;)16:40
rlandycool16:41
rlandychecking out the component fails - no idea what hit there16:41
*** udesale has quit IRC16:44
*** Trevor_V has quit IRC16:47
* marios hometime16:47
mariospanda: ack on the invite for tomorrow16:48
*** ccamacho has quit IRC16:48
*** TrevorV has joined #oooq16:50
chandankumarweshay, rlandy based on today's result on this review https://review.rdoproject.org/r/#/c/25612/ is it ok, unpin few packages and move it to a seperate review and keep testing with others?16:57
weshaychandankumar, NOOO16:57
weshaychandankumar, you mean merge?16:57
weshayor just split the tests?16:58
rlandyI think chandankumar means unpin a few in a review16:58
rlandynot upin and let it fly16:58
weshaythat's fine16:58
weshaychandankumar, don't freak me out man16:58
weshay:)16:58
rlandywaiting on https://review.opendev.org/#/c/711156/16:58
chandankumarweshay, based on job results we can break this giant review https://review.rdoproject.org/r/#/c/25612/ in small chunck16:58
weshaychandankumar++16:58
weshaythat's fine..16:59
*** marios is now known as marios|out16:59
chandankumarthen if anythign got missed, we have componenet pipeline to check16:59
rlandyweshay: pls check https://review.rdoproject.org/r/#/c/25702/16:59
weshayjust don't merge until we have it planned :)16:59
chandankumarotherwise we are delaying the pining16:59
chandankumar*unpinning16:59
rlandyyep - https://review.rdoproject.org/r/#/c/25702/ will merge as soon as depends-on does16:59
rlandyneed tempest and ovb17:00
*** tesseract-RH has quit IRC17:01
weshayrlandy++17:02
*** bogdando has quit IRC17:06
chandankumarrlandy, https://logserver.rdoproject.org/25/25325/14/check/periodic-tripleo-ci-centos-8-standalone-full-tempest-master/333fe90/logs/undercloud/var/log/tempest/tempest_run.log.txt.gz17:14
chandankumarwith concurreny 4 job ended in less than 3 hours17:14
chandankumarbut we have 2 failures17:14
rlandychandankumar++17:15
rlandythat job was driving me crazy before with no logs17:16
rlandy Details: {'type': 'IpAddressGenerationFailure', 'message': 'No more IP addresses available on network 2a5727e8-73b3-4191-82dd-51ed66fb8d3e.', 'detail': ''}17:16
rlandy^^ does that look like a temp failure?17:16
chandankumarrlandy, may be race condition or resource leakage also17:17
rlandychandankumar: ok - going to spin more17:17
rlandybut merge jobs review17:17
rlandychandankumar: thoughts about a component skip list?17:17
weshayrlandy, ya.. that's harder than you think17:18
weshaychandankumar, rlandy let's section out a part in the current skip list file for component failures...17:18
chandankumarrlandy, skip list is a big topic17:18
weshaybut do not add that feature yet17:18
chandankumarI will drop a mail tomorrow what is my thoughts on that17:18
rlandychandankumar: ack that17:18
weshaybut we can diliniate in text in case we have time in future sprints17:18
rlandyI'll spin again to see if that failure is consistent17:19
weshaylet's just add these to the normal skip list and bug it, cix etc17:19
chandankumarweshay, rlandy for current situation , replace all concurrency to 3 or 417:19
weshayrlandy, aye.. twice is good17:19
rlandybut it'd good enough to add the job definition17:19
weshaychandankumar, I would think 417:19
chandankumarless test 3 and more tests keep it 417:19
weshayrlandy++++17:19
weshayyes17:19
rlandyweshay: that's chandankumar17:19
rlandys ++17:19
rlandynot mine :)17:19
weshayyes.. chandankumar+++++++++++++++++++++++17:19
weshayas well.17:19
weshayrlandy, but add it to the component pipeline now17:19
weshayand we'll add the skips17:20
weshayif needed17:20
rlandyweshay: ack - merging job definition and spinning17:20
chandankumarrlandy, trick is here https://review.rdoproject.org/r/#/c/25325/14/.zuul.yaml17:20
weshaychandankumar, we could make the skip per component a seperate file and set the var in zuul17:20
chandankumarjsut append 2 lines you are done17:20
rlandywe can assess the skip thing tomorrow17:20
weshaybut we'll cross that bridge17:20
weshaychandankumar, we'll run into trouble if components have overlapping temest tests17:21
weshaywhich they will17:21
rlandyspin 2 in progress17:21
weshaywe closing out work faster than we can plan17:21
weshay++++17:21
rlandydon't worry upstream zuul is slowing us down bit17:21
weshayha17:21
weshayrlandy, so far the zuul forecast today is pretty sunny17:22
weshaywe just need to get contianers-multinode.. fixed and voting asap17:22
weshaybecause devels LOVE to break that job17:22
rlandyweshay: sat a small prayer for https://review.opendev.org/#/c/711156/17:22
rlandysay17:22
rlandywhich one id the fix review?17:23
rlandyis17:23
weshayrlandy, we'll know in 3hours17:24
weshay71148017:24
weshayfyi.. my list gets longer https://review.opendev.org/#/q/topic:centos-8-jobs+(status:open+OR+status:merged)17:24
chandankumarweshay, pass it to me I will take care in morning, sorry not in a mood to smash the patches17:25
weshaychandankumar, no prob man17:25
weshaychandankumar, I have minion added17:25
chandankumarweshay, minions merged na?17:25
weshayyes17:25
chandankumarI am watching this https://review.rdoproject.org/zuul/stream/3b94973fe3a64bc48cd4109549b70dc2?logfile=console.log17:26
weshayChange has been successfully merged by Zuul17:26
chandankumarweshay, after this patch, we need to coordinate to deal with neutron unpinning and ovn repo mess17:26
weshayk17:27
chandankumarweshay, rlandy for neutron we need to get all secnario in component which we run in check17:27
chandankumarnetwork is always a problem17:28
rlandychandankumar: please comment on the hackmd in the networking component line and I'll fix it17:28
chandankumarrlandy, ok17:28
rlandychandankumar: thanks - just for tracking here17:28
rlandyas we will share that with the dfs17:29
*** marios|out has quit IRC17:35
*** holser has quit IRC17:36
*** dtantsur is now known as dtantsur|afk17:40
*** derekh has quit IRC18:00
chandankumarrlandy, thanks for pushing the criteria18:05
rlandysure18:05
rlandylist updated in hackmd18:06
rlandychandankumar: dude, don't you sleep?? it's late18:07
chandankumarrlandy, will be sleeping soon18:07
rlandy:)18:07
chandankumarlast night didnot got too much sleep due to early morning  Pycon U.S. Code of Conduct Incident Response workshop it was at 4:30 AM18:08
rlandywow18:09
rlandythere is no   Pycon India Code of Conduct Incident Response workshop?18:09
chandankumarrlandy, nope, we are going to create that and taking it from US pycon18:10
chandankumarfor PyCon India 202018:10
chandankumarThanks to Sage sharp for the workshop, It was very hard to record the report when you hear something wrong has happened to someone in confernce and very hard to take measures to follow up ad take proper action so that it does not happen again without revealing the identity of reporter18:12
chandankumarsee ya18:17
chandankumarweshay, rlandy drop comments, will take over tomorrow18:18
*** chandankumar is now known as raukadah18:18
rlandyraukadah: ack - will have tempest review for you18:19
rlandythanks18:19
*** ccamacho has joined #oooq18:20
*** Goneri has quit IRC18:23
*** jbadiapa has quit IRC18:23
*** jpena is now known as jpena|off18:26
*** Goneri has joined #oooq18:27
*** amoralej is now known as amoralej|off18:35
raukadahweshay, rlandy amoralej|off ykarel|away Here is the status report on upinning https://review.rdoproject.org/r/25612 standalone plain 2,3,4 ,7,10 undercloud passed,18:44
raukadahfull tempest still running18:45
raukadahfs01 and fs010 known failure https://logserver.rdoproject.org/12/25612/10/check/distgit-tripleo-master-testing-centos-8-multinode-1ctlr-featureset010/7aee69a/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz18:45
raukadahso we are good there18:45
weshayvery cool18:45
weshayraukadah, don't work on https://review.opendev.org/#/c/711081/18:45
weshaybtw18:45
raukadahweshay, aye18:46
weshayrlandy, raukadah sshnaidm we're going to "to_build: false" in centos-8 for now for ovb jobs18:46
raukadahweshay, but I need to find a way to enjict current role in tq18:46
weshayraukadah, it's easy. I'll wip a patch18:46
weshayraukadah, but really a distraction atm18:47
raukadahweshay, ok, I will take it tomorrow18:47
raukadahrlandy, on emit release https://logserver.rdoproject.org/94/25694/2/check/periodic-tripleo-ci-centos-8-singlenode-featureset027-master/b8de8cd/job-output.txt18:49
raukadahstill timeout at tempest18:49
raukadahsomething wrong there needs to be investigated18:50
weshayraukadah, don't worry about images for now.. focus on the pinning18:50
weshayand other centos8 work18:50
raukadahweshay, we got awesome mate rlandy so no worry and marios joining the club also18:51
raukadahand sshnaidm & panda in background18:51
pandaweshay: raukadah all green again in https://review.rdoproject.org/r/24774, and code coverage on the promoted (not the stage) is above 80%. I'll remove the WIP tomorrow and we are ready for reviews.18:53
*** jaosorior has joined #oooq19:08
rlandynice19:10
rlandyraukadah: I think I know what's wrong with images19:11
rlandychecking if fix went through19:12
rlandyhttps://review.opendev.org/#/c/711408/19:17
rlandyraukadah: ^^ should fix image build - checking19:18
sshnaidmweshay, I think we can stop running 039 on master, right?19:27
weshaysshnaidm, nope.. those fuckers got it working19:27
weshay:)19:27
weshaylolz19:27
sshnaidm:(19:27
weshayleave it for now.. just switch it19:27
weshayif it doesn't work.. fine.. but let's keep it19:27
sshnaidmok19:27
raukadahrlandy, thanks19:28
rlandytripleo-ci-centos-7-containers-multinode-steinFAILURE in 3h 14m 55s19:28
rlandyUGH19:28
rlandyreall???19:29
rlandyhttps://review.opendev.org/#/c/711156/19:29
raukadahfs039 is still a thing, i think we are dropping in favor of tls standalone19:29
rlandyback again in check19:29
weshayrlandy, 2020-03-05 19:08:34 | 2020-03-05 19:08:17Z [overcloud-Controller-uzibk2hhweqt.0]: UPDATE_FAILED  resources[0]: Parameter 'ControllerImage' is invalid: Error validating value 'overcloud-full': No images unique match found for {'name': u'overcloud-full'}.19:30
rlandyweshay: where did ^^ come from?19:31
weshayrlandy, ur failing job on stein19:31
rlandyweshay: same job passed in check19:31
weshayaye.. it would have to19:31
rlandytripleo-ci-centos-7-containers-multinode-stein SUCCESS in 2h 30m 21s19:32
rlandyso now??19:32
weshayrlandy, I know.. gates happen19:38
weshaysorry?19:38
weshaytripleo is phunny sometimes/19:39
weshay?19:39
rlandyweshay: not your fault :)19:40
weshayya.. but frustrating..19:41
rlandygoing to work on the updates job - will forget about that for a few hours19:41
weshayDAM https://review.opendev.org/#/c/711480/ just failed in check :)19:41
*** saneax has quit IRC19:42
rlandyweshay: check it out ... periodic-tripleo-ci-centos-8-standalone-full-tempest-mastertestprojectmastercheck25325,14106002020-03-05T17:22:58SUCCESS20:23
rlandyhttps://review.rdoproject.org/zuul/build/0085ea6c14c7469f80149f4b94aa046320:24
rlandyis that legit passing???20:24
rlandymerging https://review.rdoproject.org/r/#/c/25717/20:24
weshayrlandy, LEGIT! Status: Pass 1524 Skip 17220:26
rlandyok - component pipeline here we come20:27
* rlandy is spending too much time texting with my nephew ... picking up his favorite word 'legit'20:28
weshaytoo legit to quit20:43
*** TrevorV has quit IRC20:44
*** sshnaidm is now known as sshnaidm|afk20:48
*** cgoncalves has quit IRC20:53
*** cgoncalves has joined #oooq20:56
*** jfrancoa has quit IRC20:57
*** cgoncalves has quit IRC21:10
rlandyweshay: https://review.rdoproject.org/r/#/c/25416/21:10
*** cgoncalves has joined #oooq21:11
*** dpawlik has quit IRC21:18
*** dsneddon has quit IRC21:48
*** jmasud has quit IRC22:26

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!