Wednesday, 2018-08-15

« Tuesday, 2018-08-14 Index Thursday, 2018-08-16 »
*** agopi has quit IRC00:32
*** rlandy has quit IRC00:43
*** agopi has joined #oooq00:57
*** rfolco|rover is now known as rfolco02:12
*** openstack has joined #oooq02:33
*** ChanServ sets mode: +o openstack02:33
*** kopecmartin has joined #oooq05:27
*** apetrich has joined #oooq05:42
*** jaosorior has joined #oooq06:25
*** jaosorior has quit IRC06:40
*** jaosorior has joined #oooq07:15
*** dtantsur|afk is now known as dtantsur08:26
*** gkadam has joined #oooq09:30
chandankumarweshay: panda|ruck: kopecmartin https://review.openstack.org/#/c/591982/ updated skip list09:32
panda|ruckchandankumar: the test value is a regular expression ?09:39
chandankumarpanda|ruck: yes, tempest will handle that09:39
panda|ruckchandankumar: thanks09:40
chandankumarpanda|ruck: I am on holiday today09:41
chandankumarso less available here09:42
panda|ruckchandankumar: get out of here!09:42
*** zul has quit IRC10:25
*** panda|ruck is now known as panda|ruck|lunch11:25
*** panda|ruck|lunch is now known as panda|ruck11:53
*** rfolco|rover has joined #oooq12:00
weshaypanda|ruck, you can drop if you want12:11
weshaythanks for joining12:11
panda|ruckweshay: ok12:14
weshaymyoung, is there a tempest squad sprint planning mtg today12:16
weshay?12:16
weshayI see the retro nvrmind12:16
panda|ruckweshay: arxcruz and chandankumar are also on holiday today12:17
weshaythanks12:17
weshaypanda|ruck, need review on https://review.openstack.org/#/c/591077/ https://review.rdoproject.org/r/#/c/15421/12:21
weshaysshnaidm, ^12:21
sshnaidmweshay, how do I run standalone job in libvirt?12:24
weshaysshnaidm, here's my invocation for f28 and standalone12:26
weshayexport WD=/var/tmp/standalone; ./quickstart.sh --release tripleo-ci/master_fedora28 -c config/general_config/featureset052.yml -E config/environments/baseos_fedora_libvirt.yml  --tags all -p quickstart-extras-standalone.yml --no-clone  --teardown all -w $WD --clean  whayutin-testbox212:26
*** trown|outtypewww is now known as trown12:26
weshayhttps://review.openstack.org/#/c/591540/4/config/release/tripleo-ci/master_fedora28.yml12:27
sshnaidmweshay, worth to add in readme..12:27
sshnaidmweshay, about second patch, where do we have job type "Standalone"? https://github.com/openstack/tripleo-quickstart/blob/402401942d781ee5eaf91b1b5cda112f0bd63c35/ci-scripts/full-deploy.sh#L1612:27
weshaysshnaidm, ya.. I want to get a check job in ci.centos12:28
sshnaidmweshay, I suppose it'll fail here https://github.com/openstack/tripleo-quickstart/blob/402401942d781ee5eaf91b1b5cda112f0bd63c35/ci-scripts/full-deploy.sh#L3612:28
weshayand I'll fill out the doc etc... but want a job running too12:28
weshayheh.. ur right12:29
* weshay fixes12:29
weshaythanks12:29
*** rlandy has joined #oooq12:31
rlandypanda|ruck: can I close this out? https://bugs.launchpad.net/tripleo/+bug/173963912:34
openstackLaunchpad bug 1739639 in tripleo "ci.centos gate are failing with THT default change" [High,Triaged] - Assigned to Gabriele Cerami (gcerami)12:34
myoungweshay: no, that's on friday12:37
*** ykarel has joined #oooq12:37
weshaysshnaidm, https://review.openstack.org/59201612:38
weshaythanks for catching that12:38
panda|ruckrlandy: close out you mean fixing >12:38
rlandysshnaidm: panda|ruck: could we get a review on https://review.openstack.org/#/c/591864?12:38
rlandypanda|ruck: that bug has been sitting without action for 8 months12:39
weshaysshnaidm, any idea why http://dashboard-ci.tripleo.org/d/cEEjGFFmz/cockpit?orgId=1 is still showing 1000+ gate failures?12:39
rlandyeither we work on it or close it out12:39
sshnaidmweshay, seems like data collection is stuck there, need to check why12:40
sshnaidmrlandy, I don't understand use case for it12:40
panda|ruckrlandy: that job taht failed doesn't even exist anymore, I think we can close it12:42
rlandypanda|ruck: ack - doing that12:42
weshaysshnaidm, see the commit message linnk https://review.openstack.org/#/c/590090/6/playbooks/tripleo-ci/vars/environment_type/ovb.yaml12:45
rlandysshnaidm: it's meed as that is a zuul var now and we want to switch case on fs12:45
weshaysshnaidm, it's so we can make the same choices w/ playbooks outside of upstream and zuul..12:45
weshayso we can have some consistency and fidelity in the way choices are made12:46
sshnaidmrlandy, weshay sorry, still confused -why to define it twice?12:48
*** ykarel is now known as ykarel|away12:49
rfolco|roversshnaidm, you mean the if/else blocks ?12:50
weshayall https://trello.com/b/0VFswmht/rdo-infra-retrospective?menu=filter&filter=label:Sprint%201712:50
sshnaidmweshay, well, grafana seems collect data well, so it's last changes that broke it12:51
myoungo/  retro board is live, see yall in a few --> https://trello.com/b/0VFswmht/rdo-infra-retrospective?menu=filter&filter=label:Sprint%201712:53
rlandymyoung: to do a technical retrospective on the on the zuul v3 design - what column would that go into?12:53
myoungrlandy: we've not done one of those, I was planning to have the discussion apart from specific cards, and track in the etherpad.  I'll create a place holder card...12:54
myoungor perhaps we just brainstorm in cards for the first few minutes.  I was planning to give a brief recap of our discussion over the past few days, including our chat with paul yesterday12:55
weshaymyoung, ya.. let's add it to the etherpad agenda12:55
myoungwhat do you think?  rlandy, weshay?12:55
weshayafter the retro12:55
myoungok12:55
weshaycan you post the agenda12:56
myoungit's there12:57
myounghttps://bluejeans.com/705085945512:57
myounghttps://etherpad.openstack.org/p/tripleo-ci-squad-meeting12:57
weshayarxcruz, retro13:02
weshaykopecmartin, arxcruz can you guys please get this fixed https://review.openstack.org/#/c/591322/13:02
weshayhttps://trello.com/b/0VFswmht/rdo-infra-retrospective13:03
weshayarxcruz, don't listen to me.. ur PTO13:04
weshaykopecmartin, can you take a look?13:04
*** ykarel|away has quit IRC13:17
weshaysshnaidm, panda|ruck https://docs.google.com/document/d/1LP23IwpCJLKCMe3wzDOvtqZZ79-eCxhxHVf7NOPeQbI/edit#13:30
*** agopi has quit IRC13:34
*** vinaykns has joined #oooq13:46
*** rfolco|rover has quit IRC13:57
*** agopi has joined #oooq14:09
kopecmartinweshay, regarding this https://review.openstack.org/#/c/591322/,  according to the last chandankumar's comment there is a different review which does the same but with a different approach14:13
kopecmartinweshay, so if i understand it correctly, that review is no longer needed14:13
weshaykopecmartin, ah k14:32
* weshay will look14:33
weshaysshnaidm, ?14:37
sshnaidmweshay, !14:37
weshaysshnaidm, can we get your ara patches through?14:40
sshnaidmweshay, yeah14:40
sshnaidmweshay, but it didn't work last time I looked14:41
weshaysshnaidm, k.. we need function14:49
weshaywe need that function/feature in ci14:49
weshaysshnaidm, have another job timing out14:49
*** d0ugal has joined #oooq14:59
sshnaidmmyoung, can you mute please?15:19
*** dsneddon has quit IRC15:20
*** dtrainor has quit IRC15:40
*** dtrainor has joined #oooq15:40
panda|rucksshnaidm: zuul variable precedence https://zuul-ci.org/docs/zuul/user/jobs.html#variables15:47
sshnaidmpanda|ruck, thanks15:48
*** vinaykns has quit IRC16:06
*** panda|ruck is now known as panda|off16:22
sshnaidmpanda|ruck, weshay rlandy just a thought about "include vars" issue and featureset overrides - https://review.openstack.org/#/c/592100 please tell what you think16:22
rlandysshnaidm: rfolco had a similar thought16:25
sshnaidmrlandy, oh, so did you discuss it?16:26
*** rf0lc0 has joined #oooq16:28
*** dtrainor_ has joined #oooq16:28
rlandysshnaidm: yeah - I am fine with that idea as long as we ca place a new playbook anywhere and it's not just an append situation16:28
rlandysshnaidm: his idea was just an append - will look at your review more closely16:29
sshnaidmrlandy, the latest vars will override, if you mean that16:30
*** dtrainor has quit IRC16:30
*** kopecmartin has quit IRC16:38
*** openstackstatus has joined #oooq16:56
*** ChanServ sets mode: +v openstackstatus16:56
rlandyweshay: RDO cloud is DOWN (20180813)- ^^ that still true? can we update?17:04
weshayrlandy, where do you see that17:05
*** gkadam has quit IRC17:05
weshaylp?17:05
weshayit's up17:05
rlandyour channel status17:05
weshayoh.. weird.. I don't see that. .but yes please17:05
weshayI probably removed it and I'm not a channel operatior17:05
rlandyok - I think I removed it17:06
rlandyhttps://logs.rdoproject.org/79/592079/1/openstack-check/legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master/27c4e03/job-output.txt.gz#_2018-08-15_15_34_20_33815317:07
rlandy^^ same error - dlrn?17:07
rlandyhttps://logs.rdoproject.org/79/592079/1/openstack-check/legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master/27c4e03/logs/undercloud/home/zuul/repo_setup.log.txt.gz#_2018-08-15_15_34_2017:09
rlandyguess so17:09
rlandyweshay: ^^ correct? same error17:09
rlandylooking at rdo errors17:09
weshaydiff err17:10
rlandyugh17:10
rlandyok17:10
weshayrlandy, url should be https://cbs.centos.org/repos/virt7-container-common-candidate/x86_64/os/17:11
weshayI think17:11
weshayrlandy, where do you see that?17:11
weshayovb jobs looking mostly good through sova17:11
rlandyhttps://logs.rdoproject.org/79/592079/1/openstack-check/legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master/27c4e03/logs/undercloud/home/zuul/17:12
weshayrlandy, passing at 69%17:12
rlandyweshay: ignore me - it's failing on the change17:12
rlandyjobs ar ok17:12
rlandylegit failure17:12
weshay%gatestatus17:14
weshayrlandy, https://review.openstack.org/#/c/591997/17:15
weshayalthough17:15
weshaywe do have something going on17:15
* weshay looks at rdo zuul17:15
weshayrlandy, https://logs.rdoproject.org/97/591997/4/openstack-check/legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/6928a85/job-output.txt.gz#_2018-08-15_16_49_16_26499617:16
weshayya.. we're 100% down17:16
rlandyweshay: see #tripleo17:16
rlandyrdocloud failing on alex's change17:16
rlandyyep - all imapcted17:17
rlandyimpacted17:17
*** gkadam has joined #oooq17:20
rf0lc0wow17:26
rlandyrf0lc0 is doing the happy dance that he is no longer ruck/rover17:31
rf0lc0haha17:32
rf0lc0rlandy, what did break?17:33
rlandyrdopkg17:33
rlandynot out fault17:33
rlandyour17:33
*** dtantsur is now known as dtantsur|afk17:40
*** trown is now known as trown|lunch17:54
*** dsneddon has joined #oooq18:02
*** sshnaidm is now known as sshnaidm|bbl18:05
*** ykarel has joined #oooq18:11
*** gkadam has quit IRC18:19
rlandyweshay: what still runs in ci.centos that we care about?18:33
weshayrlandy, check jobs18:33
weshayrdo promotion jobs18:33
weshayrlandy, as long as we have quickstart.sh supported18:34
weshaywe need ci.centos18:34
rlandyhttps://ci.centos.org/view/rdo/view/tripleo-gate/18:34
rlandybesides weirdo18:34
rlandyneed to deprecate quickstart.sh asap18:35
rlandyguess it's a good day to take a look again at internal ovb18:35
weshayhttps://ci.centos.org/view/rdo/view/promotion-pipeline/18:35
weshayhttps://ci.centos.org/view/rdo/view/tripleo-gate/18:35
*** apetrich has quit IRC18:35
weshaythe quick basic jobs are important until we deprecate18:35
weshayrlandy, I'm all for what you are saying18:37
weshayI just want to be thoughtful about it..18:37
rlandyabsolutely18:37
weshayrlandy, my perference.. would be to execute on the standalone first18:38
rlandywe can't cut and run as much as we would like  to18:38
weshaywhich should lower our footprint18:38
rlandyok - going to try internal ovb again18:39
*** vinaykns has joined #oooq19:02
myoungquestion about depracation, and openstack. Do we have a # of cycles where something (once published...like quickstart.sh) needs to remain supported ?19:03
myoungor can we hollow it out and just have quickstart.sh just be a pass thru to $ourSolution?19:03
myoungor can we just remove it with reasonable notice?19:03
*** trown|lunch is now known as trown19:05
weshaymyoung, 2 cycles19:05
weshaybut ya.. a better solution would be to reuse / improve quickstart.sh19:06
weshayto send users in the upgraded workflow19:06
myoungweshay: ack, so if we deprecate by end of this cycle, we only need to carry for S & T19:20
* myoung fiddles with thermal paste in an attempt to figure out why his TR4 is > 70C19:20
weshaymyoung, I think the right way to do this.. is to have quickstart.sh trigger a libvirt recreate19:22
*** sshnaidm|bbl is now known as sshnaidm19:22
weshaymyoung, doubt we'll have time for it though19:22
weshaynot sure yet19:22
*** ykarel has quit IRC19:29
*** apetrich has joined #oooq19:34
*** hamzy has joined #oooq19:46
rlandymyoung: weshay: trying rhos-13 gate again ... do you see any updates that this file needs? http://git.app.eng.bos.redhat.com/git/tripleo-environments.git/tree/config/release/rhos-13.yml ... running iwth fs00120:11
rlandywith20:11
*** honza has quit IRC20:11
rlandy2018-08-15 20:09:26.614699 | tripleo-ovb-centos-7 | 2018-08-15 20:09:26,614 - testenv-client - ERROR - Couldn't retrieve env20:16
rlandyhmmm20:16
* weshay looks20:24
weshayrlandy, seem right to me20:25
rlandycool20:25
myoungrlandy: looks right, is there an error from that gate?20:54
* myoung looks20:54
rlandyno - just checking as it's been a while since I used it20:54
myoungrlandy: the tq gate is hitting: stack_status_reason   | Resource CREATE failed: OverQuotaClient: resources.provision_network: Quota exceeded for resources: ['network'].20:56
myounghttps://thirdparty.logs.rdoproject.org/jenkins-tq-gate-rhos-13-ci-rhos-ovb-featureset001-128/ovb_create_stack.log20:56
myoungrlandy: ahh ok20:56
rlandyhttps://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/tq-gate-rhos-13-ci-rhos-ovb-featureset001/129/console20:57
rlandyrunning ok so far20:58
rlandywas failing before with same error that required this https://review.openstack.org/#/c/528480/1/instack_undercloud/undercloud.py20:58
*** trown is now known as trown|outtypewww21:03
rlandyhttp://logs.openstack.org/52/584452/5/check/tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates/349226f/job-output.txt.gz#_2018-08-15_20_36_16_38820221:03
rlandyweshay: know about these timeouts? something to investigate?21:03
*** agopi has quit IRC21:15
weshayrlandy, ya. we have some patches that enable ara on the overcloud from sshnaidm21:40
weshaywe need to get them working and fixed21:40
weshayhttps://bugs.launchpad.net/tripleo/+bug/178722621:40
openstackLaunchpad bug 1787226 in tripleo "tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates is timing out" [Critical,Triaged]21:40
weshayhttps://bugs.launchpad.net/tripleo/+bug/178676421:40
openstackLaunchpad bug 1786764 in tripleo "tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates times out on prepare" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami)21:40
rlandyk21:43
weshayrlandy, https://review.openstack.org/#/c/565077/21:46
weshayhttps://review.openstack.org/#/c/583799/21:46
rlandyI see21:47
*** apetrich has quit IRC21:55
*** d0ugal has quit IRC22:33
*** d0ugal has joined #oooq22:44
*** vinaykns has quit IRC22:52
*** d0ugal has quit IRC23:01
*** d0ugal has joined #oooq23:03
*** agopi has joined #oooq23:07
weshayrlandy, you still online?23:10
weshayI need to talk something through23:10
*** dsneddon has quit IRC23:30
*** dsneddon has joined #oooq23:31
*** d0ugal has quit IRC23:42
rlandyweshay: yeah - but I need to go out now23:52
rlandyI will be back later23:52
*** rlandy is now known as rlandy|afk23:52
*** d0ugal has joined #oooq23:56
« Tuesday, 2018-08-14 Index Thursday, 2018-08-16 »

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!