14:00:12 <mwhahaha> #startmeeting tripleo 14:00:12 <openstack> Meeting started Tue Sep 24 14:00:12 2019 UTC and is due to finish in 60 minutes. The chair is mwhahaha. Information about MeetBot at http://wiki.debian.org/MeetBot. 14:00:12 <mwhahaha> #topic agenda 14:00:12 <mwhahaha> * Review past action items 14:00:13 <mwhahaha> * One off agenda items 14:00:13 <mwhahaha> * Squad status 14:00:14 <mwhahaha> * Bugs & Blueprints 14:00:14 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 14:00:14 <mwhahaha> * Projects releases or stable backports 14:00:15 <mwhahaha> * Specs 14:00:15 <mwhahaha> * open discussion 14:00:16 <mwhahaha> Anyone can use the #link, #action and #info commands, not just the moderatorǃ 14:00:17 <openstack> The meeting name has been set to 'tripleo' 14:00:17 <mwhahaha> Hi everyone! who is around today? 14:00:34 <dpeacock> o/ 14:00:43 <weshay> 0/ 14:00:51 <ksambor> o/ 14:00:51 <Tengu> \o/ 14:00:52 <ekultails> o/ 14:01:05 <rlandy> o/ 14:01:26 <owalsh> o/ 14:01:26 <marios> hello 14:01:59 <chandankumar> ykarel: which branch? master ? 14:02:01 <chandankumar> \o/ 14:02:56 <rfolco> o/ 14:02:59 <mschuppert> o/ 14:03:08 <ykarel> chandankumar, yes 14:03:16 <ykarel> o/ 14:03:45 <chandankumar> patch coming 14:04:18 <mwhahaha> alright let's get on with the meeting 14:04:23 <mwhahaha> #topic review past action items 14:04:23 <mwhahaha> mwhahaha to close out train-m3 - DONE 14:04:41 <mwhahaha> I went ahead and closed out the launchpad milestone and we release m3 last week 14:04:52 <mwhahaha> any questions on that? 14:05:38 <rh-jelabarre> o/ 14:05:59 <mwhahaha> sounds like nope 14:06:02 <mwhahaha> sync with weshay on ptl things - NOT DONE 14:06:03 <mwhahaha> #action mwhahaha sync with weshay on ptl things 14:06:13 <mwhahaha> weshay: do you have time later today so we can get just mark this one done 14:06:15 <weshay> slackers 14:06:18 <weshay> yes 14:06:27 <mwhahaha> k i'll ping you a bit later 14:06:38 <mwhahaha> that's it on the past items 14:06:45 <mwhahaha> moving on to more exciting things 14:06:48 <mwhahaha> #topic one off agenda items 14:06:48 <mwhahaha> #link https://etherpad.openstack.org/p/tripleo-meeting-item 14:06:54 <mwhahaha> centos8/pcmk ; clues when it will be available? should ci switch back to pcmk when it becomes available? (gfidente/bandini) 14:07:18 <mwhahaha> IMHO the answer to pcmk would be yes 14:07:42 <mwhahaha> weshay, do we have any thoughts on the turn around time for centos8, is that a question for rdo as well? 14:08:12 <weshay> once we have a upstream nodepool image.. I'm hoping we'll see standalone running with in days.. and i'll provide an estimate then 14:08:19 <mwhahaha> k 14:08:31 <ykarel> RDO would need time to prepare 14:09:04 <bandini> mwhahaha: yes, once we can use centos8 we should switch back ci to pcmk 14:09:10 <bandini> and also work on ha by default 14:09:28 <mwhahaha> we'll also finally be able to switch to podman on ovb as well 14:09:32 <mwhahaha> and remove docker packaging 14:09:39 <mwhahaha> so there's a bunch pending centos8 14:09:51 <mwhahaha> hopefully we can switch early in the U cycle 14:09:51 <mjturek> rfolco: does this look good? ppc container builds are failing due to kolla issue and this is the fix I think https://review.opendev.org/#/c/683997/ 14:10:15 <weshay> mjturek, in a mtg fyi 14:10:20 <rfolco> mjturek, lets chat on #oooq 14:10:27 <mjturek> whoops, I'm sorry 14:10:29 <ykarel> yes hopefully for U cycle we will be able to go with CentOS 14:10:36 <bandini> aye 14:10:38 <ykarel> but more update will be there in coming days 14:10:45 <ykarel> once it's actually out 14:10:49 <mwhahaha> any other pacemaker/centos8 thoughts? 14:10:51 <gfidente> weshay mwhahaha yeah and both scenario004 and scenario001 14:11:26 <cloudnull> o/ 14:11:35 <ykarel> also we would need other SIGs to be ready apart from cloud sig 14:11:45 <ykarel> like virt and ceph to be ready with centos8 14:12:32 <mwhahaha> #link http://mirror.centos.org/centos-8/ 14:12:46 <mwhahaha> looks like we're still pending qcow cloud images, but at least a repo is available 14:13:46 <ykarel> regarding RDO and CentOS8, couple of info can be found at https://review.rdoproject.org/etherpad/p/moving-rdo-to-centos8 14:14:07 <mwhahaha> k 14:14:15 <mwhahaha> I think that's about it for now. centos8 is still a WIP 14:14:19 <mwhahaha> moving on 14:14:21 <mwhahaha> <weshay> quay registry 14:14:30 <weshay> ya... 14:14:55 <weshay> just a quick FYI.. we're exploring using quay ( key ) lolz.. in upstream. patches are being tested now 14:15:06 <weshay> expect containers to live in both docker.io and quay for some time 14:15:27 <weshay> https://review.rdoproject.org/r/#/c/22445/ <- being tested 14:15:40 <weshay> that's it 14:16:12 <weshay> mwhahaha, ^ 14:16:16 <mwhahaha> k 14:16:34 <mwhahaha> moving on 14:16:35 <weshay> next item is mine as well 14:16:38 <mwhahaha> <weshay> thoughts on a daily periodic job to update master containers w/ update_repo: "delorean-current,quickstart-centos-ceph-nautilus" ? 14:16:38 <mwhahaha> w/ standalone, or other job.. if it passes update containers in docker.io or quay 14:17:07 <weshay> ya.. I have arxcruz|rover and zbr|ruck looking at building a periodic (daily) job to update containers w/ 14:17:15 <weshay> "delorean-current,quickstart-centos-ceph-nautilus" 14:17:26 <sshnaidm> it shouldn't even pass, just build and push 14:17:32 <mwhahaha> that doesnt' seem correct 14:17:38 <weshay> and push to docker.io / quay.. to shorten the loop w/ containers prep 14:17:50 <weshay> sshnaidm, k.. we'll consider that 14:17:50 <mwhahaha> shouldn't it match our includepkgs list (tripleo-repos current-tripleo-dev) 14:18:00 <sshnaidm> anyway we update to these repos containers in every job, and it's merged only if it's passed 14:18:02 <weshay> again.. we're seeing very long container prep times in CI 14:18:06 <weshay> causing failures 14:18:07 <mwhahaha> which is the equivalent of CI 14:18:23 <mwhahaha> when was the last promotion? 14:18:23 <weshay> some examples can be found here: https://snapshot.raintank.io/dashboard/snapshot/SjZ16RtMXZCFbEIOjgZ7N7KKJvOpKBhB?orgId=2 14:18:38 <weshay> master is at 4 days 14:18:44 <bogdando> weshay: yes, it's about ~1h++ for each job 14:18:47 <mwhahaha> hmm k 14:18:58 <weshay> you can see.. we have infra issues w/ ovh and fortnebula 14:18:59 <bogdando> only to update the containers with modify image role 14:19:05 <weshay> most of the infra guys are out atm.. at ansible-fest 14:19:29 <weshay> in rdo periodic jobs w/o yum updates.. containers prep is taking 5-6 minutes 14:19:46 <ykarel> really? 14:19:59 <weshay> so we're still exploring ways to get our job times back down to a reasonable time... updates will come 14:20:13 <weshay> ykarel, aye 14:20:29 <mwhahaha> k 14:20:31 <ykarel> okk nice, any link to logs 14:20:34 <weshay> http://dashboard-ci.tripleo.org/d/si1tipHZk/jobs-exploration?orgId=1&from=now-12h&to=now&var-influxdb_filter=branch%7C%3D%7Cmaster 14:20:42 <weshay> ykarel, look for the rdo provider 14:21:10 <weshay> hrm.. let me get you a better link 14:21:11 <sshnaidm> container build takes about 15 minutes? maybe worth just to build them and not to update? :) 14:21:26 <ykarel> i see 25,21, 15 minutes 14:21:43 <mwhahaha> you mean building them in CI? 14:21:48 <sshnaidm> yeah 14:22:02 <mwhahaha> i think it'll probably take longer than that on some of these clouds 14:22:31 <mwhahaha> though maybe we would get better cache results for packages than containers 14:22:34 <mwhahaha> we could try it 14:22:38 <weshay> ykarel, use https://snapshot.raintank.io/dashboard/snapshot/S9m9pvyn4iPSELe0bd566ZlMv4KvVWwO?orgId=2 14:22:45 <mwhahaha> but it requires an undercloud to build 14:22:50 <mwhahaha> because $reasons 14:22:58 <mwhahaha> i'm not sure it'd be a simple switch 14:23:51 <weshay> http://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-scenario001-standalone-master/7b77387/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz 14:24:03 <bogdando> ykarel: just checked, http://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-scenario001-standalone-master/7b77387/logs/undercloud/var/log/tripleo-container-image-prepare.log.txt.gz is exactly 5 minutes 14:24:21 <ykarel> rhel8 14:24:32 <bogdando> ykarel: 2019-09-24 01:32:26 - 2019-09-24 01:37:21 14:24:33 <weshay> doesn't matter 14:25:03 <mwhahaha> anyway sounds like we need to continue investigating our options around containers 14:25:17 <mwhahaha> anythign else? 14:25:25 <weshay> nope 14:25:30 <weshay> aye 14:25:31 <mwhahaha> k moving on 14:25:34 <mwhahaha> #topic Squad status 14:25:34 <mwhahaha> ci 14:25:34 <mwhahaha> #link https://etherpad.openstack.org/p/tripleo-ci-squad-meeting 14:25:34 <mwhahaha> upgrade 14:25:34 <mwhahaha> #link https://etherpad.openstack.org/p/tripleo-upgrade-squad-status 14:25:35 <mwhahaha> edge 14:25:35 <mwhahaha> #link https://etherpad.openstack.org/p/tripleo-edge-squad-status 14:25:36 <mwhahaha> integration 14:25:36 <mwhahaha> #link https://etherpad.openstack.org/p/tripleo-integration-squad-status 14:25:37 <mwhahaha> validations 14:25:37 <mwhahaha> #link https://etherpad.openstack.org/p/tripleo-validations-squad-status 14:25:38 <mwhahaha> networking 14:25:38 <mwhahaha> #link https://etherpad.openstack.org/p/tripleo-networking-squad-status 14:25:39 <mwhahaha> transformation 14:25:39 <mwhahaha> #link https://etherpad.openstack.org/p/tripleo-ansible-agenda 14:25:40 <mwhahaha> any status highlights? 14:25:43 <ykarel> weshay, i see http://dashboard-ci.tripleo.org/d/si1tipHZk/jobs-exploration?orgId=1&from=now-12h&to=now&var-influxdb_filter=branch%7C%3D%7Cmaster&var-influxdb_filter=provider%7C%3D%7Crdo-cloud and in this i see 25,21,15 14:25:48 <ykarel> the other link seems not working 14:26:03 <weshay> ykarel, k.. let's go out of channel 14:26:08 <ykarel> ack 14:27:00 <bogdando> ykarel: but that was prolly a bad example, there was nothing to update at all :) 14:27:03 * ykarel checked what's with rhel8, there is dnf issue, so it didn't run completely 14:27:17 <weshay> there are never container updates in periodic.. 14:28:52 <openstackgerrit> Karthik S proposed openstack/tripleo-heat-templates master: Fixed the path for neutron-base.yaml https://review.opendev.org/684318 14:30:29 <mwhahaha> ok moving on 14:30:34 <mwhahaha> #topic bugs & blueprints 14:30:34 <mwhahaha> #link https://launchpad.net/tripleo/+milestone/train-rc1 14:30:34 <mwhahaha> For Train we currently have 27 blueprints and 521 (+15) open Launchpad Bugs. 28 train-rc1, 493 ussuri-1. 91 (-75) open Storyboard bugs. 14:30:34 <mwhahaha> #link https://storyboard.openstack.org/#!/project_group/76 14:30:46 <mwhahaha> I cleaned up the tripleo-ui story board items last week. Additionally I closed out milestone 3. I will review the open rc1 bugs to move them to ussuri-1 if they aren't really blocking the rc1. 14:30:46 <mwhahaha> #action mwhahaha review train-rc1 open bugs 14:31:13 <mwhahaha> any questions/concerns around bugs/blueprints? 14:32:19 <apetrich> amoralej, so I've narrowed down the issue to this https://i.imgflip.com/3bex5p.jpg it is a bit weird. I'm trying to understand what causes that 14:33:02 <mwhahaha> sounds like that's in reference to https://bugs.launchpad.net/tripleo/+bug/1845181 14:33:02 <openstack> Launchpad bug 1845181 in tripleo "Problem with tripleo_common git functionalities and eventlet 0.25.1" [Medium,In progress] - Assigned to Adriano Petrich (apetrich) 14:33:38 <mwhahaha> also an appropriate use of memes 14:33:42 <mwhahaha> anything else? 14:33:42 <apetrich> mwhahaha, it is. 14:34:00 <apetrich> mwhahaha, so tempted to add that to the bug 14:34:49 <openstackgerrit> Merged openstack/tripleo-heat-templates stable/stein: Enable deep_compare by default for stonith resources https://review.opendev.org/683409 14:34:50 <openstackgerrit> Merged openstack/tripleo-heat-templates stable/rocky: Enable deep_compare by default for stonith resources https://review.opendev.org/683404 14:34:52 <openstackgerrit> Merged openstack/tripleo-heat-templates stable/queens: Enable deep_compare by default for stonith resources https://review.opendev.org/683405 14:34:54 <openstackgerrit> Merged openstack/tripleo-validations master: Update the constraints url https://review.opendev.org/684161 14:34:56 <openstackgerrit> Merged openstack/tripleo-upgrade stable/stein: Fix "unbound" variable https://review.opendev.org/684191 14:35:00 <weshay> \0/ 14:35:10 <mwhahaha> alright moving on 14:35:10 <mwhahaha> #topic projects releases or stable backports 14:35:18 <mwhahaha> any release related topics? 14:37:37 <mwhahaha> moving on 14:37:40 <mwhahaha> #topic specs 14:37:40 <mwhahaha> #link https://review.openstack.org/#/q/project:openstack/tripleo-specs+status:open 14:37:54 <mwhahaha> the U cycle will be starting soon, don't forget to update your specs 14:38:10 <mwhahaha> any spec related items? 14:40:23 <mwhahaha> #topic open discussion 14:40:26 <mwhahaha> anything else? 14:43:01 <mwhahaha> alright thanks everyone 14:43:03 <mwhahaha> #endmeeting