15:03:03 <mgoddard> #startmeeting kolla
15:03:03 <openstack> Meeting started Wed Apr  8 15:03:03 2020 UTC and is due to finish in 60 minutes.  The chair is mgoddard. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:03:04 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:03:07 <mgoddard> #topic rollcall
15:03:07 <openstack> The meeting name has been set to 'kolla'
15:03:10 <mgoddard> \o
15:03:35 <yoctozepto> o/
15:03:36 <hrw> ~o~
15:03:44 <yoctozepto> mnasiadka: ?
15:03:56 <yoctozepto> osmanlicilegi: ?
15:04:03 <generalfuzz> o/
15:04:14 <mgoddard> #topic agenda
15:04:17 <mgoddard> * Roll-call
15:04:18 <mnasiadka> o/ (busy)
15:04:19 <mgoddard> * Announcements
15:04:21 <mgoddard> ** First Kolla Klub meeting tomorrow @ 15:00 UTC | https://meet.google.com/hph-pynx-vsy
15:04:23 <mgoddard> ** OpenStack (not kolla) feature freeze this week https://releases.openstack.org/ussuri/schedule.html
15:04:25 <mgoddard> ** Kolla feature freeze Apr 27 - May 01
15:04:27 <mgoddard> * Review action items from last meeting
15:04:29 <mgoddard> * CI status
15:04:31 <mgoddard> ** Kolla build forced publishing to avoid lengthy breakage periods
15:04:33 <mgoddard> ** Kolla Ansible to pin EM branches to final release images (and making it policy in docs)
15:04:35 <mgoddard> * Kolla Klub https://etherpad.openstack.org/p/kolla-sig
15:04:37 <mgoddard> ** Suggested agenda?
15:04:39 <mgoddard> * Ussuri release planning (kayobe)
15:04:41 <mgoddard> * Ussuri release planning (kolla & kolla ansible)
15:04:43 <mgoddard> #topic announcements
15:04:43 <osmanlicilegi> o/
15:04:45 <mgoddard> #info First Kolla Klub meeting tomorrow @ 15:00 UTC
15:04:48 <mgoddard> #link https://meet.google.com/hph-pynx-vsy
15:05:15 <mgoddard> we will discuss an agenda for tomorrow later in the meeting
15:05:22 <mgoddard> #info OpenStack (not kolla) feature freeze this week
15:05:29 <mgoddard> #link https://releases.openstack.org/ussuri/schedule.html
15:05:45 <mgoddard> #link Kolla feature freeze Apr 27 - May 01
15:05:50 <mgoddard> #undo
15:05:51 <openstack> Removing item from minutes: #link Kolla
15:05:55 <mgoddard> #info Kolla feature freeze Apr 27 - May 01
15:06:14 <mgoddard> We will stop accepting features after that week
15:06:26 <mgoddard> And focus on stabilising for  the release
15:06:34 <mgoddard> Any other announcements?
15:06:54 * yoctozepto (Radosław Piliszek) running for TC
15:07:13 <yoctozepto> and mgoddard officially our new-old PTL
15:07:22 <mgoddard> of course
15:07:50 <mgoddard> #info mgoddard remains Kolla PTL
15:07:55 <yoctozepto> :D
15:08:10 <mgoddard> Bad luck team, you're stuck with me for another cycle :)
15:08:20 <osmanlicilegi> :]
15:08:21 * yoctozepto does not mind
15:08:22 <mgoddard> #info Vote for yoctozepto for TC!
15:08:26 <osmanlicilegi> we're happy with you mgoddard
15:08:28 <openstackgerrit> Pierre Riteau proposed openstack/kayobe stable/train: Fix multiple CI failures  https://review.opendev.org/717031
15:08:45 <mgoddard> you're too kind
15:08:53 <mgoddard> #topic Review action items from last meeting
15:09:00 <mgoddard> There were none
15:09:06 <mgoddard> #info CI status
15:09:12 <mgoddard> #undo
15:09:13 <openstack> Removing item from minutes: #info CI status
15:09:16 <mgoddard> #topic CI status
15:09:31 <yoctozepto> mgoddard: well, you are a fully-featured lead of this project so no need to be overly humble about it ;-)
15:09:56 <mgoddard> It's been a bumpy week or two for kolla CI
15:10:01 <yoctozepto> or three
15:10:09 <yoctozepto> anyways, stein patch for sensu is pending
15:10:18 <yoctozepto> (kolla breakage)
15:10:20 <mgoddard> kayobe is still recovering, but priteau is on it
15:10:29 <yoctozepto> https://review.opendev.org/718476
15:10:29 <patchbot> patch 718476 - kolla (stable/stein) - Sensu: Remove ceph and process-checks plugins from... - 1 patch set
15:11:03 <yoctozepto> train and master should be GREEN
15:11:09 <priteau> Fixing kayobe one branch at a time...
15:11:41 <priteau> just stable/train left to fix thanksfully
15:12:12 <mgoddard> hopefully py2 drop fallout will stop soon
15:13:07 <mgoddard> We have an outstanding issue with horizon taking forever to do asset compression
15:13:08 <yoctozepto> yeah, it greatly increases our failure rates
15:13:15 <yoctozepto> train was pretty stable in that regard
15:13:25 <yoctozepto> yeah, it has been workarounded
15:13:30 <yoctozepto> but is a practical pita
15:13:51 <mgoddard> horizon team responded that they saw no difference between PyScss 1.3.4 and 1.3.7 on ubuntu 18.04
15:14:01 <mgoddard> so could be something on our side?
15:14:11 <mgoddard> hrw: did you have any thoughts?
15:15:30 <hrw> sorry, was in a call
15:15:32 <mgoddard> ok
15:15:55 <openstackgerrit> Piotr Kopec proposed openstack/kolla master: nova-compute: add ndctl to expose NVDIMMs to guests  https://review.opendev.org/718081
15:16:01 <hrw> not looked at it at all
15:16:10 <hrw> other than pointing to open a ticket
15:16:41 <mgoddard> ok
15:16:44 <mgoddard> #topic Kolla build forced publishing to avoid lengthy breakage periods
15:16:47 <mgoddard> yoctozepto: yours?
15:17:26 <yoctozepto> mgoddard: mine
15:17:49 <yoctozepto> mgoddard: can we do anything about this? it's counter-productive to wait for next day crossing our fingers
15:18:07 <mgoddard> you want a publish button?
15:18:20 <yoctozepto> does anyone remember if we visited this subject already and what is the consensus?
15:18:23 <hrw> or 'recheck with local rebuild'?
15:18:24 <yoctozepto> mgoddard: more or less
15:18:45 <yoctozepto> I want unblocked k-a without rebuilding with every change
15:19:14 <mgoddard> I would prefer that we focus on not pushing broken images
15:19:31 <mgoddard> that would be better for our users
15:19:46 <yoctozepto> fair point, but could be harder to implement, but definitely next step
15:20:09 <mgoddard> I think the push button is more of a question for zuul/infra
15:20:21 <mgoddard> we need to publish in a job to access the secret
15:20:55 <mgoddard> what if we had a job that runs when a magic file is touched?
15:20:56 <yoctozepto> mgoddard: noted, make an action for me
15:21:24 <mgoddard> echo 'yes please' > .publish
15:21:37 <mgoddard> could be run as a gate job
15:21:47 <yoctozepto> mgoddard: yeah, I was thinking of a magic file; but probably at least check queue is not gonna be able to use the secret (for obvious reasons)
15:21:58 <yoctozepto> and not entirely sure about gate
15:22:07 <yoctozepto> I will do research about the best option then
15:22:18 <yoctozepto> glad you agree we might like it
15:22:19 <mgoddard> not sure if there is anything special about the periodic queue
15:22:33 <mgoddard> it would definitely be useful
15:22:44 <yoctozepto> and what about testing before publish?
15:22:54 <yoctozepto> can we afford it?
15:23:00 <yoctozepto> at least for the core jobs?
15:23:15 <osmanlicilegi> makes sense
15:23:36 <yoctozepto> what about tiering split? how would you make the two things agree?
15:24:15 <mgoddard> there are a few things we'd need to align here
15:24:21 <mgoddard> tiering being one
15:25:01 <mgoddard> I personally like the publish, test, promote pipeline model
15:25:14 <mgoddard> as separate dependent jobs
15:25:49 <yoctozepto> mgoddard: publish under different tag?
15:25:57 <mgoddard> but it does add complexity, and a possible dependency loop
15:26:02 <mgoddard> yes, different tag
15:26:11 <yoctozepto> train-candidate
15:26:15 <mgoddard> right
15:26:29 <yoctozepto> makes sense
15:26:53 <yoctozepto> then it's only a matter of adding tags in promotion
15:27:05 <mgoddard> yes
15:27:30 <mgoddard> not sure if we would automatically promote things we don't test
15:28:32 <mgoddard> something to think about for V anyway
15:28:41 <yoctozepto> but it is brilliant
15:28:41 <hrw> separate jobs involve lot of network traffic
15:28:48 <mgoddard> that is true
15:28:53 <yoctozepto> we can give up on separate k-a periodics
15:29:02 <yoctozepto> then it would be largely the same
15:29:08 <yoctozepto> and even less concetrated
15:29:11 <hrw> build-and-temporary-publish sends GBs of data. fetch-temporary-and-test fetches those GBs. etc
15:29:24 <hrw> build-test-publish does network once
15:29:31 <yoctozepto> yeah, but we already do it via k-a periodics
15:29:35 <mgoddard> true, although it's no different than publishing then running a deploy job
15:29:35 <hrw> but needs to fit in 3h ;(
15:29:50 <yoctozepto> we could give up on those, they sometimes fail due to mixed images
15:29:54 <mgoddard> which we already do many times a day
15:29:58 <yoctozepto> so it's a win-win
15:31:03 <yoctozepto> I will give both options some real thought, we can't block ourselves that much :-)
15:31:13 <mgoddard> would be quite easy to test adding a deploy job to the pipelines
15:31:15 <yoctozepto> thanks for some food for thought
15:31:51 <mgoddard> #topic Kolla Ansible to pin EM branches to final release images (and making it policy in docs)
15:31:56 <mgoddard> yoctozepto: again?
15:32:15 <yoctozepto> mgoddard: yes
15:32:25 <yoctozepto> wdyt? simple policy change
15:32:32 <yoctozepto> makes sense / not really?
15:32:55 <yoctozepto> users might not know to switch to the em tag so we could leave k-a defaults broken
15:33:04 <yoctozepto> but we might as well change them in the final release
15:33:45 <mgoddard> what do you mean by final relese images?
15:35:10 <mgoddard> we don't actually tag images at a particular kolla release
15:35:22 <mgoddard> it looks like it should happen, but it doesn't
15:35:26 <mgoddard> https://hub.docker.com/r/kolla/centos-binary-fluentd/tags
15:36:37 <yoctozepto> mgoddard: ah, then there is no topic :-)
15:36:49 <yoctozepto> I am pretty sure I saw 8.x.x in there somewhere
15:36:59 <mgoddard> other possibility is to switch to stable branches
15:37:02 <yoctozepto> so that would be stein
15:37:05 <hrw> we published rocky ones and then stopped building them. so whatever is on hub.docker is final image
15:37:07 <mgoddard> which might be more useful
15:37:17 <yoctozepto> hrw: yeah, we stopped when we did ;-)
15:37:27 <yoctozepto> mgoddard: switch how?
15:37:42 <mgoddard> in the sources config
15:37:49 <mgoddard> as we do in master
15:37:55 <mgoddard> but use stable branches
15:38:20 <yoctozepto> you want that in em? ;d
15:39:39 <mgoddard> well otherwise any fixes after EM never get picked up
15:40:05 <yoctozepto> true
15:40:14 <yoctozepto> makes sense, however orthogonal to the topic
15:40:29 <yoctozepto> but if we don't publish em tags, then it's a no-topic
15:40:44 <yoctozepto> so we might as well discuss making em follow stable branches
15:40:53 <yoctozepto> I vote YEAH
15:41:55 <mgoddard> it makes sense to me, although I don't generally use source images
15:42:48 <mgoddard> I suppose version-check.py would need an extension
15:43:16 <mgoddard> Let's keep this one in mind and move on
15:43:18 <hrw> it has one for train which no one wants ;d
15:43:37 <yoctozepto> I do use sources, but build for myself; especially would do for em...
15:43:57 <mgoddard> #topic Kolla Klub https://etherpad.openstack.org/p/kolla-sig
15:44:02 <mgoddard> Meeting tomorro
15:44:06 <yoctozepto> w
15:44:10 <mgoddard> thanks
15:44:13 <yoctozepto> yw
15:44:27 <mgoddard> How should we track agenda & meeting notes?
15:44:33 <mgoddard> etherpad?
15:44:41 <hrw> Google Docs I would suggest
15:44:52 <yoctozepto> google meet and google docs
15:44:57 <yoctozepto> makes sense
15:44:57 <hrw> so there is a way to comment entries
15:45:00 <hrw> no meet...
15:45:06 <hrw> zoom?
15:45:10 <mgoddard> ok, lets see if that works for everyone
15:45:14 <hrw> I have to use phone for meet ;(
15:45:15 <yoctozepto> why, I like your special effecfs
15:45:20 <mgoddard> meet is announced
15:45:24 <hrw> ok
15:46:10 <osmanlicilegi> I think I'll miss tomorrow :/
15:46:26 <mgoddard> #action mgoddard to create a google doc for kolla klub meeting agenda and notes
15:46:35 <mgoddard> np osmanlicilegi, next time
15:46:39 <osmanlicilegi> yep
15:46:51 <mgoddard> #action mgoddard to send reminder to ML about tomorrow's meeting
15:47:07 <yoctozepto> what about the agenda in the first place
15:47:13 <mgoddard> oh that
15:47:18 <yoctozepto> we have quite a lot of people
15:47:24 <mgoddard> in the pad yes
15:47:30 <mgoddard> let's see who dials in :
15:47:33 <yoctozepto> we might spend an hour of fixing connection and welcoming
15:47:38 <hrw> ;D
15:47:41 <osmanlicilegi> mgoddard: are you planning to record and share on somewhere?
15:48:17 <mgoddard> I don't think meet supports recording
15:48:31 <mgoddard> but agenda & notes will be shared
15:49:07 <hrw> "This feature is available with the G Suite Enterprise and G Suite Enterprise for Education editions."
15:49:30 <mgoddard> I was thinking I could start with a couple of slides just explaining what we're doing
15:49:37 <mgoddard> then some intros
15:49:43 <hrw> but: "Through September 30, 2020, G Suite customers can use premium Google Meet video conference features, such as larger meetings (up to 250 participants), live streaming, and recording."
15:50:24 <yoctozepto> mgoddard: we must encourage really quick intros
15:50:41 <mgoddard> sure
15:50:58 <mgoddard> although I think that is what this first session is about
15:51:44 <mgoddard> then I was going to suggest we ask the group what they think the scope should be
15:51:59 <hrw> take over the world of course
15:52:13 <mgoddard> eventually
15:52:22 <mgoddard> is there anything else we should cover?
15:53:03 <yoctozepto> hmm, well, makes sense
15:53:15 <yoctozepto> it's for operators so they should decide what we are going to talk about there
15:53:22 <mgoddard> we can try to achieve something in meeting 2 :)
15:54:07 <mgoddard> sounds like we have a rough plan
15:54:11 <mgoddard> it'll be alright on the night
15:54:17 <mgoddard> #topic Ussuri release planning (kayobe)
15:54:28 <mgoddard> priteau: \o
15:54:34 <priteau> o/
15:54:47 <mgoddard> how are we looking?
15:55:19 * yoctozepto looks gorgeous (as always) but is not sure about the others ;/
15:55:20 <mgoddard> still a few centos 8 patches outstanding
15:55:36 <priteau> I've been focusing on stable branches, lots to merge in master still.
15:55:57 <hrw> do we want to finally get rid of c7 in ussuri?
15:56:09 <hrw> I can refresh patch but it needs reviews
15:56:13 <mgoddard> I'll try to rally some reviewers tomorrow
15:56:14 <yoctozepto> hrw: didn't we? (btw, this is kayobe for the moment)
15:56:29 <hrw> ops
15:56:32 <mgoddard> hrw: not yet
15:56:34 <yoctozepto> hrw: ah, you mean this part, talk later
15:56:39 <mgoddard> oops I meant yoctozepto
15:57:16 <mgoddard> #topic Ussuri release planning (kolla & kolla ansible)
15:57:28 <yoctozepto> yeah, let's drop all signs of c7
15:57:51 <hrw> will refresh https://review.opendev.org/#/c/692450/ then
15:57:52 <patchbot> patch 692450 - kolla - WIP: Remove support for CentOS 7 - 15 patch sets
15:58:23 <mgoddard> WIP might have put reviewers off
15:58:41 <mgoddard> I haven't reviewed as I edited it
15:58:53 <mgoddard> added RP+1
15:59:29 <mgoddard> I'll update https://review.opendev.org/#/c/713282/ on the k-a side
15:59:30 <patchbot> patch 713282 - kolla-ansible - Remove support for CentOS 7 - 1 patch set
15:59:52 <mgoddard> 1 minute to go
16:00:01 <yoctozepto> not really
16:00:18 <mgoddard> other centos 8 things: https://review.opendev.org/715224
16:00:19 <patchbot> patch 715224 - kolla (stable/train) - CentOS 8: Add rabbitmq-3.7.24 image (CentOS 7 only) - 2 patch sets
16:00:48 <osmanlicilegi> mgoddard: my tests were fine, nothing broken
16:00:51 <yoctozepto> bbl, thanks for meeting
16:00:58 <mgoddard> osmanlicilegi: great
16:01:07 <mgoddard> ok, let's end it there
16:01:10 <mgoddard> Thanks all
16:01:22 <osmanlicilegi> thanks all
16:01:33 <mgoddard> Keep focusing on those priority patches before feature freezr
16:01:36 <mgoddard> #endmeeting