15:00:48 <mnasiadka> #startmeeting kolla
15:00:48 <opendevmeet> Meeting started Wed Sep 29 15:00:48 2021 UTC and is due to finish in 60 minutes.  The chair is mnasiadka. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:48 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:48 <opendevmeet> The meeting name has been set to 'kolla'
15:00:56 <mnasiadka> #topic rollcall
15:01:21 <headphoneJames> o/
15:01:33 <yoctozepto> o/
15:01:38 <hrw> [°][o][o]
15:01:48 <mnasiadka> \o
15:02:13 <yoctozepto> mnasiadka: it was about the time running too fast
15:02:37 <mgoddard> \o
15:02:56 <priteau> o/
15:04:23 <mnasiadka> ok then
15:04:26 <mnasiadka> #topic agenda
15:04:30 <mnasiadka> * Review action items from the last meeting
15:04:30 <mnasiadka> * CI status
15:04:30 <mnasiadka> * Discuss switching default image type to source
15:04:30 <mnasiadka> * What do we do with Monasca?
15:04:30 <mnasiadka> * Release tasks
15:04:32 <mnasiadka> * Xena cycle planning
15:04:32 <mnasiadka> * Yoga PTG planning
15:04:34 <mnasiadka> * Open discussion
15:04:49 <mnasiadka> #topic Review action items from the last meeting
15:05:30 <mnasiadka> I've done all that were assigned to me, yoctozepto posted his change about using upstream MariaDB repos in CentOS and syncing version to what Deb/Ubuntu uses.
15:05:41 <mnasiadka> #topic CI status
15:05:53 <mnasiadka> How's the CI? green?
15:06:27 <yoctozepto> green
15:06:33 <yoctozepto> kolla STRONK
15:06:35 <yoctozepto> :D
15:06:39 <mnasiadka> Whiteboard says Kayobe is amber, priteau I guess after all those changes it should be green again?
15:06:44 <priteau> I believe Kayobe is mostly green
15:07:06 <mnasiadka> can we update the whiteboard to green and revisit once we see failures again?
15:07:13 <priteau> Updating whiteboard
15:07:15 <priteau> done
15:08:02 <mnasiadka> Do we really have Kayobe Queens as EM?
15:08:07 <mnasiadka> Kolla has been marked EOL for Queens
15:08:29 <mnasiadka> L139 in the whiteboard
15:08:57 <priteau> Needs update
15:09:01 <mgoddard> kayobe queens was unofficial IIRC
15:09:29 <mnasiadka> I think the first official was Train?
15:10:00 <mnasiadka> Anyway, let's remove it from there and done.
15:10:17 <mnasiadka> or yes, mark as unmaintained.
15:10:20 <mnasiadka> Let's move on.
15:10:34 <mnasiadka> #topic Discuss switching default image type to source
15:10:56 <yoctozepto> do we want to track EM for kolla and k-a?
15:11:02 <yoctozepto> (sorry for late question)
15:11:18 <yoctozepto> I know it's broken btw
15:11:33 <mnasiadka> Question if we're planning to fix it, or not.
15:11:39 <yoctozepto> me not
15:11:45 <yoctozepto> hrw not
15:11:49 <yoctozepto> question whether mnasiadka and/or mgoddard
15:11:53 <yoctozepto> ;-)
15:11:54 <mnasiadka> mgoddard: do we?
15:12:43 <mnasiadka> I guess if Kolla is broken, then it impacts Kayobe
15:12:54 <mgoddard> we're talking about queens still?
15:13:06 <priteau> I think EM in general so Train?
15:13:08 <yoctozepto> train
15:13:12 <yoctozepto> train is b0rken
15:13:13 <mnasiadka> no, queens is eol
15:13:23 <mnasiadka> rocky/stein/train I guess
15:13:26 <priteau> And older
15:13:38 <yoctozepto> and older is broken possibly too
15:13:45 <mgoddard> kayobe should track kolla
15:13:46 <priteau> What broke train?
15:13:50 <yoctozepto> setuptools hell
15:13:56 <yoctozepto> plus some other issues
15:13:56 <priteau> Ah yes
15:14:02 <yoctozepto> hrw may know more
15:14:03 <hrw> kolla/train is broken, EM, resting etc
15:14:14 <mgoddard> we'll probably end up fixing it at some point
15:14:32 <hrw> neither centos7 nor centos8 are buildable and it is beyond us to fix it
15:14:50 <mgoddard> there's always a way :)
15:14:59 <hrw> I refuse to work on train
15:15:01 <mnasiadka> Using the opportunity - do we want to EOL Rocky/Stein? or leave them in EM for now?
15:15:08 <mgoddard> people are still using train
15:15:21 <mgoddard> nobody's asking you to work on train hrw
15:15:26 <priteau> I have an upgrade to train in 10 days :D
15:15:38 <mnasiadka> priteau: so you have an opportunity to fix it ;-)
15:15:47 <priteau> yay
15:16:03 <hrw> mgoddard: it was to say 'in my opinion train is broken too far to be fixable'
15:16:17 <yoctozepto> it's always fixable
15:16:21 <yoctozepto> but perhaps in pain
15:17:18 <mnasiadka> Ok, seems we as a company will work on it in some time - let's leave it if it's fixable or not.
15:17:48 <mnasiadka> And we've diverted off topic already.
15:17:49 <hrw> mnasiadka: then check my 'use pip 20' patch in gerrit. will help you get CI working at all
15:18:02 <mnasiadka> hrw: thank you
15:18:24 <mnasiadka> ok, let's go back to "Discuss switching default image type to source"
15:18:25 <hrw> I abandoned it
15:18:33 <hrw> +1 for source by default
15:18:37 <opendevreview> Pierre Riteau proposed openstack/kayobe master: Bump up Ansible supported versions  https://review.opendev.org/c/openstack/kayobe/+/811410
15:18:54 <mnasiadka> mgoddard: proposed that as a topic for PTG, but I feel we could try switching in this cycle (although it's late).
15:19:07 <mnasiadka> Opinions?
15:19:28 <mgoddard> pro: pretty low risk
15:19:36 <mgoddard> con: pretty high impact
15:20:02 <opendevreview> Radosław Piliszek proposed openstack/kolla-ansible master: Do not set net.ipv4.ip_forward sysctl  https://review.opendev.org/c/openstack/kolla-ansible/+/809977
15:20:05 <mgoddard> but it's not the kind of change that needs time to settle down, so I guess we could make a late change
15:20:29 <yoctozepto> ++
15:20:31 <yoctozepto> let's switch it
15:20:34 <mnasiadka> I don't think if we would make this change early in the cycle, it would make people to read the renos better.
15:20:40 <yoctozepto> this will send the right message
15:20:49 <yoctozepto> the current default sends a wrong one ;d
15:20:49 <mnasiadka> For visibility - we can send a mail to the openstack-discuss ML
15:20:59 <mnasiadka> if that helps in anything these days...
15:21:19 <mnasiadka> ok then, anybody against switching?
15:22:17 <mnasiadka> seems not, then agreed
15:22:30 <mgoddard> who will do it?
15:22:30 <mnasiadka> Is there a volunteer to post patch(es)?
15:22:51 <mnasiadka> I guess we need one in Kolla-Ansible and a second in Kolla (and a third one in Kayobe?)
15:24:07 <mgoddard> I can do it
15:24:44 <mnasiadka> Ok then, if you'll lack time - just shout.
15:25:07 <mnasiadka> #agreed to switch default image type to source, mgoddard to post patches to do it
15:25:23 <mnasiadka> Next topic
15:25:27 <mnasiadka> #topic What do we do with Monasca?
15:25:46 <yoctozepto> I meant mostly CI job in here
15:25:52 <yoctozepto> but yeah, what do we do
15:25:56 <mnasiadka> Currently I think the log api is broken, from what I heard probably since Wallaby - but I may be mistaken.
15:26:10 <mnasiadka> And I think users that decide to deploy Monasca - think of it as the obvious choice.
15:26:28 <yoctozepto> ok, I can hide it behind the extra flag we planned
15:26:32 <yoctozepto> and also, we drop the CI job?
15:27:09 <mnasiadka> As already commented in mgoddard's patch - I think we need to at least make it visible for users, that it's broken now.
15:27:36 <mnasiadka> And if nobody fixes it in Yoga... then we need to analyze what to do.
15:29:44 <mnasiadka> mgoddard: should we drop the CI job? I think monasca-thresh also plays a role in defining alerts for thresholds - so that makes Monasca a bit crippled?
15:29:56 <mgoddard> monasca thresh works
15:29:59 <mgoddard> well
15:30:09 <yoctozepto> what does not work? only the log api?
15:30:14 <mgoddard> correct
15:30:16 <mgoddard> AFAIK
15:30:18 <yoctozepto> ok
15:30:22 <yoctozepto> ack, fair
15:30:57 <mgoddard> thresh is just a bit weird - it got fixed recently to submit jobs to storm rather than processing them locally
15:30:59 <yoctozepto> ok /me to propose the magic flag, we can update the docs accordingly
15:31:13 <yoctozepto> could we then allow to reroute logs to its central logging path?
15:31:13 <mgoddard> the implementation ended up with a container in the 'exited' state
15:31:18 <yoctozepto> despite monasca being enabled?
15:31:26 <mnasiadka> ah, so we ,,broke'' thresh?
15:31:27 <mgoddard> I didn't particularly like it, but it works
15:31:36 <mgoddard> no, as I said - thresh works
15:31:54 <mgoddard> it just now shows as 'exited'
15:31:55 <mnasiadka> but it's in exited state?
15:32:05 <mgoddard> which failed the container check script
15:32:11 <mgoddard> so I added a special case
15:32:22 <mgoddard> it's now a 'one shot'
15:32:39 <yoctozepto> what does it do in there?
15:32:43 <mnasiadka> one shot container?
15:32:45 * yoctozepto knows nothing
15:32:57 <mnasiadka> I'm also curious to understand this case :)
15:33:05 <yoctozepto> that makes the two of us
15:33:57 <mgoddard> it pushes a job to storm, then exits
15:34:20 <mgoddard> the container is kept around to allow the kolla container diff magic to work
15:34:29 <mgoddard> so a new image updates the job
15:34:43 <mgoddard> it's a bit unusual, but apparently works
15:35:00 <mnasiadka> can we just make it sleep or something, so it's not in exited state?
15:35:45 <mgoddard> can we just ignore it and move on? :)
15:35:49 <yoctozepto> well, oneshot makes sense
15:35:59 <yoctozepto> yeah, I think let's not split hair on this
15:36:07 <mnasiadka> Ok, let's then ignore it.
15:36:13 <yoctozepto> so
15:36:16 <yoctozepto> the plan is
15:36:20 <yoctozepto> document the issue
15:36:22 <yoctozepto> and hide monasca
15:36:25 <mgoddard> I suggested a different approach,but the author had already agreed on this one and didn't want to go back
15:36:28 <yoctozepto> what about the log rerouting?
15:36:51 <mgoddard> central logging can still work with monasca IIRC
15:37:00 <yoctozepto> could you check it?
15:37:14 <mgoddard> monasca_ingest_control_plane_logs
15:37:23 <mnasiadka> Is that documented in Monasca guide?
15:37:25 <mgoddard> yes
15:37:29 <mgoddard> https://docs.openstack.org/kolla-ansible/latest/reference/logging-and-monitoring/monasca-guide.html
15:37:35 <yoctozepto> yeah, but we need to drop a big warning now ;d
15:38:03 <mnasiadka> so, we drop a note/warning before "you can optionally bypass Monasca for logs"
15:38:55 <mnasiadka> does it make any sense?
15:40:10 <yoctozepto> mhm
15:40:16 <yoctozepto> you do the docs then
15:40:31 <mnasiadka> Me? I have no clue about Monasca.
15:40:42 <mnasiadka> mgoddard: keen to update the docs?
15:41:45 <mgoddard> ok
15:41:59 <opendevreview> Mark Goddard proposed openstack/kolla master: Build source type images by default  https://review.opendev.org/c/openstack/kolla/+/811743
15:43:18 <mnasiadka> #action mgoddard to update Monasca docs around using central logging, due to Monasca log problems
15:43:27 <mnasiadka> yoctozepto: what's your action? :)
15:44:03 <yoctozepto> mnasiadka: the old one still current :D
15:44:09 <mnasiadka> Ok :)
15:44:18 <mnasiadka> Let's go further... and forget about Monasca ;)
15:44:21 <yoctozepto> add the flag to hide the "bad quality" services
15:44:21 <mnasiadka> #topic Release tasks
15:44:25 <yoctozepto> ok
15:44:30 <yoctozepto> let's roll
15:45:40 <mnasiadka> Hmm, we wrote in docs that R-2 is feature freeze week for Kolla
15:45:51 <mnasiadka> R-2 was last week
15:46:52 <yoctozepto> oh-my-openstack
15:46:59 <mnasiadka> So I assume all changes that are not marked as priorities on the Kolla Whiteboard - are frozen, if needed we need to set Review Priority on those
15:47:14 <yoctozepto> all feature changes ;-)
15:47:14 <mnasiadka> And let's focus on merging what's left this week
15:47:24 <headphoneJames> any chance we could get https://review.opendev.org/c/openstack/kolla-ansible/+/692179 in? Or does that now get pushed to Yoga?
15:47:43 <mnasiadka> because we marked R-1 as prepare to RC1 and stable branch creation
15:48:02 <yoctozepto> yeah, I guess we could try to better stay on track this cycle
15:48:09 <yoctozepto> as even debian has already released lol
15:48:43 <mgoddard> do we need to update cycle highlights?
15:48:50 <mgoddard> I think we just had two initially
15:48:57 <mnasiadka> Yes, I guess so.
15:49:19 <yoctozepto> tbh, I don't think many people read those (if you have better things to do ;-) )
15:49:32 <mnasiadka> So let's mark it as a low priority item.
15:49:42 <mgoddard> I think they feed into marketing
15:49:59 <mgoddard> and foundation staff sometimes cherry pick them
15:50:06 <yoctozepto> they do
15:50:11 <mnasiadka> I understand that we have some changes left for review, that we'd like to merge this week - and we could cut stable branches and RC1 somewhere next week? Or what's the plan?
15:50:27 <mgoddard> which is how prometheus v2 got to the top of some wallaby presentation
15:50:42 <yoctozepto> it did
15:50:46 <priteau> Also here: https://www.openstack.org/software/wallaby/
15:50:53 <yoctozepto> also because of cncf cooperation
15:51:13 <mgoddard> headphoneJames: I think it's ready, I'll do a final pass through
15:53:53 <mnasiadka> Ok, we have 7 minutes left.
15:54:18 <mnasiadka> Do we agree to work on reviewing existing patches from the priority list on the whiteboard until next meeting, and then agree if we're cutting stable branches?
15:55:22 <yoctozepto> makes sense to me
15:55:39 <yoctozepto> plus the fixes I'm pouring in
15:55:43 <yoctozepto> pretty please :-)
15:55:54 <mnasiadka> obviously
15:55:56 <mgoddard> well
15:56:02 <mgoddard> when is actual feature freeze?
15:56:19 <mgoddard> next week?
15:56:19 <mnasiadka> the one we defined in docs was last week
15:56:47 <mnasiadka> R-2 Sep 20 - Sep 24
15:56:56 <priteau> My bad, I thought it was freeze at OpenStack release date
15:57:04 <mnasiadka> According to Xena Release calendar - https://releases.openstack.org/xena/schedule.html
15:57:33 <mgoddard> yeah, sounds correct
15:58:14 <mnasiadka> We just missed that, I was sure it's this week.
15:58:15 <mgoddard> which features are we hoping to merge before next week?
15:58:30 <opendevreview> Mark Goddard proposed openstack/kolla-ansible master: Deploy source type images by default  https://review.opendev.org/c/openstack/kolla-ansible/+/811744
15:59:09 <mnasiadka> So, keystone system scope is one
15:59:34 <mnasiadka> Ceph RADOS GW looks like in good shape if yoctozepto agrees
15:59:36 <mgoddard> ceph RGW pls
15:59:45 <mnasiadka> and I guess we should also remove chrony role?
15:59:56 <yoctozepto> did not we merge ceph rgw?
16:00:04 <mgoddard> not yet
16:00:06 <mnasiadka> no, you complained ;-)
16:00:09 <yoctozepto> ok, will look
16:00:09 <mgoddard> remove chrony pls
16:00:12 <yoctozepto> oh, sorry
16:00:15 <yoctozepto> yes, that one too
16:00:26 <yoctozepto> could someone write it down in one place on whiteboard
16:00:28 <yoctozepto> single list
16:00:29 <mgoddard> https://review.opendev.org/c/openstack/kolla-ansible/+/791743
16:00:31 <yoctozepto> one after the other
16:00:46 <mgoddard> we have a list already
16:00:52 <yoctozepto> ah
16:00:53 <mgoddard> could just add DEFERRED to some
16:00:57 <yoctozepto> ok
16:01:01 <mgoddard> swift won't make it
16:01:09 <mgoddard> too late for a change that big IMO
16:01:14 <yoctozepto> yeah, I don't think it's worth it either
16:01:20 <yoctozepto> it's just pain to keep it in bad shape
16:01:24 <yoctozepto> but let's release without hassle
16:01:46 <mnasiadka> I just added DEFERRED to the rest
16:02:00 <mnasiadka> ProxySQL probably also too big to get anywhere this week
16:02:57 <mgoddard> kayobe has a few with 1x +2
16:03:53 <mnasiadka> ok, I guess I need to stop the meeting, and I'll go through the Kayobe list and review those that I haven't.
16:04:03 <mnasiadka> #stopmeeting kolla
16:04:11 <mnasiadka> #endmeeting kolla