15:00:15 <gmann> #startmeeting tc
15:00:15 <opendevmeet> Meeting started Thu May 26 15:00:15 2022 UTC and is due to finish in 60 minutes.  The chair is gmann. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:15 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:16 <opendevmeet> The meeting name has been set to 'tc'
15:00:25 <gmann> tc-members: meeting time
15:00:36 <gmann> #topic Roll call
15:00:38 <gmann> o/
15:00:40 <slaweq> o/
15:00:41 <diablo_rojo_phone> o/
15:00:48 <dpawlik> o/
15:00:56 <jungleboyj> o/
15:02:04 <rosmaita> o/
15:02:40 <dansmith> oj
15:03:19 <gmann> arne_wiebalck_ informed that he will not be able to join due to holiday
15:03:20 <gmann> let's start
15:03:25 <gmann> #link https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Agenda_Suggestions
15:03:30 <spotz> o/
15:03:32 <gmann> today agenda ^^
15:04:13 <gmann> #topic Follow up on past action items
15:04:16 <gmann> gmann to schedule 'release name things' discussion call *again*
15:04:22 <gmann> that is done
15:04:32 <dansmith> ++
15:04:44 <jungleboyj> :-)
15:04:49 <diablo_rojo_phone> And so is the actual meeting
15:04:54 <rosmaita> you scared me for a minute there
15:05:10 <jungleboyj> Yeah, I was afraid we were going to have to talk again.
15:05:26 <dansmith> my ++ was to the "that is done" part
15:05:30 <gmann> :)
15:05:42 <gmann> #topic Gate health check
15:06:07 <gmann> stable/ussuri and stable/victoria is broken and temepst master too
15:06:49 <gmann> that is due to py36 and tempest master usage iun stable/ussuri and victoria
15:07:01 <dansmith> yeah I know ussuri is something the redhat people want to try to fix
15:07:10 <gmann> #link https://review.opendev.org/q/topic:ussuri-pin-tempest
15:07:38 <gmann> yeah, they are almost ready, I am fixing on projetcs side also if any is failing with old Tempest in ussuri
15:07:55 <gmann> anyways should be ready by today ir friday max
15:08:04 <dansmith> I still see a fair number of cinder-related rechecks (and some timeouts) lately, but not in a super acute way AFAICT
15:09:18 <gmann> even devstack-plugin-ceph-tempest-py3 job is very unstable now a days, reported a bug #link https://bugs.launchpad.net/devstack-plugin-ceph/+bug/1975648
15:09:36 <gmann> gate is not so healthy at least this week
15:09:41 <dansmith> hmm, that's no good
15:09:49 <rosmaita> are you seeing a lot of 500s?
15:09:54 <gmann> yeah
15:10:08 <gmann> tempest.lib.exceptions.IdentityError: Got identity error
15:10:08 <gmann> Details: Unexpected status code 500
15:10:09 <rosmaita> ok, seems like ceph and mysqld are a bad combination
15:10:15 <rosmaita> we had to do this:
15:10:19 <dansmith> is it an oom problem?
15:10:20 <rosmaita> #link https://review.opendev.org/q/topic:increase-swap
15:10:37 <rosmaita> yes, mysqld goes oom and everything that uses a db breaks
15:10:46 <jungleboyj> :-(
15:10:48 <dansmith> yeah, so trimming down the services that run along with those would be a good strategy I think
15:10:59 <slaweq> we see such issue from time to time in neutron too
15:10:59 <dansmith> like is c-bak required for those jobs?
15:11:10 <slaweq> with mysqld killed by oom-killer
15:11:41 <slaweq> and we indeed removed services which we are not using in our jobs
15:11:47 <slaweq> it helped, at least for now
15:12:22 <gmann> that is used as base in many cinder jobs so I think that is why c-bak might is enabled  but we can extract that
15:12:28 <slaweq> and we also configured swap https://github.com/openstack/neutron/blob/master/zuul.d/tempest-singlenode.yaml#L600
15:12:46 <slaweq> and since then I don't think we had similar issues again
15:12:54 <dansmith> c-bak on the job from that bug wasn't super large
15:12:58 <gmann> rosmaita: do you think we can increase  swap in devstack-plugin-ceph-tempest-py3 too?
15:13:02 <dansmith> and even mysql was only 500M, which isn't very large
15:13:12 <gmann> humm
15:13:20 <rosmaita> gmann: don't see why not
15:13:25 <dansmith> which leads to it being just general "too much stuff in a box" and not one thing running amok
15:13:35 <rosmaita> dansmith: ++
15:13:41 <dansmith> https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_293/842821/2/check/devstack-plugin-ceph-tempest-py3/2933975/controller/logs/performance.json
15:13:51 <fungi> probably something else is eating ram and mysqld just happens to be the thing the oom killer decides to sacrifice
15:14:19 <fungi> i've seen that before when, for example, apache worker forks decide to gobble up tons of memory
15:14:22 <dansmith> fungi: right, mysql is always the largest, but the dump shows nothing spiking or even larger than usual, just lots of things
15:15:40 <gmann> we can try swap increase but it seems we do not know the cause so it might hide the actual issue
15:16:08 <gmann> anyways let's debug in qa channel or so separate from this meeting
15:16:13 <rosmaita> well, we used to run with 4G swap back in the day, the decrease to 1G is fairly recent
15:16:22 <dansmith> should probably add apache to the performance.json dump
15:16:45 <gmann> +1
15:16:57 <gmann> rosmaita: ok.
15:17:02 <fungi> unrelated to that particular instability, just a heads up that opendev is currently piloting a semi-automated full zuul service upgrade process which should be zero-downtime (like most of the recent manual upgrades), but just a heads up that there could be unforeseen disruptions today/tomorrow
15:17:23 <gmann> rosmaita: dansmith fungi let's discuss in qa after meeting
15:17:29 <gmann> slaweq: +1 on shrinking the services you do not in any particular job testing
15:17:30 <rosmaita> gmann: ack
15:17:58 <gmann> fungi: ack
15:18:04 <gmann> anything else on gate?
15:19:02 <gmann> let's move then
15:19:05 <gmann> #topic New ELK service dashboard: e-r service
15:19:31 <gmann> there is note from dpawlik about status
15:19:39 <gmann> "Small summary: the opensearch cluster space seems to be stable right now (there is enough space). For the elastic search recheck, I'm working on ansible role for deploying the e-r and push it to the ci-log-processing project. Don't know about the e-r functionality progress. "
15:20:05 <gmann> also merging rdo and master branch in e-r repo is going on
15:20:31 <fungi> there is a proposed acl update for e-r which would ease that task
15:20:38 <fungi> #link https://review.opendev.org/840455 elastic-recheck: allow releasers to merge/delete
15:20:58 <gmann> +1
15:21:21 <diablo_rojo_phone> Looks fine to me
15:21:25 <gmann> anything else on this topic
15:22:19 <gmann> #topic 'SLURP' as release cadence terminology
15:22:24 <gmann> Document release notes approach
15:22:25 <slaweq> +1
15:22:35 <gmann> rosmaita: I think you have patch up
15:22:46 <rosmaita> yep
15:22:47 <gmann> #link https://review.opendev.org/c/openstack/project-team-guide/+/843457
15:22:55 <rosmaita> thanks, lost my window
15:23:24 <rosmaita> if you want to see it in html, https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_9d6/843457/1/check/openstack-tox-docs/9d6e6c8/docs/release-management.html#release-notes-for-slurp-releases
15:23:36 <gmann> anything you want to discuss/bring or need review in gerrit?
15:23:45 <rosmaita> just need some feedback
15:23:54 <gmann> cool, I will review today
15:23:55 <rosmaita> i tried to be helpful and not overly prescriptive
15:23:57 <diablo_rojo_phone> Will check it out today!
15:24:08 <gmann> thanks rosmaita for putting it up
15:24:12 <rosmaita> np
15:24:33 <gmann> #topic Release identification schema in development process/cycle
15:24:51 <gmann> no new things on this just review on what we discussed :)
15:24:54 <jungleboyj> Have it open.
15:25:03 <gmann> I have pushed the resolution #link https://review.opendev.org/c/openstack/governance/+/843214/
15:25:04 <slaweq> added to my review list for tomorrow :)
15:25:17 <gmann> and documentation #link https://review.opendev.org/c/openstack/governance/+/841800
15:25:21 <gmann> slaweq: thanks
15:25:34 <jungleboyj> Have those all open now.  :-)
15:25:43 <slaweq> those 2 also :)
15:25:43 <diablo_rojo_phone> Same
15:25:47 <gmann> please review, both are almost same content and resolution is just to record the meeting agreement
15:25:55 <gmann> cool, thanks
15:26:07 <gmann> I will remove this topic from next meeting.
15:26:21 <gmann> and everyone relax on naming things :)
15:26:25 <rosmaita> \o/
15:26:28 <slaweq> ++
15:26:53 <gmann> rosmaita: dansmith I forgot to ask, we can remove SLUPR topic also from next meeting? I think all things are fine in this too?
15:27:06 <dansmith> yep
15:27:25 <gmann> #action gmann to remove the SLURP and Release name topic from next week meeting
15:27:25 <rosmaita> sure
15:27:31 <gmann> #topic Open Reviews
15:27:33 <dansmith> just continue on things like rosmaita's patch for reno guidelines
15:27:38 <gmann> +1
15:27:56 <gmann> there are few open review for long so let's discuss them as we have time todya
15:28:24 <gmann> #link https://review.opendev.org/c/openstack/governance/+/839880
15:28:32 <gmann> improve project governance
15:28:53 <slaweq> I need to address dansmith's comment there
15:28:55 <gmann> slaweq: I am fine with dansmith suggestion to increase window for existing projects too
15:29:11 <slaweq> gmann: yeah, me too but I didn't had time to get to this today
15:29:13 <dansmith> mine and knikolla's :)
15:29:15 <gmann> yeah and after that  I think it is in good shape
15:29:36 <slaweq> ok, it will be updated tomorrow morning
15:29:38 <gmann> dansmith: which one from knikolla ?
15:29:49 <dansmith> gmann: I'm saying he agreed so it's not just me complaining :P
15:30:05 <gmann> ah yeah, sorry. yes he also pointed that
15:30:47 <gmann> but I would like other tc-members also to check it and give comment today so that slaweq can update in one shot tomorrow
15:31:03 <slaweq> gmann: thx, that would be great indeed
15:31:15 <diablo_rojo_phone> Can do
15:31:23 <gmann> related to this, slaweq updating the 'project dropping criteria' also
15:31:25 <gmann> #link https://review.opendev.org/c/openstack/governance/+/840856
15:31:30 <gmann> please check this too
15:32:05 <slaweq> and this last one is on top of https://review.opendev.org/c/openstack/governance/+/810037/ too
15:32:16 <slaweq> which I continued after ricolin
15:32:31 <gmann> diablo_rojo_phone: I think we can abandon this now ? #link https://review.opendev.org/c/openstack/governance/+/834208
15:33:06 <gmann> slaweq: I will check today. one things failed for me where there was no review in any repo. but you updated that right? I have not checked it yet
15:33:12 <diablo_rojo_phone> Yeah will do.
15:33:21 <gmann> diablo_rojo_phone: thanks
15:33:33 <slaweq> gmann: yes, I addressed that today
15:33:36 <diablo_rojo_phone> Done
15:33:38 <slaweq> good catch, thx
15:33:50 <gmann> jungleboyj: what is next step on this? can you remove the WIP and we can review/merge or you want early feedback first #link https://review.opendev.org/c/openstack/governance/+/836888
15:34:27 <jungleboyj> gmann:  I needed to finish one of the sections in there and then I saw there were comments.
15:34:37 <jungleboyj> Need to find some time to work on that.
15:34:45 <gmann> diablo_rojo_phone: thanks
15:34:58 <gmann> jungleboyj: thanks, please let me know if you need help in this.
15:35:05 <jungleboyj> gmann:  Will do.
15:35:07 <gmann> that is all from open reviews.
15:35:28 <gmann> and from agenda too, anything else from anyone today?
15:36:10 <diablo_rojo_phone> Nope
15:36:28 <gmann> ok, if nothing else let's close early and can continue on fixing gate :)
15:36:30 <jungleboyj> Nope.
15:36:31 <gmann> NOTE: next meeting is on 2nd June and video call.
15:36:43 <gmann> thanks everyone for joining
15:36:43 <jungleboyj> Okie Dokie.
15:36:47 <jungleboyj> Thank you!
15:36:47 <gmann> #endmeeting