15:00:15 #startmeeting tc 15:00:15 Meeting started Thu May 26 15:00:15 2022 UTC and is due to finish in 60 minutes. The chair is gmann. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:00:15 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 15:00:16 The meeting name has been set to 'tc' 15:00:25 tc-members: meeting time 15:00:36 #topic Roll call 15:00:38 o/ 15:00:40 o/ 15:00:41 o/ 15:00:48 o/ 15:00:56 o/ 15:02:04 o/ 15:02:40 oj 15:03:19 arne_wiebalck_ informed that he will not be able to join due to holiday 15:03:20 let's start 15:03:25 #link https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Agenda_Suggestions 15:03:30 o/ 15:03:32 today agenda ^^ 15:04:13 #topic Follow up on past action items 15:04:16 gmann to schedule 'release name things' discussion call *again* 15:04:22 that is done 15:04:32 ++ 15:04:44 :-) 15:04:49 And so is the actual meeting 15:04:54 you scared me for a minute there 15:05:10 Yeah, I was afraid we were going to have to talk again. 15:05:26 my ++ was to the "that is done" part 15:05:30 :) 15:05:42 #topic Gate health check 15:06:07 stable/ussuri and stable/victoria is broken and temepst master too 15:06:49 that is due to py36 and tempest master usage iun stable/ussuri and victoria 15:07:01 yeah I know ussuri is something the redhat people want to try to fix 15:07:10 #link https://review.opendev.org/q/topic:ussuri-pin-tempest 15:07:38 yeah, they are almost ready, I am fixing on projetcs side also if any is failing with old Tempest in ussuri 15:07:55 anyways should be ready by today ir friday max 15:08:04 I still see a fair number of cinder-related rechecks (and some timeouts) lately, but not in a super acute way AFAICT 15:09:18 even devstack-plugin-ceph-tempest-py3 job is very unstable now a days, reported a bug #link https://bugs.launchpad.net/devstack-plugin-ceph/+bug/1975648 15:09:36 gate is not so healthy at least this week 15:09:41 hmm, that's no good 15:09:49 are you seeing a lot of 500s? 15:09:54 yeah 15:10:08 tempest.lib.exceptions.IdentityError: Got identity error 15:10:08 Details: Unexpected status code 500 15:10:09 ok, seems like ceph and mysqld are a bad combination 15:10:15 we had to do this: 15:10:19 is it an oom problem? 15:10:20 #link https://review.opendev.org/q/topic:increase-swap 15:10:37 yes, mysqld goes oom and everything that uses a db breaks 15:10:46 :-( 15:10:48 yeah, so trimming down the services that run along with those would be a good strategy I think 15:10:59 we see such issue from time to time in neutron too 15:10:59 like is c-bak required for those jobs? 15:11:10 with mysqld killed by oom-killer 15:11:41 and we indeed removed services which we are not using in our jobs 15:11:47 it helped, at least for now 15:12:22 that is used as base in many cinder jobs so I think that is why c-bak might is enabled but we can extract that 15:12:28 and we also configured swap https://github.com/openstack/neutron/blob/master/zuul.d/tempest-singlenode.yaml#L600 15:12:46 and since then I don't think we had similar issues again 15:12:54 c-bak on the job from that bug wasn't super large 15:12:58 rosmaita: do you think we can increase swap in devstack-plugin-ceph-tempest-py3 too? 15:13:02 and even mysql was only 500M, which isn't very large 15:13:12 humm 15:13:20 gmann: don't see why not 15:13:25 which leads to it being just general "too much stuff in a box" and not one thing running amok 15:13:35 dansmith: ++ 15:13:41 https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_293/842821/2/check/devstack-plugin-ceph-tempest-py3/2933975/controller/logs/performance.json 15:13:51 probably something else is eating ram and mysqld just happens to be the thing the oom killer decides to sacrifice 15:14:19 i've seen that before when, for example, apache worker forks decide to gobble up tons of memory 15:14:22 fungi: right, mysql is always the largest, but the dump shows nothing spiking or even larger than usual, just lots of things 15:15:40 we can try swap increase but it seems we do not know the cause so it might hide the actual issue 15:16:08 anyways let's debug in qa channel or so separate from this meeting 15:16:13 well, we used to run with 4G swap back in the day, the decrease to 1G is fairly recent 15:16:22 should probably add apache to the performance.json dump 15:16:45 +1 15:16:57 rosmaita: ok. 15:17:02 unrelated to that particular instability, just a heads up that opendev is currently piloting a semi-automated full zuul service upgrade process which should be zero-downtime (like most of the recent manual upgrades), but just a heads up that there could be unforeseen disruptions today/tomorrow 15:17:23 rosmaita: dansmith fungi let's discuss in qa after meeting 15:17:29 slaweq: +1 on shrinking the services you do not in any particular job testing 15:17:30 gmann: ack 15:17:58 fungi: ack 15:18:04 anything else on gate? 15:19:02 let's move then 15:19:05 #topic New ELK service dashboard: e-r service 15:19:31 there is note from dpawlik about status 15:19:39 "Small summary: the opensearch cluster space seems to be stable right now (there is enough space). For the elastic search recheck, I'm working on ansible role for deploying the e-r and push it to the ci-log-processing project. Don't know about the e-r functionality progress. " 15:20:05 also merging rdo and master branch in e-r repo is going on 15:20:31 there is a proposed acl update for e-r which would ease that task 15:20:38 #link https://review.opendev.org/840455 elastic-recheck: allow releasers to merge/delete 15:20:58 +1 15:21:21 Looks fine to me 15:21:25 anything else on this topic 15:22:19 #topic 'SLURP' as release cadence terminology 15:22:24 Document release notes approach 15:22:25 +1 15:22:35 rosmaita: I think you have patch up 15:22:46 yep 15:22:47 #link https://review.opendev.org/c/openstack/project-team-guide/+/843457 15:22:55 thanks, lost my window 15:23:24 if you want to see it in html, https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_9d6/843457/1/check/openstack-tox-docs/9d6e6c8/docs/release-management.html#release-notes-for-slurp-releases 15:23:36 anything you want to discuss/bring or need review in gerrit? 15:23:45 just need some feedback 15:23:54 cool, I will review today 15:23:55 i tried to be helpful and not overly prescriptive 15:23:57 Will check it out today! 15:24:08 thanks rosmaita for putting it up 15:24:12 np 15:24:33 #topic Release identification schema in development process/cycle 15:24:51 no new things on this just review on what we discussed :) 15:24:54 Have it open. 15:25:03 I have pushed the resolution #link https://review.opendev.org/c/openstack/governance/+/843214/ 15:25:04 added to my review list for tomorrow :) 15:25:17 and documentation #link https://review.opendev.org/c/openstack/governance/+/841800 15:25:21 slaweq: thanks 15:25:34 Have those all open now. :-) 15:25:43 those 2 also :) 15:25:43 Same 15:25:47 please review, both are almost same content and resolution is just to record the meeting agreement 15:25:55 cool, thanks 15:26:07 I will remove this topic from next meeting. 15:26:21 and everyone relax on naming things :) 15:26:25 \o/ 15:26:28 ++ 15:26:53 rosmaita: dansmith I forgot to ask, we can remove SLUPR topic also from next meeting? I think all things are fine in this too? 15:27:06 yep 15:27:25 #action gmann to remove the SLURP and Release name topic from next week meeting 15:27:25 sure 15:27:31 #topic Open Reviews 15:27:33 just continue on things like rosmaita's patch for reno guidelines 15:27:38 +1 15:27:56 there are few open review for long so let's discuss them as we have time todya 15:28:24 #link https://review.opendev.org/c/openstack/governance/+/839880 15:28:32 improve project governance 15:28:53 I need to address dansmith's comment there 15:28:55 slaweq: I am fine with dansmith suggestion to increase window for existing projects too 15:29:11 gmann: yeah, me too but I didn't had time to get to this today 15:29:13 mine and knikolla's :) 15:29:15 yeah and after that I think it is in good shape 15:29:36 ok, it will be updated tomorrow morning 15:29:38 dansmith: which one from knikolla ? 15:29:49 gmann: I'm saying he agreed so it's not just me complaining :P 15:30:05 ah yeah, sorry. yes he also pointed that 15:30:47 but I would like other tc-members also to check it and give comment today so that slaweq can update in one shot tomorrow 15:31:03 gmann: thx, that would be great indeed 15:31:15 Can do 15:31:23 related to this, slaweq updating the 'project dropping criteria' also 15:31:25 #link https://review.opendev.org/c/openstack/governance/+/840856 15:31:30 please check this too 15:32:05 and this last one is on top of https://review.opendev.org/c/openstack/governance/+/810037/ too 15:32:16 which I continued after ricolin 15:32:31 diablo_rojo_phone: I think we can abandon this now ? #link https://review.opendev.org/c/openstack/governance/+/834208 15:33:06 slaweq: I will check today. one things failed for me where there was no review in any repo. but you updated that right? I have not checked it yet 15:33:12 Yeah will do. 15:33:21 diablo_rojo_phone: thanks 15:33:33 gmann: yes, I addressed that today 15:33:36 Done 15:33:38 good catch, thx 15:33:50 jungleboyj: what is next step on this? can you remove the WIP and we can review/merge or you want early feedback first #link https://review.opendev.org/c/openstack/governance/+/836888 15:34:27 gmann: I needed to finish one of the sections in there and then I saw there were comments. 15:34:37 Need to find some time to work on that. 15:34:45 diablo_rojo_phone: thanks 15:34:58 jungleboyj: thanks, please let me know if you need help in this. 15:35:05 gmann: Will do. 15:35:07 that is all from open reviews. 15:35:28 and from agenda too, anything else from anyone today? 15:36:10 Nope 15:36:28 ok, if nothing else let's close early and can continue on fixing gate :) 15:36:30 Nope. 15:36:31 NOTE: next meeting is on 2nd June and video call. 15:36:43 thanks everyone for joining 15:36:43 Okie Dokie. 15:36:47 Thank you! 15:36:47 #endmeeting