| gmann | dansmith: rosmaita ^^ updated as per your comments | 01:09 |
|---|---|---|
| *** diablo_rojo_phone is now known as Guest360 | 05:08 | |
| *** tkajinam_ is now known as tkajinam | 05:11 | |
| *** ykarel_ is now known as ykarel | 07:33 | |
| *** diablo_rojo is now known as Guest378 | 08:51 | |
| *** carloss_ is now known as carloss | 10:31 | |
| dpawlik | gmann: Probably I will leave earlier from TC meeting today. Small summary: the opensearch cluster space seems to be stable right now (there is enough space). For the elastic search recheck, I'm working on ansible role for deploying the e-r and push it to the ci-log-processing project. Don't know about the e-r functionality progress. | 11:21 |
| *** mnaser_ is now known as mnaser | 12:41 | |
| opendevreview | Brian Rosmaita proposed openstack/project-team-guide master: Add SLURP release notes strategy https://review.opendev.org/c/openstack/project-team-guide/+/843457 | 12:44 |
| opendevreview | Slawek Kaplonski proposed openstack/governance master: Add project stats check tool https://review.opendev.org/c/openstack/governance/+/810037 | 14:05 |
| opendevreview | Slawek Kaplonski proposed openstack/governance master: Update potential triggers to consider dropping of the project https://review.opendev.org/c/openstack/governance/+/840856 | 14:06 |
| gmann | dpawlik: ack, thanks for the updates | 14:33 |
| *** Guest360 is now known as diablo_rojo_phone | 15:00 | |
| gmann | #startmeeting tc | 15:00 |
| opendevmeet | Meeting started Thu May 26 15:00:15 2022 UTC and is due to finish in 60 minutes. The chair is gmann. Information about MeetBot at http://wiki.debian.org/MeetBot. | 15:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 15:00 |
| opendevmeet | The meeting name has been set to 'tc' | 15:00 |
| gmann | tc-members: meeting time | 15:00 |
| gmann | #topic Roll call | 15:00 |
| gmann | o/ | 15:00 |
| slaweq | o/ | 15:00 |
| diablo_rojo_phone | o/ | 15:00 |
| dpawlik | o/ | 15:00 |
| jungleboyj | o/ | 15:00 |
| rosmaita | o/ | 15:02 |
| dansmith | oj | 15:02 |
| gmann | arne_wiebalck_ informed that he will not be able to join due to holiday | 15:03 |
| gmann | let's start | 15:03 |
| gmann | #link https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Agenda_Suggestions | 15:03 |
| spotz | o/ | 15:03 |
| gmann | today agenda ^^ | 15:03 |
| gmann | #topic Follow up on past action items | 15:04 |
| gmann | gmann to schedule 'release name things' discussion call *again* | 15:04 |
| gmann | that is done | 15:04 |
| dansmith | ++ | 15:04 |
| jungleboyj | :-) | 15:04 |
| diablo_rojo_phone | And so is the actual meeting | 15:04 |
| rosmaita | you scared me for a minute there | 15:04 |
| jungleboyj | Yeah, I was afraid we were going to have to talk again. | 15:05 |
| dansmith | my ++ was to the "that is done" part | 15:05 |
| gmann | :) | 15:05 |
| gmann | #topic Gate health check | 15:05 |
| gmann | stable/ussuri and stable/victoria is broken and temepst master too | 15:06 |
| gmann | that is due to py36 and tempest master usage iun stable/ussuri and victoria | 15:06 |
| dansmith | yeah I know ussuri is something the redhat people want to try to fix | 15:07 |
| gmann | #link https://review.opendev.org/q/topic:ussuri-pin-tempest | 15:07 |
| gmann | yeah, they are almost ready, I am fixing on projetcs side also if any is failing with old Tempest in ussuri | 15:07 |
| gmann | anyways should be ready by today ir friday max | 15:07 |
| dansmith | I still see a fair number of cinder-related rechecks (and some timeouts) lately, but not in a super acute way AFAICT | 15:08 |
| gmann | even devstack-plugin-ceph-tempest-py3 job is very unstable now a days, reported a bug #link https://bugs.launchpad.net/devstack-plugin-ceph/+bug/1975648 | 15:09 |
| gmann | gate is not so healthy at least this week | 15:09 |
| dansmith | hmm, that's no good | 15:09 |
| rosmaita | are you seeing a lot of 500s? | 15:09 |
| gmann | yeah | 15:09 |
| gmann | tempest.lib.exceptions.IdentityError: Got identity error | 15:10 |
| gmann | Details: Unexpected status code 500 | 15:10 |
| rosmaita | ok, seems like ceph and mysqld are a bad combination | 15:10 |
| rosmaita | we had to do this: | 15:10 |
| dansmith | is it an oom problem? | 15:10 |
| rosmaita | #link https://review.opendev.org/q/topic:increase-swap | 15:10 |
| rosmaita | yes, mysqld goes oom and everything that uses a db breaks | 15:10 |
| jungleboyj | :-( | 15:10 |
| dansmith | yeah, so trimming down the services that run along with those would be a good strategy I think | 15:10 |
| slaweq | we see such issue from time to time in neutron too | 15:10 |
| dansmith | like is c-bak required for those jobs? | 15:10 |
| slaweq | with mysqld killed by oom-killer | 15:11 |
| slaweq | and we indeed removed services which we are not using in our jobs | 15:11 |
| slaweq | it helped, at least for now | 15:11 |
| gmann | that is used as base in many cinder jobs so I think that is why c-bak might is enabled but we can extract that | 15:12 |
| slaweq | and we also configured swap https://github.com/openstack/neutron/blob/master/zuul.d/tempest-singlenode.yaml#L600 | 15:12 |
| slaweq | and since then I don't think we had similar issues again | 15:12 |
| dansmith | c-bak on the job from that bug wasn't super large | 15:12 |
| gmann | rosmaita: do you think we can increase swap in devstack-plugin-ceph-tempest-py3 too? | 15:12 |
| dansmith | and even mysql was only 500M, which isn't very large | 15:13 |
| gmann | humm | 15:13 |
| rosmaita | gmann: don't see why not | 15:13 |
| dansmith | which leads to it being just general "too much stuff in a box" and not one thing running amok | 15:13 |
| rosmaita | dansmith: ++ | 15:13 |
| dansmith | https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_293/842821/2/check/devstack-plugin-ceph-tempest-py3/2933975/controller/logs/performance.json | 15:13 |
| fungi | probably something else is eating ram and mysqld just happens to be the thing the oom killer decides to sacrifice | 15:13 |
| fungi | i've seen that before when, for example, apache worker forks decide to gobble up tons of memory | 15:14 |
| dansmith | fungi: right, mysql is always the largest, but the dump shows nothing spiking or even larger than usual, just lots of things | 15:14 |
| gmann | we can try swap increase but it seems we do not know the cause so it might hide the actual issue | 15:15 |
| gmann | anyways let's debug in qa channel or so separate from this meeting | 15:16 |
| rosmaita | well, we used to run with 4G swap back in the day, the decrease to 1G is fairly recent | 15:16 |
| dansmith | should probably add apache to the performance.json dump | 15:16 |
| gmann | +1 | 15:16 |
| gmann | rosmaita: ok. | 15:16 |
| fungi | unrelated to that particular instability, just a heads up that opendev is currently piloting a semi-automated full zuul service upgrade process which should be zero-downtime (like most of the recent manual upgrades), but just a heads up that there could be unforeseen disruptions today/tomorrow | 15:17 |
| gmann | rosmaita: dansmith fungi let's discuss in qa after meeting | 15:17 |
| gmann | slaweq: +1 on shrinking the services you do not in any particular job testing | 15:17 |
| rosmaita | gmann: ack | 15:17 |
| gmann | fungi: ack | 15:17 |
| gmann | anything else on gate? | 15:18 |
| gmann | let's move then | 15:19 |
| gmann | #topic New ELK service dashboard: e-r service | 15:19 |
| gmann | there is note from dpawlik about status | 15:19 |
| gmann | "Small summary: the opensearch cluster space seems to be stable right now (there is enough space). For the elastic search recheck, I'm working on ansible role for deploying the e-r and push it to the ci-log-processing project. Don't know about the e-r functionality progress. " | 15:19 |
| gmann | also merging rdo and master branch in e-r repo is going on | 15:20 |
| fungi | there is a proposed acl update for e-r which would ease that task | 15:20 |
| fungi | #link https://review.opendev.org/840455 elastic-recheck: allow releasers to merge/delete | 15:20 |
| gmann | +1 | 15:20 |
| diablo_rojo_phone | Looks fine to me | 15:21 |
| gmann | anything else on this topic | 15:21 |
| gmann | #topic 'SLURP' as release cadence terminology | 15:22 |
| gmann | Document release notes approach | 15:22 |
| slaweq | +1 | 15:22 |
| gmann | rosmaita: I think you have patch up | 15:22 |
| rosmaita | yep | 15:22 |
| gmann | #link https://review.opendev.org/c/openstack/project-team-guide/+/843457 | 15:22 |
| rosmaita | thanks, lost my window | 15:22 |
| rosmaita | if you want to see it in html, https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_9d6/843457/1/check/openstack-tox-docs/9d6e6c8/docs/release-management.html#release-notes-for-slurp-releases | 15:23 |
| gmann | anything you want to discuss/bring or need review in gerrit? | 15:23 |
| rosmaita | just need some feedback | 15:23 |
| gmann | cool, I will review today | 15:23 |
| rosmaita | i tried to be helpful and not overly prescriptive | 15:23 |
| diablo_rojo_phone | Will check it out today! | 15:23 |
| gmann | thanks rosmaita for putting it up | 15:24 |
| rosmaita | np | 15:24 |
| gmann | #topic Release identification schema in development process/cycle | 15:24 |
| gmann | no new things on this just review on what we discussed :) | 15:24 |
| jungleboyj | Have it open. | 15:24 |
| gmann | I have pushed the resolution #link https://review.opendev.org/c/openstack/governance/+/843214/ | 15:25 |
| slaweq | added to my review list for tomorrow :) | 15:25 |
| gmann | and documentation #link https://review.opendev.org/c/openstack/governance/+/841800 | 15:25 |
| gmann | slaweq: thanks | 15:25 |
| jungleboyj | Have those all open now. :-) | 15:25 |
| slaweq | those 2 also :) | 15:25 |
| diablo_rojo_phone | Same | 15:25 |
| gmann | please review, both are almost same content and resolution is just to record the meeting agreement | 15:25 |
| gmann | cool, thanks | 15:25 |
| gmann | I will remove this topic from next meeting. | 15:26 |
| gmann | and everyone relax on naming things :) | 15:26 |
| rosmaita | \o/ | 15:26 |
| slaweq | ++ | 15:26 |
| gmann | rosmaita: dansmith I forgot to ask, we can remove SLUPR topic also from next meeting? I think all things are fine in this too? | 15:26 |
| dansmith | yep | 15:27 |
| gmann | #action gmann to remove the SLURP and Release name topic from next week meeting | 15:27 |
| rosmaita | sure | 15:27 |
| gmann | #topic Open Reviews | 15:27 |
| dansmith | just continue on things like rosmaita's patch for reno guidelines | 15:27 |
| gmann | +1 | 15:27 |
| gmann | there are few open review for long so let's discuss them as we have time todya | 15:27 |
| gmann | #link https://review.opendev.org/c/openstack/governance/+/839880 | 15:28 |
| gmann | improve project governance | 15:28 |
| slaweq | I need to address dansmith's comment there | 15:28 |
| gmann | slaweq: I am fine with dansmith suggestion to increase window for existing projects too | 15:28 |
| slaweq | gmann: yeah, me too but I didn't had time to get to this today | 15:29 |
| dansmith | mine and knikolla's :) | 15:29 |
| gmann | yeah and after that I think it is in good shape | 15:29 |
| slaweq | ok, it will be updated tomorrow morning | 15:29 |
| gmann | dansmith: which one from knikolla ? | 15:29 |
| dansmith | gmann: I'm saying he agreed so it's not just me complaining :P | 15:29 |
| gmann | ah yeah, sorry. yes he also pointed that | 15:30 |
| gmann | but I would like other tc-members also to check it and give comment today so that slaweq can update in one shot tomorrow | 15:30 |
| slaweq | gmann: thx, that would be great indeed | 15:31 |
| diablo_rojo_phone | Can do | 15:31 |
| gmann | related to this, slaweq updating the 'project dropping criteria' also | 15:31 |
| gmann | #link https://review.opendev.org/c/openstack/governance/+/840856 | 15:31 |
| gmann | please check this too | 15:31 |
| slaweq | and this last one is on top of https://review.opendev.org/c/openstack/governance/+/810037/ too | 15:32 |
| slaweq | which I continued after ricolin | 15:32 |
| gmann | diablo_rojo_phone: I think we can abandon this now ? #link https://review.opendev.org/c/openstack/governance/+/834208 | 15:32 |
| gmann | slaweq: I will check today. one things failed for me where there was no review in any repo. but you updated that right? I have not checked it yet | 15:33 |
| diablo_rojo_phone | Yeah will do. | 15:33 |
| gmann | diablo_rojo_phone: thanks | 15:33 |
| slaweq | gmann: yes, I addressed that today | 15:33 |
| diablo_rojo_phone | Done | 15:33 |
| slaweq | good catch, thx | 15:33 |
| gmann | jungleboyj: what is next step on this? can you remove the WIP and we can review/merge or you want early feedback first #link https://review.opendev.org/c/openstack/governance/+/836888 | 15:33 |
| jungleboyj | gmann: I needed to finish one of the sections in there and then I saw there were comments. | 15:34 |
| jungleboyj | Need to find some time to work on that. | 15:34 |
| gmann | diablo_rojo_phone: thanks | 15:34 |
| gmann | jungleboyj: thanks, please let me know if you need help in this. | 15:34 |
| jungleboyj | gmann: Will do. | 15:35 |
| gmann | that is all from open reviews. | 15:35 |
| gmann | and from agenda too, anything else from anyone today? | 15:35 |
| diablo_rojo_phone | Nope | 15:36 |
| gmann | ok, if nothing else let's close early and can continue on fixing gate :) | 15:36 |
| jungleboyj | Nope. | 15:36 |
| gmann | NOTE: next meeting is on 2nd June and video call. | 15:36 |
| gmann | thanks everyone for joining | 15:36 |
| jungleboyj | Okie Dokie. | 15:36 |
| jungleboyj | Thank you! | 15:36 |
| gmann | #endmeeting | 15:36 |
| opendevmeet | Meeting ended Thu May 26 15:36:47 2022 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 15:36 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/tc/2022/tc.2022-05-26-15.00.html | 15:36 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/tc/2022/tc.2022-05-26-15.00.txt | 15:36 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/tc/2022/tc.2022-05-26-15.00.log.html | 15:36 |
| spotz | Thanks | 15:36 |
| slaweq | nope | 15:36 |
| slaweq | o/ | 15:37 |
| rosmaita | brb | 15:37 |
| diablo_rojo_phone | Thanks! | 15:37 |
| rosmaita | back | 15:39 |
| *** frenzyfriday|ruck is now known as frenzy_friday | 15:54 | |
| opendevreview | Dr. Jens Harbott proposed openstack/governance master: Add a repository for the Large Scale SIG https://review.opendev.org/c/openstack/governance/+/843535 | 17:30 |
| *** diablo_rojo__ is now known as diablo_rojo | 17:33 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!