| gmann | slaweq: ralonsoh: ykarel: do you know any known issue on neutron-linuxbridge-tempest on stable/2023.2 ? it seems it is failing 100% now https://zuul.openstack.org/builds?job_name=neutron-linuxbridge-tempest&branch=stable%2F2023.2&skip=0 | 02:39 |
|---|---|---|
| gmann | it is blocking the devstack setup of stable/2023.2 https://review.opendev.org/q/topic:qa-2023-2-release+status:open | 02:39 |
| ralonsoh | gmann, yes, it should be solved with https://review.opendev.org/c/openstack/neutron/+/897438 | 06:12 |
| slaweq | gmann but You should also know that since some time linuxbridge is marked as "experimental" feature in Neutron and we support it only in best effort manner. We also don't run neutron-tempest-plugin-linuxbridge job in gate anymore (just in experimental and periodic queues) | 07:03 |
| frickler | so we should just drop it from devstack testing I guess? not sure when and why it was added initially | 07:18 |
| frickler | seems this goes back to at least the great zuul job refactoring of 2018. /me goes and makes a dropping patch. https://review.opendev.org/c/openstack/devstack/+/600990 | 07:21 |
| opendevreview | Dr. Jens Harbott proposed openstack/devstack master: zuul: Drop neutron-linuxbridge-tempest job https://review.opendev.org/c/openstack/devstack/+/897759 | 07:23 |
| ykarel | +1 | 07:24 |
| frickler | gmann: kopecmartin: ^^ we can backport that to 2023.2 to unblock things | 07:24 |
| ralonsoh | the neutron patch is enough to unblock the gate, just to let you know | 07:34 |
| frickler | ralonsoh: yes, but it neutron doesn't gate linuxbridge any longer, devstack should even less | 07:51 |
| frickler | s/it/if | 07:51 |
| ralonsoh | perfect then | 07:52 |
| opendevreview | Merged openstack/devstack master: zuul: Drop neutron-linuxbridge-tempest job https://review.opendev.org/c/openstack/devstack/+/897759 | 11:58 |
| opendevreview | Dr. Jens Harbott proposed openstack/devstack stable/2023.2: zuul: Drop neutron-linuxbridge-tempest job https://review.opendev.org/c/openstack/devstack/+/897804 | 12:44 |
| *** haleyb|away is now known as haleyb | 13:26 | |
| opendevreview | Dr. Jens Harbott proposed openstack/devstack stable/2023.2: [stable only] Make devstack-platform-rocky-blue-onyx non voting https://review.opendev.org/c/openstack/devstack/+/897631 | 13:30 |
| opendevreview | Dr. Jens Harbott proposed openstack/devstack stable/2023.2: Update branches for stable/2023.2 https://review.opendev.org/c/openstack/devstack/+/896863 | 13:30 |
| opendevreview | Dr. Jens Harbott proposed openstack/devstack stable/2023.2: Stop installing Tempest at system wide for stable/2023.2 https://review.opendev.org/c/openstack/devstack/+/896864 | 13:30 |
| opendevreview | Dr. Jens Harbott proposed openstack/devstack stable/2023.2: Cap max microversions for stable/2023.2 https://review.opendev.org/c/openstack/devstack/+/896865 | 13:30 |
| opendevreview | Dr. Jens Harbott proposed openstack/devstack stable/2023.2: Cap stable/2023.2 network, swift, volume api_extensions for tempest https://review.opendev.org/c/openstack/devstack/+/896866 | 13:30 |
| yadnesh | Centos9 telemetry jobs are failing due to " Problem: package gcc-c++-11.4.1-2.1.el9.x86_64 from appstream requires gcc = 11.4.1-2.1.el9, but none of the providers can be installed" https://zuul.opendev.org/t/openstack/build/d2d5e609b70d47ff908c256bc0fbc0b9/log/job-output.txt#3724-3727 | 14:38 |
| frickler | my suggestion is to switch to a stable distro like ubuntu or debian. or maybe rockylinux, though they also have had some mirror issues lately | 14:42 |
| slaweq | gmann hi, I wanted to ask You about S-RBAC and service role testing | 14:52 |
| slaweq | we recently merged https://review.opendev.org/c/openstack/neutron/+/886724 in Neutron so we have policies for service role already. Is there any job already to test that in CI together with other projects? | 14:53 |
| kopecmartin | #startmeeting qa | 15:00 |
| opendevmeet | Meeting started Tue Oct 10 15:00:24 2023 UTC and is due to finish in 60 minutes. The chair is kopecmartin. Information about MeetBot at http://wiki.debian.org/MeetBot. | 15:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 15:00 |
| opendevmeet | The meeting name has been set to 'qa' | 15:00 |
| kopecmartin | #link https://wiki.openstack.org/wiki/Meetings/QATeamMeeting#Agenda_for_next_Office_hours | 15:00 |
| kopecmartin | agenda ^ | 15:00 |
| frickler | \o | 15:01 |
| kopecmartin | o/ | 15:02 |
| kopecmartin | #topic Announcement and Action Item (Optional) | 15:02 |
| kopecmartin | we're past the release, although there are a few patches still needed to be merged | 15:02 |
| lpiwowar | \o | 15:02 |
| kopecmartin | #link https://review.opendev.org/q/topic:qa-2023-2-release+status:open | 15:02 |
| kopecmartin | o/ | 15:02 |
| kopecmartin | #topic Bobcat Priority Items progress | 15:03 |
| kopecmartin | #link https://etherpad.opendev.org/p/qa-bobcat-priority | 15:03 |
| kopecmartin | 'unit test coverage in Tempest' has been completed | 15:03 |
| kopecmartin | #link https://review.opendev.org/c/openstack/tempest/+/893080 | 15:04 |
| kopecmartin | the rest will be triaged during the PTG | 15:04 |
| kopecmartin | that brings us to the next topic | 15:04 |
| kopecmartin | #topic OpenStack Events Updates and Planning | 15:04 |
| kopecmartin | the next PTG will be held virtually, October 23-27, 2023 | 15:04 |
| kopecmartin | less than 2 weeks from now | 15:04 |
| kopecmartin | you can propose topics that we will discuss during PTG here | 15:05 |
| kopecmartin | #link https://etherpad.opendev.org/p/oct2023-ptg-qa | 15:05 |
| kopecmartin | i assume everyone is already registered :) | 15:05 |
| frickler | you assume wrongly | 15:05 |
| kopecmartin | oh, sorry, i remember now :D | 15:05 |
| lpiwowar | I think I should ... I'm just double-checking. | 15:06 |
| kopecmartin | anyway, anyone can join the calls regardless the registration ;) | 15:06 |
| kopecmartin | you may also influence dates of QA PTG sessions, just fill this: | 15:06 |
| kopecmartin | #link https://framadate.org/f26R3EcZ2BOo7r8Q | 15:06 |
| kopecmartin | I'll schedule something by the end of next week | 15:06 |
| kopecmartin | the topic etherpad looks tragic for now, although proposals always appear kind of last minute, so I'm not worrying | 15:07 |
| kopecmartin | #topic Gate Status Checks | 15:08 |
| kopecmartin | #link https://review.opendev.org/q/label:Review-Priority%253D%252B2+status:open+(project:openstack/tempest+OR+project:openstack/patrole+OR+project:openstack/devstack+OR+project:openstack/grenade) | 15:08 |
| kopecmartin | no patches, anything urgent to review? | 15:08 |
| frickler | the devstack series for 2023.2 | 15:08 |
| frickler | I should've marked that stack | 15:08 |
| kopecmartin | these? | 15:09 |
| kopecmartin | #link https://review.opendev.org/q/topic:qa-2023-2-release+status:open | 15:09 |
| frickler | https://review.opendev.org/c/openstack/devstack/+/896866 and below | 15:09 |
| kopecmartin | i'm monitoring it | 15:09 |
| kopecmartin | +1 | 15:09 |
| kopecmartin | actually i can already leave my vote there | 15:10 |
| lpiwowar | kopecmartin: I have these two open patches: https://review.opendev.org/c/openstack/tempest/+/896011 (tempest), https://review.opendev.org/c/openstack/devstack/+/897402 (devstack) | 15:10 |
| kopecmartin | i voted on the first one, the other one is on my list | 15:11 |
| kopecmartin | gmann: when you have a sec please ^ | 15:11 |
| frickler | oh, devstack was also affected by the Pillow bump https://zuul.opendev.org/t/openstack/build/1f99c77a838b40659f6f85d7f6e27871 | 15:12 |
| frickler | so we need to look at rewriting that (the?) blockdiag(s). but maybe there will be a cross project discussion at the ptg for that, too | 15:13 |
| kopecmartin | oh, good point | 15:14 |
| kopecmartin | moving on | 15:16 |
| kopecmartin | #topic Bare rechecks | 15:16 |
| kopecmartin | #link https://etherpad.opendev.org/p/recheck-weekly-summary | 15:16 |
| kopecmartin | all good there | 15:16 |
| kopecmartin | #topic Periodic jobs Status Checks | 15:16 |
| kopecmartin | periodic stable full | 15:16 |
| kopecmartin | #link https://zuul.openstack.org/builds?pipeline=periodic-stable&job_name=tempest-full-yoga&job_name=tempest-full-xena&job_name=tempest-full-zed&job_name=tempest-full-2023-1&job_name=tempest-full-2023-2 | 15:16 |
| kopecmartin | periodic stable slow | 15:16 |
| kopecmartin | #link https://zuul.openstack.org/builds?job_name=tempest-slow-2023-2&jjob_name=tempest-slow-2023-1&job_name=tempest-slow-zed&job_name=tempest-slow-yoga&job_name=tempest-slow-xena | 15:16 |
| kopecmartin | periodic extra tests | 15:16 |
| kopecmartin | #link https://zuul.openstack.org/builds?job_name=tempest-full-2023-2-extra-tests&job_name=tempest-full-2023-1-extra-tests&job_name=tempest-full-zed-extra-tests&job_name=tempest-full-yoga-extra-tests&job_name=tempest-full-xena-extra-tests | 15:16 |
| kopecmartin | periodic master | 15:16 |
| kopecmartin | #link https://zuul.openstack.org/builds?project=openstack%2Ftempest&project=openstack%2Fdevstack&pipeline=periodic | 15:16 |
| kopecmartin | tempest-centos9-stream-fips failed twice , hmm | 15:17 |
| kopecmartin | no, just once | 15:17 |
| frickler | plain centos too | 15:18 |
| kopecmartin | no module named pip, hopefully random | 15:18 |
| frickler | see also the comment just before the meeting | 15:18 |
| opendevreview | Katarina Strenkova proposed openstack/tempest master: [WIP] Implement purge list in tempest cleanup https://review.opendev.org/c/openstack/tempest/+/897847 | 15:18 |
| kopecmartin | the one about centos9? yeah .. it doesn't look like it's in our hands | 15:19 |
| frickler | IMHO we should stop supporting it, it breaks way too often because of uncoordinated or broken pkg updates | 15:19 |
| kopecmartin | right, that would make sense, let's have a brief PTG topic about that? | 15:20 |
| * kopecmartin added that to the etherpad | 15:21 | |
| kopecmartin | #topic Distros check | 15:22 |
| kopecmartin | cs-9 | 15:22 |
| kopecmartin | #link https://zuul.openstack.org/builds?job_name=tempest-full-centos-9-stream&job_name=devstack-platform-centos-9-stream&skip=0 | 15:22 |
| kopecmartin | debian | 15:22 |
| kopecmartin | #link https://zuul.openstack.org/builds?job_name=devstack-platform-debian-bullseye&job_name=devstack-platform-debian-bookworm&skip=0 | 15:22 |
| kopecmartin | rocky | 15:22 |
| kopecmartin | #link https://zuul.openstack.org/builds?job_name=devstack-platform-rocky-blue-onyx | 15:22 |
| kopecmartin | openEuler | 15:22 |
| kopecmartin | #link https://zuul.openstack.org/builds?job_name=devstack-platform-openEuler-22.03-ovn-source&job_name=devstack-platform-openEuler-22.03-ovs&skip=0 | 15:22 |
| kopecmartin | jammy | 15:22 |
| kopecmartin | #link https://zuul.opendev.org/t/openstack/builds?job_name=devstack-platform-ubuntu-jammy-ovn-source&job_name=devstack-platform-ubuntu-jammy-ovs&skip=0 | 15:22 |
| kopecmartin | yep, centos is broken, on master as well as 2023.2 | 15:23 |
| kopecmartin | rocky is broken on stable/2023.2 too, but there is a fix for that, isn't it? | 15:24 |
| kopecmartin | oh, right | 15:24 |
| kopecmartin | #link https://bugs.launchpad.net/devstack/+bug/2038733 | 15:24 |
| kopecmartin | that's why it was familiar, i saw the LP | 15:25 |
| kopecmartin | euler hasn't been running on master for a month due to the libvirt version incompatibility .. we'll revisit that after the PTG | 15:26 |
| kopecmartin | #link https://bugs.launchpad.net/devstack/+bug/2035224 | 15:26 |
| frickler | I wonder why that is passing on 2023.2 though | 15:27 |
| frickler | ah, no, 2023.1 is passing, nvm | 15:27 |
| kopecmartin | that's about it | 15:29 |
| kopecmartin | #topic Sub Teams highlights | 15:29 |
| kopecmartin | Changes with Review-Priority == +1 | 15:29 |
| kopecmartin | #link https://review.opendev.org/q/label:Review-Priority%253D%252B1+status:open+(project:openstack/tempest+OR+project:openstack/patrole+OR+project:openstack/devstack+OR+project:openstack/grenade) | 15:29 |
| kopecmartin | no patches | 15:29 |
| kopecmartin | #topic Open Discussion | 15:29 |
| kopecmartin | anything for the open discussion? | 15:29 |
| lpiwowar | Nothing from my side | 15:30 |
| frickler | anything on global venv? do we need to punt that to the ptg, too? | 15:30 |
| kopecmartin | nothing new, i gave it a few hours couple weeks back, couldn't figure it out | 15:31 |
| kopecmartin | yeah, we can add it to the ptg list | 15:31 |
| *** haleyb_ is now known as haleyb | 15:32 | |
| kopecmartin | #topic Bug Triage | 15:34 |
| kopecmartin | #link https://etherpad.openstack.org/p/qa-bug-triage-bobcat | 15:34 |
| kopecmartin | we have 2 bugs in tempest that have fixes looking promising: | 15:35 |
| kopecmartin | #link https://bugs.launchpad.net/tempest/+bug/2038802 | 15:35 |
| kopecmartin | #link https://bugs.launchpad.net/tempest/+bug/2034913 | 15:35 |
| kopecmartin | when you have a minute gmann, please check | 15:35 |
| kopecmartin | that's all from my side | 15:36 |
| kopecmartin | anything else to cover? | 15:36 |
| lpiwowar | kopecmartin: probably no | 15:36 |
| kopecmartin | thank you all then, see you around | 15:37 |
| kopecmartin | #endmeeting | 15:37 |
| opendevmeet | Meeting ended Tue Oct 10 15:37:06 2023 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 15:37 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/qa/2023/qa.2023-10-10-15.00.html | 15:37 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/qa/2023/qa.2023-10-10-15.00.txt | 15:37 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/qa/2023/qa.2023-10-10-15.00.log.html | 15:37 |
| lpiwowar | kopecmartin: thanks! | 15:37 |
| frickler | thx o/ | 15:38 |
| gmann | slaweq: ralonsoh frickler thanks for reply for linuxbridge job, +1 on dropping it | 15:54 |
| gmann | kopecmartin: can you review this too https://review.opendev.org/c/openstack/devstack/+/897804 | 15:54 |
| frickler | kopecmartin: and https://review.opendev.org/c/openstack/devstack/+/897631/3 , seems you only went by the topic | 16:14 |
| dansmith | artom: so, on my whitebox cpu state thing | 17:04 |
| dansmith | artom: not sure if you caught up, but I had to change the dedicated cpus to 4-5 instead of 0-1 because cpu0 can't be offlined and nova crashes on start if it tries (filed another bug for that) | 17:04 |
| dansmith | but now another test is failing that seems cpu-number-specific | 17:05 |
| dansmith | can you have a look? | 17:05 |
| artom | dansmith, yeah, I didn't look into it too deeply, but it looks pretty systematic, I suspect something's not being cleaned up properly | 17:05 |
| dansmith | I sure hope it doesn't have cpu numbers hard-coded in it | 17:05 |
| dansmith | so you think it's something not cleaned up after my test and not that rearranging the cpus is breaking it? | 17:05 |
| artom | What doesn't have CPU numbers hard-coded? | 17:06 |
| artom | Not hardcoded per se, but there's an assumption that both compute hosts come with disjoint cpu_dedicated_set and cpu_shared_set out of the box | 17:06 |
| dansmith | the test that's failing, I haven't even looked at the test itself | 17:06 |
| dansmith | disjoint, sure | 17:06 |
| dansmith | this is the change I made: https://review.opendev.org/c/openstack/whitebox-tempest-plugin/+/897579/8/.zuul.yaml | 17:06 |
| dansmith | is it complaining because that overlaps with the other compute? | 17:07 |
| artom | Oh, didn't even notice the zuul.yaml change | 17:08 |
| dansmith | that's why I'm asking, to make sure you saw that I had to change that | 17:08 |
| artom | OK, I think yeah, that broke it - the assumption is that cpu_dedicated_set and cpu_shared_set are all disjoint four-ways | 17:09 |
| artom | So the cpu_shared_set is different between the two computes | 17:09 |
| artom | And cpu_dedicated_set is disjoint between the two computes | 17:09 |
| artom | And then cpu_shared_set and cpu_dedicated_set are disjoint from each other | 17:09 |
| dansmith | okay, so, hmm | 17:10 |
| artom | If you can't offline 0, could you not boot on the node that doesn't have 0 in its dedicated set? There's bound to be 1 | 17:10 |
| dansmith | that will mean that my test would fail if it landed on the other compute | 17:10 |
| artom | You can inspect the host, get its dedicated set, and pick the one that doesn't have CPU 0 in that set | 17:11 |
| dansmith | well, I kinda think that zero should likely never be in the dedicated set regardless, so I'm not sure that's really a good fix | 17:11 |
| artom | We could try that as well | 17:11 |
| dansmith | can we remove zero from both sets on both hosts? | 17:11 |
| artom | I... _think_ so? | 17:12 |
| artom | If we've done our stuff properly, it should pass CI | 17:12 |
| artom | But the cardinality of shared and dedicated sets has to remain the same on both hosts (so 3 and 3 and 2 and 2, or whatever) | 17:12 |
| dansmith | well, if I just remove zero from both sets on both hosts, they should be good/ | 17:13 |
| dansmith | oh because one will shrink if I do that | 17:14 |
| artom | Would have to run it through CI, I don't remember off the top of my head | 17:14 |
| artom | But yeah, one will shrink | 17:14 |
| opendevreview | Dan Smith proposed openstack/whitebox-tempest-plugin master: Test nova's cpu host state management https://review.opendev.org/c/openstack/whitebox-tempest-plugin/+/897579 | 17:15 |
| dansmith | so this maybe ? | 17:16 |
| dansmith | to keep them all the same size | 17:16 |
| artom | Yep, looks sensible | 17:17 |
| artom | I can't remember if anything depends on there being 4 dedicated CPUs | 17:17 |
| artom | Like, booting 2 VMs with 2 CPUs each or something | 17:17 |
| dansmith | ack, we'll see | 17:17 |
| artom | So it might pass, but we'll need to look at the skipped tests as well | 17:17 |
| * dansmith nods | 17:17 | |
| artom | dansmith, you're finding a whole bunch of problems with how whitebox was coded :) | 18:44 |
| artom | ... Thanks? :) | 18:44 |
| dansmith | sorry man | 18:44 |
| dansmith | they seem legit tho :) | 18:44 |
| artom | Yeah, I left comments | 18:45 |
| artom | We were too coupled to the deployment (which makes sense, tests like that don't have much of a choice), but that coupling was hardcoded, and we should have done it better | 18:45 |
| dansmith | are you saying this because you've looked at the latest failure? | 18:46 |
| artom | Yes | 18:47 |
| dansmith | ack, will look after my current meeting | 18:47 |
| artom | Ack | 18:47 |
| opendevreview | Merged openstack/devstack stable/2023.2: zuul: Drop neutron-linuxbridge-tempest job https://review.opendev.org/c/openstack/devstack/+/897804 | 18:54 |
| dansmith | artom: okay so you want me to make those changes in this patch, or a separate one, or are you working on that now and I can base this on something? | 19:01 |
| artom | dansmith, I was thinking you'd introduce a new "base" patch under your test | 19:02 |
| slaweq | gmann hi, I'm not sure if You saw my question earlier so I will ask again :) | 19:02 |
| slaweq | we recently merged https://review.opendev.org/c/openstack/neutron/+/886724 in Neutron so we have policies for service role already. Is there any job already to test that in CI together with other projects? | 19:02 |
| slaweq | I'm now leaving for today but please send me here any info You have (if You have anything) and I will get back to it tomorrow morning | 19:02 |
| slaweq | thx in advance | 19:02 |
| gmann | slaweq: sorry, i read and got distracted while replying | 19:02 |
| gmann | slaweq: sure, I think existing job should validated those but I will check and update you tomorrow. | 19:04 |
| dansmith | artom: aight | 19:05 |
| opendevreview | Merged openstack/devstack stable/2023.2: [stable only] Make devstack-platform-rocky-blue-onyx non voting https://review.opendev.org/c/openstack/devstack/+/897631 | 19:06 |
| gmann | frickler: kopecmartin: base changes on jobs are merged now, can either of you want to approve these https://review.opendev.org/q/topic:qa-2023-2-release+status:open | 19:10 |
| kopecmartin | gmann: sure, done | 19:11 |
| gmann | thanks | 19:11 |
| opendevreview | Dan Smith proposed openstack/whitebox-tempest-plugin master: Test nova's cpu host state management https://review.opendev.org/c/openstack/whitebox-tempest-plugin/+/897579 | 19:14 |
| opendevreview | Dan Smith proposed openstack/whitebox-tempest-plugin master: Fix some dedicated CPU assumptions https://review.opendev.org/c/openstack/whitebox-tempest-plugin/+/897868 | 19:14 |
| opendevreview | Dan Smith proposed openstack/whitebox-tempest-plugin master: Fix some dedicated CPU assumptions https://review.opendev.org/c/openstack/whitebox-tempest-plugin/+/897868 | 20:11 |
| opendevreview | Dan Smith proposed openstack/whitebox-tempest-plugin master: Test nova's cpu host state management https://review.opendev.org/c/openstack/whitebox-tempest-plugin/+/897579 | 20:11 |
| opendevreview | Merged openstack/devstack stable/2023.2: Update branches for stable/2023.2 https://review.opendev.org/c/openstack/devstack/+/896863 | 21:19 |
| dansmith | artom: https://7135d89174e90d2f7667-4e27fa9e0f4edf99cfff42f0877124d7.ssl.cf2.rackcdn.com/897868/2/check/whitebox-devstack-multinode/c057a13/testr_results.html | 21:22 |
| dansmith | I guess that's not failing as expected because I lowered the dedicated-per-node, | 21:23 |
| dansmith | but the number matches the size of the dedicated set on each, so I'm not sure why | 21:23 |
| dansmith | ah, it's not getting set properly: | 21:24 |
| dansmith | compute.cpu_dedicated_set = 0-3 | 21:24 |
| dansmith | compute.cpu_dedicated_set = 4-7 | 21:24 |
| dansmith | hrm | 21:25 |
| dansmith | 2023-10-10 20:29:37.541 | + inc/meta-config:merge_config_file:164 : eval 'iniset $NOVA_CONF compute cpu_dedicated_set "0-3"' | 21:25 |
| dansmith | oh, duh | 21:26 |
| dansmith | okay yeah, I need to obviously trim the two sets down in the same patch where I drop that number | 21:28 |
| dansmith | that's why this all passed on the second one | 21:28 |
| opendevreview | Dan Smith proposed openstack/whitebox-tempest-plugin master: Fix some dedicated CPU assumptions https://review.opendev.org/c/openstack/whitebox-tempest-plugin/+/897868 | 21:28 |
| opendevreview | Dan Smith proposed openstack/whitebox-tempest-plugin master: Test nova's cpu host state management https://review.opendev.org/c/openstack/whitebox-tempest-plugin/+/897579 | 21:28 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!