| *** elodilles_pto is now known as elodilles | 06:35 | |
| opendevreview | Konrad Gube proposed openstack/tempest master: Add tests for the os-extend_volume_completion volume action. https://review.opendev.org/c/openstack/tempest/+/890183 | 09:02 |
|---|---|---|
| *** dmellado17042088 is now known as dmellado | 09:51 | |
| opendevreview | Katarina Strenkova proposed openstack/devstack-plugin-open-cas master: Replacedeprecated terms https://review.opendev.org/c/openstack/devstack-plugin-open-cas/+/890197 | 12:20 |
| kopecmartin | #startmeeting qa | 15:00 |
| opendevmeet | Meeting started Tue Aug 1 15:00:23 2023 UTC and is due to finish in 60 minutes. The chair is kopecmartin. Information about MeetBot at http://wiki.debian.org/MeetBot. | 15:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 15:00 |
| opendevmeet | The meeting name has been set to 'qa' | 15:00 |
| kopecmartin | #link https://wiki.openstack.org/wiki/Meetings/QATeamMeeting#Agenda_for_next_Office_hours | 15:00 |
| kopecmartin | agenda & | 15:00 |
| kopecmartin | ^^ | 15:00 |
| lpiwowar | o/ | 15:01 |
| kopecmartin | o/ | 15:02 |
| frickler | \o | 15:04 |
| kopecmartin | o/ | 15:04 |
| kopecmartin | let's get to it | 15:04 |
| kopecmartin | #topic Announcement and Action Item (Optional) | 15:05 |
| kopecmartin | btw, i've signed up the qa team for the upcoming virtual ptg | 15:05 |
| kopecmartin | and i've registered myself for the event as well | 15:05 |
| kopecmartin | #link https://ptg2023.openinfra.dev/ | 15:06 |
| kopecmartin | that's it regarding the announcements | 15:06 |
| kopecmartin | #topic Bobcat Priority Items progress | 15:06 |
| kopecmartin | #link https://etherpad.opendev.org/p/qa-bobcat-priority | 15:06 |
| kopecmartin | anything new on this front? | 15:06 |
| * kopecmartin checking the doc | 15:06 | |
| frickler | no further reviews for global venv :( | 15:07 |
| kopecmartin | #link https://review.opendev.org/c/openstack/devstack/+/558930 | 15:08 |
| kopecmartin | gmann: dansmith ^^ what do you think? | 15:08 |
| dansmith | about the venv thing? | 15:08 |
| dansmith | Ihaven | 15:09 |
| kopecmartin | yes, the patch looks ready (well, from what i can tell) | 15:09 |
| dansmith | I haven't looked at that in a long time so I'd need to look again.. I'm bummed we have to do this, but I know something has to change | 15:09 |
| dansmith | I guess I'll have a look, hopefully today | 15:09 |
| frickler | dansmith: you may also want to look at the wheel build issue mentioned in https://review.opendev.org/c/openstack/devstack/+/887547/5/.zuul.yaml | 15:10 |
| dansmith | frickler: I don't see any issue mentioned there... | 15:10 |
| frickler | # TODO(frickler): drop this once wheel build is fixed - MYSQL_GATHER_PERFORMANCE: false | 15:11 |
| frickler | L373N | 15:11 |
| dansmith | 737? | 15:12 |
| frickler | 737, yes, sorry | 15:12 |
| dansmith | but ack, I understand we need to do something for this and other reasons | 15:12 |
| dansmith | I'll pull this down and play with it while I wait for rechecks to need rechecking | 15:13 |
| kopecmartin | sounds good, thanks | 15:15 |
| kopecmartin | let's get moving | 15:15 |
| kopecmartin | #topic Gate Status Checks | 15:15 |
| kopecmartin | #link https://review.opendev.org/q/label:Review-Priority%253D%252B2+status:open+(project:openstack/tempest+OR+project:openstack/patrole+OR+project:openstack/devstack+OR+project:openstack/grenade) | 15:15 |
| kopecmartin | one patch, waiting for the CI | 15:15 |
| kopecmartin | (again) :D | 15:15 |
| kopecmartin | more like still | 15:16 |
| dansmith | that failed on one thing I've seen lately, | 15:16 |
| dansmith | which is the osc functional test failing with things that look like maybe apache running out of connections or otherwise having trouble connecting to backends | 15:16 |
| dansmith | I've seen that a handful of times this week | 15:16 |
| dansmith | I dunno really anything about that job, so I'm not sure why that might be happening | 15:17 |
| frickler | related to https://review.opendev.org/c/openstack/openstacksdk/+/888240 maybe? | 15:17 |
| kopecmartin | maybe, don't know either, the job looks quite stable | 15:17 |
| dansmith | I dunno, I've seen it with things other than placement | 15:17 |
| kopecmartin | #link https://zuul.opendev.org/t/openstack/builds?job_name=openstacksdk-functional-devstack&project=openstack/devstack | 15:17 |
| kopecmartin | in comparison to others at least | 15:17 |
| frickler | seems the same failure was in the original patch https://zuul.opendev.org/t/openstack/build/ae68c91aa4994e499bbbdfc10af0f576 | 15:18 |
| dansmith | ack, I've just seen failures on glance and devstack with it, not always with placement | 15:18 |
| dansmith | and I'm not sure I've ever seen it fail before that | 15:18 |
| frickler | I can ping stephenfin over in sdks to have a look | 15:19 |
| dansmith | kopecmartin: looking at it this way makes it look worse: https://zuul.opendev.org/t/openstack/builds?job_name=openstacksdk-functional-devstack&result=FAILURE&skip=0 | 15:20 |
| kopecmartin | :o | 15:20 |
| dansmith | perhaps it's just general instability, I just never see it fail and have seen it multiple times lately, so figured something must be up | 15:21 |
| dansmith | is that affected by the increased concurrency change/ | 15:21 |
| frickler | that at least makes it look like it didn't start yesterday | 15:22 |
| dansmith | yeah | 15:22 |
| kopecmartin | re: concurrency, would it hit just one test? | 15:22 |
| kopecmartin | but hard to tell, in openstack everything is connected in some way :D | 15:23 |
| dansmith | yeah | 15:23 |
| dansmith | anyway, it just struck me because I hadn't seen that fail on my stuff until recently | 15:23 |
| kopecmartin | right, let's keep an eye on that | 15:25 |
| kopecmartin | #topic Bare rechecks | 15:25 |
| kopecmartin | #link https://etherpad.opendev.org/p/recheck-weekly-summary | 15:25 |
| kopecmartin | considering all the rechecks we're quite ok | 15:26 |
| kopecmartin | interesting fact though | 15:26 |
| dansmith | even though there's a little cheating going on | 15:26 |
| kopecmartin | QA had the most rechecks over the last 90 days ... 480 | 15:26 |
| dansmith | *cough*kopecmartin*cough* :) | 15:26 |
| kopecmartin | :D yeah, i know , guilty as charged | 15:26 |
| dansmith | I'm just kidding, it's pretty hard to keep caring on recheck 934 | 15:27 |
| kopecmartin | plus when we can see, it's always a different job | 15:28 |
| kopecmartin | which fails | 15:28 |
| kopecmartin | #topic Periodic jobs Status Checks | 15:29 |
| kopecmartin | periodic stable full | 15:29 |
| kopecmartin | #link https://zuul.openstack.org/builds?pipeline=periodic-stable&job_name=tempest-full-yoga&job_name=tempest-full-xena&job_name=tempest-full-zed&job_name=tempest-full-2023-1 | 15:29 |
| kopecmartin | periodic stable slow | 15:29 |
| kopecmartin | #link https://zuul.openstack.org/builds?job_name=tempest-slow-2023-1&job_name=tempest-slow-zed&job_name=tempest-slow-yoga&job_name=tempest-slow-xena | 15:29 |
| kopecmartin | periodic extra tests | 15:29 |
| kopecmartin | #link https://zuul.openstack.org/builds?job_name=tempest-full-2023-1-extra-tests&job_name=tempest-full-zed-extra-tests&job_name=tempest-full-yoga-extra-tests&job_name=tempest-full-xena-extra-tests | 15:29 |
| kopecmartin | periodic master | 15:29 |
| kopecmartin | #link https://zuul.openstack.org/builds?project=openstack%2Ftempest&project=openstack%2Fdevstack&pipeline=periodic | 15:29 |
| kopecmartin | tempest-full-parallel fails more often, but with different tests, no clear pattern | 15:32 |
| kopecmartin | #topic Distros check | 15:32 |
| kopecmartin | cs-9 | 15:32 |
| kopecmartin | #link https://zuul.openstack.org/builds?job_name=tempest-full-centos-9-stream&job_name=devstack-platform-centos-9-stream&skip=0 | 15:32 |
| kopecmartin | fedora | 15:32 |
| kopecmartin | #link https://zuul.openstack.org/builds?job_name=devstack-platform-fedora-latest&skip=0 | 15:32 |
| kopecmartin | debian | 15:32 |
| kopecmartin | #link https://zuul.openstack.org/builds?job_name=devstack-platform-debian-bullseye&skip=0 | 15:32 |
| kopecmartin | focal | 15:32 |
| kopecmartin | #link https://zuul.opendev.org/t/openstack/builds?job_name=devstack-platform-ubuntu-focal&skip=0 | 15:32 |
| kopecmartin | rocky | 15:32 |
| kopecmartin | #link https://zuul.openstack.org/builds?job_name=devstack-platform-rocky-blue-onyx | 15:33 |
| kopecmartin | openEuler | 15:33 |
| kopecmartin | #link https://zuul.openstack.org/builds?job_name=devstack-platform-openEuler-22.03-ovn-source&job_name=devstack-platform-openEuler-22.03-ovs&skip=0 | 15:33 |
| kopecmartin | fedora started failing constantly, but we're about to drop that (more about that in a minute | 15:33 |
| kopecmartin | tempest-full-centos-9-stream has been failing last 2 days | 15:33 |
| opendevreview | yatin proposed openstack/devstack stable/2023.1: Use RDO official CloudSIG mirrors for C9S deployments https://review.opendev.org/c/openstack/devstack/+/890120 | 15:34 |
| kopecmartin | Failed to discover available identity versions when contacting http://localhost:5000/v3/. Attempting to parse version from URL | 15:35 |
| kopecmartin | and it ends with Bad Request | 15:35 |
| opendevreview | yatin proposed openstack/devstack stable/zed: Use RDO official CloudSIG mirrors for C9S deployments https://review.opendev.org/c/openstack/devstack/+/890221 | 15:35 |
| opendevreview | yatin proposed openstack/devstack stable/yoga: Use RDO official CloudSIG mirrors for C9S deployments https://review.opendev.org/c/openstack/devstack/+/890222 | 15:35 |
| kopecmartin | that ^^ might help actually :D | 15:36 |
| kopecmartin | perfect timing | 15:36 |
| kopecmartin | #topic Sub Teams highlights | 15:36 |
| kopecmartin | Changes with Review-Priority == +1 | 15:36 |
| kopecmartin | #link https://review.opendev.org/q/label:Review-Priority%253D%252B1+status:open+(project:openstack/tempest+OR+project:openstack/patrole+OR+project:openstack/devstack+OR+project:openstack/grenade) | 15:36 |
| kopecmartin | seems like we're ready to drop fedora | 15:37 |
| kopecmartin | #link https://review.opendev.org/c/openstack/devstack/+/885467 | 15:37 |
| kopecmartin | patches for all the consumers were proposed more than a month ago | 15:37 |
| kopecmartin | #link https://review.opendev.org/q/topic:drop-old-distros | 15:37 |
| kopecmartin | frickler: if you agree, i can +w it now | 15:38 |
| frickler | ack | 15:38 |
| kopecmartin | thanks, done | 15:39 |
| kopecmartin | #topic Open Discussion | 15:39 |
| kopecmartin | anything for the open discussion? | 15:39 |
| lpiwowar | I just have a quick question whether it would be ok to add option to tempest.conf to indicates which backup driver cinder uses. Currently the volume backup tests do not clean up swift containers. This causes failure of object storage tests with pre-prov creds. We could use this option to clean up this container when swift is used as a backup | 15:39 |
| lpiwowar | driver. | 15:39 |
| lpiwowar | #link https://bugs.launchpad.net/tempest/+bug/2028671 | 15:40 |
| kopecmartin | it's just strange that we would need an option for the cleanup of a test and not for the test itself | 15:40 |
| dansmith | agree | 15:41 |
| lpiwowar | Example of the error: | 15:41 |
| lpiwowar | #link https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_8a2/periodic/opendev.org/openstack/tempest/master/tempest-full-test-account-no-admin-py3/8a2ef81/testr_results.html | 15:41 |
| kopecmartin | what about a try except block .. attempt to delete everything except any errors, then move on and hope for the best | 15:42 |
| lpiwowar | I understand but without the option it is probably hard to tell whether swift is used as a back up driver (?). Maybe I can try to delete the containers if swift service is present. But the presence of swift service does not guarantee that it is used as a backup driver? I'm not sure. | 15:42 |
| frickler | maybe that swift test is bogus really? can't always assume a clean environment with no active containers | 15:43 |
| kopecmartin | hmm, the test could check the content of the container at the beginning and check again at the end .. as far as the test is concerned there shouldn't be any additional resources | 15:44 |
| kopecmartin | doesn't matter what was there before | 15:44 |
| dansmith | that still won't make it stable | 15:44 |
| frickler | unless it runs in parallel with the backup test | 15:44 |
| dansmith | because it could be running in parallel where more things are being created/destroyed | 15:44 |
| dansmith | it really needs to be able to look at only the things it cares about and ignore anything else | 15:44 |
| dansmith | I'm surprised tenant isolation doesn't just solve it though... | 15:45 |
| kopecmartin | oh, right | 15:45 |
| lpiwowar | dansmith: the issue only emerges when pre-prov creds are used. | 15:45 |
| dansmith | yeah exactly :) | 15:45 |
| dansmith | so the test is broken | 15:45 |
| frickler | then the test needs to be skipped in that scenario | 15:45 |
| lpiwowar | So when the test reuses a user which was used for volume back up testing ... The test fails. That is my guess. | 15:46 |
| dansmith | frickler++ | 15:47 |
| lpiwowar | frickler: So you think skipping the test when pre-prov creds are used? | 15:47 |
| frickler | yes | 15:47 |
| lpiwowar | frickler: Ok, I will propose a patch:). Maybe we can continue the discussion there. | 15:48 |
| kopecmartin | ach, the test is part of interop, which runs mostly with preprovisioned creds .. that means we'll need to kick it out | 15:48 |
| kopecmartin | lpiwowar: agree, let's start with a patch and continue there | 15:48 |
| lpiwowar | kopecmartin: That is a good point. I wasn't thinking about it. | 15:48 |
| kopecmartin | hm, if the issue really is that the test reuses a user which has left some resources behind, in that case my proposal could work - if preprov creds, save the initial content of the container and compare at the end of the test - the parallel issue whouldn't be a problem as the user is used by this test at that time | 15:50 |
| kopecmartin | unless other tests could use that user too | 15:50 |
| dansmith | that's the point though, other tests use them | 15:51 |
| dansmith | when cinder uses swift for backup | 15:51 |
| dansmith | any test that asserts you start with a blank slate doesn't make sense with shared creds, IMHO | 15:51 |
| kopecmartin | right, i hoped there more clever logic .. then skipping the test is the easiest way out of it | 15:51 |
| lpiwowar | kopecmartin: Are volume backup tests part of interop? I know it is not a nice solution but as long as volume backup tests are not in there then it should be ok. | 15:52 |
| dansmith | lpiwowar: what if I have other swift containers present before I run even with the backup tests disabled/ | 15:53 |
| dansmith | surely glance with swift would have the same problem? | 15:53 |
| lpiwowar | dansmith: Not tested. But if the containers are visible for the user in the accounts.yaml then it would probably fail. | 15:54 |
| dansmith | yeah, which will definitely happen (a lot) with glance being backed by swift, as any snapshot and potentially the test image itself would be visible | 15:54 |
| lpiwowar | Ok, from what I'm reading it seems that the issue is really on the side of the object storage tests. I thought that the fix would be required for the volume back up tests. | 15:55 |
| lpiwowar | So we have to either fix the object storage test(s) or skip them when pre-prov creds used. | 15:56 |
| kopecmartin | correct | 15:57 |
| lpiwowar | I will take a look at it more and will propose a patch:) | 15:57 |
| kopecmartin | thanks | 15:57 |
| kopecmartin | let's move on | 15:57 |
| kopecmartin | #topic Bug Triage | 15:57 |
| kopecmartin | #link https://etherpad.openstack.org/p/qa-bug-triage-bobcat | 15:57 |
| kopecmartin | numbers recorded ^ | 15:57 |
| kopecmartin | but htat's all i had time for unfortunately | 15:58 |
| kopecmartin | that's it | 15:58 |
| kopecmartin | thanks everyone | 15:58 |
| kopecmartin | #endmeeting | 15:58 |
| opendevmeet | Meeting ended Tue Aug 1 15:58:54 2023 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 15:58 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/qa/2023/qa.2023-08-01-15.00.html | 15:58 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/qa/2023/qa.2023-08-01-15.00.txt | 15:58 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/qa/2023/qa.2023-08-01-15.00.log.html | 15:58 |
| lpiwowar | thanks kopecmartin | 15:58 |
| frickler | thx | 15:59 |
| gmann | frickler: can you break the deps (currently it is dependent on focal-removal) here so that we can merge this https://review.opendev.org/c/openstack/devstack/+/885469/1 | 17:09 |
| dansmith | the dbcounter patch is about to merge | 17:11 |
| gmann | yeah, waiting for that | 17:12 |
| dansmith | in all the runs that include it I've seen much better io behavior, so I'm hopeful | 17:12 |
| opendevreview | Merged openstack/devstack master: Reduce the flush frequency of dbcounter plugin https://review.opendev.org/c/openstack/devstack/+/890136 | 17:13 |
| dansmith | \o/ | 17:13 |
| * dansmith starts more rechecks | 17:13 | |
| gmann | cool | 17:15 |
| gmann | kopecmartin: yeah there are lot of recheck but mostly on the gate fixes. hopefully we will have some improvement once they all merge | 17:26 |
| dansmith | gmann: any idea what this is? https://b5afffe3af714f1125c3-25f3a3879427e395d97921f59eb3db3c.ssl.cf2.rackcdn.com/882316/1/gate/nova-next/3a53c36/testr_results.html | 17:35 |
| dansmith | two tests failed with clashing networks.. they should be isolated... | 17:35 |
| dansmith | slaweq: ^ | 17:35 |
| dansmith | and indeed they happened within about 10s of each other, so I imagine they did clash | 17:42 |
| opendevreview | Merged openstack/devstack master: Drop Fedora support https://review.opendev.org/c/openstack/devstack/+/885467 | 17:49 |
| gmann | dansmith: yeah it should be isolated network for both test class | 17:52 |
| dansmith | gmann: so that's weird right? | 17:53 |
| gmann | dansmith: they are not creating the default network so there are chances of finding multiple network. let me update those test to create default and that way at least we pass the created network to server request | 17:54 |
| dansmith | ack | 17:54 |
| opendevreview | Ghanshyam Mann proposed openstack/tempest master: create default network for few more server tests https://review.opendev.org/c/openstack/tempest/+/890242 | 18:46 |
| frickler | meh, also need to rebase 558930 on top of fedora drop | 19:26 |
| opendevreview | Dr. Jens Harbott proposed openstack/devstack master: Add option to install everything in global venvs https://review.opendev.org/c/openstack/devstack/+/558930 | 19:29 |
| opendevreview | Dr. Jens Harbott proposed openstack/devstack master: Add debian-bookworm job https://review.opendev.org/c/openstack/devstack/+/887547 | 19:33 |
| dansmith | frickler: did you see my comments/questions? | 19:34 |
| frickler | dansmith: nope, sorry, just noticed the merge conflict, will take a look tomorrow | 19:46 |
| dansmith | frickler: ack | 20:20 |
| opendevreview | Merged openstack/devstack-plugin-ceph master: Update location of cephadm script https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/888952 | 21:27 |
| gmann | dansmith: multiple network issue fix https://review.opendev.org/c/openstack/tempest/+/890242 | 22:38 |
| gmann | kopecmartin: https://review.opendev.org/c/openstack/tempest/+/890242 | 22:38 |
| dansmith | gmann: sweet | 22:39 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!