| gmann | kopecmartin: these are ready to end support for stable/xena https://review.opendev.org/q/topic:xena-last | 03:39 |
|---|---|---|
| opendevreview | Rajat Dhasmana proposed openstack/tempest master: Add test to rebuild volume backed instance https://review.opendev.org/c/openstack/tempest/+/831018 | 10:32 |
| opendevreview | James Parker proposed openstack/whitebox-tempest-plugin master: Add socket affinity testing https://review.opendev.org/c/openstack/whitebox-tempest-plugin/+/851447 | 13:32 |
| opendevreview | James Parker proposed openstack/whitebox-tempest-plugin master: Add socket affinity testing https://review.opendev.org/c/openstack/whitebox-tempest-plugin/+/851447 | 13:36 |
| opendevreview | James Parker proposed openstack/whitebox-tempest-plugin master: Add socket affinity testing https://review.opendev.org/c/openstack/whitebox-tempest-plugin/+/851447 | 13:38 |
| opendevreview | James Parker proposed openstack/whitebox-tempest-plugin master: Add socket affinity testing https://review.opendev.org/c/openstack/whitebox-tempest-plugin/+/851447 | 13:42 |
| kopecmartin | #startmeeting qa | 15:00 |
| opendevmeet | Meeting started Tue Jun 6 15:00:16 2023 UTC and is due to finish in 60 minutes. The chair is kopecmartin. Information about MeetBot at http://wiki.debian.org/MeetBot. | 15:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 15:00 |
| opendevmeet | The meeting name has been set to 'qa' | 15:00 |
| kopecmartin | #link https://wiki.openstack.org/wiki/Meetings/QATeamMeeting#Agenda_for_next_Office_hours | 15:00 |
| kopecmartin | agenda ^ | 15:00 |
| frickler | \o | 15:00 |
| kopecmartin | o/ | 15:00 |
| lpiwowar | o/ | 15:01 |
| kopecmartin | #topic Announcement and Action Item (Optional) | 15:02 |
| kopecmartin | there's gonna be the openinfra summit next week | 15:03 |
| lpiwowar | \o/ | 15:03 |
| kopecmartin | i'm gonna cancel our regular office hour on June 13th | 15:03 |
| kopecmartin | one other announcement | 15:04 |
| kopecmartin | we're about to release xena-last tempest release | 15:04 |
| kopecmartin | #link https://review.opendev.org/q/topic:xena-last | 15:04 |
| kopecmartin | #topic Bobcat Priority Items progress | 15:05 |
| kopecmartin | #link https://etherpad.opendev.org/p/qa-bobcat-priority | 15:05 |
| * frickler didn't get to check the venv failures | 15:05 | |
| frickler | I also wanr to bump cirros, they made new releases with updated kernels | 15:07 |
| kopecmartin | me neither :/ | 15:08 |
| kopecmartin | yeah, sounds good, we could at least propose the change and see how many things will get broken | 15:08 |
| frickler | I hope not too many, unless canonical has introduced regressions into the kernel | 15:09 |
| kopecmartin | yup , we'll see | 15:09 |
| kopecmartin | #topic OpenStack Events Updates and Planning | 15:10 |
| kopecmartin | anyone else planning to be in Vancouver except me, gmann and lpiwowar? | 15:10 |
| kopecmartin | the link for the etherpad, if there are any topics to be discussed in person: | 15:11 |
| kopecmartin | #link https://etherpad.opendev.org/p/vancouver-june2023-qa | 15:11 |
| kopecmartin | moving on | 15:13 |
| kopecmartin | #topic Gate Status Checks | 15:13 |
| kopecmartin | #link https://review.opendev.org/q/label:Review-Priority%253D%252B2+status:open+(project:openstack/tempest+OR+project:openstack/patrole+OR+project:openstack/devstack+OR+project:openstack/grenade) | 15:13 |
| kopecmartin | nothing there | 15:14 |
| kopecmartin | any urgent patches to review? | 15:14 |
| lpiwowar | I updated the comment on this patch: https://review.opendev.org/c/openstack/tempest/+/881575 ... But I do not think it is urgent. | 15:14 |
| lpiwowar | It is the patch that fixes cleanup for the object_storate tests. | 15:14 |
| kopecmartin | thanks, yeah, right now there are couple of patches before we can merge this one (considering the current unstable behaviour of the CI) | 15:15 |
| lpiwowar | Got it:) | 15:15 |
| kopecmartin | #topic Bare rechecks | 15:16 |
| kopecmartin | #link https://etherpad.opendev.org/p/recheck-weekly-summary | 15:16 |
| kopecmartin | this hasn't been updated, so nothing new there | 15:16 |
| kopecmartin | #topic Periodic jobs Status Checks | 15:16 |
| kopecmartin | periodic stable full | 15:16 |
| kopecmartin | #link https://zuul.openstack.org/builds?pipeline=periodic-stable&job_name=tempest-full-yoga&job_name=tempest-full-xena&job_name=tempest-full-zed&job_name=tempest-full-2023-1 | 15:16 |
| kopecmartin | periodic stable slow | 15:16 |
| kopecmartin | #link https://zuul.openstack.org/builds?job_name=tempest-slow-2023-1&job_name=tempest-slow-zed&job_name=tempest-slow-yoga&job_name=tempest-slow-xena | 15:16 |
| kopecmartin | periodic extra tests | 15:16 |
| kopecmartin | #link https://zuul.openstack.org/builds?job_name=tempest-full-2023-1-extra-tests&job_name=tempest-full-zed-extra-tests&job_name=tempest-full-yoga-extra-tests&job_name=tempest-full-xena-extra-tests | 15:16 |
| kopecmartin | periodic master | 15:16 |
| kopecmartin | #link https://zuul.openstack.org/builds?project=openstack%2Ftempest&project=openstack%2Fdevstack&pipeline=periodic | 15:16 |
| kopecmartin | tempest-slow-2023-1 timeouts in ~50% of all the runs :/ | 15:17 |
| frickler | could these be related to mysql slowdown? | 15:19 |
| kopecmartin | maybe, it took 50 minutes to deploy devstack | 15:19 |
| kopecmartin | #link https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_6d0/periodic-stable/opendev.org/openstack/tempest/master/tempest-slow-2023-1/6d0d1c2/job-output.txt | 15:19 |
| kopecmartin | *40 | 15:20 |
| kopecmartin | and the tests took 2 hours, what's strange is that tempest was executed with concurrency=2 but i see only one thread running tests o.O | 15:21 |
| frickler | tempest run --serial --regex '\[.*\bslow\b.*\]' --concurrency=2 | 15:22 |
| kopecmartin | in a passing build, the installation of devstack takes only 30 min | 15:22 |
| frickler | --serial | 15:22 |
| kopecmartin | oh, yeah, so why is --concurrency=2 there? | 15:22 |
| frickler | to confuse casual reviewers? ;) | 15:23 |
| kopecmartin | :D for sure | 15:23 |
| frickler | maybe we need to split this into two jobs, half the tests each | 15:23 |
| frickler | or some day rekindle the discussion with opendev about flavor memory | 15:24 |
| kopecmartin | right! --serial and --parallel decide whether the --concurrency value will be used , if --serial is passed, the concurrency is ignored ... | 15:25 |
| kopecmartin | what's ironic about that is that slow job was created to avoid timeouts :D | 15:25 |
| kopecmartin | ... or try to run it in parallel? | 15:26 |
| kopecmartin | although it might timeout too but from different reasons | 15:26 |
| frickler | we could give it a try, but I fear that it will trigger oom issues | 15:26 |
| kopecmartin | gmann: what do you think? | 15:27 |
| kopecmartin | i can propose a patch to run the tests in parallel, just as an experiment | 15:28 |
| kopecmartin | the other jobs look good, not sure why centos9-stream fails again though | 15:28 |
| kopecmartin | but it's just started today, maybe it won't repeat | 15:28 |
| kopecmartin | and the tempest-full-test-account-no-admin-py3 job will be fixed once lpiwowar's patch is merged (was already linked above) | 15:29 |
| lpiwowar | kopecmartin: +1 | 15:29 |
| kopecmartin | #topic Distros check | 15:29 |
| kopecmartin | cs-9 | 15:29 |
| kopecmartin | #link https://zuul.openstack.org/builds?job_name=tempest-full-centos-9-stream&job_name=devstack-platform-centos-9-stream&skip=0 | 15:29 |
| kopecmartin | fedora | 15:29 |
| kopecmartin | #link https://zuul.openstack.org/builds?job_name=devstack-platform-fedora-latest&skip=0 | 15:29 |
| kopecmartin | debian | 15:30 |
| kopecmartin | #link https://zuul.openstack.org/builds?job_name=devstack-platform-debian-bullseye&skip=0 | 15:30 |
| kopecmartin | focal | 15:30 |
| kopecmartin | #link https://zuul.opendev.org/t/openstack/builds?job_name=devstack-platform-ubuntu-focal&skip=0 | 15:30 |
| kopecmartin | rocky | 15:30 |
| kopecmartin | #link https://zuul.openstack.org/builds?job_name=devstack-platform-rocky-blue-onyx | 15:30 |
| kopecmartin | openEuler | 15:30 |
| kopecmartin | #link https://zuul.openstack.org/builds?job_name=devstack-platform-openEuler-22.03-ovn-source&job_name=devstack-platform-openEuler-22.03-ovs&skip=0 | 15:30 |
| kopecmartin | we're about to delete the devstack-platform-ubuntu-focal job | 15:30 |
| frickler | did we talk about dropping fedora already? | 15:30 |
| kopecmartin | #link https://review.opendev.org/c/openstack/tempest/+/884952 | 15:30 |
| kopecmartin | once the CI agrees and finally merges that | 15:31 |
| kopecmartin | i think so | 15:31 |
| kopecmartin | let me try to find it | 15:31 |
| frickler | hmm, I can force merge the above, got some other forcing to do anyway | 15:31 |
| frickler | and since it is job removal only, no need to wait for random jobs passing | 15:31 |
| kopecmartin | that be great frickler , thanks | 15:32 |
| kopecmartin | found it | 15:33 |
| kopecmartin | #link https://meetings.opendev.org/meetings/qa/2023/qa.2023-05-23-15.00.log.html#l-85 | 15:33 |
| frickler | can we also delete the devstack focal platform job as followup? | 15:33 |
| kopecmartin | we talked about fedora couple weeks ago | 15:33 |
| kopecmartin | hm, one moment | 15:33 |
| kopecmartin | seems so | 15:34 |
| kopecmartin | it's not used anywhere else | 15:34 |
| kopecmartin | #lin https://codesearch.opendev.org/?q=devstack-platform-ubuntu-focal&i=nope&literal=nope&files=&excludeFiles=&repos= | 15:35 |
| frickler | oh, so I said I'd do fedora removal patches. guess I need to do that, then :D | 15:35 |
| kopecmartin | :D loop me in the reviews please | 15:36 |
| frickler | guess I'll combine focal + fedora | 15:37 |
| frickler | for the alliteration fun + to avoid conflicts | 15:37 |
| kopecmartin | sounds good | 15:38 |
| kopecmartin | #topic Sub Teams highlights | 15:38 |
| kopecmartin | Changes with Review-Priority == +1 | 15:38 |
| kopecmartin | #link https://review.opendev.org/q/label:Review-Priority%253D%252B1+status:open+(project:openstack/tempest+OR+project:openstack/patrole+OR+project:openstack/devstack+OR+project:openstack/grenade) | 15:38 |
| kopecmartin | we have there the 3 patches to tag tempest and this one | 15:39 |
| kopecmartin | #link https://review.opendev.org/c/openstack/tempest/+/884764 | 15:39 |
| kopecmartin | i'm on those | 15:40 |
| kopecmartin | one strange thing in https://review.opendev.org/c/openstack/tempest/+/885089 gmann, otherwise good to go | 15:41 |
| kopecmartin | #topic Open Discussion | 15:41 |
| kopecmartin | anything for the open discussion? | 15:41 |
| frickler | the GIT_DEPTH thing? | 15:41 |
| frickler | at first I was wondering whether we should revert the patch until we have a solution. but it also got backported already | 15:42 |
| kopecmartin | #link https://bugs.launchpad.net/devstack/+bug/2023020 | 15:43 |
| kopecmartin | i've reported that so that we have it tracked | 15:43 |
| kopecmartin | it's not that urgent | 15:43 |
| kopecmartin | the bug we fixed by that patch was a bit more pressing and was blocking some jobs | 15:43 |
| kopecmartin | anyway, it's a bug and we need to find a solution working with depth too | 15:44 |
| frickler | ok | 15:46 |
| kopecmartin | #topic Bug Triage | 15:47 |
| kopecmartin | #link https://etherpad.openstack.org/p/qa-bug-triage-bobcat | 15:47 |
| kopecmartin | this is basically all from my side | 15:48 |
| kopecmartin | thank you everyone | 15:49 |
| kopecmartin | see you in Vancouver or in 2 weeks online :) | 15:49 |
| kopecmartin | #endmeeting | 15:49 |
| opendevmeet | Meeting ended Tue Jun 6 15:49:51 2023 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 15:49 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/qa/2023/qa.2023-06-06-15.00.html | 15:49 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/qa/2023/qa.2023-06-06-15.00.txt | 15:49 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/qa/2023/qa.2023-06-06-15.00.log.html | 15:49 |
| lpiwowar | o/ | 15:50 |
| frickler | o/ | 15:50 |
| opendevreview | Merged openstack/tempest master: Remove ubuntu focal job https://review.opendev.org/c/openstack/tempest/+/884952 | 15:54 |
| rpittau | gmann, hi! similar to the timeout issues mentioned by dansmith on Friday, we're seeing an increase of timeout in the ironic unit tests since today | 16:00 |
| dansmith | rpittau: my timeouts were tempest tests and sort of a different reason it sounds like | 16:00 |
| dansmith | i.e. no deadlocks, but some lack of parallelism which is inherent in how tempest splits up its tests | 16:01 |
| rpittau | yeah, I noticed that, just wondering if there is something similar happening since apparently some workers got way fewer tests than others | 16:01 |
| dansmith | ack, well, testr is going to split tests evenly by lexical sort AFAIK, unless you control the splitting with the testr.conf (see the one in tempest's repo and the docs) | 16:02 |
| rpittau | and unfortunately it looks like they get stuck in random places | 16:02 |
| dansmith | however, if lexical ordering ends up not being equal work, then that will happen | 16:02 |
| dansmith | (i.e. if your A tests take less time than your Z tests) | 16:03 |
| dansmith | but if they're getting stuck, you need to fix that first for sure | 16:03 |
| rpittau | heh yep :) | 16:04 |
| rpittau | I'm troubleshooting in the meantime, just wanted to see if maybe there was something known going on | 16:04 |
| opendevreview | Martin Kopec proposed openstack/tempest master: Incrase timeout in tempest-slow-2023-1 job https://review.opendev.org/c/openstack/tempest/+/885370 | 16:05 |
| opendevreview | Gorka Eguileor proposed openstack/devstack-plugin-ceph master: Add service secret to libvirt when cinder enabled https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/885374 | 16:38 |
| gmann | kopecmartin: which job you mentioned to split the test and run in serial ? for many jobs I have separated the tests except slow job i think | 17:05 |
| gmann | rpittau: you can group the test to run under same worker if you know these tests are always fast otherwise it can lead to more worst situation | 17:06 |
| kopecmartin | gmann: tempest-slow-2023-1 - it's been timeouting recently https://zuul.openstack.org/builds?job_name=tempest-slow-2023-1&job_name=tempest-slow-zed&job_name=tempest-slow-yoga&job_name=tempest-slow-xena | 19:05 |
| gmann | kopecmartin: ohk but it is same as tempest-slow job also where I saw many timeout recently | 19:06 |
| kopecmartin | oh, right, i was concerned only about the periodic jobs and didn't realize that this one inherits from tempest-slow which votes on changes .. | 19:14 |
| gmann | yeah, I have one action item on this from our PTG discussion on how to make tempest-slow job more shorter or separate tests. but I have not spend time on that. | 19:18 |
| gmann | will try to check if anything we can do on that | 19:18 |
| opendevreview | Merged openstack/whitebox-tempest-plugin master: Add socket affinity testing https://review.opendev.org/c/openstack/whitebox-tempest-plugin/+/851447 | 19:31 |
| opendevreview | Ghanshyam proposed openstack/tempest master: End support for stable/xena https://review.opendev.org/c/openstack/tempest/+/885089 | 20:37 |
| opendevreview | Ghanshyam proposed openstack/tempest master: Pin stable/2023.1 constraint in tox to release 35.0.0 https://review.opendev.org/c/openstack/tempest/+/885090 | 20:37 |
| opendevreview | Ghanshyam proposed openstack/tempest master: Switch back the tox constraint to master https://review.opendev.org/c/openstack/tempest/+/885091 | 20:37 |
| opendevreview | James Parker proposed openstack/whitebox-tempest-plugin master: Convert found vlan to int https://review.opendev.org/c/openstack/whitebox-tempest-plugin/+/885413 | 21:13 |
| opendevreview | Merged openstack/tempest master: Remove closure _unshelve_server not useful https://review.opendev.org/c/openstack/tempest/+/882823 | 21:18 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!