| *** pojadhav is now known as pojadhav|ruck | 05:11 | |
| opendevreview | Abhishek Kekane proposed openstack/tempest master: Add image cache client https://review.opendev.org/c/openstack/tempest/+/825111 | 06:16 |
|---|---|---|
| opendevreview | Abhishek Kekane proposed openstack/tempest master: Add tempest coverage for new cache API https://review.opendev.org/c/openstack/tempest/+/825112 | 06:16 |
| opendevreview | Francesco Pantano proposed openstack/devstack-plugin-ceph master: Deploy with cephadm https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/826484 | 07:00 |
| opendevreview | Abhishek Kekane proposed openstack/tempest master: Add tempest coverage for new cache API https://review.opendev.org/c/openstack/tempest/+/825112 | 07:24 |
| *** jpena|off is now known as jpena | 07:37 | |
| *** pojadhav|ruck is now known as pojadhav|lunch | 07:44 | |
| opendevreview | jiang guodao proposed openstack/devstack master: [DNM] Fix the surrogatepass not allowed error https://review.opendev.org/c/openstack/devstack/+/846588 | 08:30 |
| *** pojadhav|lunch is now known as pojadhav|ruck | 08:38 | |
| kopecmartin | gmann: hi, can you check this when you have a sec, https://review.opendev.org/c/openstack/tempest/+/841719 | 09:00 |
| *** ykarel_ is now known as ykarel|lunch | 09:19 | |
| opendevreview | Gorka Eguileor proposed openstack/devstack master: Reduce memory consumption in Cinder services https://review.opendev.org/c/openstack/devstack/+/845805 | 10:16 |
| *** ykarel|lunch is now known as ykarel_ | 11:14 | |
| *** ralonsoh_ is now known as ralonsoh | 13:00 | |
| gmann | kopecmartin: ack | 13:54 |
| opendevreview | Abhishek Kekane proposed openstack/tempest master: Add image cache client https://review.opendev.org/c/openstack/tempest/+/825111 | 13:57 |
| opendevreview | Abhishek Kekane proposed openstack/tempest master: Add tempest coverage for new cache API https://review.opendev.org/c/openstack/tempest/+/825112 | 13:57 |
| kopecmartin | #startmeeting qa | 15:00 |
| opendevmeet | Meeting started Tue Jun 21 15:00:17 2022 UTC and is due to finish in 60 minutes. The chair is kopecmartin. Information about MeetBot at http://wiki.debian.org/MeetBot. | 15:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 15:00 |
| opendevmeet | The meeting name has been set to 'qa' | 15:00 |
| kopecmartin | #link https://wiki.openstack.org/wiki/Meetings/QATeamMeeting#Weekly_QA_Team_meeting | 15:00 |
| kopecmartin | the agenda for today ^^ | 15:00 |
| frickler | \o | 15:01 |
| kopecmartin | o/ | 15:01 |
| kopecmartin | let's go through the usual as alway | 15:03 |
| kopecmartin | always | 15:03 |
| kopecmartin | #topic Announcement and Action Item (Optional) | 15:03 |
| kopecmartin | none | 15:03 |
| kopecmartin | #topic Zed Priority Items progress | 15:03 |
| kopecmartin | #link https://etherpad.opendev.org/p/qa-zed-priority | 15:03 |
| kopecmartin | anything here? | 15:03 |
| kopecmartin | we started cleaning the tests with the unstable_test decorator | 15:04 |
| kopecmartin | we're exploring whether we can just remove it - means we're checking whether the bugs which were the reason for the decorator are still valid | 15:04 |
| kopecmartin | some of them are quite old, so the worst case scenario will be , we'll remove the decorators and see whether something is breaking :) | 15:05 |
| kopecmartin | that's from me on this topic | 15:05 |
| kopecmartin | #topic Gate Status Checks | 15:06 |
| kopecmartin | #link https://review.opendev.org/q/label:Review-Priority%253D%252B2+status:open+(project:openstack/tempest+OR+project:openstack/patrole+OR+project:openstack/devstack+OR+project:openstack/grenade) | 15:06 |
| kopecmartin | no reviews | 15:06 |
| kopecmartin | any urgent patches? | 15:06 |
| frickler | like we need one to make c9s non-voting | 15:07 |
| frickler | likely | 15:08 |
| frickler | for devstack | 15:08 |
| frickler | but we have that topic later | 15:08 |
| kopecmartin | seems the cs9 job has been failing lately .. yeah, we'll get to that | 15:09 |
| kopecmartin | #topic Periodic jobs Status Checks | 15:09 |
| kopecmartin | stable | 15:09 |
| kopecmartin | #link #link https://zuul.openstack.org/builds?job_name=tempest-full-yoga&job_name=tempest-full-xena&job_name=tempest-full-wallaby-py3&job_name=tempest-full-victoria-py3&job_name=tempest-full-ussuri-py3&pipeline=periodic-stable | 15:09 |
| kopecmartin | master | 15:09 |
| kopecmartin | #link https://zuul.openstack.org/builds?project=openstack%2Ftempest&project=openstack%2Fdevstack&pipeline=periodic | 15:09 |
| kopecmartin | devstack-no-tls-proxy failed | 15:10 |
| kopecmartin | although it failed before too | 15:10 |
| * kopecmartin checking hte logs | 15:10 | |
| kopecmartin | Got server fault | 15:11 |
| kopecmartin | oh , before it was a different error | 15:11 |
| frickler | I also saw some sporadic failures in OSC jobs, all network related | 15:11 |
| frickler | but nothing reproducible | 15:12 |
| kopecmartin | hopefully they won't repeat | 15:12 |
| kopecmartin | #topic Distros check | 15:13 |
| kopecmartin | cs-9 | 15:13 |
| kopecmartin | #link https://zuul.openstack.org/builds?job_name=tempest-full-centos-9-stream&job_name=tempest-full-py3-centos-8-stream&job_name=devstack-platform-centos-8-stream&job_name=devstack-platform-centos-9-stream&skip=0 | 15:13 |
| kopecmartin | fedora | 15:13 |
| frickler | failing 100% for ~6 days | 15:13 |
| kopecmartin | #link https://zuul.openstack.org/builds?job_name=devstack-platform-fedora-latest&skip=0 | 15:13 |
| kopecmartin | debian | 15:13 |
| kopecmartin | #link https://zuul.openstack.org/builds?job_name=devstack-platform-debian-bullseye&skip=0 | 15:13 |
| kopecmartin | jammy | 15:13 |
| kopecmartin | #link https://zuul.openstack.org/builds?job_name=devstack-platform-ubuntu-jammy&job_name=devstack-platform-ubuntu-jammy-ovn-source&job_name=devstack-platform-ubuntu-jammy-ovs&skip=0 | 15:13 |
| kopecmartin | hmm | 15:13 |
| * kopecmartin checking the logs of cs 9 | 15:14 | |
| kopecmartin | is there a pattern? | 15:15 |
| frickler | mostly the same 4 tests failing I think https://12f46aa5acfd725492c0-3062b6a47fa259284551b44674c48a9b.ssl.cf1.rackcdn.com/846588/2/check/devstack-platform-centos-9-stream/3a413b8/testr_results.html | 15:16 |
| kopecmartin | 'Got server fault' error on network side ? | 15:17 |
| kopecmartin | maybe this is related? | 15:20 |
| kopecmartin | #link https://bugs.launchpad.net/neutron/+bug/1979047 | 15:20 |
| frickler | seems to be similar, yes | 15:20 |
| kopecmartin | and that points to libvirt | 15:21 |
| kopecmartin | so, what should we do about it on devstack's side? | 15:22 |
| frickler | the "file descriptor null" message is in the n-cpu logs on the failing jobs | 15:24 |
| frickler | no idea what to do except make the job n-v | 15:24 |
| frickler | we could look at excluding only the failing tests, but not sure if that's worth it | 15:25 |
| frickler | also it once against confirms my claim that stream is not a stable distro | 15:25 |
| kopecmartin | yeah, ok, let's make it n-v | 15:25 |
| frickler | maybe we can find someone to build support for rockylinux instead | 15:25 |
| kopecmartin | i put that into the priority items for the cycle (rocky linux job) | 15:26 |
| kopecmartin | i'm failing to find time for that | 15:26 |
| kopecmartin | I'll ask around if someone is interested | 15:26 |
| frickler | do you want to propose the n-v patch? | 15:27 |
| kopecmartin | sure, i can so that you can vote | 15:27 |
| kopecmartin | after this meeting i need to travel and will be off for a few hours | 15:28 |
| frickler | ok, other distros look fine | 15:30 |
| opendevreview | Martin Kopec proposed openstack/devstack master: Mark devstack-platform-centos-9-stream as n-v https://review.opendev.org/c/openstack/devstack/+/847043 | 15:32 |
| kopecmartin | yeah, they do | 15:33 |
| kopecmartin | #topic Sub Teams highlights | 15:33 |
| kopecmartin | Changes with Review-Priority == +1 | 15:33 |
| kopecmartin | #link https://review.opendev.org/q/label:Review-Priority%253D%252B1+status:open+(project:openstack/tempest+OR+project:openstack/patrole+OR+project:openstack/devstack+OR+project:openstack/grenade) | 15:33 |
| kopecmartin | i see one patch there | 15:33 |
| frickler | yes, interesting tuning parameters | 15:34 |
| kopecmartin | absolutely | 15:34 |
| kopecmartin | a very nice improvement | 15:35 |
| kopecmartin | #topic Open Discussion | 15:36 |
| kopecmartin | anything for the open discussion? | 15:36 |
| frickler | not from me | 15:36 |
| kopecmartin | #topic Bug Triage | 15:38 |
| kopecmartin | #link https://etherpad.openstack.org/p/qa-bug-triage-zed | 15:38 |
| kopecmartin | the numbers were recorded | 15:38 |
| kopecmartin | gmann, just wondering, any update on this? | 15:39 |
| kopecmartin | #link https://review.opendev.org/c/openstack/tempest/+/842240 | 15:39 |
| frickler | need to clean up devstack LP one day so it gets lower than tempest again ;) | 15:39 |
| kopecmartin | :) | 15:40 |
| kopecmartin | i got few minutes today, so i triaged one or 2 | 15:40 |
| frickler | regarding ssh-able, there were some other patches recently https://review.opendev.org/q/topic:wait_until_sshable_pingable | 15:41 |
| kopecmartin | yeah, i'm aware, one was almost reverted but then gibi proposed a fix | 15:42 |
| kopecmartin | and that's already merged | 15:42 |
| kopecmartin | seems like the above one is the last one | 15:43 |
| kopecmartin | anyway, that's it from my side | 15:43 |
| kopecmartin | thanks and see you around | 15:44 |
| frickler | thx kopecmartin | 15:44 |
| kopecmartin | #endmeeting | 15:44 |
| opendevmeet | Meeting ended Tue Jun 21 15:44:33 2022 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 15:44 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/qa/2022/qa.2022-06-21-15.00.html | 15:44 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/qa/2022/qa.2022-06-21-15.00.txt | 15:44 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/qa/2022/qa.2022-06-21-15.00.log.html | 15:44 |
| dansmith | nova has also made the stream9 jobs n-v over the weekend I think | 15:45 |
| gibi | yepp due to https://bugs.launchpad.net/nova/+bug/1979047 | 15:52 |
| dansmith | and neutron it looks like | 15:53 |
| gmann | yeah in tempest also it is n-v | 16:09 |
| clarkb | I've said this in a few places, but it really feels like centos-9-stream is a good early alert to upcoming problems, but not a good stable CI platform for ensuring your own software is happy | 16:09 |
| clarkb | To me that means it is valuable to have centos-9-stream jobs on master to catch problems early. But maybe stable branches shouldn't be relying on it | 16:10 |
| *** pojadhav|ruck is now known as pojadhav|out | 16:11 | |
| dansmith | clarkb: I also agree | 16:23 |
| dansmith | of course, being n-v always means we'll just ignore it (and thus waste resources) but it definitely seems like periodics if people will really pay attention to failures, is helpful | 16:23 |
| dansmith | this current issue looks like a bug in qemu or libvirt to me, | 16:24 |
| dansmith | but if it's actually something triggered by our behavior, then knowing early is good | 16:24 |
| *** jpena is now known as jpena|off | 16:24 | |
| dansmith | but in terms of running jobs we expect to work on it, I agree, it's just not useful for that, IMHO | 16:24 |
| gmann | I knew that when I made it voting but that was to check how stable it is (as n-v job results are always get ignored). | 16:30 |
| gmann | even for master also we should think on having this distro in defined testing runtime or not and keep it as good to test to catch the possible issues. | 16:31 |
| dansmith | yeah, but it generates an exercise to make it n-v then voting again every time it breaks which requires more than just one person in a lot of cases | 16:32 |
| dansmith | so I dunno, some balance is needed for sure, but I'm not sure voting on every patch (until we have to remove it) is really the best approach | 16:33 |
| dansmith | IIRC, this is the second time we've had to make it n-v in like six months, which itself was right on the heels of a long period of breakage that clarkb had been working through | 16:33 |
| clarkb | At the summit there was talk with the centos devs about setting up third party ci to report back to them. Maybe that is a better way to leverage centos-9-stream on our end? | 16:36 |
| clarkb | basically pull it out of our reporting stream and report to the distro directly when updates break stuff. The problem with that is centos devs won't know where to start debugging devstack or tempest | 16:37 |
| gmann | clarkb: yeah, third party ci approach sounds good plan. what they said on that? | 16:41 |
| clarkb | gmann: they are set up to allow it. Figuring out what jobs to run so that they are meaningful to centos was thought to be the major issue | 16:42 |
| clarkb | I think their code review system is pagure (or was it gitlab?) zuul supports that so mostly a matter of someone deciding we want to do that and adding the connection and jobs | 16:43 |
| gmann | ok | 16:43 |
| dansmith | so that means a periodic or n-v job that rings a bell for them? I'm sure we could find a liaison to help triage for them | 16:54 |
| dansmith | if we had a daily periodic and we ring the bell (for them or us) if it fails like three days in a row or something that would help cut down the noise | 16:55 |
| dansmith | same goes for us paying attention to a periodic.. it's hard to see one failure in isolation and be worried, but three in a row or something holds more weight | 16:55 |
| clarkb | ya something like that would probably work | 16:55 |
| gmann | yeah, having liaison from there side is key thing otherwise we know it fail in n-v or periodic but do not know if any form c9s team knows about it | 17:00 |
| gmann | *anyone from | 17:00 |
| opendevreview | Ade Lee proposed openstack/tempest master: Test fips job on centos-9 https://review.opendev.org/c/openstack/tempest/+/831607 | 18:04 |
| opendevreview | Ade Lee proposed openstack/tempest master: Test fips job on centos-9 https://review.opendev.org/c/openstack/tempest/+/831607 | 18:44 |
| opendevreview | Ade Lee proposed openstack/devstack-plugin-nfs master: Add fips job https://review.opendev.org/c/openstack/devstack-plugin-nfs/+/847087 | 19:17 |
| opendevreview | Francesco Pantano proposed openstack/devstack-plugin-ceph master: Deploy with cephadm https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/826484 | 20:17 |
| opendevreview | Francesco Pantano proposed openstack/devstack-plugin-ceph master: Deploy with cephadm https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/826484 | 21:31 |
| opendevreview | Francesco Pantano proposed openstack/devstack-plugin-ceph master: Deploy with cephadm https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/826484 | 21:40 |
| opendevreview | Merged openstack/devstack master: Mark devstack-platform-centos-9-stream as n-v https://review.opendev.org/c/openstack/devstack/+/847043 | 22:04 |
| opendevreview | Ade Lee proposed openstack/devstack-plugin-nfs master: Add fips job https://review.opendev.org/c/openstack/devstack-plugin-nfs/+/847087 | 23:17 |
| opendevreview | Ade Lee proposed openstack/devstack-plugin-nfs master: Add fips job https://review.opendev.org/c/openstack/devstack-plugin-nfs/+/847087 | 23:25 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!