15:00:17 <kopecmartin> #startmeeting qa
15:00:17 <opendevmeet> Meeting started Tue Jun 21 15:00:17 2022 UTC and is due to finish in 60 minutes.  The chair is kopecmartin. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:17 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:17 <opendevmeet> The meeting name has been set to 'qa'
15:00:23 <kopecmartin> #link https://wiki.openstack.org/wiki/Meetings/QATeamMeeting#Weekly_QA_Team_meeting
15:00:27 <kopecmartin> the agenda for today ^^
15:01:10 <frickler> \o
15:01:44 <kopecmartin> o/
15:03:09 <kopecmartin> let's go through the usual as alway
15:03:13 <kopecmartin> always
15:03:14 <kopecmartin> #topic Announcement and Action Item (Optional)
15:03:19 <kopecmartin> none
15:03:29 <kopecmartin> #topic Zed Priority Items progress
15:03:29 <kopecmartin> #link https://etherpad.opendev.org/p/qa-zed-priority
15:03:37 <kopecmartin> anything here?
15:04:03 <kopecmartin> we started cleaning the tests with the unstable_test decorator
15:04:33 <kopecmartin> we're exploring whether we can just remove it - means we're checking whether the bugs which were the reason for the decorator are still valid
15:05:14 <kopecmartin> some of them are quite old, so the worst case scenario will be , we'll remove the decorators and see whether something is breaking :)
15:05:44 <kopecmartin> that's from me on this topic
15:06:27 <kopecmartin> #topic Gate Status Checks
15:06:33 <kopecmartin> #link https://review.opendev.org/q/label:Review-Priority%253D%252B2+status:open+(project:openstack/tempest+OR+project:openstack/patrole+OR+project:openstack/devstack+OR+project:openstack/grenade)
15:06:37 <kopecmartin> no reviews
15:06:40 <kopecmartin> any urgent patches?
15:07:50 <frickler> like we need one to make c9s non-voting
15:08:01 <frickler> likely
15:08:20 <frickler> for devstack
15:08:41 <frickler> but we have that topic later
15:09:35 <kopecmartin> seems the cs9 job has been failing lately  .. yeah, we'll get to that
15:09:36 <kopecmartin> #topic Periodic jobs Status Checks
15:09:43 <kopecmartin> stable
15:09:44 <kopecmartin> #link #link https://zuul.openstack.org/builds?job_name=tempest-full-yoga&job_name=tempest-full-xena&job_name=tempest-full-wallaby-py3&job_name=tempest-full-victoria-py3&job_name=tempest-full-ussuri-py3&pipeline=periodic-stable
15:09:49 <kopecmartin> master
15:09:49 <kopecmartin> #link https://zuul.openstack.org/builds?project=openstack%2Ftempest&project=openstack%2Fdevstack&pipeline=periodic
15:10:14 <kopecmartin> devstack-no-tls-proxy failed
15:10:31 <kopecmartin> although it failed before too
15:10:34 * kopecmartin checking hte logs
15:11:03 <kopecmartin> Got server fault
15:11:51 <kopecmartin> oh , before it was a different error
15:11:56 <frickler> I also saw some sporadic failures in OSC jobs, all network related
15:12:03 <frickler> but nothing reproducible
15:12:35 <kopecmartin> hopefully they won't repeat
15:13:14 <kopecmartin> #topic Distros check
15:13:23 <kopecmartin> cs-9
15:13:23 <kopecmartin> #link https://zuul.openstack.org/builds?job_name=tempest-full-centos-9-stream&job_name=tempest-full-py3-centos-8-stream&job_name=devstack-platform-centos-8-stream&job_name=devstack-platform-centos-9-stream&skip=0
15:13:29 <kopecmartin> fedora
15:13:31 <frickler> failing 100% for ~6 days
15:13:34 <kopecmartin> #link https://zuul.openstack.org/builds?job_name=devstack-platform-fedora-latest&skip=0
15:13:44 <kopecmartin> debian
15:13:45 <kopecmartin> #link https://zuul.openstack.org/builds?job_name=devstack-platform-debian-bullseye&skip=0
15:13:52 <kopecmartin> jammy
15:13:52 <kopecmartin> #link https://zuul.openstack.org/builds?job_name=devstack-platform-ubuntu-jammy&job_name=devstack-platform-ubuntu-jammy-ovn-source&job_name=devstack-platform-ubuntu-jammy-ovs&skip=0
15:13:58 <kopecmartin> hmm
15:14:11 * kopecmartin checking the logs of cs 9
15:15:35 <kopecmartin> is there a pattern?
15:16:13 <frickler> mostly the same 4 tests failing I think https://12f46aa5acfd725492c0-3062b6a47fa259284551b44674c48a9b.ssl.cf1.rackcdn.com/846588/2/check/devstack-platform-centos-9-stream/3a413b8/testr_results.html
15:17:21 <kopecmartin> 'Got server fault' error on network side ?
15:20:27 <kopecmartin> maybe this is related?
15:20:29 <kopecmartin> #link https://bugs.launchpad.net/neutron/+bug/1979047
15:20:30 <frickler> seems to be similar, yes
15:21:47 <kopecmartin> and that points to libvirt
15:22:52 <kopecmartin> so, what should we do about it on devstack's side?
15:24:20 <frickler> the "file descriptor null" message is in the n-cpu logs on the failing jobs
15:24:30 <frickler> no idea what to do except make the job n-v
15:25:05 <frickler> we could look at excluding only the failing tests, but not sure if that's worth it
15:25:36 <frickler> also it once against confirms my claim that stream is not a stable distro
15:25:51 <kopecmartin> yeah, ok, let's make it n-v
15:25:54 <frickler> maybe we can find someone to build support for rockylinux instead
15:26:18 <kopecmartin> i put that into the priority items for the cycle (rocky linux job)
15:26:26 <kopecmartin> i'm failing to find time for that
15:26:36 <kopecmartin> I'll ask around if someone is interested
15:27:34 <frickler> do you want to propose the n-v patch?
15:27:48 <kopecmartin> sure, i can so that you can vote
15:28:01 <kopecmartin> after this meeting i need to travel and will be off for a few hours
15:30:12 <frickler> ok, other distros look fine
15:32:55 <opendevreview> Martin Kopec proposed openstack/devstack master: Mark devstack-platform-centos-9-stream as n-v  https://review.opendev.org/c/openstack/devstack/+/847043
15:33:18 <kopecmartin> yeah, they do
15:33:20 <kopecmartin> #topic Sub Teams highlights
15:33:25 <kopecmartin> Changes with Review-Priority == +1
15:33:30 <kopecmartin> #link https://review.opendev.org/q/label:Review-Priority%253D%252B1+status:open+(project:openstack/tempest+OR+project:openstack/patrole+OR+project:openstack/devstack+OR+project:openstack/grenade)
15:33:55 <kopecmartin> i see one patch there
15:34:32 <frickler> yes, interesting tuning parameters
15:34:51 <kopecmartin> absolutely
15:35:23 <kopecmartin> a very nice improvement
15:36:34 <kopecmartin> #topic Open Discussion
15:36:39 <kopecmartin> anything for the open discussion?
15:36:53 <frickler> not from me
15:38:34 <kopecmartin> #topic Bug Triage
15:38:40 <kopecmartin> #link https://etherpad.openstack.org/p/qa-bug-triage-zed
15:38:44 <kopecmartin> the numbers were recorded
15:39:43 <kopecmartin> gmann, just wondering, any update on this?
15:39:45 <kopecmartin> #link https://review.opendev.org/c/openstack/tempest/+/842240
15:39:56 <frickler> need to clean up devstack LP one day so it gets lower than tempest again ;)
15:40:14 <kopecmartin> :)
15:40:23 <kopecmartin> i got few minutes today, so i triaged one or 2
15:41:21 <frickler> regarding ssh-able, there were some other patches recently https://review.opendev.org/q/topic:wait_until_sshable_pingable
15:42:52 <kopecmartin> yeah, i'm aware, one was almost reverted but then gibi proposed a fix
15:42:56 <kopecmartin> and that's already merged
15:43:06 <kopecmartin> seems like the above one is the last one
15:43:56 <kopecmartin> anyway, that's it from my side
15:44:02 <kopecmartin> thanks and see you around
15:44:11 <frickler> thx kopecmartin
15:44:33 <kopecmartin> #endmeeting