15:00:18 <kopecmartin> #startmeeting qa
15:00:18 <opendevmeet> Meeting started Tue Mar 22 15:00:18 2022 UTC and is due to finish in 60 minutes.  The chair is kopecmartin. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:18 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:18 <opendevmeet> The meeting name has been set to 'qa'
15:00:26 <kopecmartin> #link https://wiki.openstack.org/wiki/Meetings/QATeamMeeting#Weekly_QA_Team_meeting
15:00:30 <kopecmartin> agenda ^^
15:00:31 <gmann> o/
15:00:40 <lpiwowar> Hi o/
15:01:41 <kopecmartin> hello .. great,let's start
15:01:43 <kopecmartin> #topic Announcement and Action Item (Optional)
15:01:58 <kopecmartin> hmm, i don't have any announcements for this week
15:02:20 <kopecmartin> #topic Yoga Priority Items progress
15:02:26 <kopecmartin> #link https://etherpad.opendev.org/p/qa-yoga-priority
15:02:30 <kopecmartin> any updates on this front?
15:04:46 <kopecmartin> nothing specific from my side, i know there are many patches for the tempest-scenario-manager-cleanup
15:04:54 <kopecmartin> i'm slowly reviewing
15:05:05 <kopecmartin> #link https://review.opendev.org/q/topic:tempest-scenario-manager-cleanup
15:05:32 <gmann> I have also reviewed most of them and left comment
15:05:54 <kopecmartin> i saw, thank you gmann
15:06:00 <gmann> only concern I still have it many of them will have merge conflict as not all are in dependency series
15:06:06 <gmann> which will be extra review effort
15:06:39 <gmann> what we can do is if we see any patch in same plugin and not in series then we can ask to do that first.
15:06:55 <gmann> that way we can see at the end what all method left in plugin manager
15:07:19 <kopecmartin> yeah, although the extra effort will be mostly on the owner's side who'll need to rebase, a reviewer can simply check what changed between the patchsets and only revote previous vote if nothing related changed
15:07:40 <kopecmartin> gmann: +1, yep, that's a good idea
15:07:57 <kopecmartin> rpopelka: ^^
15:08:07 <kopecmartin> soniya29: ^^
15:08:49 <soniya29> kopecmartin, gmann okay
15:08:59 <kopecmartin> moving on
15:09:00 <kopecmartin> #topic OpenStack Events Updates and Planning
15:09:14 <kopecmartin> PTG is gonna be in 2 weeks
15:09:18 <kopecmartin> #link https://etherpad.opendev.org/p/qa-zed-ptg
15:09:35 <kopecmartin> our etherpad for the topics we will discuss during our sessions ^^
15:09:47 <kopecmartin> anyone can add their topic there
15:10:29 <kopecmartin> I'll assign the gathered topics to the specific slots a few days prior the ptg
15:11:22 <kopecmartin> #topic Gate Status Checks
15:11:30 <kopecmartin> #link https://review.opendev.org/q/label:Review-Priority%253D%252B2+status:open+(project:openstack/tempest+OR+project:openstack/patrole+OR+project:openstack/devstack+OR+project:openstack/grenade+OR+project:openstack/hacking)
15:11:35 <kopecmartin> no reviews there
15:11:58 <kopecmartin> #topic Periodic jobs Status Checks
15:12:03 <kopecmartin> #link https://zuul.openstack.org/builds?job_name=tempest-full-xena-py3&job_name=tempest-full-wallaby-py3&job_name=tempest-full-victoria-py3&job_name=tempest-full-ussuri-py3&job_name=tempest-full-train-py3&pipeline=periodic-stable
15:12:08 <kopecmartin> #link https://zuul.openstack.org/builds?project=openstack%2Ftempest&project=openstack%2Fdevstack&pipeline=periodic
15:12:17 <kopecmartin> seems all good there \o/
15:12:32 <gmann> on gate status, as you know this series is merged and centos9-stream job is voting on tempest gate now #link https://review.opendev.org/q/topic:wait_until_sshable_pingable
15:13:09 <gmann> I proposed to make it voting in devstack side too which is passing now #link https://review.opendev.org/c/openstack/devstack/+/834546
15:13:16 <frickler> openeuler and fedora both need to be looked at, see my earlier comments
15:13:35 <gmann> for centos9 stream?
15:13:59 <frickler> gmann: no, independent of that
15:14:04 <gmann> frickler: ohk
15:14:17 <frickler> kopecmartin: you still have the wrong xena job in your link above
15:14:27 <kopecmartin> ah, sorry
15:14:33 <gmann> yeah, not sure about fedora, openeuler and they are not in our testing runtime also.
15:14:57 <gmann> frickler: kopecmartin if there are no maintainer for those then we can remove them temporary and if anyone come back then add them back?
15:15:32 <frickler> lajoskatona: ^^ I think fedora was of some interest for neutron?
15:15:47 <frickler> gmann: I agree for openeuler
15:16:23 <kopecmartin> gmann: no need for removal now, i agree with not using it in gate, but let's keep the job around for a while more
15:16:34 <kopecmartin> i'm actively looking for new contributors
15:16:39 <gmann> kopecmartin: it consume the infra resource
15:16:53 <gmann> we can add it in experimental if anyone want to run
15:16:59 <frickler> having n-v jobs that constantly fail trains people to ignore failures there
15:17:05 <gmann> yeah
15:17:09 <frickler> which is something I want to avoid
15:17:14 <gmann> +1
15:17:16 <kopecmartin> yeah, i agree with that, but it starting passing, didn't it?
15:17:25 <gmann> fedora?
15:17:29 <kopecmartin> oh
15:17:39 <kopecmartin> i was still talking about centos streams 8/9
15:17:41 <gmann> I think we are mixing it with centos9 stream :)
15:17:46 <kopecmartin> yep
15:17:53 <frickler> fedora passes like 50%, openeuler fails 100% currently
15:17:55 <gmann> let's talk fedora and openeuler
15:18:13 <frickler> centos stream seems fine with the recent fixes, I just wonder for how long ;)
15:18:16 <lajoskatona> friskler: yea I checked and we have experimental and periodic job with fedora
15:18:25 <kopecmartin> good, let's put it to experimental queue then .. if someone starts working on that, they can easily run it as experimental
15:18:47 <kopecmartin> frickler: i wonder too :D
15:18:54 <kopecmartin> fingers crossed
15:18:55 <gmann> you mean fedora as well as openeuler in experimental and fedora in experimental we well as periodic (as it pass 50%) ?
15:18:59 <frickler> lajoskatona: do they work or do they also see high failure rates?
15:19:55 <lajoskatona> friskler: not that bad, I mean not 100% failure rate: https://zuul.openstack.org/builds?job_name=neutron-ovn-tempest-ovs-master-fedora
15:20:27 <frickler> that looks slightly better than https://zuul.opendev.org/t/openstack/builds?job_name=devstack-platform-fedora-latest&project=openstack/devstack , agreed
15:21:17 <gmann> yeah. just thinking moving it to periodic in devstack side make it less noticeable than n-v ?
15:21:43 <gmann> though n-v also does not get more attention
15:21:55 <frickler> we could still look at it weekly in this meeting
15:22:11 <gmann> to save infra resource I will vote for 1. openeuler  as experimental 2. fedore in experimental as well as periodic
15:22:13 <frickler> so maybe periodic+experimental would be fine for fedora
15:22:16 <gmann> yeah
15:22:19 <gmann> +1
15:22:31 <kopecmartin> sounds good
15:22:53 <gmann> coming back to centos9 stream
15:23:17 <gmann> frickler: I would like it to be voting because of same point you mentioned about notice of n-v jobs failure
15:23:57 <gmann> I agree it is not so stable but let's check the data as voting and if it is failing more and no active maintainer to fix it then I will bring this to TC about keeping it in distro or not
15:24:29 <gmann> as it is passing now this is best chance to check its stability and monitor
15:25:24 <kopecmartin> sounds good to me
15:25:51 <frickler> I just worried that it will block other things and we'll have to revert that
15:25:55 <soniya29> sounds good
15:26:12 <frickler> but maybe I'm too pessimistic, so let's give it a try
15:26:34 <frickler> and if it fails again soon, I can place one more nail in its coffin ;)
15:27:10 <kopecmartin> i'll add a few new links for the jobs in our agenda, so that we can regularly check how centos and fedora behave
15:27:14 <gmann> frickler: yeah, we can revert as soon as it fails and no one fix it
15:28:27 <gmann> frickler: kopecmartin one things if we want to merge it before stable/yoga which will be soon in this week or in zed master ?
15:28:51 <gmann> *thing is
15:29:21 <gmann> I mean if we merge now then it will be voting in stable/yoga too. I think it is ok?
15:29:53 <kopecmartin> i think it is ok, if it fails, we'll need to revert it either way (in master or / and in yoga)
15:29:58 <gmann> ok
15:30:33 <gmann> that is all on centos9-stream from my side.
15:31:26 <kopecmartin> ack, thanks
15:31:28 <kopecmartin> moving on
15:31:33 <kopecmartin> Changes with Review-Priority == +1
15:31:38 <kopecmartin> #link https://review.opendev.org/q/label:Review-Priority%253D%252B1+status:open+(project:openstack/tempest+OR+project:openstack/patrole+OR+project:openstack/devstack+OR+project:openstack/grenade+OR+project:openstack/hacking)
15:31:46 <kopecmartin> there are 2, one the centos9 patch
15:31:57 <kopecmartin> #link https://review.opendev.org/c/openstack/devstack/+/834546
15:31:59 <kopecmartin> and the other
15:32:04 <kopecmartin> #link https://review.opendev.org/c/openstack/devstack/+/833673
15:33:23 <kopecmartin> #topic Open Discussion
15:33:27 <kopecmartin> anything for the open discussion?
15:33:58 <lpiwowar> test_connectivity_between_vms_on_different_networks tempest test is failing (+ some other tempest plugin tests) with paramiko>2.8.1.
15:34:10 <lpiwowar> Here is a patch that probably fixes the issue (I am just working on testing it)
15:34:28 <lpiwowar> https://review.opendev.org/c/openstack/tempest/+/834686/2/tempest/lib/common/ssh.py ^^
15:34:28 <kopecmartin> oh , yeah
15:34:31 <kopecmartin> #link https://bugs.launchpad.net/tempest/+bug/1960692
15:35:35 <lpiwowar> I am not sure whether it is good approach. Disabling rsa-sha2 like this.
15:35:39 <gmann> what version we test in gate for paramiko ?
15:35:39 <frickler> didn't we fix something similar some weeks ago?
15:36:12 <lpiwowar> 2.8.1
15:36:14 <gmann> 2.8.1 https://github.com/openstack/requirements/blob/master/upper-constraints.txt#L179
15:36:17 <gmann> yeah
15:36:31 <gmann> so it is passing on that but failing with later?
15:36:43 <lpiwowar> It is probably known issue: https://opendev.org/openstack/requirements/commit/38976e9fa9d037a2c04dfd4e7b34ee39841305ec
15:36:54 <lpiwowar> The first version when it starts failing is paramiko 2.9.1
15:36:56 <gmann> I see
15:38:04 <kopecmartin> frickler: you maybe meant this? it's the only think we merged lately related to sha i can remember  - https://review.opendev.org/c/openstack/devstack/+/831245/3/lib/tls
15:38:07 <clarkb> I've been suggesting that other key type sbe used instead of rsa so that you don't have to reduce your security stance. Granted this is for testing. But it seems like the right thing is to use tools that are considered secure rather than sha1 which people are trying to move away from
15:39:16 <frickler> ah, we just capped it in u-c and never fixed it for good, ok
15:39:45 <gmann> we can test tempest fix with depends-on in requirements 2.9.1 or higher version u-c patch , that can help if that fix all things or not
15:40:07 <frickler> clarkb: I like that idea
15:41:05 <lpiwowar> @gmann Ok, I'll try to test the patch with 2.9.1 and will see.
15:42:39 <frickler> lpiwowar: would you be interested in also trying to look at changing the key type that tempest uses?
15:42:44 <lpiwowar> I have tried to experiment with the patch locally and it seems that it fixes the issue (at least for some barbican tests) but I need to check if it fixes the bug even for tempest tests.
15:43:18 <kopecmartin> lpiwowar: maybe after that we could test whether there is another way how to fix the issue, without disabling the algorithms
15:43:18 <clarkb> I think ecdsa keys make fips testing easier anyway.
15:43:24 <lpiwowar> @frickler Yeah, I can check it out.
15:44:24 <lpiwowar> @kopecmartin Ok, it sounds good
15:44:45 <kopecmartin> good, anything else?
15:45:19 <gmann> I have one topic on os-testr
15:45:36 <gmann> you might see it on ML about retirement of os-testr #link http://lists.openstack.org/pipermail/openstack-discuss/2022-March/027784.html
15:46:01 <gmann> os-testr is always confused with complete replacement by stestr but it is not
15:46:42 <gmann> you can see reply from mtreinish also on what all few CLI/utils it has other than stestr.
15:47:30 <gmann> It is not broken and its maintenance  cost is almost none. question is should we retire it and move the existing CLI/utils to other place?  OR just keep it as it is?
15:47:59 <gmann> other place is another question and if that team is ok to take/maintain those
15:48:02 <frickler> does it make sense to remove the os-testr binary and just keep what it actually needed?
15:48:37 <frickler> but I'm for keeping it in place, moving to different place isn't worth the action
15:48:45 <gmann> we can do that, test running is depreacated since long so we can remove that assuming everyone using stestr
15:48:53 <clarkb> maybe add stetsr as a dependency to os-testr so that it pulls in the replacement for that portion
15:49:45 <gmann> we can remove that part completely as it was deprecated long time ago and if anyone using it (I know few repo does) then they will be broken and have to move to stestr
15:50:10 <gmann> I think that will avoid confusion of seeing os-testr being deprecated long ago but not retire
15:51:41 <kopecmartin> i'm for either having it as is, or removing stestr only from the repo (or putting there a return right at the beginning so that no-one executes that)
15:51:45 <gmann> thanks for the idea. if that is ok for everyone I can propose and update on ML. stephen is not here so we can get his opnion in ML
15:52:01 <kopecmartin> putting stestr as a dep would encourage people to continue using os-testr, which is not what we want
15:52:28 <gmann> yeah, I will say remove it completely and ask to use stestr for test run
15:52:34 <kopecmartin> and moving the other useful things to a new repo will require work and will cause some problems along the way, so i would avoid this too
15:53:55 <gmann> yeah
15:54:31 <gmann> I will push patch and update on ML.
15:54:35 <gmann> that is all from me.
15:54:49 <kopecmartin> thank you gmann for bringing this up
15:55:01 <kopecmartin> #topic Bug Triage
15:55:05 <kopecmartin> #link https://etherpad.opendev.org/p/qa-bug-triage-yoga
15:55:17 <kopecmartin> we see a small decrease in the numbers again, yay
15:55:25 <kopecmartin> that's  all from my side too
15:55:29 <gmann> \o/
15:55:35 <lpiwowar> o/
15:55:46 <kopecmartin> if there isn't anything else, let's close the office hour
15:55:48 <gmann> at least more smooth in yoga release time at least as of now :)
15:56:04 <gmann> but still we have 1 more week to go
15:56:10 <kopecmartin> yeah
15:56:11 <gmann> yeah, nothing from me
15:56:26 <kopecmartin> thank you everyone
15:56:28 <kopecmartin> see you around
15:56:32 <gmann> thanks kopecmartin
15:56:33 <kopecmartin> #endmeeting