15:00:36 <kopecmartin> #startmeeting qa
15:00:36 <opendevmeet> Meeting started Tue Aug  8 15:00:36 2023 UTC and is due to finish in 60 minutes.  The chair is kopecmartin. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:36 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:36 <opendevmeet> The meeting name has been set to 'qa'
15:00:38 <kevko> dansmith: okay, and why do you have service_token_roles_required = False
15:01:03 <kopecmartin> #link https://wiki.openstack.org/wiki/Meetings/QATeamMeeting#Agenda_for_next_Office_hours
15:01:05 <kevko> dansmith: because there should be = True
15:01:07 <kopecmartin> agenda ^
15:01:09 <frickler> o/
15:01:35 <lpiwowar> o/
15:01:42 <kevko> dansmith: as per https://docs.openstack.org/cinder/latest/configuration/block-storage/service-token.html
15:02:42 <lpiwowar> Is gerrit down or is it only problem on my side?
15:03:30 <kopecmartin> kevko: where is that? i don't see such setting in tempest or devstack
15:04:05 <kopecmartin> lpiwowar: seems down
15:04:30 <opendevreview> Lukas Piwowarski proposed openstack/tempest master: Skip test_list_no_containers when pre-prov creds are used  https://review.opendev.org/c/openstack/tempest/+/890798
15:04:38 <kevko> kopecmartin: https://5248ae6d6484a440a059-bda51eedb42181063c5344e0473d3d05.ssl.cf2.rackcdn.com/888470/1/gate/nova-multi-cell/2964990/controller/logs/screen-c-api.txt << ctrl+f    service_token_roles_required
15:04:53 <kevko> ^^ this should be True as per https://docs.openstack.org/cinder/latest/configuration/block-storage/service-token.html
15:05:18 <frickler> gerrit look fine to me
15:05:21 <kevko> that's the reason why upstream  tempest.scenario.test_server_volume_attachment.TestServerVolumeAttachmentScenario.test_server_detach_rules is passing ...
15:05:58 <kevko> in my env where i am setting service_token_roles_required = True it's failing ...but i got 401 ...so i think the test is broken ....
15:06:09 <lpiwowar> frickler: it works me now too
15:07:21 <frickler> kevko: so maybe devstack/tempest need to adapt the deployment and then fix the test. can we go on with the meeting now, though?
15:07:32 <kopecmartin> kevko: the setting seems to be coming from keystone middleware ..
15:07:52 <frickler> likely default=false for backwards compat
15:07:54 <kopecmartin> i'll try dig deeper, but now let's start with the meeting
15:08:48 <kopecmartin> yeah, will need to find some patches .. maybe it's something we forgot to enable in tempest/devstack
15:08:55 <kopecmartin> #topic Announcement and Action Item (Optional)
15:09:04 <kopecmartin> none from my side
15:09:28 <kopecmartin> #topic Bobcat Priority Items progress
15:09:33 <kopecmartin> #link https://etherpad.opendev.org/p/qa-bobcat-priority
15:09:36 * kopecmartin checking the doc
15:10:23 <kopecmartin> seems i need to follow up on
15:10:25 <kopecmartin> #link https://review.opendev.org/c/openstack/devstack/+/558930
15:10:36 <kopecmartin> there have been recent changes
15:11:09 <frickler> I updated the venv name according to dansmith's suggestion
15:11:23 <frickler> which actually I had commented upon 5 years ago, too ;)
15:12:27 <frickler> then there's still the discussion whether to symlink only specific binaries into /usr/local or add the complete venv into the path
15:13:20 <frickler> I would like us to proceed with what we have now, but there also hasn't been a lot of reviews
15:17:40 <kopecmartin> yeah, let me revisit that right after this meeting, personally i'd rather merge this sooner than later as it may broke someone else and we'll be aproaching another release soon
15:19:14 <frickler> in particular adding the bookworm job would be good, ack
15:19:46 <kopecmartin> and as the jobs passed, it mostly works ,the rest can be figured out later
15:20:18 <kopecmartin> #topic Gate Status Checks
15:20:22 <kopecmartin> #link https://review.opendev.org/q/label:Review-Priority%253D%252B2+status:open+(project:openstack/tempest+OR+project:openstack/patrole+OR+project:openstack/devstack+OR+project:openstack/grenade)
15:20:25 <kopecmartin> nothing there
15:20:31 <kopecmartin> any urgent patches to review?
15:20:50 <lpiwowar> Can somebody please take a look at this patch: https://review.opendev.org/c/openstack/tempest/+/890653 ? It's about skipping tests that do user password update when miminum_password_age is set in keystone.conf.
15:21:04 <lpiwowar> But it is probably not urgent kopecmartin.
15:21:22 <kopecmartin> lpiwowar: right, it's on my list
15:21:38 <lpiwowar> kopecmartin: thanks!
15:21:59 <kopecmartin> #topic Bare rechecks
15:22:03 <kopecmartin> #link https://etherpad.opendev.org/p/recheck-weekly-summary
15:22:19 <kopecmartin> due to all the rechecks we made last week we quite pushed the percentage down :D
15:22:46 <kopecmartin> #topic Periodic jobs Status Checks
15:22:46 <kopecmartin> periodic stable full
15:22:46 <kopecmartin> #link https://zuul.openstack.org/builds?pipeline=periodic-stable&job_name=tempest-full-yoga&job_name=tempest-full-xena&job_name=tempest-full-zed&job_name=tempest-full-2023-1
15:22:48 <kopecmartin> periodic stable slow
15:22:50 <kopecmartin> #link https://zuul.openstack.org/builds?job_name=tempest-slow-2023-1&job_name=tempest-slow-zed&job_name=tempest-slow-yoga&job_name=tempest-slow-xena
15:22:52 <kopecmartin> periodic extra tests
15:22:54 <kopecmartin> #link https://zuul.openstack.org/builds?job_name=tempest-full-2023-1-extra-tests&job_name=tempest-full-zed-extra-tests&job_name=tempest-full-yoga-extra-tests&job_name=tempest-full-xena-extra-tests
15:22:56 <kopecmartin> periodic master
15:22:58 <kopecmartin> #link https://zuul.openstack.org/builds?project=openstack%2Ftempest&project=openstack%2Fdevstack&pipeline=periodic
15:23:26 <opendevreview> Ashley Rodriguez proposed openstack/devstack-plugin-ceph master: Remote Ceph with cephadm  https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/876747
15:25:05 <kopecmartin> tempest-all timeouts .. it seems it's ran with concurrency == 6
15:25:45 <kopecmartin> #link https://zuul.openstack.org/builds?job_name=tempest-all&project=openstack%2Ftempest&project=openstack%2Fdevstack&pipeline=periodic&skip=0
15:25:46 <kopecmartin> hm
15:27:29 <kopecmartin> #topic Distros check
15:27:29 <kopecmartin> cs-9
15:27:29 <kopecmartin> #link https://zuul.openstack.org/builds?job_name=tempest-full-centos-9-stream&job_name=devstack-platform-centos-9-stream&skip=0
15:27:31 <kopecmartin> fedora
15:27:33 <kopecmartin> #link https://zuul.openstack.org/builds?job_name=devstack-platform-fedora-latest&skip=0
15:27:35 <kopecmartin> debian
15:27:37 <kopecmartin> #link https://zuul.openstack.org/builds?job_name=devstack-platform-debian-bullseye&skip=0
15:27:39 <kopecmartin> focal
15:27:41 <kopecmartin> #link https://zuul.opendev.org/t/openstack/builds?job_name=devstack-platform-ubuntu-focal&skip=0
15:27:43 <kopecmartin> rocky
15:27:45 <kopecmartin> #link https://zuul.openstack.org/builds?job_name=devstack-platform-rocky-blue-onyx
15:27:47 <kopecmartin> openEuler
15:27:49 <kopecmartin> #link https://zuul.openstack.org/builds?job_name=devstack-platform-openEuler-22.03-ovn-source&job_name=devstack-platform-openEuler-22.03-ovs&skip=0
15:29:33 <frickler> F36 is eol, I think we need to drop that from 2023.1, too
15:29:42 <kopecmartin> right, i've just removed that from the agenda
15:30:22 <kopecmartin> frickler: should we also delete the job from stable/2023.1?
15:30:48 <kopecmartin> i see it failing there
15:31:11 <frickler> yes, that's what I just said ... or tried to
15:31:47 <kopecmartin> :D sorry
15:32:10 <kopecmartin> i read it but didn't understand it cause i was thinking about the agenda ... ah
15:32:18 * kopecmartin needs another coffee
15:32:35 <kopecmartin> so yes, i agree, will you propose a patch or should I?
15:32:59 <frickler> I can do the patch
15:33:10 <kopecmartin> prefect, thanks!
15:33:18 <gmann> o/
15:33:48 <kopecmartin> o/
15:34:15 <lpiwowar> o/
15:34:46 <kopecmartin> great, so that was fedora
15:34:56 <kopecmartin> this will help with centos9
15:34:57 <kopecmartin> #link https://review.opendev.org/q/Icd99f467d47aaafaaf3ee8f2a3c4da08842cb672
15:35:32 <kopecmartin> the rest of the distros look ok it seems
15:36:04 <kopecmartin> #topic Sub Teams highlights
15:36:04 <kopecmartin> Changes with Review-Priority == +1
15:36:09 <kopecmartin> #link https://review.opendev.org/q/label:Review-Priority%253D%252B1+status:open+(project:openstack/tempest+OR+project:openstack/patrole+OR+project:openstack/devstack+OR+project:openstack/grenade)
15:36:26 <kopecmartin> nothing there
15:36:27 <kopecmartin> #topic Open Discussion
15:36:33 <kopecmartin> anything for the open discussion?
15:37:19 <lpiwowar> I added the skip for the object storage test we talked about last week: https://review.opendev.org/c/openstack/tempest/+/890798 ... We can start the discussion there.
15:37:24 <sean-k-mooney> are there any know issue with grenede failing in keyston upgrade
15:37:53 <gmann> sean-k-mooney: not afaik
15:38:09 <gmann> it was all good till yesterday, have not checked today yet
15:38:12 <sean-k-mooney> ok cause i have seen this 3 times today on one of my patches
15:38:14 <sean-k-mooney> https://zuul.opendev.org/t/openstack/build/e4ecb12b5ea74a669f6e0ff59f478ad5/log/controller/logs/grenade.sh_log.txt#5430
15:38:22 <kopecmartin> lpiwowar: does it relate to this https://bugs.launchpad.net/tempest/+bug/2028671 ?
15:38:47 <lpiwowar> kopecmartin: yes:)
15:38:57 <sean-k-mooney> oh never mind
15:39:15 <lpiwowar> I will add the Related-Bug:
15:39:17 <gmann> ok, have not seen that. but I will monitor
15:39:19 <sean-k-mooney> its related to my patch
15:39:49 <opendevreview> Lukas Piwowarski proposed openstack/tempest master: Skip test_list_no_containers when pre-prov creds are used  https://review.opendev.org/c/openstack/tempest/+/890798
15:40:02 <gmann> just FYI, after nova-network test removal, I am removing the glance v1 APIs tests too #link https://review.opendev.org/q/topic:remove-glance-v1-tests
15:40:12 <opendevreview> Ashley Rodriguez proposed openstack/devstack-plugin-ceph master: Remote Ceph with cephadm  https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/876747
15:40:26 <gmann> it need some updates in plugins side also for config as well as scenario manager logic so doing those first
15:41:02 <kopecmartin> gmann: thanks, i added the topic to my review list
15:41:12 <gmann> with that i found there are many plugins still has their scenario manager copy, did not we removed all those ? or they are just missing
15:41:24 <gmann> lpiwowar: I think you did the most of the work in that ^^
15:41:28 <gmann> kopecmartin: thanks
15:42:18 <gmann> I can check if we can remove them but just want to check if there is any reason we did not do that while doing it for otherplugins
15:42:28 <lpiwowar> gmann: I think that it was rpopelka if I remember correctly.
15:42:32 <kopecmartin> not all the patches are merged gmann
15:42:33 <kopecmartin> #link https://review.opendev.org/q/topic:tempest-scenario-manager-cleanup
15:42:44 <kopecmartin> or maybe we missed some
15:42:45 <gmann> lpiwowar: ohk
15:43:03 <gmann> kopecmartin: nice, thanks I will check those
15:43:30 <gmann> watcher and barbican are the one and changes are not merged.
15:43:35 <gmann> this explain it.
15:44:15 <kopecmartin> dansmith: what do we wanna do with that test_rebuild_server_with_volume_attached test?
15:44:33 <gmann> kopecmartin: is it failing consistently ?
15:44:55 <lpiwowar> gmann: We recently hit an issue with this in barbican-tempest-plugin regarding the removal of scenario manager: https://review.opendev.org/c/openstack/barbican-tempest-plugin/+/889354
15:45:22 <kopecmartin> gmann: no, it doesn't seem so
15:45:33 <gmann> lpiwowar: ohk, will check this change. thanks for info
15:45:38 <ykarel> wrt rebuild test is this related to issue Host key for server does not match?
15:45:47 <ykarel> recently noticed in some of neutron jobs
15:45:58 <lpiwowar> The failure was caused by victoria job using master barbican-tempest-plugin. I'm trying to fix this here:  https://review.opendev.org/c/openstack/barbican-tempest-plugin/+/890784
15:46:05 <gmann> not sure, I think we fixed that test right ?
15:46:15 <kopecmartin> gmann: to save you some time, the barbican issue is that they use master plugin on victoria branch ... missing dependencies etc
15:46:51 <gmann> kopecmartin: I see, which they can use old plugin as victoria is too old for tempest master and barbican-tempest-plugin to use
15:46:57 <gmann> I will have a look into that
15:47:26 <kopecmartin> ykarel: seems not .. i started looking into that just an hour ago and started with only this https://5248ae6d6484a440a059-bda51eedb42181063c5344e0473d3d05.ssl.cf2.rackcdn.com/888470/1/gate/nova-multi-cell/2964990/testr_results.html
15:48:15 <ykarel> kopecmartin, ack i was talking about https://cfc27c554a994f35be4e-76bf72ffc642f12cb8e7d8393148d522.ssl.cf5.rackcdn.com/874797/41/gate/neutron-ovn-tempest-ipv6-only-ovs-release/8162e79/testr_results.html
15:48:19 <gmann> kopecmartin: I think we added wait for rebuild to finish there but need to check
15:48:30 <ykarel> and as per opensearch i see that in multiple jobs and times
15:48:38 <opendevreview> Lukas Piwowarski proposed openstack/tempest master: Skip test_list_no_containers when pre-prov creds are used  https://review.opendev.org/c/openstack/tempest/+/890798
15:50:25 <gmann> kopecmartin: we do wait for server to be active after rebuild https://github.com/openstack/tempest/blob/3e996dc6cefd5ad7136001a6eb846a9255cce961/tempest/api/compute/servers/test_server_actions.py#L158
15:51:29 <opendevreview> sean mooney proposed openstack/devstack master: add support for zswap and ksmtuned  https://review.opendev.org/c/openstack/devstack/+/890693
15:52:16 <gmann> kopecmartin: dansmith: I think what happening here is that rebuild is stuck and teardown try to detach volume and fail.
15:52:23 <kopecmartin> gmann: yes, the issue seemed to be about a race between attached volumes
15:52:38 <gmann> we can make teardown little bit more smart to handle that
15:52:53 <kopecmartin> gmann: btw, we run validation twice there, once in _test_rebuild_server and once at the end of the test
15:53:58 <gmann> kopecmartin: yeah, I think we can remove until there is any reason whoami-rajat knows
15:55:02 <gmann> let me have a look into this in my noon and will update teardown also
15:55:23 <kopecmartin> ack, sounds good, thanks
15:56:11 <kopecmartin> gmann: btw, we're feeling with frickler adventurous and would move this forward - https://review.opendev.org/c/openstack/devstack/+/558930
15:57:08 <gmann> kopecmartin: ok, i have not checked it actually but I think you and dansmith are looking into it.
15:57:46 <gmann> but do not wait for me, i might not be able to check it for  this week or so
15:58:03 <kopecmartin> okey
15:58:04 <kopecmartin> #topic Bug Triage
15:58:09 <kopecmartin> #link https://etherpad.openstack.org/p/qa-bug-triage-bobcat
15:58:15 <kopecmartin> new numbers there ^^
15:58:21 <kopecmartin> time is almost up
15:58:29 <kopecmartin> so if there isn't anything else , we can call it an hour
15:58:56 <kopecmartin> thanks everyone
15:58:56 <kopecmartin> #endmeeting