15:00:15 <kopecmartin> #startmeeting qa
15:00:15 <opendevmeet> Meeting started Tue Apr 19 15:00:15 2022 UTC and is due to finish in 60 minutes.  The chair is kopecmartin. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:15 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:15 <opendevmeet> The meeting name has been set to 'qa'
15:00:29 <kopecmartin> #link https://wiki.openstack.org/wiki/Meetings/QATeamMeeting#Weekly_QA_Team_meeting
15:00:33 <kopecmartin> agenda ^^
15:02:21 <kopecmartin> #topic Announcement and Action Item (Optional)
15:02:24 <kopecmartin> none from my side
15:02:30 <kopecmartin> #topic Zed Priority Items progress
15:02:35 <gmann> o/
15:02:36 <kopecmartin> #link https://etherpad.opendev.org/p/qa-zed-priority
15:02:41 <kopecmartin> hi gmann
15:03:22 <kopecmartin> topic: Unstable tests in Tempest monitoring
15:03:36 <kopecmartin> I found out that we have 3 tests decorated as unstable ones
15:04:16 <gmann> ok
15:05:26 <kopecmartin> test_create_object_with_transfer_encoding is associated with the following bug
15:05:28 <gmann> kopecmartin: can you please remove the +W , i forgot to keep it in -W as I am still testing it https://review.opendev.org/c/openstack/tempest/+/837777
15:05:57 <kopecmartin> gmann: sure, sorry
15:06:08 <gmann> thanks
15:06:18 <kopecmartin> so the above test is associated with the following bug:
15:06:21 <kopecmartin> #link https://bugs.launchpad.net/tempest/+bug/1905432
15:06:34 <kopecmartin> it's in progress, we can leave the test as is right now
15:06:41 <kopecmartin> although the other test
15:06:45 <kopecmartin> test_container_synchronization
15:06:54 <kopecmartin> should be fixed already
15:06:56 <kopecmartin> #link  https://bugs.launchpad.net/tempest/+bug/1317133
15:07:09 <kopecmartin> do we want to remove the unstable decorator there and watch what happens?
15:07:53 <gmann> I think we can remove and watch, its start of cycle so good to try
15:08:40 <kopecmartin> ack
15:08:49 <kopecmartin> the last one test_server_connectivity_cold_migration_revert
15:08:53 <kopecmartin> has an expired test
15:08:57 <kopecmartin> *bug
15:09:05 <kopecmartin> #link  https://bugs.launchpad.net/neutron/+bug/1836595
15:09:23 <kopecmartin> should we skip the test permanently?
15:10:13 <gmann> humm skip test permanently is not good
15:10:42 <kopecmartin> yeah, ok , maybe let me dig in a bit and find out where/what jobs execute the test
15:10:50 <kopecmartin> let's see how it behaves
15:12:14 <gmann> yeah, we can re-open it for neutron
15:12:41 <kopecmartin> sounds good
15:12:50 <kopecmartin> any other updates?
15:13:12 <kopecmartin> i'm gonna push a review soon to switch to ecdsa keys by default in tempest
15:15:34 <kopecmartin> #topic OpenStack Events Updates and Planning
15:15:38 <kopecmartin> nothing specific here
15:15:43 <kopecmartin> #topic Gate Status Checks
15:15:51 <kopecmartin> https://review.opendev.org/q/label:Review-Priority%253D%252B2+status:open+(project:openstack/tempest+OR+project:openstack/patrole+OR+project:openstack/devstack+OR+project:openstack/grenade)
15:16:04 <kopecmartin> none reviews there
15:16:10 <soniya29> i have couple of patches failing in barbican-tempest-plugin with same error
15:16:25 <soniya29> https://aad416e65f3b6b410df4-779f7e98ba8ab2ef93c3580c1febb3f0.ssl.cf2.rackcdn.com/834000/3/check/barbican-tempest-plugin-simple-crypto-victoria/07b40f8/job-output.txt
15:16:36 <kopecmartin> yeah, they fail due to this
15:16:38 <kopecmartin> #link https://bugs.launchpad.net/devstack/+bug/1968798
15:17:20 <kopecmartin> even though some workaround patches have been merged, it still fails somewhere
15:17:30 <kopecmartin> e.g. barbican-tempest-plugin or python-tempestconf
15:17:36 <kopecmartin> i don't know why
15:18:09 <gmann> soniya29: kopecmartin yeah it should be fixed now. or you saw the error after fix were merged ?
15:18:13 <soniya29> kopecmartin, my patches are blocked because of it :)
15:18:24 <soniya29> gmann, i am seeing these errors today
15:18:45 <kopecmartin> yep, it failed today, see my comment
15:18:51 <kopecmartin> https://bugs.launchpad.net/devstack/+bug/1968798/comments/5
15:18:58 <gmann> soniya29: this is victoris job which should be fixed too
15:19:20 <soniya29> gmann, it is failing on xena, ussuri as well
15:19:34 <soniya29> https://review.opendev.org/c/openstack/barbican-tempest-plugin/+/834000
15:19:47 <gmann> ussuri it will fail as we fixed only till stable/victoria
15:20:08 <soniya29> https://review.opendev.org/c/openstack/barbican-tempest-plugin/+/833801
15:21:52 <kopecmartin> i wonder why this fails as it uses master :/
15:21:53 <kopecmartin> #link https://zuul.opendev.org/t/openstack/build/025e36991b6e472ba0c204fede267104
15:23:36 <clarkb> I haven't opened the links but any use of git as a different user than ownership of the repo can cause this
15:23:37 <opendevreview> Eduardo Olivares proposed openstack/tempest master: Validate network downtime during live migration - part II  https://review.opendev.org/c/openstack/tempest/+/838518
15:24:00 <clarkb> the patches to devstack have flagged the repos that devsatck clones as safe to address this but if a plugin or job manually clones and doesn't do this they can still break
15:24:14 <clarkb> or if they interact with the zuul cloned repos directly as not the zuul user
15:24:29 <clarkb> etc etc. You'll need to dig into why they are still broken particualrly on master where the workaroudns should be sufficient
15:26:20 <kopecmartin> clarkb: does this count as interacting with it directly as not zuul user?
15:26:21 <kopecmartin> #link https://opendev.org/openinfra/python-tempestconf/src/branch/master/roles/install-plugins/tasks/main.yaml#L5
15:27:20 <clarkb> kopecmartin: no that should run out of the tempest clone done by devsatck which gets the flag set. The next task may be though as that pip installs requirements as root out of the zuul clone
15:27:41 <clarkb> er wait that doesn't pip install requirements it just gets upper constraints
15:27:50 <clarkb> it pip installs some list of plugins but I'm not sure where those plugins are located
15:28:11 <kopecmartin> it fails on the Prepare tempest venv task anyway
15:28:14 <kopecmartin> before installign plugins
15:28:21 <kopecmartin> ok, i'll try to dig more into that
15:29:11 <kopecmartin> btw, plugins_paths is just a var which references the plugins locations where they were cloned by devstack
15:29:34 <kopecmartin> so the workaround should work on that too
15:29:55 <clarkb> specifically the issue is running git commands in a git repo as a user other than the user that owns the files in the git repo
15:30:17 <clarkb> reading the files directly or navigating the repo with cd and that sort of interaction is fine. It is git itself policing this when you run git commands against a repo
15:30:35 <clarkb> installing things causes this to happen because pbr runs git commands to determine version info
15:33:07 <kopecmartin> i see, thanks, i'll check the tasks executed in more detail
15:33:59 <kopecmartin> #topic Periodic jobs Status Checks
15:34:08 <kopecmartin> stable:
15:34:09 <kopecmartin> #link https://zuul.openstack.org/builds?job_name=tempest-full-yoga&job_name=tempest-full-xena&job_name=tempest-full-wallaby-py3&job_name=tempest-full-victoria-py3&job_name=tempest-full-ussuri-py3&pipeline=periodic-stable
15:34:17 <kopecmartin> master:
15:34:19 <kopecmartin> #link https://zuul.openstack.org/builds?project=openstack%2Ftempest&project=openstack%2Fdevstack&pipeline=periodic
15:35:19 <kopecmartin> tempest-full-ussuri-py3 failed couple days ago with the same pbr error
15:35:21 <kopecmartin> #link tempest-full-ussuri-py3
15:35:27 <kopecmartin> #link https://zuul.openstack.org/build/e5fe6f0121924271b820c1ac32882640
15:35:44 <kopecmartin> which makes sense as the fix/workaround wasn't backported to ussuri
15:36:20 <kopecmartin> #link https://review.opendev.org/c/openstack/devstack/+/837749
15:37:28 <kopecmartin> #topic Distros check
15:37:33 <kopecmartin> centos 8/9 stream
15:37:40 <kopecmartin> #link https://zuul.openstack.org/builds?job_name=tempest-full-centos-9-stream&job_name=tempest-full-py3-centos-8-stream&job_name=devstack-platform-centos-8-stream&job_name=devstack-platform-centos-9-stream&skip=0
15:38:06 <kopecmartin> fedora
15:38:08 <kopecmartin> #link https://zuul.openstack.org/builds?job_name=devstack-platform-fedora-latest&skip=0
15:38:17 <kopecmartin> oh, that's really bad
15:38:28 <kopecmartin> openEuler
15:38:29 <kopecmartin> #link https://zuul.openstack.org/builds?job_name=devstack-platform-openEuler-20.03-SP2+&skip=0
15:38:42 <kopecmartin> debian
15:38:44 <kopecmartin> #link https://zuul.openstack.org/builds?job_name=devstack-platform-debian-bullseye&skip=0
15:39:41 <kopecmartin> #topic Sub Teams highlights
15:39:47 <kopecmartin> Changes with Review-Priority == +1
15:39:56 <kopecmartin> #link https://review.opendev.org/q/label:Review-Priority%253D%252B1+status:open+(project:openstack/tempest+OR+project:openstack/patrole+OR+project:openstack/devstack+OR+project:openstack/grenade)
15:40:08 <kopecmartin> no reviews
15:40:09 <kopecmartin> #topic Open Discussion
15:40:16 <kopecmartin> anything for the open discussion?
15:43:05 <tosky> very quickly, I have a question
15:43:20 <tosky> as you may know, there is an ongoing effort to switch the ceph deployment to cephadm (in devstack-plugin-cephadm)
15:43:33 <tosky> while helping with testing some patches, we realized we don't test RGW in place of swift at all
15:43:42 <tosky> so I've tried to enable it, and I hit an issue
15:43:44 <tosky> the problem we are facing now is that when you disable swift, tempest verify-config complains because it finds the swift endpoint but swift is false in the service_enabled section.
15:43:48 <tosky> So I'm not sure what to do: change the definitions services so that "s-*" means "the swift services" and "swift" generically means "any swift interface", and the latter can be set to true while the former to false?
15:43:55 <tosky> or something else
15:43:59 <tosky> let me dig the error
15:45:14 <tosky> https://57f0df93d98f0ba67cf4-594c8dbcc2892c36fc68c95ad460eff0.ssl.cf5.rackcdn.com/837842/1/check/devstack-plugin-ceph-tempest-py3/5444202/job-output.txt
15:45:22 <tosky> from https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/837842
15:45:53 <tosky> (the error is not related to cephadm, that patch doesn't use it, it's an issue we have had for a long while)
15:47:02 <opendevreview> Merged openstack/devstack stable/yoga: Write safe.directory items to system git config  https://review.opendev.org/c/openstack/devstack/+/838352
15:47:04 <opendevreview> Merged openstack/devstack stable/xena: Write safe.directory items to system git config  https://review.opendev.org/c/openstack/devstack/+/838423
15:47:07 <opendevreview> Merged openstack/devstack stable/wallaby: Write safe.directory items to system git config  https://review.opendev.org/c/openstack/devstack/+/838426
15:47:10 <opendevreview> Merged openstack/devstack stable/victoria: Write safe.directory items to system git config  https://review.opendev.org/c/openstack/devstack/+/838428
15:48:04 * kopecmartin checking the links
15:51:20 <tosky> what I'd like to avoid is to introduce another variable for devstack/tempest which really means "a generic swift interface is available", but maybe it's the only way
15:52:53 <kopecmartin> it sounds like a bug in verify_tempest_config .. i'm trying to figure out whether it's possible to make it smarter
15:53:17 <kopecmartin> .. without a new option
15:54:13 <kopecmartin> i forgot how the tool works, i need to check whether it's possible to deduce that rgw is enabled and skip the recommendation to enable swift
15:54:17 <gmann> but is swift enabled in service_available option?
15:54:30 <kopecmartin> no, it's not
15:54:38 <kopecmartin> https://57f0df93d98f0ba67cf4-594c8dbcc2892c36fc68c95ad460eff0.ssl.cf5.rackcdn.com/837842/1/check/devstack-plugin-ceph-tempest-py3/5444202/controller/logs/tempest_conf.txt
15:56:16 <gmann> and how we disabled the swift, it should be disabled in devstack in that case
15:57:18 <opendevreview> Dan Smith proposed openstack/devstack master: Gather performance data after tempest  https://review.opendev.org/c/openstack/devstack/+/837139
15:58:35 <kopecmartin> gmann: https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/837842/1/.zuul.yaml
15:58:39 <gmann> because if you are disabling the swift in tempest but enabled it in devstack then it will complain so you need to disable in devstack side also so that devstack would not cerate the swift endpoint
16:00:23 <sean-k-mooney> gmann: that seam like a sub optimial solution
16:00:35 <gmann> tosky: kopecmartin it check both 'swift' or 's-*'   https://github.com/openstack/devstack/blob/676dcaf94487665882be048cfe1f3206d6807e0f/functions-common#L2083
16:00:38 <sean-k-mooney> gmann: we might need swift for some feature but not want to actully test it with devstack
16:01:01 <sean-k-mooney> gmann: for example we talkabout using swift to store tpm data for shleve at one point
16:01:45 <sean-k-mooney> if we wanted to test that in the nova gate we woudl need to enable swift but we might not want to run the swift tempest tests
16:01:46 <gmann> sean-k-mooney: yeah so in that case you just do not run the swift test.
16:02:04 <sean-k-mooney> right so we shoudl be able to have swift enabeld in devstack adn disable it in tempest
16:02:21 <gmann> verify-config tool is mainly to tell if anything mismatch in tempest config from what your cloud has
16:02:42 <sean-k-mooney> sure but coudl it not be a warning instead of an error
16:02:44 <gmann> sean-k-mooney: not disable in tempest, just do not run the swift test via regex
16:03:18 <sean-k-mooney> gmann: right im sayting using the regex is not good form a ux point of view
16:03:35 <sean-k-mooney> it works but if i disable it in the tempest config i should not have too use the regex
16:03:53 <sean-k-mooney> i understand that conflict with verify-config's usecase
16:03:58 <gmann> yeah but making verify-config less restrictive loose overall goal of this tool
16:04:01 <gmann> yeah
16:04:27 <gmann> for current error, not sure why s-* are not working to disable service in devstack even we have this https://github.com/openstack/devstack/blob/676dcaf94487665882be048cfe1f3206d6807e0f/functions-common#L2083
16:04:35 <sean-k-mooney> ignoring the tool if i disable swift in tempest is that enough to disable the tests
16:05:13 <sean-k-mooney> gmann: is this related to ceph? is it the rados gateway?
16:05:19 <tosky> gmann: I think disabling s-* is working - the point of the job is that RGW is enabled instead
16:05:24 <tosky> gmann: so the endpoint is populated
16:05:30 <sean-k-mooney> right
16:05:42 <sean-k-mooney> so tempest shoudl treat RGW as swirft
16:05:53 <tosky> the problem is that tempest tries to set service_enabled.swift based on s-* and swift value and it found it false and it sets it to false
16:05:57 <sean-k-mooney> RGW is ment to supprot swifts api
16:06:27 <tosky> and then verify-config is called, before everything else could forcibly set service_enable.swift to true, and fails
16:06:56 <sean-k-mooney> ya so verify-cofnig either need to check the ceph plugin flag
16:07:07 <tosky> basically everything would work if "swift" meant just "swift-like interface" and then the real availability of the services was handled by s-* variables
16:07:09 <gmann> I see, I think we talked about RGW support in tempest/devstack testing  but I need to check if something we talked in bug or so
16:07:12 <sean-k-mooney> or it need to look at the keytone endpoints in perfernce to local.conf
16:07:25 <tosky> sean-k-mooney: but that would be a reverse dependency to devstack-plugin-ceph
16:07:43 <sean-k-mooney> not entirely
16:07:59 <sean-k-mooney> it does not depend on it it can simple check if the value is set or not
16:08:06 <tosky> unless you have a variable which says "expect to find a swift interface, regardless of the implementation"
16:08:08 <gmann> so that can happen if any other deployment is adding things which we told devstack/tempest not to and things fail
16:08:22 <tosky> but then we could use 'swift' in SERVICE_ENABLED for that, as I said before
16:08:43 <gmann> tosky: yes and do not run test if you do not need.
16:09:11 <gmann> tosky: I mean enable it in tempest and do not run test
16:09:26 <sean-k-mooney> tosky: https://github.com/openstack/devstack-plugin-ceph/blob/master/devstack/lib/ceph#L115=
16:09:33 <sean-k-mooney> i was hoping there woudl be something like that
16:09:40 <sean-k-mooney> which enabled or disable the RWG
16:09:45 <sean-k-mooney> that it could check
16:09:48 <tosky> sean-k-mooney: yes, but that's not the problem
16:09:58 <tosky> tempest shouldn't need to check that value
16:09:59 <sean-k-mooney> oh there is ENABLE_CEPH_RGW
16:10:18 <sean-k-mooney> right it coudl use the keystone catalog
16:10:32 <sean-k-mooney> i guess it is and comparing it to its enabeld services
16:10:38 <sean-k-mooney> based on its config
16:11:04 <sean-k-mooney> so you could add a test_cofig section to the devstack_plugin_ceph
16:11:10 <tosky> it's too late
16:11:12 <sean-k-mooney> which woudl enable swift
16:11:20 <sean-k-mooney> ok well post_config
16:11:29 <sean-k-mooney> it happens before any service is started
16:11:45 <sean-k-mooney> if you use post_cofnig jobs can still override it with test config
16:11:50 <tosky> too late :) the section in lib/tempest has no breaks between setting swift false and ruenning verify-config
16:12:46 <sean-k-mooney> oh ok
16:12:51 <sean-k-mooney> well that seam simple to fix
16:13:03 <tosky> hence the suggestion of redefining the meaning of 'swift'
16:13:03 <sean-k-mooney> just move verify-config ot after test_config phase
16:13:08 <tosky> or that
16:13:22 <tosky> I was trying to find a minimally-invasive change
16:13:31 <tosky> but it probably doesn't exist :)
16:13:34 <tosky> so I'm here
16:14:28 <sean-k-mooney> this is where it is failing yes https://github.com/openstack/devstack/blob/676dcaf94487665882be048cfe1f3206d6807e0f/lib/tempest#L661=
16:16:04 <sean-k-mooney> so ya i think the simple solution would be to move that to its own fucntion and run it either after post_config (and before test_config) preserveing the current validation scope but allowing other plugins to modify the config
16:16:19 <sean-k-mooney> or put it after test_config having it veriry the final config
16:17:11 <tosky> need to disappear for a while, but open to any solution
16:17:11 <sean-k-mooney> right now its happeing in install https://github.com/openstack/devstack/blob/676dcaf94487665882be048cfe1f3206d6807e0f/doc/source/plugins.rst#pluginsh-contract=
16:17:50 <sean-k-mooney> movign it to extra or test-config woudl be what i woudl suggest
16:18:04 <sean-k-mooney> proably extra
16:19:40 <kopecmartin> thank you for the discussion, i need to end the meeting now :)
16:19:48 <kopecmartin> we may continue next time
16:19:56 <kopecmartin> #endmeeting