| opendevreview | Ghanshyam proposed openstack/devstack stable/yoga: Write safe.directory items to system git config https://review.opendev.org/c/openstack/devstack/+/838352 | 01:43 |
|---|---|---|
| opendevreview | Ghanshyam proposed openstack/devstack stable/xena: Write safe.directory items to system git config https://review.opendev.org/c/openstack/devstack/+/838423 | 02:04 |
| opendevreview | Ghanshyam proposed openstack/devstack stable/wallaby: Write safe.directory items to system git config https://review.opendev.org/c/openstack/devstack/+/838426 | 02:31 |
| opendevreview | Ghanshyam proposed openstack/devstack stable/victoria: Write safe.directory items to system git config https://review.opendev.org/c/openstack/devstack/+/838428 | 02:47 |
| gmann | dansmith: clarkb backported '--system' one till stable/victoria https://review.opendev.org/q/I941ef5ea90970a0901236afe81c551aaf24ac1d8 | 02:48 |
| opendevreview | Wenping Song proposed openstack/tempest master: Fix: change values to items as keys and values are all used https://review.opendev.org/c/openstack/tempest/+/838430 | 03:21 |
| *** pojadhav is now known as pojadhav|ruck | 03:59 | |
| opendevreview | Merged openstack/devstack master: Add OpenStack two nodes nodeset for Centos 9 https://review.opendev.org/c/openstack/devstack/+/836841 | 05:48 |
| opendevreview | Rajat Dhasmana proposed openstack/tempest master: Add test to rebuild volume backed instance https://review.opendev.org/c/openstack/tempest/+/831018 | 05:57 |
| *** jpena|off is now known as jpena | 07:01 | |
| opendevreview | Francesco Pantano proposed openstack/devstack-plugin-ceph master: Deploy with cephadm https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/826484 | 07:01 |
| *** pojadhav|ruck is now known as pojadhav|lunch | 07:35 | |
| frickler | clarkb: I think https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_a6b/558930/22/check/tempest-full-py3/a6bd686/controller/logs/screen-keystone.txt is the real issue | 07:35 |
| *** arxcruz|out is now known as arxcruz | 08:11 | |
| *** pojadhav|lunch is now known as pojadhav|ruck | 08:30 | |
| opendevreview | Eduardo Olivares proposed openstack/tempest master: Validate network downtime during live migration https://review.opendev.org/c/openstack/tempest/+/828686 | 09:22 |
| opendevreview | Eduardo Olivares proposed openstack/tempest master: Validate network downtime during live migration https://review.opendev.org/c/openstack/tempest/+/828686 | 09:43 |
| *** icey_ is now known as icey | 09:53 | |
| sean-k-mooney | dansmith: you could perhaps try the ansibel devstack repo i have been playing with too. i have not fully figured out how to emulate the gate jobs with it yet but i was hoping to use it to be able to work on gate jobs eventually | 10:33 |
| sean-k-mooney | clarkb: ill test that out and report back. i need to start writing some docs for my molecuel roles but one of the things that was only list was to document how to test with un merged patches. | 10:43 |
| sean-k-mooney | ah you are defaulting GLOBAL_VENV to true cool so i dont need to set anyting in the local.conf | 10:43 |
| opendevreview | Andre Aranha proposed openstack/tempest master: Add multinode FIPS job to the experimental queue https://review.opendev.org/c/openstack/tempest/+/826580 | 13:12 |
| *** spotz__ is now known as spotz | 13:12 | |
| dansmith | sean-k-mooney: ah yeah good idea | 13:26 |
| dansmith | gmann: okay I was going to do that after it landed.. sorry clarkb | 13:26 |
| gmann | dansmith: also, I think I will do for ussuri also as I need to pin the tempest there so that we can declare the 'end of support of ussuri' in tempest - https://review.opendev.org/q/topic:ussuri-last | 13:28 |
| dansmith | gmann: okay | 13:35 |
| *** spotz is now known as spotz__ | 13:54 | |
| *** spotz__ is now known as spotz | 13:54 | |
| bkopilov | Hi experts ,Need help with review : | 14:06 |
| bkopilov | https://review.opendev.org/c/openstack/tempest/+/837603 | 14:07 |
| dansmith | gmann: do you want me to +W those backports even though I wrote the original? | 14:07 |
| gmann | dansmith: I think you can. | 14:07 |
| dansmith | gmann: okay, good to just +W them all, or do you want to wait for wallaby until yoga merges, etc? | 14:07 |
| dansmith | er, xena (/me not so good at alphabet) | 14:08 |
| gmann | dansmith: you can do for all, that is usually we do in devstack, if anything failing/merge conflict or so we can fix it later | 14:08 |
| dansmith | gmann: cool, here goes | 14:08 |
| gmann | dansmith: thanks | 14:08 |
| elodilles | gmann: nova and neutron grenade were working ~ a week ago for EM branches, so at least those were used o:) i'll try to look after the devstack patches that need to be backported | 14:31 |
| *** pojadhav|ruck is now known as pojadhav|out | 14:49 | |
| clarkb | frickler: thanks I agree. If I have time I'll try to look closer | 14:49 |
| kopecmartin | #startmeeting qa | 15:00 |
| opendevmeet | Meeting started Tue Apr 19 15:00:15 2022 UTC and is due to finish in 60 minutes. The chair is kopecmartin. Information about MeetBot at http://wiki.debian.org/MeetBot. | 15:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 15:00 |
| opendevmeet | The meeting name has been set to 'qa' | 15:00 |
| kopecmartin | #link https://wiki.openstack.org/wiki/Meetings/QATeamMeeting#Weekly_QA_Team_meeting | 15:00 |
| kopecmartin | agenda ^^ | 15:00 |
| kopecmartin | #topic Announcement and Action Item (Optional) | 15:02 |
| kopecmartin | none from my side | 15:02 |
| kopecmartin | #topic Zed Priority Items progress | 15:02 |
| gmann | o/ | 15:02 |
| kopecmartin | #link https://etherpad.opendev.org/p/qa-zed-priority | 15:02 |
| kopecmartin | hi gmann | 15:02 |
| kopecmartin | topic: Unstable tests in Tempest monitoring | 15:03 |
| kopecmartin | I found out that we have 3 tests decorated as unstable ones | 15:03 |
| gmann | ok | 15:04 |
| kopecmartin | test_create_object_with_transfer_encoding is associated with the following bug | 15:05 |
| gmann | kopecmartin: can you please remove the +W , i forgot to keep it in -W as I am still testing it https://review.opendev.org/c/openstack/tempest/+/837777 | 15:05 |
| kopecmartin | gmann: sure, sorry | 15:05 |
| gmann | thanks | 15:06 |
| kopecmartin | so the above test is associated with the following bug: | 15:06 |
| kopecmartin | #link https://bugs.launchpad.net/tempest/+bug/1905432 | 15:06 |
| kopecmartin | it's in progress, we can leave the test as is right now | 15:06 |
| kopecmartin | although the other test | 15:06 |
| kopecmartin | test_container_synchronization | 15:06 |
| kopecmartin | should be fixed already | 15:06 |
| kopecmartin | #link https://bugs.launchpad.net/tempest/+bug/1317133 | 15:06 |
| kopecmartin | do we want to remove the unstable decorator there and watch what happens? | 15:07 |
| gmann | I think we can remove and watch, its start of cycle so good to try | 15:07 |
| kopecmartin | ack | 15:08 |
| kopecmartin | the last one test_server_connectivity_cold_migration_revert | 15:08 |
| kopecmartin | has an expired test | 15:08 |
| kopecmartin | *bug | 15:08 |
| kopecmartin | #link https://bugs.launchpad.net/neutron/+bug/1836595 | 15:09 |
| kopecmartin | should we skip the test permanently? | 15:09 |
| gmann | humm skip test permanently is not good | 15:10 |
| kopecmartin | yeah, ok , maybe let me dig in a bit and find out where/what jobs execute the test | 15:10 |
| kopecmartin | let's see how it behaves | 15:10 |
| gmann | yeah, we can re-open it for neutron | 15:12 |
| kopecmartin | sounds good | 15:12 |
| kopecmartin | any other updates? | 15:12 |
| kopecmartin | i'm gonna push a review soon to switch to ecdsa keys by default in tempest | 15:13 |
| kopecmartin | #topic OpenStack Events Updates and Planning | 15:15 |
| kopecmartin | nothing specific here | 15:15 |
| kopecmartin | #topic Gate Status Checks | 15:15 |
| kopecmartin | https://review.opendev.org/q/label:Review-Priority%253D%252B2+status:open+(project:openstack/tempest+OR+project:openstack/patrole+OR+project:openstack/devstack+OR+project:openstack/grenade) | 15:15 |
| kopecmartin | none reviews there | 15:16 |
| soniya29 | i have couple of patches failing in barbican-tempest-plugin with same error | 15:16 |
| soniya29 | https://aad416e65f3b6b410df4-779f7e98ba8ab2ef93c3580c1febb3f0.ssl.cf2.rackcdn.com/834000/3/check/barbican-tempest-plugin-simple-crypto-victoria/07b40f8/job-output.txt | 15:16 |
| kopecmartin | yeah, they fail due to this | 15:16 |
| kopecmartin | #link https://bugs.launchpad.net/devstack/+bug/1968798 | 15:16 |
| kopecmartin | even though some workaround patches have been merged, it still fails somewhere | 15:17 |
| kopecmartin | e.g. barbican-tempest-plugin or python-tempestconf | 15:17 |
| kopecmartin | i don't know why | 15:17 |
| gmann | soniya29: kopecmartin yeah it should be fixed now. or you saw the error after fix were merged ? | 15:18 |
| soniya29 | kopecmartin, my patches are blocked because of it :) | 15:18 |
| soniya29 | gmann, i am seeing these errors today | 15:18 |
| kopecmartin | yep, it failed today, see my comment | 15:18 |
| kopecmartin | https://bugs.launchpad.net/devstack/+bug/1968798/comments/5 | 15:18 |
| gmann | soniya29: this is victoris job which should be fixed too | 15:18 |
| soniya29 | gmann, it is failing on xena, ussuri as well | 15:19 |
| soniya29 | https://review.opendev.org/c/openstack/barbican-tempest-plugin/+/834000 | 15:19 |
| gmann | ussuri it will fail as we fixed only till stable/victoria | 15:19 |
| soniya29 | https://review.opendev.org/c/openstack/barbican-tempest-plugin/+/833801 | 15:20 |
| kopecmartin | i wonder why this fails as it uses master :/ | 15:21 |
| kopecmartin | #link https://zuul.opendev.org/t/openstack/build/025e36991b6e472ba0c204fede267104 | 15:21 |
| clarkb | I haven't opened the links but any use of git as a different user than ownership of the repo can cause this | 15:23 |
| opendevreview | Eduardo Olivares proposed openstack/tempest master: Validate network downtime during live migration - part II https://review.opendev.org/c/openstack/tempest/+/838518 | 15:23 |
| clarkb | the patches to devstack have flagged the repos that devsatck clones as safe to address this but if a plugin or job manually clones and doesn't do this they can still break | 15:24 |
| clarkb | or if they interact with the zuul cloned repos directly as not the zuul user | 15:24 |
| clarkb | etc etc. You'll need to dig into why they are still broken particualrly on master where the workaroudns should be sufficient | 15:24 |
| kopecmartin | clarkb: does this count as interacting with it directly as not zuul user? | 15:26 |
| kopecmartin | #link https://opendev.org/openinfra/python-tempestconf/src/branch/master/roles/install-plugins/tasks/main.yaml#L5 | 15:26 |
| clarkb | kopecmartin: no that should run out of the tempest clone done by devsatck which gets the flag set. The next task may be though as that pip installs requirements as root out of the zuul clone | 15:27 |
| clarkb | er wait that doesn't pip install requirements it just gets upper constraints | 15:27 |
| clarkb | it pip installs some list of plugins but I'm not sure where those plugins are located | 15:27 |
| kopecmartin | it fails on the Prepare tempest venv task anyway | 15:28 |
| kopecmartin | before installign plugins | 15:28 |
| kopecmartin | ok, i'll try to dig more into that | 15:28 |
| kopecmartin | btw, plugins_paths is just a var which references the plugins locations where they were cloned by devstack | 15:29 |
| kopecmartin | so the workaround should work on that too | 15:29 |
| clarkb | specifically the issue is running git commands in a git repo as a user other than the user that owns the files in the git repo | 15:29 |
| clarkb | reading the files directly or navigating the repo with cd and that sort of interaction is fine. It is git itself policing this when you run git commands against a repo | 15:30 |
| clarkb | installing things causes this to happen because pbr runs git commands to determine version info | 15:30 |
| kopecmartin | i see, thanks, i'll check the tasks executed in more detail | 15:33 |
| kopecmartin | #topic Periodic jobs Status Checks | 15:33 |
| kopecmartin | stable: | 15:34 |
| kopecmartin | #link https://zuul.openstack.org/builds?job_name=tempest-full-yoga&job_name=tempest-full-xena&job_name=tempest-full-wallaby-py3&job_name=tempest-full-victoria-py3&job_name=tempest-full-ussuri-py3&pipeline=periodic-stable | 15:34 |
| kopecmartin | master: | 15:34 |
| kopecmartin | #link https://zuul.openstack.org/builds?project=openstack%2Ftempest&project=openstack%2Fdevstack&pipeline=periodic | 15:34 |
| kopecmartin | tempest-full-ussuri-py3 failed couple days ago with the same pbr error | 15:35 |
| kopecmartin | #link tempest-full-ussuri-py3 | 15:35 |
| kopecmartin | #link https://zuul.openstack.org/build/e5fe6f0121924271b820c1ac32882640 | 15:35 |
| kopecmartin | which makes sense as the fix/workaround wasn't backported to ussuri | 15:35 |
| kopecmartin | #link https://review.opendev.org/c/openstack/devstack/+/837749 | 15:36 |
| kopecmartin | #topic Distros check | 15:37 |
| kopecmartin | centos 8/9 stream | 15:37 |
| kopecmartin | #link https://zuul.openstack.org/builds?job_name=tempest-full-centos-9-stream&job_name=tempest-full-py3-centos-8-stream&job_name=devstack-platform-centos-8-stream&job_name=devstack-platform-centos-9-stream&skip=0 | 15:37 |
| kopecmartin | fedora | 15:38 |
| kopecmartin | #link https://zuul.openstack.org/builds?job_name=devstack-platform-fedora-latest&skip=0 | 15:38 |
| kopecmartin | oh, that's really bad | 15:38 |
| kopecmartin | openEuler | 15:38 |
| kopecmartin | #link https://zuul.openstack.org/builds?job_name=devstack-platform-openEuler-20.03-SP2+&skip=0 | 15:38 |
| kopecmartin | debian | 15:38 |
| kopecmartin | #link https://zuul.openstack.org/builds?job_name=devstack-platform-debian-bullseye&skip=0 | 15:38 |
| kopecmartin | #topic Sub Teams highlights | 15:39 |
| kopecmartin | Changes with Review-Priority == +1 | 15:39 |
| kopecmartin | #link https://review.opendev.org/q/label:Review-Priority%253D%252B1+status:open+(project:openstack/tempest+OR+project:openstack/patrole+OR+project:openstack/devstack+OR+project:openstack/grenade) | 15:39 |
| kopecmartin | no reviews | 15:40 |
| kopecmartin | #topic Open Discussion | 15:40 |
| kopecmartin | anything for the open discussion? | 15:40 |
| tosky | very quickly, I have a question | 15:43 |
| tosky | as you may know, there is an ongoing effort to switch the ceph deployment to cephadm (in devstack-plugin-cephadm) | 15:43 |
| tosky | while helping with testing some patches, we realized we don't test RGW in place of swift at all | 15:43 |
| tosky | so I've tried to enable it, and I hit an issue | 15:43 |
| tosky | the problem we are facing now is that when you disable swift, tempest verify-config complains because it finds the swift endpoint but swift is false in the service_enabled section. | 15:43 |
| tosky | So I'm not sure what to do: change the definitions services so that "s-*" means "the swift services" and "swift" generically means "any swift interface", and the latter can be set to true while the former to false? | 15:43 |
| tosky | or something else | 15:43 |
| tosky | let me dig the error | 15:43 |
| tosky | https://57f0df93d98f0ba67cf4-594c8dbcc2892c36fc68c95ad460eff0.ssl.cf5.rackcdn.com/837842/1/check/devstack-plugin-ceph-tempest-py3/5444202/job-output.txt | 15:45 |
| tosky | from https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/837842 | 15:45 |
| tosky | (the error is not related to cephadm, that patch doesn't use it, it's an issue we have had for a long while) | 15:45 |
| opendevreview | Merged openstack/devstack stable/yoga: Write safe.directory items to system git config https://review.opendev.org/c/openstack/devstack/+/838352 | 15:47 |
| opendevreview | Merged openstack/devstack stable/xena: Write safe.directory items to system git config https://review.opendev.org/c/openstack/devstack/+/838423 | 15:47 |
| opendevreview | Merged openstack/devstack stable/wallaby: Write safe.directory items to system git config https://review.opendev.org/c/openstack/devstack/+/838426 | 15:47 |
| opendevreview | Merged openstack/devstack stable/victoria: Write safe.directory items to system git config https://review.opendev.org/c/openstack/devstack/+/838428 | 15:47 |
| * kopecmartin checking the links | 15:48 | |
| tosky | what I'd like to avoid is to introduce another variable for devstack/tempest which really means "a generic swift interface is available", but maybe it's the only way | 15:51 |
| kopecmartin | it sounds like a bug in verify_tempest_config .. i'm trying to figure out whether it's possible to make it smarter | 15:52 |
| kopecmartin | .. without a new option | 15:53 |
| kopecmartin | i forgot how the tool works, i need to check whether it's possible to deduce that rgw is enabled and skip the recommendation to enable swift | 15:54 |
| gmann | but is swift enabled in service_available option? | 15:54 |
| kopecmartin | no, it's not | 15:54 |
| kopecmartin | https://57f0df93d98f0ba67cf4-594c8dbcc2892c36fc68c95ad460eff0.ssl.cf5.rackcdn.com/837842/1/check/devstack-plugin-ceph-tempest-py3/5444202/controller/logs/tempest_conf.txt | 15:54 |
| gmann | and how we disabled the swift, it should be disabled in devstack in that case | 15:56 |
| opendevreview | Dan Smith proposed openstack/devstack master: Gather performance data after tempest https://review.opendev.org/c/openstack/devstack/+/837139 | 15:57 |
| kopecmartin | gmann: https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/837842/1/.zuul.yaml | 15:58 |
| gmann | because if you are disabling the swift in tempest but enabled it in devstack then it will complain so you need to disable in devstack side also so that devstack would not cerate the swift endpoint | 15:58 |
| sean-k-mooney | gmann: that seam like a sub optimial solution | 16:00 |
| gmann | tosky: kopecmartin it check both 'swift' or 's-*' https://github.com/openstack/devstack/blob/676dcaf94487665882be048cfe1f3206d6807e0f/functions-common#L2083 | 16:00 |
| sean-k-mooney | gmann: we might need swift for some feature but not want to actully test it with devstack | 16:00 |
| sean-k-mooney | gmann: for example we talkabout using swift to store tpm data for shleve at one point | 16:01 |
| sean-k-mooney | if we wanted to test that in the nova gate we woudl need to enable swift but we might not want to run the swift tempest tests | 16:01 |
| gmann | sean-k-mooney: yeah so in that case you just do not run the swift test. | 16:01 |
| sean-k-mooney | right so we shoudl be able to have swift enabeld in devstack adn disable it in tempest | 16:02 |
| gmann | verify-config tool is mainly to tell if anything mismatch in tempest config from what your cloud has | 16:02 |
| sean-k-mooney | sure but coudl it not be a warning instead of an error | 16:02 |
| gmann | sean-k-mooney: not disable in tempest, just do not run the swift test via regex | 16:02 |
| sean-k-mooney | gmann: right im sayting using the regex is not good form a ux point of view | 16:03 |
| sean-k-mooney | it works but if i disable it in the tempest config i should not have too use the regex | 16:03 |
| sean-k-mooney | i understand that conflict with verify-config's usecase | 16:03 |
| gmann | yeah but making verify-config less restrictive loose overall goal of this tool | 16:03 |
| gmann | yeah | 16:04 |
| gmann | for current error, not sure why s-* are not working to disable service in devstack even we have this https://github.com/openstack/devstack/blob/676dcaf94487665882be048cfe1f3206d6807e0f/functions-common#L2083 | 16:04 |
| sean-k-mooney | ignoring the tool if i disable swift in tempest is that enough to disable the tests | 16:04 |
| sean-k-mooney | gmann: is this related to ceph? is it the rados gateway? | 16:05 |
| tosky | gmann: I think disabling s-* is working - the point of the job is that RGW is enabled instead | 16:05 |
| tosky | gmann: so the endpoint is populated | 16:05 |
| sean-k-mooney | right | 16:05 |
| sean-k-mooney | so tempest shoudl treat RGW as swirft | 16:05 |
| tosky | the problem is that tempest tries to set service_enabled.swift based on s-* and swift value and it found it false and it sets it to false | 16:05 |
| sean-k-mooney | RGW is ment to supprot swifts api | 16:05 |
| tosky | and then verify-config is called, before everything else could forcibly set service_enable.swift to true, and fails | 16:06 |
| sean-k-mooney | ya so verify-cofnig either need to check the ceph plugin flag | 16:06 |
| tosky | basically everything would work if "swift" meant just "swift-like interface" and then the real availability of the services was handled by s-* variables | 16:07 |
| gmann | I see, I think we talked about RGW support in tempest/devstack testing but I need to check if something we talked in bug or so | 16:07 |
| sean-k-mooney | or it need to look at the keytone endpoints in perfernce to local.conf | 16:07 |
| tosky | sean-k-mooney: but that would be a reverse dependency to devstack-plugin-ceph | 16:07 |
| sean-k-mooney | not entirely | 16:07 |
| sean-k-mooney | it does not depend on it it can simple check if the value is set or not | 16:07 |
| tosky | unless you have a variable which says "expect to find a swift interface, regardless of the implementation" | 16:08 |
| gmann | so that can happen if any other deployment is adding things which we told devstack/tempest not to and things fail | 16:08 |
| *** jpena is now known as jpena|off | 16:08 | |
| tosky | but then we could use 'swift' in SERVICE_ENABLED for that, as I said before | 16:08 |
| gmann | tosky: yes and do not run test if you do not need. | 16:08 |
| gmann | tosky: I mean enable it in tempest and do not run test | 16:09 |
| sean-k-mooney | tosky: https://github.com/openstack/devstack-plugin-ceph/blob/master/devstack/lib/ceph#L115= | 16:09 |
| sean-k-mooney | i was hoping there woudl be something like that | 16:09 |
| sean-k-mooney | which enabled or disable the RWG | 16:09 |
| sean-k-mooney | that it could check | 16:09 |
| tosky | sean-k-mooney: yes, but that's not the problem | 16:09 |
| tosky | tempest shouldn't need to check that value | 16:09 |
| sean-k-mooney | oh there is ENABLE_CEPH_RGW | 16:09 |
| sean-k-mooney | right it coudl use the keystone catalog | 16:10 |
| sean-k-mooney | i guess it is and comparing it to its enabeld services | 16:10 |
| sean-k-mooney | based on its config | 16:10 |
| sean-k-mooney | so you could add a test_cofig section to the devstack_plugin_ceph | 16:11 |
| tosky | it's too late | 16:11 |
| sean-k-mooney | which woudl enable swift | 16:11 |
| sean-k-mooney | ok well post_config | 16:11 |
| sean-k-mooney | it happens before any service is started | 16:11 |
| sean-k-mooney | if you use post_cofnig jobs can still override it with test config | 16:11 |
| tosky | too late :) the section in lib/tempest has no breaks between setting swift false and ruenning verify-config | 16:11 |
| sean-k-mooney | oh ok | 16:12 |
| sean-k-mooney | well that seam simple to fix | 16:12 |
| tosky | hence the suggestion of redefining the meaning of 'swift' | 16:13 |
| sean-k-mooney | just move verify-config ot after test_config phase | 16:13 |
| tosky | or that | 16:13 |
| tosky | I was trying to find a minimally-invasive change | 16:13 |
| tosky | but it probably doesn't exist :) | 16:13 |
| tosky | so I'm here | 16:13 |
| sean-k-mooney | this is where it is failing yes https://github.com/openstack/devstack/blob/676dcaf94487665882be048cfe1f3206d6807e0f/lib/tempest#L661= | 16:14 |
| sean-k-mooney | so ya i think the simple solution would be to move that to its own fucntion and run it either after post_config (and before test_config) preserveing the current validation scope but allowing other plugins to modify the config | 16:16 |
| sean-k-mooney | or put it after test_config having it veriry the final config | 16:16 |
| tosky | need to disappear for a while, but open to any solution | 16:17 |
| sean-k-mooney | right now its happeing in install https://github.com/openstack/devstack/blob/676dcaf94487665882be048cfe1f3206d6807e0f/doc/source/plugins.rst#pluginsh-contract= | 16:17 |
| sean-k-mooney | movign it to extra or test-config woudl be what i woudl suggest | 16:17 |
| sean-k-mooney | proably extra | 16:18 |
| kopecmartin | thank you for the discussion, i need to end the meeting now :) | 16:19 |
| kopecmartin | we may continue next time | 16:19 |
| kopecmartin | #endmeeting | 16:19 |
| opendevmeet | Meeting ended Tue Apr 19 16:19:56 2022 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 16:19 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/qa/2022/qa.2022-04-19-15.00.html | 16:19 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/qa/2022/qa.2022-04-19-15.00.txt | 16:19 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/qa/2022/qa.2022-04-19-15.00.log.html | 16:19 |
| sean-k-mooney | oh sorry didnt know this was still the meeting | 16:20 |
| sean-k-mooney | kopecmartin: please tell me to come back later in the future | 16:20 |
| kopecmartin | sean-k-mooney: it was the open discussion part, you used exactly as it should be :) | 16:20 |
| opendevreview | Brian Rosmaita proposed openstack/devstack stable/ussuri: Mark our source trees as safe for git to use as other users https://review.opendev.org/c/openstack/devstack/+/837749 | 16:37 |
| clarkb | gmann: why was ^ restored when it cannot work? | 16:42 |
| clarkb | I left notes about that in my abandonment message | 16:42 |
| opendevreview | Ghanshyam proposed openstack/devstack stable/ussuri: Mark our source trees as safe for git to use as other users https://review.opendev.org/c/openstack/devstack/+/837749 | 16:45 |
| opendevreview | Brian Rosmaita proposed openstack/devstack stable/train: Mark our source trees as safe for git to use as other users https://review.opendev.org/c/openstack/devstack/+/838556 | 16:45 |
| clarkb | rosmaita isn't here but the patches being pushed do not work | 16:46 |
| clarkb | looks like gmann's should though | 16:46 |
| gmann | clarkb: I am squashing both backport | 16:46 |
| clarkb | gmann: ok we should probably tell rosmaita to stop a minute because the patchsets being pushed are broken | 16:46 |
| gmann | yeah for stable/ussuri I am doing but he want for stable/train or so then it is fine | 16:47 |
| gmann | pinged him in cinder channel | 16:48 |
| dansmith | I really need to get a t-shirt made up that says "I shoulda checked with gmann first" | 16:49 |
| gmann | dansmith: heh. thanks for review. | 16:52 |
| opendevreview | sean mooney proposed openstack/devstack master: [WIP] refactor tempest install phases https://review.opendev.org/c/openstack/devstack/+/838559 | 17:02 |
| sean-k-mooney | tosky: ^ if you had that (untested) i think you could modify the enabled services in the devstack-ceph-plugin based on if RGW is enabled in the post-extras or extras phases | 17:03 |
| opendevreview | Brian Rosmaita proposed openstack/devstack stable/train: Mark our source trees as safe for git to use as other users https://review.opendev.org/c/openstack/devstack/+/838556 | 17:04 |
| tosky | sean-k-mooney: uh, thanks, let me see if I can adapt my patch to your change | 17:07 |
| sean-k-mooney | im not sure if that will break anything else but i think it woudl enable you to modify things by giving you a full phase to work with | 17:11 |
| gmann | clarkb: frickler stable/ussuri is ready https://review.opendev.org/c/openstack/devstack/+/837749 | 18:32 |
| clarkb | I guess those branches rely on us to review too? | 18:34 |
| clarkb | I've approved it | 18:35 |
| opendevreview | Dan Smith proposed openstack/devstack master: Gather performance data after tempest https://review.opendev.org/c/openstack/devstack/+/837139 | 18:37 |
| sean-k-mooney | clarkb: so finally got around to runnign your global vnev change and it looks like it failed because the python path is node set in the systemd unit files genergated leading too ModuleNotFoundError: No module named 'keystone' | 18:52 |
| sean-k-mooney | clarkb: i also noticed that its using system uwsgi | 18:52 |
| sean-k-mooney | https://paste.opendev.org/show/bbN8bASPdn60ST3pXe3M/ | 18:52 |
| clarkb | it should be set in the units. https://review.opendev.org/c/openstack/devstack/+/558930/22/functions-common is what attempts that | 18:53 |
| sean-k-mooney | so am i correct in thinkinging we should be use uwsgi form the venv | 18:53 |
| sean-k-mooney | and setting the path | 18:53 |
| clarkb | yes setting the path and running all the python stuff out of the venv is the intent. Fwiw a lot of stuff has changed in devstack since this was written 4 years ago | 18:53 |
| clarkb | I've spent a fair bit of time addressing those issues one by once since I rebased | 18:54 |
| clarkb | like for some reason glance has a special way of installing rootwrap rules different to everything else | 18:54 |
| sean-k-mooney | i dont see that in the file that was generated but let me double check i am not missing something in the local.conf | 18:54 |
| sean-k-mooney | i think it was enabled by default right | 18:54 |
| clarkb | yes https://review.opendev.org/c/openstack/devstack/+/558930/22/stackrc sets them by default | 18:55 |
| clarkb | its just easier for testing to start that way | 18:55 |
| clarkb | oh I think I see it | 18:55 |
| clarkb | that is in write_user_unit_file but I bet we don't use user unit files anymore or something like that | 18:55 |
| sean-k-mooney | we use system level ones | 18:56 |
| sean-k-mooney | not user uinit files | 18:56 |
| clarkb | oh and uwsgi gets its own function | 18:56 |
| sean-k-mooney | in /etc/systemd/system/devstack\@keystone.service | 18:56 |
| clarkb | I think the issue is that uwsgi gets its own function | 18:57 |
| sean-k-mooney | ack | 18:57 |
| clarkb | give me a minute and I can work on a patch update | 18:57 |
| sean-k-mooney | so you need to set it here too right https://review.opendev.org/c/openstack/devstack/+/558930/22/functions-common#1605 | 18:57 |
| sean-k-mooney | ok ill try copy pasting the path line locally quickly | 18:58 |
| opendevreview | Clark Boylan proposed openstack/devstack master: Add option to install everything in global venvs https://review.opendev.org/c/openstack/devstack/+/558930 | 18:58 |
| clarkb | something liek that maybe | 18:58 |
| clarkb | yes exactly | 18:58 |
| sean-k-mooney | yep that is what i was thinking | 18:58 |
| sean-k-mooney | i think its still going to use system uwsgi | 18:59 |
| sean-k-mooney | which might be ok | 18:59 |
| clarkb | there isn't a system uwsgi I don't think | 19:00 |
| clarkb | so it will fail if it tries to? | 19:00 |
| sean-k-mooney | /sbin/uwsgi exists | 19:01 |
| clarkb | it does `run_process "c-api" "$(which uwsgi) --procname-prefix cinder-api --ini $CINDER_UWSGI_CONF"` the key bit being the $(which uwsgi) which should find the venv first | 19:01 |
| clarkb | and that path is set in the unit | 19:01 |
| clarkb | are we installing uwsgi from the distro? | 19:01 |
| clarkb | because pip install uwsgi won't install to /sbin | 19:01 |
| sean-k-mooney | yes | 19:01 |
| sean-k-mooney | at least we do by default | 19:02 |
| sean-k-mooney | so unless you removed that it would be using distro uwsgi | 19:02 |
| clarkb | huh why would we do that .... | 19:03 |
| clarkb | anyway that should work since /sbin should also be in the path | 19:03 |
| clarkb | and then it will load the python environment in the venv in theory | 19:03 |
| clarkb | but also why are we installing uwsgi from the distro and not pypi? | 19:03 |
| sean-k-mooney | it might get pulling in i think we install python-mod-uwsgi | 19:03 |
| sean-k-mooney | +lib/apache:install_apache_wsgi:152 install_package httpd python3-mod_wsgi | 19:05 |
| sean-k-mooney | https://github.com/openstack/devstack/blob/master/lib/apache#L90-L125= | 19:07 |
| clarkb | ya ok I think it should stillwork to use the distro install | 19:07 |
| sean-k-mooney | ya it should | 19:07 |
| clarkb | since uwsgi should load the environment by running code out of the venv | 19:07 |
| sean-k-mooney | the disto somethime has a specifc version of python compiled in | 19:07 |
| sean-k-mooney | that something that could be fixed later if it was a problem | 19:08 |
| * sean-k-mooney is annoyed. i am doing a 24 hour blood pressur monitor and the tube came off the last time it did a reading and didint do any since | 19:09 | |
| sean-k-mooney | so either it swapped to 1 per hour or its given up and ill have to do it again | 19:10 |
| sean-k-mooney | hum... i turnd the bp monitor on and off again and it seam to be working... i guess ill find out in a week or two when i get the results back hopefully it will get enought that id dont have to do this yet again | 19:20 |
| sean-k-mooney | clarkb: that got to about the same place it looks liek tehre are still some path issues | 19:27 |
| sean-k-mooney | i might play around with it a bit but i think im going to go get something to eat soon | 19:27 |
| sean-k-mooney | https://paste.opendev.org/show/b7mpOyO0DWxOmAkepJGA/ | 19:28 |
| sean-k-mooney | the path is set properly in the unit file this time | 19:29 |
| sean-k-mooney | Environment = "PATH=/bin:/opt/stack/data/.venv/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/bin:/usr/local/sbin:/usr/sbin:/sbin" | 19:29 |
| sean-k-mooney | but uwsgi might need to be configure directly with the python path | 19:29 |
| sean-k-mooney | clarkb: oh... if i start python3 in the venvs and import keystone it fails | 19:37 |
| sean-k-mooney | clarkb: so wsgi is now failing because keystone is not installed into the venv | 19:37 |
| sean-k-mooney | actully now that i rentered it ti workd | 19:39 |
| sean-k-mooney | hum | 19:40 |
| sean-k-mooney | i wonder is it because i cahnged directory | 19:40 |
| sean-k-mooney | ok i must have typoed it | 19:41 |
| sean-k-mooney | clarkb: ok got it working | 19:56 |
| sean-k-mooney | clarkb: we have to explitly pass the virtual env on the uwsgi call | 19:56 |
| sean-k-mooney | clarkb: /sbin/uwsgi --procname-prefix keystone --ini /etc/keystone/keystone-uwsgi-public.ini --home /opt/stack/data/.venv/ | 19:56 |
| sean-k-mooney | then it can find everything and work properly | 19:56 |
| clarkb | sean-k-mooney: ok let me see about adding that | 20:06 |
| sean-k-mooney | this is what i was goign to test https://paste.opendev.org/show/brzdDDy4sxzPpKrYPDVy/ | 20:07 |
| clarkb | ya that is close to what I'm doing. If you want to push that update after testing I'm fine with that or I can do it concurrently | 20:09 |
| sean-k-mooney | go with your version | 20:09 |
| opendevreview | Clark Boylan proposed openstack/devstack master: Add option to install everything in global venvs https://review.opendev.org/c/openstack/devstack/+/558930 | 20:11 |
| clarkb | moved it up a bit so that the unit management is only dealing with unit stuff and not the actual command but that is close to what you've got | 20:11 |
| sean-k-mooney | ya that makes sense it keep that function generic | 20:12 |
| sean-k-mooney | that is a really unhelpful name by the way https://uwsgi-docs.readthedocs.io/en/latest/Options.html#home | 20:13 |
| sean-k-mooney | although it look like you can also use --venv | 20:13 |
| sean-k-mooney | well home pyhome venv and virtualenv all seam to be aliass | 20:13 |
| sean-k-mooney | so we might want to use venv to be more dicirptive assumeing they are all interchangeable | 20:14 |
| clarkb | ++ if this works we can update the next patchset for the next problem to use venv | 20:15 |
| sean-k-mooney | so for system python non of those exist in the help output i did contaminate my venv by isntalling uwsgi form pypi which has all of them | 20:20 |
| sean-k-mooney | which is why i said use your one because we might need to also start using it form pypi | 20:20 |
| sean-k-mooney | i can test again with your patch tomorrow in a clean env but i think this is about to finish stacking | 20:21 |
| clarkb | oh hrm | 20:22 |
| clarkb | mine is using system uwsgi though at least by default | 20:22 |
| sean-k-mooney | ya so that just finished stacking | 20:22 |
| clarkb | I guess we can toggle thato ver if necessary | 20:22 |
| sean-k-mooney | well i think it might work but be undocumeted on the system one | 20:22 |
| sean-k-mooney | not sure | 20:22 |
| clarkb | aha | 20:22 |
| clarkb | wouldn't surprise me | 20:23 |
| sean-k-mooney | it stacked | 20:23 |
| sean-k-mooney | hehe thats funny i went to run tempst smoke and tox was not installed because i did not activate the venv :) | 20:26 |
| sean-k-mooney | the tests appear to be passing | 20:26 |
| clarkb | looks like ti still failed in upstream ci. Mybe due to that flag not being valid? | 20:27 |
| clarkb | anyway progress at least | 20:27 |
| sean-k-mooney | ya perhaps let me take a look at the log quickly | 20:27 |
| sean-k-mooney | yep | 20:27 |
| sean-k-mooney | so https://zuul.opendev.org/t/openstack/build/696090fafada4ca0ad6dedb88ff1368f/log/controller/logs/screen-keystone.txt is the error i see if the virutal env is not used | 20:29 |
| sean-k-mooney | hum we dont seam to save the systemd unit files in the job output | 20:30 |
| sean-k-mooney | well actully i guess its litrally in the job-output.txt form the devtack log | 20:31 |
| sean-k-mooney | clarkb: https://zuul.opendev.org/t/openstack/build/696090fafada4ca0ad6dedb88ff1368f/log/controller/logs/devstacklog.txt#11421 | 20:32 |
| sean-k-mooney | looks like your change did not take effect | 20:32 |
| clarkb | ya it seems to have skipped myconditional | 20:33 |
| clarkb | heh I see it | 20:33 |
| clarkb | its a dumb bug | 20:33 |
| clarkb | should I try --venv in the fix? | 20:33 |
| sean-k-mooney | it shoudl be an alisa based on the new doc | 20:33 |
| sean-k-mooney | so you could | 20:34 |
| sean-k-mooney | oh | 20:34 |
| opendevreview | Clark Boylan proposed openstack/devstack master: Add option to install everything in global venvs https://review.opendev.org/c/openstack/devstack/+/558930 | 20:34 |
| sean-k-mooney | if [[ "GLOBAL_VENV" == "True" ]] ; then | 20:34 |
| clarkb | we'll find out if it works | 20:34 |
| sean-k-mooney | no $ | 20:34 |
| clarkb | yup that was the problem | 20:34 |
| sean-k-mooney | Ran: 109 tests in 443.6734 sec. | 20:34 |
| sean-k-mooney | - Passed: 104 | 20:34 |
| sean-k-mooney | - Skipped: 4 | 20:34 |
| sean-k-mooney | - Expected Fail: 0 | 20:34 |
| sean-k-mooney | - Unexpected Success: 0 | 20:34 |
| sean-k-mooney | - Failed: 1 | 20:34 |
| sean-k-mooney | so the smoke test basiclly passed | 20:34 |
| sean-k-mooney | the failed test was for horizon | 20:35 |
| opendevreview | Luigi Toscano proposed openstack/devstack-plugin-ceph master: [CI] [DNM] Test RGW in place of swift https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/837842 | 20:35 |
| sean-k-mooney | clarkb: ya horizon seams to not work but its proably something similar | 20:36 |
| sean-k-mooney | setting the python path in apache or something like that | 20:36 |
| sean-k-mooney | yep same error 0 ModuleNotFoundError: No module named 'django' | 20:37 |
| sean-k-mooney | i bet horizon has its own function too for wsgi | 20:38 |
| clarkb | ya probably | 20:38 |
| clarkb | I'm not sure how much further I'll be able to push this today. YOu should feel free to push up a new patchset whenever if you poke at it more | 20:38 |
| sean-k-mooney | ack im going to wrap up for today i can add a followup patch if i end up working on it more tonight | 20:39 |
| sean-k-mooney | lets see how your current one does in the gate | 20:39 |
| clarkb | I'll try to update it back to --home if it is obvious that --venv didn't work at least | 20:40 |
| sean-k-mooney | so ya https://github.com/openstack/devstack/blob/master/lib/horizon#L115-L137= horizon seam to use a template | 20:41 |
| sean-k-mooney | ya so there is proably a way to set it in https://github.com/openstack/devstack/blob/master/files/apache-horizon.template that but ill have to look up the apache docs | 20:43 |
| sean-k-mooney | https://modwsgi.readthedocs.io/en/master/configuration-directives/WSGIPythonPath.html#wsgipythonpath | 20:44 |
| sean-k-mooney | that is proablty what we want although ther emight be a simpler way then directly setting the python path | 20:44 |
| clarkb | https://modwsgi.readthedocs.io/en/develop/user-guides/virtual-environments.html is less clear | 20:46 |
| clarkb | they mention the config option you found but at the very end. I can psuh a new patchset to try that though as it is easy | 20:47 |
| sean-k-mooney | https://paste.opendev.org/show/bfi6ivIr3o25mzWF3nU7/ | 20:48 |
| sean-k-mooney | its not hard to get the path either | 20:49 |
| sean-k-mooney | so sure go for it but im really going to go eat this time | 20:49 |
| sean-k-mooney | chat to you tomorrow | 20:49 |
| clarkb | ya enjoy there isn't any rush on this | 20:49 |
| clarkb | there are a lot of these template files. We may need to go through and update a number of them. Or maybe stop running services with wsgi instead of uwsgi entirely in devstack? | 20:54 |
| clarkb | anyway I have a local update staged for when we have results for --venv | 20:55 |
| opendevreview | Clark Boylan proposed openstack/devstack master: Add option to install everything in global venvs https://review.opendev.org/c/openstack/devstack/+/558930 | 21:18 |
| gmann | kopecmartin: this is ready https://review.opendev.org/c/openstack/tempest/+/836715 | 21:34 |
| opendevreview | Dan Smith proposed openstack/devstack master: Gather performance data after tempest https://review.opendev.org/c/openstack/devstack/+/837139 | 22:14 |
| opendevreview | Luigi Toscano proposed openstack/devstack-plugin-ceph master: [CI] [DNM] Test RGW in place of swift https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/837842 | 22:20 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!